1 introduction

Visit pumpkin Talk www.pkslow.com for more exciting articles!

Spring Cloud Data Flow runs well locally, so why deploy it on Kubernetes? Mainly because Kubernetes can provide more flexible microservices management; Running on a cluster is more secure, stable, and makes more rational use of physical resources.

Spring Cloud Data Flow Introduction For details, see: Spring Cloud Data Flow Initial experience, running in Local mode

Deploy Data Flow to Kubernetes

In keeping with the principle of simplicity, we are still Batch task based and do not deploy stream-related components.

2.1 Downloading the GitHub code

We will make changes based on the deployment code provided by the official. Clone the official code first:

$ git clone https://github.com/spring-cloud/spring-cloud-dataflow.git
Copy the code

Let’s switch to the latest stable version of the code version:

$ git checkout v2.5.3.RELEASE
Copy the code

2.2 Creating a Permission Account

In order to allow Data Flow Server to have the permission to run tasks, can manage resources in Kubernetes, such as new Pod, so we need to create the corresponding permission account. This part of the code is consistent with the source code, do not need to modify:

(1) server – roles. Yaml

kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: scdf-role
rules:
  - apiGroups: [""]
    resources: ["services"."pods"."replicationcontrollers"."persistentvolumeclaims"]
    verbs: ["get"."list"."watch"."create"."delete"."update"]
  - apiGroups: [""]
    resources: ["configmaps"."secrets"."pods/log"]
    verbs: ["get"."list"."watch"]
  - apiGroups: ["apps"]
    resources: ["statefulsets"."deployments"."replicasets"]
    verbs: ["get"."list"."watch"."create"."delete"."update"."patch"]
  - apiGroups: ["extensions"]
    resources: ["deployments"."replicasets"]
    verbs: ["get"."list"."watch"."create"."delete"."update"."patch"]
  - apiGroups: ["batch"]
    resources: ["cronjobs"."jobs"]
    verbs: ["create"."delete"."get"."list"."watch"."update"."patch"]
Copy the code

(2) server – rolebinding yaml

kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: scdf-rb
subjects:
- kind: ServiceAccount
  name: scdf-sa
roleRef:
  kind: Role
  name: scdf-role
  apiGroup: rbac.authorization.k8s.io
Copy the code

(3) service – the yaml

apiVersion: v1
kind: ServiceAccount
metadata:
  name: scdf-sa
Copy the code

Run the following command to create an account:

$ kubectl create -f src/kubernetes/server/server-roles.yaml 
$ kubectl create -f src/kubernetes/server/server-rolebinding.yaml 
$ kubectl create -f src/kubernetes/server/service-account.yaml 
Copy the code

After executing, you can check:

$ kubectl get role
NAME        AGE
scdf-role   119m

$ kubectl get rolebinding
NAME      AGE
scdf-rb   117m

$ kubectl get serviceAccount
NAME      SECRETS   AGE
default   1         27d
scdf-sa   1         117m
Copy the code

Deploy MySQL 2.3

You can choose another database. If you already have a database, you don’t need to deploy it. You can just change the configuration when deploying the Server. Follow the official Guide here. To ensure that the deployment does not fail due to image download issues, I downloaded the image in advance:

$ docker pull mysql:5.7.25
Copy the code

The yaml file of MySQL does not need to be modified. Run the following command:

$ kubectl create -f src/kubernetes/mysql/
Copy the code

Check after execution:

$ kubectl get Secret NAME TYPE DATA AGE default-token-jhgfp kubernetes.io/service-account-token 3 27d mysql Opaque 2 98m  scdf-sa-token-wmgk6 kubernetes.io/service-account-token 3 123m $ kubectl get PersistentVolumeClaim NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE mysql Bound pvc-e95b495a-bea5-40ee-9606-dab8d9b0d65c 8Gi RWO hostpath 98m $ kubectl get Deployment NAME READY UP-TO-DATE AVAILABLE AGE mysql 1/1 1 1 98m $ kubectl get Service NAME TYPE CLUSTER-IP External-ip PORT(S) AGE mysql ClusterIP 10.98.243.130 < None > 3306/TCP 98MCopy the code

2.4 Deploying the Data Flow Server

2.4.1 Modifying the configuration file server-config.yaml

Delete unused configurations, mainly for Prometheus and Grafana, with the following results:

apiVersion: v1
kind: ConfigMap
metadata:
  name: scdf-server
  labels:
    app: scdf-server
data:
  application.yaml: |- spring: cloud: dataflow: task: platform: kubernetes: accounts: default: limits: memory: 1024Mi datasource: url: jdbc:mysql://${MYSQL_SERVICE_HOST}:${MYSQL_SERVICE_PORT}/mysql username: root password: ${mysql-root-password} driverClassName: org.mariadb.jdbc.Driver testOnBorrow: true validationQuery: "SELECT 1"Copy the code

Modify server – SVC 2.4.2. Yaml

Since I am Kubernetes running locally, change the Service type from LoadBalancer to NodePort and set port to 30093.

kind: Service
apiVersion: v1
metadata:
  name: scdf-server
  labels:
    app: scdf-server
    spring-deployment-id: scdf
spec:
  # If you are running k8s on a local dev box or using minikube, you can use type NodePort instead
  type: NodePort
  ports:
    - port: 80
      name: scdf-server
      nodePort: 30093
  selector:
    app: scdf-server
Copy the code

2.4.3 modify server – deployment. Yaml

Stream-related configuration items such as SPRING_CLOUD_SKIPPER_CLIENT_SERVER_URI:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: scdf-server
  labels:
    app: scdf-server
spec:
  selector:
    matchLabels:
      app: scdf-server
  replicas: 1
  template:
    metadata:
      labels:
        app: scdf-server
    spec:
      containers:
      - name: scdf-server
        image: Springcloud/spring - the cloud - dataflow - server: 2.5.3. RELEASE
        imagePullPolicy: IfNotPresent
        volumeMounts:
          - name: database
            mountPath: /etc/secrets/database
            readOnly: true
        ports:
        - containerPort: 80
        livenessProbe:
          httpGet:
            path: /management/health
            port: 80
          initialDelaySeconds: 45
        readinessProbe:
          httpGet:
            path: /management/info
            port: 80
          initialDelaySeconds: 45
        resources:
          limits:
            cpu: 1.0
            memory: 2048Mi
          requests:
            cpu: 0.5
            memory: 1024Mi
        env:
        - name: KUBERNETES_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: "metadata.namespace"
        - name: SERVER_PORT
          value: '80'
        - name: SPRING_CLOUD_CONFIG_ENABLED
          value: 'false'
        - name: SPRING_CLOUD_DATAFLOW_FEATURES_ANALYTICS_ENABLED
          value: 'true'
        - name: SPRING_CLOUD_DATAFLOW_FEATURES_SCHEDULES_ENABLED
          value: 'true'
        - name: SPRING_CLOUD_KUBERNETES_SECRETS_ENABLE_API
          value: 'true'
        - name: SPRING_CLOUD_KUBERNETES_SECRETS_PATHS
          value: /etc/secrets
        - name: SPRING_CLOUD_KUBERNETES_CONFIG_NAME
          value: scdf-server
        - name: SPRING_CLOUD_DATAFLOW_SERVER_URI
          value: 'http://${SCDF_SERVER_SERVICE_HOST}:${SCDF_SERVER_SERVICE_PORT}'
          # Add Maven repo for metadata artifact resolution for all stream apps
        - name: SPRING_APPLICATION_JSON
          value: "{ \"maven\": { \"local-repository\": null, \"remote-repositories\": { \"repo1\": { \"url\": \"https://repo.spring.io/libs-snapshot\"} } } }"
      initContainers:
      - name: init-mysql-wait
        image: busybox
        command: ['sh'.'-c'.'until nc -w3 -z mysql 3306; do echo waiting for mysql; sleep 3; done; ']
      serviceAccountName: scdf-sa
      volumes:
        - name: database
          secret:
            secretName: mysql
Copy the code

2.4.4 deployment Server

After modifying the file, you can deploy by executing the following command:

Download the image in advance$docker pull springcloud/spring - the cloud - dataflow - server: 2.5.3. RELEASEDeploy the Data Flow Server$ kubectl create -f src/kubernetes/server/server-config.yaml $ kubectl create -f src/kubernetes/server/server-svc.yaml $  kubectl create -f src/kubernetes/server/server-deployment.yamlCopy the code

Completes, no error can visit: http://localhost:30093/dashboard/

3 Run a Task

The easiest way to verify that the deployment is successful is to run a task. Once again, register the application, define the Task, and then execute it.

We will still use the official ready app, but note that this time we choose Docker format instead of JAR package.

Kubernetes’ Dashboard shows a newly created Pod:

4 summarizes

In this article, Spring Cloud Data Flow was successfully deployed on Kubernetes and successfully ran a task on Kubenetes, which was no longer the Local standalone mode.

Welcome to pay attention to the wechat public number “Pumpkin slow Talk”, will continue to update for you…

Read more and share more; Write more. Organize more.