This section uses the StatefulSet controller to deploy a MySQL cluster and then perform a downtime test to see if the cluster can be restored without data loss.

The cluster implemented has the following characteristics:

  1. Is a master-slave MySQL cluster
  2. One primary node and multiple secondary nodes
  3. Slave nodes can scale horizontally
  4. All write operations can only be performed on the primary node
  5. Read operations can be performed on all nodes

Service Resources required

Resources required are:

  • A ConfigMap
  • Two Service
  • A StatefulSet

ConfigMap

kubectl apply -f https://k8s.io/examples/application/mysql/mysql-configmap.yaml
Copy the code

Service

kubectl apply -f https://k8s.io/examples/application/mysql/mysql-services.yaml
Copy the code

MySQL StatefulSet

wget https://k8s.io/examples/application/mysql/mysql-statefulset.yaml
Copy the code

Modify mysql-statefulset.yaml to use the NFS dynamic storage volume created in the previous section

  volumeClaimTemplates:
  - metadata:
      name: data
      annotations:
        volume.beta.kubernetes.io/storage-class: managed-nfs-storage
    spec:
      accessModes: ["ReadWriteOnce"]
      resources:
        requests:
          storage: 1Gi
Copy the code

Create a MySQL stateful application

kubectl apply mysql-statefulset.yaml
Copy the code

Write data test

Write data

Kubectl run mysql-client --image=mysql:5.7 -i --rm --restart=Never -- mysql -h mysql-0. Mysql <<EOFCREATE DATABASE test; CREATE TABLE test.messages (message VARCHAR(250)); INSERT INTO test.messages VALUES ('hello'); EOFCopy the code

Check whether the slave machine can read data

Kubectl run mysql-client --image=mysql:5.7 -i -t --rm --restart=Never -- mysql -h mysql-read -e "SELECT * FROM kubectl run mysql-client --image=mysql:5.7 -i -t --rm --restart=Never -- mysql -h mysql-read -e "SELECT * FROM test.messages"Copy the code

The server ID changes constantly, indicating that data is read from different machines each time

Kubectl run mysql-client-loop --image=mysql:5.7 -i -t --rm --restart=Never -- bash-ic "while sleep 1; kubectl run mysql-client-loop --image=mysql:5.7 -i -t --rm --restart=Never -- bash-ic "while sleep 1; do mysql -h mysql-read -e 'SELECT @@server_id,NOW()'; done"Copy the code

Outage test

Simulate the breakdown of mysqL-2

kubectl exec mysql-2 -c mysql -- mv /usr/bin/mysql /usr/bin/mysql.off
Copy the code

You will not see the record of server 102 from the loop read SELECT @@server_id

Kubectl run mysql-client-loop --image=mysql:5.7 -i -t --rm --restart=Never -- bash-ic "while sleep 1; kubectl run mysql-client-loop --image=mysql:5.7 -i -t --rm --restart=Never -- bash-ic "while sleep 1; do mysql -h mysql-read -e 'SELECT @@server_id,NOW()'; done"Copy the code

After node mysql-2 is restored, service-id 102 will appear in the loop record

kubectl exec mysql-2 -c mysql -- mv /usr/bin/mysql.off /usr/bin/mysql
Copy the code

Expand the number of secondary nodes

[root@master01 ~]# kubectl scale statefulset mysql --replicas=5 statefulset.apps/mysql scaled [root@master01 ~]# kubectl  get pods -l app=mysql --watch NAME READY STATUS RESTARTS AGE mysql-0 2/2 Running 0 29m mysql-1 2/2 Running 0 28m mysql-2 2/2 Running 1 24m mysql-3 0/2 Init:1/2 0 9s mysql-3 1/2 Running 0 44s mysql-3 2/2 Running 1 55s ...... mysql-4 0/2 Pending 0 3s mysql-4 0/2 Init:0/2 0 3s mysql-4 0/2 Init:1/2 0 7s mysql-4 0/2 PodInitializing 0 43s mysql-4 1/2 Running 0 46s mysql-4 1/2 Running 1 51s mysql-4 2/2 Running 1 56s [root@master01 ~]# kubectl get pod -l app=mysql NAME READY STATUS RESTARTS AGE mysql-0 2/2 Running 0 32m mysql-1 2/2 Running 0 31m mysql-2 2/2 Running 1 28m mysql-3 2/2  Running 1 3m22s mysql-4 2/2 Running 1 2m27sCopy the code

After the capacity expansion is successful, server-ID 103/104 is also displayed in the select @@server_id command.

The basic principle of

Use ConfigMap to reference the primary and secondary MySQL node configuration files prepared in advance.

Then we define two services, one is mysql, one is mysql-read.

The first MySQL Service is a headless Service, which does not perform load balancing or traffic distribution. It only assigns a DNS name to each Pod. So the Pod DNS name will also be fixed.

Mysql-0. MySQL; MySQL;

The second Service is a normal Service that provides read operations. Proxies read operations to all back-end pods.

Now you can install MySQL automatically.

Since the Pod name is fixed, always use mysql-0 as the master node and POD-1-2-3 as the slave node.

Before a Pod is started, it performs two initialization containers, the first init container to copy the configuration file it is going to use, and the second init container to clone data from the previous Pod, which can be skipped if it is the master node.

The slave container needs a bucket container xtrabackup to initialize the data.

conclusion

Today, I set up mysql master and slave through K8S to store data persistently.

The Pod can automatically restore the original order after downtime, and the original data will not be lost.