Before the speech

In the practice of production environment, high availability is an unavoidable problem. K3s itself has gone through several iterations and HA scheme has been continuously optimized, forming the current relatively stable HA scheme.

Currently, two HA schemes are officially available:

  • High availability of embedded DB (Experiment)

  • Use external databases for high availability

The high availability of embedded DB is experimental at present, but this article will not introduce more, please refer to:

Rancher.com/docs/k3s/la…

Using external database to realize high availability need to build a high availability of external database, at present, K3s support SQLite/etcd/MySQL/PostgreSQL/DQLite datastore, different datastore for different usage scenarios.

At present, ali Cloud is the most widely used public cloud environment in China. We can use virtual machines to build K3s HA on Ali Cloud, and then connect the RDS of Ali Cloud, so as to avoid the trouble of maintaining a set of databases separately. This article chooses MySQL to do HA practice, PostgreSQL is similar to MySQL, this article will not repeat.

Architecture diagram

As shown in the figure above, the terminal user accesses the SLB, and the SLB forwards the traffic to the two K3s master HA at the back end. Two K3s master nodes connect to the same external database created by RDS.

Create an Ali Cloud instance

K3s requires at least two instances to form HA, so create at least two instances on Ali Cloud for demonstration:

Configure Alicloud RDS

1. Create an RDS instance, and select MySQL 5.7 as the instance type, which is officially supported by K3s. Set other parameters as required.

2. Set the whitelist to the Intranet IP address of your K3s instance. Set up after the success, we will get a network address is used as the database connection: rm-2ze64ke7q33bkq3yt.mysql.rds.aliyuncs.com

3. Create an account using a common account (KSD)

4, create database, set database name (k3s), authorization account (KSD)

In the previous mysql startup with Docker, there was no need to create the database in advance, because it will be created automatically when k3S is started. But on Ali Cloud RDS, you must first create the database required by K3s on the UI.

5. Modify database parameters

Innodb_large_prefix = ON innodb_large_prefix = ON innodb_large_prefix = ON

Jul 29 20:08:06 iZ2zed0v8rqape974mz8suZ systemd[1]: k3s.service: Service hold-off time over, scheduling restart.
Jul 29 20:08:06 iZ2zed0v8rqape974mz8suZ systemd[1]: k3s.service: Scheduled restart job, restart counter is at 11.
Jul 29 20:08:06 iZ2zed0v8rqape974mz8suZ systemd[1]: Stopped Lightweight Kubernetes.
Jul 29 20:08:06 iZ2zed0v8rqape974mz8suZ systemd[1]: Starting Lightweight Kubernetes...
Jul 29 20:08:07 iZ2zed0v8rqape974mz8suZ k3s[24934]: time="2020-07-29T20:08:07.145963348+08:00" level=info msg="Starting k3s v1.18.6+k3s1 (6f56fa1d)"
Jul 29 20:08:07 iZ2zed0v8rqape974mz8suZ k3s[24934]: time="2020-07-29T20:08:07.159363656+08:00" level=fatal msg="starting kubernetes: preparing server: creating storage endpoint: building kine: Error 1071: Specified key was too long; max key length is 767 bytes"
Jul 29 20:08:07 iZ2zed0v8rqape974mz8suZ systemd[1]: k3s.service: Main process exited, code=exited, status=1/FAILURE
Jul 29 20:08:07 iZ2zed0v8rqape974mz8suZ systemd[1]: k3s.service: Failed with result 'exit-code'.
Jul 29 20:08:07 iZ2zed0v8rqape974mz8suZ systemd[1]: Failed to start Lightweight Kubernetes.
Copy the code

After changing innodb_large_prefix to ON, click “Submit Parameters” in the upper right corner to complete the modification.

After the above steps are successful, the external database required by K3s is ready. Let’s start the K3s HA.

Implement K3s HA

Run the same command on k3s-master-1 and k3s-master-2:

curl -sfL https://docs.rancher.cn/k3s/k3s-install.sh |  \
  INSTALL_K3S_MIRROR=cn \
  K3S_DATASTORE_ENDPOINT='mysql://ksd:your_password@tcp(rm-2ze64ke7q33bkq3yt.mysql.rds.aliyuncs.com:3306)/k3s' \
  sh -s - server
Copy the code

After a while, a K3s HA environment is ready to start:

If the image of K3s is slow to be pulled on Ali Cloud, you can configure the mirror or download the offline package of the corresponding version from mirror.cnrancher.com and import the image as follows: rancher.com/docs/k3s/la…

root@k3s-master-2:~# kubectl get pods -A -o wide NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES Kube-System Local-path-provisioner - 6D59F47c7-tshFX 1/1 Running 0 16m 10.42.0.5k3s-master-1 < None > <none> kube-system metrics-server-7566d596c8-mrc94 1/1 Running 0 16m 10.42.0.3s-master-1 <none> <none> kube-system Coredns-8655855d6-sxn7v 1/1 Running 0 16m 10.42.0.4 k3s-master-1 <none> <none> kube-system helm-install-traefik-cmMSr 0/1 Completed 2 16m 10.42.0.3k3s-master-1 <none> <none> kube-system svclb-traefik-z6vlb 2/2 Running 0 11m 10.42.0.6 K3s-master-1 <none> <none> kube-system svclb-traefik-f89x6 2/2 Running 0 11m 10.42.1.2 k3s-master-2 <none> <none> Kube-system traefik-758CD5fc85-chnbc 1/1 Running 0 11m 10.42.1.3 k3s-master-2 <none> <none> root@k3s-master-2:~# root@k3s-master-2:~# kubectl get node -o wide NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE Kernel-version container-Runtime k3s-master-1 Ready master 16m V1.18.6 + K3s1 172.17.207.15 <none> Ubuntu 18.04.4lts 4.15.0-106- Generic Containerd ://1.3.3-k3s2 k3s-master-2 Ready Master 16m v1.18.6+ K3S1 172.17.207.16 <none> Ubuntu 18.04.4 LTS 4.15.0-106 - generic containerd: / / 1.3.3 - k3s2Copy the code

Provide unified access through Ali Cloud SLB

We now have highly available MySQL and K3s, but we still need a unified access point for multiple K3s servers. This can be done in the following way:

  1. L4 layer load balancer

  2. Round-robin DNS

  3. VIP or elastic IP address

Therefore, we can directly use the SLB of Ali Cloud to do L4 load balancing and forward port 6443 to the two K3s masters at the back end.

Next, we can put the k3s/etc/rancher/master node k3s/k3s yaml is copied to the local ~ /. Kube/config directory, then amend the server address to the server: Public IP https://39.106.185.201:6443 (SLB)

Kubectl get Nodes test whether SLB can be used to forward traffic to K3s master:

ksd@Hailong-MacBook-Pro  ~  kubectl get nodes
Unable to connect to the server: x509: certificate is valid for 10.43.0.1, 127.0.0.1, 172.17.207.15, 172.17.207.16, not 39.106.185.201
Copy the code

K3s mster does not trust 39.106.185.201 public IP address of SLB. To resolve this issue, update K3s master with tlS-san 39.106.185.201:

curl -sfL https://docs.rancher.cn/k3s/k3s-install.sh | \ INSTALL_K3S_MIRROR=cn \ K3S_DATASTORE_ENDPOINT='mysql://ksd:your_password@tcp(rm-2ze64ke7q33bkq3yt.mysql.rds.aliyuncs.com:3306)/k3s' \ sh -s - Server \ - TLS - SAN 39.106.185.201Copy the code

Finally, go back to the local machine and execute kubectl get Nodes again. If nothing goes wrong, you should get node information.

ksd@Hailong-MacBook-Pro  ~  kubectl get nodes NAME STATUS ROLES AGE VERSION K3S-master-2 Ready master 65m v1.18.6+ k3S1 K3s-master-1 Ready Master 65M V1.18.6 + K3S1Copy the code

Remember after

This paper only introduces how to realize THE HA of K3s with the help of THE SLB and RDS of The Ali Cloud. The operation of other public clouds is basically the same with little difference. Although no detailed test has been done, it should be supported theoretically. In a non-public cloud environment, select an appropriate datastore and HA mode based on your requirements.