Author: SRE operations blog

Blog: www.cnsre.cn/

Article address: www.cnsre.cn/posts/21111…

Related topics: www.cnsre.cn/tags/k3s/

Double Eleven collected the wool from several cloud manufacturers and built the stand-alone version of K3S at the beginning. Then I thought if I could build a K3S cluster, and then I tried to use WireGuard for networking by referring to this big guy’s article. It is light weight, convenient, efficient, and the whole data encryption transmission, is based on the public network to build a virtual LAN excellent choice.

Environment introduction

Server Introduction

Cloud vendors Public IP Address Intranet IP Address Virtual network IP address The operating system Kernel version
Tencent cloud 1 42.xx.xx.12 10.0.16.8 192.168.1.1 CentOS Linux release 7.9.2009 (Core) 5.15.2-1
Tencent cloud 2 122.xx.xxx.111 10.0.0.6 192.168.1.2 instead CentOS Linux release 7.9.2009 (Core) 5.15.2-1
Ali cloud 122.xx.xx.155 172.17.0.3 192.168.1.3 CentOS Linux release 7.9.2009 (Core) 5.15.2-1

Preparation before construction

Before setting up the k3S cluster across the cloud, we need to install WireGuard. The WireGuard has requirements on the kernel, so the kernel has been upgraded to 5.15.2-1.el7.elrepo.x86_64

Enable IP address forwarding on all nodes:

echo "net.ipv4.ip_forward = 1" >> /etc/sysctl.conf
echo "net.ipv4.conf.all.proxy_arp = 1" >> /etc/sysctl.conf
sysctl -p /etc/sysctl.conf
Copy the code

The host name change function is enabled for all nodes

# Tencent Cloud 1 execute hostnamectl set-hostname k3s-master # Tencent cloud 2 execute hostnamectl set-hostname k3s-node1 # Alibaba cloud execute hostnamectl set-hostname k3s-node2Copy the code

To upgrade the kernel

The default kernel of several servers is 3.10. Installing WireGuard requires that the kernel be upgraded to a higher version.

Before upgrading the kernel

Upgrade software packages first (not necessary)

yum update -y
Copy the code

Add iptables rules to allow native NAT:

iptables -A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
iptables -A FORWARD -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
iptables -A FORWARD -i wg0 -o wg0 -m conntrack --ctstate NEW -j ACCEPT
iptables -t nat -A POSTROUTING -s 192.168.1.1/24 -o eth0 -j MASQUERADE
Copy the code

Note:

Wg0: virtual network card defined for you

192.168.1.1: indicates your virtual IP address segment

Eth0: for your physical nic

To upgrade the kernel

All nodes are executed

Method 1:

Download the RPM package and install it.

If you want to install other kernels, you can also download them here

The RPM wget http://ftp.sjtu.edu.cn/sites/elrepo.org/linux/kernel/el7/x86_64/RPMS/kernel-ml-5.15.2-1.el7.elrepo.x86_64.rpm - the ivh kernel - ml - 5.15.2-1. El7. Elrepo. X86_64. RPMCopy the code

Method 2:

Update using package management tools

# in public key RPM -- the import https://www.elrepo.org/RPM-GPG-KEY-elrepo.org # elrepo RPM - Uvh upgrade installation http://www.elrepo.org/elrepo-release-7.0-5.el7.elrepo.noarch.rpm # load elrepo - yum kernel metadata -- disablerepo = \ * Yum --disablerepo=\* --enablerepo=elrepo-kernel install kernel-ml.x86_64 -y # yum remove kernel-tools-libs.x86_64 kernel-tools.x86_64 -y # yum remove kernel-tools-libs.x86_64Copy the code

Method 3:

Compile and install by source package.

This way can be customized, but it is also more complex, there is a need to find their own information to install, the following only gives the download address of the kernel source package of each system version

Change the default kernel version

# check the actual startup sequence grub2 - editenv # list to view the kernel insertion order grep "^ menuentry"/boot/grub/grub2. CFG | the cut - d "'" - f2 # set the default start Grub2-set-default 'CentOS Linux (5.15.2-1.el7.elrebo.x86_64) 7 (Core)' # create kernel configuration grub2-mkconfig -o /boot/grub2/grub.cfg Verify the current kernel version uname -rCopy the code

Note:

The kernel version must be higher. Otherwise, an error occurs when WireGuard is started.

[#] ip link add wg0 type wireguard
RTNETLINK answers: Operation not supported
Unable to access interface: Protocol not supported
[#] ip link delete dev wg0
Cannot find device "wg0"
Copy the code

Install the WireGuard

Execution of all nodes

CentOS kernel 5.15.2, which already contains the WireGuard kernel module, just need to install WireGuard -tools yum package.

yum install epel-release https://www.elrepo.org/elrepo-release-7.el7.elrepo.noarch.rpm
yum install yum-plugin-elrepo kmod-wireguard wireguard-tools -y
Copy the code

Configuration WireGuard

The WireGuard-Tools package provides the tools WG and WG-Quick needed for manual and automatic deployment, respectively.

First, generate Tencent Cloud 1 encryption and decryption key according to the form described in the official document

wg genkey | tee privatekey | wg pubkey > publickey
Copy the code

The privatekey and publickey files are then generated in the current directory

Note:

The key is configured to the local machine, while the public key is configured to another machine.

cat privatekey publickey
EMWcI01iqM4zkb7xfbaaxxxxxxxxDo2GJUA=
0ay8WfGOIHndWklSIVBqrsp5LDWxxxxxxxxxxxxxxQ=
Copy the code

Now we need Tencent Cloud 2 Ali Cloud which is connected to the above host through peer network, and its public IP address (the IP address that can communicate with the host is required here) is 122.xx.xx.111,122.xx.xx.155

We first install WireGuard and generate the key of Tencent Cloud 2 ali Cloud according to the above procedure.

Then write the complete Tencent Cloud 1 configuration file for WG-Quick to use in /etc/wireguard/wg0.conf of host A

[Interface] PrivateKey = EMWcI01iqM4zkb7xfbaaxxxxxxxxDo2GJUA = Address = 192.168.1.1 ListenPort [Peer] PublicKey = = 5418 Tencent Cloud 2 publicKey EndPoint = 122.xx.xxx.111:5418 AllowedIPs = 192.168.1.2/32 [Peer] publickey = Ali cloud publickey EndPoint = 122. Xx. Xx. 155:5418 AllowedIPs = 192.168.1.3/32Copy the code

Configuration instructions

Interface: section is the configuration of Tencent Cloud 1 (that is, the local machine).

Address: is the virtual IP assigned to Tencent Cloud 1,

ListenPort: indicates the port used by hosts to communicate with each other. The port uses the UDP protocol.

Peer: is the information belonging to Tencent Cloud that needs to communicate. 2. Ali Cloud information, how many hosts need to communicate, add how many Peer sections.

EndPoint: is the public IP address of Tencent Cloud 2 and Ali Cloud and the UDP port monitored by WireGuard. This IP address may not be a public network.

Pay attention to

If your machine can communicate through the Intranet, you can also use the Intranet IP address directly, of course, note that this IP address must be able to communicate with all the hosts in the LAN.

AllowedIPs: For example, if host B is assigned the Intranet IP address 192.168.1.2, the packets sent from host A to 192.168.1.2 should be forwarded to the EndPoint, which actually acts as A filter. When there are multiple peers, the IP addresses must be unique.

The privatekey and publickey generated by each node are as follows

[root @ # master node k3s - master ~] # cat privatekey publickey EMWcI01iqM4zkb7xfbaaxxxxxxxxDo2GJUA = [0 ay8wfgoihndwklsivbqrsp5ldwxxxxxxxxxxxxxxq = # node1 node root @ k3s rac-node1 ~] # cat privatekey publickey QGdNkzpnIkuvUU + 00 c6xyxxxxxxxxxk0d82qjvc = 3 izpvbzgphlm + S5szOogTDTxxxxxxxxxuKuDGn4 = # 2 nodes/root @ k3s - 2 ~ # cat privatekey publickey WOOObkWINkW/hqaAME9r+xxxxxxxxxm+r2Q= 0f0dn60+tBUfYgzw7rIihKbqxxxxxxxxa6Wo=Copy the code

The configuration files of each node are as follows

# cat/etc/master node wireguard/wg0 conf/Interface PrivateKey = EMWcI01iqM4zkb7xfbaaxxxxxxxxDo2GJUA = Address = 192.168.1.1 ListenPort = 5418 / Peer PublicKey = 3 izpvbzgphlm + S5szOogTDTxxxxxxxxxuKuDGn4 = the EndPoint = 122. Xx, XXX. 111:5418 AllowedIPs = 192.168.1.2 instead / 32 / Peer PublicKey = 0 f0dn60 + tBUfYgzw7rIihKbqxxxxxxxxa6Wo = the EndPoint = 122. Xx. Xx. 155:5418 AllowedIPs = 192.168.1.3/32Copy the code
Conf [Interface] PrivateKey = QGdNkzpnIkuvUU+00C6XYxxxxxxxxxK0D82qJVc= Address = 192.168.1.2 instead ListenPort = 5418 / Peer PublicKey = 0 ay8wfgoihndwklsivbqrsp5ldwxxxxxxxxxxxxxxq = the EndPoint = 42. Xx. Xx. Saying 18 AllowedIPs = 192.168.1.1/32 / Peer PublicKey = 0 f0dn60 + tBUfYgzw7rIihKbqxxxxxxxxa6Wo = the EndPoint = 122 xx. Xx. 155:5418 AllowedIPs = 192.168.1.3/32Copy the code
# node2 cat /etc/wireguard/wg0.conf [Interface] PrivateKey = WOOObkWINkW/hqaAME9r+ XXXXXXXXXM +r2Q= Address = 192.168.1.3 ListenPort = 5418 / Peer PublicKey = 0 ay8wfgoihndwklsivbqrsp5ldwxxxxxxxxxxxxxxq = the EndPoint = 42. Xx. Xx. Saying 18 AllowedIPs = 192.168.1.1/32 / Peer PublicKey = 3 izpvbzgphlm + S5szOogTDTxxxxxxxxxuKuDGn4 = the EndPoint = 122. Xx. Xx. 155:5418 AllowedIPs 192.168.1.2 instead of = / 32Copy the code

Start the WireGuard

After the configuration file is written, use WG-Quick to create the virtual network card,

wg-quick up wg0
Copy the code

Wg0: /etc/wireguard/wg0.conf: /etc/wireguard/wg0.conf: /etc/wireguard/wg0.conf: /etc/wireguard/wg0.conf

After installing and configuring the network card devices of Tencent Cloud 2 and Ali Cloud, you can use WG command to observe the networking situation

[root@k3s-master ~]# wginterface: wg0 public key: 0ay8WfGOIHndWklSIVBqrsp5LDWxxxxxxxxxxxxxxQ= private key: (hidden) listening port: 5418 peer: 0 f0dn60 + tBUfYgzw7rIihKbqxxxxxxxxa6Wo = the endpoint: 122 xx. Xx. 155:5418 allowed ips: 192.168.1.3/32 latest handshake: 3 minutes, 3 seconds ago Transfer: 35.40 KiB received, 47.46 KiB sent peer: 3 izpvbzgphlm + S5szOogTDTxxxxxxxxxuKuDGn4 = the endpoint: 122 xx, XXX. 111:5418 allowed ips: 192.168.1.2 instead / 32 latest handshake: 5 minutes, 6 seconds ago Transfer: 24.84 KiB received, 35.21 KiB sentCopy the code

You can see the peer-to-peer node information listed, as well as the communication measurements. You can ping the virtual IP address of another host or SSH the IP address of another host to check whether the network communication is normal.

automation

After the system restarts, the network adapter devices created by WireGuard will be lost, with automated scripts

systemctl enable wg-quick@wg0
Copy the code

Run the preceding command to generate the systemd daemon script, and the system automatically runs the up command.

Configure hot overload

Wg-quick does not provide instructions for overloading, but does provide a strip directive that converts conf files into a format that wg directives recognize.

wg syncconf wg0 <(wg-quick strip wg0)
Copy the code

Can achieve thermal overload.

With the WireGuard installed and configured, we are ready to install the K3S cluster.

Install the K3S cluster

Master Node Installation

curl -sfL http://rancher-mirror.cnrancher.com/k3s/k3s-install.sh | INSTALL_K3S_MIRROR=cn sh -s - --node-external-ip 12 --advertise-address 42.xx.xx.12 --node-ip 192.168.1.1 --flannel-iface wg0Copy the code

Parameter Description:

  • --node-external-ip 42.xx.xx.12Set an external IP address for the node. The external IP address of ali Cloud VPC is not directly bound to the VM network interface card (NIC). Therefore, I need to set this parameter to prevent the K3S component from using the internal IP address as the public IP address when setting loadBalance.
  • --advertise-address 42.xx.xx.12Used to set the address used by the Kubectl tool and child nodes to communicate. It can be IP or domain name. It will be set to the valid domain when the Apiserver certificate is created.
  • IP - node - 10.20.30.1If this parameter is not specified, the IP address on the first nic device is selected. Therefore, this IP address is usually an Intranet IP address. But I built my own virtual LAN, so I needed to specify the IP of the virtual LAN (that is, the IP of the WireGuard).
  • --flannel-iface wg0Wg0 is a network adapter device created by WireGuard. I need to use virtual LAN for communication between nodes, so I need to specify wg0 here.

In addition, because all traffic of WireGuard is encrypted, communication between nodes can be ensured through WireGuard, so there is no need to use other CNI drivers, just use the default one.

Less than a minute after the primary node executes the above command, you can see the script indicating that the installation is complete. You can run commands to view the running status of the main control terminal

systemctl status k3s
Copy the code

If so, check to see if the container is healthy

kubectl get pods -A
Copy the code

The -a parameter is used to view all namespaces. If the containers are in the running state, the installation is successful and the controlled nodes can be added.

The Agent install

With the experience of installing the master controller above, it is easier to install the Work node, and the parameters need to be adjusted

Tencent Cloud 2 execution

curl -sfL http://rancher-mirror.cnrancher.com/k3s/k3s-install.sh | INSTALL_K3S_MIRROR=cn K3S_URL = https://192.168.1.1:6443 K3S_TOKEN=K10720eda8a278bdc7b9b6d787c9676a92119bb2cf95048d6a3cd85f15717edfbe5::server:e98b986e8202885cb54da1b7e701f67e 111 --node-ip 192.168.1.2 --flannel-iface Wg0Copy the code

Aliyun Executive

curl -sfL http://rancher-mirror.cnrancher.com/k3s/k3s-install.sh | INSTALL_K3S_MIRROR=cn K3S_URL = https://192.168.1.1:6443 K3S_TOKEN=K10720eda8a278bdc7b9b6d787c9676a92119bb2cf95048d6a3cd85f15717edfbe5::server:e98b986e8202885cb54da1b7e701f67e Sh -s - --node-external-ip 122.xx.xx.155 --node-ip 192.168.1.3 --flannel-iface wg0Copy the code

Parameters need not be explained too much

  • K3S_TokenAccording to the documentation, go/var/lib/rancher/k3s/server/node-tokenGet.
  • K3S_URLThe default port number is 6443, and the IP address is the IP address of the virtual network domain. In this way, traffic is encrypted and transmitted through the WireGuard.

The other two parameters need not be said, and the same logic as the master. After the installation is complete, check the service running status as usual

systemctl status k3s-agent
Copy the code

If there is an error, find a solution based on the error.

Check the installation on the master node.

kubectl get nodes -o wide 
Copy the code

At this point, the cloudy K3S cluster has been set up.

Author: SRE operations blog

Blog: www.cnsre.cn/

Article address: www.cnsre.cn/posts/21111…

Related topics: www.cnsre.cn/tags/k3s/