My K8S essay: Kubernetes 1.17.0 deployment explanation

K8s Cluster deployment process Practice notes There are two versions: one is dedicated deployment operations, and the other is deployment related operations. This article is the latter. This article describes how to use Kubeadm to deploy Kubernetes 1.17.0 cluster on two Ubuntu 16.04 64-bit dual-core CPU VMS. The network plug-in is Flannel V0.11.0, and the image source is Ali Cloud. This paper has some practical reference significance.

A, environmental

Two Ubuntu 16.04 64-bit, 2GB of RAM, dual-core CPUS. Environment requirements and Settings: Two hosts, one for master and one for Node. The master host name is Ubuntu. Node The host name is Node. Ensure that the host name of the operating system is different. All operations are performed with the root permission (note: if you are a common user, an error will be reported when you initialize k8s). Note that k8S requires the machine to have a dual-core CPU. The K8S requires port 6443. The project directory is $HOME/k8s. The K8S version deployed for this article is 1.17.0. The image and version are as follows:

K8s. GCR. IO/kube - apiserver: v1.17.0 k8s. GCR. IO/kube - controller - manager: v1.17.0 k8s. GCR. IO/kube - the scheduler: v1.17.0 K8s. GCR. IO/kube - proxy: v1.17.0 k8s. GCR. IO/pause: 3.1 k8s. GCR. IO/etcd: rule 3.4.3-0 k8s. GCR. IO/coredns: 1.6.5 Quay. IO/coreos/flannel: v0.11.0 - amd64Copy the code

Note 1: k8s. GCR. IO using ali cloud image address registry.aliyuncs.com/google_containers replacement. Note 2: Different versions of K8S are used in different deployment periods, and the corresponding component versions are also different, which need to be downloaded again.

Version compatibility record: 2020.3.17: K8S version updated to 1.17.4(released 2020.3.12), other components remain unchanged.

K8s. GCR. IO/kube - apiserver: v1.17.4 k8s. GCR. IO/kube - controller - manager: v1.17.4 k8s. GCR. IO/kube - the scheduler: v1.17.4 K8s. GCR. IO/kube - proxy: v1.17.4 k8s. GCR. IO/pause: 3.1 k8s. GCR. IO/etcd: rule 3.4.3-0 k8s. GCR. IO/coredns: 1.6.5 Quay. IO/coreos/flannel: v0.11.0 - amd64Copy the code

The test passed.

Install Docker

apt-get install docker.io
Copy the code

The version of Docker installed for this article is 18.09.7. Run the following command to create the /etc/docker/daemon.json file:

cat > /etc/docker/daemon.json <<-EOF
{
  "registry-mirrors": [
    "https://a8qh6yqv.mirror.aliyuncs.com",
    "http://hub-mirror.c.163.com"
  ],
  "exec-opts": ["native.cgroupdriver=systemd"]
}
EOF

Copy the code

Meaning: registry-mirrors indicates the address of the mirror accelerator. Native-cgroupdriver =systemd Indicates that the cGroup driver is systemd (k8s uses this mode). The default is cgroupfs. The cause is that the k8s driver mode cannot be changed in kubeadm.conf.

Restart Docker to view cgroup:

# systemctl restart docker 
# docker info | grep -i cgroup
Cgroup Driver: systemd
Copy the code

If systemd is displayed, the modification is successful.

Deploy the K8S Master host

K8s can be deployed on a master host or a node. This section describes the master host.

3.1 close the swap

Edit the /etc/fstab file and comment out the line on which the swap partition is mounted. For example:

# swap was on /dev/sda5 during installation
UUID=aaa38da3-6e60-4e9d-bfc6-7128fd05f1c7 none swapsw  0  0
Copy the code

To perform:

# swapoff -a
Copy the code

3.2 Adding a Domestic K8S Source

Select Ali Cloud here:

# cat <<EOF > /etc/apt/sources.list.d/kubernetes.list
deb https://mirrors.aliyun.com/kubernetes/apt/ kubernetes-xenial main
EOF

Copy the code

Add the key:

# cat https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -
Copy the code

If unsuccessful, can use Ali Cloud mirror address:

curl https://mirrors.aliyun.com/kubernetes/apt/doc/apt-key.gpg | apt-key add - 
Copy the code

Of course, there are some ways to download apt-key. GPG and put it in the project directory. To perform:

# cat apt-key.gpg | sudo apt-key add -
Copy the code

Note: Domestic mirror site is to synchronize after all, synchronization may not succeed, may also be delayed synchronization, so, key or Google official prevail.

3.3 update the source

# apt-get update
Copy the code

Install kubeadm, Kubectl, Kubelet, Kubernetes-CNi and other tools.

# apt-get install -y kubeadm kubectl kubelet kubernetes-cni
Copy the code

Note 1: Installing Kubeadm automatically installs Kubectl, Kubelet, and Kubernetes-cni, so only specifying kubeadm will work. Note 2: This article was installed with version 1.17.0 and kubernetes-CNI with 0.7.5. The download file is in /var/cache/apt/archives/.

3.4 Obtaining the Image Version Required for Deployment

# kubeadm config images list
Copy the code

The output is as follows:

W1214 08:46:14.303158 8461 version.go:101] Could not fetch a Kubernetes version from the Internet: unable to get URL "https://dl.k8s.io/release/stable-1.txt": Get https://dl.k8s.io/release/stable-1.txt: net/http: Request Canceled while waiting for connection (client. Timeout exceeded while awaiting headers) W1214 08:46:14.303772 8461 version.go:102] falling back to the local client version: V1.17.0 W1214 08:46:14.304223 8461 Validation. go:28] Cannot validate kube-proxy config - no validator is available W1214 08:46:14.304609 8461 Validation. go:28] Cannot validate kubelet config - no validator is available K8s. GCR. IO/kube - apiserver: v1.17.0 k8s. GCR. IO/kube - controller - manager: v1.17.0 k8s. GCR. IO/kube - the scheduler: v1.17.0 K8s. GCR. IO/kube - proxy: v1.17.0 k8s. GCR. IO/pause: 3.1 k8s. GCR. IO/etcd: rule 3.4.3-0 k8s. GCR. IO/coredns: 1.6.5Copy the code

The previous warning message is ignored. This is to confirm the version of the mirror that kubeadm matches. Compatibility problems may occur due to different versions of components.

3.5 Pulling an Image File.

Generally, the image of k8s.gcr. IO cannot be downloaded directly in China. There are two methods: 1. When initializing K8S, use the ali Cloud image address. This address can be downloaded successfully. You can download in advance, to replace the mirror address prefix that in the previous section registry.cn-hangzhou.aliyuncs.com/google_containers can.

2. Download the aforementioned image by yourself. Use the pullk8s.sh script (note that the script must add the x attribute) :

#! /bin/bash # = "k8s.gcr. IO /"; The version is obtained by running the kubeadm config images list command. Images =(kube-apiserver:v1.17.0 kube-controller-manager:v1.17.0 Kube-proxy :v1.17.0 kube-proxy:v1.17.0 etCD :3.4.3-0 coreDNS :1.6.5) for imageName in ${images[@]}; do docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/$imageName docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/$imageName k8s.gcr.io/$imageName docker rmi registry.cn-hangzhou.aliyuncs.com/google_containers/$imageName doneCopy the code

Pull:

Chmod +x pullk8s.sh bash pullk8s.sh (or./pullk8s.sh)Copy the code

3.6 the network

Set network configuration:

Mkdir -p /etc/cni/net.d cat >/etc/cni/net.d/10-mynet.conf << -eof {"cniVersion": "0.3.0", "name": "mynet", "type": "bridge", "bridge": "cni0", "isGateway": true, "ipMasq": true, "ipam": { "type": "host-local", "subnet": "10.244.0.0/16", "routes" : [{" DST ":" 0.0.0.0/0}]}} EOF cat > / etc/the cni/net. D / 99 - loopback. Conf < < - EOF {" cniVersion ": "0.3.0", "type": "loopback"} EOFCopy the code

In practice, this step can be done without.

3.7 Downloading a Flannel mirror

Docker pull quay. IO/coreos/flannel: v0.11.0 - amd64Copy the code

Note: If downloading is not possible, use another method. Flannel Mirror information:

# docker images | grep flannel quay. IO/coreos/flannel v0.11.0 - amd64 ff281650a721 11 have a line 52.6 MBCopy the code

Note that you have downloaded the flannel image first, and the version is confirmed by the official YAML file. See below for the address.

3.8 the initialization

Version 1:

Kubeadm init - pod - network - cidr = 10.244.0.0/16 \ - image - repository registry.aliyuncs.com/google_containersCopy the code

Definition: –pod-network-cidr specifies the network segment that will be used by the subsequent network plug-in (this article uses flannel). – the mirror image repository – specifies the address, the default for k8s. GCR. IO, designated as ali cloud image address registry.aliyuncs.com/google_containers here. — pod-netword-cidr specifies the network segment of CIDR. The default is 192.168.0.0/16. The default is 192.168. Note that other parameters default.

The above command is equivalent to the following command:

Kubeadm init \ - apiserver - advertise - address = 192.168.0.102 \ - image - repository registry.aliyuncs.com/google_containers \ --kubernetes-version v1.17.0 \ --service-cidr=10.1.0.0/16\ -- pod-nett-cidr =10.244.0.0/16Copy the code

Version 2, pull the version according to the previous script:

Kubeadm init - pod - network - cidr = 10.244.0.0/16Copy the code

This article uses version one deployment.

The following information is displayed during initialization:

W1221 17:44:19.880281    2865 version.go:101] could not fetch a Kubernetes version from the internet: unable to get URL "https://dl.k8s.io/release/stable-1.txt": Get https://dl.k8s.io/release/stable-1.txt: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
W1221 17:44:19.880405    2865 version.go:102] falling back to the local client version: v1.17.0
W1221 17:44:19.880539    2865 validation.go:28] Cannot validate kube-proxy config - no validator is available
W1221 17:44:19.880546    2865 validation.go:28] Cannot validate kubelet config - no validator is available
[init] Using Kubernetes version: v1.17.0
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [ubuntu kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.0.102]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [ubuntu localhost] and IPs [192.168.0.102 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [ubuntu localhost] and IPs [192.168.0.102 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
W1221 17:50:12.262505    2865 manifests.go:214] the default kube-apiserver authorization-mode is "Node,RBAC"; using "Node,RBAC"
[control-plane] Creating static Pod manifest for "kube-scheduler"
W1221 17:50:12.268198    2865 manifests.go:214] the default kube-apiserver authorization-mode is "Node,RBAC"; using "Node,RBAC"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[apiclient] All control plane components are healthy after 17.504683 seconds
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.17" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Skipping phase. Please see --upload-certs
[mark-control-plane] Marking the node ubuntu as control-plane by adding the label "node-role.kubernetes.io/master=''"
[mark-control-plane] Marking the node ubuntu as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[bootstrap-token] Using token: 1rpp8b.axfud1xrsvx4q8nw
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 192.168.0.102:6443 --token 1rpp8b.axfud1xrsvx4q8nw \
    --discovery-token-ca-cert-hash sha256:6bf952d45bbdc121fa90583eac33f11f0a3f4b491f29996a56fc289363843e3c 
Copy the code

First, the K8S version was confirmed. Then you create configuration files such as certificates. Create a POD. Finally, the command to join the cluster is prompted. You are not advised to have a deep understanding of THE K8S concept during deployment. Finally, if kubeadm Join appears, the initialization is successful.

Kubeadm token create –print-join-command Kubeadm token create –print-join-command

W1221 21:23:37.632172 5479 Validation. go:28] Cannot validate kube-proxy config - no validator is available W1221 21:23:37.632503 5479 validation.go:28] Cannot validate kubelet config - no validator is available kubeadm join 192.168.0.102:6443 - token r9ip8i. 9 rs35l98nj3cquey -- -- the discovery - the token - ca - cert - hash sha256:6bf952d45bbdc121fa90583eac33f11f0a3f4b491f29996a56fc289363843e3cCopy the code

Definition: The token values are different but the hash values are the same.

Copy the admin.conf file to the corresponding directory of the current user as prompted. The admin.conf file will be used later (you need to copy it to the node).

# mkdir -p $HOME/.kube
# sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
# sudo chown $(id -u):$(id -g) $HOME/.kube/config
Copy the code

Note: $HOME is the HOME directory of a common user who switches to root. The directory must be the same as the user. For example, if the latelee user switches the root permission to run the admin.conf command, the admin.conf command must be run in the /home/latelee/. Kube directory, not the /root/. If this step is not performed, the following message is displayed when executing the kubectl command:

The connection to the server localhost:8080 was refused - did you specify the right host or port?
Copy the code

During initialization, an image is automatically downloaded if it does not exist. After initialization, the image is as follows:

# docker images REPOSITORY TAG IMAGE ID CREATED the SIZE registry.aliyuncs.com/google_containers/kube-proxy v1.17.0 13 days 7 d54289267dc line 116 MB registry.aliyuncs.com/google_containers/kube-apiserver v1.17.0 0 cae8d5cc64c 13 days a line 171 MB registry.aliyuncs.com/google_containers/kube-controller-manager v1.17.0 5 eb3b7486872 13 days line 161 MB Registry.aliyuncs.com/google_containers/kube-scheduler v1.17.0 78 c190f736b1 13 days line 94.4 MB Registry.aliyuncs.com/google_containers/coredns 1.6.5 70 f311871ae1 6 weekes line 41.6 MB Registry.aliyuncs.com/google_containers/etcd rule 3.4.3 0 303 ce5db0e90 8 weekes line 288 MB Registry.aliyuncs.com/google_containers/pause da86e6ba6ca1 3.1 2 years line under 742 KBCopy the code

The state of pod is as follows:

# kubectl get pods -n kube-system
NAME                             READY   STATUS    RESTARTS   AGE
coredns-9d85f5447-67qtv          0/1     Pending   0          3h26m
coredns-9d85f5447-cg87c          0/1     Pending   0          3h26m
etcd-ubuntu                      1/1     Running   0          3h27m
kube-apiserver-ubuntu            1/1     Running   0          3h27m
kube-controller-manager-ubuntu   1/1     Running   0          3h27m
kube-proxy-chqbq                 1/1     Running   0          3h26m
kube-scheduler-ubuntu            1/1     Running   0          3h27m
Copy the code

All pods are running except coreDNS, which is in Pending state. This is because the network plug-in is not deployed. This paper selects flannel.

3.9 the deployment of flannel

Run the following command to deploy flannel:

# kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
Copy the code

Definition: Deploy using the kube-flannel.yml file of the Flannel repository. For more information, such as the version number used, refer to this file. If not, you can manually download github.com/coreos/flan… Then run the kubectl apply -f kube-flannel.yml command.

# kubectl apply -f kube-flannel.yml
podsecuritypolicy.policy/psp.flannel.unprivileged created
clusterrole.rbac.authorization.k8s.io/flannel created
clusterrolebinding.rbac.authorization.k8s.io/flannel created
serviceaccount/flannel created
configmap/kube-flannel-cfg created
daemonset.apps/kube-flannel-ds-amd64 created
daemonset.apps/kube-flannel-ds-arm64 created
daemonset.apps/kube-flannel-ds-arm created
daemonset.apps/kube-flannel-ds-ppc64le created
daemonset.apps/kube-flannel-ds-s390x created
Copy the code

Note: You can also use Kube-flannel-aliyun. yml, which is faster, but it has a “Extensions /v1beta1” flag that is not supported in the current release (for unexplained reason). If the flannel image does not exist, the flannel image is automatically downloaded when you deploy the flannel image. As the flannel image has been downloaded, the system starts quickly. During startup, the flannel state changes as follows:

kube-flannel-ds-amd64-pjj5k  0/1  Init:0/1   0  3s
kube-flannel-ds-amd64-pjj5k  1/1  Running    0  9s
Copy the code

This step creates the CNI0 and flannel.1 network devices, which are listed later. When flannel is deployed, the coreDNS status changes as follows:

coredns-9d85f5447-67qtv  0/1 Pending            0  3h43m
coredns-9d85f5447-67qtv  0/1 ContainerCreating  0  3h43m
coredns-9d85f5447-67qtv  0/1 CrashLoopBackOff   2  3h44m
Copy the code

View the pod log:

# kubectl logs coredns-9d85f5447-67qtv -n kube-system .:53 [INFO] plugin/reload: Running configuration MD5 = 4 e235fcc3696966e76816bcd9034ebc7 CoreDNS - 1.6.5 Linux/amd64, go1.13.4, c2fd1b2 [FATAL] plugin/loop: Loop (127.0.0.1:40392 - > : 53) detected for zone ". ", see https://coredns.io/plugins/loop#troubleshooting. Query: "HINFO 1045765417707038110.4695109258766463637."Copy the code

The cause is a domain name resolution problem with COREDNS. Modify coreDNS ConfigMap:

kubectl edit cm coredns -n kube-system
Copy the code

By default, VIM editing is used to remove the line in the LOOP field (using the dd command). Enter :wq to save and exit. Coredns ConfigMap is as follows:

apiVersion: v1
data:
  Corefile: |
    .:53 {
        errors
        health {
           lameduck 5s
        }
        ready
        kubernetes cluster.local in-addr.arpa ip6.arpa {
           pods insecure
           fallthrough in-addr.arpa ip6.arpa
           ttl 30
        }
        prometheus :9153
        forward . /etc/resolv.conf
        cache 30
        loop
        reload
        loadbalance
    }
kind: ConfigMap
metadata:
  creationTimestamp: "2019-12-21T09:50:31Z"
  name: coredns
  namespace: kube-system
  resourceVersion: "171"
  selfLink: /api/v1/namespaces/kube-system/configmaps/coredns
  uid: 62485b55-3de6-4dee-b24a-8440052bdb66
Copy the code

Note: In theory, changing /etc/resolv.conf to 8.8.8.8 should be able to solve the problem. However, after the restart, the file is restored to network segment 127 and invalid. Delete the LOOP field to resolve the problem.

Remove all coreDNS that have problems:

# kubectl delete pod coredns-9d85f5447-67qtv coredns-9d85f5447-cg87c  -n kube-system
pod "coredns-9d85f5447-67qtv" deleted
pod "coredns-9d85f5447-cg87c" deleted
Copy the code

After the deletion, coreDNS automatically restarts. Look at pod again:

# kubectl get pod -n kube-system
NAME                             READY   STATUS    RESTARTS   AGE
coredns-9d85f5447-bhf24          1/1     Running   0          10s
coredns-9d85f5447-smgz9          1/1     Running   0          10s
etcd-ubuntu                      1/1     Running   0          3h58m
kube-apiserver-ubuntu            1/1     Running   0          3h58m
kube-controller-manager-ubuntu   1/1     Running   0          3h58m
kube-flannel-ds-amd64-pjj5k      1/1     Running   0          14m
kube-proxy-chqbq                 1/1     Running   0          3h57m
kube-scheduler-ubuntu            1/1     Running   0          3h58m
Copy the code

All pods are running. Note 1: You can modify the ConfigMap before deploying the Flannel. Note 2: The actual test found that it is ok not to delete, the state will automatically become Running.

The Master node has been successfully deployed.

View flannel network information:

# cat /run/flannel/subnet.env 
FLANNEL_NETWORK=10.244.0.0/16
FLANNEL_SUBNET=10.244.0.1/24
FLANNEL_MTU=1450
FLANNEL_IPMASQ=true
Copy the code

View the LOCAL IP address:

# ifconfig cNI0 Link encap:Ethernet HWaddr 3e: CE :1f:4a:67:d3 inet addr:10.244.0.1 Bcast:0.0.0.0 Mask:255.255.255.0 inet6  addr: fe80::3cce:1fff:fe4a:67d3/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1450 Metric:1 RX packets:269112 errors:0 dropped:0 overruns:0 frame:0 TX packets:303520 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX Bytes :18543053 (18.5 MB) TX bytes:87215483 (87.2MB) Docker0 Link Encap :Ethernet HWaddr 02:42: E4:10:57:4 a inet Addr: 172.17.0.1bcast :172.17.255.255 Mask:255.255.0.0 UP BROADCAST MULTICAST MTU:1500 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:0 (0.0b) TX bytes:0 (0.0b) ENS33 Link Encap :Ethernet HWaddr 00:0 C :29: F4 :c1:06 inet addr:192.168.0.102 Bcast: 192.168.0.255 Mask: 255.255.255.0 inet6 addr: fe80::49d4:fd5c:17ef:d637/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:202896 errors:0 dropped:0 overruns:0 frame:0 TX packets:183666 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX Bytes :27411397 (27.4 MB) TX bytes:183832306 (183.8 MB) Flannel.1 Link Encap :Ethernet HWaddr aa:2a:8b: E7:92:2b inet Addr :10.244.0.0 Bcast:0.0.0.0 Mask:255.255.255.255 UP BROADCAST RUNNING MULTICAST MTU:1450 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:38 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX Bytes :0 (0.0b) TX bytes:0 (0.0b)Copy the code

View the flannel network configuration:

# cat /etc/cni.net.d/10-flannel.conflist {"name": "cbr0", "cniVersion": "0.3.1", "plugins": [{"type": "flannel", "delegate": { "hairpinMode": true, "isDefaultGateway": true } }, { "type": "portmap", "capabilities": { "portMappings": true } } ] }Copy the code

(In question: the CNI version here is 0.3.1, which is inconsistent with the CNI version seen before. To study)

4. Node Indicates a node

K8s can be deployed on a master host or a node. This section describes node nodes.

4.1 Preconditions

Perform this operation on a node. 1. Install kubeadm, see above. 2. Download the flannel image. For details, see the preceding steps. 3, the host of the/etc/kubernetes/admin. Conf file copy to the node node/etc/kubernetes/directory. (note: the SCP command on the master node does not exist.)

4.2 Joining a Cluster

At this time, the K8S service has not been started. Run the following command to join a node:

Kubeadm join 192.168.0.102:6443 --token 1rpp8b.axfud1xrsvx4q8nW-discovery-token-ca-cert-hash sha256:6bf952d45bbdc121fa90583eac33f11f0a3f4b491f29996a56fc289363843e3cCopy the code

The following information is displayed:

[preflight] WARNING: JoinControlPane.controlPlane settings will be ignored when control-plane flag is not set. [preflight] Running pre-flight  checks [preflight] Reading configuration from the cluster... [preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml' [kubelet-start] Downloading Configuration for the kubelet from the "kubelet-config-1.17" ConfigMap in the kube-system namespace [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" [kubelet-start] Starting the kubelet [kubelet-start] Waiting for  the kubelet to perform the TLS Bootstrap... This node has joined the cluster: * Certificate signing request was sent to apiserver and a response was received. * The Kubelet was informed of the new secure connection details. Run 'kubectl get nodes' on the control-plane to see this node join the cluster.Copy the code

The required K8S image will be downloaded during cluster joining. Note that the master host has been designated as the ali source, so the node will also be the source.

The REPOSITORY TAG IMAGE ID CREATED the SIZE registry.aliyuncs.com/google_containers/kube-proxy v1.17.0 7 d54289267dc 2 weekes line 116 MB registry.aliyuncs.com/google_containers/coredns 1.6.5 70 f311871ae1 7 weekes line 41.6 MB quay. IO/coreos/flannel V0.11.0 - amd64 ff281650a721 11 have a line 52.6 MB registry.aliyuncs.com/google_containers/pause da86e6ba6ca1 3.1 2 years ago 742kBCopy the code

After the node is successfully added, the following services are running on the node:

# ps aux | grep kube root 3269 754668 86784 1.6 4.2? Ssl Dec20 18:34 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --cgroup-driver=cgroupfs - network - the plugin = the cni - pod-infra-container-image=registry.aliyuncs.com/google_containers/pause:3.1 root 3632 0.1 1.1 140104 22412 ? Ssl Dec20 2:14 /usr/local/bin/kube-proxy --config=/var/lib/kube-proxy/config.conf --hostname-override=node root 4385 0.0 1.6 407356 33704? Ssl Dec20 0:51 /opt/bin/flanneld --ip-masq --kube-subnet-mgr Root 121292 0.0 0.0 14228 1032 PTS /0 S+ 00:33 0:00 grep --color=auto kubeCopy the code

There are mainly Kubelet, Kube-proxy, Flanneld, etc.

The docker container list is as follows:

# Docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 2fde9bb78FD7ff281650a721 "/opt/bin/flanneld -..." 7 minutes ago Up 7 minutes k8s_kube-flannel_kube-flannel-ds-amd64-28p6z_kube-system_f40a2875-70eb-468b-827d-fcb59be3416b_1 aa7ca3d5825e Registry.aliyuncs.com/google_containers/kube-proxy "/ usr/local/bin/kube..." 8 minutes ago Up 8 minutes k8s_kube-proxy_kube-proxy-n6xv5_kube-system_3df8b7ae-e5b8-4256-9857-35bd24f7e025_0 Ac61ed8d7295 registry.aliyuncs.com/google_containers/pause:3.1 8 minutes "/ pause" line Up to 8 minutes k8s_POD_kube-flannel-ds-amd64-28p6z_kube-system_f40a2875-70eb-468b-827d-fcb59be3416b_0 423f9e42c082 Registry.aliyuncs.com/google_containers/pause:3.1 8 minutes "/ pause" line Up to 8 minutes k8s_POD_kube-proxy-n6xv5_kube-system_3df8b7ae-e5b8-4256-9857-35bd24f7e025_0Copy the code

View flannel network information:

Env FLANNEL_NETWORK=10.244.0.0/16 FLANNEL_SUBNET=10.244.1.1/24 FLANNEL_MTU=1450 FLANNEL_IPMASQ=trueCopy the code

View the LOCAL IP address:

# ifconfig cNI0 Link encap:Ethernet HWaddr 7a:0b:d3:cc:9c:c2 inet addr:10.244.1.1 Bcast:0.0.0.0 Mask:255.255.255.0 inet6  addr: fe80::780b:d3ff:fecc:9cc2/64 Scope:Link UP BROADCAST MULTICAST MTU:1500 Metric:1 RX packets:3 errors:0 dropped:0 Overruns :0 frame:0 TX packets:136 errors:0 dropped:0 Overruns :0 Carrier :0 Collisions :0 TxQueuelen :1000 RX bytes:84 (84.0 B) TX bytes:15701 (15.7KB) docker0 Link Encap :Ethernet HWaddr 02:42:2a:72:1c:91 inet addr:172.17.0.1 Bcast:172.17.255.255 Mask:255.255.0.0 UP BROADCAST MULTICAST MTU:1500 Metric:1 RX packets:0 errors:0 dropped:0 Overruns :0 frame:0 TX packets:0 errors:0 dropped:0 Overruns :0 Carrier :0 Collisions :0 TxQueuelen :0 RX bytes:0 (0.0b) TX Bytes :0 (0.0b) ens33 Link encap:Ethernet HWaddr 00:0 C :29:96:3c:7a inet addr:192.168.0.140 Bcast:192.168.0.255 bytes:0 (0.0b) ens33 Link encap:Ethernet HWaddr 00:0 C :29:96:3c:7a inet addr:192.168.0.140 Bcast:192.168.0.255 Mask: 255.255.255.0 inet6 addr: fe80::a5e3:6e8d:8330:34db/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:815745 errors:0 dropped:0 overruns:0 frame:0 TX packets:280001 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX Bytes :884087449 (884.0 MB) TX bytes:25231812 (25.2 MB) Flannel.1 Link Encap :Ethernet HWaddr AE :98: AE :9b: AE :ef inet Addr :10.244.1.0 Bcast:0.0.0.0 Mask:255.255.255.255 UP BROADCAST RUNNING MULTICAST MTU:1450 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:60 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX Bytes :0 (0.0b) TX bytes:0 (0.0b)Copy the code

Five, validation,

Execute on the master node:

# kubectl get Nodes NAME STATUS ROLES AGE VERSION node Ready < None > 17m v1.17.0 Ubuntu Ready Master 5h11m v1.17.0Copy the code

The two machines are in the Ready state. Node It takes about 10 seconds for a machine to change from NotReady to Ready.

Simple test pod using busyBox image. Execute on the master node:

# kubectl run -i --tty busybox --image=latelee/busybox --restart=Never -- sh
Copy the code

Wait for a moment to go to the busybox command line:

If you don't see a command prompt, Try pressing enter. / # / # / # pressing -a Linux busybox 4.4.0-21-generic #37-Ubuntu SMP Mon Apr 18 18:33:37 UTC 2016 x86_64 GNU/LinuxCopy the code

On the other command line, check the running status of pod:

# kubectl get pod -o wide 
NAME      READY   STATUS    RESTARTS   AGE   IP           NODE   NOMINATED NODE   READINESS GATES
busybox   1/1     Running   0          74s   10.244.1.4   node   <none>           <none>
Copy the code

You can see that pod is in the Running state, Running on node. View it on node:

# docker ps | grep busybox ba5d1a480294 latelee/busybox "sh" 2 minutes ago Up 2 minutes k8s_busybox_busybox_default_20d757f7-8ea7-4e51-93fc-514029065a59_0 8c643171ac09 Registry.aliyuncs.com/google_containers/pause:3.1 "/ pause" 2 minutes line Up to 2 minutes k8s_POD_busybox_default_20d757f7-8ea7-4e51-93fc-514029065a59_0Copy the code

When the master node exits from BusyBox, the Pod still exists, but it is not READY, and the Node host has no BusyBox container running.

If the verification succeeds, the K8S deployment succeeds.

Sixth, other

6.1 reset k8s

root@ubuntu:~/k8s# kubeadm reset [reset] Reading configuration from the cluster... [reset] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml' [reset] WARNING: Changes made to this host by 'kubeadm init' or 'kubeadm join' will be reverted. [reset] Are you sure you want to proceed? [y/N]: y // !!! Y [preflight] Running pre-flight checks [reset] Removing info for node "ubuntu" from the ConfigMap "kubeadm-config" in The "kube-system" Namespace W1215 11:50:28.154924 6848 removeetcdmember.go:61] [reset] failed to removeetcdmember: error syncing endpoints with etc: etcdclient: no available endpoints .Please manually remove this etcd member using etcdctl [reset] Stopping the kubelet service [reset] Unmounting mounted directories in "/var/lib/kubelet" [reset] Deleting contents of config directories: [/etc/kubernetes/manifests /etc/kubernetes/pki] [reset] Deleting files: [/etc/kubernetes/admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/bootstrap-kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf] [reset] Deleting contents of stateful directories: [/var/lib/etcd /var/lib/kubelet /var/lib/dockershim /var/run/kubernetes /var/lib/cni] The reset process does not clean CNI configuration. To do so, you must remove /etc/cni/net.d The reset process does not reset or clean up iptables rules or IPVS tables. If you wish to reset iptables, you must do so manually by using the "iptables" command. If your cluster was setup to utilize IPVS, run ipvsadm --clear (or similar) to reset your system's IPVS tables. The reset process does not clean your kubeconfig files and you must remove them manually. Please, check the contents of the $HOME/.kube/config file.Copy the code

Run the following commands to clear the directory and delete the network device:

rm -rf $HOME/.kube/* 
rm -rf /var/lib/cni/
rm -rf /var/lib/kubelet/*
rm -rf /etc/kubernetes/
rm -rf /etc/cni/
ifconfig cni0 down
ifconfig flannel.1 down
ip link delete cni0
ip link delete flannel.1

Copy the code

6.2 The Node Exits

Run the following command on master:

# Kubectl drain Node /node Cordoned evicting Pod "Busybox" // note: Pod /busybox evicted node/node evictedCopy the code

At this time prompt:

# kubectl get Nodes NAME STATUS ROLES AGE VERSION node Ready,SchedulingDisabled < None > 18h v1.17.0 Ubuntu Ready Master 23 h v1.17.0Copy the code

The node has become unschedulable, but remains Ready (because it was in the Ready state). Disable the use of this node. 2. Delete a node:

# kubectl delete node node
node "node" deleted
Copy the code

The second node is the node name. Check whether there is no node.

Wait for a moment, the kube-proxy of the node system is not running (and can check the flannel, which also stops) :

# ps aux | grep kube root 3269 754668 88712 1.6 4.3? Ssl Dec20 18:54 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --cgroup-driver=cgroupfs - network - the plugin = the cni - pod-infra-container-image=registry.aliyuncs.com/google_containers/pause:3.1 root 124216 0.0 0.0 14228 964 pts/0 R+ 00:49 0:00 grep --color=auto kubeCopy the code

Execute on node:

# kubeadm reset // Enter y to confirmCopy the code

Run the following commands to clear the directory and delete the network device (similar to but different from the master) :

ifconfig cni0 down
ip link delete cni0
ifconfig flannel.1 down
ip link delete flannel.1
rm /var/lib/cni/ -rf
rm /etc/cni/net.d/ -rf
rm /etc/kubernetes/ -rf
rm /var/lib/kubelet/ -rf

Copy the code

Note: I have yet to find a way to gracefully exit a node.

note

In practice, the author found that the first to join the cluster, then exit, delete the configuration of the node, and then re-join, before and after several attempts, all unsuccessful. Then create a Linux VM, change docker, change the host name, download Kubeadm and flannel from the beginning, add admin.conf, and finally join the cluster. (Note: the reason is unknown and will be reviewed for further understanding). K8s requires a large number of operations. You can use the VM snapshot function to save the steps step by step to reduce the repetitive workload.

The resources

During the deployment of this article, refer to the following articles and adjust according to the actual situation:

Juejin. Cn/post / 684490…
zhuanlan.zhihu.com/p/46341911
Kubernetes. IO/docs/setup /… (official)
Calico Canal related: github.com/projectcali…