Kubernetesv1.17 Cluster Ecosystem Setup Notes

preface

The last time I contacted kubernetes cluster was a year ago, at that time the official version was only V1.10, and now a year has passed, the official version has been rapidly iterated to V1.17, the community is becoming more and more mature, related ecological components are becoming more and more rich, it can be seen in the past K8S first year, How rapidly it has developed. Recently, I wanted to wrap some small things I wrote into open APIS and expose them, so I wanted to put my machines into a Kubernetes cluster, so I want to revisit the process of cluster building. All the following documents can be find in Githubhttps://github.com/lateautumn4lin/KubernetesResearch ClusterEcology directory

Kubernetes cluster construction actual combat

The first thing to do is set up a basic Kubernetes cluster.

Preparation stage

The preparation stage mainly includes two aspects: one is to prepare at least two machines to build the master-worker cluster architecture; the other is to know what software we need to install to build the most basic cluster.

1. Machine configuration

In this experiment, I chose the cloud server CVM of Tencent Cloud. Since I bought it through different accounts, the machines I chose were interconnected through the Internet. In terms of configuration, I chose the default minimum configuration of 2 cores and 4GB.

Server IP	CPU	memory	The hard disk	System version	Hostname
192.144.152.23	2 the nuclear	4GB	50GB	Cent OS 7.6	master1
49.233.81.20	2 the nuclear	4GB	50GB	Cent OS 7.6	worker1

2. Configure software

Software to be installed	version
Kubernetes	V1.17. X
Docker	18.09.7

Basic software we need to install Kubernetes and Docker, install Kubernetes we need to use Kubeadm and Kubectl tools, Kubeadm is the official recommended initial chemical tool, It is available as a General Availability (GA) in V1.13, meaning it can be used in production environments. The reason why we need Docker is that the Pod in Kubernetes needs to use CRI (Container Runtime), which is a very standard and universal CRI. Others include Containerd, Cri-O, and after v1.14 if you have multiple CRis on your machine, Kubernetes will also use Docker by default, so we’ll use Docker instead.

Check and configure phase

At this stage we mainly check our server configuration and connect our several servers together.

1. Modify Hostname and configure the Host file

Hostnamectl is used to permanently modify the hostname of worker1 and Master1 respectively, and to configure the host. The reason for doing this is that we need to mark each machine uniformly, so that we can better understand the function of each machine through hostname in the later cluster management.

hostnamectl set-hostname master1
echo "127.0.0.1 $(hostname)" >> /etc/hosts
echo "192.144.152.23 master1." " >> /etc/hosts
echo "49.233.81.20 worker1" >> /etc/hosts
Copy the code

2. Check the number of CPU cores and memory

In this step, we use lscpu command to check the architecture of our server and the number of cores in our system, because we want to build a Kubernetes cluster, the master node should not be less than 2 cores, which must be guaranteed, if the number of cores is too low, the whole cluster will be unstable and high latency.

Architecture: x86_64 This installation document does not support the ARM Architecture. CPU(s): 2 The number of CPU cores cannot be less than 2Copy the code

3. Check the network

Run the command on all nodes

[root@master1 ~]# IP route show default via 172.21.0.1 dev eth0 169.254.0.0/16 dev eth0 scope link metric 1002 172.21.0.0/20 dev eth0 proto kernel scope link SRC 172.21.0.11 [root@master1 ~]# IP address 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host LO valid_lft forever preferred_lft forever Inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000 link/ether 52:54:00:2c:42:7d brd Ff :ff:ff:ff:ff:ff :ff inet 172.21.0.11/20 BRD 172.21.15.255 scope global eth0 valid_lft forever preferred_lft forever inet6 fe80::5054:ff:fe2c:427d/64 scope link valid_lft forever preferred_lft foreverCopy the code

3.1 IP address used by Kubelet

• In the IP route show command, you can know the default network adapter of the machineeth0, such asDefault via 172.21.0.1 dev eth0
• The IP address command displays the IP address of the default network card, which Kubernetes will use to communicate with other nodes in the cluster, such as172.21.0.11
• IP addresses used by Kubernetes on all nodes must be interoperable (no NAT mapping, no security group or firewall isolation)

If the two machines are in the same Intranet, they can use the Intranet IP to communicate directly. However, our machines are in two different Tencent cloud accounts, which are isolated from each other on the Intranet. Therefore, we directly use the external IP of the machines to communicate, which is not recommended to use in the production environment.

4. Configure login exemption between devices

In this step, we need to configure the no-login between the machines to get through each machine and connect each machine in series, which is convenient for us to operate between each machine later.

4.1 Generate public and private keys for each server

SSH - the keygen - t rsaCopy the code

4.2 Adding id_rsa.pub to the authorized key

cat id_rsa.pub >> authorized_keys
Copy the code

4.3 Change the permission of the. SSH folder and its files and restart the SSH service

chmod 700 ~/.ssh
chmod 600 ~/.ssh/*
service sshd restart
Copy the code

4.4 Copy the id_rsa, id_rsa.pub, and authorized_keys files in the. SSH folder to the target server

scp ~/.ssh/authorized_keys root@192.144152.23.: ~/.ssh/
scp ~/.ssh/id* root@192.144152.23.: ~/.ssh/
Copy the code

This section describes how to configure login exemption between two machines. The same method can be used for other two machines.

Formal installation stage

After getting the above machine ready and checking and configuring all the parameters, we can start the formal installation.

1. Install Kubelet and Docker

Switch to the ClusterEcology directory and you can see the install_kubelet.sh script for a quick installation using the following command.

cat install_kubelet.sh | sh -s 1.172.
Copy the code

Let’s take a quick look at the code in this script to see what each step does

#! / bin/bash # on the master node and the worker nodes to perform the following # https://docs.docker.com/install/linux/docker-ce/centos/ # # # installation docker's reference https://docs.docker.com/install/linux/linux-postinstall/ # uninstall the old version yum remove - y docker \ docker - client \ docker-client-latest \ docker-common \ docker-latest \ docker-latest-logrotate \ docker-logrotate \ docker-selinux \ Docker-engine-selinux \ docker-engine # configure yum repository yum install -y yum-utils \ device-mapper-persistent-data \ Lvm2 yum - config - manager - add - 'http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo # installed and started docker yum IO systemctl enable docker systemctl start docker # install Yum install -y nfs-utils yum install -y wget # Disable firewall systemctl stop firewalld SeLinux setenforce 0 sed -i "s/ SeLinux =enforcing/ SeLinux =disabled/g" The/etc/selinux/config # close swap swapoff - a yes | cp/etc/fstab/etc/fstab_bak cat/etc/fstab_bak | grep -v swap > /etc/fstab # modify /etc/sysctl.conf Sed -i "s#^net.ipv4.ip_forward.*#net.ipv4.ip_forward=1#g" /etc/sysctl.conf sed -i "s#^net.bridge.bridge-nf-call-ip6tables.*#net.bridge.bridge-nf-call-ip6tables=1#g" /etc/sysctl.conf sed -i Bridge. Bridge -nf-call-iptables=1#g" /etc/sysctl.conf Add echo "net.ipv4.ip_forward = 1" >> /etc/sysctl.conf echo "net.bridge.bridge-nf-call-ip6tables = 1" >> /etc/sysctl.conf Bridge -nf-call-iptables = 1" >> /etc/sysctl.conf /etc/yum.repos.d/kubernetes.repo [kubernetes] name=Kubernetes baseurl=http://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64 enabled=1 gpgcheck=0 repo_gpgcheck=0 gpgkey=http://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg http://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg EOF # uninstall the old version yum remove - y kubelet kubeadm kubectl # Kubectl yum install -y kubelet-${1} kubeadm-${1} kubectl-${1} # The/usr/lib/systemd/system/docker. This line of service files ExecStart = / usr/bin/dockerd -h fd: / / -- containerd = / run/containerd containerd. The sock # # modified for ExecStart = / usr/bin/dockerd -h fd: / / -- containerd = / run/containerd containerd. The sock - exec - opt. Native cgroupdriver = systemd # if you don't change, # [WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". # Please follow the guide at https://kubernetes.io/docs/setup/cri/ sed -i "s#^ExecStart=/usr/bin/dockerd.*#ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock --exec-opt native.cgroupdriver=systemd#g" / usr/lib/systemd/system/docker. Service # set the docker image, improve the docker image stability # if you visit https://hub.docker.io and download speed is very stable, Also can skip this step curl - sSL https://registry.cn-hangzhou.aliyuncs.com # https://kuboard.cn/install-script/set_mirror.sh | sh - s Restart the docker, -reload systemctl restart docker systemctl enable kubelet && systemctl start kubelet docker versionCopy the code

After executing the above command, we will get the version information of the Docker if it is executed correctly

Client:
 Version:           18.097.
 API version:       1.39
 Go version:        go110.8.
 Git commit:        2d0083d
 Built:             Thu Jun 27 17:56:06 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          18.097.
  API version:      1.39 (minimum version 1.12)
  Go version:       go110.8.
  Git commit:       2d0083d
  Built:            Thu Jun 27 17:26:28 2019
  OS/Arch:          linux/amd64
  Experimental:     false
Copy the code

2. Initialize the master node

Switch to the ClusterEcology directory and you can see the init_master.sh script, where we first configure the environment variables and then use the execution script to quickly install.

# export only applies to the current shell session. After a new shell window is opened, if you want to continue the installation process, Please re-run the export command here export MASTER_IP=192.144.152.23 # to replace apiserver.demo with the dnsName export you want APISERVER_NAME=apiserver.demo # Kubernetes network segment where the container group resides. Does not exist in your physical network in advance of the export POD_SUBNET = 10.100.0.1/16 echo "${MASTER_IP} ${APISERVER_NAME}" > > / etc/hosts cat init_master. Sh | Sh - s 1.17.2Copy the code

3. Check the master initialization result

To verify the Kubernetes cluster after the previous step is installed, follow the command below.

Run the following command on the master node only and wait 3 to 10 minutes. Kubectl get pod -n kube-system -o wide watch kubectl get pod -n kube-system -o wideCopy the code

If successful, you can see the following output

[root@master1 dashboard]# kubectl get nodes -o wide NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE Kernel-version container-Runtime master1 Ready master 7h26m v1.17.2 172.21.0.11 <none> CentOS Linux 7 (Core) 3.10.0-862. El7. X86_64 docker: / / 18.9.7Copy the code

Indicates that the master node in our cluster is officially available

4. Obtain parameters of the join command

Next, we need to add the rest of our worker nodes to the cluster. To join the cluster, workers need to obtain the token and CA certificate of the whole cluster. We need to obtain the token and CA certificate from the master node first, including the joining token and CA certificate.

Kubeadm token create --print-join-command Kubeadm join apiserver.demo:6443 --token mpfjma.4vjjg8flqihor4vt --discovery-token-ca-cert-hash sha256:6f7a8e40a810323672de5eee6f4d19aa2dbdb38411845a1bf5dd63485c43d303Copy the code

5. Initialize the worker node

Execute for all worker nodes

# export only applies to the current shell session. After a new shell window is opened, if you want to continue the installation process, Please re-run the export command here export MASTER_IP=192.144.152.23 # to replace apiserver.demo with the dnsName export you want Demo echo "${MASTER_IP} ${APISERVER_NAME}" >> /etc/hosts # Replace kubeadm token create on the master node Kubeadm join apiserver.demo:6443 --token mpfjma.4vjjg8flqihor4vt --discovery-token-ca-cert-hash sha256:6f7a8e40a810323672de5eee6f4d19aa2dbdb38411845a1bf5dd63485c43d303Copy the code

After joining, we can use the following command to check whether the worker joins the cluster correctly

kubectl get nodes
Copy the code

6. Check the status of the cluster

You can only run the following command on the master node to check cluster status

kubectl get nodes -o wide
Copy the code

You can see the following output, where clusters are all Ready to indicate that nodes are available

[root@master1 ~]# kubectl get nodes NAME STATUS ROLES AGE VERSION master1 Ready master 5m3s v1.17.2 worker1 Ready <none> 2 m26s v1.17.2Copy the code

Ecological component building

After installing the Kubernetes cluster, we have a very basic cluster. There are a lot of problems that we have not solved. For example, we want to operate the whole cluster visually by clicking on the page. Or we want a package management tool like PIP in Python to manage our applications deployed in the Kubernetes cluster, or we want easy communication between our cluster and the Internet, etc., So it’s up to us to leverage other components to improve our Kubernetes ecosystem.

We choose the following software

Software to be installed	version
Kubernetes Dashboard	v2.0.3
Helm	v3.0.3
Traefik	xxx
metrics-server	xxx

1. The Dashboard plugins

Install Dashboard because it is a comprehensive management platform, is also belongs to the official project of Kubernetes, concrete can be in the warehouse to see https://github.com/kubernetes/dashboard, Although Dashboard was widely criticized for its inhumanized operation and ugly interface before, Dashboard 2.0 version has been newly released after more than half a year of closed research and development by Kubernetes Dashboard team, and its interface and operability have been greatly improved. Dashboard 2.0 is also one of the official recommended management interfaces.

1.1 Installing the Dashboard plug-in

Specific deployment plan together a few Yaml I according to the official documents, projects are under the directory ClusterEcology/InitDashboard

kubectl apply -f k8s-dashboard-rbac.yaml
kubectl apply -f k8s-dashboard-configmap-secret.yaml
kubectl apply -f k8s-dashboard-deploy.yaml
kubectl apply -f k8s-dashboard-metrics.yaml
kubectl apply -f k8s-dashboard-token.yaml
Copy the code

After executing the above commands, the Dashboard services and users are basically created. Now we need to obtain user tokens to log in to our Dashboard

1.2 Obtaining a User Token

kubectl describe secret/$(kubectl get secret -n kube-system |grep admin|awk '{print $1}') -n kube-system
Copy the code

1.3 Checking whether services are available

In the previous Yaml file, the NodePort port is set to 31001 and the type is set to NodePort to access Dashboard, so the access address is: https://192.144.152.23:31001/ into Kubernetes Dashboard page, then enter the Token ServiceAccount created in the previous step into the Dashboard, you can see the new Dashboard.

2. Helm components

The Helm component was also born out of a key pain point: although we had deployed the Kubernetes cluster environment, each microservice had to maintain a set of Yaml files, and the configuration files in each environment were different, so the cost of redeploying a new environment or migrating the environment was really high. Wouldn’t it be easier if we could use a tool like YUM to install our applications? From this point, Helm was born, giving the Kubernetes cluster a formal application market.

The Helm of the old version is divided into two parts, including the Helm Client and Tiller Server. The Helm Client is a user command line tool that manages customized package files. The Tiller Server service accepts the Client’s request and interacts with the Kubernetes cluster after parsing the request.

In the new version, that is Helm3, Tiller component is removed from Helm. Using Helm command, kubeconFig is directly used to communicate with Kubernetes cluster, so that more fine-grained permission control can be done, which is convenient to complete and use. Another benefit is that the Release name is narrowed down to Namespace, which ensures that different namespaces can use the same Release name.

2.1 Installing the Helm Client

We first need to go to the website https://github.com/kubernetes/helm/releases to download the Helm of the package.

cp helm /usr/local/bin/
Copy the code

2.2 Helm use

Initialize the Helm

helm init --client-only --stable-repo-url https://aliacs-app-catalog.oss-cn-hangzhou.aliyuncs.com/charts/
helm repo add incubator https://aliacs-app-catalog.oss-cn-hangzhou.aliyuncs.com/charts-incubator/
helm repo add seldon https://storage.googleapis.com/seldon-charts
helm repo update
Copy the code

Install a simple service

helm install seldon-core seldon/seldon-core-operator
Copy the code

You can see that Helm is already interacting with the Kubernetes cluster to generate a Seldon-core service

[root@master1 linux-amd64]# helm install seldon-core seldon/seldon-core-operator
NAME: seldon-core
LAST DEPLOYED: Tue Feb  4 22:43:58 2020
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None
[root@master1 linux-amd64]# helm list 
NAME       	NAMESPACE	REVISION	UPDATED                                	STATUS  	CHART                     	APP VERSION
seldon-core	default  	1       	2020-02-04 22:43:58.232906547 +0800 CST	deployed	seldon-core-operator-1.0.1	
Copy the code

3. Traefik components

3.1 Traefik Installation

Traefik is another Kubernetes cluster must have component, can be considered as a replacement for Nginx, as a unified gateway management tool, it has several advantages, such as a beautiful dashboard interface, can be configured based on container tag, easy to add new services, Nginx is not as complex as configuration, and does not need to restart frequently, etc., although performance and nginx will be a bit different, but as a personal use, it is still very pleasing to put down.

4. The metrics – server plug-ins

Metrics – Server is the official Kubernetes cluster resource utilization information collector, a streamlined replacement for Heapster. Metrics-server collects utilization information from Kubelet on each node in the cluster, which is the basic monitoring information of the cluster. It is mainly used by core systems such as scheduling logic.

Git clone https://github.com/kubernetes-incubator/metrics-server.git CD metrics - server/kubectl create -f deploy / + / 1.8Copy the code

After the installation is successful, you can see the specific monitoring information in the Dashboard after some time

Troubleshooting analysis

There are several types of Kubernetes cluster troubleshooting: (1) resource scheduling class (2) network communication class (3) configuration parameter class most of the problems are around the three points to carry out (not all, big guys do not jet), the following list of my installation of some problems, some problems in the installation did not involve, so later involved in the words will explain.

1. The node cannot be scheduled

We encounter the following problem during installation

1 node(s) had taints that the pod didn't tolerate
Copy the code

This means that a node is marked as unschedulable. This is the official default of K8S, because it ensures that the Master node will not be scheduled to additional containers and thus consume resources. However, in this experiment, we can set all nodes to allow scheduling to avoid this problem.

kubectl taint nodes --all node-role.kubernetes.io/master-
Copy the code

2. Mirroring problems

According to the installation of the above steps can be completely theoretically correct deployment good K8S cluster, but the installation will be depending on the speed difference, when I was in the installation and installed more than an hour, reason is because the image download slow, when we see some pod has been pending, we can use the following command to see specific situation.

kubectl describe pod calico-node-ndwqv -n kube-system
Copy the code

Use the Describe command to see how specific components are doing, although you can also use the logs command, which is not as convenient as Describe.

3. Chrome Your connection is not private

After creating Dashboard, the first time we log in to Dashboard through Chrome, we will find this error. Your connection is not private, this is caused by the error of the latest version of Chrome, we just need to modify the startup parameters.