Background: Applications are mostly large monomer used in the past, his long development cycle, it is ok to run on several machines, operations staff deployment is not frequently, monitoring difficult to also have no so big, but still facing all kinds of environmental problems, sometimes there is no problem in the test environment, production has all sorts of wonderful work on the problem, this move d personnel is a headache for delivery. The most headaches is with the rise of the service concept, service was dismantled into one large monomer small and independent service component, which makes the development iterations become more quickly, the service quantity is more and more, for a component configuration, deployment, monitoring, troubleshooting and so on a series of problems are from the original into several times, it is the disaster operations. Kubernetes came into being to solve this problem.

Environment set up

Environment to prepare

  • Number of cpus: The number of CPU cores on each node must be greater than or equal to two. Otherwise, the node cannot be started.

  • DNS network: Set the DNS that is locally connected. Otherwise, the image cannot be downloaded.

  • Linux kernel: The Linux kernel must be later than version 4.

    Prepare three virtual machines or cloud servers (but the price of the server is too high to be affordable, you can use VMware VIRTUAL machines to build clusters. If you do not know how to use VMware to build clusters, you can see my previous VMware cluster building content in advance.)

Environment depends on

#1. Give each machine a hostname for convenience
hostnamectl set-hostname k8s-master01
hostnamectl set-hostname k8s-node01
hostnamectl set-hostname k8s-node02
#Viewing host Names
hostname
#Configure IP host mapping for each machineVi /etc/hosts 192.168.70.128k8s-master01 192.168.70.129k8s-node01 192.168.70.130k8s-node02
#2, install the dependency environment, note: each machine needs to install this dependency environment
yum install -y conntrack ntpdate ntp ipvsadm ipset jq iptables curl sysstat libseccomp wget vim net-tools git iproute lrzsz bash-completion tree bridge-utils unzip bind-utils gcc

#3. Install iptables, start iptables, set the iptables rule to boot, clear the iptables rule, and save the current rule to the default rule
#Disabling the Firewall
systemctl stop firewalld && systemctl disable firewalld
#Empty the iptables 
yum -y install iptables-services && systemctl start iptables && systemctl enable iptables && iptables -F && service iptables save

#4. Close Selinux
#Close swap and permanently close virtual memory
swapoff -a && sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab
#Close the selinux
setenforce 0 && sed -i 's/^SELINUX=.*/SELINUX=disabled/' /etc/selinux/config

#5. Upgrade the Linux kernel to 4 or higherThe RPM - Uvh http://www.elrepo.org/elrepo-release-7.0-4.el7.elrepo.noarch.rpm#Install the kernel
yum --enablerepo=elrepo-kernel install -y kernel-lt
#Setup boot from the new kernelGrub2-set-default 'CentOS Linux (5.4.111-1.el7.elrebo.x86_64) 7 (Core)'#Note: after setting the kernel, you need to restart the server to take effect!!
reboot
#Query the kernel
uname -r
#6. Adjust kernel parameters for K8S
cat > kubernetes.conf <<EOF
net.bridge.bridge-nf-call-iptables=1
net.bridge.bridge-nf-call-ip6tables=1
net.ipv4.ip_forward=1
vm.swappiness=0
vm.overcommit_memory=1
vm.panic_on_oom=0
fs.inotify.max_user_instances=8192
fs.inotify.max_user_watches=1048576
fs.file-max=52706963
fs.nr_open=52706963
net.ipv6.conf.all.disable_ipv6=1
net.netfilter.nf_conntrack_max=2310720
EOF

#Copy the optimized kernel file to /etc/sysctl.d/ so that the optimized file can be called upon startup
cp kubernetes.conf /etc/sysctl.d/kubernetes.conf
#Manually refresh the optimized file to take effect immediately
sysctl -p /etc/sysctl.d/kubernetes.conf

Copy the code

Error message:

#If a manual refresh is performedSysctl: always stat/proc/sys/net/netfilter/nf_conntrack_max: don't have that file or directory#It is possible that Conntrack is not loaded
#Run the following command
lsmod |grep conntrack
modprobe ip_conntrack
#Manual flush discovery is done again
Copy the code
#7. Stop unnecessary services
systemctl stop postfix && systemctl disable postfix
#8. Set the log saving mode
#1). Create a directory to save logs
mkdir /var/log/journal
#2). Create a directory for storing configuration files
mkdir /etc/systemd/journald.conf.d
#3). Create a configuration file
cat > /etc/systemd/journald.conf.d/99-prophet.conf <<EOF
[Journal]
Storage=persistent
Compress=yes
SyncIntervalSec=5m
RateLimitInterval=30s
RateLimitBurst=1000
SystemMaxUse=10G
SystemMaxFileSize=200M
MaxRetentionSec=2week
ForwardToSyslog=no
EOF
#4). Restart systemd Journald configuration
systemctl restart systemd-journald

#9. Kube-proxy Enable ipvS preconditions
modprobe br_netfilter

cat > /etc/sysconfig/modules/ipvs.modules <<EOF
#! /bin/bash
modprobe -- ip_vs
modprobe -- ip_vs_rr
modprobe -- ip_vs_wrr
modprobe -- ip_vs_sh
modprobe -- nf_conntrack
EOF

#Use the lsmod command to see if these files are booted 
chmod 755 /etc/sysconfig/modules/ipvs.modules && bash /etc/sysconfig/modules/ipvs.modules && lsmod | grep -e ip_vs -e nf_conntrack

Copy the code

Deploy the docker

#1. Install Docker
yum install -y yum-utils device-mapper-persistent-data lvm2

#A stable repository is then configured, and the repository configuration is saved to /etc/ymp.repos. D /docker-ce.repo
yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo

#Install Docker CE for Yum
yum update -y && yum install docker-ce -y

#2. Set up the Docker Daemon file
#Create the /etc/docker directory
mkdir /etc/docker -p
#Update daemon.json file
cat > /etc/docker/daemon.json <<EOF
{
"registry-mirrors":["https://pee6w651.mirror.aliyuncs.com"]
}
EOF
#Journalctl -amu docker command: journalctl -amu docker command: journalctl -amu docker

#Create and store the Docker configuration file
mkdir -p /etc/systemd/system/docker.service.d

#3. Reload and restart the Docker service to enable self-start
systemctl daemon-reload && systemctl restart docker && systemctl enable docker

Copy the code

Kubeadm [one-click install k8S]

#1, install kubernetes, need to install kubelet, kubeadm package, but k8s website to yum source is packages.cloud.google.com, access to domestic no, at this time we can use ali cloud yum warehouse mirror.
cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=http://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=0
repo_gpgcheck=0
gpgkey=http://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg 
 http://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF

#2, install kubeadm, kubelet, kubectl (version this thing does not affect our learning, 1.15.1 is enough for learning)Yum install -y kubeadm-1.15.1 kubelet-1.15.1 kubectl-1.15.1#Start the kubelet
systemctl enable kubelet && systemctl start kubelet
#If the installation is successful, run the following command to check whether the installation is successful
kubectl --help 
Copy the code

Build a cluster

Dependent Image Import

#View the images we need for the installation
kubeadm config images list
#The output result is as followsK8s. GCR. IO/kube - apiserver: v1.15.1 k8s. GCR. IO/kube - controller - manager: v1.15.1 k8s. GCR. IO/kube - the scheduler: v1.15.1 K8s. GCR. IO/kube - proxy: v1.15.1 k8s. GCR. IO/pause: 3.1 k8s. GCR. IO/etcd: 3.3.10 k8s. GCR. IO/coredns: 1.3.1
#You can use the exported tar, it took me a long time to find this version so you can just use mine.

Copy the code

You can use him to download my extraction code: ZJDK to download the corresponding image, under the complete upload to the server /opt directory

Write scripts to import; [You can enter opt/ K8s to import one by one]

#! /bin/bashls /opt/kubeadm-basic.images > /tmp/images-list.txt cd /opt/kubeadm-basic.images for i in $(cat /tmp/images-list.txt) do  docker load -i $i done rm -rf /tmp/images-list.txtCopy the code
#Example Modify the script execution permission
chmod 755 image-load.sh 
#Execute the script
./image-load.sh
#Transfer files and images to other nodes
scp -r image-load.sh kubeadm-basic.images root@k8s-node01:/opt/ 
scp -r image-load.sh kubeadm-basic.images root@k8s-node02:/opt/ 
Copy the code

K8s deployment

#Initializing the primary node is performed only on the primary node.
#1. Remove the image resource configuration file
kubeadm config print init-defaults > kubeadm-config.yaml
#2. Modify the yamL resource fileapiVersion: kubeadm.k8s.io/v1beta2 bootstrapTokens: - groups: - system:bootstrappers:kubeadm:default-node-token token: abcdef.0123456789abcdef ttl: 24h0m0s usages: - signing - authentication kind: InitConfiguration localAPIEndpoint: AdvertiseAddress: 192.168.70.128 # Here use your own IP bindPort: 6443 nodeRegistration: criSocket: /var/run/dockershim.sock name: k8s-master01 taints: - effect: NoSchedule key: node-role.kubernetes.io/master --- apiServer: timeoutForControlPlane: 4m0s apiVersion: kubeadm.k8s.io/v1beta2 certificatesDir: /etc/kubernetes/pki clusterName: kubernetes controllerManager: {} dns: type: CoreDNS etcd: local: dataDir: /var/lib/etcd imageRepository: k8s.gcr.io kind: ClusterConfiguration kubernetesVersion: V1.15.1 # Use your own version here networking: dnsDomain: cluster.local podSubnet: 10.244.0.0/16 # Add flannel model communication here fixed. ServiceSubnet: 10.96.0.0/12 scheduler: {} # below specified in the insert ipvs network communication - apiVersion: kubeproxy. Config. K8s. IO/v1alpha1 kind: kubeProxyConfiguration featureGates: SupportIPVSProxyMode: true mode: ipvs
#3. Initialize the active node and start deployment
kubeadm init --config=kubeadm-config.yaml --experimental-upload-certs | tee kubeadm-init.log


Copy the code

Here are some common mistakes

#The following error will be reported if you have a single CPU, which is why you want to use a 2cpu vm in the first place
[ERROR NumCPU]: the number of available CPUs 1 is less than the required 2

#Docker will also get an error if you forget to start the serviceerror execution phase preflight: [preflight] Some fatal errors occurred: [ERROR CRI]: container runtime is not running: output: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running? , error: exit status 1 [ERROR Service-Docker]: docker service is not active, please run 'systemctl start docker.service' [ERROR IsDockerSystemdCheck]: cannot execute 'docker info': exit status 1 [ERROR SystemVerification]: failed to get docker info: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running? [preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=... `Copy the code

A successful build will show up as the duck below

[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace [addons] Applied essential addon:  CoreDNS [addons] Applied essential addon: kube-proxy Your Kubernetes control-plane has initialized successfully! To start using your cluster, you need To run the following as a regular user: Mkdir -p $HOME /. Kube # below three is required us to manually create sudo cp - I/etc/kubernetes/admin. Conf. $HOME/kube/config sudo chown $(id - u) : $(id -g) $HOME/.kube/config You should now deploy a pod network to the cluster. Run "kubectl apply -f [podnetwork].yaml" with  one of the options listed at: https://kubernetes.io/docs/concepts/cluster-administration/addons/ Then you can join any number of worker nodes by running the following on each as root: Kubeadm join 192.168.70.128:6443 --token abcdef.0123456789abcdef \ --discovery-token-ca-cert-hash sha256:a33be40c0ef657e7565b399e7b44ad27bdf6477fd54047c70fdd8e4ee24c2850Copy the code

Manual initialization is performed

#Create a directory, save the connection configuration cache, and authenticate files
mkdir -p $HOME/.kube
#Copy the cluster management configuration file
cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
#Authorization to the configuration file
chown $(id -u):$(id -g) $HOME/.kube/config
Copy the code

Query node information before running the command

kubectl get node
#The following information is displayed but status is notReady because flannel is not yet installed
NAME           STATUS     ROLES    AGE   VERSION
k8s-master01   NotReady    master   63m   v1.15.1
Copy the code

Install the Flannel plug-in

#Kube-flannel. yml file downloadhttps://pan.baidu.com/s/1w4IUIM7x2k85Lew-qTGY8g Numbers [ZJDK]#The Flannel deployment starts
kubectl apply -f kube-flannel.yml
#Check whether the system Po is started properly
kubectl get po -n kube-system
#It shows the duck below
NAME                                   READY   STATUS                  RESTARTS   AGE
coredns-5c98db65d4-p8x8v               0/1     Pending                 0          49m
coredns-5c98db65d4-ts4qm               0/1     Pending                 0          49m
etcd-k8s-master01                      1/1     Running                 1          48m
kube-apiserver-k8s-master01            1/1     Running                 1          49m
kube-controller-manager-k8s-master01   1/1     Running                 1          49m
kube-flannel-ds-amd64-mwbcq            0/1     Init:ImagePullBackOff   0          15m
kube-proxy-hhvqh                       1/1     Running                 1          49m
kube-scheduler-k8s-master01            1/1     Running                 1          49m
#ImagePullBackOff is displayed when some Po is not pulled successfully. This is because we failed to pull the image from the warehouse.
Copy the code

Solution Failed to pull a mirror

#Because I have changed the image source to 7 niuyun, but still can not pull down, we can more directly download the image to the local.Docker pull registry.cn-shanghai.aliyuncs.com/leozhanggg/flannel:v0.12.0-amd64 docker pull Registry.cn-shanghai.aliyuncs.com/leozhanggg/flannel:v0.12.0-arm64 docker pull Registry.cn-shanghai.aliyuncs.com/leozhanggg/flannel:v0.12.0-arm docker pull Registry.cn-shanghai.aliyuncs.com/leozhanggg/flannel:v0.12.0-ppc64le docker pull registry.cn-shanghai.aliyuncs.com/leozhanggg/flannel:v0.12.0-s390x#To play tagDocker tag registry.cn-shanghai.aliyuncs.com/leozhanggg/flannel:v0.12.0-amd64 Quay-mirror.qiniu.com/coreos/flannel:v0.12.0-amd64 docker tag Registry.cn-shanghai.aliyuncs.com/leozhanggg/flannel:v0.12.0-arm64 quay-mirror.qiniu.com/coreos/flannel:v0.12.0-arm64 Docker tag registry.cn-shanghai.aliyuncs.com/leozhanggg/flannel:v0.12.0-arm Quay-mirror.qiniu.com/coreos/flannel:v0.12.0-arm docker tag registry.cn-shanghai.aliyuncs.com/leozhanggg/flannel:v0.12.0-ppc64le Quay-mirror.qiniu.com/coreos/flannel:v0.12.0-ppc64le docker tag Registry.cn-shanghai.aliyuncs.com/leozhanggg/flannel:v0.12.0-s390x quay-mirror.qiniu.com/coreos/flannel:v0.12.0-s390x#And then execute
kubectl apply -f kube-flannel.yml
kubectl get po -n kube-system
#Found that the startup is normal. The startup result is as follows[root@k8s-master01 ~]# kubectl get po -n kube-system NAME READY STATUS RESTARTS AGE coredns-5c98db65d4-p8x8v 1/1 Running  0 62m coredns-5c98db65d4-ts4qm 1/1 Running 0 62m etcd-k8s-master01 1/1 Running 1 61m kube-apiserver-k8s-master01 1/1 Running 1 61m kube-controller-manager-k8s-master01 1/1 Running 1 61m kube-flannel-ds-amd64-mwbcq 1/1 Running 0 27m kube-proxy-hhvqh 1/1 Running 1 62m kube-scheduler-k8s-master01 1/1 Running 1 61m#When we look at the node again, it is ready
[root@k8s-master01 ~]# kubectl get node
NAME           STATUS   ROLES    AGE   VERSION
k8s-master01   Ready    master   63m   v1.15.1
Copy the code

After the primary node is successfully built, there is a log file kubeadm-init.log

cat kubeadm-init.log
#The following sentence will be found at the bottom of the document. We copy this statement and execute it on each nodeKubeadm join 192.168.70.128:6443 --token abcdef.0123456789abcdef \ --discovery-token-ca-cert-hash sha256:a33be40c0ef657e7565b399e7b44ad27bdf6477fd54047c70fdd8e4ee24c2850#We are checking the node, although it is still not ready. We just need to wait a moment.[root@k8s-master01 ~]# kubectl get node NAME STATUS ROLES AGE VERSION k8S-master01 Ready master 70m v1.15.1k8S-node01 NotReady < None > 20S V1.15.1k8S-node02 NotReady < None > 17s V1.15.1Copy the code

Note If the state is not ready for a long time, you can use the following method to check

#1. It was found that kube-Flannel-DS-AMd64 could not be downloaded for the other two nodes, so we used the above method to download the image and re-tag itkubectl get pod -n kube-system -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES Coredns-5c98db65d4-p8x8v 1/1 Running 1 22h 10.244.0.4k8s-master01 <none> <none> CoreDNS-5C98DB65D4-TS4qm 1/1 Running 1 22h 10.244.0.5k8s-master01 <none> <none> etcd-k8s-master01 1/1 Running 2 22h 192.168.70.128k8s-master01 <none> <none> Kube-apiserver -k8s-master01 1/1 Running 2 22h 192.168.70.128k8s-master01 <none> <none> Kube-controller-manager-k8s-master01 1/1 Running 2 22h 192.168.70.128k8s-master01 <none> <none> Kube - flannel-ds-amd64-2zls40/1 Init:ImagePullBackOff 0 6m39s 192.168.70.129k8s-node01 <none> <none> Kube-flannel-ds-amd64-75clq 1/1 Running 0 6m39s 192.168.70.128k8s-master01 <none> <none> kube-flannel-ds-amd64-c6MHw 0/1 Init:ImagePullBackOff 0 6m39s 192.168.70.130k8s-node02 <none> <none> kube-proxy-2m5rv 1/1 Running 1 21h 192.168.70.129k8s-node01 <none> <none> kube-proxy-fwprm 1/1 Running 1 21h 192.168.70.130k8s-node02 <none> <none> Kube-proxy-hhvqh 1/1 Running 2 22h 192.168.70.128k8s-master01 <none> <none> kube-scheduler-k8s-master01 1/1 Running 2 22h 192.168.70.128k8s-master01 <none> <none>
#2. You can also log in to the corresponding NDOE service to check node errors. You can also find that the mirror cannot be removed.Journalctl-f-u kubelet 4月 12 20:17:24 k8s-node01 kubelet[807]: E0412 20:17:24.020709 807 kuberUntime_manager. go:749] init container start failed: ImagePullBackOff: Back off - pulling image "quay-mirror.qiniu.com/coreos/flannel:v0.12.0-amd64" on April 12 20:17:24 k8s - node01 kubelet [807] : E0412 20:17:24.020784 807 pod_workers. Go :190] Error syncing Pod 69a5f6b6-ED51-45CE-9658-2706f63beaba ("kube-flannel-ds-amd64-8mt8d_kube-system(69a5f6b6-ed51-45ce-9658-2706f63beaba)"), skipping: failed to "StartContainer" for "install-cni" with ImagePullBackOff: "The Back - off pulling image \" quay-mirror.qiniu.com/coreos/flannel:v0.12.0-amd64\ ""Copy the code

Thus we have completed a K8S cluster construction. The next step is to learn its core components. And the flower