Kubernetes Full Stack Architect Chapter 2 Kubeadm high availability installation K8S cluster version 1.20

Chapter 2 Kubeadm High availability install K8S cluster version 1.20

2.1 Basic Environment Configuration

centos7.x

High availability Kubernetes cluster planning

VIP(virtual IP address) selection description

VIP(virtual IP address) should not be the same as the Intranet IP address of the company. If the ping fails, the IP address must be on the same LAN as the host. Author’s note: THE VIP(virtual IP address) must be a new IP address that is not used in the same LAN as the host.

set_hostname

##k8s-master-01
hostnamectl set-hostname k8s-master-01

##k8s-master-02
hostnamectl set-hostname k8s-master-02

##k8s-master-03
hostnamectl set-hostname k8s-master-03

##k8s-node-01
hostnamectl set-hostname k8s-node-01

##k8s-node-02
hostnamectl set-hostname k8s-node-02

Copy the code

set_hosts.sh

#curl https://gitee.com/k08s/k8s_note/raw/dev/51cto_Kubernetes/set_hosts.sh | bash

sed -i '/k8s/d'  /etc/hosts

cat << EOF >> /etc/hosts
10.11.1.108 k8s-master-01
10.11.1.109 k8s-master-02
10.11.1.110 k8s-master-03
10.11.1.111 k8s-node-01
10.11.1.112 k8s-node-02
EOF
Copy the code

close_firewall.sh

systemctl disable --now firewalld
systemctl disable --now NetworkManager
systemctl disable --now dnsmasq



Copy the code

disable_selinux.sh

setenforce 0
sed -i 's#SELINUX=enforcing#SELINUX=disabled#g' /etc/sysconfig/selinux
sed -i 's#SELINUX=enforcing#SELINUX=disabled#g' /etc/selinux/config
Copy the code

close_swap.sh

swapoff -a && sysctl -w vm.swappiness=0
sed -i.bak -r 's/(.+ swap .+)/#\1/' /etc/fstab
Copy the code

setup_ntpdate.sh

rpm -ivh http://mirrors.wlnmp.com/centos/wlnmp-release-centos.noarch.rpm
yum -y install ntpdate

ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime  
echo 'Asia/Shanghai' > /etc/timezone
ntpdate time2.aliyun.com/centos/wlnmp-release-centos
echo "*/5 * * * * root ntpdate time2.aliyun.com " >> /etc/crontab
# echo "*/5 * * * * ntpdate time2.aliyun.com" >> /var/spool/cron/root

Copy the code

ulimit_set.sh

ulimit -SHn 65535

cat <<EOF>> /etc/security/limits.conf * soft nofile 655360 * hard nofile 131072 * soft nproc 655350 * hard nproc 655350 * soft  memlock unlimited * hard memlock unlimited EOF
Copy the code

keygen_util.sh

#run only in k8s-master-01
#TODO : cfssl ? ssh-keygen 
#TODO: Remove answering questions and entering passwords?
ssh-keygen -t rsa
for i in k8s-master-01 k8s-master-02 k8s-master-03 k8s-node-01 k8s-node-02 ; 
do ssh-copy-id -i .ssh/id_rsa.pub $i; 
done
Copy the code

set_centos7_repo_using_aliyun.sh

centos7 repo aliyun

#curl https://gitee.com/k08s/k8s_note/raw/dev/51cto_Kubernetes/set_centos7_repo_using_aliyun.sh | bash

#https://developer.aliyun.com/mirror/centos

base_repo=/etc/yum.repos.d/CentOS-Base.repo

mv $base_repo ${base_repo}_`date +%s`
curl -o $base_repo https://mirrors.aliyun.com/repo/Centos-7.repo 

sed -i -e '/mirrors.cloud.aliyuncs.com/d' -e '/mirrors.aliyuncs.com/d' /etc/yum.repos.d/CentOS-Base.repo

 
Copy the code

os_upgrade.sh

yum update -y --exclude=kernel* && reboot
Copy the code

2.2 Kernel Configuration

Linux kernel upgraded to 4.18+

The author guesses: The Linux kernel with docker 19.03 below this version may have various bugs?

linux_kernel_download.sh

#run only in k8s-master-01
#TODO:Avoid entering passwords

cd /root/

wget http://193.49.22.109/elrepo/kernel/el7/x86_64/RPMS/kernel-ml-4.19.12-1.el7.elrepo.x86_64.rpm
wget http://193.49.22.109/elrepo/kernel/el7/x86_64/RPMS/kernel-ml-devel-4.19.12-1.el7.elrepo.x86_64.rpm


for i in k8s-master-01 k8s-master-02 k8s-master-03 k8s-node-01 k8s-node-02 ; 
doSCP kernel - ml - 4.19.12-1. El7. Elrepo. X86_64. RPM kernel - ml - devel - 4.19.12-1. El7. Elrepo. X86_64. RPM$i:/root/; 
done
Copy the code

linux_kernel_upgrade.sh

Check the default kernel
grubby --default-kernel
# / boot/called - 3.10.0-1160.42.2. El7. X86_64

cd /root/ && yum localinstall -y kernel-ml-*
If you restart any Linux system at this time, you will find that the first item in the GRUB menu is kernel-4.19, but the old kernel, kernel-3.10, is selected by default
The 0 in #grub2-set-default 0 refers to the first grub menu item kernel-4.19
grub2-set-default 0 && grub2-mkconfig -o /etc/grub2.cfg

grubby --args="user_namespace.enable=1" --update-kernel="$(grubby --default-kernel)"
Check the default kernel
grubby --default-kernel

After the restart, the new kernel will be used
reboot
#uname -a

Copy the code

ipvsadm_install.sh

# Production environments use IPVs instead of iptables
yum install -y ipysadm ipset sysstat conntrack   libseccomp 

# the Linux kernel 4.19 + : nf_conntrack
nf_conntrack_ipv4

Nf_conntrack is used for Linux kernel 4.19+

# Reference:? : https://docs.projectcalico.org/getting-started/kubernetes/requirements

cat <<EOF> /etc/modules-load.d/ipvs.conf
ip_vs
ip_vs_lc
ip_vs_wlc
ip_vs_rr
ip_vs_wrr
ip_vs_lblc
ip_vs_lblcr
ip_vs_dh
ip_vs_sh
ip_vs_fo
ip_vs_nq
ip_vs_sed
ip_vs_ftp
nf_conntrack
ip_tables
ip_set
xt_set
ipt_set
ipt_rpfilter
ipt_REJECT
ipip
EOF

systemctl enable --now systemd-modules-load.service 

Check whether the kernel module is loaded:
lsmod | grep   -e ip_vs  -e nf_conntrack
Copy the code

k8s_kernel_param_set.sh

To enable the kernel parameters required in the K8S cluster, all nodes must perform:

cat <<EOF> /etc/sysctl.d/k8s.conf net.ipv4.ip_forward=1 net.bridge.bridge-nf-call-iptables=1 net.bridge.brige-nf-call-ip6tables=1 fs.may_detach_mounts=1 vm.overcommit_memory=1 vm.panic_on_oom=0 fs.inotify.max_user_watches=89100 fs.file-max=52706963 fs.nr_open=52706963 net.netfilter.nf_conntrack_max=2310720 net.ipv4.tcp_keepalive_time=600 net.ipv4.tcp_keepalive_probes=3 net.ipv4.tcp_keepalive_intvl=15 net.ipv4.tcp_max_tw_buckets=36000 net.ipv4.tcp_tw_reuse=1 net.ipv4.tcp_max_orphans=327680 net.ipv4.tcp_syncookies=1 net.ipv4.tcp_max_syn_backlog=16384 net.ipv4.ip_conntrack_max=65536 net.ipv4.tcp_timestamps=0 net.core.somaxconn=16384 EOF

sysctl --system

reboot


Copy the code

2.3 Installing K8S Components

install_docker-ce_using_aliyun_mirror.sh

docker-ce repo aliyun

#curl https://gitee.com/k08s/k8s_note/raw/dev/51cto_Kubernetes/install_docker-ce_using_aliyun_mirror.sh | bash

#https://developer.aliyun.com/mirror/docker-ce
yum install -y  yum-utils device-mapper-persistent-data lvm2
yum-config-manager --add-repo https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
sed -i 's+download.docker.com+mirrors.aliyun.com/docker-ce+'/etc/yum.repos.d/docker-ce.repo yum makecache fast yum list docker-ce.x86_64 --showduplicates | sort -r yum -y install Docker - ce - 19.03.9-3. El7. Centos# yum - y install docker - ce - 19.03. *
#service docker start

mkdir /etc/docker/
cat <<EOF> /etc/docker/daemon.json
{
"exec-opts":["native.cgroupdriver=systemd"]
}
EOF

systemctl daemon-reload  && systemctl enable --now docker
Copy the code

install_k8s_using_aliyun_mirror.sh

#curl https://gitee.com/k08s/k8s_note/raw/dev/51cto_Kubernetes/install_k8s_using_aliyun_mirror.sh | bash

cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF


#yum list kubeadm --showduplicates | sort -r

#{All nodes installed: kubeadm.
# kubelet Kubectl will be installed automatically
yum install -y --nogpgcheck kubeadm 

cat <<EOF> /etc/sysconfig/kubelet
KUBELET_EXTRA_ARGS="--cgroup-driver=systemd --pod-infra-container-image=registry.cn-hangzhou.aliyuncs.com/google_containers/pause-amd64:3.2"
EOF

systemctl daemon-reload

systemctl enable --now kubelet
#}



Copy the code

2.4 Installing HA Components

keepalived_haproxy_install.sh

KeepAlived and HAProxy are installed on all master nodes
yum -y install  keepalived haproxy 


Copy the code

HAProxy_set.sh

mkdir /etc/haproxy

cat <<EOF> /etc/haproxy. CFG global maxconn 2000 ulimit-n 16384 log 127.0.0.1 local0 ERR STATS timeout 30s defaults log  global mode http option httplog timeout connect 5000 timeout client 50000 timeout server 50000 timeout http-request 15s  timeout http-keep-alive 15s frontend monitor-in bind *:33305 mode http option httplog monitor-uri /monitor frontend K8s-master bind 0.0.0.01:16443 bind 127.0.0.1:16443 mode TCP option tcplog tcp-request inspect-delay 5s default_backend k8s-master backend k8s-master mode tcp option tcplog option tcp-check balance roundrobin default-server inter 10s Downinter 5s rise 2 fall 2 slowstart 60s MaxCONN 250 maxQueue 256 weight 100 server K8S-master-01 10.0.2.18:6443 check Server K8S-master-02 10.0.2.19:6443 Check Server K8S-master-03 10.0.2.20:6443 check EOF
Copy the code

KeepAlived_master_01.sh

  
 
cat <<EOF> /etc/keepalived/keepalived.conf global_defs { router_id LVS_DEVEL script_user root enable_script_security } vrrp_script chk_apiserver{ script "/etc/keepalived/check_apiserver.sh" interval 5 weight -5 fall 2 rise 1 } Vrrp_instance VI_1{state MASTER interface enp0s3 McAst_src_ip 10.0.2.18 virtual_router_id 51 priority 101 advert_int 2 Authentication {auth_type PASS auth_pass K8SSHA_KA_AUTH} virtual_ipaddress {10.0.2.236} track_script {chk_apiserver  } } EOF
Copy the code

KeepAlived_master_02.sh

 
cat <<EOF> /etc/keepalived/keepalived.conf global_defs { router_id LVS_DEVEL script_user root enable_script_security } vrrp_script chk_apiserver { script "/etc/keepalived/check_apiserver.sh" interval 5 weight -5 fall 2 rise 1 } Vrrp_instance VI_1{state BACKUP interface enp0s3 McAst_src_ip 10.0.2.19 virtual_router_id 51 priority 100 advert_int 2 Authentication {auth_type PASS auth_pass K8S_KA_AUTH} virtual_ipaddress {10.0.2.236} track_script {chk_apiserver} } EOF
Copy the code

KeepAlived_master_03.sh

cat <<EOF> /etc/keepalived/keepalived.conf global_defs { router_id LVS_DEVEL script_user root enable_script_security } vrrp_script chk_apiserver { script "/etc/keepalived/check_apiserver" interval 5 weight -5 fall 2 rise 1 } vrrp_instance VI_1 {state BACKUP interface enp0s3 McAst_src_ip 10.0.2.20 virtual_router_id 51 priority 100 advert_int 2 Authentication {auth_type PASS auth_pass K8SHA_KA_AUTH} virtual_ipaddress {10.0.2.236} track_script {chk_apiserver } } EOF
Copy the code

check_apiserver.sh

All master nodes have check_apiserver.sh
cat <<EOF > /etc/keepalived/check_apiserver.sh #! /bin/bash err=0 # Stop KeepAlived on the master if haProxy has 4 consecutive problems For k in $(seq 1 4) do check_code=$(pgrep haproxy) if [[$check_code == ""]]; then err = $(expr $err + 1) sleep 1 continue else err=0 break fi done if [[ $err != "0" ]]; then echo "systemctl stop keepalived" /usr/bin/systemctl stop keepalived exit 1 else exit 0 fi EOF

chmod +x  /etc/keepalived/check_apiserver.sh

Copy the code

haproxy_keepalived_boot.sh

# execute only on K8S-master-01
systemctl daemon-reload
systemctl enable --now haproxy
systemctl enable --now keepalived

# k8s - master - 01
Enp0s3 has an additional IP address 10.0.2.236, which is the virtual IP address
Sending gratuitous ARP on enp0S3 for 10.0.2.236 is displayed in the messages log


K8s-master-02,k8s-master-03 try to connect to virtual IP 10.0.2.236
# ping 10.0.2.236
# Telnet 10.0.2.236 16443
Copy the code

2.5 Cluster Initialization

init_cluster__kubeadm_init.sh

Does it work? I don't know. It's not an order for now anyway.
kubeadm init --control-plane-endpoint "10.0.2.236:16443" --upload-certs

Copy the code

kubeadm-config.yaml

#/root/kubeadm-config.yaml: this file is available on all nodes.
The value of #kubernetesVersion is the return of the command kubeadm version
#cat <<EOF> /root/kubeadm-config.yaml
apiVersion: kubeadm.k8s.io/v1beta2
bootstrapTokens:
  - groups:
      - system:bootstrappers:kubeadm:default-node-token
    token: 7t2weq.bjbawausm0jaxury
    ttl: 24h0m0s
    usages:
      - signing
      - authentication
kind: InitConfiguration
localAPIEndpoint:
  advertiseAddress: 10.02.108.
  bindPort: 6443
nodeRegistration:
  criSocket: /var/run/dockershim.sock
  name: k8s-master-01
  taints:
    - effect: NoSchedule
      key: node-role.kubernetes.io/master
---
apiServer:
  certSANs:
    - 10.02.236.
  timeoutForControlPlane: 4m0s
apiVersion: kubeadm.k8s.io/v1beta2
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controlPlaneEndpoint: 10.02.236.: 16443
controllerManager: {}
dns:
# docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/coredns:1.8.4
# no: docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/coredns:v1.8.4
# are: the docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/coredns:1.8.4
  imageRepository: registry.cn-hangzhou.aliyuncs.com/google_containers
  imageTag: 1.84.
  type: CoreDNS
etcd:
  local:
    dataDir: /var/lib/etcd
imageRepository: registry.cn-hangzhou.aliyuncs.com/google_containers
kind: ClusterConfiguration
kubernetesVersion: v1.22.1
networking:
  dnsDomain: cluster.local
  podSubnet: 172.168. 0. 0/ 12
  serviceSubnet: 10.96. 0. 0/ 12
scheduler: {}
#EOF
Copy the code

Migrate: kubeadm-config.yaml: kubeadm-config-new.yaml

# /root/kubeadm-config-new.yaml
# reference: https://kubernetes.io/docs/reference/config-api/kubeadm-config.v1beta3/#kubeadm-k8s-io-v1beta3-ImageMeta
The value of #kubernetesVersion is the return of the command kubeadm version
apiVersion: kubeadm.k8s.io/v1beta3
bootstrapTokens:
- groups:
  - system:bootstrappers:kubeadm:default-node-token
  token: 7t2weq.bjbawausm0jaxury
  ttl: 24h0m0s
  usages:
  - signing
  - authentication
kind: InitConfiguration
localAPIEndpoint:
  advertiseAddress: 10.111.108.
  bindPort: 6443
nodeRegistration:
  criSocket: /var/run/dockershim.sock
  imagePullPolicy: IfNotPresent
  name: k8s-master-01
  taints:
  - effect: NoSchedule
    key: node-role.kubernetes.io/master
---
apiServer:
  certSANs:
  - 10.111.236.
  timeoutForControlPlane: 4m0s
apiVersion: kubeadm.k8s.io/v1beta3
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controlPlaneEndpoint: 10.111.236.: 16443
controllerManager: {}
dns: 
# docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/coredns:1.8.4
# no: docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/coredns:v1.8.4
# are: the docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/coredns:1.8.4
  imageRepository: registry.cn-hangzhou.aliyuncs.com/google_containers
  imageTag: 1.84.
etcd:
  local:
    dataDir: /var/lib/etcd
imageRepository: registry.cn-hangzhou.aliyuncs.com/google_containers
kind: ClusterConfiguration
kubernetesVersion: v1.22.1
networking:
  dnsDomain: cluster.local
  podSubnet: 172.168. 0. 0/ 12
  serviceSubnet: 10.96. 0. 0/ 12
scheduler: {}

Copy the code

init_cluster.sh

#init_cluster.sh: The following commands are executed on all nodes without special instructions.Docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/coredns:1.8.4# no: docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/coredns:v1.8.4
# are: the docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/coredns:1.8.4
# 2021-09-16 08:49 Progress, resolved without COREDNS: V1.8.4.
# reference: https://kubernetes.io/docs/reference/config-api/kubeadm-config.v1beta3/#kubeadm-k8s-io-v1beta3-ImageMeta


kubeadm config migrate --old-config  /root/kubeadm-config.yaml  --new-config  /root/kubeadm-config-new.yaml


##20210915 progress, 2-7 Kubeadm cluster initialization. Mp4
kubeadm  config  images  pull --config /root/kubeadm-config-new.yaml

systemctl  enable  --now  kubelet

Kubeadm init is only executed in k8S-master-01
kubeadm  init  --config  /root/kubeadm-config-new.yaml  --upload-certs

kubeadm reset -f ;  ipvsadm  --clear ; rm -fr ~/.kube


Copy the code

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf  $HOME/.kube/config

sudo chown $(id -u);  $(id -g)  $HOME/.kube/config
Copy the code