Set up the environment

  • Operating system: macOS 10.15.2
  • Vm: VMware Fusion 11.5
  • Vm system: centos 7 64
  • Kubernetes 15.1
  • Harbor 2.1.0

Cluster planning

The host name ip
k8s-master-01 192.168.8.10
harbor 192.168.8.20
k8s-node-01 192.168.8.21
k8s-node-02 192.168.8.22

Configure the VM network

1. Add a NAT network for VMware Fusion

VMware Fusion -> Preferences -> Network -> Unlock after click + to require the following Settings

  • Select Allow VMS on this network to connect to the external network (using NAT)
  • Subnet IP address (for example, 192.168.8.0)
  • The subnet mask is 255.255.255.0

2. Check vNET configuration on the MAC host

You can see the Vnet4 that we added earlier

Answer VNET_1_VIRTUAL_ADAPTER yes answer VNET_4_DHCP yes answer VNET_4_HOSTONLY_NETMASK 255.255.255.0 answer VNET_4_HOSTONLY_SUBNET 192.168.8.0 answer VNET_4_NAT yes answer VNET_4_NAT_PARAM_UDP_TIMEOUT 30 answer VNET_4_VIRTUAL_ADAPTER yes answer VNET_8_DHCP yes Copy codeCopy the code

3. Check the vnet4 configuration

cat /Library/Preferences/VMware\ Fusion/vmnet4/nat.conf

See the Vnet4 gateway IP and mask

# NAT gateway address IP = 192.168.8.2 netmask = 255.255.255.0Copy the code

4. Modify vm network configurations

Vm -> Settings -> Network Adapter Settings -> VMnet4

5. Modify the NIC configuration file of the Linux VM

vi /etc/sysconfig/network-scripts/ifcfg-ens33

The modified file looks like this (see the commented line) :

TYPE=Ethernet PROXY_METHOD=none BROWSER_ONLY=no BOOTPROTO=static Static DEFROUTE=yes IPV4_FAILURE_FATAL=no IPV6INIT=yes IPV6_AUTOCONF=yes IPV6_DEFROUTE=yes IPV6_FAILURE_FATAL=no IPV6_ADDR_GEN_MODE= stables -privacy NAME=ens33 UUID= b4bd9a95-905E-471A-9333-f7a668a8AF5d DEVICE=ens33 ONBOOT=yes IPADDR=192.168.8.10 # NETMASK=255.255.255.0 # NETMASK=255.255.255.0 # GATEWAY=192.168.8.2 # NETMASK= 192.168.8.10 # NETMASK=255.255.255.0 Vnet2 gateway IP DNS1=114.114.114.114Copy the code

6. Restart the Linux VM NIC

systemctl restart network

7, validation,

  • On a Linux VM, ping the MAC host, ping www.baidu.com
  • On a MAC host, ping 192.168.8.10, which is the static IP address just configured for the VM

System initialization

K8s-node-1 hostnamectl set-hostname k8s-master-1 hostnamectl set-hostname k8s-node-1 hostnamectl set-hostname k8s-node-2Copy the code

Configure the host

Cat >> /etc/hosts <<EOF 192.168.0.10k8S-master-01 192.168.0.21k8S-node-01 192.168.0.22k8S-node-02 EOF

After setting, synchronize to other machines

scp /etc/hosts root@k8s-node-01:/ect/hosts
scp /etc/hosts root@k8s-node-02:/ect/hosts

Copy the code

Installing dependency packages

1. Installation base dependency:

Yum install -y conntrack ntpdate NTP ipvsadm ipset jq iptables curl sysstat libseccomp wget vim net-tools git # D/centos-base. repo /etc/yum. Repos. D/centos-base. repo_bak The/etc/yum. Repos. D/CentOS - Base. 'http://mirrors.aliyun.com/repo/Centos-7.repo # update yum yum cache clean all yum makecacheCopy the code

2. Set firewall to iptables

Iptables yum install -y iptables-services &&  systemctl start iptables && systemctl enable iptables && iptables -F && service iptables saveCopy the code

3. Close selinux and Swapoff partitions

swapoff -a && sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab
setenforce 0 && sed -i 's/^SELINUX=.*/SELINUX=disabled/' /etc/selinux/config

Copy the code

4. Adjust kernel parameters for K8S

Cat > kubernetes.conf <<EOF net.bridge.bridge-nf-call-ip6tables=1 Net.ipv4. ip_forward=1 net.ipv4.tcp_tw_recycle=0 vm.swappiness=0 Overcommit_memory =1 # Do not check whether physical memory is sufficient for vm. Panic_on_oom =0 # Enable OOM fs.inotify. Max_user_instances =8192 Fs. Way. Max_user_watches = 1048576 fs file - Max = 52706963 fs. Nr_open = 52706963 net. Ipv6. Conf. All the closed ipv6 disable_ipv6 = 1 # Agreement net.net filter. Nf_conntrack_max = 2310720 EOFCopy the code

5. Adjust the system time zone

Timedatectl set-timezone Asia/Shanghai write the current UTC time to the hardware clock timedatectl set-local-rTC 0 systemctl restart rsyslog systemctl restart crondCopy the code

6. Stop unnecessary services

systemctl stop postfix && systemctl disable postfix

7, set rsyslogd and Systemd journald

After centos7, the system boot mode is changed to systemd, which has two logging systems working

Mkdir/var/log/journal # persisted log directory mkdir/etc/systemd journald. Conf. D. cat > The/etc/systemd journald. Conf. D / 99 - prophet. Conf < < EOF [Journal] # # are persisted to disk Storage = persistent compressed history log Compress = yes SyncIntervalSec=5m RateLimitInterval=30s RateLimitBurst=1000 # Maximum occupied space 10G SystemMaxUse=10G # Maximum occupied space for a log file is 200M SystemMaxFileSize=200M # Log save time 2 weeks MaxRetentionSec=2week # Do not forward logs to syslog ForwardToSyslog=no EOF systemctl restart systemd-journaldCopy the code

Upgrade the kernel to 4.4

The RPM - Uvh http://www.elrepo.org/elrepo-release-7.0-3.el7.elrepo.noarch.rpm # installation after the completion of the inspection/boot/grub2 / grub. Corresponding kernel menuentry CFG Do you have initrd16 configuration in? If not, install it again!Copy the code

yum –enablerepo=elrepo-kernel install -y kernel-lt

Setup boot from the new kernel

Grub2-set-default 'CentOS Linux (4.4.189-1.el7.elrebo.x86_64) 7 (Core)' rebootCopy the code

Install k8s

1, kube-proxy enable ipvS preconditions

Note that kube-proxy 1.13 or later strongly relies on nF_conntrack_ipv4. If kube-Proxy 1.13 or earlier is used, you need to replace nF_Conntrack with nf_conntrack_ipv4

modprobe br_netfilter cat > /etc/sysconfig/modules/ipvs.modules <<EOF #! /bin/bash modprobe -- ip_vs modprobe -- ip_vs_rr modprobe -- ip_vs_wrr modprobe -- ip_vs_sh modprobe -- nf_conntrack_ipv4 EOF chmod 755 /etc/sysconfig/modules/ipvs.modules && bash /etc/sysconfig/modules/ipvs.modules && lsmod  | grep -e ip_vs -e nf_conntrack_ipv4Copy the code

2. Install Docker

yum install -y yum-utils device-mapper-persistent-data lvm2

yum-config-manager \
--add-repo \
http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo

yum update -y  && yum install -y docker-ce

Copy the code
Mkdir /etc/docker # registries allow docker to use docker registries without permission cat > /etc/docker/daemon.json <<EOF { "exec-opts": ["native.cgroupdriver=systemd"], "log-driver": "Json-file ", "registry-mirrors": [" log-opts": {"max-size": "100m"}, "insecure-registries": {"json-file", "registry-mirrors": [" log-opts": {"max-size": "100m"}, "insecure-registries": [" hub.czp.com "]} EOF mkdir -p/etc/systemd/system/docker. Service. The d # restart daemon - reload docker service systemctl && systemctl restart docker && systemctl enable dockerCopy the code

3. Install Kubeadm on all nodes

cat > /etc/yum.repos.d/kubernetes.repo <<EOF [kubernetes] name=Kubernetes baseurl=http://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64 enabled=1 gpgcheck=0 repo_gpgcheck=0 gpgkey=http://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg http://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg EOF yum - y install kubeadm - 1.15.1 kubectl - 1.15.1 Kubelet - 1.15.1 systemctl enable kubelet. ServiceCopy the code

Once Kubeadm is installed, you can use this virtual machine as a template to clone from when you create new nodes

4. Initialize the k8S-master-01 primary node

Kubeadm config print init-defaults > kubeadm-config.yaml Modify several parameters vim kubeadm - config. Yaml imageRepository: registry.aliyuncs.com/google_containers # modified under the mirror address advertiseAddress: 192.168.8.10 # kubernetesVersion: v1.15.1 # kubeadmin "10.244.0.0/16" # must be this IP serviceSubnet: 10.96.0.0/12 # then add the following at the end to support ipvs -- apiVersion: kubeproxy.config.k8s.io/v1alpha1 kind: KubeProxyConfiguration featureGates: SupportIPVSProxyMode: true mode: ipvsCopy the code
# start initialization kubeadm init -- config = kubeadm - config. Yaml - experimental - upload - certs | tee kubeadm - init. Grep log # check if initial success "The initialized successfully" kubeadm - init. Log # after the initial success of subsequent operations mkdir -p $HOME /. Kube sudo cp - I/etc/kubernetes/admin. Conf $HOME/. Kube /config sudo chown $(id -u):$(id -g) $HOME/. Kube /config $(id -g) $HOME/. Kube /config  kubeadm-config.yaml install-k8s wget https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml kubectl apply -f kube-flannel.yml # I have a problem here. Quay. IO/coreos/flannel mirror has been unable to normal pull, I'm in the https://github.com/coreos/flannel/releases after downloading, uploaded to the virtual machine, the installation manual load to the docker. Docker [email protected]:/root docker load < flanneld-v0.13.1-rc1-amd64.docker # Kubectl get node statusCopy the code

5. Initialize k8S-Node-0xxx from the node

You can clone a VIRTUAL machine as a node based on the template mentioned above, with a change of IP address.

Get the initialization command from the log in the previous step

Kubeadm join 192.168.8.10:6443 --token abcdef.0123456789abcdef \ --discovery-token-ca-cert-hash sha256:1d853fdcfd4c933dee2c13b8e5ee7ac91fd85225d4c6f96f05d9a1997591012cCopy the code

The token is valid for 24 hours. After the token expires, you can use kubeadm token create to regenerate the token. After the command is executed successfully, wait for a while and execute it on the master

kubectl get node
Copy the code

See that node is ready.

NAME STATUS ROLES AGE VERSION K8S-master01 Ready Master 77M V1.15.1 K8S-Node-1 Ready < None > 69m v1.15.1Copy the code

6. Several common commands

  • Viewing POD Details

kubectl describe pod xxx

  • View the logs for a container within the POD

kubectl log podXXX -c containerXXXX

  • Enter the pod container. If pod has more than one container, specify it with the -c argument

kubectl exec podXXXX -it -- /bin/sh

Install the Harbo private image repository

Install the docker

yum install -y yum-utils device-mapper-persistent-data lvm2
yum-config-manager \
--add-repo \
http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
yum update -y  && yum install -y docker-ce

Copy the code
/etc/docker mkdir /etc/docker daemon. Insecure -registries Cat > /etc/docker/daemon.json <<EOF {"exec-opts": [" native-cgroupdriver =systemd"], "log-driver": "json-file", "registry-mirrors": ["https://tosnxdv7.mirror.aliyuncs.com"], "log-opts": { "max-size": "100m" } "insecure-registries": [" hub.czp.com "]} EOF mkdir -p/etc/systemd/system/docker. Service. The d # restart daemon - reload docker service systemctl && systemctl restart docker && systemctl enable dockerCopy the code

Install the docker – compose

wget https://linux-soft-ware.oss-cn-shenzhen.aliyuncs.com/docker-compose
chmod +x docker-compose
mv docker-compose /usr/local/bin

Copy the code

Install the Harbor

Download: github.com/goharbor/ha… I’m using version 2.1.0 here

# tar xf harbor-offline-installer-v2.1.0.tgz # edit configuration file vi harbor.yml # Set the domain name to be the same as the insecure-registries in docker/daemon.json. Hostname = hub.czp.com # 2, set HTTPS key and certificate HTTPS: 443 # The path of cert and key files for nginx certificate: /data/cert/server.crt private_key: /data/cert/server.keyCopy the code
Copy the code

Create a certificate

mkdir /data/cert -p


Copy the code

1. Create a key

openssl genrsa -des3 -out server.key 2048 Generating RSA private key, 2048 bit long modulus ............................................................ + + +... +++ E is 65537 (0x10001) Enter pass phrase for server.key: Enter the password 123456 Verifying - Enter pass phrase for server.key: Enter the password 123456Copy the code

2. Create a certificate

openssl req -new -key server.key -out server.csr Enter pass phrase for server.key: You are about to be asked to enter information that will be incorporated into your certificate request. What you are about to enter is what is called a Distinguished Name or a DN. There are quite a few fields but you can leave some blank  For some fields there will be a default value, If you enter '.', the field will be left blank. ----- Country Name (2 letter code) [XX]:CN State or Province Name (full name) []:GD Locality Name (eg, city) [Default City]:SZ Organization Name (eg, company) [Default Company Ltd]:DEMO Organizational Unit Name (eg, section) []:DEMO Common Name (eg, your name or your server's hostname) []:hub.czp.com Email Address []:[email protected] Please enter the following 'extra' attributes to be sent with your certificate request A challenge password []: An optional company name []:Copy the code

3, let the certificate recede password

Key Enter pass phrase for server.key.org: Enter the password 123456 Writing RSA keyCopy the code

4, signature

 openssl x509 -req -days 365 -in server.csr -signkey server.key -out server.crt
Copy the code

5. Modify the certificate permission

chmod -R 777 /data/cert
Copy the code

Install the harbor

./harbor/install.sh
Copy the code

Configuring local Hosts

10.0.0.11  hub.czp.com
Copy the code

Access to the harbor

Enter hub.czp.com and admin/12345

Test whether you can log into a private repository

 docker login https://hub.czp.com
Username: admin
Password:
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

Login Succeeded

Copy the code