Goals and objectives

  • Reduce the difficulty of installing and deploying a K8S cluster using Ansible scripts

Skills required

  • Familiar with Linux basic operation commands
  • Familiar with basic Ansible operations
  • Familiar with basic operations of Docker
  • Read the official documents of K8S, have a basic understanding of K8S, understand its terms and basic concepts. According to the keywords, quickly find the corresponding documents on the official website for reference.
  • The script uses kubeadm to create a K8S cluster. Please read the documentation
  • Understand the technical concept of load balancing and the basic working principle of HAProxy
  • Understand the basic working principles of Keepalived, high availability technology concepts

A few steps to build a cluster

  1. Preparation stage
  2. Network planning
  3. Obtaining the Installation script
  4. Modify the configuration file based on the target host and network plan
  5. Execute the script in stages

Preparation stage

The target host

  • Operating system: Ubuntu 18.04
  • The SSH certificate can be used to log in to the host as user root
  • The hardware configuration of the target host meets the minimum requirements for installing k8S. For details, see
  • Several main engines
    • Single-master cluster requires 1 master host and at least 1 worker host (for running pod load)
    • 3 Master mode load balancing + high availability mode cluster requires 3 master hosts and at least 1 worker host (for running POD loads)
  • Configure static IP addresses for all destination hosts. The hosts in the same cluster must reside on the same subnet

Control host

  • We operate through the control host to complete the entire k8S cluster creation process
  • Python3 must be installed on the control host
  • With Ansible installed, we used Ansible to create the entire cluster

Network requirements

  • The destination host is configured with a static IP address

  • The network communication between the target hosts is normal

  • The controller host can communicate with the target host properly, and the controller host can use SSH certificates to perform SSH operations on the target host. For details, see ssh-copy-id

  • The target host can access the public network so that the target host can download dependent software packages and Docker images.

  • K8s network plug-in, using Calico

    Q: Why Calico?

    A: Calico is a mature solution based on the description and evaluation in this article.

Network planning

Before creating a cluster, we need to plan the IP addresses of the cluster in a unified manner:

  • You have obtained the static IP address of each host
  • Determine the roles of each host in the cluster: master, worker
  • Based on the number of hosts (resources and costs) :
    • Single-master cluster mode
    • Multi-master cluster mode (load balancing with multiple Masters + High availability)
  • determineControl plane 的The domain name.The IP address.port
    • IP address of the control plane: in ha mode, it is the virtual IP address of the HA service. In single-master mode, the value is the IP address of the master node
    • In multi-master cluster mode, if the ports used by the control plane are the same as those used by kube-Apiserverer (default :6443), the load balancing service cannot run on the same host as the master. Otherwise, port conflicts may occur. In this case, another worker host can be selected to run the load balancing service. Alternatively, change the port number of the control plane to another value, for example, 7443. In this mode, load balancing and HA services must be set up on the three master servers before cluster creation.
    • Add a control plane domain name resolution record to the /etc/hosts file on all target hosts
  • Determine the network segment used by the POD in the K8S cluster. Generally speaking, as long as the pod is not in the same network segment as the target host, you can.
  • Ensure that the network segment used by the SERVICE in the K8S cluster is different from the network segment used by the target host and pod

Obtaining the Installation script

  • Installation script source code hosted on Github, source address

  • Get the code to the control host

    git clone https://github.com/LoveInShenZhen/k8s-ubuntu-ansible.git
Copy the code

Modify the configuration file based on the target host and network plan

The Ansible hosts file

  • The hosts file describes the target host to which our Ansible script will operate

  • The file sample is as follows. The configuration in the file is described as follows:

    # host listing file reference: http://ansible.com.cn/docs/intro_inventory.html
[nodes:children]
master
worker

[master:children]
first_master
other_master

# host_name can only contain letters, digits, and hyphens. We use host_name as the node_name of K8S.
The first Master to create the cluster[first_master] 192.168.3.151 host_name = master - 1Additional masters that need to be added to the existing cluster[other_master] 192.168.3.152 Host_name =master-2 192.168.3.153 Host_name =master-3 [worker] 192.168.3.154 host_name=work-1 192.168.3.155 host_name = work - 2# vip_interface indicates the name of the nic bound to the virtual IP address192.168.3.151 vip_interface=eth1 192.168.3.152 vip_interface=eth1 192.168.3.153 vip_interface=eth1 [all:vars] anSIBLE_ssh_user =root Ansible_ssh_private_key_file =< Replace your root certificate > ansible_python_interpreter=/usr/bin/python3Copy the code

  • Target hosts are divided into different groups based on their functions and division of labor, as follows:

    Group name instructions
    nodes K8s all in the clustermasterMainframe and allworkerThe host
    master K8s cluster management node, including twosubgroups, respectively,first_master 和 other_master
    first_master The first Master when the cluster is created
    other_master Other Master to join the existing cluster. If yesA single masterIn mode, the group member is empty
    worker K8s cluster work node for running load pods
    lb_and_ha Used to run theK8s_kube-apiserverer Load balancing service + High availabilityThe nodes of the

  • Change the IP address (host_name) of the host in the group based on the network plan

  • Host_name must be globally unique and can contain only letters, digits, and hyphens (-)

    Why is that?

    • The –node-name parameter is used to specify the node name when kubeadm init initializes the cluster and when kubeadm join adds the node to the cluster. Our script uses the hostname value for this parameter, so we need to set a unique hostname for each host.

  • In the LB_AND_HA group, each host needs to set the vip_interface parameter separately.

    Why is that?

    • Vip_interface is used to specify the name of the network adapter bound to the virtual IP address
    • The host may have more than one NETWORK adapter, so you need to specify it explicitly
    • Multiple network interface cards (nics) on a host may be in bond mode. You can use this parameter to bind virtual IP addresses to the bonded nics

  • Set anSIBLE_ssh_private_key_file to the certificate used by user root to log in to the target host using SSH

Global configuration file

  • File path: roles/common/defaults/main yml

  • The file sample is as follows. The configuration in the file is described as follows:

    ---
# defaults file for common

k8s:
    # Host domain name and port number of the control plane
    # ref: https://kubernetes.io/zh/docs/setup/production-environment/tools/kubeadm/high-availability/#%E4%BD%BF%E7%94%A8%E5%A0%86% E6%8E%A7%E5%88%B6%E5%B9%B3%E9%9D%A2%E5%92%8C-etcd-%E8%8A%82%E7%82%B9
    # kubeadm init --control-plane-endpoint "control_plane_dns:control_plane_port" ... (abbreviated)
    control_plane_dns: k8s.cluster.local
    control_plane_port: 6443
    # apiserver_advertise_address: 0.0.0.0
    apiserver_bind_port: 6443
    # ref: https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/#pod-network
    pod_network_cidr: "192.168.0.0/16"
    service_cidr: "10.96.0.0/12"
    service_dns_domain: "cluster.local"
    # optional values: registry.cn-hangzhou.aliyuncs.com/google_containers [official documentation] (https://github.com/AliyunContainerService/sync-repo)
    # GCR. Azk8s. Cn/google_containers [official documentation] (http://mirror.azure.cn/help/gcr-proxy-cache.html)
    image_repository: "registry.cn-hangzhou.aliyuncs.com/google_containers"

apt:
    docker:
        apt_key_url: https://download.docker.com/linux/ubuntu/gpg
        apt_repository: https://mirrors.tuna.tsinghua.edu.cn/docker-ce/linux/ubuntu
    k8s:
        apt_key_url: https://mirrors.aliyun.com/kubernetes/apt/doc/apt-key.gpg
        apt_repository: https://mirrors.aliyun.com/kubernetes/apt/

Number of master nodes
master_count: "{{ groups['master'] | length }}"
Is it single master mode
single_master: "{{ (groups['master'] | length) == 1 }}"
# 
first_master: "{{ groups['first_master'] | first }}"

ntpdate_server: cn.ntp.org.cn

docker:
  # the daemon. Json configuration, refer to: https://docs.docker.com/engine/reference/commandline/dockerd/#daemon-configuration-file
  daemon:
    # Docker Hub image server
    registry-mirrors: 
      - https://dockerhub.azk8s.cn
      - https://docker.mirrors.ustc.edu.cn
      - https://reg-mirror.qiniu.com
    # reference: https://kubernetes.io/zh/docs/setup/production-environment/tools/kubeadm/install-kubeadm/#%E5%9C%A8%E6%8E%A7%E5%88%B6%E5 %B9%B3%E9%9D%A2%E8%8A%82%E7%82%B9%E4%B8%8A%E9%85%8D%E7%BD%AE-kubelet-%E4%BD%BF%E7%94%A8%E7%9A%84-cgroup-%E9%A9%B1%E5%8A% A8%E7%A8%8B%E5%BA%8F
    exec-opts:
      - "native.cgroupdriver=systemd"

Copy the code

  • Parameter Description

    • k8s.control_plane_dns
      • Domain name of the control plane
    • k8s.control_plane_port
      • Ports on the control plane.
      • If we want to run the load balancing service for the control plane on the master host, we need to set this port to a different value than k8s.apiserver_bind_port
      • If the cluster is in single-master mode, the load balancing service on the control plane is not required. You can set the load balancing service to k8s.apiserver_bind_port.
      • In the case description below, we demonstrate the process of changing from a single master cluster to a 3master cluster to control the high availability and load balancing of plane services. Instead of deploying on the master machine, we choose 2 worker hosts to implement the HA+LB mode of one master and one standby mode. Therefore, the ports on the control plane are initially set to k8s.apiserver_bind_port in single-master mode. The default is 6443
    • k8s.apiserver_bind_port
      • Kube-apiserverer service port (6443 by default) –apiserver-bind-port
    • k8s.pod_network_cidr
      • See the –pod-network-cidr parameter of the kubeadm init command
    • k8s.service_cidr
      • See the –service-cidr parameter of the kubeadm init command
    • k8s.service_dns_domain
      • See the –service-dns-domain parameter of the kubeadm init command
    • k8s.image_repository
      • Refer to the –image-repository parameter of the kubeadm init command
      • By setting this parameter, the installation process will not download images from the default k8s.gcr. IO repository.
      • The mirror warehouse provided by Aliyun (Alibaba Cloud) Container Service is used for parameter configuration
      • –image-repositorySee this article for parameter usage
    • apt.docker.apt_key_url
      • The Docker’s official GPG key.
    • apt.docker.apt_repository
      • Docker’s official Repository can be found in ununtu
    • apt.k8s.apt_key_url
      • Kubernetes Image source GPG key
    • apt.k8s.apt_repository
      • Kubernetes mirror source, refer to: Alibaba Kubernetes mirror source
      • Mirror source is used to solve the problem of being walled
    • ntpdate_server
      • Domain name of the time synchronization server

Execute the script in stages

Go to the build_k8s directory (where the hosts file is located) in the script source

Step 1: Perform initialization Settings for all target hosts

ansible-playbook prepare_all_host.yml
Copy the code

Step 2: Create a single-master cluster

  1. Check k8S.control_plane_port in the global configuration file. We use the default: 6443

  2. Update /etc/hosts on all nodes to resolve the domain name of the control plane to the IP address of the first master node

    Why is that?

    • In the process of creating a cluster and adding nodes to the cluster, we need to ensure that the master kube-Apiserverer service can be accessed through the control plane domain name + control plane port. Therefore, during cluster creation, resolve the domain name of the control plane to the first Master node
    • After the other two master nodes are added to the cluster, configure the three master nodes to be highly available. After the virtual IP addresses of the control plane take effect, update /etc/hosts on all nodes, and resolve the domain names of the control plane to the virtual IP addresses.

  3. Run the following command to resolve the domain name of the update control plane. Note the parameter transfer method in the following example

    • Run the -e “key1=value1 key2=value2… Manner and
    • You need to specify parameters domain_name and domain_ip
    • Domain_name indicates the domain name of the control plane. Set the value based on the network plan
    • Domain_ip is the destination IP address that controls domain name resolution. Here, we specify the IP address of the first master
    ansible-playbook set_hosts.yml -e "Domain_name = k8s. Cluster. The local domain_ip = 192.168.3.151"
Copy the code

  4. Execute the script to start creating the first master node

    ansible-playbook create_first_master_for_cluster.yml
Copy the code

  5. SSH to master-1 and run the following command to check the cluster node information:

    kubectl get nodes
Copy the code

    The following information should appear indicating that the cluster has been successfully created, even though there is now only one Master-1 node

    NAME       STATUS     ROLES    AGE   VERSION
master-1   NotReady   master   29s   v1.18.0
Copy the code

    Run the following command on master-1 to view cluster information

    kubectl cluster-info
Copy the code

    If the following information is displayed, the cluster is in the running state

    Kubernetes master is running at https://k8s.cluster.local:7443
KubeDNS is running at https://k8s.cluster.local:7443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
Copy the code

Step 3: Add additional nodes to the cluster

Execute the script to add other master and worker nodes to the cluster

4. Add — to add one by one. In the test, it was found that during the parallel adding process, there was a certain probability that the Master could not be added due to the re-election of ETCD, and more hosts (more than 4) could join the cluster at the same time. Due to the limited hardware resources, no test was conducted.

ansible-playbook --forks 1 add_other_node_to_cluster.yml
Copy the code

To check cluster node information, run the following command on a master node:

root@master-2:~# kubectl get nodesNAME STATUS ROLES AGE VERSION Master-1 Ready Master 9M13s v1.18.0 master-2 NotReady Master 4M5s v1.18.0 master-3 Tready master 2m21s v1.18.0 work-1 NotReady < None > 110s v1.18.0 work-2 NotReady < None > 107s v1.18.0Copy the code

Step 4: Create two load balancers and proxy the last three master Kube-Apiserverer services

Note: This step is not required in single-master mode

After the third step, the cluster is up and running. However, because the domain name of the control plane is resolved to the IP address of the first master, there are three masters in the cluster, but only the first master can provide the Kube-apiserverer service of the K8S cluster through the domain name of the control plane.

Next, we select 2 hosts other than the master node (which can be the worker host or 2 additional hosts) to create a set of load balancing + HIGH availability in the form of one active and one standby

  1. Check whether the IP addresses of the two hosts [lb_AND_ha] in the hosts file and the names of the network interface cards (nics) to which the virtual IP addresses are bound are correct

    [lb_and_ha] 192.168.3.154 vip_interface=eth0 192.168.3.155 vip_interface=eth0Copy the code

  2. Check the create_haproxy.yml configuration. The following file sample is used as an example:

    ---
- name: Create a load blance using HAproxy
  hosts: lb_and_ha
  vars:
    Port for load balancing to provide services externally
    service_bind_port: "{{ k8s.control_plane_port }}"
    The haproxy.cfg.j2 template does not use this variable
    backend_server_port: "{{ k8s.apiserver_bind_port }}"
    ['192.168.3.154:6443', '192.168.3.155:6443']
    backend_servers: "{{ ansible_play_hosts_all | map('regex_replace', '^(.*)$', '\\1:' + backend_server_port) | list }}"
    Backend_servers can be used by a host outside the cluster
    # backend_servers:
    # # - "
        
         :
         
          "
         
        
    # - "192.168.3.154:6443"
    # - "192.168.3.155:6443"

    # Enable haProxy stats
    ha_stats_enable: True
    The service port of the # haProxy stats page
    ha_stats_port: 1936
    The URL of the # haProxy Stats page
    ha_stats_url: /haproxy_stats
    The user name for the # haProxy stats page
    ha_stats_user: admin
    The password to access the # haProxy stats page
    ha_stats_pwd: showmethemoney
    container_name: k8s_kube-apiserverers_haproxy
  tasks:
    - name: check parameters
      fail:
        msg: "Please setup backend_servers parameter"
      when: backend_servers = = None or (backend_servers|count) = = 0 or backend_servers[0] = = ' ' or  backend_servers[0] = = '<ip>:<port>'

      Perform basic Settings for the host
    - import_role:
        name: basic_setup

    - name: pip install docker (python package)
      pip:
        executable: /usr/bin/pip3
        name: docker
        state: present
    
    - name: mkdir -p /opt/haproxy
      file:
        path: /opt/haproxy
        state: directory

    - name: get container info
      docker_container_info:
        name: "{{ container_name }}"
      register: ha_container

    - name: setup HAproxy configuration
      template:
        backup: True
        src: haproxy.cfg.j2
        dest: /opt/haproxy/haproxy.cfg
        mode: u=rw,g=r,o=r
      notify: restart HAproxy container

    - name: create HAproxy container
      docker_container:
        detach: yes
        image: haproxy:alpine
        name: "{{ container_name }}"
        volumes:
          - "/opt/haproxy/haproxy.cfg:/usr/local/etc/haproxy/haproxy.cfg"
        network_mode: bridge
        ports:
          - "{{ service_bind_port }}:{{ service_bind_port }}"
          - "{{ ha_stats_port }}:{{ ha_stats_port }}"
        restart_policy: always
        state: started

  handlers:
    - name: restart HAproxy container
      docker_container:
        name: "{{ container_name }}"
        state: started
        restart: yes
      when: ha_container.exists
Copy the code

  3. Create load balancing services on the two assigned workers

ansible-playbook create_haproxy.yml
Copy the code

  1. After the script is run, we can view the monitoring page of HaProxy on one of the machines, for example: 192.168.3.155 http://192.168.3.155:1936/haproxy_stats, access password for create_haproxy. Yml ha_stats_pwd in value

Step 5: Configure the two load balancers to work in active/standby mode for virtual IP addresses to take effect

Now we have two load balancers providing the same service. We configure the two load balancers to work in active/standby mode and let the virtual IP address take effect

  1. Check the create_keepalive. yml configuration. The file sample is as follows.

    ---
- name: setup keepalived on target host
  hosts: lb_and_ha
  vars:
    # virtual IP
    virtual_ipaddress: 192.1683.150./ 24
    keepalived_router_id: 99
    keepalived_password: FE3C5A94ACDC
    container_name: k8s_kube-apiserverers_keepalived
  tasks:
    Perform basic Settings for the host
    - import_role:
        name: basic_setup
    
    - import_role:
        name: install_docker

    - name: pip install docker (python package)
      pip:
        executable: /usr/bin/pip3
        name: docker
        state: present

    - name: mkdir -p /opt/keepalived
      file:
        path: /opt/keepalived
        state: directory

    - name: get container info
      docker_container_info:
        name: "{{ container_name }}"
      register: the_container

    - name: copy Dockerfile to target host
      copy:
        src: keepalived.dockerfile
        dest: /opt/keepalived/Dockerfile

    - name: build keepalived image
      docker_image:
        name: keepalived:latest
        source: build
        build:
          path: /opt/keepalived
          pull: yes

    - name: setup keepalived configuration
      template:
        src: keepalived.conf.j2
        dest: /opt/keepalived/keepalived.conf
        mode: u=rw,g=r,o=r
      notify: restart keepalived container

    - name: create keepalived container
      docker_container:
        capabilities:
          - NET_ADMIN
          - NET_BROADCAST
          - NET_RAW
        network_mode: host
        detach: yes
        image: keepalived:latest
        name: "{{ container_name }}"
        volumes:
          - "/opt/keepalived/keepalived.conf:/etc/keepalived/keepalived.conf"
        restart_policy: always
        state: started

  handlers:
    - name: restart keepalived container
      docker_container:
        name: "{{ container_name }}"
        state: started
        restart: yes
      when: the_container.exists

Copy the code

  2. Ensure that the virtual IP address configuration item virtual_ipAddress is the same as planned

    Note: In the configuration of the virtual IP address, the IP address must be identified by the number of subnet mask bits, for example, /24

  3. Execute script to configure high availability on 3 masters (Keepalived)

    ansible-playbook create_keepalived.yml
Copy the code

  4. After the command is executed, check whether the virtual IP address can be pinged. If the virtual IP address can be pinged, the virtual IP address in active/standby mode takes effect

Step 6: Resolve the domain name of the control plane to a virtual IP address

So far, the domain name of the control plane still refers to master-1. Then we need to resolve the domain name of the control plane to the virtual IP address, so that the load balancing service created in the previous step can be accessed by the domain name of the control plane.

We need to update /etc/hosts of all nodes to resolve the domain name of the control plane to the virtual IP address

Execute the script command:

In this example, the virtual IP address is 192.168.3.150

ansible-playbook set_hosts.yml -e "Domain_name = k8s. Cluster. The local domain_ip = 192.168.3.150"
Copy the code

Ping the domain name of the control plane on the host to verify that the domain name is correctly resolved to the virtual IP address.

On the master host, run the kubectl command to check whether the kube-apiserverer of the control plane can be accessed through the domain name and port of the control plane:

kubectl get nodes
Copy the code