preface

The above set up the highly available Kubernetes cluster, originally planned to continue to explore the actual application deployment and how to transform the existing technical architecture to make Kubernetes initial landing. However, after building the cluster, I still want to find a graphical management tool. After all, MY focus is more on how to implement the existing technical architecture and process to the cloud native, so the overly complicated and complex command line operation and maintenance mode is not suitable for me at present.

That’s when Rancher was found. After preliminary information retrieval, Rancher’s technical direction, popularity, community activity, and platform technical ability are all considered excellent and mainstream, so this part of content learning is intersperned.

Rancher’s main knowledge points come from official documents. The official documents are comprehensive and detailed. Good English documents can also be compared with English documents, but there are some minor differences.

The overall architecture

So for using Rancher itself and the managed cluster architecture, I drew an architecture diagram

Rancher high availability installation

The first thing we need to do is the installation of Rancher. Rancher supports a variety of installation methods, the simplest is standalone installation, using Docker to run an image on the line. I use a high availability installation. The latest version of 2.5 already supports installing Rancher on various Kubernetes clusters, such as existing Kubernetes clusters, or RKE, K3s, Amazon EKS, and more. It is recommended that the Kubernetes cluster be used exclusively to run Rancher for performance purposes.

After a brief comparison, we decided to use Rancher’s own K3s to deploy the Rancher application. Kubernetes and K3S

K3s is intended to be a fully compatible Kubernetes distribution with the following major changes from K8S:

  1. Old, Alpha, non-default features have been removed.
  2. Removed most of the internal cloud provider and storage plug-ins and can be replaced with plug-ins.
  3. SQLite3 is added as the default storage mechanism, etCD3 is still in effect, but it is no longer the default.
  4. Packaged in a simple launcher, you can handle a lot of LTS complexity and options.
  5. Minimized to no operating system dependencies, requiring only a kernel and cgroup mount.

And K3s is a standard compliant Kubernetes release that has been officially certified by CNCF, which has obvious advantages in the development environment or lightweight requirements.

The preparatory work

  • Mysql External database

    K3s uses a data store other than ETCD to run Kubernetes, making it somewhat more flexible. We can use an off-the-shelf database, such as Mysql, which we need to create here.

  • Load balancing

    When our Rancher node is running on K3s, a load balancer is required to direct traffic to multiple nodes. The official example uses Nginx to do L4 load balancing, with the following configuration file:

    worker_processes 4;
worker_rlimit_nofile 40000;

events {
    worker_connections 8192;
}

stream {
    upstream rancher_servers_http {
        least_conn;
        server 10.128.2.53:80 max_fails=3 fail_timeout=5s;   # # rancher node
        server 10.128.2.52:80 max_fails=3 fail_timeout=5s;
    }
    server {
        listen 80;
        proxy_pass rancher_servers_http;
    }

    upstream rancher_servers_https {
        least_conn;
        server 10.128.2.53:443 max_fails=3 fail_timeout=5s;
        server 10.128.2.52:443 max_fails=3 fail_timeout=5s;
    }
    server {
        listen     443;
        proxy_passrancher_servers_https; }}Copy the code

    To start nginx, you need to enable the Stream module.

  • Install kubectl

    Kubectl needs to be installed on the K3s machine to facilitate cluster management through the command line. Kubectl will also need to be installed on nodes in the Kubernetes cluster installed via Rancher if they need to be managed from the command line. As for how to install it, there is no need to describe it.

K3s management

  • The installation

    The installation of K3s is quite simple, and due to the introduction of external mysql database for storage, the nodes of K3s have become stateless. The requirement for the number of nodes does not require a certain three starting points. The official example is two nodes. The following command is used to install k3s

    curl -sfL http://rancher-mirror.cnrancher.com/k3s/k3s-install.sh | INSTALL_K3S_MIRROR=cn sh -s - server \ - datastore - the endpoint = "mysql: / / k3s: k3s123456 @ TCP (10.11.0.194:3306)/k3s"
#To install the specified version
curl -sfL https://get.k3s.io |  INSTALL_K3S_VERSION=vX.Y.Z sh -s - server \
--datastore-endpoint="mysql://username:password@tcp(hostname:3306)/database-name"
Copy the code

    Wait until the startup succeeds

    Now you can go through

    k3s kubectl get nodes
Copy the code

    To check the status of the K3S node. To access through kubectl tools, you need to configure the Kubeconfig file

    cp /etc/rancher/k3s/k3s.yaml ~/.kube/config
Copy the code

    The default server address in the ~/. Kube /config file is 127.0.0.1, which needs to be changed to the DNS of the front-end load balancer. Change the nginx configuration. Here I use haProxy + Keepalived for the cluster deployed according to Kubeadm, so HERE I continue to use the server address configured is keepalived VIP.

  • uninstall

    K3s stores the uninstallation file on the local PC. If you want to delete K3s, perform this operation

    /usr/local/bin/k3s-uninstall.sh
Copy the code

Helm installation

First paste the Helm document address.

  • What is Helm?

    Helm is a package management tool for Kubernetes that manages chart – pre-configured package resources, similar to Ubuntu’s APT and CentOS’s YUM

  • Helm installation

    About installation, the official installation part said very detailed, two ways to take.

  • Add the Rancher source

    If you install Rancher in Helm mode, you need to add a new Rancher source, which can be a stable version or the latest version

    #Stable version
helm repo add rancher-stable http://rancher-mirror.oss-cn-beijing.aliyuncs.com/server-charts/stable
#The latest version
helm repo add rancher-latest http://rancher-mirror.oss-cn-beijing.aliyuncs.com/server-charts/latest
Copy the code

Certificate generated

Rancher Server requires SSL/TLS configuration by default to ensure access security.

Tip: If you want to terminate SSL/TLS access externally, look at using an external TLS load balancer.

There are three recommended options for the source of the certificate that will be used to terminate TLS in Rancher Server:

  • Self-signed certificates generated by Rancher:In this case, you need to install in a clustercert-manager. Rancher usecert-managerIssue and maintain certificates. Rancher will generate its own CA certificate and use that CA to sign the certificate. And then,cert-managerResponsible for managing the certificate.
  • Let ‘s Encrypt:The Let’s Encrypt option is also requiredcert-manager. But, in this case,cert-managerUsed in conjunction with the special Issuer,cert-managerAll operations (including request and validation) required to obtain a certificate issued by Let’s Encrypt are performed. This configuration uses HTTP authentication (HTTP-01), the load balancer must have a public DNS record that can be accessed from the public network.
  • Use your own certificate:This option enables you to use certificates issued by your own authoritative CA or self-signed CA certificates. Rancher will use this certificate to protect WebSocket and HTTPS traffic. In this case, you must upload the name separately fortls.crtandtls.keyCertificate in PEM format and related key. If you use a private CA, you must also upload the certificate. This is because your nodes may not trust this private CA. Rancher takes this CA certificate and generates a checksum from it that various Rancher components use to verify their connection to Rancher.

The above is the official description of the certificate. Rancher provides three certificate methods. Domestic documents suggest choosing the method of your own certificate, for the reason that CERt-Manager needs extra energy in operation and maintenance and is persuaded to eat enough. Again, the following operations are based on the use of their own certificates!!

  • SSL Certificate Generation

    The official provided the script to generate the certificate, I posted here

    #! /bin/bash -ehelp () { echo ' ================================================================ ' echo ' --ssl-domain: If the primary domain name is not specified, the default domain name is www.rancher.local. If IP access services are used, the primary domain name can be ignored. 'echo' -- SSL-trusted - IP: generally, an SSL certificate trusts only the access requests of the domain name. Sometimes, you need to use the IP address to access the server. In this case, you need to add extended IP addresses to the SSL certificate. 'echo' -- SSL-trusted -domain: If you want to access multiple domains, add the extended domain name (SSL_TRUSTED_DOMAIN), separated by commas. 'echo' --ssl-size: SSL encryption bits (default: 2048); 'echo' --ssl-cn: country code (2-letter code), default cn; /create_self-signed-cert.sh --ssl-domain=www.test.com --ssl-trusted-domain=www.test2.com \ ' Echo '-- SSL - trusted - IP = 1.1.1.1 2.2.2.2, 3.3.3.3 - SSL - size = 2048 - SSL - date = 3650' echo ' ================================================================' } case "$1" in -h|--help) help; exit;; esac if [[ $1 == '' ]]; then help; exit; fi CMDOPTS="$*" for OPTS in $CMDOPTS; do key=$(echo ${OPTS} | awk -F"=" '{print $1}' ) value=$(echo ${OPTS} | awk -F"=" '{print $2}' ) case "$key" in --ssl-domain) SSL_DOMAIN=$value ;; --ssl-trusted-ip) SSL_TRUSTED_IP=$value ;; --ssl-trusted-domain) SSL_TRUSTED_DOMAIN=$value ;; --ssl-size) SSL_SIZE=$value ;; --ssl-date) SSL_DATE=$value ;; --ca-date) CA_DATE=$value ;; --ssl-cn) CN=$value ;; esac done
#CA Configuration
CA_DATE=${CA_DATE:-3650}
CA_KEY=${CA_KEY:-cakey.pem}
CA_CERT=${CA_CERT:-cacerts.pem}
CA_DOMAIN=cattle-ca

#SSL Configuration
SSL_CONFIG=${SSL_CONFIG:-$PWD/openssl.cnf}
SSL_DOMAIN=${SSL_DOMAIN:-'www.rancher.local'}
SSL_DATE=${SSL_DATE:-3650}
SSL_SIZE=${SSL_SIZE:-2048}

## country code (2-letter code), default CN;CN=${CN:-CN} SSL_KEY=$SSL_DOMAIN.key SSL_CSR=$SSL_DOMAIN.csr SSL_CERT=$SSL_DOMAIN.crt echo -e "\033[32m -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- \ [0 m "echo - e" 033\033 [32 m | generated SSL Cert | \ [0 m "echo - e" 033\033 [32 m -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- \ [0 033 m "if [[- e. / ${CA_KEY}]]; then the echo - e" 033 \ [32 m = = = = > 1. Found existing CA private key, ${CA_KEY} = ${CA_KEY} \033[0m" mv ${CA_KEY}" ${CA_KEY}"-bak openssl genrsa -out ${CA_KEY} ${SSL_SIZE} else echo -e "\033[32m ====> 1 ${CA_KEY} \033[0m" openssl genrsa -out ${CA_KEY} ${SSL_SIZE} fi if [[-e./${CA_CERT}]]; then echo -e "\ [033 32 m = = = = > 2. Found existing CA certificate, first backup" ${CA_CERT} "to" ${CA_CERT} "- bak, \033[0m" mv ${CA_CERT}" ${CA_CERT}"-bak openssl req-x509-sha256-new-nodes -key ${CA_KEY} -days ${CA_DATE} - out ${CA_CERT} - subj "/ C = ${CN} / CN = ${CA_DOMAIN}" the else echo - e \ 033 "[32 m = = = = > 2. Generate new CA certificate ${CA_CERT} \ 033 [0 m" openssl req -x509 -sha256 -new -nodes -key ${CA_KEY} -days ${CA_DATE} -out ${CA_CERT} -subj "/C=${CN}/CN=${CA_DOMAIN}" fi echo -e "\033[32m ====> 3. Generate the Openssl configuration file ${SSL_CONFIG} \033[0m" cat > ${SSL_CONFIG} <<EOM [req] req_extensions = v3_req distinguished_name = req_distinguished_name [req_distinguished_name] [ v3_req ] basicConstraints = CA:FALSE keyUsage = nonRepudiation, digitalSignature, keyEncipherment extendedKeyUsage = clientAuth, serverAuth EOM if [[ -n ${SSL_TRUSTED_IP} || -n ${SSL_TRUSTED_DOMAIN} ]]; then cat >> ${SSL_CONFIG} <<EOM subjectAltName  = @alt_names [alt_names] EOM IFS="," dns=(${SSL_TRUSTED_DOMAIN}) dns+=(${SSL_DOMAIN}) for i in "${!dns[@]}"; do echo DNS.$((i+1)) = ${dns[$i]} >> ${SSL_CONFIG} done if [[ -n ${SSL_TRUSTED_IP} ]]; then ip=(${SSL_TRUSTED_IP}) for i in "${! IP / @}"; do echo IP $(+ 1) (I) = ${IP [$I]} > > ${SSL_CONFIG} done fi fi echo - e \ 033 "[32 m = = = = > 4. SSL KEY generation service ${SSL_KEY} \033[0m" openssl genrsa -out ${SSL_KEY} ${SSL_SIZE} echo -e "\033[32m ====> 5. \033[0m" openssl req -sha256 -new -key ${SSL_KEY} -out ${SSL_CSR} -subj "/C=${CN}/CN=${SSL_DOMAIN}" -config ${SSL_CONFIG} echo -e "\033[32m ====> 6. Generate service SSL CERT ${SSL_CERT} \033[0m" openssl x509-sha256-req -in ${SSL_CSR} -ca ${CA_CERT} \ -CAkey ${CA_KEY} -CAcreateserial -out ${SSL_CERT} \ -days ${SSL_DATE} -extensions v3_req \ -extfile ${SSL_CONFIG} echo -e "\033[32m ====> 7. Certificate making is complete \033[0m" echo echo -e "\033[32m ====> 8. Output result \033[0m" echo in YAML format "----------------------------------------------------------" echo "ca_key: |" cat $CA_KEY | sed 's/^/ /' echo echo "ca_cert: |" cat $CA_CERT | sed 's/^/ /' echo echo "ssl_key: |" cat $SSL_KEY | sed 's/^/ /' echo echo "ssl_csr: |" cat $SSL_CSR | sed 's/^/ /' echo echo "ssl_cert: | "cat $SSL_CERT | sed 's ^ / / /' echo echo - e" 033 \ [32 m = = = = > 9. Additional CA certificate to the Cert file \ [0 033 m "cat ${CA_CERT} > > ${SSL_CERT} echo "ssl_cert: | "cat $SSL_CERT | sed 's ^ / / /' echo echo - e" 033 \ [32 m = = = = > 10. Rename the service certificate \ 033 [0 m "echo" cp ${SSL_DOMAIN} key TLS. Key "cp  ${SSL_DOMAIN}.key tls.key echo "cp ${SSL_DOMAIN}.crt tls.crt" cp ${SSL_DOMAIN}.crt tls.crtCopy the code

    The instructions are as follows

    --ssl-domain: specifies the primary domain name required for generating SSL certificates. If this domain name is not specified, the default domain name is www.rancher.local. If IP access services are used, the domain name can be ignored. -- SSL-trusted - IP: Generally, SSL certificates only trust access requests by domain names. Sometimes, you need to use IP addresses to access the server. In this case, you need to add extended IP addresses to SSL certificates. -- SSL-trusted -domain: If you want to access multiple domains, add the extended domain name (TRUSTED_DOMAIN). Separate multiple domains with commas (,). -- SSL-size: SSL encryption bits (2048 by default); Ssl-cn: country code (2-letter code), default cn; Generate an example use of a 10-year validity period: . / create_self - signed - cert. Sh \ - ssl-domain=rancher.xxx.com - SSL - trusted - IP = 10.11.0.220 10.128.2.52, 10.128.2.53 --ssl-size=2048 --ssl-date=3650Copy the code

  • Certificate submitted

    #New rancher namespace
kubectl create namespace cattle-system
#Create a secret
kubectl create secret generic tls-ca -n cattle-system --from-file=cacerts.pem
kubectl create secret tls tls-rancher-ingress -n cattle-system --cert=./tls.crt --key=./tls.key
Copy the code

Install the Rancher

#Helm, install the rancher
helm install rancher rancher-stable/rancher \
  --namespace cattle-system \
  --set hostname=rancher.xxx.com \
  --set ingress.tls.source=secret \
  --set privateCA=true

#Helm, update the rancher
helm upgrade --install rancher rancher-stable/rancher \
  --namespace cattle-system \
  --set hostname=rancher.xxx.com \
  --set ingress.tls.source=secret \
  --set privateCA=true

Copy the code

You can use the kubectl command to check the startup status. This process will take a while before the entire Rancher cluster becomes active.

kubectl get ingress -n cattle-system
kubectl -n cattle-system rollout status deploy/rancher
Copy the code

Use the Rancher

If Intranet IP domain name mapping is used before accessing the Rancher page, you need to configure the domain name and IP address mapping on the local host. Browser access domain name

When the above page appears, you can set your password and use Rancher normally.

Set Ali Source

In system Settings

Set to

registry.cn-hangzhou.aliyuncs.com
Copy the code

Create a custom cluster

Earnest advice

To start with, the cluster I used to create was not clean and had both Java processes and Docker deployed services on it. After the cluster was created, the Java process was normal, but all Docker deployed services could not be accessed. I guess the reason is that CNI was set up during the cluster creation, which affected the network Settings of Docker.

So I strongly suggest that when creating a custom cluster using Rancher, it’s best to use a clean, service-free machine and avoid having Docker deployed services on it.

Create the cluster

I won’t go into the details of how Rancher creates a cluster, but you just have to go through the next step. Finally, the command for the node machine to join the cluster appears, and you just execute the command on the relevant machine. The whole cluster creation process let me sum up in one word: wait! It could be a long time…

Many images are downloaded on each node machine and multiple containers are started

The Rancher image also displays the current cluster information

It’s a long process for me, at least a lot longer than the seven or eight minutes that some people say.

After the cluster status is not successful, you can download the Kubeconfig file to the local, then you can operate the cluster through Kubectl.

Dashboard error and kubectl command line 1006 issue

After the Kubernetes cluster is successfully created, two problems are found:

  • Click cluster dashboard error, into the blank page
  • Execute the kubectl command line1006 error appears after the button is clicked

Check the POD status from the kubectl command line

Cattle-cluster-agent two nodes are found to be in CrashLoopBackOff state and are being restarted continuously. So look at the log

The rancher.xxx.com domain name cannot be accessed, and nodes in the cluster do not know who Rancher.xxx.com is

kubectl -n cattle-system patch deployments cattle-cluster-agent --patch '{ "spec": { "template": { "spec": {" hostAliases ": [{" hostnames:"/" rancher.xxx.com ", "IP" : "10.11.0.220}]}}}} '#Configuration cattle - node - agentkubectl -n cattle-system patch daemonsets cattle-node-agent --patch '{ "spec": { "template": { "spec": { "hostAliases": [{" hostnames: "/" rancher.xxx.com ", "IP" : "10.11.0.220}]}}}} 'Copy the code

All is normal after configuration.

Clean up the node

If you want to uninstall a node that has been installed in a cluster, you need to run the cleanup script provided by the system. Otherwise, the system is not guaranteed to succeed next time.

In fact, this process took me quite a bit of time, the official documentation is complete, but the order is debatable, and of course I have my own pot. I’ll leave you with this article and I’ll cover deploying applications in the future. See you then.