The author

Liang Hao, Tencent TEG engineer, cloud native open source fan, SuperEdge developer, is now in charge of tKEX-TEG container platform operation and maintenance related work.

Wang Dong, TKE background research and development engineer of Tencent Cloud, focuses on the native field of container cloud, and is the core developer of SuperEdge. He is now in charge of the work related to the privatization of TKE Edge container of Tencent cloud Edge.

Li Tengfei, Tencent container technology RESEARCH and development engineer, Tencent Cloud TKE background research and development, SuperEdge core development member.

Addon SuperEdge allows native K8s clusters to manage edge applications and node video links

Edge capacity

After talking about Addon SuperEdge, what will your normal Kubernetes cluster have:

It manages both central applications and nodes as well as edge applications and nodes

You can manage applications and nodes in a Kubernetes cluster, or you can manage applications and nodes on the edge, without making any difference. This design will Kubernetes distributed logic concept and the physical location of the actual IDC room perfect fit, to achieve the real meaning of cloud side a cloud.

Can Join nodes at any location

An edgeadm join command adds a node to the central cluster whenever it has access to the Kube-apiserver, regardless of where it is or how it exists. With a Kubernetes cluster, you can manage all the machines in a unified way, pooling the computing power of each IDC room, and maximizing the utilization of the machines.

At the same time manage dozens of edge dot, and have grayscale ability

The ServiceGroup design of SuperEdge allows an application to be deployed to dozens of nodes at the same time as long as it is submitted once, and the flow loop within each node is realized. Moreover, each node is allowed to be different on the basis of the basic application, no matter the gray scale or the differentiated configuration can be supported.

Edge nodes have the capability of edge autonomy

The network between the edge node and kube-Apiserver is unpredictable. It may be a dedicated line or WIFI. Both the public network and the Intranet may exist. SuperEdge allows edge nodes to operate without affecting edge services even if the cloud side is disconnected. Even if the power is cut off and the edge services are restarted, the edge services can be pulled up normally.

Addon SuperEdge edge capabilities

The installation conditions

  • The Kubernetes cluster has been created, and all kube-controller-Managers have been enabled
--controllers=*,bootstrapsigner,tokencleaner
Copy the code

parameter

  • At present, only Kubernetes cluster built through kubeadm is supported. Other Kubernetes cluster is not not supported, but may have compatibility problems, see the principle part;

  • If there is no Kubernetes cluster built by Kubeadm, you can use edgeadm init one key to create a native Kubernetes cluster, reference: using edgeadm one key to create edge k8S cluster or native K8S cluster;

  • Support for Kubernetes v1.16~ V1.19. Edgeadm provides the installation package of Kubernetes V1.18.2.

    • Users need to consider edge nodeskubeletCompatibility with the original Kubernetes version, unification is best;
    • Other Kubernetes versions can be seen in the one-click Install Edge Kubernetes cluster 5. Custom Kubernetes static installation package, made by yourself.

Download the edgeadm static installation package

Download the edgeadm static installation package from any Master node and copy it to the edge node to be added to the cluster.

Arch =amd64 version= v0.4.0&& rm -rf edgeadm-linux-* && wget https://superedge-1253687700.cos.ap-guangzhou.myqcloud.com/$version/$arch/edgeadm-linux-$arch-$version.tgz && tar -xzvf edgeadm-linux-* && cd edgeadm-linux-$arch-$version && ./edgeadmCopy the code

Current support currently support [[AMD64, ARM64], two systems, download their own machine corresponding architecture, other systems can compile edgeADM and make the corresponding system installation package, For details, see One-click Kubernetes Cluster Installation. 5. Customize the Static Installation Package of Kubernetes

Addon SuperEdge

Addon edge capability components on any Master node in the original cluster

/edgeadm addon edge-apps --ca.cert < cluster CA certificate address > --ca.key < cluster CA certificate key path > --master-public-addr < External IP address of the Master node/Internal IP address/domain name of the Master node >:<Port>Copy the code

Among them:

  • – ca. Cert: cluster ca certificate path, the default/etc/kubernetes/pki/ca. CRT
  • – ca. Key: cluster ca certificate key path, the default/etc/kubernetes/pki/ca. Key
  • –master-public-addr: specifies the IP address of the edge node to access the Kube-apiserver service. Default :< internal IP address of the master node >:< port >

If the edgeadm addon edge-apps procedure is correct, the terminal will print the following log:

. I0606 12:52:51.976165 26593 deploy_tunnel.go:35] Deploy tunnel-coredns.yaml success! Create tunnel-cloud.yaml success!Copy the code

If any problem occurs during the execution, an error message is returned and the installation of edge components is interrupted. You can run the./edgeadm detach command to uninstall edge components and restore the cluster.

./edgeadm detach edge-apps --ca.cert < cluster CA certificate address > --ca.key < cluster CA certificate key path >Copy the code

The original Kubernetes cluster becomes a Kubernetes that manages cloud applications as well as delivering and managing edge applications.

Join edge node

The installation conditions

The edge node complies with the minimum requirements of KUbeadm. The minimum disk space is 2C2G and the disk space is not less than 1G.

⚠️ Note: provide clean machines as much as possible to avoid other factors causing installation errors. There are container services on the machine that may be cleaned up during installation, please check carefully before executing.

Create the token of the Join edge node

After the edge capability component addon is successful, Join edge node and kubeadm are used similarly. You can run the following command to obtain:

./edgeadm token create --print-join-command 
Copy the code

If there is no problem during the execution, the terminal will output the Join command

. Edgeadm join <Master node internal IP>:Port --token XXXX \ --discovery-token-ca-cert-hash sha256: XXXXXXXXXXCopy the code

Note: The validity period of the created token is 24h, the same as that of kubeadm. After the expiration, you can run the above command again to obtain the Join command.

The edge node is added to the original cluster

Download the edgeadm static installation package from 2.<2>. Or upload the edgeadm static installation package to the edge node by other means, and then run the Join command obtained from 3.<2> on the edge node:

/edgeadm join <Master Node extranet IP address /Master node internal IP address/domain name >:Port --token XXXX \ --discovery-token-ca-cert-hash sha256: XXXXXXXXXX --install-pkg-path <edgeadm Kube-* static installation package address /FTP path >Copy the code

Note: You can use edgeadm create token –print-join-command to replace the service address of the join prompt command kube-apiserver with the external IP address of the Master node or the internal IP address of the Master node if necessary. Depending on whether you want the edge node to access the Kube-Apiserver service from the Internet or from the Intranet, the default output is the internal IP address of the Master node.

Among them:

  • –install-pkg-path: Kubernetes static installation package path;
  • < External IP address of the Master node/Internal IP address of the Master node/domain name of the Master node >:Port is the IP address of the edge node to access the Kube-Apiserver service

If nothing goes wrong and the new Node is successfully added to the cluster, the following output is displayed:

This node has joined the cluster:* Certificate signing request was sent to apiserver and a response was received.* The Kubelet was informed of the new secure connection details.Run 'kubectl get nodes' on the control-plane to see this node join the cluster.
Copy the code

If any problem occurs during the operation, an error message is displayed and node adding is interrupted. You can run the./edgeadm reset command to roll back the operation and Join the node again.

IO /edge-node=enable, which is convenient for subsequent applications to select and schedule applications to edge nodes through nodeSelector.

If you have any problems with the above operations, you can join SuperEdge’s Slack, Google forum, wechat group and communicate with us, and you can also raise Issues in the community to give us feedback.

Realize the principle of

The main steps

In fact, the principle of Addon SuperEdge implementation is very simple, mainly divided into two steps:

  • Step 1: Addon SuperEdge edge capability component;
  • Step 2: Prepare the configuration required for the edge node to Join the native Kubernetes cluster;

The first step, which is not detailed, is to Addon the edge power components into the native Kubernetes cluster. Users can also easily uninstall the SuperEdge Edge power components using edgeadm Detach.

The key is the second step: Join the edge nodes into the native Kubernetes cluster, facing the following problems:

1. Kube-apiserver certificate processing

The kube-Apiserver server certificate of the native Kubernetes cluster may not have the external IP address or the external access domain name, so how can the edge node access the Kube-Apiserver using the public IP address?

2. The user’s Kubernetes cluster is not built by Kubeadm

Kubeads-config, kubelet-config, cluster-info, etc. Kubeads-config, kubelet-config, cluster-info, etc.

3. The user’s Kubernetes cluster is set up by Kubeadm

If the user’s Kubernetes cluster is built by Kubeadm, how can the edge node Join the user’s Kubernetes with the public network? How can the edge node ensure that the user’s original Join common node is not affected? …

So the core of implementing Addon Superedge is not the edge capability component of Addon Superedge, but how to Join edge nodes into the user’s original Kubernetes cluster without having any impact on the user’s Kubernetes cluster.

Design practice

The overall implementation idea is to prepare the configuration of Kubeadm Join node for any Kubernetes cluster built by users, and then Join edge nodes like Kubeadm Join.

1. Kube-apiserver certificate processing

For kube-apiserver, we did not replace the user’s original Kubernetes cluster certificate and restart kube-apiserver, so how do we implement the edge node to add to the user’s Kubernetes cluster through the public network? The answer is by writing to the edge node hosts. Kube -apiserver for Kubernetes

kuberneteskubernetes.defaultkubernetes.default.svckubernetes.default.svc.cluster.local
Copy the code

When the user executes the Join command:

/edgeadm join<Master external IP address /Master internal IP address/domain name >:Port...Copy the code

We give the edge node the hosts:

External IP address of the Master node/Internal IP address of the Master node kubernetes.defaultCopy the code

Then the edge node accesses the cloud kube-apiserver through kubernetes.default:Port, which not only circumvents the problem that the kube-Apiserver certificate does not sign the public IP address, but also successfully joins the edge node to the user’s Kubernetes cluster. For domain names, we do not write hosts, so users need to pay attention to whether the given domain name is signed in the kube-apiserver certificate. What if the user’s Kube-Apiserver certificate is not signed to kubernetes. Default?

When the user joins the edge node, the user can only use the domain name instead of the IP address. The user needs to ensure that the given domain name has signed the address on the Kube-Apiserver certificate of Kubernetes, and the edge node can access it.

2. The user’s Kubernetes cluster is not built by Kubeadm

Processing method is not built by Kubeadm, it becomes built by Kubeadm, as long as the information required by Kubeadm join is ready, then it can be similar to Kubeadm join way to join the edge node cluster. The specific implementation is to prepare the conditions for edgeadm join edge node during the stage of edgeadm addon edge-apps installation and deployment of edge components. The typical conditions are as follows:

  • Create the kube-public namespace and the ConfigMap of cluster-info in the namespace
  • Create a ConfigMap containing kubeadm-config and kubelet-config in the kube-system namespace
  • The RBAC permission required by Kubelet to access ConfigMap and other resources in the cluster when an edge node is added to the cluster

3. The user’s Kubernetes cluster is set up by Kubeadm

When executing edgeadm addon edge-apps, we will check the ConfigMap to be added. If the cluster information of the user already exists, it will not be added. If the cluster information set up by user kubeadm, such as ConfigMap of cluster-info, is not enough for us, we will generate a new one to avoid changing and overwrite the user’s original one.

After the above configuration, edge nodes can join the cluster using the edgeadm join command. For more details, see the source edgeadm addon edge-apps for details.

Collaboration and open source

TKE Edge Edge container management service Edge computing power core components have been open source to the SuperEdge project, welcome to build Edge computing, participate in the construction of SuperEdge open source project, let you develop Edge power benefit more people.

SuperEdge version:

  • SuperEdge – V0.4.0

  • SuperEdge – V0.3.0

  • SuperEdge – V0.2.0

TKE Edge

  • SuperEdge open source project

  • Tencent Cloud joined with a number of ecological partners to launch a blockbuster open source SuperEdge edge container project

  • TKE Edge Container series: Install edge K8s cluster and native K8s cluster with edgeadm

  • Learn about SuperEdge from 0 to N, these dry goods must see! 【18 Dry Goods collection 】

  • [TKE Edge Container series] Break down Intranet barriers and add hundreds or thousands of edge nodes from the cloud at a time

  • SuperEdge cloud Edge Tunnel features: Edge node operation and maintenance from the cloud SSH

  • 【TKE edge container series 】 this article reads SuperEdge edge container architecture and principle

  • SuperEdge distributed health check edges

  • 【TKE edge container series 】 the article reads SuperEdge topological algorithm

  • 【TKE Edge container series 】 the article reads SuperEdge cloud side tunnel

Related materials of landing cases:

  • Edge container practice of Tencent WeMake Industrial Internet Platform: To build a more efficient industrial Internet

  • After the explosion. With the edge container, it can realize the workload of seven or eight people in a week in seconds

  • Construction of industrial Internet Platform based on edge container technology

  • Deploy EdgeX Foundry using TKE Edge