Kubernetes integration with KubeEdge

Author: Zhu Han

Recently, partners have many questions about the integration of edge nodes on the use of KubeSphere V3.1. Here is the guide document address, and then we can summarize the problems in this aspect here, so as to facilitate follow-up troubleshooting of partners, and welcome you to continue to add.

Official Guide document portal

• Activate kubeedge
• Edge node addition

1. The IP address and port are open

If you use ks-Installer to install KubeEdge, activate KubeEdge and configure an external IP address for the master node.

Assume that the IP address of the master node in the cluster is 192.168.10.7, and the following external network ports need to be opened:

The firewall is required to pass the extranet port.

If you forgot to set the KubeEdge component IP address in ks-Installer, cloudHub cannot get up, you can use the following command:

 kubectl -n kubeedge edit cm cloudcore
Copy the code

2. Obtain edge node logs and metrics

There are many cases where the edge node and the master node are not in the same LAN, so we design the use of external network port communication above. In addition, because KubeEdge completes kubectl exec, obtains logs and metrics, relies on virtual IP to carry out corresponding IPtable forwarding to cloudhub corresponding port, and then obtains edge data. Therefore, the virtual IP address bound to the edge node and the name of the edge node must be unique, which should be maintained according to the rules. Note that the virtual IP address cannot be the internal IP address of the edge node, and it is better to choose the network end that does not conflict with the internal network. To ensure that the metrics-server component is enabled, update to version 0.4.1 or higher to accommodate KubeEdge (the current version mainly obtains edge metrics from metrics-server).

3. If edge nodes use external IP and port for communication, some Daemonset nodes have strong tolerance, such as Calico. Patch should be given to them to avoid scheduling to the edge end

#! /bin/bash

NodeSelectorPatchJson='{"spec":{"template":{"spec":{"nodeSelector":{"node-role.kubernetes.io/master": "","node-role.kubernetes.io/worker": ""}}}}} '
NoShedulePatchJson='{"spec":{"template":{"spec":{"affinity":{"nodeAffinity":{"requiredDuringSchedulingIgnoredDuringExecution":{"nodeSelecto rTerms":[{"matchExpressions":[{"key":"node-role.kubernetes.io/edge","operator":"DoesNotExist"}]}]}}}}}}}'

edgenode="edgenode"
if [ The $1 ]; then
        edgenode="The $1"
fi

namespaces=($(kubectl get pods -A -o wide |egrep -i $edgenode | awk '{print $1}' ))
pods=($(kubectl get pods -A -o wide |egrep -i $edgenode | awk '{print $2}' ))
length=${#namespaces[@]}

for((i=0; i<$length; i++));do
        ns=${namespaces[$i]}
        pod=${pods[$i]}
        resources=$(kubectl -n $ns describe pod $pod | grep "Controlled By" |awk '{print $3}')
        echo "Patching for ns: $ns, resources: $resources"
        kubectl -n $ns patch $resources --type merge --patch "$NoShedulePatchJson"
        sleep 1
done
Copy the code

4. kubelet cgroup driver: “cgroupfs” is different from docker cgroup driver: “systemd”

Cause: The cgroup used by kubelet and master is inconsistent

Solutions:

Add config “execu-opts “: [” native-cgroupdriver =systemd”], then restart Docker, systemctl daemon-reload && systemctl restart kubelet

Reference Description:

• kubeedge/kubeedge#1772 (comment) cgroup driver shouldn’t be taken care by the keadm, I think this is also true for “kubeadm”, all that care about the cgroup driver is actually which cri you are using, for example, in your case, you are using docker, so you just need to update the docker configuration to use “systemd” instead.

5. How to join KubeSphere management when edge nodes and Kubernetes are in the same LAN?

If edge nodes and Kubernetes cluster are on the same LAN, you can use nodePort to join edge nodes. The default nodePort is 3000-30004. Therefore, when edge nodes join the cluster, external ports 10000-10004 should be changed to 30,000-30004. X :10000 –quicport 10001 –certport 10002 — TunnelPort 10004 –cloudcore-ipport=192.168.x.x:30000 — Quicport 30001 –certport 30002 — TunnelPort 30004 Specify the application scenarios of edge nodes.

See the Guide guide for more information.

6. The lowest version of Docker supported by edge node Pod Metrics

The edge of the end support Docker version should be greater than or equal to v19.3.0, specific reasons can reference kubesphere.com.cn/forum/d/449…

KubeEdge v1.6.2 has been released and the bug has been fixed.

The edge needs to be upgraded to V1.6.2, you can also add the following configuration to modify:

apiVersion: v1
kind: ConfigMap
metadata:
  name: edge-watcher-config
  namespace: kubeedge
data:
  version: v1.6.2
  region: zh
Copy the code

7. Node Metrics displays abnormal exclusion guidelines

• Check whether the metrics-server service is normal. Check whether the version of the metrics-server deployment is normal (v0.4.1 or higher). Check whether kubectl Top is abnormal.

  kubectl apply -f https://raw.githubusercontent.com/kubesphere/ks-installer/master/roles/metrics-server/files/metrics-server/metrics-serve r.yamlCopy the code

• All the other nodes have metrics, but the edge nodes don’t have metrics

  • To check whether the iptables pod is properly forwarded, run the following command to check which node the iptables POD is deployed on:

  kubectl get pods -n kubeedge -o wide
  Copy the code

  • Obtain the NAT forwarding rule 10350 on the corresponding node and ensure that the forwarding destination is the K8S master or node IP address

  • If the forwarding destination is not correct, use the following method:

  kubectl -n kubeedge edit cm edge-watcher
  Copy the code

  

  Restart edge-Watcher-Controller-Manager Deployment after modification

8. Certificate issues

kubectl delete secret casecret cloudcoresecret -n kubeedge
Copy the code

Cloudcore needs to be restarted.

This article is published by OpenWrite!