Preface:

On August 2nd, Broadcloud officially launched FabEdge, an open source network solution for edge computing scenarios based on K8S and Kubedge. Since its release, FabEdge has received a lot of attention from developers who have made valuable suggestions for FabEdge. At the same time, we noticed that during the process of installing and deploying FabEdge, users encountered the challenge of not being able to experience FabEdge because they could not build Kubernetes + Kubedge cluster.

In response, the FabEdge team has introduced the one-click deployment of K8S and Kubedge. This article will show you how to quickly deploy clusters and get a hands-on experience with the FabEdge project.

FabEdge Installation Guide video introduction

Rapidly deploy a K8S cluster

The installation conditions

• Comply with the minimum requirements of Kubeadm, Master && Node minimum 2 GB, disk space is not less than 10 GB.

⚠️ Note: provide clean machine as far as possible, avoid installation errors caused by other factors.

Supported operating systems

• Ubuntu 18.04.5 Server 4.15.0-136- Generic (Recommended)

• Ubuntu 20.04.2 Server 5.4.0-66 – generic

• CentOS Linux release 7.9.2009 (Core)

• CentOS Linux release 7.8.2003 (Core)

Deploy the K8S cluster

1. Install the K8S Master node

Using Ubuntu 18.04.5 as an example, run the following commands:

Root @ master: ~ # curl http://116.62.127.76/FabEdge/fabedge/main/deploy/cluster/install-k8s.sh | bash -Copy the code

Copy the code

⚠️ Note: If the loading time is too long, the network speed is slow. Please wait patiently

If the following information is displayed, the installation is successful:

PLAY RECAP *********************************************************************master                     : ok=15   changed=13   unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
Copy the code

Copy the code

2. Add k8S edge nodes

Root @ master: ~ # curl http://116.62.127.76/FabEdge/fabedge/main/deploy/cluster/add-edge-node.sh | bash - s - - host - vars ansible_hostname={hostname} ansible_user={username} ansible_password={password} ansible_host={edge-node-IP}Copy the code

Copy the code

Parameter Description:

• Ansible_hostname Specifies the hostname of the edge node

• Ansible_user Specifies the user name of the edge node

• Ansible_password Configures the password for edge nodes

• Ansible_host Specifies the IP address of the edge node

  For example, set the host name edge1, user name root, password pwd111, and IP address 10.22.45.26 of the edge node as follows:

  Root @ master: ~ # curl http://116.62.127.76/FabEdge/fabedge/main/deploy/cluster/add-edge-node.sh | bash – s – – host – vars Ansible_hostname = edge1 ansible_user = root ansible_password = pwd111 ansible_host = 10.22.45.26

Copy the code

If the following information is displayed, the installation is successful:

PLAY RECAP *********************************************************************edge1                      : ok=13   changed=10   unreachable=0    failed=0    skipped=1    rescued=0    ignored=0
Copy the code

Copy the code

3. Confirm that the node is successfully added

root@master:~# kubectl get nodeNAME STATUS ROLES AGE VERSIONedge1 Ready Agent, Edge 22m V1.19.3 - KubeEdge -v1.5.0master Ready master node 32 m v1.19.7Copy the code

Copy the code

⚠️ Note: If no password is configured on the edge node, you need to configure the SSH certificate.

Configure the SSH certificate for the master node.

root@master:~# docker exec -it installer bashroot@master:~# ssh-copy-id {edge-node-IP}
Copy the code

Copy the code

FabEdge deployment

Close the rp_filter

Execute the following command on all cloud nodes:

root@master:~# for i in /proc/sys/net/ipv4/conf/*/rp_filter; do echo 0 >$i; Done# save configuration root@master:~# vi /etc/sysctl.conf.. net.ipv4.conf.default.rp_filter=0net.ipv4.conf.all.rp_filter=0.. # confirm the configuration take effect root @ master: ~ # sysctl -a | grep rp_filter | grep -v arp.. net.ipv4.conf.cali18867a5062d.rp_filter = 0net.ipv4.conf.cali6202a829553.rp_filter = 0..Copy the code

Copy the code

Check the nodelocaldns service status

• Verify that the pod of Nodelocaldns starts properly on all edge nodes

  Root @ master: ~ # kubectl kube get Po – n – system – wide o | grep nodelocaldnsnodelocaldns – 4 m2jx 1/1 Running 25 m 10.22.45.30 0 Master nodelocaldns- P5h9k 1/1 Running 0 35m 10.22.45.26 edge1

Copy the code

Get Fabedge

root@master:~# git clone https://github.com/FabEdge/fabedge.git
Copy the code

Copy the code

Generate a certificate for StrongSwan

• Generate certificates for each edge node, using edge1 as an example:

  root@master: Kubectl get node NAME STATUS ROLES AGE VERSION EDge1 Ready agent, EDGE 47M V1.19.3 – KubeEdge -v1.1.0 master Ready Master,node 57m v1.19.7# Cloud execute, generate certificate root@master: # docker run – rm – v/ipsec. D: / ipsec. D fabedge/strongswan: latest/genCert sh # edge1 login edge node, create a directory on the edge nodes edge1 root @ edge1: # mkdir -p /etc/fabedge/ipsec root@edge1: # CD /etc/fabed/ipsec root@edge1: /etc/fabed/ipsec # mkdir -p cacerts certs private # copy the generated certificate to the edge node, # note the certificate name: Edge1_cert -> edgecert.pem, edge1.ipsec. Secrets -> ipsec. # scp /ipsec.d/cacerts/ca.pem @edge1:/etc/fabedge/ipsec/cacerts/ca.pemroot@master: # scp /ipsec.d/certs/edge1_cert.pem @edge1:/etc/fabedge/ipsec/certs/edgecert.pemroot@master: # scp /ipsec.d/private/edge1_key.pem @edge1:/etc/fabedge/ipsec/private/edge1_key.pemroot@master: # scp /ipsec.d/edge1.ipsec.secrets @edge1:/etc/fabedge/ipsec/ipsec.secrets

Copy the code

• Generate a certificate for the Connector service and copy it to the node running the Connector service. Take master as an example:

  root@master: Kubectl get node NAME STATUS ROLES AGE VERSION EDge1 Ready agent,edge 62m V1.19.3 – kubeEdge -v1.1.0 master Ready Master,node 72m v1.19.7 # create certificate on master root@master: # docker run – rm – v/ipsec. D: / ipsec. D fabedge/strongswan: latest/genCert sh connector # on the master, create the directory root @ master: # mkdir -p /etc/fabedge/ipsec root@master: # CD/etc/fabedge/ipsec root @ master: / etc/fabedge ipsec# mkdir -p cacerts certs private # on the master, root @ master copy certificate: # cp /ipsec.d/cacerts/ca.pem /etc/fabedge/ipsec/cacerts/ca.pemroot@master: # cp /ipsec.d/certs/connector_cert.pem /etc/fabedge/ipsec/certs/connector_cert.pemroot@master: # cp /ipsec.d/private/connector_key.pem /etc/fabedge/ipsec/private/connector_key.pemroot@master: # cp /ipsec.d/connector.ipsec.secrets /etc/fabedge/ipsec/ipsec.secrets

Copy the code

Creating a namespace

• Create namespace for fabedge resources. Default is Fabedge.

  root@master:~# kubectl create ns fabedge

Copy the code

The deployment of the Connector

• Select a node in the cloud and run connector to mark the node. Take master as an example:

  root@master: # kubectl get node NAME STATUS ROLES AGE VERSION EDge1 Ready agent,edge 107M V1.19.3 – kubeEdge -v1.1.0 master Ready The master node 117 m v1.19.7 root @ master: # kubectl label no master node-role.kubernetes.io/connector=root@master:~# kubectl get node NAME STATUS ROLES AGE VERSION EDGE1 Ready Agent, Edge 108M V1.19.3 – KubeEdGE-V1.1.0 Master Ready Connector, Master, Node 118M V1.19.7

Copy the code

• Modify connector configuration

Modify edgePodCIDR, IP, and sbunets based on the actual environment

root@master:~# vi ~/fabedge/deploy/connector/cm.yaml
Copy the code

Copy the code

data: connector.yaml: | tunnelConfig: /etc/fabedge/tunnels.yaml certFile: /etc/ipsec.d/certs/connector_cert.pem viciSocket: /var/run/charon.vici # period to sync tunnel/route/rules regularly syncPeriod: 5m edgePodCIDR: 10.10.0.0/16 # Namespace for Fabedge resources fabedgeNS: fabedge debounceDuration: 5s tune. yaml: | # connector identity in certificate id: C=CN, O=StrongSwan, CN=connector # connector name name: cloud-connector ip: # IP address of node, which runs connector subnets: -10.233.0.0/17 # CIDR used by pod & service in cloud cluster - 10.22.45.30/32 # IP address of all cloud cluster - 10.22.45.31/32-10.22.45.32/32Copy the code

Copy the code

⚠ ️ note:

**CIDR: ** Classless inter-domain Routing (CIDR) is a method of classifying IP addresses for assigning IP addresses to users and Routing IP packets efficiently over the Internet.

EdgePodCIDR: Select a large network segment, from which each edge node will be assigned a small segment, and each edge POD will be assigned an IP address from this small segment, which cannot conflict with the network segment of cloud POD or service.

IP: IP address of the connector service. Ensure that the edge node can ping this IP address.

Root @ edge1: ~ # ping 10.22.45.30Copy the code

Copy the code

Subnets: Service clusterIP CIDR and POD clusterIP CIDR are required

For example, if service clusterIP CIDR is 10.233.0.0/18 and podClusterIPCIDR = 10.233.64.0/18 then subnets are 10.233.0.0/17

To obtain service clusterIP CIDR and POD clusterIP CIDR, perform the following steps:

# service clusterIP CIDRroot@master:~# grep -rn "service-cluster-ip-range" /etc/kubernetes/manifests# pod clusterIP CIDRroot@master:~# calicoctl.sh get ipPool
Copy the code

Copy the code

**nodeSubnets: ** The IP addresses of all cloud nodes need to be added

• Create configMap for connector

  root@master:~# kubectl apply -f ~/fabedge/deploy/connector/cm.yaml

Copy the code

• The deployment of the connector

  root@master:~# kubectl apply -f ~/fabedge/deploy/connector/deploy.yaml

Copy the code

• Modify the Calico configuration

Cidr is the previously assigned edgePodCIDR, and disabled is true

root@master:~# vi ~/fabedge/deploy/connector/ippool.yaml
Copy the code

Copy the code

ApiVersion: projectcalico.org/v3kind: IPPoolmetadata: name: fabedgespec: blockSize: 26 cidr: 10.10.0.0/16 natOutgoing: false disabled: trueCopy the code

Copy the code

• Create the calico pool

  Calico commands may vary depending on the environment root@master:# calicoctl.sh create –filename=/root/fabedge/deploy/connector/ippool.yamlroot@master:# calicoctl.sh get IPPool –output yaml ## export DATASTORE_TYPE=kubernetesroot@master:# export KUBECONFIG=/etc/kubernetes/admin.confroot@master:~# calicoctl get ipPoolNAME CIDR SELECTOR default-pool 10.231.64.0/18 all() fabedge 10.10.0.0/16 all()

Copy the code

Configuring edge Nodes

• Modify the edgecore configuration file

  root@edge1:~# vi /etc/kubeedge/config/edgecore.yaml

Copy the code

A) disable edgeMesh

edgeMesh:  enable: false
Copy the code

Copy the code

B) to enable the CNI

CniBinDir: /opt/cni/bin cniCacheDirs: /var/lib/cni/cache cniConfDir: NetworkPluginName: cni networkPluginMTU: 1500Copy the code

Copy the code

C) Configure the domain name and DNS

Edged: clusterDNS: "169.254.25.10" clusterDomain: "root-cluster"Copy the code

Copy the code

You can perform the following operations on the cloud to obtain related information

root@master:~# kubectl get cm nodelocaldns -n kube-system -o jsonpath="{.data.Corefile}"root-cluster:53 {... The bind 169.254.25.10... }root@master:~# grep -rn "cluster-name" /etc/kubernetes/manifests/kube-controller-manager.yaml20: - --cluster-name=root-cluster# In this example, domain is root-cluster and DNS is 169.254.25.10Copy the code

Copy the code

• Install the CNI plug-in

  root@edge1: # mkdir -p cni /opt/cni/bin /etc/cni/net.d /var/lib/cni/cacheroot@edge1: # cd cniroot@edge1: / cni# wget github.com/containerne… Tar XVF cni-plugins-linux-amd64-v0.9.1.tgzroot@edge1: /cni# cp bridge host-local loopback /opt/ cnI /bin

Copy the code

• Restart edgecore

  root@edge1:~# systemctl restart edgecore

Copy the code

• Verify that the edge node is ready

  root@master:~# kubectl get node NAME STATUS ROLES AGE VERSION Edge1 Ready agent, Edge 125m v1.19.3- KubeEdge -v1.1.0 master Ready connector, master, 135 m v1.19.7 node

Copy the code

The deployment of the Operator

• Create a Community CRD

  root@master:~# kubectl apply -f ~/fabedge/deploy/crds

Copy the code

• Modifying a Configuration File

Modify edge-network-CIDR based on the actual environment

root@master:~# vi ~/fabedge/deploy/operator/fabedge-operator.yaml
Copy the code

Copy the code

apiVersion: apps/v1kind: Deploymentmetadata: name: fabedge-operator namespace: fabedge labels: app: fabedge-operatorspec: replicas: 1 selector: matchLabels: app: fabedge-operator template: metadata: labels: app: fabedge-operator spec: containers: - name: operator image: fabedge/operator:latest imagePullPolicy: IfNotPresent args: - Namespace =fabedge - -edge-network-cidr=10.10.0.0/16 # Network used by edge pod - -agent-image=fabedge/agent - -strongswan-image=fabedge/strongswan - -connector-config=connector-config - -endpoint-id-format=C=CN, O=StrongSwan, CN={node} - -v=5 hostNetwork: true serviceAccountName: fabedge-operatorCopy the code

Copy the code

⚠ ️ note:

Edge-network-cidr specifies the edgePodCIDR assigned to modify Connector configuration in Deploy Connector

• Create Operator

  root@master:~# kubectl apply -f ~/fabedge/deploy/operator

Copy the code

Verify that services are started properly

root@master:~# kubectl get po -n fabedgeNAME                               READY   STATUS    RESTARTS   AGEconnector-5947d5f66-hnfbv          2/2     Running   0          35mfabedge-agent-edge1                2/2     Running   0          22sfabedge-operator-dbc94c45c-r7n8g   1/1     Running   0          55s
Copy the code

Copy the code

About FabEdge

FabEdge is an open source network solution based on Kubernetes and KubeEdge. In the edge computing scenario, FabEdge solves such problems as complex container network configuration management, disintegrated network without communication, lack of service discovery, lack of topology awareness, and inability to provide nearby access.

Fabedge supports weak network environments such as 4/5G, WiFi, LoRa, etc. Supports dynamic IP addresses of edge nodes, applicable to scenarios such as the Internet of Things and the Internet of vehicles.

Making:

Github.com/FabEdge/fab…

Official website:

www.fabedge.io

Official Email:

[email protected]

Wechat Group: Open the “Boyun” public account, click the menu bar “Scan code to enter the group”