Brief introduction:With the rapid development of 5G, IoT and other technologies, edge computing has been more and more widely used in telecommunications, media, transportation, logistics, agriculture, retail and other industries and scenarios, and has become a key way to solve the data transmission efficiency in these fields. At the same time, the form, scale and complexity of edge computing are increasing day by day, and the support of operation and maintenance means and operation and maintenance capabilities in the field of edge computing to the innovation speed of edge business is becoming increasingly weak. As a result, Kubernetes has quickly become a key element of edge computing, helping companies better run their containers at the edge, maximizing resources and shorting development cycles.

Author | He Linbo (Xinsheng)

background

​

With the rapid development of 5G, IoT and other technologies, edge computing has been more and more widely used in telecommunications, media, transportation, logistics, agriculture, retail and other industries and scenarios, and has become a key way to solve the data transmission efficiency in these fields. At the same time, the form, scale and complexity of edge computing are increasing day by day, and the support of operation and maintenance means and operation and maintenance capabilities in the field of edge computing to the innovation speed of edge business is becoming increasingly weak. As a result, Kubernetes has quickly become a key element of edge computing, helping companies better run their containers at the edge, maximizing resources and shorting development cycles.

However, if native Kubernetes is directly applied to the edge computing scene, many problems still need to be solved. For example, cloud and edge are generally located in different network planes, and edge nodes are generally located inside the firewall. The adoption of cloud (center) edge collaborative architecture will cause the operation and maintenance monitoring ability of native K8S system to face the following challenges:

  • K8S native operation and maintenance capability is missing (such as Kubectl logs/exec, etc., cannot be executed)
  • Mainstream community monitoring operation component does not work (e.g. Prometheus/ Metrics – Server)

​

In order to help enterprises solve the challenges of native Kubernetes on application life cycle management, cloud side network connection, cloud side operation and maintenance collaboration, heterogeneous resource support and other aspects under the edge scenario, OpenYurt, the edge computing cloud native open source platform based on K8S, came into being. It is also an important part of CNCF in the original map of the edge cloud. This article describes in detail how the Yurt-Tunnel, one of the core components of OpenYurt, extends the native K8S system’s relevance in edge scenarios.

Yurt-Tunnel design idea

​

Since the edge can access the cloud, consider building a tunnel back through the edge of the cloud, so that the cloud (the center) can actively access the edge based on the tunnel. At that time, we also investigated many open source Tunnel solutions. From the aspects of capability and ecological compatibility, we finally chose to design and implement the overall solution of Yurt-Tunnel based on ANP, which has the advantages of safety, non-intrusion, extensibility and efficient transmission.

implementation

To build a secure, non-intrusive, scalable backchannel solution in the K8S cloud edge integration architecture, the solution needs to include at least the following capabilities.

  • Cloud side tunnel construction
  • Self-management of certificates at both ends of the tunnel
  • Cloud component requests are seamlessly backed back into the tunnel

​

The Yurt-Tunnel architecture module is shown below:

3.1 Cloud side tunnel construction

​

  • When the edge Yurt-tunnel-agent is started, the connection with Yurt-tunnel-server is established and registered based on the access address, and the health status of the connection is periodically checked and the connection is rebuilt.

​

# # https://github.com/openyurtio/apiserver-network-proxy/blob/master/pkg/agent/client.go#L189 yurt - tunnel, agent registration information: "agentID": {nodeName} "agentIdentifiers": ipv4={nodeIP}&host={nodeName}"
  • When the Yurt-tunnel-server receives a request from the cloud component, it needs to forward the request to the corresponding Yurt-tunnel-agent. This is because in addition to forwarding the initial request, the session of the request also has subsequent data return or continuous forwarding of data (such as Kubectl exec). So you need to forward the data in both directions. The need to support concurrent forwarding of requests from cloud components also means that a separate identity needs to be established for each request lifecycle. So there are two ways to design it.

​

Option 1: The initial cloud side connection only notifies the forwarding request, and the tunnel-agent establishes a new connection with the cloud to handle the request. The problem of independent identification of requests is solved well with new connections, and concurrency is also solved well. However, the need to establish a connection for each request consumes a lot of resources.

Scheme 2: only the initial cloud edge connection is used to forward the request. In order to reuse the same connection for a large number of requests, it is necessary to encapsulate each request and add independent identification to solve the appeal of concurrent forwarding. At the same time, due to the need to reuse a connection, it is necessary to decouple connection management and request life cycle management, that is, the state migration of request forwarding needs to be managed independently. This scheme involves packet unpacking, request processing state machine, etc., the scheme will be more complex.

  • The ANP component that OpenYurt has chosen is scheme 2 above, which is consistent with our original design.

​

# https://github.com/openyurtio/apiserver-network-proxy/blob/master/konnectivity-client/proto/client/client.pb.go#L98 # Type Packet struct {type PacketType 'protobuf:"varint,1,opt,name=type,proto3,enum=PacketType" json:"type,omitempty"` // Types that are valid to be assigned to Payload: // *Packet_DialRequest // *Packet_DialResponse // *Packet_Data // *Packet_CloseRequest // *Packet_CloseResponse Payload isPacket_Payload `protobuf_oneof:"payload"` }

​

  • Packet\ _dialResponse and Packet\ _dialResponse where Packet\ _dialResponse is used to identify the request and Packet\ _dialResponse. It’s the equivalent of RequestId in the tunnel. The request and associated data are encapsulated in Packet\_Data. PACKET \ _CLOSEReRequest and PACKET \ _CLOSEReResponse are used for forwarding link resource collection. For details, please refer to the following sequence diagram:



​

  • What the RequestInterceptor module does

​

From the above analysis, it can be seen that before Yurt – Tunnel – Server forwards the request, the requester needs to make an HTTP Connect request to construct the forwarding link. However, it is difficult to add corresponding processing to open source components such as Prometheus and Metrics – Server. Therefore, a request hijacker module, Interceptor, is added into Yurt- Tunnel – Server to make HTTP Connect requests. The relevant code is as follows:

# https://github.com/openyurtio/openyurt/blob/master/pkg/yurttunnel/server/interceptor.go#L58-82 proxyConn, err := net.Dial("unix", udsSockFile) if err ! = nil { return nil, fmt.Errorf("dialing proxy %q failed: %v", udsSockFile, err) } var connectHeaders string for _, h := range supportedHeaders { if v := header.Get(h); len(v) ! = 0 { connectHeaders = fmt.Sprintf("%s\r\n%s: %s", connectHeaders, h, v)} fmt.fprintf (proxyConn, "CONNECT %s HTTP/1.1\r\nHost: %s%s\r\n\r\n", addr, "127.0.0.1", connectHeaders) br := bufio.NewReader(ProxyConn) res, err := http. readResponse (br, nil) if err ! = nil { proxyConn.Close() return nil, fmt.Errorf("reading HTTP response from CONNECT to %s via proxy %s failed: %v", addr, udsSockFile, err) } if res.StatusCode ! = 200 { proxyConn.Close() return nil, fmt.Errorf("proxy error from %s while dialing %s, code %d: %v", udsSockFile, addr, res.StatusCode, res.Status) }

​

3.2 Certificate Management

In order to ensure long-term secure communication between the cloud side channel and support HTTPS request forwarding, the Yurt-Tunnel needs to generate its own certificates and maintain automatic rotation of certificates. The specific implementation is as follows:

# 1. Yurt-tunnel-server: # https://github.com/openyurtio/openyurt/blob/master/pkg/yurttunnel/pki/certmanager/certmanager.go#L45-90 - certificate store location: /var/lib/yurt-tunnel-server/pki - CommonName: "Kube-apiserver-kubelet-client" // Webhook verifier-organization for kubelet server: {"system:masters", } // Auto-Approve -Subject Alternate Name values for Webhook validation and Yurt-tunnel-server certificates for Kubelet Server: {x-tunnel-server-svc, x-tunnel-server-internal-svc IPS and DNS Names} -keyUsage: "Any" # 2. Yurt - Tunnel - Agent certificate: # https://github.com/openyurtio/openyurt/blob/master/pkg/yurttunnel/pki/certmanager/certmanager.go#L94-112 - certificate store location: /var/lib/yurt-tunnel-agent/pki - CommonName: "yurttunnel-agent" - Organization: {" openYurt :yurttunnel"} // Auto-Approve -Subject Alternate Name values for the yurt-tunnel-agent certificate: {nodeName, nodeIP} - KeyUsage: "Any" # 3. Yurt-Tunnel Certificate Application (CSR) is approved by yurt-tunnel server https://github.com/openyurtio/openyurt/blob/master/pkg/yurttunnel/pki/certmanager/csrapprover.go#L115 - - monitor CSR resources Filter CSR that is not yurt-tunnel (no "openyurt:yurttunnel" in the Organization) -approve CSR # 4 that has not been Approved. Automatic certificate rotation processing # https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/client-go/util/certificate/certificate_manager.g o#L224

3.3 Seamless diversion cloud component requests to the tunnel

​

The need to seamlessly forward requests from the cloud component to the yurt-tunnel-server also means that no changes to the cloud component are required. Therefore, it is necessary to analyze the requests of cloud components. Currently, there are mainly two types of component operation and maintenance requests:

  • Type 1: Access directly with an IP address, such as http://{nodeIP}:{port}/{path}
  • Type 2: Access using a domain name, such as http://{nodeName}:{port}/{path}

​

Different schemes are needed for different types of diversion requests.

  • Scenario 1: Use iptables dnat rules to ensure that requests of type 1 are seamlessly forwarded to Yurt-tunnle-server
Iptables rules: iptables rules: https://github.com/openyurtio/openyurt/blob/master/pkg/yurttunnel/iptables/iptables.go # yurt - tunnel - the iptables server maintenance DNAT rules are as follows: [root@xxx /]# iptables -nv-t nat-l Output Tunnel Server TCP -- * * 0.0.0.0/0 0.0.0.0/0 /* Edge Tunnel Server Port */ [root@xxx /]# iptables -nv-t nat-l tunnel-port tunnel-port 10255 TCP -- * * 0.0.0.0/0 0.0.0.0/0 TCP DPT :10255 /* jump To port 10255 */ tunnel-port-10250 TCP -- * * 0.0.0.0/0 0.0.0.0/0 TCP DPT :10250 /* jump to port 10250 */ [root@xxx /]# Iptables -nv-t nat-l tunnel-port -10255 RETURN TCP -- * * 0.0.0.0/0 127.0.0.1 /* RETURN request to access node directly */ TCP DPT :10255 RETURN TCP -- * * 0.0.0.0/0 172.16.6.156 /* RETURN request to access node directly */ TCP DPT :10255 DNAT TCP -- * * 0.0.0.0/0 0.0.0.0/0 /* DNAT to Tunnel for Access Node */ TCP DPT :10255 to:172.16.6.156:10264

​

  • Scenario 2: Use the DNS domain name to resolve the access address of the nodeName to the Yurt-Tunnel so that the Type 2 request is forwarded seamlessly to the Yurt-Tunnel
# x - tunnel - server - SVC and x - tunnel - server - internal - SVC different purposes: - x - tunnel - server - SVC: The main Expose port is 10262/10263, which is used to access the Yurt-Tunnel-Server from the public network. For example, Yurt - Tunnel - Agent - X - Tunnel - Server - Internal - SVC: mainly used for cloud components to access from internal network, such as Prometheus, Metrics - Server, etc. 1. Yurt-tunnel-server creates or updates the Yurt-tunnel-nodes configMap to Kube-Apiserver, where the format of the tunnel-nodes field is: {x - tunnel - server - internal - SVC clusterIP} {nodeName}, 2. Mount the yurt tunnel-nodes configmap in coredns pod, ConfigMap DNS Records 3. Also configure port mapping in X-Tunnel-Server-Internal SVC, 10250 to 10263,10255 to 10264 4. With the above configuration, it is possible to seamlessly forward http://{nodeName}:{port}/{path} requests to Yurt - Tunnel-Servers

​

  • Cloud request extension:

​

If the user needs to access other ports on the edge (other than 10250 and 10255), add the corresponding DNAT rules to iptables or the corresponding port mapping to x-tunnel-server-internal-svc, as shown below:

Iptables dnat rule: iptables NAT rule: iptables NAT rule: iptables NAT rule: iptables NAT rule: iptables NAT rule: iptables NAT rule [root@xxx /]# iptables -nv-t nat-l tunnel-port tunnel-port 9051 TCP -- * * 0.0.0.0/0 0.0.0.0/0 TCP DPT :9051 /* jump to Port 9051 */ [root@xxx /]# iptables -nv-t nat-l tunnel-port -9051 RETURN TCP -- * * 0.0.0.0/0 127.0.0.1 /* RETURN Request to access node directly */ TCP DPT :9051 RETURN TCP -- * * 0.0.0.0/0 172.16.6.156 /* RETURN request to access Node Directly */ TCP DPT :9051 DNAT TCP -- * * 0.0.0.0/0 0.0.0.0/0 /* DNAT to tunnel for access node */ TCP DPT :9051 Spec: ports: -name: HTTPS port: 10250 protocol: to:172.16.6.156:10264 # x-tunnel-server-internal-svc spec: ports: -name: HTTPS port: 10250 protocol: to:172.16.6.156:10264 # x-tunnel-server-internal-svc spec: ports: -name: HTTPS port: 10250 protocol: TCP targetPort: 10263-name: HTTP port: 10255 Protocol: TCP targetPort: 10264-name: dnat-9051 # 9051 protocol: TCP targetPort: 10264

Of course, the iptables dnat rules and service port mappings mentioned above are automatically updated by Yurt – Tunnel – Server. The user only needs to add the port configuration to the yurt-tunnel-server-cfg configmap. The details are as follows:

# Note: Due to uncontrolled certificate, the new port only supports forwarding apiVersion: v1 data: dnat-ports-pair from Yurt - Tunnel - Server 10264: Type: ConfigMap metadata: name: Yurt-tunnel-server-cfg namespace: kube-system

The recent planning

​

  • Support for the EgressSelector function of Kube-APIServer
  • Verify Yurt – Tunnel-Server multi-instance deployment validation
  • Support yurt-tunnel-agent to configure multiple yurt-tunnel-server addresses
  • Support certificate storage directory customization
  • Support for more refined definition of the Usage of certificates, ensuring that the scope of use of certificates is controllable
  • Support for yurt-tunnel-server certificates to be automatically updated when the access address of yurt-tunnel-server is changed
  • Support for yurt-tunnel-agent to automatically refresh yurt-tunnel-server access addresses
  • Support for non-NodeIP/Nodename type request forwarding (such as cloud access edge for non-host network POD)
  • Support access to cloud Pod from edge Pod through Tunnel
  • Support for Yurt-Tunnel stand-alone deployment (unbound K8s)
  • Support more protocol forwarding, such as GRPC, WebSocket, SSH, etc

​

Welcome to the OpenYurt community

​

As the core of AliCloud’s edge container service ACK@Edge, OpenYurt has been commercialized in dozens of industries such as CDN, audio and video broadcast, Internet of Things, logistics, industrial brain, urban brain and so on, serving millions of CPU cores. We are excited to see that more and more developers, the open source community, enterprises and credit institutions are embracing the OpenYurt concept and are joining the effort to build OpenYurt. For example, VMware, Intel, Xin Xin Service, China MerchantBureau, Zhejiang University, Edgex Foundry Community, Ekuiper Community, etc. We also welcome more friends to build the OpenYurt community, to prosper the cloud native edge computing ecology, and let the true cloud native create value in more edge scenarios.

Welcome to the OpenYurt community pinning group:

Go to https://openyurt.io/en-us/ to OpenYurt’s website!

Copyright Notice:The content of this article is contributed by Aliyun real-name registered users, and the copyright belongs to the original author. Aliyun developer community does not own the copyright and does not bear the corresponding legal liability. For specific rules, please refer to User Service Agreement of Alibaba Cloud Developer Community and Guidance on Intellectual Property Protection of Alibaba Cloud Developer Community. If you find any suspected plagiarism in the community, fill in the infringement complaint form to report, once verified, the community will immediately delete the suspected infringing content.