Introduction: With the rapid development of 5G, IoT and other technologies, edge computing is more and more widely used in telecommunications, media, transportation, logistics, agriculture, retail and other industries and scenarios, becoming a key way to solve the data transmission efficiency in these fields. At the same time, with the increasing morphology, scale and complexity of edge computing, the operation and maintenance methods and operation and maintenance capabilities in the field of edge computing are increasingly weak in supporting the innovation speed of edge business. As a result, Kubernetes quickly became a key element in edge computing, helping enterprises to better run containers on the edge, maximize resources and shorten development cycles.

Author | He Linbo (Xinsheng)

background

With the rapid development of 5G, IoT and other technologies, edge computing has been more and more widely used in telecommunications, media, transportation, logistics, agriculture, retail and other industries and scenarios, becoming a key way to solve the data transmission efficiency in these fields. At the same time, with the increasing morphology, scale and complexity of edge computing, the operation and maintenance methods and operation and maintenance capabilities in the field of edge computing are increasingly weak in supporting the innovation speed of edge business. As a result, Kubernetes quickly became a key element in edge computing, helping enterprises to better run containers on the edge, maximize resources and shorten development cycles.

However, if the native Kubernetes is directly applied to the edge computing scenario, many problems still need to be solved, such as cloud and edge nodes are generally located in different network planes, and edge nodes are generally located inside the firewall. The adoption of cloud (center) edge collaborative architecture will lead to the following challenges for the operation and maintenance monitoring capability of the native K8s system:

  • K8s native operation and maintenance capabilities missing (such as kubectl logs/exec cannot be executed)
  • Community mainstream monitoring o&M components not working (e.g. Prometheus/ Metrics-Server)

In order to help enterprises solve the challenges of native Kubernetes in the edge scenarios, such as application life cycle management, cloud side network connection, cloud side side operation and maintenance coordination, heterogeneous resource support and other aspects, the edge computing cloud native open source platform OpenYurt based on K8s came into being. It is also an important part of CNCF in the original map of edge cloud. This article will detail how Yurt-Tunnel, one of the core components of OpenYurt, extends the capabilities of the native K8s system in edge scenarios.

Yurt Tunnel design roadmap

Since the edge can access the cloud, it is important to consider building a reverse-penetrating tunnel at the cloud side to ensure that the cloud (center) can proactively access the edge based on the tunnel. At that time, we also investigated many open source Tunnel schemes. In terms of capability and ecological compatibility, we finally designed and implemented the overall solution of YURt-Tunnel based on ANP, which has the advantages of security, non-intrusion, scalability and high transmission efficiency.

implementation

To build a secure, non-intrusive and scalable reverse channel solution in K8s cloud side integration architecture, the solution should include at least the following capabilities.

  • Cloud side tunnel construction
  • Self-management of certificates at both ends of the tunnel
  • Cloud component requests are seamlessly streamed backwards into the tunnel

The architecture module of yurt-Tunnel is shown as follows:

3.1 Cloud side tunnel construction

  • When the edge Yurt-tunnel-Agent is started, it establishes and registers a connection with the Yurt-tunnel-server based on the access address, periodically checks the health status of the connection, and reestablishes the connection.

    Github.com/openyurtio/…

    Registration information of yurt-tunnel-agent:

    “agentID”: {nodeName} “agentIdentifiers”: ipv4={nodeIP}&host={nodeName}”

  • When the Yurt-tunnel-server receives a request from the cloud component, it forwards the request to the corresponding Yurt-Tunnel-Agent. This is because in addition to forwarding the initial request, the request session is followed by either a data return or a continuous forwarding of the data (such as Kubectl exec). Therefore, two-way data forwarding is required. The need to support concurrent forwarding of requests from cloud components means that a separate identity needs to be established for each request lifecycle. So there are two kinds of design options.

Scheme 1: The initial cloud-side connection only notifies the forwarding request, and the tunnel-agent establishes a new connection with the cloud to process the request. The problem of separate identification of requests can be solved by new connections, and concurrency can also be solved. But establishing a connection for each request consumes a lot of resources.

Scheme 2: Only use the initial cloud side connection to forward requests. In order to reuse the same connection for a large number of requests, each request needs to be encapsulated and an independent identifier added to solve the demands of concurrent forwarding. At the same time, as a connection needs to be reused, connection management and request lifecycle management need to be decoupled, that is, the state migration of request forwarding needs to be independently managed. The scheme involves packet unpacking, request processing state machine, etc., which is more complicated.

  • The ANP component selected by OpenYurt adopts scheme 2 above, which is consistent with our original design.

    Github.com/openyurtio/…

    Data formats and data types for cloud side communication

    type Packet struct { Type PacketType protobuf:"varint,1,opt,name=type,proto3,enum=PacketType" json:"type,omitempty" // Types that are valid to be assigned to Payload: // *Packet_DialRequest // *Packet_DialResponse // *Packet_Data // *Packet_CloseRequest // *Packet_CloseResponse Payload isPacket_Payload protobuf_oneof:"payload" }

  • Request forwarding link construction is encapsulated in Packet_DialRequest and Packet_DialResponse, where Packet_DialResponse.ConnectID is used to identify request. Equivalent to requestID in a tunnel. The request and associated data are encapsulated in Packet_Data. Packet_CloseRequest and Packet_CloseResponse are used to reclaim forwarding link resources. For details, please refer to the following sequence diagram:

  • The function of the RequestInterceptor module

According to the above analysis, before yurt-tunnel-server forwards a request, the requesting end initiates an Http Connect request to construct a forwarding link. However, it is difficult to add corresponding processing for open source components such as Prometheus and Metrics-Server. Therefore, the request hijacking module Interceptor is added to Yurt-Tunnel-Server to initiate Http Connect requests. The relevant codes are as follows:

# https://github.com/openyurtio/openyurt/blob/master/pkg/yurttunnel/server/interceptor.go#L58-82 proxyConn, err := net.Dial("unix", udsSockFile) if err ! = nil { return nil, fmt.Errorf("dialing proxy %q failed: %v", udsSockFile, err) } var connectHeaders string for _, h := range supportedHeaders { if v := header.Get(h); len(v) ! = 0 { connectHeaders = fmt.Sprintf("%s\r\n%s: %s", connectHeaders, h, v)}} FMT.Fprintf(proxyConn, "CONNECT %s HTTP/1.1\r\nHost: %s%s\r\n\r\n", addr, "127.0.0.1", connectHeaders) br := bufio.newreader (proxyConn) res, err := http.readResponse (br, nil) if err ! = nil { proxyConn.Close() return nil, fmt.Errorf("reading HTTP response from CONNECT to %s via proxy %s failed: %v", addr, udsSockFile, err) } if res.StatusCode ! = 200 { proxyConn.Close() return nil, fmt.Errorf("proxy error from %s while dialing %s, code %d: %v", udsSockFile, addr, res.StatusCode, res.Status) }Copy the code

3.2 Certificate Management

To ensure long-term secure communication over the cloud side channel and support HTTPS request forwarding, the YURt-Tunnel needs to generate its own certificate and maintain automatic certificate rotation. The concrete implementation is as follows:

Yurt-tunnel-server certificate: # https://github.com/openyurtio/openyurt/blob/master/pkg/yurttunnel/pki/certmanager/certmanager.go#L45-90 - certificate store location: /var/lib/yurt-tunnel-server/pki - CommonName: Kube-apiserver-kubelet-client: kube-apiserver-kubelet-client: {"system:masters", "Openyurt :yurttunnel"} // Auto approve - Subject Alternate Name values for Kubelet server webhook check and Yurt-tunnel-server certificate: {x-tunnel-server-svc, ips and DNS names of x-tunnel-server-internal-svC} -keyUsage: {x-tunnel-server-svC, IPS and DNS names of x-tunnel-server-internal-SVC} -keyUsage: "Any" # 2. Yurt-tunnel-agent # https://github.com/openyurtio/openyurt/blob/master/pkg/yurttunnel/pki/certmanager/certmanager.go#L94-112 - certificate store location: /var/lib/yurt-tunnel-agent/pki - CommonName: "yurttunnel-agent" - Organization: {" openYurt :yurttunnel"} // For yurt-tunnel-agent certificate auto approve - Subject Alternate Name values: {nodeName, nodeIP} - KeyUsage: "Any" # 3. All CSR requests are approved by yurt-tunnel-server https://github.com/openyurtio/openyurt/blob/master/pkg/yurttunnel/pki/certmanager/csrapprover.go#L115 - - monitor CSR resources Filter CSR # 4 that is not a Yurt tunnel (there is no "OpenYurt :yurttunnel" in the Organization) - Approve unapproved CSR # 4. Automatic certificate rotation processing # https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/client-go/util/certificate/certificate_manager.g o#L224Copy the code

3.3 Seamless Diversion The cloud component requests to the tunnel

Since requests from the cloud component need to be seamlessly forwarded to the Yurt-Tunnel-server, there is no need to modify the cloud component. Therefore, requests of cloud components need to be analyzed. Currently, o&M requests of components mainly fall into the following two types:

  • Type 1: Use the IP address to access the IP address, for example, http://{nodeIP}:{port}/{path}
  • Type 2: Access using domain names, for example, http://{nodeName}:{port}/{path}

Different schemes are required for different types of diversion requests.

  • Scheme 1: Use iptables DNAT rules to ensure that requests of type 1 are seamlessly forwarded to yurt-tunnel-server

    Iptables rulesGithub.com/openyurtio/…

    The iptables DNAT rules maintained by yurt-tunnel-server are as follows:

    [root@xxx /]# iptables -nv -t nat -L OUTPUT TUNNEL-PORT tcp — * * 0.0.0.0/0 0.0.0.0/0 /* edge tunnel server port */

    [root@xxx /]# iptables -nv -t nat-l tunnel-port tunnel-port-10255 TCP — * * 0.0.0.0/0 0.0.0.0/0 TCP DPT :10255 /* jump To port 10255 / tunnel-port-10250 TCP — * * 0.0.0.0/0 0.0.0.0/0 TCP DPT :10250 / jump to port 10250 */

    [root@xxx /]# iptables -nv -t nat-l tunnel-port-10255 RETURN TCP — * * 0.0.0.0/0 127.0.0.1 /* RETURN request to access Node directly/TCP DPT :10255 RETURN TCP — * * 0.0.0.0/0 172.16.6.156 / RETURN request to access node directly/TCP DPT :10255 DNAT TCP — * * 0.0.0.0/0 0.0.0.0/0 / DNAT to tunnel for access node */ TCP DPT :10255 to:172.16.6.156:10264

  • Scheme 2: Use the DNS domain name to resolve nodeName as the access address of Yurt-tunnel-server, so that type 2 requests are seamlessly forwarded to Yurt-tunnel

    Different uses of x-tunnel-server-SVC and X-tunnel-server-internal-svC are as follows:

    • X-tunnel-server-svc: indicates the expose 10262/10263 port, which is used to access the YURt-tunnel-server from the public network. Such as the yurt – tunnel – agent
    • X-tunnel-server-internal-svc: used to access cloud components, such as Prometheus and metrics-server, from the internal network

    Principles of DNS domain name resolution:

    1. Yurt-tunnel-server Creates or updates the Yurt-tunnel-Nodes configmap to kube-Apiserver. The format of the tunnel-Nodes field is as follows: {x-tunnel-server-internal-svC clusterIP} {nodeName}, ensure that the mapping between all nodeName services and Yurt-tunnel-server services is recorded
    2. Yurt-tunnel-nodes Configmap is mounted in coreDNS Pod, and DNS records of Configmap are used using the host plugin
    3. At the same time, configure port mapping in x-tunnel-server-internal-svC to map 10250 to 10263 and 10255 to 10264
    4. Through the above configuration, http://{nodeName}:{port}/{path} requests can be seamlessly forwarded to Yurt-tunnel-Servers

  • Cloud request extension:

If the user needs to access other ports on the edge (other than 10250 and 10255), add the corresponding DNAT rules in iptables or add the corresponding port mapping in X-tunnel-server-internal-svC, as shown below:

Iptables dNAT rule [root@xxx /]# iptables -nv -t nat-l tunnel-port tunnel-port-9051 TCP -- * * 0.0.0.0/0 0.0.0.0/0 TCP DPT :9051 /* jump to Port 9051 */ [root@xxx /]# iptables -nv -t nat-l tunnel-port-9051 RETURN TCP -- * * 0.0.0.0/0 127.0.0.1 /* RETURN Request to access node Directly */ TCP DPT :9051 RETURN TCP -- * * 0.0.0.0/0 172.16.6.156 /* RETURN request to access Node directly */ TCP DPT :9051 DNAT TCP -- * * 0.0.0.0/0 0.0.0.0/0 /* DNAT to tunnel for access node */ TCP DPT :9051 To :172.16.6.156:10264 # X-tunnel-server-internal-svC Add port mapping spec: ports: - name: HTTPS port: 10250 protocol: TCP targetPort: 10263-name: HTTP port: 10255 protocol: TCP targetPort: 10264-name: dNAT-9051 9051 protocol: TCP targetPort: 10264Copy the code

Of course, iptables DNAT rules and service port mappings are automatically updated by yurt-tunnel-server. You only need to add port configurations in yurt-tunnel-server-cfg configmap. Details are as follows:

ApiVersion: v1 data: dnat-ports-pair: Kind: ConfigMap metadata: name: yurt-tunnel-server-cfg Namespace: kube-systemCopy the code

The recent planning

  • Support kube-Apiserver’s EgressSelector function
  • Verify the multi-instance deployment of Yurt-Tunnel-Server
  • Yurt-tunnel-agent Multiple Yurt-tunnel-server addresses are supported
  • You can customize the certificate storage directory
  • More refined definition of certificate Usage is supported to ensure that the certificate Usage range is controllable
  • Support for Yurt-tunnel-server When the access address of the Yurt-tunnel-server changes, the Certificate of the Yurt-tunnel-server is automatically updated
  • Supports automatic refreshing of the access address of yurt-tunnel-agent for yurt-tunnel-server
  • Support for non-nodeIP /NodeName type request forwarding (such as non-host network Pod cloud access edge)
  • Supports access to cloud Pod through edge Pod through Tunnel
  • Support independent deployment of YURt-Tunnel (not bound to K8S)
  • Supports more protocols, such as gRPC, WebSocket, and SSH

Welcome to the OpenYurt community

As the core of Alibaba Cloud edge container service ACK@Edge, OpenYurt has been commercialized in dozens of industries such as CDN, audio and video live broadcasting, Internet of Things, logistics, industrial brain, urban brain and so on, with the service scale reaching millions of CPU cores. We are pleased to see that more and more developers, the open source community, enterprises, and academic organizations are now embracing the idea of OpenYurt and joining the team to build OpenYurt. For example, VMware, Intel, Shenxin Services, China Merchants, Zhejiang University, EdgeX Foundry community, eKuiper community, etc. We also welcome more friends to build the OpenYurt community to flourish the cloud native edge computing ecology and create value in more edge scenarios in the true sense of cloud native.

The original link

This article is the original content of Aliyun and shall not be reproduced without permission.