background

On the edge of the scenario at the edge of the cluster nodes distribution in different areas, and the edges between nodes and the cloud is a one-way network, edge node can access the cloud, the cloud can’t directly access edge nodes, has done a lot of inconvenience to the operations of edge node, if you can from the cloud can simplify the SSH login to edge node of operational work. In response to this requirement, the SuperEdge project extended the capabilities of the tunnel component by adding an SSH module that allows users to SSH to edge nodes from the cloud.

Demand analysis

Nodes in the edge cluster are distributed in different areas. The edge node can access the cloud master node, but the master node cannot access the nodes in the edge area. Users want to use SSH to log in to the master node to perform o&M.

Conventional scheme

SSH port forwarding technology can be used to realize SSH o&M edge node function, as shown in the figure below:

  1. Edge nodes nod-a and Nod-B bind port -a and port -b of the cloud master-A node to local port 22 (the port of the SSH Server) through SSH remote forwarding (Ssh-r)
  2. The user establishes an SSH tunnel with master -a through SSH dynamic forwarding (ssh-d) and listens to the local port
  3. All requests from the local port are transmitted to master-A through SSH connection and forwarded by master-A. For example, to log in to node -a using SSH: SSH -o ProxyCommand= “nc -x 5 -x 127.0.0.1:local-port 127.0.0.1 port -a” [email protected], 127.0.0.1 port -a is the destination address requested by Master -A for forwarding.

Disadvantages of the conventional scheme:

  • As shown in Figure 2, Nod-A and Nod-B map the local port 22 to different ports on master-A. SSH login to the target node requires specifying the port mapped on master-A. When there are many edge nodes, port mapping management is very complicated. It directly leads to the infeasibility of the scheme.
  • As shown in Figure 1, sshClient->local-port, user->master -a, master -a ->port -a, master -a ->node-A, Nod-a ->sshServer: A total of five connections need to be established.
  • The SSH connection between the cloud node and the cloud node needs to be established on the side node, and the connection cannot be disconnected and reconnected. Therefore, maintenance is difficult.

Tunnel scheme

Architecture design

  1. The SSH Client requests the tunnel-cloud service to leak to the nodeport-1 port of the extranet. The service forwards the request to tunnel-cloud pod-a or tunnel-cloud pod-b based on the load balancing policy.
  2. If the request is forwarded to POD-A (R.1.1), since Nod-A does not tunnel with POD-A, POD-A queries coreDNS for podIp of PoD-B tunnling with Nod-A and forwards the request to POD-B (R.1.2).
  3. After receiving the request, POD-b forwards the request to the tunnel-edge of Nod-A through the tunnel, and the tunnel-edge forwards the request to the SSH Server.

Advantages of this scheme

  • Tunnels between cloud and edge nodes do not map ports, avoiding port management.
  • It reduces the number of connections established during the request process and reduces the probability of error
  • The tunnel on the cloud and the side has a disconnection and reconnection mechanism to reduce maintenance costs

The establishment of the Yunbian tunnel

The gRPC open source project is used to build a long connection tunnel, and gRPC realizes the disconnection and reconnection mechanism.

  1. Tunnel-edge sends a request to tunnel-cloud to set up a gRPC connection. Tunnel-edge sends a request to tunnel-cloud’s nodeport-2 to set up a gRPC connection. The request contains the node name and token information of the node. The tunnel-cloud service forwards requests to tunnel-cloud POD based on the load balancing policy, as shown in Figure 3, and forwards requests to tunnel-cloud POD-b
  2. Tunnel-cloud registers the node name information of the node where the podIp and tunnel-edge of the pod reside with the CoreDNS. Tunnel-cloud authenticates the token information in the request information of the tunnel-edge. After the authentication is successful, The node name in the node request information and the podIp of this POD are written to coreDNS
  3. Tunnel-cloud Returns a message indicating that the gRPC connection is established successfully
  4. A custom protocol message is sent between tunnel-edge and tunnel-cloud over a long gRPC connection. StreamMsg For the field definition of a custom protocol message, please refer to this article to read SuperEdge cloud side tunnel

Implementation of tunnel-cloud forwarding

  1. The SSH Client requests the nodeport-1 port of the tunnel-cloud service. An SSH Client sends an HTTP request with the method “Connect” through netcat or Crokscrew. ProxyCommand= “nc-x connect-x tunnel-cloudip: nodeport-1 nod-a 22”);

    • -x: indicates the protocol type. Connect is HTTP Connect
    • -x: indicates the IP address and port of the HTTP Server. The specified tunnel-cloudip: nodeport-1 indicates the IP address and port that the tunnel-cloud service leaks to the Internet
    • Node-a indicates the node name of the side node
    • 22 is the listening port of the SSH Server on the side node

    The tunnel-cloud service forwards the SSH Client’s request to tunnel-cloud POD according to the load balancing policy, as shown in Figure 3. If the request is forwarded to tunnel-cloud PoD-a, The HTTP Server of tunnel-cloud receives the CONNECT nodes-a :22 HTTP/1.0 message, where nodes-a is the node name of the side node, 22 is the port monitored by the SSH Server on the side node. Since Nod-a does not establish A cloud-side tunnel with tunnel-cloud pod-a, the HTTP Server will ask coreDNS to obtain the podIp of tunnel-cloud corresponding to nod-a’s node name. That is, tunnel-cloud pod-b. Pod-a forwards the request to PoD-B

  2. Tunnel-cloud sends a custom protocol message (streammsg. Type: connecting) to the tunnel-edge. The tunnel-cloud obtains the tunnel information from the cloud and the side nodes according to the HTTP CONNECT request. In addition, the SVN sends a user-defined protocol message to the tunnel-edge through the cloud-side tunnel to set up a TCP connection with the SSH Server. The message type is connecting

  3. Tunnel-edge sends a TCP connection message to the SSH Server. After receiving a user-defined connecting protocol message, the tunnel-edge sends a TCP connection request based on the IP address and port of the SSH Server in the message

  4. The SSH Server returns a message to tunnel-edge indicating that the TCP connection is established successfully

  5. Tunnel-edge returns a user-defined protocol message (streammsg. Type is conneted) to tunnel-cloud. After receiving the message that the connection is established, tunnel-edge sends a TCP connection Type to tunnel-cloud Connected’s custom protocol message

  6. Tunnel-cloud Returns a reponse message with the status of the SSH Client being 200. After receiving the connected user-defined protocol message, the SSH Client returns a message with the status code being 200. HTTP / 1.1 200 Connection established

  7. SSH Client Sends SSH protocol messages to tunnel-cloud After receiving the response message with status code 200, the SSH Client sends SSH protocol data through the tunnel established with tunnel-cloud

  8. Tunnel-cloud encapsulates SSH messages as user-defined protocol messages (streammsg. Type is content) and sends them to Tunel-Edge

  9. Tunnel-edge sends the SSH protocol message to the SSH Server over the TCP connection

  10. The SSH Server sends the SSH protocol message to tunnel-edge

  11. Tunnel-edge encapsulates SSH protocol messages as content messages and sends them to tunnel-cloud

  12. Tunnel-cloud returns the SSH protocol message to the SSH Client

Logging In to a Node using SSH

Link to video of functional demonstration

precondition

Install Corkscrew, Mac directly using BREW installation

brew install corkscrew
Copy the code

Install netcat(centos)

yum install -y netcat
Copy the code

Logging In to a Node using SSH

To log in to edge node nod-a-1 over SSH, run the following command:

Ssh-o ProxyCommand= "Corkscrew masterIP cloud-port nod-a-1 22" [email protected]Copy the code

or

SSH -o ProxyCommand= "nc -x connect -x masterIP:cloud-port node a-1 22" [email protected]Copy the code
  • MaterIP: External IP address of the node where the master node resides
  • Cloud-port: specifies the NodePort. The cloud-port can be obtained from the Server port of the corresponding SSH module
kubectl -n edge-system get svc tunnel-cloud -o=jsonpath='{range .spec.ports[*]}{.name}{"\t"}{.nodePort}{"\n"}{end}' | grep ssh | awk '{print $2}'
Copy the code

conclusion

In this solution, users can quickly log in to the side node over SSH to implement O&M without manual configuration. In addition, the tunnel provided by this solution is easier to maintain and more stable than the traditional solution, and the user experience is improved.

future

Support for unified SSH operation and maintenance of edge nodes from the cloud is an enhanced function of tunnel components. It is also the realization of the outlook of SuperEdge cloud side tunnel. After SuperEdge open source, community partners also put new demands on tunnel components, which are roughly as follows:

  • The cloud Apiserver can access the Webhook server on the edge
  • Cross-region access between services is supported

Collaboration and open source

The new feature of SSH o&M edge node of cloud side Tunnel has been open source in SuperEdge Release 0.4.0, welcome to experience. We will also continue to improve the capabilities of the Tunnel for more complex edge network scenarios, and we welcome companies, organizations and individuals interested in edge computing to work together on the SuperEdge Edge Container project.