The introduction
In SuperEdge version 0.2.0, Lite-Apiserver has received significant architectural upgrades and feature enhancements. This paper will analyze the edge autonomy capability of SuperEdge from the perspective of lite-Apiserver implementation and its collaboration with other SuperEdge components, so as to provide reference for research and selection.
Autonomy of edge nodes
In the edge computing scenario of cloud-edge collaboration, edge nodes connect to the cloud through the public network. The edge nodes are numerous, the network environment is complex, and the network quality is uneven. In the case of weak network or disconnection from the cloud, the edge node should continue to work normally and the running services should not be affected, so as to achieve the purpose of autonomy of the edge node. To achieve autonomy of edge nodes, the following scenarios need to be handled:
- The edge node is disconnected from the cloud, but it is normal. The business container running on it should not be expelled, and no new business container is scheduled to the node
- When the edge node is disconnected from the cloud, the Kubernetes components and business containers on the edge node can continue to run
- When the edge node is disconnected from the cloud, Kubernetes components and business containers on the edge node can run after the edge node is restarted
- After the edge node is restored, the data on the edge node is consistent with that on the cloud center
SuperEdge uses edge-Health, a distributed node health check component, for scenario 1 and Lite-Apiserver for scenario 2, 3, and 4.
Lite-apiserver is a lightweight Apiserver that runs on edge nodes. It proxies all components and business containers on the nodes to access requests from cloud Kube-Apiserver and performs efficient caching of the request results. In the case of cloud edge disconnection, these caches are utilized to provide services and achieve edge autonomy.
Lite-apache Server design features
Lite-apiserver not only meets the requirements of autonomy of edge nodes, but also meets the following design features:
All Client types are supported
As the only “exit” from edge nodes to cloud Kube-Apiserver, Lite-Apiserver needs to support all types of clients, Includes Kubernetes components running in the form of bin (such as Kubelet, etc.) or POD (such as Flannel \kube-proxy, etc.), and business containers accessing Kube-Apiserver in the way of InCluster. Further, if the edge node network environment is special, need to use proxy to access cloud Kube-Apiserver, only for Lite-Apiserver proxy, all components can normally access cloud Kube-Apiserver, do not need to do a separate configuration for each component.
Supports caching of all types of resources
Support caching of all types of Resources, Kubernetes built-in Resources and Custom Resources. Kube-apiserver has a variety of resources. If only part of the resource types are cached or only Kubernetes built-in resource types are supported, components or services may fail because the corresponding cache cannot be read when the cloud edge is disconnected. The effect of edge node autonomy cannot be achieved. Of course, supporting caches of all types of Resources (especially Custom Resources) also presents challenges in data parsing and processing.
security
Edge nodes are widely distributed and the environment is complex, which is more likely to cause security risks. Security issues are also becoming more and more important in edge computing and Kubernetes management. It is a common access control scheme to grant an access permission to Lite-Apiserver so that all requests made by lite-Apiserver to the cloud kube-Apiserver are discarded. Since lite-Apiserver needs to access and process all types of resources, this permission must be a “super” permission. In this case, a malicious program on a single edge node can use Lite-Apiserver to operate on all resources in the cluster, potentially damaging the entire cluster. Therefore, from the perspective of security, lite-Apiserver is not designed to have a “super” permission. Instead, lite-Apiserver can use the authentication and authentication methods of Kubernetes components and service containers to access cloud Kube-Apiserver.
Supports a variety of cache storage
According to IDC’s definition of Edge computing layering, Edge is divided into Heavy Edge (Edge data center) and Light Edge (low-power computing platform). For different scenarios, Lite-Apiserver can use different cache storage strategies to achieve better results. In Light Edge, Lite-Apiserver uses file storage caching to reduce its own overhead and improve its versatility. In Heavy Edge, Lite-Apiserver can use KV storage to improve read and write performance.
In the following sections, we describe how Lite-Apiserver achieves the above functional requirements and design features in terms of its architecture and key technologies.
Lite-apache Server architecture and key technologies
architecture
The lite-Apiserver architecture is shown in the figure
In general, Lite-Apiserver starts an HTTPS Server to accept all Client requests. Select a ReverseProxy according to the Common Name in the REQUEST TLS certificate (default is used if the request does not have the MTLS certificate) and forward the request to Kube-Apiserver. When the cloud side network is normal, the client returns the corresponding result (HTTPS Response) and stores the response in the cache asynchronously as required. When the cloud edge is disconnected, the access to Kube-Apiserver times out, and the cached data is obtained from the cache and returned to the client to achieve the purpose of edge autonomy.
- The HTTPS Server listens on localhost port (51003 in SurperEdge) to receive HTTPS requests from clients.
- Cert Mgr && Transport Mgr Cert Mgr is responsible for managing TLS client certificates connected to Kube-Apiserver. It periodically loads configured TLS certificates and notifies Transport Mgr to create or update the corresponding Transport if there is an update. Transport Mgr is responsible for managing Transport. It receives notification from Cert Mgr to create a new transport or close the old connection with the transport whose certificate has been updated.
- Proxy Selects a ReverseProxy based on the Common Name in the REQUEST MTLS certificate (default is used if the Request does not have the MTLS certificate). Forward the request to Kube-Apiserver. If the request is successful, the result is returned directly to the Client and the data is cached by calling Cache Mgr. If the request fails, data is read from Cache Mgr to Client.
- Cache Mgr caches the result data of Get and List respectively according to the type of Client, and updates the corresponding List data according to the return value of Watch.
The key technology
1. HTTPS Server
Under the current architecture, lite-Apiserver only processes all requests of the local node. Using HTTP Server can meet performance and security requirements. However, most components and service containers use the client-Go library to access Kube-Apiserver. If HTTP Server is used, the authentication and authentication information of the client is lost, which does not meet the requirements of permission management. Therefore, the HTTPS Server must be used. The TLS Server certificate of Lite-Apiserver must be issued by a CA that is the same as the Kube-Apiserver Server certificate.
2. Support InCluster access
A common Client accesses kube-Apiserver using the URL specified by the kube-apiserver. When using lite-Apiserver, Simply replace the original KUbe-apiserver URL with the lite-apiserver address. The recommended way to access kube-apiserver in Pod is through the DNS name kubernetes.default. SVC, which will be resolved to the service IP, which will then be routed to kube-apiserver. Using Lite-Apiserver in this scenario requires a little “magic”. In SuperEdge, application-grid-Wrapper is deployed on each edge node in the form of DaemonSet, and kube-proxy only returns endpoints within the region to access the closed loop within the region. Application-grid-wrapper changes the endpoint of the kubernetes Service to lite-apiserver and returns it to the local kube-proxy. You can support InCluster access.
3. Supports Client Bootstrap Token and certificate rotation
Lite-apiserver uses the Client’s own authentication and authentication methods to access the Cloud cloud Kube-Apiserver. For the static token, bootstrap token, and Service Account, lite-Apiserver only needs to transparently transmit the authentication information contained in the Http Request Header. For TLS Client certificate authentication, Lite-apiserver loads all TLS Client certificates used by the Client by reading the configuration file, and uses these certificates to construct the corresponding HTTPS request kube-apiserver. To support Client Bootstrap Token and certificate rotation, Lite-Apiserver periodically loads and updates these certificates. By default, the certificate issued by Kube-Controller-Manager lasts one year. Lite-apiserver loads the TLS client certificate within a short period. However, if the loading cycle of the certificate is too long, kubelet uses the Bootstrap Token to update the certificate in a timely manner. To handle these scenarios, Lite-Apiserver uses an “elegant” certificate loading policy: When the certificate is loaded incorrectly or expires, it enters the quick loading mode with a period of 1s. If all certificates are successfully loaded, the system enters the common loading mode every 30 minutes. When the certificate is updated, lite-apiserver uses the closeAll method provided by client-go to close the existing connection to prevent authentication failure.
4. Cache resolution and update
Lite-apiserver needs to support caching of all types of resources, and caching resolution and updating is one of the keys to lite-Apiserver implementation. Lite-apiserver caches each Client’s Get and List requests to resources separately, which wastes some storage space but avoids complex resource version conversion. To Watch type of request as a result, lite – apiserver using unstructured. UnstructuredJSONScheme parsing a resource meta information, and then the corresponding List data update.
Looking forward to
SuperEdge has attracted a lot of attention since it was officially open source. In the rapid iterative development of SuperEdge, Lite-Apiserver also has a number of extensibility points. We welcome your active participation to build an excellent cloud native edge container project.
- Resources that are frequently updated are cached in the memory to improve performance
- Supports more types of storage
- Performance and memory optimization to accommodate a wider range of edge scenarios