Currently, e-commerce platforms will use Docker-based container technology to host key business segments during the 618 campaign, including the simplest product photo display, order details page, and so on.
Through container transformation, each business section of the e-commerce platform is decouped and can be independently developed, deployed and online, so that the background business system has higher stability, scalability and security. Even if there is a problem in a link, the smooth operation of the platform during peak value can be guaranteed.
Image is the cornerstone of Docker container, only through it can create container, and Registry is the repository of Docker image. However, in practical applications, file transfer becomes the bottleneck of image distribution due to the frequent downloading of images from Registry to run container applications (such as releasing new versions and making patches, etc.).
P2P accelerated image download is an effective solution, but how to ensure the security of P2P transmission of user data in the public cloud environment is particularly critical. This paper mainly expounds the security of P2P acceleration of public cloud Docker image from the security reinforcement of link layer and business layer.
Question:
When using Docker to run containerized applications, the host usually first downloads the corresponding image from Registry service (such as Docker Hub). This mirroring mechanism works well in a development environment where team members can easily share the same image. In actual production environment, however, when a large number of host need download mirror from the Registry at the same time run container applications (such as a new release, patch, etc), Registry services tend to become the bottleneck of image distribution, applied the mirror takes a long time to transfer to all host, make application release cycle greatly extended.
Many enterprises have proposed P2P solutions to accelerate image download, but they are only used in private clouds and internal environments, and are not used in public clouds. A large part of the reason is the security problem of public cloud using P2P, how to ensure the security of user data in P2P transmission has become a difficulty. We design and implement a P2P image distribution system to ensure user data security. This paper expounds its security.
Architecture:
Huawei P2P container image distribution system consists of three components: Proxy, BT client, and BT Tracker.
Client Proxy
The Client Proxy is deployed on each node of the cluster and configured as the Docker Http Proxy. It intercepts the image download request of the Docker Daemon, notifies the Client to download the image, and finally imports the image into the Docker Daemon.
BT the client
The BT client and Tracker deployed on the cluster nodes together form a complete P2P file transfer system. In the whole image distribution process, they use BT protocol to complete image download.
BT Tracker
Tracker is a part of the BT system that stores metadata information and seed information required by BT clients during the download process and assists each BT client to complete the entire communication process.
Security:
First, we limited P2P downloads across clusters to minimize data leakage between tenants.
After that, security at the link level and service level is enhanced.
Link security
When we think of link security, the first thing that comes to mind is encryption.
Symmetric encryption The server and client use the same secret key for encryption and decryption. As long as the secret key is not public and the secret key is secure enough, the link is secure. But using the same symmetric encryption key across the network is tantamount to public transmission, and if the key is hijacked, all the data on the link can be tampered with.
And then of course we think about HTTPS, how is it secure? Let’s take a look at how HTTPS is implemented.
In the process of data transmission, HTTPS adopts symmetric encryption and decryption, but it adds the process of handshake negotiation during connection establishment.
What is a public key?
Public key is a concept in asymmetric encryption. Asymmetric encryption is based on a secret key pair. Data is encrypted with one secret key and can only be decrypted with the other secret key. The server saves the private key and sends the public key to the client.
Let’s assume a scenario where we generate a secret key pair, where the client encrypts the data using the public key and the server decrypts the data using the private key. So even if the user hijacks the public key, he cannot hijack and tamper with the user’s data. However, the link from the server to the client is not secure.
HTTPS uses this feature of asymmetric encryption to ensure that the transfer of symmetric secret keys is secure, and finally, symmetric encryption is used to transmit data.
Meaning of certificate:
However, this raises a new question: what if the public key is hijacked?
HTTPS, of course, will not be so easily hijacked. To address the appeal, it introduced digital certificates and third-party institutions. A certificate is issued by a third-party certification authority using a public key. The certificate contains not only the public key, but also the signature (encrypted by the private key of the issuing node), expiration date, issuing authority, web address, and expiration date.
HTTPS returns a certificate instead of a private key. When receiving a certificate, the client verifies the certificate. An authoritative list of third-party organizations (including their public keys) is maintained on each machine. When a client needs a public key, it can locally search for the public key based on the issuer information. The client uses the public key of the issuing authority to verify the validity of the signature and the integrity of the certificate to ensure that the public key is not tampered with.
HTTPS ensures link security by using the private key, certificate, and CA (issuing authority). In P2P scenarios, BT clients are peer to peer, and they transfer data to each other. Therefore, the server verification Client is more important than the HTTPS Client verification server. And since BT Client is a node deployed on the user, there is also the risk of the certificate and private key being hijacked.
How did we do that
- Between the Client
Data transmitted between BT clients must be encrypted to prevent link data from being hijacked. However, if only HTTPS is added, although the link is encrypted, the Client may be impersonated. As long as the impersonator directly shakes hands with the server without verifying the certificate of the server, he can obtain the data he wants from other BT clients.
We learn from HTTPS implementation, using two-way authentication mode.
A certificate is required, and a unified CA (issuing authority) is required first. Therefore, we store the certificate and private key in Tracker as the issuing authority. When Proxy obtains the seed, it returns the CA, and the user verifies the certificate of the client.
Then, it is dangerous to use only one certificate pair and put it in Bt Client. There is a high possibility that the certificate will be intercepted by intrusion. Therefore, we obtain the certificate from Tracker, obtain the temporary certificate private key generated by Tracker at the same time, and add it to the download queue of Bt Client. When BT clients are connected to each other, they must verify the validity of each other’s certificates (such as signatures and issuing authorities), and then request and download data from each other.
In this mode, the links between clients are secure.
(1) The link is encrypted by the certificate, so it is not feasible to directly intercept the link
(2) Even if we imitate the BT Client, since each connection of the Client requires bidirectional certificate verification, if we want to intercept data in this way, we must request Tracker to obtain it, and the access to Tracker is HTTPS first, and then we have done the security verification of the business layer (mentioned in the security of the business layer below). Nor is it feasible.
- Docker Daemon to Proxy
We need to hijack Docker’s request in Proxy, because Docker uses HTTPS to access Registry when it is not configured, so Proxy hijack Docker request must maintain HTTPS connection with Docker.
We made the client agent listen only on the localhost port, eliminating the possibility of external use of the agent. At the same time, the client proxy binds a set of temporary self-signed certificates and CA certificates issued to The Registry domain, which are used to hijack the Docker Daemon’s requests and add the CA certificate to the machine’s trust certificate.
The certificate bound by the proxy is only stored in memory. Even if the CA certificate and server certificate of the current node are obtained in a specific way, data of other nodes cannot be intercepted.
- From user node to Registry, Tracker
First, to ensure link security, both Regstry and Tracker bind HTTPS certificate private keys purchased from authoritative third-party organizations. Both Proxy and BT clients verify the validity of the certificate when accessing them, and send requests only when the certificate is valid. This prevents the possibility of Regstry and Tracker being counterfeits.
Business encryption
After securing the link, we also harden the layer of service security. First let’s take a look at JWT Token.
Json Web Token (JWT) is a JSON-based development standard (RFC 7519) implemented for the transfer of declarations between network application environments. The token is designed to be compact and secure, especially suitable for single sign-on (SSO) scenarios in distributed sites. The JWT declaration is generally used to pass authenticated user identity information between the identity provider and the service provider to obtain resources from the resource server, and to add some additional declaration information necessary for other business logic. The token can also be used directly for authentication or can be encrypted.
When we use the Docker command to download the image, Docker will first go to Registy to obtain the Token, and later in the process of obtaining the image layer, the Token will be used for authentication. The Token group must contain the following information:
(1) User information
(2) Resource information, which is the name of the operation mirror and namespace
(3) Permission information: Whether you have PULL/PUSH permission
(4) Certificate used to parse Token signature
Using the feature of resolving Token certificate in JWT Token, we added Token verification in the communication between BT clients. After the certificate verification succeeds, the client needs to send a Token to the server for verification. In order to prevent the Token from being counterfeited, the intruder uses the third party to generate the Token. We use the server to intercept the certificate in the Token intercepted from Docker to parse and verify the Token of the client to prevent this situation.
At the same time, the Proxy interface to access the Tracker will also carry the Token, and the Tracker will verify the permission of the Token to complete security verification at the service layer to prevent certificate and seed theft.
Through the security hardening at the link layer and service layer, the possibility of user data theft is almost zero. If you have better suggestions, feel free to comment.
Click to follow, the first time to learn about Huawei cloud fresh technology ~
