Certificate is an indispensable part of Kubernetes, its role mainly has two functions, one is to ensure the integrity and security of data, and the other is to enable the server and client to verify each other’s identity through the certificate. Kubernetes is a distributed system. Components communicate with each other over the network. It is necessary to use certificates, namely HTTPS, to achieve secure transmission. HTTPS allows clients to verify the identity of the Server, for example to ensure that Kubectl and Kubelet are accessing a real API Server. At the same time, the Server can also enable TLS bidirectional verification to verify the identity of the client. For example, when we use Kubectl to access the API Server, the API Server will verify our client certificate (stored in KubeconFig) to judge the validity of the user, and further combine the permission management. Check whether the user has the corresponding permission. When we use kubeadm to create a cluster, kubeadm will create various certificates for us. Based on the cluster created by Kubeadm, this article summarizes the various certificates-related knowledge we need to know when deploying and operating a Kubernetes cluster.

The roots of Kubernetes CA

Certificates over the Internet need to be issued by an authoritative CA and cost a considerable amount of money, but since Kubernetes aims to build trust within the cluster, self-signed cas are sufficient. Kubernetes requires a root CA, and all certificates in the cluster (except for Kubelet’s server certificate) need to be issued by the root CA or an intermediate CA based on the CA. When using kubeadm init to create the control plane, kubeadm generates the root CA for us, along with the associated certificates. Kubeadm The default directory for storing the certificate of the created cluster is /etc/kubernetes/pki.

[root@master-1 ~]# ls -lh /etc/kubernetes/pki/total amount 56k-rw-r --r-- 1 root root 1.3k 10月 5 20:36apiserver.crt # 1 root root 1.1K 10月 5 20:36 apiserver-etcd-client. CRT # Apiserver client certificate for accessing ETCD -rw-------. 1 Root root 1.7K 10月 5 20:36 apiserver.key # Certificate private key -rw-------. 1 root root 1.7K 10月 5 20:36 apiserver.key # Certificate private key -rw-r--r--. 1 root root 1.1K 10月 5 20:36 apiserver-kubelet-client. CRT # apiserver -rw------- Root 1.1K 9月 8 22:58 ca.crt # root CA certificate 1 root root 1.7k 9月 8 22:58 ca.key # Root CA private key drwxr-xr-x. 2 root root 162 9月 8 22:58 etcd -rw-r--r--. 1 root Root 1.1K 9月 8 22:58 front-proxy-ca. CRT -rw-------. 1 root root 1.1K 9月 8 22:58 front-proxy-ca.key -rw-r--r-- 1 root Root 1.1K 10月 5 20:36 front-proxy-client. CRT -rw-------. 1 root root 1.7K 10月 5 20:36 front-proxy-client.key -rw------- 1 root root 1.7k 9月 8 22:58 sa.key # kube-controller-manager Private key used to sign service account token -rw-------. 1 root root 451 9月 8 22:58 sa.pub # kube-apiserver unlocks the public key required for the service Account token signatureCopy the code

The root CA certificate needs to be stored on all nodes in the cluster because all components in the cluster need to use the CA certificate to verify certificate signatures.

Server certificate

All components that expose the HTTPS service require a server certificate. In Kubernetes, Kube-Apiserver and Kubele provide HTTPS services externally. Kube-apiserver: kube-apiserver: kube-apiserver: kube-apiserver: kube-apiserver: kube-apiserver: kube-apiserver: kube-apiserver

[root@master-1 ~]# cat /etc/kubernetes/manifests/kube-apiserver.yaml ... - command: - kube - apiserver - - the client - ca - file = / etc/kubernetes/pki/ca. CRT # specified ca certificate... - - the TLS - cert - file = / etc/kubernetes/pki/apiserver CRT # specified server certificate - - the TLS - private - key - the file = / etc/kubernetes/pki/apiserver. Key # specify private key...Copy the code

Note that when kube-Apiserver is exposed via elastic IP, NAT gateway, or load balancing, the corresponding DNS Name or IP address should be added to the SANs (Subject Alternative Name) field of the server certificate. Otherwise, the client may report an error because the access address does not match the certificate. Kubeadm will help us set some default SANs. You can use Openssl to view the contents of the certificate:

[root@master-1 ~]# openssl x509 -noout -text -in /etc/kubernetes/pki/apiserver.crt Certificate: Data: ... Signature Algorithm: sha256WithRSAEncryption Issuer: CN=kubernetes Validity Not Before: Sep 8 14:58:50 2021 GMT Not After : Sep 8 14:58:51 2022 GMT Subject: CN=kube-apiserver Subject Public Key Info: .... X509v3 extensions: X509v3 Key Usage: critical Digital Signature, Key Encipherment X509v3 Extended Key Usage: TLS Web Server Authentication X509v3 Subject Alternative Name: # look here DNS: master - 1, DNS: kubernetes, DNS: kubernetes. Default, DNS: kubernetes. Default. SVC, DNS: kubernetes. Default. SVC. Cluster. The local, IP Address: 10.96.0.1, IP Address: 192.168.33.220 Signature Algorithm: sha256WithRSAEncryption ...Copy the code

It can be seen that the Subject Alternative Name field already contains node hostname (master-1), service DNS (kubernetes.default, etc.), Service IP address (10.96.0.1) and Node IP address (192.168.33.220).

When creating a cluster using kubeadm, we can specify sans in init with –apiserver-cert-extra-sans. Kubeadm will set the sans field when generating the certificate.

Client Certificate

Kubernetes components verify the identity of the client through TLS two-way verification. All client certificates issued by the cluster root CA are considered authenticated and HTTPS connections can be established. Further, Kubernetes reads the user name and user group information from the client certificate. Determine whether the client has the Authorization based on RBAC authentication policies. In client certificates, Common Name indicates the user Name and Organization indicates the user group.

Kubelet client certificate

Kubelet When accessing kube-apiserver, you need to bring the client certificate (TLS biometric verification). The default directory for saving the certificate is /var/lib/kubelet/pki.

[root@master-1 ~]# ls-lh /var/lib/kubelet/pki total amount 12k-rw -------. 1 root root 2.7k 9月 8 22:59 Kubelet-client-2021-09-08-22-59-01. pem LRWXRWXRWX. 1 root root 59 9月 8 22:59 kubelet-client-current-pem -> The/var/lib/kubelet/pki/kubelet - the client - the 2021-09-08-22-59-01. Pem - rw - r - r -. 1 root root on September 8 after 2.2 K kubelet. CRT -rw-------. 1 root root 1.7k 9月 8 22:58 kubelet.keyCopy the code

Kubelet-client-2021-09-08-22-59-01.pem indicates the client certificate, and kubelet-client-current. Pem is a soft link of the certificate. The reason why certificates are dated is because Kubelet’s certificates are automatically updated when they are about to expire, so wearing the time is convenient to distinguish between new and old certificates. Check the certificate information:

[root@master-1 ~]# openssl x509 -noout -text -in /var/lib/kubelet/pki/kubelet-client-current.pem Certificate: Data: Version: 3 (0x2) Serial Number: 7265397798501666783 (0x64d3e0cdd398cfdf) Signature Algorithm: sha256WithRSAEncryption Issuer: CN=kubernetes Validity Not Before: Sep 8 14:58:50 2021 GMT Not After : Sep 8 14:58:55 2022 GMT Subject: O=system:nodes, CN=system:node:master-1 # O indicates Organization, CN indicates Common Name Subject Public Key Info:....Copy the code

You can see that the kubelet user group is System: Nodes and the user name is System :node:master-1. With this information, Kube-Apiserver can use the Node Authorizer to restrict Kubelet to only reading and modifying resources on this Node.

Kubernetes manages API requests from Kubelet using an authorization called Node Authorizer. To be authorized by the Node Authorizer, Kubelet must use a credential that identifies them as the System: Nodes user group with the user name System: Node:

. Reference: Node Authorizer kubernetes. IO/docs/refere…

Kube-apiserver client certificate

Similarly, when kube-Apiserver calls kubelet (exec/logs command), Kubelet also asks to verify kube-Apiserver’s client certificate. Save the certificate path of the/etc/kubernetes/pki/apiserver – kubelet – client. The CRT. Check the certificate information:

[root@master-1 ~]# openssl x509 -noout -text -in /etc/kubernetes/pki/apiserver-kubelet-client.crt
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 5903840273946896753 (0x51eea773033fe971)
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN=kubernetes
        Validity
            Not Before: Sep  8 14:58:50 2021 GMT
            Not After : Oct  5 12:36:44 2022 GMT
        Subject: O=system:masters, CN=kube-apiserver-kubelet-client
        ....
Copy the code

The user group for this certificate is System: Masters, which is Kubernetes’ built-in user group to which Kubernetes automatically binds cluster administrator rights.

# ClusterRole/cluster-admin contains all operation permissions that are automatically bound to the System :masters user group when kube-apiserver is started. [root@master-1 ~]# kubectl get clusterRoleBinding cluster-admin -owide NAME ROLE AGE USERS GROUPS SERVICEACCOUNTS cluster-admin ClusterRole/cluster-admin 27d system:mastersCopy the code

Kubectl client certificate

When we use Kubectl to access Kube-Apiserver, we also need to provide the client certificate. Kubectl client certificates are stored in KubeconFig:

[root@master-1 ~]# cat ~/.kube/config apiVersion: v1 clusters: - cluster: certificate-authority-data: "Said the CA certificate, a long base64 encoding > 1) server: https://192.168.33.220:6443 name: kubernetes contexts: - the context: cluster: kubernetes user: kubernetes-admin name: kubernetes-admin@kubernetes current-context: kubernetes-admin@kubernetes kind: Config preferences: {} users: - name: kubernetes-admin user: client-certificate-data: < represents the client certificate, a long base64 encoding > ② Client-key-data: < represents the client certificate private key, a long base64 encoding >Copy the code

Kubeconfig = kube-apiserver; kubeconFig = Kube-apiserver; kubeconFig = Kube-apiserver;

② The client certificate is encoded in Base64 and stored in the client-certificate-data field. Take a look at the client certificate section:

[root@master-1 ~]# grep "client-certificate-data" ~/.kube/config \ | sed 's/\s*client-certificate-data:\s*//' \ | base64  -d | openssl x509 -noout -text Certificate: Data: Version: 3 (0x2) Serial Number: 368716738867724927 (0x51df1eba2f3827f) Signature Algorithm: sha256WithRSAEncryption Issuer: CN=kubernetes Validity Not Before: Sep 8 14:58:50 2021 GMT Not After : Sep 8 14:58:54 2022 GMT Subject: O=system:masters, CN=kubernetes-admin # O for Organization, CN for Common Name....Copy the code

The kubeconfig user name is kubernetes-admin, and the user group is system:masters. Kubeconfig in the example is the cluster administrator certificate created by default by kubeadm. The cluster administrator can issue the certificate of any user group and user name and make it into KubeconFig, combined with RBAC permission management to control permissions.

Kube – controller – manager and kube – the scheduler

Kube-controller-manager and Kube-Scheduler also require client certificates to access kube-Apiserver. Their client certificates are stored in their kubeconFig credentials.

[root@master-1 ~]# ls -ls /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf 8 -rw-------. 1 root Root on October 5, 5486 joined the/etc/kubernetes/controller - manager. Conf. 8 - rw. -- -- -- -- -- -- -- 1 root root on October 5, 5438 clients /etc/kubernetes/scheduler.confCopy the code

The user name of kube-controller-manager is system:kube-controller-manager, and the user name of kube-Scheduler is system: Scheduler. Even for internal components, Kubernetes will bind only necessary permissions for these two users based on the principle of minimum permissions.

System: prefix is a keyword reserved by Kubernetes to describe the built-in system users and system user groups that are bound by default when Kube-Apiserver starts

Certificate of rotation

The certificate has an expiration time. The default expiration time of the cluster built by Kubeadm is 1 year. You need to update the certificate in time to prevent the cluster from working due to the certificate expiration.

Manual rotation

With the exception of kubelet’s client/server certificates, which can be rotated automatically, other certificates need to be updated manually. Kubeadm provides two update methods:

usekubeadm upgradeAfter the cluster is upgraded, the certificate is updated and the validity period is extended by one year.
usekubeadm cert renewCommand to manually update the certificate to extend the validity period by one year. Restart the control plane.

For security reasons, kubeadm does not provide a configuration to modify the certificate expiration time, and recommends that we upgrade the cluster at least once a year. But the production cluster is difficult to do 1 year upgrade so frequently, if the security requirements are not so high, modify kubeadm source code, the certificate default time is set a bit longer is also a good choice.

Detailed documentation: kubernetes. IO/docs/tasks /…

Automatic rotation

Kubelet’s client certificates are generally not our concern and can be updated automatically. Server certificates can also be configured for automatic update, but this is more complicated and will be explained separately in kubelet’s server Certificates section. When kubelet found his client certificate expiration, will create a CSR (CertificateSigningRequest) resources to apply for a new certificate. Kube-controller-manager will decide whether to approve the CSR based on kubelet’s current permissions. Once approved, Kube-Controller-Manager will issue a new certificate for Kubelet and save it to the CSR. Kubelet then downloads the certificate from the CSR and saves it to a local disk for certificate rotation.

Detailed documentation: kubernetes. IO/docs/refere…

The root CA rotate

The root CA certificate created by Kubeadm has a looser default validity of 10 years. Kubernetes does not provide a method to automatically update the root CA. You can refer to the documentation for manual updates.

CSR (CertificateSigningRequest)

Kubernetes provides a set of Certificate API (Certificate API), which is used to realize the application and automatic issuance of certificates. A certificate applicant, such as Kubelet, can request a certificate signature from a specified certificate signer (specified by the CSR singerName field) by creating a CSR resource. The CSR request may be approved or rejected. After the CSR request is approved, the signer will sign the certificate and save the signed certificate in the STATus. Certificat field of the CSR, and the whole issuing process is completed. Applicants for certificates can obtain signed certificates from Status. Certificate. Kube-controller-manager has some signers built in to handle CSR requests corresponding to singerName:

kubernetes.io/kube-apiserver-client, apply for a certificate to access API Server,There is no automatic approval.
kubernetes.io/kube-apiserver-client-kubelet.kubeletApply for a client certificate to access the API Server,It may be automatically approved.

kubernetes.io/kubelet-serving.kubeletServer certificate of,There is no automatic approval.
kubernetes.io/legacy-unknown, certificate application for third-party applications,There is no automatic approval.

IO /kube-apiserver-client-kubelet, kube-Controller-manager, kube-controller-manager, kube-controller-manager The certificate is then issued.

CSR approval/rejection

After the CSR is submitted, it needs to be approved by the approver (either a person or a program) before subsequent certificate issuing operations can be carried out. Kube-controller-manager has a built-in approval controller that automatically approves certain CSR requests. However, to prevent conflicts with other approvers, Kube-Controller-Manager does not explicitly reject any CSR. For CSRS that will not be handled by Kube-Controller-Manager, we can use API methods such as kubectl command line:

# CSR $kubectl certificate deny # CSR $kubectl certificate deny <certificate-signing-request-name>Copy the code

Or implement a special controller to automatically approve or reject.

Kubelet client certificates are automatically approved

For the CSR of kubernetes. IO/Kube-Apiserver-client-Kubelet type, Kube-Controller-Manager decides whether to approve the CSR according to whether the applicant has the corresponding RBAC permission. Kubelet creates CSR requests in both cases:

When joining the cluster for the first time, the client certificate has not been generated.kubeletYou need to create CSR resources to apply, which is the TLS boot phase (TLS bootstrapping).
When the client certificate is about to expire,kubeletCertificate rotation is initiated and a CSR request is created to apply for a new certificate.

For both scenarios, Kubernetes provides two default permissions (ClusterRoles) :

nodeclient: When a node creates a certificate for the first time,kubeletThere is no official client certificate yet and it is in the TLS boot phase. At this timekubeletWill usebootstrap tokenAuthentication mode to request API Server.kubeadm initTo create thebootstrap tokenThe owning user group issystem:bootstrappers:kubeadm:default-node-token.kubeadmWill be responsible for thenodeclientThe permission is assigned to the user group.
selfnodeclient: When a node requests certificate rotation,kubeletYou already have an official client certificate.kubeletThe certificate belongs tosystem:nodesUser groups,kubeadmWill be responsible for theselfnodeclientThe permission is assigned to the user group.

Two # default permissions/root @ master - 1 ~ # kubectl get clusterrole - l kubernetes. IO/bootstrapping = rbac - defaults | grep nodeclient system:certificates.k8s.io:certificatesigningrequests:nodeclient 2021-09-08T14:59:17Z System: certificates. K8s. IO: certificatesigningrequests: selfnodeclient T14:2021-09-08 # kubeadm Lord Z will be responsible for the binding the two permissions to the corresponding user group [root@master-1 ~]# kubectl get clusterrolebinding kubeadm:node-autoapprove-bootstrap kubeadm:node-autoapprove-certificate-rotation -owide NAME ROLE AGE USERS GROUPS SERVICEACCOUNTS kubeadm:node-autoapprove-bootstrap ClusterRole/system:certificates.k8s.io:certificatesigningrequests:nodeclient 27d system:bootstrappers:kubeadm:default-node-token kubeadm:node-autoapprove-certificate-rotation ClusterRole/system:certificates.k8s.io:certificatesigningrequests:selfnodeclient 27d system:nodesCopy the code

Kube-controller-manager checks the permission and automatically approves the CSR request.

Reference: kubernetes. IO/docs/refere…

Issue of CSR Certificate

When the CSR request is approved, the issuer can issue the certificate. Kube-controller-manager also has a built-in issuing controller. Turn on the built-in signing controller by setting the –cluster-signing-cert-file and –cluster-signing-key-file startup parameters for kube-controller-manager, which represent the certificate and private key used for signing, respectively. CA certificate for cluster:

[root@master-1 pki]# cat /etc/kubernetes/manifests/kube-controller-manager.yaml
apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: null
  labels:
    component: kube-controller-manager
    tier: control-plane
  name: kube-controller-manager
  namespace: kube-system
spec:
  containers:
  - command:
    - kube-controller-manager
    ...
    - --cluster-signing-cert-file=/etc/kubernetes/pki/ca.crt
    - --cluster-signing-key-file=/etc/kubernetes/pki/ca.key
    ...
Copy the code

Prior to 1.18, Kube-Controller-Manager issued certificates for all approved CSRS. After 1.18, Kube-Controller-Manager limited the CSR singerName and only issued certificates for the above four CSR requests that specified the singerName. Similarly, CSR requests that do not automatically issue certificates can also be issued manually through Kubectl, or by implementing a dedicated controller to automatically issue certificates.

The document reference: kubernetes. IO/docs/refere…

Kubelet server certificate

Kubelet also exposes HTTPS services. Its clients are Mainly Kube-Apiserver and some monitoring components, such as Metric-Server. Kube-apiserver needs to access Kubelet to get container logs and execute commands (Kubectl logs/exec). Monitoring components need to access the CAdvisor interface exposed by Kubelet to get monitoring information. Ideally, we need to configure Kubelet CA certificates into Kube-Apiserver and Metry-Server to verify kubelet server certificates for security. This is not possible using the default cluster setup method and requires some extra work.

As mentioned earlier, all Kubernetes certificates except kubelet server certificates are issued by the cluster root CA (or an intermediate CA based on the root CA). Kubelet’s certificate does not require this. In fact, when Kubelet starts up, if no server certificate path is specified, it creates a self-signed CA certificate and uses that CA to issue itself a server certificate.

Kubelet can also use Serving certificates. Kubelet itself provides an HTTPS terminal that contains several features. To secure these ends, Kubelet can do one of the following:

Use the keys and certificates set with — tlS-private-key-file and –tls-cert-file

If the key and certificate are not provided, a self-signed key and certificate is created

Request the service certificate from the cluster server through the CSR API

The document reference: kubernetes. IO/docs/refere…

This means that each kubelet root CA is different, which results in client components such as Metric-Server and even Kube-Apiserver not being able to verify Kubelet’s server certificates. To deal with this, metrical-server needs to add –kubelet-insecure- TLS to skip the validation of server certificates, which kube-apiserver does not validate by default.

Can you apply for a server certificate using A CSR just like you apply for a client certificate? The CSR issuer uses the root CA of the cluster to issue the server certificate for each Kubelet. Then Kube-Apiserver and other components can configure the root CA of the cluster to implement HTTPS server certificate verification. The answer is yes, we can enable this feature by setting serverTLSBootstrap to true in the Kubelet configuration file, that is, using a CSR to apply for server certificates instead of using a self-signed CA to issue certificates. This configuration also enables automatic rotation of server certificates. However, this process is not fully automated. As mentioned in the certificate Rotation section, kubelet’s server certificate CSR request (singerName kubernetes.io/kubelet-serving CSR request), It will not be automatically approved by kube-Controller-Manager, which means we need to manually approve these CSRS, or use a third party controller.

Why doesn’t Kubernetes automatically approve kubelet’s server certificate? Isn’t that convenient? The reason was for security reasons – Kubernetes did not have sufficient ability to tell whether the CSR should be approved or not.

The HTTPS server certificate is used to prove that “I am myself” to the client, preventing communication between the client and someone pretending to be “ME”, that is, preventing man-in-the-middle attacks. When applying for a certificate to an authoritative CA organization, we need to provide a series of proof materials to prove that the site is mine, including that I am the owner of the domain name of the site, and the CERTIFICATE will be issued only after the CA has approved it. However, the K8S cluster itself is not capable of identifying kubelet identity, because node IP and DNS name may change, and K8S itself is not capable of determining which IP and DNS are legitimate, which falls within the responsibility of the infrastructure manager. If your cluster is provided by a cloud vendor, your cloud vendor can provide a controller to determine the validity of a CSR request and approve a valid CSR request. In a self-built cluster, only the cluster administrator can determine whether the node IP address and DNS name contained in the CSR request are valid. If kube-Controller-Manager automatically issues these certificates, there is a risk of a man-in-the-middle attack.

This PR describes a specific scenario of a man-in-the-middle attack. Assume that the service bar on node A uses HTTPS to expose the service, and the server certificate is obtained through A CSR request and issued by the cluster root CA. Assuming that an intruder has access to node A, he can easily use kubelet’s client certificate access to create A CSR request to request A server certificate with barService IP and DNS name barService DNS. If kube-Controller-Manager automatically passes and issues this certificate, then the intruder can use this certificate, in conjunction with kube-proxy on the node, to hijack all traffic passing through the BAR service.

If you are interested in this topic, you can check out the discussion of these two issues: github.com/kubernetes/… Github.com/kubernetes/…

conclusion

Kubernetes as a distributed system, the use of HTTPS encryption communication between components is a basic requirement. The certificate is also used as an access certificate for clients. The server uses TLS bidirectional verification to authenticate clients and manages user permissions in combination with RBAC and Node Authorizer authentication schemes. To address the challenge of certificate management, Kubernetes provides the ability to rotate certificates automatically, kubelet’s TLS auto-boot, and apply/issue certificate apis to reduce management complexity. However, we should pay special attention to the validity of cluster certificates and update them timely to avoid expiration. Kubelet’s server certificate is special. If you want to follow the best HTTPS security practices, that is, verify HTTPS server certificates, you can enable kubelet’s automatic server certificate rotation and use a secure way to approve Kubelet’s server certificate CSR request.

What you need to know about Kubernetes certificates