Stupid not clear kubernetes certificate

Kubeadm generated a pile of certificates is not very confusing, these things are not so magical, to dig into his underwear.

root@k8s-master:/etc/kubernetes/pki# tree . |-- apiserver.crt |-- apiserver-etcd-client.crt |-- apiserver-etcd-client.key |-- apiserver.key |-- apiserver-kubelet-client.crt |-- apiserver-kubelet-client.key |-- ca.crt  |-- ca.key |-- etcd | |-- ca.crt | |-- ca.key | |-- healthcheck-client.crt | |-- healthcheck-client.key | |-- peer.crt | |-- peer.key | |-- server.crt | `-- server.key |-- front-proxy-ca.crt |-- front-proxy-ca.key |-- front-proxy-client.crt |-- front-proxy-client.key |-- sa.key `-- sa.pub 1 directory, 22 filesCopy the code

From the RSA

To understand the role of certificates, you first need to understand some principles and have some basic knowledge, such as what asymmetric encryption, public key, private key, digital signature, etc. Let’s start with the RSA algorithm.

Asymmetric encryption generates a key pair, such as sa.key sa.pub above, one for encryption and one for decryption.

Plaintext + Public key => Ciphertext

Ciphertext + Private key => Plaintext

Then without the private key, it would be difficult to decrypt the ciphertext.

Look at the principles in more detail, skip the principles section below if you don’t want to:

Let’s say we want to encrypt the word Caesar, let’s turn it into a string of numbers, like Ascii X = 067097101115097114 and that’s what we want to encrypt. Now let’s encrypt X.

1. Take two very large primes P and Q and take their product N = P * Q and let M = (p-1)(q-1).
2. Find a number E where E and M have no common divisor except 1
3. Find a number D where E times D divided by M is remainder 1, E times D mod M is equal to 1

Now E is the public key and can be encrypted publicly to anyone

D is the private key used for decryption. Make sure you keep it yourself

The connection between the public key and the private key N is public, why is this public, because it’s easy to calculate N from P and Q, but it’s very difficult to decompose N into P and Q, so it’s very difficult to solve with existing computer computing power

Now to encrypt:

Pow (X,E) mod N = Y, Y is ciphertext, now without D(private key) you can’t figure out X(plaintext)

Decryption:

Pow (Y,D) mod N = X, X is plaintext, so plaintext comes out.

Math is not very magic, can now be considered sa.key = D sa.pub = E

A digital signature

Suppose you wrote a letter to your boss saying “I admire you boss” and then asked a colleague to send it to him. How can you be sure that you wrote the letter and how can you prevent them from changing it to “You are a moron boss “?

You can do this by first generating a key pair, giving the public key to the boss, then making a hash digest of the message, encrypting the digest with the private key, and the result is the signature

The boss gets the letter and decrypts it with the public key. The hash value is the same as the hash value of the letter, so the boss is sure that you wrote the letter

So a digital signature is an application of encryption. The difference from a fully encrypted message is that it’s public and your colleagues can see you puffing on your boss.

The digital certificate

Root certificates and certificates

Usually, you need to apply for a certificate from an “authority” when configuring the HTTPS service.

Here’s how it works:

1. The website creates a key pair that provides public keys and organizational and personal information to authorities
2. Certificate issued by authority
3. Browsing web friends using the authority of the root certificate public key decryption signature, compared with the abstract, to determine the legitimacy
4. The client authenticates domain name information validity time, etc. (Browsers are basically built-in CA public keys of various authorities)

This certificate contains the following:

1. Applicant’s public key
2. Applicant’s organizational and personal information
3. CA information, valid time, serial number, etc
4. Signature of the above information

The root certificate is also called a self-signed certificate, which is a certificate issued by the user. The Certificate Authority (CA) is called the Certificate authorization center. The CA Certificate in K8S is the root Certificate.

Certificate of kubernetes

With the above foundation, let’s officially begin…

First category:

Key pair: sa.key sa.pub Root certificate: ca.crt etcd/ CA private key: ca.key

First of all, other certificates are issued by the CA root certificate. Kubernetes and ETCD use a different CA. It is important to know whether the certificate is used for client verification or server verification. One by one:

Service Account Key pair sa.key sa.pub

The token is signed by the kube-controller-Manager using the sa.key. The master node verifies the signature through the public key sa.pub. For example, kube-proxy is run in the form of POD. In POD, service Account is directly used to authenticate with Kube-apiserver. In this case, there is no need to create a certificate for kube-proxy, and the token verification is directly used

Root certificate

pki/ca.crt
pki/ca.keyCopy the code

It is the K8S cluster certificate issuing authority

Apiserver certificate

pki/apiserver.crt
pki/apiserver.keyCopy the code

Kubelet certificate

pki/apiserver-kubelet-client.crt
pki/apiserver-kubelet-client.keyCopy the code

Kubelet needs to proactively access Kube-Apiserver, and Kube-Apiserver needs to proactively initiate requests to Kubelet. Therefore, both parties need to have their own root certificates and server certificates and client certificates issued using the root certificates. In Kube-apiserver, server certificates for HTTPS access and client certificates with CN user name information are generally explicitly specified. In the startup configuration of Kubelet, only the CA root certificate is specified, but the server certificate for HTTPS access is not explicitly specified. When the server certificate is generated, the server address or host name is usually specified. The relative change of Kube-apiserver is not very frequent. The IP address or host name/domain name that can be used as a Kube-apiserver can be pre-assigned at the beginning of cluster creation. However, because the kubelet deployed on a node changes frequently due to the size of the cluster, it is impossible to predict all the IP information of the node. Therefore, kubelet does not specify the server certificate explicitly, but only the CA root certificate. Kubelet automatically generates the server certificate based on the local host information and saves it to the configured cert-dir folder

Certificate of Aggregation

Proxy root certificate:

pki/front-proxy-ca.crt
pki/front-proxy-ca.keyCopy the code

Client certificates issued by agent root certificates:

pki/front-proxy-client.crt
pki/front-proxy-client.keyCopy the code

For example, when kubectl proxy is used for access, kube-apiserver uses this certificate to verify whether the client certificate is issued by the kube-apiserver.

Etcd root certificate

pki/etcd/ca.crt
pki/etcd/ca.keyCopy the code

Etcd Peer certificate for communication between etCd nodes

Issued by the root certificate

pki/etcd/peer.crt
pki/etcd/peer.keyCopy the code

Lip probe client certificate in POD

pki/etcd/healthcheck-client.crt
pki/etcd/healthcheck-client.keyCopy the code

To view the YAML active configuration:

Liveness:       exec [/bin/sh -ec ETCDCTL_API=3 etcdctl \
  --endpoints=https://[127.0.0.1]:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/healthcheck-client.crt \
  --key=/etc/kubernetes/pki/etcd/healthcheck-client.key get foo] \
  delay=15s timeout=15s period=10s #success=1 #failure=8Copy the code

Apiserver Certificate for accessing the ETCD

pki/apiserver-etcd-client.crt
pki/apiserver-etcd-client.keyCopy the code

Note the difference between a client certificate and a server certificate. A server certificate usually verifies an address, domain name, etc.

Code implementation

Kubeadm wrote the certificate down to a year (client-go did), a sad story that resulted in Sealos having to strip out the certificate generation logic to allow the installation to support arbitrary expiration times.

It may bea bit tiring to look at the kubeadm code directly. The sealos/ Cert directory is stripped of the core code to make it easier to read.

The following in order to highlight the core logic and delete some error handling code details, interested can read github.com/fanux/sealos/cert source code

Key pair generation

// create sa.key sa.pub for service Account
func GenerateServiceAccountKeyPaire(dir string) error {
    key, err := NewPrivateKey(x509.RSA)
    pub := key.Public()
    err = WriteKey(dir, "sa", key)
    return WritePublicKey(dir, "sa", pub)
}Copy the code

Generate the private key, where keyType is x509.rsa

func NewPrivateKey(keyType x509.PublicKeyAlgorithm) (crypto.Signer, error) {
    if keyType == x509.ECDSA {
        return ecdsa.GenerateKey(elliptic.P256(), rand.Reader)
    }
    return rsa.GenerateKey(rand.Reader, rsaKeySize)
}Copy the code

Generating a CA Certificate

Ca.crt (self-signed certificate) ca.key(private key)

func NewCaCertAndKey(cfg Config) (*x509.Certificate, crypto.Signer, error) {
    key, err := NewPrivateKey(x509.UnknownPublicKeyAlgorithm)
    cert, err := NewSelfSignedCACert(key, cfg.CommonName, cfg.Organization, cfg.Year)
    return cert, key, nil
}Copy the code

To generate a self-signed certificate based on the private key, NotAfter is the expiration time of the certificate. Instead of writing death, we have a friendly variable:

// NewSelfSignedCACert creates a CA certificate
func NewSelfSignedCACert(key crypto.Signer, commonName string, organization []string, year time.Duration) (*x509.Certificate, error) {
    now := time.Now()
    tmpl := x509.Certificate{
        SerialNumber: new(big.Int).SetInt64(0),
        Subject: pkix.Name{
            CommonName:   commonName,
            Organization: organization,
        },
        NotBefore:             now.UTC(),
        NotAfter:              now.Add(duration365d * year).UTC(),
        KeyUsage:              x509.KeyUsageKeyEncipherment | x509.KeyUsageDigitalSignature | x509.KeyUsageCertSign,
        BasicConstraintsValid: true,
        IsCA:                  true,
    }

    certDERBytes, err := x509.CreateCertificate(rand.Reader, &tmpl, &tmpl, key.Public(), key)
    return x509.ParseCertificate(certDERBytes)
}Copy the code

Note the CommonName and Organization fields, which are useful if you create a k8s user and specify which user group the user belongs to.

For example, fanux in the certificate belongs to sealyun, so generating a Kubeconfig is equivalent to having fanux as a user. In this way, K8S only need to verify the signature when doing authentication, instead of accessing the database to do authentication, which is very beneficial to the horizontal expansion of Apiserver.

Generate other certificates

The key pair is generated by itself, and the root certificate information is carried with the visa

func NewCaCertAndKeyFromRoot(cfg Config, caCert *x509.Certificate, caKey crypto.Signer) (*x509.Certificate, crypto.Signer, error) {
    key, err := NewPrivateKey(x509.UnknownPublicKeyAlgorithm)
    cert, err := NewSignedCert(cfg, key, caCert, caKey)

    return cert, key, nil
}Copy the code

Usages on the server or client must be specified. Note the difference between SelfSign and the above

// NewSignedCert creates a signed certificate using the given CA certificate and key
func NewSignedCert(cfg Config, key crypto.Signer, caCert *x509.Certificate, caKey crypto.Signer) (*x509.Certificate, error) {
    serial, err := rand.Int(rand.Reader, new(big.Int).SetInt64(math.MaxInt64))
    if len(cfg.CommonName) == 0 {
        return nil, errors.New("must specify a CommonName")
    }
    if len(cfg.Usages) == 0 {
        return nil, errors.New("must specify at least one ExtKeyUsage")
    }

    certTmpl := x509.Certificate{
        Subject: pkix.Name{
            CommonName:   cfg.CommonName,
            Organization: cfg.Organization,
        },
        DNSNames:     cfg.AltNames.DNSNames,
        IPAddresses:  cfg.AltNames.IPs,
        SerialNumber: serial,
        NotBefore:    caCert.NotBefore,
        NotAfter:     time.Now().Add(duration365d * cfg.Year).UTC(),
        KeyUsage:     x509.KeyUsageKeyEncipherment | x509.KeyUsageDigitalSignature,
        ExtKeyUsage:  cfg.Usages,
    }
    certDERBytes, err := x509.CreateCertificate(rand.Reader, &certTmpl, caCert, key.Public(), caKey)
    return x509.ParseCertificate(certDERBytes)
}Copy the code

All certificates in Kubernetes

Root Certificate List

var caList = []Config{
    {
        Path:         BasePath,
        BaseName:     "ca",
        CommonName:   "kubernetes",
        Organization: nil,
        Year:         100,
        AltNames:     AltNames{},
        Usages:       nil,
    },
    {
        Path:         BasePath,
        BaseName:     "front-proxy-ca",
        CommonName:   "front-proxy-ca",
        Organization: nil,
        Year:         100,
        AltNames:     AltNames{},
        Usages:       nil,
    },
    {
        Path:         EtcdBasePath,
        BaseName:     "ca",
        CommonName:   "etcd-ca",
        Organization: nil,
        Year:         100,
        AltNames:     AltNames{},
        Usages:       nil,
    },
}Copy the code

List of other signing certificates

var certList = []Config{ { Path: BasePath, BaseName: "apiserver", CAName: "kubernetes", CommonName: "Kube-apiserver ", Organization: nil, Year: 100, AltNames: AltNames{// add DNSNames to the server IP user custom domain name: []string{ "apiserver.cluster.local", "localhost", "master", "kubernetes", "kubernetes.default", "Kubernetes. Default. SVC,"}, IPs: [] net. IP {127,0,0,1} {and},}, Usages: [] x509. ExtKeyUsage {x509. ExtKeyUsageServerAuth}, / / purpose is server-side validation}, {Path: BasePath, the BaseName: "apiserver-kubelet-client", CAName: "kubernetes", CommonName: "kube-apiserver-kubelet-client", Organization: []string{"system:masters"}, Year: 100, AltNames: AltNames{}, Usages: []x509.ExtKeyUsage{x509.ExtKeyUsageClientAuth}, }, { Path: BasePath, BaseName: "front-proxy-client", CAName: "front-proxy-ca", CommonName: "front-proxy-client", Organization: nil, Year: 100, AltNames: AltNames{}, Usages: []x509.ExtKeyUsage{x509.ExtKeyUsageClientAuth}, }, { Path: BasePath, BaseName: "apiserver-etcd-client", CAName: "etcd-ca", CommonName: "kube-apiserver-etcd-client", Organization: []string{"system:masters"}, Year: 100, AltNames: AltNames{}, Usages: []x509.ExtKeyUsage{x509.ExtKeyUsageClientAuth}, }, { Path: EtcdBasePath, BaseName: "Server ", CAName: "etcd-ca", CommonName: "etcd", // kubeadm etcd server certificate common name: Nil, Year: 100, AltNames: AltNames{}, // When called, need to add the node name, node IP, etc. to lead lead: []x509.ExtKeyUsage{x509.ExtKeyUsageServerAuth, x509.ExtKeyUsageClientAuth}, }, { Path: EtcdBasePath, BaseName: "Peer ", CAName: "etcd-ca", CommonName: "etcd-peer", Organization: nil, Year: 100, AltNames: AltNames{}, // same as etcd Server lead lead: []x509.ExtKeyUsage{x509.ExtKeyUsageServerAuth, x509.ExtKeyUsageClientAuth}, }, { Path: EtcdBasePath, BaseName: "healthcheck-client", CAName: "etcd-ca", CommonName: "kube-etcd-healthcheck-client", Organization: []string{"system:masters"}, Year: 100, AltNames: AltNames{}, Usages: []x509.ExtKeyUsage{x509.ExtKeyUsageClientAuth}, }, }Copy the code

It is important to note that the server verification certificate is installed with the IP and domain name, and etCD commonName is also set to node name.

Check the certificate information generated at the end:

apiserver:

[root@iZ2ze4ry74x8bh3cweeg69Z pki]# openssl x509 -in /etc/kubernetes/pki/apiserver.crt -text -noout
Certificate:
...
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN=kubernetes
        Validity
            Not Before: Mar 31 09:18:06 2020 GMT
            Not After : Mar  8 09:18:06 2119 GMT
        Subject: CN=kube-apiserver
...
        X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment
            X509v3 Extended Key Usage: 
                TLS Web Server Authentication
            X509v3 Subject Alternative Name: 
                DNS:iz2ze4ry74x8bh3cweeg69z, DNS:kubernetes, DNS:kubernetes.default, DNS:kubernetes.default.svc, DNS:kubernetes.default.svc.cluster.local, DNS:apiserver.cluster.local, DNS:apiserver.cluster.local, IP Address:10.96.0.1, IP Address:172.16.9.192, IP Address:127.0.0.1, IP Address:172.16.9.192, IP Address:172.16.9.193, IP Address:172.16.9.194, IP Address:10.103.97.2
    Signature Algorithm: sha256WithRSAEncryptionCopy the code

etcd server:

[root@iZ2ze4ry74x8bh3cweeg69Z pki]# openssl x509 -in /etc/kubernetes/pki/etcd/server.crt -text -noout Certificate: Data: Version: 3 (0x2) Serial Number: 1930981199811083392 (0x1acc392ba2b27c80) Signature Algorithm: sha256WithRSAEncryption Issuer: CN=etcd-ca Validity Not Before: Mar 31 09:18:07 2020 GMT Not After : Mar 8 09:18:07 2119 GMT Subject: CN=iz2ze4ry74x8bh3cweeg69z ... X509v3 Extended Key Usage: TLS Web Server Authentication, TLS Web Client Authentication X509v3 Subject Alternative Name: DNS: iz2ze4ry74x8bh3cweeg69z, DNS: localhost, IP Address: 172.16.9.192, IP Address: 127.0.0.1, IP Address:0:0:0:0:0:0:0:1 Signature Algorithm: sha256WithRSAEncryptionCopy the code

Generate the user certificate and kubeconfig

Now there is an intern Fanux coming to the company, and he also wants to use K8S, but he is not sure to give admin’s Kubeconfig to him, then how to do? With that in mind, how do you allocate a separate Kubeconfig for fanux

1. Load the root certificate and private key from disk

2. Generate a certificate for the fanux user. The common name is fanux

3. Pem format

4. Write kubeconfig, write disk

   func GenerateKubeconfig(conf Config) error{ certs, err := cert.CertsFromFile(conf.CACrtFile) caCert := certs[0] cert := EncodeCertPEM(caCert) caKey,err := TryLoadKeyFromDisk(conf.cakeyFile) TryLoadKeyFromDisk(conf.cakeyFile) Can be multiple clientCert clientKey, err: = NewCertAndKey (caCert, caKey, conf. The User, the conf. Groups, the conf. DNSNames, conf. IPAddresses) encodedClientKey,err := keyutil.MarshalPrivateKeyToPEM(clientKey) encodedClientCert := EncodeCertPEM(clientCert) // Config := &api. config {Clusters: map[string]*api.Cluster{conf.clusterName: {Server: The conf. Apiserver, / / cluster address Such as https://apiserver.cluster.local:6443 CertificateAuthorityData: Contexts: map[string]*api.Context{CTX: // root certificate for HTTPS},}, Contexts: map[string]*api.Context{CTX: {// Triplet information, User name fanux, cluster name above, and namespace: Map [string]* api.authinfo {// alter user (); User:&api.AuthInfo{ClientCertificateData: encodedClientCert, // pem ClientKeyData: EncodedClientKey // private key in pem format},}, } err = clientcmd.writetofile (*config, conf.output) return nil} err = clientcmd.writetofile (*config, conf.output) return nil}Copy the code

The user certificate and private key are generated, and the user is fanux and the group is the user group, just like the signature certificate:

func NewCertAndKey(caCert *x509.Certificate, caKey crypto.Signer, user string, groups []string, DNSNames []string,IPAddresses []net.IP) (*x509.Certificate, crypto.Signer, error) {
    key,err := rsa.GenerateKey(rand.Reader, 2048)
    serial, err := rand.Int(rand.Reader, new(big.Int).SetInt64(math.MaxInt64))

    certTmpl := x509.Certificate{
        Subject: pkix.Name{
            CommonName:   user,
            Organization: groups,
        },
        DNSNames:     DNSNames,
        IPAddresses:  IPAddresses,
        SerialNumber: serial,
        NotBefore:    caCert.NotBefore,
        NotAfter:     time.Now().Add(time.Hour * 24 * 365 * 99).UTC(),
        KeyUsage:     x509.KeyUsageKeyEncipherment | x509.KeyUsageDigitalSignature,
        ExtKeyUsage:  []x509.ExtKeyUsage{x509.ExtKeyUsageClientAuth},
    }
    certDERBytes, err := x509.CreateCertificate(rand.Reader, &certTmpl, caCert, key.Public(), caKey)
    cert,err := x509.ParseCertificate(certDERBytes)
    return cert,key,nil
}Copy the code

Then the friend’s Kubeconfig is generated without any permissions:

kubectl --kubeconfig ./kube/config get pod
Error from server (Forbidden): pods is forbidden: User "fanux" cannot list resource "pods" in API group ...Copy the code

Finally play RBAC can be, here is directly bound to an administrator rights

kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: user-admin-test
subjects:
- kind: User
  name: "fanux" # Name is case sensitive
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: ClusterRole
  name: cluster-admin  # using admin role
  apiGroup: rbac.authorization.k8s.ioCopy the code

conclusion

Certificate and K8S authentication principles can be very useful when installing clusters and developing multi-tenant container platforms, and hopefully this article will give you a thorough understanding of the whole.

` ` `

This article is published by OpenWrite!