K8s Series 3 - How to configure etCD HTTPS certificate?

In K8s, Kube-Apiserver uses ETCD for persistent storage of REST Object resources. This article introduces how to configure and generate self-signed HTTPS certificate, build ETCD cluster for Apiserver to use, and attach related pit records.

1. Install CFSSL

CD/data/work wget https://github.com/cloudflare/cfssl/releases/download/v1.6.0/cfssl_1.6.0_linux_amd64 - O CFSSL wget https://github.com/cloudflare/cfssl/releases/download/v1.6.0/cfssljson_1.6.0_linux_amd64 - O cfssljson wget https://github.com/cloudflare/cfssl/releases/download/v1.6.0/cfssl-certinfo_1.6.0_linux_amd64 - O CFSSL - certinfo chmod + x  cfssl* mv cfssl* /usr/local/bin/ chmod +x cfssl* mv cfssl_linux-amd64 /usr/local/bin/cfssl mv cfssljson_linux-amd64 /usr/local/bin/cfssljson mv cfssl-certinfo_linux-amd64 /usr/local/bin/cfssl-certinfoCopy the code

2. Create a CA certificate

cat > ca-csr.json <<EOF { "CN": "etcd-ca", "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "ST": "Beijing", "L": "Beijing", "O": "etcd-ca", "OU": "etcd-ca" } ], "ca": { "expiry": "87600 h"}} EOF CFSSL gencert - initca ca - CSR. Json | cfssljson - bare ca = > generate: ca - key. Pem, ca. CSR, ca. PemCopy the code

3. Configure the CA certificate policy

cat > ca-config.json <<EOF
{
  "signing": {
      "default": {
          "expiry": "87600h"
        },
      "profiles": {
          "etcd-ca": {
              "usages": [
                  "signing",
                  "key encipherment",
                  "server auth",
                  "client auth"
              ],
              "expiry": "87600h"
          }
      }
  }
}
EOF
Copy the code

4. Configure the ETCD to request a CSR

cat > etcd-csr.json <<EOF { "CN": "etcd", "hosts": [" 127.0.0.1 ", "etcd0-0. Etcd", "etcd1-0. Etcd", "etcd2-0. Etcd]", "key" : {" algo ":" rsa ", "size" : 2048}, "names" : [{ "C": "CN", "ST": "Beijing", "L": "Beijing", "O": "etcd", "OU": "etcd" }] } EOFCopy the code

5. Generate the ETCD certificate

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=etcd-ca etcd-csr.json | cfssljson -bare etcd Pem, etcd. CSR, etcd.pemCopy the code

6. Create an etcd cluster

Yaml file: github.com/k8s-club/et…

kubectl apply -f etcd-cluster.yaml
Copy the code

7. View DNS resolution

Dnsutils installation: kubernetes. IO/docs/tasks /…

Kubectl exec it-n etcd dnsutils -- nslookup etcd Server: 9.165.x.x Address: 9.165.x.x#53 Name: kubectl exec it-n etcd dnsutils -- nslookup etcd Server: 9.165.x.x#53 Name: Etcd. Etcd. SVC. Cluster. The local Address: 9.165. 7.0.x.x Name: etcd. Etcd. SVC. Cluster. The local Address: 9.165. 7.0.x.x Name: Etcd. Etcd. SVC. Cluster. The local Address: 9.165. 7.0.x.xCopy the code

8. Check the ETCD cluster status

kubectl exec -it -n etcd etcd0-0 -- sh /usr/local/bin/etcdctl --write-out=table --cacert=/etc/etcd/ssl/ca.pem --cert=/etc/etcd/ssl/etcd.pem --key=/etc/etcd/ssl/etcd-key.pem --endpoints=https://etcd0-0.etcd:2379,https://etcd1-0.etcd:2379,https://etcd2-0.etcd:2379 endpoint health +---------------------------+--------+-------------+-------+ | ENDPOINT | HEALTH | TOOK | ERROR | + -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- + + -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- - + -- -- -- -- -- -- -- + | | https://etcd0-0.etcd:2379 true 13.551982 ms | | | | https://etcd1-0.etcd:2379 | true 13.540498 ms | | | | | https://etcd2-0.etcd:2379 true 23.119639 ms | | | +---------------------------+--------+-------------+-------+ /usr/local/bin/etcdctl --write-out=table --cacert=/etc/etcd/ssl/ca.pem --cert=/etc/etcd/ssl/etcd.pem --key=/etc/etcd/ssl/etcd-key.pem --endpoints=https://etcd0-0.etcd:2379,https://etcd1-0.etcd:2379,https://etcd2-0.etcd:2379 endpoint status +--------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--- -----------------+--------+ | ENDPOINT | ID | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT  APPLIED INDEX | ERRORS | +--------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- - + -- -- -- -- -- -- -- -- + | | http://etcd0-0.etcd:2379 4 dde210279eea33a | 3.4.13 20 kB | | true | false | 2 | | | 9 9 20669865 d12a473b http://etcd1-0.etcd:2379 | | | | 3.4.13 | 20 kB | | false false | 2 | 9 9 | | | | F17922d1ed63113 http://etcd2-0.etcd:2379 | 3 | 3.4.13 20 kB | | | false false | 2 | | 9 9 | | +--------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--- -----------------+--------+Copy the code

9. Verify etCD read and write

kubectl exec -it -n etcd etcd0-0 -- sh /usr/local/bin/etcdctl --cacert=/etc/etcd/ssl/ca.pem --cert=/etc/etcd/ssl/etcd.pem --key=/etc/etcd/ssl/etcd-key.pem put hello world OK /usr/local/bin/etcdctl Pem --cert= /etc/etcd-ssl/etcd-key. pem --key= /etc/etcd-ssl/etcd-key. pem get hello hello world View all information keys: /usr/local/bin/etcdctl --cacert=/etc/etcd/ssl/ca.pem --cert=/etc/etcd/ssl/etcd.pem --key=/etc/etcd/ssl/etcd-key.pem get "" --keys-only --prefix hello /usr/local/bin/etcdctl --cacert=/etc/etcd/ssl/ca.pem --cert=/etc/etcd/ssl/etcd.pem --key=/etc/etcd/ssl/etcd-key.pem get "" --prefix hello worldCopy the code

10. Configure apiserver to request CSR

cat > apiserver-csr.json <<EOF
{
  "CN": "apiserver",
  "hosts": [
    "*.etcd"
  ],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [{
    "C": "CN",
    "ST": "Beijing",
    "L": "Beijing",
    "O": "apiserver",
    "OU": "apiserver"
  }]
}
EOF
Copy the code

11. Generate the Apiserver certificate

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=etcd-ca apiserver-csr.json | cfssljson -bare Pem, apiserver. CSR, and apiserver.pem are generatedCopy the code

12. Create the extension – apiserver

Yaml: Use ConfigMap to mount the *. Pem certificate to Apiserver

containers:
  - image: xxxxx:latest
    args:
    - --etcd-servers=https://etcd0-0.etcd:2379
    - --etcd-cafile=/etc/kubernetes/certs/kube-apiserver-etcd-ca.crt
    - --etcd-certfile=/etc/kubernetes/certs/kube-apiserver-etcd-client.crt
    - --etcd-keyfile=/etc/kubernetes/certs/kube-apiserver-etcd-client.key
Copy the code

kubectl apply -f apiserver.yaml
Copy the code

13. Pothole record

13.1 The certificate hosts is incorrect

log: etcd0-0: {" level ":" warn ", "ts" : "the 2021-08-19 T11:55:07. 755 z", the "caller" : "embed/config_logging. Go: 279", "MSG" : "rejected Connection ", "remote - addr" : "127.0.0.1:41226" and "server - the name" : ""," error ":" the TLS. first record does not look like a TLS handshake"} etcd1-0: {" level ", "info", "ts" : "the 2021-08-19 T11: bloweth. 830 z", the "caller" : "embed/serve. Go: 191", "MSG" : "serving client traffic securely","address":"[::]:2379"} {" level ", "info", "ts" : "the 2021-08-19 T11: bloweth. 838 z", the "caller" : "etcdserver/server. Go: 716", "MSG" : "the initialized peer connections;  fast-forwarding election Ticks ", "30 dd90df9a304e97 local - member - id" : ""," forward - ticks ": 18," forward - duration ":" 4.5 ", "s election - ticks" : 20, "election - t imeout":"5s","active-remote-members":2} {" level ", "info", "ts" : "the 2021-08-19 T11: bloweth. 867 z", the "caller" : "the membership/cluster. Go: 558", "MSG" : "set initial cluster Version ", "80 c7f1f6c2848777 cluster - id" : ""," local - member - id ":" 30 dd90df9a304e97 cluster - ", "version" : "3.4"} {" level ", "info", "ts" : "the 2021-08-19 T11: bloweth. 867 z", the "caller" : "API/capability. Go: 76", "MSG" : "enabled" capabilities for Version ", "cluster - version" : "3.4"} etcd2-0: {" level ":" warn ", "ts" : "the 2021-08-19 T11:54:17. 782 z", the "caller" : "embed/config_logging. Go: 270", "MSG" : "rejected Connection ", "remote - addr" : "9.165 7.0.x.x: 40180" and "server - the name" : "etcd2-0. Etcd", "IP addresses -" : [" 0.0.0.0 ", "127.0.0.1"], "DNS - nam es":["etcd0-0.etcd","etcd1-0.etcd","etcd2-0.etcd"],"error":"tls: \"9.165.x.x\" does not match any of DNSNames [\"etcd0-0.etcd\" \"etcd1-0.etcd\" \"etcd2-0.etcd\"] (lookup etcd1-0.etcd On 9.165.x.x:53: no such host)"}Copy the code

Solution: Reconfigure the correct hosts domain name

13.2 Certificate hosts Configuration Pits

"Hosts" : [" 127.0.0.1 ", "etcd0-0. Etcd", "*. Etcd" / / allow * generic domain name, but can't be empty "or" *].Copy the code

13.3 DNS Settings

You are advised to set *.xxx.ns. SVC, so that no revisa is required after capacity expansion

Reference: kubernetes. IO/docs/concep…

Go code reference is as follows:

func genEtcdWildcardDnsName(namespace, serviceName string) []string {
	return []string{
		fmt.Sprintf("%s.%s.%s", serviceName, namespace, "svc"),
		fmt.Sprintf("*.%s.%s.%s", serviceName, namespace, "svc"),
		fmt.Sprintf("%s.%s.%s", serviceName, namespace, DnsBase),
		fmt.Sprintf("*.%s.%s.%s", serviceName, namespace, DnsBase),
	}
}
Copy the code

13.4 The leader/ Follower is successfully established, but an access error occurs

# /usr/local/bin/etcdctl put hello world {" level ":" warn ", "ts" : "the 2021-08-19 T12: thus says. 200 z", the "caller" : "clientv3 / retry_interceptor. Go: 62", "MSG" : "retrying of unary invoker Failed ", "target" : "the endpoint: / / client - 05 ed1825 - e70f - 492 - a - af94-03 c633d0affc / 127.0.0.1:2379", "attempt" : 0, "error" : "the RPC error:  code = DeadlineExceeded desc = latest balancer error: all SubConns are in TransientFailure, latest connection error: connection closed"} Error: context deadline exceededCopy the code

Solution: Etcdctl requires certificate access

/usr/local/bin/etcdctl --cacert=/etc/etcd/ssl/ca.pem --cert=/etc/etcd/ssl/etcd.pem --key=/etc/etcd/ssl/etcd-key.pem put hello world
Copy the code

13.5 You cannot switch between HTTP and HTTPS

Create a cluster using HTTP and then create a cluster using HTTPS.

tls: first record does not look like a TLS handshake
Copy the code

If the cluster is set up successfully, the connection protocol (HTTP/HTTPS) is written to the ETCD store and cannot be changed.

Solution: If you really need to switch the protocol, try the following method

Delete data: After deleting data, re-establish the cluster
Do not delete data: You can use snapshot & Restore to perform snapshot and restore operations

13.6 Can apiserver directly use the ETCD certificate generated in Step 5?

After verification, it is possible to use etCD certificates directly, but this is not recommended for production.

In production, you are advised to independently generate certificates for apiserver(or other applications). You can flexibly configure certificates using a generic domain name (*.xx.xx) and different expiration times to facilitate cluster management.

mo4tech.com (Moment For Technology) is a global community with thousands techies from across the global hang out!Passionate technologists, be it gadget freaks, tech enthusiasts, coders, technopreneurs, or CIOs, you would find them all here.

K8s Series 3 – How to configure etCD HTTPS certificate?

1. Install CFSSL

2. Create a CA certificate

3. Configure the CA certificate policy

4. Configure the ETCD to request a CSR

5. Generate the ETCD certificate

6. Create an etcd cluster

7. View DNS resolution

8. Check the ETCD cluster status

9. Verify etCD read and write

10. Configure apiserver to request CSR

11. Generate the Apiserver certificate

12. Create the extension – apiserver

13. Pothole record

13.1 The certificate hosts is incorrect

13.2 Certificate hosts Configuration Pits

13.3 DNS Settings

13.4 The leader/ Follower is successfully established, but an access error occurs

13.5 You cannot switch between HTTP and HTTPS

13.6 Can apiserver directly use the ETCD certificate generated in Step 5?

K8s Series 3 – How to configure etCD HTTPS certificate?

1. Install CFSSL

2. Create a CA certificate

3. Configure the CA certificate policy

4. Configure the ETCD to request a CSR

5. Generate the ETCD certificate

6. Create an etcd cluster

7. View DNS resolution

8. Check the ETCD cluster status

9. Verify etCD read and write

10. Configure apiserver to request CSR

11. Generate the Apiserver certificate

12. Create the extension – apiserver

13. Pothole record

13.1 The certificate hosts is incorrect

13.2 Certificate hosts Configuration Pits

13.3 DNS Settings

13.4 The leader/ Follower is successfully established, but an access error occurs

13.5 You cannot switch between HTTP and HTTPS

13.6 Can apiserver directly use the ETCD certificate generated in Step 5?

Related Posts

Circular linked lists of data structures

Memory model for the JVM

Solution to Bug – Master/slave switchover “failed “?