background

Etcd is a distributed key-value data storage service written by Golang to store configuration data. It can also be used for distributed locks, distributed queues, message publishing and subscription, service registration and discovery, and more.

An odd number of nodes are deployed to facilitate master selection. Data consistency between nodes is maintained through RAFT protocol. K8s uses ETCD as its database, mainly kube-Apiserver components and ETCD data communication.

In order to facilitate in-depth learning of ETCD, a local ETCD cluster with certificate authentication is first deployed to facilitate debugging and development.

steps

(1) Download etCD and certificate tool CFSSL

# etcdctl is an etcd CLI tool. Convenient and interactive wget etcd unzip at https://github.com/etcd-io/etcd/releases/download/v3.4.10/etcd-v3.4.10-darwin-amd64.zip Etcd -v3.4.10- Darwin -amd64. Zip mv etcd-v3.4.10- Darwin -amd64/etcd /usr/local/bin/etcd mv etcd-v3.4.10- Darwin -amd64/etcdctl /usr/local/bin/etcdctl # brew install etcd # CFSSL curl -o CFSSL https://pkg.cfssl.org/R1.2/cfssl_darwin-amd64 curl -o cfssljson chmod at https://pkg.cfssl.org/R1.2/cfssljson_darwin-amd64 +x CFSSL cfssljson sudo mv CFSSL cfssljson /usr/local/bin/ # https://pkg.cfssl.org/R1.2/cfssl_linux-amd64 wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64 wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64 chmod + x cfssl_linux amd64 cfssljson_linux - amd64 cfssl-certinfo_linux-amd64 mv cfssl_linux-amd64 /usr/local/bin/cfssl mv cfssljson_linux-amd64 /usr/local/bin/cfssljson mv cfssl-certinfo_linux-amd64 /usr/bin/cfssl-certinfoCopy the code

(2) Generate etCD server and client certificates

Run the root certificate ca.sh script:

# Docs: https://kubernetes.io/zh/docs/tasks/tls/managing-tls-in-a-cluster/

# Certificate Authority
## cfssl: https://github.com/cloudflare/cfssl
cat > ca-config.json <<EOF
{
  "signing": {
    "default": {
      "expiry": "8760h"
    },
    "profiles": {
      "kubernetes": {
        "usages": ["signing", "key encipherment", "server auth", "client auth"],
        "expiry": "8760h"
      }
    }
  }
}
EOF

####
# CN: Common Name
# https://blog.cloudflare.com/introducing-cfssl/
####
cat > ca-csr.json <<EOF
{
  "CN": "Kubernetes",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "L": "BeiJing",
      "O": "Kubernetes",
      "OU": "CA",
      "ST": "BeiJing"
    }
  ]
}
EOF

cfssl gencert -initca ca-csr.json | cfssljson -bare ca
Copy the code

Run the server certificate script etcd-server.sh:

cat > etcd-csr.json <<EOF { "CN": "etcd", "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "L": "BeiJing", "O": "system:masters", "OU": "etcd", "ST": "BeiJing" } ] } EOF cfssl gencert \ -ca=./ca.pem \ -ca-key=./ca-key.pem \ -config=./ca-config.json \ - the hostname = 127.0.0.1, kubernetes. Default \ - profile = kubernetes etcd - CSR. Json | cfssljson - bare etcd # - > etcd - key. Pem, etcd.pemCopy the code

Run the client certificate script etcd-client.sh:

cat > etcd-client-csr.json <<EOF { "CN": "etcd-client", "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "L": "BeiJing", "O": "system:masters", "OU": "etcd", "ST": "BeiJing" } ] } EOF cfssl gencert \ -ca=./ca.pem \ -ca-key=./ca-key.pem \ -config=./ca-config.json \ - the hostname = 127.0.0.1, kubernetes. Default \ - profile = kubernetes \ etcd - the client - CSR. Json | cfssljson - bare etcd - client # - > etcd-client-key.pem, etcd-client.pemCopy the code

Finally, the root certificate, server certificate for etCD startup, and client certificate for etCDCTL interaction are generated:

The client certificate can also reuse the server certificate directly, so there is no need to regenerate a client certificate. To be more secure, generate a client certificate.

(3) Start an ETCD cluster

Inside the Makefile, write:

PWD := $(shell pwd) etcd1-auth: etcd \ --name=infra1 \ --client-cert-auth=true \ --cert-file=$(PWD)/tls/etcd.pem \ --key-file=$(PWD)/tls/etcd-key.pem \ --peer-client-cert-auth=true \ --peer-cert-file=$(PWD)/tls/etcd.pem \ --peer-key-file=$(PWD)/tls/etcd-key.pem \ --trusted-ca-file=$(PWD)/tls/ca.pem \ --peer-trusted-ca-file=$(PWD)/tls/ca.pem \ - initial - advertise - peer - urls \ = https://127.0.0.1:42380 - listen - peer - url = https://127.0.0.1:42380 \ - listen - the client - urls \ = https://127.0.0.1:42379 - advertise - the client - url = https://127.0.0.1:42379 \ --initial-cluster-token=etcd-cluster-0 \ --initial-cluster 'infra1 = https://127.0.0.1:42380, infra2 = https://127.0.0.1:52380, infra3 = https://127.0.0.1:62380' \ --initial-cluster-state=new \ --data-dir=$(PWD)/infra1.etcd \ --logger=zap \ --log-outputs=stderr etcd2-auth: etcd \ --name=infra2 \ --client-cert-auth=true \ --cert-file=$(PWD)/tls/etcd.pem \ --key-file=$(PWD)/tls/etcd-key.pem \ --peer-client-cert-auth=true \ --peer-cert-file=$(PWD)/tls/etcd.pem \ --peer-key-file=$(PWD)/tls/etcd-key.pem \ --trusted-ca-file=$(PWD)/tls/ca.pem \ --peer-trusted-ca-file=$(PWD)/tls/ca.pem \ - initial - advertise - peer - urls \ = https://127.0.0.1:52380 - listen - peer - url = https://127.0.0.1:52380 \ - listen - the client - urls \ = https://127.0.0.1:52379 - advertise - the client - url = https://127.0.0.1:52379 \ --initial-cluster-token=etcd-cluster-0 \ --initial-cluster 'infra1 = https://127.0.0.1:42380, infra2 = https://127.0.0.1:52380, infra3 = https://127.0.0.1:62380' \ --initial-cluster-state=new \ --data-dir=$(PWD)/infra2.etcd \ --logger=zap \ --log-outputs=stderr etcd3-auth: etcd \ --name=infra3 \ --client-cert-auth=true \ --cert-file=$(PWD)/tls/etcd.pem \ --key-file=$(PWD)/tls/etcd-key.pem \ --peer-client-cert-auth=true \ --peer-cert-file=$(PWD)/tls/etcd.pem \ --peer-key-file=$(PWD)/tls/etcd-key.pem \ --trusted-ca-file=$(PWD)/tls/ca.pem \ --peer-trusted-ca-file=$(PWD)/tls/ca.pem \ - initial - advertise - peer - urls \ = https://127.0.0.1:62380 - listen - peer - url = https://127.0.0.1:62380 \ - listen - the client - urls \ = https://127.0.0.1:62379 - advertise - the client - url = https://127.0.0.1:62379 \ --initial-cluster-token=etcd-cluster-0 \ --initial-cluster 'infra1 = https://127.0.0.1:42380, infra2 = https://127.0.0.1:52380, infra3 = https://127.0.0.1:62380' \ --initial-cluster-state=new \ --data-dir=$(PWD)/infra3.etcd \ --logger=zap \ --log-outputs=stderr list: ETCDCTL_API=3 etcdctl member list \ --write-out=table \ --endpoints=https://127.0.0.1:42379 \ --cacert $(PWD)/ TLS /ca.pem  \ --cert $(PWD)/tls/etcd-client.pem \ --key $(PWD)/tls/etcd-client-key.pemCopy the code

Make etcd1-auth, make etcd2-auth, and make etcd3-auth on three terminals, respectively, to build a 3-node ETcd cluster, And the etCD1 / ETCD2 / ETCD3 listening port on the client is 42379/52379/62379 respectively. And can read and write data normally:

In addition, ETCD supports data persistence. K-v data is stored in B +tree. Etcd data is stored in the current directory, including SNAP and WAL files. A SNAP file is a snapshot file that stores full data. Wal is incremental data. When the number of wal keys reaches a certain number, a snapshot is automatically generated. This design improves performance and reduces memory stress:

In conclusion, etCD as a distributed KV storage service is a very good tool with good compact performance and easy maintenance.

reference

Etcd. IO/docs/v3.4.0…