Home page address

The problem background

Production environment Kuberneties cluster was built on November 7, 2019, using Kubeadm combined with domestic mirror, refer to the article

Kubernetes cluster setup

On November 8, 2020, all cluster services in the production environment were found to be invalid and unavailable

The process

1. Check whether the service is started

[root@k8smaster1 ~]$ docker ps
CONTAINER ID        IMAGE                                               COMMAND                  CREATED             STATUS              PORTS               NAMES
65dff767f422        b95b1efa0436                                        "kube-controller-m..."20 hours ago Up 20 hours k8s_kube-controller-manager_kube-controller-manager-k8smaster1_kube-system_7a524400c16990c1a69c237c9da4c7f2_11 1 b26acecbcea registry.cn-hangzhou.aliyuncs.com/imooc/pause:3.1"/pause"                 20 hours ago        Up 20 hours                             k8s_POD_kube-controller-manager-k8smaster1_kube-system_7a524400c16990c1a69c237c9da4c7f2_1
9c6449df0a18        00638a24688b                                        "kube-scheduler --..."20 hours ago Up 20 hours k8s_kube-scheduler_kube-scheduler-k8smaster1_kube-system_baebad22afce708e52c6d3f886ff9424_10 028 a1177cc9b registry.cn-hangzhou.aliyuncs.com/imooc/pause:3.1"/pause"                 20 hours ago        Up 20 hours                             k8s_POD_kube-scheduler-k8smaster1_kube-system_baebad22afce708e52c6d3f886ff9424_1
6c6bdc2c2644        registry.cn-hangzhou.aliyuncs.com/imooc/pause:3.1   "/pause"                 20 hours ago        Up 20 hours                             k8s_POD_kube-apiserver-k8smaster1_kube-system_0e954d211e160f48f5ca9fd42d295c10_1
fe4a12daa7d7        registry.cn-hangzhou.aliyuncs.com/imooc/pause:3.1   "/pause"                 20 hours ago        Up 20 hours                             k8s_POD_etcd-k8smaster1_kube-system_6119961323d801d05a7dd23e429cda3f_1
Copy the code

The container service is normal, but the Kubelet service reported an error message, indicating that apiserver cannot be connected

[root@k8smaster1 ~]$systemctl status kubelet -l ● kubelet.service - kubelet: The Kubernetes Node Agent Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; Vendor preset: disabled) Drop - In: / usr/lib/systemd/system/kubelet. Service. D └ ─ 10 - kubeadm. Conf Active: Active (running) since 日 2020-11-08 22:16:07 CST; Line 20 h Docs: https://kubernetes.io/docs/ Main PID: 15313 (kubelet) Tasks: 16 Memory: 68.4 M CGroup: / system. Slice/kubelet service └ ─ 15313 / usr/bin/kubelet - the bootstrap - kubeconfig = / etc/kubernetes/bootstrap - kubelet. Conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --cgroup-driver=systemd - network - the plugin = the cni - pod-infra-container-image=registry.cn-hangzhou.aliyuncs.com/imooc/pause:3.1 November 09 18:43:06 K8smaster1 kubelet [15313] : E1109 18:43:06. 967416 15313 reflector. Go: 126] k8s. IO/kubernetes/PKG/kubelet/kubelet go: 442: Failed to list * v1. Service: Get https://192.168.1.14:6443/api/v1/services?limit=500&resourceVersion=0: Dial TCP 192.168.1.14:6443: connect: K8smaster1 kubelet[15313]: E1109 18:43:07.009411 15313 kubelet.go:2244] kubelet.go:2244"k8smaster1"Not found Nov 09 18:43:07 k8sMaster1 kubelet[15313]: E1109 18:43:07.109665 15313 kubelet.go:2244] node not found Nov 09 18:43:07 k8smaster1 kubelet[15313]: E1109 18:43:07.109665 15313 kubelet.go:2244] node"k8smaster1"Not found 11月 09 18:43:07 k8smaster1 kubelet[15313]: I1109 18:43:07.165755 15313 kubelet_node_status.go:283] Setting node annotation toenableVolume Controller Attach/Detach 11月 09 18:43:07 K8sMaster1 kubelet[15313]: E1109 18:43:07. 167106 15313 reflector. Go: 126] k8s. IO/client - go/informers/factory go: 133: Failed to list * v1beta1. CSIDriver: Get https://192.168.1.14:6443/apis/storage.k8s.io/v1beta1/csidrivers?limit=500&resourceVersion=0: Dial TCP 192.168.1.14:6443: connect: K8smaster1 kubelet[15313]: I1109 18:43:07.169000 15313 kubelet_node_status.go:72] Attempting to register Node K8sMaster1 11月 09 18:43:07 k8smaster1 Kubelet [15313]: E1109 18:43:07.209831 15313 kubelet.go:2244"k8smaster1"Not found 11月 09 18:43:07 k8sMaster1 kubelet[15313]: E1109 18:43:07.310013 15313 kubelet.go:2244] node not found 11月 09 18:43:07 k8smaster1 kubelet[15313]: E1109 18:43:07.310013 15313 kubelet.go:2244] node"k8smaster1"Not found 11月 09 18:43:07 k8smaster1 kubelet[15313]: E1109 18:43:07.362049 15313 kubelet_node_status.go:94] Unable to register node"k8smaster1"With API server: Post https://192.168.1.14:6443/api/v1/nodes: dial TCP 192.168.1.14:6443: connect: K8smaster1 kubelet[15313]: E1109 18:43:07.410174 15313 kubelet.go:2244] kubelet.go:2244"k8smaster1" not found
Copy the code

2. Check the APIServer

The Apiserver port is properly connected

[root@k8smaster1 ~]$Telnet 192.168.1.14 6443 Trying 192.168.1.14... Telnet: connect to address 192.168.1.14: Connection refusedCopy the code

Reset the firewall and check again. The connection is normal

[root@k8smaster1 ~]$ iptables -F && iptables -X && iptables -F -t nat && iptables -X -t nat && iptables -P FORWARD ACCEPT [root@k8smaster1 ~]$Telnet 192.168.1.14 6443 Trying 192.168.1.14... Connected to 192.168.1.14.Copy the code

But the service is still inaccessible

3. Restart kubelet

[root@k8smaster1 ~]$ systemctl restart kubelet
[root@k8smaster1 ~]$ docker ps
CONTAINER ID        IMAGE                                               COMMAND                  CREATED             STATUS              PORTS               NAMES
4383926ef6c9        ecf910f40d6e                                        "kube-apiserver --..."   3 seconds ago       Up 2 seconds                            k8s_kube-apiserver_kube-apiserver-k8smaster1_kube-system_0e954d211e160f48f5ca9fd42d295c10_1362
88eb6134fe3c        2c4adeb21b4f                                        "etcd --advertise-..."   3 seconds ago       Up 2 seconds                            k8s_etcd_etcd-k8smaster1_kube-system_6119961323d801d05a7dd23e429cda3f_1768
65dff767f422        b95b1efa0436                                        "kube-controller-m..."20 hours ago Up 20 hours k8s_kube-controller-manager_kube-controller-manager-k8smaster1_kube-system_7a524400c16990c1a69c237c9da4c7f2_11 1 b26acecbcea registry.cn-hangzhou.aliyuncs.com/imooc/pause:3.1"/pause"                 20 hours ago        Up 20 hours                             k8s_POD_kube-controller-manager-k8smaster1_kube-system_7a524400c16990c1a69c237c9da4c7f2_1
9c6449df0a18        00638a24688b                                        "kube-scheduler --..."20 hours ago Up 20 hours k8s_kube-scheduler_kube-scheduler-k8smaster1_kube-system_baebad22afce708e52c6d3f886ff9424_10 028 a1177cc9b registry.cn-hangzhou.aliyuncs.com/imooc/pause:3.1"/pause"                 20 hours ago        Up 20 hours                             k8s_POD_kube-scheduler-k8smaster1_kube-system_baebad22afce708e52c6d3f886ff9424_1
6c6bdc2c2644        registry.cn-hangzhou.aliyuncs.com/imooc/pause:3.1   "/pause"                 20 hours ago        Up 20 hours                             k8s_POD_kube-apiserver-k8smaster1_kube-system_0e954d211e160f48f5ca9fd42d295c10_1
fe4a12daa7d7        registry.cn-hangzhou.aliyuncs.com/imooc/pause:3.1   "/pause"                 20 hours ago        Up 20 hours                             k8s_POD_etcd-k8smaster1_kube-system_6119961323d801d05a7dd23e429cda3f_1
Copy the code

Check kubelet status, still error

[root@k8smaster1 ~]$systemctl status kubelet -l ● kubelet.service - kubelet: The Kubernetes Node Agent Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; Vendor preset: disabled) Drop - In: / usr/lib/systemd/system/kubelet. Service. D └ ─ 10 - kubeadm. Conf Active: Active (running) since a 2020-11-09 18:52:24 CST; 24 s line Docs: https://kubernetes.io/docs/ Main PID: 31759 (kubelet) Tasks: 15 Memory: 30.1 M CGroup: / system. Slice/kubelet service └ ─ 31759 / usr/bin/kubelet - the bootstrap - kubeconfig = / etc/kubernetes/bootstrap - kubelet. Conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --cgroup-driver=systemd - network - the plugin = the cni - pod-infra-container-image=registry.cn-hangzhou.aliyuncs.com/imooc/pause:3.1 November 09 18:52:48 K8smaster1 kubelet[31759]: E1109 18:52:48.063832 31759 Controller. go:115] Failed to ensure node lease exists, will retryin3.2 s, the error: Get https://192.168.1.14:6443/apis/coordination.k8s.io/v1beta1/namespaces/kube-node-lease/leases/k8smaster1? timeout=10s: net/http: request canceledwhile waiting for connection (Client.Timeout exceeded whileAwaiting HEADERS) November 09 18:52:48 K8smaster1 kubelet[31759]: E1109 18:52:48.152723 31759 kubelet.go:2244] node"k8smaster1"Not found 11月 09 18:52:48 k8smaster1 kubelet[31759]: E1109 18:52:48.252964 31759 kubelet.go:2244] node"k8smaster1"Not found 11月 09 18:52:48 k8sMaster1 kubelet[31759]: E1109 18:52:48.353226 31759 kubelet.go:2244] node"k8smaster1"Not found Nov 09 18:52:48 k8smaster1 kubelet[31759]: E1109 18:52:48.453452 31759 kubelet.go:2244] node not found Nov 09 18:52:48 k8smaster1 kubelet[31759]: E1109 18:52:48.453452 31759 kubelet.go:2244] node"k8smaster1"Not found Nov 09 18:52:48 k8smaster1 kubelet[31759]: E1109 18:52:48.553682 31759 kubelet.go:2244] node not found Nov 09 18:52:48 k8smaster1 kubelet[31759]: E1109 18:52:48.553682 31759 kubelet.go:2244] node"k8smaster1"Not found Nov 09 18:52:48 k8smaster1 kubelet[31759]: E1109 18:52:48.653965 31759 kubelet.go:2244] node not found Nov 09 18:52:48 k8smaster1 kubelet[31759]: E1109 18:52:48.653965 31759 kubelet.go:2244] node"k8smaster1"Not found 11月 09 18:52:48 k8sMaster1 kubelet[31759]: E1109 18:52:48.754181 31759 kubelet.go:2244] node"k8smaster1"Not found 11月 09 18:52:48 k8sMaster1 kubelet[31759]: E1109 18:52:48.854401 31759 kubelet.go:2244] node"k8smaster1"Not found 11月 09 18:52:48 k8smaster1 kubelet[31759]: E1109 18:52:48.954576 31759 kubelet.go:2244] node"k8smaster1" not found
Copy the code

4. Check API Server logs

Check the container ID

[root@k8smaster1 ~]$ docker ps -l
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS               NAMES
27a3b7787748        ecf910f40d6e        "kube-apiserver --..."   18 seconds ago      Up 18 seconds                           
Copy the code

See the log

[root@k8smaster1 ~]$ docker logs 27a
Flag --insecure-port has been deprecated, This flag will be removed inI1109 10:55:42.281078 1 Server. Go :559] External host was not specified, Using 192.168.1.15 I1109 10:55:42.281264 1 server.go:146] Version: V1.14.0i1109 10:55:42.759000 1 plugins. Go :158] Loaded 9 Mutating admission controller(s) successfullyinthe following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,De FaultStorageClass, MutatingAdmissionWebhook I1109 10:55:42. 759031 1 plugins. Go: 161] the Loaded validating admission controller(s) successfullyinthe following order: LimitRanger,ServiceAccount,Priority,PersistentVolumeClaimResize,ValidatingAdmissionWebhook,ResourceQuota. E1109 10:55:42.759943 1 Prometheus. Go :138] Failed to register depth metric admission_quota_controller: Duplicate metrics collector registration FAILED to register E1109 10:55:42.759976 1 Prometheus. Go :150 metric admission_quota_controller: Duplicate the metrics collector registration attempted E1109 10:55:42. 760007 1 Prometheus. Go: 162] failed to register latency  metric admission_quota_controller: Duplicate metrics collector registration failed to register CHS E1109 10:55:42.760038 1 Prometheus. Go :174 work_duration metric admission_quota_controller: Duplicate metrics collector registration failed to register CHS E1109 10:55:42.760065 1 Prometheus. Go :189 unfinished_work_seconds metric admission_quota_controller: Duplicate metrics collector Registration failed to register CHS E1109 10:55:42.760086 1 Prometheus. Go :202 longest_running_processor_microseconds metric admission_quota_controller: Duplicate metrics collector registration 语 言 口 语 I1109 10:55:42.760102 1 plugins.go:158] Loaded 9 Mutating admission controller(s) successfullyinthe following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,De FaultStorageClass, MutatingAdmissionWebhook I1109 10:55:42. 760110 1 plugins. Go: 161] the Loaded validating admission controller(s) successfullyinthe following order: LimitRanger,ServiceAccount,Priority,PersistentVolumeClaimResize,ValidatingAdmissionWebhook,ResourceQuota. I1109 10:55:42.763844 1 client. Go :352]""I1109 10:55:42.763866 1 Client.go :352] Scheme""Not registered, fallback to default Scheme I1109 10:55:42.764295 1 ASM_AMd64.s :1337] ccResolverWrapper: Sending new addresses to cc: [{127.0.0.1:2379 0 <nil>}] I1109 10:55:42.764449 1 asM_amd64.s :1337] balancerWrapper: Got UPDATE ADDR FROM Notify: [{127.0.0.1:2379 <nil>}] W1109 10:55:42.771517 1 Clientconn. go:1251] GRPC: Addrconn. createTransport failed to connect to {127.0.0.1:2379 0 <nil>}. Err: Connection error: desc ="transport: authentication handshake failed: x509: certificate has expired or is not yet valid". Reconnecting... I1109 10:55:43.756910 1 Client.go :352] Parsed scheme:""I1109 10:55:43.756939 1 Client.go :352] Scheme""Not registered, fallback to default Scheme I1109 10:55:43.756999 1 ASM_AMd64.s :1337] ccResolverWrapper: Sending new addresses to cc: [{127.0.0.1:2379 0 <nil>}] I1109 10:55:43.757077 1 asM_amd64.s :1337] balancerWrapper: Got UPDATE ADDR from Notify: [{127.0.0.1:2379 <nil>}] W1109 10:55:43.763355 1 Clientconn. go:1251] GRPC: Addrconn. createTransport failed to connect to {127.0.0.1:2379 0 <nil>}. Err: Connection error: desc ="transport: authentication handshake failed: x509: certificate has expired or is not yet valid". Reconnecting... W1109 10:55:43.770837 1 Clientconn. go:1251] GRPC: Addrconn. createTransport failed to connect to {127.0.0.1:2379 0 <nil>}. Err: Connection error: desc ="transport: authentication handshake failed: x509: certificate has expired or is not yet valid". Reconnecting... W1109 10:55:44.764994 1 Clientconn. go:1251] GRPC: Addrconn. createTransport failed to connect to {127.0.0.1:2379 0 <nil>}. Err: Connection error: desc ="transport: authentication handshake failed: x509: certificate has expired or is not yet valid". Reconnecting... W1109 10:55:45.541971 1 Clientconn. go:1251] GRPC: Addrconn. createTransport failed to connect to {127.0.0.1:2379 0 <nil>}. Err: Connection error: desc ="transport: authentication handshake failed: x509: certificate has expired or is not yet valid". Reconnecting... W1109 10:55:46.657328 1 Clientconn.go :1251] GRPC: Addrconn. createTransport failed to connect to {127.0.0.1:2379 0 <nil>}. Err: Connection error: desc ="transport: authentication handshake failed: x509: certificate has expired or is not yet valid". Reconnecting... W1109 10:55:47.938720 1 Clientconn. go:1251] GRPC: Addrconn. createTransport failed to connect to {127.0.0.1:2379 0 <nil>}. Err: Connection error: desc ="transport: authentication handshake failed: x509: certificate has expired or is not yet valid". Reconnecting... W1109 10:55:48.730961 1 Clientconn. go:1251] GRPC: Addrconn. createTransport failed to connect to {127.0.0.1:2379 0 <nil>}. Err: Connection error: desc ="transport: authentication handshake failed: x509: certificate has expired or is not yet valid". Reconnecting... W1109 10:55:51.899665 1 Clientconn. go:1251] GRPC: Addrconn. createTransport failed to connect to {127.0.0.1:2379 0 <nil>}. Err: Connection error: desc ="transport: authentication handshake failed: x509: certificate has expired or is not yet valid". Reconnecting... W1109 10:55:52.961615 1 Clientconn. go:1251] GRPC: Addrconn. createTransport failed to connect to {127.0.0.1:2379 0 <nil>}. Err: Connection error: desc ="transport: authentication handshake failed: x509: certificate has expired or is not yet valid". Reconnecting... W1109 10:55:59.193916 1 Clientconn. go:1251] GRPC: Addrconn. createTransport failed to connect to {127.0.0.1:2379 0 <nil>}. Err: Connection error: desc ="transport: authentication handshake failed: x509: certificate has expired or is not yet valid". Reconnecting... W1109 10:56:00.025333 1 Clientconn. go:1251] GRPC: Addrconn. createTransport failed to connect to {127.0.0.1:2379 0 <nil>}. Err: Connection error: desc ="transport: authentication handshake failed: x509: certificate has expired or is not yet valid". Reconnecting... F1109 10:56:02.764150 1 Storage_decorator. go:57] Unable to create storage backend: Config (& registry / {{[https://127.0.0.1:2379] / etc/kubernetes/pki/apiserver - etcd - client. The key /etc/kubernetes/pki/apiserver-etcd-client.crt /etc/kubernetes/pki/etcd/ca.crt}false true 0xc0007a94d0 apiextensions.k8s.io/v1beta1 <nil> 5m0s 1m0s}), err (context deadline exceeded)
Copy the code

The last message is authentication Handshake failed: X509: Certificate has expired or is not yet valid

Query the solution with this information

The solution

External recommendation

Finally, I found a solution on this website

Kubeadm certificate expiration time is adjusted

Using solution

Note The production cluster version is 1.14, which supports automatic certificate rotation. Therefore, manually issue a certificate and then enable automatic certificate renewal

Reissue of certificate

Issuing certificates directly on domestic networks will cause an error when you cannot connect to Google

[root@k8smaster1 ~]$kubeadm alpha certs renew all I1109 19:33:40.266778 22487 version.go:96] Could not fetch a Kubernetes version from the internet: unable to get URL"https://dl.k8s.io/release/stable-1.txt": Get https://storage.googleapis.com/kubernetes-release/release/stable-1.txt: net/http: request canceled (Client.Timeout exceeded whileAwaiting headers) I1109 19:33:40.266978 22487 Version.go :97] Falling back to thelocalclient version: V1.14.0i1109 19:33:51.139051 22487 version.go:96] Could not fetch a Kubernetes version from the Internet: unable to get URL"https://dl.k8s.io/release/stable-1.txt": Get https://storage.googleapis.com/kubernetes-release/release/stable-1.txt: net/http: request canceled (Client.Timeout exceeded whileAwaiting headers) I1109 19:33:51.139111 22487 Version.go :97] Falling back to thelocalclient version: V1.14.0i1109 19:34:01.511909 22487 version.go:96] Could not fetch a Kubernetes version from the Internet: unable to get URL"https://dl.k8s.io/release/stable-1.txt": Get https://storage.googleapis.com/kubernetes-release/release/stable-1.txt: net/http: request canceled while waiting for connection (Client.Timeout exceeded whileAwaiting HEADERS) I1109 19:34:01.511967 22487 Version.go :97] Falling back to thelocalclient version: V1.14.0i1109 19:34:11.945411 22487 version.go:96] Could not fetch a Kubernetes version from the Internet: unable to get URL"https://dl.k8s.io/release/stable-1.txt": Get https://storage.googleapis.com/kubernetes-release/release/stable-1.txt: net/http: request canceled (Client.Timeout exceeded whileAwaiting HEADERS) I1109 19:34:11.945504 22487 Version.go :97] Falling back to thelocalclient version: V1.14.0i1109 19:34:22.145905 22487 version.go:96] Could not fetch a Kubernetes version from the Internet: unable to get URL"https://dl.k8s.io/release/stable-1.txt": Get https://storage.googleapis.com/kubernetes-release/release/stable-1.txt: net/http: request canceled (Client.Timeout exceeded whileAwaiting HEADERS) I1109 19:34:22.145975 22487 Version.go :97] Falling back to thelocalclient version: V1.14.0i1109 19:34:32.532514 22487 version.go:96] Could not fetch a Kubernetes version from the Internet: unable to get URL"https://dl.k8s.io/release/stable-1.txt": Get https://storage.googleapis.com/kubernetes-release/release/stable-1.txt: net/http: request canceled (Client.Timeout exceeded whileAwaiting HEADERS) I1109 19:34:32.532561 22487 Version.go :97] Falling back to thelocalclient version: V1.14.0i1109 19:34:43.335574 22487 version.go:96] Could not fetch a Kubernetes version from the Internet: unable to get URL"https://dl.k8s.io/release/stable-1.txt": Get https://storage.googleapis.com/kubernetes-release/release/stable-1.txt: net/http: request canceled while waiting for connection (Client.Timeout exceeded whileAwaiting HEADERS) I1109 19:34:43.335628 22487 Version.go :97] Falling back to thelocalThe client version: v1.14.0Copy the code

Prepare a kubeadm.conf file to connect to the domestic image

apiVersion: kubeadm.k8s.io/v1beta1
kind: ClusterConfiguration
kubernetesVersion: v1.14.0 #--> Change this to the version of your cluster
imageRepository: registry.cn-hangzhou.aliyuncs.com/google_containers 
Could not fetch a Kubernetes version from the Internet: unable to get URL "https://dl.k8s.io/release/stable-1.txt"
Copy the code

Re-issue, and restart Kubelet

[root@k8smaster1 ~]$ kubeadm alpha certs renew all --config=/root/kubeadm.conf
[root@k8smaster1 ~]$ systemctl restart kubelet
Copy the code

To use the kubectl command after re-issuing, regenerate the ~/. Kube /config configuration file

[root@k8smaster1 kubernetes]$ kubeadm init phase kubeconfig all --config=/root/kubeadm.conf
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf"Kubeconfig file [root @ k8smaster1 kubernetes] $cp - I/etc/kubernetes/admin. Conf ~ /. Kube/config cp: whether or not covered"/root/.kube/config"? yCopy the code

Normal Use of commands

[root@k8smaster1 kubernetes]$ kubectl get pods NAME READY STATUS RESTARTS AGE glusterfs-hfdmh 1/1 Running 0 351d glusterfs-k766z 1/1 Running 0 351d glusterfs-rrc7x 1/1 Running 0 351d heketi-68f9dfdfbf-2k58b 1/1 Running 0 351d nginx 1/1 Running 1 370d nginx-ds-29dbc 1/1 Running 5 366d nginx-ds-4w6cn 1/1 Running 1 370d nginx-ds-6lhsk 0/1 Evicted 0 216d  nginx-ds-xq4h7 1/1 Running 3 366d tomcat-demo-6bc7d5b6f4-75rgc 0/1 Evicted 0 351dCopy the code

Restart all services in the cluster

The certificate is automatically renewed

Kubelet certificates are divided into server and client. K8s 1.9 enables automatic rotation of client certificates by default, but automatic rotation of server certificates needs to be enabled by users

Query kubelet configuration file location

[root@k8smaster1 ~]$ find / -name 10-kubeadm.conf
/usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf
[root@k8smaster1 ~]$ vi /usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf
Copy the code

Add the following configuration

# in the/etc/systemd/system/kubelet. Service. D / 10 - kubeadm. Conf increase the following parameters
Environment="KUBELET_EXTRA_ARGS=--feature-gates=RotateKubeletServerCertificate=true"
Copy the code

Add the controller-manager parameter

apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: null
  labels:
    component: kube-controller-manager
    tier: control-plane
  name: kube-controller-manager
  namespace: kube-system
spec:
  containers:
  - command:
    - kube-controller-manager
    - --allocate-node-cidrs=true
    # Validity of certificate
    - --experimental-cluster-signing-duration=87600h0m0s
    The certificate is automatically issued
    - --feature-gates=RotateKubeletServerCertificate=true
    - --authentication-kubeconfig=/etc/kubernetes/controller-manager.conf
    - --authorization-kubeconfig=/etc/kubernetes/controller-manager.conf
    - - the bind - address = 127.0.0.1
    - --client-ca-file=/etc/kubernetes/pki/ca.crt
    - - cluster - cidr = 172.22.0.0/16
    - --cluster-signing-cert-file=/etc/kubernetes/pki/ca.crt
    - --cluster-signing-key-file=/etc/kubernetes/pki/ca.key
    - --controllers=*,bootstrapsigner,tokencleaner
    - --kubeconfig=/etc/kubernetes/controller-manager.conf
    - --leader-elect=true
    - --node-cidr-mask-size=24
    - --requestheader-client-ca-file=/etc/k
Copy the code

Create an RBAC object

cat > ca-update.yaml << EOF apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: annotations: rbac.authorization.kubernetes.io/autoupdate: "true" labels: kubernetes.io/bootstrapping: rbac-defaults name: system:certificates.k8s.io:certificatesigningrequests:selfnodeserver rules: - apiGroups: - certificates.k8s.io resources: - certificatesigningrequests/selfnodeserver verbs: - create --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: kubeadm:node-autoapprove-certificate-server roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: system:certificates.k8s.io:certificatesigningrequests:selfnodeserver subjects: - apiGroup: rbac.authorization.k8s.io kind: Group name: system:nodes EOF
Copy the code
[root@k8smaster1 ~]$ kubectl create -f ca-update.yaml
clusterrole.rbac.authorization.k8s.io/system:certificates.k8s.io:certificatesigningrequests:selfnodeserver created
clusterrolebinding.rbac.authorization.k8s.io/kubeadm:node-autoapprove-certificate-server created
Copy the code