Background:

Continuous upgrade process: Kubernetes 1.16.15 to 1.17.17,Kubernetes 1.17.17 to 1.18.20, and Kubernetes 1.18.20 to 1.19.12

Cluster configuration:

1. Refer to official documentation

Reference: https://kubernetes.io/zh/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade/

2. Confirm the upgradable version and upgrade plan

yum list --showduplicates kubeadm --disableexcludes=kubernetes

The latest version of 1.20 is version 1.20.9-0. K8S-master-03 node should be upgraded first according to personal habits

3. Upgrade the K8S-master-03 node control plane

K8S-master-03 is still executed:

1. Yum upgrades the Kubernetes plugin

Yum install kubeadm-1.20.9-0 kubelet-1.20.9-0 kubectl-1.20.9-0 --disableexcludes=kubernetes

2. The vacated node checks whether the cluster can be upgraded

Still considered a refresher drain command:

kubectl drain k8s-master-03 --ignore-daemonsets
sudo kubeadm upgrade plan

3. Upgrade to 1.20.9

Kubeadm upgrade the apply 1.20.9

[root@k8s-master-03 ~]# sudo systemctl daemon-reload [root@k8s-master-03 ~]# sudo systemctl restart kubelet [root@k8s-master-03 ~]# kubectl uncordon k8s-master-03 node/k8s-master-03 uncordoned [root@k8s-master-03 ~]# kubectl get  nodes [root@k8s-master-03 ~]# kubectl get pods -n kube-system

4. Upgrade other control plane (K8S-master-01 K8S-master-02)

Yum install kubeadm-1.20.9-0 kubelet-1.20.9-0 kubectl-1.20.9-0 --disableexcludes=kubernetes sudo kubeadm upgrade node sudo systemctl daemon-reload sudo systemctl restart kubelet

5. Work node upgrade

Yum install kubeadm-1.20.9-0 kubelet-1.20.9-0 kubectl-1.20.9-0 --disableexcludes=kubernetes sudo kubeadm upgrade node sudo systemctl daemon-reload sudo systemctl restart kubelet

Note: there is no empty node for individuals, depending on their needs

6. Verify the upgrade

 kubectl get nodes

7. Others — V1.20.0 disables key Inks

Since my Prometheus Oprator is a branch of 0.4, I’m ready to uninstall and reinstall. The version gap is too wide. Now you don’t want to branch out and use the mainline version: install Prometheus-Oprator in Kubernetes 1.20.5. There is no problem with the basic process, but there is one problem: I used NFS to upgrade my StorageClass cluster to Kubernetes1.16:

kubectl get sc

The last

kubectl get pods -n monitoring
kubectl logs -f prometheus-operator-84dc795dc8-lkl5r -n monitoring

Look at the key words:

Additional seems to be an automatic discovery configuration, right?

First comment out the automatically discovered configuration in the prometheus-prometheus.yaml file. Well, the service is still not working, look at the log again:

kubectl logs -f prometheus-operator-84dc795dc8-lkl5r -n monitoring

There is nothing new out there, but take a look at PV, PVC is not created. Take a look at the NFS POD log:

kubectl get pods -n nfs
kubectl logs -f nfs-client-provisioner-6cb4f54cbc-wqqw9 -n nfs

class “managed-nfs-storage”: unexpected error getting claim reference: selfLink was empty, can’t make reference

Baidu is the key product of Ink.https://www.orchome.com/10024



Modify the kube-apiserver.yaml of the three master nodes

Then PV, PVC created successfully Prometheus service started successfully. Then take a look back at my Additional Autodiscovery configuration:

I am inInstall Prometheus-Oprator in Kubernetes 1.20.5



Try my old version of this file? :

cat <<EOF > prometheus-additional.yaml - job_name: 'kubernetes-service-endpoints' kubernetes_sd_configs: - role: endpoints relabel_configs: - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape] action: keep regex: true - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scheme] action: replace target_label: __scheme__ regex: (https?) - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_path] action: replace target_label: __metrics_path__ regex: (.+) - source_labels: [__address__, __meta_kubernetes_service_annotation_prometheus_io_port] action: replace target_label: __address__ regex: (([^ :] +)? ::\d+)? ; (\d+) replacement: $1:$2 - action: labelmap regex: __meta_kubernetes_service_label_(.+) - source_labels: [__meta_kubernetes_namespace] action: replace target_label: kubernetes_namespace - source_labels: [__meta_kubernetes_service_name] action: replace target_label: kubernetes_name EOF

kubectl delete secret additional-configs -n monitoring
kubectl create secret generic additional-configs --from-file=prometheus-additional.yaml -n monitoring

Look at the log started up again. First suspect that there is a problem with prometheus-additional. YAML in my configuration file. Of course this is personal. The main emphasis is on the master node kube-apiserver.yaml:

- --feature-gates=RemoveSelfLink=false

​