No need to rebuild the cluster, easily replace the certificate!

Hailong Wang is the Technical Manager of Rancher China community, responsible for the maintenance and operation of Rancher China Community. I have 6 years of experience in cloud computing and experienced the technical transformation from OpenStack to Kubernetes. I have rich practical experience in operation and peacekeeping no matter the underlying operating system Linux, virtual KVM or Docker container technology.

Before the speech

The Rancher Server certificate issue has been a frequent issue in the Rancher technology community. If the user did not think enough about setting up the Rancher environment for the first time, it is common to rebuild the cluster when certificates need to be replaced later.

This article shows you how to replace the Rancher certificate in an existing cluster without having to rebuild the cluster.

The version of Rancher used at the time of writing this article is V2.5.x, and other versions may operate slightly differently.

The following sections show how certificates can be replaced for single-node and highly available Rancher Server.

Replace the Rancher Server certificate of a single node

This section shows how to replace the default certificate for single-node Rancher with a self-signed certificate.

preparation

Suppose you have built a single-node Rancher and created a downstream business cluster:

The Rancher certificate is dynamic and issued by DynamicListener-ca. This is the default certificate generated by Rancher:

Because Rancher Agent needs to use the domain name to connect to Rancher Server after replacing the certificate, the business cluster will be unable to connect, so it is necessary to download the Kubeconfig of the business cluster from Rancher UI in advance. And switch the context to demo-rancher-demo2. After switching, k8S API-Server can be accessed directly without using Rancher API.

 1. # kubectl config get-contexts
 2. CURRENT   NAME                 CLUSTER              AUTHINFO  
  
 3. NAMESPACE
 4.         demo                 demo                 demo
          demo-rancher-demo2   demo-rancher-demo2   demo
 5. List item
 6. # kubectl config use-context demo-rancher-demo2
 7. Switched to context "demo-rancher-demo2".
 8. List item
 9. # kubectl config current-context
 10. demo-rancher-demo2
 11. List item
 12. # kubectl get nodes
 13. NAME            STATUS   ROLES                      AGE
 14. VERSION
 15. rancher-demo2   Ready    controlplane,etcd,worker   58m   v1.19.6
Copy the code

Rancher Server is replaced with a self-signed certificate

Backup Rancher Server

Refer to the official documentation for upgrading single-node Rancher:

Docs. The rancher. Cn/docs/ranche…

1. # docker ps 2. CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 3. b0e3062667a2 rancher/rancher "entrypoint.sh" 2 hours ago Up 2 hours 0.0.0.0:80->80/ TCP, List item 5. # docker stop silly_swanson 6. Silly_swanson 7. List item 8. # docker  create --volumes-from silly_swanson --name rancher-data rancher/rancher 9. aca6f6c791f7caa870e96e1c9f6370a6f015af62c65d73f1f6f80c56587a7542Copy the code

2. Start Rancher Server with custom certificates based on backup

docker run -d --privileged --volumes-from rancher-data \ --restart=unless-stopped \ -p 80:80 -p 443:443 \ -v $PWD/certs/cert.pem:/etc/rancher/ssl/cert.pem \ -v $PWD/certs/key.pem:/etc/rancher/ssl/key.pem \ -v $PWD/certs/ca. Pem: / etc/rancher/SSL/cacerts pem \ - ring \ rancher/rancher: v2.5.5Copy the code

Note:

• Do not modify Rancher Server if your requirement is to replace certificates only

In this example, rancher/ Rancher :v2.5.5. Otherwise, you need to upgrade the image.

• This example is to replace the certificate with a self-signed certificate. If you want to replace the certificate with another type of certificate, please refer to other options in Upgrading single-node Rancher.

Re-log in to the Rancher UI and the status of the downstream cluster is Unavailable:

Cluster-agent container error:

kubectl logs -f cattle-cluster-agent-77cfbbff8b-ldt9n -n cattle-system time="2021-02-09T09:32:08Z" level=error msg="Remotedialer proxy error" error="x509: Cannot validate certificate for 192.168.64.55 because it doesn't contain any IP SANs" W0209 09:32:15.106448 1 Warnings. Go: 77] apiextensions. K8s. IO/v1beta1 CustomResourceDefinition is deprecated in v1.16 +, unavailable in v1.22 +; use apiextensions.k8s.io/v1 CustomResourceDefinition time="2021-02-09T09:32:18Z" level=info msg="Connecting to WSS: / / 192.168.64.55 / v3 / connect with token j6t7l2wkvz9k9xbdfnlf6kgxngfl89htl2svnmb4qgkrv7wl9ccp5m" Time = "2021-02-09 T09: then Z" level = info MSG = "Connecting to the proxy" url = "WSS: / / 192.168.64.55 / v3 / connect" time="2021-02-09T09:32:18Z" level=error msg="Failed to connect to proxy. Empty dialer response" error="x509: Cannot validate certificate for 192.168.64.55 because it doesn't contain any IP SANs"Copy the code

Delete daemonset. Apps /cattle-node-agent and deployment.apps/cattle-cluster-agent via kubectl

 1. kubectl -n cattle-system delete daemonset.apps/cattle-node-agent
    deployment.apps/cattle-cluster-agent
 2. daemonset.apps "cattle-node-agent" deleted
 3. deployment.apps "cattle-cluster-agent" deleted
Copy the code

4. Go to the Setting-> Advanced Settings page and change the server-URL address to the domain name defined by you, for example, Rancher-demo.kingsd. top

5. Export cluster YAML

Create API tokens (user -> API & Keys) and save Bearer tokens on Rancher UI

After selecting the cluster, find its ClusterID in the Rancher UI (format C-XXxxx) and find it in the address bar.

RANCHERURL, CLUSTERID, TOKEN (host needs to install CURL and JQ)

3. # Cluster ID 4. CLUSTERID="c-sxjz5" 5. TOKEN="token-89z7s:lmg8cszl69vjj9pqr5bjst6shs6mht2n5wxtx6hlz8xpl962hxkprf" 7. # Valid certificates 8. curl -s -H "Authorization: Bearer ${TOKEN}" "${RANCHERURL}/v3/clusterregistrationtokens? clusterId=${CLUSTERID}" | jq -r '.data[] | select(.name ! = "system") | .command' 9. # Self signed certificates curl -s -k -H "Authorization: Bearer ${TOKEN}" 10. "${RANCHERURL}/v3/clusterregistrationtokens? clusterId=${CLUSTERID}" | jq -r '.data[] | select(.name ! = "system") | .insecureCommand'Copy the code

After the preceding command is executed successfully, the command for importing the cluster is displayed:

curl --insecure -sfL https://rancher-demo.kingsd.top/v3/import/rc2gb9qgl4wxm7tjwr5d6krqb9c8d8pkm8bt9fmtj8hrhx62fvqkgk.yaml | kubectl apply -f  -Copy the code

The cattle-cluster-agent command can then be executed on a host with kubeconfig and will regenerate the cattle-cluster-agent:

1. curl --insecure -sfL https://rancherdemo.kingsd.top/v3/import/rc2gb9qgl4wxm7tjwr5d6krqb9c8d8pkm8bt9fmtj8hrhx62fvqkgk.yaml | kubectl apply -f - 2. clusterrole.rbac.authorization.k8s.io/proxy-clusterrole-kubeapiserver unchanged 3. clusterrolebinding.rbac.authorization.k8s.io/proxy-role-binding-kubernetes-master unchanged 4. namespace/cattle-system unchanged 5. serviceaccount/cattle unchanged 6. clusterrolebinding.rbac.authorization.k8s.io/cattle-admin-binding unchanged 7. secret/cattle-credentials-cfff3df unchanged 8. clusterrole.rbac.authorization.k8s.io/cattle-admin unchanged  deployment.apps/cattle-cluster-agent createdCopy the code

validation

Cattle-cluster-agent and cattle-node-Agent will run again after a moment:

kubectl -n cattle-system get pods
NAME                                   READY   STATUS    RESTARTS   AGE
cattle-cluster-agent-c9774fcdd-rwdk5   1/1     Running   2          2m14s
cattle-node-agent-r58w4                1/1     Running   0          2m13s
Copy the code

The service cluster status changes to Active again:

At this point, the Rancher Server certificate has been replaced with a self-signed certificate:

Replace the Rancher HA certificate

This section shows how to replace the Rancher HA from a self-signed certificate to a trusted certificate

In this example, L4 is used to build Rancher HA. If L7 is used to build Rancher HA, the certificate can be replaced from LB.

preparation

Suppose you have set up a highly available self-signed Rancher cluster and created a downstream business cluster:

The Rancher certificate is a self-signed rancher.kingsd.top issued by ca-Rancher.kingsd. top:

Replace the self-signed certificate with a trusted certificate

1. Remove self-signed certificates from secret

 1. kubectl -n cattle-system delete secret tls-rancher-ingress
 2. kubectl -n cattle-system delete secret tls-ca
Copy the code

2. Re-create tlS-rancher-ingress with trusted certificates

 1. kubectl -n cattle-system create secret tls tls-rancher-ingress \  
 2. --cert=rancher.kingsd.top.pem \
 3. --key=rancher.kingsd.top.key
Copy the code

3. Get the value passed with –set from the currently installed Rancher Helm Chart.

 1. # helm get values rancher -n cattle-system
 2. USER-SUPPLIED VALUES:
 3. hostname: rancher.kingsd.top
 4. ingress:
 5. tls:
 
 6. source: secret
 7. privateCA: true
Copy the code

4. Append all values from the previous step to the command with –set key=value.

1. # helm upgrade rancher rancher-latest/rancher \ 2. --namespace cattle-system \ 3. --set hostname=rancher.kingsd.top \  4. --set ingress.tls.source=secretCopy the code

Because only self-signed certificates require the privateCA: true parameter, you need to delete this parameter when updating the cluster to a trusted certificate

This is an example, and there may be more values to append from the previous step. Alternatively, you can export the current value to a file and reference that file when upgrading.

For details, see section high Availability Upgrade Guide in the Chinese documentation:

Docs. The rancher. Cn/docs/ranche…

In this example, the certificate is replaced with a trusted certificate. If you want to replace the certificate with another type of certificate, see other parameters in the High Availability Upgrade Guide.

validation

Wait a moment, and when the Rancher Pod is successfully updated, you can access Rancher Server from your browser.

At this point, the Rancher Server certificate has been replaced with a trusted certificate:

Remember after

Whether the certificate is replaced by a single node or a highly available Rancher Server, the idea is basically the same. The new certificate is used to update the original cluster. If the domain name is changed during the certificate replacement process or the agent of the service cluster cannot connect to the Rancher Server, You can modify the server-URL by referring to the procedure for replacing the single-node Rancher Server certificate in this article, and then re-create the cattle-cluster-Agent.