Hailong Wang is the Technical Manager of Rancher China community, responsible for the maintenance and operation of Rancher China Community. I have 6 years of experience in cloud computing and experienced the technical transformation from OpenStack to Kubernetes. I have rich practical experience in operation and peacekeeping no matter the underlying operating system Linux, virtual KVM or Docker container technology.
Before the speech
The Rancher Server certificate issue has been a frequent issue in the Rancher technology community. If the user did not think enough about setting up the Rancher environment for the first time, it is common to rebuild the cluster when certificates need to be replaced later.
This article shows you how to replace the Rancher certificate in an existing cluster without having to rebuild the cluster.
The version of Rancher used at the time of writing this article is V2.5.x, and other versions may operate slightly differently.
The following sections show how certificates can be replaced for single-node and highly available Rancher Server.
Replace the Rancher Server certificate of a single node
This section shows how to replace the default certificate for single-node Rancher with a self-signed certificate.
preparation
Suppose you have built a single-node Rancher and created a downstream business cluster:
The Rancher certificate is dynamic and issued by DynamicListener-ca. This is the default certificate generated by Rancher:
Because Rancher Agent needs to use the domain name to connect to Rancher Server after replacing the certificate, the business cluster will be unable to connect, so it is necessary to download the Kubeconfig of the business cluster from Rancher UI in advance. And switch the context to demo-rancher-demo2. After switching, k8S API-Server can be accessed directly without using Rancher API.
1. # kubectl config get-contexts
2. CURRENT NAME CLUSTER AUTHINFO
3. NAMESPACE
4. demo demo demo
demo-rancher-demo2 demo-rancher-demo2 demo
5. List item
6. # kubectl config use-context demo-rancher-demo2
7. Switched to context "demo-rancher-demo2".
8. List item
9. # kubectl config current-context
10. demo-rancher-demo2
11. List item
12. # kubectl get nodes
13. NAME STATUS ROLES AGE
14. VERSION
15. rancher-demo2 Ready controlplane,etcd,worker 58m v1.19.6
Copy the codeRancher Server is replaced with a self-signed certificate
Backup Rancher Server
Refer to the official documentation for upgrading single-node Rancher:
Docs. The rancher. Cn/docs/ranche…
1. # docker ps 2. CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 3. b0e3062667a2 rancher/rancher "entrypoint.sh" 2 hours ago Up 2 hours 0.0.0.0:80->80/ TCP, List item 5. # docker stop silly_swanson 6. Silly_swanson 7. List item 8. # docker create --volumes-from silly_swanson --name rancher-data rancher/rancher 9. aca6f6c791f7caa870e96e1c9f6370a6f015af62c65d73f1f6f80c56587a7542Copy the code2. Start Rancher Server with custom certificates based on backup
docker run -d --privileged --volumes-from rancher-data \ --restart=unless-stopped \ -p 80:80 -p 443:443 \ -v $PWD/certs/cert.pem:/etc/rancher/ssl/cert.pem \ -v $PWD/certs/key.pem:/etc/rancher/ssl/key.pem \ -v $PWD/certs/ca. Pem: / etc/rancher/SSL/cacerts pem \ - ring \ rancher/rancher: v2.5.5Copy the codeNote:
- Do not modify Rancher Server if your requirement is to replace certificates only
In this example, rancher/ Rancher :v2.5.5. Otherwise, you need to upgrade the image.
- This example is to replace the certificate with a self-signed certificate. If you want to replace the certificate with another type of certificate, please refer to other options in Upgrading single-node Rancher.
Re-log in to the Rancher UI and the status of the downstream cluster is Unavailable:
Cluster-agent container error:
kubectl logs -f cattle-cluster-agent-77cfbbff8b-ldt9n -n cattle-system time="2021-02-09T09:32:08Z" level=error msg="Remotedialer proxy error" error="x509: Cannot validate certificate for 192.168.64.55 because it doesn't contain any IP SANs" W0209 09:32:15.106448 1 Warnings. Go: 77] apiextensions. K8s. IO/v1beta1 CustomResourceDefinition is deprecated in v1.16 +, unavailable in v1.22 +; use apiextensions.k8s.io/v1 CustomResourceDefinition time="2021-02-09T09:32:18Z" level=info msg="Connecting to WSS: / / 192.168.64.55 / v3 / connect with token j6t7l2wkvz9k9xbdfnlf6kgxngfl89htl2svnmb4qgkrv7wl9ccp5m" Time = "2021-02-09 T09: then Z" level = info MSG = "Connecting to the proxy" url = "WSS: / / 192.168.64.55 / v3 / connect" time="2021-02-09T09:32:18Z" level=error msg="Failed to connect to proxy. Empty dialer response" error="x509: Cannot validate certificate for 192.168.64.55 because it doesn't contain any IP SANs"Copy the codeDelete daemonset. Apps /cattle-node-agent and deployment.apps/cattle-cluster-agent via kubectl
1. kubectl -n cattle-system delete daemonset.apps/cattle-node-agent
deployment.apps/cattle-cluster-agent
2. daemonset.apps "cattle-node-agent" deleted
3. deployment.apps "cattle-cluster-agent" deleted
Copy the code4. Go to the Setting-> Advanced Settings page and change the server-URL address to the domain name defined by you, for example, Rancher-demo.kingsd. top
5. Export cluster YAML
Create API tokens (user -> API & Keys) and save Bearer tokens on Rancher UI
After selecting the cluster, find its ClusterID in the Rancher UI (format C-XXxxx) and find it in the address bar.
RANCHERURL, CLUSTERID, TOKEN (host needs to install CURL and JQ)
3. # Cluster ID 4. CLUSTERID="c-sxjz5" 5. TOKEN="token-89z7s:lmg8cszl69vjj9pqr5bjst6shs6mht2n5wxtx6hlz8xpl962hxkprf" 7. # Valid certificates 8. curl -s -H "Authorization: Bearer ${TOKEN}" "${RANCHERURL}/v3/clusterregistrationtokens? clusterId=${CLUSTERID}" | jq -r '.data[] | select(.name ! = "system") | .command' 9. # Self signed certificates curl -s -k -H "Authorization: Bearer ${TOKEN}" 10. "${RANCHERURL}/v3/clusterregistrationtokens? clusterId=${CLUSTERID}" | jq -r '.data[] | select(.name ! = "system") | .insecureCommand'Copy the codeAfter the preceding command is executed successfully, the command for importing the cluster is displayed:
curl --insecure -sfL https://rancher-demo.kingsd.top/v3/import/rc2gb9qgl4wxm7tjwr5d6krqb9c8d8pkm8bt9fmtj8hrhx62fvqkgk.yaml | kubectl apply -f -Copy the codeThe cattle-cluster-agent command can then be executed on a host with kubeconfig and will regenerate the cattle-cluster-agent:
1. curl --insecure -sfL https://rancherdemo.kingsd.top/v3/import/rc2gb9qgl4wxm7tjwr5d6krqb9c8d8pkm8bt9fmtj8hrhx62fvqkgk.yaml | kubectl apply -f - 2. clusterrole.rbac.authorization.k8s.io/proxy-clusterrole-kubeapiserver unchanged 3. clusterrolebinding.rbac.authorization.k8s.io/proxy-role-binding-kubernetes-master unchanged 4. namespace/cattle-system unchanged 5. serviceaccount/cattle unchanged 6. clusterrolebinding.rbac.authorization.k8s.io/cattle-admin-binding unchanged 7. secret/cattle-credentials-cfff3df unchanged 8. clusterrole.rbac.authorization.k8s.io/cattle-admin unchanged deployment.apps/cattle-cluster-agent createdCopy the codevalidation
Cattle-cluster-agent and cattle-node-Agent will run again after a moment:
kubectl -n cattle-system get pods
NAME READY STATUS RESTARTS AGE
cattle-cluster-agent-c9774fcdd-rwdk5 1/1 Running 2 2m14s
cattle-node-agent-r58w4 1/1 Running 0 2m13s
Copy the codeThe service cluster status changes to Active again:
At this point, the Rancher Server certificate has been replaced with a self-signed certificate:
Replace the Rancher HA certificate
This section shows how to replace the Rancher HA from a self-signed certificate to a trusted certificate
In this example, L4 is used to build Rancher HA. If L7 is used to build Rancher HA, the certificate can be replaced from LB.
preparation
Suppose you have set up a highly available self-signed Rancher cluster and created a downstream business cluster:
The Rancher certificate is a self-signed rancher.kingsd.top issued by ca-Rancher.kingsd. top:
Replace the self-signed certificate with a trusted certificate
1. Remove self-signed certificates from secret
1. kubectl -n cattle-system delete secret tls-rancher-ingress
2. kubectl -n cattle-system delete secret tls-ca
Copy the code2. Re-create tlS-rancher-ingress with trusted certificates
1. kubectl -n cattle-system create secret tls tls-rancher-ingress \
2. --cert=rancher.kingsd.top.pem \
3. --key=rancher.kingsd.top.key
Copy the code3. Get the value passed with –set from the currently installed Rancher Helm Chart.
1. # helm get values rancher -n cattle-system
2. USER-SUPPLIED VALUES:
3. hostname: rancher.kingsd.top
4. ingress:
5. tls:
6. source: secret
7. privateCA: true
Copy the code4. Append all values from the previous step to the command with –set key=value.
1. # helm upgrade rancher rancher-latest/rancher \ 2. --namespace cattle-system \ 3. --set hostname=rancher.kingsd.top \ 4. --set ingress.tls.source=secretCopy the codeBecause only self-signed certificates require the privateCA: true parameter, you need to delete this parameter when updating the cluster to a trusted certificate
This is an example, and there may be more values to append from the previous step. Alternatively, you can export the current value to a file and reference that file when upgrading.
For details, see section high Availability Upgrade Guide in the Chinese documentation:
Docs. The rancher. Cn/docs/ranche…
In this example, the certificate is replaced with a trusted certificate. If you want to replace the certificate with another type of certificate, see other parameters in the High Availability Upgrade Guide.
validation
Wait a moment, and when the Rancher Pod is successfully updated, you can access Rancher Server from your browser.
At this point, the Rancher Server certificate has been replaced with a trusted certificate:
Remember after
Whether the certificate is replaced by a single node or a highly available Rancher Server, the idea is basically the same. The new certificate is used to update the original cluster. If the domain name is changed during the certificate replacement process or the agent of the service cluster cannot connect to the Rancher Server, You can modify the server-URL by referring to the procedure for replacing the single-node Rancher Server certificate in this article, and then re-create the cattle-cluster-Agent.