Linkerd 2.10(Step by Step) — 3. Automatic rotation control plane TLS &Webhook TLS credentials

Linkerd 2.10 series

• Linkerd V2 Service Mesh
• Tencent Cloud K8S deployment Service Mesh — Linkerd2 & Traefik2 deployment emojivoto application
• Learn about the basic features of Linkerd 2.10 and step into the era of Service Mesh
• Linkerd 2.10(Step by Step) — 1. Add your service to Linkerd
• Linkerd 2.10(Step by Step) — 2. Automated Canary publishing

Linkerd 2.10 中文 版

• linkerd.hacker-linner.com

Automatic rotation control plane TLS credentials

Linkerd’s automatic mTLS function generates TLS certificates for agents using a set of TLS credentials: Trust anchors, Issuer certificates, and private keys. While Linkerd automatically rotates the TLS certificates for the data plane agent every 24 hours, it does not rotate the TLS credentials used to issue those certificates. In this document, we describe how to automatically rotate issuer certificates and private keys using an external solution.

(Note that Linkerd’s trust anchor must still be manually rotated on the long-lived cluster)

Cert manager

Cert-manager is a popular project for making TLS credentials from external sources available to Kubernetes clusters.

First, install cert-Manager on your cluster.

If you want to install cert-Manager >= 1.0, you need Kubernetes >= 1.16. The traditional custom resource definition for Kubernetes <= 1.15 in cert-Manager does not have the keyAlgorithm option, so the certificate will be generated using RSA and not compatible with Linkerd.

For more details on version requirements, see the V0.16 to V1.0 upgrade notes.

Cert Manager acts as a Certificate Authority (CA) on the cluster

In this case, instead of getting credentials from an external source, we configure them as a CA on the cluster and have it periodically reissue Linkerd’s Issuer Certificate and private key.

First, create the namespace that cert-Manager will use to store its Linkerd-related resources. For simplicity, we recommend using the default Linkerd control plane namespace:

kubectl create namespace linkerd
Copy the code

Save the signing key pair as Secret

Next, using the STEP tool, create a signing key pair and store it in Kubernetes Secret in the namespace created above:

step certificate create root.linkerd.cluster.local ca.crt ca.key \ --profile root-ca --no-password --insecure && kubectl  create secret tls \ linkerd-trust-anchor \ --cert=ca.crt \ --key=ca.key \ --namespace=linkerdCopy the code

For longer-lived trust anchor certificates, the –not-after argument is passed to the step command with the desired value (for example –not-after=87600h).

The Issuer of the referencing the secret

With Secret, we can create a cert-manager “Issuer” resource that references it:

cat <<EOF | kubectl apply -f - apiVersion: cert-manager.io/v1 kind: Issuer metadata: name: linkerd-trust-anchor namespace: linkerd spec: ca: secretName: linkerd-trust-anchor EOF
Copy the code

Issuing certificates and writing them to a secret

Finally, we can create a cert-manager “Certificate” resource that uses the Issuer to generate the required certificates:

cat <<EOF | kubectl apply -f - apiVersion: cert-manager.io/v1 kind: Certificate metadata: name: linkerd-identity-issuer namespace: linkerd spec: secretName: linkerd-identity-issuer duration: 48h renewBefore: 25h issuerRef: name: linkerd-trust-anchor kind: Issuer commonName: identity.linkerd.cluster.local dnsNames: - identity.linkerd.cluster.local isCA: true privateKey: algorithm: ECDSA usages: - cert sign - crl sign - server auth - client auth EOF
Copy the code

In the YAML listing above, the Duration key instructs cert-Manager to treat the certificate as valid for 48 hours, and the renewBefore key instructs cert-Manager to attempt to issue a new certificate 25 hours before the current one expires. These values can be customized to your liking.)

At this point, cert-Manager can now use the Certificate Resource to obtain the TLS credentials. This credential will be stored in secret named Linkerd-identity-Issuer. To verify your newly issued certificate, you can run:

kubectl get secret linkerd-identity-issuer -o yaml -n linkerd
Copy the code

Now we just need to tell Linkerd to use these credentials.

Due to a bug in cert-Manager, if you use version 0.15 of Cert-Manager with experimental controllers, It issues a certificate that is incompatible with Linkerd version <= stable-2.8.1.

Your Linkerd-Identity pod may crash with the following log output:

"Failed to initialize identity service: failed to read CA from disk:
unsupported block type: 'PRIVATE KEY'"
Copy the code

Some possible solutions to this problem are:

• Upgrade Linkerd to includefixEdge version> = edge - 20.6.4.
• Upgrade cert-Manager to version> = 0.16. (How to upgrade)
• Disable the cert-Manager experimental Controllers. (docs)

Alternative CA providers

Instead of using Cert Manager as the CA, you can configure Cert Manager to rely on many other solutions, such as Vault. You can find more details here on how to set up an existing certificate manager to use different types of issuers.

Third-party certificate management solution

It is important to note that the mechanism provided by Linkerd can also be used outside of cert-Manager. Linkerd reads linkerd-identity-issuer Secret, and if it is of type kubernetes. IO/TLS, uses the content as its TLS credentials. This means that any solution that can rotate them by writing TLS certificates to this key can be used to provide dynamic TLS certificate management.

Use these credentials in the CLI installation

For CLI installations, the Linkerd control plane should be installed with the — identity-external-Issuer flag, which instructs Linkerd to read certificates from linkerd-identity-issuer Secret. Whenever the certificate and key stored in Secret are updated, the Identity service automatically detects the change and reloads the new credentials.

Look! We have set up automatic rotation of TLS credentials on the Linkerd control plane. If you want to monitor the update process, you can check the Issue Updated event issued by the service:

kubectl get events --field-selector reason=IssuerUpdated -n linkerd
Copy the code

Installation using Helm

For Helm installation, instead of running Linkerd install, set identityTrustAnchorsPEM to the value of ca.crt in Linkerd-identity-issuer Secret:

helm install linkerd2 \
  --set-file identityTrustAnchorsPEM=ca.crt \
  --set identity.issuer.scheme=kubernetes.io/tls \
  --set installNamespace=false \
  linkerd/linkerd2 \
  -n linkerd
Copy the code

For Helm versions lower than V3, the –name flag must be passed specifically. In Helm V3, it is deprecated and is the first parameter specified above.

Automatic rotation of Webhook TLS credentials

The Linkerd control plane contains several components, called Webhooks, which are called directly by Kubernetes itself. Traffic from Kubernetes to Linkerd Webhooks is secured using TLS, so each Webhooks requires a secret containing TLS credentials. These certificates are different from those used by the Linkerd agent to secure poD-to-POD communication and use a completely separate trust chain.

By default, when Linkerd is installed with Linkerd CLI or Linkerd Helm Chart, TLS credentials are automatically generated for all Webhooks. If these certificates expire or need to be regenerated for any reason, performing Linkerd Upgrade (using Linkerd CLI or using Helm) will regenerate them.

This workflow works for most users. However, if you need to automatically rotate these Webhook certificates on a regular basis, you can use cert-Manager to manage them automatically.

Install the Cert manager

First, install cert-Manager on your cluster and create the namespace that cert-Manager will use to store its Webhook related resources. For simplicity, we recommend using the default linkerd namespace: linkerd

# control plane core
kubectl create namespace linkerd

# viz (ignore if not using the viz extension)
kubectl create namespace linkerd-viz

# viz (ignore if not using the jaeger extension)
kubectl create namespace linkerd-jaeger
Copy the code

Save the signing key pair as Secret

Next, we will use the STEP tool to create a signing key pair for each Webhook certificate:

step certificate create webhook.linkerd.cluster.local ca.crt ca.key \
  --profile root-ca --no-password --insecure --san webhook.linkerd.cluster.local

kubectl create secret tls webhook-issuer-tls --cert=ca.crt --key=ca.key --namespace=linkerd

# ignore if not using the viz extension
kubectl create secret tls webhook-issuer-tls --cert=ca.crt --key=ca.key --namespace=linkerd-viz

# ignore if not using the jaeger extension
kubectl create secret tls webhook-issuer-tls --cert=ca.crt --key=ca.key --namespace=linkerd-jaeger
Copy the code

Creating publishers that reference Secrets (Issuers)

With Secrets, we can create cert-manager “Issuer” resources that reference them:

cat <<EOF | kubectl apply -f - apiVersion: cert-manager.io/v1 kind: Issuer metadata: name: webhook-issuer namespace: linkerd spec: ca: secretName: webhook-issuer-tls --- # ignore if not using the viz extension apiVersion: cert-manager.io/v1 kind: Issuer metadata: name: webhook-issuer namespace: linkerd-viz spec: ca: secretName: webhook-issuer-tls --- # ignore if not using the jaeger extension apiVersion: cert-manager.io/v1 kind: Issuer metadata: name: webhook-issuer namespace: linkerd-jaeger spec: ca: secretName: webhook-issuer-tls EOF
Copy the code

Issue a certificate and write it to secrets

Finally, we can create the cert-Manager “Certificate” resource, which uses the issuer (Issuers) to generate the required Certificate:

cat <<EOF | kubectl apply -f -
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: linkerd-proxy-injector
  namespace: linkerd
spec:
  secretName: linkerd-proxy-injector-k8s-tls
  duration: 24h
  renewBefore: 1h
  issuerRef:
    name: webhook-issuer
    kind: Issuer
  commonName: linkerd-proxy-injector.linkerd.svc
  dnsNames:
  - linkerd-proxy-injector.linkerd.svc
  isCA: false
  privateKey:
    algorithm: ECDSA
  usages:
  - server auth
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: linkerd-sp-validator
  namespace: linkerd
spec:
  secretName: linkerd-sp-validator-k8s-tls
  duration: 24h
  renewBefore: 1h
  issuerRef:
    name: webhook-issuer
    kind: Issuer
  commonName: linkerd-sp-validator.linkerd.svc
  dnsNames:
  - linkerd-sp-validator.linkerd.svc
  isCA: false
  privateKey:
    algorithm: ECDSA
  usages:
  - server auth
---
# ignore if not using the viz extension
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: tap
  namespace: linkerd-viz
spec:
  secretName: tap-k8s-tls
  duration: 24h
  renewBefore: 1h
  issuerRef:
    name: webhook-issuer
    kind: Issuer
  commonName: tap.linkerd-viz.svc
  dnsNames:
  - tap.linkerd-viz.svc
  isCA: false
  privateKey:
    algorithm: ECDSA
  usages:
  - server auth
---
# ignore if not using the viz extension
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: linkerd-tap-injector
  namespace: linkerd-viz
spec:
  secretName: tap-injector-k8s-tls
  duration: 24h
  renewBefore: 1h
  issuerRef:
    name: webhook-issuer
    kind: Issuer
  commonName: tap-injector.linkerd-viz.svc
  dnsNames:
  - tap-injector.linkerd-viz.svc
  isCA: false
  privateKey:
    algorithm: ECDSA
  usages:
  - server auth
---
# ignore if not using the jaeger extension
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: jaeger-injector
  namespace: linkerd-jaeger
spec:
  secretName: jaeger-injector-k8s-tls
  duration: 24h
  renewBefore: 1h
  issuerRef:
    name: webhook-issuer
    kind: Issuer
  commonName: jaeger-injector.linkerd.svc
  dnsNames:
  - jaeger-injector.linkerd.svc
  isCA: false
  privateKey:
    algorithm: ECDSA
  usages:
  - server auth
EOF
Copy the code

At this point, cert-Manager can now use these Certificate resources to obtain TLS certificates. These credentials are stored in linkerd-proxy-injector-k8s-TLS, Linkerd-SP-validator-k8s-TLS, tap-k8S-TLS, tap-Injector-k8s-TLS, and linkerd-proxy-injector-k8s-TLS, respectively Jaeger – Injector – K8s – TLS these secrets.

Now we just need to tell Linkerd to use these credentials.

Use these credentials in the CLI installation

To configure Linkerd to use credentials from cert-Manager instead of generating its own, we generate a supplementary configuration file:

CA=$(awk '{ print " " $0 }' ca.crt)

cat > config.yml <<EOF proxyInjector: externalSecret: true caBundle: | $CA profileValidator: externalSecret: true caBundle: | $CA EOF

# ignore if not using the viz extension
cat > config-viz.yml <<EOF tap: externalSecret: true caBundle: | $CA tapInjector: externalSecret: true caBundle: | $CA EOF

# ignore if not using the jaeger extension
cat > config-jaeger.yml <<EOF webhook: externalSecret: true caBundle: | $CA EOF
Copy the code

Now we can install Linkerd using these configuration files:

linkerd install --values=config.yml | kubectl apply -f -

# ignore if not using the viz extension
linkerd viz install --values=config-viz.yml | kubectl apply -f -

# ignore if not using the jaeger extension
linkerd jaeger install --values=config-jaeger.yml | kubectl apply -f -
Copy the code

Installation using Helm

For Helm installation, we can directly configure the Helm value:

helm install linkerd2 \
  --set installNamespace=false\ -set proxyInjector.externalSecret=true \
  --set-file proxyInjector.caBundle=ca.crt \
  --set profileValidator.externalSecret=true \
  --set-file profileValidator.caBundle=ca.crt \
  linkerd/linkerd2 \
  -n linkerd

# ignore if not using the viz extension
helm install linkerd-viz \
  --set installNamespace=false\ -set tap.externalSecret=true \
  --set-file tap.caBundle=ca.crt \
  --set tapInjector.externalSecret=true \
  --set-file tapInjector.caBundle=ca.crt \
  linkerd/linkerd-viz \
  -n linkerd-viz

# ignore if not using the jaeger extension
helm install linkerd-jaeger \
  --set installNamespace=false\ -set webhook.externalSecret=true \
  --set-file webhook.caBundle=ca.crt \
  linkerd/linkerd-jaeger \
  -n linkerd-jaeger
Copy the code

When installing Linkerd with Helm, you must also provide issuer trust root and issuer credentials, as described in installing Linkerd with Helm.

For Helm versions lower than V3, the –name flag must be passed specifically. In Helm V3, it is deprecated and is the first parameter specified above.