Kubernetes MutatingAdmissionWebhook: An inside look at Kubernetes MutatingAdmissionWebhook

Diving into Kubernetes MutatingAdmissionWebhook

Admission Controllers are a useful tool for intercepting requests to the Kubernetes API Server before data persistence. However, because it needs to be compiled into binaries by the cluster administrator in Kube-Apiserver, it is not very flexible to use. Starting with Kubernetes 1.7, Initializers and External Admission Webhooks were introduced to solve this problem. In Kubernetes 1.9, Initializers remain in alpha, while External Admission Webhooks have been promoted to beta, And is divided into MutatingAdmissionWebhook and ValidatingAdmissionWebhook.

MutatingAdmissionWebhook and ValidatingAdmissionWebhook both add up to is a special type of admission controllers, a processing resources change, a process validation. Validation is defined by matching to MutatingWebhookConfiguration rules.

In this article, I will delve into the details of MutatingAdmissionWebhook and implement a usable Webhook Admission Server step by step.

The benefits of Webhooks

Kubernetes cluster administrators can use Webhooks to create additional resource change and validation access plug-ins that work through Apiserver’s access chain without recompiling Apiserver. This allows developers to customize the entry logic for many actions, such as creating, updating, and deleting any resource, giving developers a great deal of freedom and flexibility. The number of applications available is huge. Some common uses include:

  1. Make changes before creating the resource. Istio is a typical example of injecting an Envoy Sidecar container into the target Pods for traffic management and rule enforcement.
  2. Automatic configurationStorageClass. Listening to thePersistentVolumeClaimResources, and automatically add corresponding ones to them according to preset rulesStorageClass. Users don’t need to careStorageClassCreation.
  3. Validate complex custom resources. Ensure that a custom resource can be created only after it has been defined and all dependencies are created and available.
  4. Namespace limits. In a multi-tenant system, prevent resources from being created in a pre-reserved namespace.

In addition to the usage scenarios listed above, more applications can be created based on WebHooks.

Webhooks and Initializers

Based on community feedback, and the use case for alpha versions of External Admission Webhooks and Initializers, the Kubernetes community decided to upgrade Webhooks to beta, And it is divided into two kinds of webhooks (MutatingAdmissionWebhook and ValidatingAdmissionWebhook). These updates bring Webhooks into line with other Admission Controllers and enforce mutate-before-validate. Initializers can change before Kubernetes resources are created, enabling dynamic access control. If you’re not familiar with Initializers, check out this article.

So, what is the difference between Webhooks and Initializers?

  1. WebhooksIt can be used for more operations, including “mutate” and “admit” for “add, delete, or modify” resources. However,InitializersYou cannot “admit” for “delete” resources.
  2. WebhooksDo not query resources before creating them. However,InitializersYou can listen for uninitialized resources by using parameters? includeUninitialized=trueTo implement.
  3. Due to theInitializersThe “pre-created” state is persisted to etCD as well, thus introducing high latency and a burden on etCD, especially if apiserver upgrades or fails; However Webhooks consume less memory and computing resources.
  4. Webhooks 比 InitializersThe insurance against failure is stronger.WebhooksYou can configure a failure policy to prevent resources from being hung when they are created. However,InitializersIt is possible to block all resources while trying to create them.

In addition to the differences listed above, Initializer had a number of known issues over a long period of development, including quota replenishment errors. The upgrade to beta indicates that Webhooks will be a target in the future. If you need more stable operations, I recommend using Webhooks.

How does MutatingAdmissionWebhook work

Be persisted to the front of the etcd MutatingAdmissionWebhook in resources, according to the rules of its request to intercept, blocking rule definition in MutatingWebhookConfiguration. MutatingAdmissionWebhook Implements changes to resources by sending access requests to webHook Servers. Webhook Server is just a simple HTTP server.

The following diagram details how MutatingAdmissionWebhook works:

MutatingAdmissionWebhook requires three objects to run:

MutatingWebhookConfiguration

MutatingAdmissionWebhook need according to MutatingWebhookConfiguration apiserver to register. During the registration process, MutatingAdmissionWebhook needs to state:

  1. How to connectwebhook admission server;
  2. How to validatewebhook admission server;
  3. webhook admission serverThe URL path;
  4. Webhook requires rules that operate on objects;
  5. webhook admission serverHow to handle errors encountered during processing.

MutatingAdmissionWebhook itself

MutatingAdmissionWebhook is an admission controller in plug-in form that can be configured into apiserver. MutatingAdmissionWebhook plug-in can be obtained from the MutatingWebhookConfiguration all interested in admission webhooks.

The MutatingAdmissionWebhook then listens for apiserver requests, intercepts those that meet the criteria, and executes in parallel.

Webhook Admission Server

Webhook Admission Server is just an HTTP Server attached to k8S Apiserver. For each apiserver request, MutatingAdmissionWebhook sends an admissionReview to the relevant Webhook Admission Server. Webhook Admission Server then decides how to change resources.

MutatingAdmissionWebhook tutorial

Writing a complete Webhook Admission Server can be daunting. For convenience, we write a simple Webhook Admission Server to inject nginx Sidecar containers and mount volumes. Complete code in kube-mutating-webhook-tutorial. This project references the Kubernetes Webhook example and Istio Sidecar injection implementation.

In the following paragraphs, I will show you how to write a working containerized Webhook Admission Server and deploy it to a Kubernetes cluster.

precondition

MutatingAdmissionWebhook Kubernetes version 1.9.0 and requests more than its admissionregistration k8s. IO/v1beta1 API is available. Make sure the following command:

kubectl api-versions | grep admissionregistration.k8s.io/v1beta1
Copy the code

Its output is:

admissionregistration.k8s.io/v1beta1
Copy the code

In addition, MutatingAdmissionWebhook and ValidatingAdmissionWebhook access controller need to be in the right order to join kube apiserver admission – control tag.

Write Webhook Server

Webhook Admission Server is a simple HTTP Server that follows the Kubernetes API. I pasted some pseudocode to describe the main logic:

sidecarConfig, err := loadConfig(parameters.sidecarCfgFile)
pair, err := tls.LoadX509KeyPair(parameters.certFile, parameters.keyFile)

whsvr := &WebhookServer {
    sidecarConfig:    sidecarConfig,
    server:           &http.Server {
        Addr:        fmt.Sprintf(":%v".443),
        TLSConfig:   &tls.Config{Certificates: []tls.Certificate{pair}},
    },
}
	
// define http server and server handler
mux := http.NewServeMux()
mux.HandleFunc("/mutate", whsvr.serve)
whsvr.server.Handler = mux

// start webhook server in new rountine
go func(a) {
    if err := whsvr.server.ListenAndServeTLS("".""); err ! =nil {
        glog.Errorf("Filed to listen and serve webhook server: %v", err)
    }
}()
Copy the code

A detailed explanation of the above code:

  1. sidecarCfgFileContains the Sidecar injector template, which is defined in ConfigMap below;
  2. certFile 和 keyFileIt’s a secret key pair. It’s gonna bewebhook serverTLS communication with Apiserver;
  3. Line 19 turns on HTTPS Server to listen on port 443'/mutate'.

Let’s focus on the main logic of handling the serve function:

// Serve method for webhook server
func (whsvr *WebhookServer) serve(w http.ResponseWriter, r *http.Request) {
	var body []byte
	ifr.Body ! =nil {
		if data, err := ioutil.ReadAll(r.Body); err == nil {
			body = data
		}
	}

	var reviewResponse *v1beta1.AdmissionResponse
	ar := v1beta1.AdmissionReview{}
	deserializer := codecs.UniversalDeserializer()
	if _, _, err := deserializer.Decode(body, nil, &ar); err ! =nil {
		glog.Error(err)
		reviewResponse = toAdmissionResponse(err)
	} else {
		reviewResponse = mutate(ar)
	}

	response := v1beta1.AdmissionReview{}
	ifreviewResponse ! =nil {
		response.Response = reviewResponse
		response.Response.UID = ar.Request.UID
	}
	// reset the Object and OldObject, they are not needed in a response.
	ar.Request.Object = runtime.RawExtension{}
	ar.Request.OldObject = runtime.RawExtension{}

	resp, err := json.Marshal(response)
	iferr ! =nil {
		glog.Error(err)
	}
	if_, err := w.Write(resp); err ! =nil {
		glog.Error(err)
	}
}
Copy the code

The serve function is a simple HTTP processor with HTTP Request and Response Writer parameters.

  1. First assemble the request intoAdmissionReview, includingobject,oldobject 及 userInfo.
  2. The Webhook main function is then triggeredmutateTo create apatchTo inject the Sidecar container and mount the volume.
  3. Finally, theadmission decisionAnd additionalpatchAssemble the response and send it back to apiserver.

You can do whatever you want with the implementation of the function Mutate. I’ll use my implementation as an example:

// main mutation process
func (whsvr *WebhookServer) mutate(ar *v1beta1.AdmissionReview) *v1beta1.AdmissionResponse {
	req := ar.Request
	var pod corev1.Pod
	iferr := json.Unmarshal(req.Object.Raw, &pod); err ! =nil {
		glog.Errorf("Could not unmarshal raw object: %v", err)
		return &v1beta1.AdmissionResponse {
			Result: &metav1.Status {
				Message: err.Error(),
			},
		}
	}
	
	// determine whether to perform mutation
	if! mutationRequired(ignoredNamespaces, &pod.ObjectMeta) { glog.Infof("Skipping mutation for %s/%s due to policy check", pod.Namespace, pod.Name)
		return &v1beta1.AdmissionResponse {
			Allowed: true, 
		}
	}

	annotations := map[string]string{admissionWebhookAnnotationStatusKey: "injected"}
	patchBytes, err := createPatch(&pod, whsvr.sidecarConfig, annotations)
	
	return &v1beta1.AdmissionResponse {
		Allowed: true,
		Patch:   patchBytes,
		PatchType: func(a) *v1beta1.PatchType {
			pt := v1beta1.PatchTypeJSONPatch
			return &pt
		}(),
	}
}
Copy the code

As you can see from the above code, the function mutate requests mutationRequired to determine whether the change is allowed. For permitted requests, the function mutate retrieves the modified body ‘patch’ from the other function createPatch. Note here function mutationRequired tricks, we skip with annotations sidecars – injector – webhook. Morven. Me/inject: true. This will be mentioned later when deploying deployment. See here for the full code.

Write a Dockerfile and build it

Create a build script:

dep ensure
CGO_ENABLED=0 GOOS=linux go build -a -installsuffix cgo -o kube-mutating-webhook-tutorial .
docker build --no-cache -t morvencao/sidecar-injector:v1 .
rm -rf kube-mutating-webhook-tutorial

docker push morvencao/sidecar-injector:v1
Copy the code

Write a Dockerfile file for dependencies as follows:

FROM alpine:latest

ADD kube-mutating-webhook-tutorial /kube-mutating-webhook-tutorial
ENTRYPOINT ["./kube-mutating-webhook-tutorial"]
Copy the code

To build the container manually, you need the Docker ID account and change the image name and tag (in the Dockerfile and deployme.yaml files) to your own, then execute the following command:

[root@mstnode kube-mutating-webhook-tutorial]#./build Sending build context to Docker daemon 44.89MB Step 3: FROM alpine:latest ---> 3fd9065eaf02
Step 2/3 : ADD kube-mutating-webhook-tutorial /kube-mutating-webhook-tutorial
 ---> 432de60c2b3f
Step 3/3 : ENTRYPOINT ["./kube-mutating-webhook-tutorial"]
 ---> Running in da6e956d1755
Removing intermediate container da6e956d1755
 ---> 619faa936145
Successfully built 619faa936145
Successfully tagged morvencao/sidecar-injector:v1
The push refers to repository [docker.io/morvencao/sidecar-injector]
efd05fe119bb: Pushed
cd7100a72410: Layer already exists
v1: digest: sha256:7a4889928ec5a8bcfb91b610dab812e5228d8dfbd2b540cd7a341c11f24729bf size: 739
Copy the code

Write Sidecar injection configuration

Now we create a Kubernetes ConfigMap with container and volume information to inject into the target pod:

apiVersion: v1
kind: ConfigMap
metadata:
  name: sidecar-injector-webhook-configmap
data:
  sidecarconfig.yaml: |
    containers:
      - name: sidecar-nginx
        image: Nginx: 1.12.2
        imagePullPolicy: IfNotPresent
        ports:
          - containerPort: 80
        volumeMounts:
          - name: nginx-conf
            mountPath: /etc/nginx
    volumes:
      - name: nginx-conf
        configMap:
          name: nginx-configmap
Copy the code

As you can see from the listing above, you need another ConfigMap that contains the Nginx conf. For full yamL see nginxconfigmap.yaml.

Then deploy the two ConfigMaps to the cluster:

[root@mstnode kube-mutating-webhook-tutorial]# kubectl create -f ./deployment/nginxconfigmap.yaml
configmap "nginx-configmap" created
[root@mstnode kube-mutating-webhook-tutorial]# kubectl create -f ./deployment/configmap.yaml
configmap "sidecar-injector-webhook-configmap" created
Copy the code

Create Secret that contains the Secret key pair

Since access control is a high-security operation, it is necessary to provide TLS for external Webhook Servers. As part of the process, we need to create a TLS certificate signed by Kubernetes CA to secure communication between Webhook Server and Apiserver. For the complete steps for CSR creation and approval, see here.

For simplicity, we referenced Istio’s script and created a similar script called webhook-create-signed-cert.sh to automatically generate certificates and key pairs and add them to secret.

#! /bin/bashwhile [[ $# -gt 0 ]]; do case ${1} in --service) service="$2" shift ;; --secret) secret="$2" shift ;; --namespace) namespace="$2" shift ;; esac shift done [ -z ${service} ] && service=sidecar-injector-webhook-svc [ -z ${secret} ] && secret=sidecar-injector-webhook-certs [ -z ${namespace} ] && namespace=default csrName=${service}.${namespace} tmpdir=$(mktemp -d) echo "creating certs in tmpdir ${tmpdir} " cat <<EOF >> ${tmpdir}/csr.conf [req] req_extensions = v3_req distinguished_name = req_distinguished_name [req_distinguished_name] [ v3_req ] basicConstraints = CA:FALSE keyUsage = nonRepudiation, digitalSignature, keyEncipherment extendedKeyUsage = serverAuth subjectAltName = @alt_names [alt_names] DNS.1 = ${service} DNS.2 = ${service}.${namespace} DNS.3 = ${service}.${namespace}.svc EOF openssl genrsa -out ${tmpdir}/server-key.pem 2048 openssl req -new -key ${tmpdir}/server-key.pem -subj "/CN=${service}.${namespace}.svc" -out ${tmpdir}/server.csr -config  ${tmpdir}/csr.conf
# clean-up any previously created CSR for our service. Ignore errors if not present.
kubectl delete csr ${csrName} 2>/dev/null || true

#create server cert/key CSR and send to k8s API
cat <<EOF | kubectl create -f -
apiVersion: certificates.k8s.io/v1beta1
kind: CertificateSigningRequest
metadata:
  name: ${csrName}
spec:
  groups:
  - system:authenticated
  request: $(cat ${tmpdir}/server.csr | base64 | tr -d '\n')
  usages:
  - digital signature
  - key encipherment
  - server auth
EOF

# verify CSR has been created
while true; do
    kubectl get csr ${csrName}
    if [ "$?" -eq 0 ]; then
        break
    fi
done

# approve and fetch the signed certificate
kubectl certificate approve ${csrName}
# verify certificate has been signed
for x in $(seq 10); do
    serverCert=$(kubectl get csr ${csrName} -o jsonpath='{.status.certificate}')
    if [[ ${serverCert} != '' ]]; then
        break
    fi
    sleep 1
done
if [[ ${serverCert} == '' ]]; then
    echo "ERROR: After approving csr ${csrName}, the signed certificate did not appear on the resource. Giving up after 10 attempts." >&2
    exit 1
fi
echo ${serverCert} | openssl base64 -d -A -out ${tmpdir}/server-cert.pem


# create the secret with CA cert and server cert/key
kubectl create secret generic ${secret} \
        --from-file=key.pem=${tmpdir}/server-key.pem \
        --from-file=cert.pem=${tmpdir}/server-cert.pem \
        --dry-run -o yaml |
    kubectl -n ${namespace} apply -f -
Copy the code

After running the script, the secret containing the certificate and the secret key pair is created:

[root@mstnode kube-mutating-webhook-tutorial]# ./deployment/webhook-create-signed-cert.sh creating certs in tmpdir /tmp/tmp.wXZywp0wAF Generating RSA private key, 2048 bit long modulus ........................................... + + +... +++ e is 65537 (0x10001) certificatesigningrequest "sidecar-injector-webhook-svc.default" created NAME AGE REQUESTOR CONDITION sidecar-injector-webhook-svc.default 0s https://mycluster.icp:9443/oidc/endpoint/OP#admin Pending certificatesigningrequest "sidecar-injector-webhook-svc.default" approved secret "sidecar-injector-webhook-certs" createdCopy the code

Create the Deployment and Service for the Sidecar injector

The Deployment comes with a POD which runs the Sidecar-Injector container. The container runs with special parameters:

  1. sidecarCfgFileThe configuration file for the Sidecar injector, mounted from the ConfigMap created abovesidecar-injector-webhook-configmap.
  2. tlsCertFile 和 tlsKeyFileIs a Secret key pair mounted from Secretinjector-webhook-certs.
  3. alsologtostderr,v=4 和 2 > &1Is a log parameter.
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: sidecar-injector-webhook-deployment
  labels:
    app: sidecar-injector
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: sidecar-injector
    spec:
      containers:
        - name: sidecar-injector
          image: morvencao/sidecar-injector:v1
          imagePullPolicy: IfNotPresent
          args:
            - -sidecarCfgFile=/etc/webhook/config/sidecarconfig.yaml
            - -tlsCertFile=/etc/webhook/certs/cert.pem
            - -tlsKeyFile=/etc/webhook/certs/key.pem
            - -alsologtostderr
            - -v=4
            - 2> &1
          volumeMounts:
            - name: webhook-certs
              mountPath: /etc/webhook/certs
              readOnly: true
            - name: webhook-config
              mountPath: /etc/webhook/config
      volumes:
        - name: webhook-certs
          secret:
            secretName: sidecar-injector-webhook-certs
        - name: webhook-config
          configMap:
            name: sidecar-injector-webhook-configmap
Copy the code

Service exposes the POD with app= Sidecar-Injector label and makes it accessible in the cluster. This Service will be defined MutatingWebhookConfiguration clientConfig part of access, the default port spec. The ports, the port needs to be set for 443.

apiVersion: v1
kind: Service
metadata:
  name: sidecar-injector-webhook-svc
  labels:
    app: sidecar-injector
spec:
  ports:
  - port: 443
    targetPort: 443
  selector:
    app: sidecar-injector
Copy the code

Then deploy the above Deployment and Service to the cluster and verify that the Webhook Server of the Sidecar injector is running:

[root@mstnode kube-mutating-webhook-tutorial]# kubectl create -f ./deployment/deployment.yaml
deployment "sidecar-injector-webhook-deployment" created
[root@mstnode kube-mutating-webhook-tutorial]# kubectl create -f ./deployment/service.yaml
service "sidecar-injector-webhook-svc" created
[root@mstnode kube-mutating-webhook-tutorial]# kubectl get deployment
NAME                                  DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
sidecar-injector-webhook-deployment   1         1         1            1           2m
[root@mstnode kube-mutating-webhook-tutorial]# kubectl get pod
NAME                                                  READY     STATUS    RESTARTS   AGE
sidecar-injector-webhook-deployment-bbb689d69-fdbgj   1/1       Running   0          3m
Copy the code

Dynamically configure the Webhook access controller

Which webhook MutatingWebhookConfiguration in the concrete expression of admission server is being used and what resources under the control of the access to the server. Suggest you before you create MutatingWebhookConfiguration deployment webhook admission server, and to ensure their normal work. Otherwise, the request will be accepted unconditionally or rejected according to the failure rule.

Now, we are according to the following content creation MutatingWebhookConfiguration:

apiVersion: admissionregistration.k8s.io/v1beta1
kind: MutatingWebhookConfiguration
metadata:
  name: sidecar-injector-webhook-cfg
  labels:
    app: sidecar-injector
webhooks:
  - name: sidecar-injector.morven.me
    clientConfig:
      service:
        name: sidecar-injector-webhook-svc
        namespace: default
        path: "/mutate"
      caBundle: ${CA_BUNDLE}
    rules:
      - operations: [ "CREATE" ]
        apiGroups: ["] ""
        apiVersions: ["v1"]
        resources: ["pods"]
    namespaceSelector:
      matchLabels:
        sidecar-injector: enabled
Copy the code

Line 8: name – The name of webhook, which must be specified. Multiple Webhooks are sorted in the order provided; Line 9: clientConfig – describes how to connect to webhook Admission Server and TLS certificates; Line 15: rules – describes the resources and operations handled by the WebHook Server. In our example, only requests to create Pods are intercepted; Line 20: namespaceSelector – namespaceSelector determines whether to send an access request to the Webhook Server for the resource based on whether the resource object matches the selector.

Before deploying MutatingWebhookConfiguration, we need to ${CA_BUNDLE} to replace the default apiserver caBundle. Let’s write a script to automatically match:

#! /bin/bashset -o errexit set -o nounset set -o pipefail ROOT=$(cd $(dirname $0)/.. /.. /; pwd) export CA_BUNDLE=$(kubectl get configmap -n kube-system extension-apiserver-authentication -o=jsonpath='{.data.client-ca-file}' | base64 | tr -d '\n') if command -v envsubst >/dev/null 2>&1; then envsubst else sed -e "s|\${CA_BUNDLE}|${CA_BUNDLE}|g" fiCopy the code

Then execute:

[root@mstnode kube-mutating-webhook-tutorial]# cat ./deployment/mutatingwebhook.yaml |\
>   ./deployment/webhook-patch-ca-bundle.sh >\
>   ./deployment/mutatingwebhook-ca-bundle.yaml
Copy the code

We do not see any logs describing webhook Server receiving an access request, as if the request was not sent to Webhook Server. So one possibility is that this is MutatingWebhookConfiguration triggered the configuration. To reconfirm MutatingWebhookConfiguration we will find the following content:

namespaceSelector:
      matchLabels:
        sidecar-injector: enabled
Copy the code

Control the Sidecar injector through namespaceSelector

Configuration in our MutatingWebhookConfiguration namespaceSelector, also means that must be met under the namespace of the conditions of resources that can be sent to the webhook server. Sidecar-injector =enabled this namespace has been labeled default;

[root@mstnode kube-mutating-webhook-tutorial]# kubectl label namespace default sidecar-injector=enabled
namespace "default" labeled
[root@mstnode kube-mutating-webhook-tutorial]# kubectl get namespace -L sidecar-injector
NAME          STATUS    AGE       sidecar-injector
default       Active    1d        enabled
kube-public   Active    1d
kube-system   Active    1d
Copy the code

Now we configure MutatingWebhookConfiguration will create in pod when injected into sidecars container. Delete running pods and verify that a new pod with a Sidecar container has been created:

[root@mstnode kube-mutating-webhook-tutorial]# kubectl delete pod sleep-6d79d8dc54-r66vz
pod "sleep-6d79d8dc54-r66vz" deleted
[root@mstnode kube-mutating-webhook-tutorial]# kubectl get pods
NAME                                                  READY     STATUS              RESTARTS   AGE
sidecar-injector-webhook-deployment-bbb689d69-fdbgj   1/1       Running             0          29m
sleep-6d79d8dc54-b8ztx                                0/2       ContainerCreating   0          3s
sleep-6d79d8dc54-r66vz                                1/1       Terminating         0          11m
[root@mstnode kube-mutating-webhook-tutorial]# kubectl get pod sleep-6d79d8dc54-b8ztx -o yaml
apiVersion: v1
kind: Pod
metadata:
  annotations:
    kubernetes.io/psp: default
    sidecar-injector-webhook.morven.me/inject: "true"
    sidecar-injector-webhook.morven.me/status: injected
  labels:
    app: sleep
    pod-template-hash: "2835848710"
  name: sleep-6d79d8dc54-b8ztx
  namespace: default
spec:
  containers:
  - command:
    - /bin/sleep
    - infinity
    image: tutum/curl
    imagePullPolicy: IfNotPresent
    name: sleep
    resources: {}
    volumeMounts:
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: default-token-d7t2r
      readOnly: true
  - image: nginx:1.12.2
    imagePullPolicy: IfNotPresent
    name: sidecar-nginx
    ports:
    - containerPort: 80
      protocol: TCP
    resources: {}
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /etc/nginx
      name: nginx-conf
  volumes:
  - name: default-token-d7t2r
    secret:
      defaultMode: 420
      secretName: default-token-d7t2r
  - configMap:
      defaultMode: 420
      name: nginx-configmap
    name: nginx-conf
...
Copy the code

You can see that the Sidecar container and volume have been successfully injected into the application. At this point, we have successfully created a running sidecar injector with MutatingAdmissionWebhook. With namespaceSelector, we can easily control whether pods in a specific namespace need to be injected into a Sidecar container.

There is a problem, however. According to the configuration above, all Pods under the default namespace will be injected into the Sidecar container, without exception.

Control the Sidecar injector through annotations

Thanks to the flexibility of The MutatingAdmissionWebhook, you can easily customize the change logic to filter resources with specific annotations. Remember the annotations sidecars mentioned above – injector – webhook. Morven. Me/inject: “true”? This can be used as another form of control in the Sidecar injector. In Webhook Server I wrote some logic to skip the line of pod without this annotation.

Let’s try it out. In this case, we create another application, sleep without annotation in its podTemplateSpec sidecars – injector – webhook. Morven. Me/inject: “true” :

[root@mstnode kube-mutating-webhook-tutorial]# kubectl delete deployment sleep
deployment "sleep" deleted
[root@mstnode kube-mutating-webhook-tutorial]# cat <<EOF | kubectl create -f -
apiVersion: extensions/v1beta1
> kind: Deployment
> metadata:
>   name: sleep
> spec:
>   replicas: 1
>   template:
>     metadata:
>       labels:
>         app: sleep
>     spec:
>       containers:
>       - name: sleep
>         image: tutum/curl
>         command: ["/bin/sleep"."infinity"]
>         imagePullPolicy: IfNotPresent
> EOF
deployment "sleep" created
Copy the code

Then verify that the Sidecar injector skipped the pod:

[root@mstnode kube-mutating-webhook-tutorial]# kubectl get deployment NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE sidecar-injector-webhook-deployment 1 1 1 1 45m sleep 1 1 1 1 17s [root@mstnode kube-mutating-webhook-tutorial]# kubectl  get pod NAME READY STATUS RESTARTS AGE sidecar-injector-webhook-deployment-bbb689d69-fdbgj 1/1 Running 0 45m sleep-776b7bcdcd-4bz58 1/1 Running 0 21sCopy the code

It turns out that the Sleep application contains only one container, with no additional containers or volume injection. We then annotate the Deployment and verify that it is injected into sidecar after rebuilding:

[root@mstnode kube-mutating-webhook-tutorial]# kubectl patch deployment sleep -p '{"spec":{"template":{"metadata":{"annotations":{"sidecar-injector-webhook.morven.me/inject": "true"}}}}}'
deployment "sleep" patched
[root@mstnode kube-mutating-webhook-tutorial]# kubectl delete pod sleep-776b7bcdcd-4bz58
pod "sleep-776b7bcdcd-4bz58" deleted
[root@mstnode kube-mutating-webhook-tutorial]# kubectl get pods
NAME                                                  READY     STATUS              RESTARTS   AGE
sidecar-injector-webhook-deployment-bbb689d69-fdbgj   1/1       Running             0          49m
sleep-3e42ff9e6c-6f87b                                0/2       ContainerCreating   0          18s
sleep-776b7bcdcd-4bz58                                1/1       Terminating         0          3m
Copy the code

As expected, pods are injected with additional Sidecar containers. At this point, we have a working Sidecar injector that can be controlled by namespaceSelector or finer grained by annotations.

conclusion

MutatingAdmissionWebhook is one of the simplest ways to extend Kubernetes functionality. It works by controlling resource changes with new rules.

This capability enables more work modes and supports more ecosystems, including the service grid platform Istio. Starting with Istio 0.5.0, the automatic injection section of Istio has been rewritten from initializers to MutatingAdmissionWebhook.

reference

  • Blog. Kubernetes. IO / 2018/01 / ext…
  • Docs.google.com/document/d/…
  • V1-8. Docs. Kubernetes. IO/docs/admin /…
  • Github.com/kubernetes/…