The author | with science Alibaba senior engineer

This article is compiled from lecture 24 of “CNCF X Alibaba Cloud Native Technology Open Course”. Click “Read the original article” to direct to the course page.

Pay attention to “Alibaba cloud original” public number, reply keywords “entry”, you can download K8s series of articles PPT from zero entry.

This article will illustrate how to extend the Kubernetes API with the help of Operator development framework. The content is mainly divided into three parts: Firstly, the knowledge related to Operator will be briefly introduced; Then the Operator development framework will be introduced and the whole development process will be explained in detail with a case study. Finally, we will reillustrate how the Operator works in light of the workflow of the case.

I. Operator Overview

The basic concept

First, introduce the basic concepts involved in the content of this article.

  • CRD (Custom Resource Definition): allows users to customize Kubernetes resources, is a type;
  • CR (Custom Resourse): a concrete instance of CRD;
  • Webhook: This is essentially an HTTP callback registered with Apiserver. When an Apiserver specific event occurs, the registered Webhooks are queried and the corresponding message is forwarded.

According to the different processing types, it can be generally divided into two categories: one may modify the incoming object, called mutating Webhook; A class that reads only incoming objects is called Validating Webhook.

  • Work queues: The core component of controllers. It monitors resource changes in the cluster and stores related objects, including their actions and keys, such as a Create action of Pod, as an event in the queue.
  • Controller: It cycles through the above work queues, pushing the cluster state towards the desired state according to its own logic. Different controllers deal with different types. Replicaset controllers, for example, focus on the number of copies and handle poD-related events.
  • Operator: Operator is a mechanism for describing, deploying, and managing Kubernetes applications. In terms of implementation, it can be understood as CRD with optional Webhook and Controller to implement user business logic. Operator = CRD + Webhook + Controller.

Common operator mode

Workflow:

  1. The user creates a custom resource (CRD);
  2. Apiserver forwards the CRD request to Webhook based on a list of passes it has registered.
  3. Webhook generally does the default setting and parameter verification for this CRD. After webhook processing, the corresponding CR will be written to the database and returned to the user.
  4. At the same time, the controller monitors the custom resource in the background and processes the special operations associated with the custom resource according to the business logic.
  5. The above processing generally causes state changes within the cluster, and the controller monitors these associated changes and records these changes in the CRD state.

Here is a high-level overview, which will be reorganized in combination with cases.

Second, operator Framework actual combat

Operator framework overview

Before we begin, let’s introduce the Operator Framework. In fact, it provides the framework of Webhook and Controller for users. Its main significance lies in helping developers to shield some common low-level details. It does not need developers to realize message notification trigger, failure re-join the team, etc., and only needs to pay attention to the operation and maintenance logic implementation of managed applications.

There are two main operator frameworks: KubeBuilder and Operator-SDK.

There’s really no essential difference between the two, and at their core they both use the official Controller-Tools and Controller-Runtime. However, the details are slightly different. For example, KubeBuilder has more complete testing and deployment and code generation scaffolding. The operator-SDK has better support for upper level operations such as Ansible Operator.

Kuberbuildere of actual combat

The actual combat here is kuberBuilder. The case uses SidercarSet under the Kruise project of Ali Cloud.

SidercarSet is responsible for the Pod to insert sidecar container (also called secondary container), for example, can insert some monitoring, log collection to enhance the Pod function. The SidercarSet is then updated in reverse to record the state of these auxiliary containers based on the inserted state and the state of the Pod.

Step 1: Initialize

Action: Create a new GitLab project and run “KubeBuilder init –domain=kruise.io”.

Domain Specifies the Group domain of the CRD object to be registered later.

Pull depends on code base, generate code framework, generate Makefile/Dockerfile and other tool files.

Let’s take a look at what it actually generates _(the actual build file has been partially deleted for demonstration purposes) :

The specific content we can be in actual combat when their own detailed confirmation.

Step 2: Create the API

Operation: Running “KubeBuilder create API –group apps –version v1alpha1 –kind SidecarSet –namespace=false” actually creates not only the API, which is CRD, It also generates the skeleton of the Controller.

-group + domian = CRD group: apps.kruise.io;

  • Version is generally divided into three types according to community standards:
    • V1alpha1: this API is unstable, CRD may be discarded, fields may be adjusted at any time, do not rely on;
    • V1beta1: API is stable, backward compatibility will be guaranteed, features may be adjusted;
    • V1: API and features are stable;
  • Kind: This type of CRD is similar to the community native Service concept;
  • Namespaced: This CRD is globally or namespace-unique, similar to Nodes and pods.

Its parameters fall into two basic categories. Group, Version, and kind basically correspond to the three important components of CRD meta-information. Here are some common standards that you can refer to when you actually use them. Namespaced is used to specify whether the CRD we just created is globally unique (like Node) or namespace-unique (like Pod). False is used here to specify that SidecarSet is globally unique.

Effect interpretation:

The framework for CRD and Controller is generated, followed by manual padding.

The actual result is shown below:

Let’s focus on the blue part of the picture. Sidecarsettypes.go is where the CRD is defined and we need to fill it in. Sidecarsetcontroller. go is used to define controllers and also needs to be populated.

Step 3: Fill CRD

1. Generate the CRD is located in the “PKG/apis/apps/v1alpha1 / sidecarset_types go”, usually need to carry on the following two operations:

(1) Adjust notes

Code Generator relies on annotations to generate code, so annotations need to be adjusted sometimes. Here is a list of SidecarSet notes that need to be adjusted in this field:

+ genClient :nonNamespaced: generates non-namespace objects; + kubebuilder: subresource: status: status child resources; +kubebuilder:printcolumn:name=”MATCHED”,type=’integer’,JSONPath=”.status.matchedPods”,description=”xxx”: Kubectl get sidecarset

(2) Fill the field

The padding field is an important part of making the user’s CRD actually valid and meaningful.

  • SidecarSetSpec: Fills the CRD description.
  • SidecarSetStatus: Fills the CRD status information.

2. Run make to regenerate the code

It should be noted that the r&d personnel do not need to participate in the CRD GRPC interface, codec and other low-level implementation of controller.

The actual fill is shown below:

To do this, we define two main pieces of information in the SidecarSetSpec (left) : a Selector that selects which pods to inject; One is to define Containers for Sidecar Containers.

The status information is defined in SidecarSetStatus (right), where MatchedPods reflect how many pods the SidecarSet actually matches, and UpdatedPods reflect how many pods have been injected, ReadyPods reflect how many pods are working.

You can refer to this document for the complete content.

Step 4: Generate the Webhook framework

1. Generate mutating webhook, run:

“kubebuilder alpha webhook –group apps –version v1alpha1 –kind SidecarSet –type=mutating –operations=create”

“kubebuilder alpha webhook –group core –version v1 –kind Pod –type=mutating –operations=create”

2. Generation of Valhook, run:

“kubebuilder alpha webhook –group apps –version v1alpha1 –kind SidecarSet –type=validating –operations=create,update”

Parameter interpretation:

  • Group /kind describes the resource object to be processed.
  • Type describes what type of framework needs to be generated;
  • Operations Describes operations that concern resource objects.

Effect interpretation:

  • A Webhook framework is generated, followed by manual padding code

The actual generated results are as follows:

We executed three commands that generated three different handlers to populate (shown in blue). I won’t mention it here, but will explain it in detail in the next step of the fill operation.

Step 5: Populate the Webhook

The generated Webhook handlers are located in:

  • – pkg/webhook/defaultserver/sidecarset/mutating/xxxhandler.go
  • – pkg/webhook/defaultserver/sidecarset/validating/xxxhandler.go
  • – pkg/webhook/defaultserver/pod/mutating/xxxhandler.go

Generally, rewriting and filling include the following two parts:

  • Whether the K8s client needs to be injected depends on whether additional resources are needed in addition to the incoming CRD. In this case, we need to inject Pod and SidecarSet, so we need to inject K8s client.
  • The key method to populate webhooks is mutatingSidecarSetFn or validatingSidecarSetFn. Since the pointer to the resource object to be operated on has already been passed in, the hook is done by directly adjusting the property of the object.

Let’s look at the actual filling result.

Sidecarset mutating, Sidecarset mutaing and pod mutating are defined in step 4.

First, setDefaultSidecarSet mutating. This is what mutaing most often does.

In the figure on the right side validating is also very standard and validates some fields in SidecarSet.

Pod Mutaing is not shown here, it’s a little different, mutaingSidecarSetFn is not set by default, it gets the value of setDefaultSidecarSet and inject it into pod.

Step 6: Populate the controller

Generated by the controller framework in PKG/controller/sidecarset/sidecarset_controller. Go. There are three main points that need to be modified:

  • Modify permission comments. //+ kuberBuilder :rbac; groups=apps,resources=deployments/status,verbs=get; update; Path annotations, which we can modify to suit our own needs, will eventually generate RBAC rules;

 

  • Added team logic. The default code framework will fill in the enqueue logic of CRD itself (for example, the addition, deletion and modification of SidecarSet object will be added to the work queue). If the triggering mechanism of resource object needs to be associated (for example, SidecarSet also needs to pay attention to the change of Pod), it needs to manually add its enqueue logic.

 

  • Populate the business logic. Modify the Reconcile function to loop through the work queue. The Reconcile function mainly does two parts: “Complete the business logic according to the Spec” and “feedback the business logic result back to status”. Note that if the Reconcile function returns ERR in error, the team will be rejoined by default.

Let’s look at the result of populating the Controller of the SidecarSet:

AddPod first retrieves the Pod’s corresponding SidecarSet and queues it for reconciliation processing.

Reconcile SidercarSet is taken out and matched pods are selected based on Selector. Finally, the cluster state is calculated based on the current status information of the Pod. The state of the cluster is then backcoded into the CRD state.

3. Work flow of SidecarSet

Finally, let’s go through the SidecarSet workflow again to understand how the operator works.

  1. The user creates a SidecarSet.
  2. After receiving the SidecarSet, Webhook sets the default value and verifies configuration items. After these two operations are complete, the actual entry is completed and returned to the user;
  3. The user creates a Pod;
  4. Webhook takes back the corresponding SidecarSet and takes the Container out of it and pours it into the Pod, so the Pod is already loaded with the sidecar.
  5. The controller constantly polls in the background to see how the cluster status changes. The injection in step 4 triggers the enlistment of the SidecarSet, and the Controller increments the Update Pods of the SidecarSet by one.

This is a simple implementation of SidecarSet early on.

Here we add one more question. Generally webhooks are handled by controllers for business logic and status updates, but this is not required, and one or the other may not be required. In the example above, the main business logic is done by Webhook without the controller’s involvement.

Iv. Summary of this paper

This is the end of the main content of this article, here is a simple summary for you:

  • Operator is CRD with optional Webhook and controller, under Kubernetes system to extend user business logic a set of mechanisms;
  • Kubebuilder is an official, standardized Operator framework with high community recognition.
  • It is easy to implement an Operator by filling in the user-defined code as described above.

“Alibaba Cloud originators pay close attention to technical fields such as microservice, Serverless, container and Service Mesh, focus on cloud native popular technology trends and large-scale implementation of cloud native, and become the technical circle that knows most about cloud native developers.”