To be proficient in reading and developing, you probably need to know the following:

  • The K8S REST API model needs to be understood.
  • The HTTP protocol requires proficiency.
  • Understand openAPI specifications.
  • Good command of Linux.
  • Understanding of the theoretical framework of HELM.
  • Understanding of K8S CRD, controller concept, and RBAC authority model.
  • Understanding and use of HELM private library index.yaml.
  • Configure and use core DNS.

Overall structure drawing

What’s helm

Helm is the package manager for K8S and the de facto standard for complex application deployment on the K8S platform. Contains application packaging, deployment, upgrade, rollback, uninstall and other life-cycle management functions.

Architecture changes

Helm was upgraded from V2 to V3, removing an important component tiller and making the overall architecture simpler.

The deficiency of HELM architecture in cloud management platform development

Helm currently has no official GA API. The download, deployment, upgrade and uninstall of chart all depend on CLI. In a multi-cluster environment, the CLI cannot meet the service requirements of the platform.

By looking at github Issue, the community has two possible solutions:

  • Encapsulates the CLI as an API. Each cluster needs to deploy the helm binary file to the master node using SSH or Ansible, adding burden to the underlying deployment work.
  • CRD. Package helm’s core capabilities as Docker images and deploy them to k8S clusters to provide capabilities in the manner of Controllers. Use CRD to complete the deployment, uninstallation, upgrade, rollback and other business actions of release.

The biggest problem with CLI mode is that it does not conform to the idea of cloud native, and cli mode is locked with the helm version. If you want to upgrade the helm, you need to re-adapt the console content. The problem with CRD is that officially there is no GA yet. But still expect the controller approach.

Our team initially tried the first approach, but it didn’t work well. At that time, I happened to find that The Cloud had opened source helm V3 Controller Captain, so I made a second attempt based on the community Captain, and finally completed the function development.

At that time, a github search for Helm Controller found two warehouses, one provided by Rancher and one provided by Linsparrow cloud. After a simple test, Captain was installed and tested successfully at one time. Combined with internal discussion, we finally decided to develop based on Captain.

captain

Making: github.com/alauda/capt…

introduce

The Helm 3 Design Proposal exists for a while and currently it is still under heavy development. Captain comes as the first implementation of Helm v3 Controller based on the Proposal. This project is based on the core helm v3 code, acting as a library. Since it’s not officially released yet (alpha stage for now), some modifications were made to help implement this controller on a fork: alauda/helm (will be deprecated once Helm’s library is released).

Captain is the open source Helm V3 controller for Linsparrow cloud. Internally, it relies on helm Library. So the core logic is consistent with helm Client. After helm’s official GA, it can be migrated back to the official version, which is so easy for Java with interface oriented programming.

Open source based on Apache 2.0 protocol

Captain internally deploys helm into Deployment.

Quick installation test

Installation steps:

kubectl create ns captain-system
kubectl create clusterrolebinding captain --serviceaccount=captain-system:default --clusterrole=cluster-admin
kubectl apply -n captain-system -f https://raw.githubusercontent.com/alauda/captain/master/artifacts/all/deploy.yamlCopy the code

Uninstall:

kubectl delete -n  captain-system -f https://raw.githubusercontent.com/alauda/captain/master/artifacts/all/deploy.yaml
kubectl delete ns captain-systemCopy the code

Install nginx chart

kind: HelmRequest
apiVersion: app.alauda.io/v1alpha1
metadata:
  name: nginx-ingress
spec:
  chart: stable/nginx-ingressCopy the code

Viewing deployment Results

root@VM-16-12-ubuntu:~/demo# kubectl get pods
NAME                                             READY   STATUS    RESTARTS   AGE
nginx-ingress-controller-57987f445c-9rhv5        1/1     Running   0          16s
nginx-ingress-default-backend-7679dbd5c9-wkkss   1/1     Running   0          16s
root@VM-16-12-ubuntu:~/demo# kubectl get hr
NAME            CHART                  VERSION   NAMESPACE   ALLCLUSTER   PHASE    AGE
nginx-ingress   stable/nginx-ingress             default                  Synced   23sCopy the code

Chart repo issue

By default, Captain comes with stable’s official helm warehouse. There is no problem with helm’s official warehouse address, but if a docker image is used in chart image, it cannot be downloaded. When testing is using aliyun developer.aliyun.com/hub/ provide warehouse address. In this way, the Captain Controller can download the chart image successfully.

When the test is over, we need to connect K8S with chart private library on the Intranet and create a YAML file of ChartRepo

apiVersion: app.alauda.io/v1alpha1
kind: ChartRepo
metadata:
  name: cloud
  namespace: captain-system
spec:
  url: https://harbor.offline/chartrepo/libraryCopy the code

Then use kubectl create -f fileName to add it to K8S. It should be noted that harbor is used to manage the docker image and helm image. Because of the docker problem, we used the self-signed certificate. The certificate will be verified. We have communicated with the official and solved the problem. Now Captain has been ga and can be used directly without worrying about the certificate.

RBAC permission problem

In the management of cloud platform, we use K8S API through Servicecount. After installing Captain, we need an extra step to add permissions, which is different from using user Account in the command line

kubectl create clusterrolebinding default --serviceaccount=default:default --clusterrole=cluster-admin

Captain SDK problem

Captain only provides the GO and Python SDK. Based on this, we will definitely package a Captain Java SDK.

At the bottom of the architecture, we use the official SDK of K8S for development.

Based on the simple understanding of CRD and K8S OpenAPI, combined with the official instructions, we tried to carry out the action generated by SDK, but failed. See issue for details. We also contacted the author and learned that Captain did not make verification based on Schema, but used Webhook for internal verification. Based on this background, it is impossible to directly use OpenAPI specification to generate SDK, so we directly use kubectl-V9 to verify the message and develop the code.

When using kubectl command line for any operation, add -v9 parameter, you can obtain detailed HTTP packet information

root@master:/home/kylin# kubectl get pod -v9 I0414 22:42:53.981748 16582 loader.go:359] Config loaded from file: /root/.kube/config I0414 22:42:54.042173 16582 round_trippers. Go :419] curl -k -v -xget -h "Accept: application/json; as=Table; v=v1beta1; g=meta.k8s.io, application/json" -H "User-Agent: Kubectl/v1.15.5 (Linux/amd64) kubernetes / 20 c265f "https://192.168.4.139:6443/api/v1/namespaces/default/pods? Limit =500' I0414 22:42:54.077898 16582 round_trippers https://192.168.4.139:6443/api/v1/namespaces/default/pods? Limit =500 200 OK in 35 milliseconds I0414 22:42:54.077959 round_pppers. Go: 544] Response Headers: I0414 22:42:54.078006 16582 round_trippers. Go :447] Content-type: Application /json I0414 22:42:54.078054 16582 round_trippers. Go :447] Date: Tue, 14 Apr 2020 14:42:54 GMT I0414 22:42:54.078394 16582 Request. Go :947] {"kind":"Table","apiVersion":"meta.k8s.io/v1beta1","metadata":{"selfLink":"/api/v1/namespaces/default/pods","resourceVer Sion ":"14332801"},"columnDefinitions": Complete message too long, omitted! Go :564] no kind "Table" is registered for version "meta-k8s. IO /v1beta1" in scheme "k8s.io/kubernetes/pkg/api/legacyscheme/scheme.go:30" NAME READY STATUS RESTARTS AGE busybox 1/1 Running 970 39d nginx-1585049022-b4f4c56c9-dvspz 1/1 Running 24 12d nginx-deployment-5bd886c88c-28d6q 0/1 Pending 0 2d1h nginx-deployment-5bd886c88c-968pd 0/1 MatchNodeSelector 0 4d3h nginx-deployment-5bd886c88c-dnh8q 0/1 MatchNodeSelector 0  4d3h nginx-deployment-5bd886c88c-pk9xz 0/1 Pending 0 2d1hCopy the code

This is combined with the CustomObjectsApi provided by the k8s official Java SDK. It is relatively easy to develop a complete set of interfaces corresponding to the chart mirror life cycle.

The deployment of

var customObjectsApi = new  CustomObjectsApi(apiClient);
var json = new JsonObjectBuilder()  
        .set("apiVersion", "app.alauda.io/v1alpha1")  
        .set("kind", "HelmRequest")  
        .set("metadata", new JsonObjectBuilder().set("name", name).build())  
        .set("spec", new JsonObjectBuilder()  
                .set("chart", chart)  
                .set("namespace", namespace)  
                .set("releaseName", name)  
                .set("values", Map2JsonUtil.map2Json(params))  
                .set("version", version)  
        ).build();

customObjectsApi.createNamespacedCustomObject("app.alauda.io","v1alpha1", "default", "helmrequests", json, null);Copy the code

uninstall

var customObjectsApi = new  CustomObjectsApi(apiClient);
customObjectsApi.deleteNamespacedCustomObject("app.alauda.io", "v1alpha1", "default","helmrequests", "test-nginx",new  V1DeleteOptions().gracePeriodSeconds(0L).propagationPolicy("Foreground"),null, null, null);Copy the code

upgrade

Here you can choose to patch or directly replace, which is consistent with the concept of K8S.

The rollback

Unlike Deployment, Captain does not have the native support for rollback. It needs to save the parameters of each installation or upgrade externally, and then replace the parameters of the specified version to simulate rollback.

Other instructions

  • Overall, we completed the first version of the App Store and interface tuning in three weeks. This was much faster than expected using the CLI, and we only needed to add two additional lines to our Ansbile deployment script to install Captain.
  • With a private Helm REPO, by default, the coreDNS within the cluster forwards non-cluster addresses to the native/etc/resolv.conf, this time must ensure that the K8S host machine/etc/resolv.confChange the DNS address to the Intranet DNS server address. Otherwise the Captain Controller cannot find the private Helm repo, error timeout.
  • In the development process, if you encounter problems that cannot be located, you can directly view the log of captain-controller to handle them.
  • For network problems, it is best to deploy a BusyBox with built-in nsLookup, wGET and other tools. Facilitate network detection. 
      apiVersion: v1
  kind: Pod
  metadata:
    name: busybox
    namespace: default
  spec:
    containers:
    - name: busybox
      image: busybox:1.28.4
      command:
      - sleep
      - "3600"
      imagePullPolicy: IfNotPresent
      restartPolicy: AlwaysCopy the code

    usekubectl create -f busybox.yamlBusybox deployment is completekubectl exec -it busybox shEnter inside the container and usenslookup.wgetSuch as network detection.

    This article is published by OpenWrite, a blogging tool platform