The previous article introduced the role of Choerodon Agent based on HELM2 in the continuous delivery deployment pipeline of Toothfish platform and its implementation principle.

Now that the latest version of HELM has arrived at HELM3, continuing to use HELM2 will face the following problems:

The HELM2 version uses an older K8S API, which makes it difficult for Choerodon Agent to support older VERSIONS of K8S
The HELM2 architecture is a client-server architecture. Tiller-pod requires high cluster permissions, which is inconvenient for permission management in a cluster. At the same time, Choerodon Agent is a client, so it is not convenient to debug when problems occur

Therefore, Choerodon Agent needs to support HELM3 and migrate instances installed under HELM2.

Helm2 is different from HELM3

Tiller is deleted

As shown in the figure, deploying releases in HelM2 requires a Tiller-Pod, but in HelM3 the instance is deployed directly through KubeconFig

In HELM2, releases are global resources; in HELM3, releases are stored in their respective namespaces
Values supports the JSON Schema validator, which automatically checks all input variable formats
Removed the helm Serve command for local temporary Chart Repository
Helm Install no longer generates a Release name by default unless — generate-name is specified
Helm Cli changed its name individually

Helm2 to HELM3 migration

Helm2 to HELM3 migration includes the following steps:

Migrate helM2 configurations
Release migration of Helm2
Clear the helM2 configuration, release data, and Tileer Deployment

Data migration is performed using Helm-2TO3

Install the 2TO3 plug-in

helm3 plugin install https://github.com/helm/helm-2to3

Copy the code

Plug-in features

Supported functions:

Migrate helM2 configurations
Migration of Helm2 releases
Clear helM2 configuration, REL Ease data, and Tiller Deployment

Migrate helM2 configurations

First, you need to migrate the helM2 configuration and data folders, including the following:

Chart starters
Repositories
Plugins

Run the following command to start the migration:

helm3 2to3 move config

Copy the code

Migrate the helM2 instance

Run the following command to start the migration:

helm3 2to3 convert [ReleaseName]

Copy the code

Example Clear helM2 data

If no error occurs after the migration, you can run this command to clear helM2 data, including the following information:

Configuration(Helm homedirectory)
v2release data
Tiller deployment

Run the following command to clear the data:

helm3 2to3 cleanup

Copy the code

Note: If you run the clear command, all deleted data cannot be recovered. So if you don’t have to, keep the old data

Upgrade process of Choerodon Agent

The change from HELM2 to HELM3 is very large, so Choerdon Agent calls helm also change greatly. There are two parts that need to be changed

Obtaining, installing, upgrading, and uninstalling the HELM client requires reconstruction
The helM2 installed instances need to be migrated to HelM3; otherwise, Choerodon Agent cannot continue to manage the previous instances after the upgrade

Helm client refactoring

In helM2, Choerodon Agent directly uses helm source code as part of Choerodon Agent code. In HELM3, direct secondary development of helM3 source code, and then through dependency reference. The advantage of this is to decouple the HELM code from Choerodon Agent code, which is conducive to the update and upgrade of helm related code.

IO /release, choeroodn. IO /command, etc. Therefore, the secondary development of HelM3 is mainly to add resource label, for example, Install operation, other operations (upgrade, delete) are basically the same.

1. Change the module name

For secondary development, the first step is to change the module name of the project, which is also the most troublesome step because it requires modifying all the package reference paths in the code

As shown, the module in the go.mod file is modified as follows

github.com/choerodon/helm => github.com/open-hand/helm

Copy the code

Then modify the reference path in the code file

2. Add add label logic

Find helm3 through breakpoint debugging, installation logic by open – hand – helm/PKG/action/install. Go: : Run () method, insert tag step in this method. The unnecessary code is omitted below

func (i *Install) Run(chrt *chart.Chart, vals map[string]interface{}, valsRaw string) (*release.Release, error) {
  //... the omitted step is that Helm verifies the chart package rendering, generates the K8S object, and saves it in the Resources variable

    // The following changes are made, traversing the Resources object and adding the Label
    for _, r := range resources {
        err = action.AddLabel(i.ImagePullSecret, i.Command, i.AppServiceId, r, i.ChartVersion, i.ReleaseName, i.ChartName, i.AgentVersion, "", i.TestLabel, i.IsTest, false, nil)
        iferr ! = nil {return nil, err
        }
    }
  //... the omitted step is that Helm updates resources to the cluster
}

Copy the code

Then look at the open – hand – helm/PKG/agent/action/label. Go: : AddLabel () method

func AddLabel(imagePullSecret []v1.LocalObjectReference,
    command int64,
    appServiceId int64,
    info *resource.Info,
    version, releaseName, chartName, agentVersion, testLabel, namespace string,
    isTest bool,
    isUpgrade bool,
    clientSet *kubernetes.Clientset) error {
    // This method is more content, not shown here, can refer to the source code. This function is to add different Label values according to different resource types
}

Copy the code

3. Choerodon Agent references the helm library of secondary development

Refer to helM3 source code install command initialization method, will be divided into the following steps

Obtain helm configuration information

// Get helm configuration information
func getCfg(namespace string) (*action.Configuration, *cli.EnvSettings) {
settings := cli.New()
settings.SetNamespace(namespace)
actionConfig := &action.Configuration{}
helmDriver := os.Getenv("HELM_DRIVER")
iferr := actionConfig.Init(settings.RESTClientGetter(), settings.Namespace(), helmDriver, debug); err ! =nil {
    log.Fatal(err)
}
return action config, settings
}

Copy the code

Create the Install action object

	installClient := action.NewInstall(
		cfg,
		chartPathOptions,
		request.Command,
		request.ImagePullSecrets,
		request.Namespace,
		request.ReleaseName,
		request.ChartName,
		request.ChartVersion,
		request.AppServiceId,
		envkube.AgentVersion,
		"".false)

Copy the code

ValueOpts := getValueOpts(request. values) p := get. All(envSettings) vals, err := valueOpts.MergeValues(p) if err ! = nil { return nil, err }

// Check chart dependencies to make sure all are present in /charts chartRequested, err := loader.Load(cp) if err ! = nil { return nil, err }

validInstallableChart, err := isChartInstallable(chartRequested) if ! ValidInstallableChart {return nil, err} ··

4. Invoke the install method and Run the go ··· · responseRelease, err := installClient.Run(chartRequested, vals, request.Values) ···Copy the code

Specific logic can see choerodon – cluster – agent/PKG/helm/helm. Go: : InstallRelease ()

To sum up, there are four steps:

Obtain configuration objects -> Generate operation objects -> verify chart package and generate values -> Perform the operation

Migrate the installed Releases

The migration of Release requires the Helm migration tool, which is directly integrated into the Choerodon Agent project code.

In the startup logic of the Choeordon Agent, the first command received is agent_init. This command is responsible for namespace creation and listening, so the Release migration logic is placed in this step.

The whole process is shown in the figure below:

Begin from choerodon – cluster – agent/PKG/command/agent/agent. Go: : InitAgent () method to start ` ` ` go func InitAgent (opts * commandutil opts, CMD * model.packet) ([]* model.packet, * model.packet) {// ··· · The omitted step is to handle initialization parameters

For _, envPara := range agentInitOpts.Envs {nsList = append(nsList, nsList) envPara.Namespace) err := createNamespace(opts, envPara.Namespace, envPara.Releases) if err ! = nil { return nil, commandutil.NewResponseError(cmd.Key, cmd.Type, err) } }

// the omitted steps are to enable gitOPS listening, Controller listening, and return cluster information}

2. In [choerodon - cluster - agent/PKG/command/agent/agent. Go: : createNamespace ()] (HTTP: / / https://github.com/open-hand/choerodon-cluster- Agent/blob/master/PKG/command/agent/agent go# L196) began to initialize the namespace ` ` ` go func createNamespace (opts * commandutil opts, namespaceName string, releases []string) error { ns, err := opts.KubeClient.GetKubeClient().CoreV1().Namespaces().Get(namespaceName, metav1.GetOptions{}) if err ! = nil {// If the namespace does not exist, If errors.isnotFound (err) {_, err = opts.KubeClient.GetKubeClient().CoreV1().Namespaces().Create(&corev1.Namespace{ ObjectMeta: metav1.ObjectMeta{ Name: namespaceName, Labels: map[string]string{model.HelmVersion: Annotations := ns. annotations := ns. annotations // If the namespace exists, Then check the labels if _, OK := labels[model.helmversion]; ! Return update(OPts, Releases, namespaceName, Labels, Annotations)} return nil}Copy the code

In choerodon – cluster – agent/PKG/command/agent/agent. Go: : update () in the migrating instance ` ` ` go func update (opts * commandutil opts, releases []string, namespaceName string, labels, annotations map[string]string) error { releaseCount := len(releases) upgradeCount := 0

// Select * from choerodon; // select * from Choerodon; The Choerodon namespace is created directly without the model.helmVersion tag // The user then creates the PV directly, which causes the Choerodon to be included in the environment management without the label (a problem if Prometheus or cert-Manager is installed through the agent) // So the default choeordon does not require the helm migration if namespaceName ! = “choerodon” && releaseCount! = 0 { for i := 0; i < releaseCount; i++ { getReleaseRequest := &helm.GetReleaseContentRequest{ ReleaseName: releases[i], Namespace: namespaceName, }
```
/ / check to see if the instance helm3 management, if it is upgradeCount plus one, if not, the migration operation and then add 1 _, err: = opts. HelmClient. GetRelease (getReleaseRequest) if err! = nil {// The instance does not exist. Attempt to migrate if strings.Contains(err.error (), helm.ErrReleaseNotFound) { helm2to3.RunConvert(releases[i]) if opts.ClearHelmHistory { helm2to3.RunCleanup(releases[i]) Else {// instances exist means instances are managed by HELM3. Then upgradeCount+ 1 if opts.clearhelmHistory {helm2to3.RunCleanup(Releases [I])} upgradeCount++}} if releaseCount! = upgradeCount { return fmt.Errorf("env %s : failed to upgrade helm2 to helm3 ", namespaceName) }Copy the code
```
}

Labels = nil {labels = make(map[string]string)}

labels[model.HelmVersion] = "helm3"
_, err := opts.KubeClient.GetKubeClient().CoreV1().Namespaces().Update(&corev1.Namespace{
    ObjectMeta: metav1.ObjectMeta{
        Name:        namespaceName,
        Labels:      labels,
        Annotations: annotations,
    },
})
return err

Copy the code

This completes the migration logic of Choerodon Agent to Release

FAQ Resolution

Sometimes, after the Choerodon Agent is restarted, a startup failure occurs. In the log, instance migration fails and tiller-Pod does not have an error

The problem may be that the label of the namespace fails to be added after the instance migration is complete. The solution is to manually add the “Choerodon. IO /helm-version” : “HELm3” tag to the namespace to indicate that instances of the namespace have been migrated and do not need to be migrated again

After the resource is modified, the deployment fails and the error message timeout is displayed

Check whether the three commit values are consistent at the DevOPS environment level as follows:

Devops has a problem with GitLab’s Webhook callback and should be checked from GitLab’s Webhook execution records and Devops logs
The first two are consistent, and the third is behind the first two: Indicates that the DevOPS synchronization related GitOPS operation is complete, but there is a problem with the Choerodon Agent synchronization Gitops library.

Kubectl -n Choerodon delete [podName] Delete The Choerodon Agent and restart it.

If Choerodon Agent has not synchronized the Gitops library after the restart is complete, you should consider whether the key of the environment library in GitLab is the same as that stored in the Devops database. Verify whether the gitOPS library can be pulled by this key locally. If not, the best way to reset the gitOPS library key is to delete the environment and start again. If yes, it indicates that there is a problem with the connection between Choerodon Agent and GitLab. Check the port development of GITLab and the access and connection between Choerodon Agent and the external network

All three Commits are consistent, but the instance deployment fails and the access to the Chart repository is prompted to time out: this is a common problem. After investigation, it was found that the network connection between Choerodon Agent and Chart Museum was faulty

This article was originally written by the toothfish technical team.The official website of toothfish

About the toothfish

Choerodon’s full-scene performance platform provides systematic methodology and collaboration, testing, DevOps and container tools to help enterprises streamline their requirements, design, development, deployment, testing and operations processes to improve management efficiency and quality in a one-stop shop. From team collaboration to DevOps tool chain, from platform tools to systematic methodology, toothfish fully meets the needs of collaborative management and engineering efficiency, running through the whole end-to-end process, helping team efficiency become faster, stronger and more stable. Poke here to try the toothfish

Full scene efficiency platform Pig Toothfish Agent — Helm component upgrade

Helm2 is different from HELM3

Helm2 to HELM3 migration

Data migration is performed using Helm-2TO3

Install the 2TO3 plug-in

Plug-in features

Migrate helM2 configurations

Migrate the helM2 instance

Example Clear helM2 data

Upgrade process of Choerodon Agent

Helm client refactoring

1. Change the module name

2. Add add label logic

3. Choerodon Agent references the helm library of secondary development

Migrate the installed Releases

FAQ Resolution

Related Posts

Crawler get started to give up series 07: JS obconfusion, eval encryption, font encryption three anti – crawling technology

Pycharm, Idea and Goland have been officially sinicized

ROC curve was used to assess customer churn warning model | more challenges