An overview of the

User services need to migrate images in batches in the upper cloud system or during cloud migration. Based on this background, Tencent cloud container expert team developed image batch migration tool: Image-Transfer. The tool supports batch migration between a variety of cloud vendors’ image warehouses, and supports Tencent cloud image warehouse TCR personal VERSION CCR one-click full migration to Tencent cloud image warehouse enterprise version TCR. This article begins with an introduction to the pain points of mirror migration in the cloud/migration process on the business. Then, the design idea, function modules and best practices of image-Transfer are introduced in detail. The tool has been officially open source, project address: github.com/tkestack/im…

Binary package download address: github.com/tkestack/im…

How to migrate an image in the cloud service?

Service cloud mainly includes the following scenarios: self-built IDC cloud, third-party cloud migration, and remote DISASTER recovery (hybrid cloud). In these scenarios, whether cloud or cloud migration, the migration process mainly consists of the following steps.

  1. Network planning. VPC configuration and subnet division.
  2. Data migration. Object storage, file storage, etc.
  3. Application migration. Services, configurations, etc.

During data migration, if a large number of services use containerized deployment, batch image migration is required. At present, most cloud vendors’ mirror warehouse services do not provide the ability of batch migration of mirror warehouses. To migrate services, download the original image, change the tag, and upload the image to the destination image repository. This process has the following disadvantages:

  • Time consuming and energy consuming. For a single mirror migration, there are three steps and you need to keep an eye on it, and if it fails, you need to deal with it further.
  • Complete human operation, prone to errors. When modifying the tag of the original image, it is easy to make errors due to manual modification.
  • When the number of mirrors is large, the cost increases and the cloud ascending progress is slow. If there are hundreds or thousands of mirrors, such human operation, the migration progress will be very slow, and the migration cost will greatly increase.

The principle of image – transfer

Design goals

Pain points for mirror migration. We want to develop an image batch migration tool. For users, it only needs simple configuration to realize batch migration of images without manual intervention, improve the cloud/migration progress of services and reduce costs. Specifically, the tool has the following design goals.

  1. Simple configuration and no complex input is required. We hope that this tool is easy to use for users, just need to configure the source and destination mirror address and authentication, to complete the batch migration.

  2. Fast and efficient migration of massive images, reducing migration costs. For a large number of mirror migration requirements, we want the tool to be efficient and fast with the help of high concurrency technologies.

  3. Certain fault tolerance ability, reduce manual intervention maintenance. During the migration process, we hope that the tool can do some error identification and automatic repair, minimize manual maintenance, improve operation and maintenance efficiency.

  4. There is no dependency on the operating environment, improving the universality of the tool. We want the tool to run on any Linux, MAC OS, or Windows operating system, without having to rely on Docker and other programs.

  5. Support Tencent cloud CCR one-click full migration mode. At present, Tencent cloud container image service TCR enterprise version officially launched, Tencent Cloud TCR personal version (CCR) will gradually reduce maintenance in the future, until offline. Therefore, tools need to support full one-click migration of CCR repositories to TCR Enterprise Edition.

  6. Supports custom QPS speed limit. Batch migration of mirrors and frequent invocation of the mirror warehouse interface may cause the crash of the mirror warehouse. Therefore, QPS must be restricted.

architecture

Image-transfer is composed of multiple modules. The following figure shows the image-Transfer architecture diagram.

  • General mode input (default) : used to accept image migration tasks delivered by users. Includes image migration configuration files and authentication configuration files. This pattern is used to migrate images between cloud vendors.

  • CCR One-click Migration mode input: You need to add –ccrToTcr=true to the tool input parameter. This mode is used for one-click migration of CCR repositories to TCR Enterprise Edition. In addition to adding the –ccrToTcr=true parameter, you also need the authentication profile and Tencent cloud secret profile.

  • Pipeline: This module is the core of the tool to handle image migration. Processes the image migration tasks delivered by users, including processing the synchronization rules of the image warehouse according to the migration configuration file, and performing hierarchical image pull and transfer tasks. The high concurrency pipeline model is adopted to improve the migration speed.

  • Retry Task: This phase retries tasks that failed to transfer in the pipeline. The number of retries depends on user input parameters. The default value is 2.

Pipeline

As the tool is written in Golang language, the core processing module adopts the go pipeline high concurrency model. The whole pipeline module is divided into three small modules.

Multiple coroutines handle mirror warehouse synchronization rules

The image migration configuration file entered by the user is processed and parsed to obtain the address (including repO and tag) of each source image to be transmitted and the corresponding address of the destination image. Then, a job is created for each source mirror address and destination mirror address. If the tag of the source mirror is not specified in the mirror configuration file, all the tags under the repO are pulled and a job is created for each tag. Golang’s multi-coroutine method is used in this process to increase the processing speed. The number of coroutines can be specified by the user in the input parameter –proc, default is 5. After each job is created, it is put into the Task Channel and waits to be consumed.

Task transport channel

The Task channel can be seen as a simple piece of middleware, implemented by Golang’s channel, where every job produced is put to be consumed. This design can ensure that the job produced by the producer will be immediately put into the consumption line. Once the consumption end is free, the consumption can be processed. Improve job processing efficiency.

Multiple coroutines process tasks

These coroutines are the consumer side of the job. After getting a job, the system first pulls the manifest of the source address of the job to determine whether it is a multiple MANIFEST image, then pulls each BLOB, transfers the BLOB to the destination address, and finally transfers the MANIFEST to the destination address. The whole process uses caching. Data does not fall, improve efficiency. Golang’s multi-coroutine method is used in this process to increase the processing speed. The number of coroutines can be specified by the user in the input parameter — ROUTINES, which defaults to 5.

Mirror migration best practices

This section describes how to use image-Transfer to migrate images in batches in different scenarios. The scenarios are as follows:

  1. Image migration between different cloud vendors. For example, from Ali cloud image warehouse ACR to Tencent cloud image warehouse TCR.

  2. Migrate the open source/self-built image warehouse to the cloud. For example, from harbor mirror warehouse to Tencent cloud mirror warehouse TCR.

  3. Tencent Cloud TCR Personal edition (CCR) one-click migration to Tencent Cloud image warehouse enterprise EDITION TCR.

Tool installation:

First of all, the tool is downloaded and compiled, there are two ways, one is to directly obtain binary files, the second is to download the source code compiled.

Download binary release package:

Github.com/tkestack/im…

Examples are compiled by downloading source code to demonstrate:

The compilation process is also very simple, enter the source directory, directly make.

git clone https://github.com/tkestack/image-transfer.git
cd ./image-transfer
make
Copy the code

After compiling, the image-Transfer binaries are generated in the current directory. Ready to use. Next, a best practice demonstration.

Best practice Scenario 1: Mirror migration between different cloud vendors

Take the migration from Ali cloud mirror warehouse ACR to Tencent cloud mirror warehouse TCR as an example.

1. Prepare auth.json access certificate information file of TCR of Tencent Cloud image warehouse and ACR of Ali cloud image warehouse
   grant-test.tencentcloudcr.com:
   username: xxx
   password: xxx
   grant-test2.tencentcloudcr.com:
   username: xxx
   password: xxx
   registry.cn-hangzhou.aliyuncs.com:
   username: xxx
   password: xxx
   ccr.ccs.tencentyun.com:
   username: xxx
   password: xxx
   registry.hub.docker.com:
   username: xxx
   password: xxx
Copy the code

The configuration is simple. Enter the address of the source mirror warehouse and the address of the destination mirror warehouse. Enter the user name and password of the mirror repository.

Where insecure indicates whether registry is an HTTP service. If so, the insecure field needs to be true and the default is false, which is optional.

The user of the destination mirror warehouse needs to have push and create warehouse permissions, if not provided, the default is anonymous access.

The TCR access certificate of Tencent Cloud can be obtained as follows:The access certificate of Ali Cloud mirror warehouse ACR is obtained as follows:

2. Prepare the image rule file rule-yaml to be migrated
registry.cn-hangzhou.aliyuncs.com/grantzhao/sichenzhao:xx": "grant-test.tencentcloudcr.com/grantzhao/sichenzhao
Copy the code

This file is the source image and destination image to be transferred. The file rules are as follows: Source mirror address: destination mirror address The source mirror address can be tag, no tag, or multiple tags. When a single tag is specified: The destination address may or may not contain the tag. If no tag is included, the tag of the source image is used. If no tag is specified: The destination address must contain a tag. Specifies how the tag: when multiple tag between commas in English, such as grant-test.tencentcloudcr.com/grantzhao/sichenzhao:1.0, 2.0, 3.0. In this case, the destination address cannot contain a tag. By default, the source address tag is used.

3. Run the tool
   ./image-transfer --routines=5 --securityFile=./security.yaml --ruleFile=./rule.yaml --ns=default \
   --registry=grant-test.tencentcloudcr.com --retry=2 --qps=100
Copy the code

Parameter Description:

–ns Specifies a default NS. If the NS of the destination repository is empty, the default NS is used instead.

–registry specifies a default registry that will be replaced if the destination repository registry is empty.

–routines=5, indicating that the number of concurrent requests is set to 5 The default value is 5.

— Retry =2, which indicates that the number of retries is 2. The default value is 2.

–securityFile: specifies the authentication file.

–ruleFile, specifies the mirror repository configuration file.

— QPS, which limits the QPS of requests to 100/s or less.

4. Running result

The last line

################# Finished, 0 transfer jobs failed, 0 jobs generate failed #################
Copy the code

Indicates that the operation is successful.

Best practice Scenario 2: Moving an open source/self-built mirror repository to the cloud

Take the migration from open source image warehouse Docker Hub to Tencent cloud image warehouse TCR as an example.

1. Prepare the access certificate information file of Docker Hub and TCR of Tencent cloud image warehouse: security.yaml
   grant-test2.tencentcloudcr.com:
   username: xxx
   password: xxx
   registry.hub.docker.com:
   username: xxx
   password: xxx
Copy the code
2. Prepare the image rule file image.json to be migrated
   sichenzhao/private-test:xxx": "grant-test2.tencentcloudcr.com/grantzhao/sichenzhao
Copy the code
3. Run the tool
   ./image-transfer --routines=5 --securityFile=./security.yaml --ruleFile=./rule.yaml --ns=default \
   --registry=grant-test.tencentcloudcr.com --retry=2
Copy the code
4. Running result

The last line

################# Finished, 0 transfer jobs failed, 0 jobs generate failed #################
Copy the code

Indicates that the operation is successful.

Best practice Scenario 3: One-click migration of Tencent Cloud TCR Personal Edition (CCR) to Tencent Cloud Image Warehouse Enterprise EDITION TCR

The usage in this scenario is slightly different from the above two scenarios. The main performance is the change of input parameters.

1. Prepare the image authentication configuration file security.yaml
   grant-test.tencentcloudcr.com:
   username: xxx
   password: xxx
   grant-test2.tencentcloudcr.com:
   username: xxx
   password: xxx
   ccr.ccs.tencentyun.com:
   username: xxx
   password: xxx
Copy the code
2. Prepare the Tencent cloud secret configuration file secret.yaml

For TCR’s one-click migration mode, the user name and password of the warehouse are not required as access authentication, but the Secret information of Tencent cloud.

   ccr:
   secretId: xxx
   secretKey: xxx
   tcr:
   secretId: xxx
   secretKey: xxx
Copy the code

Note: The file format is as shown above, only the secretId and secretKey entries can be modified.

If CCR’s secret information is not available, TCR’s will be used instead. On the contrary, if there is no SECRET information of TCR, CCR will be used instead.

Secret information can be obtained as follows:Contains secretid and secretkey information

3. Run the tool

The parameter input here is slightly different from the above two scenarios.

   ./image-transfer --ccrToTcr=true --routines=5 --securityFile=./security.yaml --secretFile=./secret.yaml --tcrName=tcr-test \
   --retry=3 --tcrRegion=ap-guangzhou --ccrRegion=ap-guangzhou --qps=3000
Copy the code

Parameter Description:

–ccrToTcr=true: TCR one-click full migration mode is enabled.

–secretFile, providing secret.yaml configuration file.

–tcrName= tCR-test, specifies the name of the destination TCR repository.

–tcrRegion: Specifies the region where the destination TCR warehouse is located.

–ccrRegion, which specifies the region where the source CCR repository resides.

4. Running result

One-click batch migration takes a long time because all images of CCR need to be transferred to TCR.

As you can see, 16 jobs failed. Finally, the tool lists the source and destination mirror addresses of the failed job. After checking the failed jobs in the warehouse, it is found that the tags of these jobs are invalid. Therefore, the transfer failed.

conclusion

This paper introduces the image batch transfer tool: image-Transfer in detail from the aspects of problem analysis, design objectives, principle analysis and best practices. Everyone is welcome to contribute source code, also welcome to issue request.