Build a hybrid cloud online data synchronization based on Kafka transit -CloudCanal actual combat

junyu-cloudcanal

This article introduces how to implement mysql synchronization data to Kafka to achieve hybrid cloud online data synchronization

Introduction to the

This paper mainly describes how to use CloudCanal to quickly build a secure hybrid cloud online data ecology, which is fully achieved by using CloudCanal SAAS capabilities.

This scheme has the following characteristics

Using Kafka to do data transfer
The public network port is disabled for both databases
Internet data is encrypted using SSL
The data egress is authenticated by the user name and password, and an IP address whitelist is configured to enhance security control

In the example, the cloud database, cloud messaging product, and self-built database can be replaced with self-built resources or various cloud resources in your current environment.

Technical point

The main difficulty of hybrid cloud data ecology is network security. Some users are concerned about traffic resource loss due to the large amount of synchronized data transmitted.

The solution based on CloudCanal pays more attention to the problems at the network security level and achieves one-way isolation of sensitive resource network, link authentication and transmission encryption.

Here is a brief exampleInternet model 和 Line modeData goes up and down the cloud.

For example, “chestnuts”

This case mainly demonstrates the Internet model of cross-cloud data migration and synchronization, the specific scenario is how to carry out data on the cloud (self-built room database -> Ali Cloud cloud database), and long-term maintenance of hybrid cloud data system. Of course, the same scheme can also be used in reverse, but in the data source selection of the external network Intranet is different.

Install CloudCanal

Prepare an ECS and a vm in the self-built equipment room, and install the ECS client according to the ECS client installation document and the self-built machine client installation document.

For Kafka

Buy page in Aliyun Kafka

Purchase the corresponding Kafka, verification capabilities can be purchased in quantity instances
- Select a public network /VPC instance and select a large public network traffic
Select a deployment instance2.x.xVersion,Maximum message sizeSlightly larger (say 4MB) is recommended.
Enter the instance,To create a Consumer Group,“And write down the name

Subaccount authorization and add data source

In accordance with theAli Cloud sub-account preparationDocument, create or authorize subaccounts, andRemember the sub-accounts AK and SKGrant basic database accessAliyunRDSFullAccess.AliyunKafkaFullAccess
Add respectivelySelf-built database under the cloud,Aliyun RDS for MySQL 和 Ali cloud Kafka
- When adding Aliyun resources, select Automatically add migration machine whitelist in the second step
- The Kafka user name and password can be found in the security configuration section at the bottom of aliyun Kafka instance details page
- Please download the TLS file from the Aliyun Kafka root certificate

Start building data

The source database is a self-built database in the cloud, IUD 20:60:20, 1~~2 KB/ piece of data, per table 2~~4 concurrency, 2~4 changes per transaction, RPS 1000 or so.

Use CloudCanal to create a data synchronization task in the cloud

Select the data source and select the appropriate options
- Select a cluster in the cloud or a self-built cluster
- 2, 3 Select the internal network for the source database and the public network for the peer Kafka database
- You can select the compatible open source Canal message format or the built-in CloudCanal message format
Select Data Synchronization and select Initialize data
Select the table. Do not change the peer topic here. Generate the table according to the default rules.
Select the column, oktailoringSome columns are dropped without migration synchronization
Create a confirmation
Tasks are running normally

Use CloudCanal to create a cloud migration synchronization task

Select the data source and select the appropriate options
- Procedure 1 Select the cluster where the ECS client resides
- Both 2 and 3 choose Intranet access to Kafka and RDS for MySQL, respectively
- 4 Fill in the Consumer Group you created earlier in the Aliyun Kafka console
- 5 Select a message format that is consistent with that of the cloud task
Skip the table and column selections and click Next
Create a confirmation
The two tasks are running properly
Cloud consumption task creationBecause it is inTasks run in the cloudAfter that, so the cloud needs to consume the task sitebackTo cover all data before creating a task under the cloud.

Check data

In order to verify the data in this case, we are lazy and directly open the RDS for MySQL public network link, and connect to the CloudCanal cluster under the cloud to do a data verification directly (the production environment is forbidden ！！！！).

To make the verification results more clear, stop producing data
Apply for RDS for MySQL public IP address and change the public IP address of the instance on the CloudCanal data source management page
Creating a Verification Task
- 1 Select a cloud or local cluster
- 2, 3 Source end self-built MySQL select Intranet access, target end RDS for MySQL select extranet access
Set the task type toData validation
Skip the table and column selection in the middle and click all the way
Create a confirmation
The task is completed and the result is correct

conclusion

This article briefly describes how to use CloudCanal SAAS capabilities to quickly build a secure, cross-internet data migration synchronization solution.

This scheme has the following features:

The public network port is disabled for both databases
Internet data is encrypted using SSL
The data egress is authenticated by the user name and password, and an IP address whitelist is configured to enhance security control

If you think this post is good, please forward, like, try it out (SAAS or download the community version).