This article introduces how to implement mysql synchronization data to Kafka to achieve hybrid cloud online data synchronization

junyu-cloudcanal

Introduction to the

This paper mainly describes how to use CloudCanal to quickly build a secure hybrid cloud online data ecology, which is fully achieved by using CloudCanal SAAS capabilities.

This scheme has the following characteristics

  • Using Kafka to do data transfer
  • The public network port is disabled for both databases
  • Internet data is encrypted using SSL
  • The data egress is authenticated by the user name and password, and an IP address whitelist is configured to enhance security control

In the example, the cloud database, cloud messaging product, and self-built database can be replaced with self-built resources or various cloud resources in your current environment.

Technical point

The main difficulty of hybrid cloud data ecology is network security. Some users are concerned about traffic resource loss due to the large amount of synchronized data transmitted.

The solution based on CloudCanal pays more attention to the problems at the network security level and achieves one-way isolation of sensitive resource network, link authentication and transmission encryption.

Here is a brief exampleInternet model 和 Line modeData goes up and down the cloud.

For example, “chestnuts”

This case mainly demonstrates the Internet model of cross-cloud data migration and synchronization, the specific scenario is how to carry out data on the cloud (self-built room database -> Ali Cloud cloud database), and long-term maintenance of hybrid cloud data system. Of course, the same scheme can also be used in reverse, but in the data source selection of the external network Intranet is different.

Install CloudCanal

  • Prepare an ECS and a vm in the self-built equipment room, and install the ECS client according to the ECS client installation document and the self-built machine client installation document.

For Kafka

  • Buy page in Aliyun Kafka

    Purchase the corresponding Kafka, verification capabilities can be purchased in quantity instances

    • Select a public network /VPC instance and select a large public network traffic

  • Select a deployment instance2.x.xVersion,Maximum message sizeSlightly larger (say 4MB) is recommended.

  • Enter the instance,To create a Consumer Group,“And write down the name

Subaccount authorization and add data source

  • In accordance with theAli Cloud sub-account preparationDocument, create or authorize subaccounts, andRemember the sub-accounts AK and SKGrant basic database accessAliyunRDSFullAccess.AliyunKafkaFullAccess
  • Add respectivelySelf-built database under the cloud,Aliyun RDS for MySQL 和 Ali cloud Kafka
    • When adding Aliyun resources, select Automatically add migration machine whitelist in the second step
    • The Kafka user name and password can be found in the security configuration section at the bottom of aliyun Kafka instance details page
    • Please download the TLS file from the Aliyun Kafka root certificate

Start building data

  • The source database is a self-built database in the cloud, IUD 20:60:20, 12 KB/ piece of data, per table 24 concurrency, 2~4 changes per transaction, RPS 1000 or so.

Use CloudCanal to create a data synchronization task in the cloud

  • Select the data source and select the appropriate options

    • Select a cluster in the cloud or a self-built cluster
    • 2, 3 Select the internal network for the source database and the public network for the peer Kafka database
    • You can select the compatible open source Canal message format or the built-in CloudCanal message format

  • Select Data Synchronization and select Initialize data

  • Select the table. Do not change the peer topic here. Generate the table according to the default rules.

  • Select the column, oktailoringSome columns are dropped without migration synchronization

  • Create a confirmation

  • Tasks are running normally

Use CloudCanal to create a cloud migration synchronization task

  • Select the data source and select the appropriate options

    • Procedure 1 Select the cluster where the ECS client resides
    • Both 2 and 3 choose Intranet access to Kafka and RDS for MySQL, respectively
    • 4 Fill in the Consumer Group you created earlier in the Aliyun Kafka console
    • 5 Select a message format that is consistent with that of the cloud task

  • Skip the table and column selections and click Next

  • Create a confirmation

  • The two tasks are running properly

  • Since the cloud consumption task is created after the cloud task runs, the cloud consumption task site needs to be traced back to before the cloud task is created to cover all the data.

Check data

In order to verify the data in this case, we are lazy and directly open the RDS for MySQL public network link, and connect to the CloudCanal cluster under the cloud to do a data verification directly (the production environment is forbidden ！！！！).

  • To make the verification results more clear, stop producing data

  • Apply for RDS for MySQL public IP address and change the public IP address of the instance on the CloudCanal data source management page

  • Creating a Verification Task

    • 1 Select a cloud or local cluster
    • 2, 3 Source end self-built MySQL select Intranet access, target end RDS for MySQL select extranet access

  • Set the task type to data verification

  • Skip the table and column selection in the middle and click all the way

  • Create a confirmation

  • The task is completed and the result is correct

conclusion

This article briefly describes how to use CloudCanal SAAS capabilities to quickly build a secure, cross-internet data migration synchronization solution.

This scheme has the following features:

  • The public network port is disabled for both databases
  • Internet data is encrypted using SSL
  • The data egress is authenticated by the user name and password, and an IP address whitelist is configured to enhance security control

If you think this post is good, please forward, like, try it out (SAAS or download the community version). Join CloudCanal fan group to get first-hand information and more benefits, please add our little assistant wechat: suhuayue001

CloudCanal- free and easy to use enterprise-level data synchronization tool, welcome to taste. To learn more about the product, check out the official website: www.clougence.com CloudCanal community: www.askcug.com/