The MySQL database can use the open source Canal component to detect data changes. Canal open source code address: github.com/alibaba/can…

Canal dependence conditions:

  • The principle of Canal is based on the mysql binlog technology, so it is necessary to enable the mysql binlog writing function here. It is recommended to set the binlog mode to ROW. If the configuration of the target database is inconsistent, you may need to modify the database configuration of the target database.
[mysqld] log-format=ROW # select ROW mode server_id=1 # configure mysql replaction to be defined.Copy the code

In master-slave mode, the server_id of the target database must be specified, not the default 0.

Canal after V1.0.26, the serverId as slave is automatically generated.

  • The principle of Canal is to simulate itself as mysql Slave, so there must be relevant permissions for mysql slave here.
CREATE USER canal IDENTIFIED BY 'canal';  
GRANT SELECT, REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO 'canal'@'%';
-- GRANT ALL PRIVILEGES ON *.* TO 'canal'@'%' ;
FLUSH PRIVILEGES;
Copy the code

Test environment installation:

  1. Install MySQL and set permissions
  2. Install canal, modify the configuration to connect to the corresponding database, or install directly through Docker.
  • Install github.com/alibaba/can… Wget github.com/alibaba/can…
  • Github.com/alibaba/can docker setup…
  1. Install the zookeeper wget mirrors.shu.edu.cn/apache/zook…
  2. When Kafka is installed, data change messages are delivered to Kafka

The test data

A canal and MySQL environment was configured on the test machine.

  • Database configuration:
# position info canal. The instance. The master. The address = 127.0.0.1:3306 canal. The instance. The master. The journal. The name = canal.instance.master.position= canal.instance.master.timestamp= canal.instance.master.gtid= # username/password canal.instance.dbUsername=root canal.instance.dbPassword=canal canal.instance.connectionCharset = UTF-8 canal.instance.defaultDatabaseName =test # enable druid Decrypt database password canal.instance.enableDruid=falseCopy the code

Canal listening on port 8930:

Id = 1 canal. IP = 0.0.0.0 canal.port = 8930Copy the code

Test example:

  • DML Deletes data records

Canal print: ================> binlog[mysql-bin.000003:3991] , name[test,teacher] , eventType : DELETE the = = = = = = = = = = = = = = = = = = = = RowData = = = = = = = = = = = = = = = = = = DELETE: id: 3 update = false th_name: period of teachers update = false age: 35 update=false teach_course: Chinese update=false school_age: 10 update=false school: tsinghua update=false TTT: update=falseCopy the code
  • DDL Deletes fields

SQL: "ALTER table teacher drop column TTT" ddlSchemaName: "test"Copy the code