Small knowledge, big challenge! This article is participating in the creation activity of “Essential Tips for Programmers”.

Deploy mysql

docker run --name mysql -e MYSQL_ROOT_PASSWORD=123456 -d -p 3306:3306 mysql:5.7
Copy the code

After mysql is started, you need to change the configuration, because the container is started, you need to enter the mysql container to execute

Docker exec -it mysqlCopy the code

Because the mysql container does not install vi vim or other command tools by default, echo is used to append file contents

# Enable binlog to configure echo"[mysqld]" >> /etc/mysql/my.cnf
echo "server_id=1"  >> /etc/mysql/my.cnf
echo "log_bin = mysql-bin"  >> /etc/mysql/my.cnf
echo "binlog_format = ROW"  >> /etc/mysql/my.cnf
Copy the code

After the above configuration changes are complete, exit the mysql container and restart

Log in to mysql and check whether the binlog configuration takes effect

# enable binlogshow variables like 'log_bin'; Query the details of the binlog fileshow master logs;
Copy the code

Configure maxwell requirements

Account secrets and permissions required to create Maxwell

CREATE USER 'maxwell'@The '%' IDENTIFIED BY 'XXXXXX';
GRANT ALL ON maxwell.* TO 'maxwell'@The '%';
GRANT SELECT, REPLICATION CLIENT, REPLICATION SLAVE ON *.* TO 'maxwell'@The '%';
Copy the code

Create the test library and a test table

SET FOREIGN_KEY_CHECKS=0;
create database test;
use test;
CREATE TABLE `maxwell` (
  `id` int(11) DEFAULT NULL,
  `daemon` varchar(255) DEFAULT NULL
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;
Copy the code

The deployment of kafka

Kafka 2.2.1 depends on ZooKeeper, so zooKeeper is installed first

Zk docker run -d --name zookeeper-server -e ALLOW_ANONYMOUS_LOGIN=yes bitnami/zookeeper:latest # Kafka docker run -d  --name kafka-server -e ALLOW_PLAINTEXT_LISTENER=yes -e KAFKA_CFG_ZOOKEEPER_CONNECT=zookeeper-server:2181 bitnami/kafka:2.21.
Copy the code

After Kafka is started, you need to create a Topic for Maxwell to use because Maxwell does not actively create topics

# create a1To copy,5Topic kafka-topics. Sh --create --zookeeper IP:2181 --replication-factor 1 --partitions 5 --topic maxwell
Copy the code

The deployment of maxwell

docker run -it -d --name maxwell --rm zendesk/maxwell bin/maxwell \
--user=maxwell --password=XXXXXX --host=mysql_ip --port=3306 --producer=kafka \
--kafka.bootstrap.servers=kafka_ip:9092 --kafka_topic=maxwell
Copy the code

After the startup is complete, view the Maxwell container to output logs, as shown in the following figure

Maxwell docker logs -f maxwellCopy the code

Generate the data

Write a script that writes data to the test.maxwell table every second

#!/bin/Bash # generate data script HOSTNAME="ip"
PORT="3306"
USERNAME="root"
PASSWORD="123456"

DBNAME="test"
TABLENAME="maxwell"# write a total1000The datafor i in {1.1000.}
do
   insert_sql="insert into ${TABLENAME} values(${i},'Stanislaw Lem')"
   mysql -h${HOSTNAME}  -P${PORT}  -u${USERNAME} -p${PASSWORD} ${DBNAME} -e"${insert_sql}"
   sleep 1
done
Copy the code

Verify collection results

After running the script to generate data, the test.maxwell table in mysql starts to write data continuously.

Now verify that there is data in Kafka

# View messages, view kafka- from the beginningconsole-consumer.sh --topic maxwell --bootstrap-server ip:9092 --from-beginning
Copy the code

You can see that Maxwell successfully parses the binlog and sends the message to Kafka in JSON format

Database: test table: Maxwell type: insert data: ID :100,daemon:”Stanislaw Lem”

At this point, Maxwell collects binlogs and sends them to Kafka. In the next article, we will share flink1.12 consumption Kafka messages