This article describes how to synchronize mysql database information to ElasticSearch using Go-mysql-ElasticSearch.

1. The go – mysql – elasticsearch profile

Go-mysql-elasticsearch is a service that automatically synchronizes mysql data to ElasticSearch.

It first uses mysqldump to get the raw data, and then incrementally synchronizes the data with binlog.

Github address: github.com/siddontang/…

Here are a few caveats:

  • 1. The Mysql binlog must be in ROW mode; otherwise, an error will be reported.
  • 2. The user permission for connecting to Mysql must be larger.

2. Install

2.1 installation go

Install the go

yum install -y go
Copy the code

Install godep

go get github.com/tools/godep
Copy the code

Download the go-mysql-ElastiSearch plugin

go get github.com/siddontang/go-mysql-elasticsearch
Copy the code

Into the corresponding directory, for example, I use the following directory/root/go/src/github.com/siddontang/go-mysql-elasticsearch

cd /root/go/src/github.com/siddontang/go-mysql-elasticsearch
Copy the code

compile

make
Copy the code

2.2 Enable binlog for Mysql

Mysql > enable binlog;

Enter the mysql

mysql -uroot -p
Copy the code

Enter the password and run the following command to check whether binlog is enabled

show variables like '%log_bin%';
Copy the code

As shown in the figure, ON means ON, or OFF if not.

If this function is not enabled, add the following configuration to my.cnf (server-id can be set as required, here is 1, log-bin is the log location, be sure to give log write permission, otherwise error will be reported, binlog_format is the mode, here must be ROW) :

server-id=1
log-bin=/usr/local/mysql-log/mysql-bin.log
binlog_format="ROW"
Copy the code

Restart mysql after the Settings are complete.

service mysqld restart
Copy the code

As shown in the figure, the restart is successful. If it fails, you can check the error log.

3. Configure the go – mysql – elasticsearch

You can configure go-mysql-elasticSearch as follows: github.com/siddontang/…

The configuration files tested in this article are as follows:

MySQL > select * from user where username = 'password'
my_addr = "ip:3306"     
my_user = "root"
my_pass = "* * *"

# Elasticsearch address
es_addr = "IP: port"
    
# Where data is stored
data_dir = "./var"
    
# Inner Http status address
stat_addr = "127.0.0.1:12800"
    
# pseudo server id like a slave
server_id = 1001
    
# mysql or mariadb
flavor = "mysql"
    
Mysql backup file, skip if not set or set to empty
# mysqldump = "mysqldump"
    
# minimal items to be inserted in one bulk
bulk_size = 128
    
# force flush the pending requests if we don't have enough items >= bulk_size
flush_bulk_time = "200ms"
    
# Ignore table without primary key
skip_no_pk_table = false
    
MySQL data source, schema, tables
[[source]]
schema = "test"
tables = ["link_info"]

[[rule]]
schema = "test"
table = "link_info"
index = "test_mysql2"
type = "link_info"

Copy the code

4. Run the go – mysql – elasticsearch

After the configuration is complete, run go-mysql-ElasticSearch

bin/go-mysql-elasticsearch -config=river.toml
Copy the code

The operation succeeds as shown in the figure.

5. Inspection

View es-head, as shown

Select * from the go-mysql-elasticSearch console where id = 5; select * from the go-mysql-ElasticSearch console where id = 5

The next time you look at es-head, as shown, the data has also changed.

6. Summary

Since I have never been in production, I only evaluate my personal test use. Installation and data synchronization feel very friendly, because combined with binlog, it can realize synchronous increase, deletion and change. For the online said log is few and immature and so on, here does not evaluate.