1. Compare migration tools

  1. redis-migrate-tool

    Redis-migration-tool is a real-time data migration tool between heterogeneous clusters of Redis, which has not been updated for two years. It is a perfect tool, especially for data verification. For details, see GitHub

  2. RedisShake

    RedisShake is a RedisShake tool that supports real-time synchronization of heterogeneous clusters based on Redis-port. Compared with Redis-Migration-tool, I think its advantages are that it supports synchronization of prefix keys and supports multi-DB synchronization. The redis- Migrate -tool can only synchronize data to the db0 database if the source database is migrated to the db0 database. This is not applicable to services that are migrated to the db0 database

  3. Redis -port redis-port is a redis data migration tool, which is used to migrate data from Redis to Codis. It has not been updated for a long time

2. Install and configure the migration tool

  1. The installation of RedisShake

    Download binary package directly to install:

    Download address

  2. Modifying configuration files

    Redis-shak. conf configuration file:

    The version number of the current configuration file. Do not change this value.
conf.version = 1
# -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -
# id
id = redis-shake
# log file will be printed to stdout (e.g. /var/log/redis-shak.log) if not configured
log.file =
# log level: "none", "error", "warn", "info", "debug". default is "info".
log.level = info
# pid path (e.g. /var/run/)
Pid = '{pid_path}/{id}.pid
pid_path = 

# pprof port.
system_profile = 9310
# restful port, set -1 means disable, in `restore` mode RedisShake will exit once finish restoring RDB only if this value
# is -1, otherwise, it'll wait forever.
-1 indicates that the metric port is disabled. If the mode is' restore ', the RDB will exit only when it is set to -1. Otherwise, the RDB will remain blocked.
http_profile = 9320

# parallel routines number used in RDB file syncing. default is 64.
# How many concurrent threads are started to synchronize an RDB file.
parallel = 32

Standalone, Sentinel, Cluster and Proxy modes are supported. Note: Proxy is only used in RUMP mode.
source.type = sentinel
Source redis address. For sentinel or open source cluster mode, enter the format "master name: pull the address of the role master or slave@sentinel ", other clusters
# Architecture, such as COdis, TwemProxy, Aliyun Proxy, etc., requires the configuration of all master or slave DB addresses.Source. The address = mymaster: [email protected]:26379; 10.10.20.102:26379; 10.10.20.103:26379# password of db/proxy. even if type is sentinel.
source.password_raw = 
# auth type, don't modify it
source.auth_type = auth
# tls enable, true or false. Currently, only support standalone.
# open source redis does NOT support tls so far, but some cloud versions do.
source.tls_enable = false
# input RDB file.
# used in `decode` and `restore`.
# if the input is list split by semicolon(;) , redis-shake will restore the list one by one.
In the case of decode or restore, this parameter indicates the RDB file to read. Input list is supported, for example, rdb.0. rdb.1; rdb.2
# redis-shake will be restored one by one.
source.rdb.input = local

Parallel =3; parallel=3; parallel=3
Select * from RDB; select * from RDB; select * from RDB; select * from RDB;
Len (source.address) or len(rdB.input).
source.rdb.parallel = 0
# for special cloud vendor: ucloud
# used in `decode` and `restore`.
# ucloud RDB files with slot prefix: ucloud_cluster
source.rdb.special_cloud = 

Type of destination Redis, support standalone, Sentinel, Cluster and Proxy modes.Target. type = cluster target.address = 10.10.20.101:7000; 10.10.20.102:7000; 10.10.20.103:7000# password of db/proxy. even if type is sentinel.
target.password_raw = Redis1234{}
# auth type, don't modify it
target.auth_type = auth
# all the data will be written into this db. < 0 means disable.
target.db = 0
# tls enable, true or false. Currently, only support standalone.
# open source redis does NOT support tls so far, but some cloud versions do.
target.tls_enable = false
# output RDB file prefix.
# used in `decode` and `dump`.
If you decode or dump the RDB prefix in the output, for example, if the input is 3 dB, then dump is:
# ${output_rdb}.0, ${output_rdb}.1, ${output_rdb}.2
target.rdb.output = local_dump

Target. Version = 4.0
target.version =

# Used to handle expired key values, which the destination side needs to add when the migration ends are inconsistent
fake_time =

If the source destination has a duplicate key, whether to overwrite it
Rewrite means the source side overwrites the destination side.
# None indicates that the process exits once it occurs.
# ignore: preserves the destination key and ignores the synchronization key of the source. This value is not used in RUMP mode.
key_exists = rewrite

# the specified db is passed, such as 0; 5; 10 will allow DB0, DB5, db10 to pass, and the rest to be filtered
filter.db.whitelist =
The specified db is filtered, such as 0; 5; 10 will cause DB0, DB5, db10 to be filtered, and others to be passed
filter.db.blacklist =
# filter keys by prefix. Only keys with a specified prefix are allowed to pass through. For example, if you specify ABC, it will pass ABC, abc1, abcxxx
filter.key.whitelist =
# support key filter by prefix, do not allow the specified prefix key through, semicolon separation. For example, specifying ABC will block ABC, abc1, abcxxx
filter.key.blacklist =
# filter given slot, multiple slots are separated by '; '.
# e.g., 1; 2; 3
# used in `sync`.
# select filter slot, only the specified slot will pass
filter.slot =
# filter lua script. true means not pass. However, in redis 5.0, the lua 
# converts to transaction(multi+{commands}+exec) which will be passed.
# control not to allow lua scripts to pass, true means not to pass
filter.lua = false

If the value of the key exceeds the specified value, the value of the key exceeds the specified value
#, then batches are written one by one. If the destination end is Codis, set this parameter to 1. For details, see FAQ.
If the destination version is smaller than the source version, set this parameter to 1.
big_key_threshold = 524288000

# enable metric
# used in `sync`.
# Whether to enable metric
metric = true
# print in log
Print the metric to log
metric.print_log = false

# sender information.
# sender flush buffer size of byte.
# used in `sync`.
# The length of bytes in the send cache, exceeding this threshold will force the cache to flush the send
sender.size = 104857600
# sender flush buffer size of oplog number.
# used in `sync`. flush sender buffer when bigger than this threshold.
If the number of packets sent to the cache exceeds this threshold, the packets will be forcibly flushed from the cache. If the destination end is cluster, this value is used
# will take up some memory.
sender.count = 4095

Queue for metric statistics of delay
sender.delay_channel_size = 65535

# enable keep_alive option in TCP when connecting redis.
# the unit is second.
# 0 means disable.
# TCP keep-alive specifies the keepalive parameter, in seconds. 0 indicates that TCP keep-alive is disabled.
keep_alive = 0

# Number of scans per time. If no value is specified, the default value is 100.
scan.key_number = 50
Some versions have special formats, different from the normal scan command, we have special adaptations. Currently supports the cluster version of Tencent Cloud "Tencent_cluster"
# and ali cloud cluster version "aliyun_cluster", notes master and slave version does not need to configure, only for the cluster version.
scan.special_cloud =
# used in `rump`.
# we support to fetching data from given file which marks the key list.
# Some cloud versions do not support either sync/psync or SCAN, we support reading all key lists from files and fetching them: one key per line.
scan.key_file =

# limit the rate of transmission. Only used in `rump` currently.
QPS = 1000 means pass 1000 keys per second. Default is 500,000(0 means default)
qps = 200000

# enable resume from break point, please visit xxx to see more details.
# breakpoint continuation switch
resume_from_break_point = false

# replace hash tag.
# used in `sync`.
replace_hash_tag = false

Copy the code

    Main configuration items:

    # Source Redis type, sentinel mode uses sentinel
source.type = sentinel
Source Redis address information, if sentinel, format: sentinel name: primary/secondary identifier @ node address informationSource. The address = mymaster: [email protected]:26379; 10.10.20.102:26379; 10.10.20.103:26379# Redis password information, no password authentication, leave blank
source.password_raw = 
The authentication identifier of the source Redis, regardless of whether there is password authentication, does not need to be changed
source.auth_type = auth
The Redis target is a cluster
target.type = cluster
Redis address information, multiple addresses; Notice the unified master node or slave node information to be entered hereTarget. Address = 10.10.20.101:7000; 10.10.20.102:7000; 10.10.20.103:7000# Target Redis password information
target.password_raw = Redis1234{}
The authentication identifier of the target Redis is not modified
target.auth_type = auth
If the value is negative, this function is disabled. If a library is specified, all the source data will be written
target.db = 0
# When the same keys exist, the solution is:
Rewrite means the source side overwrites the destination side.
# None indicates that the process exits once it occurs.
# ignore: preserves the destination key and ignores the synchronization key of the source. This value is not used in RUMP mode.
key_exists = rewrite

Copy the code

3. Data synchronization

  1. Data synchronization commands

    ./redis-shake -conf=redis-shake.conf -type=sync
Copy the code

    Type supports sync, restore, dump, decode, and rump. Full + Incremental synchronization Select Sync

  2. Log Result

    Synchronization is divided into three stages:

    A. Wait until the source end saves the RDB

    2019/06/06 15:14:56 [INFO] dbSyncer[0] + waiting source rdb
2019/06/06 15:14:57 [INFO] dbSyncer[0] - waiting source rdb
2019/06/06 15:14:57 [INFO] dbSyncer[0] + waiting source rdb
Copy the code

    B. In the full synchronization stage, percentage is displayed:

    2019/06/06 15:15:41 [INFO] dbSyncer[0] total=924836132 - 9155943 [ 0%] entry=2109 2019/06/06 15:15:42 [INFO] dbSyncer[0]  total=924836132 - 16107663 [ 1%] entry=4411 2019/06/06 15:15:43 [INFO] dbSyncer[0] total=924836132 - 22914262 [ 2%] entry=6750 2019/06/06 15:15:44 [INFO] dbSyncer[0] total=924836132 - 29707595 [ 3%] entry=9060 2019/06/06 15:15:45 [INFO]  dbSyncer[0] total=924836132 - 35741354 [ 3%] entry=11067 2019/06/06 15:15:46 [INFO] dbSyncer[0] total=924836132 - 42911547 [ 4%] entry=13480Copy the code

    C. Incremental synchronization. Sync RDB done is displayed.

    2019/07/09 16:34:05 [INFO] dbSyncer[0] sync:  +forwardCommands=1      +filterCommands=0      +writeBytes=4
2019/07/09 16:34:06 [INFO] dbSyncer[0] sync:  +forwardCommands=0      +filterCommands=0      +writeBytes=0
2019/07/09 16:34:07 [INFO] dbSyncer[0] sync:  +forwardCommands=0      +filterCommands=0      +writeBytes=0
2019/07/09 16:34:08 [INFO] dbSyncer[0] sync:  +forwardCommands=0      +filterCommands=0      +writeBytes=0
2019/07/09 16:34:09 [INFO] dbSyncer[0] sync:  +forwardCommands=0      +filterCommands=0      +writeBytes=0
2019/07/09 16:34:10 [INFO] dbSyncer[0] sync:  +forwardCommands=0      +filterCommands=0      +writeBytes=0
2019/07/09 16:34:11 [INFO] dbSyncer[0] sync:  +forwardCommands=0      +filterCommands=0      +writeBytes=0
Copy the code

    ForwardCommands indicates the number of sent commands, filterCommands indicates the number of filtered commands, such as opinfo or filter, and writeBytes indicates the number of sent bytes.

4. Data verification and processing

  1. Data verification tool

    Redis-full-check Verifies data by fully comparing the redis data on the source end and the destination end.

    Redis-full-check

    Download address

  2. Verify execution command

    ./redis-full-check -s '10.10.20.102:6379' -p ' ' -t '10.10.20.101:7000; 10.10.20.102:7000; 10.10.20.103:7000 '  -a 'Redis1234{}' --targetdbtype=1 --metric  --result=result.1            
Copy the code

    If the source address and target address are synchronized, the metric function is enabled, and the result result is recorded.

    Detailed parameter description:

      -s, --source= source redis library address (IP :port) Split different dB, only need to configure either master or slave. For example: 10.1.1.1:1000; 10.2.2.2:2000; 10.3.3.3:3000.
  -p, --sourcepassword=Password Source redis library Password
      --sourceauthtype=Auth-type Indicates the permission to manage the source library. This parameter is useless under open source REIDS.
      --sourcedbtype=               Category of the source library, 0: DB (Standalone, master/slave), 1: Cluster (Cluster), 2: AliCloud
      --sourcedbfilterlist=         The logical db whitelist that the source library needs to fetch, starting with a semicolon (;) Split, for example: 0; 5; 15 indicates that DB0, DB5, and DB15 are all fetched
  -t, --target= target Destination redis library address (IP :port)
  -a, -- targetPassword =Password Specifies the target redis library Password
      --targetauthtype=Auth-type Indicates the permission to manage the destination library. This parameter is useless under open source REIDS.
      --targetdbtype=               Reference sourcedbtype
      --targetdbfilterlist=         Reference sourcedbfilterlist
  -d, --db=Sqlite3 -db-file Specifies the location of the Sqlite3 DB for the different keys stored, default result.db
      --comparetimes=COUNT Compares the number of rounds
  -m, -- compareMode = Comparison mode, where 1 compares the length of the value, 2 compares the existence of the key, and 4 ignores the comparison of large keys
      --id=                         Used to make metric
      --jobid=                      Used to make metric
      --taskid=                     Used to make metric
  -q, -- QPS = QPS rate limit threshold
      --interval=Second Indicates the interval between rounds
      --batchcount=COUNT Number of batch aggregations
      --parallel=COUNT Number of concurrent coroutines for comparison. Default: 5
      --log=The FILE log FILE
      --result=FILE Inconsistent results are recorded in the result FILE in the format of 'db diff-type key field'.
      --metric=The FILE metric FILE
      --bigkeythreshold=COUNT Threshold for splitting large keys. Comparemode =4
  -f, --filterlist=FILTER list of keys to compare, semicolon (;) Segmentation. For example: "ABC * | efg | m *" said 'ABC', 'abc1', 'efg', 'm', 'mxyz' contrast 'efgh', 'p'.
  -v, --version
Copy the code

  3. Check the comparison results

    Query result result information:

    [root @ localhost redis - full - check - 1.4.8] # less result. 1Copy the code

    Verification result:

    0       lack_target     single23        
1       lack_target     test    
9       value   single2
Copy the code

    The first column represents the database number, the second column represents the conflict type, and the third column represents the conflict key.

    Conflict types include:

    Lack_soure: Field exists in the key of the source end, but field does not exist in the key of the destination end.

    Lack_target: Field does not exist on the source key, but on the destination key.

    Value: Field exists in the key of the source end and the key of the destination end, but the value of field is different.

    To see more specific information, you can query through SQLite:

    [root@localhost redis-full-check-1.4.8]# sqlite3 result.db.3 SQLite version 3.7.17 2013-05-20 00:56:22 Enter ".help" for  instructions Enter SQL statements terminated with a ";" sqlite> .tables FINAL_RESULT field key sqlite> select * from key; 1|single23|string|lack_target|0|2|0 2|test|string|lack_target|1|4|0 3|single2|string|value|9|1|9Copy the code

    Id, key, type, conflict_type, db, source_len, target_len

This article was created and shared by Mirson. For further communication, please add to QQ group 19310171 or visit www.softart.cn