[Redis] Docker compose deployment cluster mode

The master-slave replication solves the performance problem of Redis, and the sentinel mode solves the availability problem of Redis. However, with the increase of data volume, a single Redis service node still faces the problem of large data volume. Therefore, Redis launched the cluster mode, where multiple masters can be set. The amount of data is fragmented for storage.

6. Docker image after version X supports the deployment cluster mode. As Redis requires a cluster to have at least three master nodes, this drill builds a Redis cluster with three master nodes and three slave nodes.

Pay attention to the point

Because Redis cluster does not support network forwarding, Docker needs to pay attention to network Settings when building Redis cluster. It is ok to build a subnet, the cluster can run and can be used, but the host is not able to use the cluster, can only be used within the subnet, such as:

zohar@ZOHAR-LAPTOP:~$redis-cli -c -p 6380 127.0.0.1:6380> ping PONG 127.0.0.1:6380> set zohar@ZOHAR-LAPTOP:~$redis-cli -c -p 6381 127.0.0.1:6381> Ping PONG 127.0.0.1:6381> get hello -> Redirected to slot [866] located at 172.26.0.101:6379 > # Client stuckCopy the code

Since the hello key has been written on 6380, when we cut to the machine 6381 to query, 6381 will automatically redirect our customer to the node where the key is stored. However, because we define the cluster on the Intranet, the IP and address of the Intranet are used for the jump, and the external network cannot be connected, so the client cannot be used.

So create Docker create Redis cluster directly using the host network is best, that is, use “host” mode. Due to problems with the Docker Host mode based on WSL2, I operate directly on the Debian real machine based on Host mode, and USE WSL2 based on Bridge mode.

Network mode based on Host

The directory structure

In cluster mode, the Master and Slave are automatically determined and all the machines are called nodes.

├─ data │ ├─ └ ├─ data │ ├─ ├─ ├─ ├─ data │ ├─ data │ ├─ ├─ ├─ Node3 │ ├ ─ ─ data │ └ ─ ─ redis. Conf ├ ─ ─ node4 │ ├ ─ ─ data │ └ ─ ─ redis. Conf ├ ─ ─ node5 │ ├ ─ ─ data │ └ ─ ─ redis. Conf └ ─ ─ node6 ├ ─ ─ data └ ─ ─ redis. ConfCopy the code

Compose File

version: "3"

services:
  node1:
    image: redis
    container_name: redis-cluster-node-1
    network_mode: "host"
    volumes:
      - "./node1/redis.conf:/etc/redis.conf"
      - "./node1/data:/data"
    command: ["redis-server"."/etc/redis.conf"]
    restart: always

  node2:
    image: redis
    container_name: redis-cluster-node-2
    network_mode: "host"
    volumes:
      - "./node2/redis.conf:/etc/redis.conf"
      - "./node2/data:/data"
    command: ["redis-server"."/etc/redis.conf"]
    restart: always

  node3:
    image: redis
    container_name: redis-cluster-node-3
    network_mode: "host"
    volumes:
      - "./node3/redis.conf:/etc/redis.conf"
      - "./node3/data:/data"
    command: ["redis-server"."/etc/redis.conf"]
    restart: always

  node4:
    image: redis
    container_name: redis-cluster-node-4
    network_mode: "host"
    volumes:
      - "./node4/redis.conf:/etc/redis.conf"
      - "./node4/data:/data"
    command: ["redis-server"."/etc/redis.conf"]
    restart: always

  node5:
    image: redis
    container_name: redis-cluster-node-5
    network_mode: "host"
    volumes:
      - "./node5/redis.conf:/etc/redis.conf"
      - "./node5/data:/data"
    command: ["redis-server"."/etc/redis.conf"]
    restart: always

  node6:
    image: redis
    container_name: redis-cluster-node-6
    network_mode: "host"
    volumes:
      - "./node6/redis.conf:/etc/redis.conf"
      - "./node6/data:/data"
    command: ["redis-server"."/etc/redis.conf"]
    restart: always
Copy the code

Node configuration

All nodes have the same configuration except port and cluster-announce-bus-port:

port 6371
protected-mode no
daemonize no

################################ REDIS CLUSTER  ###############################

cluster-enabled yes
cluster-config-file nodes.conf
cluster-node-timeout 15000
# cluster-announce-ip 127.0.0.1
# cluster-announce-port 6380
cluster-announce-bus-port 16371
Copy the code

For six nodes, the value of Port is 6371 to 6376, and the value of bus-port is 16371 to 16376.

For the node configuration file cluster-config-file, it is automatically generated and we do not need to configure it. We only need to define the file name.

Start the service configuration cluster

Start all services:

docker-compose up -d
Creating redis-cluster-node-6 ... done
Creating redis-cluster-node-1 ... done
Creating redis-cluster-node-3 ... done
Creating redis-cluster-node-2 ... done
Creating redis-cluster-node-4 ... done
Creating redis-cluster-node-5 ... done
Copy the code

Check whether the node is running properly.

docker ps
CONTAINER ID   IMAGE  COMMAND                  CREATED          STATUS          PORTS  NAMES
4db611346310   redis  "Docker - entrypoint. S..."   12 minutes ago   Up 12 minutes          redis-cluster-node-5
0f98fa0209af   redis  "Docker - entrypoint. S..."   12 minutes ago   Up 12 minutes          redis-cluster-node-2
59415143e37d   redis  "Docker - entrypoint. S..."   12 minutes ago   Up 12 minutes          redis-cluster-node-4
0a8856052297   redis  "Docker - entrypoint. S..."   12 minutes ago   Up 12 minutes          redis-cluster-node-3
30f965ef01b3   redis  "Docker - entrypoint. S..."   12 minutes ago   Up 12 minutes          redis-cluster-node-1
151f70be6dc6   redis  "Docker - entrypoint. S..."   12 minutes ago   Up 12 minutes          redis-cluster-node-6
Copy the code

At this point, we can see that the nodes.conf file is generated on each node.

Tree node1 node1 ├ ─ ─ data │ ├ ─ ─ dump. The RDB │ └ ─ ─ nodes. The conf └ ─ ─ redis. Conf cat node1 / data/nodes. Conf bcaaee5d3304868c74f55868713b517f29db08f0 :0@0 myself,master - 0 0 0 connected vars currentEpoch 0 lastVoteEpoch 0Copy the code

As you can see, each node writes its own information to nodes.conf.

Starting a cluster:

–cluster-replicas sets the number of replicas per Master.

If you don’t have a redis client installed, it doesn’t matter. Create a Redis container and share the data volume. Then copy all the Redis tools in the /usr/local/bin/ directory to the data volume. Then the host will have all the Redis programs, including Redis – CLI.

Redis -cli --cluster create 127.0.0.1:6371 127.0.0.1:6372 127.0.0.1:6373 127.0.0.1:6374 127.0.0.1:6375 127.0.0.1:6376 --cluster-replicas 1 >>> Performinghashslots allocation on 6 nodes... Master[0] -> Slots 0 - 5460 Master[1] -> Slots 5461 - 10922 Master[2] -> Slots 10923 - 16383 Adding replica 127.0.0.1:6375 to 127.0.0.1:6371 Adding Replica 127.0.0.1:6376 to 127.0.0.1:6372 Adding Replica 127.0.0.1:6374 to >>> Trying to optimize Slaves Allocationfor anti-affinity
[WARNING] Some slaves are inthe same host as their master M: 2 a84fbf4c63ad7e0fda88783c41aadf87a403463 127.0.0.1:6371 slots: [0-5460] (5461 slots) master M: 8457 fba97e70cffa25e42ce3fe8099081ea244f6 127.0.0.1:6372 slots: [5461-10922] (5462 slots) master M: 7644693 f18b72312e2d9d6e0ad472a98ab3d33be 127.0.0.1:6373 slots: [10923-16383] (5461 slots) master S: Afbc31f6bcbb3e07925ebe694221a185f4f47d17 127.0.0.1:6374 replicates 7644693 f18b72312e2d9d6e0ad472a98ab3d33be S: C4a5391101de8b046d4494399f29ec479e4ab4b0 127.0.0.1:6375 replicates a84fbf4c63ad7e0fda88783c41aadf87a403463 2 S: 8457 fba97e70cffa25e42ce3fe8099081ea244f6 cbed04ea790b849dd9dc66b27d64b69c98cd8db7 127.0.0.1:6376 replicates Can Iset the above configuration? (type 'yes' to accept):
Copy the code

You will be asked if this configuration works, and you will be asked to set the cluster according to this configuration by typing yes:

Waiting forthe cluster to join .... >>> Performing Cluster Check (using node 127.0.0.1:6371) M: 2 a84fbf4c63ad7e0fda88783c41aadf87a403463 127.0.0.1:6371 slots: [0-5460] (5461 slots) master 1 additional up (s) M: 7644693 f18b72312e2d9d6e0ad472a98ab3d33be 127.0.0.1:6373 slots: [10923-16383] (5461 slots) master 1 additional up (s) M: 8457 fba97e70cffa25e42ce3fe8099081ea244f6 127.0.0.1:6372 slots: [5461-10922] (5462 slots) master 1 additional up (s) S: c4a5391101de8b046d4494399f29ec479e4ab4b0 127.0.0.1:6375 slots: (0 slots) slave replicates 2a84fbf4c63ad7e0fda88783c41aadf87a403463 S: Afbc31f6bcbb3e07925ebe694221a185f4f47d17 127.0.0.1:6374 slots: (0 slots) slave replicates 7644693f18b72312e2d9d6e0ad472a98ab3d33be S: Cbed04ea790b849dd9dc66b27d64b69c98cd8db7 127.0.0.1:6376 slots: (0 slots) slave replicates 8457fba97e70cffa25e42ce3fe8099081ea244f6 [OK] All nodes agree about slots configuration. >>> Checkfor open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
Copy the code

The nodes.conf information under each node is changed to add information about other nodes:

The cat node1 / data/nodes. The conf e1ba5da93c357ec7d98c7b96455773525f1f8890 127.0.0.1:6375 @ 16375 slave bcaaee5d3304868c74f55868713b517f29db08f0 0 1629280926614 1 connected 9f2decedb68110c12e5e4f320b560438c39915e6 127.0.0.1:6376 @ 16376 slave af0bb933d8d155f4a18c0d99a969bbd2239c8ec0 0 1629280925611 2 connected Bcaaee5d3304868c74f55868713b517f29db08f0 127.0.0.1:6371 @ 16371 myself, master 1629280924000 connected 1 0 0-5460 0985 adaae5b03bc13039fd2af7bcdecb29a12bac 127.0.0.1:6374 @ 16374 slave e7eb0cb150332c13f993014c93db99e63aaf5935 0 1629280926000 3 connected af0bb933d8d155f4a18c0d99a969bbd2239c8ec0 127.0.0.1:6372 @ 16372 master - 0 1629280924607 2 Connected to 5461-10922 e7eb0cb150332c13f993014c93db99e63aaf5935 127.0.0.1:6373 @ 16373 master - 0 1629280927617 3 connected 10923-16383 vars currentEpoch 6 lastVoteEpoch 0Copy the code

Validation of the cluster

You can view cluster configuration information from Cluster Info or node information from Cluster Nodes.

Zohar @ VM - 4-2 - debian: ~ $redis - cli - c - p 6372 127.0.0.1:6372 > cluster nodes bcaaee5d3304868c74f55868713b517f29db08f0 127.0.0.1: master 6371 @ 16371-0, 1629283228000 1 connected e1ba5da93c357ec7d98c7b96455773525f1f8890 0-5460 127.0.0.1:6375 @ 16375 slave bcaaee5d3304868c74f55868713b517f29db08f0 0 1629283228868 1 connected 9 f2decedb68110c12e5e4f320b560438c39915e6 127.0.0.1:6376 @ 16376 slave af0bb933d8d155f4a18c0d99a969bbd2239c8ec0 0 1629283229873 2 connected 0985 adaae5b03bc13039fd2af7bcdecb29a12bac 127.0.0.1:6374 @ 16374 slave e7eb0cb150332c13f993014c93db99e63aaf5935 0 1629283229000 3 connected af0bb933d8d155f4a18c0d99a969bbd2239c8ec0 127.0.0.1:6372 @ 16372 myself, master 1629283226000 2 0 connected e7eb0cb150332c13f993014c93db99e63aaf5935. 5461-10922 127.0.0.1:6373@16373 master-0 1629283226000 3 Connected 10923-16383 127.0.0.1:6372>Copy the code

Reading and writing tests:

127.0.0.1:6372> Set Hello World -> Redirected to slot [3030] located at 127.0.0.1:6371 OK 127.0.0.1:6371> get Hello "World" 127.0.0.1:6371> Set test jump -> Redirected to slot [6918] located at 127.0.0.1:6372 OK 127.0.0.1:6372> get test  "jump"Copy the code

As you can see, since the Slot after the Hello hash is on 6371, it automatically jumps to 6371 and performs the write. Since the Slot after Test is hashed on 6372, it automatically jumps to 6372 and performs a write.

Disable the Master node:

docker stop redis-cluster-node-1
redis-cluster-node-1
Copy the code

The logs of other nodes are as follows:

# redis-cluster-node-1 1:M 18 Aug 2021 10:42:46.364 * FAIL message received from E7eb0cb150332c13f993014c93db99e63aaf5935 about bcaaee5d3304868c74f55868713b517f29db08f0 1: M 18 Aug 2021 10:42:46. 365 # Cluster state changed: Fail 1: M 18 Aug 2021 10:42:47. 070 # Failover auth granted to e1ba5da93c357ec7d98c7b96455773525f1f8890 for epoch 1:7 M 18 Aug 2021 10:42:47.112 # Cluster state changed: OKCopy the code

Log in to other nodes and query node information.

127.0.0.1:6372 > cluster nodes bcaaee5d3304868c74f55868713b517f29db08f0 127.0.0.1:6371 @ 16371 master, fail - 1629283349279 1629283346000 1 disconnected e1ba5da93c357ec7d98c7b96455773525f1f8890 127.0.0.1:6375 @ 16375 master - 0, 1629283432687 Connected 0-5460-9 f2decedb68110c12e5e4f320b560438c39915e6 127.0.0.1:6376 @ 16376 slave af0bb933d8d155f4a18c0d99a969bbd2239c8ec0 0 1629283432000 2 connected 0985adaae5b03bc13039fd2af7bcdecb29a12bac 127.0.0.1:6374 @ 16374 slave e7eb0cb150332c13f993014c93db99e63aaf5935 0 1629283431682 3 connected Af0bb933d8d155f4a18c0d99a969bbd2239c8ec0 127.0.0.1:6372 @ 16372 myself, master 1629283433000 2 0 connected. 5461-10922 E7eb0cb150332c13f993014c93db99e63aaf5935 127.0.0.1:6373 @ 16373 master - 0 1629283433691 3 connected. 10923-16383Copy the code

As you can see, node 6375 is promoted to Master.

Based on Bridge network mode

The directory structure

This parameter is consistent with the Host mode.

Cluster subnet configures / ├ ─ ─ docker - compose. Yml ├ ─ ─ node1 │ ├ ─ ─ data │ └ ─ ─ redis. Conf ├ ─ ─ 2 │ ├ ─ ─ data │ └ ─ ─ redis. Conf ├ ─ ─ Node3 │ ├ ─ ─ data │ └ ─ ─ redis. Conf ├ ─ ─ node4 │ ├ ─ ─ data │ └ ─ ─ redis. Conf ├ ─ ─ node5 │ ├ ─ ─ data │ └ ─ ─ redis. Conf └ ─ ─ node6 ├ ─ ─ data └ ─ ─ redis. ConfCopy the code

Compose File

Defines a subnet redis-cluster:

version: "3"

networks:
  redis-cluster:
    driver: bridge
    ipam:
      config:
        - subnet: 172.26. 0. 0/ 24

services:
  node1:
    image: redis
    container_name: redis-cluster-node-1
    ports:
      - "6371:6379"
    volumes:
      - "./node1/redis.conf:/etc/redis.conf"
      - "./node1/data:/data"
    command: ["redis-server"."/etc/redis.conf"]
    restart: always
    networks:
      redis-cluster:
        ipv4_address: 172.26. 0101.

  node2:
    image: redis
    container_name: redis-cluster-node-2
    ports:
      - "6372:6379"
    volumes:
      - "./node2/redis.conf:/etc/redis.conf"
      - "./node2/data:/data"
    command: ["redis-server"."/etc/redis.conf"]
    restart: always
    networks:
      redis-cluster:
        ipv4_address: 172.26. 0102.

  node3:
    image: redis
    container_name: redis-cluster-node-3
    ports:
      - "6373:6379"
    volumes:
      - "./node3/redis.conf:/etc/redis.conf"
      - "./node3/data:/data"
    command: ["redis-server"."/etc/redis.conf"]
    restart: always
    networks:
      redis-cluster:
        ipv4_address: 172.26. 0103.

  node4:
    image: redis
    container_name: redis-cluster-node-4
    ports:
      - "6374:6379"
    volumes:
      - "./node4/redis.conf:/etc/redis.conf"
      - "./node4/data:/data"
    command: ["redis-server"."/etc/redis.conf"]
    restart: always
    networks:
      redis-cluster:
        ipv4_address: 172.26. 0104.

  node5:
    image: redis
    container_name: redis-cluster-node-5
    ports:
      - "6375:6379"
    volumes:
      - "./node5/redis.conf:/etc/redis.conf"
      - "./node5/data:/data"
    command: ["redis-server"."/etc/redis.conf"]
    restart: always
    networks:
      redis-cluster:
        ipv4_address: 172.26. 0105.

  node6:
    image: redis
    container_name: redis-cluster-node-6
    ports:
      - "6376:6379"
    volumes:
      - "./node6/redis.conf:/etc/redis.conf"
      - "./node6/data:/data"
    command: ["redis-server"."/etc/redis.conf"]
    restart: always
    networks:
      redis-cluster:
        ipv4_address: 172.26. 0106.
Copy the code

Node configuration

The configurations of all nodes are the same and do not need to be modified:

port 6379
protected-mode no
daemonize no

################################ REDIS CLUSTER  ###############################

cluster-enabled yes
cluster-config-file nodes.conf
cluster-node-timeout 15000
# cluster-announce-ip 127.0.0.1
# cluster-announce-port 6380
cluster-announce-bus-port 16379
Copy the code

Redis provides cluster-announce-ip and cluster-announce-port to enable clustering to work in forwarding networks, but I have tried it and found it ineffective, so I will study it later.

Start the service configuration cluster

Starting a cluster:

docker-compose up -d
Creating redis-cluster-node-5 ... done
Creating redis-cluster-node-6 ... done
Creating redis-cluster-node-3 ... done
Creating redis-cluster-node-4 ... done
Creating redis-cluster-node-2 ... done
Creating redis-cluster-node-1 ... done
Copy the code

Start the cluster inside the instance:

docker exec -it redis-cluster-node-1 /bin/bash
root@53026834c158:/data# redis-cli --cluster create 172.26.0.101:6379 172.26.0.102:6379 172.26.0.103:6379 172.26.0.104:6379 172.26.0.105:6379 172.26.0.106:6379-1 - cluster - replicas
>>> Performing hashslots allocation on 6 nodes... Master[0] -> Slots 0 - 5460 Master[1] -> Slots 5461 - 10922 Master[2] -> Slots 10923 - 16383 Adding replica 172.26.0.105:6379 to 172.26.0.101:6379 Adding replica 172.26.0.106:6379 to 172.26.0.102:6379 Adding replica 172.26.0.104:6379 to 172.26.0.103:6379 M: 5857 fffad0e75e119e86fffef8b606900a0b886a 172.26.0.101:6379 slots: [0-5460] (5461 slots) master M: 98 d5586036314b23a0992f8db2ab21238ebc13d9 172.26.0.102:6379 slots: [5461-10922] (5462 slots) master M: 0371736 e14b0d7ce8e7e5c0685b8a2a75d757822 172.26.0.103:6379 slots: [10923-16383] (5461 slots) master S: Ca417fac5e3417ac906f1b5a4d854eae3751820e 172.26.0.104:6379 replicates 0371736 e14b0d7ce8e7e5c0685b8a2a75d757822 S: D619095c3bbe18ed027c538f49aeaafbdbe327a9 172.26.0.105:6379 replicates 5857 fffad0e75e119e86fffef8b606900a0b886a S: 6 a081bce8790e2a616c2b352da95ad877b8057f7 172.26.0.106:6379 replicates 98 d5586036314b23a0992f8db2ab21238ebc13d9 Can Iset the above configuration? (type 'yes' to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting forThe cluster to join. >>> Performing Cluster Check (using node 172.26.0.101:6379) M: 5857 fffad0e75e119e86fffef8b606900a0b886a 172.26.0.101:6379 slots: [0-5460] (5461 slots) master 1 additional up (s) s: D619095c3bbe18ed027c538f49aeaafbdbe327a9 172.26.0.105:6379 slots: (0 slots) slave replicates 5857fffad0e75e119e86fffef8b606900a0b886a M: 0371736 e14b0d7ce8e7e5c0685b8a2a75d757822 172.26.0.103:6379 slots: [10923-16383] (5461 slots) 1 additional master Up (s) s: ca417fac5e3417ac906f1b5a4d854eae3751820e 172.26.0.104:6379 slots: (0 slots) slave replicates 0371736e14b0d7ce8e7e5c0685b8a2a75d757822 S: 6 a081bce8790e2a616c2b352da95ad877b8057f7 172.26.0.106:6379 slots: (0 slots) slave replicates 98d5586036314b23a0992f8db2ab21238ebc13d9 M: 98 d5586036314b23a0992f8db2ab21238ebc13d9 172.26.0.102:6379 slots: [5461-10922] (5462 slots) 1 additional master replica(s) [OK] All nodes agree about slots configuration. >>> Checkfor open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
Copy the code

Querying node information:

root@53026834c158:/data# redis-cli -c127.0.0.1:6379 > cluster nodes d619095c3bbe18ed027c538f49aeaafbdbe327a9 172.26.0.105:6379 @ 16379 slave 5857fffad0e75e119e86fffef8b606900a0b886a 0 1629284311000 1 connected 5857fffad0e75e119e86fffef8b606900a0b886a 172.26.0.101:6379 @ 16379 myself, master 1629284312000 connected 1 0 0-5460, 0371736 e14b0d7ce8e7e5c0685b8a2a75d757822 172.26.0.103: master 6379 @ 16379-0 1629284313881 3 connected ca417fac5e3417ac906f1b5a4d854eae3751820e. 10923-16383 172.26.0.104:6379 @ 16379 slave e14b0d7ce8e7e5c0685b8a2a75d757822 0371736 0 1629284312000 3 connected 6 a081bce8790e2a616c2b352da95ad877b8057f7 172.26.0.106:6379 @ 16379 slave d5586036314b23a0992f8db2ab21238ebc13d9 0 98 1629284314885 2 connected 98 d5586036314b23a0992f8db2ab21238ebc13d9 172.26.0.102:6379 @ 16379 master - 0 1629284313000 2 connected 5461-10922Copy the code

Validation of the cluster

• Container Intranet:

  127.0.0.1:6379> set hello world
  OK
  127.0.0.1:6379> set test redsi
  -> Redirected to slot [6918] located at 172.26.0.102:6379
  OK
  Copy the code

• The host:

  Redis -cli -c -p 6371 127.0.0.1:6371> get hello "world" 127.0.0.1:6371> get test -> Redirected to slot [6918] located at 172.26.0.102:6379 # stuckCopy the code

  Service switchover cannot be performed.