If you think the article is useful or well written, please click “like” on the left.

This article has been included in my personal blog: Geekvic. Top, welcome to chat!

Redis is highly available and supports three cluster modes: master-slave, Sentinel, and Cluster.

A master-slave mode

define

In a Redis cluster, several Redis servers are assigned to replicate another Redis server. We define the replicated server as the master server, and the servers that replicate the master server are called replica servers. This mode is called the master-slave replication mode.

role

  • Provides multiple copies of data and supports high availability.
  • Implement read/write separation (the primary node writes data and the secondary node reads data. The primary node periodically synchronizes data to the secondary node to ensure data consistency).
  • Separation of tasks, such as backup work and computing work from the server.

Characteristics of master-slave replication

  • The primary node can perform read and write operations. When the data on the primary node changes, the data is automatically synchronized to the secondary node.
  • Slave nodes are generally read-only;
  • A master can have multiple replicas, but a replica can only have one master.
  • If one replica is down, the read and write of the other replicas and master are not affected. Data will be synchronized from the master after the replica is restarted.
  • If the master is down, read data on the Replica node is not affected, but the replica node no longer provides write services. After the master restarts, Redis will provide write services again.
  • If the master is down, a new master is not selected from the replica node.

Working mechanism

After replica starts, send SYNC to master. After receiving the SYNC command, the master saves the snapshot (RDB persistence) and caches the commands for saving the snapshot in the background. Then the master sends the saved snapshot file and cached commands to the replica. The execution command that the replica loads the snapshot file and cache after receiving the snapshot file and command.

After replication initialization, each write command received by the master is sent to the slave to ensure data consistency between the master and slave.

Primary/secondary replication setup

Environment to prepare

Realize one master and two slave, on the same server, master port 6379; Two replicas, one port 6380 and the other port 6381, as shown in the figure below.

configuration

Conf file to generate redis6380.conf and redis6381.conf

The master configuration

You can disable the RDB snapshot function and assign backup work to the Replica node. The AOF function can be turned on or off.

Up the configuration

  • Replicaof: Configures the IP address and port of the master server, and establishes a master/slave relationship.
  • Masterauth: If the master requires password authentication, you need to configure the master password. Otherwise, the connection fails.
  • Replica-read-only: the default value is yes. The default value is read-only.

The specific configuration of 6380 slave service is as follows, and the configuration of 6381 is similar.

port 6380

pidfile /var/run/redis_6380.pid

save 900 1
save 300 10
save 60 10000

dbfilename dump6380.rdb

replicaof localhost 6379

replica-read-only yes
    
appendonly yes

appendfilename "appendonly6380.aof"
Copy the code

test

Run the following command to connect to the primary and secondary Redis services through the Redis client

./redis-cli -p 6379
./redis-cli -p 6380
./redis-cli -p 6381
Copy the code

Check the replica information in the primary service

info replication
Copy the code

According to the information

# Replication
role:master
connected_slaves:2
slave0:ip=::1,port=6380,state=online,offset=123023,lag=0
slave1:ip=::1,port=6381,state=online,offset=123023,lag=0
master_replid:37d7bc32b5aca031ba762585137da938c78a77dc
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:123023
second_repl_offset:-1
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:1
repl_backlog_histlen:123023
Copy the code

View information from the service

# Replication
role:slave
master_host:localhost
master_port:6379
master_link_status:up
master_last_io_seconds_ago:8
master_sync_in_progress:0
slave_repl_offset:123205
slave_priority:100
slave_read_only:1
connected_slaves:0
master_replid:37d7bc32b5aca031ba762585137da938c78a77dc
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:123205
second_repl_offset:-1
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:1
repl_backlog_histlen:123205
Copy the code

Add a key named name to the Redis master service and check whether synchronous replication is implemented in the slave service, as shown below.

127.0.0.1:6379 >setName Vic OK 127.0.0.1:6379> 127.0.0.1:6380> get name"vic"127.0.0.1:6381 > get the name"vic"
Copy the code

The primary/secondary replication configuration is complete.

Pits encountered in master-slave replication mode configuration

According to the previous configuration, I found that the master/slave configuration of Redis did not take effect, and the slave service was not connected to the master service, as shown below.

master_link_status:down
Copy the code

I thought it was a firewall problem. The port of 6379 was not opened, but I ran the netstat -lntp command to check that the port was open. I wondered whether the master service configured a password, and the slave service did not.

You can fix this problem by commenting out the bind 127.0.0.1 in the main service’s redis.conf file.

# bind 127.0.0.1
Copy the code

The guard mode

An overview of the

Sentinel mode is a special mode where Sentinel is an independent process that monitors multiple instances of Redis by sending commands and waiting for the Redis server to respond, as shown in the figure below. In the primary/secondary services, when the primary service fails, failover can be automatically implemented without manual operation. Therefore, this mode is recommended.

The principle of

The role of sentinels

  • The Redis server is sent back to monitor its running status, both primary and secondary, by sending commands
  • When the sentry detects that the master is down, it automatically switches the slave to master and notifies the other slave servers in public-subscribe mode to modify their configuration files and make them switch hosts.

In Redis high availability architectures, there is often not just one Sentinel, but three or more. To make it more reliable, we can monitor it with multiple sentries. Each sentry is also monitored, creating a multi-sentry mode

failover

When the primary node is detected to be down, disconnect all slave nodes that are connected to the primary node, select one slave node as the primary node, and connect the other slave nodes to the new primary node. And inform the client of the latest server address.

Relevant concepts

Subjective offline

SDOWN (subjectively down), directly translated as subjective offline, that is, the current Sentinel instance considers a redis service as unavailable.

Objective offline

ODOWN can be translated objectively, that is, the master is considered to be IN an SDOWN state by many sentinel instances — we objectively down — and ODOWN can be simply interpreted as the master being determined as “unavailable” by the cluster. Failover will be enabled.

Sentinel environment setup

Based on the original master-slave replication test environment, configure sentinel.conf and start the sentinel process.

Service type The master/slave port
Redis The main 6379
Redis from 6380
Redis from 6381
Sentinel – 26379
Sentinel – 26380
Sentinel – 26381

configuration

Conf file. The main configuration parameters are as follows. The difference between the three files is that the port number is 26379 to 26381.

# Port on which the Sentinel service is currently running
port 26379  
The first configuration tells Sentinel to monitor a primary server named MyMaster, whose IP address is 127.0.0.1 and port number is 6379, and to judge this primary server as invalid requires at least 2 Sentinels to agreeSentinel Monitor MyMaster 127.0.0.1 6379 2If myMaster does not respond within 10 seconds, myMaster is considered down
sentinel down-after-milliseconds mymaster 10000
If mysater is not started after 20 seconds, start failover
sentinel failover-timeout mymaster 20000  
When failover is performed, a maximum of 1 slave server synchronizes with the new master server
sentinel parallel-syncs mymaster 1
Copy the code

Start the Sentry process

/usr/local/bin/redis-sentinel sentinel.conf --sentinel &
/usr/local/bin/redis-sentinel sentinel6380.conf --sentinel &
/usr/local/bin/redis-sentinel sentinel6381.conf --sentinel &
Copy the code

The validation test

When the sentry cluster shuts down the 6379 master service, the sentry cluster determines that the 6379 master service is down, elects one of the two secondary services to upgrade to master, notifes the other service, and automatically changes its configuration file to point to the new master. For example, if 6380 becomes master, 6381 will automatically become replica of 6380. Of course, if we start 6379 now, 6379 will also become Replica of 6380, so as to realize automatic fault transfer of Redis.

In the pit of

23617:X 19 Aug 2020 16:01:43.844 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
23617:X 19 Aug 2020 16:01:43.844 # Sentinel ID is 6c4ad4b0df82b09a7d3078927182b59da3becb5f
23617:X 19 Aug 2020 16:01:43.844 # +monitor master mymaster 127.0.0.1 6379 quorum 123617:X 19 Aug 2020 16:01:43.845 * +slave Slave [::1]:6380 @mymaster 127.0.0.1 6379 23617:X 19 Aug 2020 16:01:43.850 * +slave Slave [::1]:6381 ::1 6381 @ mymaster 127.0.0.1 6379 23617:X 19 Aug 2020 16:01:53.867# +sdown slave [::1]:6381 ::1 6381 @ mymaster 127.0.0.1 6379
23617:X 19 Aug 2020 16:01:53.867 # +sdown slave [::1]:6380 @mymaster 127.0.0.1 6379
23617:X 19 Aug 2020 16:03:55.606 # +sdown master mymaster 127.0.0.1 6379
23617:X 19 Aug 2020 16:03:55.606 # +odown master mymaster 127.0.0.1 6379 #quorum 1/1
23617:X 19 Aug 2020 16:03:55.606 # +new-epoch 1
23617:X 19 Aug 2020 16:03:55.606 # +try-failover master mymaster 127.0.0.1 6379
23617:X 19 Aug 2020 16:03:55.610 # +vote-for-leader 6c4ad4b0df82b09a7d3078927182b59da3becb5f 1
23617:X 19 Aug 2020 16:03:55.610 # + election-leader master myMaster 127.0.0.1 6379
23617:X 19 Aug 2020 16:03:55.610 # + fail-state-select-slave master mymaster 127.0.0.1 6379
23617:X 19 Aug 2020 16:03:55.669 # - fail-abort -no-good-slave master mymaster 127.0.0.1 6379
23617:X 19 Aug 2020 16:03:55.735 # Next failover delay: I will not start a failover before Wed Aug 19 16:09:55 2020
Copy the code

Cause: Split brain

Split-brain, that is, one of the master’s machines is suddenly disconnected from the normal network and cannot be connected to other slave machines, but in fact the master is still running. At this point, the sentry might assume that the master is down and initiate an election, switching the other slaves to master. At this point, there will be two masters in the cluster, which is called split brain.

The solution

min-slaves-to-write 1
min-slaves-max-lag 10
Copy the code

Indicates that there must be at least one slave and the delay of data replication and synchronization cannot exceed 10 seconds. If the data replication and synchronization delay for all slaves exceeds 10 seconds, then the master will not receive any more requests.

Cluster mode

Sentry and master-slave can’t automatic fault recovery solution, but at the same time there are also difficult to expansion and single storage, the problem of limited ability to read and write, and before is a redis cluster is all the amount of data, when the concurrency value is very high, will meet the bottleneck of memory, concurrent, storage and so on, so you need a real distributed architecture for load balancing.

Introduction to the

Cluster mode realizes distributed storage of Redis data, data fragmentation, each Redis node stores different content, and solves the problem of online node shrinkage (offline) and capacity expansion (online).

Redis Cluster hashes keys to different hash slots based on the CRC16 algorithm. A Redis cluster provides a total of 16,384 hash slots. In theory, the maximum number of nodes in a cluster is 16,384. The official redis recommendation is no more than 1000. Each Redis Instance is responsible for a portion of the hash slot. To add or delete a node, the Redis cluster reshard the slot. The Redis Cluster ensures smooth slot movement.

  • Advantages: No center node (all Redis nodes are peer node, synchronous data using a Gossip protocol), data on multiple Redis instances, according to a tank to store distribution can be node smooth expansion/contraction capacity, while the number of nodes to change, only need to dynamically change the node is responsible for the tank, it is transparent for the client. No need to rely on middleware, low operation and maintenance costs.
  • Disadvantages: Heavy reliance on Redis-trib, lack of monitoring management, slow detection of Failover node, delay in spreading messages to final consistency.

Communication mechanism

Redis Cluster nodes communicate with each other using the Gossip protocol.

Gossip protocols

The Gossip protocol contains multiple messages, including ping, pong, meet, fail, and so on.

  • Meet: A node sends a meet to a new node to join the cluster, and then the new node starts communicating with other nodes.
  • Ping: Each node frequently sends ping messages to other nodes, including its own status and cluster metadata, to exchange metadata with each other.
  • Pong: Returns ping and meeet, containing its own status and other information, and is also used for information broadcasts and updates.
  • Fail: After judging that another node fails, a node sends a fail message to other nodes to inform them that a node is down

Distributed addressing algorithm

  • The hash algorithm
  • Consistent hash algorithm
  • Hash Slot algorithm (used by Redis Cluster)

The Redis cluster has a fixed hash slot of 16384, calculates the CRC16 value for each key, and then modulates the hash slot of 16384 to obtain the hash slot corresponding to the key.

Each master in a Redis cluster will hold some slots. For example, if there are three masters, each master may hold more than 5000 Hash slots. Hash Slot makes it easy to add and remove nodes. Adding a master moves part of the hash slot of another master, and removing a master moves its Hash slot to another master. The cost of moving Hash Slot is very low. The API of the client can use the same hash tag for the specified data.

Any machine down, the other two nodes, does not affect. Because key is looking for a hash slot, not a machine.

Principle of ha and active/standby switchover

The high availability principle of Redis Cluster is almost similar to sentry.

  1. Check whether the node is down
  2. Filter by node
  3. Slave node election

Compared to Sentinel mode, Redis Cluster is powerful and directly integrates replication and Sentinel capabilities.

Cluster Environment Setup

A Redis cluster usually consists of multiple nodes, and the number of nodes must be at least six to ensure a complete high availability cluster. Cluster-enabled yes must be enabled for each node to enable Redis to run in cluster mode.

Therefore, three master and three slave services are set up. The master service is 7000,7001,7002, and the slave service is 7003,7004,7005. As shown in the figure below.

The configuration file

Edit the redis. Conf file, copy 5 copies, and replace 7000 with the following configuration:

port 7000
daemonize yes
pidfile /home/redis/tmp/scripts-redis/cluster/pid/pid_7000.pid
bind 127.0.0.1

loglevel notice
logfile /home/redis/tmp/scripts-redis/cluster/log/log_7000.log
dbfilename "dump.rdb"
dir "/home/redis/tmp/scripts-redis/cluster/data/7000"

cluster-enabled yes
cluster-config-file "/home/redis/tmp/scripts-redis/cluster/config/nodes_7000.conf"
cluster-node-timeout 5000
Copy the code

Start the service

/usr/local/bin/redis-server redis7000.conf 
/usr/local/bin/redis-server redis7001.conf 
/usr/local/bin/redis-server redis7002.conf 
/usr/local/bin/redis-server redis7003.conf 
/usr/local/bin/redis-server redis7004.conf 
/usr/local/bin/redis-server redis7005.conf 
Copy the code

After starting the command, the following figure shows the result.

Create the cluster

Redis-cli --cluster create 127.0.0.1:7000 127.0.0.1:7001 127.0.0.1:7002 127.0.0.1:7003 127.0.0.1:7004 127.0.0.1:7005 --cluster-replicas 1Copy the code

After the command is executed, the following figure is displayed.

Cluster validation

Connect to the client in cluster mode and set the key.

[root@VM_0_4_centos bin]# redis-cli -c -p 7000127.0.0.1:7000 >setA B -> Redirected to slot [15495] located at 127.0.0.1:7002 OK 127.0.0.1:7002>setShort short -> Redirected to slot [2103] located at 127.0.0.1:7000 OK 127.0.0.1:7000> get short"short"127.0.0.1:7000 >Copy the code

After the primary service of port 7000 is stopped, the key stored in port 7000 is automatically stored on 7005, and the cluster automatically switches the primary service to 7005.

127.0.0.1:7000> shutdown
not connected> 
[root@VM_0_4_centos bin]# redis-cli -c -p 7000
Could not connect to Redis at 127.0.0.1:7000: Connection refused
not connected> set a b [EX seconds|PX milliseconds] [NX|XX] [KEEPTTL]
[root@VM_0_4_centos bin]# redis-cli -c -p 7001
127.0.0.1:7001> get short
-> Redirected to slot [2103] located at 127.0.0.1:7005
"short"
127.0.0.1:7005> get name
-> Redirected to slot [5798] located at 127.0.0.1:7001
"hello"127.0.0.1:7001 >Copy the code
127.0.0.1:7005 > info replication# Replicationrole:master connected_slaves:0 master_replid:5b3a8ba01eb3c6afa931cb6143a72340f283e48c master_replid2:47821ec4918bb29baa1f4546f4111f57af88efaa master_repl_offset:1107 second_repl_offset:1108 repl_backlog_active:1 repl_backlog_size:1048576 repl_backlog_first_byte_offset:1 repl_backlog_histlen:1107 127.0.0.1:7005 >Copy the code

conclusion

Redis is highly available and supports three cluster modes: master-slave, Sentinel, and Cluster.

The cluster mode is the best and recommended.

Left hand typing, right hand shooting, a basketball loving coder ~~~

Sync personal blog: Geekvic. top/post/ d254C2…

Disclaimations: All articles on this blog are licensed under by-NC-SA unless otherwise stated. Reprint please indicate the source!