In the last article, WE talked about the master-slave replication of Redis and how it works. In the next article, we will talk about the sentry mechanism of Redis high availability based on replication

The guard mechanism

What is sentry mechanism

In master-slave replication mode, after the primary server is down, the secondary server cannot be automatically transferred to the primary server. You can only manually failover the secondary server

Sentinel mechanism (Sentinel) is the high availability solution of Redis. Sentinel system consists of one or more Sentinel instances to manage the master server and all connected slave servers. You can realize fault discovery, automatic failover, configuration center, and client notification

• Monitoring: Sentinel continuously checks whether your primary and secondary servers are functioning properly
• Notification: Sentinel can send notifications to administrators or other applications via the API when a monitored Redis server has a problem
• Automatic failover: When a primary server does not work properly, Sentinel starts an Automatic failover operation. It upgrades one of the secondary servers of the failed primary server to the new primary server and replicates the other secondary servers of the failed primary server. When a client tries to connect to a failed primary server, the cluster also returns the address of the new primary server to the client, allowing the cluster to use the new primary server in place of the failed server

Looking at the figure above, we can understand that the sentry mechanism is a monitoring system based on replication, which can fail over in time. So let’s first understand how Redis fault diagnosis and failover

Fault judgement

Failure determination, which is what Sentinel is doing, when a Sentinel is started, the following tasks are performed periodically

• Each Sentinel process will start withOnce per secondIs sent to known master servers, slave servers, and other Sentinel instancesPINGThe command
• Each instance is from the last valid replyPINGCommand time exceededdown-after-millisecondsOption, then the instance is marked asSubjective offline. An effective response could be:+PONG,-LOADINGor-MASTERDOWN
• If aMasterThe primary server is marked asSubjective offline, then all sentinels that monitor the master server will use theOnce per secondVerify that the primary server is indeed in the mainline offline state
• If there are enough sentinels (at least as many as specified in the configuration file), confirm one within the specified time rangeMasterThe primary server isSubjective offlineState, then the master server is marked asObjective offline
• In general, each Sentinel willEvery 10 secondsIs sent to known master and slave serversINFOCommand when a primary server is marked asObjective offlineWhen sendingINFOThe command frequency will be changed to once per second
• The objective offline status of the primary server will be removed if a sufficient number of Sentinels do not allow the primary server to go offline. When the master server redirects to SentinelPINGWhen the command returns a valid reply, the subjective offline status of the master server is removed

Subjective offline (Subjectively Down, SDOWN) refers to a single Sentinel instance that determines the offline status of a server

Objectively Down (ODOWN) refers to an SDOWN judgment of servers that is made by an Objectively number of Sentinel instances and communicated with each other through the Sentinel is-master-down-by-addr command

Build sentinel instance

To prepare

Here I have prepared 3 virtual machines, 1 master and 2 slave Redis servers and 3 sentinel instances. On the official website, it is recommended to deploy at least three Sentinel instances to ensure robustness. The specific addresses and ports are as follows

1 The configuration of the primary and secondary Redis services is as follows. Modify the Redis

Set to run in the background
daemonize yes
# bind host address, commented out here, open IP connection
# bind 127.0.0.1

The password can be set on the main service
requirepass 123456
Copy the code

Adds the address information of the primary server to the secondary server

Set the address of the primary server
# replicaof <masterip> <masterport>Replicaof 192.168.249.20 6379Set the primary server password
# masterauth <master-password>
masterauth 123456
Copy the code

You can copy the sentinel.conf file from the Redis installation directory as follows

Set to run in the background
daemonize yes
logfile "26379.log"
Quorum 2 indicates that failover should occur when two or more sentinels deem the primary server unavailable
# sentinel monitor <master-name> <ip> <redis-port> <quorum>Sentinel Monitor MyMaster 192.168.249.20 6379 2Set the server password
# sentinel auth-pass <master-name> <password>
sentinel auth-pass mymaster 123456

# specifies the number of milliseconds needed for Sentinel to determine that the server is offline
# sentinel down-after-milliseconds <master-name> <milliseconds>
sentinel down-after-milliseconds mymaster 30000
# specifies the maximum number of slave servers that can synchronize with the new master server during a failover. The smaller the number, the longer the failover will take. However, a larger number means that more slaves are unavailable due to Replication.
This value can be set to 1 to ensure that only one slave at a time is unable to process command requests
# sentinel parallel-syncs <master-name> <numreplicas>
sentinel parallel-syncs mymaster 1
# specify failover timeout
# sentinel failover-timeout <master-name> <milliseconds>
sentinel failover-timeout mymaster 180000
Copy the code

Sentinel can automatically discover other Sentinels that are also monitoring the master server through the publish/subscribe function of the Redis instance, as well as obtain information from other slave servers through the master server, so we only need to configure the address of the master server

Start the

The startup sequence is master, slave, and then the three sentinels

Start the Redis service

./redis-server .. /redis.confCopy the code

Sentinel is actually a Redis service, which can be regarded as a special mode of Redis service. There are two kinds of commands to start Sentinel

./redis-server .. /sentinel.conf --sentinel ./redis-sentinel .. /sentinel.confCopy the code

The effect of starting Sentry is the same in both ways, and the configuration file must be used to start the run, because the system needs this file to save the current state for loading upon reboot

Viewing host Status

We can connect to the Sentinel with the client and use sentinel Master

to see the status of the master server it monitors

$./redis-cli -h 192.168.249.20 -p 26379 192.168.249.20:26379> Sentinel master mymaster 1)"name"
 2) "mymaster"
 3) "ip"
 4) "192.168.249.20"
 5) "port"
 6) "6379"
 7) "runid"
 8) "2a58b746fe9c963e69887efbc877aa3143eb6ec6"
 9) "flags"
10) "master"
11) "link-pending-commands"
12) "0"
13) "link-refcount"
14) "1"
15) "last-ping-sent"
16) "0"
17) "last-ok-ping-reply"
18) "167"
19) "last-ping-reply"
20) "167"
21) "down-after-milliseconds"
22) "30000"
23) "info-refresh"
24) "2941"
25) "role-reported"
26) "master"
27) "role-reported-time"
28) "2006632"
29) "config-epoch"
30) "0"
31) "num-slaves"
32) "2"
33) "num-other-sentinels"
34) "2"
35) "quorum"
36) "2"
37) "failover-timeout"
38) "180000"
39) "parallel-syncs"
40) "1"
Copy the code

We see some monitoring information about the master server

Num-slaves refers to the number of slave servers; Num-other-sentinels The number of other sentinels; Flags is master. When the host goes offline, it changes to S_down or o_DOWN

In addition, to view the status of other slave servers and other sentinels, use the following command

sentinel slaves <master-name>
sentinel sentinels <master-name>
Copy the code

failover

We can manually simulate a failover situation when the primary server goes down.

To obtain the address information of the current primary server, the client can run the sentinel get-master-addr-by-name

command

192.168.249.20:26379> sentinel get-master-addr-by-name mymaster
1) "192.168.249.20"
2) "6379"
Copy the code

You can see that the current main service address is 192.168.249.20. At this point, we can either manually disable the service or use the following command to put the server to sleep for 40 seconds

$ ./redis-cli -h 192.168.249.20  -p 6379
192.168.249.20:6379> auth 123456
OK
192.168.249.20:6379> debug sleep 40
OK
(40.01s)
Copy the code

When we look at the address of the main service again, we should see different information

192.168.249.20:26379> sentinel get-master-addr-by-name mymaster
1) "192.168.249.22"
2) "6379"
Copy the code