The guard mode

Redis series

Redis series cache principle & design

A quick primer on the Redis series

Redis series of data structures

Redis series Redis persistence mechanism

Redis series of master-slave replication

Redis series sentinel mode

Basic introduction

Sentinel is Redis High Availability solution:

  • A Sentinel cluster consisting of one or more Sentinel instances can monitor one or more primary and multiple slave servers.

  • When the primary server goes offline, Sentinel can upgrade one of the secondary servers under the primary server to continue the service of the primary server, thus ensuring the high availability of Redis.

  • The illustration

Sentinel mode setup steps

Make a copy of the sentinel.conf file

Cp sentinel.conf ‐26379.conf cp sentinel.conf ‐26380.conf cp sentinel.conf sentinel‐26381.confCopy the code

2. Modify related configurations

# Sentinel instance runs on port 26379 by default
port 26379

# change 'daemonize' from 'no' to 'yes'
daemonize yes

IP port of the redis master node monitored by Sentinel
#master-name Specifies the name of the master node that can be named by itself. The name can contain only letters a-z, numbers 0-9, and ".-_". When these quorum sentinels consider the master node to be disconnected, the master node is objectively considered to be disconnected
#sentinel monitor <master-name> <ip> <redis-port> <quorum>Sentinel Monitor Master 127.0.0.1 6379 2# When requirePass Foobared is enabled in the Redis instance, all clients connected to the Redis instance must provide the password
Note that the primary and secondary authentication passwords must be the same as the primary and secondary authentication passwords
#sentinel auth-pass <master-name> <password>

sentinel auth-pass master MySUPER--secret-0123passw0rd

# set the number of milliseconds after which the primary node does not answer sentinel sentinel
#sentinel down-after-milliseconds <master-name> <milliseconds>
sentinel down-after-milliseconds master 3000

# This configuration item specifies the maximum number of slaves that can synchronize the new master at the same time during a failover. The smaller the number is, the longer it takes to complete the failover. This means that more slaves are unavailable due to replication. Setting this value to 1 ensures that only one slave at a time is unable to process command requests.
#sentinel parallel-syncs <master-name> <numslaves>

sentinel parallel-syncs master 1

Failover -timeout can be used in the following ways:
#1. The interval between two failover operations for the same sentinel and the same master.
#2. Start time counting when a slave synchronizes data from an incorrect master. Until the slave is corrected to synchronize data to the correct master.
#3. The time required to cancel an ongoing failover.
#4. Maximum time required to configure all Slaves to point to the new Master when performing failover. However, even after this timeout, slaves will still be configured correctly to point to the master, but not according to the rules configured for parallel syncs
#sentinel failover-timeout <master-name> <milliseconds>
sentinelf ailover-timeout master1 80000
Copy the code
docker run -it --name redis-sentinel2639 -v /Users/yujiale/docker/redis/conf/sentinel6379.conf:/etc/redis/sentinel.conf - v/Users/yujiale/docker/redis/data26379: / data - network localNetwork - IP 172.172.0.16 -d redis: 6.2.6 redis - sentinel /etc/redis/sentinel.confCopy the code

3. Start sentinel instance

Enable redis-master and redis-slaver

Conf in the redis-master directory./redis-server redis.conf in the redis-slaver1 directory./redis-server redis.conf in the redis-slaver2 directory./redis-server redis.confCopy the code

# start redis – sentinel

In the redis-sentinel1 directory./redis-sentinel sentinel.conf in the redis-sentinel2 directory./redis-sentinel sentinel.conf In the redis-sentinel3 directory./redis-sentinel sentinel.confCopy the code

4. Check the startup status

Execute the process

1. Start and initialize Sentinel

Sentinel is a special Redis server that does not persist

After the Sentinel instance is started, each Sentinel creates two network connections to the primary server

  • Command connection: Used to send commands to the master server and receive responses

  • Subscription connection: the — sentinel — : Hello channel used to subscribe to the primary server

2. Obtain Master information

By default, Sentinel sends the info command to the monitored primary server every 10 seconds to obtain information about the primary server and its subordinate secondary servers.

3. Get information from Salve

Sentinel also establishes command connections and subscription connections to slave servers when Sentinel detects new slave servers on the primary server.

Sentinel sends the info command to the slave server every 10 seconds after the command connection is established and logs the slave server’s information.

4. Send messages to master and slave servers as subscriptions

By default, Sentinel sends messages every 2s to all monitored master servers and to the — Sentinel — : Hello channel that slave servers subscribe to. The messages carry Sentinel’s own information and the master server’s information.

5. Channel information received from master server and slave server

After Sentinel establishes a subscription connection with the master or slave server, Sentinel sends the following command to the server through the subscription connection

The subscribe - sentinel - : helloCopy the code

The Sentinels only create command connections with each other, but do not create subscription connections, because sentinels can sense the addition of new Sentinels by subscribes to the master or slave server. Once the new Sentinels join, the sentinels that are aware of each other communicate with each other through command connections.

6. Detect subjective offline status

Sentinel sends PING instances once a second to all instances (primary, secondary, and other sentinels) that have been connected to it and returns invalid replies in down-after-milliseconds (except +PONG, -loading, and -masterdow) N) if an instance does not respond in up-after-milliseconds, Sentinel considers the instance to be down.

7, check the objective offline status

When a Sentinel determines that a master server is subjectively offline

Sentinel sends query commands to all other Sentinels that are also monitoring this master server

The host

SENTINEL is-master-down-by-addr <ip> <port> <current_epoch> <runid>
Copy the code

Other Sentinel replies

<down_state> <leader_runid> <leader_epoch>
Copy the code

Determine if they also consider the primary server offline. If the primary server is judged to be subjective offline by all Sentinel instances that reach quorum in the Sentinel configuration, the primary server is judged to be objective offline (ODown).

8. Elect Leader Sentinel

When a primary server is judged to be objectively offline, all sentinels monitoring the primary server select a Leader Sentinel to perform a failover via raft.

The sentry election

Raft

Raft protocol is a protocol designed to address consistency issues in distributed systems.

The node states described by Raft protocol are Leader, Follower, and Candidate.

Term: Raft protocols slice time into terms, which can be considered a “logical time”.

The election process

Raft uses the heartbeat mechanism to trigger the Leader election

  • After the system starts, all nodes are initialized as followers and the term is 0.

  • A node retains its Follower status if it receives RequestVote or AppendEntries

  • If the node does not receive the AppendEntries message for a period of time and the Leader is not found within the timeout period, the Follower becomes a Candidate and starts running for the Leader.

    • Once converted to Candidate, the node immediately starts the following things:

      • Add your own term.
      • Start a new timer.
      • Vote for yourself.
      • Send RequestVote to all other nodes and wait for the other nodes to reply.

  • If the node receives a majority of the nodes’ consent votes before the timer expires, it becomes the Leader. At the same time send AppendEntries to all other nodes to indicate that you have become the Leader.

  • Each node can only vote one time per term, which is a first-come, first-served policy. Followers will vote for the first node that receives a RequestVote.

  • The Raft protocol’s timer takes a random timeout, which is key to electing a Leader.

  • In the same term, the node that becomes Candidate first votes first, thus gaining the majority of votes.

Sentinel’s leader election process

  • 1. After a Sentinel determines that the master is offline objectively, the Sentinel will first see if it has voted. If it has voted for other sentinels, it will not become the Leader within a certain period of time.

  • 2. If the Sentinel has not already been voted on, it becomes a Candidate.

  • 3. Sentinel needs to do several things:

    • Update failover status to start
    • If the current epoch is added by 1, it is equivalent to entering a new term, which in Sentinel is the term in the Raft protocol.
    • The is-is master-down-by-addr command is sent to another node to request a vote. The command will carry its own epoch.
    • Vote for yourself (leader, Leader_epoch)

  • 4. When other sentries receive this order, they may agree or refuse to be the leader. (By judging the epoch)

  • 5. A Candidate will continuously count his votes until he finds that more than half of the votes approve him to be the Leader and exceeds its quorum configuration, at which point he becomes the Leader.

  • 6. Other Sentinels wait for the Leader to select the master from the slave and remove the indication of objective offline when the new master is detected to work normally.

failover

Once the Leader Sentinel is elected, the Leader Sentinel performs a failover operation on the primary server that goes offline

  • 1. It will upgrade one Slave of the invalid Master to the new Master, and make the other slaves of the invalid Master copy the new Master.

  • 2. When a client attempts to connect to the invalid Master, the cluster returns the address of the new Master to the client. In this way, the cluster can replace the invalid Master with the current Master.

  • 3. After the Master and Slave servers are switched, the contents of the redis.conf configuration files of Master, redis.conf and sentinel.conf configuration files of Slave will be changed accordingly. The redis.conf configuration file of the Master server will have an extra line of Replicaof configuration, and the sentinel.conf monitoring target will be changed accordingly.

Primary server selection

  • 1. Filter out the offline nodes
  • 2. Select the node with the highest slave-priority. If yes, continue to select the node
  • 3. Select the system node with the largest replication offset, because the larger the replication offset is, the more complete the data replication will be. If yes, the system returns the data
  • 4. Select the node with the smallest run_id, because a smaller run_id means fewer restarts