Writing in the front
Before, a friend went to an interview and was asked about Redis sentinel mechanism. In fact, many friends should have been asked about this question! Here’s how to answer that question! At the same time, I drew a map with XMind to record the study notes of Redis and some interview analysis (the source file has detailed remarks and references for some nodes:
What is the sentry mechanism?
Redis’ Sentinel system is used to manage multiple Redis servers. The system performs the following three tasks:
Monitoring: Sentinels constantly check that your Master and Slave are working properly.
** Sentinel can send notifications to administrators or other applications via the API when a monitored Redis has a problem.
Automatic failover :** When a Master does not work properly, Sentinel starts an automatic failover operation. It upgrades one of the failed Master’s slaves to the new Master and makes the other slaves of the failed Master replicate the new Master. If a client attempts to connect to an invalid Master, the cluster returns the address of the new Master to the client, enabling the cluster to replace the invalid Master.
1. Why sentinels?
The sentry mechanism was introduced to address the shortcomings of master-slave replication! Here are the drawbacks of redis master-slave replication:
- If the primary node is faulty, it cannot provide services. You need to manually modify the configuration to change the secondary node to the primary node
- Primary/Secondary replication Write capability of the primary node The write capability of the primary node is limited
- The storage capacity of a single node is also limited
2. High availability of Sentinel
Principle: When the primary node fails, Redis Sentinel automatically completes the fault discovery and transfer, and informs the application party to achieve high availability.
In fact, the whole process only needs one sentinel node to complete. We first implemented the election mechanism using Raft algorithm and selected one sentinel node to complete the transfer and notification
3. Regular monitoring task of sentry
Task 1: Each sentinel node will send the info command to the master node and the slave node every 10 seconds to obtain the topological structure diagram. During sentinel configuration, only the monitoring of the master node is required. By sending info to the master node, the information of the slave node can be obtained, and the new slave node can be immediately sensed when it joins.
Task 2: Every sentry node will send the judgment of the sentry node on the master node and the information of the current sentry node to the designated channel of redis data node every two seconds. Meanwhile, each sentry node will subscribe to this channel to learn the information of other sentries and their judgment on the master node. This is done via publish and subscribe messages.
Task 3: Every second, each sentry sends a ping command to the master node, the slave node and other sentry nodes for heartbeat detection, which is also an important basis for the sentry to judge whether the node is normal.
Objective referral: When the subjectively offline node is the master node, the sentinel 3 node will seek the judgment of other sentinels on the master node through the command sentinel IS-Masterdown-by-addr. When the quorum (election) number is exceeded, the sentinel node will consider that the master node is indeed problematic and will be objectively offline. Most sentinels agree to go offline, that is, go offline objectively.
4. Lead the sentry election process
- Each sentinel node that is online can become a leader. When it confirms (for example, Sentinel 3) that the primary node is offline, it sends is-master-down-by-addr to the other sentinels to solicit judgment and request that it be set as the leader. The leader handles failover.
- When other sentries receive this order, they can either approve or reject it as leader.
- If sentry 3 finds himself in an election where the number of votes is greater than or equal to Num (Sentinels)/2+1, he becomes the leader, and if not, continues to elect …………
5. Failover mechanism
Sentinel node periodically monitors to find out whether the primary node is faulty: Sentinel would send a heartbeat PING to the master to confirm that the master was alive, and if the master did not respond to PONG or returned an error message within a “certain amount of time”, the Sentinel would subjectively (unilaterally) consider the master to be unavailable
When the primary node fails, the Sentinel3 node is jointly elected as the leader by the three Sentinel nodes, and the load handles the failover of the primary node
Failover is performed by the Sentinel3 lead node, in the same process as master-slave replication, but automatically
Process:
- Disconnect slave-1 from the original slave node, upgrade the master node,
- Points the slave node slave-2 to the new master node
- Notifies the client that the primary node is replaced
- Change the oldMaster node (oldMaster) to a slave node pointing to the new master node
Redis Sentinel topology after failover
5. Sentinel mechanism – Failover details – Identify the master node
- Filter out unhealthy (offline or offline) slave nodes that have not responded to the sentry’s ping
- Select salve-priority from the node with the highest priority (redis.conf)
- Select the slave node with the most complete replication offset
conclusion
Redis sentry functions:
- Monitor the primary and secondary databases.
- When the primary database fails, the secondary database can be automatically converted to the primary database for automatic switchover.
Redis related articles recommended
- Question 23 Redis23
- Redis publish subscription mechanism, interview will know!
- Redis read/write separation difficult to understand? This paper analyzes Redis read-write separation technology
- Finally make clear the Redis scene design, need to master all here!
- Enterprise-level Redis monitoring, refined to each project instance
- Hardcore! I spent 5 hours developing this Redis cache solution to take you off the ground!
Three things to watch ❤️
If you find this article helpful, I’d like to invite you to do three small favors for me:
- Like, forward, have your “like and comment”, is the motivation of my creation.
- Pay attention to the public account “ARCHITECTURE notes of A Wind”, regularly share dry goods content!
- Also look forward to the follow-up article ing🚀