Last time, we studied the master-slave repository cluster pattern. In this mode, if the slave fails, the client can continue to send requests to the master or other slave libraries for related operations, but if the master fails, it will directly affect the synchronization of the slave library, because the slave library has no corresponding master library to replicate data.
Moreover, if the client sends all read requests, the slave library can continue to provide the service, which is acceptable in a pure read business scenario. However, once there is a write request, according to the requirements of read/write separation in master/slave mode, the master library must complete the write operation. At this point, there is no instance to service the client’s write request, as shown below:
Neither write service interruption nor data synchronization from the library is acceptable. So, if the master library dies, we need to run a new master library, such as switching a slave library to the master as the master. This involves three questions:
- Is the main library really down?
- Which slave library should be selected as the master library?
- How do I inform slave libraries and clients about the new master library?
Which brings us to the sentry mechanism. In Redis master-slave cluster, the sentinel mechanism is the key mechanism to realize the automatic switch between master and slave libraries, which effectively solves the three problems of failover in master-slave replication mode.
Next, let’s study the sentry mechanism together.
Basic flow of sentry mechanism
Sentinel is essentially a Redis process that runs in special mode at the same time that the master and slave library instances are running. Sentinels are responsible for three main tasks: monitoring, master selection (master library selection), and notification.
Let’s start with the surveillance. Monitoring is when the sentinels periodically PING all the master and slave libraries while they are running to check if they are still online. If the slave library does not respond to the sentinel’s PING command within the specified time, the sentinel marks it as “offline.” Similarly, if the master library does not respond to the sentinel’s PING command within the specified time, the sentinel determines that the master library is offline and begins the process of automatically switching the master library.
The process begins by performing the sentry’s second task, master selection. When the master library dies, the sentinel needs to select an instance of the slave library from many slave libraries according to certain rules and take it as the new master library. Once this step is complete, you now have a new master library in your cluster.
Then, the sentry performs one final task: notification. When performing the notification task, the sentinel sends the connection information of the new master database to the other slave databases, which then run replicaof to establish a connection with the new master database and replicate the data. At the same time, sentry notifies the clients of the connection information of the new master library and asks them to send the requested actions to the new master library.
I drew a picture showing the three tasks and their respective goals.
Among the three tasks, the notification task is relatively simple. The sentinel only needs to send the information of the new master library to the slave library and the client, so that they can establish a connection with the new master library, and does not involve the logic of decision. However, in the two tasks of monitoring and choosing the master, the sentry needs to make two decisions:
- In the monitoring task, the sentry needs to determine whether the master library is offline.
- In the primary selection task, the sentry also decides which slave library instance to select as the primary library.
Next, we will discuss how to determine the offline status of the main library.
The first thing you need to know is that sentry has two types of referrals for main libraries: subjective referrals and objective referrals. So why are there two judgments? What are the differences and connections?
Subjective and objective logoff
Let me first explain what “subjective logoff” is.
The Sentinel process uses the PING command to determine the status of the instance by detecting its own network connection to the master and slave libraries. If the sentinel finds that the response to the PING from the primary or secondary library has timed out, it marks it as “subjective offline.”
If you are checking from the repository, the sentry simply marks it as a “subjective offline”, since offline from the repository is generally not too disruptive and the cluster’s external services are not interrupted.
However, if you are checking the master library, sentry cannot simply mark it as “subjective offline” to enable a master/slave switch. There is a good chance that the sentry misjudged that the main library was not faulty. However, once a master/slave switchover is initiated, subsequent master selection and notification operations incur additional computing and communication overhead.
To avoid these unnecessary costs, we need to pay special attention to miscarriages of justice.
First, we need to know what miscarriage of justice is. Simply, the main library was not actually offline, but the sentry mistakenly thought it was. Misjudgment generally occurs when the cluster network pressure is high, the network congestion, or the main library itself pressure is high.
Once the sentry determines that the master is offline, it starts to select the new master and synchronizes data between the slave and the new master. This process has its own costs, for example, the sentry takes time to select the new master and the slave takes time to synchronize with the new master. In the case of miscalculation, the main library itself does not need to be switched at all, so the overhead of this process is not valuable. Because of this, we need to determine whether there are misjudgments and reduce them.
So how do you reduce miscarriages of justice? In our daily life, when we have to make a judgment on something important, we often discuss it with our family or friends before making a decision.
The Sentinel mechanism is similar in that it is typically deployed in a multi-instance cluster pattern, also known as a sentinel cluster. By introducing multiple sentinel instances to judge, one sentinel can avoid misjudging the main library offline because of its own bad network condition. At the same time, the probability of multiple sentinels’ network being unstable at the same time is smaller, and the misjudgment rate can be reduced by them making decisions together.
In this lesson, you just need to understand the sentinel cluster’s role in reducing misjudgment. As for the specific operation mechanism, we will focus on it in the next class.
It is not up to a sentinel to determine whether the main library is offline or not. The main library is marked as “offline objectively” only when most sentinel instances have judged that the main library has been “offline subjectively”, which means that the main library has become an objective fact. The rule of judgment is that the minority is subordinate to the majority. At the same time, this further triggers the sentry to start the master-slave switching process.
And just to make it easier for you, let me draw another picture to show you the logic here.
As shown in the figure below, the Redis master-slave cluster has one master library, three slave libraries, and three sentinel instances. On the left side of the image, Sentry 2 determines that the master library is “subjectively offline,” but Sentry 1 and 3 determine that the master library is online. At this point, the master library is still judged to be online. On the right side of the image, both Sentry 1 and 2 judge the master library to be “subjectively offline.” At this point, the master library is marked as “objectively offline” even though Sentry 3 still judges the master to be online.
To put it simply, the criterion of “objective offline” is that when there are N sentinel instances, it is better to have N/2 + 1 instances to judge the master database as “subjective offline” before the master database can be judged as “objective offline”. In this way, the probability of misjudgment can be reduced, and unnecessary master/slave library switching caused by misjudgment can be avoided. (Of course, the number of instances that make a “subjective offline” decision is up to the Redis administrator).
Ok, so here you can see that we can more accurately determine if the main library is offline, thanks to the joint determination mechanism of multiple Sentinel instances. If the master library does go offline, the sentry begins the next decision process, which is to select a slave library from among many slave libraries to be the new master library.
How do I select a new master library?
In general, I refer to the sentry’s selection of a new master library as “screening + scoring.” To put it simply, in multiple slave libraries, we first remove the unqualified ones from the library according to certain screening conditions. Then, we scored the remaining slave libraries one by one according to certain rules, and selected the slave library with the highest score as the new master library, as shown in the figure below:
In the preceding paragraph, we need to pay attention to the two “must”. Now, we need to consider what the “must” refers to.
Let’s start with the criteria for filtering.
In general, we must first ensure that the selected slave library is still running online. However, when the slave library is online at the time of the master selection, it just means that the slave library is in good condition, not that it is the best fit for the master library.
Imagine if, during master selection, a slave library works, and we select it as the new master library to start using. But soon its network went down, and we had to choose a new owner. This is obviously not what we were hoping for.
Therefore, in the selection of master, in addition to checking the current online status of the slave library, but also to determine its previous network connection status. If the slave library is constantly disconnected from the master library, and the number of disconnections exceeds a certain threshold, we have reason to believe that the network condition of the slave library is not very good, and we can screen the slave library out.
How do you know? You use the configuration item down-after-milliseconds * 10. Down-after-milliseconds is the maximum connection times we’ve determined that the master/slave library is disconnected. If both the primary and secondary nodes fail to contact each other in down-after-milliseconds, we assume that the primary and secondary nodes are disconnected. If disconnections occur more than 10 times, the slave library is not networked well enough to be the new master library.
Okay, so we’ve filtered out the slave libraries that don’t fit the master library, and we’re done filtering.
Now it’s time to score the rest of the slave library. We can do three rounds of scoring according to three rules, which are the priority of the slave library, the progress of the slave library and the slave library ID number. As long as the slave library has the highest score in a particular round, it is the master library, and the main selection process is over. If the highest scoring slave library does not appear, then the next round is continued.
Round 1: The slave library with the highest priority gets the highest score.
You can use the slave-priority configuration item to set different priorities for different slave libraries. For example, if you have two slave libraries with different memory sizes, you can manually assign a higher priority to the instance with the larger memory. When selecting a master, the sentry gives a high score to the slave library with the highest priority. If one of the slave libraries has the highest priority, it is the new master library. If the priority from the library is the same, the sentry starts the second round of scoring.
Round 2: The slave library that synchronizes most closely with the old master gets the highest score.
This rule is based on the fact that if you select the slave that synchronizes most closely with the old master as the master, the new master will have the latest data on it.
How do you determine the synchronization progress between the slave and the old master?
Last time I showed you how a command propagates when a master/slave library is synchronized. During this process, the master library records the position of the current latest write operation in repl_backlog_buffer with master_REPL_offset, while the slave library records the current replication progress with slave_REPL_offset.
At this point, we want to find the slave library whose slave_REPL_offset needs to be closest to master_REPL_offset. If the slave library slave_REPL_offset is closest to master_REPL_offset of all slave libraries, it gets the highest score and can be used as the new master library.
As shown in the figure below, if master_REPL_offset for the old master is 1000 and Slave_REPL_offset for slave libraries 1, 2 and 3 is 950, 990 and 900 respectively, slave library 2 should be chosen as the new master library.
Of course, if two slave libraries have the same slave_REPL_offset value (for example, slave_REPL_offset for both library 1 and library 2 is 990), then they need to be scored in the third round.
Round 3: Those with smaller ids score higher from the library.
Each instance will have an ID, similar to the slave library number here. Currently, Redis has a default rule when selecting a master library: under the same priority and replication progress, the slave library with the smallest ID will be selected as the new master library with the highest score.
At this point, the new master library is selected, and the “master selection” process is complete.
Let’s review the process again. First, the sentinel will filter out some unqualified slave libraries according to online status and network status. Then, it will score the remaining slave libraries according to priority, replication progress and ID number. As long as there is a slave library with the highest score, it will be selected as the new master library.
summary
In this class, we learned about the sentry mechanism, which is an important guarantee for the continuous service of Redis. To be specific, data synchronization between master and slave clusters is the basic guarantee of data reliability. When the master library fails, the automatic master/slave switchover is the key support for continuous service.
The sentinel mechanism of Redis automatically completes the following three functions, thus realizing the automatic switch between master and slave libraries, which can reduce the operation and maintenance overhead of Redis cluster:
- Monitor the running status of the master library and judge whether the master library is offline objectively;
- After the main library is offline objectively, the new main library is selected.
- When the new master library is selected, the slave library and the client are notified.
In order to reduce the misjudgment rate, the sentinel mechanism is usually deployed in multi-instance mode in practical application. Multiple sentinel instances judge whether the master database is offline objectively through the principle of “the minority is subordinate to the majority”. In general, we can deploy three sentinels, and if two of them decide that the main library is “subjectively offline,” the switching process can begin. Of course, if you want to improve your judgment even further, you can increase the number of sentinels by, say, five.
However, using multiple sentinel instances to reduce the misjudgment rate is actually equivalent to forming a sentinel cluster. Therefore, we face some new challenges, such as:
- There are instances of sentinel cluster hang, what to do, will affect the master library status judgment and choose the master?
- Most instances of the Sentinel cluster have reached a consensus on which instance should perform the master/slave switchover after determining that the master library is “objectively offline”?
In order to understand these problems, we have to refer to the sentinel cluster, and next time, we will talk more about the mechanism and problems of sentinel cluster.
Each lesson asking
Well, as usual, LET me give you a quick question. In this class, I mentioned that the sentry mechanism enables automatic switching between master and slave libraries, which is a key support for uninterrupted service. I also mentioned that switching between master and slave libraries takes a certain amount of time. So, please consider whether the client can handle the request properly during this switch. If you want your application to be unaware of service interruptions, is there anything else you need to sentry or do with the client?
You are welcome to discuss with me in the comments section, and you are welcome to help me share today’s content with more people and help them solve their problems together. I’ll see you next time.