How to ensure high concurrency and availability of Redis?
January 23, 2024
by Katherine Todd
No Comments
The interview questions
How to ensure high concurrency and availability of Redis? Can you introduce the master slave replication principle of Redis? Can you introduce redis’ sentry principle?
Interviewer psychoanalysis
In fact, ask this question, mainly test you, Redis single machine can bear how high concurrency? How to expand capacity to carry more concurrency if a single machine fails? Will Redis fail? How can redis be highly available if it will hang?
These are questions that you must consider in your project, and if you haven’t, you haven’t given enough thought to the problems in your production system.
Analysis of interview questions
If you are using Redis caching, you must consider how to use Redis to add multiple machines, ensure that Redis is highly concurrent, and how to ensure that Redis does not die when it dies, i.e. redis is highly available.
Since this section is quite extensive, it will be divided into two sections.
Redis master-slave architecture
Redis is highly available based on sentry
Redis mainly relies on the master-slave architecture to achieve high concurrency. Generally speaking, many projects are actually enough. A single master is used to write data, a single machine tens of thousands of QPS, multiple slave is used to query data, and multiple slave instances can provide 10W QPS per second.
If you want to accommodate large amounts of data while achieving high concurrency, you need redis clusters, which can provide hundreds of thousands of concurrent reads and writes per second.
Redis is highly available, if it is master slave architecture deployment, then add sentinel can be achieved, any instance down, you can carry out master/standby switchover.
Redis master-slave architecture
A single REDis can carry QPS ranging from tens of thousands to tens of thousands. For caches, this is generally used to support high read concurrency. Therefore, the architecture is made into a master-slave architecture, with one master and many slaves. The master is responsible for writing and copying data to other slave nodes, and the slave nodes are responsible for reading. All read requests go from the node. In this way, it is also easy to achieve horizontal expansion and support high read concurrency.
Redis replicates data to slave nodes asynchronously. However, starting from Redis2.8, slave nodes periodically confirm the amount of data they replicate each time.
A master node can be configured with multiple slave nodes.
Slave nodes can also connect to other slave nodes.
The slave node does not block the normal operation of the master node.
The slave node does not block its own query operation when performing replication. Instead, it uses the old data set to provide services. However, when the replication is complete, the old data set needs to be deleted and the new data set needs to be loaded. At this time, the external service will be suspended.
The slave node is used for horizontal capacity expansion and read/write separation. The expanded slave node improves read throughput.
Note that if you use a master/slave architecture, it is recommended to enable persistence for master nodes. It is not recommended to use slave nodes as data hot standby for master nodes, because then if you turn off persistence for master nodes, The data may be empty when the master is down and restarted, and may be lost as soon as the slave node is replicated.
In addition, the master of various backup schemes, also need to do. If all local files are lost, select an RDB from the backup to restore the master. This ensures that data is available at startup. Even with the high availability mechanism described later, the slave node can automatically take over the master node. However, it is also possible that the Master node automatically restarts before Sentinel detects a master failure, or that all slave node data on it may be wiped clean.
The core principles of Redis master-slave replication
When a slave node is started, it sends a PSYNC command to the master node.
If this is the first time that the slave node connects to the master node, a full resynchronization full replication is triggered. At this point, the master starts a background thread to generate a snapshot file of the RDB and cache all the new write commands received from the client. After the RDB file is generated, the master sends the RDB file to the slave. The slave first writes the RDB file to the local disk and then loads the RDB file from the local disk to the memory. The master then sends the write commands cached in the memory to the slave and the slave synchronizes the data. If the slave node is disconnected from the master node due to a network fault, the slave node automatically reconnects to the slave node. After the connection, the master node copies only the missing data to the slave node.
Breakpoint continuation for master/slave replication
Since redis2.8, breakpoint continuation of master/slave replication is supported. If the network connection is down during master/slave replication, the replication can continue where the last replication was made, rather than starting from scratch.
Master nodes maintain a backlog in memory. Both master and slave nodes maintain a replica offset and a Master run ID. Offset is stored in the backlog. If the network connection between the master and slave breaks down, the slave asks the master to continue replication from the last replica offset. If no corresponding offset is found, a resynchronization operation is performed.
Locating the master node based on host+ IP is unreliable. If the master node restarts or data changes, the slave nodes should be distinguished based on different RUN ids.
Diskless replication
The master creates the RDB in memory and sends it to the slave instead of landing on its own disk. Simply enable repl-diskless-sync yes in the configuration file.
repl-diskless-sync yes
Wait 5 seconds before starting replication, as more slaves need to reconnect
repl-diskless-sync-delay 5
Copy the code
Handling expired Keys
The slave does not wait for the master key to expire. If the master expires a key or discards a key via the LRU, a del command is emulated and sent to the slave.
The complete process of replication
When the slave node starts, it saves the master node information, including the host and IP address of the master node, but the replication process does not start.
The slave node has a scheduled task that checks every second to see if there are new master nodes to be connected and replicated. If so, the slave node establishes socket connections with the master node. The slave node then sends the ping command to the master node. If master has requirePass set, slave node must send masterauth’s password for authentication. The master node performs full replication for the first time and sends all data to the slave node. Later, the master node asynchronously copies the write command to the slave node.
Full amount of copy
Master performs bgSave to generate an RDB snapshot file locally.
The master node sends the RDB snapshot file to the slave node. If the RDB replication time exceeds 60 seconds (Repl-timeout), the slave node considers that the replication fails. You can adjust this parameter appropriately. Generally 100MB, 6GB file transfer per second, probably more than 60 seconds)
When the master node generates an RDB, it caches all new write commands to the memory. After the master node saves the RDB, the master node copies the new write commands to the slave node.
If memory buffer consumption exceeds 64MB continuously during the replication or exceeds 256MB at a time, the replication is stopped and the replication fails.
client-output-buffer-limit slave 256MB 64MB 60
Copy the code
After receiving the RDB, the slave node clears its old data, reloads the RDB into its own memory, and provides services based on the old data version.
If AOF is enabled on the slave node, BGREWRITEAOF is immediately executed to override the AOF.
Incremental replication
If the master-slave network connection breaks during the full replication, incremental replication is triggered when the slave reconnects to the master.
The master takes some of the missing data directly from its own backlog and sends it to slave nodes. The default backlog is 1MB.
The master retrieves the backlog from the offset in the psync sent by the slave.
heartbeat
Both the primary and secondary nodes send heartbeat information to each other.
By default, the master sends a heartbeat every 10 seconds, and the slave node sends a heartbeat every 1 second.
Asynchronous replication
After receiving a write command, the master writes data internally and asynchronously sends the data to the slave node.
How can Redis be highly available
A system is highly available if it is available 99.99% of the time within 365 days.
The failure of one slave does not affect availability, as other slaves provide the same external query service with the same data.
But what happens if the master node dies? I can’t write data. When I write to the cache, it all fails. What’s the point of a slave node? With no master to copy data to them, the system is virtually unusable.
The high availability architecture of Redis is called failover, also known as master/slave switchover.
When a master node fails, it automatically detects the fault and switches a slave node to the master node. This is called active/standby switchover. This process implements high availability under the master-slave architecture of Redis.
The Sentinel-based high availability of Redis will be explained in more detail later.
The Redis Sentinel cluster implements high availability
Introduction to the Sentry
Sentinel is Chinese name for sentinel. Sentinel is a very important component in redis cluster architecture. It has the following functions:
Cluster monitoring: Monitors whether the Redis master and slave processes are working properly.
Message notification: If a Redis instance fails, the sentry is responsible for sending a message as an alarm notification to the administrator.
Failover: If the master node fails, it is automatically transferred to the slave node.
Configuration center: Notifies the client client of the new master address if failover occurs.
Sentinel is used to achieve high availability of redis cluster, itself is also distributed, as a sentinel cluster to run, work with each other.
During failover, determining whether a master node is down requires the agreement of most of the sentinels, which relates to distributed elections.
Even if some of the sentinels fail, the sentinels will still work, because if a failover system that is an important part of the high availability mechanism is itself a single point of failure, it will be bad.
The core knowledge of sentinels
Sentinels need at least three instances to be robust.
Sentinel + Redis master-slave deployment architecture does not guarantee zero data loss, only high availability of redis cluster.
For sentry + Redis master-slave complex deployment architectures, try to do adequate testing and practice in both test and production environments.
The sentinel cluster must have more than two nodes deployed, and if the sentinel cluster has only two sentinel instances deployed, quorum = 1.
If quorum=1 is configured, if the master fails, as long as only one sentry in S1 and S2 thinks the master is down, the switch can be performed, and s1 and S2 will elect a sentry to perform the failover. But at the same time, the majority needs to be running.
If at this point only the M1 process is down and Sentry S1 is running properly, then failover is OK. However, if the entire M1 and S1 machine goes down, and the sentinel only has 1, there is no majority to allow a failover, even though there is an R1 on the other machine.
A classic 3-node sentinel cluster looks like this:
Quorum =2 if M1 fails, there are two sentinels remaining. S2 and S3 can agree that master is down, and then elect one to perform failover. Majority of the three sentinels is 2, so the remaining two sentinels are running. You can allow failover.
Data loss during redis Sentry active/standby switchover
Two conditions that result in data loss
Data may be lost during the active/standby switchover:
Data loss caused by asynchronous replication
Because master->slave replication is asynchronous, some data may break down before it is replicated to the slave, and the data may be lost.
Data loss due to split-brain
Split-brain, that is, one of the master’s machines is suddenly disconnected from the normal network and cannot be connected to other slave machines, but in fact the master is still running. At this point, the sentry might assume that the master is down and initiate an election, switching the other slaves to master. At this point, there will be two masters in the cluster, which is called split brain.
Even though a slave is switched to the master, the client may continue to write data to the old master before switching to the new master. Therefore, when the old master is recovered, it is attached to the new master as a slave, and its own data is cleared and data is copied from the new master. The new master does not have the data that the client wrote later, so the data is lost.
Solutions to data loss problems
Perform the following configuration:
min-slaves-to-write 1
min-slaves-max-lag 10
Copy the code
Indicates that there must be at least one slave and the delay of data replication and synchronization cannot exceed 10 seconds.
If the data replication and synchronization delay for all slaves exceeds 10 seconds, then the master will not receive any more requests.
Reduce the loss of asynchronous replication data
With min-rabes-max-lag this configuration ensures that if the slave replicated data and ack delay was too long, it would think that too much data was lost after the master went down and would reject the write request. In this way, the data loss caused by the failure of the master to synchronize some data to the slave can be reduced to a manageable extent.
Reduce data loss in split-brain
If a master has a split brain and loses contact with other slaves, the above two configurations ensure that if the master cannot continue to send data to a specified number of slaves and the slave does not send itself an ACK message for more than 10 seconds, the write request will be rejected. So in the split-brain scenario, at most 10 seconds of data is lost.
Sdown and ODOWN conversion mechanisms
Sdown is a subjective outage, just a sentry who thinks a master is down is subjective outage
Odown is an objective outage, and if the sentinels of the quorum quantity feel that a master is down, it is an objective outage
Sdown does a simple test: if a sentry pings a master for more than the number of milliseconds specified for IS-IS master-down-after-milliseconds, the master is down. If a sentinel receiving a quorum quantity within a specified period of time is also considered sdown by other sentinels, it is considered ODown.
Automatic discovery mechanism for sentinel cluster
Sentinels discover each other through redis’ pub/sub system, where each sentinel sends a message to the __sentinel__: Hello channel, where all other sentinels can consume the message and become aware of the presence of the other sentinels.
Every two seconds, each sentinel will send a message to the __sentinel__: Hello Channel corresponding to master+ Slaves that they monitor about their host, IP and RUNId as well as the monitoring configuration of the master.
Every sentinel will also monitor __sentinel__: Hello Channel corresponding to every master+ Slaves they monitor, and then sense the presence of other sentinels who are also monitoring this master+ Slaves.
Each sentry also exchanges master monitoring configurations with other sentries to synchronize the monitoring configurations with each other.
Automatic correction of slave configurations
The sentry is responsible for automatically correcting some configurations of the slave. For example, if the slave is to become a potential master candidate, the sentry makes sure that the slave copies the data of the existing master. If the slaves are connected to the wrong master, such as after a failover, then the sentinels ensure that they are connected to the correct master.
Slave -> Master Election algorithm
If a master is considered to be oDown and the majority of sentries allow a master/slave switchover, a majority sentry will perform the master/slave switchover.
Duration of disconnection from master
Slave priority
Copy the offset
run id
If a slave has disconnected from the master for more than 10 times the number of down-after-milliseconds, plus how long the master has been down, then it is considered unfit to vote master.
(down-after-milliseconds * 10) + milliseconds_since_master_is_in_SDOWN_state
Copy the code
The slave is then sorted:
The slave priority is sorted by slave priority. The lower slave Priority is, the higher priority is.
If the slave priority is the same, check replica offset to see which slave replicates more data. The lower the offset is, the higher the priority is.
If both conditions are the same, select a slave with a smaller RUN ID.
The quorum and majority
Each time a sentinel switches master/standby, the quorum sentinels must first consider the switch odown and then elect a sentinel to do the switch, which must also be authorized by the majority sentinels to officially perform the switch.
If quorum < majority, such as five sentinels, majority is 3, and quorum is set to 2, then three sentinels authorization can perform the switch.
However, if quorum >= majority, then all sentinels of the quorum number must be authorized, such as five sentinels, and quorum is 5, then all five sentinels must agree on authorization before the switch can take place.
configuration epoch
Sentinel will monitor a set of Redis Master + Slaves and have the corresponding monitoring configuration.
The sentry performing the switch gets a Configuration epoch from the new master (Salve -> Master) to which it is switching. This is a version number that must be unique for each switch.
If the first elected sentry fails to switch, the other sentries will wait for fail-timeout and then continue to switch. At this time, a new Configuration epoch will be obtained as the new version.
Spread the configuration
After the switch is complete, the sentry updates the master configuration locally and synchronizes it to the other sentries via pub/ SUB messaging.
The previous version number is important here because messages are published and listened to through a channel, so when a sentinel makes a new switch, the new master configuration follows the new version number. The other sentinels update their master configuration based on the size of the version number.
Pay attention to my wechat official number, the first time to get the update of my blog remind, more surprise waiting for you ~
Scan the QR code below or search for wechat account shenshan_laoyuan
This article was automatically published by ArtiPub
