Redis guarantees high concurrency and high availability
January 24, 2024
by Tammy Hood
No Comments
If you use Redis cache technology, you must consider how to use Redis to add multiple machines, to ensure that Redis is high concurrency, and also how to ensure that Redis does not die when it dies, Redis high availability
Redis high concurrency: master-slave architecture, one master with many slaves, generally speaking, many projects are actually enough, single master for writing data, single machine tens of thousands of QPS, multiple slave for querying data, multiple slave instances can provide 100,000 QPS per second.
While redis has high concurrency, it also needs to accommodate a large amount of data: one master, many slaves, each instance holds complete data, for example, Redis has only 10GB of memory, in fact, you can only accommodate 10GB of data. If your cache is going to hold a lot of data, tens of gigabytes, or even hundreds of gigabytes, or terabytes, then you need a Redis cluster, and with a Redis cluster, you can have hundreds of thousands of concurrent reads and writes per second.
Redis high availability: If you deploy master slave architecture, in fact, it is added to the sentinel can be implemented, any instance down, the automatic master/standby switchover. The following details
How does Redis handle read requests with read/write separation over 100,000 + QPS?
1. The relationship between high concurrency in Redis and high concurrency in the whole system
Redis to engage in high concurrency, it is necessary to do a good job of the underlying cache, so that fewer requests directly to the database, because the high concurrency of the database implementation is more troublesome, and some operations and transaction requirements, so it is difficult to achieve a very high concurrency. Redis concurrency is not enough for the whole system concurrency, but redis is a very important part of the large cache architecture that supports high concurrency. To realize the high concurrency of the system, the cache middleware and the cache system must be able to support the high concurrency first, and then after a good overall cache architecture design (multi-level cache, hot cache), can really support the high concurrency.
2. Redis cannot support the bottleneck of high concurrency
The bottleneck of Redis not supporting high concurrency is mainly the stand-alone problem, that is, there is a limit to the performance of a single Redis machine, no matter how good it is.
3. How to support higher concurrency
Single Redis cannot support high concurrency, to support higher concurrency can be read and write separation. Caches generally support high read concurrency and have fewer write requests, so they can be separated based on master/slave architecture.
Configure a master machine to write data, configure multiple slaves to read data, and synchronize data to the slave after the master receives data. In this way, the slave can configure multiple machines to improve the overall concurrency.
Security implications of Redis Replication and Master persistence for master-slave architectures
1. Principle of Redis Replication.
A master node has several slave nodes attached to it. Data is written to the master node. After the master writes data, data is asynchronously synchronized to all slave nodes to ensure the consistency of data on all nodes.
2. The core mechanism of Redis Replication
(1) Redis replicates data to slave nodes asynchronously, but since Redis2.8, Slave nodes periodically confirm the amount of data they replicate each time. (2) A master node can be configured with multiple Salve nodes. (3) The slave node can also connect to other slave nodes. (4) The slave node does not block the normal operation of the master node. (5) When performing replication, the slave node does not block its own operations and uses the old data to provide services. However, when the replication is complete, the old data needs to be deleted and new data needs to be loaded. In this case, the external services are suspended. (6) The slave node is used for horizontal capacity expansion and read/write separation. The expanded slave node can improve throughput.
3. Security implications of master persistence for master-slave architecture
If you use this master-slave architecture, you must enable persistence for the Master Node. It is not recommended to use the slave node as a hot backup for the master node, because if the master node fails, the data on the master node will be lost. After the restart, the data will be empty, and other slave nodes will copy the data to empty. So all the data on all the nodes is lost. Make multiple cold backups of backup files in case the entire machine goes down and backup RDB data is lost.
① When a slave node is started, it sends a PSYNC command to the master node.
(2) If the slave node is reconnected to the master node, the master node will only copy the missing data to the slave node. If the master node is connected for the first time, full resynchronization is triggered.
③ When full resynchronization starts, the master starts a background thread to generate an RDB snapshot file and stores all new write commands received from the client in the memory.
④ The master node sends the generated RDB file to the slave node. The slave node writes the RDB file to the local disk and loads it to the memory. The master node then sends the write commands cached in memory to the slave node, which synchronizes the data.
⑤ If the slave node is disconnected from the master node due to a network fault, the slave node automatically reconnects to the master node.
⑥ If the master finds that there are multiple slave nodes to reconnect, it only starts an RDB save operation and serves all slave nodes with one data copy.
2. Resumable data transmission for the primary/secondary replication
Support for resumable breakpoints is available from Redis2.8. If the network suddenly goes down during a master-slave replication, the replication can continue where it left off, rather than starting from scratch. Principle: Master nodes create a backlog in memory. Both master and slave nodes save a replica offset and a master ID. The offset is stored in the backlog. If the connection between the master and slave breaks, the slave tells the master to continue copying from the replica offset. But if offset is not found, a full resynchronization operation is performed.
3. Diskless replication
Diskless replication means that the master directly creates RDB files in memory and sends them to the slave without saving data on its own disk. Setting Mode Repl-diskless-sync and repl-diskless-sync-delay are configured. Repl-diskless-sync: this parameter ensures diskless replication. Repl-diskless-sync-delay: this parameter indicates that replication starts after a certain period of time. In this case, multiple slave nodes can be connected again.
4. Handle expired keys
The slave does not expire the key, but waits for the master to expire the key. If the master expires a key or discards a key, the master simulates sending a del command to the slave, which then deletes the key.
4. A further in-depth analysis of redis Replication’s complete flow history and principles
1. Complete process of replication
(1) The slave node is started and only the master node information is saved, including the host and IP address of the master node. However, data replication has not started yet. The host and IP address of the master node are configured in slaveOf in the redis.conf file.
(2) The slave node has a scheduled task to check whether a new master node needs to be connected to a replication node every second. If a new master node needs to be connected to a replication node, the slave node establishes a socket network connection with the master node.
③ The slave node sends the ping command to the master node.
(4) If requirePass is set for master, slave node must send master Auth’s password for password authentication.
⑤ The master node performs full replication for the first time and sends all data to the slave node.
⑥ The master node continuously downwrites the data to the slave node asynchronously.
2. Data synchronization mechanism
This is when the slave connects to the master for the first time, performing full replication. (1) Both the master and the slave maintain an offset. The master accumulates offset on its own, and the slave accumulates offset on its own. The slave reports its offsets to the master every second, and the master saves each slave’s offsets. This is not specific for full replication, mainly because the master and slave need to know their data offset, so as to know the data inconsistency between each other
② A backlog master node has a backlog in memory. The default value is 1 MB. When the master node copies data to the slave node, the master node synchronizes the data from the backlog. The backlog is primarily used for incremental replication when full replication breaks.
③ Master Run ID Redis You can view the master run ID on the info server. Purpose: Slave uses it to locate a unique master. Why not use host+ IP: It is unreliable to use host+ IP to locate the master. If the master node restarts or data changes, the slave node should be differentiated according to different master Run ids. If the run ids are different, a full replication is required. To restart Redis without changing the run ID, run redis-cli debug reload.
3. Full replication process and mechanism
① Master executes bgSave to generate a local RDB file.
② The master node sends the RDB snapshot file to the slave node. If the replication time of the RDB file exceeds 60 seconds (Repl-timeout), the slave node fails to replicate the RDB file. You can adjust this parameter appropriately.
③ For gigabit network card machine, the general transmission of 100M per second, transmission of 6G files is likely to exceed 60 seconds.
④ When the master node generates an RDB file, it caches all the new write commands to the memory. After the MASTER node saves the RDB file, the master node copies the new write commands to the slave node.
⑤ Check the client-output-buffer-limit slave parameter, for example, [client-output-buffer-limit slave 256MB 64MB 60], indicating that memory cache consumption exceeds 64MB during the replication period. Or if the value exceeds 256MB at a time, the replication fails.
⑥ After receiving the RDB file, the slave node clears its own data and reloads the RDB file into its own memory. In this process, the slave node provides services based on the old data.
⑦ If AOF is enabled on the slave node, BRREWRITEAOF will be executed immediately
RDB generation, RDB copying over the network, slave old data cleansing, slave aof rewrite, takes time
If the amount of data to be replicated ranges from 4 GB to 6 GB, the full replication takes 1 and a half minutes to 2 minutes
4. Incremental replication process and mechanism
(1) If the network connection between the master and slave breaks during the full replication, the slave reconnects to the master to trigger the supplement replication.
② The master directly obtains the missing data from its own backlog and sends it to the slave node.
③ The master gets the data from the backlog based on the offset in the psync sent by the slave.
5. The heartbeat
Both the master and slave send heartbeat information to each other. By default, the master node sends a message every 10 seconds, and the slave node sends a message every 1 second.
6. Asynchronous replication
Each time the master receives a write command, it now writes data internally and asynchronously sends it to the slave node
5. How to achieve 99.99% high availability under the master-slave architecture of Redis?
1. What is 99.99% high availability?
High availability (HA), an IT term referring to the ability of a system to perform its functions without interruption. Is one of the criteria for system design. High availability systems can run longer than the components that make up the system. High availability is usually achieved by improving the fault tolerance of the system. Defining what makes a system highly available requires a case-by-case analysis. It is measured in terms of the amount of time a system has been damaged, rendered unusable, and returned from non-operational to operational conditions, compared to the total operating time of the system. The calculation formula is:
A (availability), MTBF(mean time to failure), MDT(mean time to repair) online systems and mission-critical systems typically require availability to meet 5 9 criteria (99.999%).
availability
Annual failure time
99.9999%
32 seconds
99.999%
5 minutes and 15 seconds
99.99%
52 minutes and 34 seconds
99.9%
Eight hours and 46 minutes
99%
Three days, 15 hours and 36 minutes
2. Redis is unavailable
Redis cannot contain single instance unavailability, master/slave schema unavailability. (1) The master node of the master and slave architecture is suspended. If the master node is suspended, the cache data cannot be written, and the data in the slave cannot expire, which leads to unavailability.
If it is a single instance, the redis process may die for other reasons. Or the machine on which Redis is deployed is broken.
Unavailable consequences: first of all, the cache is not available, then the request will go database directly, if the influx of a large number of requests over the carrying capacity of database, the database was hanging off, if not timely handle the cache problem, so there are too many requests, database restart may hang up again soon, as a direct result of the whole system is unavailable.
3. How to achieve high availability
① Make sure every Redis has a backup. ② Ensure that after the current REDIS failure, you can quickly switch to the backup REDis. To solve this problem, introduce the following sentry mechanism.
6. Redis Sentinel architecture related basic knowledge explanation
1. What is a sentry?
Sentinal is a very important component of redis cluster architecture. It mainly has the following functions: ① Cluster monitoring, responsible for monitoring whether the Redis master and slave process is working properly. (2) Message notification. If a Redis instance fails, the sentry is responsible for sending a message as an alarm notification to the administrator. (3) Failover. If the master fails, it will be automatically transferred to the slave. 4) Configure the center, if the failure occurs, notify the client client to connect to the new master.
2. Core knowledge of sentinels
(1) Sentinels themselves are distributed and need to be run as a cluster. All sentinels work together. ② During failover, a master is down and most of the sentries need to agree. ③ Even if some sentinels fail, the sentinels cluster can still work normally. ④ Sentinels need at least three instances to ensure their robustness. ⑤ Sentinel + Redis master-slave structure, is unable to guarantee zero data loss, will only ensure the high availability of redis cluster. ⑥ Corresponding to sentry + Redis master-slave architecture, repeated tests and drills should be done before re-use.
3. Why cannot the Sentinel cluster work properly with two nodes?
More than two nodes must be deployed in the Sentinel cluster. If the cluster has only two sentinel instances deployed, quorum=1(the number of sentinels agreed to perform failover).
4. Classic 3-node sentinel cluster
Configuration: quorum = 2，majority=2
If M1’s machine goes down, then there are two sentinels left, S2 and S3 can agree that the master is down, and then elect one to failover
Majority of the 3 sentinels is 2, so the remaining 2 sentinels are running to allow failover
7. Data loss problem of Redis Sentry active/standby switchover: asynchronous replication, cluster brain split
1. Two data loss scenarios
(1) Data loss caused by asynchronous replication Data is copied asynchronously from the master to the slave. In this case, the master may break down before some data is copied to the slave, and the data is lost.
What is split-brain? Split-brain is when a master machine suddenly disconnects from the normal network and cannot be connected to other slave machines, but in fact the master is still running. At this point, the sentry might assume that the master is down and initiate an election, switching the other slaves to master. At this point, there will be two masters in the cluster, which is called split brain.
At this time, a slave is switched to the master, but data written to the old master may be lost before the slave is switched to the new master. Therefore, when the old master is recovered, it is attached to the new master as a slave, and its own data is cleared and data is copied from the new master
2. Resolve data loss caused by asynchronous replication split
To solve this problem, two parameters need to be configured: min-rabes-to-write 1 and min-rabes-max-lag: indicates that at least one slave is performing data replication and synchronization with a delay of no more than 10 seconds. If the delay for all slave data synchronization and replication exceeds 10 seconds, the master will no longer accept any requests. Min-rabes-max-lag ensures that if the slave replicated data and ack delay is too long, it will consider that too much data was lost after the master went down and reject the write request. In this way, the data loss caused by the failure of the master to synchronize some data to the slave can be reduced to a manageable extent.
(2) Reduce split data loss if a master has split data and loses contact with other slaves, the above two configurations ensure that if the master cannot continue to send data to a specified number of slaves and the slave does not send itself an ACK message for more than 10 seconds, the client will reject the write request directly. In this way, the old master will not accept new data from the client, thus avoiding data loss.
This configuration ensures that if you lose a connection to any slave and find no ack from any slave after 10 seconds, you will reject any new write requests. So in the split-brain scenario, at most 10 seconds of data is lost
8. In-depth analysis of several core underlying principles of Redis Sentinel (including slave election algorithm)
1. Sdown and ODOWN states
Sdown is a subjective outage, just a sentry who thinks a master is down is subjective outage
Odown is an objective outage, and if the sentinels of the quorum quantity feel that a master is down, it is an objective outage
Sdown does something very simple: if a sentry pings a master for more than the number of milliseconds specified for IS-IS master-down-after-milliseconds, the master is down
The condition of sDOWN to ODOWN conversion is very simple. If a sentry receives a quorum quantity within a specified period of time and other sentries consider the master to be SDown, the master is considered to be ODown.
2. Field discovery mechanism of the Sentinel cluster
Each sentinel sends a message to the __sentinel__: Hello channel, where other sentinels can consume the message and sense the presence of other sentinels.
② Every two seconds, each sentinel sends a message to the __sentinel__: Hello channel corresponding to the master+slave he/she monitors. The message contains his/her host, IP, run ID, and the monitoring configuration for the master.
(3) Each sentinel will also listen to the __sentinel__: Hello Channel, r, corresponding to each master+slave he monitors, and then sense the presence of other sentinels who are also listening to the master+slave.
④ Each sentry synchronizes the monitoring configuration of the master based on the monitoring configuration of other sentry switches.
3. Self-correction of the slave configuration
The sentry is responsible for automatically correcting some configurations of the slave. For example, if the slave is to become a potential master candidate, the sentry ensures that the slave is copying existing master data. If the slaves are connected to the wrong master, such as after a failover, the sentinels make sure they are connected to the correct master.
4. Election algorithms
If a master is considered to be oDown and the majority of sentinels allow a master/slave switchover, then a master/slave switchover is performed by a majority of sentinels. The election takes into account the following conditions: (1) The disconnection duration between the slave and the master, (2) the priority of the slave, (3) the offset of the slave data, (4) the RUN ID of the slave
First, if a slave is unconnected to the master by more than 10 times the down-after-millisecondes and the master’s age then it is not considered fit to be elected master. (down-after-milliseconds * 10 + milliseconds_since_master_is_in_SDOWN_state).
The remaining slaves are sorted as follows: (1) First, they are sorted according to their priorities. The lower the slave priority is, the higher the priority is. ② If the priorities are the same, look at replica offset. The more data that slave copies, the farther the offset is, the higher the priority is. (3) Select the slave with the smallest RUN ID.
5. Quorum and majority
Each time a sentinel switches over, the quorum sentinels must first consider the switch oDown and then elect a sentinel authorized by the majority to perform the switch.
If quorum is < majority, such as five sentinels, majority is three (more than half), and quorum is set to two, then three sentinels are required to perform the switch.
If quorum >= majority, the switch can only take place if all the sentinels of the quorum are authorized. For example, if quorum is 5, all the sentinels must agree on authorization.
6.configuration epoch
Sentry monitors a set of Redis master+ Slave and has the corresponding monitoring configuration.
The sentry performing the switch gets a Configuration epoch from the new master (Salve -> Master) to which it is switching. This is a version number that must be unique for each switch.
If the first elected sentry fails to switch, the other sentries will wait for fail-timeout and then continue to switch. At this time, a new Configuration epoch will be obtained as the new version.
Configuraiton propagation
After the switch is complete, the sentry updates the master configuration locally and synchronizes it to the other sentries via pub/ SUB messaging.
The previous version number is important here because messages are published and listened to through a channel, so when a sentinel makes a new switch, the new master configuration follows the new version number.
The other sentinels update their master configuration based on the size of the version number.
