Redis series of directories

Redis series – distributed lock

Redis series – Cache penetration, cache breakdown, cache avalanche

Why Is Redis so fast?

Redis series — Data Persistence (RDB and AOF)

Redis series – consistent hash algorithm

Redis series – High Availability (Master slave, Sentinel, Cluster)

Redis series – Things and Optimism lock

Redis series — Geospatial: Do you have Lao Wang next door?

Bitmaps: Did you check in today?

What is a Bloom filter?!

High Availability, also known as High Availability (HA), is one of the factors that must be considered in the architecture design of distributed systems. It usually refers to the design to reduce the time when the system cannot provide services.

In real production, if Redis only deploys one node, the rectification service will not be available when the machine fails. This is what we call a single point of failure.

If more than one Redis is deployed, the entire system can still provide services when one or several fail, thus improving service availability.

Today we will talk about the three modes of Redis high availability: master-slave mode, Sentinel mode, and cluster mode.

I. Master-slave mode

In general, high availability of a system is achieved by deploying multiple machines. Redis also needs to deploy multiple machines to avoid a single point of failure.

Because multiple machines are deployed, data synchronization between different machines is involved.

For this purpose, Redis provides redis with replication. When data changes in one Redis database, the changes are automatically synchronized to other Redis machines.

In redis multi-machine deployment, these machine nodes are divided into two types: master node and slave node. Generally, the primary node can perform read and write operations, while the secondary node can only perform read operations. At the same time, because the primary node can write data, the data will change. When the data of the primary node changes, the changed data will be synchronized to the secondary node, so that the data of the secondary node can be consistent with that of the primary node. A master node can have multiple slave nodes, but a slave node can have only one master node, which is called a master multi-slave structure.

1.1. Machine planning

The name of the machine IP port
master 192.168.1.10 6379
slave1 192.168.1.11 6379
slave2 192.168.1.12 6379
slave3 192.168.1.13 6379

1.2. The configuration

Active Node Configuration

Configure the active node according to the normal configuration.

Secondary Node Configuration

Use the default configuration to start the machine, which is the master node. If you want your machine to become a slave node, you need to configure the parameters for master/slave replication on the CONF server.

  • In the redis.conf file of the secondary node, specify the information about the primary node (the login password of the primary node and parameters related to the primary/secondary replication can be configured if necessary). The configuration of the three slave nodes is the same.
Configure the IP address and port of the primary nodeSlaveof 192.168.1.10 6379Starting with redis2.6, slave nodes are read-only by default
slave-read-only yes
If the primary node has a login password, it is 123456
masterauth 123456
Copy the code
  • You u can also use the redis-server command without configuring the above file. When starting the slave node, use the –slaveof parameter to specify who the master node is.
6379. / redis - server -- -- slaveof 192.168.1.10Copy the code
  • You can also start the Redis machine without the above file, and then passredis-cliCommand line executionSlaveof 192.168.1.10 6379Specify who the primary node is.

When the system is running, if the master hangs, you can manually run the slaveof no one command on a slave library (such as Slave1) to change slave1 into the new master. Run slaveof 192.168.1.11 6379 on slave2 and Slave3 to point the primary node of both machines to the new master. At the same time, the lost master is started as a new slave pointing to the new master.

Run the slaveof no one command to disable the replication function of the secondary server. At the same time, the original synchronized data sets will not be discarded.

1.3. Start the machine

Start the master node first, and then start the slave nodes one by one.

1.4. Master/slave replication mechanism

  • Connect to the primary database from the secondary database and send SYNC command.
  • After receiving the SYNC command, the primary database can execute the BGSAVE command to generate an RDB file and use a buffer to record all write commands executed thereafter.
  • After the primary database BGSAVE is executed, snapshot files are sent to all secondary databases, and write commands executed continue to be recorded during sending.
  • After receiving the snapshot file from the database, discard all old data and load the received snapshot.
  • After the primary database snapshot is sent, write commands in the buffer are sent to the secondary database.
  • The snapshot is loaded from the database, command requests are received, and write commands are executed from the autonomous database buffer. (Complete from database initialization)
  • The master database sends the same write command to the slave database for each write command executed, receives and executes the received write command from the slave database (after the initialization of the database is complete)
  • After 2.8, commands during disconnection will be passed to the slave database for incremental replication after disconnection and reconnection.
  • When the master and slave are first connected, full synchronization is performed. After full synchronization is complete, incremental synchronization is performed. Of course, slave can initiate full synchronization at any time if needed. Redis’s policy is to try incremental synchronization first anyway, and if this fails, to require a full synchronization from the slave machine.

1.5. Advantages and disadvantages of master-slave mode

advantages

  • Supports master/slave replication. The master machine automatically synchronizes data to the slave machine, enabling read/write separation.
  • In order to share the load of the Master’s read operation, the Slave server can provide read-only service for the client, but the Master must still complete the write service.
  • Slave can also accept the connection and synchronization requests of other Slaves so as to effectively divide the synchronization pressure of Master;
  • Master provides services to Slaves in a non-blocking manner. Therefore, during master-slave synchronization, clients can still submit query or modify requests.
  • The Slave also blocks data synchronization. During synchronization, if a client submits a query request, Redis returns the data before synchronization.

disadvantages

  • Redis does not have automatic fault tolerance and recovery functions. The downtime of the slave host will lead to the failure of some front-end read and write requests, which can be recovered only after the machine restarts or manually switch the front-end IP address.
  • When the host is down, some data cannot be synchronized to the slave host in a timely manner. After the IP address is switched, data inconsistency may occur, which reduces system availability.
  • If multiple slaves are disconnected, do not restart them at the same time. As soon as the Slave starts, a sync request is sent to fully synchronize with the host. When multiple slaves restart, the Master I/O surge may result in downtime.
  • Redis is difficult to support online capacity expansion, which becomes complicated when the cluster capacity reaches its maximum.
  • Data in the master and slave nodes of Redis is the same, reducing memory availability

2. Sentinel mode

In the master/slave mode, when the master server breaks down, you need to manually switch a slave server to the master server. This requires manual intervention and makes services unavailable for a period of time. This approach is not recommended, in actual production, we give priority to the sentinel mode. In this mode, if the master is down, the sentry automatically elects the master and points other slaves to the new master.

In master-slave mode, Redis also provides the sentinel command, redis-sentinel, which is an independent process and runs independently as a process. The sentry process monitors multiple instances of Redis running by sending commands to all Redis machines and waiting for the Redis server to respond.

There can be more than one sentry, usually an odd number of sentries is used to facilitate decision-making elections. Sentinels can be deployed with redis machines or on other machines. Multiple sentries form a sentry cluster. Sentries directly communicate with each other to check whether the sentries are running normally and elect a new master when the master is down

What sentinel mode does:

  • By sending commands, Redis servers return to monitor their running status, including master and slave servers;
  • When the sentry detects that the master is down, it automatically switches the slave to the master and passes throughPublish and subscribe modelThrough the other slave servers, modify the configuration file to make them switch hosts;
  • However, a single sentry process monitoring the Redis server can also cause problems, so we can use multiple sentries to monitor. Each sentry is also monitored, creating a multi-sentry mode.

Sentinel is much like zooKeeper in Kafka clusters.

2.1. Machine planning

The name of the machine IP port
master 192.168.1.10 6379
slave 1 192.168.1.11 6379
slave 2 192.168.1.12 6379
slave 3 192.168.1.13 6379
sentinel 1 192.168.1.14 26379
sentinel 2 192.168.1.15 26379
sentinel 3 192.168.1.16 26379

The Sentinel process and Redis are deployed on different machines to avoid the sentinel process being unavailable due to redis downtime.

2.2. The configuration

The configuration of redis. Conf is the same as that of the master/slave mode. Here is the sentinel configuration.

The sentinel process on each machine requires a sentinel configuration file, sentinel.conf. The sentinel configuration is the same for all three machines.

# Disable protected mode
protected-mode no
Sentinel monitor = sentinel monitor, mymaster = sentinel monitor, mymaster = mymaster
#192.168.1.10 indicates the monitored primary server, 6379 indicates the port, and 2 indicates that failover is performed only when two or more sentinels consider the primary server to be unavailable.Sentinel Monitor MyMaster 192.168.1.10 6379 2Mymaster is the service name and 123456 is the Redis server password
sentinel auth-pass mymaster 123456
Copy the code

2.3. Start the machine

Start the master node first, and then start the slave nodes one by one.

After the redis cluster is started, run the redis-sentinel /path/to/sentinel.conf command to start the three sentinels on the host where the sentinel cluster resides.

2.4. Sentry mode work

  • Each Sentinel process sends a PING command to the Master, Slave, and other Sentinel processes in the cluster at a frequency of once per second.
  • The Sentinel process marks an instance as SDOWN if it has taken longer than the value specified in the down-after-milliseconds option for the last effective PING response.
  • If a Master server is marked as a subjective offline (SDOWN), all Sentinel processes that are monitoring the Master server confirm once per second that the Master server is indeed in a subjective offline state
  • The Master server is marked as ODOWN when a sufficient number of Sentinel processes (greater than or equal to the value specified in the configuration file) confirm that the Master server is in a subjective SDOWN state within the specified time period.
  • In general, each Sentinel process sends INFO commands to all Master and Slave servers in the cluster every 10 seconds.
  • When the Master server is marked as ODOWN by the Sentinel process, the Sentinel process sends INFO commands to all slaves of the offline Master server once every 10 seconds instead of once every second.
  • The objective offline status of the Master Master server is removed if a sufficient number of Sentinel processes do not agree to take the Master offline. If the Master server reping the Sentinel process and returns a valid response, the subjective offline status of the Master server will be removed.

If the master is down and Sentinel 1 detects the failure first, the system does not perform a failover immediately to select a new master. Sentinel 1 only determines that the master is unavailable, and this phenomenon becomes subjective offline. When sentinels at the back also detect that the primary server is unavailable and the number reaches a certain value, a vote is conducted between sentinels. The result of the vote is initiated by Sentinel 1 and failover is performed. After the switch is successful, each sentinel will switch the host from the secondary server monitored by himself through the publish and subscribe mode, which is called objective offline process. This makes everything transparent to the client.

2.5. Advantages and disadvantages of master-slave mode

advantages

  • The Sentinel mode is based on the master-slave mode, which has all the advantages of the master-slave mode.
  • The master and slave can be switched automatically, making the system more robust and more available.

disadvantages

  • With the disadvantages of master-slave mode, the data on each machine is the same and the availability of memory is low.
  • Redis is difficult to support online capacity expansion, which can be complicated when the cluster capacity reaches its maximum.

Third, cluster mode

A first mistake: Redis does not use a consistent hash algorithm for clustering itself, but instead uses slots. This is a mistake many people make. I’m going to leave a little bit of a hole here, and I’m going to publish a redis series on Consistent Hash Algorithms.

The Sentinel mode of Redis can basically achieve high availability, read and write separation, but in this mode, every Redis server stores the same data, which is a waste of memory, so in Redis3.0, the Cluster mode is added to realize distributed storage of Redis, and the data is sharded. That is, each Redis node stores different content;

The six Redis here are not independent of each other. Each node communicates with other nodes through the cluster bus. A special port number is used for communication, that is, the external service port number plus 10000. For example, if the port number of a node is 6379, the port number used to communicate with other nodes is 16379. Communication between Nodes uses a special binary protocol.

To the client, the cluster is treated as a whole, and the client can connect to any node to perform operations, just like a single Redis instance. If the client’s operation key is not assigned to the node, Redis will return a steering instruction pointing to the correct node. It’s kind of like a 302 redirect for a browser page.

According to official recommendations, at least three master nodes must be deployed in a cluster. The recommended deployment mode is three master nodes and three slave nodes. During testing, you can also deploy the six instances on a single machine, separated by ports.

3.1. Machine planning

The name of the machine IP port
master 1 192.168.1.11 6379
master 2 192.168.1.12 6379
master 3 192.168.1.13 6379
slave 1 192.168.1.21 6379
slave 2 192.168.1.22 6379
slave 3 192.168.1.23 6379

3.2. The configuration

Modify the redis.conf configuration file:

Enable redis cluster mode
cluster-enabled yes
Redis-cluster. conf this file is automatically generated after the cluster is started and does not need to be manually configured.
cluster-config-file redis-cluster.conf
Copy the code

3.3. Machine start

After the six Redis services are successfully started, the six machines are still independent although the cluster is configured to start. Add the six machines to a cluster using cluster management commands.

The redis-tri.rb tool can be used to quickly deploy clusters.

Simply run redis-trib.rb create –replicas 1 192.168.1.11:6379 192.168.1.21:6379 192.168.1.12:6379 192.168.1.22:6379 192.168.1.22:6379 192.168.1.13:6379 192.168.1.23:6379 creates a cluster.

After the command is executed, a response log is displayed. You can view the relationship between machines in the cluster based on related logs. The logs are as follows:

>>> Performing hashslots allocation on 6 nodes... Master[0] -> Slots 0 - 5460 Master[1] -> Slots 5461 - 10922 Master[2] -> Slots 10923 - 16383 Adding replica 192.168.1.21:6379 to 192.168.1.11:6379 Adding Replica 192.168.1.22:6379 to 192.168.1.12:6379 Adding replica 192.168.1.23:6379 to 192.168.1.13:6379 M: 80 c80a3f3e33872c047a8328ad579b9bea001ad8 192.168.1.11:6379 slots: [0-5460] (5461 slots) master S: B4d3eb411a7355d4767c6c23b4df69fa183ef8bc 192.168.1.21:6379 replicates ee9a8d7f72b1d45a9093838efd0e501f1 6788453 M: 4 d74ec66e898bf09006dac86d4928f9fad81f373 192.168.1.12:6379 slots: [5461-10922] (5462 slots) master S: B6331cbc986794237c83ed2d5c30777c1551546e 192.168.1.22:6379 replicates c80a3f3e33872c047a8328ad579b9bea001ad8 80 M: 6788453 ee9a8d7f72b1d45a9093838efd0e501f1 192.168.1.13:6379 slots: [10923-16383] (5461 slots) master S: 277 daeb8660d5273b7c3e05c263f861ed5f17b92 192.168.1.23:6379 replicates 4 d74ec66e898bf09006dac86d4928f9fad81f373 Can Iset the above configuration? (type 'yes' to accept): yes                  Enter yes to accept the above configuration
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Copy the code

The redis-cluster.conf file is automatically generated after the command is executed.

To log in to the cluster: redis-cli -c -h 192.168.1.11 -p 6379 -a 123456 # -c To log in to the cluster.

View CLUSTER information: 192.168.1.11:6379> CLUSTER INFO

List node information: 192.168.1.11:6379> CLUSTER NODES # List node information.

Add data:

192.168.1.11:6379 >setName AAA -> Redirected to Slot [13680] located at 192.168.1.13:6379Data was finally written to 192.168.1.13:6379
OK
Copy the code

Get data:

192.168.1.11:6379> get name
-> Redirected to slot [13680] located at 192.168.1.13:6379                Finally, 192.168.1.13:6379 is used to read data
"aaa"
Copy the code

3.4. Operation mechanism

Redis-trib.rb (” slots “); redis-trib.rb (” slots “); redis-trib.rb (” slots “); Another is cluster, which can be understood as a cluster management plug-in, similar to a sentinel.

When our access Key arrives, Redis will calculate a result according to crC16 algorithm, and then calculate the remainder of the result and 16384, so that each Key will correspond to a hash slot with a number between 0 and 16383. Through this value, we can find the node corresponding to the corresponding slot. Then jump directly to the corresponding node for access operation.

After data is written to the master node, the data is synchronized to all slave nodes corresponding to the master node.

To ensure high availability, a Redis-cluster cluster adopts the master-slave mode, where one master node corresponds to one or more slave nodes. When other master nodes ping master 1, if the communication between more than half of the master nodes and master 1 times out, master 1 is considered to be down. The slave node of Master 1, Slave 1, becomes the master node and continues to provide services.

If master 1 and its slave node, Slave 1, both go down, the entire cluster goes into a fail state because the slot mapping of the cluster is incomplete. If more than half of the masters in a cluster fail, the cluster enters the Fail state regardless of whether there are slaves.

Redis-cluster adopts the idea of decentralization, there is no central node, the client is directly connected with the Redis node, there is no need for intermediate agent layer, the client does not need to connect all nodes of the cluster, but can connect any available node in the cluster.

3.5. Cluster capacity expansion

Scaling up a Redis cluster means adding machines to the cluster, scaling down means removing machines from the cluster and reassigning 16383 slots to nodes in the cluster (data migration).

Scaling is also done using the cluster management tool Redis-tri.rb.

For capacity expansion, the redis-tri.rb add-node is used to add the new machine to the cluster. Although the new machine is already in the cluster, it is still not usable without slots. The new node does not work until slots from the old node is assigned to the new node using redis-tri.rb reshard for shard rehash (data migration).

When scaling down, first use the slots on the machine that redis-tri.rb reshard removes, and then use the redis-tri.rb add-del to remove the machine.

3.8. Advantages and disadvantages of the cluster model

advantages

Decentralized, data storage is distributed among multiple nodes according to slot, and data distribution can be dynamically adjusted among nodes.

Scalability: linear expansion to more than 1000 nodes, nodes can be added or deleted dynamically;

High availability: The cluster is still available when some nodes are unavailable. By adding the Slave as the standby data copy, an automatic failover can be implemented. Status information is exchanged between nodes through the Gossip protocol and the role is promoted from Slave to Master through the voting mechanism.

Reduces o&M costs and improves system scalability and availability.

disadvantages

1.Redis Cluster is a Cluster architecture without central nodes, which relies on Goss protocol (rumor spreading) to cooperate with automatic repair of Cluster status

However, GosSIp has message delay and message redundancy. When there are too many nodes in the cluster, the nodes need to PING or PANG communication continuously. Therefore, unnecessary traffic occupies a large number of network resources. Although Reds4.0 has been optimized for this, the problem still exists.

2. Data migration

Redis Cluster can dynamically expand or shrink nodes. This process is still semi-automatic and requires manual intervention. Data migration is required during capacity expansion.

And Redis in order to ensure the consistency of migration, migration all operations are synchronous operation, perform the migration, at the ends of the Redis will enter the length of the blocking state, for the Key, the time can be neglected, but if the Key of memory usage is too big, serious when can contact failover within a cluster, causing unnecessary switch.

Four,

Master/slave mode: If the master node fails, you need to manually specify a new master node.

Sentry mode: After the master node fails, the sentry process actively elects a new master. The availability is high, but each node stores the same data, wasting memory space. The data volume is not large and the cluster size is not large. This parameter is used when automatic fault tolerance and DISASTER recovery are required.

Cluster mode: This mode is used when a large amount of data is required and QPS requirements are high. Redis Cluster was officially launched after Redis 3.0, which was late. At present, there are not many successful cases that can be proved in large-scale production environment, and it needs time to test.

Done, done!

[spread knowledge, share value], thank small partners attention and support, I am [Zhuge small ape], a hesitation in the struggle of the Internet migrant workers!!