preface

Caching should now be the first step in performance optimization for high concurrency scenarios, relieving database stress while improving access efficiency, and Redis should be the first choice for most applications. But Redis performance good again as soon as possible, in today’s high concurrency scenario, a server is responsible for reading and writing, the performance of the machine and will avoid the bottleneck of memory, to that there must be a friend of the cluster, right, thinking that’s right, just before the cluster, the optimization tactics of the mode of master-slave replication can solve many problems, if the master-slave mode also resist high concurrency, It’s not too late to cluster again; Let’s start with master slave replication of Redis.

For a better demonstration, a cloud server, Linux environment; Convenient at the same time, but also more in line with the actual application scenarios;

The body of the

Master/slave replication: There are multiple Redis servers, one of which is the master server and the other is the slave server. The master/slave relationship can be realized by command or configuration. Replication refers to synchronizing data from the primary server to the secondary server. Data can be unidirectional synchronized from the primary server to the secondary server only. Its functions are as follows:

  • Read/write separation: The primary server replicates the write and each secondary server reads the data. According to the 80-80 principle, 80% of operations are read, and only 20% are written, so to a certain extent, it also solves the single machine bottleneck problem;
  • Data persistence is more secure. When multiple servers are engaged in persistent operations, the failure of any one server does not affect data recovery, avoiding single point of failure.
  • Others: Master-slave replication is a prerequisite for sentry and Redis clustering, as discussed later.

All right, old rule, once you know how it works, let’s do it first.

Implement master/slave replication

When each Redis server is started, it is the Master server by default, which can be viewed using the command info replication, as shown in the following figure:

Start to build the environment of one master and two slave, as follows:

1. Modify the configuration file

Since it is simulated on the same machine, three copies of the Redis configuration file are made. The main modifications are as follows:

  • Conf, redis6388.conf, and redis6399.conf respectively.
  • Port: port. The three configuration files are changed to 6379, 6388, and 6399 respectively to avoid port conflicts on the same machine.
  • Pidfile: Change the file name to redis. Pid, redis6388.pid, and redis6399.pid to avoid file conflicts.
  • Dbfilename: specifies the persistant filename to avoid persistant file conflicts. Change the values to dump. RDB, dump6388. RDB, and dump6399.rdb respectively.

2. Start the Redis server

Then start redis-server by specifying configuration files respectively. Conf,./redis-server redis6388.conf, and./redis-server redis6399.conf in the bin directory of redis. Can through the command ps – ef | grep redis view start redis effect, is as follows:

3.1 Implementing the Master/Slave Relationship Using Commands

By default, the three started servers are all primary servers. You can run a simple command to create a master/slave relationship. Server 6379 is the primary server and server 6388 and server 6399 are secondary servers.

  1. Connect two servers 6388 and 6399 respectively and run slaveof masterip masterport.

  2. There is no problem with the display from the node. Take a look at the status information of the master node and connect it to 6379, as follows:

  3. Now that the master-slave relationship is maintained, let’s look at data replication. Usually the master node is responsible for writing, synchronizing data to the slave node, and the slave node is responsible for reading. Now that there is no data on the three servers, write some data to the primary node to see if it can be synchronized to the secondary node:

Note: If the master/slave relationship is implemented through commands, the master/slave relationship will be lost when the slave server restarts and you have to run commands manually. Therefore, configuration files are recommended.

3.2 Implementing primary/secondary Configuration through configuration files

The REPLICATION module can be configured as follows: The REPLICATION module can be configured as follows: the REPLICATION module can be configured as follows:

  • replicaof <masterip> <masterport>: Specify the IP address and port number of the primary node, for example:Replicaof 127.0.0.1 6379;
  • masterauth <master-password>: Specifies the password of the primary node. If the password is configured for the primary node, the password can be directly configured.

After modifying these two items, restart the server, which is as simple as running the command line. However, the primary/secondary relationship established through the configuration file will not be lost when the server is shut down. Here the configuration file demo is not screenshots, but the implementation method is different, other are the same as the command line;

A small summary

  • The slave server does not match the master server. That is, configure only the slave server and do not configure the master server.

  • A master node has multiple slave nodes, and a slave node can only have one master node. A slave node can also act as the master of another slave node, as follows:

    Although 6388 is the master node of 6399, it still plays the role of the slave node of 6379, but it is connected to other slave nodes.

  • If the primary server is disconnected and the primary server is reconnected, the primary/secondary relationship remains. After the secondary server is disconnected, the reconnection relationship disappears. You need to manually specify the primary node again.

  • If you want to disconnect and reconnect the master/slave relationship, you are advised to disconnect and reconnect the master/slave relationship through a configuration file.

Is it very simple to implement the master-slave replication setup, whether it is commands or configuration files, it is very easy to achieve; However, you may also have a question: how is the data synchronized between the master and slave nodes? What are the other parameters for master-slave replication?

Principle of master-slave replication

Through the above practice, the data written in the master server, without being aware of the synchronization to the slave server, what happened in the middle? So a quick look at that;

Before Redis2.8, all synchronization modes were synchronized in full mode. After that, to improve efficiency, data replication modes were divided into two types: full replication and partial replication:

  • Full replication: Data on the primary server is synchronized to the secondary server. Full synchronization is usually performed when the data is initialized from the server.
  • Partial replication: Unsynchronized incremental data is synchronized to the secondary server. When data cannot be synchronized due to network interruption, data can be partially synchronized during the interruption after the network is restored.

In order to facilitate the log analysis, two Redis servers are used to set up the primary and secondary servers. Remove the relationship established before and restart it by modifying the configuration file.

In this case, 6388 serves as the Master node and 6399 serves as the Slave node. When connecting to 6399, run the slaveof 127.0.0.1 6388 command to configure the Master/Slave relationship, the following logs are generated on the Master/Slave nodes:

The figure above shows the process of full replication at the beginning of the master-slave relationship:

Brief description:

  1. The slave node establishes a connection with the master node and sends a synchronization request psync.
  2. The master node sends information to the slave node, replID and offset, which are then the key data to determine whether it is partially replicated.
  3. The fork process of the main node generates RDB files for all data.
  4. Write commands received during the master node are stored in the replication buffer;
  5. When the master node RDB file is completed, it is sent to the slave node.
  6. After receiving files from nodes, delete old data first.
  7. After clearing data from the node, load the received data into memory;
  8. The master node sends data in the replication buffer to the slave node;
  9. Commands are received and executed from the node, and the latest data is finally synchronized.

The master-slave relationship communicates through the network. Network interruption or jitter may occur, so that data in a short time cannot be synchronized to the slave node in a timely manner. Ideally, when the connection is restored, only the data during the interruption is synchronized to improve the synchronization efficiency.

Brief description:

  1. When the network between the master node and the slave node is interrupted, the slave node tries to reconnect to the master node.
  2. During this time, write commands received by the master node are logged into the replication buffer;
  3. When the network is restored and the secondary node is connected to the master node, the synchronization request psync will be issued with the replID and offset sent by the master node.
  4. After receiving the request from the slave node, the master node verifies whether the replID received matches the replID of the master node. If the replID does not match, full replication is performed. In addition, it verifies whether the offset of offset data is within the legal range. If the amount of data during the interruption is too large and the data in the replication buffer is exceeded, and the offset of the primary and secondary nodes is inconsistent, full replication is also performed.
  5. When the replID and offset passed from the node are validated, a partial copy is made and the latest offset is recorded.

Here, I will not simulate part of the replication process and leave it to my partner. The demonstration process is as follows: For the two machines that have set up a master-slave relationship, manually simulate the network disconnection. During the disconnection, data is written to the primary server. After a while, the network is restored and logs of the primary and secondary servers are checked.

Parameters related to primary/secondary replication

In the configuration file shown above, only some parameters are configured. Other parameters can be easily configured. The following describes the meanings of related parameters based on the content of master/slave replication:

  • Slaveof: Set the local computer as the slave computer, specify the IP address and port number of the host.
  • Masterauth: If the host needs a password, use this setting;
  • Slave-serve-stale-data: Whether to continue to obtain data if the slave server is disconnected from the host or data is being synchronized. If the value is set to Yes, data can be read normally. If the value is set to No, an error message is displayed.
  • Slave-read-only: sets the slave host to read-only mode. The default value is yes, and read-only mode is strongly recommended.
  • Repl-diskless-sync: Indicates whether to use the diskless mode for data transfer from master to slave, that is, Socket mode. This mode is not adopted by default. If the disk performance is poor and the network environment is good, you can use this mode.
  • Repl-diskless-sync-delay: when data is transferred in diskless mode, the server waits for a specified time before transferring data. The server waits for the data to enter the transfer queue, improving the data transfer efficiency. The default value is 5 seconds.
  • Rep-ping-slave-period: sets the interval for sending ping messages from the slave machine to the host, which is regarded as heartbeat detection.
  • Repl-timeout: specifies the timeout period for data transfer between the master and slave machines. If the timeout period is exceeded, the slave machine will be reconnected. If the primary and secondary replication data is large, you can change the size to 60 seconds by default.
  • Repl-disable-tcp-nodelay: indicates whether TCP_NODELAY is used for primary and secondary replication data. The default value is no, indicating that data is not enabled and the host synchronizes data immediately to ensure data consistency failure. If the value is set to yes, smaller TCP packets are merged and sent together, which has a high latency but improves bandwidth performance. However, you are advised not to enable the value.
  • Repl-backlog -size: sets the size of the replication buffer. This buffer is used for incremental data to be synchronized after the slave machine is disconnected and reconnected. Full replication is not required within a certain period of time to improve synchronization efficiency. The default value is 1mb and can be changed as required.
  • Repl – backlogs – TTL: Sets the backlog buffer to be released after the slave machine is disconnected. The default value is 3600s.
  • Slave-priority: used for sentinel mode selection. When a host is down, a slave with a higher priority is selected to replace the down host for rapid recovery. The default value is 100.
  • Min-rabes-to-write 3: when the number of slave servers is less than three, the master server can only read and cannot write. It is generally used with min-rabes-max-lag. It is not used by default and configured as required.
  • Min-rabes-max-lag 10: When the delay of the slave server exceeds 10 seconds, the master server can only read and cannot write. It is used with min-rabes-to-write and is not used by default and configured as required.

What are the problems with master/slave replication

Master-slave replication relieves single-node performance and storage bottlenecks, so what’s the problem?

  • The master-slave replication architecture will have latency, as soon as possible, and also, especially when the data volume and concurrency are large; The current master-slave architecture does not have a good way to handle latency, nor does MySql and SqlServer;
  • Expired Key data cannot invalidate slave server data in early versions, which can be resolved after upgrade to Redis3.2, because expiration judgment is added.
  • If the primary node is down, you can manually reconfigure the primary/secondary relationship on the secondary serverslaveof no oneCommand rewrite to return to the master server role, and then reconfigure the master/slave relationship;
  • If there are too many secondary nodes, the I/O of the primary node increases dramatically when the initial full data synchronization or multiple secondary nodes are disconnected and reconnected.

conclusion

The master-slave replication drill seems simple, but it still needs to gain experience in practice and match relevant configuration parameters to make it more suitable for application scenarios. The configuration is not fixed, but whether the application scenario is suitable. With the main node down can not automatic election problem, next time on this basis to talk about sentinel mode, let automatic election is not a problem;

A handsome guy who was made ugly by the program, pay attention to “Code variety circle “, learn with me ~~~