“This is the 16th day of my participation in the November Gwen Challenge. See details of the event: The Last Gwen Challenge 2021”.

Dry goods warning comes, this article will first explain to you how to build master-slave replication, and then introduce the principle of master-slave replication in detail, patience, you will harvest full.

Set up master/slave replication

First modify the configuration file of the primary node

Daemonize yes // Enable the daemon thread logfile "6379.log" dbfilename "dump-6379.rdb" protected-mode no To enable this function, configure bind IP or access password port 6379 bind 0.0.0.0Copy the code

Then copy the two configuration files and change the file names

Then make changes to the slave node configuration file

Daemonize yes logfile "6378.log" dbfilename "dump-6378. RDB "protected-mode no port 6378 bind 0.0.0.0 slaveof 192.168.121.133 6379Copy the code

Start three nodes

./redis-server .. /conf/redis-6379-master.conf ./redis-server .. /conf/redis-6378.conf ./redis-server .. /conf/redis-6377.confCopy the code

Accessing the Client


./redis-cli -p 6379

Copy the code

You can see that there are two slave nodes

Now let’s look at the slave node

You can see from the node

Master-slave setup is successful

Principle of master-slave replication

Master/slave replication has changes before and after Redis2.8, which are explained here

Copy the old version

Redis replication is divided into two operations: sync and command propagation

  • The synchronization operation is used to update the slave server database state to the same state as the master server is currently in

  • When the database status of the primary server is changed and the status of the primary and secondary databases is inconsistent, the command propagates the database status of the primary and secondary databases back to the consistent state.

synchronous

When the slave server establishes a master-slave relationship with the master server (the slave server sends the slaveof command), the slave server needs to perform synchronization first. The slave server needs to send the SYNC command to the master server to complete synchronization

  • The secondary server sends the SYNC command to the primary server

  • The primary server receiving the SYNC command executes the BGSAVE command, generates an RDB file in the background, and uses a cache to record all write commands executed from now on.

  • When the master BGSAVE command is complete, the master sends the generated RDB file to the slave server, which reads and stores the data from the RDB file into its own library.

  • The master server sends all the write commands recorded in the buffer to the slave server, which executes the write commands and stores the data.

Command transmission

Assume a primary and a finished synchronous operation from the server, and their state of database is consistent, if the client sends a write command to the primary server as master-slave is inconsistent, the master server to execute commands transmitted from the server, the primary server will perform the write command sent to from the server, so that the master-slave returned to a consistent state.

Old copy defect

There are two types of replication from the slave to the master:

  • Primary replication: The secondary server has not replicated any primary server before, or the primary server to be replicated is different from the primary server to be replicated last time

  • Replication after disconnection: The replication between the primary and secondary servers in the command propagation phase is interrupted for various reasons, but the secondary server reconnects to the primary server through automatic reconnection and continues to replicate the primary server.

For short term after copying, old replication efficiency is very low, when reconnection, will send the SYNC to the master server from the server, and the primary RDB file containing all of the data will be sent to the server, to achieve the final data, the total amount of push is actually can be optimized, can only write no synchronization during the break command synchronizes, But the old copy didn’t implement this mechanism.

The new copy

Since 2.8, Redis uses PSYNC instead of SYNC to perform replication synchronization.

The PSYNC command supports full and partial resynchronization modes.

  • Full resynchronization is used to handle the initial replication. The steps of the SYNC command are basically the same as those of the previous version

  • Partial resynchronization is used to deal with post-disconnection duplication. The master server can send only write commands executed during disconnection to the slave server. The slave server can restore consistency by receiving and executing these write commands.

Partial duplicate copy implementation

The partial repetition system consists of the following three parts:

  • The replication offsets of the master and slave servers

  • Replication backlog buffer for the primary server

  • Running ID of the server

Copy offset

Each replication party maintains a replication offset. Each time the master server propagates N bytes of data to the slave server, it adds N to its own replication offset. Each time the slave server receives N bytes of data propagated from the master server, it adds N to its own replication offset.

Replication backlogs

The replication backlog buffer is a fixed-length first-in, first-out queue maintained by the primary server, with a default size of 1MB.

When the master server propagates commands, it not only sends write commands to all slave servers, but also queues them into the replication backlog.

When the slave server reconnects to the master server, the slave server sends its replication offset to the master server through the PSYNC command. The master server uses this replication offset to decide which synchronization operation to perform on the slave server:

  • If the data after the offset exists in the replication backlog buffer, the master server performs partial resynchronization on the slave server.

  • If the data after the offset does not exist in the replication backlog buffer, the master performs a full resynchronization operation on the slave.

Server Running ID

Each Redis server will have its own run ID.

When the slave makes the first replication to the master, the master passes its run ID to the slave, and the slave saves the run ID.

When the secondary server SMS and reconnects to the primary server, the secondary server sends the previously saved running ID to the currently connected primary server.

If the running ID sent by the secondary server is the same as that of the current primary server, the primary server tries to perform partial resynchronization first.

If the run ids are not equal, the primary server performs a full resynchronization on the secondary server.

The PSYNC command is implemented

  • If the slave server has not replicated any master servers before, starting a new replication from the slave server will send PSYNC to the master? -1 command to proactively request full resynchronization from the primary server.

  • If the secondary server has replicated a primary server, the secondary server will send the PSYNC command to the primary server when starting a new replication: runid is the ID of the primary server that was replicated last time,offset is the current offset of the primary server. The primary server uses these two parameters to determine which synchronization operation to perform.

  • If the host returns a +FULLRESYNC response, it indicates a FULLRESYNC operation, and if the host returns a +CONTINUE response, it indicates that the primary server will perform a partial resync operation with the secondary. If the primary server returns -err, the primary server version is earlier than 2.8 and cannot recognize the PSYNC command. The secondary server sends a SYNC command to perform a full resynchronization.

Copy the overall process

  • Run slaveof from the node

  • Save the slaveof primary node information from the node

  • The scheduled task inside the node detects information about the primary node and connects to the primary node using sockets

  • If the connection is established successfully, you can run the ping command and get a response. Otherwise, the connection is reconnected

  • If the primary node has set permissions, permission verification is required

  • If the verification succeeds, data is synchronized

The heartbeat

During command propagation, the secondary server sends commands to the primary server once per second by default

Replication_offset Specifies the current replication offset of the server.

Sending the REPLCONF ACK command does three things

  • Check the network connection status of the primary and secondary servers

This command is used by the master and slave servers to check whether the network connection is normal, and if the master server does not receive the REPLCONF ACK command from the slave server for more than a second, the master server knows that there is a problem with the connection

  • Assist in implementing min-Salves configuration options

Redis can prevent write commands from being executed when the master server is unsafe with min-rabes-to-write and min-salves-mas-lag options

After setting, if the number of secondary servers is less than three, or the delay value of three secondary servers is greater than or equal to 10 seconds, the primary server rejects the write command.

  • Detection command loss

If the write command propagated from the master to the slave is lost halfway through due to a network failure, then when the slave sends the REPLCONF ACK command to the master, the master sends a different offset from the slave, finds the missing data in the replication backlog buffer, and resends it to the slave.