Redis master-slave synchronization (II) after 2.8 ways

Next, “Redis master slave synchronization (a) before 2.8 version of the way”, in the last, spoke to the 2.8 version before the way has defects, everyone still remember, today will talk about from 2.8 version of the master slave synchronization mode improvement

Let’s review the shortcomings of the pre-2.8 redis master-slave synchronization

• If the existing data is synchronized after the sync operation is completed, at this point, the master server is in the command propagation stage between the slave server and the master server, at this point, the following command is transmitted, and then the connection between the master server and the slave server is disconnected
• After the connection between the primary and secondary servers is re-established, the synchronization between the primary and secondary servers starts again
• At this point in the sync operation, the master generates an RDB file that contains the key from the first command development in the first synchronization
• While the sync operation can still be done, there are many keys that do not need to be synchronized again
• Since the main server needs to generate RDB files during the sync operation, this operation takes up a lot of resources of the server: CPU, memory, IO
• The transfer of RDB files between the master and slave servers consumes server bandwidth
• Loading RDB files from the server also consumes a large amount of server resources: CPU, memory, AND IO
• Therefore, no unnecessary sync operation can consume server resources and degrade service performance

In summary, the master/slave synchronization mode of the old version was very low in case of disconnection and reconnection, so version 2.8 started to passPSYNCCommand in place ofSYNCCommand to perform a synchronization operation

PSYNC command

• PSYNCCommands have two modes:Full resynchronization modeandPartial resynchronization modeTwo modes

Full resynchronization mode

• Full synchronization mode is used for processingThe first copysituation
• Full synchronization mode execution process andSYNCThe command is consistent. Let’s review the process as follows:

• 1. The secondary server sends the slaveof command to the primary server
• 2. The primary server receives the slaveof command
• 3. Run the BGSAVE command
• 4. Generate an RDB file in the background
• 5. Use a buffer to record all commands executed since the beginning
• 6. The BGSAVE command is executed on the primary server
• 7. The primary server sends the RDB file generated by the BGSAVE command to the secondary server
• 8. Accept the RDB file from the server and load it
• 9. Update your database status to the database status when the master server runs the BGSAVE command
• 10. The master server sends all write commands recorded in the buffer to the slave server
• 11. Run these commands from the server to complete the synchronization

Partial resynchronization mode

• Partial synchronization mode is mainly handledReplication scenario after disconnection and reconnection
• Other andSYNCThe commands are consistent. The master node will synchronize the commands during the disconnection to the slave node only after the connection is reconnected

Let’s take a look at what happens when we break and reconnectPSYNCCommand Execution Process

• And this process depends on three main parts

Replication offset of the primary and secondary servers

• Both the master and slave services maintain a replication offset
• Each time the master propagates n bytes to the slave, it adds n to the offset it maintains
• Each time the slave service receives data propagated from the master server, it adds the offset it maintains to the number of bytes of the data size
• Master slave offset chestnut:

Offset of the primary server: offset=100 Offset of the secondary server A: offset=100 Offset of the secondary server B: offset=100Copy the code

• In this case, the primary server and secondary server A and B are synchronized
• When the primary service synchronizes 100 bytes of data to the secondary service

Offset of the primary server: offset=100 Offset of secondary server A: offset=200 (secondary server B and the primary server are disconnected) Offset of secondary server B: offset=100Copy the code

• In this case, the primary server and secondary server A are synchronized
• The offset value of server B is 100 due to disconnection of service B, so it can be judged that they are not in data synchronization state

Replication backlog of the primary server

• Copy the backlogged cacheIs a queue maintained by the primary server with a default size of 1MB
• The queue isFixed lengthandFirst in first outthe
• When the primary server executes on the secondary serverCommand transmissionNot only will all commands be sent to all slave services
• It also writes all the commandsCopy the backlogged cache, as shown in the figure:

• becauseCopy the backlogged cacheisThe queue, is to write the bytes of each command of the main service to the queue, consisting of two parts,The offsetandCommand byte value
• The queue structure is as follows:

Node1:
offset=1
val=s
Node2:
offset=2
val=e
Node3:
offset=3
val=t
Node4:
offset=4
val=k
Node5:
offset=5
val=1
Node6:
offset=6
val=v
Node7:
offset=7
val=1
Copy the code

• See the queue structure, which is the command of the master serviceset k1 v1Each byte of and byte corresponding toThe offsetDeposit toCopy the backlogged cacheIn the
• When the slave server reconnects to the master server, the slave server passesPSYNCTo his ownThe replication offset is offsetSend it to the master server, which determines whether it is used based on the replication offsetComplete synchronizationorPart of the synchronization, the specific operations are as follows:

Run ID of the redis service

• When each Redis service starts, it has its own run ID
• Each ID will never be repeated. How do I do that
• When the primary replication is performed between the secondary service and the primary server, the secondary service saves the running ID of the primary service
• Run the command after the connection is reconnectedPSYNCThe command is used to compare the ids. If the ids are the same, the active service is checkingCopy offsetDetermine whether full or partial synchronization is required. If the ids of the two main services are different, select this optionComplete synchronization

PSYNCDetailed command execution process

Today, I talked about the way after the 2.8 version of the master/slave synchronization of Redis, welcome everyone to communicate, point out some mistakes in the article, let me deepen my understanding, wish you no bugs, thank you!