Redis persistence

RDB(Snapshot Persistence)

By default, Redis saves the in-memory database snapshot in a file named dump.rdb.

You can set Redis to automatically save a snapshot of the dataset when the condition of “at least 1000 keys changed in 60 seconds” is met: Save 60 1000 At the same time, the Save can be configured multiple times, that is, multiple RDB policies can be supported. Note that the save caused by configuring this policy in the configuration is bgSave and does not block the client thread executing the command.

Rdbfilename specifies the dump filename and dir specifies the path to RDB.

You can also run the redis client command to manually generate an RDB snapshot. You can run the save or bgsave command to generate a dump. RDB file. Each time you run the command, all memory snapshots are saved to a new RDB file and the original RDB file is overwritten.

Copy-on-write (COW) for BGSave

Redis uses the copy-on-write technology provided by the operating system to process write commands while generating snapshots. Since the bgSave child process is generated by fork, it can share all of the main thread’s memory data. Once the bgSave child thread runs, it starts reading the main thread’s memory data and writing it to an RDB file. If the main thread changes a piece of data, the data will be copied, creating a copy of the data, and then the bgSave child thread will write the copy to the RDB file. The main thread can still modify the original data directly.

The advantages of RDB

RDB recovery speed, because RDB is stored in memory snapshot, is a binary file, reading will be very fast.

The disadvantage of RDB

Although RDB recovery is fast, RDB persistence is always full, because it saves a snapshot of the current data. If there is 8GB of data in memory, then Redis will write the 8GB of memory directly to the RDB snapshot file in disk, which is obviously very low performance.

RDB functionality is not very durable, and if Redis fails for some reason, the server will lose data that was recently written to and still hasn’t been saved to the snapshot.

AOF(append-only file)

Due to the limitations of RDB, Redis adds a fully durable persistence scheme: AOF persistence, logging every modified instruction in the file appendone.aof (first written to OS cache and fsync to disk at intervals).

Open appendonly

Write a command set Darkness 123

Look at the appendone.aof file again

It did find that the command had been written to the AOF file.

*2
$6
SELECT
$1
0
*3
$3
set
$8
darkness
$3
123
Copy the code

This is resP protocol format. The number after * indicates how many arguments the command has, and the number after the parameter indicates how many characters the parameter has. For example, setdarKNess123 command, starting with \*3, indicates that the command has three arguments, and the number after the first parameter indicates how many characters the parameter has. For example, in set Darkness 123, starting with \*3, the command has three arguments, and the number after the first parameter means how many characters it has. For example, in setdarkness123, starting with \*3, the command has three arguments, and the first 3 means that set is three characters. The second 8 for darkness is eight characters, the third 8 for darkness is eight characters, the third 8 for darkness is eight characters, and the third 3 for 123 is three characters.

Note: If you run a set command with an expiration time, the aOF file does not record the original command executed, but the timestamp of the expiration of the key. For example, set Name Darkness EX 1000 would be in the following AOF file: \* would still be 3 instead of 5, and the command would be pexpireat instead.

*3
$3
set
$4
name
$8
darkness
*3
$9
PEXPIREAT
$4
name
$13
1647523736190
Copy the code

When AOF is enabled, from then on, whenever Redis executes a command that changes the data set, such as set, the command is appended to the end of the AOF file. When Redis is restarted, the program can recreate the dataset by re-executing the commands in the AOF file.

You can configure the policy (frequency) for Redis to fsync data to disk

Appendfsync always: Fsync is performed every time a new modification command is appended to the AOF file. Very slow, but safest. Appendfsync evernsec: fsync once per second, fast enough, and only lost in the event of a failure1Appendfsync no: passes data to the operating system without fsync, faster and less secure. The recommended (and default) measure is fsync once per second, a fsync policy that balances speed and securityCopy the code

AOF rewrite

There may be too many useless instructions in AOF files, so AOF periodically generates AOF files based on the latest in-memory data.

For example, the following commands are executed:

The client then executes the bgrewriteaof command to rewrite AOF

*3
$3
SET
$2
readcount
$1
5
Copy the code

The following two configurations control the frequency of automatic AOF rewriting

Auto-aof -rewrite-min-size 64mb: indicates that aof files are automatically rewritten only when the size is at least 64mb. The recovery speed of aof files is very fast. Auto-aof -rewrite-percentage is not significant100The: aOF file size has increased since the last rewrite100% will trigger rewriting againCopy the code

Note: When aOF is overwritten, Redis will fork out a subprocess to do it, which will not have much impact on redis normal command processing.

The advantages of aof

Aof basically does not lose data, and saves commands every time, so as not to save them in full when the memory is large. In this way, the command execution speed will not be affected because the data set grows too fast for persistence.

The disadvantage of aof

Aof because save command set, so every time to reload the persistent data, equivalent to the entire redis before during the running of all the commands replay again, nature is very inefficient, although aof provides aof rewriting mechanism combined a lot of orders, but the performance of pain points or not only by this means can solve.

Redis4.0 Hybrid Persistence

As mentioned above, when aOF starts to load persistent data replay commands, it takes a lot of time, resulting in very slow redis startup. However, if RDB is used, the risk of data loss is too great. Therefore, another solution is enabled after Redis4.0, which is hybrid persistence. Aof + RDB is commonly used.

Enable mixed persistence with the following configuration (AOF must be enabled first)

aof-use-rdb-preamble yes
Copy the code

Hybrid persistence is dominated by AOF. Because AOF doesn’t lose data, it still uses AOF to record every command, but instead of just aOF, when aOF is rewritten (aofrewrite), Instead of simply converting the memory data into RESP commands and writing them into aOF files, RDB snapshots will be taken for the memory before rewriting, and RDB snapshots will be written into a new AOF file together with incremental AOF commands to modify the memory data. The new file is not initially called appendone.aof, but will be renamed after rewriting the new AOF file to override the original aOF file and complete the replacement of the old and new aOF files.

Therefore, when Redis restarts, the contents of RDB can be loaded first, and then the incremental AOF log can completely replace the previous full AOF file replay, so the restart efficiency is greatly improved.

For example, if set Name darkness triggered aofrewrite and set Darkness name was triggered later, the aOF file would be the same as the one before Set Name Darkness. Set Darkness name (resp); create a new appendone.aof file.

The hybrid persistent AOF file structure is as follows:

Redis data backup policy

  1. This section describes how to write a crontab scheduling script to copy an RDB or AOF backup file to a directory every hour. Only the backup file generated in the last 48 hours is retained.

  2. Keep a copy of the current day’s data backup to a directory every day, can keep the latest month’s backup.

  3. Every time you copy a backup, delete the backup that is too old.

  4. Assign a copy of the backup on the current machine to another machine every night to prevent machine damage.

Redis master-slave architecture

Redis master-slave principle

If a slave is configured for the master, regardless of whether the slave is connected to the master for the first time, it sends a PSYNC command to the master to request data replication.

After receiving the PSYNC command, the master will persist the data in the background and generate the latest RDB snapshot file using bgSave. During the persistence, the master will continue to receive requests from the client, which will be cached in memory. When the persistence is complete, the master sends the RDB file data set to the slave, and the slave persists the received data to generate the RDB, which is then loaded into memory. The master then sends the commands previously cached in memory to the slave.

When the connection between the master and slave is disconnected for some reason, the slave can automatically reconnect to the master. If the master receives multiple concurrent connection requests from the slaves, it will persist only once, not once. This persistent data is then sent to multiple slaves concurrently connected.

Primary/Secondary replication (full replication)

  1. The psync command synchronizes data and establishes a socket connection with the master before sending the command.

  2. Master: The system receives the psync command and runs the BGSave command to generate the latest RDB snapshot data.

  3. Master: After the master starts doing the RDB, it continues to cache new data, which are essentially write commands called repl buffers.

  4. Master: Sends RDB data to the slave.

  5. Slave: Clears old data and loads the RDB of the master node.

  6. Master: repL buffer that sends cached data.

  7. Slave: Executes write commands in the buffer to the memory.

  8. Master: continuously sends commands to the secondary node over a socket connection to ensure data consistency between the primary and secondary nodes.

Master/slave replication (partial replication/Breakpoint continuation)

When the master and salve are disconnected and reconnected, the entire data is copied. However, starting from Redis2.8, Redis uses the PSYNC command that can support partial data replication to synchronize data with the master. After the network connection is disconnected and reconnected, the slave and master can only perform partial data replication (resumable transmission).

The master and all its slaves maintain offset and the master process ID. Therefore, when the network connection is disconnected, The slave requests the master to continue the unfinished replication, starting with the subscript of the recorded data. If the master process ID changes, or if the slave node offset is old and no longer in the master cache queue, a full copy will be performed.

  1. Slave: The slave is disconnected from the master.

  2. Master: The master continues to cache new data sets (essentially write commands), called the Repl Backlog Buffer.

  3. Slave: Restores the network and establishes a socket connection with the master again.

  4. Slave: sends the PSYNC(offset) command to the master.

  5. Master: Identifies the offset sent by the slave. If the offset is in the repl Backlog buffer, the master synchronizes the cached data after the offset to the slave node at once. If not, full synchronization is performed.

  6. Master: continuously sends commands to the secondary node over a socket connection to ensure data consistency between the primary and secondary nodes.

Master/slave replication storm

If a master master has many slave nodes, then when these slave nodes concurrently send PSYNC to the master node, the master node will be under great pressure to synchronize data to each slave node. So if there are too many slave nodes, you can have some slave nodes synchronize data from some slave nodes, as shown in the figure below.

Set up a redis with 1 master and 2 slave

Download redis-5.0.14, download link

Installation:

/****** Configure master ******/
// Create the redis6379 directory
mkdir redis6379
// Decompress the downloaded package
tar -zxvf redis-5.014.
// Go to the decompressed redis root directory
cd redis-5.014.
// Use make to install, PREFIX= the output of the current directory PWD
make PREFIX=/opt/redis/redis6379/redis-5.014. install
/ / configure redis. Conf
protected-mode no
port 6379
daemonize yes
bind 0.0. 0. 0
pidfile "/var/run/redis_6379.pid"
appendonly yes
aof-use-rdb-preamble yes
/****** Configure master end ******/

/****** Configure slave ******/
// Create directory redis6380
mkdir redis6380
// make setup is the same as above, configuration needs to be different
// Copy 6379's redis.conf and modify some configurations
port 6380
replicaof 127.0. 01. 6379
replica-read-only yes
Copy the code

Now that you have everything installed, start

Go to the redis6379 directory and enter bin/redis-server redis.conf in the redis root directory

Go to the redis6380 directory and enter bin/redis-server redis.conf in the redis root directory

Go to the redis6381 directory and enter bin/redis-server redis.conf in the redis root directory

See whether have already started: ps – ef | grep redis

Log in to the primary node using a client and run the info replication command to view primary/secondary information: bin/redis-cli -p 6379

It can be seen that there are two connected slaves. The role of 6379 is master. The specific IP ports of the two slaves can also be clearly displayed.

Check whether data on the master node can be synchronized to the slave node

  1. The slave node was first connected to test data write and read. The data cannot be written and cannot be read from testkey.

  2. Connect to the master node and write testkey 123.

  3. Connect 6380 to 6381 and find that testKey is ready to read

Redis Sentinel architecture

Sentinel sentinel is a special Redis service that does not provide read and write services. Sentinel sentinel is mainly used to monitor instance nodes of Redis.

In sentinel architecture, the client finds out the master node of Redis from sentinel for the first time, and then accesses the master node of Redis directly, instead of accessing the master node of Redis through sentinel agent every time. When the redis master node changes, the sentry senses it and elects a new master node to notify the client. Therefore, to use the Sentinel architecture, clients must implement redis subscriptions to subscribe to sentinel published node changes.

Build a 3-node cluster redis Sentinel

  1. Create sentinel1, Sentinel2, and Sentinel3 folders, drop the downloaded Redis package into them, unzip it, and make the installation.

  2. If you go to the redis root directory, you’ll see a sentinel.conf file and modify the following properties

protected-mode no
bind 0.0. 0. 0
port 26379
daemonize yes
pidfile "/var/run/redis-sentinel-26379.pid"
logfile "26379.log"
// It is recommended to use the same directory for the three Sentinel nodes in the following configuration. I test if the cluster cannot be built with different directories
dir "/opt/redis/sentineldata"
// If the name below is not myMaster, the startup will report an error, because there are other references to the name, if you want to change it all
// At least two sentinels in the cluster must consider the master to be dead before a new master is elected
sentinel monitor mymaster 127.0. 01. 6379 2
Copy the code

So modify the three sentinel.conf files and start one by one

Start command: bin/redis-sentinel sentinel.conf

Vim sentinel.conf, shift+G to the bottom, you can see that sentinel.conf has loaded the sentinel information and the primary and secondary nodes of redis (each sentinel node in the sentinel cluster will have it).

Test whether the Sentinel cluster can vote properly

  1. Take out Redis on 6379

  2. If you do not stop cat sentinel.conf, you will find that the primary/secondary information in the CONF changes after more than 10 seconds.

6379 becomes a slave node, which of course means 6381 becomes the master node.

  1. Check whether 6381 can execute the write command and check the master/slave info. It is found that 6381 can write commands, and the slave node has only 6380 but no 6379. This is because 6379 has not been started.

  1. If you start 6379, it will automatically become a slave node

Sentinel configuration file modified after the election

  1. Modified the configuration of each sentinel node, such as master/slave information at the end of the sentinel.conf file, and master node configuration. In the following figure, we initially configured 6379, but after the election, monitor in sentinel.conf changed to 6381.

  1. Modified redis.conf for each node of redis master and slave, such as replicaof.
  • 6379 add Replicaof 6381

  • Change replicaof 6380 to 6381

  • 6381 deletes the replicaof attribute

Because the startup configuration of the real Redis-server is modified, when 6379 is started, it is directly slave node.