This blog is the third in the Redis series, which focuses on Redis’s two persistence mechanisms: RDB and AOF.

The first two articles in this series can be viewed by clicking on the following links:

Redis series (1) : Introduction to Redis and environment installation.

Redis series (2) : Redis 5 data structures and common commands

1. Why persistence?

Because Redis is an in-memory database, it stores its own data in memory. Once the Redis server process exits or the computer running the Redis server is down, the data in the Redis server will be lost.

In order to avoid data loss, Redis provides a persistence mechanism to save the data stored in memory to disk, which is used to quickly recover the data stored in memory when the Redis server process exits or the computer running the Redis server stops and the data is lost.

Redis provides two types of persistence, which are:

  1. RDB persistence
  2. AOF persistence

Next, let’s go through them one by one.

2. RDB persistence

RDB persistence is to save the data in Redis at a point in time to an RDB file, as shown below:

RDB persistence is also called snapshot persistence because of the above properties of RDB persistence.

This file is a compressed binary through which you can restore the data in Redis when generating the RDB file, as shown below:

2.1 Creating an RDB File

Redis provides two commands to create RDB files, one is SAVE and the other is BGSAVE.

The SAVE command blocks the Redis server process until the RDB file is created, and the server cannot process any command requests while the server process is blocked, as shown below:

The BGSAVE command gives birth to a child process, which is then responsible for creating the RDB file, and the server process (parent) continues to process the command request, as shown below:

The above description is also the difference between these two commands. This is the main point, which is often asked in interviews.

The BGSAVE command is recommended because it can be executed without blocking the server process.

This command can be executed manually, as shown in the screenshot above, but it is recommended to set the save option in the Redis server configuration file so that the server automatically executes the BGSAVE command once in a while.

Multiple save criteria can be set with the Save option, and the server will execute the BGSAVE command if any of the criteria is met.

The default conditions for setting the save option are as follows:

save 900 1

save 300 10

save 60 10000

The default configuration condition indicates that the BGSAVE command will be executed if any of the following three conditions are met:

  • The server modifies the database at least once within 900 seconds (15 minutes)
  • The server made at least 10 changes to the database within 300 seconds (five minutes)
  • The server made at least 10,000 changes to the database within 60 seconds (that is, 1 minute)

When the BGSAVE command is executed, the log output is as follows:

The generated RDB file is saved according to the name and path in the Redis configuration file. The two related configurations are as follows:

The resulting RDB file is shown as follows (the screenshot shows the path of the RDB file in the local Windows environment. The path may be slightly different in Linux) :

2.2 Loading the RDB File

First, let’s make it clear that the purpose of loading the RDB file is to restore the data previously stored in Redis after the Redis server process restarts.

However, there is no special command for Redis to load the RDB file, but it is automatically executed when the Redis server starts.

In addition, whether the RDB file will be loaded when the Redis server is started depends on whether the AOF persistence function is enabled on the server.

  1. RDB files are used by the server to restore data only when AOF persistence is turned off.
  2. If AOF persistence is enabled on the server, the server will use AOF files to restore data in preference.

The above judgment logic is shown in the figure below:

AOF persistence is turned off on Redis servers by default, so Redis servers load RDB files when they start up,

The startup log is as follows:

2.3 Server Status

To create and load an RDB file, there are three possible server states:

  1. When the SAVE command is executed, the Redis server blocks, and all command requests sent by the client are blocked. The command requests sent by the client are processed only after the server executes the SAVE command and starts accepting command requests again.
  2. When the BGSAVE command is executed, the Redis server does not block, and the Redis server can continue to process command requests sent by the client.
  3. The server blocks while loading the RDB file until the RDB file is successfully loaded.

3. AOF persistence

AOF persistence records database data by saving write commands executed by the Redis server, as shown in the following figure:

By default, AOF persistence is turned off. If you want to turn it on, you can modify the configuration shown below:

For example, assuming no data is stored in Redis, we execute the following command:

Then we see that the Redis server generates a file called appendone.aof, which opens and we see that the three write commands executed above are stored in this file: appendone.aof

3.1 Implementation of AOF persistence

When AOF persistence is enabled, Redis server will append the executed write command to the end of the server state AOF buffer in the protocol format (as shown in the screenshot above) after executing a write command. The Redis server then decides when to write and synchronize the contents of the AOF buffer to the AOF file based on the appendfsync option value in the configuration file.

The appendfsync option has the following three values:

  1. always

    Always is the safest (least lost data) in terms of security, because even in the event of a downtime, the database will only lose command data generated in one event loop.

    Always is the slowest in terms of efficiency, because the server writes everything in the AOF buffer to the AOF file and synchronizes the AOF file at each event loop.

  2. everysec

    In terms of security, in Everysec mode, the database loses command data for only a second in the event of a downtime.

    In terms of efficiency, everysec mode is fast enough because the server writes everything in the AOF buffer to an AOF file at each event loop and synchronizes the AOF file in child threads every second.

  3. no

    From safety, no mode, if there is downtime, the database will be lost the last synchronization AOF file after all the write command data, is uncertain, because the server in each event loop will AOF all contents of the buffer writes to AOF file, as for when to AOF file synchronization, it is controlled by the operating system.

    In terms of efficiency, no mode is about as efficient as Everysec mode.

The default value of the appendfsync option is everysec, which is also recommended for efficiency and security.

3.2 Loading the AOF file

Because the AOF file contains all the write commands needed to rebuild the database, the Redis server simply reads and re-executes the write commands saved in the AOF file to restore the data before the Redis server was shut down.

Redis reads AOF files and restores the database as follows:

  1. Create a pseudo client with no network connection

    Because Redis commands can only be executed in the client context, and the commands used to load AOF files come directly from AOF files and not from a network connection, the server uses a pseudo-client with no network connection to execute the write commands saved by AOF files.

    The effect of the command executed by the pseudo client is exactly the same as that executed by the client with network connection.

  2. Parse and read a write command from the AOF file.

  3. Use pseudo clients to execute read read write commands.

  4. Continue steps 2 and 3 until all write commands in the AOF file have been executed.

The above steps are shown in the figure below:

If the Redis server has AOF persistence enabled, then the Redis server loads AOF files when it starts.

The startup log is as follows:

3.3 AOF rewrite

Because AOF persistence records database data by saving write commands executed, with the increase of Redis server running time, the contents of AOF files will become more and more, and the size of files will become larger and larger. If not controlled, it will have the following two disadvantages:

  1. Excessive use of server disk space may affect the Redis server or even the entire host computer.
  2. The larger the AOF file, the more time it takes to use the AOF file for database restoration.

For example, run the following command on the client:

To record the state of the list key, the AOF file needs to hold the six commands executed above.

In order to solve the problem of the increasing size of AO files, Redis provides the AOF file rewriting function. That is, the Redis server creates a new AOF file to replace the existing AOF file. The old and new AOF files hold the same database data, but the new AOF file does not contain any redundant commands that waste space. So the size of new AOF files is usually much smaller than the size of old AOF files.

3.3.1 Implementation principle of AOF rewrite

AOF file rewrite does not require any reading, parsing, or writing of existing AOF files, but rather reads the server’s current database data.

Using the list key above as an example, the old AOF file stored six commands to record the state of the list key, but the result of the list key was “C”, “D”, “E”, “F”, “G”, so when the AOF file was rewritten, You can replace the previous six commands with a single RPUSH list “C” “D” “E” “F” “G” command, thus reducing the number of commands required to save the list key from six to one.

According to the principle above, if the Redis server stores enough key-value pairs, the new AOF file generated by the AOF file rewrite will have many, many fewer redundant commands, thus greatly reducing the size of the AOF file.

To sum up, the realization principle of AOF file rewriting is as follows:

The current value of the key is read from the database, and a single command is used to record the key-value pair instead of multiple commands.

3.3.2 AOF background rewrite

Because AOF file rewriting does a lot of file writing, the thread that performs this operation is blocked for a long time.

Because the Redis server uses a single thread to process command requests, if the server process performed this operation directly, the server would not be able to process the command requests sent by the client during the rewriting of the AOF file.

To avoid these problems, Redis places the AOF file rewrite function in a child process. This has two advantages:

  1. The server process (parent) can continue to process command requests while the child process does the AOF file rewrite.
  2. The child process has a copy of the server process’s data, and using the child process instead of the thread keeps the data secure without using locks.

The steps of AOF background rewrite are as follows:

  1. The server process creates the child process, which begins the AOF file rewrite

  2. From the time the child process is created, all write commands executed by the server process are written not only to the AOF buffer, but also to the AOF rewrite buffer

    The purpose of writing to the AOF buffer is to synchronize to the original AOF file.

    The purpose of writing the AOF rewrite buffer is because the server process continues to process command requests while the child process is rewriting the AOF file,

    The new command may modify the existing database, so that the server’s current database data and the rewritten AOF file are lost

    The saved database data is inconsistent.

  3. The child process completes the AOF rewrite and sends a signal to the parent process. Upon receiving the signal, the parent process performs the following operations:

    1. Write all the contents of the AOF rewrite buffer to the new AOF file. This ensures that the database data stored in the new AOF file is the same as the current database data on the server.

    2. Rename the new AOF file, atomically overwrite the existing AOF file, and complete the replacement of the old and new AOF files.

Redis provides the BGREWRITEAOF command to perform the above steps, as shown below:

Appendone. aof (list); appendone. aof (list);

In addition to executing the BGREWRITEAOF command manually, Redis also provides two configuration items for automatically executing the BGREWRITEAOF command:

auto-aof-rewrite-percentage 100

auto-aof-rewrite-min-size 64mb

This configuration says that Redis will automatically execute the BGREWRITEAOF command when the AOF file is larger than 64MB and the AOF file is at least twice (100%) larger than the size since the last rewrite.

4. Difference between RDB persistence and AOF persistence

From the above explanation, we can see that the two persistence methods provided by Redis are different, which can be summarized as the following four points:

  1. implementation
  2. The file size
  3. security
  4. priority

Let’s go through them.

4.1 Implementation Method

RDB persistence is implemented by saving the data stored by the Redis server at a point in time to an RDB file.

AOF persistence is implemented by saving all write commands executed by the Redis server to an AOF file.

4.2 File Volume

As can be seen from the above implementation, RDB persistence records results, while AOF persistence records processes, so AOF files generated by AOF persistence will have the problem of increasing size. Redis provides AOF rewrite function to reduce the size of AOF files.

4.3 security

AOF persistence is more secure than RDB persistence, that is, less data is lost in the event of a machine failure than RDB persistence.

Because RDB persistence loses data written since the last RDB persistence, AOF persistence loses data written in less than 1s (using the default Everysec configuration).

4.4 priority

Due to the above security problems, if the Redis server has AOF persistence enabled, the Redis server will use THE AOF file to restore the data when it is started. If the Redis server does not have AOF persistence enabled, the Redis server will use the RDB file to restore the data when it is started. Therefore, AOF files have a higher priority than RDB files.

5. Source code and reference

Josiah L. Carlson, Reids in Action

Redis Design and Implementation by Huang Jianhong