Redis AOF and RDB persistence policy principles

Data persistence

By default, Redis data is stored in memory. To avoid data loss caused by the unexpected Redis process exit, Redis provides RDB and AOF to implement persistent storage of data.

Second, RDB mechanism

The RDB mechanism generates snapshots of Redis data at a specified interval and saves the snapshots to hard disks. RDB is more suitable for periodic data backup scenarios. The RDB mechanism can be triggered manually or automatically:

2.1 Manual Triggering

You can manually trigger the RDB mechanism in the following two ways:

• Save: The save command blocks the current Redis service until the RDB backup process is complete, during which time all queries on the client are blocked.
• Bgsave: The Redis process forks a child process, blocking only during the fork phase, after which persistent operations are performed by the child process.

2.2 Automatic Trigger

In addition to manually using commands, Redis’s RDB mechanism is automatically triggered in certain scenarios:

• inredis.confConfiguration of thesave m nIs triggered automatically if n modification operations occur within m secondsbgsave;
• If a full copy operation is performed on the secondary node, it is automatically performed on the primary nodebgsaveAnd send the generated RDB file to the slave node.
• performdebug reloadIs triggered when the Redis command is reloadedsaveOperation;
• performshutdownCommand is used by default if AOF persistence is not enabledbgsavePersist.

2.3 Related Configurations

1. File directory

By default, the RDB file is saved in the working directory of Redis. The default file name is dump. RDB and can be changed statically or dynamically.

• Static configuration: Modify the working directory dir and database storage filename dbfilename in redis.conf.

• Dynamic change: Run the following command on the CLI:

  config set dir{newDir}
  config set dbfilename{newFileName}
  Copy the code

2. Compression algorithm

Redis uses the LZF algorithm to compress the generated RDB files by default, which can reduce the space occupied and the amount of data transmitted over the network. However, the compression process consumes CPU computing resources. You can choose whether to enable the compression based on the actual situation. This can be dynamically modified by modifying the rdbcompression configuration in redis.conf or by using the following command:

config set rdbcompression{yes|no}
Copy the code

3. AOF mechanism

AOF is another persistent method provided by Redis. It records each write operation in a separate log, and then performs these operations again upon restart, so as to achieve the command of data recovery.

3.1 Execution Principles

After the AOF mechanism is enabled, all write commands are appended to the aOF_buf buffer and data in the buffer is periodically synchronized to the disk according to the specified policy. In addition to recording each command, AOF will fork out a sub-process to rewrite the AOF file at appropriate time. During the rewriting process, Redis will merge the statements that can be merged and delete the invalid ones, thus reducing the size of the AOF file. In this way, the file occupies less space and can be loaded faster during data recovery.

3.2 Synchronization Policy

Redis provides three synchronization policies that control the behavior of the AOF buffer to synchronize data to disk, controlled by the appendfsync parameter:

Write and fsync operation instructions:

• The write operation triggers the write delay mechanism. Linux provides a page buffer in the kernel to improve disk I/O performance. The write operation returns directly after being written to the system buffer. Synchronous operations depend on system scheduling mechanisms, such as when buffer page space is full or when a specific time period is reached. If the system breaks down before file synchronization, data in the buffer will be lost.
• Fsync forces disk synchronization for a single file operation. The fsync operation will be blocked until the data is returned after being written to the disk, ensuring data persistence security.

The default Redis synchronization mechanism is Everysec, which ensures both performance and data security. In the event of an unexpected outage, at most one second of data can be lost.

3.3 Related Configurations

To use the AOF function, set appendOnly to yes and default to no. The default filename of AOF is appendone. AOF, which can be changed by changing the value of appendfilename. The default file is saved in the same location as the RDB file, in the Redis working directory.

4. Comparative analysis

4.1 Advantages and disadvantages

The advantages of RDB

• RDB generates a memory snapshot at a time to provide a higher file compression ratio and is suitable for backup and full replication scenarios.
• RDB files are usually smaller than the equivalent AOF files for the same data set, so data recovery using RDB is much faster than the AOF method.
• RDB maximizes the performance of Redis because the parent process only forks out a child process, which does not perform disk I/O and other operations.

The disadvantage of RDB

• RDB cannot implement real-time persistence of data. Assume that the interval of each persistence is 5 minutes. If a service outage occurs 3 minutes after the last persistence, all data within 3 minutes will be lost.
• Forking is a heavyweight operation and can be time-consuming if the data set is large.

The advantages of AOF

• AOF can implement real-time or second-level persistence operations to ensure the least loss of data.
• If the logs are interrupted suddenly and the half-write command is used, you can use the redis-check-aof tool to repair the logs to minimize data loss.

The disadvantage of AOF

• AOF files are usually larger than equivalent RDB files of the same data set.
• Depending on the synchronization strategy chosen, AOF may be slower than RDB.

4.2 Suggestions

According to Redis official recommendation, both persistence mechanisms can be used together to ensure data security. In Redis official long-term plan, AOF and RDB may be unified into a single persistence model in the future. Note that in this case, Redis will rebuild the dataset using AOF files when Redis restarts, as it guarantees minimal data loss.

The resources

1. Fu Lei, Zhang Yijun. Redis Development, Operation and Maintenance. China Machine Press. 2017-3-1
2. Official documentation: Redis Persistence

For more articles, please visit the full stack Engineer manual at GitHub.Github.com/heibaiying/…