Redis is an in-memory database that keeps data in memory and reads and writes much faster than traditional databases that keep data on disk. But once the process exits, Redis data is lost.

To solve this problem, Redis provides two persistence schemes, RDB and AOF, to save data in memory to disk to avoid data loss. Redis RDB persistence Redis RDB persistence Redis RDB persistence

AOF(Append Only File) persistence records every write command in an independent log. When Redis restarts, commands in the AOF file are re-executed to recover data. The main function of AOF is to solve the real-time of data persistence.

RDB and AOF

In his article “Redis Persistence and Decryption”, Antirez describes the advantages and disadvantages of RDB and AOF:

  • The RDB is a compact binary file that represents a backup of Redis data at a point in time. Ideal for backup, full copy, and other scenarios. For example, perform bgSave backups every 6 hours and copy RDB files to remote machines or file systems for disaster recovery.
  • Redis loads RDB recovery data much faster than AOF
  • RDB data cannot be persisted in real time, while AOF data can.

Let’s take a look at how AOF implements real-time persistence.

Implementation of AOF persistence

As shown in the figure above, the implementation of AOF persistence can be divided into command append, file write, file sync, file rewrite and restart load. The process is as follows:

  • All write commands are appended to the AOF buffer.
  • The AOF buffer synchronizes data to disks based on the corresponding policy.
  • As AOF files become larger, they need to be rewritten periodically to achieve compression.
  • When Redis restarts, AOF files can be loaded for data recovery.

Command to add

When AOF persistence is enabled, Redis appends a write command to the end of the AOF buffer maintained by the Redis server in the protocol format (RESP) after executing a write command.

For example, the SET mykey myValue command is logged into the AOF buffer in the following format.

"*3\r\n$3\r\nSET\r\nA $5\r\nmykey\r\n$7\r\nmyvalue\r\n"
Copy the code

Redis protocol format will not be described in this article. The reason why AOF directly adopts text protocol format is that all write commands need to be appended, so the protocol format is directly adopted to avoid secondary processing overhead.

File writing and synchronization

Before Redis terminates an event loop, it calls the flushAppendOnlyFile function to determine whether the contents of the AOF cache need to be written to and synchronized to the AOF file.

The behavior of the flushAppendOnlyFile function is determined by the value of the appendfsync option in the redis.conf configuration. This option has three optional values: always, Everysec, and no:

  • alwaysRedis writes all the contents of the AOF buffer to the AOF file and synchronizes the AOF file in each event loop, soalwaysThe efficiency isappendfsyncThe option is the worst of the three, but it’s also the safest in terms of security. AOF persistence also loses only command data generated in an event loop in the event of a failure.
  • everysecRedis writes all the contents of the AOF buffer to the AOF file in each event loop and synchronizes the AOF file in child threads every second. In terms of efficiency, this model is fast enough. Only one second of command data is lost in the event of a malfunctioning outage.
  • noRedis writes all the contents of the AOF buffer to the AOF file in each event loop. The synchronization of AOF files is controlled by the operating system. This mode has the highest speed, but requires a long synchronization interval. Therefore, a large amount of data may be lost when a fault occurs.

In Linux, the write operation triggers the delayed write mechanism. Linux provides a page cache in the kernel to provide disk IO performance. The write operation returns directly after writing to the system buffer. Disk synchronization depends on the system scheduling mechanism. For example, the buffer page space is full or a certain period of time is reached. If the system breaks down before file synchronization, data in the buffer will be lost.

Fsync, on the other hand, forcibly synchronizes a single file and blocks it until it returns after writing to disk, ensuring data persistence.

The three appendfsync values represent three different policies for calling fsync. The more frequently fsync cycles are called, the worse the read and write efficiency is, but the higher the security, the less data is lost in the event of an outage.

The I/O for Linux and the role of the individual system calls are shown below. For details, see “Talking About Linux I/O.”

AOF data restoration

The AOF file contains all the write commands needed to rebuild Redis data, so Redis can restore the state before Redis is shut down by reading and re-executing the write commands saved in the AOF file.

Redis reads the AOF file and restores the database state as follows:

  • Create a fake client without a network connection, because Redis commands can only be executed in the client context, and the commands used to load AOF files come directly from the AOF file and not from the network connection. Therefore, the server uses a pseudo client without network connection to execute the write command saved by AOF file, and the effect of executing the command on the pseudo client is exactly the same as that of executing the command on the client with network connection.
  • Analyze and extract a write command from the AOF file.
  • Use pseudo clients to execute read write commands.
  • Continue steps 2 and 3 until all write commands in the AOF file have been processed.

When the above steps are complete, the database state saved in the AOF file will be restored in its entirety.

AOF rewrite

Because AOF persistence records the state of Redis by keeping write commands executed, as Redis runs for a long time, the contents of AOF files become larger and larger, and if left unchecked, Oversized AOF files may affect Redis or even the host computer.

To solve the problem of bloated AOF files, Redis provides AOF file rewrite. With this feature, Redis can create a new AOF file to replace an existing AOF file. The old and new AOF files hold the same Redis state, but the new AOF file does not contain any honor commands that waste space, so the size of the new AOF file is usually much smaller than the size of the old AOF file.

As shown in the figure above, the state of the key named list is recorded before the override, and the AOF file holds five commands. After the override, you only need to hold one command.

AOF file rewrite does not require any reading, parsing, or writing of existing AOF files, but rather reads the current database state of the server. First read the current value of the key from the database, and then use a command to record the key value pair, instead of recording the previous multiple commands, this is the realization principle of the AOF rewrite function.

In practice, to avoid client input buffer overflows when executing commands, the AOF override checks the number of elements in a list, hash, set, and ordered set of keys that may contain multiple elements. If the number exceeds the REDIS_AOF_REWRITE_ITEMS_PER_CMD (typically 64) constant, use multiple commands to record the value of the key instead of one command.

Rewrite is triggered by three main mechanisms:

  • Call bgrewriteaof manually, and rewrite will be delayed if a child process is running, or rewrite will be triggered again.
  • Manually enabling the AOF function with a configuration directive will trigger rewrite to write data from the current database to the rewrite file if there is no RDB child.
  • In the Redis timer, rewriting conditions that trigger once or have reached the configured AOF file size are automatically triggered when there is rewrite that needs to exit and no RDB or rewrite child processes are running.

AOF background rewrite

The AOF override does a lot of writing, and the thread calling it is blocked for a long time, so Redis performs the AOF override in the child process.

  • The Redis process can continue to process client command requests while the child process does the AOF rewrite.
  • The child process has a copy of the parent process’s memory data, which can ensure data security even if locking is not applicable.

However, during the AOF restart of the child process, Redis receives client commands that modify the existing database state, resulting in inconsistency between the current state of the data and the database state stored in the rewritten AOF file.

To do this, Redis sets up an AOF rewrite buffer, which is used after the server creates the child process. When Redis executes a write command, it sends the write command to both the AOF buffer and the AOF rewrite buffer.

When the child completes the AOF rewrite, it sends a signal to the parent. Upon receiving the signal, the parent calls a signal handler and performs the following:

  • Write all contents of the AOF rewrite buffer to the new AOF file to ensure that the database state saved in the new AOF file is the same as the current state of the server.
  • Rename the new AOF file, overwrite the existing AOF file atomically, and complete the replacement of the old and new files
  • Continue processing client request commands.

During the whole AOF background rewriting process, only the signal processing function will block the main Redis process. At other times, the AOF background rewriting will not block the main process.

Afterword.

We will continue to learn about Redis replication and clustering.

Personal blog address: Remcarpediem