Redis provides two types of persistence: RDB and AOF

RDB

Redis DataBase (RDB) means a snapshot of Redis memory data ata point in time. RDB persistence can be triggered in three ways:

  • Manually triggered by the SAVE command: The main Redis process performs persistent actions that block other event processing (including time and file events)
  • The BGSAVE command is triggered manually: Redis performs persistent actions in the fock() child process. Blocks during fork(), and the main process stops blocking after fork()
  • Configure automatic triggering: You can configure the SAVE option to enable the Redis server to automatically trigger the function corresponding to the BGSAVE command to persist when specified conditions are met

Question 1: How does Redis monitor whether the automatic trigger condition of configuration is met? Answer: Redis’s time event function serverCorn() executes every 100ms by default (controlled by Redis’s event driver, Reactor), This function checks whether the current Redis state meets the RDB persistence criteria and executes asynchronous persistence using the rdbSaveBackground function corresponding to the BGSAVE command.

Question 2: Why does Redis implement asynchronous persistence by fork() rather than by child thread? Answer: Because the RDB file has the meaning of “snapshot of in-memory data at a point in time,” asynchronous persistence using child threads does not guarantee that the data is the data at the point in time when persistence was started. The SAVE command blocking persistence is guaranteed, and BGSAVE is guaranteed through child processes, because data is isolated between processes.

Question 3: Will the fork() command block for a long time? How is it guaranteed? Answer: Not long. The Linux fork() function uses the COW mechanism (Copy On Write) to ensure that the fork() function does not directly Copy all the memory data of the parent process into the memory space of the child process, but copies the page that is written when the Write operation occurs, thus realizing the fast return of the fork() function. For details, see the Linux fork() COW mechanism.

Question 4: Will Redis generate multiple RDB files overwrite each other? How is that done? Answer: When Redis does RDB persistence, it first creates a temporary file, writes the data in Redis to the file and brushes it to disk, and finally changes the temporary file to an official file to overwrite the old and new files.

AOF

AOF(Append Only File) is persisted in the form of writing operation logs, which means that Redis data changes are continuously recorded. AOF persistence can be divided into four blocks: command appending, file write, file rewrite, and file load.

Command to add

Command appending writes operations to the AOF buffer in protocol format (RESP– the communication protocol over which Redis client servers interact). Interest in RESP can be learned separately.

File is written to

Every time Redis completes an event loop, the flushAppendOnlyFile() function is called to determine whether the contents of the AOF buffer need to be synchronized to the AOF file. The behavior of the flushAppendOnlyFile() function is controlled by the appendfsync option in the redis.conf configuration:

  • Always: The contents of the AOF buffer are written to the AOF file and flushed before the end of each event loop. Worst performance and safest. An outage only loses data in one event loop
  • Everysec: Writes the contents of the AOF buffer to the AOF file at the end of each event loop and flushers the child thread every 1s.
  • No: The contents of the AOF buffer are written to the AOF file before the end of each event loop, and the flushing action is controlled by the operating system.

Question: what is the relationship between writing files and brushing disk? Answer: Write on Linux triggers delayed write. Linux provides page buffers in the kernel to optimize disk IO performance. The write operation returns directly after writing to the system buffer. The disk synchronization operation (disk flushing) depends on the scheduling mechanism of the operating system. For example, the buffer page space is full or a specific period of time is reached. Data in the buffer may be lost due to system downtime before disk flushing. Fsync forces the operating system to flush for single file operations to ensure data persistence.

The three appendfsync values here represent three different fsync policies. The more frequently fsync is invoked, the lower the read and write efficiency, but the higher the security.

File to rewrite

AOF records operation logs. As time goes by, files become larger and larger, so file rewriting is provided. AOF file rewrite does not rely on existing AOF files, but reads Redis memory data and writes new AOF files. The file rewrite trigger mechanism consists of two:

  • The BGREWRITEAOF command is triggered manually
  • Configuring automatic triggering

The AOF overrides do a lot of writing, so Redis also chooses to execute in the child process. As with the RDB BGSAVE command, the child process has a snapshot of the memory data at the time the BGREWRITEAOF command was executed.

Question 1: When the child process rewrites the AOF file, the main process is still modifying the data in the event cycle. How to ensure that the rewritten AOF file does not lose data? Answer: Redis introduces AOF rewrite buffers. During AOF rewrite, the main process writes not only to the AOF buffer but also to the AOF rewrite buffer when performing a write operation. After the child process completes the snapshot rewrite operation, it can append the contents of the AOF rewrite buffer. If the contents of the AOF rewrite buffer are also written, you can rename the AOF file to make the new file overwrite the old file.

Question 2: AOF files do not have the meaning of “snapshot of data at a time” as RDB files do. Can the AOF rewrite be done directly with child threads instead of child processes? Solution: this did not see the source is not too clear. I feel that if Redis commands are idempotent and there is nothing else to do when AOF is overwritten, it is ok to use child threads.

File is loaded

File loading is simple because the operation logs are recorded in the AOF file and the Redis command can only be executed in the client context. Therefore, the process of loading AOF file is to create a pseudo client without network connection, and realize data recovery by replaying AOF file command.

doubt

Question 1: How does RDB persist expired keys? How do RDB files load with expired keys? Is the logic consistent when loading the RDB files from the primary and secondary servers? Answer: Not sure. The expiration key may not be written when RDB files are written. RDB files are skipped if they are loaded from the master server and loaded altogether if they are loaded from the slave server.

Question 2: How are expired keys handled in AOF persistence? How are expired keys handled when AOF files are loaded? Is the logic consistent when loading AOF files on the master and slave servers? Answer: Not sure. If the stale key is not obsolete, no special operation is performed. If the stale key is obsolete, a DEL operation is appended to the AOF file. Replay the command when loading the AOF file.

Question 3: Based on the above two questions, how do the primary and secondary servers process expired keys in replication mode? Answer: Not sure. It is possible that the primary server notifies the secondary server after the expired key is eliminated. If the data is read directly from the slave server, the slave server does not eliminate the direct return value even if it encounters the expiration key on the lower version Redis, and the slave server will return null if it encounters the expiration key on the higher version Redis.