This is the 12th day of my participation in the August More Text Challenge. For details, see:August is more challenging

Introduction to AOF persistence

AOF persistence records the state of the database by using write commands executed by the Redis server. As shown in the figure below:

Here’s an example:

For the above three commands, the server will produce an AOF file containing the following:

*2\r\n$6\r\nSELECT\r\n$1\r\n0\r\n
*3\r\n$3\r\nSET\r\n$3\r\nmsg\r\n$5\r\nhello\r\n
*5\r\n$4\r\nSADD\r\n$6\r\nfruits\r\n$5\r\napple\r\n$6\r\nbanana\r\n$6\r\ncherry\r\n
*5\r\n$5\r\nRPUSH\r\n$7\r\nnumbers\r\n$3\r\n128\r\n$3\r\n256\r\n$3\r\n512\r\n
Copy the code

In this AOF file, all the other commands we sent from the client are used to specify that the database SELECT command was automatically added by the server.

Second, the realization of AOF persistence

The implementation of AOF persistence can be divided into three steps: append, file write, and sync.

2.1 Adding Commands

When AOF persistence processing is enabled, the server executes a write command and appends the executed write command to the end of the aOF_buf buffer of the server state in protocol format:

struct redisServer {
    // ...
    // AOF buffer
    sds aof_buf;
    // ...
};
Copy the code

2.2 AOF file writing and synchronization

The Redis server process is an event loop in which file events receive and reply commands from the client, and time events execute scheduled functions such as serverCron.

The server calls flushAppendOnlyFile each time it terminates a loop, because the server may use write commands to append content to the aof_buf buffer when handling file events. Consider whether you need to write and save the contents of the AOF_buf buffer to an AOF file. This process can be represented by the following pseudocode:

def eventLoop(a):
    whileTrue: # Processing file events, receiving command requests and sending command replies # Processing command requests may add new content to the AOF_buf bufferprocessFileEvents(a)# Process time eventsprocessTimeEvents(a)Consider whether to write and save the contents of aof_buf to the AOF fileflushAppendOnlyFile(a)
Copy the code

The behavior of the flushAppendOnlyFile function is determined by the value of the appendfsync option configured on the server. The behavior generated by each value is shown in the following table:

If the user does not actively set the value for the appendfsync option, the default value for the appendfsync option is everysec.

AOF file loading and data restoration

Because the AOF file contains all the write commands needed to rebuild the database state, the server can restore the database state before the server was shut down by simply reading in and re-executing the write commands stored in the AOF file.

The restoration process is as follows:

A pseudo client is a client that does not have a network connection.

3, AOF rewrite

Because AOF persistence records the state of the database by storing the write commands that are executed, the size of the AOF file will grow as the server runs over time. If left unchecked, An oversized AOF file is likely to affect the Redis server or even the entire host computer, and the larger the SIZE of the AOF file, the more time it takes to restore using the AOF file.

To solve the problem of AOF swelling, Redis provides the AOF rewrite function. With this feature, the Redis server can create a new AOF file to replace the existing AOF file. The old and new AOF files hold the same database state, but the new AOF file does not contain any redundant commands that waste space, so the size of the new AOF file is usually much smaller than the old AOF file.

3.1 Implementation of AOF file rewrite

Although Redis will use the command “AOF rewrite” to generate new AOF files to replace old AOF files, in practice, AOF rewrite does not require any reading, analysis, or writing of existing AOF files. It is done by reading the current database state of the server.

Consider a case where the server executes the following command on the list key:

redis> RPUSH list "A"  "B"  //["A","B"]
(integer) 2
redis> RPUSH list "C"  //["A","B","C"]
(integer) 3
redis> RPUSH list "D" "E"  //["A","B","C","D","E"]
(integer) 5
redis> LPOP list //["B","C","D","E"]
"A"
Copy the code

Then the server must write four commands to the AOF file in order to keep the state of the current list key.

If the server wants to record the status of the list key in as few commands as possible, the simplest and most efficient way is not to read and analyze the contents of the existing AOF file, but to read the list of keys directly from the database, and then replace the four commands in the AOF file with the Rpush command. You can reduce the number of commands needed to save the list key from four to one.

With the exception of the list and collection keys listed above, all other types of keys can be used in the same way to reduce the number of commands in the AOF file. This is how the AOF rewrite function works.

The entire rewrite process can be expressed in pseudocode as follows:

def aof_rewrite(new_aof_file_name)Create a new AOF file f= create_file(new_aof_file_nameforDb in reddisserver. db: # Ignore empty databaseif db.is_empty(): continue# write SELECT command, specify database number f.wirite_command ("SELECT"+ db.id) # Traverse all the keys in the databaseforKey in db: # Ignore expired keysif key.is_expired(): continueOverride key based on key typeifkey.type == String: rewrite_string(key) elif key.type == List: rewrite_list(key) elif key.type == Hash: rewrite_hash(key) elif key.type == Set: rewrite_set(key) elif key.type == SortedSet: Rewrite_sorted_set (key) # If the key has an expiration time, the expiration time will also be overriddenifKey.have_expire_time (): rewrite_expire_time(); Value = GET(key) value = GET(key) Value) def rewrite_list(key) # def rewrite_list(key) ,itemN = LRANGE(key,0.- 1RPUSH, key, item1, item2,.... , itemN) # Other types are similar.Copy the code

Because the new AOF files generated by the aof_rewrite function contain only the commands necessary to restore the current state of the database, the new AOF files do not waste any disk space.

3.2 AOF background rewrite

Because the aof_rewrite function does a lot of writing, the calling thread is blocked for a long time. But Redis didn’t want the AOF rewrite to cause the server to be unable to process the request, so Redis decided to put the AOF rewrite program in a child process, which serves two purposes at once:

• The server process (parent process) can continue processing command requests while the child process is doing an AOF rewrite.
• The child process has a copy of the server process’s data, and using the child process instead of the thread ensures data security without using locks.

Use the child process, however, there is a problem to be solved, because the child process in AOF rewriting, the server process is still in processing command request, and the new command may modify the existing state of the database and making the current database server status and rewritten AOF files do not match the saved database state.

To solve this data inconsistency, the Redis server sets up an AOF override buffer. This buffer is used after the server creates a child process. When the Redis server executes a write command, it sends the write command to both the AOF buffer and the AOF override buffer. The process is as follows:

The above process ensures that:

• The contents of the AOF buffer are periodically written and synchronized to the AOF file, and processing of existing AOF files will proceed as usual.
• From the time the child is created, all write commands executed by the server are recorded in the AOF override buffer.

When the child finishes rewriting the AOF, it sends a signal to the parent, which, upon receiving the signal, calls a signal handler and performs the following:

• All the contents of the AOF rewrite buffer are written to the new AOF file. The new AOF file will hold the same database state as the server’s current database state.
• The new AOF file is renamed and the existing AOF file is overwritten atomically to complete the replacement of the old and new AOF files.

After this signal handler executes, the parent can continue receiving command requests as usual.

During the entire AOF background rewrite process, only the signal handler executes to block the server process (parent process), which minimizes the impact of THE AOF rewrite on server performance.