Daily sentence

Those who do not have the opportunity to do great things, because they have not done small things to prove their strength. In the more obscure place, the more active light. Prove yourself by doing small things well before others dare to do big ones.

The premise to review

The summary of AOF

  • The Redis AOF is similar to the Log mechanism. Every write operation is written to the disk. In the event of a system crash, the AOF can be used to recover data.

  • Every time a command with a write operation is run by the Redis server, the command is logged to the AOF file. Since it’s just an append to a file operation (stored in aof_buf and using the asynchronous method fsync/fdatasync), writes to the disk tend to be very fast.

The realization of AOF

The Redis AOF mechanism includes two things, AOF and Rewrite.

  • AOF is similar to a common DATABASE management system log recovery point. When AOF files swell with write commands, rewrite is run when the file size reaches a critical level.

  • Rewrite will, like Replication, fork out a child process, create a temporary file, iterate through the database, and output every key and value pair to the temporary file.

    • The output format is Redis command. However, in order to reduce the file size, multiple key and value pairs are collected and expressed in one command.

    • Write operations during rewrite are stored in aof_rewrite_buf_block in memory. After rewrite succeeds, these operations are copied to temporary files that replace AOF files.

If AOF is closed, then the rewrite operation can be performed with the bgrewriteaof command.

The process of AOF

  • Redis Server starts, initializes the AOF state if the AOF mechanism is enabled, and reads the AOF file if it exists. As Redis continues to receive commands, each write command is added to the AOF file, either when the AOF file expands to the rewrite threshold and needs to be overwritten or when the bgreWriteAof command is received from the client.

  • Fork out a child process to rewrite, while the parent process continues to accept the operation commands, now all write commands are added to an aof_rewrite_buf_blocks buffer.

  • When the child process ends the rewrite process, the parent process receives an exit signal from the child process, adds aOF_rewrite_buf_blocks buffer to the AOF file after rewriting, and then switches the AOF file fd to replace the old file with the new file. Rewrite done, move on to step two.

The key point

  • Since write operations are usually cached, it is possible that the AOF operation is not written to the hard disk. You can force output to the hard disk using fsync()/fdatasync().

  • The frequency of fsync() can be specified by using the flush policy in the configuration file. You can choose to force fsync for each event loop or to run fsync at least once per second.

  • ** When the rewrite child process starts, the commands received by the parent process are added to aof_rewrite_buf_blocks, so that when the rewrite process succeeds, these commands are added to the new file.

  • In the rewrite process, the original AOF can be added or not. Due to performance problems, if fsync() continues to be executed in the rewrite process, I/O performance will be damaged and Redis performance will be affected. So generally forbid fsync() to old AOF files during rewrite. This policy can be modified in the configuration file. (this is the synchronization mechanism for AOF operations in the rewrite process, which can be temporarily stopped and not saved to aof files).

  • After the rewrite ends, when the new rewrite file is renamed to the file specified in the configuration, if the old AOF exists, the old file will be unlinked and deleted.

    • The problem is that when rewriting files are migrated to the main thread, the rename(oldPath, newpath) process overwrites the old file. In this case, the rename process unlinks (oldfD), and the unlink operation blocks the main thread.
    • At this time, we need to * * a similar libeio (software. Schmorp. DE/PKG/libeio….

Automatic bgrewriteaof

  • To avoid the size of the aof file, we will periodically bgrewriteaof the aof file to restructure. Previously, we would configure crontab as an extra command to execute this command during peak hours. This extra script task of adding a workaroud is very bad in a large cluster because it is not easy to check and errors cannot be detected immediately.

  • The automatic bgreWriteaof function is then added directly to redis. First for the Aof file, the server object adds a field to record the size of the aOF file server.appendonly_CURRENT_size, which is maintained each time the Aof changes.

aof.c

Nwritten = write(server.appendfd, server.aofbuf, sdslen(server.aofbuf)); . server.appendonly_current_size += nwritten;Copy the code

AofUpdateCurrentSize is also called to maintain this field after bgrewriteAof is completed or after the instance is started to load aof data. It also records the size of the Aof file at this time. Used to determine the Aof growth rate next.

aof.c


aofUpdateCurrentSize();
server.auto_aofrewrite_base_size = server.appendonly_current_size;

Copy the code

With the current and base values we can determine the growth of the Aof file. Two more parameters need to be configured to determine whether bgreWriteAof needs to be triggered automatically.

redis.h


int auto_aofrewrite_perc; /* Rewrite AOF if % growth is > M and... * /
off_t auto_aofrewrite_min_size; /* the AOF file is at least N bytes. */

Copy the code
  • Auto_aofrewrite_perc: BgreWriteAof is triggered when the size of an AOF file exceeds the baseline percentage. By default, this value is set to 100, meaning that bgreWriteAof is triggered when the current AOF is twice the base size. Setting it to 0 disables automatic triggering.

  • Auto_aofrewrite_min_size: Specifies the size of the current AOF file before triggering. Avoid unnecessary actions when the AOF is small. The default size is 64mb.

Both parameters can be statically configured in conf or dynamically changed using the config set.

Redis 127.0.0.1:6379> config get auto-aof-rewrite-percentage 1) "auto-aof-rewrite-percentage" 2) "100" redis 127.0.0.1:6379> config get auto-aof-rewrite-min-size 1) "auto-aof-rewrite-min-size" 2) "1048576" redis 127.0.0.1:6379> Config get auto-aof-rewrite-min-size 1) "auto-aof-rewrite-min-size" 2) "1048576" redis 127.0.0.1:6379> config set Auto-do-rewrite-percentage 200 OK redis 127.0.0.1:6379> config set auto-do-rewrite-min-size 10485760 OKCopy the code

Then there is the main logic that triggers the check. The serverCron time event checks the existing status and parameters each time to determine whether bgreWriteAof needs to be started.

redis.c

if (server.bgsavechildpid == -1 && server.bgrewritechildpid == -1 && server.auto_aofrewrite_perc && server.appendonly_current_size > server.auto_aofrewrite_min_size) { long long base = server.auto_aofrewrite_base_size ? server.auto_aofrewrite_base_size : 1; long long growth = (server.appendonly_current_size*100/base) - 100; If (growth >= server.auto_aofrewrite_perc) {redisLog(REDIS_NOTICE, "Starting automatic acceleration of AOF on % LLD %% growth", growth); rewriteAppendOnlyFileBackground(); }}Copy the code

If the percentage of aof file growth is greater than auto_aofrewrite_perc, then the next bgreWriteAof is automatically triggered.

Delay bgrewriteaof

This is a small change. When bgreWriteAof is manually triggered, it will delay the operation if there is a backup with bgSave. Set server. aofrewrite_Scheduled =1 to wait until the next serverCron after bgsave ends.

Set the aofrewrite_scheduled = 1

aof.c


 void bgrewriteaofCommand(redisClient *c) {
     if(server.bgrewritechildpid ! =- 1) {addReplyError (c,"Background append only file rewriting already in progress");
     } else if(server.bgsavechildpid ! =- 1) {
         server.aofrewrite_scheduled = 1; AddReplyStatus (c,"Background append only file rewriting scheduled");
     } else if(rewriteAppendOnlyFileBackground () = = REDIS_OK) {addReplyStatus (c,"Background append only file rewriting started");
     } else{addReply (c, Shared. Err); }}Copy the code

Trigger bgrewriteaof

redis.c

    if (server.bgsavechildpid == - 1 && server.bgrewritechildpid == - 1 &&
        server.aofrewrite_scheduled){
         rewriteAppendOnlyFileBackground();
    }
Copy the code