Data Persistence Strategy (RDB) in Redis

Redis is an in-memory database, all data is stored directly in memory, so, if the Redis process abnormally exits, or the server itself abnormally goes down, the data stored in Redis will disappear, never to be found again.

As an excellent data middleware, Redis must have its own persistent data backup mechanism. There are two main persistence strategies in Redis, which are used to back up data stored in memory to disk and reload backup files when the server restarts.

RDB and AOF are two kinds of data persistence strategies in Redis, which are two different persistence strategies, one is based on memory snapshot, the other is based on operation log, so this article will first talk about RDB based on memory snapshot persistence strategy.

What is RDB persistence policy

Redis Database (RDB) : snapshot persistence policy. RDB is the default redis persistence policy. You can open redis.conf and see these three configurations by default.

Save 900 1 This command is executed within 900 secondssetSave 300 10 Perform 10 operations within 300 secondssetSave 60 10000 10000 times within 60 secondssetOperation, the system persists 1 timeCopy the code

RDB is divided into two types, one is synchronous, call the save command can trigger Redis to generate RDB file backup, but this is a synchronous command, until the backup is complete, redis server does not respond to any client requests. The other is asynchronous, where the BGSave command is invoked and the Redis server forks a child process for RDB file backup, while the main process still responds to client requests.

Obviously, the asynchronous RDB generation strategy is the dominant one, and no one would block the REDis service with the save command to generate AN RDB file in a production environment, except in some special cases.

The above two commands, save and BGSave, need to be triggered manually by sending a request to the client, which we call active triggering.

And the configuration triggers that we talked about in a hurry, we call them passive triggers, and passive triggers have some configurations, so let’s look at them.

1. Save configuration

The Save configuration is a very important configuration that configures when the Redis server automatically triggers bgSave asynchronous RDB backup file generation.

Basic syntax:

save <seconds> <changes>
Copy the code

The bgSave command is invoked when the keys in the Redis database change once within a second.

2. Configure dbfilename

The dbfilename configuration item determines the name of the generated RDB file. The default value is dump.rdb.

dbfilename dump.rdb
Copy the code

3. Rdbcompression configuration

Rdbcompression is used to enable compression in RDB files.

rdbcompression yes(|no)
Copy the code

If RDBCompression is set to yes, then RDB file generation on behalf of Redis is performed. If a string object is encountered and its string value takes up more than 20 bytes, LZF compression is performed on the string.

4. Stop-writes-on-bgsave-error is configured

Stop-writes-on-bgsave-error Specifies whether to stop the redis write service if an error occurs during RDB backup file generation. The function is enabled by default.

stop-writes-on-bgsave-error yes(|no)
Copy the code

5. Configure dir

Dir Specifies the directory for storing RDB files. The default directory is the current directory.

dir ./
Copy the code

6. Rdbchecksum is configured

Rdbchecksum Specifies whether redis uses the CRC64 algorithm to check whether RDB files are corrupted. This function is enabled by default. If you need to improve performance, you can disable this function.

rdbchecksum yes(|no)
Copy the code

Saveparams and dirty counters

We have two fields in the redisServer structure:

The saveParams structure is defined as follows:

struct saveparam { time_t seconds; // seconds int changes; // number of changes};Copy the code

As you can imagine, the SAVE configuration in the above configuration file corresponds to two parameters that trigger bgSave as many times as the database changes in seconds.

Mapping to code is our SaveParam structure, where each saveParam structure corresponds to a line of save configuration that is eventually read into redisServer as an array of SaveParam.

Ps: The introduction of this is to pave the way for us to analyze the source code implementation of RDB file generation later.

In addition, there are two fields in the redisServer data structure:

The dirty field records how many more times the entire Redis database has been modified since the last successful RDB file backup, including the save and BGSave commands. The dirty_before_BGSAVE field can be understood as the total number of database changes during the last BGsave command backup.

There are also time fields related to persistence, the last time a successful RDB backup was performed, the last time a BGSave command was executed, and so on.

Let’s also paste and paste the source code to see how Redis performs RDB backup file generation.

int serverCron(....) {... // If a child is already performing RDB generation, or AOF recovery, or a child has not returnedif(server.rdb_child_pid ! = -1 || server.aof_child_pid ! = -1 || ldbPendingChildren()) { int statloc; pid_t pid; // See if the process returns a signalif ((pid = wait3(&statloc,WNOHANG,NULL)) ! = 0) { int exitcode = WEXITSTATUS(statloc); int bysignal = 0;if(WIFSIGNALED(statloc)) bysignal = WTERMSIG(statloc); // Persist exceptions and print logsif (pid == -1) {
                serverLog(LL_WARNING,"wait3() returned an error: %s. "
                    "rdb_child_pid = %d, aof_child_pid = %d",
                    strerror(errno),
                    (int) server.rdb_child_pid,
                    (int) server.aof_child_pid);
            } else if(pid = = server. Rdb_child_pid) {/ / success persistence RDB file, call methods careful RDB file to cover the old RDB backgroundSaveDoneHandler (exitcode bysignal);if(! bysignal && exitcode == 0) receiveChildInfo(); }else if(pid = = server. Aof_child_pid) {/ / successful execution AOF, replace the existing file AOF backgroundRewriteDoneHandler (exitcode bysignal);if(! bysignal && exitcode == 0) receiveChildInfo(); }else{// The child process succeeded, but the returned PID type is abnormal and cannot matchif(! ldbRemoveChild(pid)) { serverLog(LL_WARNING,"Warning, detected child with unmatched pid: %ld", (long)pid); }} // If the child process is not finished, the dictionary is not allowed to proceedrehashupdateDictResizePolicy(); closeChildInfoPipe(); }}else{... }}Copy the code

ServerCron is executed every 100 milliseconds (it may be different in later Versions of Redis, this article is based on 4.0). It will first determine whether the RDB or AOF child process has completed successfully, and if so, it will replace and overwrite old files. Let’s move on to the else part.

int serverCron(....) {...if(server.rdb_child_pid ! = -1 || server.aof_child_pid ! = -1 || ldbPendingChildren()) { .......... }else{// If no child process has done RDB file generation // iterate through the saveParams array to retrieve the Save configuration item in our configuration filefor(j = 0; j < server.saveparamslen; j++) { struct saveparam *sp = server.saveparams+j; // Determine whether the save configuration conditions are met based on the dirty counters we introduced earlierif(server.dirty >= sp->changes && server.unixtime-server.lastsave > sp->seconds && (server.unixtime-server.lastbgsave_try > CONFIG_BGSAVE_RETRY_DELAY | | server lastbgsave_status = = C_OK)) {/ / log serverLog (LL_NOTICE,"%d changes in %d seconds. Saving...", sp->changes, (int)sp->seconds); rdbSaveInfo rsi, *rsiptr; rsiptr = rdbPopulateSaveInfo(&rsi); RdbSaveBackground (server.rdb_filename,rsiptr);break; }} //AOFif (server.aof_state == AOF_ON &&
             server.rdb_child_pid == -1 &&
             server.aof_child_pid == -1 &&
             server.aof_rewrite_perc &&
             server.aof_current_size > server.aof_rewrite_min_size)
         {
            long long base = server.aof_rewrite_base_size ?
                            server.aof_rewrite_base_size : 1;
            long long growth = (server.aof_current_size*100/base) - 100;
            if (growth >= server.aof_rewrite_perc) {
                serverLog(LL_NOTICE,"Starting automatic rewriting of AOF on %lld%% growth",growth); rewriteAppendOnlyFileBackground(); }}}}Copy the code

If there is no child process to generate the RDB file, then loop over whether our save configuration item is satisfied, if so, call rdbSaveBackground for the actual RDB file generation. Let’s continue with this core approach:

int rdbSaveBackground(char *filename, rdbSaveInfo *rsi) {
    pid_t childpid;
    long long start;

    if(server.aof_child_pid ! = -1 || server.rdb_child_pid ! = 1)return C_ERR;

    server.dirty_before_bgsave = server.dirty;
    server.lastbgsave_try = time(NULL);
    openChildInfoPipe();

    start = ustime();
    if ((childpid = fork()) == 0) {
        int retval;
        closeListeningSockets(0);
        redisSetProcTitle("redis-rdb-bgsave");
        retval = rdbSave(filename,rsi);
        if (retval == C_OK) {
            size_t private_dirty = zmalloc_get_private_dirty(-1);

            if (private_dirty) {
                serverLog(LL_NOTICE,
                    "RDB: %zu MB of memory used by copy-on-write",
                    private_dirty/(1024*1024));
            }

            server.child_info_data.cow_size = private_dirty;
            sendChildInfo(CHILD_INFO_TYPE_RDB);
        }
        exitFromChild((retval == C_OK) ? 0:1); }else {
        server.stat_fork_time = ustime()-start;
        server.stat_fork_rate = (double) zmalloc_used_memory() * 1000000 / server.stat_fork_time / (1024*1024*1024); /* GB per second. */
        latencyAddSampleIfNeeded("fork",server.stat_fork_time/1000);
        if (childpid == -1) {
            closeChildInfoPipe();
            server.lastbgsave_status = C_ERR;
            serverLog(LL_WARNING,"Can't save in background: fork: %s",
                strerror(errno));
            return C_ERR;
        }
        serverLog(LL_NOTICE,"Background saving started by pid %d",childpid);
        server.rdb_save_time_start = time(NULL);
        server.rdb_child_pid = childpid;
        server.rdb_child_type = RDB_CHILD_TYPE_DISK;
        updateDictResizePolicy();
        return C_OK;
    }
    return C_OK; 
}
Copy the code

At the core of rdbSaveBackground are calls to the fork and rdbSave functions. Fork is a system call that copies a child with almost identical memory data as the parent.

The fork function blocks. When the child is copied, subsequent code segments are executed simultaneously by the parent and child processes. In other words, the parent and child processes execute code concurrently after the fork, but the order of execution is not guaranteed.

In the parent process, fork returns the child process ID, and in the child process fork returns zero.

RdbSaveBackground (‘ temp-%d.rdb ‘, ‘temp-%d.rdb’, ‘temp-%d.rdb’, ‘temp-%d.rdb’); Information such as time.

As for how the rdbSave function writes to the RDB file, this is also very simple, RDB files have a fixed protocol specification, the program only needs to write data according to the protocol, we will talk about it in more detail later.

To sum up, serverCron, a function that is executed periodically, reads the save configuration in the configuration file and determines whether the conditions are met. If the conditions are met, rdbSaveBackground is called to fork out a child process to complete the RDB file writing and generate temporary files. After the temporary file is successfully written, the old RDB file is replaced and the child process exits.

Ps: fork the child process must remember to exit, otherwise the main process will copy a child process each time, resulting in OOM service.

RDB file structure analysis

Any file format will have their own coding protocol, in Java bytecode, picture format file, we here RDB file, all have their own a set of agreed upon agreement, specific to each byte position should be what kind of field data, it is established, according to the agreement is written to the binary encoding, Field bytes are also read by protocol.

The RDB protocol specifies that the entire file contains the following fields:

Among them, the first part is a fixed five bytes, redis called it Magic Number, fixed five characters “R”, “E”, “D”, “I”, “S”.

We add a key-value pair to redis database 0, execute the save command to generate the RDB file, and then open the binary.

We use the OD command and output the binary with the ASCII option, you will find that the first five bytes are our fixed redis five characters.

The next field, REDIS_VERSION, is four bytes long and describes the current RDB version. For example, the RDB file version corresponding to redis-4.0 is 0008.

The next field is Aux Fields, officially called auxiliary field, which was added after RDB 7 and mainly contains the following Fields:

Redis-ver: indicates the version number
Redis – bits: OS the Arch
Ctime: time when an RDB file is created
Used-mem: indicates the used memory size
Repl-stream-db: database selected in the server.master client
Repl-id: indicates the replication ID of the current instance
Repl-offset: offset of the current instance replication

Then there is the DATABASE section, which stores the real data in our dictionary. The RDB file generated by multiple databases in Redis will only be written to the DATABASE with data, and the format of this section is as follows:

Corresponding to our example above is this part:

Our rdb.h file header has some definitions for constants:

#define RDB_OPCODE_AUX 250
#define RDB_OPCODE_RESIZEDB 251
#define RDB_OPCODE_EXPIRETIME_MS 252
#define RDB_OPCODE_EXPIRETIME 253
#define RDB_OPCODE_SELECTDB 254
#define RDB_OPCODE_EOF 255
Copy the code

RDB_OPCODE_SELECTDB specifies the number of the database to be opened. In this case, the number of the database to be opened is zero.

The hexadecimal fb is 251 in decimal, which corresponds to RDB_OPCODE_RESIZEDB, which identifies the current database capacity, i.e. how many keys there are. We only have one key here.

The next step is to save our key-value pairs in the following format:

Type is a byte that identifies the type of the current key-value pair. That is, the object type.

#define RDB_TYPE_STRING 0
#define RDB_TYPE_LIST 1
#define RDB_TYPE_SET 2
#define RDB_TYPE_ZSET 3
#define RDB_TYPE_HASH 4
#define RDB_TYPE_ZSET_2 5 
#define RDB_TYPE_MODULE 6
#define RDB_TYPE_MODULE_2 7 
/* Object types for encoded objects. */
#define RDB_TYPE_HASH_ZIPMAP 9
#define RDB_TYPE_LIST_ZIPLIST 10
#define RDB_TYPE_SET_INTSET 11
#define RDB_TYPE_ZSET_ZIPLIST 12
#define RDB_TYPE_HASH_ZIPLIST 13
#define RDB_TYPE_LIST_QUICKLIST 14
Copy the code

A key is always a string, consisting of a string length prefix followed by its own content, followed by the content of the value.

The EOF field identifies the end of the RDB file, takes up one byte, and has a fixed value equal to 255, or hexadecimal ff, which can be found in the rdb.h header.

The CHECK_SUM field stores the eight-byte checksum of the RDB file, which is used to check whether the RDB file is damaged.

Above, we briefly introduced the composition of the RDB file, in fact, it is only a point ah, each type of object encoding is not the same, but also some compression of the object techniques and so on, we can not be all exhaustive here.

In general, the RDB file composition has a basic understanding of the line, in fact, few people have nothing to analyze the DATA in the RDB file, even if there is also through the analysis of tools, such as RDB-tools, manual analysis is also too burst.

Ok, that’s all we have to say about RDB. In the next article, we’ll look at AOF as a persistence strategy. Bye!

Focus on the public not getting lost, a programmer who loves to share.
Public number reply “1024” add the author wechat together to discuss learning!
All the case code materials used in each article are uploaded to my personal Github
Github.com/SingleYam/o…
Welcome to step on it!

Data Persistence Strategy (RDB) in Redis

What is RDB persistence policy

Saveparams and dirty counters

RDB file structure analysis

Related Posts

Maven relies on mediation for source code parsing (5) : the latter overrides the former declared in the same file

Linux Software Management – Decompress installation

MySQL storage engine and index