Redis uses the bgSave command to persist data to disk. At startup time, RDB files generated by THE bgSave can be loaded from disk to recover data

The save command blocks and is not recommended

Introduction to RDB persistence mechanism

The RDB structure of Redis is roughly as follows

In the case of hashtable

REDIS|db_version|SELECTDB|0|REDIS_TYPE_HASH|hash_size|key1_len|key1_value|key1_value_len|key1_value|EOF|checksum
Copy the code
  • REDIS: Identifier placed at the beginning of a file
  • Db_version: specifies the current RDB version
  • SELECTDB: identifier. The next thing to read is the database subscript in the server
  • 0: indicates the 0th DB. The default value is 16
  • REDIS_TYPE_HASH: Saves the hashTable structure in db
  • Hash_size: The number of elements in a hashTable
  • Key1_len: number of bytes of the first key
  • Key1_value: the literal value of the first key
  • Key1_value_len: specifies the number of bytes of the value corresponding to the first key
  • Key1_value: specifies the value of the first key
  • EOF: Identifier without data
  • Checksum: checksum of the RDB file to check the integrity of the content

Call bgSave for storage

When a bgSave command is executed, Redis forks a child process so that the execution of other commands is not blocked

Code.SLICE.source("if ((childpid = fork()) == 0) {" +
"/ /..." +
" retval = rdbSave(filename,rsi);" +
" if (retval == C_OK) {" +
"/ /..." +
" server.child_info_data.cow_size = private_dirty;" +
" sendChildInfo(CHILD_INFO_TYPE_RDB);" +
"}" +
" exitFromChild((retval == C_OK) ? 0:1);" +
" } else {" +
" /* Parent */" +
"/ /..." +
" server.rdb_save_time_start = time(NULL);" +
" server.rdb_child_pid = childpid;" +
" server.rdb_child_type = RDB_CHILD_TYPE_DISK;" +
" updateDictResizePolicy();" +
" return C_OK;" +
"}")
.interpretation("Create a child process, the child process is responsible for rDB-related processing, the parent process remembers the child process ID, and returns the current bgSave execution, that is, bgSave does not block the execution of other commands.");
   
Copy the code

When storing data into the RDB, the REDIS string is first written to the file header, spelling out the current VERSION of the RDB

Code.SLICE.source("snprintf(magic,sizeof(magic),\"REDIS%04d\",RDB_VERSION);" +
" if (rdbWriteRaw(rdb,magic,9) == -1) goto werr;")
.interpretation("First write down the REDIS string and the RDB version in the file.");
Copy the code

Then go through all the databases in redis server, write data one by one, according to the different types of data, use different types to identify, and then write down the corresponding length, and then store the value, for example, the value of the object to be stored is hashTable

Code.SLICE.source("else if (o->type == OBJ_HASH) {" +
" /* Save a hash value */" +
" if (o->encoding == OBJ_ENCODING_ZIPLIST) {" +
" size_t l = ziplistBlobLen((unsigned char*)o->ptr);" +
"" +
" if ((n = rdbSaveRawString(rdb,o->ptr,l)) == -1) return -1;" +
" nwritten += n;" +
"" +
" } else if (o->encoding == OBJ_ENCODING_HT) {" +
" dictIterator *di = dictGetIterator(o->ptr);" +
" dictEntry *de;" +
"" +
" if ((n = rdbSaveLen(rdb,dictSize((dict*)o->ptr))) == -1) {" +
" dictReleaseIterator(di);" +
" return -1;" +
"}" +
" nwritten += n;" +
"" +
" while((de = dictNext(di)) ! = NULL) {" +
" sds field = dictGetKey(de);" +
" sds value = dictGetVal(de);" +
"" +
" if ((n = rdbSaveRawString(rdb,(unsigned char*)field," +
" sdslen(field))) == -1)" +
"{" +
" dictReleaseIterator(di);" +
" return -1;" +
"}" +
" nwritten += n;" +
" if ((n = rdbSaveRawString(rdb,(unsigned char*)value," +
" sdslen(value))) == -1)" +
"{" +
" dictReleaseIterator(di);" +
" return -1;" +
"}" +
" nwritten += n;" +
"}" +
" dictReleaseIterator(di);" +
" } else {" +
" serverPanic(\"Unknown hash encoding\");" +
"}" +
"}")
.interpretation("Using hash encoding as an example, look at the underlying implementation.")
.interpretation("1: If the underlying hash implementation is ziplist, get the length of the ziplist and save the ziplist as a string.")
.interpretation("2: The underlying implementation of hash is hasttable, so iterate over keys and values one by one, convert them into strings and store them again");
Copy the code

When all the data records are complete, the EOF end marker is written and the checksum is added to complete the in-memory data serialization and storage to disk

Code.SLICE.source("if (rdbSaveType(rdb,RDB_OPCODE_EOF) == -1) goto werr;")
        .interpretation("Write the EOF flag to indicate that all db data has been written.");
Code.SLICE.source("cksum = rdb->cksum;" +
        " memrev64ifbe(&cksum);" +
        " if (rioWrite(rdb,&cksum,8) == 0) goto werr;")
        .interpretation("Write checksum, complete memory data written.");
Copy the code

Start loading

It is loaded during the startup of Redis, which is essentially stored deserialization process, starting with reading the string Redis

 Code.SLICE.source("if (rioRead(rdb,buf,9) == 0) goto eoferr;" +
    " buf[9] = '\\0';" +
    " if (memcmp(buf,\"REDIS\",5) ! = 0)")
    .interpretation("Read first 9 bytes of file, first 5 must be REDIS characters, otherwise error");
Copy the code

Deserialization can then follow the rules of serialization until the read is complete

Code.SLICE.source("while(1) {..." +
"if ((type = rdbLoadType(rdb)) == -1) goto eoferr;" +
"..." +
" else if (type == RDB_OPCODE_EOF) {" +
" /* EOF: End of file, exit the main loop. */" +
" break;" +
"..." +
"else if (type == RDB_OPCODE_RESIZEDB){... }" +
"..." +
"if ((key = rdbLoadStringObject(rdb)) == NULL) goto eoferr;" +
"if ((val = rdbLoadObject(type,rdb)) == NULL) goto eoferr;" +
"}")
.interpretation("Loop through the contents of the file, reading the next type first.")
.interpretation("1: End of EOF")
.interpretation("2: after reading the corresponding flag, continue reading the following bytes until the key is read.")
.interpretation("3: read key, read val");
Copy the code

Value The following uses hashTable as an example to construct the corresponding structure

  Code.SLICE.source("else if (rdbtype == RDB_TYPE_HASH) {" +
    " len = rdbLoadLen(rdb, NULL);" +
    "..." +
    " o = createHashObject();" +
    "/ *... * /" +
    " while (o->encoding == OBJ_ENCODING_ZIPLIST && len > 0) {" +
    " len--;" +
    " /* Load raw strings */" +
    " if ((field = rdbGenericLoadStringObject(rdb,RDB_LOAD_SDS,NULL))" +
    " == NULL) return NULL;" +
    " if ((value = rdbGenericLoadStringObject(rdb,RDB_LOAD_SDS,NULL))" +
    " == NULL) return NULL;" +
    "" +
    " /* Add pair to ziplist */" +
    " o->ptr = ziplistPush(o->ptr, (unsigned char*)field," +
    " sdslen(field), ZIPLIST_TAIL);" +
    " o->ptr = ziplistPush(o->ptr, (unsigned char*)value," +
    " sdslen(value), ZIPLIST_TAIL);" +
    "" +
    " /* Convert to hash table if size threshold is exceeded */" +
    " if (sdslen(field) > server.hash_max_ziplist_value ||" +
    " sdslen(value) > server.hash_max_ziplist_value)" +
    "{" +
    " sdsfree(field);" +
    " sdsfree(value);" +
    " hashTypeConvert(o, OBJ_ENCODING_HT);" +
    " break;" +
    "}" +
    " sdsfree(field);" +
    " sdsfree(value);" +
    "}" +
    "..."+
    " /* Load remaining fields and values into the hash table */" +
    " while (o->encoding == OBJ_ENCODING_HT && len > 0) {" +
    " len--;" +
    " /* Load encoded strings */" +
    " if ((field = rdbGenericLoadStringObject(rdb,RDB_LOAD_SDS,NULL))" +
    " == NULL) return NULL;" +
    " if ((value = rdbGenericLoadStringObject(rdb,RDB_LOAD_SDS,NULL))" +
    " == NULL) return NULL;" +
    "" +
    " /* Add pair to hash table */" +
    " ret = dictAdd((dict*)o->ptr, field, value);" +
    " if (ret == DICT_ERR) {" +
    " rdbExitReportCorruptRDB(\"Duplicate keys detected\");" +
    "}" +
    "}" +
    "}")
    .interpretation("Take hashtable as an example, read the corresponding length of data, create objects, and parse them into Ziplist or HashTable for storage according to the encoding mode of objects.");
 
Copy the code

conclusion

  1. Bgsave does not block other redis commands.
  2. RDB serializes in-memory objects by setting the type representation of the data, then the amount of data, then the length of the data value, and then the data itself
  3. To start parsing loaded RDB files is to deserialize them according to established save rules

Advantages and disadvantages of RDB

  • Advantages: RDB is a compact binary file, suitable for backup, full copy scenarios; It recovers much faster than AOF
  • Disadvantages: Not suitable for real-time persistence, high cost of real-time operation; The old Redis service is not compatible with the RDB files generated by the new Redis

The appendix

RDB start loading source bgSave source code books: Redis design and implementation, Redis development and operation of AOF mechanism introduction