This is the sixth day of my participation in the August More text Challenge. For details, see: August More Text Challenge

The introduction

Since Redis is a in-memory database, the data state is in memory, so once the service restarts, all the data in memory will be lost. To solve this problem, Redis provides two persistence schemes, RDB and Aof.

RDB

Persistence of RDB can be performed either manually or periodically depending on the configuration. Either manually or automatically, data is stored in an RDB file at a certain time. The RDB file itself is a binary compressed file that can be used to recover data when Redis starts.

In this way, even if Redis fails, the data is not lost because dump. RDB is saved on the hard disk. As mentioned earlier, there are currently two ways to generate RDB files, one is SAVE and the other is BASAVE.

SAVE

SAVE is manually saved. It blocks the Redis process until the RDB file is created. During the creation, all commands cannot be processed.

127.0.0.1:6379> save
OK
27004:M 31 Jul 15:06:11.761 * DB saved on disk
Copy the code

BGSAVE

Unlike the SAVE command, the BGSAVE command does not block the Redis process. With BGSAVE, the redis process forks a child process to SAVE the RDB, and the main process continues to execute the command.

127.0.0.1:6379> BGSAVE
Background saving started
27004:M 31 Jul 15:07:08.665 * Background saving terminated with success
Copy the code

BGSAVE is mutually exclusive with other IO commands during execution:

  1. During BGSAVE, all SAVE commands are rejected to prevent the parent and child processes from being executed at the same time, which may cause some race problems.
  2. During BGSAVE, if there is a new BGSAVE, it will be rejected, which is also a competition problem.
  3. If a BGREWRITEAOF comes during BGSAVE, BGREWRITEAOF will be delayed until after BGSAVE.
  4. If BGREWRITEAOF is running, then the BGSAVE command will be rejected.

Both BGSAVE and BGREWRITEAOF are handled by two sub-processes that target different files. There is no conflict between them, except that both may require a lot of IO, which is not very friendly to the service itself.

The realization of the BGSAVE

You can configure bgSave to be performed at regular intervals

save 900 1 
save 300 10
save 60 10000
Copy the code
  1. The password is modified at least once within the 900s
  2. At least 10 changes have been made within 300 seconds
  3. The value is modified at least 10000 times within 60s

Bgsave can be performed as long as one of the above conditions is met. There are two parameters involved to keep track of times and times. Dirty counter and LastSave.

  • The DIRTY counter records how many changes (writes, deletes, updates, and so on) the server has made to the state of the database (all databases in the server) since the last successful execution of the SAVE command or BGSAVE command.
  • The lastsave property is a UNIX timestamp that records the last time the server successfully executed a SAVE command or a BGSAVE command.

The above two indicators are based on Redis serverCron, which is a program that is executed periodically, every 100ms by default. Each time serverCron executes, it iterates through all the conditions, checks if the count is ok, if the time is OK, then executes a BGsave, records the latest lastSave time, and rewrites the dirty to 0.

RDB file structure

Redis files are stored in binary format and roughly store the following data:

  1. REDIS is a fixed beginning
  2. Db_version specifies the RDB version.
  3. A database is the data we store
  4. The terminator. When you read the terminator when you load it, you’re done
  5. Check_sum Indicates the checksum to check whether the RDB file is corrupted

Database is our key concern:

Since keys are classified as expired and non-expired, the structure is also different. More than one expiration time is required.

The import

Unlike the two save commands, redis does not have a special user import command. Redis checks for RDB files at startup and automatically imports them if they exist.

27004:M 31 Jul 14:46:51.793 # Server started, Redis version 3.2.12
27004:M 31 Jul 14:46:51.793 * DB loaded from disk: 0.000 seconds
27004:M 31 Jul 14:46:51.793 * The server is now ready to accept connections on port 6379
Copy the code

RDB loaded from disk The service is blocked during RDB loading. Of course, if AOF is also enabled, then AOF is preferred for recovery, and RDB is only selected for recovery if AOF is not enabled. Expired keys are also automatically filtered during import.

AOF

The principle of

Unlike RDB persistence, Redis also provides AOF persistence, which is a mode of appending commands. Suppose that:

RPUSH list 1 2 3 4
RPOP list
LPOP list
LPUSH list 1
Copy the code

*2$6SELECT$10*6$5RPUSH$4list$11$12$13$14*2$4RPOP$4list*2$4LPOP$4list*3$5LPUSH$4list$11

strategy

Aof is first written to aOF_buf buffer. Redis provides three options for flushing buF buffer data to disk, which serverCron handles according to the policy.

appendfsync always
appendfsync everysec
appendfsync no
Copy the code
  1. Always: Writes and synchronizes all the contents of the AOF_buf buffer to the AOF file
  2. Everysec: writes all the contents of the AOF_buf buffer to the AOF file. If the time of the last synchronization is longer than 1s, the synchronization is performed again, and the synchronization is done by one thread.
  3. No: the aof_buf file is written to the AOF file but is not synchronized. The synchronization time is determined by the operating system.

This explains writing AOF files and synchronizing AOF.

In modern operating systems, to improve the efficiency of file writing, when we call write to write a piece of data, the operating system does not immediately write the data to the disk, but to a buffer, when the buffer is full or a certain amount of time, the actual flush to the disk. There is a risk that the data in memory will be lost if the machine goes down before it can be flushed to disk, so the operating system provides a synchronization function called fsync that allows users to decide when to synchronize.

  1. Always means every time serverCron executes, fsync is done immediately, which is the least efficient. The advantage is that if the system goes down, the data can be lost for a maximum of one cycle (100ms).
  2. No is definitely the most efficient because it doesn’t fsync, it waits for the OPERATING system to synchronize itself, so if the service goes down, the most data will be lost
  3. Everysec is a compromise, which ensures both high efficiency and low data loss (up to 1s of data loss).

AOF rewrite

As you add more commands, the size of the Aof gets bigger, for example

incr num
incr num
incr num
incr num
Copy the code

Num = 4; set num = 4; set num = 4;

Rewriting is also not an analysis of existing AOF

Rewriting is reading the existing key from the database and replacing it with a command if possible.

  1. Create a new Aof
  2. Traversing the database
  3. Iterate over all keys
  4. Ignore expired
  5. Write aof

Not all of them can be replaced with a single command: for example, sadd can only add up to 64 items at a time, and if more than 64 items are added in batches.

sadd key 1 2 ... 64 sadd key 64 66 ... .Copy the code

Child override

The rewrite of Aof involves a lot of IO, which is not appropriate to do in the current process, and of course is to fork a child process to do it. The reason for not doing this is to avoid some locking problems. Use child process issue to consider is when the child to write, main process is still in a steady stream of receiving new requests, so for this kind of circumstance redis set a aof rewrite buffer, the buffer in the child began to use when creating, so when a new request to besides writing aof buffer, The Aof override buffer is also written, and this process does not block. After the child process finishes rewriting, it will send a signal to the main process. After receiving the signal, the main process will resynchronize the data from the rewrite buffer to the new AOF file, and then rename the new AOF, and the atom overwrites the old AOF file to complete the rewrite. This process is blocked.

Rewrite the timing

  1. Performed manuallybgrewriteaof
  2. Through the configuration, automatically trigger:
Auto-aof -rewrite-percentage 100 //100 indicates that the current aof file is rewritten only when the current aof file is twice the size of the last rewrite. Auto-aof -rewrite-min-size 64mb // the default minimum rewrite size is 64mbCopy the code

The import

When redis starts, it creates a dummy client and executes the commands in the aof file.

conclusion

  1. There are currently two persistence schemes for Redis: RDB and Aof
  2. The RDB stores data in binary format
  3. Aof stores the commands that are executed
  4. Aof allows for less data loss in the event of an accident
  5. When redis is started, if aof is enabled, the RDB is recovered first
  6. Generally, both are enabled in the production environment

Refer to The Design and Implementation of Redis