Redis has two types of persistence:

  • Redis Database (RDB) : Stores data in memory as a snapshot to hard disks
  • AOF (Append Only File) : Every time the command is executed, it will try to trigger the disk flushing operation. It has good real-time performance, less data loss, but performance loss

RDB

concept

Save a copy of redis memory data to binary file

triggered

  • Active Save: The entire Redis Server blocks until complete
  • Bgsave: Snapshot backup is done through an asynchronous process
  • Automatic trigger: Set the trigger time according to the rules, n times of modification within m seconds (Save M n)

Bgsave process

  1. Fork out a child process
  2. The child process shares memory with the main process (copy on write), and the child process reads the data directly and writes it to the binary file
  3. Replace the original RDB

Copy-on-write (COW)

The parent and child processes share a piece of memory data. When the parent process modifies the data, the child process copies the data to its own physical space

In this way, you can avoid large copies of data and reduce memory consumption

conclusion

advantages

  • Simple mechanism: The snapshot mechanism is simple
  • Low write loss: Unlike AOF, there is no need for additional writes to aOF_buf after each write
  • Small file: Compressed binary files occupy small space during data backup
  • Fast recovery: Based on binary direct recovery

disadvantages

  • Data loss: Data is lost when the backup interval breaks down
  • Tight performance during backup: Although there is no extra performance overhead during normal write, there is a large amount of I/O during backup, which consumes performance

AOF

concept

  1. Redis appends data to aOF_buf each time it executes a write command
  2. Flush data from aOF_buf to disk according to the flush time set by appendfysNC

Time to flush the disk (appendfysNC)

  • Always: The disk will fall every time, with poor performance, but no data will be lost
  • “No” : Do not actively flush disks. The performance is best, but the risk of data loss is high
  • Everysec: Timed brush once per second, compromise solution

File to rewrite

Aof files need to be rewritten periodically to keep them from ballooning indefinitely, reducing recovery time in the event of a failure

By directly reading Redis and constructing AOF files, storage space can be effectively reduced, mainly in the following aspects:

  1. Skip expired, invalid keys
  2. Condense several commands into one
  3. Effectively reduce repeated commands

The overwritten pseudocode looks like this:

F :=createFile(path) #for db rangeRedis.db (): # traversal keyfor key rangeDb: # skip the expired keyif key.isExpired():
            continue# set the timeoutif key.hasExpireTime():
            f.write(expiredCmd(key))
f.close(a)Copy the code

The compression scheme is mainly about a number of repeated and scattered commands for the same key, compressed into one command. For example:

Lpush queue r1

Lpush queue r2

Lpush queue r2

It can be compressed into one strip

Lpush queue r1 r2

Background rewrite of the child process

Because it is necessary to traverse the redis on the line, direct synchronization will block the main process, thus denying access, equivalent to stopping the copy data (but do not worry about the rewrite process, because the new write is blocked).

To solve this problem, subprocess-based background rewriting was born: the subprocess copies data by using a copy of the main process’s data

In this case, the main process is still in service and new writes are inevitable. To ensure consistency, a new buffer will be created during file rewriting to store changes during the rewriting process, as shown in the following figure:

After the rewrite is complete, the finishing touches are done:

  1. Write the data from the aOF file overwritten buffer to the aOF file just overwritten
  2. Replace the existing AOF file with the rewritten AOF file

Note that these two operations block the main process to ensure data consistency

restore

Relatively simple, is to create a local pseudo client, read aOF file, one time to write server redis

conclusion

advantages

  • Good fault tolerance: After a failed command is skipped, recovery can be continued
  • Easy to read: Each command is a write statement, readable
  • Support background rewrite: avoid infinite file bloat

disadvantages

  • Slow recovery: Statements are inserted one by one, which takes time to execute
  • Large space: it is a complete command, and of course it uses more space

reference

Redis persistence (RDB and AOF)

Differences between RDF and AOF

AOF rewriting of Redis and its implementation principle

Linux IO synchronization: sync, fsync, and fdatasync

Copy-on-write Techniques (Detailed version)

Linux copy-on-write