An important reason why Redis is so fast is that Redis is a high-performance key-value in-memory database. All data is stored in memory. To ensure that the memory data is not lost after downtime or restart, Redis provides a persistence mechanism to save the database state in memory to disk. And it can be quickly recovered at startup time.

RDB persistence mechanism

RDB is a function provided by Redis to save a snapshot of the in-memory database status, preventing data loss in memory. The RDB file generated by RDB persistence is a compressed binary file that can be used to restore the memory state of Redis when it is started to save the RDB file.

Because RDB files are stored in binary files and are saved every once in a while, the size and execution speed of RDB files are relatively fast. When Redis restarts, RDB files can be used to quickly restore data in memory. However, the disadvantage of RDB files is that data may not be complete. An RDB snapshot is a full backup of data at a point in time

Write to RDB file

RDB files can be generated using either the SAVE or BGSAVE commands

  • SAVE Creates an RDB file

    The SAVE command blocks the Redis server process until the RDB file is created, and the server cannot process any requests while the server process is blocked

  • BGSAVE creates the RDB file

    The BGSAVE command forks a child process that generates the RDB file, and the main Redis process can continue receiving client requests.

    BGSAVE’s main process blocks only once during the fork phase, which is short compared to Save

    • COW mechanism

      When the child process creates an RDB file, it writes all the data in the file to the file. If the main process updates the data in memory while processing the client request, the child process and the parent process share the memory data, causing data problems.

      Therefore, when modifying shared resources, copy on Write (COW) copies the shared resources, locks the resources, and points the reference of the original container to the new container. In this way, the memory for operations between processes is not the same and data problems are not generated.

      The disadvantage of COW is that COW copies the memory. In this case, the data in the memory has two copies of the same memory, which is equivalent to half of the memory. Sufficient memory is required to implement COW.

    • COW mechanism of Redis

      1. After creating the child process, Redis does not copy data at all. The main process shares data with the child thread. The main process continues to provide read and write services and the child process performs write RDB services
      2. Although data is not copied, the kernel sets the permissions of all memory pages in the main process to read-only. Pointers to data accessed by the main and child processes point to the same memory address
      3. A page-fault is triggered when a write operation is performed on the main process because the permission is already read-only. In interrupt processing, a copy of the memory page that needs to be written is made, and the copied old data is given to the child process, which then writes to the original data page.

      Data is stored in pages. Therefore, when the main process operates data, the page of data to be operated can be independently copied. On the premise of data correctness, the waste of memory can be reduced

      The COW mechanism of Redis has also been optimized to prevent memory waste and prevent large memory data from having enough memory to copy.

  • Automatic save policy

    The automatic RDB file generation policy can be configured in the Redis configuration file:

    save 900 1
save 300 10
save 60 10000
Copy the code

    Description:

    • BGSAVE is executed when there is one change within 900s

    • BGSAVE is executed when there are 10 modifications within 300 seconds

    • BGSAVE is executed when there are 10000 modifications within 60 seconds

AOF persistence mechanism

The AOF persistence mechanism is different from the RDB persistence mechanism. The RDB periodically saves the snapshots in memory to disk, while the AOF file saves the write commands executed by Redis every time and saves the database status by recording the write commands.

AOF file writes

  • AOF_BUF buffer

    Modern operating systems, when the write function will be written to the input file, the operating system will usually write data temporarily saved to a memory buffer, wait until the buffer space fill up or forced to call a function to refresh buffer, the operating system will buffer the data written to disk, because every time written to disk operation is slower.

    After executing a command, the Redis server appends the command to the end of the aOF_buf buffer, and then determines whether the aOF_buf buffer needs to be flushed to disk (AOF file). You can configure whether the command needs to be flushed to disk each time you run the command or whether the command can be flushed to disk at the same time after several times. You can choose an appropriate solution in different scenarios to ensure data security and efficiency.

  • AOF persistence mechanism

    The aOF_buf refresh behavior is controlled through the appendfsync configuration, which can be configured from three options

    • always

      The server needs to flush the aOF_buf buffer to the AOF file every time it executes a command, so always is the slowest, but the safest, losing at most one command

    • everysec

      The server flusher the aOF_buf buffer to the AOF file every second, which synchronizes the file buffer to the file

    • no

      Each time a write command is executed, data in the AOF_buf buffer is not actively flushed to the AOF file, and the operating system decides when to flush.

    Always is the most secure policy, but frequent flushing to disk will affect the performance of Redis. Everysec is a relatively safe and efficient policy, and the data can be lost at most 1s. Most scenarios can be received, and NO can be used without requiring data reliability.

AOF file loading and restore

In the AOF file, all write commands are saved, so when restoring the database state of Redis, you only need to execute all commands in the AOF file again.

However, Redis creates a dummy client before executing the commands in the AOF file, because Redis commands can only be executed in the client context, reading one command at a time and iterating until the command is executed.

When Redis restarts to recover memory data, it will preferentially use AOF file to recover data, because AOF file data is relatively complete. However, the AOF file records all write commands, so if the time is long, the volume of the AOF file is very large, and when the database is restarted to restore the state, it will be slow, resulting in Redis cannot provide services quickly.

AOF rewrite mechanism

Because of the large size of AOF files, Redis server database state recovery and disk file storage, Redis provides a rewriting mechanism to solve the problem of large AOF files.

The implementation of AOF rewrite in Redis is somewhat similar to that of RDB, because it requires a lot of operations and may block the main process from processing the request, so it adopts the way of background rewrite.

In simple terms, AOF rewrite is to write key values from the current database into a new AOF file and then overwrite the previous AOF file.

  • AOF rewrite step

    • A child process is created to override the AOF, and the master continues processing the command request

    • The child from the value of each key now read from the database, then multiple commands into a command, use a command multiple orders prior to the completion of the operation, and by some data has expired or has been deleted, AOF this part has not need to record data in the file, the same, command also no longer need to record the AOF file.

      For example, multiple SADD commands can be combined into one SADD key1 value1 key2 value2… Command. Only one command needs to be recorded in the AOF file

    • During AOF rewrite, if there is a command that needs to change the current database data, Redis records the current operation instruction through an AOF rewrite buffer

      So during AOF rewrite, if the client executes a write command, the command is first executed, then written to the AOF buffer, mainly to synchronize the command to the existing AOF file, and then written to the AOF rewrite cache to synchronize data to the new AOF file

    • After the child process completes the AOF rewrite, it will notify the main process that the rewrite is complete, and the main process will write the data of the AOF rewrite buffer to the new AOF file. At this time, the state of the new AOF file is consistent with that of the database.

    • For the new AOF file with the same name, atomic overwrite the current AOF file, complete the replacement of the old and new AOF files, subsequent command synchronization will be written to the new AOF file

    The command to rewrite AOF is BGREWRITEAOF

  • Redis 4.0’s AOF persistence mechanism

    AOF no longer records all write commands, but incremental write commands. When RDB starts persistence, AOF starts incremental write commands for the current write operation. After RDB persistence is complete, no write commands are recorded. Only incremental logs are recorded between the start of RDB persistence and the end of persistence, which is relatively small.

    During Redis recovery, data in memory can be quickly recovered to ensure efficiency and data security.

The hybrid persistence mechanism of Redis4.0

In Redis 3.x, the persistence mechanism can only choose between AOF and RDB. Redis 4.0 does not choose between RDB persistence and AOF persistence.

Enable mixed persistence

In version 4.0, mixed persistence is disabled by default. You can use the aof-use-rdb-preamble parameter to control mixed persistence. Yes indicates that mixed persistence is enabled, and no indicates that mixed persistence is disabled.

Mixed persistence

Mixed persistence is also implemented through bgreWriteaof. The difference is that when mixed persistence is enabled, the child process forked out first writes the full copy of the shared memory to the AOF file in RDB mode, and then writes the increment command of the rewrite buffer to the file in AOF mode. After writing, the main process is notified to update the statistics. Replace old AOF files with new AOF files in RDB format and AOF format. To put it simply: the first half of the new AOF file is full data in RDB format and the second half is incremental data in AOF format, as shown below:

Comparison of two persistence mechanisms

You can see the pros and cons of both persistence mechanisms:

  • Only RDB snapshots are used to restore data. Because the snapshot time granularity is large, a large amount of data will be lost.
  • Using only AOF replay to recover data, log performance is slower than RDB. In the case of large Redis instances, it takes a long time to start up.
Hybrid persistence mechanisms

Store the contents of the RDB file with the incremental AOF log file. Here the AOF log is no longer the full log, but rather the incremental AOF log that occurs between the beginning of persistence and the end of persistence, which is usually small. Is equivalent to:

  • A large amount of data is captured using coarse-grained (time) RDB snapshots, providing high performance and fast recovery time.
  • Incremental data uses fine-grained (temporal) AOF logging to ensure that data is not lost.

When Redis is restarted, the RDB can be loaded first, and then the incremental AOF log can be replayed completely instead of the previous full AOF file replay, which greatly improves the restart efficiency.

Of course, some of the problems are solved, but the hybrid persistence of Redis4.0 also has corresponding disadvantages: poor compatibility, aOF files are mixed storage RDB and AOF format, in earlier versions of the log format is not recognized files

summary

  • The AOF persistence mechanism records the state of the database by saving write commands
  • The RDB persistence mechanism saves snapshots of the database at regular intervals
  • There are different options for AOF persistence. The default value is once per second. Instead of saving a write command directly to an AOF file, the write command is first saved to the AOF buffer, and then the data in the AOF buffer is flushed to a disk file (AOF file).
  • Redis provides AOF rewrite mechanism to solve the problem of large AOF files. During AOF rewrite, there will be AOF rewrite buffer to ensure data consistency during rewrite
  • The essence of AOF rewrite is to read the current database key-value pairs to achieve command merge
  • Redis 4.0 hybrid persistence mechanism, better ensure Redis memory database recovery efficiency, data security and load efficiency to ensure balance.

Wechat public number fingertips on the code, welcome to pay attention to ~ learn together and progress together

Original is not easy, click a praise and then go ~ welcome to pay attention to, to bring you more wonderful articles!

Your likes and attention are the biggest motivation for writing articles