1, RDB (Redis DataBase)

1.1 concept

At a specified interval, a Snapshot is written to a disk. The Snapshot file is directly read into the memory during recovery. This function is enabled by default.

1.2 How is Backup Performed

Redis forks a separate subprocess for persistence, writing data to a temporary file that will be used to replace the last persistent file after the persistence process has finished. During the entire process, the main process does not perform any IO operations, which ensures extremely high performance. If large-scale data recovery is required and data integrity is not sensitive, the RDB method is more efficient than the AOF method, but may cause data loss after the last persistence.

1.3 the Fork
  • Fork copies the same process as the current one. All data values of the new process (variables, environment variables, program counters, etc.) are the same as those of the original process, but the new process is a child of the original process.
  • Fork () produces a child that is exactly the same as the parent, but will later exec the system call. Linux introduced copy-on-write techniques for efficiency.
  • Generally, the parent process and child process share the same physical memory. Only when the contents of each segment of the process space need to be changed, the parent process copies the contents to the child process.
1.4 RDB Persistence Process
  1. When the bgsave command is executed, the parent Redis process determines whether there are currently executing child processes and returns if there are.
  2. The parent process forks a child process (which blocks during the fork) using the info STATS command to view the latest_fork_usec option to see how long the last fork took, in microseconds.
  3. After the parent forks, the Background saving started message is returned and the fork block is cleared.
  4. Fork the child process starts to generate a temporary snapshot file from the parent process’s memory data and then replaces the original file. Use the lastsave command to see when the RDB was last generated, for the rdb_last_save_time option of info.
  5. Send a completion message to the parent when the backup is complete, as shown in the rdb_* option under Info Persistence.

1.5 RDB-related Configurations
Dump. RDB dbfilename Dump. RDB # dump. / # Default trigger snapshot configuration # 3600 seconds at least one key sends changes save 3600 1 save 300 10 save 60 10000 # When Redis cannot write to disk, Disable Redis write operations directly. Yes is recommended. Stop-writes-on-bgsave-error yes # You can set whether to compress snapshots stored in disk. If the value is yes, Redis uses the LSF algorithm for compression. If you don't want to use CPU for compression, you can turn this feature off. Yes. Rdbcompression yes # After storing snapshots, redis can also use the CRC64 algorithm for data validation. However, this adds about 10% of the performance cost, and you can turn this feature off if you want the maximum performance gain. Yes. rdbchecksum yesCopy the code
1.6 Advantages VS Disadvantages

1.6.1 advantage

  • An RDB is a snapshot taken at a point in time, a compact single file, and is more used for data backup. Can be backed up by hour or media, easy to recover from different versions of the time.
  • Single files are easily transferred to remote services for failover.
  • The compressed binary file is used to load data files when the service is restarted, which is faster than the AOF mode.

1.6.2 disadvantage

  • RDB stores files in encrypted binary format. Due to compatibility issues between Redis versions, RDB files may not be used in other versions.
  • Redis uses copy-on-write technology when forking, but it consumes a lot of performance when the data is large.
  • If the Redis goes down unexpectedly, all changes made after the last snapshot will be lost.
1.7 other

1.7.1 the save VS bgsave

  • Save: Save only, other commands are blocked. Manual saving is not recommended.
  • Bgsave: Redis takes snapshots asynchronously in the background and responds to client requests.

1.7.2 Dynamically Stopping RDB Persistence

redis-cli config set save " "
Copy the code

Similarly, other dynamic config:

redis-cli config set XX
Copy the code

2, AOF (Append Only File)

2.1 concept

Each write operation is recorded in the form of a log (incremental save), all write instructions performed by Redis are recorded (read operations are not recorded), only files can be appended but files can not be overwritten, Redis will initially read the file to rebuild the data, in other words, When redis is restarted, write instructions are executed from front to back according to the contents of the log file to complete data recovery. This function is disabled by default.

2.2 AOF Persistence Process
  1. The client request write command is appended to the AOF buffer by append.
  2. The AOF buffer synchronizes the operation sync to the AOF file on disk according to the AOF persistence policy [always, Everysec, no].
  3. When the AOF file size exceeds the rewrite policy or manual rewriting, the AOF file will be rewritten to reduce the AOF file size.
  4. When the Redis service restarts, it loads write operations in AOF files to restore data.

2.3 Setting of AOF synchronization frequency
  • Appendfsync always: Always synchronize, every Redis write is logged immediately; Poor performance but good data integrity.
  • Appendfsync everysec (default) : Synchronization is performed once per second and logs are logged once per second. Data in this second may be lost if the system goes down.
  • Appendfsync no: Redis does not initiate synchronization and assigns the synchronization time to the operating system.
2.4 Rewrite the compression

Against 2.4.1 concept

In order to avoid this situation, a new rewriting mechanism is added. When the size of AOF files exceeds the set threshold, Redis will start the content compression of AOF files and only retain the minimum instruction set that can restore data. You can use the command bgrewriteaof.

2.4.2 principle

If an AOF file grows too large, a new process will be forked to rename the file (also write temporary files first and then rename the file). Rewrite after redis4.0 means that a snapshot of the RDB is attached to the new AOF header in binary form as the existing historical data, replacing the original ledger operation.

  • No – appendfsync – on – rewrite:
  • If no-appendfsync-on-rewrite=yes, the AOF file is only written to the cache, the user request will not block, but the cache data will be lost during this period (reducing data security and improving performance).
  • If no-appendfsync-on-rewrite=no, it would still flush data to disk, but might block when overwritten (data safety, but performance degradation).

2.4.3 Triggering Mechanism

Redis will record the size of the AOF when it was last rewritten, triggered by default when the AOF file size lies to you twice the size of the rewritten file and the file is larger than 64M.

Overwriting can save a lot of disk space and reduce recovery time. However, each rewrite will have a certain burden, so the Redis will be rewritten only when certain conditions are met.

Auto-aof-rewrite-percentage: specifies the percentage of rewrites. The default value is “rewrites” when the number of rewrites reaches 100% (the number of rewrites is twice that of the original file).

Auto-aof -rewrite-min-size: specifies the reference value for rewriting. Default is 64MB. If both configurations are correct, the system starts to rewrite.

If the current AOF size of Redis >= base_size + base_size * 100% (default), if the current AOF size >= 64MB (default), Redis will rewrite AOF.

2.4.4 Rewrite the process

  1. Bgrewriteaof triggers rewriting and checks whether BGSave or BgreWriteAOF is running. If yes, the command is executed after the command is executed.
  2. The main process forks a child process to perform the rewrite operation to ensure that the main process does not block.
  3. The child process traverses the data in Redis memory to the temporary file, and the client’s write request simultaneously writes the AOF_BUF cache and the AOF_rewrite_buF rewrite cache to ensure that the original AOF file is intact and the new data modification action during the generation of the new AOF file is not lost.

4.1 After the child process writes a new AOF file, it sends a signal to the main process, and the parent process updates the statistics. 4.2 The main process writes data in aOF_rewrite_buf to the new AOF file. 5. Use the new AOF file to overwrite the old AOF file to complete the AOF rewrite.

2.5 Advantages vs. Disadvantages

2.5.1 advantage

  • More robust backup mechanism, low data loss probability.
  • Readable, because the data is stored in text protocol format, you can view commands directly.

2.5.2 disadvantage

  • It takes up more disk space than RDB.
  • The backup recovery speed is slower.
  • AOF is less efficient than RDB during heavy writes and loads.

3. Official advice

  • RDB persistence allows you to take snapshots of your data at specified intervals.
  • The AOF persistence mode records each write operation to the server. When the server is restarted, these commands will be executed again to restore the original data. The AOF command saves each write operation to the end of the file using the Redis protocol.
  • Cache only: If you only want your data to live while the server is running, don’t use any persistence.
  • You are advised to enable both persistence modes. In both cases, when Redis restarts, the AOF file will be loaded first to restore the original data. In general, the data saved by AOF file is more complete than that saved by RDB file.
  • You are not advised to enable AOF only. Because RDB is better for backup databases (AOF is constantly changing) and there are no potential AOF bugs.
  • Since RDB files are only used for backup purposes, it is recommended to persist RDB files only on the slave and backup them only once every 15 minutes. Only save 9001 is retained.
  • If you use AOF, the benefit is that you lose less than two seconds of data in the worst case, and you can just start with a simpler load AOF file. The costs are: first, continuous IO, and then AOF rewrite, which inevitably blocks new data into new files. So as long as hard drives are licensed, AOF rewrite should be minimized.