AOF persistence

AOF persistence means that redis operation logs are appended to disk files. AOF persistence records every write and delete operation processed by the server in the form of logs. The query operation is not recorded, but is recorded in the form of text. You can open the file to see detailed operation records.

AOF implementation

  • AOF(Append Only File) persistence records each write command in an independent log, and then executes the commands in the AOF file to recover data upon restart.
  • The main role of AOF is to solve the real-time of data persistence, which has been the mainstream way of Redis persistence.

AOF advantage

  • This mechanism can lead to higher data security, that is, data persistence.

  • Redis provides three synchronization strategies: synchronization per second, synchronization per modification, and unsynchronization.

    • Synchronization per second: In fact, synchronization per second is also done asynchronously, which is very efficient, but if the system goes down, the data modified within a second will be lost.

    • Every change: Whereas every change synchronization, we can think of it as synchronous persistence, that is, every data change that occurs is recorded to disk immediately.

    • Out of sync: Predictably, this approach is the least efficient. As for asynchrony, needless to say, I think everyone understands it correctly.

  • Because the mechanism writes log files in append mode, the existing content in log files will not be damaged even if the log file is down.

  • If the system crashes after only half of the data is written in this operation, don’t worry. Before the next Redis startup, we can use the Redis-check-aOF tool to help solve the data consistency problem.

  • If the log is too large, Redis can automatically enable the rewrite mechanism, compressing and slimming the associated AOF files.

    • Redis continuously writes modification data to the old disk file in Append mode, and creates a temporary new file to record which modification commands were executed in the meantime.

    • Therefore, data security can be better ensured when switching over rewrite.

AOF includes a well-formatted, easy-to-understand log file for recording all changes. In fact, we can also use this file to complete data reconstruction.

The AOF command writes

The AOF command writes directly to the text protocol format. AOF files are plain text files whose contents are the contents of the original communication protocol sent by the Redis client to Redis.

For example, the command set hello world appends the following text to the AOF buffer:

$3
set
$5
hello
$5
world
Copy the code
  • $3: The length of the set instruction. Set length == 3
  • Set: indicates the set instruction
  • $5: The value length of the Hello key
  • Hello: the key values
  • $5: The length of the world value
  • World: the value value
  1. Why does AOF adopt text protocol format directly? Possible reasons are as follows:
    • Text protocols have good compatibility.

– After AOF is enabled, all write commands contain append operations and adopt the protocol format to avoid secondary processing overhead. – The text protocol is readable, facilitating direct modification and processing. 2. Why does AOF append commands to aOF_buF?

- Redis uses a single thread response command. If the command is appended directly to the disk each time the AOF file is written, the performance depends entirely on the current disk load. - Write to buffer aOF_buf first. Another benefit is that Redis can provide multiple buffer synchronization policies to balance performance and security.Copy the code

AOF configuration

In the redis redis.conf configuration file:

Redis does not enable AOF (APPEND ONLY File) persistence by default. It is enabled using appendonly:

  1. The AOF file is saved in the same location as the RDB file
Both are dir /path with the dir argumentCopy the code
  1. Redis is disabled by default. To enable redis, manually change no to yes
appendonly yes 
Copy the code
  1. Specifies the local database file name. The default value is appendone.aof
appendfilename "appendonly.aof"
Copy the code

  1. Redis supports three different brush modes. Redis provides multiple AOF buffer synchronization file policies controlled by the appendfsync parameter:

    • Every time a data change occurs, it is written to the AOF file

      • Forcing a write to disk every time a write command is received is the most guaranteed complete persistence, but it is also the slowest. It is generally not recommended
      appendfsync always
Copy the code
      • On a typical STAT disk, Redis can only support a few hundred TPS writes, which is the safest and slowest way to do so, and is not recommended because it runs counter to the high performance features of Redis.

    • It synchronizes once per second and writes the data changes that occurred in the past one second to the AOF file

      • Forcing writes to disk once per second, a good compromise between performance and persistence, is the recommended approach.
      appendfsync everysec
Copy the code
      • It is a recommended synchronization policy and the default configuration. It balances performance and data security. In theory, only 1s of data can be lost when the system breaks down suddenly. (Strictly speaking, the maximum loss of 1s data is not accurate)

    • No active synchronization, high efficiency but data persistence, is determined by the operating system

      • OS – dependent write, usually about 30 seconds, the best performance but least persistence, not recommended.
      appendfsync no
Copy the code
      • The operating system synchronizes AOF files every 30 seconds and synchronizes a large amount of disk data. This improves performance, but data security is not guaranteed.

  1. Redis AOF file synchronization policy:

    • Always (safest but slowest) : Synchronous persistence, where every data change is written to disk immediately. Poor performance when data integrity is good (slow, secure)
    • Everysec (default synchronization policy) : Factory default recommendation, asynchronous recording once per second (default)
    • No (fastest but unsafe) : out of sync

  2. Does not block fsync() calls in the main process when BGSAVE or BGREWRITEAOF is executing (default: no, adjust to yes if delay is a problem).

no-appendfsync-on-rewrite no
Copy the code

Although AOF records commands in AOF files every time an operation is performed to change the contents of the database, in fact, due to the operating system’s caching mechanism, the data is not actually written to the disk, but instead goes to the system’s disk cache. By default, the system performs synchronization every 30 seconds to write data in the disk cache to disks. If the system exits unexpectedly during this 30 seconds, data in the disk cache will be lost. In general, AOF persistence enabled applications cannot tolerate this loss, which requires Redis to ask the system to synchronize cached content to hard disk after writing AOF files.

AOF rewrite mechanism

As commands write to AOF, files get bigger and bigger. To solve this problem, Redis introduced AOF rewriting to reduce file size. AOF file rewriting is the process of converting data in the Redis process into write commands to synchronize to a new AOF file.

AOF rewriting principle

Why can rewritten AOF files become smaller? There are several reasons:

  1. Data that has timed out in the process is no longer written to the file.
  2. Old AOF files contain invalid commands such as del key1, set a 111, set a 222, etc. Overrides are generated directly using in-process data, so that the new AOF file retains only the write commands for the final data.
  3. Multiple write commands can be combined into one. For example, lpush list A, lpush list B, and lpush list C can be converted into lpush list A, b, and C.

To prevent overflow of the client buffer caused by large merged data, split the list, set, hash, and zset types into multiple entries bounded by 64 elements.

AOF rewriting principle

AOF works by apending writes to a file, which becomes more and more redundant. So Redis is smart enough to add rewriting. When the AOF file size exceeds a set threshold, Redis compresses the contents of the AOF file.

Redis forks a new process, reads the data in memory, and writes it back to a temporary file. No old files are read. Finally, replace the old AOF file.

Cases requiring compression rewrite:

  • Another problem with AOF is that persistent files get bigger and bigger. For example, if we called the INCR test command 100 times, we would have to save all 100 commands in the file, but 99 of them are unnecessary.

    • To restore the state of the database, simply save a SET test 100 file.

  • To merge persistent files that overwrite AOF, Redis provides the bgrewriteaof command. After receiving this command, Redis will use the similar way to the snapshot to save data in memory, in the form of a command to a temporary file, and finally to replace the original file, so as to achieve control of AOF file merging rewrite (will be received in the process of rewriting a new instructions and generate new rewrite AOF file merging).

Note: Because of the simulated snapshot process, the AOF file is not read when overwriting the old AOF file. Instead, the entire database contents in memory are command overwritten into a new AOF file

AOF rewrite purpose
  • In addition to reducing file footprint, AOF rewrite has another purpose: smaller AOF files can be loaded faster by Redis.
The AOF rewrite process can be triggered manually and automatically:
  • Configure the rewrite trigger mechanism
    • Manual trigger: Directly invoke the bgrewriteaof command
    • Automatic trigger: Determine the automatic trigger time according to the auto-aof-rewrite-min-size and auto-aof-rewrite-percentage parameters
      • Auto-aof -rewrite-min-size: limits the minimum number of aOF files that can be rewritten, usually when the AOF file is too small even if it contains redundant commands.
      • Auto-aof -rewrite-percentage: If the current AOF file size exceeds the percentage of the last rewritten AOF file size, it will be rewritten again. If it has not been rewritten before, it will be rewritten based on the aOF size at startup.

Note that when an AOF rewrite request is executed, the parent process still responds to the command. Redis uses the “AOF rewrite buffer “to store this new data, preventing it from being lost during the generation of a new AOF file.

auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb
Copy the code

Triggered when the AOF file size is double the size after rewrite and the file is larger than 64M. Generally set to 3G, 64M is too small, here “double” and “64M” can be changed through the configuration file.

Criteria for choosing between AOF and RDB (combined with previous article)

The trade-off is between the need for consistency of data and the need for performance. Whether the system is willing to sacrifice some performance for higher cache consistency (AOF), or whether the system is willing to disable backup during frequent write operations for higher performance, and then perform backup (RDB) when you run save manually.

AOF implementation process

After AOF persistence is enabled, Redis writes every command that changes data in Redis to an AOF file on the hard disk.

AOF’s workflow operations include:

  • Command write (append)
  • File Synchronization
  • Document rewrite
  • Restart load

The AOF process is as follows:

  1. All write commands are appended to the aOF_buf (buffer).
  2. The AOF buffer synchronizes data to disks based on the corresponding policy.
  3. As AOF files become larger, they need to be rewritten periodically to achieve compression.
  4. When the Redis service restarts, the AOF file can be loaded for data recovery.

AOF restarts to load

  • Both AOF and RDB files can be used for data recovery when the server restarts.

    1. If AOF persistence is enabled and AOF files exist, AOF files are preferentially loaded.

    2. When AOF is closed or the AOF file does not exist, load the RDB file.

    3. After loading AOF/RDB file city, Redis starts successfully.

    4. Redis fails to start and prints error messages when there are errors in the AOF/RDB file.

Recover data from AOF files

In normal cases, copy the appendone. aof file to the bin directory of the redis installation directory and restart the Redis service. However, in actual development, data restoration fails because the appendone. aof file format is abnormal. You can run the redis-check-aof –fix appendone. aof command to rectify the fault. Experience from the following operation demonstration.

If you plan to use Redis persistence. It is recommended that both RDB and AOF be enabled. In fact, RDB is more suitable for data backup, leave a back-up. AOF is broken, RDB is broken.

Advantages and disadvantages of AOF

AOF advantages

Higher data integrity and consistency

AOF disadvantage

  • AOF files are usually larger than RDB files for the same number of data sets. RDB can recover large data sets faster than AOF.

  • Depending on the synchronization strategy, AOF is usually slower than RDB in running efficiency. In summary, the synchronization per second policy is relatively efficient, and the synchronization disable policy is as efficient as RDB.

  • Data integrity and consistency are higher because AOF records more and more content, files are bigger and data recovery is slower and slower.

  • AOF files are usually larger than RDB files for the same number of data sets. RDB can recover large data sets faster than AOF.

conclusion

The synchronization per second policy is efficient, and the synchronization disable policy is as efficient as the RDB.

  • The data integrity of AOF is higher than that of RDB, but the record content is too much, which will affect the efficiency of data recovery.
  • The criteria for selecting RDB and AOF is whether the system is willing to sacrifice some performance for higher cache consistency (AOF), or whether the system is willing to disable backup for higher performance when the write operation is frequent, and then perform backup (RDB) when the SAVE operation is run manually.
  • Redis allows both AOF and RDB to be enabled at the same time, ensuring data security and making backup and other operations very easy. After restarting Redis, Redis will use AOF files to recover data, because AOF persistence may lose less data.
  • If you only intend to use Redis for caching, you can turn persistence off.