Daily sentence

The effort is called dream, don’t work hard is a dream, the effort you pay, is the most clear time in this life. It is not inspirational sayings or inspirational stories that inspire you to keep moving forward, but you who are full of positive energy and who have been working hard on the way forward. There will be no pie falling from the sky. Come on!

The premise to review

Redis itself stores and manipulates data in memory; If the data is stored only in memory, this can cause the following problems:

  1. Data disappears with process exit: When the Server is powered off or the Redis Server process exits, memory will be released and data will be lost. Some people may think that it is just a cache, and the data is gone, so they can read it from the database again and put it in it. Just imagine, if it is a high concurrency scenario, the database is not under great pressure.

  2. Important data cannot be recovered: Data cannot be recovered after it is lost. For some important data, it only exists in Redis, but there is no relational database. If the data is lost, it cannot be recovered. For example, brush gift list, if the data loss, users certainly do not want to;

Persistent mode

Redis has two solutions for data persistence, as follows:

  • Redis DataBase (RDB) : snapshot data in the Redis memory is stored on physical disks ata specified interval. The data is stored in a *. RDB file in binary format and the restored data can be loaded directly.

  • AOF(Append Only File) : in log format, each write command is recorded in the *. AOF File in Append – Only mode. The File cannot be modified but can Only be appended. Follow-up Data recovery Automatically run commands in log files to restore data.

  • Blending is a combination of RDB and AOF: both can be easily done through configuration files.

RDB persistence

The theory will come later, let’s see the actual operation first, and then make a summary; The default configuration is as follows:

Conditions are configured with Save Seconds Changes so that RDB persistence occurs automatically if the condition is triggered. The default configuration contains the following three conditions. If one of them is met, the data is automatically saved to disk:

save 900 1:900Seconds (15Minutes) at least1Change the value of key; save300 10:300At least for a second (five minutes)10Change the value of key; save60 10000:900Seconds (1Minutes) at least10000Change the value of key;Copy the code

Other RDB configuration items

What if the RDB backup file fails to write? These are explained in detail in the SNAPSHOTTING section of the configuration file and are provided with relevant configuration items to set.

  • Stop-writes-on-bgsave-error: The default value is yes. When RDB fails to back up data, Redis stops receiving data to ensure data consistency. If you do not have high requirements on data consistency, you can disable them (set them to no), but you are advised to enable them all.

  • Rdbcompression: the default value is yes. After compression is enabled, LZF algorithm will be used to compress backup file dump. RDB. However, it will consume some CPU performance for processing, but the impact is not significant.

  • Rdbchecksum: The default value is yes, that is, after this function is enabled, backup file data will be verified, but will consume CPU performance. If you want to improve performance, you can disable this function.

  • Dbfilename: The default value is dump. RDB, that is, the default backup filename is dump. RDB.

  • Dir: the default directory is the current directory, that is, the directory where backup files are stored.

RDB manually triggers backup

The above mentioned automatic trigger backup, in fact, in practical application scenarios, some requirements are very urgent. If you require to wait until the backup is completed, it is better to have a short interval. If the interval is more than 5 minutes, it is estimated that the person waiting to handle the problem will have to go to the room to remove the tile.

Redis also provides manual backup as follows:

  • Save: Execute the save command directly, but block the operation of the main process. You can only perform other operations after the backup is complete.

  • Bgsave: When the bgsave command is executed, the main process forks a sub-process for backup without blocking the main process. When the data is too large, there may be a short time in the fork, but the impact is not significant. The above automatic backup actually ends up in bgSave mode.

  • Flushall: Running the flushall command triggers an RDB backup, but the backup file is empty

Stopping automatic Backup

The server can be configured in a configuration file or shut down by running a command. However, the server becomes invalid after being restarted by running a command. Therefore, you are advised to configure the server in a configuration file.

  • Configuration file: Remove all save configurations or configure a save seconds changes and restart redis-server.

  • Command method: Run the config set save seconds changes command on the client. The default value is restored after the redis-server restarts.

  1. When bgSave persistence is triggered (the bgSave command is configured or executed manually), the main process forks a child process to persist. Forking duplicates a child process that has the same data as the original process. The main process does not participate in any persistent IO operations.

  2. In order not to affect the use of the original RDB file, the child process writes snapshot data to the temporary file first.

When snapshot data is fully backed up to a temporary file, the original RDB file is replaced to obtain the latest RDB file.

Note: When executing the save command, it will block and cannot do anything else until the snapshot data is persisted.

Advantages and disadvantages of RDB persistence

Every technology that solves existing problems certainly creates new ones, so what are the pros and cons of RDB for persistence?

advantages

  • RDB stores more compact data files. Compared with AOF, the file size of the same data is smaller. It is faster than AOF in the case of massive data persistence.

  • In RDB, bgSave mode has little effect on the main process. It only consumes resources when the main process forks its child process, but the effect is not significant. Automatic backup background uses bgSave mode;

disadvantages

  • The RDB may lose data that was not backed up for the last time. If the server dies before the last backup, the last data will be lost.

  • When there is a large amount of data, the main process may fork the child process, resulting in a slight lag.

AOF persistence

The disadvantages of RDB, largely due to the possibility of losing data before the last backup, are unacceptable for some important data. The emergence of AOF greatly reduces the risk of data loss. Let’s not talk about it so much. Let’s talk about it.

AOF is not enabled by default. If you open the configuration file, RDB backup will not be affected.

To disable RDB backup:

Comment the configuration of the previous close.

Enable AOF backup

As mentioned in the previous article, first locate the APPEND ONLY MODE configuration block and turn the AOF backup on appendOnly Yes

Appendone.aof file is generated when redis-server is started.

Appendonly. Aof file

Try opening appendonly. Aof to see how it differs from dump. RDP;

  • Appendonly. Aof records write commands, read commands, and add-ons.

  • Similar to RDB, when redis-server is restarted, the command of automatically loading AOF file is executed successively to recover data.

AOF Other configuration items

For each function can be completed through the configuration items, the use is very convenient;

  • Appendonly: default no, AOF persistence is not enabled; This can be enabled by setting it to yes;

  • Appendfilename: default appendone. aof, which represents the generated AOF log filename and can be changed.

  • Appendfsync: set the policy of synchronizing commands to disk through fsync once per second. The following command synchronization policies are available: always: The system synchronizes data to disks through fsync whenever there is a write command, which provides high data integrity but low efficiency. Everysec: Commands are synchronized to disk through fsync once per second. This may result in loss of data in one second because the machine may hang up before commands are synchronized. All things considered, this strategy is recommended. No: the data is not synchronized and processed by the operating system. This data cannot be secure.

  • Auto – aof – rewrite – percentage with: By default, aOF will be rewritten when the current aOF file size is twice that of the last one. In order to avoid the situation where the aOF file is rewritten when the ratio reaches the trigger condition but the file size is too small. Therefore, auto-aof-rewrite-min-size is used to set the minimum rewriting size of aof file. That is, when the current AOF file size reaches the ratio and the file size is at least the value set by auto-aof-rewrite-min-size, rewriting is triggered.

  • Auto-aof -rewrite-min-size: the default value is 64mb and is used with auto-aof-rewrite-percentage.

AOF triggers rewriting

When too many write commands are executed, the AOF file will grow excessively, and it is unnecessary for some repetitive commands to exist in the AOF file, as shown in the following figure:

In the figure above, a1 Key is written for several times and the final value is 10. It can be seen that it is not best to record only one write command with the final value in AOF file, thus reducing the size of AOF file. The size of the AOF file is not enough to trigger the override automatically, so let’s see if the AOF file is optimized as follows:

The rewritten AOF file is really what we want, don’t you think Redis is more awesome? Overriding can be triggered in two ways:

  • Automatic trigger: If the values of auto-aof-rewrite-percentage and auto-aof-rewrite-min-size meet the specified value, auto-aof-rewrite-percentage and auto-aof- rewrite-size automatically trigger rewriting.

  • Manual triggering: Run the bgrewriteaof command on the client.

AOF rewrite process

After reading this article, you will be able to solve the puzzle of Redis persistence.

  1. When triggered to rewrite the AOF file, the main process forks a child process, the child process according to the existing data in memory to simplify the command, rewrite the new AOF file;

  2. When the child process is overwriting the AOF file, if there is a new write command, it is stored in the rewrite buffer (AOF_REWRITE_BUFFER_BLOCK) and synchronized to the original AOF file.

  3. When the child process completes rewriting, the main process is notified to write the new command in the rewrite buffer to the new AOF file, and the original AOF file is replaced with the new AOF file.

  4. Finally, the optimized AOF file is obtained to reduce the file size.

AOF file repair

How to solve the legitimacy of AOF file content? It may be due to sudden events, such as downtime, resulting in incomplete writing of AOF file. It is also possible for someone to maliciously add non-standard data. How does Redis handle this? Here is the simulated manual modification of AOF file, as follows:

Run the redis-check-aof –fix filename command as prompted to fix the file as follows:

Redis can also repair RDB files, the article does not show, but friends remember to try, use redis-check-rdb this tool, in the Windows version redis does not provide this tool, to Linux with a high version of the actual operation.

Advantages and disadvantages of AOF persistence

The emergence of AOF solves the problem of RDB losing the last unsaved data, greatly reducing the risk of data loss, but it also brings related problems.

advantages
  • Reduce the risk of data loss by up to one second if lost;
  • Log in the way of adding, fast;
  • Automatically optimize AOF files, rewrite files when they are too large, simplify AOF files;
disadvantages
  • For the same big data, AOF files are larger than RDB files and occupy disk space.
  • For big data recovery, the speed is not as fast as RDB.

Mixed persistence

With Redis4.0, mixed persistence configuration is enabled; Hybrid persistence is a persistence scheme combining the advantages of RDB and AOF to solve the problem of slow data recovery using AOF.

The principle is to add the RDB snapshot data in the first half of the AOF file, followed by the command record of incremental data. Aof-use-rdb-preamble Yes, the mixed persistence mode is enabled by default for higher versions of Redis.

  • Advantages: It solves the problem of slow data recovery of simple AOF;

  • Disadvantages: not compatible with older Redis scenarios;

Which persistence is appropriate?

  • If the data integrity requirements are not very high, you can accept a short period of data loss, RDB snapshot persistence is the best choice.

  • If the data integrity requirements are strict, using AOF log format for persistence is appropriate.

If redis 4.0 or higher, hybrid persistence can be used to reduce the recovery time of pure AOF files.

If it’s just caching, caching data is not important, concurrency is not very high, you don’t need to turn on persistence;

Note: If RDB and AOF are enabled at the same time instead of mixed persistence, redis-server will use AOF file for data recovery first, because AOF file is relatively complete.