From: juejin. Cn/post / 684490…

Redis persistence

Java great Ape handsome growth manual, GitHub JavaEgg, N line Internet development necessary skills weapon spectrum

Redis data is all in memory. If there is a sudden outage, all data will be lost. Therefore, there must be a mechanism to ensure that Redis data will not be lost due to failure, and this mechanism is Redis persistence mechanism.

Redis has two types of persistence: snapshots (RDBFiles) and add-on files (AOFFile)

RDB (Redis DataBase)

What is the

Writes a Snapshot of an in-memory data set to disk at a specified time interval. A Snapshot is restored by reading the Snapshot file directly into memory.

Redis forks a separate subprocess for persistence, writing data to a temporary file that will be used to replace the last persistent file after the persistence process is complete. During the whole process, the main process does not perform any IO operations, which ensures high performance. If large-scale data recovery is required and data integrity is not very sensitive, the RDB mode is more efficient than the AOF mode. The downside of RDB is that data can be lost after the last persistence.

> What ? Isn’t Redis single-process?

Redis uses the multi-process COW(Copy On Write) mechanism of the operating system to implement snapshot persistence. Fork is the main method of creating processes On unix-like operating systems. Copy On Write (COW) is an optimization strategy used in computer programming.

Fork

Fork copies the same process as the current one. All data values of the new process (variables, environment variables, program counters, etc.) are the same as those of the original process, but are a new process and a child of the original process. The child process reads the data and serializes it to disk

RDB saves dump. RDB files by default

You can set Redis to automatically save the dataset once the condition of “at least M changes to the dataset in N seconds” is met.

You can also manually tell Redis to SAVE the dataset by calling SAVE or BGSAVE.

For example, the following Settings will automatically save the data set once Redis meets the condition that at least 1000 keys have been changed within 60 seconds:

save 60 1000

This type of persistence is called a snapshot.

Configuration position: SNAPSHOTTING

How do I trigger an RDB snapshot

  • Default snapshot configuration in the configuration file

    RDB: cp dump. RDB dump_new.rdb

  • The command save or bgsave

    • Save: Save only, other ignore, all block
    • BGSAVE: Redis takes snapshots asynchronously in the background and responds to client requests. You can run the lastsave command to obtain the time when the last snapshot was successfully executed
    • The dump. RDB file is also generated when you run the flushall command, but it is empty and meaningless

How snapshots work

When Redis needs to save the dump. RDB file, the server does the following:

  1. Redis calls fork(), producing a child with both a parent and a child.
  2. The child process writes the data set to a temporary RDB file.
  3. When the child process finishes writing to the new RDB file, Redis replaces the old RDB file with the new one and deletes the old RDB file.

This way of working allows Redis to benefit from copy-on-write mechanisms.

How to restore

Move the backup file (dump. RDB) to the redis installation directory and start the service (CONFIG GET dir).

advantage

  • Once done this way, your entire Redis database will contain only one file, which is perfect for file backups. For example, you might want to archive the last 24 hours of data every hour and the last 30 days of data every day. With this backup strategy, we can easily recover from catastrophic system failures. Suitable for large-scale data recovery
  • RDB is a great choice for disaster recovery. It is very easy to compress a single file and transfer it to another storage medium.
  • Maximize performance. As for Redis server, the only thing it needs to do to start persistence is fork out the child process, and then the child process will do the work of persistence, so that the server process can not perform IO operations.
  • Compared to AOF, RDB starts more efficiently if the data set is large.

disadvantage

  • RDB is not a good choice if you want to ensure high availability of data, i.e. avoid data loss as much as possible. If the system goes down before scheduled persistence, all data that has not been written to disk before is lost (all changes since the last snapshot are lost).
  • Since the RDB is forked to assist in data persistence, the in-memory data is cloned, and roughly twice as much expansibility needs to be considered, so if the data set is large, the entire server may be out of service for hundreds of milliseconds, or even a second.

How to stop

Redis -cli config set save”

conclusion

  • RDB is a very compact file

  • When RDB saves RDB files, the only thing the parent process needs to do is fork out a child process. The child process does all the following work. The parent process does not need to do other IO operations, so THE RDB persistence method can maximize the performance of Redis

  • Compared to AOF, RDB is faster for recovering large data sets

  • Data loss risks are high

  • The RDB often forks the child process to save the data set to the hard disk. When the data set is large, the fork process can be very time-consuming and may cause Redis to fail to respond to client requests for some milliseconds

AOF (Append Only File)

What is the

Each write operation is recorded in the form of a log, and all write instructions performed by Redis are recorded (read operations are not recorded). Only files can be appended but files cannot be overwritten. When Redis starts, it reads the file to reconstruct data, which is called “replay”. In other words, redis restarts and executes write instructions from front to back based on the contents of the log file to complete data recovery.

AOF defaults to the ** appendone.aof ** file

Configuration position: APPEND ONLY MODE

AOF startup/repair/recovery

  • To restore normal

    • Initialize: Set Yes Modify default appendonly no to Yes
    • Make a copy of the aOF file with data and save it to the corresponding directory (config get dir)
    • Recovery: Restart Redis and reload

  • Abnormal return

    • Initialize: Set Yes Modify default appendonly no to Yes
    • Back up the bad AOF file
    • Redis-check-aof –fix to fix + aof files
    • Recovery: Restart Redis and reload

Rewrite (AOF)

  • When the size of the AOF file exceeds the set threshold, Redis will start the content compression of the AOF file. Only the minimum instruction set that can restore the data can be usedbgrewriteaof
  • Rewriting principle: If the AOF file grows too large, a new process is forked to rename the file (also write temporary files first and then rename), traversing the data in the memory of the new process, and each record has a Set statement. The operation of overwriting an AOF file, rather than reading the old AOF file, commands the entire contents of the database in memory to rewrite a new AOF file, similar to a snapshot
  • Triggering mechanism: Redis records the AOF size of the last rewrite. The default configuration is to trigger when the AOF file size is double the size of the last rewrite and the file size is greater than 64M

The durability of AOF

You can configure how often Redis fsync data to disk.

There are three options:

  • Fsync is executed every time a new command is appended to the AOF file: very slow and very safe.
  • Fsync once per second: fast enough (similar to using RDB persistence) and only 1 second of data lost in the event of a failure.
  • Never fsync: Hand over data to the operating system for processing. The faster and less safe option.

The recommended (and default) measure is fsync once per second, a fsync policy that balances speed and security.

A policy of always fsync is very slow to use in practice, and frequent calls to fsync make it impossible to pick up fast.

What if the AOF file is wrong?

The server may stop when a program is writing to an AOF file, and if the stop results in corrupt AOF files, Redis will refuse to load the AOF file when it restarts to ensure that data consistency is not compromised.

When this happens, you can fix the faulty AOF file by:

  1. Create a backup of the existing AOF files.
  2. Use the redis-check-aof program that comes with Redis to fix the original AOF file.

$ redis-check-aof –fix

  1. (Optional) Compare the restored AOF file with the original AOF backup using diff -u to see the differences between the two files.
  2. Restart the Redis server, wait for the server to load the repaired AOF file, and restore the data.

AOF operation

AOF rewriting, like RDB snapshot creation, makes clever use of the copy-on-write mechanism.

Here are the steps to perform the AOF rewrite:

  1. Redis executes fork() and now has both parent and child processes.
  2. The child process starts writing the contents of the new AOF file to the temporary file.
  3. For all newly executed write commands, the parent process accumulates them into an in-memory cache while appending the changes to the end of the existing AOF file: thus, the existing AOF file is safe even if an outage occurs in the middle of a rewrite.
  4. When the child process finishes rewriting, it sends a signal to the parent process, which, upon receiving the signal, appends all the data in the memory cache to the end of the new AOF file.
  5. Done! Redis now atomically replacing old files with new ones, after which all commands are appended directly to the end of new AOF files.

advantage

  • This mechanism can lead to higher data security, that is, data persistence. Redis provides three synchronization strategies: synchronization per second, synchronization per modification, and no synchronization. In fact, synchronization per second is also done asynchronously, and it is very efficient, except that if the system goes down, the data modified within a second will be lost. With each change synchronization, we can think of it as synchronous persistence, meaning that every data change that occurs is recorded to disk immediately. Predictably, this approach is the least efficient. As for asynchrony, needless to say, I think everyone understands it correctly.
  • Because the mechanism writes log files in append mode, the existing content in log files will not be damaged even if the log file is down. However, if the system crashes after only writing half of the data, don’t worry. Before the next Redis startup, we can use the Redis-check-aOF tool to help us solve the problem of data consistency.
  • Redis can automatically enable the rewrite mechanism if the log is too large. Redis keeps writing changes to the old disk file in Append mode, and creates a new file to record which changes were executed during that time. Therefore, data security can be better ensured when switching over rewrite.
  • AOF includes a well-formatted, easy-to-understand log file for recording all changes. In fact, we can also use this file to complete data reconstruction. As a result, the contents of AOF files are easy to read and parse. Exporting AOF files is also very simple: For example, if you accidentally execute the FLUSHALL command, as long as the AOF file isn’t overwritten, stop the server, remove the FLUSHALL command at the end of the AOF file, and restart Redis, You can restore the data set to the state it was in before the FLUSHALL execution.

disadvantage

  • AOF files are usually larger than RDB files for the same number of data sets. The recovery speed is slower than that of RDB.
  • Depending on the synchronization strategy, AOF tends to run more slowly than RDB. In summary, the synchronization per second policy is relatively efficient, and the synchronization disable policy is as efficient as RDB.

conclusion

  • The AOF file is an appending only log file
  • Redis can automatically rewrite AOF in the background when AOF files become too large
  • AOF files orderly store all the write operations to the database. These write operations are stored in the Redis protocol format, so the contents of AOF files are easy to read and analyze
  • For the same data set, the size of AOF files usually needs to be larger than the size of RDB files
  • Depending on the fsync strategy used, AOF may be slower than RDB

How do I switch from RDB persistence to AOF persistence

In Redis 2.2 or above, it is possible to switch from RDB to AOF without rebooting:

  1. Create a backup of the latest dump. RDB file.
  2. Put the backup in a safe place.
  3. Run the following two commands:
 redis-cli> CONFIG SET appendonly yes 	

 redis-cli> CONFIG SET save ""Copy the codeCopy the code
  1. Ensure that the number of database keys does not change after the command is executed.
  2. Make sure the write command is correctly appended to the end of the AOF file.

The first command executed in Step 3 enables the AOF function: Redis blocks until the initial AOF file is created, after which Redis continues processing command requests and begins appending write commands to the end of the AOF file.

Step 3 Run the second command to disable the RDB function. This step is optional, or you can use both RDB and AOF if you prefer.

Don’t forget to turn on the AOF feature in redis.conf! Otherwise, after the server restarts, the configuration previously SET through the CONFIG SET will be forgotten and the program will start the server with the original configuration.

Which one

  • RDB persistence allows you to take snapshots of your data at specified intervals

  • The AOF persistence mode records each write operation to the server. When the server is restarted, these commands will be executed again to restore the original data. The AOF command saves each write operation to the end of the file using the Redis protocol. Redis can also bgrewriteaof AOF files, so that the size of AOF files is not too large

  • Cache only: If you only want your data to live while the server is running, you can also do without any persistence.

  • Enable both persistence methods

    • In this case, when Redis restarts, AOF files will be loaded first to recover the original data, because AOF files usually hold more complete data sets than RDB files.
    • RDB data is not real-time, and server restart will only look for AOF files when both are used. Should we just use AOF? Not recommended, as RDB is more suitable for backing up databases (AOF is constantly changing and not good for backing up), quick restart, and there are no potential AOF bugs left as a back-up measure.

Performance Suggestions

  • Since RDB files are only used for backup purposes, it is recommended to persist RDB files only on the Slave and backup them only once every 15 minutes, leaving only the save 9001 rule.
  • If you Enalbe AOF, the benefit is that you lose less than two seconds of data in the worst case. The startup script is simpler and you just load your own AOF files. The cost is constant IO and the inevitable blocking of writing new data to new files in the rewrite process. The frequency of AOF rewrite should be minimized as long as hard drives are permitted; the default base size of AOF rewrite, 64M, is too small and can be set to more than 5G. The default size exceeds 100% of the original size and overrides can be changed to an appropriate value.
  • If AOF is not enabled, only master-slave Replication can be used to implement high availability. It saves a lot of IO and reduces the system volatility that comes with rewriting. The trade-off is that if the Master/Slave goes down at the same time, the data will be lost for more than ten minutes, and the startup script will compare the RDB files in both Master/Slave and load the newer one.