Redis persistence mechanism

Why persist

If Redis finds that Redis data is empty when it accesses again, a cache penetration occurs. More importantly, because the data in Redis is empty, there is no key that the client wants to access, causing a large number of requests to hit the database instantly, causing a cache avalanche (a small number of keys is penetration, a large number of keys is avalanche). At this point, the database might hang. And can not guarantee that Redis does not break down, so it is necessary to quickly restore the contents of the inside when Redis breaks down. So you need to do a persistence. Persistence is for recovering data, not storing it

RDB

Redis DataBase (RDB) is the default Redis storage mode. The RDB mode is implemented by using snapshotting.

Method of triggering a snapshot

The snapshot rules must be customized

  • Save 900 1 # Snapshots are taken if at least one key has been changed within 15 minutes (900 seconds).

  • Save 300 10 # Snapshots are taken if at least 10 keys have been changed within 5 minutes (300 seconds).

  • Save 60 10000 # indicates that at least 10000 keys have been changed in 1 minute

When the condition that the dataset has at least M changes within N seconds is met, the dataset is automatically saved once.

Run the save or bgsave command

The dump. RDB file can be generated by executing the save or bgsave command. Each time the command is executed, all redis memory is snapshot to a new RDB file and the original RDB snapshot file is overwritten.

Comparison between SAVE and BGSave:

The command save bgsave
IO types synchronous asynchronous
Whether to block other redis commands is No (there is a short block when generating child execution to call the fork function)
The complexity of the O(n) O(n)
advantages No extra memory will be consumed Do not block client commands
disadvantages Block the client command Child processes are forked, consuming memory

Configure automatic RDB file generation using bgSave mode in the background.

Run the flushall command

flushall
Copy the code

Before clearing Redis, save the current Redis snapshot

Performing a master-slave replication (first time)

The RDB file is generated during the first master/slave replication, and the current Redis snapshot is saved

RDB executes the process

  • Process analysis

      1. The Redis parent determines whether it is currently executing a save or bgSave/bgrewriteAof (aOF file rewrite command) child, and if it is executing the BGsave command returns directly.
      1. The parent process forks (calling an operating system function to copy the main process) to create a child process. The parent process is blocked and Redis cannot execute any commands from the client.
      1. After the parent fork, the bgsave command returns “Background Saving Started” and no longer blocks the parent and can respond to other commands.
      1. The child process creates an RDB file, generates a temporary snapshot file based on the memory snapshot of the parent process, and then atomic replaces the original file. (RDB is always complete)
      1. The child sends a signal to the parent indicating completion, and the parent updates the statistics.
      1. The parent forks the child and resumes work.

RDB file structure

  • 1. The first 5 bytes are REDIS
  • 2, 4-byte “RDB” version number (not Redis version number), currently 9, after filling 0009
  • 3. Auxiliary fields are in the form of key-value
  • 4. Store database numbers
  • 5. Dictionary size
  • 6. Expired keys
  • 7. Main data is stored in the form of key-value
  • 8. Closing sign
  • 9, checksum, is to see whether the file is damaged, or whether it has been modified

Advantages and disadvantages of RDB

advantages

  • RDB is a binary compressed file that takes up little space and is easy to transfer (to SLAver)

  • The main process forks child processes to maximize Redis performance. The main process cannot be too large. The main process blocks during replication

disadvantages

  • Data integrity is not guaranteed and all data changed since the last snapshot will be lost

AOF

AOF(Append Only File) is another method of Redis persistence. Redis is disabled by default. When AOF persistence is enabled, Redis records all commands (and their parameters) written to the database (RESP) to the AOF file to record the database state.

This will restore the commands to their original state when Redis restarts by simply playing them back sequentially. AOF records the process, RDB only the results

AOF persistent implementation

Configure redis. Conf

# Redis can be modified.confAppendonly yes # The location of the AOF file is the same as the location of the RDB file, which is set using the dir parameter. Dir./ # The default filename is appendone.aof. Appendfilename can be changed to appendfilename appendone.aofCopy the code

AOF principle

AOF files store redis commands. The whole process of synchronizing commands to AOF files can be divided into three stages:

  • Command propagation :Redis sends the command executed, the parameters of the command, and the number of parameters of the command to the AOF program.

  • Cache add-on: The AOF program converts the command to the network communication protocol format according to the received command data, and then appends the protocol content to the AOF cache of the server.

  • File write and save: The contents of the AOF cache are written to the end of the AOF file. If the AOF save criteria are met, fsync or fdatasync will be called to actually save the contents to disk.

Command transmission

When a Redis client needs to execute a command, it sends the protocol text to the Redis server over a network connection. Upon receiving the request from the client, the server selects the appropriate command function based on the content of the protocol text, and converts each parameter from the string text to a Redis StringObject (StringObject). The command parameters are propagated to the AOF program each time the command function is successfully executed.

Cache additional

When a command is propagated to the AOF program, the program converts the command from a string object back to the original protocol text based on the command and its arguments. After the protocol text is generated, it is appended to the aof_buf end of the redis. H /redisServer structure.

The redisServer structure maintains the state of the Redis server, while the AOF_BUF field holds all the protocol text (RESP) waiting to be written to AOF files.

File write and save

When the server routine tasks function is executed, or the event handler is executed, aof. C/flushAppendOnlyFile function will be called, this function performs the following two jobs:

  • WRITE: Writes the cache in aOF_buf to the AOF file based on the condition.

  • SAVE: Call the fsync or fdatasync function depending on the condition to SAVE the AOF file to disk.

AOF save mode

Redis currently supports three AOF save modes, which are:

  • AOF_FSYNC_NO: not saved.

  • AOF_FSYNC_EVERYSEC: Saves every second. (the default)

  • AOF_FSYNC_ALWAYS: Saves each command. (Not recommended)

AOF_FSYNC_NO

Never fsync, handing the data to the operating system for processing. The faster and less safe option.

SAVE will only be executed in any of the following cases:

  • Redis is closed

  • The AOF function is disabled

  • The system’s write cache was flushed (either the cache was full or a periodic save operation was performed)

The SAVE operation in all three cases causes the Redis main process to block.

AOF_FSYNC_EVERYSEC

SAVE is in principle executed every second, because the SAVE operation is invoked by the back fork, so it does not block the main server process and only loses 1 second of data in the event of a failure.

AOF_FSYNC_ALWAYS

WRITE and SAVE are executed each time a command is executed. Fsync is executed every time a new command is appended to the AOF file, which is very slow and very safe.

Because SAVE is executed by the main Redis process, the main process is blocked and cannot accept command requests during SAVE execution.

The impact of AOF save mode on performance and security

A comparison of the three models

AOF rewrite

AOF records the changing process of data, which is getting bigger and bigger, and needs to rewrite the “slimming”.

Redis can automatically rewrite the AOF in the background (Fork child process) when the AOF becomes too large.

The rewritten new AOF file contains the minimum set of commands needed to restore the current data set.

In fact, AOF rewrite does not require any writing or reading of the original AOF file. It refers to the current value of the key in the database.

For example

set s1 11
set s1 22
set s1 33

lpush list1 1 2 3
lpush list1 4 5 6
Copy the code

AOF rewritten

set s1 33

lpush list1 1 2 3 4 5 6
Copy the code

Redis doesn’t want the AOF override to cause the server to be unable to handle requests, so Redis decides to put the AOF override in a child process.

  • 1. The main process can continue to process command requests while the child processes AOF rewrite.

  • 2. The child process has a copy of the data of the main process. Using the child process instead of the thread can ensure the security of the data in the case of avoiding locking.

However, there is a problem with using child processes: while the child processes the AOF rewrite, the main process needs to continue processing commands, and new commands may modify existing data, making the data in the current database inconsistent with the data in the rewritten AOF file.

To solve this problem, Redis has added an AOF rewrite cache, which is enabled after the child is forked out. When the main Redis process receives a new write command, it appends the protocol content of the write command to this cache, in addition to the existing AOF file.

Rewrite process analysis

Redis will continue to append commands to existing AOF files as new AOF files are created, and the existing AOF files will not be lost even if downtime occurs during the rewrite process. Once the new AOF file is created, Redis switches from the old AOF file to the new AOF file and starts appending the new AOF file.

When the child process is performing an AOF rewrite, the main process needs to do the following three things:

  • Process command requests.
  • Appends the write command to an existing AOF file.
  • Appends the write command to the AOF rewrite cache

. This ensures that existing AOF functionality will continue and that no data will be lost even if an outage occurs during AOF rewrite. All commands that make changes to the database are logged into the AOF rewrite cache.

When the child completes the AOF rewrite, it sends a completion signal to the parent, which calls a signal handler and does the following:

  • Writes everything in the AOF rewrite cache to the new AOF file.
  • Rename the new AOF file to override the original AOF file.

——-> new AOF file —-> overwrite old when step 1 is completed, the state of the existing AOF file, the new AOF file and the database are exactly the same.

When step 2 is complete, the program completes the interchangeover of the old and new AOF files. After this signal handler completes, the main process can continue to accept command requests as usual

. During the entire AOF background rewrite, only the last write cache and rename operations block the main process. At other times, AOF background rewrite does not block the main process, which minimizes the performance impact of AOF rewrite.

AOF override trigger mode

  • 1. Configure triggering
# indicates that the current AOF file size will be overwritten if the current AOF file size exceeds the previous AOF file size by what percentage. Auto -aof-rewrite-percentage 100 # specifies the maximum aof file size allowed to be overwritten if the aof file size is less than 64mb. No need to optimize auto-aof-rewrite-min-size 64MBCopy the code
  • 2. Run bgrewriteaof
127.0.0.1:6379 > bgrewriteaof 'BackgroundappendonlyfilerewritingstartedCopy the code

AOF rewrite summary

Mixed persistence

Both RDB and AOF have their advantages and disadvantages, and Redis 4.0 starts to support mixed persistence of RDB and AOF.

When mixed persistence is turned on, aofrewrite writes the RDB directly to the top of the AOF file.

Head of RDB + body of AOF —-> appendone.aof

Enable mixed persistence

aof-use-rdb-preambleyes
Copy the code

AOF file loading and data restoration

If mixed persistence is enabled, when AOF is rewritten, it no longer converts the memory data into RESP commands and writes them into AOF files. Instead, it takes RDB snapshots of the memory before the rewriting and writes the RDB snapshots together with the incremental AOF commands to modify the memory data into a new AOF file. The new file is not initially called appendone.aof, but will be renamed after rewriting the new aOF file, overwriting the original aOF file and completing the replacement of the old and new aOF files.

Therefore, when Redis restarts, the contents of RDB can be loaded first, and then the incremental AOF log can completely replace the previous full AOF file replay, so the restart efficiency is greatly improved.

  • Create a fake client with no network connection

Since Redis commands can only be executed in the client context, and the commands used to load AOF files come directly from AOF files and not from the network connection, the server uses a pseudo-client with no network connection to execute the write commands saved by AOF files. The effect of the command executed by the pseudo client is exactly the same as that executed by the client with network connection

  • Analyze and read a write command from the AOF file

  • 3. Use the pseudo client to execute the read write command

  • 4. Perform Steps 2 and 3 until all write commands in the AOF file are processed

Redis data backup policy

  • 1. Write a crontab scheduling script to copy an RDB or AOF backup file to a directory every hour, reserving only the backup file generated in the last 48 hours

  • 2. Back up one copy of the current day’s data to a directory every day, which can save the latest one month’s backup

  • 3. Every time I copy a backup, I delete the old backup

  • 4. Make a copy of the backup on the current machine to another machine every night to prevent machine damage

RDB versus AOF

  • 1. RDB stores a snapshot of the data at a certain time, using binary compressed storage, AOF stores operation commands, using text storage (mixed)

  • 2. High RDB performance and low AOF performance

  • 3. RDB will lose all data changed after the last snapshot when the configuration is triggered. If AOF is set to save once per second, the data will be lost for a maximum of 2 seconds

  • 4. Redis runs in primary server mode and RDB does not store expired key/value pairs. Redis runs in secondary server mode and RDB stores expired key/value pairs. When AOF writes to a file, a del command is appended to expired keys. When AOF overwrites, expired keys and del commands are ignored.