preface

Redis is an in-memory database, so the data is in memory. If there is a sudden outage, the data will be lost (assuming that non-volatile memory is not used), and Redis provides a persistence mechanism to prevent this from happening

If you use Redis only as a cache and don’t need to quickly generate cached data after an outage, you can eliminate persistence and improve performance

Redis provides the following two methods of persistence:

• Redis database (RDB) : periodically saves data in memory ata certain point to disks
• AOF (Append only File) : Persists data by continuously incrementing redis write commands

The principle and implementation are described below

RDB

The snapshot principle

Redis implements data preservation by persisting snapshots taken in memory at a certain point in time to disk

How does Redis “capture” a snapshot of a moment in time?

A simple idea is that during persistence, no requests are processed, that is, the save command is executed

However, this can block online business. It is better to persist while responding to customer requests, also known as bgsave commands

In snapshot persistence mode, ensure that consistent snapshots are obtained

What does that mean? In the process of persistence, if the first half of the memory is persisted and the second half is persisted, the second half of the memory may be written to the first half and the second half of the data at the same time. The first half has been persisted and will not be modified. In this way, the data of the two parts are “inconsistent”

Here’s an example:

1. I started out with A=1, B=2
2. Serialize A=1 to disk
3. A request changes A and B to 2 in memory, where A=1 on disk
4. Continue serialization, write B=2 to disk
5. The problem is that there is illegal data on disk, because there is never A time in memory when A=1, B=2, i.eAn older version of AandA new version of BIt’s mixed up

So how do you get a consistent snapshot?

• Mysql Innodb’s repeatable read isolation level uses MVCC to ensure that each read operation obtains consistent data without blocking other read and write requests
• Redis uses multiple processesWhen writing copy(Copy on write) technology to achieve

When the Redis server receives the BGSave command, the fork function is called to produce a child process that creates the RDB file, and the main process continues to respond to the client request

When a child process is created, it shares data segments with its parent process, which is a memory saving mechanism of the operating system

Only when a process needs to make changes to the data will it make a copy for its own use and modify the copied page. In other words, the pages traversed by the child process during snapshot generation are not modified, and are always just forked out.

The page just forked out is definitely a snapshot of consistency at that point in time, so there is no problem serializing it to disk

To save time

At what point does Redis trigger a child fork to generate a snapshot file?

To save the configuration file, run the following command:

save 900 1

save 300 10

save 60 3600
Copy the code

The above configuration means that Redis automatically executes the bgsave command when:

• 900 secondsAt leastA keySend changes (add, modify, delete)
• 300 secondsAt leastTen keySend a change
• 60 secondsAt least3600个keySend a change

Why do I need to configure multiple rules?

Imagine if there were only intermediate rules (at least 10 key changes in 300 seconds)

If only 9 keys change in redis, then no matter how long it takes, those 9 keys will never be persisted, because the condition for sending at least 10 changes in 300 seconds is never met

Therefore, it is better to have a save XXX 1 configuration pocket in the Save configuration item to ensure that all changes in redis can be persisted within a certain period of time

If the number of changes is large in a short period of time, the persistence will be carried out at an interval of 300 seconds based on the unique configuration. If it goes down, changes made between the last persistence and the outage are lost. That’s up to 300 seconds of data lost

Therefore, in the Save configuration item, it is better to have a save XXX (less than 300 seconds) XXX (more than 10) configuration, so that more changes in a short time can be made earlier and more frequently. If you go down, you’ll only lose data at shorter intervals

How do you do that? It’s very simple

The following use go code examples, but no special syntax, does not affect the understanding

Redis maintains two variables:

• dirty: How many modifications have been performed since the last BGSave
• lastsave: Time of the last BGSave

And the configured rule saveConfig:

Type saveConfig struct {Change int // how many keys are changed save 60 3600 time int // how many keys are changed save 60 3600 time}Copy the code

Each configuration is tried in each event loop, and bgSave is executed if a configuration condition is met

Interval := time.now ().sub (lastsave) // Try each configuration for saveConfig := range saveConfigs {// If a configuration condition is met, run bgSave if dirty >= saveConfig.Change && Interval >= saveConfig.Time {bgSave () break}}}Copy the code

AOF

Unlike RDB, which stores data in Redis as a snapshot, AOF records the state of the database by storing write commands executed by Redis. If the AOF file records all the commands ever, you can recover the data by replaying them in an empty Redis

Synchronization strategies

A write command goes through the following three steps from generation to writing to AOF file:

Command append, file write, file synchronization

After a write command is executed, it is appended to the memory buffer

At the end of the event loop, the pseudo-code for an event loop looks like this:

Func eventLoop() {for {// processFileEvents() {// processFileEvents(); ProcessTimeEvents () // flushAppendOnlyFile()}}Copy the code

At the beginning of the event loop, the file event, the client request read and write, is performed, which is where the command append is done

Before the end of an event loop, Redis uses the appendfsync value to decide what to do with the aOF command previously added to the memory buffer:

• Always: Writes all the contents of the AOF_buf buffer and synchronizes the aOF file

  • Since each event loop synchronizes data to disk, which is known to be much slower than memory, this configurationThe lowest efficiency, butThe highest securityBecause you can only lose data for one event loop if it goes down

• Everysec: Writes all the contents of the AOF_buf buffer to the AOF file. If the time since the last synchronization is more than 1 second, the synchronization is performed

  • The frequency of synchronization has changed from every time cycle to every second,Improved efficiencyIn the event of an outage, at most 1 second of data will be lost

• No: All the contents of the AOF_buf buffer are written to the AOF file, but synchronization is not performed. The synchronization time is determined by the operating system

  • In terms of efficiencyThe fastestSo there is no need to wait for data to be synchronized to disk each time, but when to synchronize data is uncontrollable, yesThe risk of losing data over a long time range

File write and synchronization: In order to improve the efficiency of modern operating systems, when users write some data to disk (file write), the operating system usually stores the data in the memory buffer temporarily, and then flushes the data to disk after it fills up or exceeds a certain period of time. The operating system also provides file synchronization functions

It is worth noting that when configured to always, the disk is not synchronized every time a command is written, but once after an event loop. Multiple commands may be executed in a single event loop

Appendfsync is typically configured to everysec in production environments to preserve high performance while minimizing data loss

AOF rewrite

As the program runs, aOF files become larger and larger. If left unprocessed, database restarts or downtime using AOF files will take longer and longer to restore, even exceeding disk capacity limits. So Redis will periodically slim down aOF files to hold only the necessary data

So what is unnecessary data?

As an example, suppose the following six commands were historically executed against list

rpush list "A" // ["A"]

rpush list "B" // ["A","B"]

rpush list "C" // ["A","B","C"]

lpop list // ["B","C"]

lpop list // ["C"]

rpush list "D" "E" // ["C","D","E"]
Copy the code

But these six commands can be replaced with one:

rpush list "C" "D" "E"
Copy the code

As a result, both footprint and recovery time are reduced

Rewriting can be done in two ways:

1. Analyze the content about list in the existing AOF file and rewrite it

2. Read the list in memory and replace the list in the AOF file with rpush XXX

   1. Just like the one up hererpush list "C" "D" "E"

Obviously, the second method is simple and efficient, unlike the first method, which needs to design a complex algorithm to compare and process AOF files

Redis is a single-threaded application, and if you put aOF overrides on the main thread, redis will not be able to process customer responses during the overrides

To avoid this, Redis puts the work of rewriting the AOF file into the child process. This has the following advantages:

• The main thread can continue to provide external services, not affected by the aOF file rewrite
• The child process is based on the data at the moment of the fork and is not affected by subsequent actions of the main thread, which is a general advantage of the forked child approach

The child process generates a new AOF file based on the memory snapshot

The main thread continues to receive and execute new write commands during the subprocess rewrite. It may be that the actual state of the database after the subprocess finishes rewriting is inconsistent with the rewritten AOF file. Therefore, it is necessary to append the new write command during this period to the rewritten AOF file, and then replace the rewritten AOF file with the original AOF file, so that the REWRITING of AOF is complete

In order to solve the problem of data inconsistency, redis set up aof rewrite buffer, during the period of the child to rewrite, new write command in addition to be written to the original aof file, will be written to aof rewrite the buffer, so that after the child finished rewriting, can know what is the new order, will these new command appended to rewrite good aof files

Why does the new command need to be written to the original AOF file? The original AOF persistence logic is guaranteed to work properly and will not be affected if the rewrite fails

Appending the new command to the rewritten aOF file does not need to be executed in the child process, but in the main thread, because

• These new write commands won’t be many and won’t affect the main thread too much
• If you use child processes, you also need to consider how to merge new commands during appending, and ultimately need a synchronous operation to merge

Mixed persistence

If RDB is used alone, a lot of data may be lost, but if AOF is used alone, data recovery is much slower than RDB. So Redis 4.0 introduced hybrid persistence, which puts RDB files together with incremental AOF log files. In this case, the AOF log is the incremental update log from the end of RDB persistence to the current moment, which is usually small

This hybrid persistence approach includes RDBThe advantage of quick recoveryAnd AOFThe advantage of not losing large amounts of data

conclusion

• The RDB uses the copy-on-write technology to capture snapshots for persistence. You need to take various conditions into account when configuring and saving the snapshots
• Configure AOF synchronization policies based on service requirements. To avoid large files, you need to rewrite AOF files
• Hybrid persistence combines the best of both