Most of the time we need persistent data, which is to write data in memory to disk, mostly for later reuse (such as restarting a machine, recovering data after a machine failure), or to back up data to a remote location in case of a system failure.

An important difference between Redis and Memcached is that Redisz supports persistence and supports two different persistence operations. One Redis persistence method is called Snapshotting (RDB) and the other method appends only files (AOF). Each of these methods has its advantages. Here’s why we need persistence, what they are, how to use them, and how to choose the right method for you.

Why persistence

Redis acts as a key-value in-memory database (NoSQL). Data is stored in memory and all operations are performed in memory when processing client requests, as shown below:

What’s wrong with that?

In fact, as long as a little computer basic knowledge of the people know, stored in the memory of the data, as long as the server shut down (caused by various reasons), the data in the memory will disappear, not only the server shut down the opportunity to cause the data disappear, Redis server daemon exit, the data in the memory will also disappear.

For projects that only use Redis as a cache, data disappearance may not be a problem, and data can be re-loaded from the data source. However, if the business data submitted by users is directly stored in Redis, and Redis is used as a database, in which important business data is stored, The impact of Redis’s memory data loss could be devastating.

In order to avoid the loss of data in memory, Redis provides the support of persistence. We can choose different ways to save data from memory to hard disk for persistent storage.

RDB

RDB is a snapshot storage persistence mode. Specifically, the Redis memory data at a certain point is saved to a file on the hard disk. The default file name is dump. RDB.

Enable the RDB persistence mode

Enabling RDB persistence is simple. Clients can send save or BGSave commands to the Redis server to have the server generate an RDB file, or specify RDB trigger conditions from the server configuration file.

1. The save command

The save command is a synchronous operation

# Synchronize data to disk
> save
Copy the code
save 900 1           # After 900 seconds (15 minutes), Redis automatically triggers the BGSAVE command to create a snapshot if at least one key has changed.

save 300 10          # After 300 seconds (5 minutes), Redis automatically triggers the BGSAVE command to create a snapshot if at least 10 keys have changed.

save 60 10000        After 60 seconds (1 minute), Redis automatically triggers the BGSAVE command to create a snapshot if at least 10,000 keys have changed.
Copy the code

When a client sends a save command to the server for persistence, the server blocks requests from other clients following the save command until data synchronization is complete.

If the amount of data is too large, synchronizing the data will take a long time, during which time the Redis server cannot receive other requests, so it is best not to use the save command in a production environment.

2, bgsave

Unlike the save command, the BGsave command is an asynchronous operation.

Save the data set to disk asynchronously
> bgsave
Copy the code

So, in contrast to the save command, the Redis server uses a sub-process for IO writing for BGSave, while the main process can still receive other requests, but the Forks sub-process is synchronous, and all forks sub-processes cannot receive other requests, which means that If the forks sub-process takes too long (usually too fast), the BGSave command can still block requests from other customers.

3. The server configuration is triggered automatically

In addition to sending commands from the client, save in the Redis configuration file can specify conditions that trigger RDB persistence, such as RDB data synchronization for at least as many writes in as many seconds.

For example, we can specify the following options in the redis.conf configuration file:

At least one write command is reached within 900s
save 900 1
At least 10 write commands in # 300s
save 300 10
Reach at least 10000 write commands in 60 seconds
save 60 10000
Copy the code

The configuration file is then loaded when the server is started

Start the server to load the configuration file
redis-server redis.conf

Copy the code

This is similar to the bgsave command. When the trigger condition is reached, data synchronization will be performed by a sub-process in Forks. However, it is best not to trigger RDB persistence in this way because the trigger time is too short and RDB files will be written frequently, affecting server performance. If the time is too long, data will be lost.

4. RDB file

There are three ways to get the server to generate RDB files, either by the main process or by a child process, as follows:

  • 1. Generate a temporary RDB file and write data.
  • 2. Complete data writing and replace the official RDB file with a temporary file.
  • 3. Delete the original DB file
Whether to compress RDB files
rdbcompression yes

# RDB file name
dbfilename redis-6379.rdb

# RDB file save directory
dir ~/redis/
Copy the code

5. Several advantages of RDB

  • 1. Compared with AOF, data recovery through RDB file is faster.
  • 2. RDB files are very suitable for data backup.
  • 3. Data backup through RDB has little impact on Redis server performance because it is generated by sub-process.

6. Several disadvantages of RDB

  • 1. If the server is down, the RDB method will cause the loss of data in a certain period of time. For example, if we set the synchronization once in 10 minutes or once in 5 minutes when 1000 writes are reached, then if the server freezes before triggering conditions are reached, the data in this period will be lost.
  • Using the save command will block the server until the direct data synchronization is complete to receive subsequent requests.
  • Using the BGsave command in the Forks process, if the volume of data is too high, the Forks process will block and the forks process will consume memory.

AOF

Unlike RDB, which stores snapshots at a certain point in time, THE AOF persistence mode records every write operation command from the client to the server, and stores these write operations to the end of the file with the suffix AOF in Redis protocol. When the Redis server restarts, the AOF file commands are loaded and run to recover data.

Enable AOF persistence

AOF persistence is not enabled by default in Redis. You can enable AOF persistence in the configuration file as shown in the following Redis.

# Enable aOF
appendonly yes

# aof file name
appendfilename "appendonly.aof"

Write policy,always means that every write operation is saved to aof file, everysec or no
appendfsync always

Aof files are not overwritten by default
no-appendfsync-on-rewrite no

# save directory
dir ~/redis/

Copy the code

Three write strategies

In the configuration file above, we can specify the write policy through the appendfsync option, which has three options

appendfsync always
# appendfsync everysec
# appendfsync no

Copy the code

1, always

Every write operation on the client is saved to an AOF document, which is a safe strategy, but each write request has IO operation, so it is slow.

2, everysec

Appendfsync’s default policy is to write to aOF files once per second, so many may lose 1s of data.

3, no

The Redis server is not responsible for writing the AOF, but leaves it up to the operating system to handle when the AOF file is written. Faster, but also the least safe option, not recommended

AOF file overwritten

AOF appends every write operation on the client to the end of the AOF file. For example, when the INCR command is executed on a key multiple times, the AOF saves each command line in the AOF file, and the AOF file becomes very large.

incr num 1
incr num 2
incr num 3
incr num 4
incr num 5
incr num 6
...
incr num 100000

Copy the code

To solve this problem, Redis supports aOF file rewriting. By rewriting AOF, you can generate a minimum set of commands to restore the current data, such as the number of commands in the above example, which can be rewritten as:

set num 100000
Copy the code

The aOF file is a binary file, and instead of saving each command directly as in the example above, Redis uses its own format, which is just for demonstration purposes

There are two ways to write

The no-appendfsync-on-rewrite option in the redis.conf configuration file allows you to set whether to enable rewriting. This option overwrites every fsync and affects server performance, so the default is no and is not recommended.

Aof files are not overwritten by default
no-appendfsync-no-rewrite no
Copy the code

The client can also send the bgrewriteaOF command to the server to have the server do AOF rewriting.

Let the server asynchronously override the append aOF file command
> bgrewriteaof
Copy the code

AOF overwriting is also asynchronous, meaning that if an AOF file is written, the main Redis process forks a child process for it

Benefits of rewriting aOF files

  • Compress AOF files to reduce disk usage.
  • Aof commands are compressed into a minimum command set to speed up data recovery

The AOF file is damaged

If the Redis server is down when writing the AOF log file, the AOF log file will be in the wrong format. When the Redis server is restarted, the Redis server will refuse to load the AOF file. You can repair the AOF and recover the data by following the steps.

  • Back up existing AOF files, just in case
  • Run the redis-check-aof command to repair aOF files. The command format is as follows:
Fix the AOF log file
$ redis-check-aof -fix file.aof
Copy the code
  • Restart Redis server, shelf ah has been repaired AOF file, restore data.

AOF advantages

AOF simply apends log files, so it has less impact on server performance, is faster than RDB, and consumes less memory.

The disadvantage of AOF

  • The size of the log file generated in AOF mode is too large
  • Data recovery is slower than RDB.

Choose RDB or AOF

Through the above comparison, we understand the advantages and disadvantages of RDB and AOF, how to choose? Through the comparison below, we can compare AOF and RDB, from several aspects in the applications according to their actual needs, choose to RDB or AOF, in fact, if you want to secure enough data, can open both ways, but two persistence ways for IO operations at the same time, will seriously affect the server performance, so sometimes have to make a choice.

When both RDB and AOF are enabled, Redis preferentially uses AOF logs to recover data, so AOF saves more complete files than RDB files

Redis 4.0 optimizes persistence

Redis 4.0 starts to support mixed persistence of RDB and AOF (disabled by default and enabled by aof-use-rdb-preamble).

If mixed persistence is turned on, the CONTENTS of the RDB are written directly to the beginning of the AOF file when AOF is overwritten. The advantage of this is that you can combine the advantages of RDB and AOF to load quickly without losing too much data. Of course, there are shortcomings, the RDB part of AOF is that the compressed format is no longer AOF format, poor readability.

The above article is from 10 minutes to thoroughly understand Redis’ persistence mechanism: RDB and AOF