Redis provides two ways to persist:

  • RDB: Snapshots can be taken of your data at specified intervals.
  • AOF: Records each write operation to the server. When the server is restarted, these commands are executed again to restore the original data.

Through the introduction of the following content, this paper hopes to give you a more comprehensive and clear understanding of the two persistence methods, and understand the idea of saving data, and apply it to your own system design.

  • Persistent configuration
  • How RDB and AOF persistence work
  • How do I recover data from persistence
  • Performance and practice recommendations

Persistent configuration

In order to use persistence, we need to know how to enable persistence.

Persistent configuration of RDB

#Time strategy
save 900 1
save 300 10
save 60 10000

#The file name
dbfilename dump.rdb

#File Saving Path
dir /home/work/app/redis/data/

#Whether the main process stops writing if persistence fails
stop-writes-on-bgsave-error yes

#Whether the compression
rdbcompression yes

#Check whether to import
rdbchecksum yes
Copy the code

The configuration is very simple. Here’s what persistent time policy means.

  • save 900 1If only one write command is generated within 900 seconds, a snapshot is generated, which can be interpreted as a backup
  • save 300 10If there are 10 writes in 300 seconds, a snapshot is generated

The following is similar, so why do you need to configure so many rules? Since Redis certainly does not have balanced read and write requests from time to time, we are free to customize the circumstances under which backups are triggered in order to balance performance and data security. So here is the proper configuration according to their own Redis writes.

Stop-writes-on-bgsave-error yes this configuration is also very important. When the backup process fails, the main process stops accepting new writes. This is to protect persistent data consistency. If your own service has a comprehensive monitoring system, you can disable this configuration. Otherwise, enable it.

As for the configuration of RDBCompression yes, it is not necessary to enable it. After all, Redis itself is a CPU intensive server, enabling compression will bring more CPU consumption, and CPU is more valuable than the cost of hard disk.

Of course, if you want to disable RDB configuration, it is very easy to just say: save “” on the last line of save.

AOF the configuration of the

#Whether to enable AOF
appendonly yes

#The file name
appendfilename "appendonly.aof"

#synchronously
appendfsync everysec

#Whether the AOF is synchronized during rewrite
no-appendfsync-on-rewrite no

#Override trigger configuration
auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb

#What if there is an error when loading aOF
aof-load-truncated yes

#File rewrite strategy
aof-rewrite-incremental-fsync yes
Copy the code

Again, focus on some key configurations:

Appendfsync everysec has three modes:

  • Always: Synchronizes every write command to aOF immediately, slowly but safely
  • Everysec: Sync once per second is a compromise
  • No: Redis does not process and hands it over to the OS, which is very fast, but also the most insecure

In general, the Everysec configuration is used to balance speed and security, with a maximum loss of 1s of data.

Aof -load-truncated yes If this configuration is enabled, a log is written to the client if the aOF tail is incorrect during loading. If this configuration is set to no, the client stops when errors are detected and must be repaired before reloading.

The working principle of

For the principle part, we’ll focus on how RDB and AOF do persistence and how they work.

The frequency of scheduled task execution can be set by Hz 10 in the configuration file (this configuration means that the scheduled task is executed 10 times within 1s, that is, the scheduled task is triggered once every 100ms). The maximum value can be 500, but it is not recommended to exceed 100. A larger value indicates that the execution frequency increases, which consumes more CPU and affects the read and write performance of the main process.

Scheduled tasks use Redis’s own implementation of TimeEvent, which periodically invokes commands to complete scheduled tasks that may block the main process and degrade Redis performance. Therefore, when configuring Redis, we must consider some configurations that trigger scheduled tasks as a whole and adjust them according to the actual situation.

The principle of RDB

RDB persistence in Redis can be triggered in two ways: manually triggered and Redis timed.

For RDB-style persistence, manual triggering can be used:

  • Save: blocks the current Redis server until persistence is complete and should be disabled online.
  • Bgsave: This trigger forks a child process that is responsible for persistence, so blocking only happens when the child process is forked.

The automatic trigger scenarios mainly include the following:

  • According to oursave m nConfiguration rules are automatically triggered.
  • When the primary node sends the RDB file to the secondary node for full replication, the primary node triggers the replicationbgsave;
  • performdebug reloadWhen;
  • performshutdownIs triggered if aOF is not enabled.

Since save is rarely used, let’s focus on how the bgSave command does RDB persistence.

Note here that fork operations block, causing Redis read and write performance to degrade. We can control the maximum memory for a single Redis instance to minimize Redis event consumption at fork. And the frequency of automatic triggers mentioned above to reduce the number of forks, or use manual triggers to do persistence according to your own mechanism.

The principle of AOF

The entire AOF process can be roughly divided into two steps, one is the real-time write of the command (1s loss in the appendfsync Everysec configuration) and the second is the rewriting of the AOF file.

The process of adding increments to files is as follows: Command write = append to aof_buf = synchronize to aOF disk. So why write buF first before synchronizing to disk? Writing data to disks in real time results in high DISK I/OS, which affects overall performance.

Aof overrides are used to reduce the size of AOF files and can be triggered manually or automatically. See the configuration section above for the rules for automatic triggering. The fork also happens at the rewrite step, where the main process is blocked.

Manual triggering: bgreWriteaOF. Automatic triggering is triggered according to the configuration rules. Of course, the overall time of automatic triggering is also related to the frequency of Redis scheduled tasks.

Let’s take a look at a rewritten flowchart:

There are four key points to add to the figure above:

  1. During rewrite, the main process is still responding to commands, to ensure the integrity of the final backup; Therefore, it will still be written to the old AOF file, ensuring that the data is not lost if the rewrite fails.
  2. A BUF is also reserved for the child process to prevent the newly written file from losing data, in order to also write to the new file in response to the rewrite.
  3. Rewrite is to directly generate the current memory data corresponding command, do not need to read the old AOF file for analysis, command merge.
  4. AOF files directly using the text protocol, mainly good compatibility, easy to add, high readability can be considered to modify repair.

Either RDB or AOF writes a temporary file first and then rename replaces the file.

Recover data from persistence

After data is backed up and persisted, how can we recover data from these persistent files? If a server has both RDB files and AOF files, who should be loaded?

To recover data from these files, you just need to restart Redis. Again, let’s look at the diagram to understand the process:

On startup, the AOF file is checked to see if it exists, and if not, the RDB is loaded. So why load AOF first? Because AOF saves more complete data, we know from the above analysis that AOF basically loses at most 1s of data.

Performance and Practice

From the above analysis, we know that snapshots of the RDB and overwriting of the AOF require forking, which is a heavyweight operation that blocks Redis. So in order not to affect the Redis main process response, we need to keep blocking as low as possible.

  1. Reduce the frequency of forking, such as manually triggering RDB snapshots and AOF overrides;
  2. Control the maximum memory used by Redis to prevent long fork time;
  3. Use better hardware;
  4. Configure Linux memory allocation policies to avoid fork failures due to insufficient physical memory.

What are we supposed to do online? I offer some practical experience of my own.

  1. If the data in Redis is not particularly sensitive or the generated data can be overwritten in some other way, persistence can be turned off. If the data is lost, it can be replaced in another way.
  2. Create your own policy to periodically check on Redis, and then manually trigger backups and rewrites of data;
  3. If multiple instances are deployed on a single machine, you need to prevent multiple machines from running persistent and rewrite operations at the same time to prevent memory, CPU, and I/O resource competition and make persistence serial.
  4. The master and slave machines can be added, and one slave machine can be used for backup processing. The other machines can normally respond to the commands of the client.
  5. RDB persistence and AOF persistence can be used together.

The contents of this article are mostly operational considerations, but we developers have learned this knowledge, which can sometimes help us find weird bugs. The following sections cover master/slave replication and clustering in Redis.

Public id: dayuTalk