In this article we will introduce the mechanisms associated with Redis high availability. In order to achieve high availability, Redis mainly has the following aspects to ensure:

Data persistence Master/slave replication automatic failover clusteringCopy the code

In this article, we will introduce the foundation of Redis’s high availability guarantee: data persistence. Because Redis relies on Redis persistence for both master-slave replication and automatic failover. At the same time, Redis data persistence can also be used for data backup to ensure data security.

Redis is an in-memory database, its data is stored in memory, if the instance goes down, then all data is lost. How to ensure data integrity and security is also one of the important mechanisms to improve the high availability of services.

Redis provides a perfect persistence mechanism, which can persist the data in memory to the disk, which is convenient for us to back up data and quickly recover data.

In this article, we will analyze how Redis data persistence is implemented. What’s the difference between RDB and AOF that we hear a lot? And different scenarios for their use.

Redis provides two data persistence modes:

RDB: a data snapshot file is generated. AOF: a log file of real-time append commandsCopy the code

They correspond to different usage scenarios, and we will analyze them in turn.

RDB is introduced

RDB stands for Redis Database Backup File, also known as Redis data snapshot.

You can have Redis locally generate an RDB snapshot file that contains nearly complete data content for the entire instance by executing the save or bgSave commands.



Its advantages are as follows:

RDB file data is written in compression, so the size of the RDB file is smaller than the entire instance memory. When an instance is recovered from an outage, the RDB file can be loaded quickly and the data in the file can be quickly recovered in a short timeCopy the code

Its disadvantages are also obvious:

Because it is a snapshot of the data at a certain point in time, the cost of RDB file generation is relatively high, which will consume a lot of CPU and memory resourcesCopy the code

Therefore, RDB is suitable for the following scenarios:

Primary/Secondary full data Synchronization Database backup is not sensitive to lost data, and data can be quickly recovered after instances are downCopy the code

Redis master/slave full synchronization is done using RDB files, which we’ll cover in more detail in a later article.

This shows that RDB is very suitable for data backup, we can periodically have Redis generate RDB file, and then backup the snapshot file.

Redis also provides configuration items that trigger the generation of RDB files on a regular basis:

# At least 1 write in the last 15 minutes Save 900 1 # At least 10 write in the last 5 minutes Save 300 10 # At least 10000 write in the last 1 minute Save 60 10000Copy the code

If any of the above conditions are met, Redis automatically generates a new RDB file to reduce the difference between the RDB data content and the instance data.

Copy On Write RDB files can be generated by executing both save and BGSave commands On Redis, but the former is executed in the foreground, which means that the entire instance is blocked while the RDB file is generated. No request can be processed until the RDB is generated. Generating RDB files is very time consuming, which is clearly unacceptable.

So we usually choose to bgSave and let Redis generate RDB files in the background so that Redis can still process client requests without blocking the entire instance.

However, it is not said that background RDB generation is free of cost. In order to realize the background to Write the snapshot of memory data into the file, Redis adopts the Copy On Write technology provided by the operating system, which is also known as fork system call.

The fork system call produces a child that shares the same memory address space as the parent so that the process has the same memory data as the parent at that point in time.

Although the child and parent process share the same memory address space, but in the fork, the operating system need to copy the memory of the parent page table to the child, if the Redis instance memory footprint is very large, so its memory page table will be very big, the copy will be more time consuming, and at the same time the process will consume large amounts of CPU resources. The parent process is also blocked and unable to process client requests until the copy is complete.

After fork, the child can scan all of its own memory data and write all of its data to the RDB file.

After that, the parent process still processes requests from clients. When processing Write commands, the parent process reallocate the new memory address space and request new memory usage from the operating system, which is no longer shared with the child process. This process gives the name Copy On Write. In this way, the parent process will gradually separate the memory. The parent process will apply for new memory space and change the memory data, while the child process’s memory data is not affected.

As you can see, generating an RDB file consumes not only CPU resources, but also up to twice as much memory.

When we execute info in Redis, we can see how long it takes to fork the child process. We can use this time to evaluate whether the fork time is as expected. At the same time, we should ensure that the Redis machine has enough CPU and memory resources, and properly set the timing of RDB generation.

AOF is introduced

AOF stands for Append Only File. Unlike RDB, AOF records detailed information about each command, including complete command types, parameters, and so on. Whenever a write command is generated, it is written to the AOF file in real time.

AOF can be enabled via configuration files:

Appendonly yes # AOF filename appendfilename "appendonly. AOF "# appendfsync everysecCopy the code

Flush Mode After AOF is enabled, Redis records the commands of each write operation in a file and stores the commands in the disk persistently. To ensure data file security, Redis also provides the time when files are flushed:

Appendfsync always: Flush disks every time data is written, which has the greatest impact on performance and occupies a high DISK I/O footprint. Appendfsync Everysec: Appendfsync no: Flush disks according to the operating system mechanism. The impact on performance is minimal and data security is low. Data loss during node downtime depends on the operating system flush mechanismCopy the code

As can be seen from the above, the advantage of AOF compared with RDB is that AOF data files are updated in a more timely manner and more complete data is stored than RDB. In this way, complete data can be recovered as far as possible during data recovery to reduce the risk of data loss.

If both RDB and AOF files exist, Redis will use AOF files for data recovery.

But its drawbacks are easy to see:

As time goes by, AOF files will become larger and larger. AOF file flushing will increase the load of disk I/O and may affect Redis performance (when flushing per second is enabled).Copy the code

AOF rewriting For the first case, Redis provides the AOF slimming function, which can be set to automatically trigger THE AOF rewriting when the AOF file is large. Redis will scan the entire instance data and generate a new AOF file to achieve the slimming effect. But this rewriting process also consumes a lot of CPU resources.

# AOF specifies the maximum size of the AOF file to trigger auto-aof-rewrite-percentage. 100 # AOF specifies the maximum size of the AOF file to trigger auto-aof-rewrite-min-size 64mbCopy the code

Because AOF minimizes the risk of data loss, it is generally suitable for business scenarios that are sensitive to data loss, such as those involving money transactions.

Performance impact If the flush time of AOF is set to flush every write, the write performance of Redis will be greatly reduced, because each write command will be returned only after the file is written to the disk. When the write volume is large, the disk I/O burden will be increased. Performance and data security are not compatible, and while Redis provides a mechanism for real-time flushing, it is rarely used in real scenarios.

We usually choose to flush every second to ensure good write performance and lose up to 1 second of data in case of instance downtime, so as to achieve a balance between performance and security.

conclusion

Our summary of RDB and AOF is shown in the table below.



We need to choose an appropriate persistence mode for different business scenarios. We can also use it in combination with the advantages of RDB and AOF to ensure the security of Redis data and take into account its performance.