Redis Core Part 2 – Data persistence

1. Introduction

The first reaction to a discussion of why Redis is so fast might be: Redis is an in-memory database, it stores its own database in memory; Since the data is in memory, if the server process exits, the data should be lost; But most of the time, we restart the process, the data is still there, and most of the time, Redis has more than 10 GIGABytes of data, when the process exits, it can quickly recover the 10 gigabytes of data, these sound and memory storage features and contrary to each other; To achieve this, Redis provides a data persistence feature, which can save Redis database state in memory to disk, so that downtime can be quickly recovered.

2.RDB memory snapshot

Memory snapshot refers to the full amount of data in Redis memory at a certain moment.

RDB trigger mechanism:

• Manually run the following commands:

  • Save: The Redis server blocks when the save command is executed, so all command requests sent by the client are rejected while the save command is being executed
  • Bgsave: The bgsave command gives birth to a child process that is then responsible for creating the RDB file

• The server configuration option is executed periodically to save the database state at a point in time to a snapshot file.

For example, if we provide the following configuration to the server:

save 900 1
save 300 10
save 70  10000
Copy the code

The BGSAVE command is executed if any of the following three conditions are met:

• The server made at least one change to the database within 900 seconds.
• The server made at least 10 changes to the database within 300 seconds.
• The server made at least 10,000 changes to the database within 60 seconds.

2.2.1 principle

By manually executing commands or redis internal time events, the full amount of data at a certain time can be saved. But do not know to have a friend thought that the amount of data, when the amount of data to achieve more than 10 G, also could not have finished the instantaneous, the backup of that time, the data is constantly changing, at the same time, the data is written to disk didn’t also so fast, that how to ensure the backup data is T1, T2, T3 when data? Of course, Redis cannot make a copy of the data in memory. By keeping two copies of the same data in memory at the same time, Redis memory utilization is less than 50%, which is unacceptable for users

2.2.2 COW

Cow: Copy On Write. When executing the BgSave command, Redis forks a sub-process, and the snapshot persistence is handed over to the sub-process. The child process keeps a copy of the full data index pointer of the parent process and does not make a full copy of the data when the child process is created. When data changes, such as modification, the parent process copies the original data to the new memory, and then modifies the data content by changing the pointer to d. So in this case, you can interpret it as playing with a pointer

Linux parent-child relationship:

• The parent process’s data changes that the child process cannot see.
• The child process’s data modification cannot affect the parent process, and the parent process’s modification cannot affect the child process.

2.2.3 Advantages and disadvantages of RDB

Disadvantages:

• There is no zip support and only one dump.rdb file.
• Data loss is relatively more, the window data at time point is easy to lose.
• Generate RDB file frequency is not good to grasp, too low frequency, downtime will lose more data; The disk read/write pressure is too high.

Advantages:

• Fast recovery

2.3 AOF (append only file)

Redis not Only provides RDB persistence function, but also provides AOF(Append Only File) persistence function. Unlike RDB persistence, AOF records database state by saving key value pairs in the database. AOF records database state by saving write commands executed by the Redis server.

When a client sends a write command, Redis writes the command in protocol format to the AOF buffer. The AppendfSync configuration is then read before the Redis server completes each event loop and determines whether the contents of the AOF buffer should be written and saved to the AOF file.

• Appendfsync:
  • Always: Synchronizes AOF buffers to AOF files
  • Eveerysec: Synchronizes the buffer of AOF to the AOF file. If the last synchronization takes more than a second, the AOF file is synchronized again, and a single thread is responsible for this synchronization
  • No: Writes the contents of the AOF buffer to the AOF file, but does not synchronize the file. The operating system decides when to synchronize the file.

Note: Synchronization here is the disk file cache to synchronize to disk. Normally, when we write to a file, there are two steps that happen, write to the file buffer, the buffer is full, and then write to disk.

For this configuration, in fact, most open source projects are the same idea, between performance and reliability, such as Kafka data synchronization, in the case of Redis, to achieve high performance, choose No policy; To ensure high reliability, choose the Always policy. If you want to allow a bit of data loss, but you don’t want performance to suffer too much, choose the Everysec policy.

2.3.2 pros and cons

Disadvantages:

• AOF files are easy to cause infinite size. When the Redis service process runs for a long time, AOF files are too large, which will not only affect the append, but also greatly affect the data recovery rate.

Advantages:

• Less data loss

2.3.3 AOF Rewrite (AOF_rewrite)

AOF persistence is executed by saving write command to record the state of the database, so as the server is running the passage of time, AOF the contents of the file will be more and more, the volume of the file will be more and more big, if not controlled, the volume is too large AOF file is likely to affect Redis server, even the entire host computer, And the larger the size of the AOF file, the more time it takes to restore the data using the AOF file. When the restore time is half an hour, or even a minute, the backup may become meaningless

2.3.3.1 aof_rewrite principle

The AOF file can be rewritten by executing the BGREWRITEAOF command. After executing the command, redis service will fork a child process, the child will be the snapshot of data written to the AOF file (RDB in front of the principle has said that the child can’t see the parent process data modification), at the same time redis server will maintain a AOF rewrite the buffer, the buffer will be during the period of the child process to create new AOF file, Records all write commands executed by the server. When the child process has finished creating a new AOF file, the server appends all the contents of the rewrite buffer to the end of the new AOF file, making the database state held by the old and new AOF files consistent. Finally, the server completes the AOF file rewrite by replacing the old AOF file with the new one