Redis memory KV database, once the server down, the data in memory will be lost!

Mysql database solution, from the back-end database to recover this data, there are problems

Problem 2: The data is read from a slow database, and the performance is not as good as reading from Redis. As a result, the application using the data is slow to respondCopy the code

Redis has two main mechanisms for persistence,

AOF (Append Only File) Logs and RDB snapshotsCopy the code

AOF solve

In Redis, after data is written to memory, the Log is written to the database. Unlike the traditional database Write Ahead Log and WALl, wal writes to the Log first, then writes to the library

Why does AOF run the command before logging?

Cause 1: Traditional database logs, such as redo logs, record modified data, whereas AOF records every command received by Redis. These commands are stored as text. For example, SET TESTKEY TESTVALUE, AOF records SET TESTKEY TESTVALUE ->redis ->AOF

* 3. $3; set; $7; testkey; $9; testvalue;Copy the code

• “*3” indicates that the current command has three parts. Each part starts with “$+ number” and is followed by a specific command, key, or value.

• The number indicates the number of bytes of the command, key, or value in this section

• “$3 set” indicates that this section has three bytes, which is the “set” command.

To avoid extra checking overhead, Redis does not first check the syntax of commands when logging to AOF. Therefore, if the log is recorded before the command is executed, the wrong command may be recorded in the log. When Redis uses the log to recover data, it may make an error.

In post-write log mode, the system executes commands first. Only commands that are successfully executed are recorded in logs. Otherwise, the system reports errors to the client. So one of the big benefits of Redis using post-write logging is that you can avoid logging incorrect commands.Copy the code

Cause two: Redis records logs after the command is executed. Therefore, the current write operation is not blocked.

AOF has two risks

A risk

If a command goes down just after it is executed without logging, the command and its data are at risk of being lostCopy the code

The second risk

While AOF avoids blocking the current command, it may risk blocking the next operation. This is because the AOF logs are also executed in the main thread. If the disk is under heavy write pressure when the log files are written to disk, the write to disk will be slow and subsequent operations will not be able to be performedCopy the code

The risk of three

AOF files that are too large can be very, very slow to recover because you need to execute the commands line by lineCopy the code

Three write back strategies

The AOF mechanism provides three options. The three optional values of the AOF configuration item appendfsync: Always, write the log back to disk synchronously immediately after each write command is executed; Everysec, write back per second: after each write command is executed, only the log is written to the memory buffer of AOF file first, and the contents of the buffer are written to disk every second. No, operating system-controlled write back: After each write command is executed, the log is first written to the memory buffer of the AOF file. The operating system decides when to write the contents of the buffer back to disk.

None of the three write back strategies can solve the problem of preventing main thread blocking and reducing data loss.

“Always Synchronous write back” can basically do not lose data, but it has a slow down operation after each write command, which inevitably affects the main thread performance. Everysec writes back once a second to avoid the performance overhead of synchronous write back. Although the impact on system performance is reduced, commands that do not fall off disks in the last second will still be lost in the event of a breakdown. So, this is a compromise between not affecting main thread performance and not losing data. “No operating system control write back” after writing buffer, you can continue to execute subsequent commands, but the timing of disk fall is not in the hands of Redis, as long as the AOF record is not written back to disk, once the downtime corresponding data will be lost;

Write back timing of the three strategies, and a summary of their advantages and disadvantages:

What if the log file is too large? — AOF rewriting mechanism

The AOF rewrite mechanism is that Redis creates a new AOF file according to the current state of the database, that is, reads all the key/value pairs in the database, and records the write of each key/value pair with a command. For example, when the key-value pair "testkey" : "testValue" is read, the override mechanism records the set testkey testvalue command. In this way, when you need to restore, you can run the command again to write "testkey" : "testValue".Copy the code

The override mechanism has a “polychange-one” feature, which means that multiple commands in the old log file become one command in the new log after rewriting. The AOF file records the write commands received one by one in an appending manner. When a key-value pair is repeatedly modified by multiple write commands, the AOF file records the corresponding multiple commands. When overwritten, it is based on the current state of the key value pair, it generates the corresponding write command legend:

Will AOF overwriting block?

Unlike the 'AOF logs' written back by the main thread, the rewrite process is done by the backend child bgreWriteAof to avoid blocking the main thread and causing database performance degradation.Copy the code

The process of rewriting is summed up as “one copy, two logs”.

“One copy” means:

Each time a rewrite is performed, the main thread forks out the background bgrewriteaof child. Fork sends a copy of the main thread's memory to the bgrewriteaof child, which contains the latest data from the database. The bgrewriteAof child process can then write the copied data as operations, one by one, to the rewrite log without affecting the main thread.Copy the code

“Two logs” means:

First: The first log is the AOF log that is being used, and Redis writes this operation to its buffer. This way, even if there is an outage, the OPERATION of the AOF log is still complete and ready for recovery. Second: log, which means the new AOF rewrite log. This operation is also written to the buffer of the rewrite log. In this way, the rewrite log will not lose the latest operations. After all operation records of copying data are rewritten, the latest operations recorded in the rewrite log will also be written into a new AOF file to ensure that the latest database status is recorded. At this point, we can replace the old file with the new AOF file. Summary: Redis performs a memory copy for each AOF rewrite. Then, two logs are used to ensure that the newly written data is not lost during the rewrite process. Also, because Redis uses additional threads for data rewriting, this process does not block the main thread.Copy the code

Here is:

When will AOF rewrite be triggered? There are two configuration items that control when AOF rewriting is triggered:

Auto-aof -rewrite-min-size: specifies the minimum size of aof files. The default value is 64MB. Auto-aof -rewrite-percentage: specifies the maximum size of aof files. This value is calculated by dividing the difference between the current AOF file size and the AOF file size since the last rewrite by the AOF file size since the last rewrite. It's the ratio of the size of the current AOF file to the size of the AOF file since the last rewrite.Copy the code

When Redis forks the AOF file, the potential blocking risks include: forking the AOF file and writing by the parent process during the AOF rewrite

A, fork, fork, fork, fork, fork, fork, fork, fork, fork, fork, fork, fork, fork Is in order to avoid disposable copy a lot of memory data to the child process caused by the blocking problem for a long time, but the fork the child process need to copy the necessary data structures, and one of them is to copy the memory page table (virtual memory and physical memory mapping index table), the copy process will consume a large amount of CPU resources, copy to complete the whole process will be blocked before, The blocking time depends on the memory size of the entire instance. The larger the instance, the larger the memory page table, and the longer the fork blocks. After the page table is copied, the child and the parent point to the same memory address space, that is, the child is created, but does not apply for the same memory size as the parent. When do parent and child processes really have memory separation? Realistic copying, as the name implies, is copying the actual data in memory while the write takes place, and in this process, the parent process may block, as described below. A, fork a child process that points to the same memory address space as the parent process. In this case, the child process can perform AOF overwrite and write all data in memory to the AOF file. But at this time the parent is still there will be a flow to write, if the parent process is the key of an existing operation, so this time the parent process will copy the key corresponding to the real memory data, apply for a new memory space, so gradually, and his son began to separation process memory data, gradually have a parent and child process independent memory space. The default memory allocation is 4K. If the parent process operates on a bigkey, re-allocating large chunks of memory takes longer and may cause blocking risks. In addition, if Huge Page (Page size: 2M) is enabled in the operating system, the probability of blocking when the parent process applies for memory will be greatly increased. Therefore, Huge Page must be disabled on the Redis machine. After each fork generates an RDB or AOF rewrite, you can see in the Redis log how much memory the parent process has reallocated.Copy the code

Huge Page is recommended to be turned off when actually using Redis.

A Huge page is friendly to improve THE TLB hit ratio. Because a Huge page can be used to reduce the number of page entries with the same memory capacity, TLB can cache more page entries and reduce the TLB miss overhead. However, this mechanism is not very friendly to a fork-prone system like Redis, especially if Redis has a lot of write requests. Because after fork, the parent process changes the data by copying at write time, the replication granularity is one memory page. If only one 256B of data is modified, the parent process needs to read the original memory page and then map to the new physical address to write. Reading and writing magnifies reading and writing. If the memory page is larger (for example, a large 2MB page), the read and write magnification is more severe, which affects Redis performance.Copy the code