00 preface

If the Redis server is down and all the data in the memory is lost, how to recover the data? Some people say that it is easy to read from the MySQL database.

This method has two problems. First, it is risky to frequently access the MySQL database. Second, it is slow. From the interface, reading from MySQL is not as fast as reading from Redis.

Far brother, far brother, what can we do? Teach me.

I use the middle finger to touch small fat of the next bar, say: fool, we can do persistent ah. The persistence of Redis is divided into two kinds, one is AOF, the other is RDB. Here, sit on your brother’s lap, and I’ll give you a good lecture.

As usual, here’s a brain map:

0.1 What is Persistence?

Persistence is the saving of data (such as objects in memory) to a permanent storage device (such as a disk). The main application of persistence is to store objects in memory in a database, in disk files, XML data files, and so on. Persistence is a mechanism for converting program data between persistent and transient states.

0.2 past wonderful

1, xiao Pang asked me: how is the select statement executed?

2, my girlfriend asked me: what is the principle of MySQL index?

3, xiao Pang asked me: what is the use of MySQL log?

4. Lao Wang asked me: What is the principle of MySQL transaction and MVCC?

5, my girlfriend asked me: what is the lock mechanism of MySQL?

Six thousand words long, 38 figure explosive liver Redis foundation!

01 How to understand Redis single thread?

It must be stated that Redis single thread means that Redis network IO and key/value pair reads and writes are completed by one thread (main thread), which is also the main process of Redis to provide external key/value storage service. But other Redis functions, such as persistence, asynchronous deletion, and cluster data synchronization, are actually performed by additional threads.

Why is 1.0 Redis fast?

Based on the memory

  • Data is stored in memory, reducing some unnecessary I/O operations, operation speed is very fast.

Efficient data structures

  • The underlying data structure supports different data types and supports Redis to store different data.
  • Different data structures are designed to minimize the time complexity of data storage.

Rational threading model

  • The I/O multiplexing model listens for multiple client connections simultaneously;

  • Single thread does not need to perform context switch during execution, reducing time consumption.

02 AOF Persistence

AOF (Append Only File) persistence records database state by keeping write commands executed by the Redis server. That is, whenever Redis executes a command that changes the data SET (such as SET), the command is appended to the end of the AOF File.

Alter the redis.conf configuration file (default: appendonly No) and set no to yes to enable AOF persistence:

appendonly yes
Copy the code

You can also run the following command on the client, but the command will become invalid after the Redis server restarts.

192.16817.101.:6379> config set appendonly yes
OK
Copy the code

The implementation of AOF persistence can be divided into two steps: command append (append) and file write back to disk.

2.0 Command Addition

When AOF persistence is enabled, Redis appends a write command to the end of the aof_buf buffer of the server state that has not yet been written to appendone.aof.

2.0.1 AOF format

AOF stores Redis write commands, such as set testkey testValue, as shown in the following figure:

Where, “*3” means that the current command has three parts, each of which begins with + number ∗∗, followed by a specific command, key, or value. Here, ∗∗ numbers ∗∗ denotes how many bytes there are of commands, keys, or values in this section. For example, ∗∗+ the number ** begins, followed by a specific command, key, or value. Here, the ** number ** indicates the total number of bytes of the command, key, or value in this section. For example, **+ the number ∗∗ begins, followed by a specific command, key, or value. Here, ∗∗ numbers ∗∗ denotes how many bytes there are of commands, keys, or values in this section. For example, ∗∗3 set means that this part has 3 bytes, which is the set command.

2.0.2 What are the advantages and disadvantages of post-log writing?

AOF logs are recorded after a write. In MySQL, redo logs and binlogs are recorded before a write. The writing process looks like this:

What are the advantages of writing it?

  • The command syntax is not checked when AOF is recorded. After AOF is written, only the commands that are successfully executed are recorded. (Avoid saving the wrong command, the recovery of the end of the calf)
  • Recording after execution does not block the current write operation

What are the defects after writing?

  • Response data will be lost if the crash occurs after executing a command before logging.
  • The writing of AOF is handled by the main thread, and if it takes a long time to write, it will affect the main thread to process subsequent requests.

Did you notice? The two defects after the write are both AOF writing to disk, let’s see how it is written.

2.1 AOF Writes to disks

AOF provides three options, which are the three optional values of the AOF configuration item appendfsync.

  • Always, synchronous write back: Logs are written back to the disk synchronously immediately after each write command is executed.

  • Everysec (default), write back per second: after each write command is executed, the log is written to the memory buffer of the AOF file first, and the contents of the buffer are written to disk every second.

  • No, operating system-controlled write back: After each write command is executed, the log is first written to the memory buffer of the AOF file. The operating system decides when to write the contents of the buffer back to disk.

2.1.0 Advantages and disadvantages of the three strategies

None of the three write back strategies can solve the problem of preventing main thread blocking and reducing data loss. The main reasons are:

  • Always (synchronous write back) basically does not lose data, but it has a slow drop operation after each write command, affecting the main thread performance;
  • No (operating system control write back) after the buffer is written, continue to execute the following command, but the time of the disk is not in the hands of Redis, as long as the AOF record is not written back to the disk, once the corresponding data will be lost;
  • Everysec (Write back per second) uses the write back frequency once a second to avoid the performance overhead of Always. Although it reduces the impact on system performance, commands that do not fall in the last second will still be lost in the event of a breakdown.

To sum up: want high performance, choose No strategy; For high reliability, select the Always policy. Allow for a bit of data loss, but don’t expect performance to suffer too much, and choose the Everysec policy.

2.2 AOF Recovers data

Anyway, look at the picture:

2.3 AOF rewrite

I don’t know if you noticed? AOF files constantly append write commands to the end of the file to record database state. Write commands are increasing, and AOF is getting bigger and bigger.

Some commands update the same data multiple times, but it can be combined into the same command. For example, LPUSH makes six changes to the list data, but AOF only needs to record the last change. Because when the log is restored, only the last change command needs to be executed.

To handle this, Redis provides a rewriting mechanism for AOF. Its variable one function merges six write commands into one. As follows:

If some of your keys have hundreds or thousands of changes, the space savings from overwriting can be significant.

2.3.1 Triggering rewrite

There are two ways to trigger, one is to call the command BGREWRITEAOF; One is to modify configuration file parameters.

Style #192.16817.101.:6379> BGREWRITEAOF
Background append onlyFile started # Mode 2 Auto-aof-rewrite-percentage 100The ratio of the current AOF file size to the last AOF file size overridden auto-aof-rewrite-min-size 64MB # The minimum size of the fileCopy the code

2.3.2 Rewrite steps

  1. Create child process for AOF rewrite
  2. Appends the client’s write command to the AOF rewrite buffer
  3. When the child process completes the AOF rewrite, it sends a signal to the parent process
  4. After receiving the signal, the parent process writes all the contents of the AOF rewrite buffer to the new AOF file
  5. Rename the new AOF file to override the existing AOF file

2.4 Related Configurations

# Enable the AOF function appendonlynoAppendfilename "appendonly. AOF "# three ways to write an AOF file appendfsync always appendfsync everysec appendfsyncnoWhether to continue writing AOF files when overwriting AOFno-appendfsync-on-rewrite no# Autorewrite AOF file condition auto-aof-rewrite-percentage 100# % auto-aof-rewrite-min-size 64MB # size # Whether to ignore the last potentially problematic instruction aof-load-truncated yes
Copy the code

2.5 the advantages and disadvantages

advantages

  1. AOF files are highly readable and easy to analyze
  2. If the AOF file is too large, it will be overwritten automatically
  3. The append form is written to the end without having to read the file again

disadvantages

  1. For the same amount of data, AOF is generally larger than RDB
  2. AOF recovery requires a replay command, and the recovery speed is slow
  3. Depending on the Fsync policy, AOF may be slower than RDB

03 RDB Persistence

RDB persistence refers to saving a snapshot of Redis data generated in memory in a binary file named dump. RDB (file name can be changed) when save, BGSave is entered on the client side or the configuration file automatically saves a snapshot.

3.1 the save command

The save command blocks the Redis server process until the RDB file is created, and the server cannot process any command requests while the Redis server is blocked. Type Save on the client side

127.0. 01.:6379> save
OK
Copy the code

After the snapshot is generated, a message is displayed indicating DB saved ondisk.

1349:M 25 Apr 13:16:48.935 * DB saved on disk
Copy the code

3.2 bgsave command

When bgSave is executed, the main thread creates a subprocess dedicated to writing to the RDB file, avoiding blocking by the main thread, which is the default configuration for Redis RDB file generation.

127.0. 01.:6379> bgsave
Background saving started
Copy the code

The SAVE command was rejected during the execution of the bgsave command. Cannot execute two BGSAVE commands at the same time; You cannot run BGREWRITEAOF and BGSAVE at the same time.

3.3 Write Data during BGSave

When bgSave is executed, the Redis main thread reads and writes data normally. Read operation, the main thread and bgSave child thread do not affect each other; During Write operations, Redis uses copy-on-write (COW) technology to generate a Copy of the modified data. The BGSave child thread then writes the replica data to the RDB.

For example, during BGSave, the main thread modifies the key value pair C as follows:

But what if there is an outage in the process? For example, a snapshot is taken at time T0, and another snapshot is taken at time T0+t. However, after the main thread modified data 5 and 9 in time t, Redis broke down and RDB did not record the modified data.

When Redis restarts to restore data, data 5 and 9 will be lost and cannot be restored.

How do you do that? We need to remember that those numbers have been modified.

3.4 Mixed Persistence

As shown in the figure below, it takes extra space to record the data modified at time T, and Redis is an in-memory database, so space is very precious. Therefore, it is not advisable to record directly to memory.

The less memory overhead is to log incremental writes in T time to the AOF log, which preserves quick RDB recovery without taking up extra space.

As shown in the figure, changes at T1 and T2 are recorded in AOF logs. When the second full snapshot is taken, the AOF logs are cleared, because the changes at this time are recorded in the snapshot, and AOF logs are not needed for recovery.

Fortunately, Redis 4.0 began to provide this RDB + AOF persistence mode. The enabled configuration item is AOF – use-rDB-preamble yes, which needs to be realized with the rewriting mechanism of AOF.

# Enable mixed persistence redis> config set aof-use-rdb-Preamble yes OK # AOF rewrite redis> BGREWRITEAOF
Background append only file rewriting started
Copy the code

Before a second full snapshot is taken, the format looks like this: the first half is RDB and the second half is AOF incremental logs. If it goes down at this time, restore data directly with appendone.aof.

3.5 Advantages and disadvantages of RDB

advantages

  1. Binary data, recovery is faster than AOF
  2. RDB bgSave mode main thread does not block

disadvantages

  1. Redis will lose some data when it goes down unexpectedly (hybrid persistence can be resolved)
  2. When the amount of data is large, the process of fork is very time-consuming, and the child process will be blocked, during which Redis cannot respond to the client’s request.

04 How to Choose?

  1. If data cannot be lost, use both memory snapshot and AOF.
  2. If minute-level data loss is allowed, only RDB can be used.
  3. If only AOF is used, everysec’s configuration option is preferred because it strikes a balance between reliability and performance.

05 Data Restoration process

07 summary

This article mainly explains the principle of Redis AOF and RDB persistence and the advantages and disadvantages of both. After comparing the two, I also summarized the process of Redis hybrid persistence. Finally, I give you some advice on how to choose a persistence solution.

Full text nearly 5000 words, 12 pictures, I hope to help you. Ok, so that’s doggie’s summary of Redis persistence. Thanks to all the tech community leaders for their efforts, especially geek time, which is awesome. If I have seen further, it is by standing on your shoulders. Hope you found this article helpful, and we’ll see you in the next article

08 Big factory interview questions & e-books

If you read here and like this article, please click on it.

I don’t know what to give you. Just a few hundred ebooks and the latest 2021 interview materials. Wechat search JavaFish reply ebook send you 1000+ programming ebook; Reply to the interview and send some interview questions; Reply 1024 send you a complete set of Java video tutorials.

The interview questions have answers, as follows: If you need them, come and take them, absolutely free, no tricks.

Shoulders of giants

  • Redis Design and Implementation
  • juejin.cn/post/6844903648976175118
  • time.geekbang.org/column/article/272852
  • blog.csdn.net/wsdc0521/article/details/106765809
  • blog.csdn.net/weixin_33810006/article/details/90394921