This is the 7th day of my participation in the November Gwen Challenge. Check out the details: The last Gwen Challenge 2021

Mind mapping:

Zero, preface,

RDB(Redis DataBase) : Memory snapshot. The snapshot file will not be lost even if the system is down.

Advantages:

  • RDB is a binary compressed file that takes up little space and is easy to transfer (to SLAver)

  • The main process forks child processes to maximize Redis performance. The main process should not be too large, the amount of data in Redis should not be too large, and the main process is blocked during replication

Disadvantages: Data integrity is not guaranteed and all data changed since the last snapshot will be lost

Consider three key questions:

  1. What data does the snapshot take? This is related to the efficiency of snapshot execution.

    Redis data is stored in memory. To ensure the reliability of all data, Redis performs a full snapshot, that is, records all data in memory to disk.

  2. Can data be added, deleted, or changed when taking snapshots? This is related to whether Redis is blocked and can handle requests properly at the same time.

  3. How often do you take snapshots?

1. What data does the snapshot take for?

Redis data is in memory, and to ensure the reliability of all data, it performs a full snapshot.

That is, all the data in memory is recorded to disk.

RedisTwo commands are provided to generateRDBFile:

  • save: Executed in the main thread, causing blocking.
  • bgsave: Creates a child process dedicated to writingRDBFile, avoid blocking the main thread, this is alsoRedis RDBDefault configuration for file generation.

That of coursebgsaveCommand to execute a full snapshot, which ensures data reliability and avoids data synchronizationRedisPerformance impact.


2. Can the data be added, deleted, or modified when making snapshots?

Of course, data in Redis can also be added and deleted when making snapshots, otherwise the technology is useless.

Let’s take a look at snapshots in Redis: At a point in time, data is saved in memory to disk.

So let’s look at what data operations are:

  • Read operation: The snapshot is not affected because data in the memory is read.
  • Write operations: will affect, stopping write operations for snapshots is definitely not acceptable.

So what technology guarantees snapshot consistency?

Redis uses copy-on-write (COW) technology provided by the operating system to process Write operations while performing snapshots.

In plain English, when data is modified, it is copied, making a copy of the data.

Go deeper:

The child process copies the page table of the main process, so it can read the original data of the main process through the page table mapping, and when new data is written or modified, the main process writes the new data or modified data to a new physical memory address and modifies the main process’s own page table mapping. So, the child reads something like a copy of the original data, and the main thread can modify it as usual.

Here’s an example:

  1. Main process: modify the key pair C, create a new memory generated key pair C’, and reference to the memory address of the key pair C’.
  2. The child process:forkThe main process will make a copy of the main processA page table(Data can be found through page table mapping)
  3. Main process: change key-value pair C, and change the mapping of its page table to page table ‘
  4. Child process: keep the original page table, can find the original data. So it won’t be affected.


3. How often do you take snapshots?

There are many aspects of this investigation, how often to synchronize depends on the actual situation:

  • Read/write Request Ratio
  • Machine memory size
  • Actual number of write requests per minute

Can I take a snapshot every second? Can’t. Here’s why:

  1. Frequent full snapshots also incur two costs:
    • High disk pressure: Full data is frequently written onto the disk
    • Blocking the main process:bgsaveThe child process needs to passforkThe operation is created from the main thread, which blocks.

So how do you do that?

  1. The configuration parameters are executed periodically: for example, if at least one key is changed within 15 minutes, a snapshot is generated
  2. Timing: Trigger commands with scripts

Redis 4.0 proposes a hybrid approach to using AOF logs and RDB memory snapshots:

  1. Memory snapshots are performed at a certain frequency.
  2. Between snapshots, run theAOFLogs record all command operations during this period.




1. Snapshot trigger mode

Redis provides two commands to generate RDB files, save and BGSave.

Trigger mode:

  1. The snapshot rules must be customized
  2. performsaveorbgsaveThe command
  3. performflushallThe command
  4. Performing a master-slave replication (first time)

(1) Configure parameters periodically

Save: Executed in the main thread, blocking;

In redis.conf, configure: Save how many seconds, how much data changed

Funnel design provides performance

save ""        # Do not use RDB storage, cannot master/slave
save 900 1     # indicates that a snapshot is taken if at least one key is changed within 15 minutes (900 seconds).
save 300 10    # indicates that a snapshot is taken if at least 10 keys have been changed within 5 minutes (300 seconds).
save 60 10000  # indicates that a snapshot is taken if at least 10000 keys are changed within 1 minute.
Copy the code

(2) The command is triggered explicitly

Bgsave: Create a subprocess that is dedicated to writing to RDB files, avoiding blocking on the main thread, which is also the default configuration for Redis RDB file generation.

The BGSave child process is generated by the main thread fork and can share all memory data of the main thread. Once the BGSave child process runs, it starts reading the main thread’s memory data and writing it to an RDB file.

Redis uses copy-on-write (COW) technology provided by the operating system to process Write operations while performing snapshots.

Enter the bgsave command on the client

127.0.0.1:6379> bgsave
Background saving started
Copy the code




Second,RDBExecute the process

The execution process is shown as follows:

  1. The Redis parent determines whether it is currently executing save, or a child of bgSave/bgrewriteAof (aOF file rewrite command), and if it is executing the bgsave command returns directly.

  2. The parent process forks (calls OS functions to copy the main process) to create a child process. The parent process is blocked during this replication and Redis cannot execute any commands from the client.

  3. After the parent fork, the bgsave command returns “Background Saving Started” and no longer blocks the parent and can respond to other commands.

  4. The child process creates an RDB file, generates a temporary snapshot file based on the memory snapshot of the parent process, and then atomic replaces the original file. (RDB is always complete)

  5. The child sends a signal to the parent indicating completion, and the parent updates the statistics.

  6. The parent forks the child and resumes work.




Three,RDBFile structure

The file structure is shown as follows:

File structure field description:

  1. The first 5 bytes are fixedREDISstring
  2. 4 bytes"RDB"Version number (noRedisVersion number), currently 9, and 0009 after padding
  3. Secondary fields tokey-valueAs shown in the figure below:

  1. Storing database numbers
  2. The size of the dictionary
  3. overduekey
  4. Main data tokey-valueStorage in the form of
  5. End mark
  6. The checksum, which is to see if the file is corrupted or has been modified.

The dump. RDB file can be viewed using the Winhex tool.

Here, open with vi and enter the command :%! XXD -g 1 Switch to the hexadecimal mode

00000000: 52 45 44 49 53 30 30 30 39 ef bf bd 09 72 65 64 REDIS0009.... Red 00000010:69 73 2D 76 65 72 35 2E 30 2E 35 EF bf bd 0a 72 IS-ver5.0.5.... r 00000020: 65 64 69 73 2d 62 69 74 73 ef bf bd 40 ef bf bd edis-bits... @... 00000030: 63 74 69 6d 65 ef bf bd 01 ef bf bd 5d 75 73 65 ctime....... ] use 00000040: 64 2d 6d 65 6d ef bf bd ef bf bd 18 0a 20 20 20 d-mem........ 00000050: 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 00000060: 20 20 20 20 20 20 20 20 20 20 20 20 20 20 ef bf .. 00000070: bd 0a 20 20 20 20 20 20 20 20 20 20 20 20 20 20 .. 00000080:20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 61 6f 66 2d 70 72 65 61 6d 62 6c 65 aof-preamble 000000a0: ef bf bd ef bf bd ef bf bd 01 70 72 6f 64 75 63 .......... produc 000000b0: 74 5f 73 6b 75 5f 79 79 ef bf bd ef bf bd 34 ef t_sku_yy...... 4. 000000c0: bf bd 25 ef bf bd 40 ef bf bd 37 ef bf bd 0a .. %... @... 7...Copy the code