This is the second day of my participation in the August Text Challenge.More challenges in August

preface

Redis is a NoSQL database based on memory. Once a server outage occurs, data will be lost. To prevent data loss, Redis uses persistence technologies, RDB and AOF respectively.

RDB persistence

RDB persistence refers to writing snapshots of data sets in memory to binary files on disk (dump.rdb) at regular intervals. So how does Redis trigger persistent snapshots?

1.1 save

When one of the clients executessaveCommand, Redis will proceedSnapshot persistenceRedis is blocked and will not respond to the client’s request until the persistence process is complete. When the persistence is complete, theNew RDB filereplaceOld RDB file.Obviously, blocking has its drawbacks. If Redis has a large amount of data in memory, the persistence process will consume a large amount of time, during which Redis will not be able to respond to clients, degrading the user experience.

1.2 bgsave

When the client executes the BGSave command, Redis forks a child process that is the same as the main process and cannot respond to the client during the fork. When fork is complete, the in-memory data is snapshot persisted through the child process, while the main Redis process can still respond to the client. When the persistence is complete, the old RDB file is replaced with the new one.

Obviously, do it manually every timebgsaveCommands, or very troublesome, after all, we can not always wait on the server to execute commands. So that leads to automatic triggeringbgsaveCommand.

At least 10 write commands are executed in 1 # 300s. At least 10000 write commands are executed in 10 # 60s. Save 60 10000Copy the code

Related parameters

If the RDB file encounters an error, the main process will accept writes. Yes: stop writing, if no: redis continues to provide services. Stop-writes -on-bgsave-error yes # Whether to compress snapshot mirroring. Yes: compression, but requires some CPU consumption. "No" : not compressed, requiring more disk space. Rdbcompression yes # a CRC64 check is placed at the end of the file. There will be a 10% or so performance drop when storing or loading RBD files. To maximize performance, you can turn this off. Rdbchecksum yes # Snapshot filename dbfilename dump. RDB # Snapshot directory dir /var/lib/redisCopy the code

1.3 the advantages and disadvantages

(1) RBD file is a binary file, is a snapshot of the memory data set, storage is very compact, the file is small. So good for backup and disaster recovery. Faster than AOF when recovering large amounts of data. (2) RDB files are binary files, so it is suitable for master/slave replication. (3) If the server goes down, data will be lost for a period of time.

AOF persistence

To compensate for the RDB’s loss of data over time, Redis introduced another approach, aOF. The Redis server appends each write command to the end of the file using the write function. When Redis restarts, the in-memory database data is rebuilt by re-executing the write commands saved in the file.

  1. The client issues the bgrewriteaof command
  2. Fork a child process that is the same as the main process.
  3. The parent process continues to process client requests except for writing the write command to the original AOF file. The received write command is also cached in the AOF rewrite buffer. This ensures that there will be no problem if the child process rewrite fails.
  4. The child process writes to the new AOF file according to the command merge rules based on the memory snapshot.
  5. When a child process writes a memory snapshot to a temporary file, it sends a signal to notify the parent process. The parent process then writes cached write commands to the temporary file as well.
  6. The parent process can now replace the old AOF file with a temporary file and rename it, and subsequent write commands will start appending to the new AOF file.
Appendonly yes # specify AOF filename appendfilename appendonly. # appendfsync always # force write to disk every time a write command is received, similar to MySQL sync_binlog=1, # is the safest. However, this mode has the slowest speed and is not recommended. Appendfsync everysec # Force writes to disk once per second to balance performance and persistence. Recommended. # appendfsync no # completely dependent on OS writes, typically around 30 seconds, best performance but persistence # least guaranteed, not recommended. When the log is overwritten, it does not append to the command, but only puts it in the buffer to avoid collisions with the command append. Setting this parameter to yes means that rewrite will not be fsynced and will be stored in memory. It is recommended that yes no-appendfsync-on-rewrite yes # automatically start a new log rewrite process when the current AOF file size is twice the size of the AOF file obtained from the last log rewrite. Auto-aof -rewrite-percentage 100 # Specifies the minimum value for the current AOF file to start a new log rewrite process, avoiding frequent rewrites due to the small file size when Reids is started. auto-aof-rewrite-min-size 64mbCopy the code

Normally we configure appendfsync everysec to persist once per second. If the server is down, 1 second of data will be lost.

2.1 File Rewriting

Because the AOF file holds some write commands, the problem is that the file is too large. How to solve this problem? That is, get rid of some useless commands. Redis regenerates a new AOF file (no useless commands) from the data state, replacing the old AOF file with the new AOF file.

// Overwrite the previous file incr k11  
set  k2 a  
set  k2 b  
incr k1 2  
incr k1 3 
set  k2 c  
del  k3  
...  
incr k1 100// Rewrite the file incr k1100
set  k2 c 
Copy the code

Multiple commands can be combined into one command. To prevent client buffer overflow caused by a large command, operations such as list,set,hash, and zset are divided into multiple commands with 64 elements bound.