Follow me for the latest knowledge, classic interview questions and micro service technology sharing

1. Persistence

1.1 Introduction to Persistence

Persistence. Persistence is the mechanism for converting program data between persistent and transient states, i.e. storing data (such as objects in memory) on a permanentstorage device (such as a disk).

1.2 Redis Persistence

Redis is an in-memory database. In order to prevent server data loss after server downtime and server process exit, Redis provides the persistence function, that is, the in-memory data in REDis is persisted to the disk. Redis provides different levels of persistence:

  • RDB persistence: Snapshots can be taken for data storage at specified intervals.
  • AOF persistence: Records each write operation to the server. When the server is restarted, these commands are executed again to restore the original data. AOF saves each write operation to the end of the file using redis protocol. Redis can also rewrite AOF files in the background so that AOF files are not too large.

If AOF persistence is enabled on the server. The server will use AOF files to restore data in preference. The server uses RDB files to restore data only when AOF persistence is turned off

2. RDB persistence

2.1 RDB File Format

An RDB file is a compressed binary file (default filename: dump.rdb) consisting of several parts in the RDB format:

2.2 RDB file persistent creation and loading

During Redis persistence, the RDB program saves the current in-memory database state to a disk file. Upon Redis restart, the RDB program can restore the database state by loading the RDB file.

2.3 Working Mode

When Redis needs to save the dump. RDB file, the server does the following:

  • Redis called forks. It has both a parent and a child.
  • The child process writes the data set to a temporary RDB file.
  • When the child process finishes writing to the new RDB file, Redis replaces the old RDB file with the new one and deletes the old RDB file.

This way of working allows Redis to benefit from copy-on-write mechanisms.

2.4 Creation Mode

SAVE

When the command is executed, the server blocks and rejects the command request sent by the client

    redis> save
Copy the code

BGSAVE

Asynchronous operation, when executing this command, the child process performs the save work, and the server can continue to let the main thread process the command requests sent by the client

 redis>bgsave
Copy the code

Automatically created

Since the BGSAVE command can be executed without blocking the server process, the user can customize the save attribute so that the server automatically executes the BGSAVE command once in a while (Redis is set to have at least M changes to the data set in N seconds). Automatic data set saving operation).

Such as: /* The server made at least one change to the database within 900 seconds */ Save 900 1 /* The server made at least 10 changes to the database within 300 seconds */ Save 300 10 /* The server made at least one change to the database within 60 seconds, At least 10000 database changes */ Save 60 10000Copy the code

The BGSAVE command is executed as long as one of the conditions is met

2.5 Default RDB configuration

################################ SNAPSHOTTING ################################ # # Save the DB on disk: Automatic persistence for a given number of seconds and a given number of write operands to the database. # save <seconds> <changes> # save 900 1 save 300 10 save 60 10000 Yes stop-writes-on-bgsave-error yes # Use LZF to compress string objects for persistence. Rdbcompression yes # Whether to checksum RDB files, usually yes rdbchecksum yes # RDB persistent filename dbfilename dump. RDB # Persistent file storage directory dir./Copy the code

3. AOF persistence

3.1 Introduction to AOF Persistence

AOF persistence records database state by saving write commands executed by the Redis server

AOF persistence:

  1. Append command append: when AOF persistence is enabled, the server appends the end of the server state’s AOF_buf buffer after executing a write command in protocol format.

    Reids >SET KET VAULE // Protocol format \r\n$3\r\nSET\r\n$3\r\nKEY\r\n$5\r\nVAULE\r\nCopy the code

  2. File writing and synchronization SYNC: The Redis server process is an event loop. This file event is responsible for receiving command requests from clients and sending command responses to clients. When append is executed, the server calls flushAppendOnlyFile to determine whether the contents of the AOF buffer need to be written and saved to the AOF file

     redis> SET msg "Ccww"
 redis> SADD persistence "rdb" "aof"
 redis> RPUSH size 128 256 512
Copy the code

3.2 AOF Persistence Policy

The AOF persistence policy (i.e. buffer content written and synchronized to AOF) can be selected by configuring appendfsync:

  • Always: Writes and synchronizes all contents of the aOF_buf buffer to an AOF file. Fsync is performed every time a new command is appended to the AOF file.
  • Everysec (default) : If the last time the AOF was synchronized is now more than one second, the AOF file is synchronized again by writing the entire contents of the AOF_buf buffer to the AOF file, and the synchronization is performed by a dedicated thread.
  • No: Writes all the contents of the AOF_buf buffer to an AOF file, but does not synchronize the AOF file. The timing of synchronization is determined by the operating system (OS).

Efficiency and security of AOF persistence strategy:

  • Always: the slowest, but the most secure, even if there is a failure, persistence will only lose the command data of one event loop
  • Everysec: Speed and security, downtime is only a second of command data loss
  • “No” : writes the fastest, but takes the longest time in a single synchronization. In addition, all command data after uploading and synchronizing AOF files will be lost if downtime occurs.

3.3 AOF rewrite

Because AOF persistence appends executed write commands to the AOF file, the size of the AOF file becomes larger over time as more write commands are added. The large size of AOF files affects the Reids server and even the host server.

To solve the problem of bloated AOF files, Redis provides AOF file rewriting:

  • Generate a new AOF file that does not store any redundant commands that waste space, and the old and new AOF files store the same database state
  • The new AOF file is implemented by reading key-value pairs from the database, and the program does not need to read, analyze, or write to the existing AOF file.
  • To prevent buffer overflows, overwrite lists, hashes, sets, and zsets to record a set of the same commands when the number of constants exceeds the set number.
  • Redis 2.4 can be configured to trigger AOF rewriting automatically, The trigger parameters auto-aof-rewrite-percentage(percentage) and auto-aof-rewrite-min-size(minimum size) are used to trigger aOF rewrites.

AOF rewrite functions:

  • Reduce disk usage
  • Accelerating Data Recovery

The Redis server uses a single thread to process command requests. The aOF_rewrite function was heavily invoked by the Redis server. During the AOF rewrite process, the AOF rewrite was unable to process command requests sent by the client.

  1. The server process can continue processing command requests while the child process does the AOF rewrite
  2. The child process has a copy of the server process’s data, ensuring data security.

In order to solve this problem, Redis uses the AOF rewrite buffer to implement:

  • Execute the command, appending the command to both the AOF buffer and the AOF rewrite buffer
  • When the AOF child process finishes rewriting, it sends a signal to the parent process, and the parent process writes all the contents of the AOF rewrite buffer to the new AOF file. The database state saved in the new AOF file is the same as the current database state of the server.
  • Rename the new AOF file, atomically overwrite the existing AOF file, and complete the replacement of the old and new AOF files.

3.4 AOF Persisting default parameters

# # # # # # # # # # # # # # # # # # # # # # # # # # # # # # APPEND ONLY MODE # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # open appendonly AOF persistent way no AOF persistence file name Appendfilename "appendonly. Aof "# fsync buffer data to disk every second appendfsync everysec # appendfsync no # Whether data is not synchronized to aOF file during rewrite No-appendfsync-on-rewrite no # Indicates the rate at which AOF file rewrites are triggered. Auto-off-rewrite-percentage 100 # Indicates the minimum size at which AOF file rewrites are triggered Aof-load-truncated yes # Whether to turn on the mixing switch aof-use-rdb-preamble yesCopy the code

4. Summary and choice of persistence mode

4.1 Advantages and disadvantages of RDB

The advantages of RDB

  • RDB file is a very compact, it preserved the some point data set, very suitable for a backup data sets, such as you can in each hour to save the data in the past 24 hours, at the same time every day to save data in the past 30 days, even out of the question you can restore to a different version of the data set according to demand.
  • Based on the compactness of RDB files, it is easy to copy data to a remote data center, which is ideal for disaster recovery.
  • When RDB saves RDB files, the only thing the parent process needs to do is fork out a child process. The child process does all the following work. The parent process does not need to do other IO operations, so THE RDB persistence method can maximize the performance of Redis.
  • Compared to AOF, RDB is faster for recovering large data sets.

The disadvantage of RDB

  • RDB is not for you if you want to minimize data loss in the event that Redis unexpectedly stops working, such as a power outage. Although you can configure different save time point (for example, every five minutes and to the operation of the data set has 100 writing), is Redis to complete save the entire data set is a relatively heavy work, you will usually every 5 minutes or more for a complete preservation, one thousand outage in Redis accident, you may lose a few minutes of data.
  • The RDB often forks the child process to save the data set to the hard disk. When the data set is large, the fork process can be very time-consuming and may cause Redis to fail to respond to client requests for some milliseconds. If the data set is large and the CPU performance is not very good, this can last up to a second. AOF also requires forking, but you can adjust the frequency of rewriting log files to improve the durability of the data set.

4.2 Advantages and disadvantages of AOF

Advantages of AOF:

  • Using AOF makes your Redis more durable: Use different fsync policies: no fsync, fsync per second, fsync every time you write. With the default fsync per second policy,Redis still performs well (fsync is handled by background threads and the main thread does its best to handle client requests), and you can lose up to 1 second of data in the event of a failure.
  • The AOF file is an append only log file, so there is no need to write seek. Even if the full write command is not executed for some reason (disk space is full, the write process is down, etc.), you can use the redis-check-aof tool to fix the problem.
  • Redis can automatically rewrite AOF files when they become too large: the rewritten new AOF file contains the minimum set of commands needed to restore the current data set. The entire rewrite operation is absolutely safe because Redis continues to append commands to existing AOF files while creating new AOF files, and the existing AOF files will not be lost even if an outage occurs during the rewrite. Once the new AOF file is created, Redis switches from the old AOF file to the new AOF file and starts appending the new AOF file.
  • AOF files orderly store all writes to the database in the Redis protocol format, so the contents of AOF files are easy to read and parse. Exporting AOF files is also very simple (for example, if you accidentally execute the FLUSHALL command but as long as the AOF file isn’t overwritten, stop the server, remove the FLUSHALL command at the end of the AOF file, and restart Redis, You can restore the data set to the state before FLUSHALL was executed.

AOF faults:

  • AOF files are usually larger than RDB files for the same data set.
  • Depending on the fsync strategy used, AOF may be slower than RDB. Fsync per second performance is still very high under normal conditions, and turning off fsync allows the AOF to be as fast as the RDB, even under high loads. However, RDB can provide more guaranteed maximum latency when handling large write loads.

4.3 How to Choose which Persistence mode to Use?

In general, if you want to achieve data security comparable to PostgreSQL, you should use both persistence features.

If you care deeply about your data, but can still afford to lose it within minutes, you can use RDB persistence only.

There are many users who only use AOF persistence, but this is not recommended: In addition to providing periodic SNAPSHOTS for database backups and restoring data sets faster than AOF, RDB also avoids the aforementioned AOF bugs.

Is everyone still ok? If you like, move your hands to show 💗, point a concern!! Thanks for your support!

Welcome to pay attention to the public number [Ccww technology blog], original technical articles launched at the first time