state
Redis is an in-memory database that runs in memory and may contain multiple databases as well as multiple key-value pairs in the database. These non-empty databases in Redis and their key-value pairs are called states.
Redis data is stored in memory. If you do not save the state of the database in memory to the hard disk, then when the Redis process ends, all the contents of the database will not be saved, and the next Redis startup will be an empty database.
To solve this problem, Redis uses persistence, which saves data in Redis to hard disk for persistent storage and prevents accidental data loss.
RDB persistence
RDB persistence can be stored manually using the save command, or the server can be configured to automatically perform the storage periodically.
RDB reads the state of Redis to get the information currently stored in the database, and then compresses it into an RDB file. When the Redis database data needs to be restored, the saved information is read from the RDB file to restore the data.
Create an RDB file
There are two commands to create RDB files, one is SAVE and the other is BGSAVE.
- The Save command blocks
RedisProcess, so fromSaveNone of the user’s commands can be executed.
127.0.0.1:6379 > save OKCopy the codeBGSaveCommand willforkA child process completes the RDB file creation, and the server process continues to execute the command request.
127.0.0.1:6379> bgsave
Background saving started
Copy the codeIn fact, the job of creating the RDB is done by the rdbSave() function, which is simply called at a different time from the BGSAVE command. The pseudocode is as follows:
function SAVE(a){
rdbSave(); // Call directly, blocking
}
function BGSAVE(a){
// Create a child process
pid = fork();
if(pid==0) {
// pid=0 indicates that the child process is used for execution
// So the main process is not blocked
rdbSave();
} else if (pid > 0) {# The parent process continues to execute the command sent by the user.... }}Copy the codeAutomatically save RDB files
Since BGSAVE does not block user requests during execution, you can set commands in the conf file to make Redis automatically save the current database state and generate RDB files.
The default parameters for running BGSAVE in conf are as follows:
save 900 1 BGSAVE is executed if there is at least one change to the database within 900 seconds
save 300 10 BGSAVE is executed if the database has at least ten changes within 300 seconds
save 60 10000 BGSAVE is executed if the database has at least 10000 changes within 60 seconds
Copy the codeYou can configure multiple Saves in the CONF file, and BGSAVE is automatically executed if one of them is met.
Set save conditions
During server startup, read the Save configuration in the CONF file, generate an array that holds all the save criteria,
struct saveparam *saveparams;
struct saveparam{
// save Number of seconds configured
time_t seconds;
// save the number of times the configuration changes
int changes;
}
Copy the codeSuppose the configuration is:
save 900 1 BGSAVE is executed if there is at least one change to the database within 900 seconds
save 300 10 BGSAVE is executed if the database has at least ten changes within 300 seconds
save 60 10000 BGSAVE is executed if the database has at least 10000 changes within 60 seconds
Copy the codeThe result is:
Dirty counter +lastsave attribute
The dirty counter records how many times the server has changed the database state, including adding, deleting, and updating operations, since the last SAVE or BGSAVE was successfully executed.
Lastsave records the UNIX timestamp from the last time the save was performed.
Check that the conditions for BGSAVE execution are met
Redis has a periodic function serverCron that executes every 100ms by default. One of its jobs is to check that the save option in the CONF file is met, and if so, BGSAVE is executed.
function serverCron(a){...// Iterate to perform the check
for(saveparam param:server.saveparams){
int diff = now() - lastsave();// The interval since the last update
if(diff >= param.seconds && dirty >= param.changes) { BGSAVE(); }}... }Copy the codeThe program iterates through all the save parameters and, if any of them are satisfied, performs the BGSAVE operation.
RDB summary
RDBThe file can restore the serverRedisAll key-value pair data for the database.SAVECommand execution of save operations blocks the server.BGSAVECommands are executed in the background without blocking the server.- in
confSet in thesaveParameter to configureBGSAVEThe execution conditions of.
advantage
This is the only backup file in redis.
Suitable for large-scale data recovery. Maximize performance.
Persistence is done by forking the child process while the main process continues to process the client’s requests.
Compared with THE AOF mechanism, if the data set is large, data recovery is faster and more efficient at startup.
disadvantage
If the server suddenly goes down, data that has not been persisted will be lost.
Do not use this method if you have high requirements on data integrity.
As a result of forking a process that is identical to the current process and contains all the data for the current process, doubling the amount of data in memory affects performance.
AOF
RDB stores the state of the database, while AOF stores all writes performed by Redis (reads have no effect on the state of the database and do not need to be logged).
When the server executes:
127.0.0.1:6379 >setMSG VMS OK 127.0.0.1:6379> sadd schools Qinghua Beida USTC (integer) 3 127.0.0.1:6379> Rpush numbers 12 3 (integer) 3
Copy the codeThe AOF file holds:
* 2$6
SELECT
The $10 * 3$3
set
$3
msg
$3
vms
*5
$4
sadd
$7
schools
$7
qinghua
A $5
beida
$4
ustc
*5
A $5
rpush
$7
numbers
The $1
1
The $1
2
The $1
3
Copy the codeThese commands are the write commands that have just been executed. When the Redis server is started again, the database state can be restored from the AOF file.
AOF append implementation
- Command to add
When the AOF command is enabled, the server adds a write command to the end of the AOF_BUF buffer in a specified format after executing it.
- Write and synchronize
The Redis server is a loop of events in which
- File event: accepts a command request from a client
- Time events: Execute commands that need to be executed periodically
Server end of the event a file at a time, because of the possible execution write command, it will make some content is appended to the AOF_BUF buffer (setting buffer in order to improve the efficiency of file writing, is the purpose of staging will need to write data to a memory buffer, the buffer is full or over a certain time when writing data to the hard disk), So every time the file event ends, the flushAppendOnlyFile function is called to determine that the contents of the AOF_BUF buffer should be added to the AOF file. Pseudo code:
Function eventLoop(){while(true){processFileEvents(); ProcessTimeEvents (); FlushAppendOnlyFile (); flushAppendOnlyFile(); }}Copy the codeFlushAppendOnlyFile determines when to execute AOF based on the conf configuration.
# appendfsync always # AOF_BUF all content is added to the AOF file
appendfsync everysec Add AOF_BUF files to AOF files every second
# appendfsync no # When to join is decided by the operating system
Copy the codeappendfsync alwaysIf there is a sudden power outage, the command that has been executed by the event loop will be lost at most once, because it will be written to AOF after each execution, but since the AOF file is written each time, it is slightly less efficient.appendfsync everysecThe command executed in a maximum of one second is lostappendfsync noNumber of lost commands = number of commands executed since last written to AOF.
Data reduction
Redis reads AOF files as follows:
- Create a dummy client because
RedisCommands can only be executed in the client context and loadedAOFThe commands in the file are obtained fromAOFInstead of coming from the client, create a dummy client to executeAOFCommand in. - Read a write command from AOF
- The pseudo client executes this command
- Repeat 2-3 until the command in the AOF file is executed.
AOF rewrite
If the AOF file size is too large, it usually affects server performance, and there are a few things you can do to mitigate this. For example:
127.0.0.1:6379> RPUSH list A B # [A,B]
(integer) 2
127.0.0.1:6379> RPUSH list C # [A,B,C]
(integer) 3
127.0.0.1:6379> RPUSH list D E # [A,B,C,D,E]
(integer) 5
127.0.0.1:6379> LPOP list # [B,C,D,E]
"A"127.0.0.1:6379 > LPOP list# [C,D,E]
"B"
127.0.0.1:6379> RPUSH list F G # [C,D,E,F,G]
(integer5)Copy the codeAll of this can be done with one command
RPUSH list C,D,E,F,G
Copy the codeThe AOF operation that implements this functionality is override.
Overridden implementation
AOF rewriting does not require reading analysis of existing AOF files, but directly reads the state of the server. After executing these commands:
127.0.0.1:6379> RPUSH list A B # [A,B]
(integer) 2
127.0.0.1:6379> RPUSH list C # [A,B,C]
(integer) 3
127.0.0.1:6379> RPUSH list D E # [A,B,C,D,E]
(integer) 5
127.0.0.1:6379> LPOP list # [B,C,D,E]
"A"127.0.0.1:6379 > LPOP list# [C,D,E]
"B"
127.0.0.1:6379> RPUSH list F G # [C,D,E,F,G]
(integer5)Copy the codeIf you want to minimize the size of the AOF file, you can read the list key directly from the database and get: C,D,E,F,G.
So the AOF rewrite can be summarized as:
- Read the current value of the key from the database
- With a command (e.g
RPUSH list C,D,E,F,G) to record the key-value pair instead of the previous multiple commands.
The whole process:
function aof_rewrite(newfilename){
f = create_file(newfilename); // New aOF file
for(Database db: server.Databases){
if(db.empty) continue; // If the database is empty, skip it
for(Key k:keys){ // iterate over all keys
if key.is_expired() continue;
f.write(k); // Write the key and its corresponding data
}
}
f.close();
}
Copy the code- AOF rewriting is done through child processes
Because the child process is used, when the child process overwrites the AOF, the server may process new data and change the state of the database after the rewriting of the AOF. In this case, the current database state may be inconsistent with the state of the server recorded after the rewriting of the AOF. To solve this problem, Redis sets up an AOF rewrite buffer, which is used after the child process. After executing a write command, Redis sends the write command to both the AOF buffer and the AOF rewrite buffer. This ensures that: From the time the child process is created, all write commands are logged into the AOF rewrite buffer.
When the child is finished, it sends a signal to the parent, which calls the signal handler and executes:
- Write the contents of the AOF rewrite buffer to the written AOF to ensure data consistency.
- Rename the new AOF file, atomically overwrite the existing AOF file, and complete the replacement of the old and new AOF files.
Advantages and disadvantages
advantage
Improved data security and integrity.
By default, data can be lost at most one second per second.
Background asynchronous, very efficient.
The redis-check-aof –fix mechanism is provided to ensure data correctness.
disadvantage
Compared with RDB files, AOF files are much larger and the data recovery efficiency is low.
Although AOF is background asynchronous fsync to append log files, both synchronization per second and synchronization per modification cost performance.
If both persistence methods are enabled, redis restarts using AOF files to recover data. Because AOF files hold more complete data than RDB, RDB is at greater risk of losing data.
reference
- Redis Design and Implementation
- Juejin. Cn/post / 684790…