“This is the sixth day of my participation in the Gengwen Challenge. For more details, see” Gengwen Challenge “.

Persistence of Redis

RDB persistence

Redis will fork and create a child process to persist. Write data to a temporary file that will replace the last persistent RDB file when persistence ends. During this period, the main Redis process will not participate in persistence to ensure high performance of Redis.

Trigger:

  1. If AOF persistence is not enabled when the shutdown command is executed on the client, RDB persistence is triggered.

  2. The redis configuration file has the following default configurations. RDB persistence is triggered when the following conditions are true, and this is done using the save command. But the save command blocks the main process. Generally, using the bgsave command, fork out a child process for persistence operation.

    save 900 1     #after 900 sec (15 min) if at least 1 key changed
save 300 10    #after 300 sec (5 min) if at least 10 keys changed
save 60 10000  #after 60 sec if at least 10000 keys changed
Copy the code

  3. When you run the flushall command to flush data in the memory, persistence is triggered and disks are cleared.

Pros and cons

Advantages: Fast data recovery, suitable for large-scale data recovery, and suitable for cold backup.

Disadvantages: If it is a sudden downtime, the loss of more data. When a large amount of data is generated, the performance of Redis is affected by the persistent generation of snapshot RDB files.

AOF persistence

After AOF is enabled, all commands are appended to the AOF buffer and written to the persistent file of the AOF on disk according to the corresponding write policy. Redis is a log file that records write operations to redis. However, since the AOF files are recorded in a sequence of commands, the AOF files can expand rapidly. When the amount reaches, the rewrite operation will be triggered to achieve the purpose of compression (fork child process to do this).

Redis uses a single-threaded response command, and if each AOF file write command is appended directly to the disk, the performance bottleneck is entirely due to the current disk load. As another benefit of writing to buffer AOF_buf first, Redis can provide multiple strategies for buffer synchronization to the disk, with tradeoffs in terms of performance and security.

Trigger:

  1. AOF is not enabled in the redis profile by default. To enable AOF, open it in the redis profileappendonly no —>appendonly yes. It can also be set when Redis is already running:CONFIG SET appendonly yes, but the Settings will be invalid when Redis restarts.
  2. The configured write policy is triggered. Procedure
    1. appendfsync everysec(Default) : synchronizes commands to the AOF persistent file once per second, which is highly efficient and may cause loss of data for 1 second.
    2. appendfsync no: Never synchronizes, just hand over the data to the operating system. Faster, less secure methods. Typically, Linux will use this configuration to refresh data every 30 seconds, but this depends on the precise tuning of the kernel.
    3. appendfsync alwaysAppend data to AOF file every time a change is triggered. Low efficiency, but very safe.

Override mechanism:

  1. Default Settings. For example, after the last AOF rewrite, it was 128mb. Then it will continue to write AOF logs of 128MB, and if it grows by more than 100% of the previous 256MB, it may trigger a rewrite. However, at this point, it needs to compare with min-size and 64MB, and only when 256MB > 64MB will it trigger the rewrite.

    To reduce the number of overwrites, the following parameters need to be increased, but this depends on the amount of data stored in redis.

    auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb
Copy the code

Advantages and disadvantages:

Advantage: To better protect data against loss, generally, AOF performs the fsync operation on a background thread every second (default synchronization policy) to lose up to one second of data.

AOF log files are written in Append-only mode, so there is no disk addressing overhead, the write performance is very high, and the files are not easily damaged, and even if the tail of the file is damaged, it is easy to fix (using redis-check-aof –fix).

Even if the log file is too large and a background rewrite operation occurs, the read and write of the client will not be affected. When you rewrite logs, you compress the guidelines and create a minimal log to recover data from. When a new log file is created, the old log file is written as usual. When the new log file after the merge is ready, the new log file and the old log file are exchanged.

Commands for AOF log files are logged in a very readable way, and this feature is ideal for emergency recovery in case of catastrophic misdeletes. For example, if you use the flushall command to flush out all data, you can immediately copy the AOF file, delete the last flushall command, and put the AOF file back. In this case, you can automatically restore all data in the file

Disadvantages: AOF log files are generally larger than RDB data snapshot files for the same data.

When AOF is enabled, write QPS will be lower than RDB because AOF is configured to fsync log files once per second. However, once fSYNC per second is still high performance.

The more complex command log /merge/ playback approach, such as AOF, is more vulnerable to bugs than the RDB-based approach of persisting complete data snapshots one at a time. However, AOF is designed to avoid the bugs caused by the rewrite process, so each time the rewrite is not based on the old instruction log merge, but based on the data in memory at the time, the instructions are rebuilt, which is much more robust.

How to choose between RDB and AOF

  1. Don’t just use RDB because that will cause you to lose a lot of data

  2. Don’t just use AOF, either, because that has two problems. First, you use AOF for cold backup, and there is no RDB for cold backup, so the recovery is faster. Second, RDB is more robust by simply generating a snapshot of the data each time, avoiding the bugs of complex backup and recovery mechanisms like AOF

  3. The persistence mechanism of AOF and RDB is integrated. AOF is used to ensure that data is not lost and is the first choice for data recovery. RDB can be used for different degrees of cold backup, and can also be used for fast data recovery when AOF files are lost or corrupted

So adults, all of them!

AOF and RDB work simultaneously:

  1. If RDB is snapshotting, then Redis will not execute AOF rewrite. If Redis executes AOF rewrite, then RDB snapshotting will not be executed
  2. If the RDB is executing snapshotting and you run the BGREWRITEAOF command, the AOF command is executed only after the RDB snapshot is generated
  3. With both RDB snapshot files and AOF log files, when Redis restarts, the AOF is preferred for data recovery because the logs are more complete

Redis Persistent file loading process

  1. First, it checks whether AOF is enabled. If an AOF file exists, it loads the file directly
  2. If the AOF file is not found, it will be started directly and the RDB file will not be loaded
  3. If AOF is not enabled, the SYSTEM loads the RDB file and uses the RDB to recover data

Data Backup Scheme

  1. Write the crontab scheduling script to back up data
  2. Copy a backup copy of the RDB to a directory every hour and keep only the backups in the last 48 hours
  3. Keep a copy of the RDB of the current day to a directory every day, and only keep the backup of the latest month
  4. Every time you copy a backup file, you delete the old one (whether to delete the old one depends on the actual situation).
  5. Send a copy of all the data on the current server to a remote cloud server (either for backup or local storage) every night.

Run the redis_rdb_backup_account. sh command to backup data every hour

#! /bin/sh 

cur_date=`date +%Y%m%d%k`
rm -rf /usr/local/redis/snapshotting/$cur_date
mkdir /usr/local/redis/snapshotting/$cur_date
#Backup stored locally
cp /var/redis/6379/dump.rdb /usr/local/redis/snapshotting/$cur_date
#When backup is sent to the backup server, no password is required for login
#SCP - rq/var/redis / 6379 / dump. The RDB [email protected]: / opt

#Delete the RDB files generated in the last 48 hours
del_date=`date -d -48hour +%Y%m%d%k`
rm -rf /usr/local/redis/snapshotting/$del_date
Copy the code

Run this script every hour: 0 0 * * *? sh /usr/local/redis/shell/redis_rdb_backup_hourly.sh

Run the redis_rdb_backup_daily.sh command to backup data once a day

#! /bin/sh 

cur_date=`date +%Y%m%d`
rm -rf /usr/local/redis/snapshotting/$cur_date
mkdir /usr/local/redis/snapshotting/$cur_date
cp /var/redis/6379/dump.rdb /usr/local/redis/snapshotting/$cur_date

del_date=`date -d -1day +%Y%m%d`
rm -rf /usr/local/redis/snapshotting/$del_date
Copy the code

Execute the backup script once a day: 0 0 0 * *? sh /usr/local/redis/shell/redis_rdb_backup_daily.sh

Data Recovery Scheme

(1) If the Redis process is suspended, restart the Redis process and directly recover data based on the AOF log file

(2) If the machine where the Redis process is running is down, then restart the machine, try to restart the Redis process, and try to restore data directly based on the AOF log file. The AOF is not damaged and can be restored directly based on the AOF. AOF append-only, sequential write, if the AOF file is damaged, then use redis-check-aof fix

(3) If the latest AOF and RDB files in Redis are lost/damaged, then you can try to restore data based on a copy of the latest RDB data currently on the machine.

Current latest AOF and RDB files are lost/damaged to unrecoverable, generally not machine fault, artificial. Then delete the damaged files. Go to the backup server to find the latest RDB backup, hour level backup is ok, the hour level must be the latest backup, copy to redis, you can restore to a certain hour data

Stop redis, close aof, copy RDB backup, restart Redis, confirm data recovery, directly modify the redis configuration, open aof, redis will write the corresponding data log in memory to the Aof file

At this point, the aof and RDB data files are synchronized

If the actual parameters in the configuration file are not modified persistently, stop redis again, manually modify the configuration file, open the aof command, and restart Redis again

(4) If all the RDB files on the current machine are damaged, then pull the latest RDB snapshot from the remote cloud service to restore the data

(5) If significant data errors are found, for example, all the data are polluted by the online program in an hour, and all the data are wrong, then an earlier time point can be selected to restore the data