Section 1: RDB of Redis persistence

1. General introduction to Redis persistence operation

The official website: www.redis.io

Redis is an in-memory database. If you do not save the in-memory database state to disk, the database state in the server will disappear once the server process exits. So Redis provides persistence

Redis saves internal data as a “backup” in the form of files on the hard disk. After the server restarts, the data on the hard disk will be automatically restored to the memory (Redis). The process of saving data to the hard disk is called “persistence” effect.

Redis has two types of persistence:

1. One is “(RDB) (Redis DataBase)” snapshot persistence: This operation is enabled by default. All data in Redis memory in a certain period is directly saved in a copy and stored in the hard disk (written as a file). If there is too much data (10-20GB), this operation is not suitable for frequent operation.
2. One is “AOF Persistence” (Append Only File), in which every “write” (add/modify/delete) performed by the user is backed up to a File, and the specific write is executed when the data is restored. Enable AOF persistence (this empties data inside Redis, preferably before Redis uses it). Simply put, write all content data to a file in the form of an append.

2. What is RDB

Write a Snapshot of the data set in Redis memory to disk at a specified time interval (backup, i.e. Redis is the RDB by default, and generally you do not need to change this configuration

How is the backup performed?

Redis forks a separate subprocess for persistence. It writes data to a temporary file (dump. RDB) and replaces the last persistent file with the latest temporary file when the persistence process is complete. The file is stored on disk.

During the entire process, the main process does not perform any IO operations, which ensures extremely high performance. If large-scale data recovery is required and the integrity of data recovery is not very sensitive, RDB is more efficient than AOF. The downside of RDB is that data can be lost after the last persistence.

The Fork?

Fork copies the same process as the current one. All data values of the new process (variables, environment variables, program counters, etc.) are the same as those of the original process, but are a new process and a child of the original process

In Linux, fork() produces a child that is exactly the same as the parent, but will later exec the system call. For efficiency, Linux introduced the copy-on-write technique.

Generally, the parent process and child process share the same physical memory. Only when the contents of each segment of the process space change, the parent process copies the contents to the child process.

3. RDB snapshot configuration

Every time we start Redis, a dump. RDB file is automatically generated. This file will be generated in the directory in which you start Redis.

SNAPSHOTTING: SNAPSHOTTING: SNAPSHOTTING: SNAPSHOTTING: SNAPSHOTTING: SNAPSHOTTING: SNAPSHOTTING: SNAPSHOTTING: SNAPSHOTTING

# # Save <seconds> <changes> # # Redis Will Save the DBifboth the given number of seconds and the given # number of write operations against the DB occurred. # Will write data to disk after the specified number of seconds and number of data changes900 sec (15 min) if at least 1 key changed
#   after 300 sec (5 min) if at least 10 keys changed
#   after 60 sec if at least 10000Keys changed # The following example will write data to disk: #900Seconds (15Minutes) later, and at least1Times change #300Seconds (5Minutes) later, and at least10Times change #60In seconds, and at least10000# # Note: You can disable completely by Commenting out all"save"# note: You can disable saving by commenting out all "save" # # It is also possible to remove all the previously configured save # points by adding a save directive with a single empty string argument # like in the following example: # As in the following example, it is also possible to delete all previously saved configurations by adding a single empty string to point to: # # save""

save 900 1    //1 change in 15 minutes
save 300 10
save 60 10000

# By default Redis will stop accepting writes ifRDB snapshots are enabled # (at least one save point) and the latest background save failed. Redis will stop accepting writes and the latest background save will fail. #This will make the user aware (in a hard way)that data is not persisting # on disk properly, Otherwise chances are that no one will notice and some # disaster will happen. Otherwise the change will go unnoticed # and some disaster will occur. # # If the background saving process will start working again Redis will # automatically allow writes again. # If the background save process starts working again, Redis will automatically allow write operations again. # # Howeverif you have setup your proper monitoring of the Redis server
# and persistence, you may want to disable this feature so that Redis will
# continue to work as usual even ifThere are problems with disk, # permissions, and so forth. # However, if you have Redis server and persistence set up properly, You may want to disable this feature # so that even if The Redis disk has a problem it can continue to work as usual, etc. # stop-writes-on-bgsave-error yes # Compress string objects using LZF when dump .rdb databases? # export. RDB database when using LZF compressed string objects? # FordefaultThat's set to 'yes' as it's almost always a win. # Default to 'yes' because this is almost always correct. # If you want to save some CPU in the saving child set it to 'no' but # the dataset will likely be biggerifYou have compressible values or keys. # Set this value to "no" if you want to save CPU while saving child nodes, but if you have compressible keys or # values the data files may be larger. Rdbcompression yes # Since version 5 of RDB a CRC64 checksum is placed at the end of the file. The CRC64 checksum is placed at the end of the file. # This makes the format more resistant to corruption but there is a performance # hit topay (around 10%) when saving and loading RDB files, so you can disable it 
# forMaximum performances. # This makes the file format more reliable, but there is a performance drain (approx. 10%) when producing and loading RDB files, # so you can turn it off for better performance. # # RDB files created with checksum disabled have a checksum of zero that will # tell the loading code to skip the Create RDB file with checksum 0, This will tell The code to skip checking when loading rdbChecksum yes # The filename where to dump The DB # dbfilename dump. RDB # The working # # The DB will be written insidethisDirectory, with the filename specified # above using the 'dbfilename' configuration directive. # # The Append Only File will also be created insidethisDirectory. # The accumulated files will also be placed under this directory. # # Note that you must specify a directory here, not a file nameCopy the code

This snapshot configuration is mainly used for persistent operations. Here are some important configurations:

The save:

This is used to configure the persistence conditions that trigger Redis, that is, when data in memory is saved to disk. The default configuration is as follows:

• Save 900 1: if at least one key value changes within 900 seconds, the Redis saves the value, that is, carries out persistent operation
• Save 300 10: If at least 10 key values change within 300 seconds, the value is saved. If the value is too long, the value can be changed to 120s
• Save 60 10000: If the values of at least 10000 keys change within 60 seconds, the keys are saved

Of course, if you just use Redis caching and don’t need persistence, you can comment out all save lines to disable the save function. You can disable it with an empty string: save “”

Remember: after the configuration is changed, Redis restart:

Stop – writes – on – bgsave – error:

The default value is yes. When RDB is enabled for persistence and the last background save fails, decide whether Redis should stop receiving data. This makes the user aware that the data has not been persisted to disk correctly, otherwise no one will notice that a disaster has occurred. If Redis is restarted, you can start receiving data again

rdbcompression

The default value is yes. You can set whether to compress snapshots stored in disks. If so, Redis uses the LZF algorithm for compression. You can turn this off if you don’t want to use CPU for compression, but the snapshot stored on disk will be large.

rdbchecksum

The default value is yes. After storing the snapshot, we can also have Redis use the CRC64 algorithm to validate the data, but this adds about 10% of the performance cost and can be turned off for maximum performance gains.

dbfilename

Set the name of the snapshot file. The default is dump.rdb

dir

Set the directory where the snapshot file is stored. The configuration item must be a directory, not a file name. Use dbfilename above as the filename to save. To view the information, run the config get dir command

4. RDB persistence process

The copy-on-write technique uses a temporary file [RDB file] to write content and then replaces the RDB file persisted last time with a temporary file

Trigger mechanism:

1. If the SAVE rule is met, the RDB rule is automatically triggered
2. performflushallCommand will also trigger RDB rules
3. Exiting Redis also generates RDB files
4. The backup automatically generates onedump.rdbfile

For RDB-style persistence, manual triggering can be used:

1. Save: blocks the current Redis server until persistence is complete and should be disabled online.

   1.  set 112 good
       ok
       save
       ok
       #This will be done immediately, without waiting for the time in the default configuration, manually 
      Copy the code

   2. Save and block everything else. Manually save persistence. Is not recommended.

2. Bgsave: This trigger forks a child process that is responsible for persistence, so blocking only happens when the child process is forked.

   1. Redis takes snapshots asynchronously in the background and responds to client requests, such as data input from the foreground
   2. You can run the lastsave command to obtain the time when the last snapshot was successfully executed

The automatic trigger scenarios for persistence mainly include the following:

1. According to oursaveConfiguration rules automatically trigger RDB rules.
2. In full replication, the master sends the RDB file to the slave, and the master triggers bgSave ‘.
3. performdebug reloadWhen;
4. performshutdownIf aOF is not enabled when you exit Redis, it will also trigger and generate RDB files
5. performflushallCommand will also trigger RDB rules

The backup automatically generates a dump. RDB file. Since save is rarely used, let’s focus on how the bgsave command does RDB persistence. The flow chart is as follows

1. The Redis parent determines whether it is currently executing save, or a child of bgSave/bgrewriteAof (aOF file rewrite command), and if it is executing the bgsave command returns directly.
   1. bgsave/bgrewriteaofFor performance reasons, two concurrent child processes that perform a large number of disk writes at the same time may cause serious performance problems.
2. The parent process forks to create a child process. The parent process blocks and Redis cannot execute any commands from the client.
3. After the parent process forks, the bgsave command returns “Background Saving Started” and stops blocking the parent process and can respond to other commands.
4. The child process generates a temporary snapshot file RDB file based on the parent process memory snapshot, and then atomic replaces the original file.
5. The child sends a signal to the parent indicating completion, and the parent updates the statistics.

5. Redis backup and restore

5.1, process,

From the above exercise, the backup and recovery flow of RDB files can be derived

RDB file backup:

• Through the firstconfig get dirThe queryrdbDirectory of files
• will*.rdbCopy the file cp to the current Redis startup directory

RDB restoration:

• To shutdown Redis: shutdown– a new dump. RDB file is generated, empty
• Copy the above backup files to your working directorycp dump2.rdb dump.rdb. Overwrite the empty RDB file
• Overrides start Redis and backup data is loaded directly

5.2, practice

1. Open two clients, one for practice testing and one for viewing.

2. Client 1: Start Redis

CD user/local/bin /redis installation directory redis-server /etc/redis.conf redis -cli -p 6379 // Multiple port startup modeCopy the code

After logging in to client 1, set 12 values: simulated change 10 times command [set save 120 10]

set k1 v1
set k2 v2
set k3 v3
set k4 v4
set k5 v5
set k6 v6
set k7 v7
set k8 v8
set k9 v9
set k10 v10
set k11 v11
set k12 v12
Copy the code

Client 2: View files in the bin directory of the redis installation directory

cd user/local/bin
ls-l
Copy the code

Result: Dump. RDB file is not found. Why? Because 120s is not yet here, persistence mechanism is not triggered, after 120s it will be here, so we have finished the backup of our content, to dump. RDB file [automatic completion]. Because 12 keys were added in 120 seconds, the dump. RDB file was generated.

3. Once we have a backup, we should restore it during project development. Note: in enterprise development, we often have a working machine and a backup machine, because if one day our working machine goes down, we can restore it from the backup machine. Here we demonstrate the backup machine on client 2 by copying a dump. RDB file

#Make a copy of the dump. RDB file
cp dump.rdb dump_bk.rdb
#Look, there are two RDB files
ls-l
Copy the code

4. Client 1 Check the key

Keys * #v1-v12Copy the code

5. Failure of client 1 simulation development machine:

FLUSHALL shutdown // exitCopy the code

6. Client 2 Check the files in the bin directory again

Flushhall and shutdown can be used to trigger persistenceCopy the code

FLUSHALL and shutdown both trigger persistence and generate an empty dump. RDB file.

7, client 1: reIDS server should read the RDB file from the hard disk and restore the contents of the file to the memory, but we check the key *, find that there is nothing in redis memory, which means that the restoration failed. Why?

Conf redis -cli -p 6379 keys * (empty list or set)// EmptyCopy the code

The dump. RDB file is null because the last save persistence file was overwritten by the FLUSHALL operation.

8, delete the new RDB file, do not use this to overwrite, because it is empty, use the original copy of dump_bk. RDB file restore

rm -f dump.rdb
cp dump_bk.rdb dump.rdb
Copy the code

9. Client 1, restart to view

redis-server /etc/redis.conf
redis -cli -p 6379  
keys *  
Copy the code

It is found that the data is recovered, but k12 is missing. This is the defect of the RDB – the last data may disappear

6. Summary of RDB

RDB is a very compact file, RDB to save the RDB file, the only thing the parent process needs to do is fork out a child process, the child process does all the following work, the parent process does not need to do other IO operations, so THE RDB persistence method can maximize the performance of Redis

Compared with AOF, RDB is faster to restore large data sets, but the risk of data loss is greater. RDB often needs to fork the child process to save the data set to the hard disk. When the data set is large, the fork process is very time-consuming, and may cause Redis to fail to respond to client requests in some boma seconds

advantage

• Suitable for large-scale data recovery
• Data integrity and consistency requirements are not high more suitable for use
• Save disk space
• Fast recovery

disadvantage

• At Fork, the data in memory is cloned, and roughly twice as much expansibility needs to be considered
• Although Redis uses copy-on-write for forking, it can be a performance drain if the data is large.
• Backups are made at regular intervals in the backup cycle, so if Redis unexpectedly goes down, all changes since the last snapshot will be lost.

How to stop

• Dynamic stop RDB:redis-cli config set save ""To disable the save policy, empty value is given after save

Section 2: AOF of Redis Persistence

1. Background

We know that the RDB has one drawback: So if Redis down accidentally fall, will lose all changes after the last snapshot, cause have difficulty in recovery, up from work, the truth about the impact is not big, but someone will pursue the perfect, solved the problem, but from the actual effect, the possible loss of around 1 s data, hence better restored.

Before we get to Aof, let’s talk about a mathematical problem, such as adding numbers from 1 to 100. Gauss didn’t add them one by one in a more neat way. Similarly, suppose we have another requirement here: do as follows

Set k1 1 # write operation incr k1...... Omit 100,000 identical commandsCopy the code

As above, within our redis, have the same action like a lot, we aof for backup, to command the same clone, execution, restore the time to read it again, this ensures absolutely right, and each time for 1 s, you write a backup I a, but some questions like this: When a large data command like the one described above occurs, the appendone.aof file will experience a resource explosion that takes up disk space. But from the computer’s point of view, we can just do incR k1 100,000, and we’re done. So now let’s look at AOF, the persistence mechanism.

2. What is AOF

Each write operation is recorded in the form of a log (incremental save), all write instructions performed by Redis are recorded (read operations are not recorded), only files can be appended but files can not be overwritten, Redis will initially read the file to rebuild the data, in other words, When redis is restarted, write instructions are executed from front to back according to the contents of the log file to complete data recovery.

3. AOF persistence process

1. The client request write command is appended to the AOF buffer by append.
2. The AOF buffer synchronizes the write operation sync to the disk’s AOF file according to the AOF persistence policy [always, Everysec,no].
3. When the AOF file size exceeds the rewrite policy or manual rewriting, the AOF file will be rewritten to reduce the AOF file size.
4. When the Redis service restarts, it loads the write operations in AOF files to restore data.

4. AOF configuration

# By default Redis asynchronously dumps the dataset on disk. This mode is
# good enough in many applications, but an issue with the Redis process or
# a power outage may result into a few minutes of writes lost (depending on
# the configured save points)# By default, Redis asynchronously dumps data sets to disk. This pattern is good enough for many applications, But The Redis process or power outage may cause write loss of several minutes (depending on The savepoint configured) # # # The Append Only File is an alternative persistence mode that provides # much  better durability. For instance using thedefault data fsync policy
# (see later in the config file) Redis can lose just one second of writes in a
# dramatic event like a server power outage, or a single write ifsomething # wrong with the Redis process itself happens, But the operating system is # still running correctly. # Append Only files are another persistence mode that provides better persistence. For example, using the default data fsync # policy (see later in the configuration file) Redis will lose only one second of write # in the event of an emergency such as a server power outage, or only one write # if the Redis process itself is faulty but the operating system is still running. # # AOF and RDB persistence can be enabled at the same time without problems. # If the AOF is enabled on startup Redis will load the AOF, That is the file # with the better fix. # Redis loads AOF if they are enabled at startup, This is a file with better durability guarantee # # Please check HTTP://redis.io/topics/persistence for more information.

appendonly no

# The name of the append only file (default: "appendonly.aof")

appendfilename "appendonly.aof"

# The fsync(a) call tells the Operating System to actually write data on disk
# instead of waiting for more data in the output buffer. Some OS will really flush
# data on disk, some other OS will just try to do it ASAP.
# fsync(a)The call tells the operating system to actually write data to disk instead of waiting for more data in the output buffer. # Some operating systems actually flush the data on disk, while others just try to do it as quickly as possible. # # Redis supports three different modes: Don't fsync, just let the OS flush the data when it wants. Faster. # no: Don't fsync, just let the OS flush the data when it wants. Faster. 'always' : fsync after every write to the append only log. Slow, everything is synchronized after every write. Slow, safe # everysec: fsync only one time every second. Compromise. # everysec: Sync only once per second. A compromise. # # Thedefault is "everysec", as that's usually the right compromise between
# speed and data safety. It's up to you to understand if you can relax this to
# "no" that will let the operating system flush the output buffer when
# it wants, for better performances (but if you can live with the idea of
# some data loss consider the default persistence mode that's snapshotting),
# or on the contrary, use "always" that'S very slow but a bit safer than # everysec. # The default is "everysec", as this is usually the right tradeoff between speed and data security. # this depends on whether you can put this value to "no", so that the operating system when the need to refresh the output buffer, in order to obtain better performance of # (however, if you can accept the idea of some data loss, please consider the durability of the default mode, namely the snapshot), or # instead, use "always", this is very slow, But a little safer than every second. # # More details please check the following article: # http://antirez.com/post/redis-persistence-demystified.html # # If unsure, use"everysec"# If in doubt, use "everysec". # appendfsync always appendfsync everysec # appendfsync no # When the AOF fsync policy is set to always or everysec, and a background # saving process (a background save or AOF log background rewriting) is
# performing a lot of I/O against the disk, in some Linux configurations
# Redis may block too long on the fsync(a) call. Note that there is no fix for
# this currently, as even performing fsync in a different thread will block
# our synchronous write(2)Call. # When the AOF fsync policy is set to always or everysec and the process is saved in the background(Background save or AOF log background # rewrite)In some Linux configurations, Redis may perform heavy I/O operations on disksfsync(a)Blocking on call too long. Note that this has not been fixed yet, because even performing fsync in a different thread would block our # syncwrite(2)The call. # # In order to mitigatethis problem it's possible to use the following option
# that will prevent fsync(a) from being called in the main process whileA # BGSAVE or BGREWRITEAOF is in progress. # To alleviate this problem, you can use the following options to prevent # calls in the main process when BGSAVE or BGREWRITEAOF is executedfsync(a). # # This means thatwhile another child is saving, the durability of Redis is
# the same as "appendfsync none". In practical terms, this means that it is
# possible to lose up to 30 seconds of log in the worst scenario (with the
# default Linux settings)# This means that Redis has the same persistence as "appendfsync None" when another child process is being saved. In practice, this means that in the worst case you can lose 30 seconds of logging (using the default Linux Settings) # # If you have latency problems turnthisTo "yes". Otherwise leave it as # "no" that is the point of view of delay. # If you have delay problems, Change this option to Yes. Otherwise, setting it to "no" is the safest option from a durability perspective. Appendfsync-on-rewrite no # Automatic rewrite of the append only file rewrite the log file implicitly calling # BGREWRITEAOF when the AOF log size grows by the specified percentage. # When the log size increases by a specified percentage, Redis can implicitly call BGREWRITEAOF to automatically rewrite the log file # # This is how it works: Redis remembers the size of the AOF file after the # latestrewrite (if no rewrite has happened since the restart, the size of
# the AOF at startup is used)# Here's how it works: Redis remembers the size of the AOF file after the last rewrite(If no overwriting occurs after restart, # use the size of AOF at startup)
#
# This base size is compared to the current size. If the current size is
# bigger than the specified percentage, the rewrite is triggered. Also
# you need to specify a minimal size for the AOF file to be rewritten, this
# is useful to avoid rewriting the AOF file even ifThe percentage increase # is reached but it is still pretty small. Rewriting is triggered if the current size is greater than the specified percentage. # In addition, you need to specify a minimum size for the AOF files to be overwritten, which is useful to avoid overwriting AOF files, even if the percentage increase of # is reached, but it is still very small. # Specify a percentage of zero in order to disable the automatic AOF # rewrite feature. If auto-aof overwriting is disabled, auto-aof-rewrite-percentage 100 auto-aof-rewrite-min-size 64mb # An AOF file may be found to be truncated at the end during the Redis # startup process, when the AOF data gets loaded back into memory. # This may happen when the system where Redis is running # crashes, especially when an ext4 filesystem is mounted without the # data=ordered option (however this canThe last thing that happens when Redis itself # crashes or aborts but the operating system still works correctly). When the AOF data is loaded back into memory, you may find that the AOF file is truncated. # This can happen when the system running Redis crashes, especially when the # ext4 filesystem is mounted without the data=ordered option (however, this does not happen when Redis itself crashes or aborts, # # Redis can either exit with an error when this happens, or load as much # data as possible (the default now) and start if the AOF file is found # to be truncated at the end. When this happens, Redis can exit with an error, or load as much data as possible, and if it finds that The AOF file is truncated at The end #, it can start. The following options control this behavior. # # If aof-load-truncated is set to yes, a truncated AOF file is loaded and # the Redis server starts emitting a log to inform the user of the event. # Otherwise  if the option is set to no, the server aborts with an error # and refuses to start. When the option is set to no, The user requires # to fix the AOF file using the "redis-check-aof" utility before to restart # the server -load-truncated is set to yes, a truncated AOF file is loaded, and Redis server # starts sending logs to notify the user of the event. Otherwise, if this option is set to no, the server will abort with an error and refuse to start the # action. When this option is set to no, the user needs to repair the # aof file using the "redis-check-aof" utility before restarting the server. # # Note that if the AOF file will be found to be corrupted in the middle # the server will still exit with an error. This option only applies when # Redis will try to read more data from the AOF file but not enough bytes # will be found. Note that if the AOF file is destroyed in the middle, the server will still exit with an error. This option only works when Redis tries to # read more data from an AOF file and does not find enough bytes. aof-load-truncated yes # When rewriting the AOF file, Redis is able to use an RDB preamble in the # AOF file for faster rewrites and recoveries. When this option is turned # On the rewritten AOF file is composed of two different stanzas: # When rewriting AOF files, Redis is able to use the RDB prologues in AOF files for faster rewriting and recovery. With this option # enabled, the rewritten AOF file consists of two distinct sections:  # # [RDB file][AOF tail] # # When loading Redis recognizes that the AOF file starts with the "REDIS" # string and loads * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * Then continue loading the AOF # tail. aof-use-rdb-preamble yesCopy the code

1. AOF is disabled by default

Unlike RDB, RDB is enabled by default and can be configured in redis.conf

appendonly no  ---yes
Copy the code

After RDB is enabled, the default generated file is dump. RDB, and the default generated file is appendone.aof after AOF is enabled

The path to save the AOF file is the same as the RDB path [default Redis startup directory].

2, AOF and RDB are enabled at the same time, redis to whom?

AOF and RDB are enabled at the same time, and the system takes AOF data by default (data will not be lost).

K2 v2 set k3 v3 set k4 v4 set k5 v5 set k6 v6 set k7 v7Copy the code

As shown above: This is the last time we redis set inside, we open AOF in configuration, restart after redis, but our keys * command but found no data [it should be restored, morally RDB have contents 】, this is because we started the AOF, because there is no any open after we write command, So aof, this file is empty, and you listen to AOF. So there’s no content. Aof files are generated when reids is enabled.

3, FLUSHALL

Set k1 v1 // Write operation set k2 v2 set k3 v3 set K4 V4 FLUSHALLCopy the code

Flushhall: a new RDB file will be generated in flushhall. Aof will also be generated in flushhall. No data will be generated in flushhall by keys *. FLUSHALL is not used in the aOF command because it flushhall is not used in the aof command. When you delete the FLUSHALL command and restart Redis, it can be automatically restored.

5. AOF backup/repair/restore

Although the backup mechanism and performance of AOF are different from those of RDB, the backup and restoration operations are the same as those of RDB. Backup files are copied to the Redis working directory when restoration is required and loaded immediately after system startup.

2. Normal recovery

• Change the default appendonly no to yes
• Make a copy of the aOF file with data and save it to the corresponding directory (see config get dir).
• Recovery: Restart Redis and reload

Abnormal return

• Change the default appendonly no to yes
• Back up the bad AOF file
• ifThe AOF file is damaged[Contains error content], to pass/usr/local/bin/redis-check-aof--fix appendonly.aofCommand to restore [ls-l]
• Recovery: Restart Redis and then reload

6. AOF synchronization frequency setting

appendfsync always

Always synchronize, every Redis write is logged immediately; Poor performance but good data integrity

appendfsync everysec

It is recommended by factory default that the operation is asynchronous and synchronized every second. The data in this second may be lost if the system goes down.

appendfsync no

Redis does not actively synchronize data, but leaves the synchronization timing to the operating system, which is the most efficient

Rewrite compression

7.1 overview,

What is:

When the size of AOF files exceeds the set threshold, Redis will start the content compression of AOF files. Only the minimum instruction set that can restore data can be retained. You can use the command bgrewriteaOF

Rewriting principle, how to implement rewriting

If the AOF file grows too large, a new subprocess will be forked to rename the file (also write temporary files first and then rename the file). Rewrite after redis4.0 means that the snapshot of the RDB is attached to the head of the new AOF in the form of secondary system, as the existing historical data, replacing the original running account operation.

No – appendfsync – on – rewrite:

If no-appendfsync-on-rewrite=yes, no aOF files are written to the cache, the user request will not block, but the cached data will be lost if downtime occurs during this time. (Reduces data security and improves performance)

If no-appendfsync-on-rewrite=no, it would still flush data to disk, but would block when overwritten. (Data security, but performance degradation)

Trigger mechanism, when to override

Redis will record the AOF size of the last rewrite. The default configuration is to trigger when the AOF file size is double the size of the last rewrite and the file size is greater than 64M, although rewriting can save a lot of disk space and reduce recovery time. However, each rewrite still has a certain burden, so the Redis will be rewritten only when certain conditions are met.

• auto-aof-rewrite-percentage: ** Sets the baseline value for rewriting, starting when the file reaches 100% (** triggered when the file is twice the size of the original rewritten file)
• auto-aof-rewrite-min-size: Sets the baseline value for rewriting, minimum file size 64MB. Start rewriting when you reach this value.

For example, if the file reaches 70MB and starts to be rewritten, go down to 50MB, when will the next file be rewritten? 100MB

When the system is loaded or when it was rewritten last time, Redis will record the AOF size and set it to base_size,

Redis overwrites AOF if the current size of Redis is >= base_size +base_size*100% (default) and the current size is >= 64MB (default). [Actual 3G start]

7.2. Rewrite the process

1. Bgrewriteaof triggers rewriting to check whether BGSave or BgreWriteAOF is running. If yes, the command is executed after the command is executed.
2. The main process forks a child process to perform the rewrite operation to ensure that the main process does not block.
3. The child process traverses the data in redis memory to the temporary file, and the client’s write request writes both the AOF_BUf buffer and the AOF_rewrite_buf rewrite buffer to ensure that the original AOF file is intact and the new data modification action during the generation of the new AOF file is not lost.
4. 4.1- After the child process writes the new AOF file, it signals the main process, and the parent process updates the statistics. 4.2- The main process writes data from aOF_rewrite_buf to the new AOF file.
5. Complete the AOF rewrite by overwriting the old AOF file with the new AOF file.

8, summary

advantage

• The backup mechanism is more robust and the probability of data loss is lower.
• Read log text, by operating AOF robust, can handle misoperations.

disadvantage

• Takes up more disk space than RDB.
• Restoring backups is slow, so RDB persistence is the default
• There is some performance pressure to synchronize every read and write.
• Some bugs exist, causing the recovery failure.

A small summary

AOF the file is a only for additional log files, Redis in AOF file becomes too large, the volume automatically in the background to rewrite AOF AOF file orderly saved to the database to perform all of the write operation, the write operation to Redis protocol format, therefore AOF the file content is very easy to read, Parsing files is also easy

AOF files are usually larger than RDB files for the same data set, and depending on the fsync strategy used, AOF may be slower than RDB

Which is better: The official recommendation is to use both.

• If you are not sensitive to data, you can choose to use RDB alone. AOF alone is not recommended because of the potential for bugs.
• If only do pure memory cache, can not use, that is, all closed

Website suggest

• RDB persistence allows you to take snapshots of your data at specified intervals
• The AOF persistence mode records each write operation to the server. When the server is restarted, these commands will be executed again to restore the original data. The AOF command saves each write operation to the end of the file using the Redis protocol.
• Redis can also rewrite AOF files in the background so that AOF files are not too large
• Cache only: If you only want your data to live while the server is running, you can also do without any persistence.

Enable both persistence modes:

• In this case, when Redis restarts, AOF files will be loaded first to recover the original data, because AOF files usually hold more complete data sets than RDB files.
• RDB data is not real-time, and server restart will only look for AOF files when both are used. Should we just use AOF?
• Not recommended, as RDB is more suitable for backing up databases (AOF is constantly changing and not good for backing up), quick restart, and there are no potential AOF bugs left as a back-up measure.

Performance Suggestions