1. Selection and selection of Redis persistence

1. What is persistence

  • Redis keeps all data in memory and updates to data are asynchronously saved to disk.

2. Persistence mode

  • The snapshot
    • MySQL Dump
    • Redis RDB
  • Write the log
    • MySQL Binlog
    • Hbase HLog
    • Redis AOF

2. RDB

1. What is RDB

2. Trigger mechanism – There are three main ways

1. Save (sync)

  • The client sends a save command to Redis and Redis generates the RDB file for us.
  • Problem: Since it is a synchronization command, if our save is very slow (with a lot of data), redis will block.
  • File policy: If an old RDB file exists, the new one replaces the old one.
  • Complexity: O(N).

2. Bgsave (asynchronous)

  • The client executes the BGsave command, which uses the Linux fork() function to generate a Redis child of the main process and let the child process complete RDB generation. After the RDB is generated, the main process is told that the RDB is generated successfully.
  • If fork execution is very slow (and in most cases very fast), redis will still block.
  • Redis responds normally to the client because heavy operations like createRDB are handled by child processes.
  • The file policy and complexity are the same as for Save.
The command   save   bgsave
IO types synchronous asynchronous
blocking   是 Yes (blocking occurs at fork)
The complexity of the   O(N)   O(N)
advantages No extra memory will be consumed Do not block client commands
disadvantages Block the client command It forks and consumes memory

3. Automatic

  • Redis provides the Save configuration for automatic persistence
configuration   seconds   changes
save   900   1
save   300   10
save   60   10000
  • 10,000 data changes in 60 seconds, 10 data changes in 300 seconds, and one data change in 900 seconds, for automatic persistence. RDB file generation is an internal call to BGSave.
  • The specific configuration can be modified.
  • Unable to control the RDB generation frequency.

4. Configuration

  • Let’s look at the configuration given in the redis default configuration file
  • Save: automatic persistence
  • Dbfilename: indicates the RDB name
  • Dir: indicates the directory where log files such as RDB are stored
  • Stop-writes-on-bgsave-error: specifies whether to stop writing when a BGSave error occurs
  • Rdbcompression: Whether RDB files are compressed
  • Rdbchecksum: indicates whether to checksum RDB files
save 900 1
save 300 10
save 60 10000
dbfilename dump.rdb
dir ./
stop-writes-on-bgsave-error yes
rdbcompression yes
rdbchecksum yes
Copy the code

5. Optimal configuration

  • Disable automatic persistence
  • Dbfilename: uses the port number to distinguish files
  • Dir: Disks are divided according to the number of Redis
dbfilename dump-${port}.rdb
dir /bigdistpath
stop-writes-on-bgsave-error yes
rdbcompression yes
rdbchecksum yes
Copy the code

3. Trigger mechanism – Not to be ignored

1. Full replication

2. debug reload

3. shutdown

4. Test

  • Modify the configuration file for remote connection
    • The bind 0.0.0.0
    • protected-mode no
  • Add some random data using Jedis
127.0.0.1:6379 > dbsize (integer) 1000000

127.0.0.1:6379> info memory
# MemoryUsed_memory: 105221280 used_memory_human: 100.35 MCopy the code
  • We are ready to execute commands in advance in another window
127.0.0.1:6379 >setHello word OK 127.0.0.1:6379> get helloCopy the code
  • Save in the first window, followed by Get Hello, is blocked
127.0.0.1:6379> save
OK
(4.77s)

127.0.0.1:6379> get hello
"word"(2.22 s)Copy the code
  • We can also go to the data directory (configuration file specified) to see the generated RDB file
[root@localhost redis]# cd data/
[root@localhost data]# ll1 root root 7365 12月 21 14:29 6379.log -rw-r--r-- 1 root root 4809 12月 20 23:40 6382.log -rw-r--r--. 1 root root 53666783 December 21 14:29 dump-6379.rdbCopy the code
  • Let’s verify BGSIVE next
127.0.0.1:6379> bgsave
Background saving started
Copy the code
  • Call the child process using command discovery while get Hello does not block
[root@localhost redis]# ps -ef | grep redis-root 4987 1 1 14:21 ? 00:00:15 redis-server 0.0.0.0:6379 root 5271 3826 0 14:33 PTS /0 00:00:00 redis-cli root 5304 4987 56 14:36? 00:00:01 redis-rdb-bgSave 0.0.0.00:6379 root 5306 5143 0 14:36 PTS /2 00:00:00 grep --color=auto redis-Copy the code
  • Finally, let’s look at the automatic persistence strategy
#save 900 1
#save 300 10
save 60 5
Copy the code
  • You can flush data using flushall
127.0.0.1:6379> flushall
OK
(0.70s)
127.0.0.1:6379> dbsize
(integer) 0
Copy the code
  • Insert data
127.0.0.1:6379 >setB OK 127.0.0.1:6379 >setC d OK 127.0.0.1:6379 >setE f OK 127.0.0.1:6379 >setG h OK 127.0.0.1:6379 >setI j OK 127.0.0.1:6379 >set m n
OK
Copy the code
  • Automatic persistence is found in logs
[root@localhost data]# tail -f 6379.log

5475:M 21 Dec 14:45:03.075 * 5 changes in 60 seconds. Saving...
5475:M 21 Dec 14:45:03.077 * Background saving started by pid 5494
5494:C 21 Dec 14:45:05.018 * DB saved on disk
5494:C 21 Dec 14:45:05.019 * RDB: 10 MB of memory used by copy-on-write
5475:M 21 Dec 14:45:05.073 * Background saving terminated with success
Copy the code

5. To summarize

  • The RDB is a Redis memory-to-disk snapshot for persistence.
  • Sava usually blocks Redis.
  • Bgsava does not block Redis, but forks a new process.
  • The save autoconfiguration meets either of these criteria and is executed.
  • Some triggers cannot be ignored.

3. AOF

1. What’s wrong with RDB

1. Time consumption and performance consumption

  • The RDB generation process is to dump the data in memory to the hard disk to form an RDB file.
  • First of all, it is time-consuming. We need to dump all the data, which is an O(N) process, and the write itself will consume a lot of CPU.
  • The second is a memory drain, and we know bgSIVE has a fork process.
  • There is also the consumption of IO performance.

2. If the alarm is uncontrollable, data is lost

configuration action
T1 Execute multiple write commands
T2 The conditions for automatic RDB creation are met
T3 Execute multiple write commands again
T4 outage
  • Data writes between T3 and T4 are lost

2. AOF operation principle and creation

  • When the client writes a piece of data, it appends a write command to the log

3. AOF’s three strategies

  • always
    • When redis executes a write command, it actually does not write directly to the file system, but to the disk buffer. Buffers are flushed to hard disk according to some policy.
    • Always means that every command fsync is sent to the hard disk so that redis writes data without loss.
  • everysec
    • Everysec means fsync buffer to hard disk every second.
    • The hard disk is properly protected during high write volume.
    • The downside is that a second of data can be lost if something goes wrong.
    • This is a default value.
  • no
    • The OS decides fsync, and the operating system decides when to write to the hard disk.
The command   always   everysec   no
advantages No data loss Fsync every second No need to worry
disadvantages IO overhead is large, the general SATA disk only a few hundred TPS Lose one second of data uncontrolled

4. AOF rewritten

  • set hello world set hello java set hello hehe => set hello hehe
  • incr counter incr counter => set counter 2
  • rpush mylist a rpush mylist b rpush mylist c => rpush mylist a b c
  • Expired data => Rewriting procedure is useless

Simplifying some commands that can be optimized serves two purposes

  • Reduce disk usage
  • Accelerated recovery rate

5. Two ways of AOF rewrite implementation

1. bgrewriteaof

  • The client sends a command, bgrewriteaof, to Redis, which returns OK and executes it asynchronously. Redis receives this command and forks a child process to complete the AOF rewrite.
  • Here the AOF rewrite is to go back to the data in redis memory, go back to the AOF file, and rewrite from redis memory.

2. AOF overwrites the configuration

  • configuration
The configuration of meaning
auto-aof-rewrite-min-size Size required for AOF file rewrite
auto-aof-rewrite-percentage AOF file growth rate
  • statistical
Statistics of meaning
aof-current-size AOF Current size (in bytes)
aof-base-size Size of AOF last started and rewritten in bytes
  • Automatic trigger time
    • aof-current-size > auto-aof-rewrite-min-size
    • (aof-current-size – aof-base-size) / aof-base-size > auto-aof-rewrite-percentage

6. AOF configuration

  • Let’s modify the configuration given in the redis default configuration file
appendonly yes
appendfilename "appendonly.aof"
appendfsync everysec
dir /home/redis/data
no-appendfsync-on-rewrite yes
auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb
Copy the code
  • Do some simple things
127.0.0.1:6379 >setHello world OK 127.0.0.1:6379 >setHello Java OK 127.0.0.1:6379 >setHello redis OK 127.0.0.1:6379> incr counter (integer) 1
127.0.0.1:6379> incr counter
(integer) 2
127.0.0.1:6379> rpush list a
(integer1 127.0.0.1:6379> rpush list b (integer) 2
127.0.0.1:6379> rpush list c
(integer) 3
Copy the code
  • Check whether the AOF file is generated
[root@localhost redis]# cd data
[root@localhost data]# ll1 root root 21736 12月 21 16:57 6379.log -rw-r--r-- 1 root root 4809 12月 20 23:40 6382.log -rw-r--r-- 1 root root 277 December 21 16:58 appendonly. Aof -rw-r--r-- 1 root root 127 December 21 15:22 dump-6379.rdbCopy the code
  • Viewing file Contents
[root@localhost data]# more appendonly.aof * 3$3
set
A $5
hello
A $5
world
Copy the code

  • Let’s look at the file format a little bit

    • *3 indicates that the following command takes three arguments
    • $3 represents the length of the following data in bytes

  • So let’s do a rewrite

127.0.0.1:6379> Bgrewriteaof Background append only file startedCopy the code
  • Open appendone.aof again and we see that only set Hello redis remains.
* 3$3
SET
A $5
hello
A $5
redis
Copy the code

7. Choice between RDB and AOF

1. Comparison between RDB and AOF

The command   RDB   AOF
Startup priority   低   高
volume   小   大
Recovery rate   快   慢
Data security Lost data By strategy
Light and heavy   重   轻

2. Optimal RDB policy

  • “Closed”
  • Centralized management
  • Master slave, slave open?

3. AOF best strategy

  • “On” : cache and storage
  • AOF rewrite centralized management
  • everysec

4. Best strategy

  • Small shard
  • Cache or store
  • Monitoring (hard disk, memory, load, network)
  • Enough memory

4. Develop common problems in operation and maintenance

1. The fork

1. Synchronize operations

2. It depends on the amount of memory: The larger the memory, the longer it takes (depending on the type of machine)

3. The info: latest_fork_usec

4. Improve the fork

  • Physical machines or virtualization technologies that support efficient fork operations are preferred.
  • Controls the maximum available memory for redis instances: maxMemory.
  • Configure the Linux memory allocation policy properly: vm. Overcommit_memory =1.
  • Reduce the frequency of forking: for example, relax the automatic trigger time of AOF rewriting, unnecessary full copy.

2. Subprocess overhead and optimization

1. CPU

  • Overhead: RDB and AOF file generation, which is CPU intensive.
  • Optimization: No CPU binding, no CPU intensive deployment.

2. The memory

  • Cost: fork memory cost, copy-on-write.
  • Optimization: ban supports large page of memory allocation: echo never > / sys/kernel/mm/transparent_hugepage/enabled.

3. The hard disk

  • Overhead: AOF and RDB file writing, can be combined with ioSTAT, IOTOP analysis.
  • Optimization:
    • Do not deploy with high disk load services: storage services, message queues, etc.
    • Do not append AOF: no-appendfsync-on-rewrite = yes.
    • The disk type depends on the amount of data to be written, for example, SSD.
    • The standalone multi-instance persistent file directory can be divided into disks.

3. AOF append is blocked

  • If AOF is used, we usually use a strategy of flushing every second.
  • First, the main thread writes to the AOF buffer, while another AOF synchronization thread synchronizes every second flush and records the time of the last sync.
  • The main thread also compares the time to the last AOF. If it is more than two seconds since the last synchronization, the main thread will block.
  • For the solution, see the above section on disk optimization.

The last

You can follow my wechat public number to learn and progress together.