Standalone Redis installation

Note: This series uses Redis 6.2.5, the test environment is the local VMVare VM, and the system image is centos-7-x86_64.

To build a standalone version of Redis, you need to download two installation packages:

  • IO /download Redis
  • Download tcl:sourceforge.net/projects/tc…

The IP address of the local VM is 192.168.241.130. Upload the two installation packages to the /usr/local/src directory on the server.

Unpack the installation

Switch to the /usr/local/src directory

# cd /usr/local/src
Copy the code

If TCL is not installed first, an error message will be displayed indicating that TCL is required when installing Redis.

#The tar - ZXVF tcl8.5.10 - SRC. Tar. Gz

# cdTcl8.5.10 / Unix /

# ./configure

# make & make install
Copy the code

Unzip and install Redis.

# cd /usr/local/src

#The tar - ZXVF redis - 6.2.5. Tar. Gz

# cdRedis - 6.2.5

# make && make test

# make install
Copy the code

Redis startup solution

  • 1. Start the script

This is redis startup script. Copy it to /etc/init.d and rename it redis_6379 to indicate that Redis wants to listen on port 6379.

# cd /usr/local/ SRC/redis - 6.2.5 / utils /

# cp redis_init_script /etc/init.d/

# cd /etc/init.d/

# mv redis_init_script redis_6379
Copy the code

In the redis_6379 script, you can see the following configuration:

  • REDISPORT: Redis port. The default value is 6379
  • EXEC: Redis server
  • CLIEXEC: Redis client
  • PIDFILE: Redis program will generate a PID file after running, that is/var/run/redis_6379.pid
  • CONF: redis configuration file location,/etc/redis/6379.conf

  • 2. Modify the configuration

CONF specifies the redis configuration file directory as /etc/redis/ and file name as 6379.conf.

Create the configuration file directory, data persistence directory, log directory:

# mkdir -p /etc/redis

# mkdir -p /var/redis/6379

# mkdir -p /var/redis/log
Copy the code

Conf file to /etc/redis, rename it to 6379.conf, and grant read/write permissions:

# cp /usr/local/ SRC/redis - 6.2.5 / redis. Conf/etc/redis

# cd /etc/redis/

# mv redis.conf 6379.conf

# chmod +666 6379.conf
Copy the code

Modify the configuration in 6379.conf as follows, and keep the default Settings for the rest:

configuration value instructions
daemonize yes Let Redis run as a daemon process
bind Local IP address (192.168.241.130) Bind the local IP, otherwise redis cannot be accessed externally
pidfile /var/run/redis_6379.pid Set the pid file location of redis
port 6379 Set the listening port number of Redis
dir /var/redis/6379 Sets the location where persistent files are stored
logfile /var/redis/log/6379.log Set the log file location

/etc/init.d/redis_6379 /redis_6379 /redis_6379

  • 3. Start the machine

Modify the redis_6379 startup script by adding the following two lines to it:

# cd /etc/init.d/

# vim redis_6379
Copy the code

Add the following two lines of comment:

# chkconfig: 2345 90 10 
# description: Redis is a persistent key-value database
Copy the code

Configure boot:

# chkconfig redis_6379 on
Copy the code
  • 4. Start Redis
# cd /etc/init.d

# ./redis_6379 start
Copy the code

Verify whether Redis is started successfully:

# ps -ef|grep redis
Copy the code

  • 5. Client connection

Connect to the server through redis-cli command:

# redis - 192.168.241.130 cli - hCopy the code

And do some simple command tests:

Data persistence

The meaning of persistence

Redis is generally used for caching, data are in memory, once Redis down or the server where Redis is down, it will lead to the loss of all the data in memory, then it may take great efforts to recover the original data. Retrieving data from the database and then caching it into Redis puts a lot of strain on the database and slows down applications that use the cached data.

So it is critical for Redis to persist data and avoid recovery from back-end databases. Without persistence, Redis will lose all data in the event of a catastrophic failure.

For production-level Redis architecture, persistence is also essential. Persistence is mainly used for disaster recovery and data recovery, so as to ensure high availability of Redis. In this way, after Redis becomes unavailable, Redis can be restored as soon as possible, become available, and then provide external services.

There are two main mechanisms for Redis persistence: AOF (Append Only File) logs and Redis DataBase (RDB) snapshots. If we want Redis to be used only as a pure memory cache, we can disable the persistence mechanism of RDB and AOF.

AOF log

The AOF mechanism takes each write command as a log and writes it to a log file in appends-only mode. When Redis restarts, the entire data set can be reconstructed by playing back the write command in the AOF log.

  • 1. Post log

We may be familiar with logging before you write (WAL), where you write to the log before you write to the data; AOF is post-write logging. That is, commands are executed to write data to the memory before logs are recorded.

This is because AOF records the execution of commands. The command is executed first and logs are generated only after the command is successfully executed. Otherwise, an error message is reported to the client. These commands do not need to be syntactically checked during logging, thus avoiding additional checking overhead.

However, this post-write logging mechanism also has some risks, that is, after the command is executed, the command and data may be lost before logging. However, the impact is generally small and data can be recovered from the database.

  • 2. Write back strategy

When AOF persistence is enabled, it is important to remember that logs are not written directly to disk. Instead, logs are written to the AOF buffer, the OS cache, and fsync synchronizes data from the buffer to disk files at regular intervals.

This brings us to the three types of fsync synchronization:

  • always: Synchronous write back. Each time a log is written, fsync is immediately called to write the log back to the disk.
  • everysec: write back every second. Fsync is called once a second to write the contents of the buffer to disk. This is the default value.
  • no: The operating system decides when to write buffer contents back to disk.

Advantages and disadvantages of the three write back strategies:

Write back to strategy Write back to the time advantages disadvantages
always Synchronous write back The reliability is high, and data is not lost Every write command falls immediately, which has a significant impact on performance and leads to low throughput
everysec Per second to write back The performance is moderate Data will be lost within 1 second of the outage
no Operating system control Performance is good This policy is not controlled by Redis and can result in massive data loss

Each policy has its own advantages and disadvantages. Therefore, you need to select an appropriate write back policy based on the system’s requirements for high performance and reliability. For high performance, choose the No policy; To ensure high reliability, choose the Always policy. If you want to allow a bit of data loss, but you don’t want performance to suffer too much, choose the Everysec policy. Generally, use the default Everysec policy.

  • 3. AOF rewriting mechanism

Redis only writes to one AOF file, so if you keep logging to that file, it gets bigger and bigger.

Large AOF files can cause performance problems:

  • The file system itself has a limit on the file size and cannot save large files
  • If the file is too large, it will be inefficient to add command records to it
  • If you want to do disaster recovery, you have to replay all the commands in the log file, and the recovery process will be very slow and will affect the normal use of Redis

So, when the AOF file gets big enough, Redis does AOF rewriting.

AOF rewrite is to create a new AOF file based on the data in memory at the time, read all the key/value pairs in the database, and then write each key/value pair to the new AOF file with a write command. This will create a smaller AOF file and delete the older, larger file.

Does the AOF rewrite block the main thread?

The AOF rewrite is actually done by the backend process bgrewriteAOF so that the main thread can continue processing command requests. On each rewrite, the main thread forks out the bgrewriteaof child and makes a copy of the main thread’s memory to the bgrewriteaof child. The bgrewriteAof child can then read the data into commands to write to a new AOF file without affecting the main thread.

However, there is a problem with using child processes. During the AOF rewrite, the main thread can continue to process write requests, causing the database state of the server to be inconsistent with the database state stored in the rewritten AOF file.

The Redis server then sets up an AOF rewrite buffer, which is used after the server creates the child process. When the main thread finishes executing a write command, it sends the write command to both the AOF buffer and the AOF rewrite buffer.

When the child process completes the AOF rewrite, it sends a signal to the parent process. The parent process receives the signal and blocks the main process. The parent process then appends the command in the AOF rewrite buffer to the new AOF file, and the data state in the new AOF file is the same as that in the main process. Finally, delete the original AOF file, rename the new AOF file, and complete the replacement of the old and new files.

In other words, during the whole AOF background rewrite process, only the stage after receiving the signal will block the server process. In other times, AOF background rewrite will not block the parent process, which minimizes the impact of AOF rewrite on the server performance.

To summarize, draw a graph of the entire AOF non-blocking rewrite process:

RDB snapshot

Failure recovery using AOF log files requires executing all the commands in the logs one by one. If there are many operation logs, recovery will be slow.

Here comes another persistence mechanism that can be quickly recovered: RDB memory snapshots.

The so-called memory snapshot is to write the state of the data in memory at a certain moment to a disk file. Compared with AOF, RDB records data at a certain point in time, rather than commands, so when data recovery is done, RDB files can be read directly into memory and recovery can be completed quickly.

1. Generate an RDB snapshot

Redis has two commands that can be used to generate RDB snapshots:

  • Save: blocks the redis main thread and the server cannot process any command requests until the RDB file is created.

  • Bgsave: a child process is forked to create the RDB file, and the main thread continues to process command requests.

A full RDB snapshot is generated each time. The more full data is generated, the larger the RDB file is, and the more time it takes to write data to disks. So RDB snapshots are typically generated in the background using BGSave, which is the default configuration.

When a snapshot is created, data is first written to a temporary RDB file. When the new RDB file is written, Redis replaces the original RDB file with the new RDB file and deletes the old RDB file.

In addition, the save, BGsave, and bgrewriteaof commands are not executed at the same time, mainly to avoid concurrent contention or concurrent execution of a large number of disk writes affecting main thread performance.

2. Copy while writing

The BGSave child process is generated by the main thread fork and can share all memory data of the main thread. Once the BGSave child process runs, it starts reading the main thread’s memory data and writing it to an RDB file.

However, the main thread can also receive write commands, and the data in memory can be changed during snapshot creation, which can compromise the integrity of the snapshot.

To avoid blocking Write operations On the main thread, Redis uses copy-on-write (COW) technology:

  • If the main thread is a read operation, the main thread and the BGSave child process do not affect each other.
  • If the main thread modifies a piece of data, the piece of data will be changedcopyMake a copy of the data. The main thread then makes changes on this copy of data. The BGSave child process can continue to write the original data to the RDB file.

The copy-on-write technology enables the main thread to process write requests normally when the snapshot is executed. This ensures the integrity of the snapshot and does not affect the performance of the main thread.

3. Snapshot timing

RDB snapshot generation can be performed synchronously or asynchronously by manually invoking the save or bgsave commands.

In the Redis configuration file, multiple savepoints can be set with save, and at each savepoint, a specified number of keys are checked for changes, and if so, a new RDB snapshot is generated.

Such as:

  • save 3600 1: One key change occurs within 3600 seconds
  • save 300 100: 100 key changes in 300 seconds
  • save 60 10000: 10,000 key changes in 60 seconds

AOF versus RDB

1. Advantages of AOF

  • AOF can better protect data from loss. Generally, if the AOF is set to perform fsync every second, the data of a maximum of one second will be lost.

  • AOF log files are written in append-only mode, which has high write performance and is not easy to damage files.

  • When AOF files become too large, Redis automatically rewrites them in the background without affecting main thread performance.

  • AOF records command by command, and files are easy to read and analyze, making it ideal for emergency recovery in the event of catastrophic deletions. If you accidentally use the FLUSHALL command to flush everything out, as long as the AOF file hasn’t been rewritten, you simply stop the server, delete the last command, and restart Redis.

2. Disadvantages of AOF

  • AOF log files are usually larger than RDB data snapshot files for the same data. In addition, all commands are replayed during recovery, which is slower than RDB.

  • After AOF is enabled, it is generally configured to perform fsync every second. Compared with RDB, AOF has a greater impact on redis performance. However, fsync is usually performed once per second and still performs well.

  • AOF is not suitable for cold backup. First, data recovery is slow; second, regular backup is not convenient, and you may have to write complex scripts to do it.

3. Advantages of RDB

  • The RDB is a very compact file that holds data sets from a certain point in the past, with a new RDB file generated every once in a while. This characteristic is very suitable for cold backup and disaster recovery of data, such as every hour can save the data in the past 24 hours, every day to save data in the past 30 days, then you can send these files to some remote safety storage, when the problem can be restored to different points in time according to the demand data set.

  • The RDB snapshot is done in the child process of fork and has very little impact on the performance of the redis main thread providing read and write services. AOF writes to disk every time it executes a write command. Although it writes to the cache first and then fsync to disk, it is definitely slower than RDB.

  • When the data set to be restored is large, it is faster to restart and restore directly based on RDB files than aOF-based recovery.

4. RDB disadvantages

  • Generally, RDB snapshot files are generated every 5 minutes or longer. Once Redis is down, the data in the last 5 minutes may be lost. If you rely on RDB for the first-priority recovery scheme, more data will be lost.

  • The RDB execution requires a fork process. When the data is large, the fork process takes a long time, and the services provided to the client may be suspended for milliseconds or even seconds. So the frequency of RDB should not be much.

5. How to choose

If we want Redis to be used only as a pure memory cache, we can disable the persistence mechanism of RDB and AOF.

Of course, in a production environment, both AOF and RDB persistence mechanisms are commonly used:

  • If you only use RDB, you can lose a lot of data.

  • If only AOF is used, it is not suitable for cold standby and disaster recovery is not as fast as RDB.

  • So use a combination of AOF and RDB persistence mechanisms:

    • AOF ensures data loss and is the first choice for data recovery.

    • RDB is used for periodic cold backup and fast data recovery when AOF files are lost or unavailable.

If both RDB and AOF persistence mechanisms are used, they will be used when Redis restartsAOFTo rebuild the data, because the data in AOF is more complete.

AOF persistence

1. Enable AOF

AOF persistence is turned off by default, RDB persistence is turned on by default.

Aof-related configurations are as follows:

configuration The default value instructions
appendonly no Whether to enable AOF
appendfilename “appendonly.aof” AOF File name
appendfsync everysec The value can be always, everysec, or no

You simply set appendOnly Yes to enable AOF.

2. Verify AOF recovery data

After configuration modification, stop Redis, delete files in data directory, and restart Redis:

# cd /etc/init.d/

# ./redis_6379 stop

# rm -rf /var/redis/6379/*

# ./redis_6379 start
Copy the code

At this point you can see that an appendone.aof file has been generated in the data directory:

# cd /var/redis/6379/ && ls
appendonly.aof
Copy the code

Connect to the Redis client and write a command:

# redis-cli -h 192.168.241.130

# 192.168.241.130:6379> set name bojiangzhou
Copy the code

Looking at the AOF file, you can see that the command has been written:

[root@node0 6379]# cat /var/redis/6379/appendonly.aof
*2
$6
SELECT
$10 * 3$3
set
$4
name
$11
bojiangzhou
Copy the code

Kill -9 to kill the Redis process (simulate the redis failure and downtime), delete the PID file, and restart Redis:

# ps -ef|grep redisroot 3973 1 0 19:04 ? 00:00:00 /usr/local/bin/redis-server 192.168.241.130:6379 root 4039 2564 0 19:07 PTS /0 00:00:00 grep --color=auto redis
# kill3973-9

# rm -rf /var/run/redis_6379.pid 

# ./redis_6379 start
Copy the code

If the client connects to Redis, verify that the data is still there. Here is the data recovered from AOF:

#Redis - cli - h 192.168.241.130192.168.241.130:6379 > get the name "bojiangzhou"Copy the code

3. AOF rewrite

You can manually send the bgrewriteAOF command to perform AOF overrides, but the redis background usually does AOF overrides automatically.

AOF rewrite has the following parameters:

configuration The default value instructions
auto-aof-rewrite-percentage 100 Percentage increase in file size compared to the last rewrite
auto-aof-rewrite-min-size 64mb The minimum size of a file when AOF is overwritten. Default is 64MB
no-appendfsync-on-rewrite no When overwriting, AOF does not perform fsync
  • Rewrite the timing

The timing of AOF rewriting is controlled by the first two parameters. First, the AOF file should be larger than auto-aof-rewrite-min-size. Second, the size of AOF file is larger than that of the last AOF rewriting. Overwriting occurs only when the ratio is greater than auto-aof-rewrite-percentage.

For example, when an AOF file reaches 64MB, it can be overwritten, and then it will continue to append logs to the rewritten AOF file. When the rewritten AOF file reaches 128MB, the size of the file exceeds 64MB, and (128-64)/64 * 100% = 100%, and the increment ratio reaches 100%. You’re ready to rewrite.

  • no-appendfsync-on-rewrite

Bgrewriteaof involves a lot of disk operations. This causes the main thread to block while writing AOF files, so you can set the no-appendfsync-on-rewrite parameter:

① When set to no: AOF overwrites, the main thread executes fsync as defined in appendfsync. This does not lose data, but requires a certain amount of congestion.

Set appendfsync to yes: This is equivalent to setting appendfsync to no during rewrite, which does not block disk I/O but may lose some data.

Set to yes if the application system cannot tolerate delays and can tolerate a small amount of data loss. If the application system cannot tolerate data loss, set this parameter to no.

4. Repair damaged AOF files

When Redis append data to AOF file, the server may be down, which may cause AOF file damage. When Redis restarts, it will refuse to load the broken AOF file.

When this happens, back up the AOF files and fix the broken AOF files with the redis-check-aof –fix command, which probably deletes the problematic commands. Then restart Redis.

RDB persistence

1. RDB configuration

The RDB snapshot has the following configurations:

configuration The default value instructions
save save 3600 1

save 300 100

save 60 10000		 RDB snapshot checkpoint
stop-writes-on-bgsave-error yes Whether Redis stops receiving data after RDB fails to save data
rdbcompression yes Whether to compress RDB files for storage
rdbchecksum yes Whether to verify snapshot files
dbfilename dump.rdb RDB file name

2. Snapshot timing

RDB snapshots can be generated manually by using the save and bgSave commands, but are generally automatically generated by the Redis background.

RDB is enabled by default. If you want to disable RDB, you can configure an empty string for save:

save ""
Copy the code

The RDB executes snapshot checkpoints by default:

save 3600 1
save 300 100
save 60 10000
Copy the code
  • RDB snapshots are generated when 10000 keys change within 60 seconds
  • RDB snapshots are generated when 100 keys change within 300 seconds
  • If one key changes within 3600 seconds, an RDB snapshot is generated

We can set up multiple checkpoints or adjust the frequency of RDB snapshots as needed.

3. Verify RDB recovery data

The server only uses RDB files to restore data state when AOF is turned off, so turn AOF off first:

appendonly no
Copy the code

Then set up a checkpoint: RDB files are generated when 1 key changes within 10 seconds

save 10 1
Copy the code

Then stop Redis, delete the files in the data directory, and restart Redis:

# cd /etc/init.d/

# ./redis_6379 stop

# rm -rf /var/redis/6379/*

# ./redis_6379 start
Copy the code

The client then connects to Redis and writes a piece of data:

#Redis - cli - h 192.168.241.130192.168.241.130:6379> set name bojiangzhou OK 192.168.241.130:6379> get name bojiangzhou OKCopy the code

Then kill -9 to kill the Redis process within 10 seconds. No files are found in the data directory.

# ps -ef|grep redisroot 5829 1 0 20:30 ? 00:00:00 / usr/local/bin/redis - server 192.168.241.130:6379
# kill5829-9

# rm -rf /var/run/redis_6379.pid

# cd /etc/redis/6379 && ls
Copy the code

Note: Redis will immediately create a full RDB snapshot of the data in memory when it exits using redi-CLI shutdown as a safe exit mode.

If you restart Redis, you will find that the data is lost. This means that redis is down before the RDB snapshot can be generated, and the 10 seconds of data is lost.

#./redis_6379 start # redis-cli -h 192.168.241.130 192.168.241.130:6379> get name (nil)Copy the code

If you kill the redis process 10 seconds later, you will find that the dump. RDB file is generated in the data directory.

# cd /var/redis/6379/ && ls
dump.rdb
Copy the code

At this point, you will find that the data is still there, which verifies that the RDB file was used to restore the data.

After validation is complete, AOF: appendOnly Yes is enabled again.

Then delete the set checkpoint save 10 1.

The data backup

As mentioned earlier, RDB snapshots are suitable for cold backups. To prevent damage or loss of AOF and RDB files, regular backups of RDB files are also performed in production environments. Generally, the data is saved every hour for the past 24 hours, and every day for the past 30 days.

These files can be sent to some remote safety storage, simple point can also be in the native other directories, such as my below RDB file backup to local/usr/local/redis/snapshot directory.

1. Backup every hour

Back up dump. RDB every hour and delete data from 24 hours ago.

Create redis_rdb_hourly_backup.sh in /etc/redis:

#! /bin/sh

#Backup directory
dir=/usr/local/redis/snapshot

#Current date time (yyyyMMddHH)
cur_date=`date +%Y%m%d%H`

#Create a new directory
rm -rf $dir/$cur_date
mkdir -p $dir/$cur_date

#Copy the RDB file to the backup directory
cp /var/redis/6379/dump.rdb $dir/$cur_date

#Delete a directory created 24 hours ago
del_date=`date -d -24hour +%Y%m%d%k`
rm -rf $dir/$del_date
Copy the code

You can run the script once to verify that the directory for the current hour has been created and dump. RDB has been copied over.

# sh redis_rdb_hourly_backup.sh

# ls /usr/local/redis/snapshot/
2021090100

# ls /usr/local/redis/snapshot/2021090100/
dump.rdb
Copy the code

Then create the crontab schedule and run this script hourly:

# crontab -e

0 * * * * sh /etc/redis/redis_rdb_hourly_backup.sh
Copy the code

2. Make daily backups

Back up dump. RDB every hour and delete data from a month ago.

Create redis_rdb_daily_backup.sh in /etc/redis:

#! /bin/sh

#Backup directory
dir=/usr/local/redis/snapshot

#Current date time
cur_date=`date +%Y%m%d`

#Create a new directory
rm -rf $dir/$cur_date
mkdir -p $dir/$cur_date

#Copy the RDB file to the backup directory
cp /var/redis/6379/dump.rdb $dir/$cur_date

#Delete directories from a month ago
del_date=`date -d -1month +%Y%m%d`
rm -rf $dir/$del_date
Copy the code

Dump. RDB = dump. RDB = dump. RDB = dump.

# ls /usr/local/redis/snapshot/20210901, 2021090100,
# ls /usr/local/redis/snapshot/20210901
dump.rdb
Copy the code

Create a crontab schedule and run this script once a day:

# crontab -e

0 0 * * * sh /etc/redis/redis_rdb_daily_backup.sh
Copy the code

3. RDB restores data

If the redis process or server is down, simply restart the Redis process and restore data based on the AOF log file. If the synchronization policy is appendfSync Everysec, at most one second of data is lost.

If the current AOF and RDB files of Redis are corrupted or lost, the data can be restored with the recently backed up copy of the RDB.

If for some reason the online cache data is contaminated, we can also select the backup data of a certain hour to restore.

Then follow these steps to restore the data:

  • Stop Redis first

  • Modify the configuration file /etc/redis/6379.conf to disable AOF: appendonly no. Because Redis starts with AOF to recover data by default, AOF needs to be turned off

  • Delete appendone. aof and dump. RDB files in /var/redis/6379

  • Copy backup dump. RDB to /var/redis/6379

  • If you start Redis, you will find that the data is restored based on dump.rdb

  • AOF: config set appendonly yes the data in the memory is synchronized to the appendone. AOF file and the data in the AOF and RDB files are synchronized

  • Then manually modify the /etc/redis/6379.conf file and open AOF: appendonly yes

  • You can restart Redis again