First, the basic installation of Redis

Redis installation and server startup

#Basic environment installation, here is centos7
yum install gcc-c++
#Download, unzip, and compile. Here is the latest version 6.0.9 used.Wget https://download.redis.io/releases/redis-6.0.9.tar.gz tar XZF redis - 6.0.9. Tar. Gz CD redis - 6.0.9 make#The binaries now compiled can be found in the SRC directory. You can also find the redis. Conf file
mv redis.conf /etc/redis/redis.conf
#Then edit the configuration file vi /etc/redis/redis.confDaemonize yes # make Redis run bind 0.0.0.0 # for any address to access, of course, production based on the actual situation of the company. requirepass test123; You need to restart Redis to configure the password of redis. Config set requirepass test123 # Configure the password in redis-cli without restarting#Then start the Redis server as a configuration file.
redis-server /etc/redis/redis.conf
Copy the code

Client connection:

The default port is 6379. The default host is 127.0.0.1. If you need to enter a password, use the -a option
redis-cli -p 6379 -a password -h 127.0.0.1
# Close the server.
# 1. This can be turned off directly from the redis- CLI command line (without losing snapshot data). You can then use the ps command - ef | grep redis check to see if the server shut down
127.0.0.1:6379 > shutdown
# 2. You can directly enter the command in the terminal window to close, actually also using the client command.
redis-cli -p xxx -a xxx -h xxx shutdown
# 3. Can kill the process directly, using ps - ef | grep redis view then use kill 9 pid XXX to kill the process. Killing the abnormal shutdown directly will not achieve backup.
Copy the code

Two, Redis basic data types and application scenarios

2.1 type String

  1. Set the value of the key set key value [EX seconds] [PX milliseconds] [NX | XX]

    • EX seconds: Sets the expiration time of the key tosecondsSeconds. performset key value EX secondsThe effect is equivalent to executionsetex key seconds value 。
    • PX milliseconds: Sets the expiration time of the key tomillisecondsMilliseconds. Run set key value PX millisecondsPsetex key milliseconds Value 。
    • NX: Sets the key only when the key does not exist. performset key value NXThe effect is equivalent to executionsetnx key value 。
    • XX: Sets a key only when the key already exists.
    Set name "wangyun" EX 10 #=> Ok Set name to "wangyun" and expire in 10 seconds. PTTL name #=> 3212 Set name "wangyun" PX 10 # Set name to "wangyun" and expire after 10 milliseconds. Set not-exists-key "value" NX #=> The setting is successful if the Ok key does not exist. NX #=> nil set name "value" NX #=> nil Exists exists-key #=> 0 The key does not exist set exists-key "value" XX #=> nil because the key does not exist set failed.Copy the code

  2. Setnx Key value Is set only when the key does not exist

    Set the key value to value only if the key does not exist. If the key already exists, the SETNX command does nothing.

    Exists job #=> 0 setnx job "coding" #=> 1 This setting is successful because the key does not exist. Setnx job "move brick" #=> 0Copy the code

  3. Setex Key seconds value Sets the value of the key and its expiration time

    Set the value of the key to value and the lifetime of the key to seconds. If the key already exists, the SETEX command overwrites the existing value.

    setex name wangyun 10 #=> OK
#The above command is equivalent to the following two commands.SET name wangyun #=> OK EXPIRE name seconds #=>1Copy the code

  4. Psetex key milliseconds Value psetex key milliseconds Value

    This command is similar to the SETEX command, but it sets the lifetime of the key in milliseconds, rather than seconds, as the SETEX command does.

    psetex mykey 1000 "Hello" #=> OK
pttl mykey #=> 800
Copy the code

  5. Get key

    If key does not exist, return the special value nil; Otherwise, return the value of key.

    If the key key value is not a string, an error is returned because the GET command can only be used for string values.

    get db #=> nil
set db redis #=> OK
get db #=> "redis"
Copy the code

  6. Set the value of the key and return the old value getSet key Value

    Returns the old value of the given key. If the key has no old value, that is, it did not exist before it was set, then the command returns nil. The command returns an error if the key exists but is not a string.

    getset name wangyun #=> nil
getset name wy #=> "wangyun"
Copy the code

  7. Strlen key Returns the length of the string value stored by key

    Strlen returns the length of the string. Returns 0 if key does not exist. An error is returned when key is not a string.

    set mykey "Hello world" #=> OK
strlen mykey #=> 11
Copy the code

  8. Append key value to the end of the key

    If the key already exists and its value is a string, the APPEND command appends the value to the end of the key’s existing value. If the key doesn’t exist, APPEND simply sets the key’s value to value, as if it were SET key value.

    exists myphone #=> 0
append myphone apple #=>5
append myphone " xiaomi" #=> 12
Copy the code

  9. Change the key value setrange key offset value

    The string value stored by the key is overwritten with the value argument, starting with the offset.

    The SETRANGE command will ensure that the string is long enough to set value to the specified offset. If the key originally stored the string length is smaller than the offset (for example, the string is only 5 characters long, but you set offset to 10), Then the space between the original character and the offset will be filled with zerobytes (zerobytes, “\x00”).

    Set email "[email protected]" #=> OK setrange email 5 "@qq.com" #=> "[email protected]Copy the code

  10. Getrange key start end Getrange Key start end

    Returns the specified portion of the string value stored by key. The range of the string is determined by the start and end offsets (both included). A negative offset means counting from the end of the string, -1 for the last character, -2 for the penultimate character, and so on.

    SET greeting "hello, my friend" #=> OK getrange 0 4 #=> "Hello" Getrange 0 -1 #=> "Hello, my friend" from the first to the last getrange 0 100 #=>" Hello, my friend" from the range of the actual string. Excess parts are automatically omittedCopy the code

  11. Add 1 incR key to the value of key

    If the key does not exist, its value is initialized to 0 before INCR is executed.

    If the value stored by the key cannot be interpreted as a number, the INCR command returns an error.

    The INCR command is a string operation. Because Redis does not have a dedicated integer type, values stored by key are interpreted as decimal 64-bit signed integers when INCR is executed.

    set page_view 20 #=> OK
incr page_view #=> 21
#Incrby key increment Indicates the key value plus increment 
incrby page_view 23 #=> 44
#This command adds a float to the value of key and returns a string.Incrbyfloat page_view 2.3 #=> "46.3"Copy the code

  12. Decr key minus the value of key by 1

    If the key does not exist, the value of the key is initialized to 0 before DECR is performed.

    If the value stored by the key cannot be interpreted as a number, the DECR command returns an error.

    exists nums #=> 0
decr nums #=> -1
set num "1" #=> OK
decr num #=> 0
decrby num 23 #=> -23
decrby num -20 #=> -3
Copy the code

  13. Mset key value [key value]

    Set values for multiple keys simultaneously. If a given key already exists, the MSET overwrites the old value with the new value.

    MSET is an atomic operation. All given keys are set at the same time. There is no case where some keys are set but others are not.

    Mset date "2012.3.30" time "11:00 a.m." weather "sunny" #=> OK mget date time weatherCopy the code

  14. Msetnx key value [key value] Sets a value for all given keys if and only if none of the given keys exist.

    Sets a value for all given keys if and only if none of the given keys exist.

    The MSETNX command refuses to set all keys even if only one given key already exists.

    MSETNX is an atomic operation in which all given keys are either set or not set at all, and no third state is possible.

    MSETNX rmdbs "MySQL" nosql "MongoDB" key-value-store "redis" #=> 1
Copy the code

Type 2.2 hash

  1. Hset hash field value hset hash field value

    If the given hash table does not exist, a new hash table is created and the HSET operation is performed.

    If the field already exists in the hash table, its old value will be overwritten by the new value value.

    #When the HSET command creates a new field in the hash table and successfully sets its value, the command returns 1. If the field already exists in the hash table, and the HSET command successfully overwrites its old value with the new value, the command returns 0.
hset website google "www.g.cn" #=> 1
hset webside google "google.com" #=> 0
Copy the code

  2. Sets the value of the field field in the hash if it does not exist. hsetnx hash field value

    Set the field value to value if and only if it does not already exist in the hash table.

    If the given field already exists in the hash table, the command abandons the set operation.

    If the hash table does not exist, a new hash table is created and the HSETNX command is executed.

    #The HSETNX command returns 1 on success or 0 if the set operation is abandoned because the given domain already exists.hsetnx database key-value-store Redis #=> 1 hsetnx database key-value-store Riak #=> 0 hget database key-value-store #=> "Redis" so the command will not be executed if the given field already exists in the hash table.Copy the code

  3. Returns the value of the given field in the hash table. hget hash field

    If the given field does not exist in the hash table, or if the given hash table does not exist, the command returns nil.

    hset homepage redis redis.com #=> 1
hget homepage redis #=> "redis.com"
Copy the code

  4. Checks whether the given field exists in the hash table. hexists hash field

    The HEXISTS command returns 1 if the given field exists and 0 if it does not exist.

    hexists phone myphone #=> 0
hset phone myphone iphone #=> 1
hexists phone myphone #=> 1
Copy the code

  5. Some other methods used in hashing

    #Removes one or more specified fields from the hash table key. Non-existing fields are ignored. hdelhashThe field [field]...
hdel website google baidu #=> 2
hdel website not-exists-field #=> 0

#Returns the hash tablehashNumber of middle domains. hlenhash
hlen website #=> 1

#Returns the hash tablehashThe string length of the value associated with the given field field. hstrlenhash field 
hmset myhash f1 "HelloWorld" f2 "99" f3 "-256" #=> 0
hstrlen myhash f1 #=> 10

#For the hash tablehashThe value of field plus increment can be negative. hincrbyhash field increment
hexists counter page_view #=> 0
hincrby counter page_view 200 #=> 200
hget count page_view #=> "200"

#For the hash tablehashIncrement increment is added to the field field. hincrbyfloathash field incrementHset MyKey field 10.5 #=> 1 hincrByFloat mykey field 0.1 #=> 10.6
#Set multiple field-value pairs to the hash table at the same timehashIn the. hmset key field value [field value]
#If the key does not exist, an empty hash table is created and the HMSET operation is performed.
# 当 hashError returned when not a hash type.
hmset website google www.google.com yahoo www.yahoo.com #=> OK

#Returns the hash tablehash, the value of one or more given fields. hmgethash field [field ...]
#Returns nil if the given field does not exist in the hash table.
hmset pet dog "doudou" cat "nounou" #=> OK 
hmget pet dog cat lion #=> "doudou" "nounou" nil

#Returns the hash tablehashAll fields in. hkeyshash
hmset website google www.google.com yahoo www.yahoo.com #=> OK
hkeys website #=> "google" "yahoo"

#Returns the hash tablehashValues for all fields in. hvalshash
hvals website #=> "www.google.com" "www.yahoo.com"

#Returns the hash tablehashIn, all fields and values. hgetallhashHset people jack" Jack Sparrow" #=> OK hset people gump "Forrest gump "#=> OK hgetall people 1) "Jack" # field 2) "jack Sparrow" # value 3) "gump" 4) "Forrest gump"Copy the code

2.3 list

  1. Insert one or more values into the header of the list Lpush list value [value…]

    If there are multiple values, each value is inserted in the header from left to right: For example, if you execute the command LPUSH myList a b c on the empty list myList, the values of the list will be C B A, This is equivalent to atomically executing LPUSH myList a, LPUSH myList b, and LPUSH mylist C.

    If the key does not exist, an empty list is created and an LPUSH operation is performed.

     lpush languages python
Copy the code

3. Redis persistence

3.1 Differences between RDB and AOF in Persistence Mode:

RDB persistence refers to writing a snapshot of an in-memory data set to disk at a specified interval.

Specific process:

  1. Redis uses fork to make a copy of the current process (parent process) (child process);
  2. The parent process continues to receive and process commands from the client, while the child process begins to write data from memory to temporary files on the hard disk.
  3. When the child process finishes writing all data, it replaces the old RDB file with the temporary file. The snapshot operation is complete.

AOF persistence records every write and delete operation processed by the server in the form of logs. The query operation is not recorded in the form of text. You can open the file to see detailed operation records. Redis can also rewrite AOF files in the background so that the size of the AOF file does not exceed the actual size needed to store the state of the dataset.

3.2 Detailed process of RDB and AOF persistence

3.2.1 PERSISTENCE of RDB

Triggering RDB persistence can be manually triggered or automatically triggered

Manual trigger:

  1. The save command: blocks the current Redis server until the RDB process is complete. It will cause a long block for instances with large memory and is not recommended for online environments
  2. Bgsave: The Redis process forks to create a child process. The RDB persistence process is the responsibility of the child process and ends automatically. Blocking occurs only during the fork phase, which is usually very short

Automatic trigger:

  1. Use the save related configuration in the configuration file, such assave m n. Indicates that it is automatically triggered when the data set is modified for n times within m secondsbgsave 。
  2. If a full replication is performed on the secondary node, the primary node automatically performs the replicationbgsavegenerateRDBThe file is sent to the slave node.
  3. performdebug reloadCommand reloadRedis, will also automatically triggersaveOperation.
  4. It is executed by defaultshutdownIf the command is not enabledAOFThe persistence function is performed automaticallybgsave.

Bgsave is the mainstream rDB-triggered persistence method. The flow chart is as follows:

Specific procedures performed by BGSave:

  1. performbgsaveCommand, the Redis parent determines if there are currently executing children, such as RDB/AOF, and if there arebgsaveThe command returns directly.
  2. Parent process executionforkAction to create a child process,forkThe parent process will block during the operationinfo statsCommand to seelatest_fork_usecOption to get the most recent oneforkOperation time, in microseconds
  3. The parent processforkAfter completion,bgsaveThe command returns the “Background saving Started” message and stops blocking the parent process. You can continue to respond to other commands.
  4. The child process creates an RDB file, generates a temporary snapshot file based on the parent process memory, and then atomic replaces the original file. performlastsaveCommand to obtain the time when the RDB was generated for the last timerdb_last_save_timeOptions.
  5. The process signals completion to the parent process, which updates the statistics.
3.2.2 AOF Persistence

The main role of AOF is to solve the real-time of data persistence, which has been the mainstream way of Redis persistence.

AOF workflow operations: command write (append), file synchronization (sync), file rewrite (rewrite), load (restart)

Specific process description:

  1. All write commands are appended toaof_bufIn (buffer).
  2. The AOF buffer synchronizes data to disks based on the corresponding policy.
  3. As AOF files become larger, they need to be rewritten periodically to achieve compression.
  4. When the Redis server restarts, AOF files can be loaded for data recovery.

Question and answer:

  1. Why does AOF append commands to aOF_buf?

    Redis uses single-threaded response commands, so if every time you write an AOF file, the command is appended directly to the disk, the performance depends entirely on the current disk load. Writing to buffer AOF_buf first has another benefit. Redis can provide multiple buffer synchronization policies to balance performance and security.

  2. Why can rewritten AOF files become smaller?

    • Data that has timed out in the process is no longer written to the file.
    • The old AOF file contains invalid commands, such as del key1, hdel key2, srem keys, set A111, set A222, etc. Overrides are generated directly using in-process data, so that the new AOF file retains only the write commands for the final data.
    • Multiple write commands can be combined into one, for example:lpush list a,lpush list b,lpush list cCan be translated into:lpush list a b c. To prevent client buffer overflow caused by a large single command, thelist,set,hash,zsetAnd other types of operations, with 64 elements as the boundary split into multiple.

  3. How is the AOF rewrite process triggered manually and automatically?

    • Manual trigger: Directly invoke the bgrewriteaof command.
    • Automatic trigger: Determine the automatic trigger time according to the auto-aof-rewrite-min-size and auto-aof-rewrite-percentage parameters.
    • Auto-aof -rewrite-min-size: indicates the minimum size of a file when aof rewriting is run. The default size is 64MB.
    • Auto-aof-rewrite-percentage: specifies the ratio of the current AOF file space (AOF_current_size) to the aOF file space after the last rewrite (aOF_base_size).

  4. What if the AOF file is wrong?

    The server may stop while the program is writing to the AOF file. If the AOF file fails due to the stop, Redis will refuse to load the AOF file when it restarts to ensure that data consistency is not compromised.

    When this happens, you can fix the faulty AOF file by:

    • Create a backup of the existing AOF files.

    • Use the redis-check-aof program that comes with Redis to fix the original AOF file.

    • Restart the Redis server, wait for the server to load the repaired AOF file, and restore the data.

Recovery process after restart:

Specific process description:

  1. If AOF persistence is enabled and AOF files exist, the AOF file is loaded preferentially and the following log is displayed:

    DB loaded from append only file: 5.841 seconds
Copy the code

  2. When AOF is disabled or the AOF file does not exist, the RDB file is loaded and the following log is displayed:

DB loaded from disk: 5.586 seconds
Copy the code
  1. After the AOF/RDB file is successfully loaded, Redis is successfully started.
  2. Redis fails to start and prints error messages when there are errors in the AOF/RDB file.

3.3 Advantages and Disadvantages of RDB(Redis DataBase)

Advantages:

  1. Once done this way, your entire Redis database will contain only one file, which is perfect for file backups. For example, you might want to archive the last 24 hours of data every hour and the last 30 days of data every day. With this backup strategy, we can easily recover from catastrophic system failures.
  2. RDB is a great choice for disaster recovery. It is very easy to compress a single file and transfer it to another storage medium.
  3. Maximize performance. As for Redis server, the only thing it needs to do to start persistence is fork out the child process, and then the child process will do the work of persistence, so that the server process can not perform IO operations.
  4. Compared to AOF, RDB starts more efficiently if the data set is large.

Disadvantages:

  1. If you need to avoid losing data in the event of a server failure, the RDB is not for you. Although Redis allows you to set different save points to control how often RDB files are saved, it is not an easy operation because RDB files need to hold the state of the entire data set. So you’ll probably save your RDB file at least once every 5 minutes. In this case, you could lose several minutes of data in the event of a malfunctioning outage.
  2. Redis requires it every time the RDB is savedfork()Create a child process that does the actual persistence. When the data set is large,fork()It can be time consuming, causing the server to stop processing the client in so-and-so milliseconds; If the data set is very large and CPU time is very tight, this stop time can even take a full second. Although the AOF rewrite also needs to be donefork()However, no matter how long the interval between the AOF rewrite is, the durability of the data will not be compromised.

3.4 Advantages and disadvantages of AOF(short for AppendOlny File)

Advantages:

  1. This mechanism can lead to higher data security, that is, data persistence. Redis provides three synchronization policies: synchronization per second (the default), synchronization per change, and unsynchronization. In fact, synchronization per second is also done asynchronously, and it is very efficient, except that if the system goes down, the data modified within a second will be lost. With each change synchronization, we can think of it as synchronous persistence, meaning that every data change that occurs is recorded to disk immediately. Predictably, this approach is the least efficient.
  2. The AOF file is an append only log file, so writing to the AOF file is not requiredseek, even if the log contains incomplete commands for some reason (the disk is full at write time, the write stops halfway, etc.),redis-check-aofTools can also easily fix this problem.
  3. Redis can automatically rewrite the AOF in the background when the AOF file becomes too large: the rewritten new AOF file contains the minimum set of commands needed to restore the current data set. The entire rewrite operation is absolutely safe because Redis continues to append commands to existing AOF files while creating new AOF files, and the existing AOF files will not be lost even if an outage occurs during the rewrite. Once the new AOF file is created, Redis switches from the old AOF file to the new AOF file and starts appending the new AOF file.
  4. AOF files orderly store all writes to the database in the Redis protocol format, so the contents of AOF files are easy to read and parse. Exporting AOF files is also very simple: For example, if you accidentally execute the FLUSHALL command, as long as the AOF file isn’t overwritten, stop the server, remove the FLUSHALL command at the end of the AOF file, and restart Redis, You can restore the data set to the state it was in before the FLUSHALL execution.

Disadvantages:

  1. AOF files are usually larger than RDB files for the same data set. RDB can recover large data sets faster than AOF.
  2. Depending on the synchronization strategy used, AOF may be slower than RDB. While synchronization per second performance is still very high under normal conditions, turning off synchronization allows THE AOF to be as fast as the RDB, even under heavy load. However, RDB can provide more guaranteed maximum latency when handling large write loads.
  3. AOF has had a bug in the past where an AOF file could not restore the dataset as it was saved when it was reloaded due to certain commands. The test suite adds tests for this: they automatically generate random, complex data sets and reload them to make sure everything is okay. Although this kind of bug is not common in AOF files, RDB bugs are almost impossible by comparison.

3.5 How to choose RDB and AOF

Summary of RDB and AOF:

How to choose:

  1. The criteria for choosing between the two options is whether the system is willing to sacrifice some performance in exchange for higher cache consistency (AOF), or whether the system is willing to disable backup when the write operation is frequent in exchange for higher performance, and then perform backup (RDB) when the save operation is run manually. RDB is more eventually consistent. But the production environment is more of a combination of the two.
  2. Redis can also use both AOF and RDB persistence. In this case, when Redis restarts, it will use AOF files to restore the dataset in preference, since AOF files usually hold more complete datasets than RDB files.
  3. If large-scale data recovery is required and the integrity of data recovery is not very sensitive, RDB is more efficient than AOF. The downside of RDB is that data can be lost after the last persistence.

3.6 Configuration

3.6.1 RDB Persistence Configuration:

Redis dumps a snapshot of the dataset into the dump. RDB file. The Redis server can also change the frequency of snapshot dump via the configuration file. After opening the 6379.conf file, we can search for save and see the following configuration information:

  1. Save 900 1 # After 900 seconds (15 minutes), if at least one key has changed, the memory snapshot is dumped.
  2. Save 300 10 # Dump the memory snapshot after 300 seconds (5 minutes) if at least 10 keys have changed.
  3. Save 60 10000 # Dump the memory snapshot after 60 seconds (1 minute) if at least 10000 keys have changed.

The relationship between conditions is “or”. If one of the conditions is met, a snapshot is taken.

By default, Redis stores snapshot files in the dump. RDB file in the current directory (you can view them by entering config get dir in the Redis – CLI command line). You can specify the storage path and filename of the snapshot file by specifying dir and dbfilename respectively.

3.6.2 AOF Persistence Configuration:

Enable AOF persistence in the configuration file:

appendonly yes
Copy the code

The directory is set using the dir parameter. The default filename is appendone. AOF, which can be modified using the appendfilename parameter:

appendfilename appendonly.aof
Copy the code

By default, the AOF mode provides three persistence schemes:

  1. Appendfsync always # Writes to the AOF file every time a data change occurs, so its performance is also affected.
  2. Appendfsync everysec # when setappendfsync 为 everysecBy default, Redis will do this every secondfsyncCall to write the data in the buffer to disk. But when this timefsyncWhen the call duration exceeds 1 second. Redis will take a delayfsyncWait a second. That is, in two secondsfsyncThis timefsyncIt doesn’t matter how long it takes. At this time due tofsyncThe file descriptor is blocked, so the current write operation is blocked. The upshot is that, in most cases, Redis will do it every secondfsync. In the worst case, it happens every two secondsfsyncOperation. This operation is called in most database systemsgroup commitTo combine data from multiple write operations and write logs to disks at one time.
  3. Appendfsync no # Redis does not actively call fsync to synchronize the AOF log content to disk, so it all depends on OS debugging. On most Linux operating systems, fsync is performed every 30 seconds to write the data in the buffer to disk.