Why is #Redis fast?

Memory storage: Redis uses in-memory storage with no overhead on disk IO

Single-threaded implementation: Redis uses a single thread to process requests, avoiding the overhead of thread switching and lock resource contention between multiple threads

Non-blocking IO: Redis uses multiplexing IO technology to select the optimal IO implementation in Poll, Epool, and kqueue

Optimized data structure: Redis has many implementations of optimized data structure that can be directly applied. The application layer can directly use the native data structure to improve performance

Data structure operation

String

operation

SET key value MSET key value [key value...] // SET key value NX // store a non-existing string key value GET key // obtain a string key value MGET key [key...] DEL key [key...] EXPIRE key seconds atom add/subtract INCR key atom add/subtract INCR key atom add/DECR key // INCRBY key Increment // Add Increment DECRBY key Decrement to the values stored in the key // Subtract Decrement from the values stored in the keyCopy the code

scenario

  1. Single-value or object caching

  2. A distributed lock

SETNX product:10001 true SETNX product:10001 true // 0 failed to obtain the lock... Performing the service operation DEL product:10001 // Releasing the lock after performing the service SET product:10001 true ex 10 Nx // Prevent deadlock caused by unexpected program terminationCopy the code

RedisDB Data structure

BitMap

Redis actually only supports 5 data types. There is no such type as BitMap, which is based on the string type of Redis.

In Redis, Bitmaps can be thought of as an array of bits, each cell of which can store only zeros and ones. The subscripts of the array are called offsets in Bitmaps. Save a space

operation

# set a value where value can only be0 和 1Setbit key offset value getbit key offset value within the specified range is1The number of# start and end are in bytesBitcount key start end ## operations Displacement operator, enumeration valueAND with the operation & OR OR operation | XOR (exclusive OR ^ NOT take the ~# result The computed result is stored in this key# key1... Keyn can be multiple keys, separated by Spaces,notWhen BITOP processes strings of different lengths, the missing part of the shorter string is treated as one key0. The return value is the length (in bytes) of the string saved to destkey, which is equal to the length of the longest string in the input key. Bitop [operations] [result] [key1] [keyn...] # return the first occurrence of the specified value in the specified key (0/1[key] [value] # return the length of a bytestrlen [key]
Copy the code

Note, however, that the maximum length of the string in Redis is 512 MB, so the offset value of the BitMap is also capped. The maximum value is:

8 * 1024 * 1024 * 512 = 2^32

Since C stores a delimiter at the end of a string, the upper limit of a BitMap’s offset is actually:

(8 * 1024 * 1024 * 512) -1 = 2^ 32-1

scenario

  1. User sign in

Many sites offer check-ins and need to show the last month’s check-ins, which can be done using a BitMap. Offset = (Today is the day of the year) % (the number of days in the year), key = Year: user ID.

  1. Active User Statistics (user logins)

Use the date as the key, and then the user ID is offset, set to 1 if active that day. You can specify by yourself how active you are.

If 20201009 active user status is: [1,0,1,1,0]

The total number of active users for two consecutive days:

Statistics of active users from 20201009 to 20201010:


bitop and dest1 20201009 20201010Dest1 The median value is1Offset is the ID of the active user for two consecutive days bitcount dest1Copy the code

Statistics of active users from 20201009 to 20201010:

bitop or dest2 20201009 20201010 
Copy the code
  1. Check whether the user is online

If you need to provide an interface to query whether the current user is online, you can also consider using BitMap. The user ID is offset. If the user is online, the value is set to 1. If the user is offline, the value is set to 0.

SDS

The bottom implementation of String is simple dynamic String SDS (Simple Dynamic String), which can be modified. Similar to ArrayList in Java, it preallocates redundant space to reduce frequent memory allocations.

String in key value in Redis and String in list are all strings.

  1. Implement bloom filter

Developer.aliyun.com/article/773…

Structure:

 struct sdshdr{
   int len;
   int free;
   char buf[];
}
Copy the code

Why redis uses SDS instead of C strings

  1. Evaluates the difference in string length

For C, the way to calculate the length of the string is to traverse and stop at 0, so the complex pair is O(n), while SDS directly stores the length of the string and the complexity is O(1).

  1. Ensure the security of storing binary data

Because SDS is not marked with 0 ending, when storing binary data of video and audio, it will not terminate in the case of \0, which naturally ensures binary security.

  1. Memory management strategy (pre-allocated memory and lazy space release strategy)

Redis is a high-speed cache database, which requires frequent operations on strings. If the memory allocation is wrong, it will lead to serious consequences. Even if the memory allocation is ok, frequent memory allocation is very time-consuming, so it should be avoided

Lazy space release strategy

The lazy space release strategy is first used in SDS, which is used to optimize the string shortening operation of SDS. When shortening a string saved by SDS, the program does not immediately use full memory allocation to reclaim the shortened extra bytes. Instead, it uses the free member of the table header to record the bytes and wait for future use. The source code is as follows

void sdsclear(sds s) {  // Reset SDS BUF space, lazy release
    struct sdshdr *sh = (void*) (s-(sizeof(struct sdshdr)));
    sh->free += sh->len;    Len = new free. Len = new free
    sh->len = 0;            // Used space becomes 0
    sh->buf[0] = '\ 0';         // The string is null
}
Copy the code

Preallocated memory

If the character string length is less than 1 MB, the space is doubled, that is, 100% redundant space is reserved. When the length exceeds 1 MB, only 1 MB more redundant space is allocated for each expansion to avoid excessive redundant space waste.

  1. C compatible function library (string is automatically followed by \0)

SDS architecture after version 3.2

En and free are both int types, which are both 4byte, i.e. 32bit, and can represent a range of about 4.2 billion, which greatly causes a waste of space. Therefore, SDS has been changed to some extent after 3.2, and corresponding types will be created according to different string lengths

typedef char *sds;

/* Note: sdshdr5 is never used, we just access the flags byte directly. * However is here to document the layout of type 5 SDS strings. */
struct __attribute__(((packed__)) sdshdr5 {
    unsigned char flags; /* 3 lsb of type, and 5 msb of string length */
    char buf[];
};
struct __attribute__(((packed__)) sdshdr8 {
    uint8_t len; /* used */
    uint8_t alloc; /* excluding the header and null terminator */
    unsigned char flags; /* 3 lsb of type, 5 unused bits */
    char buf[];
};
struct __attribute__(((packed__)) sdshdr16 {
    uint16_t len; /* used */
    uint16_t alloc; /* excluding the header and null terminator */
    unsigned char flags; /* 3 lsb of type, 5 unused bits */
    charbuf[]; }; .Copy the code

Sdshdr5 represents the data length with 5 bit bits, sdSHdr8 represents the data length with 8bit bits, and so on. The memory allocation of SDSHdr5 is shown in the figure,flag occupies 1 byte, the first three bits of storage type of 8bit, The last five digits of flag represent len only in sdshdr5. The actual meaning of flag is to store type information, a separate SDSHDR5 can further save memory.

SDS is a continuous space, so the pointer to the string can be moved forward by one position to obtain the type information, and then the len information.

When the length of data to be stored exceeds 31, sdSHDR8 is used to represent the memory allocation of SDSHDR8

3. Counter (read volume) or distributed system global serial number

INCR article:readcount:{article id}

GET article:readcount:{article id}

INCRBY orderId 1000 // Redis generates serial numbers in batches to improve performance

  1. Web session sharing

Spring Session + Redis implements session sharing

Hash

HSET key field value HSETNX key field Value HSETNX key field Value HMSET key field value [field value... HGET key field HMGET key field [field...] HDEL key field [field...] HGETALL key HINCRBY key field increment HINCRBY key field increment HLEN key HGETALL key HINCRBY key field increment Increment increment increment increment increment increment increment increment increment increment increment increment increment increment increment increment increment incrementCopy the code

scenario

  1. Object cache shopping cart
 
HMSET  user  {userId}:name  zhuge  {userId}:balance  1888
HMSET  user  1:name  zhuge  1:balance  1888
HMGET  user  1:name  1:balance  
Copy the code
  1. Advantages and disadvantages compared to String

Advantages:

  1. Facilitate the management
  2. It consumes less memory and CPU

Disadvantages:

  1. You cannot set an expiration time for each field, but only for the key
  2. Redis cluster architecture is not suitable for large-scale use, because if the object of hash structure is very large, it can only be stored on a node of the cluster, resulting in too much pressure to access a single machine. However, it can be optimized to make the key be stored in segments using certain rules.

List

operation

LPUSH key value [value ...] RPUSH key value [value...] RPUSH key value [value...] RPOP Key // Remove and return the first element of the key list. // Remove and return the last element of the key list. LRANGE key start stop BLPOP key [key...] [key...] [key...] [key...] BRPOP key [key...] BRPOP key [key...] BRPOP key [key...] Timeout // If there is no element in the list, block and wait timeout seconds. If timeout=0, block and waitCopy the code

scenario

  1. Stack (Microblog information flow)

Stack = LPUSH + LPOP

  1. The queue

Queue = LPUSH + RPOP

  1. Blocking queue

Blocking MQ = LPUSH + BRPOP

ziplist

Ziplist is a specially coded two-way linked list designed to improve storage efficiency. Ziplist can be used to store strings or integers, where integers are encoded as true binary representations rather than as sequences of strings. It can provide push and POP operations at both ends of the table with O(1) time complexity.

Disadvantages of ordinary linked lists

  1. Too many address Pointers take up extra memory
  2. The discontinuity results in memory fragmentation, resulting in memory waste

The data structure

The parts are contiguous in memory and have the following meanings:

  1. Zlbytes: 32bits, indicating the total number of bytes (including the four bytes) occupied by ziplist.
  2. Zltail: 32bit: indicates the offset bytes of the last entry in the Ziplist table. Makes it easy to find the last item (without traversing the entire Ziplist) and to perform a push or pop at the end of the Ziplist quickly.
  3. Zllen: 16 bits: indicates the number of entries in ziplist. The zllen field has only 16 bits, so the maximum that can be expressed is 2^16-1. It is important to note that if the number of items in ziplist exceeds the maximum number of 16 bits, ziplist can still be represented. How do I represent that? Zllen is the number of items in ziplist if it is less than or equal to 2^16-2 (i.e., not equal to 2^16-1). Otherwise, that is equal to 16 bits are all 1, then zllen does not represent the number of data items, at this time to know the total number of data items in ziplist, then you must traverse the ziplist from beginning to end each data item, to count out.
  4. Entry: Indicates the data entry that stores data. The length varies. An entry also has its own internal structure, which will be explained later.
  5. Zlend: The last byte of ziplist, which is a closing flag and has a fixed value of 255.

Let’s look again at the composition of each data item

:

<prevrawlen><len><data>

We see two more fields before the actual data () :

: Indicates the total number of bytes occupied by the previous data item. The purpose of this field is to allow Ziplist to traverse from back to front (from the position of the latter item, just offset prevrawlen bytes forward to find the previous one). This field is variable-length encoded. : represents the data length of the current data item (that is, the length of the part). Variable length coding is also used. So how do and encode variable length?

Ziplist is designed to put items next to each other into a contiguous memory space, which is not very good at making changes. Any changes to the data can trigger memory realLOC, which can lead to memory copying.

ziplist

quicklist Zhangtielei.com/posts/blog-…

Quicklist was introduced after version 3.2.

According to the above mentioned, Ziplist will introduce frequent memory application and release, while LinkedList will also cause memory waste due to Pointers, and each node exists independently, resulting in a lot of memory fragments. Therefore, Combining the characteristics of the two structures, quickList is designed.

Set unordered Set

operation

SADD key member [member ...] SREM key member [member...] SREM key member [member... Smember key smember key SCARD key SISMEMBER key member SRANDMEMBER key [count] SPOP key [count] SRANDMEMBER key [count] SINTER key [key...] // Intersection operation SINTERSTORE destination key [key..] SUNION key [key..] SUNIONSTORE destination key [key...] SDIFF key [key...] // SDIFFSTORE destination key [key...] // Store the difference set result ina new collection destinationCopy the code

scenario

  1. Do the intersection search set

Such as lottery procedures, wechat praise, common attention, may know, screening and so on

Zset (sorted_set) A collection of sorted fields with score values

operation

ZSet ZADD key score member [[score member]... // add ZREM key member [member...] ZINCRBY key increment Member ZINCRBY key increment Member ZINCRBY key increment member ZINCRBY key increment member ZRANGE key start stop [WITHSCORES] ZRANGE key start stop [WITHSCORES] ZREVRANGE key start stop [WITHSCORES] ZREVRANGE key start stop [WITHSCORES] ZREVRANGE key start stop [WITHSCORES ZUNIONSTORE destkey destkey [key...] ZUNIONSTORE destkey [key...] ZINTERSTORE destkey numkeys key [key... // Calculate the intersectionCopy the code

scenario

  1. list
Zset set operation to implement leaderboards 1) click news ZINCRBY hotNews:20190819 1 Guardian Hong Kong 2) display the top 10 of the day ZREVRANGE hotNews:20190819 0 9 WITHSCORES 3) seven day search list calculation ZUNIONSTORE hotNews:20190813-20190819 7 hotNews:20190813 hotNews:20190814... ZREVRANGE hotNews:20190813-20190819 0 9 WITHSCORESCopy the code

Is Redis single threaded?

Redis single-threaded network IO and key/value read/write modules are executed by a single thread, while other modules such as persistence, asynchronous deletion, cluster data synchronization, etc., are executed by additional threads. In RedIS 6, network IO can be multithreaded, which is disabled by default

Why is Redis single thread so fast?

  1. Memory-based operation
  2. IO multiplexing (To be added)
  3. Reduced CPU context switching overhead

scan Bottom layer to be replenished

SCAN cursor [MATCH pattern] [COUNT count]

The scan parameter provides three parameters. The first parameter is the cursor integer value (hash bucket index value), the second parameter is the key’s regular pattern, and the third parameter is the number of keys traversed at a time (reference value, the number of underlying traversals is not necessarily the number of results that meet the criteria). The cursor value is 0 for the first traversal, and then the first integer value in the returned result is used as the cursor for the next traversal. Iterate until the returned cursor value is 0.

Note: Scan is not perfect, however. If keys change (add, delete, modify) during scan, the traversal effect may encounter the following problems: The newly added keys may not be traversed, and repeated keys may be traversed, which means that scan cannot guarantee the complete traversal of all keys, which we need to consider during development

The SCAN directive is a series of instructions that, in addition to traversing all keys, can also traverse a specified collection of containers.

Zscan traverses the zset set elements,

Hscan traverses the elements of the Hash dictionary,

Sscan traverses the elements of the set.

Note:

The first argument to the SSCAN, HSCAN, and ZSCAN commands is always a database key.

The SCAN command does not need to provide any database keys in the first parameter — it iterates over all database keys in the current database.

How to locate the big key?

  1. Write a script to iterate over the keys, get the data structure of the key by type, and scan the first N names of each type by len or size

If it is a string, use strlen;

If the structure is a list, check by llEN.

If it is a hash structure, check by hlen.

If the structure is set, scard is used to determine.

If it is sorted set, check it by zcard. 2. redis-cli

Redis -cli - Bigkeys will analyze keys sizeCopy the code

Redis-cli –bigkeys -i 0.1 0.1 seconds per 100 sleep scans to prevent increased OPS (redis-cli –bigkeys -i 0.1

Pay attention to

This method ensures that the longest key is found, but does not necessarily require the largest memory usage.

There are two list keys: numberlist – [0,1,2], stringlist – [” 123456789123456789 “],

Because of llEN, numberList is greater than StringList.

Stringlist is actually more memory intensive. The other three data structures hash, set, and sorted set have this problem.

View slow logs and logs about executing slow commands

SLOWLOG subcommand [argument]

Subcommands include:

Get, slowlog get [argument], gets the number of slow logs specified in the argument argument.

Slowlog len, the total number of slow logs.

Reset, usage: slowlog reset, clears slow logs.

redis-cli slowlog get 5

Slowlog-log-slower indicates the time required for a command to be saved to slowlog. You can run the config set slowlog-log-slower than 2000 command to set slowlog-slower than 2000 without restarting Redis. The default is 1 s

Note: The units are subtle, 2000 subtle is 2 milliseconds.

Rename -command renames a dangerous command

Rename -command flushdb flushddbb Flushes the DATABASE

rename-command flushall flushallall

rename-command keys keysys

Check the number of keys

Last line of dbsize or info

Redis persistence

RDB Snapshot (Snapshot)

By default, Redis stores in-memory database snapshots in a binary file named dump.rdb.

You can set Redis to automatically save the dataset once the condition of “at least M changes to the dataset in N seconds” is met.

For example, the following Settings will automatically save the data set if Redis meets the condition that at least 1000 keys have been changed within 60 seconds:

Save 60 1000 // To disable RDB, comment out all save policiesCopy the code

The dump. RDB file can be generated manually by running the save or bgsave command on the Redis client. Each time the command is executed, all redis memory snapshots are taken to a new RDB file and the original RDB snapshot file is overwritten.

Bgsave’s copy-on-write (COW) mechanism

Redis uses copy-on-write (COW) technology provided by the operating system to process Write commands while generating snapshots. In simple terms, the BGSave child process is generated by the main thread fork and can share all memory data of the main thread. Once the BGSave child process runs, it starts reading the main thread’s memory data and writing it to an RDB file. At this point, if the main thread also reads these data, then the main thread and the BGSave child do not affect each other. However, if the main thread modifies a piece of data, that piece of data is copied, making a copy of that data. The bgSave child then writes the copy to the RDB document, while the main thread can still modify the original data directly.

AOF (append-only file)

Snapshots are not durable: If Redis is down for some reason, the server loses data that was recently written to and is still not saved to the snapshot. As of version 1.1, Redis added a fully durable persistence method: AOF persistence, logging each modified instruction in the file appendone.aof (write to OS cache first and fsync to disk at intervals)

Note that if you run a set command with an expiration time, the aOF file does not record the original command executed, but the timestamp of the expiration of the key

Appendonly yes appendfsync always: Fsync is executed every time a new command is appened to the AOF file, very slow and very safe. Appendfsync everysec: Fsync once per second, fast enough, and only lost in case of failure1Seconds of data. Appendfsync no: passes data to the operating system without fsync. The faster and less safe option.Copy the code

AOF rewrite

There may be too many useless instructions in AOF files, so AOF periodically generates AOF files based on the latest data in memory

Such as:

127.0. 01.:6379> incr readcount
 (integer)13 127.0.0.1:6379 > incrreadcount
 (integer)2 127.0.0.1:6379 > incrreadcount
 (integer)3 127.0.0.1:6379 > incrreadcount
 (integer)4 127.0.0.1:6379 > incrreadcount
 (integer) 5
Copy the code

After the rewrite:

*3
 $3
 SET
 $2
 readcount
 $1
 5
Copy the code

The following two configurations control the frequency of AOF automatic rewriting

1# auto ‐ aof ‐ rewrite ‐ min ‐ size 64 MB// Aof files must be at least 64MB before they are automatically rewrittenSoon, there's not much point in rewriting2# auto ‐ aof ‐ rewrite ‐ percentage100 // if the size of the aOF file has increased by 100% since the last rewrite, the rewrite will be triggered again
Copy the code

Of course, AOF can also be manually rewritten. Enter the Redis client and run the bgrewriteAOF command to rewrite AOF

Note that AOF overwriting Redis forks a subprocess to do it (similar to the BGsave command) without much impact on normal redis command processing

RDB and AOF

Command RDB AOF

Both production environments can be enabled. If there are both RDB files and AOF files when Redis is started, aOF files will be preferred to recover data because AOF is generally more complete.

Redis 4.0 Hybrid Persistence

When rebooting Redis, we rarely use RDB to restore memory state because a lot of data is lost. We usually use AOF log replay, but replay AOF log performance is much slower than RDB, so it can take a long time to start up with a large Redis instance. Redis 4.0 addresses this problem with a new persistence option, hybrid persistence. Mixed persistence can be enabled by the following configuration (AOF must be enabled first) :

# aof ‐ use ‐ RDB ‐ preamble yesCopy the code

If mixed persistence is enabled, when AOF is rewritten, it no longer converts the memory data into RESP commands and writes them into AOF files. Instead, it takes RDB snapshots of the memory before the rewriting and writes both RDB snapshots and incremental AOF commands to modify memory data into a new AOF file. The new file is not called appendone.aof at the beginning, but will be renamed after rewriting the new aOF file to override the original aOF file and complete the replacement of the old and new aOF files.

Therefore, when Redis restarts, the contents of RDB can be loaded first, and then the incremental AOF log can completely replace the previous full AOF file replay, so the restart efficiency is greatly improved.

The hybrid persistent AOF file structure is as follows

Redis data backup policy: Periodically back up AOF or RDB files

  • This section describes how to write a crontab scheduling script to copy an RDB or AOF backup file to a directory every hour. Only the backup file generated in the last 48 hours is retained
  • Every day, one copy of the current day’s data is backed up to a directory, which can save the latest one month’s backup
  • Every time I copy a backup, I delete the old backup
  • Make a copy of the backup on your current machine to another machine every night to prevent machine damage

Redis master-slave architecture

The characteristics of

A master server can be configured with multiple slave nodes, and a slave node can also be configured with sub-slave nodes. When data changes on the master server, commands can be sent to synchronize data to the slave server. Generally, the slave server is read-only. If the primary server is faulty, manually restore the primary server. It provides hot backup of data, facilitates recovery of single points of failure, and loads part of the read load. It is the basis of high availability architecture.

Set up

Steps for setting up master/slave architecture of Redis and configuring slave nodes:

  1. Make a copy of the redis.conf file
  2. Change related configurations to the following values:

  • port  6380

  • Pidfile /var/run/redis_6380.pid # Write the pid process number to the pidfile configuration file

  • logfile “6380.log”

  • Dir /usr/local/redis‐5.0.3/6380 # specifies the directory for storing data

  • Set the value of bind to bind 0.0.0.0. The default value is 127.0.0.1. This parameter can be configured only on the local PC. In the current approach, both the code and the Redis client in Linux can be used

    1. Configure multiple IP addresses in bind 192.168.64.129 127.0.0.1

    2. Set bind to 0.0.0.0

  • Configure the primary/secondary replication

replicaof 192.168. 060.  6379# from the native6379Copy data to redis instance, redis5.0Slaveof replica‐read‐only yes #Copy the code

If we need the slave to verify the replication of the master, we can configure the requirepass option on the master to set the password to be used on the slave server, You can use the command config set masterauth or set masterauth in the configuration file.

test

Enable slave node Redis ‐server redis. Conf to connect slave node Redis ‐cli ‐p6380The test in6379Write data to the instance,6380Whether the instance can synchronize newly modified data in a timely mannerCopy the code

The principle of

www.cnblogs.com/kismetv/p/9…

The master-slave replication process of Redis is roughly divided into three stages: connection establishment, data synchronization, and command propagation

Establish a connection

Run replicaof command on the secondary server to establish a connection according to the IP and port number. The secondary server will send ping, and the master server will restore Pong, indicating that it is available. Otherwise, disconnect and initiate reconnection.

192.168.249.20:6379 > info replication… = 192.168.249.22 slave0: IP and port = 6379, state = online, offset = 700, lag = 0 = 192.168.249.21 slave1: IP and port = 6379, state = online, offset = 700, lag = 0…

Replication code data synchronization

In the command propagation phase, the master server propagates the same write command to the slave server for each write command executed by the master server. In this phase, the slave server also sends heartbeat detection with offset every second to check the network condition. Check whether the command executed is complete (replication backlog buffer is also used here), and if incremental replication is performed after the network is recovered, it is possible to perform full replication. Two conditions must be met for performing incremental replication. First, the RUNID sent from the secondary node must be the same as the ID of the existing primary node, and the commands after offset must exist in the replication backlog cache.

Full amount of copy

The secondary server connects to the primary server and sends Psync


, where runid is the rUNId of the primary node that was last replicated, and offset is the replication offset saved from the secondary node when the last replication expires. Upon receiving the request, the primary server performs bgSave to generate RGB files and continues to record all subsequent commands using a buffer. The primary server sends snapshots to all secondary servers. After the secondary server gets the snapshots, the old data is deleted and the snapshots are loaded. After sending the snapshot, the primary server sends the write command of the buffer to the secondary server.

Incremental replication (incremental synchronization, breakpoint continuation)

When the slave connects to the master, PSYNC


is executed. The master compares the offset with the master and sends the command after the offset.

The offset

The master and slave nodes each maintain an offset, which represents the number of bytes sent by the master to the slave server.

Copy the backlogged cache

Is a fixed-size queue that backs up the latest data sent from the master node to the slave node. It records the offset of each byte. By default, it is 1 MB. To increase the probability of partial replication in the event of a network outage, the size of the replication backlog buffer can be increased as needed (by configuring the repl-backlog-size).

Pipeline

Client can one-time send multiple requests without waiting for a server response, for after all command to send up the ring should be one-time reading service, so that we can greatly reduce the network transmission cost of multiple command execution pipeline execution of multiple commands network overhead in fact only equivalent to a command network overhead. Note that redis must cache all command processing results before finishing all command processing. The more commands you pack, the more memory the cache consumes. So it’s not that the more commands you pack, the better. Each command sent in the pipeline is executed immediately by the server. If the command fails, it will receive a message in the subsequent response. A pipeline does not express the semantics of “all commands succeed together”, where the previous command fails and the subsequent command does not affect execution.

Reduces network IO and increases memory consumption

Pipeline pl = jedis.pipelined();
23 for (int i = 0; i < 10; i++) {
24 pl.incr("pipelineKey");
25 pl.set("zhuge" + i, "zhuge");
26 }
27 List<Object> results = pl.syncAndReturnAll();
28 System.out.println(results);
Copy the code

The lua script

  1. Reduce network overhead
  2. Atomic operation
  3. Instead of a transaction
  4. Execution is single-process single-thread

The command line

EVAL script numkeys key [key ...]  arg [arg ...]Copy the code

java

String script = " local count = redis.call('get', KEYS[1]) " +
 " local a = tonumber(count) " +
 " local b = tonumber(ARGV[1]) " +
 " if a >= b then " +
 " redis.call('set', KEYS[1], a‐b) " +
 " return 1 " +
 " end " +
 " return 0 ";
 Object obj = jedis.eval(script, Arrays.asList("product_count_10016"),
Arrays.asList("10"));
42 System.out.println(obj);
Copy the code

Redis Sentinel high availability architecture

Sentinel sentinel is a special Redis service that does not provide read and write services. Sentinel sentinel is mainly used to monitor redis instance nodes. In sentinel architecture, the client will find out the master node of Redis from sentinel for the first time, and then directly access the master node of Redis, instead of accessing the master node of Redis through Sentinel agent every time. When the master node of Redis changes, the sentinel will immediately perceive it. In addition, the new redis master node is notified to the client (in this case, the Client of Redis generally implements the subscription function to subscribe to the sentinel node change message).

Redis Sentinel architecture setup steps:

1 1, copy the sentinel.conf file2Cp sentinel. Conf sentinel ‐26379.conf
3
4 2, modify related configurations to the following values:5 port  26379
6 daemonize yes
7 pidfile "/ var/run/redis ‐ sentinel ‐ 26379. Pid"
8 logfile "26379.log"
9 dir "/ usr/local/redis ‐ 5.0.3 / data"
10# Sentinel monitor <master‐redis‐name> <master‐redis‐ IP > <master‐redis‐port> <quorum>11# quorum is a number that indicates how many sentinels consider a master to be invalid (the value is typically: sentinel total /2  +
1), master is truly invalidated12 sentinel monitor mymaster 192.168. 060.  6379 2The name # myMaster is arbitrary and will be used for client access13
14 3, start the sentinel instance15SRC/redis ‐ sentinel sentinel ‐26379.conf
16
17 4, view the sentinel info18SRC/redis ‐ cli ‐ p26379
19 127.0. 01.: 26379 >info
20You can see that the Sentinel info has identified the redis master and slave21
22 5You can configure two sentinel ports by yourself26380 和 26381Notice that the numbers in the preceding configuration file must be changed23
Copy the code

After the sentinel cluster has been started, metadata information of sentinel cluster will be written into all sentinel configuration files (added at the bottom of the file).

Sentinel known‐ Replica MyMaster 192.168.0.60 6380 # represents the slave node information of the redis master node

Sentinel known‐ Replica MyMaster 192.168.0.60 6381 # represents the slave node information of the redis master node

Sentinel known ‐ sentinel mymaster 192.168.0.60 26380 # 52 d0a5d70c1f90475b4fc03b6ce7c3c35760f represent a perceived other sentinel sentinel node Known ‐ sentinel mymaster 192.168.0.60 26381 e9f530d3882f8043f76ebb8e1686438ba bd5ca6 # for perceived other sentinel node

If the primary redis node fails, the Sentinel cluster will re-elect a new primary Redis node, modify the cluster metadata information of all Sentinel node configuration files, and change the port 6379 corresponding to MyMaster previously configured in the Sentinel file to 6380

Sentinel Monitor MyMaster 192.168.0.60 6380

When the redis instance of 6379 is started again, the Sentinel cluster can join the Redis node on port 6379 as a slave node based on the cluster metadata information

Sentry’s Spring Boot integration Redis connection code: automatically subscribing to the Sentry service when configured

server:
2 port:  
3
4 spring:
5 redis:
6 database:  
7 timeout:  
8Sentinel: # Sentinel mode9Master: myMaster # cluster name of the primary server10 nodes: 192.168. 060.: 26379 ,192.168. 060.: 26380 ,192.168. 060.: 26381
Copy the code

List of methods in the RedisTemplate corresponding to the Redis client command:

High availability cluster mode

Inadequate Sentry mode:

  1. Access transients exist
  2. With a single master node, high concurrency cannot be achieved
  3. Do not set the memory size of a single primary node to too large. Otherwise, the persistent file size will affect data recovery or master/slave synchronization

The efficiency of

Redis cluster is a distributed server farm consisting of multiple master and slave node groups with replication, high availability and sharding features. The Redis cluster can perform node removal and failover without sentinel sentinel ∙. Each node needs to be set to cluster mode, which has no central nodes and can scale horizontally and linearly to tens of thousands of nodes according to official documentation (no more than 1000 nodes are officially recommended). Redis clusters provide better performance and high availability than previous versions of Sentinel mode, and cluster configuration is very simple.

Set up

The redis cluster requires at least three master nodes. Here we set up three master nodes and a slave node for each master node, with a total of six Redis nodes. Here we deploy six Redis instances on three machines, with one master and one slave for each machine.

Step 1: Create the folder Redis ‐ Cluster under /usr/local on the first machine and then create them separately2The folders are as follows (1‐p /usr/local/redis‐cluster (2) the mkdir8001 8004Step 1: Copy the previous redis.conf configuration file to8001Below, modify the following content: (1) daemonize yes (2) the port8001(Set the port number of each machine separately) (3Pidfile /var/run/redis_8001.pid4Dir/usr/local/redis ‐ cluster /8001/ (Specify the location of the data file, you must specify a different directory location, otherwise the data will be lost)5) cluster‐ Enabled YES (6) cluster ‐ config ‐ file nodes ‐8001.Conf (cluster node information file, 800x better match port) (7) cluster ‐ node ‐ timeout10000
 (8)# bind 127.0. 01.Bind is the IP address of the network adapter on the machine. If multiple network adapters can be configured with multiple IP addresses, it indicates which NETWORK adapter IP addresses can be used by clients to access the Intranet.9)protected‐ Mode NO (Off protection mode) (10Appendonly yes Add the following configuration to set the password: (11)requirepass zhuge (Set redis access password)
(12)masterauth zhuge (Set the password for accessing cluster nodes as above.)20 Step 3: Copy the modified configuration file to 8004 and change the port numbers in items 2, 3, 4, and 6. You can replace them with batch numbers: :%s/ source string/destination string /g Step 4: The other two machines also need to do the above steps, the second machine uses 8002 and 8005, and the third machine uses 8003 and 8006. Step 5: Start each of the six Redis instances, Then check whether (1) start/usr/local/redis ‐ 5.0.3 / SRC/redis ‐ server/usr/local/redis ‐ cluster / 800 * / redis. Conf. (2) ps ‐ ef | grep redis Step 6: Use redis‐ CLI to create the entire Redis cluster(Prior to Redis5 clustering was implemented using the Ruby script Redis ‐trib.rb)Create a secondary server node for each primary server node you create. Create a secondary server node for each primary server node you create. If you do not disable the firewall, enable the Redis service port and cluster node gossip communication port 16379(The default is to add 1W to the redis port number)# systemctl stop firewalld # systemctl disable firewalld # systemctl disable firewalld Don't copy the following command to create a cluster. /usr/local/redis‐5.0.3/src/redis‐cli ‐a zhuge ‐replicas create ‐‐cluster 1 1 92.168.0.63:8001 192.168.0.62:8002 192.168.0.63:8003 192.168.0.61:8004 192.168.0.62:8005 192. 168.0.63:8006 Step 7 verify the cluster: 39 (1) Connect to either client:./redis‐ CLI ‐ C ‐ H ‐p ‐a password for accessing the server. ‐c denotes the cluster mode, specifying the IP address and port number5.03./ SRC /redis‐cli ‐a zhuge ‐c ‐h192.168. 061.‐ p800* (2Verify: Cluster info (view cluster information), Cluster Nodes (view node list) (3) perform data manipulation validation (4To shut down clusters one by one, run the /usr/local/redis‐ command5.03./ SRC /redis‐cli ‐a zhuge ‐c ‐h192.168. 060.‐ p800* shutdown
Copy the code

Spring Boot for the cluster integrates Redis connection code

<? xml version="1.0" encoding="UTF-8"? > <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0. 0</modelVersion> <parent> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-parent</artifactId>  <version>2.19..RELEASE</version> <relativePath /> <! -- lookup parent from repository --> </parent> <groupId>com.fiberhomebj</groupId> <artifactId>xingyv-webback-starter</artifactId> <version>0.01.-snapshot </version> <name>xingyv-webback-starter</name> <description> Star domain web background </description> <properties> <java.version>1.8</java.version> </properties> <dependencies> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-web</artifactId> </dependency> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-data-redis</artifactId> </dependency> <dependency> <groupId>org.apache.commons</groupId>  <artifactId>commons-pool2</artifactId> </dependency> </dependencies> <build> <plugins> <plugin> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-maven-plugin</artifactId> </plugin> </plugins> </build> </project>Copy the code
spring:
  redis:
    database: 0
    timeout: 3000
    password: zhuge
    cluster:
      nodes: 192.1688.8.:8001.192.1688.9.:8002.192.1688.10.:8003.192.1688.8.:8004.192.1688.9.:8005.192.1688.10.:8006
    lettuce:
      pool:
        max-active: 50
        min-idle: 10
        max-idle: 100
        max-wait: 1000
Copy the code
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.data.redis.core.StringRedisTemplate;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;

@RestController
public class IndexController {

  private static final Logger logger = LoggerFactory.getLogger(IndexController.class);

  @Autowired
  private StringRedisTemplate stringRedisTemplate;

  @RequestMapping("/test_cluster")
  public void testCluster(a) throws InterruptedException {
      stringRedisTemplate.opsForValue().set("mxx"."666");
      System.out.println(stringRedisTemplate.opsForValue().get("mxx")); }}Copy the code

Analysis of Cluster Principle

The Redis Cluster divides all data into 16384 slots, some of which each node is responsible for. Slot information is stored in each node. When the client of the Redis Cluster connects to the Cluster, it also gets a slot configuration information of the Cluster and caches it locally to the client. In this way, when the client is looking for a key, it can directly locate the target node. In addition, slot information may be inconsistent between the client and server. Therefore, a correction mechanism is required to verify and adjust slot information.

Slot location algorithm

By default, the Cluster hashes the key using the CRC16 algorithm to obtain an integer value, and then modulates 16384 to obtain the specific slot. HASH_SLOT = CRC16(key) mod 16384

Jump relocation

When the client sends an instruction to a wrong node, the node will find that the slot where the key of the instruction is located is not managed by itself, and then it will send a special jump instruction to the client to carry the node address of the target operation, telling the client to connect to this node to obtain data. After receiving the command, the client not only redirects to the correct node, but also synchronously updates and corrects the local slot mapping table cache. All subsequent keys use the new slot mapping table.

Redis communication mechanism between cluster nodes

Redis cluster nodes use gossip protocol to communicate with each other. There are two ways to maintain cluster metadata (cluster node information, primary and secondary roles, number of nodes, and data shared by each node) : Centralized and Gossip

Centralized:

The advantage lies in the update and reading of metadata, with very good timeliness. Once the metadata changes, it will be immediately updated to the centralized storage, and other nodes can immediately perceive when they read it. The disadvantage is that all the updating pressure of metadata is concentrated in one place, which may lead to the storage pressure of metadata data. Many middleware use ZooKeeper to centrally store metadata.

Gossip:

The Gossip protocol contains multiple messages, including ping, pong, meet, fail, and so on.

Meet: a node sends a meet to a new node to join the cluster, and then the new node will communicate with other nodes.

Ping: Each node frequently pings each other, including its own status and its own cluster metadata, to communicate with each other

Ping exchange metadata (like self-aware cluster node addition and removal, hash slot information, etc.);

Pong: the return of ping and meet messages, containing its own status and other information, can also be used for information broadcast and update;

Fail: After judging that another node fails, a node sends a fail message to inform other nodes that the specified node is down.

The gossip protocol has the advantage that metadata updates are scattered rather than centralized in one place. The update requests will be sent to all nodes intermittently, which has a certain delay and reduces the pressure. The disadvantage is that the metadata update is delayed and some operations of the cluster may be delayed.

Port 10000 for the gossip communication

Each node has a dedicated port for communication between nodes. This port is the number of its own service +10000. For example, port 17001 is used for communication between nodes. Each node will send ping messages to other nodes at regular intervals, and the other nodes will return pong messages after receiving the ping messages.

The network jitter

Real world computer room networks are not always smooth, and they often have all kinds of minor problems. Network jitter, for example, is a very common phenomenon, where parts of a connection suddenly become inaccessible and then quickly recover. To solve this problem, the Redis Cluster provides clusterNodeTimeout. Clusternodetimeout indicates that a node is faulty and a primary/secondary switchover is required only when the node is disconnected within the timeout period. Without this option, network jitter can cause a primary/secondary failover (reduplication of data).

Redis cluster election principle analysis

  1. When the slave finds that its master becomes FAIL, it increses its currentEpoch (current era, period) and broadcasts FAILOVER_AUTH_REQUEST to all nodes.
  2. Only the master responds, determines the validity of the requester, and sends FAILOVER_AUTH_ACK, sending only one ACK for each epoch based on the first-come-first-cast principle
  3. The slave will become a new master after receiving more than half of the master’s ACKS.
  4. Slave Broadcasts Pong messages to inform other cluster nodes

The slave node does not attempt to initiate the election immediately after the master node enters the FAIL state, but has a certain delay to ensure that the FAIL state is spread in the cluster. If the slave tries to vote immediately, other masters may not be aware of the FAIL state and may refuse to vote.

• Delay calculation formula:

DELAY = 500ms + random(0 ~ 500ms) + SLAVE_RANK * 1000ms

•SLAVE_RANK Rank of the total amount of data copied from the slave master. A smaller Rank indicates that the replicated data is newer. In this scenario, the slave with the latest data would (in theory) call the election first.

Split cluster data loss problem

What is Redis cluster split brain?

Cluster split brain of Redis refers to the fact that the Redis master node is in a different network partition from the Redis slave node and sentinel cluster due to network problems. At this time, because the Sentinel cluster cannot perceive the existence of the master, Therefore, promote the slave node to master node. There are two different master nodes, like a brain split into two. In the case of cluster split brain, if the client continues to write data based on the original master node, the new master node cannot synchronize the data. When the network problem is resolved, the Sentinel cluster reduces the original master node to slave node, and then synchronizes data from the new master node. There will be a lot of data loss.

The circumvention method can add parameters to the Redis configuration (this method is not 100% to avoid data loss, see cluster leader election mechanism) :

Min ‐ replicas ‐ to ‐ write1 // The minimum number of slaves that need to be synchronized to successfully write data. This number can mimic more than half of the mechanism configurations, for exampleA total of three nodes in the cluster can be configured1And leader is2, more than half, equivalent to at least half step synchronization is considered successful writeCopy the code

Note: This configuration may affect the availability of the cluster to some extent. For example, if there is less than one slave, the cluster cannot provide services even if the leader is normal.

Services can be provided only when the cluster is complete

Cluster-require-full-coverage the default value is yes, indicating that cluster integrity is required. All nodes in a small cluster are hung and cannot be accessed by clients

Why does a Redis cluster need at least three master nodes and an odd number of nodes is recommended?

To elect a new master, more than half of the master nodes in the cluster must agree. If there are only two master nodes, if one of them fails, the condition for electing a new master cannot be met. An odd number of master nodes can save one node on the basis of meeting the election condition. For example, compared with a cluster with three master nodes and four master nodes, if everyone fails a master node, they can elect a new master node. If both master nodes fail, it is impossible to elect new master nodes. Therefore, the odd number of master nodes is more to save machine resources.

Redis cluster support for batch operation commands

For the original batch operation commands of multiple keys, such as MSET and MGET, redis cluster only supports the case that all keys are placed in the same slot. If the slots obtained by keys are different, the execution will fail. Because of the guarantee of atomicity, some may succeed and some may fail when they are placed on different machines. If multiple keys must be used in the Redis cluster using the mset command, add {XX} to the front of the key to hash only the values in braces. This ensures that different keys will be placed in the same slot, as shown in the following example: 1 mset {user1}:1:name zhuge {user1}:1:age 18 If the hash slot values calculated by name and age are different, redis will only use user1 in curly braces to compute hash slot values. Therefore, the calculated slot values must be the same, and they all end up in the same slot.

Horizontal scaling of Redis High Availability Clusters (adding and deleting small clusters)

Horizontal scaling of Redis High Availability Clusters (adding and deleting small clusters)