Still deleting data with keys fuzzy match? This is a bomb ready to explode!

There is no instruction in Redis to delete a specific prefix key in batch, but we often need to delete according to the prefix, so how to do? You might get the following answer after a search

redis-cli --raw keys "ops-coffee-*" | xargs redis-cli del
Copy the code

Directly match all keys in Linux through redis keys command, and then call system command xargs to delete. It seems very perfect, but actually it is a huge risk

Due to the single-threaded service mode of Redis, command keys will block normal business requests. If you match too many keys at a time or encounter a large key in DEL, it will directly cause service unavailability and even cause the risk of Redis downtime

So we should avoid this approach in production. What’s an elegant way to do it? SCAN!

SCAN introduction and use

Redis supports the scan command from version 2.8. The basic usage of the scan command is as follows:

SCAN cursor [MATCH pattern] [COUNT count]
Copy the code

Cursor: Cursor, SCAN command is a cursor based iterators, SCAN every time command is invoked, a new cursor will be returned to the user, the user need to use the new cursor in next iteration as SCAN cursor command parameter, to the continuation of the previous iteration, until the server returns a value of 0 to the users of the cursor, A complete traversal is completed

MATCH: a matching rule, for example, traversing all keys starting with ops-coffee- can be written as ops-coffee-*, and keys containing -coffee- can be written as *-coffee-*

The COUNT: The COUNT option allows the user to tell the iteration command how many elements should be returned from the dataset in each iteration. COUNT is just a hint to the incremental iteration command, and does not represent the actual number returned. For example, if you set COUNT to 2, you may return 3 elements. But the returned element data is positively correlated with the COUNT setting, which defaults to 10

The following is an example of an iteration of the SCAN command:

127.0.0.1:6379> scan 0 MATCH ops-coffee-* 
1) "38"
2)  1) "ops-coffee-25"
    2) "ops-coffee-19"
    3) "ops-coffee-29"
    4) "ops-coffee-10"
    5) "ops-coffee-23"
    6) "ops-coffee-5"
    7) "ops-coffee-14"
    8) "ops-coffee-16"
    9) "ops-coffee-11"
   10) "ops-coffee-15"
   11) "ops-coffee-7"
   12) "ops-coffee-1"
127.0.0.1:6379> scan 38 MATCH ops-coffee-* COUNT 1000
1) "0"
2)  1) "ops-coffee-13"
    2) "ops-coffee-9"
    3) "ops-coffee-21"
    4) "ops-coffee-6"
    5) "ops-coffee-30"
    6) "ops-coffee-20"
    7) "ops-coffee-2"
    8) "ops-coffee-12"
    9) "ops-coffee-28"
   10) "ops-coffee-3"
   11) "ops-coffee-26"
   12) "ops-coffee-4"
   13) "ops-coffee-31"
   14) "ops-coffee-8"
   15) "ops-coffee-22"
   16) "ops-coffee-27"
   17) "ops-coffee-18"
   18) "ops-coffee-24"
   19) "ops-coffee-17"
Copy the code

The SCAN command returns an array of two elements, the first array element is the new cursor for the next iteration, and the second array element is an array containing all the iterated elements

The above example means scanning all keys prefixed with ops-coffee-

The first iteration uses 0 as a cursor to start a new iteration, matches the key prefixed with OPs-coffee -, and returns the cursor value 38 along with the traversed data

The second iteration uses the cursor returned from the first iteration, which returns the value 38 for the first element, and forces the command to scan more elements for this iteration by setting the COUNT option to 1000

On the second call to SCAN, the command returns a cursor 0, indicating that the iteration has ended and the entire data set has been fully traversed

The time complexity of KEYS command is O(n), while SCAN command will decompose the traversal operation into m operations with time complexity of O(1), so as to solve the server blocking caused by traversal of a large amount of data using KEYS command. The following commands can achieve the purpose of elegant deletion:

redis-cli --scan --pattern "ops-coffee-*" | xargs -L 2000 redis-cli del
Copy the code

The xargs -L command represents the number of lines read by XARgs at a time, that is, the number of keys deleted at a time. If too many lines are read at a time, XARgs will report an error

Elegant deletion of several other data structures

There are several other SSCAN, HSCAN, and ZSCAN commands that are similar to Redis data types:

> sscan ops-coffee 0 MATCH v1*
1) "Seven"
2) 1) "v15"
   2) "v13"
   3) "v12"
   4) "v10"
   5) "v14"
   6) "v1"
Copy the code

Different from the SCAN command, these commands need to add a key parameter, such as the above OPs-coffee

For a large set key, ssCAN provides an elegant batch deletion using the following code:

import redis

def del_big_set_key(key_name):
    r = redis.StrictRedis(host='localhost', port=6379)
    
    # count indicates the number of elements removed at a time, 300 at a time
    for key in r.sscan_iter(name=key_name, count=300):
        r.srem(key_name, key)

del_big_set_key('ops-coffee')
Copy the code

For a large hash key, hsCAN gracefully delete it using the following code:

import redis

def del_big_hash_key(key_name):
    r = redis.StrictRedis(host='localhost', port=6379)
    
    # hscan_iter (); # hscan_iter ()
    for key in r.hscan_iter(name=key_name, count=300):
        r.hdel(key_name, key[0])

del_big_hash_key('ops-coffee')
Copy the code

For large ordered sets of the delete is relatively simple, directly according to the zremrangeByrank rank range delete

import redis

def del_big_sort_key(key_name):
    r = redis.StrictRedis(host='localhost', port=6379)

    while r.zcard(key_name) > 0:
        # check if there are elements in the set, if there are, delete the elements ranked 0-99
        r.zremrangebyrank(key_name, 0, 99)

del_big_sort_key('ops-coffee')
Copy the code

If you want to delete a large list, use the llEN method to determine the number of elements, and then ltrim to remove the range of elements

At this point, the elegant deletion of Redis’s five data structures is all realized, and the production environment is preferred to use ~

Related articles recommended reading:

  • Remember a weird troubleshooting experience
  • Nginx has several security-related configurations