Redis refers to a large set data type, such as (set/hash/list/sorted set). A key contains many elements. Because Redis is a single thread, the main thread is busy deleting the big key when deleting the big key(ten million level set set) or cleaning the expired big key data, which will cause redis to block, crash, and application exceptions.

A case in point

Online Redis as a real-time deduplication tool, there are 60 million user GUids, such a set, if the direct use of del deletion, will cause redis severely blocked.

10.1.254.18:6380> info memory # memory used_memory:15175740016 used_memory_human: 14.13g used_memory_rss:22302339072 Used_memory_peak: 22351749192 used_memory_peak_human: 20.82 G used_memory_lua: 36864 mem_fragmentation_ratio: 1.47 Mem_allocator :jemalloc-3.6.0 10.1.254.18:6380> scard Helper_2019-03-12 (INTEGER) 64530980 10.1.254.18:6380> del Helper_2019-03-12 (INTEGER) 1 (81.23s) 10.1.254.18:6380> Info memory # memory usED_memory :8466985704 Used_memory_human: 7.89 G used_memory_rss: 10669453312 used_memory_peak: 22351749192 used_memory_peak_human: 20.82 G Used_memory_lua: 36864 mem_fragmentation_ratio: 1.26 mem_allocator: jemalloc - 3.6.0Copy the code

Helper_2019-03-12 (select * from helper_2019-03-12 (select * from helper_2019-03-12)) Fortunately, we used connection pooling and there were no problems.

Java batch deletion

In this case, the ssCAN command should be used to batch delete set elements. Here is an example of Java code batch deleting a set in Redis:

private static void test2(a){
    // Connect to the Redis server
    Jedis jedis = new Jedis("0.0.0.0".6379);
    jedis.auth("123456");

    // Batch delete
    try {
        ScanParams scanParams = new ScanParams();
        // Delete 500 entries at a time
        scanParams.count(500);
        String cursor = "";
        while(! cursor.equals("0")){
            ScanResult<String> scanResult=jedis.sscan("testset", cursor, scanParams);
            // Return 0 to indicate completion of traversal
            cursor = scanResult.getStringCursor();
            List<String> result = scanResult.getResult();
            long t1 = System.currentTimeMillis();
            for(int m = 0; m < result.size(); m++){ String element = result.get(m); jedis.srem("testset", element);
            }
            long t2 = System.currentTimeMillis();
            System.out.println("Delete"+result.size()+"Piece of data, time:"+(t2-t1)+"Milliseconds, cursor."+cursor); }}catch (JedisException e){
        e.printStackTrace();
    }finally {
        if(jedis ! =null){ jedis.close(); }}}Copy the code

For other sets, there are corresponding methods.

  • Hash key: Run the hscan command to obtain 500 fields at a time, and then run the hdel command.
  • Set key: scan 500 elements in the set each time using SSCAN command, and then delete one element each time using SREM command;
  • List key: deletes large list keys without using the scan command. Remove a few elements at a time with the ltrim command.
  • Sorted set key: Deletes a large ordered set key. Similar to List, sortedSet removes the top 100 elements at a time using the zremrangebyrank command.

Delete Python scripts in batches

For redis monitoring and cleaning, usually using some Python scripts to do, simple, lightweight. With Java, even a small task must be packaged and released, if there is no set of perfect development, release process, or more trouble. At this point, a lot of people tend to write Python scripts, and most people who know Python know Java.

Here, again, deleting a set is an example:

# -*- coding:utf-8 -*-

import redis

def test() :
    When StrictRedis is created, the connection is managed by the pool, so we don't have to worry about whether the connection needs to be released actively
    re = redis.StrictRedis(host = "0.0.0.0",port = 6379,password = "123")
    key = "test"
    for i in range(100000):
        re.sadd(key, i)

    cursor = '0'
    cou = 200
    whilecursor ! =0:
        cursor,data = re.sscan(name = key, cursor = cursor, count = cou)
        for item in data:
            re.srem(key, item)
            print cursor

if __name__ == '__main__':
                    test()
Copy the code

Lazyfree mechanism for background deletion

In order to solve the problem of redis blocking when using the del command to delete large keys or using the flushdb or flushall database, the lazyfree mechanism is introduced in Redis 4.0, which can put the deletion in the background and let the background subthread (BIO) execute the deletion. Avoid main thread blocking.

The use of lazy free can be divided into two types: the first type is active deletion corresponding to the DEL command, and the second type is expired key deletion and maxMemory key expulsion.

Take the initiative to delete

The UNLINK command is a lazy free implementation of the delete key function like DEL. The only difference is that when UNLINK removes a set key, if the set key has more than 64 elements (more on that later), it assigns the actual memory free operation to a separate BIO.

127.0.0.1:7000> UNLINK myList (integer) 1 FLUSHALL/FLUSHDB ASYNC 127.0.0.1:7000> FLUSHALL ASYNC // Cleans instance data asynchronouslyCopy the code

Passive delete

Lazy free applies to passive deletion. Currently, there are four scenarios, and each scenario corresponds to a configuration parameter. The default is off.

lazyfree-lazy-eviction no
lazyfree-lazy-expire no
lazyfree-lazy-server-del no
slave-lazy-flush no
Copy the code

lazyfree-lazy-eviction

When redis memory usage reaches maxMeory and the elimination strategy is set; Whether to use lazy free mechanism in passive elimination of keys;

Because lazy free is enabled in this scenario, memory may not be freed in time using the eliminate key, resulting in redis exceeding the maxMemory limit. When using this scenario, combine service tests.

lazyfree-lazy-expire

For keys with TTL, whether redis uses the lazy free mechanism when it deletes them after they expire.

You are advised to enable this scenario because TTL is an adaptive adjustment speed.

lazyfree-lazy-server-del

An operation for instructions that have an implicit DEL key when handling an existing key. For example, with the rename command,redis removes target keys when they already exist. If these target keys are big keys, this introduces performance problems that block deletion. This parameter is used to resolve such problems. You are advised to enable it.

slave-lazy-flush

Full data synchronization is performed for the slave. Before loading the MASTER RDB file, the slave will run flushall to clean up its own data scenarios.

Parameter Settings Determine whether to use abnormal flush. If the memory does not change much, you are advised to enable it. This reduces the full synchronization time, thereby reducing the memory usage increase caused by the output buffer burst in the primary library.

Expire and EVICT optimization Redis will enter the activeExpireCycle loop to delete expired keys when idle. Each loop will calculate an execution time first. In the loop, it does not traverse the entire database, but randomly selects some keys to check whether they expire. So sometimes time is not run out (it’s faster to clean up expired keys when doing asynchronous deletes), and the remaining time can be handed over to freeMemoryIfNeeded for execution.

Reference links:

  • Mysql.taobao.org/monthly/201…
  • Blog.csdn.net/liu1390910/…
  • Blog.csdn.net/wsliangjian…

Follow the public account: Java big data and data warehouse, learn big data technology.