[REDIS SCAN] SCAN clears invalid keys, resulting in slow log surge

​

1 Problem Scene

private List<String> scanWithLimit(String pattern, int limit, int type) { Preconditions.checkArgument(StringUtils.isNotBlank(pattern)); List<String> list = Lists.newArrayList(); AtomicInteger count = new AtomicInteger(); AtomicInteger expired = new AtomicInteger(); this.redisTemplate.execute((RedisConnection connection) -> { Cursor<byte[]> cursor = null; try { cursor = connection.scan(ScanOptions.scanOptions().count(30000).match(pattern).build()); while (cursor.hasNext()){ String key = new String(cursor.next(), StandardCharsets.UTF_8); //list.add(key); long expire = redisCacheUtil.getExpireMillis(RedisKeys.BLANK, key); if(expire == -1 || type == 2){ expired.getAndIncrement(); redisCacheUtil.delBatch("", key); }else if(expire < -1){ expired.getAndIncrement(); }else{ } int total = count.getAndIncrement(); if(total > limit){ break; } } return list; } catch (Exception e) { return null; }finally { if(null ! = cursor){ try { cursor.close(); } catch (IOException e) { return null; }}}}); return list; }Copy the code

2 Slow log on site

​

3 The implementation used by the above code

org.springframework.data.redis.connection.lettuce.LettuceKeyCommands#scan(org.springframework.data.redis.core.ScanOption s)

Incrementally iterate the keys space over the whole Cluster. // Notice the Incrementally iterate the keys space over the whole Cluster. ->31 So the node scans as many keys as possible at a time, but the execution time of 32 nodes is time-consuming, which is beyond expectation, temporarily MARK.

​