Some thoughts on Redis hot Key

Yesterday I was talking to a colleague who has changed jobs. I asked him about some interview problems he had recently encountered. I also want to accumulate knowledge of this aspect. In one of them, he said that when he was interviewing a company, the interviewer asked him about hot Key’s solution. So in view of this conversation and the Internet to check some data after the thinking of a summary. Easy to follow up their own reference.

What is a hot Key

In fact, for hot Key, a lot of online search, here I quote a paragraph on the Internet.

From the data based on data on the user’s consumption is much larger than the production point of view, we use of the software such as zhihu, most people usually just browsing, don’t go to ask questions and published articles, published their own articles or occasionally, this is a typical read more write less, of course this kind of situation is not so easy to lead to the generation of hot spots.

Some unexpected events in daily work and life, such as the price reduction promotion of some popular commodities during “Double 11”, when one of these commodities is clicked and purchased tens of thousands of times, a large demand will be formed. In this case, a single Key will be generated, which will cause a hot spot. Similarly, hot news and comments that are widely published and browsed will also generate hot news. In addition, when the Server reads and accesses data, data is often sliced and sharded. During this process, corresponding keys are accessed on a host Server. When the access exceeds the host Server limit, hot keys may occur.

How to solve it?

There are no more than two solutions for hot keys found on the Internet

  • Server cache: Cache hotspot data to server memory
  • Backup hot Key: the hot Key+ random number is randomly allocated to other Redis nodes. In this way, hot key accesses will not all hit the same machine.

In fact, the premise of these two solutions is to know what the hot Key is, so how to find the hot Key?

Hot spot detection

  1. Make predictions based on experience: for example, if you know the start of an activity in advance, use this Key as a hot Key
  2. Client collection: Statistics the data before operating Redis
  3. Packet capture for evaluation: Redis uses TCP protocol to communicate with the client. The communication protocol is RESP, so it can intercept and parse packets
  4. At the proxy layer, each REDis request is collected and reported
  5. Redis comes with command query: Redis4.0.4 provides itRedis - cli - hotkeysYou can find the hot Key

If you need to use the Redis command to query data, you must set the memory evicting policy to allkeys-lfu or volatile-lfu. Otherwise, an error is returned. Enter Redis and run the config set maxmemory-policy allkeys-lfu command.

Server side cache

Assuming that we have counted some hot keys and cached them to the server, there is still a problem. That is, how to ensure data consistency between Redis and server hot Key. The solution I think of here is to use the message notification mechanism of Redis to establish a monitor for the hot Key client. When the hot Key is updated, the client will be updated accordingly.

The main code is as follows. The listener class is responsible for receiving the event of Redis, and then screening the hot Key for the corresponding action

public class KeyExpiredEventMessageListener implements MessageListener { @Autowired private RedisTemplate redisTemplate;  @Override public void onMessage(Message message, byte[] pattern) { String key = new String(message.getChannel()); key = key.substring(key.indexOf(":") + 1); String action = new String(message.getBody());if (HotKey.containKey(key)){
            String value = redisTemplate.opsForValue().get(key)+"";
            switch (action){
                case "set":
                    log.info("Hotspot Key:{} modification",key);
                    HotKeyAction.UPDATE.action(key,value);
                    break;
                case "expired":
                    log.info("Hot Key:{} expire delete",key);
                    HotKeyAction.REMOVE.action(key,null);
                    break;
                case "del":
                    log.info("Hot Key:{} delete",key);
                    HotKeyAction.REMOVE.action(key,null);
                    break; }}}}Copy the code

Create a data structure ConcurrentHashMap to store hot keys, and set the corresponding operation method. Here set false data, set two hot keys directly in the static code block

public class HotKey {

    private static Map<String,String> hotKeyMap = new ConcurrentHashMap<>();
    private static List<String> hotKeyList = new CopyOnWriteArrayList<>();

    static {
        setHotKey("hu1"."1");
        setHotKey("hu2"."2");
    }

    public static void setHotKey(String key,String value){
        hotKeyMap.put(key,value);
        hotKeyList.add(key);
    }

    public static void updateHotKey(String key,String value){
        hotKeyMap.put(key,value);
    }

    public static String getHotValue(String key){
        return hotKeyMap.get(key);
    }

    public static void removeHotKey(String key){
        hotKeyMap.remove(key);
    }

    public static boolean containKey(String key){
        returnhotKeyList.contains(key); }}Copy the code

In fact, using Redis event notification mechanism is not good, because as long as the event notification is enabled, then every Key change will be sent a message, which will increase the burden of Redis server for no reason. Of course, I’m just going to briefly demonstrate that there are many ways to do this notification scheme.

Backing up hotspot Keys

The Redis cluster contains 16384 Hash slots, and the cluster uses CRC16(key) % 16384 to calculate which slot the key belongs to. So the same Key should be calculated with the same value, how to distribute the Key to other machines? As long as the random number is followed, it can ensure that the same Key is distributed on different machines, and access through Key+ random number.

The pseudocode is as follows

Const M = N * 2 // generate random number random = GenRandom(0, BakHotKey = hotKey + "_" + random data = redis.get (bakHotKey)ifSET(bakHotKey, expireTime + GenRandom(0,5))}Copy the code

Code address Github

Refer to the article

  • Mysql.taobao.org/monthly/201…
  • www.cnblogs.com/rjzheng/p/1…