This is the 25th day of my participation in the “Redis Challenge”.

preface

  • [Redis prequel] Continue to update! Various interspersed updates! Gnome male -“

Redis compared to data except for their structural subversion! And they’re stored in different places. Whereas traditional databases store data on hard disk and require IO for every data operation, Redis stores data in memory. So let me explain a little bit what I mean by IO. I/O is the input stream output stream between the data in the disk and memory interaction! Redis is left with IO operations directly in memory. That’s one of the reasons Redis is so fast

  • Memory is valuable relative to hard disk. Our usual computer is also hard disk memory is hundreds of times. Since memory is so valuableredisAnd store the data in memory soredisYou can’t store it with impunity. This would require theredisAnd the developers to optimize accordingly
  • First of all,redisIn the configuration file (redis.conf) bymaxmemoryParameter specifies theredisSet the total memory domination size!
  • The second is to ask us developers to thinkredisSet the key expiration time according to your own requirements. So unnecessary data can beredisThe expulsions strategy is clear. Thus saving memory for others to use

This article overridesredisOn the basis of this, the author here has been installed by defaultredis . And actually usedredis

Memory allocation

  • maxmemorySpecify the size. inredis.confYou can specify it in the configuration file. His unit is byte.

  • The comments in the figure above are for you to explain, the actual # configuration needs to be newline ha!! Helpful hints
  • And we can connectredisthroughconfigCommand to set

  • Both methods can be set, the former is global Settings and still valid after restart! The latter is a temporary setting that will be reloaded after the restartredis.confIs configured in.

Key expired

  • Up here we started fromredisSet memory limits from the perspective of itself, so you don’t have to worry about them eating up system memory! Now it’s up to us developers to constrain ourselves from the design perspective.

Setting the Expiration Time

expire key time
Copy the code

  • Set the default unit of expiration time in seconds.

  • You can run the TTL command to view the remaining expiration time

  • After many executionsttlYou can see that the remaining time is decreasing! When reduced to 0, it is given the eviction strategy eviction! Notice that this is an eviction strategy and eviction does not mean that it is removed immediately

Update expiration Time

  • del keyIf you delete the key directly, the expiration of the key will not exist! The author also counts this as the update expiration time
  • set getsetIf the key and value commands are reset, the expiration time is overwritten as -1

  • set 、 getsetincludingdelStrictly, overwrite expiration time. The one that really updates the expiration time is expire. Expire is the latest one!
  • Above actually modified key ability should send original expired time invalid! Because this key is not that key. butappend,incrIf you want to change the value, you will not change the expiration Settings

Elimination strategy

  • According to the above configuration we can put ourredisThe maximum memory is set to1MB, set the size of whatever it is better to be small. And then we go throughJavaThe little program keeps goingredisFill in. Eventually an error will be reported when there is insufficient memory

  • The error is reported because memory is full and the new key isredisRejected! It’s not just new additions that are rejected, even when we want to changeredisThe value of key in is also unavailable
public static void main(String[] args) {
    Jedis jedis = new Jedis("39.102.60.114".6379);
    jedis.auth("Qq025025");
    Integer index = 1;
    while (true) { String uuid = UUID.randomUUID().toString(); jedis.set(index.toString(), uuid); System.out.println(index++); }}Copy the code

  • OOM Command not allowed when used Memory > MAXMemory. The number of printed and Redis keys in the code is the same; Indicating that our default elimination strategy is outright rejection

  • When redis memory is full, any write will be rejected!

  • When there’s not enough memory, you just reject it? It is also mentioned above that we need to set the key expiration to free up memory for other keys that need to use! How do expired keys get cleared? Which brings us to our elimination strategy

Volatile -lru [rarely used recently]

When a memory alarm is generated, Redis will remove keys that are rarely used recently and whose expiration time has been set, even if the key has not expired yet. OOM Command not allowed when used Memory > maxMemory If no matching key is available, the noeviction will throw the same error as the default noeviction policy

  • First of all, we’re looking atredis.confWe rarely use policies these days.maxmemory-policy volatile-lru. And then restart ourredisService. After the restartflushallClean up all data. We’re going through theJavaProgram to regenerate the next data!

  • In the Java program, we set the expiration time for the first 100 keys
public static void main(String[] args) {
    Jedis jedis = new Jedis("39.102.60.114".6379);
    jedis.auth("Qq025025");
    Integer index = 1;
    while (true) {
        String uuid = UUID.randomUUID().toString();
        if (index < 100) {
            jedis.setex(index.toString(),360, uuid);
        } else{ jedis.set(index.toString(), uuid); } System.out.println(index++); }}Copy the code

  • A simple analysis of why the program counter is greater thanredisNumber of keys in the library! Because we set the expiration time for the first 100. When memory is running outredisThe current set the expiration time of the key will be the least recently used key for elimination! So our counter is going to be greater than the number of keys. Because some of the keys were cleared! We get the first 100 keys are null, which means they are deleted! So why isn’t the counter 100 more than it was last time. That’s because each time we store a UUID, the length is not fixed. And the elimination strategy itself is also a memory hog

Strategy to summarize

  • The least recently used elimination strategy is demonstrated above! There are other strategies
Noeviction: Reject write requests, provide read requests normally so that no data will be lost (default policy) 2. Volatile -lru: Attempts to eliminate keys with expiration time. Keys that are rarely used are eliminated, but keys without expiration time are not eliminated. 3. Volatile-ttl: This is almost the same as volatile-lru, but it uses the TTL value of the key and first disables the key with the lowest TTL; 4. Volatile -random: same as above, except that it randomly selects the keys in the key set in a very random way; 5. Allkeys-lru: this is different from volatile lru in that allkeys are eliminated. 6. Allkeys-random: in this way, allkeys are randomly eliminated.Copy the code
  • Which elimination strategy to use needs to be combined with our own project scenarios to use!!

Overdue delete

  • Above we from [key expiration], [elimination strategy] two Angle analysisredis. These two alone are not fully efficient use of memory! The phase-out policy is triggered when memory is running out. Student: So when the key that was set to expire is actually expired and the memory is still available? Does this scenario require the expiration key to be removed?
  • becauseredisIs single threaded, so how do you clear expired keys without affecting external services when they expire? The answer is no. Strictly speaking, it can’t be solved because a single line can only do one thing at a time! Since it can’t be solved then we can reach a state of coordination! If there is an expiration key at the same time then the clear key is very fast and the time to block the external service is very short maybe in milliseconds set to nanoseconds!
  • However, if tens of thousands of keys expire at the same time, if you want to delete tens of thousands of keys, it will take a long time to block the external service! This is definitely not acceptable. A long block time will result in a client connection timeout. This is even more unacceptable in concurrent scenarios! soredisHow to deal with the same time too much data expired scenarios, his delete expired key method is slightly different!

Time to clear

  • Set a timer for each expiration key to clear the key when it expires!
  • This method can do real-time data cleaning to ensure that memory will not be occupied for a long time! Improved memory usage!
  • But then there are problems, every one of themkeyYou need to set a timer for tracking.redisHere I guess should be to enable another thread for timing tracking! Here is cleared still help to answer please?
  • When the same time expiredkeyA lot of times! Our CPU needs to constantly execute these timers, resulting in CPU resources. It will eventually affectredisPerformance of the service

On a regular basis to remove

  • Regular deletions are what we’ve shown above,redisevery100msExecute a timer. The task of the timer is to randomly select 20 expired oneskey. Check whether the alarm is cleared. What is wrong in the illustration above is not 10S, but every other100ms. Please forgive my carelessness!!

  • Periodic deletion and periodic deletion are the opposite! Periodic deletion is a time limit that is added to the processing of key sets to ensure high availability of services. The total duration of each execution cannot exceed25ms. That is to say, for the client, the server delay will not exceed25ms 。
  • The advantage is that it does not require the CPU to operate frequently on key cleanup! Because he is regularly cleared so it will lead to a part of the data has not cleared enough time to cause memory use will be occupied!

Inert to clear

  • About lazy delete we also often use this way in peacetime development! When the data is expiredredisNot in a hurry to erase the data, but wait until the key is requested again! There is no problem in the final effect.
  • As well as regular cleanup, it ensures that the CPU does not have to switch frequently! However, the disadvantage is obvious: many expired keys will still existredisIn the.

Lazy cleanup + regular cleanup

  • We said at the beginning that high availability and real-time cleanup of expired keys is not possible! Since we can’t do that, we need to make a tradeoff between CPU and memory!redisInternal is the use of lazy cleanup and regular cleanup of two ways to use, ultimately take care of a balance between CPU and memory!

conclusion

  • I believe you are a little vague about the above three concepts. [Key expiration], [Elimination Strategy], [Expiration clearance]
  • The first thing [key expiration] isredisFor us as developers. We can reasonably set the expiration time of the key according to our business needs, so as to ensure high availability of memory
  • Secondly, how to clear the expired key we set before? We can’t clear the expired key all at once, because too much data will cause the service lag. At this time we need to clear the key code by periodically clearing slow down the lag. inredis.confIn which we can sethz 10On behalf of1SAnd that’s what we talked about above100MSThe origin of the. But just regular erasure can create missing data so we need to add lazy cleanup to ensure that the data is accurately and in real time to the client.
  • So what about the elimination strategy? The expired key is cleared if the user has not requested the expired key and more expired keys are generated as the service. At this timeredisThere are also many expired and invalid keys piled up in the service. What if you run out of memory? At this time we need to set the elimination strategy than ifvolatile-lruThe least recently used set expired key will be cleared to ensure that as much as possible to receive valid data!
  • That’s why all three are designed! With these three themes in mind, it’s easy to understand why cache avalanches, cache crashes, and cache breakdowns occur.
  • About the aboveredisCommon disaster scenarios, we continue in the next chapter to analyze how to produce and how to repair the discussion.