As an in-memory database, memory size is critical to Redis. More memory means more data can be stored. But do not know you have encountered such a situation, obviously the space is very large, but the use of memory is not very ideal.
Why does this happen? In this episode, we’ll take a look at this “weird” incident.
Sit tight and get ready to go!
Note: Mind mapping
Check memory usage
In order to know the memory usage of Redis, we need to obtain relevant information.
Viewing memory information in Redis is easy, just type “Info Memory” on the command line to see the various related data. Here I list some of the more important metrics:
Used_memory: used memory size.
Used_memory_rss: redis physical memory size.
Mem_fragmentation_ratio: memory fragmentation rate.
There is a term for memory fragmentation, which can be used to describe current memory usage.
Specific calculation method:
For the memory fragmentation rate, it is most reasonable to keep it between 1 and 1.5.
What is memory fragmentation
Given the memory fragmentation rate, what is memory fragmentation?
The definition is that a contiguous chunk of free space becomes unusable because it is smaller than the space to be allocated, which is memory fragmentation as a whole.
Here’s an example:
Let’s say you have a block of 100MB of contiguous free memory, and you’re applying for a block of 30MB each time. So after three requests, you only have 10MB left, and the fourth request will fail. If no other space is freed and each request is greater than 10MB, the remaining space is fragmentation for the entire memory.
The cause of memory fragmentation
Redis is most commonly used for writing, modifying, and deleting data. Each of these operations generates a certain amount of memory fragmentation after execution.
Write data
Redis allocates memory based on a fixed size. To reduce the number of allocations, Redis allocates space based on the nearest fixed value of the requested memory.
What does that mean if Redis allocates memory in terms of 8 bytes, 16 bytes, 32 bytes, 48 bytes, etc. When you want to store 18 bytes of data, Redis allocates 32 bytes (because 32 is the closest fixed value to 18). If the rewritten data requires less than 14 bytes of memory, Redis no longer allocates the data.
It’s like if you have different boxes, in order to put things in, you need to find the closest size box to put them in. But once you’re inside, you realize there’s room for a few small things, so you don’t have to look for boxes anymore.
However, this way of allocating space introduces a certain amount of memory fragmentation. We can think of a partition of a fixed size as a box of different volumes, and each box has a certain amount of space left over. This remaining space is memory fragmentation.
Modify the data
When the key-value pairs are modified, they may become larger or smaller, taking up extra space or freeing up unused space accordingly.
As shown in the figure, currently A, B and C occupy 3, 2 and 4 bytes respectively. When A is changed from 3 bytes to 2 bytes, 1 byte space will be left, and 1 byte fragment will appear.
What if I change data A from 3 bytes to 4 bytes? In order to maintain the spatial continuity of data A, the operating system copies data B to another space. Another fragment of 1 byte will appear.
Delete the data
Once you understand modifying data, deleting data is easy to understand. As in the previous example, data B is deleted, freeing 2 bytes of space. This creates a two-byte fragment for the entire memory space.
How to resolve memory fragmentation
You may be wondering, what’s the harm of memory fragmentation?
Let’s do the boxes on the top. If you think about it, if you have to load all these boxes into the car and ship them away, each box has free space (memory fragmentation), then the efficiency and cost performance of the operation will be very low. Similarly, in Redis, the actual utilization is low due to the large amount of fragmentation.
So is there a way to fix memory fragmentation?
Over to
The first way is simple: just tear it down and start again. That is, Redis directly restart the thing, memory a power off the world will be clean. But there are many pitfalls to this brute force approach.
If you do this in a production environment, you have to burn incense beforehand, and hopefully nothing will go wrong. If you haven’t persisted, then don’t burn it, it won’t work. If there is persistence, the recovery time depends on the size of your persistent file and is not available at this stage. Vexed?
Spatial displacement
So is there a less exciting way?
Yes, higher versions of Redis provide a way to clean up memory fragmentation. In a word, it’s spatial permutation.
What’s the substitution? Our goal is to eliminate memory fragmentation, so don’t we just rearrange the used memory data together? Let the discontinuous space become continuous, the rest of the space, continue to allocate.
Let me draw a picture:
However, it is easy to say that there are performance losses between theory and practice.
During multiple data copies, single-threaded Redis can only wait, unable to respond to client requests. At this time can only stare, performance is too affected.
Cool, what should I do then? ! Don’t worry. There’s a mitigation strategy. You keep reading.
Redis has a special parameter set for automatic defragmentation cleaning: ActiveDefRag Yes.
This command starts the cleanup function, which is not enough. There are other conditions in Redis to enable cleanup.
The following parameters can be cleaned if any of the conditions are met:
Active-defrag -ignore-bytes 100mb: clear fragments when the size of fragments reaches 100mb.
Active-defrag-threshold-lower 10: enable clearing when the number of fragments exceeds 10%.
Active-defrag-threshold-upper 100: clear as many memory fragments as possible if the memory fragment exceeds 100%.
In the process of processing, to avoid the impact on normal requests, and ensure the performance. Redis also provides parameters to monitor THE CPU usage to ensure that the cleanup can proceed properly only when:
Active-defrag-cycle-min 5: Indicates that the proportion of CPU time occupied by memory fragments is not lower than this value to ensure normal cleaning.
Active-defrag-cycle-max 75: indicates that the CPU time occupied by clearing memory fragments is not higher than this value. Once this is exceeded, the cleanup is stopped to avoid blocking Redis with a large number of memory copies and delaying other requests during cleanup.
conclusion
Check memory usage
To view Redis memory information, run the Info Memory command. According to the memory fragmentation rate, you can clear the memory fragmentation within a certain time. Memory fragments cause the fault
When writing data, Redis allocates memory according to a fixed size to reduce the number of times. Additional memory space is freed or occupied when data is modified, and space is freed when data is deleted. This results in different levels of memory fragmentation. How to resolve memory fragmentation
If the Redis is not persisted, an accident may occur. In the case of persistence, the recovery speed depends on the size of the file.
Through space permutation, that is, the used memory data is rearranged together.