“This is the fifth day of my participation in the Gwen Challenge in November. Check out the details: The Last Gwen Challenge in 2021.”

When using Redis, we often encounter such a problem: after deleting data, the amount of data is not large, why use the top command to view, but still find that Redis occupies a lot of memory?

In fact, this is because when data is deleted, the memory space freed by Redis is managed by the memory allocator and is not immediately returned to the operating system. As a result, the operating system still records that it allocated a large amount of memory to Redis.

However, there is a potential risk that Redis may not be able to free up contiguous memory, which is likely to remain idle. This leads to a problem: While there is free space, Redis can’t use it to store data, reducing not only the amount of data Redis can actually store, but also the cost-return Redis has to run the machine.

Let’s use a metaphor. We can compare the memory space of Redis to the number of seats on a high-speed train. If a high-speed railway has a large number of seats but carries a small number of passengers, then the efficiency and cost of running a high-speed railway will be low and high, and the cost performance will be reduced. The same is true of Redis. If you rent a cloud host with 16GB of ram to run Redis and only store 8GB of data, your ROI on renting that cloud host will be cut in half, which is definitely not what you want.

Therefore, in this lesson, I will talk to you about the memory storage efficiency of Redis, explore why the data has been deleted, but the memory is unused, and the corresponding solution.

What is memory fragmentation?

In general, memory space is idle, usually because the operating system has serious memory fragmentation. So, what is memory fragmentation?

For your convenience, LET me explain with the help of a high-speed train seat. You and your two friends need exactly three tickets to travel on a high-speed train. However, you want to sit together so you can talk on the way. However, when choosing a seat, you find that consecutive seats are no longer available. So you had to take another train. As a result, you need to change your travel time, and three seats are vacant on this train.

In fact, the number of empty seats on this train matches the number of you, but they are scattered, as shown below:

The memory fragmentation of the operating system is easy to understand with what we might call “car seat fragmentation.” While the rest of the operating system total memory space enough, however, is used to apply for a continuous address space N bytes, but in the rest of the memory space, there is no size for N bytes of contiguous space, so, the remaining space is memory fragments (such as in the picture above “free 2 bytes” and “free 1 byte”, is one such fragments).

So what causes memory fragmentation in Redis? Next, I’ll take you to have a look. Only when we understand the causes of memory fragmentation, can we take appropriate medicine to make full use of the memory space occupied by Redis and increase the amount of stored data.

How are memory fragments formed?

In fact, the formation of memory fragmentation has internal and external causes of two levels. In simple terms, the internal cause is the memory allocation mechanism of the operating system, and the external cause is the load characteristics of Redis.

Internal cause: Allocation policy of the memory allocator

The allocation strategy of the memory allocator determines that the operating system cannot “allocate on demand”. This is because memory allocators typically allocate memory at a fixed size, rather than allocating it to programs at the exact size of the memory requested by the application.

Redis allocates memory using liBC, Jemalloc, and TCMALloc allocators. Jemalloc is used by default. Next, I’ll take jemalloc as an example to explain. Other allocators have similar problems.

One of jemalloc’s allocation strategies is to partition memory in a series of fixed sizes, such as 8 bytes, 16 bytes, 32 bytes, 48 bytes… , 2KB, 4KB, 8KB, etc. Jemalloc allocates a certain amount of memory to a program when it is closest to a fixed value.

The distribution itself is to reduce the number of distribution. For example, If Redis requests 20 bytes of space to hold data, Jemalloc will allocate 32 bytes. If the application needs to write 10 bytes of data, Redis will not need to request more space from the operating system, because the 32 bytes allocated are already sufficient, thus avoiding an allocation.

However, if Redis requests a different size of memory from the allocator each time, this allocation runs the risk of fragmentation, which is due to Redis’s externality.

External cause: different key – value pair sizes and deletion operations

Redis usually serves as a common cache system or key-value database. Therefore, data of different business applications may be stored in Redis, resulting in key-value pairs of different sizes. As a result, When Redis requests memory space allocation, it has its own space requirements of different sizes. This is the first external cause.

However, as we have just mentioned, the memory allocator can only allocate memory according to a fixed size. Therefore, the allocated memory space is usually larger than the requested memory space, which will cause a certain amount of fragmentation and reduce the memory storage efficiency.

For example, application A holds 6 bytes of data and Jemalloc allocates 8 bytes according to the allocation policy. If application A does not store new data, the extra 2 bytes of space is memory fragmentation, as shown below:

The second external cause is that these key-value pairs can be modified and deleted, resulting in space expansion and release. Specifically, on the one hand, if the modified key-value pair becomes larger or smaller, it needs to take up additional space or free up unused space. On the other hand, deleted key-value pairs no longer require memory space, which is then freed up to form free space.

I drew the following picture to help you understand.

At first, applications A, B, C, and D hold 3, 1, 2, and 4 bytes of data, respectively, and occupy the corresponding memory space. Then, application D deletes 1 byte, and the 1 byte space is free. Next, application A modifies the data from 3 bytes to 4 bytes. In order to maintain the spatial continuity of A’s data, the operating system needs to copy B’s data to another space, for example, the space that D has just released. At this point, applications C and D also delete 2 bytes and 1 byte of data, respectively, resulting in 2 bytes and 1 byte of free fragmentation in the entire memory space. If application E wants a continuous space of 3 bytes, it can’t be satisfied. Because, although the amount of space is enough, it’s debris space, it’s not continuous.

Here, we know the internal and external factors causing memory fragmentation. Among them, the memory allocator policy is the internal cause, and the load of Redis is the external cause, including the change of memory space caused by the modification and deletion of key pairs of different sizes.

A large amount of memory fragmentation can cause the actual memory utilization of Redis to be low, and we will solve this problem. However, before we can solve the problem, we need to determine whether there is any memory fragmentation during Redis operation.

How do I determine whether there are memory fragments?

Redis is a memory database. The level of memory utilization is directly related to the running efficiency of Redis. To enable users to monitor real-time memory usage, Redis provides the INFO command, which can be used to query detailed information about memory usage.

INFO memory # memory used_memory:1073741736 USED_memory_human: 1024.00m used_memory_rss:1997159792 Used_memory_rss_human: 1.86 G... Mem_fragmentation_ratio: 1.86Copy the code

There is a mem_fragmentation_ratio, which is the current memory fragmentation rate of Redis. So how is this fragmentation rate calculated? This is the result of dividing the two indices used_memory_rss and used_memory from the command above.

mem_fragmentation_ratio = used_memory_rss/ used_memory
Copy the code

Used_memory_rss is the physical memory space actually allocated by the operating system to Redis, which contains fragments; Used_memory is the space Redis actually requested to use to store data.

Let me give you a quick example. For example, if a Redis application uses 100 bytes (usED_memory) and the operating system actually allocates 128 bytes (usED_memory_rss), mem_fragmentation_ratio is 1.28.

So, knowing this metric, how do we use it? Here, I provide some empirical thresholds:

  • Mem_fragmentation_ratio is greater than 1 but less than 1.5. That makes sense. That’s because the factors I just described are unavoidable. After all, internal memory allocators must be used, and allocation policies are common and not easily modified; The external factor is determined by the Redis load and cannot be restricted. Therefore, memory fragmentation is normal.
  • Mem_fragmentation_ratio is greater than 1.5. This indicates that the memory fragmentation rate has exceeded 50%. Generally, at this point, we need to take some measures to reduce the memory fragmentation rate.

How do I clear memory fragments?

When Memory fragmentation occurs in Redis, a “simple and crude” solution is to restart the Redis instance. Of course, this is not an “elegant” approach, after all, rebooting Redis has two consequences:

  • If the data in Redis is not persisted, then the data is lost;
  • Even if Redis data is persisted, we still need to restore it through AOF or RDB. The recovery time depends on the size of AOF or RDB. If there is only one Redis instance, services cannot be provided in the recovery phase.

So, what’s another good idea?

Fortunately, Redis itself provides a method for automatic defragmentation from 4.0-RC3 onwards, so let’s look at the basic mechanism.

Defragmentation, simply put, is “move out of the way, merge space”.

Let me also take the seat selection of the high-speed train as an example to explain. You and your friend didn’t want to waste time, so they bought three seats that were not together. But when you get on the bus, you and your friend sit together again by switching seats with someone else.

That said, the mechanism of debris removal is easy to understand. When data splits a contiguous memory space into discrete chunks, the operating system copies the data elsewhere. At this point, the data copy needs to be able to empty the space occupied by the data, the original discontinuous memory space into contiguous space. Otherwise, if the data is copied and does not form contiguous memory space, it is not clean.

Let me draw a picture to illustrate.

Before defragmentation, this 10-byte space has one 2-byte and one 1-byte free space, but the two Spaces are not contiguous. Before clearing debris, the operating system copies the data of application D to the 2-byte free space and releases the space occupied by D. Then, copy B’s data into D’s original space. This makes the last three bytes of the 10-byte space a contiguous space. This is the end of the debris cleanup.

However, there is a caveat: defragmentation comes at a cost, as the operating system has to copy multiple copies of data to new locations to free up space, which can cost time. Because Redis is single-threaded, Redis can only wait while data is copied, which results in Redis not being able to process requests in a timely manner, resulting in poor performance. In addition, sometimes the order of data copy needs to be paid attention to. Just like the example of cleaning up memory fragmentation, the operating system needs to copy D first and release the space of D before copying B. This sequential requirement can further increase Redis wait time, resulting in performance degradation.

So what can be done to minimize the problem? This brings us to the parameters Redis sets specifically for the automatic memory defragmentation mechanism. You can set parameters to control when defragmentation starts and ends, as well as how much CPU it consumes, thereby reducing the performance impact of defragmentation on Redis’s own request processing.

First, Redis needs to enable automatic memory defragmentation. You can set the ActiveDefrag configuration item to yes as follows:

config set activedefrag yes
Copy the code

This command only enables the automatic cleanup function, but the specific cleanup time is controlled by the following two parameters. Each of these parameters sets a condition that triggers a memory cleanup, and if both conditions are met, the cleanup begins. During the cleaning process, stop automatic cleaning as long as one condition is not met.

  • Active-defrag -ignore-bytes 100mb: indicates that the memory fragment will be cleared when the number of bytes reaches 100mb.
  • Active-defrag-threshold-lower 10: indicates that the memory fragment space will be cleared when the proportion of the total space allocated by the OS to Redis reaches 10%.

As much as possible in order to reduce the influence of debris to Redis normal request processing, automatic memory fragments cleaning function when executed, will monitor cleaning operation takes up CPU time, and also set up two parameters, respectively, are used to control the cleaning operation takes up CPU time ratio, the lower limit, not only ensure clean up work can be carried out as normal and, It also avoids degrading Redis performance. The two parameters are as follows:

  • Active-defrag-cycle-min 25: indicates that the proportion of CPU time used in automatic cleaning is not less than 25% to ensure normal cleaning.
  • Active-defrag-cycle-max 75: indicates that the proportion of CPU time used in the automatic cleaning process is not higher than 75%. Once the CPU time is exceeded, the cleaning will be stopped, so as to avoid a large number of memory copies blocking the Redis during the cleaning, resulting in increased response delay.

The automatic defragmentation mechanism takes into account not only the defragmentation space ratio and the impact on Redis memory efficiency, but also the CPU time ratio of the defragmentation mechanism itself and the impact on Redis performance when controlling the defragmentation start and stop time. In addition, the cleaning mechanism also provides four parameters, which can be flexibly used according to the data volume requirements and performance requirements of practical applications. It is recommended that you use this mechanism in practice.

summary

In this lesson, I learned with you about the memory efficiency of Redis. One of the key technical points is to identify and deal with memory fragmentation. Simply put, it’s “three ones” :

  • The info memory command is a good tool to check fragmentation rates;
  • The fragmentation threshold is a good rule of thumb to help you decide effectively whether to clean up debris;
  • Automatic memory fragmentation cleaning is a good method to avoid the actual memory utilization of Redis caused by fragmentation and improve the cost return rate.

Memory fragmentation is nothing to be afraid of, what we need to do is to understand it, pay attention to it, and borrow efficient ways to solve it.

Finally, one more tip for you: Automatic defragmentation involves memory copying, which is a potential risk for Redis. If you experience slow Redis performance in practice, remember to check the log to see if defragmentation is taking place. If Redis is indeed cleaning up debris, THEN I recommend that you turn down the value of active-defrag-cycle-max to mitigate the impact on normal request processing.

Each lesson asking

As usual, LET me give you a quick question. In this lesson, I mentioned that you can use mem_fragmentation_ratio to determine if The current memory fragmentation rate of Redis is severe, and the empirical threshold I give is always greater than 1. If mem_fragmentation_ratio is less than 1, what happens to Redis memory usage? What is the impact on Redis performance and memory utilization?

You’re welcome to leave your thoughts and answers in the comments, and share them with me. If you find today’s content helpful, feel free to share them with your friends or colleagues, and we’ll see you next time.