“This is the fourth day of my participation in the Gwen Challenge in November. Check out the details: The Last Gwen Challenge in 2021.”

Last time, I introduced two ways to determine Redis slowness, response latency and baseline performance. In addition, I have shared with you two ways to troubleshoot and resolve problems from the Redis command operation level.

However, if you find that Redis is not executing a large number of slow query commands and deleting a large number of out-of-date keys at the same time, is there nothing we can do?

Of course not! I have a lot more tricks up my sleeve that I’m going to share with you in this lesson.

If the last lesson didn’t work, then you need to focus on other mechanisms that affect performance, namely file systems and operating systems.

Redis will persist data to disk, and this process depends on the file system to complete. Therefore, the mechanism of the file system writing data back to disk directly affects the efficiency of Redis persistence. In addition, during the process of persistence, Redis is also receiving other requests, and the efficiency of persistence will affect the performance of Redis in processing requests.

On the other hand, Redis is an in-memory database, and memory operations are very frequent. Therefore, the memory mechanism of the operating system will directly affect the processing efficiency of Redis. For example, if Redis runs out of memory, the operating system will initiate swap, which will directly slow Down Redis.

Then, from these two aspects, I will continue to introduce to you how to further solve the problem of Redis slowing down.

File system: in AOF mode

Redis is an in-memory database, you may ask, so why is its performance related to the file system?

As I mentioned earlier, Redis uses AOF logs or RDB snapshots to ensure data reliability. AOF logs provide three log write back policies: No, Everysec, and always. These three write back strategies rely on the completion of two system calls to the file system, namely write and fsync.

Write simply writes the log to the kernel buffer and returns without waiting for the log to actually be written back to disk. Fsync, on the other hand, needs to write log records back to disk before returning, which takes a long time. The table below shows the system calls performed by the three write back strategies.

When the write back policy is set to Everysec and always, Redis needs to call fsync to write logs back to disk. However, the implementation of the two write back strategies is not quite the same.

When using Everysec, Redis allows one second of action logs to be lost, so the Redis main thread does not need to ensure that every action log is written back to disk. Also, fsync takes a long time to execute, and if it is executed in the Redis main thread, it can easily block the main thread. So, when the write back policy is set to Everysec, Redis uses background child threads to asynchronously complete fsync operations.

In the case of always, Redis needs to ensure that every operation log is written back to disk. If this is done asynchronously with the backstop thread, the main thread will not know whether each operation has been completed in time, which does not meet the requirements of always. Therefore, the ALWAYS policy does not use the backend child thread to execute.

In addition, when using AOF logs, Redis performs AOF rewrite to generate a new AOF log file with a reduced size in order to avoid increasing the size of the log file. AOF rewriting itself takes a long time and tends to block the main Redis thread, so Redis uses child processes to do AOF rewriting.

However, there is a potential risk that the AOF rewrite will do a lot of I/O to the disk, and fsync will not return until the data is written to the disk, so when the PRESSURE of the AOF rewrite is high, fsync will block. Although fsync is executed by the backend child thread, the main thread monitors the progress of fsync execution.

When the main thread executes fsync once with the backtable child thread and needs to write the newly received operation record back to disk again, it blocks if the main thread finds that the last fsync has not been completed. So, if the fsync performed by the backend child threads blocks frequently (such as AOF overrides that consume a lot of disk IO bandwidth), the main thread will also block, causing Redis performance to slow down.

To help you understand, let me draw another diagram to show how the fsync backend child thread and main thread are affected when disk pressure is low and high.

Well, at this point, you already know that the main I/O thread is generally not blocked due to the presence of the fsync backend thread and AOF override child. However, if there is a large amount of writing to the AOF override child during log overwriting, the fsync thread will also block, which in turn blocks the main thread, resulting in increased latency. Now, I’ll give you the troubleshooting and solution.

First, you can check the appendfsync configuration item in the Redis configuration file. The value of this configuration item indicates which AOF log write back policy the Redis instance is using, as shown below:

If the AOF write back policy is configured with Everysec or always, confirm the service’s requirements on data reliability and determine whether every second or operation needs to be logged. Some businesses do not understand the Redis AOF mechanism and may directly use always configuration with the highest data reliability. In fact, in some scenarios (such as Redis for caching), lost data can be retrieved from the back-end database without requiring high data reliability.

If a business application is sensitive to latency but allows a certain amount of data to be lost, set the no-appendfsync-on-rewrite configuration item to yes, as shown below:

no-appendfsync-on-rewrite yes
Copy the code

When this configuration item is set to yes, the fsync operation is not performed during AOF rewrite. That is, the Redis instance writes the write command to memory and returns it without calling the background thread for fsync. Of course, if the instance goes down at this point, data will be lost. On the other hand, if this configuration item is set to no (which is also the default), the Redis instance will still call the background thread for fsync when AOF is overwritten, which will block the instance.

If you really need high performance and high reliability, I suggest you consider using a fast SOLID-state drive as a writing device for AOF logs.

High-speed solid-state drives offer 10 times more bandwidth and concurrency than traditional mechanical drives. When AOF rewrite and fsync background threads are executed simultaneously, SSDS can provide sufficient disk I/O resources to reduce disk I/O resource competition between AOF rewrite and Fsync background threads, thus reducing the impact on Redis performance.

Operating system: Swap

If Redis’s AOF logging configuration is just NO, or if the AOF mode is not adopted, is there any problem causing performance to slow down?

Next, I’ll talk about a potential bottleneck: memory swap for the operating system.

Memory swap is a mechanism for the OPERATING system to swap memory data back and forth between the memory and disk. It involves disk read and write operations. Therefore, the performance of both the data swap process and the data swap process is affected by slow disk read and write operations.

Redis is an in-memory database, which consumes a large amount of memory. If you do not control the memory usage or run it together with other applications that require large amounts of memory, it may be affected by swap, resulting in slow performance.

This is especially important for a Redis in-memory database: normally, the Redis operation is performed directly by accessing memory, but once the swap is triggered, the Redis request is not performed until the disk data is read or written. Also, instead of using the Fsync thread to read and write AOF log files, swap triggers the main Redis I/O thread, which greatly increases Redis response time.

Speaking of which, I’d like to share with you an example of a performance degradation I’ve encountered due to swap.

Under normal circumstances, we run an instance that takes 300 seconds to complete 50 million GET requests, but on one occasion it took nearly four hours to complete 50 million GET requests. After problem replay, we find that the machine where the instance is located has already had a swap after Redis took nearly 4 hours to process the request. From 300s to 4 hours, the latency increases by nearly 48 times, and you can see the serious impact swap has on performance.

So, when does a swap trigger?

In general, swap is triggered mainly because the physical machine is out of memory. For Redis, there are two common situations:

  • Redis instances themselves use a large amount of memory, resulting in insufficient memory available on the physical machine;
  • Other processes running on the same machine as the Redis instance do a lot of file reading and writing. File reads and writes occupy system memory, which results in less memory allocated to Redis instances, triggering Redis to swap.

For this problem, I also offer you a solution: increase the memory of the machine or use Redis cluster.

The operating system itself records the swap usage of each process in the background, that is, how much data has been swapped. You can first check the Redis process number by using the following command, in this case 5332.

$ redis-cli info | grep process_id
process_id: 5332
Copy the code

Then, go to the process directory in the /proc directory on the Redis machine:

$ cd /proc/5332
Copy the code

Finally, run the following command to check the usage of the Redis process. Here, I’ve only captured some of the results:

$cat smaps | egrep '^(Swap|Size)'
Size: 584 kB
Swap: 0 kB
Size: 4 kB
Swap: 4 kB
Size: 4 kB
Swap: 0 kB
Size: 462044 kB
Swap: 462008 kB
Size: 21392 kB
Swap: 0 kB
Copy the code

Each line of Size indicates the Size of the memory block used by the Redis instance, and the Swap below Size indicates how much of the memory block has been swapped out to disk. If these two values are equal, the memory area has been completely swapped out to disk.

As an in-memory database, Redis itself uses a lot of memory blocks of different sizes, so you can see a lot of Size rows, some are very small (4KB) and some are very large (462044KB). Different memory blocks are swapped out to disk in different sizes. For example, in the preceding result, the first 4KB memory block is swapped out, and the Swap below it is also 4KB. In addition, the 462044KB memory block was swapped out 462008KB, which is about 462MB.

One important thing to note here is that when you have swap sizes of hundreds of Megabytes, or even gigabytes, it means that your Redis instance is under a lot of memory pressure and is likely to slow down. Therefore, the size of swap is an important indicator to check whether Redis performance slowdowns are caused by swap.

Once a memory swap occurs, the most straightforward solution is to increase machine memory. If the instance is in a Redis sliced cluster, you can increase the number of instances of the Redis cluster to share the amount of data served by each instance, thereby reducing the amount of memory required by each instance.

Of course, if the Redis instance shares the machine with another program that operates on a large number of files, such as a data analyst, you can migrate the Redis instance to run on a separate machine to meet its memory requirements. If the instance happens to be the master library in the Master/slave cluster of Redis and the slave library has a large amount of memory, you can also consider switching the master/slave library from the large memory slave library to the master library to handle client requests.

Operating system: large memory pages

In addition to memory swap, there is another memory-related factor, namely Transparent Huge Page (THP), which also affects Redis performance.

The Linux kernel has supported the memory big page mechanism since 2.6.38, which supports page allocations of 2MB, while regular page allocations are performed at a granularity of 4KB.

Many people think: “Redis is an in-memory database, isn’t the big page in memory just enough to meet the needs of Redis? It also reduces the number of times you allocate the same amount of memory, which is also redis-friendly.”

In fact, system design is usually a trade-off process, we call it trade-off. A lot of mechanics are usually both good and bad. Redis’ use of large memory pages is a typical example.

Although large memory pages can benefit Redis in terms of memory allocation, it is important to remember that Redis needs to persist data to ensure data reliability. This writing process is performed by an additional thread, so the Redis main thread can still receive client write requests at this point. Write requests from clients may modify data that is being persisted. In this process, Redis uses a copy-on-write mechanism, that is, when data is to be modified, Redis does not directly modify the data in memory, but makes a copy of the data and then modifies it.

If large memory pages are used, Redis will need to copy 2MB of large pages even if the client requests only 100 MB of data to be modified. In contrast, with the regular memory page mechanism, only 4KB is copied. In comparison, you can see that when the client requests a lot of changes or new data is written, the large page mechanism will cause a large number of copies, which will affect the normal Redis fetch operation, resulting in slow performance.

So what to do? Very simple, close the memory large page, and that’s it.

First, we need to check the large memory pages. To do this, run the following command on the machine where the Redis instance is running:

cat /sys/kernel/mm/transparent_hugepage/enabled
Copy the code

If the result is always, the memory large page mechanism is enabled. If the value is never, the memory large page mechanism is disabled.

When deployed in a real production environment, I recommend that you do not use the memory page mechanism, and the operation is very simple, just execute the following command:

echo never /sys/kernel/mm/transparent_hugepage/enabled
Copy the code

summary

In this lesson, I will show you how to deal with Redis slowness from both the file system and operating system dimensions.

In order to facilitate your application, I have sorted out a Checklist consisting of 9 checkpoints for you. I hope you can follow these steps to check one by one when the performance of Redis slows down and solve the problem efficiently.

  1. Gets the baseline performance of the Redis instance in the current environment.
  2. Is a slow query command used? If so, replace the slow query command with another command, or put the aggregate calculation command on the client side.
  3. Is the expiration time set to the same for expired keys? For keys that are deleted in batches, you can add a random number to the expiration time of each key to avoid simultaneous deletion.
  4. Is there a bigkey? If you are running Redis 4.0 or above, you can use the asynchronous thread mechanism directly to reduce the main thread blocking. If the version is earlier than Redis 4.0, you can use the SCAN command to iterate the deletion. You can run the SCAN command on the client to query and aggregate bigkeys.
  5. What is the Redis AOF configuration level? Is this level of reliability really needed at the business level? If high performance is required and data loss is allowed, set the no-appendfsync-on-rewrite option to yes to avoid AOF overwriting and fsync competing for disk I/O resources and causing Redis delays. Of course, if high performance and high reliability are required, it is best to use a high-speed solid state disk as the write disk for AOF logs.
  6. Is the Redis instance using too much memory? Did a swap occur? If so, increase machine memory, or use a Redis cluster to spread out the number of key-value pairs and memory stress in a single Redis machine. Also, avoid Redis sharing machines with other memory-hungry applications.
  7. Is the transparent large page mechanism enabled in the running environment of the Redis instance? If so, simply turn off the memory page mechanism.
  8. Is the Redis primary/secondary cluster running? If yes, limit the data size of the master library instance to 2~4GB, so that the slave library will not be blocked due to loading large RDB files during the master/slave replication.
  9. Are machines running Redis instances using multi-core cpus or NUMA architectures? With multi-core cpus, you can bind physical cores to Redis instances. When using the NUMA architecture, be careful to run the Redis instance and the network interrupt handler on the same CPU Socket.

In fact, there are many other factors that affect system performance, and these two lessons will cover solutions to the most common problems.

Don’t panic if you run into something unusual. Here’s a tip: Check for annoying “neighbors”. Specifically, Redis is running on a machine that uses memory, disk IO, and network IO, such as database programs or data collection programs. If so, I suggest you migrate these programs to run on another machine.

To ensure the high performance of Redis, we need to provide Redis with sufficient computing, memory, and IO resources, and provide it with a “quiet” environment.

Each lesson asking

In these two lessons, I have introduced you to systematically locating, troubleshooting, and solving Redis slowdowns. So, I’d like to ask you for a chat, have you ever experienced Redis slowing down? If so, how did you solve it?

Feel free to share your experience in the comments, and if you find today’s content helpful, feel free to share it with your friends or colleagues, and we’ll see you next time.