background
The GaussDB(for Redis) service team found a delay jitter problem caused by fork in the process of supporting the cloud of a customer’s business. In an attitude of being responsible to the customer, we explored the performance impact of fork in detail. In addition, the latest GaussDB(for Redis) version has solved the jitter problem and eliminated the internal fork usage. Compared with native Redis, the potential performance of fork is completely solved.
The problem focus
During the cloud cable commissioning of huawei GaussDB(for Redis) service on a customer, delay jitter occurs once every five minutes. Huawei Cloud GaussDB(for Redis) team solved the jitter problem and confirmed that the jitter was caused by fork. Fork is an important dependency of open source Redis. I hope this article can help you fully understand the influence of fork when using open source Redis, so as to choose a better solution.
Phenomenon of the problem
When a customer’s services are connected to GaussDB(for Redis), the system has regular delay jitter every 5 minutes.
In normal cases, the message delay is 1-3ms, and the jitter delay is about 300ms. Usually after a period of pressure measurement began to appear jitter; Once the jitter appears, it is kept regularly at 1 time every 5 minutes; The duration of each jitter is within 10ms. The following figure shows a jitter message captured in the system slow log (sensitive information is masked) :
Problem analysis
1) Check jitter sources:
Because the time distribution of faults is very regular, the impact of scheduled tasks should be excluded first, including: Agent: Periodic statistics reporting task that interconnects with management and control tasks Kernel: Periodic operations of the execution engine (Redis protocol parsing) and storage engine (Rocksdb) (including RocskDB statistics and WAL clearing) mask the two types of scheduled tasks, but jitter still exists.
After the elimination failed, the decision was made to return to positive orientation. By adding segment time statistics for data access paths, it is found that memory operations (including allocate and memcpy) take significantly longer at jitter time. Basically, the long delay is blocking memory operations.
Since it is identified as the jitter of the system-level operation, the next step is to capture whether the system is abnormal at the time of jitter. The method we adopt is to periodically capture top information through scripts and analyze system changes. As luck would have it, a key piece of information was caught immediately after the script was deployed: every time there was jitter, an FRM-timer process appeared in the system. This process is a child process of GaussDB(for Redis). It is instantaneous and exits after 1 to 2 seconds.
In order to confirm the influence of this process, we also captured perF information, and found that when this process appeared, Kmalloc, memset_SSE, memcopy_SSE and other kernel system calls increased. It can be inferred from the above information that the FRM-timer process should be forked out, and the jitter source can be basically locked on the action of fork FRM-timer. 2) Determine the code causing jitter:
The next step is to analyze the history of FRM-Timer. Since this identifier is not in our code, it is up to laton to collaborate with our library sibling. After a joint investigation, it is confirmed that FRM-Timer is a timer processing thread in the log library Liblog. If the thread forks an anonymous child, the parent’s thread name is reused, as if the Redis process created a child named FRm-timer. Since FRM-Timer handles timer tasks for all modules in Liblog, which module triggered the fork? Here we take a more subtle approach by adding code to the timer processing logic: if the processing takes longer than 30ms, STD :: abort() is called to exit to generate the core stack. By analyzing core stack and checking codes, the codes causing jitter were confirmed as follows:
The above code is used to archive the logs periodically, executing a system call every 5 minutes to run the script to archive the logs. The source code of the Linux system call is as follows, which is actually a process of forking a child process and then calling execl.
This analysis leaves us with one final question to answer: is the jitter caused by fork, or by script content? To this end, we designed a set of test cases: Use case 1: change the script content to the simplest echo operation; Use case 2: simulate a frM-timer-like thread in the Redis process, and trigger the thread to fork using the command; Use case 3: Example 4: Simulate a thread similar to FRM-timer in the Redis process and trigger the thread to fork and then excel operation case 4: Simulate a thread similar to FRM-timer in the Redis process and trigger the thread to execute system operation case 5: Simulate a thread similar to FRM-timer in Redis process, and trigger the thread to perform vfork first and then Excel operation by command. Final verification result:
Use case 1: There is jitter. Use case 2: There is jitter. Use case 3: There is jitter. Use case 4: There is jitter. Use case 5: No jitter. The results of case 1 show that jitter has nothing to do with script content. The results of use cases 2, 3, and 4 show that the root cause of jitter is the fork operation. The result of use case 5 further confirms that the root cause of jitter is the fork operation. The final fault causes are shown as follows:
3) Further explore the influence of fork:
Fork, as we all know, is the Linux (strictly POSIX) system call for creating child processes, and historically, the mainstream view has mostly praised it; In recent years, however, as the technology has evolved, there have been objections: some argue that fork is a relic of a previous era, outdated and harmful in modern operating systems. Radical views even suggest that it should be abandoned altogether. (see appendix 1,2) fork one of the main issues currently being criticized is its performance. The common understanding of fork is that it uses copy-on-wirte copy-on-write policy, so it is not sensitive to its performance impact. In practice, although the shareable data content does not need to be replicated when forked, the overhead of copying its associated kernel data structures (including page directories, page tables, VM_AREA_STRUc, and so on) is also significant. The article in Appendices 1 and 2 describes fork overhead in detail, and the problem we encountered this time is a vivid example: for delay-sensitive applications like Redis, one fork can cause messages to jitter by 100 times, which is unacceptable for the application. 4) Native Redis fork problem:
4.1 native Redis is also plagued by fork problems (see appendix 3,4,5), including the following scenarios:
1) Data backup
An RDB file is generated during backup, so Redis needs to trigger a fork.
2) Master/slave synchronization
In full replication scenarios (including initial replication or other cases where heaps are severe), the primary node needs to generate an RDB file to speed up synchronization, as well as triggering a fork.
3) AOF rewrite
A fork can also occur when AOF files are large and need to be merged and overwritten.
4.2 Impacts of the above fork problems on native Redis are as follows:
1) Service jitter
Native Redis uses a single-threaded architecture. If such fork occurs during business peak hours such as e-commerce promotion and hot events, Redis will be blocked, resulting in an avalanche of business impact.
2) Memory utilization is only 50%
When Fork, the child process needs to copy the memory space of the parent process. Although COW is used, enough space should be reserved in case of unexpected events. Therefore, the memory utilization is only 50%, which also doubles the cost.
3) Impact of capacity scale
In order to reduce the impact of fork, the maximum memory of a single process of native Redis in the production environment is usually controlled within 5G, resulting in the capacity of native Redis instances being greatly limited and unable to support massive data.
Solution Modify the periodic archive logic in the log library liblog to stop fork child processes. The system checks and corrects fork calls in GaussDB(for Redis) code (including the used library code). According to the final result of investigation, only this problem point of this time actually involved fork. After the current change, the delay of GaussDB(for Redis) remains stable and is not affected by fork performance. Note: GaussDB(for Redis) is independently developed by Huawei Cloud based on the save-computing separation architecture. Therefore, there is no scenario where the fork of native Redis is invoked.
conclusion
This paper explores the performance impact of system call fork (for Redis) by analyzing a delay jitter problem caused by fork. The latest GaussDB(for Redis) version has solved this jitter problem and eliminated internal fork usage, completely resolving the performance issues of fork compared to native Redis. Hope that through the analysis of this problem, can bring you some inspiration, convenient for you to better selection.