This article is a practical experience post by uU colleagues, which is a complete case of problem discovery, problem analysis and problem solving. I hope it will be helpful to share with you.

cause

It started when a colleague of the company posted a problem in an internal email group. After running a business program using Go1.8.3 for some time, some goroutine stuck in a ForkLock. The colleague thought this was a go1.8.3 bug. It did not reappear after upgrading to go1.10. To get to the bottom of it, a colleague posted an issue on Github:

Github.com/golang/go/i…

I took a look at the business code of this problem. The approximate usage is that the parent process invokes the Command under OS /exec to open the child process to execute shell commands. Command is followed by a call to the golang encapsulated forkExec to open the child process and execute the Command. ForkExec uses ForkLock.

Problem analysis

ForkLock exists to avoid the following situations: In the case of multiple Goroutines forking exec, in order for the child process to inherit only the file descriptors it needs, the parent process needs to add the O_CLOEXEC flag when creating these file descriptors, so that these descriptors are closed in the child process. The child process opens the descriptor it needs to inherit as needed.

On Linux after 2.6.27, opening a file or pipe and setting O_CLOEXEC is an atomic operation, so this is not a problem. However, golang requires a kernel version of 2.6.23 or higher. Open and set O_CLOEXEC are two operations, and if a fork occurs between the two operations, the child process may inherit file descriptors that it does not need and therefore need to lock. Focus on the source code for forkExec:

From the signs of the problem, one of the Goroutines must have got stuck in the forkExecPipe or forkAndExecInChild, and the lock was not released, so some goroutines were unable to reach the lock and starved to death. ForkExecPipe last calls kernel pipe2, and forkAndExecInChild last calls kernel Clone and exec.

Reason for speculation

Pipe2 is a quick system call, so it is possible that the block system calls are clone and exec, plus the problem does not recast on GO1.10. Compare the forkAndExecInChild function with the go1.8 code:

go1.8

go1.9

Go1.9 added CLONE_VFORK and CLONE_VM. Clone with SIGCHILD can be considered similar to fork(do_fork is called at the end). The problem with fork is that the larger the memory footprint of the parent process, the worse the performance.

Bugzilla.redhat.com/show_bug.cg…

This case was proposed in 2011 and updated in July this year. The problem in this case is that although the Linux kernel introduced copy-on-write mechanism, page table entries are still copied when fork. The larger the process virtual memory is, the more page table entries need to be copied. So the slower the fork. Someone in Golang’s discussion group has tested that the heap size is 2G, and the fork time can reach the level of milliseconds. Normally, it is tens of microseconds, a thousand-fold difference.

Go1.9 adds these two parameters to make the child process and the parent process share memory, which is equivalent to calling vfork. It does not need to copy the page table entries, which speeds up the creation speed. From the test effect, it is stable in dozens of subtle.

So it is a reasonable guess that in programs written below go1.9, when the memory footprint of the program is large enough and processes are created frequently enough, ForkLock waits can result.

The experiment demonstrates

I wrote a test program using GO1.8.3 and tested it on a 2-core 4G VM (kernel 3.10.0-693.17.1.el7.x86_64).

At external intervals of 10 seconds, the program is given a SIGUSR1 signal, which prints the runtime stack. After some time, parts of the Goroutine take longer and longer to access the ForkLock. See the following two figures:

This is not the case with GO1.9 and above, which I think is already telling. To solve this problem, upgrade to go1.9 or later.

Write in the last

The purpose of vfork is to solve the performance problem caused by forking the page table entries, and most scenarios fork after the call exec, exec will delete all the page table to reset the new page table, there is no need to copy the page table entries. If the child changes a variable, the parent process will be affected, and the kernel will suspend the parent process so that the child can execute first. These limitations basically limit vfork to exec scenarios, not fork.

Before the release of GO1.9 was ready for vfork, it was suggested that the code was not robust enough because rawVforkSyscall would return and execute instructions in the parent process, giving the process a chance to break both shared stacks. Therefore, a commit is made to cause rawVforkSyscall to return without doing anything in the parent process segment to resolve the interaction, as shown in the figure below:

If you’re interested in learning more, check out the Commit review, with speakers like Rob Pike.

Go-review.googlesource.com/c/go/+/4617…

For more technical stuff, check out cloud Computing, where we’re here to change the future with cloud computing.