What happened was that for a service online, when it started, RSS continued to rise as the number of tasks increased, but after the business peak, the number of tasks decreased, RSS did not decrease, but remained at a high level.

So who’s holding the memory? In order to locate the problem, I continuously collected all Go Runtime memory indicators of the process, RSS and other indicators of the process, and drew a line graph in the time dimension:

In line with the DRY principle, I made the collection and drawing part into an open source library, so that the business code can be easily accessed to draw line charts of the above style and view them in real time through the web page. Git address: github.com/q191201771/…

The metrics in the figure, VMS and RSS, are common to any Linux process. Sys, HeapSys, HeapAlloc, HeapInuse, HeapReleased, and HeapIdle are the memory records of the Go Runtime.

  • Linux Memory Management: The difference between RSS and VSZ
  • Go Pprof Memory Runtime

Simply put, RSS can be considered the actual memory footprint of a process and the most important memory metric for a process’s external performance. HeapReleased is the memory that the Go process returned to the operating system. In the old article “How to Analyze The Memory Usage of A Golang Program,” WE experimented with how HeapReleased went up and RSS went down with garbage collection.

In this case, however, as you can see from the graph, HeapReleased went up and RSS never went down.

Let’s break it down. (I will not repeat the interpretation of each indicator below, just look at the above two articles)

First of all, the number of business tasks continued to increase from the start time of 03-13 17:47:17, then began to decline after 22:17:17, and then began to rise again after the start time of 03-14 16:17:27. And then the cycle repeats. This is characteristic of the actual memory requirements of the business.

  • The overall waveform of VMS and RSS was consistent, and maintained at a certain difference, which was in line with expectations.
  • Sys and RSS almost overlap, indicating that this is indeed the memory used by the Go code, as expected.
  • The waveforms of HeapSys and Sys are consistent and remain at a relatively small difference, indicating that most of the memory is heap memory, as expected.
  • HeapInuse and HeapAlloc are constantly oscillating with consistent waveforms and maintained at a certain difference value. They rise in the peak period and decline in the low peak period, meeting expectations.
  • HeapIdle oscillates before the first peak and then continues to be contrary to the waveform of HeapInuse, indicating that it plays a role of cache and meets expectations.
  • HeapIdle and HeapReleased waveforms are consistent with expectations.

So back to the original question, why did HeapReleased go up and RSS not go down?

This is because the memory requested by Go with Mmap is freed with Madvise. See the go/ SRC /runtime/mem_linux.go code.

When madvise marks a portion of memory as no longer in use, there are two ways MADV_DONTNEED and MADV_FREE (passed in with the flag argument) :

  • MADV_DONTNEEDMarked memory, if used again, triggers a page-missing interrupt
  • MADV_FREEMarked memory is not released until the kernel runs out of memory. This block of memory can still be reused before being freed. This feature is derived fromLinux 4.5Version kernel start support

Obviously, MADV_FREE is a space-for-time optimization.

  • inGo to 1.12Previously, Go Runtime in LinuxsysUnsedusemadvise(MADV_DONTNEED)
  • inGo to 1.12Later, inMADV_FREEPreferred when availableMADV_FREE

See github.com/golang/go/i…

After Go 1.12, there is a way to forcibly roll back the use of MADV_DONTNEED by adding GODEBUG=madvdontneed=1 before executing the program. See github.com/golang/go/i…

Ok, now that we know why RSS doesn’t release, let’s go back to our own question and conclude.

In fact, in our case, the process has exclusive access to the resources of the execution environment, meaning that the machine has only one core business process, and the memory is mainly used by it.

So we know that it is not the upper business that we write that holds the memory by mistake, but the optimization that we do at the bottom. We can use it happily.

On the other hand, the oscillations of HeapInuse and GC time should be used to observe whether the upper layer services apply and release heap memory too frequently, and whether it is necessary to optimize the upper layer services, such as reducing heap memory and adding memory pools.

Ok, this article will be written here first. Recently, there are two memory cases of online actual business, which also use the above pprofPlus drawing analysis. I will write an article to share when I have time.

This article, by Yoko, respect the achievements of working people, reprinted please note: pengrl.com/p/20033/

This article was automatically published by ArtiPub