What do you know about Go GC optimization? It is more common to adjust the GC firing frequency by pacing the GC.

  • Set the GOGC

  • Set the debug. SetGCPercent ()

The principle and effect of both methods are the same. The default GOGC value is 100, which means that the heap triggered by the next GC is twice the size of the heap after the current GC.

As we all know, GO’s GC is mark-clean. When GC is triggered, variables are fully traversed for marking, and when the marking is over, the cleanup is performed, and the objects marked white are garbage collected. It’s important to note that the reclamation here is just the markup memory that can be returned to the operating system, not immediately, which is why you see RSS in the Go app remain so high. The entire Go program (STW) is suspended during the entire garbage collection process. The time taken for Go garbage collection depends mainly on the amount of time spent marking, and the cleanup process is very fast.

Disadvantages of setting up GOGC

1. The GOGC is imprecise in the way it sets the ratio

Setting the GOGC is basically a common way of tuning the Go GC. In most cases, we don’t need to adjust the GOGC. On the one hand, the programs that don’t involve memory intensive are too memory sensitive, and on the other hand, the GOGC is not accurate in setting the ratio. It is difficult to precisely control the threshold at which we want to trigger the garbage collection.

2. The GOGC setting is too small

The GOGC setting is so small that it triggers GC too frequently, resulting in too much ineffective CPU waste, and the performance of the program is particularly noticeable. For example, in the case of API interfaces, the result is periodic time changes in the interface. When you grab the CPU profile, most of the time is spent on GC processing.

As shown in the figure above, this is a Prometheus query operation and we see that most of the CPU is consumed in the GC operation. This is also the case in production environments, where the GOGC Settings are too small, resulting in excessive consumption on the GC.

3. For some programs, the memory footprint is low and GC is easily triggered

Businesses that are sensitive to the elapsed time of the API interface may also experience periodic elapsed time fluctuations of the interface if the GOGC is set to default. Why is that?

Because the interface itself has a low memory footprint and uses less memory after each GC, this threshold can easily be triggered if the GOGC is set at a GC pace that is twice the heap after the last GC, making it easy for the interface to incur additional consumption due to GC triggering.

4. GOGC Settings are very large and sometimes trigger OOM

So how do you adjust? Should the GOGC be as big as possible? This does reduce the frequency of GC firing, but this value needs to be set to a very large value to be effective. The problem with this is that the GOGC Settings are too large, and if these interfaces suddenly receive a large amount of traffic, the failure to trigger GC for a long time may result in OOM.

As a result, the GOGC is not very friendly for some scenarios, but is there any way to accurately control memory so that it can accurately control GC at multiples of 10G?

GO memory ballast

And that’s where Go Ballast comes in. What is Go Ballast? It is simple to initialize a very large slice throughout the Go application lifecycle.

func main() {
  ballast := make([]byte, 10*1024*1024*1024) // 10G 
  
  // do something
  
  runtime.KeepAlive(ballast)
}

Copy the code

This code initializes ballast and uses runtime.KeepAlive to ensure that GC does not recycle ballast.

By taking advantage of this feature, the GC can only be triggered at a time of 10G, which allows precise control of the trigger timing of the GO GC.

One question you might have here is, isn’t it 10 gigabytes of physical memory to initialize a 10 gigabyte array? The answer is no.

package main

import (
    "runtime"
    "math"
    "time"
)

func main() {
    ballast := make([]byte, 10*1024*1024*1024)

    <-time.After(time.Duration(math.MaxInt64))
    runtime.KeepAlive(ballast)
}

Copy the code
$ ps -eo pmem,comm,pid,maj_flt,min_flt,rss,vsz --sort -rss | numfmt --header --to=iec --field 5 | numfmt --header - from - unit = 1024 - to 6 | = iec field, the column - t | egrep "[t] est | [P] I" % MEM COMMAND PID MAJFL MINFL RSS VSZ 0.1 test 12859 0 1.6k 344M 11530184Copy the code

This result was verified in CentOS Linux Release 7.9, and we saw that the actual RSS physical memory usage was only 344M, but VSZ virtual memory usage was 10GB.

By extension, when we suspect that the elapsed time of our interface is due to frequent GC firing, how do we know for sure? The first thing you’ll want to do is periodically grab pPROF for analysis, which is fine, but too cumbersome. You can actually draw the graph based on the GC trigger time, which can be obtained using LastGC of Runtime.memStats.

Production environment verification

  • GOGC = 30 before green line adjustment

  • Yellow line adjusted GOGC default value, ballast = 50G

In this chart, the performance of ballast is obviously favored under the same flow pressure

conclusion

This article simply describes the application of Go Ballast, and Go Ballast is the official approved plan. For details, please refer to Issue 23044[1]. Many open source programs, such as TiDB [2] and Cortex [3], have implemented Go Ballast. If your program suffers from GOGC problems or is periodically time-consuming, try Go Ballast.

I highly recommend you to read Twitch. TV [4], which will give you a better understanding of GOGC and the application of Ballast.

The resources

[1]

Issue 23044: github.com/golang/go/i…

[2]

Tidb: github.com/pingcap/tid…

[3]

Architecture: github.com/cortexproje…

[4]

Twitch TV this article: blog. Twitch. TV/en / 2019/04 /…