When it comes to letting Go programs monitor the resource usage of their own processes, let’s talk about the metrics that need to be monitored first. Generally speaking, the most common metrics to talk about a process are memory usage, CPU usage, and the number of threads created. Since the Go language maintains its own Goroutine on top of the thread, the resource metric for the Go process also needs to add the number of goroutines created.

And because many services are now deployed in Kubernetes cluster, a Go process is often a Pod, but the container resources are shared with the host, only in the creation of the specified upper limit of the use of its resources, so in the acquisition of CPU and Memory these information also need to be discussed separately.

How do I use Go to get metrics for a process

We will first discuss how to obtain these metrics in normal host and virtual machine situations, which will be discussed in the next section.

The resource usage of the Go process can be obtained by using the Gopstuil library, which shields the differences between various systems and helps us easily obtain various system and hardware information. Gopsutil divides different functions into different sub-packages. The main modules it provides are:

  • cpu: System CPU related modules;
  • disk: System disk-related modules;
  • docker: Docker-related modules;
  • mem: Memory related modules;
  • net: Network related;
  • process: process-related module;
  • winservices: Windows service-related modules.

We only use its Process subpackage here to get process-related information.

Statement: the process module need to import “github.com/shirou/gopsutil/process” is introduced into the project, after the demo code can use OS omit the import relevant information of the unity of module and error handling, the instructions in advance.

Creating a process object

The process module’s NewProcess returns a process object with the specified PID. The method checks for the existence of the PID and returns an error if it does not exist. We can get various information about the process through other methods defined on the Process object.

p, _ := process.NewProcess(int32(os.Getpid()))
Copy the code

CPU usage of the process

The CPU usage of a process is calculated by calculating the change in CPU usage of the process within a specified period of time

cpuPercent, err := p.Percent(time.Second)
Copy the code

The above returns the percentage of total CPU time, but if you want to visualize the percentage, you can calculate the percentage of individual cores.

cp := cpuPercent / float64(runtime.NumCPU())
Copy the code

Memory usage, number of threads, and number of Goroutines

These three indicators are too easy to get and let’s put them together

// Get the memory usage of the process
mp, _ := p.MemoryPercent()
// The number of threads created
threadCount := pprof.Lookup("threadcreate").Count()
/ / number of Goroutine
gNum := runtime.NumGoroutine()
Copy the code

The preceding method of obtaining the process resource ratio is accurate only in vm and physical machine environments. Linux containers like Docker rely on Linux Namespace and Cgroups technology to achieve process isolation and resource restrictions, is not.

At present, many companies deploy K8s cluster, so if the resource usage of Go process is obtained from Docker, it needs to calculate accurately according to the resource upper limit allocated by Cgroups to the container.

Obtain process indicators in a container environment

In Linux, the operating interface of Cgroups exposed to users is the file system, which is organized in the way of files and directories under the /sys/fs/cgroup path of the operating system. Under /sys/fs/cgroup, there are many subdirectories such as Cpuset, CPU and memory. Each subdirectory represents the type of resource on the system that can currently be restricted by Cgroups.

To monitor the Go process memory and CPU metrics, we need only know cpu.cfs_period_us, cpu.cfs_quota_us, and memory.limit_in_bytes. The first two parameters must be used together to limit the CPU time that a process can only be allocated to cfs_quota within a cfs_period. The value can be simply interpreted as the number of cores that a container can use = cfs_quota/cfs_period.

So the way to get the CPU percentage of the Go process in the container needs to be adjusted a little bit, using the formula we gave above to calculate the maximum number of cores the container can use.

cpuPeriod, err := readUint("/sys/fs/cgroup/cpu/cpu.cfs_period_us")

cpuQuota, err := readUint("/sys/fs/cgroup/cpu/cpu.cfs_quota_us")

cpuNum := float64(cpuQuota) / float64(cpuPeriod)
Copy the code

Then divide the percentage of all CPU time occupied by p. Cent by the number of cores calculated to calculate the percentage of CPU occupied by the Go process in the container.

cpuPercent, err := p.Percent(time.Second)
// cp := cpuPercent / float64(runtime.NumCPU())
/ / for adjustment
cp := cpuPercent / cpuNum
Copy the code

Limit_in_bytes specifies the maximum amount of memory that the Go process can use in the container

memLimit, err := readUint("/sys/fs/cgroup/memory/memory.limit_in_bytes")
memInfo, err := p.MemoryInfo
mp := memInfo.RSS * 100 / memLimit
Copy the code

The RSS in the process memory information above is called resident memory, which is the amount of memory allocated to the process in RAM and allowed to be accessed by the process. The cGroups implementation provides containerd with a readUint to read container resources.

func readUint(path string) (uint64, error) {
	v, err := ioutil.ReadFile(path)
	iferr ! =nil {
		return 0, err
	}
	return parseUint(strings.TrimSpace(string(v)), 10.64)}func parseUint(s string, base, bitSize int) (uint64, error) {
	v, err := strconv.ParseUint(s, base, bitSize)
	iferr ! =nil {
		intValue, intErr := strconv.ParseInt(s, base, bitSize)
		// 1. Handle negative values greater than MinInt64 (and)
		// 2. Handle negative values lesser than MinInt64
		if intErr == nil && intValue < 0 {
			return 0.nil
		} else ifintErr ! =nil &&
			intErr.(*strconv.NumError).Err == strconv.ErrRange &&
			intValue < 0 {
			return 0.nil
		}
		return 0, err
	}
	return v, nil
}
Copy the code

I’ll link to their source code in the references below.

conclusion

On the complete source code of this article and some process monitoring related valuable information has been income to my “Go development Reference book”, there is a need to click on the link to get read, if you want to try in the container environment you need to start a Docker or Kubernetes cluster to test.

If you want to start K8s, amway down my Kubernetes study notes, package will not be ~!

Why, you might ask, is it useful to let Go monitor itself? That must be able to do some service governance things based on this, the specific application scenarios will be shared later, interested can pay attention to a wave.

Refer to the link

  • Contianerd utils: github.com/containerd/…
  • What is RSS: stackoverflow.com/questions/7…