This article is collected in GitHub mountain moon blog: shfshanyue/blog, including my problems encountered in practical work, thinking about business and learning in the direction of the full stack

  • Front-end engineering series
  • Node advanced series

To start, ask a question:

Do you know how much memory the Node service in your production environment normally takes up? Or what order of magnitude?

When interviewing Node candidates, this question is enough to weed out half of the self-proclaimed Node savvy, but when it’s not answered, I tend to add another question so as not to miss candidates with excellent wireless experience:

How do I know how much memory a process consumes?

When Node is used as the server language in production environment, excessive concurrency or code problems causing OOM (out of memory) or CPU full load are common problems in the server. In this case, it is easy to find problems by monitoring THE CPU and memory, combined with logs and Release.

This chapter describes how to monitor memory changes in the local environment and production environment

A Node application instance

So, how do you dynamically monitor memory changes in a Node process?

The following is an example of a Node Server that has a memory leak problem and is a condensed version of a problem that Yamatsuki has been locating in production for a long time.

The memory leak caused the memory in a single container to jump from 400 MB to 700 MB, and occasionally OOM occurred under the 800 MB container resource limit, resulting in a restart. The problem was not located at that time (the problem was discovered too late, the timing data from half a month ago had been swallowed, so the Release was not located), so the resource limit was increased to 1000M. CTX. Request mounted a large field in the database

const Koa = require('koa')
const app = new Koa()

function getData () {
  return Array.from(Array(1000)).map(x= > 10086)
}

app.use(async (ctx, next) => {
  ctx.data = getData()
  await next()
})

app.use(ctx= > {
  ctx.body = 'hello, world'
})

app.listen(3200, () = >console.log('Port: 3200'))
Copy the code

Process memory monitoring

Some issues need to be killed locally and in the test environment to avoid a larger impact in the production environment. Knowing how to monitor memory locally is critical.

Pidstat is the sysstat series of Linux performance debugging tool package, should use it to debug Linux performance problems, including memory, network, IO, CPU etc.

This is not just a trial withnodeAnd applies to all processes, includingpython.javaAs well asgo

# -r: indicates the output memory indicator
# -p: Specifies the PID
# 1: Output once every second
# 100: Output 100 times
$ pidstat -r -p pid 1 100
Copy the code

Before using pidstat, you need to find the PROCESS PID

How to find the PID of a Node process

You can find the PROCESS PID in Node by using process.pid

> process.pid
16425
Copy the code

Although pids can be found by writing code, they are intrusive and not very practical. So how do you find pids by non-invasive means? There are two ways to do this

  1. By combining redundant parameterspsPositioning process
  2. By port numberlsofPositioning process
$ node index.js shanyue

# The first method: quickly locate the PID by redundant parameters
$ ps -ef | grep shanyue
root     31796 23839  1 16:38 pts/5    00:00:00 node index.js shanyue

# Second method: Locate pid by port number
lsof -i:3200
COMMAND   PID USER   FD   TYPE    DEVICE SIZE/OFF NODE NAME
node    31796 root   20u  IPv6 235987334      0t0  TCP *:tick-port (LISTEN)
Copy the code

Use pidstat to monitor memory

As you can see from the above code, the PID of node service is 31796. In order to observe the dynamic change of memory, apply another stress test

$ ab -c 10000 -n 1000000 http://localhost:3200/
Copy the code
# -r: indicates the output memory indicator
# -p: Specifies the PID
# 1: Output once every second
# 100: Output 100 times$pidstat -r -p 31796 1 100 Linux 3.10.0-957.21.3.el7.x86_64 (Shuifeng) 2020 07月02日 _x86_64_ (2 CPU) UID PID minflt/s Majflt /s VSZ RSS %MEM Command 19:20:39 sec. 0 11401 0.00 0.00 566768 19800 0.12 node 19:20:40 sec. 0 11401 0.00 0.00 566768 19800 0.12 node 19:20:42 0.00 579024 37792 0.23 node 19:20:42 0.00 600716 59988 0.37 Node 19 hours 20:43 0 11401 5417.82 0.00 611420 70900 0.44 node 19 hours 20:44 0 11401 3901.00 0.00 627292 85928 0.53 node 0 11401 2390.00 0.00 623964 83696 0.51 node 19.20min 47.0 11401 1764.00 0.00 625500 85204 0.52 nodeCopy the code

The implications for output indicators are as follows

  • RSS: Resident Set Size, resident memory set, can be understood as memory, this is the memory metric we need to monitor
  • VSZ: virtual sizeVirtual memory

As you can see from the output, the memory went from 19M to 85M when the stress test was applied.

Use top to monitor memory

Pidstat is a Linux performance tool under SYSstat, but how do you locate memory changes on the MAC?

In this case, you can use top/htop

$ htop -p 31796
Copy the code

Monitor memory in the production environment

Since most production environments are deployed in K8S, the memory monitoring of an application in the production environment is essentially the memory monitoring of K8S for a Workload/Deployment. The data flow of the metric for memory monitoring is as follows:

k8s -> metric server -> prometheus -> grafana

The architecture diagram is as follows:

The above pictures are taken from the following article

  • Kubernetes Monitoring with Prometheus
  • Kubernetes monitoring architecture

Finally, a real-time image of memory monitoring for an application can be collected in Grafana:

Because this part of the design content is too much, I will be introduced in the following chapters

This applies not only to Node services, but to everything on K8Sworkload

conclusion

This chapter describes how to monitor the local and production environments of Node services

  1. The local usehtop/toporpidstatMonitor process memory
  2. Use in production environmentk8s/metric-server/prometheus/grafanaMonitors node application memory

When a memory leak is detected in a service, how can the problem be resolved? So we’ll talk about that in the next article

  1. How does the production environment monitor memory for the entire application
  2. How can I quickly locate an OOM in the production environment
  3. Real production environment several OOM sample locations

This article is collected in GitHub mountain moon blog: shfshanyue/blog, including my problems encountered in practical work, thinking about business and learning in the direction of the full stack

Pay attention to my

Scan code to add my wechat, remarks into the group, join the advanced front-end advanced group

Add me wechat to get you into the interview communication group

Welcome to pay attention to the public number [full stack growth road], regular push Node original and full stack growth article

Add me wechat to get you into the interview communication group