background

At present, containerization and microservices are a trend of server-side development. However, under this microservice architecture, we will encounter some difficulties in actual enterprise development: there is a natural contradiction between the more stable server-side API and the diverse and flexible user demands.

To more generally describe some actual development scenarios: Android, IOS, PC and M stations for the same nature of the interface requirements of the field is not consistent, resulting in front-end development and server development often because of the increase and decrease of the field generated a lot of communication overhead.

To solve this dilemma, some companies have added a LAYER of BFF between the traditional front end and back end to achieve the purpose of development and maintenance. Obviously, Node.js, which is familiar with the front end, is an ideal language for this BFF layer implementation.

However, this introduces some new problems (typically introducing a new problem to solve a problem), and the Node.js runtime is a black box for most developers compared to the traditional, more mature Java language. There is no corresponding ecological chain tool to ensure the stability of the BFF layer ———— for example, there is a memory leak on the line and the process is intermittently OOM, how should we deal with the location.

This article aims to expand and discuss the memory leak problem encountered in node.js development under this background.

Heap snapshot shallow probe

Taking a heap snapshot

To analyze and locate a memory leak, we first need to obtain the node.js process at the time of the leak and the reference relationship between each object on the heap, the file that saves each object on the heap and its reference relationship is the heap snapshot. The V8 engine provides an interface that makes it easy to get a heap snapshot in real time. Here are three different ways to get a heap snapshot.

heapdump

To install the heapdump module, run the following command:

npm install heapdumpCopy the code

This module needs to be introduced in code:

const heapdump = require('heapdump');Copy the code

The heapdump module provides two ways to take a snapshot of the current heap of a process. The first way is to use custom logic in your code (either timer set to fetch, or long connection switch hot start). Here is an example:

'use strict';
const heapdump = require('heapdump');
const path = require('path');
setInterval(function() {
    let filename = Date.now() + '.heapsnapshot';
    heapdump.writeSnapshot(path.join(__dirname, filename));
}, 30 * 1000);Copy the code

Here a heap snapshot is output to the current current every 30 seconds.

The second method is to start the node. js process with the heapdump module, and use the usR2 semaphore to trigger the heap snapshot:

Kill -usr2 < Node.js process PID>Copy the code

The advantage of this approach is that there is no need to build logic into the code, and you can only SSH to the server to get a heap snapshot through the semaphore when needed.

v8-profiler

To install the heapdump module, run the following command:

npm install v8-profilerCopy the code

V8-profiler provides the transform stream to output heap snapshot files, which can be better generated for some large heap snapshot files:

'use strict'; const v8Profiler = require('v8-profiler-node8'); const snapshot = v8Profiler.takeSnapshot(); Const transform = snapshot.export(); Transform. on('data', data => console.log(data)); // Delete transform.on('finish', snapshot.delete.bind(snapshot));Copy the code

In node.js v6.x, the v8-profiler can be downloaded directly into binary via Node-pre-gyp. No local compilation is required, and it is relatively friendly for some non-Mac classes.

Node.js performance platform

Node.js performance platform currently integrates heap snapshot acquisition into runtime, so as long as the application is connected to the platform, the heap snapshot can be obtained online without changing the business code for analysis:

Click the heap snapshot button to generate a heap snapshot. Click the file option on the left of the navigation bar to see the heap snapshot generated just now:

At this point, click dump to the cloud, you can download the analysis anytime, anywhere.

Heap snapshot content parsing

Field meaning

When you open the heap snapshot from the previous section with any document reader, you can see that its contents are essentially one large JSON:

{
    snapshot: {},
    nodes: [],
    edges: [],
    strings: []
}Copy the code

It is easy to guess between nodes and edges. Obviously, the nodes array stores the information of each node in the memory relation, while the array of edges stores the relation between each node in the memory relation.

The snapshot node contains only one meta node. If you expand the meta node, you can view the description of node and edge.

• Node_fields: array, the length of the array is the length of the nodes array.
• Meta-node_types: an array of elements representing each bit of a node. The first of the six bits represents the node type, and the node type is also in a limited array.
• Meta. Edge_fields: An array whose length is an edge is represented by the corresponding number of edges in the array of edges.
• Meta-node_types: array, where the elements represent each bit of an edge. The first of the three bits represents the edge type, and the edge type is also in a limited array.

Finally, there is the Strings array, which has a simpler meaning. It actually holds the names of Node and Edge inside.

The overall diagram is as follows:

Nodes and edges

Through the above information, we can obtain the description of each node and each edge required by the memory diagram, but the relationship between nodes and edges is still missing to complete the diagram.

Note that the meta. Node_fields that describe node information has an item called edge_count. This clearly describes the number of edges of the node, and edges of the array are arranged in order. We can construct a diagram as follows;

Edge_fields contains a to_node entry, which refers to the node to which the edge points.

Locating memory Leaks

Based on the previous section, we can get a memory diagram similar to the following:

We can consider that node 5 is the source of a memory leak, accumulating a large amount of memory that has not been released properly. At this point, if we release its parent node 3, the path 1->2->4->5 can still reach 5 from the root node, that is, releasing node 3 alone cannot disconnect the reference of node 5. Similarly, node 4 can be obtained.

In this example, node 5 can only be freed if we disconnect the reference to node 2. In other words, all paths from root node 1 to node 5 go through node 2, which means node 2 is the direct dominator of node 5. This is where the concept of a dominance tree is introduced, which is very helpful for analyzing memory leaks.

We convert the memory reference diagram in the figure above into a domination tree, as shown below:

Retained size is calculated from the dominant leaf node 8 up to the root node. Retained size of each node = self size of the node + retained size of its children. Finally, we can see where the memory is accumulated. We can assume that these memory accumulation nodes are likely to be the ones that are not being collected properly and thus causing memory leaks.

After obtaining these suspected leak points, the memory diagram is restored to locate the corresponding code logic fragment, and the final memory leak confirmation must go back to these code logic fragment to see if they actually produce some unexpected memory that cannot be freed.

In actual combat

The following are two real online memory leakage cases. After the heap snapshot is obtained online, the suspected leakage point of memory accumulation is found through the above positioning analysis process, and the leakage code is finally located after restoration. Interested students can take a look at the details:

• Event listener leaks
• Module hot update leakage

Reference Documents:

• heapdump
• v8-profiler
• devtoolx
• Troubleshooting – Memory leak