Node.js is prone to memory overflow when it performs intensive operations or operates on large arrays and objects. This is because the operating environment of Node.js relies on the V8 engine. If you frequently perform operations such as large data operations, you need to be aware of the limitations of the Node.js operating environment. This article covers

  • Memory overflow problem

  • Why do I run out of memory

  • 2.1 V8 Memory Allocation Mechanism

  • 2.2 Causes of Memory Overflow

3. How to solve the memory overflow problem

Example 1: When we need to batch process some data (such as updating user information), we may need a large for or while loop to complete the update of all data, such as:

for (var i = 0; i < 10000000; i++) {

((i) => {

var site = {};

site.name = ‘koala’;

Site. domain = ‘programmer growth refers to north ‘;

// This is a save or update operation

setTimeout(()=>{

console.log(i, site);

}, 0)

})(i)

}

Large amount of operation Data Example 2: The object needs to be frequently created or destroyed, or the operation object itself is large, for example:

var sites = [];

for (var x=0; x<5000; x++){

var site=[];

for (var y=0; y<5000; y++){

Site = [y, ‘koala’, ‘koala’];

sites.push(site);

}

}

Both types of operations will have errors like the following:

<— Last few GCs —>

…

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed – process out of memory

Abort trap: 6

As we all know, V8 is the JavaScript engine that Google uses in Chrome. In a browser environment, calculations generally don’t require much memory. V8 allocates about 700MB of running memory per process on a 32-bit system and 1.4GB on a 64-bit system. 2.2 Causes of Memory Overflow Node.js programs can overflow memory for three reasons: 1. V8 allocates a small amount of memory; 2. Improper use by programmers. In example 1, the amount of memory required for each operation is small, but V8 memory cannot be freed in time due to the for loop. As the program runs more and more time, memory usage will increase, and eventually lead to memory overflow. In example 2, it is possible that the created object itself does not exceed the memory limit. But in addition to the objects themselves: creating objects, referencing objects, node.js programs themselves, etc., all require memory space, which can easily lead to memory overflow. During node.js application development, it is essential to understand V8 memory allocation and JavaScript language limitations. We should balance the advantages and disadvantages in the application, considering the memory and the efficiency of the program. Here are some suggestions to prevent memory overflow: 1. Use async/await to prevent event accumulation and change the code execution order to synchronize with await operation. This gives V8 a chance to reclaim memory, effectively addressing overflow caused by excessive event accumulation. We can use the await method to handle:

async function dbFuc() {

for (let i = 0; i < 10000000; i++) {

var site = {};

site.name = ‘koala’;

Site. domain = ‘programmer growth refers to north ‘;

// This is a save or update operation

await console.log(i, site);

}

}

dbFuc();

V8 reclaims memory once for each cycle, so it doesn’t overflow again. However, this will inevitably lead to the reduction of operating efficiency, and should be balanced between the speed and safety, control the safe number of cycles. Note: In actual development, the above method solves memory overflow, but still causes process blocking. You can start a process/thread to solve the blocking problem. Node.js provides an application runtime parameter, max-old-space-size, which can be used to specify the amount of memory used by V8 to avoid running out of memory to some extent. For example, we can specify 4G memory when running the example 2 program:

node –max-old-space-size=4096 app

3. Memory used by node.js programs that use non-V8 memory falls into two categories:

  • V8 memory: JavaScript built-in objects such as arrays, strings, etc., using “V8 memory” at runtime
  • System memory: Buffer is a Node.js extension that uses the underlying system memory and does not take up V8 memory space. The related file system FS and Stream operations do not take up V8 memory. Fs and Stream have a look at my two articles
  • Node.js advanced fs file module learning
  • Node.js is used for backend development, stream is used for backend development.

When the program allows, data should be stored in buffers rather than converted to JS objects such as strings to avoid excessive use of V8 memory.