Node.js is prone to memory overflow when it performs intensive operations or operates on large arrays and objects. This is because the operating environment of Node.js relies on the V8 engine. If you frequently perform operations such as large data operations, you need to be aware of the limitations of the Node.js operating environment.

This article covers

  1. Memory overflow problem
  2. Why do I run out of memory
    • 2.1 V8 Memory Allocation Mechanism
    • 2.2 Causes of Memory Overflow
  3. How to solve the memory overflow problem

Koala is dedicated to sharing the complete Node.js technology stack, from JavaScript to Node.js, to back-end database. Wish you become an excellent senior Node.js engineer. [Programmer growth refers to north] Author, Github blog open source project github.com/koala-codin…

1. Memory overflow

Here are two common types of memory overflow problems we encounter in Node.js applications:

Intensive operation

Example 1: When we need to batch some data (such as updating user information), we might need a large for or while loop to complete all data updates, such as:

for (var i = 0; i < 10000000; i++) {
    ((i) = > {
        var site = {};
        site.name = 'koala';
        site.domain = 'Programmer growth is north';
        // This is a save or update operation

        setTimeout(() = >{
            console.log(i, site);
        }, 0)
    })(i)
}
Copy the code

The operation requires a large amount of data

Example 2: The object needs to be created/destroyed frequently, or the object itself is large, as in:

var sites = [];
for (var x=0; x<5000; x++){var site=[];
    for (var y=0; y<5000; y++){ site = [y,'koala'.'Programmer growth is north']; sites.push(site); }}Copy the code

Both types of operations will have errors like the following:

<-- Last few GCs -->...... FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - process outof memory
Abort trap: 6
Copy the code

2. Why does the memory overflow

2.1 V8 Memory Allocation Mechanism

As we all know, V8 is the JavaScript engine that Google uses in Chrome. In a browser environment, calculations generally don’t require much memory. V8 allocates about 700MB of running memory per process on a 32-bit system and 1.4GB on a 64-bit system.

2.2 Causes of Memory Overflow

There are three reasons why node.js programs run out of memory:

    1. V8 itself allocates less memory
    1. The JavaScript language itself is limited
    1. Improper use by programmers.

In example 1, the amount of memory required for each operation is small, but V8 memory cannot be freed in time due to the for loop. As the program runs more and more time, memory usage will increase, and eventually lead to memory overflow.

In example 2, it is possible that the created object itself does not exceed the memory limit. But in addition to the objects themselves: creating objects, referencing objects, node.js programs themselves, etc., all require memory space, which can easily lead to memory overflow.

3. Resolve memory overflow

An understanding of V8 memory allocation and JavaScript language limitations is essential to node.js application development. We should balance the advantages and disadvantages in the application, considering the memory and the efficiency of the program. The following are some suggestions to prevent memory overflow:

3.1. Use async/await to prevent event accumulation and become synchronous operation

Await changes the code execution order to synchronous. This gives V8 a chance to reclaim memory, effectively addressing overflow caused by excessive event accumulation. We can use the await method to handle:

async function dbFuc() {
for (let i = 0; i < 10000000; i++) {
    var site = {};
    site.name = 'koala';
    site.domain = 'Programmer growth is north';
    // This is a save or update operation

    await  console.log(i, site);

    }
}
dbFuc();
Copy the code

V8 reclaims memory once for each cycle, so it doesn’t overflow again. However, this will inevitably lead to the reduction of operating efficiency, and should be balanced between the speed and safety, control the safe number of cycles. Note: In actual development, the above method can solve the memory overflow, but still cause the process blocking, can start a process/thread to solve the blocking problem (see my article “in-depth understanding node.js processes and threads”).

3.2. Increase V8 memory space

Node.js provides an application run parameter, max-old-space-size, which can be used to specify the amount of memory used by V8 to avoid running out of application memory to some extent. For example, we can specify the use of 4G memory when running the example 2 program: node –max-old-space-size=4096 app

3.3. Use non-V8 memory

The memory used by Node.js programs falls into two categories:

  • V8 memory: JavaScript built-in objects such as arrays, strings, etc., using “V8 memory” at runtime
  • System memory: Buffer is an extension of Node.js that uses the underlying system memory and does not occupy V8 memory space. Buffer-related file system FS and stream operations do not take up V8 memory.

(Note: FS and STREAM are two modules that I have covered in detail in the Node progression series, so I won’t repeat them here.)

When the program allows, data should be stored in buffers rather than converted to JS objects such as strings to avoid excessive use of V8 memory. (For buffer, see the article “Node Advancement – Exploring Buffer objects not stored in V8 heap Memory.”)

Node series original articles

Learn more about processes and threads in Node.js

To learn node. js, it is necessary to know stream first

Exports: module. Exports: module

Understand the Events module thoroughly

Node.js advanced fs file module learning

Pay attention to my

  • Welcome to add my wechat (COder_QI), pull you into the technology group, long-term exchange and learning…
  • Welcome to “programmer growth refers to the north”, a heart to help you grow the public number…