Concepts and understandings?

As the node.js API, the Buffer class is responsible for manipulating binary data and is a typical combination of js and c++ modules. Commonly known as buffer zone.

Buffer is similar to an array in JS, but its elements are two-digit hexadecimal values ranging from 0 to 255 (8-bit unsigned integer Uint8Array).[FF is the largest two-digit hexadecimal value, i.e. 255].

We’ve already mentioned that data flow is data moving from one point to another, but how exactly do they move?

Usually data is moved in order to process or read data, and decisions are made based on the data. In this process, the data may need to reach a minimum or a maximum amount to be processed. Therefore, if the data is arriving faster than the process can consume it, then the extra data needs to wait somewhere to be processed. On the other hand, if the process consumes data faster than it arrives, then the few data that arrive early need to wait for a certain amount of data to arrive before being sent out for processing.

That “waiting area” is Buffer! It is a small physical location in a computer, usually in RAM, where data is temporarily collected, waited for, and finally sent out for processing in a stream.

We can think of the entire stream and buffer process as a bus station. At a bus stop, the bus does not leave until there are a certain number of passengers or a specific time. In addition, passengers may arrive at different times and at different speeds. Neither the passenger nor the bus station can control the arrival time of passengers. Passengers who arrive early have to wait for the bus to leave. By the time some passengers arrive, they are already full or the bus has left and need to wait for the next bus.

No matter what the situation, there is always a waiting place. This is node.js’s Buffer!

Js does not control the speed or time of arrival of data, nor does it control the speed of flow. It can only decide when to send data. If the time is not up, Node.js will put them in a buffer, a small place in RAM, until they can be sent out for processing.

A typical example is when you watch a Youku video and you can see Buffer working. If your connection is fast enough, the stream will be fast enough to immediately fill the Buffer and send it out for processing, then fill another Buffer, then send it out, then send another, and then another, until the stream is complete.

Instead, after processing the first set of data that arrives, the video gets stuck, which means the program is collecting more data or waiting for more data to arrive. When the buffer is full and processed, the player continues to play the video. While playing, more data will continue to arrive and wait in the buffer.

Buffer used?

1. Convert bits, bytes, KB, MB, and GB

Since buffer always involves byte size conversion, memory application, declare size conversion relationship.

    1 Byte = 8Bits (i.e. 1B=8b)1 KB = 1024 Bytes
    1 MB = 1024 KB
    1 GB = 1024 MB
Copy the code
  • Bit means “Bit” or “Bit”. It is the basis of computer operations and belongs to the category of binary.
  • Byte means “Byte” and is the basic unit of calculation for the size of a computer file.

The two have different applications. Bit is usually used as the unit of data transmission, because the transmission of the physical layer, data link layer is transparent to the user, and this communication transmission is based on binary transmission. At the application layer, bytes are used to indicate the size of the file, which is the size of the data visible to the user. For example, a character is 1byte, if it is a Chinese character, it is 2byte

A brief answer to the actual combat:

/** * The unit is byte. The format is MB
const format = function (bytes) {
  return (bytes / 1024 / 1024).toFixed(2) + " MB";
};

const print = function () {
   const memoryUsage = process.memoryUsage();
   console.log(
    JSON.stringify({
      rss: format(memoryUsage.rss),
      heapTotal: format(memoryUsage.heapTotal),
      heapUsed: format(memoryUsage.heapUsed),
      external: format(memoryUsage.external),
    })
  );
};
Copy the code

Output information is: {” RSS “:” 93.06 MB “, “heapTotal” : “4.26 MB”, “heapUsed” : “2.63 MB”, “external” : “0.90 MB”}. For more information on memory, see another blog post: V8 Memory-Node Memory-Whether I’m big or not

2. Create and operate the buffer

Node.js automatically creates buffers when processing streams, or we can create our own buffers using the API provided by Nodejs. Depending on your needs, there are several different ways to create buffers.

The buffer provides buffer. from, buffer. alloc, buffer. allocUnsafe, and buffer. allocUnsafeSlow methods to apply for memory.

// Create a new buffer that can hold 10 bytes
const buf1 = Buffer.alloc(10);

const buf2 = Buffer.from("hello buffer")

// The second argument specifies the encoding format
const buf3 = Buffer.from('10'); // <Buffer 31 30>
const buf2 = Buffer.from('10'.'utf8');  // <Buffer 0a>
Copy the code

Once the buffer has been created, you can begin to interact with it.

// Check the structure of the buffer
buf1.toJSON()
// { type: 'Buffer', data: [ 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 ] }

buf2.toJSON()
//{ type: 'Buffer',data: [ 104, 101, 108, 108, 111, 32, 98, 117, 102, 102, 101, 114 ]}

buf1.length / / 10
buf2.length / / 12

/ / write operations
buf1.write("Buffer really rocks!")

// decode
buf1.toString() // 'Buffer rea'
// Because buf1 was created with only 10 bytes allocated. Any excess will not be stored.
Copy the code

Convert a string to a buffer.

const buf = Buffer.from('Good future JS'.'utf8');

console.log(buf); // <Buffer e5 a5 bd e6 9c aa e6 9d a5 4a 53>
console.log(buf.length); // 11, the last two bytes are "JS", the front three are a single length of 3

console.log(buf.toString('utf8')); // Good future JS
Copy the code

3. The API list

  • Buffer.alloc(size[, fill[, encoding]])It is used to request the specified size of memory space

Size, specify the length of the buffer, but cannot exceed buffer.kmaxLength. If it is not a number, an error is reported. Fill, which specifies the value to initialize buffer, defaults to 0. Encoding, specifies the encoding of fill if fill is a string, defaults to ‘utf8’.

  • Buffer.allocUnsafe(size)The size parameter specifiesbufferThis method returns an uninitialized buffer, and therefore may retain sensitive data, resulting in information leakage. Recommendedbuffer.fill(0)Function initializationbuffer.
const buf = Buffer.allocUnsafe(10);
Prints: (contents may vary): 
      
console.log(buf);

buf.fill(0);

// Prints: <Buffer 00 00 00 00 00 00 00 00 00 00>
console.log(buf);

Copy the code
  • Buffer.allocUnsafeSlow(size)Memory allocation directly through c++; Old value padding is not done. Except these two points have to do withBuffer.allocUnsafe(size)Other features of the.
// allocate memory directly from the c++ module level
const buf4 = Buffer.allocUnsafeSlow(10);
console.log(buf4);  //
Copy the code
  • Buffer.from(array)Receiving an array as an argument converts the values in the array to hexadecimal.
const bufArr = Buffer.from([1.2.3]);
console.log(bufArr);  // <Buffer 01 02 03>
Copy the code
  • Buffer.from(arrayBuffer[, byteOffset[, length]])

Buffer and memory

Buffer objects are allocated not in V8’s heap, but in Node’s C++ level. The requested memory is then managed at the JavaScript level.

In order to efficiently manage memory, Node adopts slab dynamic memory management mechanism. “Slab” refers to an allocated area of memory. In other words, passing a specified size in a Buffer applies to “slab”. Then slab has the following three states:

  • Empty: is created for the first time and has not yet been allocated data
  • Partial: The space is partially allocated, but not fully allocated
  • Full: Fully assigned

Node will classify Buffer objects according to the current requested memory size. If the requested memory size is less than 4k (for example, the first request), it will be stored in the initialized slab unit, that is, the so-called 8K pool for reading various data. If the memory size is still less than 4k and the space in the first initialized 8K pool is sufficient, the memory will be stored in the first initialized 8K pool.

For example, if the remaining space of the initialized 8K pool is 2k, and another memory space larger than 2K and smaller than 4K is applied, a new slab unit space will be applied, and the remaining 2k memory of the last initialized slab unit will be wasted and can no longer be used.

If the allocated memory is greater than 4k, then instead of going to the 8k pool, node directly applies for an exclusive slab cell space through the C++ layer.

One final note: In either case, the resulting Buffer is JavaScript level, which means it can be managed by V8’s garbage collection mechanism. In fact, Node does a lot of work, the most important is to combine JS and C++.

Here are the apis and conditions for using the 8K pool:

Buffer.allocunsafe Indicates the size of the data passed in (0 < size < 4 * 1024) buffer.concat Indicates the size of the data passed in (0 < size < 4 * 1024) buffer.from does not have a single parameter ArrayBuffer instance and incoming data size (0 < size < 4 * 1024)

Conclusion:

  • V12.x /lib/buffer.js#L156 (); v12.x/lib/buffer.js#L156

  • There are small Buffer objects and large Buffer objects according to the requested memory size. In the case of a small Buffer (less than 4KB), determine whether the remaining space of this slab is sufficient. If enough is available to use the remaining space allocation, the offset will increase. If not, then createPool is called to create a new slab space to allocate.

For large buffers (larger than 4KB), createUnsafeBuffer(size) is created. The reason to determine the difference between large objects or small objects, just hope that small objects do not every time the application to the system to apply for memory call.

Whether it is a small Buffer or a large Buffer, memory allocation is done at the C++ level and memory management is done at the JavaScript level. In the end, the Buffer itself can be reclaimed by V8’s garbage collection flag. Those parts of out-of-heap memory can only be given to C++.

Application scenarios

Files are associated with network I/O and are associated with streams, except that streams wrap things around them without the developer having to manually create buffers. The Chrome V8 engine limits the amount of memory that can be used (1.4GB for 64-bit and 1.0GB for 32-bit) and requires streaming.

const fs = require("fs");
fs.readFile("./mv.jpg".(err, res) = > {
  if (err) throw err;
  console.log("= = = =", res.length, Buffer.isBuffer(res), res); // buffer true
});
Copy the code

Refer to the article

  • www.freecodecamp.org/news/do-you…
  • www.cnblogs.com/copperhaze/…
  • Es6 Arraybuffer in-depth understanding and www.cnblogs.com/jixiaohua/p…