This part of node.js will be unfamiliar to those who are transiting from the front end to node.js, where simple string operations already meet basic business requirements, and buffers and streams can sometimes seem mysterious. Back on the server side, if you want to be more than just a node.js developer, you should take a look at Buffer to uncover this mystery and take your understanding of Node.js to a new level.

The interview guide

  • What is the difference between a Buffer and a Cache?

Buffer first

Prior to the introduction of TypedArray, the JavaScript language had no mechanism for reading or manipulating binary data streams. The Buffer class was introduced as part of the Node.js API to interact with octet streams in TCP streams, file system operations, and other contexts. Node.js can be used to process or interact with binary stream data.

The Buffer is used to read or manipulate binary data streams without requiring as part of the Node.js API. It is used for scenarios that require large amounts of binary data, such as network protocols, databases, images, and file I/O. The size of the Buffer is determined when it is created and cannot be adjusted. In memory allocation this Buffer is provided by the C++ layer rather than V8, which will be explained later.

I don’t know if you think it’s easy here? But what are binary, Stream, and Buffer? Here’s an attempt at a brief introduction.

What is binary data?

When we think of binary, we might think of a code command like 010101, as shown below:

Binary data, as shown in the figure above, uses numbers 0 and 1. In order to store or display some data, the computer needs to convert the data to binary representation. For example, if I want to store the number 66, the computer will first convert the number 66 into binary 01000010 representation. I remember that I first came into contact with this in the C language course in university, and the conversion formula is as follows:

128 64 32 16 8 4 2 1
0 1 0 0 0 0 1 0

In the example above with numbers, we know that numbers are just one of the data types. Other data types include strings, images, files, and so on. For example, an English M operation will be converted to binary representation in JavaScript when the corresponding ASCII code is retrieved by ‘M’.charcodeat ().

What is a Stream?

Stream is an abstraction of input and output devices, which can be files, networks, memory, etc.

Streams are directional. When an application reads data from a data source, it opens an input stream. The data source can be a file, a network, etc. For example, we read data from an A.txt file. Instead, when our program needs to write data to a specified data source (file, network, etc.), it opens an output stream. When there are large file operations, we need a Stream to pipe data out bit by bit.

For example

We now have a big pot of water that needs to be poured over a vegetable patch. If we pour all the water in the pot into the vegetable patch, how much force (the strength here is like the performance of the hardware in a computer) is needed to move it. If we take a hose and pour water into our vegetable patch bit by bit, this time with less effort can be done.

What is a Stream? What is the relationship between Stream and Buffer? See below for more information about the Stream itself. Please follow the public account “Nodejs Stack” and we will cover it separately later.

What is a Buffer?

As we have seen in the Stream above, data flows from one end to the other. How does it flow?

Typically, data is moved in order to process or read it and make decisions based on it. Over time, each process has a minimum or maximum amount of data. If the data arrives faster than the process consumes, a small number of early arrivals are left in the waiting area waiting to be processed. Conversely, if the data arrives more slowly than the data consumed by the process, the data that arrived earlier will have to wait for a certain amount of data to arrive before it can be processed.

The waiting area here is the Buffer, which is a small physical unit in the computer, usually located in the computer’s RAM. These concepts can be difficult to understand, so don’t worry about illustrating them further with an example.

Bus stop examples of rides

Taking a bus stop as an example, usually the bus will come every dozens of minutes, before arriving at this time even if the passengers are full, the bus will not leave in advance, passengers who arrive early need to wait at the station first. Suppose there are too many passengers arriving, and those who arrive later have to wait at the bus stop for the next one.

In the above example of the bus stop waiting area, corresponding to our Node. Js in the Buffer (Buffer), the speed of the other passengers arrived, we can’t control, and only when to start, we can control, corresponding to our program is we can’t control the time of arrival of the data stream, you can do is to decide when to send data.

Basic use of Buffer

Now that we know some of the concepts of Buffer, let’s take a look at some of the basic uses of Buffer. Instead of listing all the API uses, we will only list some of the common ones, and refer to Node.js Chinese for more details.

Create a Buffer

In versions of Node.js prior to 6.0.0, Buffer instances were created using the Buffer constructor, which allocates the returned Buffer new Buffer() in different ways depending on the arguments provided.

It can now be created using buffer.from (), buffer.alloc (), and buffer.allocunsafe ()

Buffer.from()

const b1 = Buffer.from('10');
const b2 = Buffer.from('10'.'utf8');
const b3 = Buffer.from([10]);
const b4 = Buffer.from(b3);

console.log(b1, b2, b3, b4); // <Buffer 31 30> <Buffer 31 30> <Buffer 0a> <Buffer 0a>
Copy the code

Buffer.alloc

Returns an initialized Buffer that guarantees that the newly created Buffer will never contain old data.

const bAlloc1 = Buffer.alloc(10); // Create a buffer of 10 bytes

console.log(bAlloc1); // <Buffer 00 00 00 00 00 00 00 00 00 00>
Copy the code

Buffer.allocUnsafe

Create a new uninitialized Buffer of size bytes. Since buffers are uninitialized, allocated memory segments may contain sensitive old data. If the Buffer is readable, it may leak its old data. This is not safe and should be used with caution.

const bAllocUnsafe1 = Buffer.allocUnsafe(10);

console.log(bAllocUnsafe1); // <Buffer 49 ae c9 cd 49 1d 00 00 11 4f>
Copy the code

Buffer character encoding

The conversion between Buffer instances and JavaScript strings can be realized by using character encodings, which are currently supported as follows:

  • ‘ASCII’ – applies to 7-bit ASCII data only. This encoding is fast and, if set, strips high bits.
  • ‘UTf8’ – Multi-byte encoded Unicode character. Many web pages and other document formats use UTF-8.
  • ‘UTF16le’ – a 2 – or 4-byte, little-endian encoded Unicode character. Proxy pairs (U+10000 to U+10FFFF) are supported.
  • Alias for ‘UCS2’ – ‘UTf16LE’.
  • ‘base64’ – Base64 encoding. This encoding also correctly accepts the “URL and filename safety letters” specified in section 5 of RFC 4648 when creating buffers from strings.
  • ‘latin1’ – a method of encoding buffers as single-byte encoded strings (defined by IANA in RFC 1345, p. 63, as complementary blocks to Latin-1 and C0/C1 control codes).
  • Alias of ‘binary’ – ‘latin1’.
  • ‘hex’ – Encodes each byte into two hexadecimal characters.
const buf = Buffer.from('hello world'.'ascii');
console.log(buf.toString('hex')); // 68656c6c6f20776f726c64
Copy the code

The string and Buffer types interconvert

String to Buffer

Buffer.form() is implemented in the same way that encoding is stored in utF-8

const buf = Buffer.from('Node.js Technology Stack '.'UTF-8');

console.log(buf); // <Buffer 4e 6f 64 65 2e 6a 73 20 e6 8a 80 e6 9c af e6 a0 88>
console.log(buf.length); / / 17
Copy the code

Buffer is converted to a string

ToString ([encoding], [start], [end]). The default encoding is UTF-8. Pass start and end to implement partial conversion (careful here)

const buf = Buffer.from('Node.js Technology Stack '.'UTF-8');

console.log(buf); // <Buffer 4e 6f 64 65 2e 6a 73 20 e6 8a 80 e6 9c af e6 a0 88>
console.log(buf.length); / / 17
console.log(buf.toString('UTF-8'.0.9)); / / the Node. Js �
Copy the code

Node.js = node.js = node.js = node.js = node.js = node.js = node.js = node.js = node.js

Why is there garbled characters during conversion?

First of all, the default encoding used in the above example is UTF-8. The problem is that a Chinese character takes up 3 bytes in UTF-8. The buF corresponding byte is 8a, 80 e6. This will cause characters to be truncated and garbled.

Let’s change the scope of the sample:

const buf = Buffer.from('Node.js Technology Stack '.'UTF-8');

console.log(buf); // <Buffer 4e 6f 64 65 2e 6a 73 20 e6 8a 80 e6 9c af e6 a0 88>
console.log(buf.length); / / 17
console.log(buf.toString('UTF-8'.0.11)); / / the Node. Js
Copy the code

You can see that the output is normal

Buffer memory mechanism

The node.js garbage collection section explains how V8 is used to manage node.js garbage collection, but it does not mention how Buffer data is collected. Let’s look at the memory collection mechanism of Buffer.

Since Buffer needs to process a large amount of binary data, if it is used only a little bit, it will cause frequent memory requests to the system. Therefore, the memory occupied by Buffer is no longer allocated by V8, but is applied at the C++ level of node.js. Memory allocation in JavaScript. Therefore, this part of memory is called off-heap memory.

Note: the following buffer.js source code is node.js v10.x, address: github.com/nodejs/node…

Principle of Buffer memory allocation

Node.js uses slab mechanism for pre-application and post-allocation, which is a dynamic management mechanism.

Using buffer. alloc(size) to pass a specified size will apply for a fixed-size memory area. Slab has the following three states:

  • Full: Indicates the full allocation status
  • Partial: indicates the partial allocation status
  • Empty: no assigned state

8 KB limit

Node.js uses an 8KB limit to distinguish small objects from large objects, and you can see the following code in buffer.js

Buffer.poolSize = 8 * 1024; // Line 102, node.js version is v10.x
Copy the code

As mentioned in the Introduction to Buffers section, buffers are created with predetermined sizes and cannot be adjusted.

Buffer object allocation

In the following code example, calling createPool() directly at load time initializes an 8 KB memory space directly, making it more efficient to allocate memory the first time. It also initializes a new variable poolOffset = 0 that records how many bytes have been used.

Buffer.poolSize = 8 * 1024;
varpoolSize, poolOffset, allocPool; .// Middle code is omitted

function createPool() {
  poolSize = Buffer.poolSize;
  allocPool = createUnsafeArrayBuffer(poolSize);
  poolOffset = 0;
}
createPool(); / / 129 rows
Copy the code

In this case, the newly constructed slab is as follows:

Now let’s try to allocate a Buffer object of size 2048 as follows:

Buffer.alloc(2 * 1024)
Copy the code

Now let’s first look at what our current slab memory looks like, okay? As follows:

So what does this allocation process look like? Let’s look at buffer.js another core method allocate(size)

// https://github.com/nodejs/node/blob/v10.x/lib/buffer.js#L318
function allocate(size) {
  if (size <= 0) {
    return new FastBuffer();
  }

  // If the allocated space is smaller than buffer. poolSize shifts to the right, the result is 4KB
  if (size < (Buffer.poolSize >>> 1)) {
    if (size > (poolSize - poolOffset))
      createPool();
    var b = new FastBuffer(allocPool, poolOffset, size);
    poolOffset += size; // The used space is accumulated
    alignPool(); // 8 bytes memory alignment processing
    return b;
  } else { // C++ level application
    returncreateUnsafeBuffer(size); }}Copy the code

After reading the above code, you can clearly see when small Buffer objects are allocated and when large Buffer objects are allocated.

Buffer memory allocation summary

This content is really difficult to understand, read several node.js related books, park Ling’s “simple and profound Node.js” Buffer section is quite detailed, I recommend you to read it.

  1. An 8KB memory space is initialized at first load, as shown in the buffer.js source code
  2. According to the requested memory size, it can be divided into small Buffer objects and large Buffer objects
  3. In the case of a small Buffer, it continues to determine whether the slab space is sufficient
    • If there is enough space to use the remaining space and update the slab allocation state, the offset increases
    • If there’s not enough space, there’s not enough slab space, then a new slab space will be created to allocate
  4. For large buffers, the createUnsafeBuffer(size) function is used directly
  5. For both small and large Buffer objects, memory allocation is done at the C++ level, memory management is done at the JavaScript level, and can eventually be reclaimed by V8’s garbage collection flag.

Buffer Application Scenarios

The following are some examples of Buffer applications in real business, and you are welcome to add them in the comments section!

I/O operations

As for I/O, it can be file or network I/O. The following is to read the input. TXT information by stream and then write it to output. TXT. If you don’t understand the relationship between Stream and Buffer, please refer to the introduction of Buffer. , what is Buffer?

const fs = require('fs');

const inputStream = fs.createReadStream('input.txt'); // Create a readable stream
const outputStream = fs.createWriteStream('output.txt'); // Create a writable stream

inputStream.pipe(outputStream); // Pipe read and write
Copy the code

You don’t need to create your own buffer in a Stream, it will be created automatically in a Node.js Stream.

zlib.js

Zlib.js is one of the core libraries of Node.js, which utilizes the Buffer function to manipulate binary data streams, and provides compression or decompression functions. Refer to the source code zlib.js source code

encryption

Buffer is used in some encryption and decryption algorithms. For example, crypto. CreateCipheriv’s second parameter key is of type String or Buffer. Here is a simple encryption example, focusing on initializing an instance with buffer.alloc () (described above) and then filling it with the fill method. Here is a look at the use of this method.

buf.fill(value[, offset[, end]][, encoding])

  • Value: The first argument is the content to fill
  • Offset: offset, starting position of filling
  • “End” : indicates the offset of the buF
  • Encoding: code set

The following is a symmetric encryption Demo for ciphers

const crypto = require('crypto');
const [key, iv, algorithm, encoding, cipherEncoding] = [
    'a123456789'.' '.'aes-128-ecb'.'utf8'.'base64'
];

const handleKey = key= > {
    const bytes = Buffer.alloc(16); // Initialize a Buffer instance with each entry filled with 00
    console.log(bytes); // <Buffer 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00>
    bytes.fill(key, 0.10) / / fill
    console.log(bytes); // <Buffer 61 31 32 33 34 35 36 37 38 39 00 00 00 00 00 00>

    return bytes;
}

let cipher = crypto.createCipheriv(algorithm, handleKey(key), iv);
let crypted = cipher.update('Node.js Technology Stack ', encoding, cipherEncoding);
    crypted += cipher.final(cipherEncoding);

console.log(crypted) // jE0ODwuKN6iaKFKqd3RF4xFZkOpasy8WfIDl8tRC5t0=
Copy the code

Buffer VS Cache

What is the difference between a Buffer and a Cache?

Buffer

A Buffer is used to process binary streaming data and Buffer the data. It is temporary. For streaming data, a Buffer is used to temporarily store the data and store it in hard disk after it is buffered to a certain size. Video players are a classic example, sometimes you will see a buffered icon, which means that the buffer is not full, and when the data reaches the full buffer and is processed, the buffered icon disappears and you can see some image data.

Cache

Cache can be regarded as an intermediate layer, which can permanently Cache hotspot data to make access faster. For example, we request data from hard disks or other third-party interfaces through Memory and Redis for caching, so as to store data in the Cache area of Memory. In this way, access to the same resource is faster, which is also an important point of performance optimization.

For a discussion from Zhihu, click on more

Buffer VS String

Stress test to see how both String and Buffer perform?

const http = require('http');
let s = ' ';
for (let i=0; i<1024*10; i++) {
    s+='a'
}

const str = s;
const bufStr = Buffer.from(s);
const server = http.createServer((req, res) = > {
    console.log(req.url);

    if (req.url === '/buffer') {
        res.end(bufStr);
    } else if (req.url === '/string') { res.end(str); }}); server.listen(3000);
Copy the code

I tested the above examples in a virtual machine. You can also test them on your local computer using the AB test tool.

Test string

Take a look at the following important parameters and compare them through buffer transmission

  • Complete requests: 21815
  • Requests per second: 363.58 [#/ SEC] (mean)
  • Transfer rate: 3662.39 [Kbytes/ SEC] Received
$ab - 200 - c t 60 http://192.168.6.131:3000/stringCopy the code

Test the buffer

The total number of requests transferred through buffer was 50,000, the QPS more than doubled, and the bytes transferred per second was 9138.82 KB, which shows that the performance can be nearly doubled by converting data to buffer in advance.

  • Complete requests: 50000
  • Requests per second: 907.24 [#/ SEC] (mean)
  • Transfer rate: 9138.82 [Kbytes/ SEC] Received
$ab - 200 - c t 60 http://192.168.6.131:3000/bufferCopy the code

In the HTTP transmission, binary data is transferred. In the above example, the /string interface directly returns a string. In this case, HTTP converts the string to Buffer before transmission, which is transmitted in binary data and then gradually returned to the client as a Stream. Returning Buffer directly, however, eliminates each conversion operation and improves performance.

In some Web applications, static data can be pre-converted to Buffer for transmission, which can effectively reduce CPU reuse (repeated string to Buffer operations).

Reference

  • Nodejs. Cn/API/buffer….
  • Node.js Buffer
  • Do you want a better understanding of Buffer in Node.js? Check this out.
  • A cartoon intro to ArrayBuffers and SharedArrayBuffers
  • buffer.js v10.x
By May Jun link: HTTPS://github.com/Q-Angelo/Nodejs-RoadmapSource: nodejs.js technology stackCopy the code