If you’re learning about buffers for the first time, you might be unfamiliar with them, because they don’t exist in JavaScript on the front end, because the front end just does some string manipulation or DOM basic manipulation to meet your business needs.

What is a buffer?

A buffer is the underlying memory of a Node that is allocated through C++ and JS. This is the buffer that holds the files. The question is, why is it called a cache? Before we get to the bottom of this, we need to explain V8’s memory limitations.

When we declare variables in our code and assign values, the memory of the objects we use is allocated in the heap. If there is not enough free heap memory allocated for new objects, heap memory will continue to be allocated until the heap size exceeds V8’s limit. (About 1.4 GB on 64-bit systems and 0.7 GB on 32-bit systems)

When V8 is doing garbage collection, JS is stagnant, waiting for the garbage collection to complete and then resume. Therefore, the garbage collection time is related to our performance. When the memory is too large, it takes longer, so V8 has to limit this.

Under this limitation, Node will not be able to directly operate large memory objects. For example, if a 2 GB file cannot be read into memory for string analysis, the file will be read into buffer and temporarily stored, so it is called buffer.

const fs = require('fs')

fs.readFile('./a.js', { encoding: 'utf8' }, (err, data) => {
  if(! err) { fs.writeFile('./b.js', data, { encoding: 'utf8' }, (err, data) => {
      if (err) {
        console.log('Write failed')}}}else {
    console.log('Read failed')}})Copy the code

Instead of using fs.readFiles and fs.writeFile to operate on large files directly, use fs.createReadStream and fs.createWriteStream to stream large files.

let reader = fs.createReadStream('a.js');
let writer = fs.createWriteStream('b.js');
reader.pipe(writer);
Copy the code

The idea behind streaming files is to use buffers. Read and write a little at a time.

Although buffers are requested memory and are not limited by V8 memory, physical memory is still limited.

Understand the Buffer

A Buffer is an object like Array, but it is primarily used to manipulate bytes.

Because Buffer is so common, Node loads it when the process starts and places it on a global object. Therefore, when using Buffer, it can be used directly without requiring ().

A Buffer object is similar to an array in that its elements are hexadecimal two-digit numbers ranging from 0 to 255.

let buf = Buffer.from('hello'.'utf8')
console.log(buf)
// <Buffer 68 65 6c 6c 6f 20 77 6f 72 6c 64>
console.log(buf.length) // 5
console.log(buf[0]) // 104
Copy the code

You can access the length attribute for length (which is the length of a byte, 3 bytes for Chinese and 1 byte for English in UTF8 encoding), and you can access elements by subscript.

The important thing to note about subscript access elements is that there is a computational process to go through. We take obtaining the first letter as an example. First, h is found to be 68 in hexadecimal according to the ASCII code comparison table. Then, the toString method in JS passes in a parameter of 2 to indicate that it is converted into binary to obtain 1101000, and then the number changed to 10 becomes 104. So the first element accessed by subscript is 104, as shown below.

let buf = Buffer.from('hello'.'utf8')
console.log(buf)
// <Buffer 68 65 6c 6c 6f 20 77 6f 72 6c 64>
console.log((0x68).toString(2))
// 1101000
Copy the code

The first argument is a string to create a Buffer containing a string, and the second argument is the encoding of the specified character. The default value is utf8.