The JavaScript language itself only has string data types, not binary data types. But when dealing with things like TCP streams or file streams, you must use binary data. Therefore, in Node.js, a Buffer class is defined to create a Buffer dedicated to binary data.

In Node.js, the Buffer class is the core library shipped with the Node kernel. The Buffer library gives Node.js a way to store raw data, allows Node.js to work with binary data, and is likely to be used whenever data needs to be moved during I/O operations in Node.js. The raw data is stored in an instance of the Buffer class. A Buffer is similar to an array of integers, but it corresponds to a block of raw memory outside of V8 heap memory.

The buffer store is in hexadecimal format, which saves the node’s binary to hexadecimal storage. In general, utF8 is 3 bytes and GBK/GB2312 is 2 bytes.

The use of the Buffer

// Create a Buffer of length 10 filled with zero.
const buf1 = Buffer.alloc(10);

// Create a Buffer of length 10,
// All of them are filled with bytes with the value '1'.
const buf2 = Buffer.alloc(10.1);

// Create an uninitialized buffer of length 10.
// This method is faster than calling buffer.alloc (),
// However, the returned Buffer instance may contain old data,
// So you need to rewrite it with fill(), write(), or some other function that fills the contents of the Buffer.
const buf3 = Buffer.allocUnsafe(10);

// Create a Buffer containing bytes [1, 2, 3].
const buf4 = Buffer.from([1.2.3]);

// Create a Buffer containing bytes [1, 1, 1, 1],
// All entries are truncated with '(value & 255)' to fit the range 0-255.
const buf5 = Buffer.from([257.257.5, -255.'1']);

// Create a Buffer containing utF-8 encoded bytes of the string 'test' :
// [0x74, 0xC3, 0xa9, 0x73, 0x74]
// [116, 195, 169, 115, 116]
const buf6 = Buffer.from('tést');

// Create a Buffer containing latin-1 bytes [0x74, 0xe9, 0x73, 0x74].
const buf7 = Buffer.from('tést'.'latin1');

Copy the code

A Buffer object can be converted to a string using the toString method, which takes an argument specifying what type of string to convert. Such as UTF8, base64 and so on.

const buf = Buffer.from('hello world'.'utf8');

console.log(buf.toString('hex'));
// Print: 68656c6c6f20776f726c64
console.log(buf.toString('base64'));
// Print: aGVsbG8gd29ybGQ=

console.log(Buffer.from('fhqwhgads'.'utf8'));
// Print: 
      
console.log(Buffer.from('fhqwhgads'.'utf16le'));

      
Copy the code

Converting a Buffer to a string using one of the above methods is called decoding; Converting a string to a Buffer is called encoding.

  • ‘UTf8 ‘: Multi-byte encoded Unicode character. Many web pages and other document formats use UTF-8. This is the default character encoding. When decoding Buffer to a string that does not specifically contain valid UTF-8 data, the Unicode replacement character U+FFFD � is used to indicate these errors

  • ‘UTF16LE ‘: Unicode character with multi-byte encoding. Unlike ‘UTf8’, each character in a string is encoded in two or four bytes. Node.js supports only utF-16 minor endian variants.

  • ‘latin1’: Latin-1 stands for ISO-8859-1. This character encoding only supports Unicode characters from U+0000 to U+00FF. Each character is encoded with a single byte. Characters outside the range are truncated and mapped to characters within the range.

Node.js also supports two binary-to-text encodings. For binary-to-text encodings, the naming convention is reversed: converting a Buffer to a string is usually called encoding, while converting a string to a Buffer is called decoding.

  • ‘base64’: Base64 encoding. This encoding also correctly accepts the “URL and filename safety letters” specified in section 5 of RFC 4648 when creating buffers from strings. Whitespace characters (such as Spaces, tabs, and newlines) contained in base64-encoded strings are ignored.

  • ‘hex’: Encodes each byte into two hexadecimal characters. Data truncation may occur when decoding a string that contains only valid hexadecimal characters.

  • ‘ASCII ‘: applies only to 7-bit ASCII data. When encoding a string as Buffer, this is equivalent to using ‘latin1’. When decoding Buffer to a string, using this encoding will additionally unset the highest bit of each byte before decoding it to ‘latin1′. In general, there should be no reason to use this encoding when encoding or decoding plain ASCII text, as’ UTf8 ‘(or ‘latin1’ if known data is always plain ASCII) would be a better choice. This is provided for traditional compatibility only.

  • ‘binary’: alias for ‘latin1’ For more background on this encoding, see binary Strings. The name of the encoding can be misleading, as all of the encodings listed here convert between strings and binary data. For converting between strings and buffers, ‘UTF-8’ is usually the correct choice.

  • ‘UCS2 ‘: alias for’ UTf16LE ‘. Ucs-2 formerly referred to a variant of UTF-16 that did not support characters with code points greater than U+FFFF. In Node.js, these code points are always supported.

Buffer.from('1ag'.'hex');
// Prints 
      
       , and the data is truncated when the first non-hexadecimal value ('g') is encountered.
      

Buffer.from('1a7g'.'hex');
// Prints 
      
       , which is truncated when the data ends with a number ('7').
      

Buffer.from('1634'.'hex');
// Print 
      
       , all data are available.
      
Copy the code

Modern Web browsers follow the WHATWG coding standards by aliasing ‘latin1’ and ‘ISO-8859-1’ to ‘WIN-1252’. This means that when performing an operation such as http.get(), if the returned character set is one of those listed in the WHATWG specification, the server may actually return ‘WIN-1252’ encoded data, while using ‘latin1’ encoded data may incorrectly decode characters

Buffer with TypedArray

Here’s a quick overview of TypedArray. As we all know, it’s impossible to manipulate files directly on traditional Web front ends. Early browsers only provide file-input tags on form forms for uploading files.

ES6 provides ArrayBuffer and TypedArray, so that the front end can also directly edit binary data. The input tag of type file in a web page can also be converted to binary through FileReader, and then edit.

ArrayBuffer: Represents a chunk of binary data in memory from which we can directly create binary objects and then use the associated methods and properties. An ArrayBuffer object represents raw binary data that can only be stored rather than edited directly.

TypedArray is really used to read and write simple types of binary data. Typed is a generic term that refers to specific types of data, such as Int8Array and Uint8Array.

var ab = new ArrayBuffer(32)
var iA = new Int8Array(ab)
iA[0] = 97;// Change the first digit of binary data to 97, which is the asCLL code of lowercase letter A;
var blob = new Blob([iA], {type: "application/octet-binary"});// Convert binary code to bloB type
var url = URL.createObjectURL(blob);
Copy the code

The Buffer instance is also a Uint8Array and TypedArray instance of JavaScript. All TypedArray methods are also available on Buffer. However, there are subtle incompatibilities between Buffer’s API and TypedArray’s API.

Buffers and iterators

Buffer instances can use for.. The of syntax iterates:

const buf = Buffer.from([1.2.3]);

for (const b of buf) {
  console.log(b);
}
/ / print:
/ / 1
/ / 2
/ / 3
Copy the code

In addition, the buf.values(), buf.keys(), and buf.entries() methods can also be used to create iterators.

Buffer.alloc(size[, fill[, encoding]])

  • Size Expected length of the new Buffer.

  • The fill | | | for pre-filled with the new value of the Buffer. Default value: 0.

  • Encoding If fill is a string, this is its character encoding. Default: ‘utf8’.

Allocate a new Buffer of size bytes. If fill is undefined, the Buffer is filled with zero.

const buf = Buffer.alloc(5);

console.log(buf);

      
Copy the code

If size is greater than buffer.constants.MAX_LENGTH or less than 0, ERR_INVALID_OPT_VALUE is thrown.

If fill is specified, the allocated Buffer is initialized by calling buf.fill(fill).

const buf = Buffer.alloc(5.'a');

console.log(buf);
// Print: 
Copy the code

If both fill and encoding are specified, the allocated Buffer is initialized by calling buf.fill(fill, encoding).

const buf = Buffer.alloc(11.'aGVsbG8gd29ybGQ='.'base64');

console.log(buf);

      
Copy the code

Calling buffer.alloc () may be much slower than the alternative buffer.allocunsafe (), but ensures that the contents of a newly created Buffer instance will never contain sensitive data from previously allocated data, including data that may not have been allocated to Buffer.

If size is not a number, TypeError is raised.

Buffer.allocUnsafe(size) Creates a new Buffer with size bytes. If size is greater than buffer.constants.MAX_LENGTH or less than 0, ERR_INVALID_OPT_VALUE is thrown.

The underlying memory of a Buffer instance created in this way is uninitialized. The contents of the newly created Buffer are unknown and may contain sensitive data. Using buffer.alloc (), you can create a Buffer instance that is initialized with zero.

The Buffer module preallocates a Buffer instance with an internal size of buffer. poolSize as a quick allocation memory pool. The new Buffer(size) constructor used to create a new Buffer instance using buffer.allocunsafe (), or buffer.from (array), or buffer.concat (), or deprecated, but only if size is less than or equal to Buffer.poolSize >> 1 (buffer. poolSize divided by two and rounded down).

The use of this pre-allocated internal memory pool is the key difference between calls to buffer.alloc (size, fill) and buffer.allocunsafe (size).fill(fill). Specifically, buffer.alloc (size, fill) never uses the internal Buffer pool, Buffer.allocunsafe (size).fill(fill) will use the internal Buffer pool if the size is less than or equal to half of buffer.poolsize. This difference, while subtle, can be important when applications need the additional performance provided by buffer.allocunsafe ().

Once a buffer is declared, its length cannot be increased.

Buffer.concat(list[, totalLength])

  • A list < Buffer [] > | < Uint8Array [] > Buffer array or Uint8Array arrays to be combined.

  • TotalLength the totalLength of the combined Buffer instances in the list.

Returns a new Buffer that incorporates all the Buffer instances in the list.

If there are no elements in the list, or totalLength is 0, a Buffer of length 0 is returned.

If no totalLength is provided, it is calculated by adding the lengths of the Buffer instances in the list.

If totalLength is provided, it is cast to an unsigned integer. If the combined Buffer length in the list is greater than totalLength, the result is truncated to the totalLength length.

// Create a single Buffer from an array of three instances of Buffer.

const buf1 = Buffer.alloc(10);
const buf2 = Buffer.alloc(14);
const buf3 = Buffer.alloc(18);
const totalLength = buf1.length + buf2.length + buf3.length;

console.log(totalLength);
// Print: 42

const bufA = Buffer.concat([buf1, buf2, buf3], totalLength);

console.log(bufA);
// 打印: <Buffer 00 00 00 00 ...>
console.log(bufA.length);
// Print: 42

Copy the code

Buffer.isBuffer(obj)

Return true if obj is a Buffer, false otherwise.

Buffer.isEncoding(encoding)

Encoding Indicates the name of the character encoding to be checked.

Returns true if encoding is the name of the supported character encoding, false otherwise.

console.log(Buffer.isEncoding('utf-8'));
// Print: true

console.log(Buffer.isEncoding('hex'));
// Print: true

console.log(Buffer.isEncoding('utf/8'));
// Print: false

console.log(Buffer.isEncoding(' '));
// Print: false
Copy the code

buf.copy(target[, targetStart[, sourceStart[, sourceEnd]]])

  • Target | to copy into the Buffer or Uint8Array.

  • TargetStart Target Number of bytes to skip before starting writing. Default value: 0.

  • SourceStart BuF Specifies the offset from which to start the copy. Default value: 0.

  • SourceEnd BuF Offset (not included) to end the copy. Default value: buf.length.

  • Returns: Number of bytes copied.

  • Copy data from a region in BUF to a region in target, even if target’s memory region overlaps that of BUF.

TypedArray#set() performs the same operation and is available for all TypedArray, including Node.js’s Buffer, although it takes a different function argument.

// Create two Buffer instances.
const buf1 = Buffer.allocUnsafe(26);
const buf2 = Buffer.allocUnsafe(26).fill('! ');

for (let i = 0; i < 26; i++) {
  // 97 is the ASCII decimal value of 'a'.
  buf1[i] = i + 97;
}

// copy data at offsets 16 to 19 from 'buf1' to 'buf2' at offsets 8.
buf1.copy(buf2, 8.16.20);
// This is equivalent to:
// buf2.set(buf1.subarray(16, 20), 8);

console.log(buf2.toString('ascii'.0.25));
// Print:!!!!!!!! qrst!!!!!!!!!!!!!
Copy the code
// Create a 'Buffer' and copy the data from one area of the same 'Buffer' to another overlapping area.

const buf = Buffer.allocUnsafe(26);

for (let i = 0; i < 26; i++) {
  // 97 is the ASCII decimal value of 'a'.
  buf[i] = i + 97;
}

buf.copy(buf, 0.4.10);

console.log(buf.toString());
/ / print: efghijghijklmnopqrstuvwxyz
Copy the code

buf.equals(otherBuffer)

const buf1 = Buffer.from('ABC');
const buf2 = Buffer.from('414243'.'hex');
const buf3 = Buffer.from('ABCD');

console.log(buf1.equals(buf2));
// Print: true
console.log(buf1.equals(buf3));
// Print: false
Copy the code

buf.fill(value[, offset[, end]][, encoding])

  • The value of value | | | used in buf.

  • Offset starts filling buF’s offset. Default value: 0.

  • End Specifies the offset of buF (not included). Default value: buf.length.

  • Encoding Specifies the character encoding of value if value is a string. Default: ‘utf8’.

  • Return: a reference to buf.

  • Populate the BUF with the specified value. If offset and end are not specified, the entire buf is populated:

  • The fs module reads the buffer

// Fill 'Buffer' with the ASCII character 'h'.

const b = Buffer.allocUnsafe(50).fill('h');

console.log(b.toString());
/ / print: HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH
Copy the code

If value is not a string, Buffer, or integer, it is converted to a uint32. If the resulting integer is greater than 255 (decimal), buf will be populated with value & 255.

If fill() ends up writing a multi-byte character, then only bytes that are appropriate for buf are written:

// Fill 'Buffer' with characters that take up two bytes in UTF-8.

console.log(Buffer.allocUnsafe(5).fill('\u0222'));

      
Copy the code

If value contains invalid characters, truncate the invalid characters. If there is no data after truncation, do not fill:

const buf = Buffer.allocUnsafe(5);

console.log(buf.fill('a'));
// Print: 
console.log(buf.fill('aazz'.'hex'));

      
console.log(buf.fill('zz'.'hex'));
// Throw an exception.
Copy the code
  • Stream file reading
const fs = require('fs');
const path = require('path');
const rs = fs.createReadSteam(path.resolve(__dirname, 'name.txt'), {
    flags: 'r'.// r w Read and write permissions
    highWaterMark: 4.// How many bytes are expected to be read each time
    encoding: null.autoClose: true.// Close the file
    start: 0.end: 5 // Slice (start, end) contains end
})
Rs.emit ('data', 123); rs.emit('data', 123);
const arr = [];
rs.on('data'.function(chunk) {
    arr.push(chunk);
    rs.pause(); // Pause the trigger of the data event rs.play(); Continue to
});
rs.on('end'.function() {
    console.log(Buffer.concat(arr).toString()); // Complete reading
});

rs.on('error'.function(error) {
    console.log(error);
})
Copy the code

File compression zlib

File compression based on zlib library and buffer

const zlib = require('zlib');
const fs = require('fs');
const rs = fs.cerateReadStream('jquery.js');
const ws = fs.cerateWriteStream('jquery.js.gz');
const gz = zlib,createGzip();
rs.pipe(gz).pipe(ws);
ws.on('error'.(err) = > {
    console.log('failure');
})
ws.on('finish'.() = > {
    console.log('complete')})Copy the code

Server zip file


const http = require('http'); // Service module
const fs = require('fs'); // Read the file
const zlib = require('zlib'); // Compression module

const server = http.cerateServer((req, res) = > {
    const rs = fs.cerateReadStream(`www${req.url}`);
    res.setHeader('content-encoding'.'gzip'); // Set the response header
    const gz = zlib.createGzip(); / / gz compression
    rs.pipe(gz).pipe(res); // Return to the front end after compression
    rs.on('error'.err= > { // Failed to listen
        res.setHeader(404);
        res.write('Not Found');
        res.end();
       console.log('Read failed')
    })
    rs.on('finish'.err= > { // The transfer is complete
       console.log('Write done')
    })
}).listen(8080);

Copy the code

[Reference -] (nodejs.org/dist/latest…)