Are you always as confused as I am when I encounter words like Buffer,Stream, and binary data in Node.js? They are not considered common and are only suitable for node.js experts and package developers.

In fact, these words are very important, especially for web development with Node.js without any CS degree.

Of course, if you choose to remain a normal Node.js developer, you may never use them directly. But if you want to take your understanding of Node.js to the next level, you do need to dig deeper into many of Node’s core features. Such as Buffer. That’s exactly why I’m writing this post — to help demystify some of these features and take Node.js learning to the next level.

Before we start, take a look at the node.js documentation for buffers

… mechanism for reading or manipulating streams of binary data. The Buffer class was introduced as part of the Node.js API to make it possible to interact with octet streams in the context of things like TCP streams and file system operations.

Let’s rephrase it in plain English:

The Buffer class was introduced as part of the Node.js API to make it possible to manipulate or interact with binary data streams.Copy the code

Let’s take a closer look at Buffers, Streams, and binary data.

What is binary data

You probably already know that computers store and represent data in binary files. Binary is just a set of ones and zeros. For example, here are five different binaries with five different groups of ones and zeros:

10, 01, 001, 1110, 00101011
Copy the code

Each 1 and 0 in binary is called a Bit, which is a short form of binary number.

To store or represent a piece of data, a computer needs to convert that data into a binary representation. For example, to store the number 12, the computer needs to convert it to a binary representation, 1100.

But at work, number is not the only data type. Usually there are strings,images,videos. Computers know how to represent all data types in binary. For example, how does a computer represent the string “L” in binary? To store any character in a binary file, the computer first converts the character to a number, and then converts the number to a binary representation. For the string “L”, the computer first converts L to a number representing L.

Open the browser console and type L. charcodeat(0). The console displays the number 76, which is the numerical representation of the character “L”. But how does a computer know the exact number each character represents? How does it know what L is in terms of 76?

Character set

Character set A rule defined to represent the exact number of each character. We have different definitions of these rules, the most popular include Unicode and ASCII. JavaScript works well with the Unicode character set. So, browser Unicode dictates that 76 should stand for L.

We have seen how computers represent characters in numbers. After converting it to a number, the computer converts 76 into its binary representation.

A character encoding

Just as there are character set rules that define how numbers should represent characters, there are rules that define how numbers should be represented in binary files. Specifically, how many bits are used to represent a number. This is called character encoding.

One rule for character encoding is UTF-8. Utf-8 declaration characters should be encoded in bytes. A byte is a collection of eight bits — eight ones and zeros. Therefore, UTF-8 specifies that eight ones and zeros should be used to represent any character in binary.

As mentioned in the previous example, the number 12 is represented as 1100 in binary, but in UTF-8 it should be 8 bits. So UTF-8 states that the computer needs to add more bits to the left of the less than 8-bit binary number to make it a byte. So 12 should be stored as 00001100.

Therefore, 76 is stored as 01001100 under UTF-8 rules

This is how computers store strings or characters in binary files. Similarly, computers dictate rules for how pictures and videos should be converted, encoded, and stored in binary files. Computers store all data types in binary files.

Now that we know what binary data is, let’s look at what binary data flow is.

flow

A Stream in JS simply represents a sequence of data that moves from one point to another over time. The whole idea is that you have a lot of data to work with, but you don’t have to wait until all the data is available to start working with it. Basically, this big data is broken up and sent as chunks. Thus, from the original definition of Buffer, this simply means that binary data is moving through the file system. For example, move text stored in file1.txt to file2.txt.

But how exactly does Buffer help us interact with or manipulate binary data in a stream? What exactly is a Buffer?

Buffer

We’ve already mentioned that data flows are data moving from one point to another, but how exactly do they move?

Usually data is moved in order to process or read data and make decisions based on that data. In this process, there may be a minimum or maximum amount of data to be processed. Therefore, if data is arriving faster than the process can consume it, the excess data needs to be processed by waiting somewhere. On the other hand, if the process is consuming data faster than it can arrive, the small amount of data that arrives early needs to wait for a certain amount of data to arrive before it is sent out for processing.

That “waiting area” is the Buffer! It is a small physical location in a computer, usually in RAM, where data is temporarily collected, waited on, and finally sent out for processing in a stream.

We can think of the entire Stream and buffer process as a bus station. At a particular bus station, the bus will not leave until there is a certain number of passengers or a special time. In addition, passengers may arrive at different times and at different speeds. Neither the passenger nor the bus station can control the time when the passenger arrives at the station. Passengers who arrive early have to wait for the bus to depart. When some passengers arrive, the passengers are already full or the bus has already left and needs to wait for the next bus.

No matter what the situation, there is always a place to wait. This is node.js’s Buffer! Js does not control how fast or when data arrives, nor does it control the speed of the flow. It can only decide when to send data. If the time is not up, Node.js will place them in buffer, a small location in RAM, until they are sent out for processing.

A typical example is when you watch a streaming video and you can see the buffer working. If your Internet connection is fast enough, the stream will be fast enough to immediately fill the Buffer and send it out for processing, then fill another Buffer, then send it out, then send another, and then send another until the stream is complete.

But if your connection is slow, the video will get stuck after processing the first set of incoming data, which means the program is collecting more data, or waiting for more data to arrive. When the buffer is filled and processed, the player continues playing the video. While playing, more data will continue to arrive and wait in buffer.

Interact with Buffer

Buffers are created automatically by Node.js during the processing of a stream, or we can create buffers ourselves using the API provided by Nodejs. There are several different ways to create buffers, depending on your needs.

// Create an empty buffer of size 10.
// A buffer that only can accommodate 10 bytes
const buf1 = Buffer.alloc(10)

// Create a buffer with content
const buf2 = Buffer.from("hello buffer")
Copy the code

Once the buffer has been created, you can start interacting with it.

// Check the structure of the buffer
buf1.toJSON()
// { type: 'Buffer', data: [ 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 ] }

buf2.toJSON()
//{ type: 'Buffer',data: [ 104, 101, 108, 108, 111, 32, 98, 117, 102, 102, 101, 114 ]}

buf1.length / / 10
buf2.length / / 12

/ / write operations
buf1.write("Buffer really rocks!")

// decode
buf1.toString() // 'Buffer rea'
// Because buf1 was created with only 10byte space allocated. Excess will not be stored.
Copy the code

For more interactive apis, check out the official documentation,

Hopefully this introduction has helped you better understand Node.js Buffer.

For more js articles, check out Github