Want to do a simple Web API, this time you need to build a Web server, ASP.NET needs IIS to build the server, PHP needs to use Apache/Nginx to achieve, for the novice before the start to see so many steps, may have to give up, However, it is So Easy to start a Web server in Node.js. It can be achieved with a few lines of simple code using Net, Dgram, HTTP, HTTPS and other modules.

Quick navigation

  • A network model
  • Introduction to TCP
  • Net module builds a TCP service
  • TCP sticky packet problem

The interview guide

  • What is THE TCP protocol? When do you choose TCP? Refer to Interview1
  • What is TCP sticky packet? How to solve it? Refer to Interview2

A network model

Most students are familiar with HTTP and HTTPS, which are usually used for interaction between browser and server, or interaction between server and server. The other two Net and Dgram may be relatively unfamiliar. These two are implemented based on the transport layer of network model, corresponding to TCP and UDP protocols respectively. The following figure shows the relationship between the OSI seven-layer model and the TCP/IP five-layer model. Dotted lines are used in the middle to mark the transport layer, and the upper application layer (HTTP/HTTPS, etc.) is also based on this layer of TCP protocol to achieve, so I want to use Node.js to do server development. The.NET module is also a must, and is the focus of this article.

Introduction to TCP

Interview1: Some concepts to be clear, what is TCP protocol? When do you choose TCP?

TCP is a transmission control protocol. Most of the time, we use this protocol because it is a more reliable data transfer protocol. It has three characteristics:

  • Link-oriented: requires the host to be online and establish links.
  • Byte stream oriented: You give me a bunch of byte streams and I send them to you, but EACH time I send them, I select a byte and send it with a serial number, which is the number of the least numbered byte in the sent byte.
  • Reliability: Ensure that the data reaches the host in an orderly manner. Every time a data is sent, the host expects to receive a reply from the host. If the host receives a reply within a specified period of time, the host considers the host to have received the data.

TCP is connection-oriented and reliable. One of the most significant features of TCP is that there is a three-way handshake before transmission, which is implemented as follows:

During a TCP three-way handshake, the client and server each provide a socket to form a link. The client and server then use this link to send data to each other.

Net module builds a TCP service

Now that we know some of the concepts of TCP, let’s create a TCP server and client instance. Here we need to use the Node.js Net module, which provides some interfaces for the underlying communication. This module can be used to create stream-based TCP or IPC servers (net.createserver ()) and clients (net.createconnection ()).

Creating a TCP Service

You can create a TCP Server link using new Net.server, or you can create a TCP object using the factory function net.createserver (). The internal implementation of createServer() also internally calls the Server constructor to create a TCP object. The same as new Net.server, the code looks like this:

Github.com/nodejs/node…

function createServer(options, connectionListener) {
  return new Server(options, connectionListener);
}
Copy the code

Github.com/nodejs/node…

function Server(options, connectionListener) {
  if(! (this instanceof Server))
    return new Server(options, connectionListener);

  // The Server class still inherits EventEmitter, which is outside the scope of this section
  EventEmitter.call(this); .Copy the code

TCP Service Events

Before you start coding, check out the related events at nodejs.cn/api/net.htm… Here, I will not introduce all of them. Here, I will introduce some common ones and explain them through code examples. On this basis, I can refer to the official website to practice some other events or methods.

TCP Server Events

  • Listening: server.listen();
  • CreateServer (function(socket) {}); net.createserver (function(socket) {}
  • Close: triggered when the server is closed (server.close()). If there are any connections, this event will not be raised until all connections end
  • Error: Traps errors. For example, when listening to an existing port, the error: Listen EADDRINUSE error is reported

TCP connection event method

  • Data: When one end calls the write() method to send data, the other end receives it through the socket.on(‘data’) event, which can be interpreted as reading data
  • End: Indicates that the socket link appears once each time. For example, after the client sends a message, it will receive the message by pressing Ctrl + C
  • Error: indicates the error information about the listening socket
  • Write: Write is a method (socket.write()) where the data event is to read the data, and the write method in this case is to write the data to the other end,

TCP server code implementation

const net = require('net');
const HOST = '127.0.0.1';
const PORT = 3000;

// Create a TCP service instance
const server = net.createServer();

// Listen on the port
server.listen(PORT, HOST);

server.on('listening', () = > {console.log(The service has been started in${HOST}:${PORT}`);
});

server.on('connection', socket => {
    // The data event reads data
    socket.on('data', buffer => {
        const msg = buffer.toString();
        console.log(msg);

        // The write method writes data and sends it back to the client
        socket.write(Buffer.from('hello' + msg));
    });
})

server.on('close', () = > {console.log('Server Close! ');
});

server.on('error', err => {
    if (err.code === 'EADDRINUSE') {
        console.log('Address in use, trying again... ');

        setTimeout((a)= > {
            server.close();
            server.listen(PORT, HOST);
        }, 1000);
    } else {
        console.error('Server exception:', err); }});Copy the code

TCP client code implementation

const net = require('net');
const client = net.createConnection({
    host: '127.0.0.1'.port: 3000
});

client.on('connect', () = > {// Send data to the server
    client.write('Nodejs Technology Stack ');

    setTimeout((a)= > {
        client.write('JavaScript ');
        client.write('TypeScript ');
        client.write('Python ');
        client.write('Java ');
        client.write('C ');
        client.write('PHP ');
        client.write('ASP.NET ');
    }, 1000);
})

client.on('data', buffer => {
    console.log(buffer.toString());
});

// ECONNREFUSED is used when listening to an unenabled port
client.on('error', err => {
    console.error('Server exception:', err);
});

client.on('close', err => {
    console.log('Client link down! ', err);
});
Copy the code

Source code implementation address

https://github.com/Q-Angelo/project-training/tree/master/nodejs/net/chapter-1-client-server 
Copy the code

Client and server Demo tests

Start the server first, then start the client, the client is called three times, and the print result is as follows:

The service side

The $node server.js service is enabled at 127.0.0.1:3000# for the first time,Nodejs technology stack JavaScript TypeScript Python Java C PHP ASP.NET# for the second timeNodejs technology stack JavaScript TypeScript Python Java C PHP ASP.NETCopy the code

The client

$ node client.js
# for the first time,Hello Nodejs technology stack hello JavaScript hello TypeScript Python Java C PHP ASP.NET# for the second timeHello Nodejs technology stack hello JavaScript TypeScript Python Java C PHP ASP.NETCopy the code

On the client side, I used client.write() to send data several times, but only the data outside setTimeout is normal. SetTimeout seems to send data consecutively, but it will return randomly. Why? Take a look at the following TCP sticky packet problem introduction.

TCP sticky packet problem

Interview2: What is TCP sticky packet? How to solve it?

The example above finally raises the question, why does a client continuously send data to a server and receive a merge return? It is also common in TCP stick package problem, the client (send) at the end of the before sending would send a short period of time have more than one block of data buffer together (the sender buffer), forming a large block of data sent, along with all the same the receiver also has a receiver buffer, received the data for the receiving end of buffer, The program then reads some of the data from here and consumes it, again to reduce I/O consumption for performance optimization.

Question to consider: When does the data start to be sent when it reaches the buffer?

This depends on the TCP congestion control, is any time to determine the number of bytes that can be sent out within the control factor, is to stop the sender to the receiver means of congestion, the link between the reference wikipedia: zh.wikipedia.org/wiki/TCP congestion control…

TCP Sticky packet solution?

  • Scheme 1: Delay sending
  • Scheme 2: Turn off Nagle algorithm
  • Scheme 3: Package sealing/unpacking

Scheme 1: Delay sending

The simplest solution is to set the mode of delay sending and sleep for a period of time. However, although this solution is simple, its disadvantages are also obvious. The transmission efficiency is greatly reduced, and it is obviously not suitable for scenes with frequent interaction.

client.on('connect', () => {
    client.setNoDelay(true);
    // Send data to the server
    client.write('Nodejs Technology Stack ');

    const arr = [
        'JavaScript '.'TypeScript '.'Python '.'Java '.'C '.'PHP '.'ASP.NET '
    ]
    
    for (let i=0; i<arr.length; i++) {
        (function(val, k){
            setTimeout((a)= > {
                client.write(val);
            }, 1000 * (k+1)) }(arr[i], i)); }})Copy the code

The console runs the Node client.js command and everything seems to be ok without sticky packets, but this is only used in scenarios with low interaction frequency.

$node client.js Hello Nodejs technology stack hello JavaScript hello TypeScript hello Python hello Java hello C hello PHP hello ASP.NETCopy the code

Source code implementation address

https://github.com/Q-Angelo/project-training/tree/master/nodejs/net/chapter-2-delay
Copy the code

Scheme 2: Nagle algorithm

Nagle algorithm is an algorithm to improve the transmission efficiency of the network, avoid the network is filled with a large number of small data blocks, it is expected to send as large a data block as possible, so every time a data block is requested to send TCP, TCP will not immediately send, but wait for a short period of time to send.

Nagle’s ability to aggregate small chunks of data and send them together to reduce network congestion is still helpful when the network is full of small chunks of data, but this is not necessary in all scenarios, such as REPL terminal interactions where a user enters a single character to get a response, So in Node.js you can set the socket.setnodelay () method to turn Nagle off.

const server = net.createServer();

server.on('connection', socket => {
    socket.setNoDelay(true);
})
Copy the code

Disabling Nagle algorithm is not always effective, because it is merged on the server side. TCP will first store the received data in its own buffer, and then notify the application to receive it. If the application layer fails to retrieve data from the TCP buffer due to network or other reasons, multiple data blocks will be stored in the TCP buffer. It will form a sticky bag again.

Scheme 3: Package sealing/unpacking

In front of the two solutions are not very ideal, here are the third kind of packet/unpacking, is currently the industry with more, used here, length encoding good communication both sides agreed format, message can be divided into fixed Header (Header) and variable length of the message Body (Body), read the message Header when parsing get the length of the content to take up, When the number of bytes of the body content read later equals the number of bytes of the header, we consider it to be a complete package.

Message Header Number (Header) Message body length (Header) The message Body (Body)
SerialNumber bodyLength body
2 (bytes) 2 (bytes) N (bytes)

Prior knowledge Buffer

I’ll do this by coding, but before I start I want you to know something about buffers. What exactly are buffers in Node.js? , I will list the buffers needed for this time to explain, which is helpful for those who do not know Buffer.

  • Buffer.alloc(size[, fill[, encoding]]) : initializes a size of Buffer space, filling 0 by default, or specifying fill for custom filling
  • Buf. writeInt16BE(value[, offset]) : value indicates the Buffer value to be written, offset indicates the position from which the offset starts to be written
  • Buf. writeInt32BE(value[, offset]) : This parameter is the same as that of writeInt16BE. The difference is that writeInt16BE indicates that a 16-bit integer is first written to a high value, while writeInt32BE indicates that a 32-bit integer is first written to a high value
  • Buf. readInt16BE([offset]) : indicates a 16-bit integer with high priority to be read. Offset is the number of bytes to be skipped before reading
  • Buf. readInt32BE([offset]) : 32-bit integer with high priority to be read. Offset is the number of bytes to be skipped before reading

Encoding/decoding implementation

The bottom layer of TCP transmission is based on binary data, but our application layer is usually easy to express strings, numbers, etc. Here, the first step in the implementation of encoding, we need to first convert our data into binary data through Buffer, and when we take out the data, we also need to decode the operation, all in the code, the implementation is as follows:

// transcoder.js

class Transcoder {
    constructor () {
        this.packageHeaderLen = 4; // The length of the head
        this.serialNumber = 0; // Define the package number
        this.packageSerialNumberLen = 2; // The bytes occupied by the packet sequence number
    }

    /** * encoding * @param {Object} data Buffer Object data * @param {Int} serialNumber packet serialNumber, which is automatically generated when the client encodes, and which needs to be passed in when the server encodes after decoding */
    encode(data, serialNumber) {
        const body = Buffer.from(data);

        const header = Buffer.alloc(this.packageHeaderLen);
        header.writeInt16BE(serialNumber || this.serialNumber);
        header.writeInt16BE(body.length, this.packageSerialNumberLen); // Skip the first two digits of the packet sequence number

        if (serialNumber === undefined) {
            this.serialNumber++;
        }
       
        return Buffer.concat([header, body]);
    }

    /** * decode * @param {Object} buffer */
    decode(buffer) {
        const header = buffer.slice(0.this.packageHeaderLen); // get the baotou
        const body = buffer.slice(this.packageHeaderLen); // Get the package tail
    
        return {
            serialNumber: header.readInt16BE(),
            bodyLength: header.readInt16BE(this.packageSerialNumberLen), // Since the first two digits are skipped during the coding phase, the decoding should also be skipped
            body: body.toString(),
        }
    }

    * 1. If the current buffer size is smaller than the packet header, it is definitely not a complete packet, so it returns 0. * 2. Otherwise return the full packet length * @param {*} buffer */
    getPackageLength(buffer) {
        if (buffer.length < this.packageHeaderLen) {
            return 0;
        }
    
        return this.packageHeaderLen + buffer.readInt16BE(this.packageSerialNumberLen); }}module.exports = Transcoder;
Copy the code

Client Transformation

const net = require('net');
const Transcoder = require('./transcoder');
const transcoder = new Transcoder();
const client = net.createConnection({
    host: '127.0.0.1'.port: 3000
});

let overageBuffer=null; // Last Buffer remaining data

client.on('data', buffer => {
    if (overageBuffer) {
        buffer = Buffer.concat([overageBuffer, buffer]);
    }

    let packageLength = 0;

    while (packageLength = transcoder.getPackageLength(buffer)) {
        const package = buffer.slice(0, packageLength); // Retrieve the entire packet
        buffer = buffer.slice(packageLength); // Delete the packets that have been removed. This method is used to intercept the packets that have been removed from the buffer

        const result = transcoder.decode(package); / / decoding
        console.log(result);
    }

    overageBuffer=buffer; // Log the remaining incomplete packages
}).on('error', err => { // ECONNREFUSED is used when listening to an unenabled port
    console.error('Server exception:', err);
}).on('close', err => {
    console.log('Client link down! ', err);
});

client.write(transcoder.encode('0 Nodejs Stack '));

const arr = [
    '1 JavaScript '.'2 TypeScript '.'3 Python '.'4 Java '.'5 C '.'6 PHP '.'7 ASP.NET '
]

setTimeout(function() {
    for (let i=0; i<arr.length; i++) {
        console.log(arr[i]); client.write(transcoder.encode(arr[i])); }},1000);
Copy the code

Server Transformation

const net = require('net');
const Transcoder = require('./transcoder');
const transcoder = new Transcoder();
const HOST = '127.0.0.1';
const PORT = 3000;
let overageBuffer=null; // Last Buffer remaining data

const server = net.createServer();

server.listen(PORT, HOST);

server.on('listening', () = > {console.log(The service has been started in${HOST}:${PORT}`);
}).on('connection', socket => {
    // The data event reads data
    socket
        .on('data', buffer => {
            if (overageBuffer) {
                buffer = Buffer.concat([overageBuffer, buffer]);
            }
        
            let packageLength = 0;
        
            while (packageLength = transcoder.getPackageLength(buffer)) {
                const package = buffer.slice(0, packageLength); // Retrieve the entire packet
                buffer = buffer.slice(packageLength); // Delete the packets that have been removed. This method is used to intercept the packets that have been removed from the buffer
        
                const result = transcoder.decode(package); / / decoding
                console.log(result);
                socket.write(transcoder.encode(result.body, result.serialNumber));
            }
        
            overageBuffer=buffer; // Log the remaining incomplete packages
        })
        .on('end'.function(){
            console.log('socket end')
        })
        .on('error'.function(error){
            console.log('socket error', error);
        });
}).on('close', () = > {console.log('Server Close! ');
}).on('error', err => {
    if (err.code === 'EADDRINUSE') {
        console.log('Address in use, trying again... ');

        setTimeout((a)= > {
            server.close();
            server.listen(PORT, HOST);
        }, 1000);
    } else {
        console.error('Server exception:', err); }});Copy the code

Run the test

The console executes Node server.js to start the server, and then executes Node client.js to start the client test. The output is as follows:

$ node client.js
{ serialNumber: 0, bodyLength: 18, body: '0 Nodejs Stack ' }
1 JavaScript 
2 TypeScript 
3 Python 
4 Java 
5 C 
6 PHP 
7 ASP.NET 
{ serialNumber: 1, bodyLength: 13, body: '1 JavaScript ' }
{ serialNumber: 2, bodyLength: 13, body: '2 TypeScript ' }
{ serialNumber: 3, bodyLength: 9, body: '3 Python ' }
{ serialNumber: 4, bodyLength: 7, body: '4 Java ' }
{ serialNumber: 5, bodyLength: 4, body: '5 C ' }
{ serialNumber: 6, bodyLength: 6, body: '6 PHP ' }
{ serialNumber: 7, bodyLength: 10, body: '7 ASP.NET ' }
Copy the code

In the above results, in the setTimeout function, we first send multiple data at the same time, and then return one by one. At the same time, the serial number, length and body of the packet defined by the packet header are printed, and the packet sticking problem mentioned above is also solved. Wrapping/unpacking is a bit complicated, the above code has been as simple as possible to introduce the implementation of the idea, the following code address, can be used as a reference to their own can also use different ways to achieve

https://github.com/Q-Angelo/project-training/tree/master/nodejs/net/chapter-3-package
Copy the code
Link: https://github.com/Q-Angelo/Nodejs-Roadmap Source: Nodejs.js technology stackCopy the code