¿

This article includes the following contents:

  • WebSocketChapter 4 – Connection handshake
  • WebSocketProtocol Chapter 5 – Data Frames
  • nodejs wsLibrary source code analysis – connection handshake process
  • nodejs wsLibrary source code analysis – data frame parsing process

reference

A deep dive into the WebSocket protocol

ws – github

This article will not cover the basic knowledge of WebSocket, such as concept, definition, explanation and purpose. It is a bit dry and long. Markdown is about 800 lines

1. Connect the handshake process

There is a common statement about websockets: Websockets reuse HTTP handshake channels, which refer to:

The client negotiates an upgrade protocol with the WebSocket server through HTTP requests. After the upgrade, subsequent data exchanges follow the WebSocket protocol

1.1 Client: Apply for a protocol upgrade

The client initiates a protocol upgrade request. According to the WebSocket protocol specification, the request header must contain the following content

GET/HTTP/1.1 Host: localhost:8080 Origin: http://127.0.0.1:3000 Connection: Upgrade Upgrade: websocket Sec-WebSocket-Version: 13 Sec-WebSocket-Key: w4v7O6xFTi36lq3RNcgctw==Copy the code
  • Request line: The request method must be GET, HTTP version at least 1.1
  • The request must contain Host
  • If the request is coming from a browser client, Origin must be included
  • The request must contain Connection and its value must contain the “Upgrade” flag
  • The request must contain Upgrade, whose value must contain the “websocket” keyword
  • The request must contain sec-websocket-version, whose value must be 13
  • The request must contain an SEC-websocket-key to provide basic protection, such as unintentional connections

1.2 Server: Response protocol upgrade

The response header returned by the server must contain the following content

HTTP/1.1 101 Switching Protocols Connection:Upgrade Upgrade: websocket Sec- websocket-Accept: Oy4NRAQ13jhfONC7bP8dTKb4PTU=Copy the code
  • Response:HTTP / 1.1 101 Switching separate Protocols
  • The response must contain Upgrade with a value of “weboscket”
  • The response must contain Connection with a value of “Upgrade”
  • The response must contain sec-websocket-Accept, calculated from the sec-websocket-key at the request header

1.3 Calculation of sec-websocket-key /Accept

The specification mentions:

The sec-websocket-key value is a randomly generated 16-byte random number encoded in base64 (see chapter 4 of RFC4648)

For example, the randomly selected 16 bytes are:

// Hexadecimal digits 1 to 16 0x01 0x02 0x03 0x04 0x05 0x06 0x07 0x08 0x09 0x0A 0x0B 0x0C 0x0D 0x0E 0x0f 0x10Copy the code

The base64 encoding value is AQIDBAUGBwgJCgsMDQ4PEA==

The test code is as follows:

const list = Array.from({ length: 16 }, (v, index) => ++index)
const key = Buffer.from(list)
console.log(key.toString('base64'))
// AQIDBAUGBwgJCgsMDQ4PEA==
Copy the code

The sec-websocket-accept value is calculated as follows:

  1. willSec-Websocket-KeyThe value of and258EAFA5-E914-47DA-95CA-C5AB0DC85B11Joining together
  2. throughSHA1Compute the abstract and convert it tobase64string

There is no need to worry about the magic string 258eafa5-e914-47DA-95CA-C5AB0DC85b11. It is just a GUID, probably generated randomly when writing an RFC

The test code is as follows:

const crypto = require('crypto')

function hashWebSocketKey (key) {
  const GUID = '258EAFA5-E914-47DA-95CA-C5AB0DC85B11'

  return crypto.createHash('sha1')
    .update(key + GUID)
    .digest('base64')
}

console.log(hashWebSocketKey('w4v7O6xFTi36lq3RNcgctw=='))
// Oy4NRAQ13jhfONC7bP8dTKb4PTU=
Copy the code

1.4 Functions of sec-websocket-key

As briefly mentioned above, its role is to provide basic protection and reduce malicious connections. Further elaboration is as follows:

  • KeyThis prevents the server from receiving illegal messagesWebSocketConnect, for examplehttpRequest a connection towebsocketIn this case, the server can reject the request
  • KeyCan be used to initially ensure server awarenesswsProtocol, however, does not rule out some HTTP server only processingSec-WebSocket-Key, does not come truewsagreement
  • KeyReverse proxy caching can be avoided
  • Make an Ajax request in the browser,Sec-Websocket-KeyAnd the associated headers are disallowed, so that clients do not accidentally request protocol upgrades when they send Ajax requests

Finally, it is important to note that sec-websocket-key /Accept is not used to ensure data security, because its calculation/conversion formula is public and very simple. The main purpose is to prevent unexpected situations

2. Data frames

The minimum unit of WebSocket communication is frame, which consists of one or more frames to form a complete message. During the process of data exchange, the sender and receiver need to do the following:

  1. Sender: The message is cut into multiple frames and sent to the server
  2. Receiver: Accepts message frames and reassembles the associated frames into a complete message

The format of data frames, as the core, may seem hard to understand at first glance, but the author of this article has ordered that it be understood and rushed

2.1 Detailed description of data frame format

  • FIN: 占1bit

    • 0Indicates not the last shard of the message
    • 1Represents the last shard of the message

  • RSV1, RSV2, RSV3: each 1bit, generally all 0, related to Websocket expansion, if the non-zero value and no Websocket expansion, connection error

  • Opcode: bit 4

    • %x0: indicates that the data transmission adopts a data fragment. The current data frame is one of the data fragments
    • %x1: indicates that this is a text frame
    • %x2: indicates a binary frame
    • %x3-7: Reserved action code for later defined non-control frames
    • %x8: Indicates that the connection is down
    • %x9: Indicates a heartbeat request (ping)
    • %xA: indicates a heartbeat response (pong)
    • %xB-F: Reserved action code for later defined non-control frames

  • Mask: 占1bit

    • 0Indicates that no mask xOR operation is performed on the data payload
    • 1Represents mask xOR operations on data payloads

  • Payload Length: 7, 7+16, or 7+64bit

    • 0 ~ 125: The data length is equal to the value
    • 126: The following two bytes represent a 16-bit unsigned integer with a value of the length of the data
    • 127: The following 8 bytes represent a 64-bit unsigned integer with a value of the length of the data

  • Masking-key: a value of 0 or 4bytes

    • 1: Carries a 4-byte Masking-key
    • 0: there is no Masking – key
    • The purpose of the mask is not to prevent data leakage, but to prevent problems such as proxy cache contamination attacks that existed in earlier versions of the protocol

  • Payload Data: Payload data

I think if you know the difference between byte and bit, this part is fine

2.2 Data Transfer

Each WebSocket message may be divided into multiple data frames. When receiving a data frame, the WebSocket determines whether it is the last data frame based on the FIN value

Data frame transfer example:

  1. FIN=0, Opcode=0x1: Sent text type, message not yet sent, and subsequent frames
  2. FIN=0, Opcode=0x0: The message is not completed, and there are subsequent frames, followed by the previous one
  3. FIN=1, Opcode=0x0: The message is sent, no subsequent frames, followed by the last one to form a complete message

3. Ws library source code analysis: connection handshake process

Although I was using socket. IO, I stumbled upon WS, which was used quite heavily, with weekly downloads six times that of socket. IO

In NodeJS, the upgrade event of the HTTP module is triggered whenever a negotiated upgrade request is encountered. This is the entry point for implementing WebSocketServer. The native sample code is as follows:

// Create an HTTP server. const srv = http.createServer( (req, res) => { res.writeHead(200, {'Content-Type': 'text/plain' });
  res.end('Response content');
});
srv.on('upgrade', (req, socket, head) => {// specific processing to implement the Websocket service});Copy the code

And, in general use, it is an extension of an existing httpServer to implement WebSocket, rather than creating a separate WebSocketServer

The example code ws uses, based on an existing httpServer, is

const http = require('http');
const WebSocket = require('ws');

const server = http.createServer();
const wss = new WebSocket.Server({ server });

server.listen(8080);
Copy the code

The existing httpServer is passed as a parameter to the websocket. Server constructor, so the core entry point for source code analysis is:

new WebSocket.Server({ server });
Copy the code

From this pointcut, you can fully reproduce the join handshake process

3.1 Analyzing the WebSocketServer class

Because httpServer is passed in as an argument, its constructor becomes quite simple:

Class WebSocketServer extends EventEmitter {constructor(options, callback) {super(if(options.server) {this._server = options.server} // Listen for eventsif (this._server) {
      this._removeListeners = addListeners(this._server, {
        listening: this.emit.bind(this, 'listening'),
        error: this.emit.bind(this, 'error'HandleUpgrade (req, socket, head, (ws) => {this.emit(req, socket, head);'connection', ws, req)})}})}}} // This is a very nice piece of code, return a function to remove multiple event listeners while binding multiple event listenersfunction addListeners(server, map) {
  for (const event of Object.keys(map)) server.on(event, map[event]);

  return function removeListeners() {
    for(const event of Object.keys(map)) { server.removeListener(event, map[event]); }}; }Copy the code

As you can see, in the constructor, a listener for the upgrade event is registered for the httpServer. When triggered, the this.handleUpgrade function is executed, which is the next step

3.2 Filtering illegal requests: handleUpgrade function

This function is used to filter out illegal requests, including:

  • The Sec – WebSocket – Key value

  • The Sec – WebSocket – Version value

  • The path of the WebSocket request

The key codes are as follows:

const keyRegex = /^[+/0-9A-Za-z]{22}==$/;

handleUpgrade(req, socket, head, cb) {
  socket.on('error', socketOnError) // Get sec-websocket-key const key = req.headers['sec-websocket-key'] !== undefined
    ? req.headers['sec-websocket-key']
    : falseSec-websocket-version const version = +req.headers['sec-websocket-version'Const extensions = {}; // For illegal requests, break the handshakeif( req.method ! = ='GET'|| req.headers.upgrade.toLowerCase() ! = ='websocket'| |! key || ! keyRegex.test(key) || (version ! == 8 && version ! = = 13) | | / / this function is to Websocket request path of judgment, and option. The path, don't start! this.shouldHandle(req) ) {returnAbortHandshake (socket, 400)} // For valid requests, update it! this.completeUpgrade(key, extensions, req, socket, head, cb) }Copy the code

AbortHandshake: 400 bad Request abortHandshake: 400 bad request abortHandshake:

const {  STATUS_CODES } = require('http');

functionAbortHandshake (socket, code, message, headers) {// net. socket is also a duplex stream, so it can be read or writtenif (socket.writable) {
    message = message || STATUS_CODES[code];
    headers = {
      Connection: 'close'.'Content-type': 'text/html'.'Content-Length': Buffer.byteLength(message), ... headers }; Socket. Write (` HTTP / 1.1${code} ${STATUS_CODES[code]}\r\n` +
        Object.keys(headers)
          .map((h) => `${h}: ${headers[h]}`)
          .join('\r\n') +
        '\r\n\r\n'+ message ); } // Remove the error listener socket.removelistener ('error', socketOnError); Socket.destroy (); // Make sure there are no more I/O activities on the socket. }Copy the code

If all goes well, we come to the completeUpgrade function

3.3 Completing the handshake: The completeUpgrade function

This function is mainly used to return the correct response, trigger related events, record values, etc. The code is relatively simple

const { createHash } = require('crypto');
const { GUID } = require('./constants');
const WebSocket = require('./websocket');

function completeUpgrade(key, extensions, req, socket, head, cb) {
  // Destroy the socket if the client has already sent a FIN packet.
  if(! socket.readable || ! socket.writable)returnSocket. Destroy () // Generate sec-websocket-accept const digest = createHash('sha1')
    .update(key + GUID)
    .digest('base64'); // set Headers const Headers = ['the HTTP / 1.1 101 Switching separate Protocols'.'Upgrade: websocket'.'Connection: Upgrade',
    `Sec-WebSocket-Accept: ${digest}`]; // Create an instance of Websocket const ws = new Websocket(null) this.emit('headers', headers, req); Socket.write (headers. Concat ('\r\n').join('\r\n'));
  socket.removeListener('error', socketOnError); // ws.setsocket (socket, head, this.options. MaxPayload); // Set records the client in the connected stateif (this.clients) {
    this.clients.add(ws);
    ws.on('close', () => this.clients.delete(ws)); } // Trigger the connection event cb(ws); }Copy the code

At this point, the entire handshake phase is complete, but the processing of data frames is not yet involved

4. Ws library source code analysis: data frame processing

At the end of the previous chapter, the following code is revealed in completeUpgrade:

ws.setSocket(socket, head, this.options.maxPayload);
Copy the code

Enter the setSocket method in the WebSocket class, the data frame processing code can be simplified as:

Class WebSocket extends EventEmitter {
  ...
  setSocket(Socket, head, maxPayload) {// Instantiate a writable stream for processing data frames this._extensions, maxPayload ); receiver[kWebSocket] = this; socket.on('data', socketOnData); }}function socketOnData(chunk) {
  if (!this[kWebSocket]._receiver.write(chunk)) {
    this.pause();
  }
}
Copy the code

Many event handling, such as error, end, close, etc., are omitted here because they are irrelevant to the purpose of this article, and some apis are not covered

So the core pointcut is the Receiver class, which is the core for processing data frames

4.1 Basic structure of Receiver class

The Receiver class inherits from writable streams and needs to clarify two basic concepts:

  • streamAll streams areEventEmitterAn instance of the
  • Implementing writable flows requires implementationwritable._writeMethod for internal use
const { Writable } = require('stream') class Recevier extends Writable { constructor(binaryType, extensions, maxPayload) { super() this._binaryType = binaryType || BINARY_TYPES[0]; // nodebuffer this[kWebSocket] = undefined; / / the WebSocket instance references this. _extensions = extensions | | {}; / / the WebSocket protocol to expand this. _maxPayload = maxPayload | 0; // 100 * 1024 * 1024 this._bufferedBytes = 0; This._buffers = []; // Record buffer data this._compressed =false; // Whether to compress this._payloadLength = 0; // Data frame PayloadLength this._mask = undefined; // data frames Mask this._fragmented = 0; // Whether data frames are fragmented this._masked =false; // Data frame Mask this._fin =false; // Data frame FIN this._opcode = 0; Opcode this._totalPayloadLength = 0; This._messagelength = 0; _fragments = []; This. _state = GET_INFO; // Flag bit for startLoop function this._loop =false; // Flag bit for startLoop function} _write(chunk, encoding, cb) {if (this._opcode === 0x08 && this._state == GET_INFO) returncb(); this._bufferedBytes += chunk.length; this._buffers.push(chunk); this.startLoop(cb); }}Copy the code

As you can see, every time a new data frame is received, it is logged in the _buffers array and the parsing process startLoop begins immediately

4.2 Data frame parsing process: startLoop function

startLoop(cb) {
  let err;
  this._loop = true;

  do {
    switch (this._state) {
      case GET_INFO:
        err = this.getInfo();
        break;
      case GET_PAYLOAD_LENGTH_16:
        err = this.getPayloadLength16();
        break;
      case GET_PAYLOAD_LENGTH_64:
        err = this.getPayloadLength64();
        break;
      case GET_MASK:
        this.getMask();
        break;
      case GET_DATA:
        err = this.getData(cb);
        break;
      default:
        // `INFLATING`
        this._loop = false;
        return; }}while (this._loop);

  cb(err);
}
Copy the code

The parsing process is simple:

  • GetInfo parses data such as FIN, RSV, OPCODE, MASK, and PAYLOAD LENGTH

  • The payload length is divided into three types:

    • 0~125: Call haveLength

    • The getPayloadLength16 method is triggered and the haveLength method is called

    • GetPayloadLength64 (haveLength)

  • In the haveLength method, if there is a mask, the getMask method is called first, then the getData method is called

The overall flow and state are controlled by this._loop and this._state, which is more intuitive

4.3 Consume Buffer: consume method

The first step should be analyzing the getInfo method, but the consume method is involved. This provides a neat way to consume the obtained Buffer. The function takes an argument, n, representing the number of bytes to consume, and returns the number of bytes consumed

If we need to get the first byte of the data frame (including FIN + RSV + OPCODE), just pass this.consume(1)

The record value this._buffers is a buffer array, which initially stores complete data frames. As consumption progresses, the data will become smaller.

  1. The number of bytes consumed is exactly equal to the number of bytes of a chunk

  2. The number of bytes consumed is less than the number of bytes of a chunk

  3. The number of bytes consumed is greater than the number of bytes of a chunk

In the first case, just move out + return

if (n === this._buffers[0].length) return this._buffers.shift()
Copy the code

In the second case, just crop + return

if (n < this._buffers[0].length) {
  const buf = this._buffers[0]
  this._buffers[0] = buf.slice(n)
  return buf.slice(0, n)
}
Copy the code

In the third case, which is a bit more complicated, we first apply for a buffer space of the size of bytes to be consumed to store the returned buffers

// It doesn't matter if the buffer space is initialized, because eventually it will all be overwrittenCopy the code

In this case, it is guaranteed to be larger than the first chunk, but it is not certain that it will be larger than the first chunk after a chunk is consumed (the index moves forward after consumption), so a loop is required

// do... While can avoid a meaningless judgment, first execute a loop body, then judge the conditiondo{const buf = this._buffers[0] // If the buffer length is larger than the first chunk, remove the + bufferif(n >= buf.length) { this._buffers.shift().copy(dst, dst.length - n); } // If the length is less than one chunk, crop + copyelseBuf. copy(DST, dst.length -n, 0, n); // buf.copy(DST, dst.length -n, 0, n); this._buffers[0] = buf.slice(n); } n -= buf.length; }while (n > 0)
Copy the code

4.4 Analyzing data Frames: getInfo method

A minimum data frame must contain the following data:

FIN (1 bit) + RSV (3 bit) + OPCODE (4 bit) + MASK (1 bit) + PAYLOADLENGTH (7 bit)
Copy the code

At least 2 bytes, so less than 2 bytes of data frame is an error, simplified getInfo as follows

getInfo() {
  if (this._bufferedBytes < 2) {
    this._loop = false
    return} const buf = this._fin = (buf[0] &0x80) === 0x80 this._opcode = buf[0] &0x0f This._payloadlength = buf[1] &0x7f this._masked = (buf[1] &0x80) === 0x80 // Three cases of the Payload Lengthif (this._payloadLength === 126) this._state = GET_PAYLOAD_LENGTH_16
  else if (this._payloadLength === 127) this._state = GET_PAYLOAD_LENGTH_64
  else return this.haveLength()
}
Copy the code

The core here is the meaning of the location operator &. Let’s start with FIN, which is the first bit in the data frame

// The value of FIN is denoted by [], where X represents the subsequent bit in the first byte [] XXXXXXX // the hexadecimal number 0x80 represents the binary 10000000 // The result is bit-independent of the following 7 bits []0000000 // therefore, This._fin = (buf[0] & 0x80) === 0x80Copy the code

OPCODE and PAYLOAD LENGTH are the same

[][][][] [][][][] [][][][] [][][][] With the press, 0111, 1111 in can x [] [] [] [] [] [] [] [] 0111 & 1111 (0 x7f)Copy the code

4.5 Payload Length Three scenarios and the size end

The three cases are as follows:

  • 0-125: The actual length of the load is a number between 0-125

  • 126: The actual payload length is the value of a 16-bit unsigned integer represented by the next two bytes

  • 127: The actual payload length is the value of a 64-bit unsigned integer represented by the following 8 bytes

If this sounds convoluted, look at the code, for example, branch 126:

getPayloadLength16() {
  if (this._bufferedBytes < 2) {
    this._loop = false;
    return;
  }

  this._payloadLength = this.consume(2).readUInt16BE(0);
  return this.haveLength();
}
Copy the code

As you can see, the core of the processing length is readUInt16BE(0), which involves the size side:

  • The Big endian considers the first byte to be the highest bit, similar to what we know about the size of a decimal number

  • The Little endian considers the first byte to be the least significant byte

Then, the specification says that the next two bytes represent a 16-bit unsigned integer value, which naturally refers to the big end

Big end vs small end comparison:

// If the following two bytes are binary 1111 1111 0000 0001 // convert to hexadecimal 0xFF 0x01 // the big end outputs 65281 console.log(buffer. from([0xff, 0x01]).uint16be (0).toString(10))Copy the code

In addition to this, the 7 + 64 mode has a little extra processing, which looks like this:

getPayloadLength64() {
  if (this._bufferedBytes < 8) {
    this._loop = false;
    return;
  }

  const buf = this.consume(8);
  const num = buf.readUInt32BE(0);

  //
  // The maximum safe integer in JavaScript is 2^53 - 1. An error is returned
  // if payload length is greater than this number.
  //
  if (num > Math.pow(2, 53 - 32) - 1) {
    this._loop = false;
    return error(
      RangeError,
      'Unsupported WebSocket frame: payload length > 2^53 - 1'.false, 1009); } this._payloadLength = num * Math.pow(2, 32) + buf.readUInt32BE(4);return this.haveLength();
}
Copy the code

4.6 Obtaining load data: getData

Before obtaining the payload, if the getInfo value of mask is 1, the getMask operation is required to obtain the mask Key(a total of four bytes).

getMask() {
  if (this._bufferedBytes < 4) {
    this._loop = false;
    return;
  }

  this._mask = this.consume(4);
  this._state = GET_DATA;
}
Copy the code

The source code for getData is simplified as follows

GetData (cb) {// data is buffer.alloc (0)letdata = EMPTY_BUFFER; Payload data = this.consume(this._payloadLength) // If there are masks, decode them according to the mask keyif(this._masked) unmask(data, this._mask) // Logs it into the fragmented array this._fragments.push(data) // If this data frame represents: connection disconnection, heartbeat request, heartbeat responseif (this._opcode > 0x07) returnThis.controlmessage (data) // If the data frame represents: data fragment, text frame, binary framereturn this.dataMessage()
}
Copy the code

4.7 Assembly load data: dataMessage

Next, look at the dataMessage() function, which combines data from multiple frames and is simpler after simplification

dataMessage() {
  if (this._fin) {
    const messageLength = this._messageLength
    const fragments = this._fragments

    const buf = concat(fragments, messageLength)
    this.emit('message', buf.toString())}} //function concat(list, totalLength) {
  if (list.length === 0) return EMPTY_BUFFER;
  if (list.length === 1) return list[0];

  const target = Buffer.allocUnsafe(totalLength);
  let offset = 0;

  for (let i = 0; i < list.length; i++) {
    const buf = list[i];
    buf.copy(target, offset);
    offset += buf.length;
  }

  return target;
}
Copy the code

5. To summarize

This article is long and is not the kind of interview questions in small pieces of knowledge, reading urgent need patience, has tried to avoid Posting large sections of code, can see here I want to give you money

Through this analysis, two key phases in WebSocket are fully introduced and replicated:

  • Connection handshake phase

  • Data exchange stage

In my opinion, the key is the use of Node.js buffer module and stream module, which is also the biggest part of the harvest