Node HTTP Server source code parsing

This article focuses on the main process of generating services and processing requests through HTTP, but does not cover other functions.

Using the example

const http = require('http');

http.createServer((req, res) = > {
  res.end('hello word');
}).listen(8080);
Copy the code

In this example, from generating the service, to receiving the request, and finally to responding to the request, there are four major parts of the work, respectively:

callhttp.createServerTo generate a service
calllistenFunction listening port
Receive the request and generatereqandresobject
Execute the business function, executeres.endRespond to the request

HTTP. CreateServer and listen

// lib/http.js
function createServer(opts, requestListener) {
  return new Server(opts, requestListener);
}

// lib/_http_server.js
function Server(options, requestListener) {
  if (typeof options === 'function') {
    requestListener = options;
    options = {};
  }
  // ...
  if (requestListener) {
    // When both the REq and RES objects are generated, the Request event is raised and the business function processes the request
    this.on('request', requestListener);
  }

  // The Connection event can be seen in the NET Server class and is triggered when the three-way handshake is complete
  this.on('connection', connectionListener);
}
ObjectSetPrototypeOf(Server.prototype, net.Server.prototype);
ObjectSetPrototypeOf(Server, net.Server);

function connectionListener(socket) {
  / / here is to perform connectionListenerInternal function is introduced to this and the socket parameters
  defaultTriggerAsyncIdScope(
    getOrSetAsyncId(socket), connectionListenerInternal, this, socket
  );
}

// The callback function after the connection event is triggered. This function is explained in the section "Parsing to generate REq and RES objects"
function connectionListenerInternal(server, socket) {
  // ...
}
Copy the code

When the http.createServer function is called, an instance of a Server is returned, which is inherited from the NET Server class. Therefore, HTTP Server instances also have the ability to listen for port generation services and communicate with clients. The listen function called in the previous example is actually listen in NET Server.

During the instance of the Server object, the Request and Connection events are listened for, respectively.

connection: This is listening tonetIn theconnectionEvent, which is triggered by the server when the client initiates a request and the TCP three-way handshake succeedsconnectionEvents.connectionEvent callback functionconnectionListenerInternalWe’ll talk about that in the next section.
request: whenreqandresOnce the objects have all been initialized successfully, they are publishedrequestEvent, as we saw in the previous coderequestEvent callback functionrequestListenerIs that the developer callshttp.createServerIs passed to the callback function, which receives itreqandresTwo objects.

Generate req and RES objects

When the client TCP request is successfully connected to the server, the server will trigger the Connection event, and then an HTTP-Parser is instantiated to parse the client request. When the client data is successfully parsed, a REQ object is generated. Next, let’s look at the reQ object generation process.

// lib/_http_server.js
function Server(options, requestListener) {
  // ...
  // The connection event is triggered when the client and server complete the three-way handshake
  this.on('connection', connectionListener);
}

function connectionListener(socket) {
  / / here is to perform connectionListenerInternal function is introduced to this and the socket parameters
  defaultTriggerAsyncIdScope(
    getOrSetAsyncId(socket), connectionListenerInternal, this, socket
  );
}

/ * * *@param {http Server} server
 * @param {net Socket} socket* /
function connectionListenerInternal(server, socket) {
  // ...
  // The parsers.alloc function is executed using an HTTPParser object that returns a free list allocation
  const parser = parsers.alloc();
  // Request parser initialization work
  parser.initialize(
    HTTPParser.REQUEST,
    new HTTPServerAsyncResource('HTTPINCOMINGMESSAGE', socket),
    server.maxHeaderSize || 0,
    server.insecureHTTPParser === undefined ?
      isLenient() : server.insecureHTTPParser,
    server.headersTimeout || 0,); parser.socket = socket; socket.parser = parser;// ...
}

// lib/_http_common.js
const parsers = new FreeList('parsers'.1000.function parsersCb() {
  // The http-parser library is used as the request parser
  const parser = new HTTPParser();
  cleanParser(parser);
  // ...
  return parser;
});
Copy the code

The HTTP Server uses an HTTP-Parser instance as the parser for client requests. Note that the Free List data structure is used to allocate the Parser objects.

// lib/internal/freelist.js
class FreeList {
  constructor(name, max, ctor) {
    this.name = name;
    this.ctor = ctor;
    this.max = max;
    this.list = [];
  }

  // Assign an object
  alloc() {
    return this.list.length > 0 ?
      this.list.pop() :
      // cTOR is the method passed in to instantiate FreeList
      ReflectApply(this.ctor, this.arguments);
  }

  // Release the object when it runs out
  free(obj) {
    if (this.list.length < this.max) {
      this.list.push(obj);
      return true;
    }
    return false; }}Copy the code

This section applies to the free List data structure. This data structure is used to reduce the performance cost of object creation and destruction by maintaining a queue of fixed length in which all objects are of the same size. When an object needs to be used, the system preferentially obtains free objects from the queue. If no object is available in the queue, the system creates an object with the same size as the object stored in the queue for the program to use. After an object is used, it is not destroyed directly. Instead, it is pushed into a queue until it is later pushed out for use.

With the Free List in mind, let’s move on to client request parsing.

// lib/_http_common.js
const parsers = new FreeList('parsers'.1000.function parsersCb() {
  const parser = new HTTPParser();

  cleanParser(parser);

  // Bind callback functions to these events
  parser[kOnHeaders] = parserOnHeaders;
  parser[kOnHeadersComplete] = parserOnHeadersComplete;
  parser[kOnBody] = parserOnBody;
  parser[kOnMessageComplete] = parserOnMessageComplete;

  return parser;
});
Copy the code

Http-parser parses client requests based on events:

kOnHeaders: Continuously parses the request headers
kOnHeadersComplete: Request header parsing is complete
kOnBody: Constantly parses the request body
kOnMessageComplete: Request body parsing is complete

During data transmission, TCP unpacks the data that exceeds the remaining space of the buffer. As a result, the same request packet may be sent to the server for multiple times. KOnHeaders and kOnBody are used to concatenate the split data, combining the same request.

When the request header is resolved, the kOnHeadersComplete callback is executed, where the REQ object is generated.

// lib/_http_common.js
const { IncomingMessage } = require('_http_incoming');
// The callback function executed after the request header is parsed
function parserOnHeadersComplete(versionMajor, versionMinor, headers, method, url, statusCode, statusMessage, upgrade, shouldKeepAlive) {
  const parser = this;
  const { socket } = parser;
  // ...
  // In most cases socket.server[kIncomingMessage] equals IncomingMessage
  const ParserIncomingMessage = (socket && socket.server && socket.server[kIncomingMessage]) || IncomingMessage;
  const incoming = parser.incoming = new ParserIncomingMessage(socket);
  // ...
  return parser.onIncoming(incoming, shouldKeepAlive);
}

// lib/_http_incoming.js
function IncomingMessage(socket) {
  // ...
}
Copy the code

The IncomingMessage object instantiated in the kOnHeadersComplete callback is the REQ object. The callback ends by executing the parser.onIncoming function to generate the RES object.

// lib/_http_server.js
function connectionListenerInternal(server, socket) {
  // ...
  // This is the last function executed by the kOnHeadersComplete callback
  parser.onIncoming = FunctionPrototypeBind(parserOnIncoming, undefined, server, socket, state);
  // ...
}

Parser. onIncoming(incoming, shouldKeepAlive); // The fourth argument is the req object. The req object is the incoming object passed in when parser.onIncoming(incoming, shouldKeepAlive)
function parserOnIncoming(server, socket, state, req, keepAlive) {
  // ...
  ArrayPrototypePush(state.incoming, req);

  // Instance the res object
  const res = new server[kServerResponse](req);

  if (socket._httpMessage) {
    ArrayPrototypePush(state.outgoing, res);
  }

  // ...
  // This event is raised when res.end is called
  res.on('finish', FunctionPrototypeBind(resOnFinish, undefined, req, res, socket, state, server));
  // ...
    server.emit('request', req, res); // Issue the Request event and execute the createServer function to call the incoming business handler
  // ...
}

// The ServerResponse here inherits from the OutgoingMessage class, which will be covered later
this[kServerResponse] = options.ServerResponse || ServerResponse;
Copy the code

When both the REQ and RES objects are initialized and stored, the createServer function is executed to call the incoming business handler function. When reQ is generated, the side executes parserOnIncoming to generate the RES object, and registers the Finish event in the RES object. This event is triggered when the business code executes res.end. When both the REQ and RES objects are ready, a Request event is issued and both the REQ and RES objects are passed in. The callback function for the Request event is the one passed in when the business code calls http.createserver.

Res. The end

const http = require('http');

http.createServer((req, res) = > {
  res.end('hello word');
}).listen(8080);
Copy the code

When the business process is complete, the res.end() function is actively called in the business code in response to the client request. Let’s see.

// lib/_http_server.js
function ServerResponse(req) {
  FunctionPrototypeCall(OutgoingMessage, this);
  // ...
}

ObjectSetPrototypeOf(ServerResponse.prototype, OutgoingMessage.prototype);
ObjectSetPrototypeOf(ServerResponse, OutgoingMessage);
Copy the code

The ServerResponse class inherits from the OutgoingMessage class. The res.end method used in business is also defined in OutgoingMessage. Let’s look at the OutgoingMessage class implementation.

// lib/_http_outgoing.js
function OutgoingMessage() {
  // ...
  this._header = null;
  // ...
}

OutgoingMessage.prototype.end = function end(chunk, encoding, callback) {
  / /...
  if (chunk) {
    // ...
    write_(this, chunk, encoding, null.true);
  }

  // Subscribe to the Finish event, which is the callback passed in when res.end is called
  if (typeof callback === 'function')
    this.once('finish', callback);

  // ...
    // Write the response data to the content of the response request using write_, and then execute the _send binding finish. When the data response is complete, the finish function is triggered
    const finish = FunctionPrototypeBind(onFinish, undefined.this);
    this._send(' '.'latin1', finish);
}

function write_(msg, chunk, encoding, callback, fromEnd) {
  // ...
  len = Buffer.byteLength(chunk, encoding);
  // ...
  if(! msg._header) {if(fromEnd) { msg._contentLength = len; }}/ /...
  _header is null. The _implicitHeader function is overridden in lib/_http_server.js. The _implicitHeader execution will assign a header+CRLF value to msg._header
  if(! msg._header) { msg._implicitHeader(); }// ...
    ret = msg._send(chunk, encoding, callback);
  // ...
}

OutgoingMessage.prototype._send = function _send(data, encoding, callback) {
  if (!this._headerSent) {
    if (typeof data === 'string' &&
        (encoding === 'utf8' || encoding === 'latin1'| |! encoding)) {The _implicitHeader function is generated as the _header assignment response header +CRLF, so the final value of data here is the response header +CRLF+ response body
      data = this._header + data;
    } else {
      const header = this._header;
      ArrayPrototypeUnshift(this.outputData, {
        data: header,
        encoding: 'latin1'.callback: null
      });
    }
    this._headerSent = true;
  }
  return this._writeRaw(data, encoding, callback);
};

OutgoingMessage.prototype._writeRaw = _writeRaw;
function _writeRaw(data, encoding, callback) {
  const conn = this.socket;
  // ...

  if (conn && conn._httpMessage === this && conn.writable) {
    // ...
    // The contents of the response are added to the response buffer and written out back to the user. When written out, the callback function is executed
    return conn.write(data, encoding, callback);
  }
  // ...
}
Copy the code

Res.end, when executed, has two main flows:

callwrite_Function, which first generates the response header and then stores the response header to_header, then regenerate the response content, and write the response content (response header +CRLF+ response body) to the user through socket.
callres._sendtosocket.writewritesfinishThe callback function is executed when the server response is fully written outfinishThe function,finishThe function is published internallyfinishEvents. There are two listening points in the programfinishEvents:
- parserOnIncomingGenerated in a functionresObject on which to listenfinishEvents;
- res.endSubscribed once in the functionfinishEvent, where the callback function is primarily a business code callres.endIs passed to the callback function.

// Response header content processing
// lib/_http_server.js
ServerResponse.prototype._implicitHeader = function _implicitHeader() {
  this.writeHead(this.statusCode);
};

ServerResponse.prototype.writeHead = writeHead;
function writeHead(statusCode, reason, obj) {
  // ...
  this._storeHeader(statusLine, headers);
  // ...
}

// lib/_http_outgoing.js
OutgoingMessage.prototype._storeHeader = _storeHeader;
function _storeHeader(firstLine, headers) {
  // ...
    this._last = true;
  // ...
  this._header = header + CRLF;
  this._headerSent = false;
  // ...
}
Copy the code

The _implicitHeader execution stores the response header +CRLF content in res._header. The response header has been processed and will be returned to the client with the response body when socket.write is required to respond to the request.

// lib/_http_server.js
function parserOnIncoming(server, socket, state, req, keepAlive) {
  // Notice that the finish event in the RES object is also subscribed
  res.on('finish',
         FunctionPrototypeBind(resOnFinish, undefined,
                               req, res, socket, state, server));
}

function resOnFinish(req, res, socket, state, server) {
  // Clear the req object stored in state
  ArrayPrototypeShift(state.incoming);
  clearRequestTimeout(req);
  clearIncoming(req);
  / / close the res
  process.nextTick(emitCloseNT, res);
  // Close the socket connection
  if (res._last) {
    if (typeof socket.destroySoon === 'function') {
      socket.destroySoon();
    } else {
      socket.end(); // The socket is disconnected}}}function emitCloseNT(self) {
  self.destroyed = true;
  self._closed = true;
  self.emit('close');
}
Copy the code

When the Finish event is triggered, the program first removes the buffered REq and RES objects and then closes the socket connection, and the client request is processed.

Node HTTP Server source code parsing

Using the example

HTTP. CreateServer and listen

Generate req and RES objects

Res. The end

Related Posts

Promise core logic, in-depth Jane out, see don’t understand beat to death me

A simple visualization tool to analyze the size of each version of the package file.

[Luffy]_ Full array II