Pure of heart, life is full of sweet and joy. — Leo Tolstoy

Socket hang up first appeared in a service load test and was later resolved. Recently, this problem was reported again when node.js service migrated to K8S container. After checking the cause, it was found that the container’S CPU and memory size were limited. Here is a summary of what Socket hang up is, when it happens, and how to solve it.

About the author: May Jun, Nodejs Developer, moOCs certified author, love technology, like sharing, welcome to pay attention to Nodejs technology stack and Github open source project www.nodejs.red

What is a Socket hang up

What is a Socket hang up?

Socket hang up means that the socket (link) is hung up. No matter which language you use, you should have encountered it more or less, but have you ever thought about why? For example, in Node.js, the system provides a default timeout of 2 minutes for the HTTP server. If a request exceeds this time, the HTTP server will close the request. When a client tries to return a request and finds that the socket has been “hung up”, it sends a socket hang up error.

To understand a problem, or to practice more, the following is a small demo of the problem and then combine with node.js HTTP related source code to further understand Socket hang up? Also recommend that you look at the stack overflow of the universal also has a discussion on the issue above stackoverflow.com/questions/1… .

Replicating Socket hang up

The service side

To start an HTTP service, define the /timeout interface to delay the response for 3 minutes

const http = require('http');
const port = 3020;

const server = http.createServer((request, response) = > {
    console.log('request url: ', request.url);

    if (request.url === '/timeout') {
        setTimeout(function() {
            response.end('OK! ');
        }, 1000 * 60 * 3)
    }
}).listen(port);

console.log('server listening on port ', port);
Copy the code

The client

const http = require('http');
const opts = {
  hostname: '127.0.0.1'.port: 3020.path: '/timeout'.method: 'GET'}; http.get(opts, (res) => {let rawData = ' ';
  res.on('data', (chunk) => { rawData += chunk; });
  res.on('end', () = > {try {
      console.log(rawData);
    } catch (e) {
      console.error(e.message); }}); }).on('error', err => {
  console.error(err);
});
Copy the code

After starting the server and then starting the client about 2 minutes later or directly killing the server, the following error is reported. You can see the corresponding error stack

Error: socket hang up
    at connResetException (internal/errors.js:570:14)
    at Socket.socketOnEnd (_http_client.js:440:23)
    at Socket.emit (events.js:215:7)
    at endReadableNT (_stream_readable.js:1183:12)
    at processTicksAndRejections (internal/process/task_queues.js:80:21) {
  code: 'ECONNRESET'
}
Copy the code

The node.js HTTP client source code does not receive any response, so it is considered that the socket has ended. A connResetException(‘socket hang up’) error is therefore emitted at L440.

// https://github.com/nodejs/node/blob/v12.x/lib/_http_client.js#L440

function socketOnEnd() {
  const socket = this;
  const req = this._httpMessage;
  const parser = this.parser;

  if(! req.res && ! req.socket._hadError) {// If we don't have a response then we know that the socket
    // ended prematurely and we need to emit an error on the request.
    req.socket._hadError = true;
    req.emit('error', connResetException('socket hang up'));
  }
  if (parser) {
    parser.finish();
    freeParser(parser, req, socket);
  }
  socket.destroy();
}
Copy the code

Socket hang up Solution

1. Set the TIMEOUT period of the HTTP Server socket

By default, the timeout value of the server is 2 minutes. If it does, the socket will automatically destroy itself. You can use the server.setTimeout(msecs) method to set the timeout to a larger value. Passing a 0 will turn off the timeout mechanism

// https://github.com/nodejs/node/blob/v12.x/lib/_http_server.js#L348
function Server(options, requestListener) {
  // ...

  this.timeout = kDefaultHttpServerTimeout; // The default value is 2 x 60 x 1000
  this.keepAliveTimeout = 5000;
  this.maxHeadersCount = null;
  this.headersTimeout = 40 * 1000; // 40 seconds
}
Object.setPrototypeOf(Server.prototype, net.Server.prototype);
Object.setPrototypeOf(Server, net.Server);


Server.prototype.setTimeout = function setTimeout(msecs, callback) {
  this.timeout = msecs;
  if (callback)
    this.on('timeout', callback);
  return this;
};
Copy the code

The modified code looks like this:

const server = http.createServer((request, response) = > {
    console.log('request url: ', request.url);

    if (request.url === '/timeout') {
        setTimeout(function() {
            response.end('OK! ');
        }, 1000 * 60 * 3)
    }
}).listen(port);

server.setTimeout(0); // Set the timeout period
Copy the code

If you do not set setTimeout, you can also catch such errors on the HTTP client, put them in the queue and initiate retry. If the probability of such errors is high, you should check whether the corresponding service has abnormal problems such as slow processing.

ECONNRESET VS ETIMEDOUT

Note the difference between ECONNRESET and ETIMEDOUT

ECONNRESET is read timeout. {“code”:”ECONNRESET”} error occurs when the server is too slow to respond properly, such as the socket hang up example described above.

ETIMEDOUT refers to the timeout that occurs when a client initiates a connection with a remote server. Here is an example of a request from the Request module.

const request = require('request');

request({
  url: 'http://127.0.0.1:3020/timeout'.timeout: 5000,
}, (err, response, body) => {
  console.log(err, body);
});
Copy the code

In the above example, {code: ‘ETIMEDOUT’} error is reported after approximately 5 seconds, and the stack is as follows:

Error: ETIMEDOUT
    at Timeout._onTimeout (/Users/test/node_modules/request/request.js:677:15)
    at listOnTimeout (internal/timers.js:531:17)
    at processTimers (internal/timers.js:475:7) {
  code: 'ETIMEDOUT'
}
Copy the code