Practice of several HTTP file transfer schemes based on NodeJS

The HTTP file transfer scheme based on NodeJS plays a very important role in the development of front and back end full stack at present stage. In this paper, I will realize HTTP transfer of large files through several schemes. Before implementing the functionality, we write a large file via nodejs’ FS module and generate a local file in the project:

const fs = require('fs');
const writeStream = fs.createWriteStream(__dirname + "/file.txt");
for(let i = 0; i <=100000; i++) {
  writeStream.write(`${i}- I'm${i}The date file \ n `."utf-8");
}
writeStream.end();
Copy the code

After the above code runs successfully, a 3.2MB text file will be generated in the current execution directory, which will be used as the “large file material” for the following scheme. Before listing the large file transfer schemes, we’ll encapsulate the two public methods we’ll use later: the file read method and the file compression method:

// Encapsulate the method to read the file
const readFile = async (paramsData) => {
  return new Promise((resolve, reject) = > {
    fs.readFile(paramsData, (err, data) = > {
      if(err) {
        reject('File read error');
      } else{ resolve(data); }})})}// Encapsulate the file compression method
const gzip = async (paramsData) => {
  return new Promise((resolve, reject) = > {
    zlib.gzip(paramsData, (err, result) = > {
      if(err) {
        reject('File compression error');
      } else{ resolve(result); }})})}Copy the code

1. Transfer compressed data in large files

Browsers send requests with accept and Accept -* headers, which tell the server the file types supported by the current browser, the list of supported compression formats, and the supported languages. The accept-Encoding field in the request header is used to tell the server what Encoding (usually some compression algorithm) the client can understand. The server selects a method supported by the client and notifys the client of the choice via the content-encoding response header, which tells the browser that the returned JS script has been processed by the GZIP compression algorithm

/ / request header
accept-encoding: gzip, deflate, br 
Copy the code

/ / response headers
cache-control: max-age=2592000 
content-encoding: gzip 
content-type: application/x-javascript
Copy the code

Based on the accept-encoding and Content-Encoding fields, let’s verify that gzip is not enabled and that gzip is enabled.

// Implement a simple file reading server (without gzip enabled)
const server = http.createServer(async (req, res) => {
  res.writeHead(200, {
    "Content-Type": "text/plain; charset=utf-8"});const buffer = await readFile(__dirname + '/file.txt');
  res.write(buffer);
  res.end();
})
server.listen(3000.() = > {
  console.log('Server started successfully')})Copy the code

// Implement a simple file reading server (with gzip enabled)
const server = http.createServer(async(req, res) => {
  res.writeHead(200, {
    "Content-Type": "text/plain; charset=utf-8"."Content-Encoding": "gzip"
  });
  const buffer = await readFile(__dirname + '/file.txt');
  const gzipData = await gzip(buffer);
  res.write(gzipData);
  res.end();
})
server.listen(3000.() = > {
  console.log('Server started successfully')})Copy the code

2. Data is transmitted in blocks

Chunking can be used for scenarios where a large HTML table is generated using data obtained from a database query, or when a large number of images are transferred.

Transfer-Encoding: chunked
Transfer-Encoding: gzip, chunked
Copy the code

The value of the transfer-Encoding field in the response header is chunked, indicating that the data is sent in a series of chunks. Note that transfer-Encoding and Content-Length fields are mutually exclusive, that is, the two fields cannot appear at the same time in the response packet.

// Data is transferred in blocks
const spilitChunks = async() = > {const buffer = await readFile(__dirname + '/file.txt');
  const lines = buffer.toString('utf-8').split('\n');
  let [chunks, i, n] = [[], 0, lines.length];
  while(i < n) {
    chunks.push(lines.slice(i, i+= 10));
  };
  return chunks;
}
const server = http.createServer(async(req, res) => {
  res.writeHead(200, {
    "Content-Type": "text/plain; charset=utf-8"."Transfer-Encoding": "chunked"."Access-Control-Allow-Origin": "*"});const chunks = await spilitChunks();
  for(let i =0; i< chunks.length; i++) {
    setTimeout(() = > {
      let content = chunks[i].join("&");
      res.write(`${content.length.toString(16)}\r\n${content}\r\n`);
    }, i * 1000);
  }
  setTimeout(() = > {
    res.end();
  }, chunks.length * 1000);
})
server.listen(3000.() = > {
  console.log('Server started successfully')})Copy the code

2. Data is transmitted through data streams

When using Node.js to return a large file to the client, using a stream to return the file stream can avoid processing large files and occupy too much memory. The specific implementation is as follows. When file data is returned as a stream, the value of the TRANSFER-Encoding field of the HTTP response header is chunked, indicating that the data is sent in a series of chunks.

const server = http.createServer((req, res) = > {
  res.writeHead(200, {
    "Content-Type": "text/plain; charset=utf-8"."Content-Encoding": "gzip"."Transfer-Encoding": "chunked"
  });
  fs.createReadStream(__dirname + "/file.txt")
    .setEncoding("utf-8")
    .pipe(zlib.createGzip())
    .pipe(res);
})

server.listen(3000.() = > {
  console.log('Server started successfully')})Copy the code