Basic concepts of Node

1.1 What is a Node

Node.js is an open source and cross-platform JavaScript runtime environment. Run the V8 JavaScript engine (the core of Google Chrome) outside of the browser to improve performance using techniques such as event-driven, non-blocking, and asynchronous input/output models. Node.js is a server-side, non-blocking I/O, event-driven JavaScript runtime environment.

To understand Node, there are several basic concepts: non-blocking async and event-driven.

  • Non-blocking asynchrony: Nodejs uses a non-blocking I/O mechanism. When an I/O operation is performed, it does not cause any blocking. When the operation is completed, it is notified to execute the operation in the form of time. For example, after executing the code that accesses the database, it immediately moves to the code that follows, putting the code that handles the result returned by the database into the callback function, thus increasing the efficiency of the program.
  • Event-driven: Event-driven is when a new request comes in, the request will be pushed into an event queue, and then through a loop to detect the event status change in the queue. If the event status change is detected, the corresponding processing code will be executed, usually a callback function. For example, when a file is read, the corresponding state is triggered and processed by the corresponding callback function.

1.2 Application scenarios and disadvantages of Node

1.2.1 the pros and cons

Node.js is suitable for I/O intensive applications. The CPU usage of the application is still low when the application is in the running limit and most of the time is used for I/O disk memory reading and writing operations. Disadvantages are as follows:

  • Not suitable for CPU intensive applications
  • Only single-core cpus are supported and cpus cannot be fully utilized
  • Reliability is low, once a part of the code crashes, the whole system crashes

For the third point, a common solution is to use an Nnigx reverse proxy to open multiple processes to bind multiple ports, or to open multiple processes to listen on the same port.

1.2.1 Application Scenarios

After familiarizing ourselves with Nodejs’ advantages and disadvantages, we can see that it is suitable for the following application scenarios:

  • Good at I/O, bad at calculation. Because Nodejs is a single thread, too much computation (synchronization) will block the thread.
  • With a large number of concurrent I/ OS, there is no need for very complex processing within the application.
  • Cooperate with WeSocket to develop real-time interactive applications with long connections.

Specific application scenarios are as follows:

  1. User form collection system, background management system, real-time interaction system, examination system, networking software, high concurrency Web applications.
  2. Based on web, Canvas and other multiplayer online games.
  3. Web-based real-time chat client, chat room, live graphics.
  4. Single-page browser application.
  5. Operates databases and provides JSON-based apis for both front-end and mobile.

All objects in Node

In browser JavaScript, window is the global object, while in Nodejs the global object is global.

In NodeJS, it is not possible to define a variable in the outermost layer because all user code belongs to the current module and is only available in the current module, but can be passed outside the module through the use of exports objects. So, in NodeJS, variables declared with var are not global variables and only apply to the current module. A global object like the one above is in a global scope, and any global variable, function, or object is an attribute value of that object.

2.1 Common Global Objects

Common Node global objects include the following:

  • Class:Buffer
  • process
  • console
  • ClearInterval, setInterval
  • ClearTimeout, setTimeout
  • global

Class:Buffer can be used to process binary and non-Unicode encoded data. The raw data is stored in the Buffer Class instantiation. A Buffer is like an array of integers that is allocated memory in the V8 heap’s original storage space and cannot be resized once a Buffer instance is created.

Process A process represents a process object that provides information and control about the current process. If we need to pass a parameter during the execution of the Node program, we want to get the parameter in the process built-in object. For example, we have the following file:

process.argv.forEach((val, index) => {
   console.log(`${index}: ${val}`);
});
Copy the code

When we need to start a process, we can use the following command:

Nodeindex. js parameter...Copy the code

Console Console is used to print stdout and stderr. The most common one is console.log. The command to clear the console is console.clear. If you need to print the call stack of a function, you can use the command console.trace.

ClearInterval, setInterval setInterval is used to set timers. The syntax is as follows:

setInterval(callback, delay[, ...args])
Copy the code

The clearInterval is used to clear the timer, and the callback is repeated every delay millisecond.

ClearTimeout, setTimeout

Like setInterval, setTimeout is mainly used to set the delayers, while clearTimeout is used to clear the set delayers.

Global global is a global namespace object. We can put process, console, setTimeout, and so on in global, for example:

Console. log(process === global.process) // Prints trueCopy the code

2.2 Global objects in modules

In addition to system-supplied global objects, there are some that just appear in modules and look like global variables, like this:

  • __dirname
  • __filename
  • exports
  • module
  • require

__dirName The __dirname command is used to obtain the path of the current file, excluding the file name. For example, if you run node example.js in /Users/ MJR, it will print the following:

console.log(__dirname); // Prints: /Users/ MJRCopy the code

__filename The __filename command is used to obtain the path and filename of the current file, including subsequent file names. For example, if you run node example.js in /Users/ MJR, it prints the following:

console.log(__filename); // Prints: /Users/ MJR /example.jsCopy the code

Exports module. Exports exports the contents of a specified module, which can then be accessed using require().

exports.name = name; exports.age = age; exports.sayHello = sayHello;Copy the code

Require require is mainly used to import modules, JSON, or local files, which can be imported from node_modules. You can import local modules or JSON files using relative paths, which are processed according to the directory name defined by __dirname or the current working directory.

Third, talk about the understanding of process

3.1 Basic Concepts

As we know, process is the basic unit of resource allocation and scheduling in a computer system, the basis of the operating system structure, and the container for threads. When we start a JS file, it is actually open a service process, each process has its own independent space address, data stack, like another process can not access the variables of the current process, data structure, only after data communication, between processes can share data.

The Process object is a Node global variable that provides information about and controls the current Node.js process. Since JavaScript is a single-threaded language, there is only one main thread for starting a file through Node XXX.

3.2 Common Attributes and methods

Common attributes of process are as follows:

  • Process. env: Environment variables, such as’ process.env.node_env ‘to obtain configuration information about different environment items
  • Process. nextTick: This is often mentioned when talking about EventLoop
  • Process. pid: obtains the ID of the current process
  • Process. ppid: indicates the parent process of the current process
  • Process.cwd () : Gets the working directory of the current process
  • Process. platform: Obtains the operating system platform on which the process is running
  • Process.uptime () : indicates the running time of the current process, for example, the uptime value of the pm2 daemon

Process events: Process. on(‘ uncaughtException ‘,cb) catches the exception message, and process.on(‘ exit ‘, CB) pushes out the listener

  • Three standard streams: process.stdout standard output, process.stdin standard input, and process.stderr standard error output
  • Process. title: Specifies the name of the process. Sometimes you need to give a name to the process

Talk about your understanding of FS module

4.1 What is FS

Fs (filesystem) is a filesystem module that provides read and write capabilities for local files. It is basically a simple package of POSIX file operation commands. It can be said that all operations with files are implemented through the FS core module.

Before use, fs module needs to be imported as follows:

const fs = require('fs');
Copy the code

4.2 Basic File Knowledge

In the computer, there are some basic knowledge about files as follows:

  • Permission bit mode
  • Identify a flag
  • The file is described as fd

4.2.1 Permission bit mode

Permission is assigned to the file owner, file owning group, and other users. The permission types are divided into read, write, and execute users. If a file has the permission bit 4, 2, and 1, if a file does not have the permission bit 0. For example, on Linux, run the following command to view file permission bits:

Drwxr-xr-x 1 PandaShen 197121 0 Jun 28 14:41 core-rw-r --r-- 1 PandaShen 197121 293 Jun 23 17:44 indexCopy the code

In the first ten digits, D is a folder, and – is a file. The last nine digits represent the permissions of the current user, the user’s owning group, and other users. The three digits represent read (R), write (W), and execute (x) respectively.

4.2.2 identifies a

The identifier bits represent the operations on files, such as readable, writable, readable, writable, and so on, as shown in the following table:

4.2.3 File Description FD

The operating system assigns a numeric identifier called a file descriptor to each open file, and file operations use these file descriptors to identify and track each particular file.

The Window system uses a different but conceptually similar mechanism to track resources. For the user’s convenience, NodeJS abstracts the differences between operating systems and assigns numerical file descriptors to all open files.

In NodeJS, file descriptors are incremented each time a file is operated on. File descriptors usually start at 3, because there are three special descriptors: 0, 1, and 2. They represent process.stdin (standard input), process.stdout (standard output), and process.stderr (error output).

4.3 Common Methods

Since fs module is mainly used to operate files, common file operation methods are as follows:

  • File to read
  • File is written to
  • File appending
  • File copy
  • Create a directory

4.3.1 File Reading

Common file reading methods are readFileSync and readFile. ReadFileSync indicates synchronous reading, as follows:

const fs = require("fs");

let buf = fs.readFileSync("1.txt");
let data = fs.readFileSync("1.txt", "utf8");

console.log(buf); // <Buffer 48 65 6c 6c 6f>
console.log(data); // Hello
Copy the code
  • The first parameter is the path to read the file or the file descriptor.
  • The second argument is options. The default value is null, including encoding (null by default) and flag (flag bit by default r). The encoding can also be passed directly.

ReadFile is an asynchronous reading method. The first two parameters of readFile are the same as those of readFileSync. The last parameter is the callback function, which has two parameters err and data.

const fs = require("fs"); fs.readFile("1.txt", "utf8", (err, data) => { if(! err){ console.log(data); // Hello } });Copy the code

4.3.2 File Writing

File writing requires two methods, writeFileSync and writeFile. WriteFileSync indicates synchronous write, as shown below.

const fs = require("fs");

fs.writeFileSync("2.txt", "Hello world");
let data = fs.readFileSync("2.txt", "utf8");

console.log(data); // Hello world
Copy the code
  • The first parameter is the path to the file to be written to or the file descriptor.
  • The second argument is the data to be written, of type String or Buffer.
  • The third argument is options, which defaults to null. It contains encoding (utF8 by default), flag (w by default), and mode (0o666 by default). It can also be passed directly to encoding.

The first three parameters of writeFile are the same as the first three parameters of writeFileSync. The last parameter is the callback function, which contains one parameter err (error). The callback function is executed after the file is successfully written to the data.

const fs = require("fs");

fs.writeFile("2.txt", "Hello world", err => {
    if (!err) {
        fs.readFile("2.txt", "utf8", (err, data) => {
            console.log(data);       // Hello world
        });
    }
});
Copy the code

4.3.3 Appending Files

AppendFileSync and appendFile are used to append files to files. AppendFileSync indicates synchronous write, as follows.

const fs = require("fs");

fs.appendFileSync("3.txt", " world");
let data = fs.readFileSync("3.txt", "utf8");
Copy the code
  • The first parameter is the path to the file to be written to or the file descriptor.
  • The second argument is the data to be written, of type String or Buffer.
  • The third argument is options, which defaults to null, and contains encoding (utF8 by default), flag (flag bit by default), and mode (permission bit by default, 0o666), which can also be passed directly to encoding.

AppendFile: async appendFile: async appendFile: async appendFile: asynC appendFile: asynC appendFile: asynC appendFile: asynC appendFile: asynC appendFile: asynC appendFile: asynC

const fs = require("fs");

fs.appendFile("3.txt", " world", err => {
    if (!err) {
        fs.readFile("3.txt", "utf8", (err, data) => {
            console.log(data); // Hello world
        });
    }
});
Copy the code

4.3.4 Creating a Directory

There are two methods for creating directories: mkdirSync and mkdir. MkdirSync is created synchronously. The parameter is the path of a directory. No return value is returned.

Fs.mkdirsync ("a/b/c");Copy the code

Mkdir is created asynchronously and the second parameter is a callback function, as shown below.

fs.mkdir("a/b/c", err => { if (! Err) console.log(" created successfully "); });Copy the code

Talk about your understanding of Stream

5.1 Basic Concepts

A Stream is a means of data transmission and an end-to-end exchange of information. It is sequential. It reads data and processes content block by block and is used to read input or write output sequentially. In Node, a Stream is divided into three parts: Source, Dest, and Pipe.

Pipe (dest); pipe(dest); pipe(dest);

5.2 Traffic Classification

In Node, streams can be divided into four categories:

  • Writable stream: A stream that can write data, such as fs.createWritestream (), which can be used to write data to a file.
  • Readable stream: a stream that can read data, such as fs.createreadstream (), which reads content from a file.
  • Duplex stream: a stream that is both readable and writable, such as net.socket.
  • Conversion stream: A stream that can modify or transform data as it is written and read. For example, in a file compression operation, you can write compressed data to a file and read decompressed data from a file.

In the HTTP server module of Node, request is a readable stream and Response is a writable stream. For the FS module, it can handle both readable and writable file streams. Both readable and writable streams are one-way and easier to understand. The Socket is bidirectional and can be read and written.

5.2.1 duplex flow

In Node, the most common full-duplex communication is websocket, because both sender and receiver are independent methods, and there is no relationship between sending and receiving.The basic usage method is as follows:

const { Duplex } = require('stream'); const myDuplex = new Duplex({ read(size) { // ... }, write(chunk, encoding, callback) { // ... }});Copy the code

5.3 Application Scenarios

Common usage scenarios for streams are:

  • The GET request returns the file to the client
  • File operations
  • The low-level operations of some packaging tools

5.3.1 Network Request

A common use scenario for streams is network requests, such as a stream that returns a file. Res is also a Stream object that returns file data through a PIPE pipe.

const server = http.createServer(function (req, res) { const method = req.method; // get request if (method === 'get ') {const fileName = path.resolve(__dirname, 'data.txt'); let stream = fs.createReadStream(fileName); stream.pipe(res); }}); server.listen(8080);Copy the code

5.3.2 File Operations

File reading is also a stream operation, creating a readStream and a writeStream, and passing the data through the pipe.

Const fs = require('fs') const path = require('path') // 'data.txt') const fileName2 = path.resolve(__dirname, 'data-bak.txt') // Read file stream const readStream = fs.createreadstream (fileName1 CreateWriteStream = fs.createWritestream (fileName2) // Pipe to execute copy, readstream. pipe(writeStream) // Readstream. on('end', function () {console.log(' copy done ')})Copy the code

In addition, some packaging tools, such as Webpack and Vite, involve a lot of streaming.

6. Event cycle mechanism

6.1 What is a Browser Event Loop

Node.js maintains an event queue in the main thread. When a request is received, it places the request in the queue as an event and continues to receive other requests. When the main thread is idle (when there are no requests for access), it starts to loop the event queue and check whether there are any events in the queue to be processed. There are two ways to do this: if it is a non-I /O task, it is handled by itself and returns to the upper level call through the callback function; If it is an I/O task, a thread is pulled from the thread pool to handle the event, the callback function is specified, and the rest of the queue continues to loop.

When the I/O task in the thread completes, the specified callback function is executed, and the completed event is placed at the end of the event queue, waiting for the event loop. When the main thread loops through the event again, it is processed directly and returned to the upper level call. This process is called an Event Loop, and it works as shown below.

From left to right, and from top to bottom, Node.js is divided into four layers: the application layer, the V8 engine layer, the Node API layer, and the LIBUV layer.

  • Application layer: JavaScript interaction layer, common node.js modules, such as HTTP, FS
  • V8 engine layer: The V8 engine parses JavaScript syntax and interacts with the underlying API
  • Node API layer: Provides system calls for upper-layer modules, usually implemented in C language, to interact with the operating system.
  • LIBUV layer: it is the low-level encapsulation of cross-platform, realizing the event loop, file operation, etc., and is the core of Node.js asynchronous implementation.

In Node, what we call event loops are implemented based on Libuv, a multi-platform library that focuses on asynchronous IO. The EVENT_QUEUE above looks like it has only one queue, but EventLoop actually has six phases, each with a corresponding fifO callback queue.

6.2 Six stages of the Event cycle

The event cycle can be divided into six phases, as shown below.

  • Timers: Performs the callback of timer (setTimeout, setInterval) in this stage.
  • I/O event callback phase (I/O callbacks) : I/O callbacks that are deferred until the next iteration of the loop, i.e., some I/O callbacks that were not executed in the previous loop.
  • Idle and prepare: Used for internal use only.
  • Poll: Retrieves new I/O events; Perform I/ O-related callbacks (in almost all cases, except for closed callback functions, those scheduled by timers and setImmediate()), where Node will block at the appropriate time.
  • Check phase: The setImmediate() callback is performed here
  • Close callback: Some closed callback functions, such as socket.on(‘close’,…)

Each phase corresponds to a queue, and when the event loop enters a phase, the callback will be executed within that phase until the queue is exhausted or the maximum number of callbacks has been executed, and the next phase of processing will proceed, as shown in the figure below.

Seven, EventEmitter

7.1 Basic Concepts

As mentioned above, Node uses EventEmitter, and EventEmitter is the foundation for Node to implement EventEmitter. Based on EventEmitter, almost all modules of Node inherit this class. These modules have their own events, can bind, trigger listeners, and implement asynchronous operations.

Many objects in Node.js emit events, such as the fs.readStream object, which fires an event when a file is opened. These EventEmitter objects are instances of events.EventEmitter, which binds one or more functions to named events.

7.2 Basic Usage

The Events module of Node only provides the EventEmitter class, which implements the basic pattern of Node’s asynchronous event-driven architecture: the observer pattern.

In this mode, the observed (subject) maintains a set of observers sent (registered) by other objects, registers the observer when new objects are interested in the subject, unsubscribes when they are not, and notifies the observer in turn when the subject updates, using the following pattern.

const EventEmitter = require('events') class MyEmitter extends EventEmitter {} const myEmitter = new MyEmitter() Function callback() {console.log(' Trigger event! ') } myEmitter.on('event', callback) myEmitter.emit('event') myEmitter.removeListener('event', callback);Copy the code

In the above code, we register an event named Event through the on method of the instance object, fire the event through the EMIT method, and removeListener is used to unlisten for the event.

In addition to the methods described above, other common methods are as follows:

  • Emitter. AddListener /on(eventName, listener) : Adds a listener event of type eventName to the end of the event array.
  • Emitter. PrependListener (eventName, Listener) : Adds a listener of type eventName to the header of the event array.
  • Emitter. Emit (eventName[,…args]) : Emits an eventName monitoring event.
  • Emitter. RemoveListener/off (eventName, listener) : remove type eventName listen for an event.
  • Emitter. Once (eventName, listener) : Adds a listener whose type is eventName. It can be executed once and deleted.
  • Emitter. RemoveAllListeners ([eventName]) : removes all types for eventName listen for an event.

7.3 Implementation Principles

EventEmitter is a constructor that contains an object containing all events.

class EventEmitter { constructor() { this.events = {}; }}Copy the code

Where, the function structure of listening events stored in events is as follows:

{
  "event1": [f1,f2,f3]，
  "event2": [f4,f5]，
  ...
}
Copy the code

The first argument is the type of the event, and the second argument is the argument that triggers the event function. The implementation is as follows:

emit(type, ... args) { this.events[type].forEach((item) => { Reflect.apply(item, this, args); }); }Copy the code

After the emit method is implemented, three instance methods, on, addListener, and prependListener, are successively implemented, which add event listener triggering functions.

on(type, handler) { if (! this.events[type]) { this.events[type] = []; } this.events[type].push(handler); } addListener(type,handler){ this.on(type,handler) } prependListener(type, handler) { if (! this.events[type]) { this.events[type] = []; } this.events[type].unshift(handler); }Copy the code

To remove event listeners, use the removeListener/on method.

removeListener(type, handler) { if (! this.events[type]) { return; } this.events[type] = this.events[type].filter(item => item ! == handler); } off(type,handler){ this.removeListener(type,handler) }Copy the code

Implement the once method, encapsulate it when you pass in the event listener handler, use the closure properties to maintain the current state, and use the fired attribute value to determine if the event function has been executed.

once(type, handler) { this.on(type, this._onceWrap(type, handler, this)); } _onceWrap(type, handler, target) { const state = { fired: false, handler, type , target}; const wrapFn = this._onceWrapper.bind(state); state.wrapFn = wrapFn; return wrapFn; } _onceWrapper(... args) { if (! this.fired) { this.fired = true; Reflect.apply(this.handler, this.target, args); this.target.off(this.type, this.wrapFn); }}Copy the code

Here is the completed test code:

class EventEmitter { constructor() { this.events = {}; } on(type, handler) { if (! this.events[type]) { this.events[type] = []; } this.events[type].push(handler); } addListener(type,handler){ this.on(type,handler) } prependListener(type, handler) { if (! this.events[type]) { this.events[type] = []; } this.events[type].unshift(handler); } removeListener(type, handler) { if (! this.events[type]) { return; } this.events[type] = this.events[type].filter(item => item ! == handler); } off(type,handler){ this.removeListener(type,handler) } emit(type, ... args) { this.events[type].forEach((item) => { Reflect.apply(item, this, args); }); } once(type, handler) { this.on(type, this._onceWrap(type, handler, this)); } _onceWrap(type, handler, target) { const state = { fired: false, handler, type , target}; const wrapFn = this._onceWrapper.bind(state); state.wrapFn = wrapFn; return wrapFn; } _onceWrapper(... args) { if (! this.fired) { this.fired = true; Reflect.apply(this.handler, this.target, args); this.target.off(this.type, this.wrapFn); }}}Copy the code

8. Middleware

8.1 Basic Concepts

Middleware is a kind of software between application system and system software. It uses basic services (functions) provided by system software to connect various parts of application system or different applications on the network, so as to achieve the purpose of resource sharing and function sharing. In Node, middleware primarily refers to methods that encapsulate the details of HTTP request handling. For example, in Web frameworks such as Express and KOA, the essence of middleware is a callback function, and the parameters include the request object, the response object and the function to execute the next middleware. The architecture diagram is as follows.

Typically, within these middleware functions, we can execute business logic code, modify request and response objects, return response data, and so on.

8.2 koa

Koa is a popular Web framework based on Node. It does not support many functions. All functions can be extended through middleware. Koa does not bundle any middleware, but rather provides an elegant way for developers to write server-side applications quickly and happily.

Koa middleware uses the Onion ring model, passing in two parameters each time the next middleware executes:

  • CTX: encapsulates the request and response variables
  • Next: Goes to the next middleware function to execute

From the previous introduction, we know that Koa middleware is essentially a function, which can be either async or normal. Middleware encapsulation for KOA is as follows:

// async app.use(async (CTX, next) => {const start = date.now (); await next(); const ms = Date.now() - start; console.log(`${ctx.method} ${ctx.url} - ${ms}ms`); }); // App. use((CTX, next) => {const start = date.now (); return next().then(() => { const ms = Date.now() - start; console.log(`${ctx.method} ${ctx.url} - ${ms}ms`); }); });Copy the code

Of course, we can also use middleware to encapsulate several common functions during HTTP requests:

Token check

module.exports = (options) => async (ctx, Next) {try {// Obtain token const token = ctx.header.authorization if (token) {try {// verify function verify token, Await verify(token)} catch (err) {console.log(err)}} // enter the next middleware await next()} catch (err) { console.log(err) } }Copy the code

The log module

const fs = require('fs') module.exports = (options) => async (ctx, next) => { const startTime = Date.now() const requestTime = new Date() await next() const ms = Date.now() - startTime; let logout = `${ctx.request.ip} -- ${requestTime} -- ${ctx.method} -- ${ctx.url} -- ${ms}ms`; AppendFileSync ('./log.txt', logout + '\n')}Copy the code

There are many third-party middleware for Koa, such as KOA-BodyParser, KOA-static, etc.

8.3 Koa Middleware

Koa-bodyparser The KOA-BodyParser middleware converts our POST requests and form submission query strings into objects that are attached to ctx.request.body for us to value on other middleware or interfaces.

// file: my-koa-bodyparser.js const queryString = require(" queryString "); module.exports = function bodyParser() { return async (ctx, next) => { await new Promise((resolve, Reject) => {// Let dataArr = []; Ctx.req. on("data", data => dataarr.push (data)); Ctx.req. on("end", () => {// Get the requested data Type json or form let contentType = ctx.get(" content-type "); Let data = buffer.concat (dataArr).tostring (); let data = buffer.concat (dataArr).tostring (); If (contentType === "application/x-www-form-urlencoded") {// If form submission, Ctx.request. body ctx.request.body = queryString.parse (data); } else if (contentType === "applAction /json") { Body ctx.request.body = json.parse (data); ctx.request.body = json.parse (data); } // Execute the successful callback resolve(); }); }); // continue to await next(); }; };Copy the code

The purpose of the KOA-static middleware is to help us process static files when the server receives a request, for example.

const fs = require("fs"); const path = require("path"); const mime = require("mime"); const { promisify } = require("util"); // convert stat and access to Promise const stat = promisify(fs.stat); Const access = promisify(fs.access) module.exports = function (dir) {return async (CTX, next) => { / let realPath = path.join(dir, ctx.path); / let realPath = path.join(dir, ctx.path); Try {// get stat object let statObj = await stat(realPath); HTML if (statobj.isfile ()) {ctx.set(" content-type ", '${mime.getType()}; charset=utf8`); ctx.body = fs.createReadStream(realPath); } else { let filename = path.join(realPath, "index.html"); // If the file does not exist, execute next in catch and give another middleware to process await access(filename); Ctx. set(" content-type ", "text/ HTML; charset=utf8"); ctx.body = fs.createReadStream(filename); } } catch (e) { await next(); }}}Copy the code

In summary, when implementing middleware, the individual middleware should be simple enough, with a single responsibility, and the middleware code should be efficient, with repeated retrieval of data through caching when necessary.

9. How to design and implement JWT authentication

What is 9.1 JWT

JWT (JSON Web Token), essentially a string writing specification, is used for secure and reliable communication between the user and the server, as shown in the following figure.In the current development process of separating the front end from the back end, using token authentication mechanism for authentication is the most common solution, and the process is as follows:

  • When the server verifies that the user account and password are correct, it issues a token to the user. This token is used as a credential for subsequent user access to some interface.
  • Subsequent access will use this token to determine when the user is authorized to access.

The Token is divided into three parts: Header, Payload, Signature.Splicing. The header and payload are stored in JSON format, but are encoded, as shown in the following diagram.

9.1.1 the header

Each JWT is accompanied by header information, which mainly declares the algorithm used. The name of the field to declare the algorithm is ALG, and there is also a tyP field, JWT is the default. In the following example, the algorithm is HS256:

{  "alg": "HS256",  "typ": "JWT" } 
Copy the code

Since JWT is a string, we also need to Base64 encode the above, resulting in the following string:

eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9        
Copy the code

9.1.2 payload

The payload is the body of the message, which holds the actual content, namely the data declaration of the Token, such as the user id and name. By default, it also carries the issuing time of the Token, iAT. You can also set the expiration time, as follows:

{
  "sub": "1234567890",
  "name": "John Doe",
  "iat": 1516239022
}
Copy the code

After the same Base64 encoding, the string is as follows:

eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6IkpvaG4gRG9lIiwiaWF0IjoxNTE2MjM5MDIyfQ
Copy the code

9.1.3 Signature

In general, set a secretKey and perform HMACSHA25 algorithm on the results of the first two. The formula is as follows:

Signature = HMACSHA256(base64Url(header)+.+base64Url(payload),secretKey)
Copy the code

Therefore, even if the first two parts of the data are tampered with, as long as the encryption key used by the server is not leaked, the resulting signature will be inconsistent with the previous signature.

9.2 Design and Implementation

Typically, the use of tokens is divided into two parts: generating tokens and verifying tokens.

  • Token generation: After the login is successful, a token is issued.
  • Verify token: Verifies the token when accessing certain resources or interfaces.

9.2.1 generated token

With the help of third-party library JsonWebToken, a token is generated through jsonWebToken’s sign method. Sign takes three arguments:

  • The first parameter refers to Payload.
  • The second is the secret key, which is server-side specific.
  • The third parameter is option, which defines the token expiration time.

Here is an example of a front-end generation token:

const crypto = require("crypto"), jwt = require("jsonwebtoken"); Let userList = []; let userList = []; Class UserController {static async login(CTX) {const data = ctx.request.body; if (! data.name || ! data.password) { return ctx.body = { code: "000002", message: }} const result = userlist. find(item => item.name === data.name && item.password === = Crypto.createhash ('md5').update(data.password).digest('hex')) if (result) {// Generate token const token = jwt.sign({name: Result. name}, "test_token", // secret {expiresIn: 60 * 60} // expiration time: 60 * 60 s); Return ctx.body = {code: "0", message: "login succeeded ", data: {token}}; } else {return ctx.body = {code: "000002", message: "username or password error"}; } } } module.exports = UserController;Copy the code

After the front-end receives the token, it will generally be cached through localStorage and then put the token into the HTTP request header Authorization. Regarding the Authorization Settings, Bearer is required to be added in front of the Authorization, notice that there are Spaces behind as follows.

axios.interceptors.request.use(config => { const token = localStorage.getItem('token'); config.headers.common['Authorization'] = 'Bearer ' + token; // Check the Authorization return config; })Copy the code

9.2.2 validation token

First of all, we need to use KOA-JWT middleware for verification, which is relatively simple and can be verified before route jump, as follows.

App. Use (koajwt ({secret: 'test_token}). Unless ({/ / configure white list path: [/ \ \ / API/register /, / \ \ / API/login /]}))Copy the code

When using the KOA-JWT middleware for verification, note the following:

  • Secret must be consistent with sign.
  • Unless can be used to configure interface whitelists, i.e. which urls can not be verified, such as login/registration.
  • The verification middleware must be placed before the route to be verified, and the preceding URL cannot be verified.

The method for obtaining user token information is as follows:

Router.get ('/ API /userInfo', Async (CTX,next) =>{const authorization = ctx.header.authorization // Obtain JWT const token = authorization.replace('Beraer ','') const result = jwt.verify(token,'test_token') ctx.body = result }Copy the code

Note: the above HMA256 encryption algorithm is in the form of a single secret key, once disclosed, the consequences are very dangerous.

In a distributed system, each subsystem obtains a secret key from which it can issue and authenticate tokens, but some servers only need to authenticate tokens. At this time, asymmetric encryption can be adopted, using private key to issue tokens, public key to verify tokens, and the encryption algorithm can choose asymmetric algorithms such as RS256.

In addition, JWT authentication also needs to pay attention to the following points:

  • The payload part is simply encoded, so it can only be used to store non-sensitive information that is logically necessary.
  • Need to protect the encryption key, once leaked the consequences of unimaginable.
  • To avoid token hijacking, you are advised to use HTTPS.

X. Node performance monitoring and optimization

10.1 Node Optimization Points

Node, as a server language, is particularly important in terms of performance, which is generally measured as follows:

  • CPU
  • memory
  • I/O
  • network

10.1.1 CPU

For CPU indicators, pay attention to the following points:

  • CPU load: The total number of processes occupied and waiting for the CPU during a period of time.
  • CPU usage: CPU usage, equal to 1 – Idle Time/total CPU time.

These two indicators are used to evaluate the current CPU busy level of the system. Node applications generally do not consume much CPU. If the CPU usage is high, the application has many synchronous operations, blocking the callback of asynchronous tasks.

10.1.2 Memory Specifications

Memory is a very easy metric to quantify. Memory usage is a common indicator of a system’s memory bottleneck. The usage of the internal memory stack is also a quantifiable metric for Node, and you can use the following code to get data about memory:

// /app/lib/memory.js const os = require('os'); Const {RSS, heapUsed, heapTotal} = process.memoryUsage(); const {RSS, heapUsed, heapTotal} = process.memoryUsage(); // Get free system memory const sysFree = os.freemem(); Const sysTotal = os.totalmem(); // Get system memory const sysTotal = os.totalmem(); Module.exports = {memory: () => {return {sys: 1 - sysFree/sysTotal, // sys: 1 - sysFree/sysTotal: HeapUsed/headTotal, // Node heap memory usage Node: RSS/sysTotal, // Node heap memory usage ratio}}}Copy the code
  • RSS: indicates the total memory occupied by node processes.
  • HeapTotal: indicates the total heap memory.
  • HeapUsed: Actual heap memory usage.
  • External: the amount of memory used by external programs, including the C++ programs at the Node core.

The maximum memory capacity of a Node process is 1.5GB. Therefore, control the memory usage properly.

10.13 disk I/O

Hard disk I/O overhead is very expensive, hard disk I/O takes 164,000 times the CPU clock cycle of memory. Memory IO is much faster than disk IO, so caching data in memory is an effective optimization. Common tools include Redis, memcached, and so on.

In addition, not all data needs to be cached. Only the data with high access frequency and high generation cost should be considered. In other words, you should consider caching if it affects your performance bottleneck, and cache avalanche and cache penetration should be solved.

10.2 Monitoring

Performance monitoring is generally achieved by means of tools, such as Easy-Monitor and Ali Node performance platform.

Easy-monitor 2.0 is adopted here, which is a lightweight Node.js project kernel performance monitoring + analysis tool. In the default mode, only require once in the project entry file, without changing any business code to enable kernel-level performance monitoring analysis.

Easy-monitor is also relatively simple to use and is introduced in the project entry file as follows.

const easyMonitor = require('easy-monitor'); EasyMonitor (' Project name ');Copy the code

Open your browser and visit http://localhost:12333 to see the process interface. For more details, please refer to the official website

10.3 Node Performance Optimization

There are several ways to optimize Node performance:

  • Use the latest version of Node.js
  • Use streams correctly
  • Code level optimization
  • Memory management optimization

10.3.1 Using the latest Node.js version

The performance improvements in each release come from two main aspects:

  • V8 version update
  • Node.js internal code update optimization

10.3.2 Using streams correctly

In Node, many objects are streamed, and a large file can be sent as a stream without having to read it into memory.

const http = require('http'); const fs = require('fs'); Http. createServer(function (req, res) {fs.readFile(__dirname + '/data.txt', function (err, data) { res.end(data); }); }); Http.createserver (function (req, res) {const stream = fs.createreadStream (__dirname + '/data.txt'); stream.pipe(res); });Copy the code

10.3.3 Code Level optimization

Merge query, multiple queries merge once, reduce the number of database queries.

For user_id in userIds let account = user_account.findone (user_id) // Correct const user_account_map = {} // Note that this object will consume a lot of memory. user_account.find(user_id in user_ids).forEach(account){ user_account_map[account.user_id] = account } for user_id in userIds var account = user_account_map[user_id]Copy the code

10.3.4 Memory Management Optimization

In V8, there are two main memory generations: the new generation and the old generation:

  • Cenozoic: Objects live for a short time. Newborn objects or objects that have been garbage collected only once.
  • Old generation: The object lives for a long time. Objects that have undergone one or more garbage collections.

If the memory space of the new generation is insufficient, it is directly allocated to the old generation. By reducing memory footprint, you can improve server performance. If there is a memory leak, it can also cause a large number of objects to be stored in the old generation, which can greatly degrade server performance, as shown in the following example.

const buffer = fs.readFileSync(__dirname + '/source/index.htm'); app.use( mount('/', async (ctx) => { ctx.status = 200; ctx.type = 'html'; ctx.body = buffer; leak.push(fs.readFileSync(__dirname + '/source/index.htm')); })); const leak = [];Copy the code

When leaky memory is very large, it can cause a memory leak and should be avoided.

Reducing memory usage can significantly improve service performance. The best way to save memory is to use pooling, which stores frequently used, reusable objects and reduces creation and destruction. For example, if you have an image request interface, every time you make a request, you need to use the class. It is not appropriate to have to new these classes every time, because they are frequently created and destroyed in the event of a large number of requests, causing memory jitter. The object pool mechanism is used to save the frequently created and destroyed objects in an object pool, so as to avoid repeated initialization operations and improve the performance of the framework.