Drill into node.js's module loading mechanism by writing the require function

Module is a very basic and important concept in Node.js. Various native libraries are provided by module, and third-party libraries are managed and referenced by module. This paper will start from the basic module principle, and finally we will use this principle to implement a simple module loading mechanism, that is, to implement a REQUIRE by ourselves.

The full code for this article has been uploaded to GitHub:Github.com/dennis-jian…

A simple example

As always, before we talk about the principles, let’s do a simple example, and we’ll start with that example and work our way through the principles. Module. exports allows you to export js objects of almost any type, including strings, functions, objects, arrays and so on. Let’s create an A.js and export the simplest hello world:

// a.js 
module.exports = "hello world";
Copy the code

And then another b.js to export a function:

// b.js
function add(a, b) {
  return a + b;
}

module.exports = add;
Copy the code

Js and require them. The require function returns the value of the module. Exports file:

// index.js
const a = require('./a.js');
const add = require('./b.js');

console.log(a);      // "hello world"
console.log(add(1.2));    // b produces an addition function, which can be used directly. The result of this line is 3
Copy the code

Require will run the target file first

Module. Exports = XXX is just one line of code that, as we’ll see later, modifs the module’s exports property. For example, let’s have another C.JS:

// c.js
let c = 1;

c = c + 1;

module.exports = c;

c = 6;
Copy the code

Exports = module. Exports = c; exports = module. Exports = c; exports = module. C = 2 c = 2 c = 2 c = 2 C = 2 C = 2 C = 6

const c = require('./c.js');

console.log(c);  // the value of c is 2
Copy the code

C = 6; c = 6; c = 6; Module. exports does not affect what if it is a reference type? Let’s try it straight away:

// d.js
let d = {
  num: 1
};

d.num++;

module.exports = d;

d.num = 6;
Copy the code

Then require it in index.js:

const d = require('./d.js');

console.log(d);     // { num: 6 }
Copy the code

We found that assigning d.num after module.exports still worked because d is an object and a reference type that we can modify. Module. exports allows you to change the value of a reference type outside the module, such as index.js:

const d = require('./d.js');

d.num = 7;
console.log(d);     // { num: 7 }
Copy the code

`require`and`module.exports`Not black magic

We can see through the previous example, the require and module exports do not complex, we first assume that there is a global object {}, initial condition is empty, when you require a file, will take out this file, If module.exports exists in this file, add the value of module.exports to the object when running this line of code, and the key is the corresponding file name. The object looks like this:

{
  "a.js": "hello world"."b.js": function add(){},
  "c.js": 2."d.js": { num: 2}}Copy the code

When you require a file again, if the object has a value in it, it will be returned to you. If not, repeat the previous steps, execute the object and add its module.exports to the global object and return it to the caller. This global object is what we often hear about as a cache. There’s nothing dark about require and module.exports, they just run and fetch the value of the object file and add it to the cache. Because d.js is a reference type, you can change the value of the reference anywhere you get it. If you don’t want the value of your module to change, you need to write the module to handle it, for example, use object.freeze (). Methods like object.defineProperty ().

Module type and loading sequence

This section is full of concepts, which are a bit boring, but also something we need to understand.

Module type

There are several types of Node.js modules. The previous ones are actually file modules. In summary, there are two main types:

Built-in module: is what Node.js provides natively, for examplefs.httpWait, these modules are loaded when the Node.js process starts.

File module: The modules we wrote earlier, and the third party modules, i.enode_modulesThe following modules are file modules.

Load order

The loading order refers to the order in which and where to find X when we require(X). There are detailed pseudocodes in the official document, which can be summarized as follows:

The built-in modules are loaded preferentially, even if there is a file with the same name.

Not a built-in module. Go to the cache first.

If the cache does not exist, find the file in the corresponding path.

If no corresponding file exists, load this path as a folder.

If you can’t find any files or folders, gonode_modulesFind below.

I can’t find it yet.

Load folders

If you can’t find a file, find a folder, but it is not possible to load the entire folder. There is also a loading order for the folder:

Let’s see if there’s anything under this folderpackage.jsonIf there are, look insidemainFields,mainIf the field has a value, load the corresponding file. So if you’re looking at some third-party library source code and you can’t find the entry, look it uppackage.jsonThe inside of themainFields, for examplejquerythemainThe field looks like this:"main": "dist/jquery.js".

If there is nopackage.jsonorpackage.jsonThere is nomainJust looking forindexFile.

If you can’t find either of these steps, you’re reporting an error.

Supported file types

Require supports three main file types:

.js:.jsFile is the most commonly used file type, loading will first run the entire JS file, and then the above mentionedmodule.exportsAs arequireThe return value of.

.json:.jsonFile is a plain text file, used directlyJSON.parseJust return it as an object.

.node:.nodeFiles are C++ compiled binaries, a type that pure front ends rarely touch.

handwritten`require`

Now that we’ve covered the principle, let’s go to the big one, which is implementing a require. The implementation of require is the implementation of the entire node.js module loading mechanism, we need to solve the problem:

Find the corresponding file by the path name passed in.

Execute the found file and inject at the same timemoduleandrequireThese methods and properties are used by the module file.

Return module’smodule.exports

This article’s handwritten code all refer to node.js official source code, function name and variable name as far as possible to keep consistent, in fact, is a condensed version of the source code, we can compare to see, when writing specific methods I will also affix the corresponding source address. The overall code is in this file: github.com/nodejs/node…

The Module class

Node.js Module loading functions are all in the Module class, the whole code uses the idea of object-oriented, if you are not very familiar with the OBJECT-ORIENTED JS can see this article first. The constructor of the Module class is also not complicated. It mainly initializes some values. To distinguish it from the official Module name, we call our own class MyModule:

function MyModule(id = ' ') {
  this.id = id;       // This id is the path to require
  this.path = path.dirname(id);     // Path is the built-in Node.js module that gets the folder path of the passed parameter
  this.exports = {};        // The exported object is placed here and initialized as an empty object
  this.filename = null;     // The file name of the module
  this.loaded = false;      // loaded indicates whether the current module is loaded
}
Copy the code

Method the require

We have been using the require is actually an instance of the Module class method, the content is very simple, first do some parameters check, and then call the module. _load method, source code see here: github.com/nodejs/node… . The code for the lite version is as follows:

MyModule.prototype.require = function (id) {
  return Module._load(id);
}
Copy the code

MyModule._load

Mymodule. _load is a static method, which is the real body of the require method, and what it actually does is:

First check whether the requested module already exists in the cache, if so, return the cache module directlyexports.

If it’s not in the cache, thennewaModuleInstance, with which to load the corresponding module and return the module’sexports.

Create (null) is used to initialize the Object. Create (null), so that the created prototype points to NULL. Let’s do the same:

MyModule._cache = Object.create(null);

MyModule._load = function (request) {    // Request is the road force parameter we pass in
  const filename = MyModule._resolveFilename(request);

  // Check the cache first. If the cache exists and has been loaded, return the cache directly
  const cachedModule = MyModule._cache[filename];
  if(cachedModule ! = =undefined) {
    return cachedModule.exports;
  }

  // If the cache does not exist, we load the module
  // Create an instance of MyModule before loading it and call the instance method load to load it
  Module.exports is returned directly after loading
  const module = new MyModule(filename);
  
  // Load caches the module before, so that if there are any circular references it will be returned to the cache, but the exports in the cache may not be present or complete
  MyModule._cache[filename] = module;
  
  module.load(filename);
  
  return module.exports;
}
Copy the code

The above code corresponds to the source code here: github.com/nodejs/node…

Mymodule._resolvefilename and myModule.prototype. load are also called from the source code.

`MyModule._resolveFilename`

Mymodule. _resolveFilename () ¶ Mymodule. _resolveFilename (); mymodule. _resolveFilename (); Built-in modules, relative paths, absolute paths, folders, third-party modules, etc. If it’s a folder or third-party module, parse package.json and index.js inside. Here we mainly talk about the principle, so we only implement relative path and absolute path to find files, and support automatic add JS and JSON suffix two:

MyModule._resolveFilename = function (request) {
  const filename = path.resolve(request);   // Get the absolute path of the passed argument
  const extname = path.extname(request);    // Get the file name extension

  // If there is no file suffix, try adding.js and.json
  if(! extname) {const exts = Object.keys(MyModule._extensions);
    for (let i = 0; i < exts.length; i++) {
      const currentPath = `${filename}${exts[i]}`;

      // If the concatenated file exists, return the concatenated path
      if (fs.existsSync(currentPath)) {
        returncurrentPath; }}}return filename;
}
Copy the code

There is also a static variable myModule._extensions in the source code. This variable is used to store the corresponding processing methods for various files, which we will implement later.

Mymodule. _resolveFilename: github.com/nodejs/node…

`MyModule.prototype.load`

Mymodule.prototype. load is an instance method that actually loads modules. This is also an entry point for loading different types of files that correspond to a method in myModule._extensions:

MyModule.prototype.load = function (filename) {
  // Get the file name extension
  const extname = path.extname(filename);

  // Call the corresponding postfix handler to process it
  MyModule._extensions[extname](this, filename);

  this.loaded = true;
}
Copy the code

Note that this in this code refers to the Module instance, because it is an instance method. See the corresponding source here: github.com/nodejs/node…

Load js files: myModule._extensions [‘.js’]

Mymodule._extensions can be used to load files of.js type. Mymodule._extensions can be used to load files of.js type.

MyModule._extensions['.js'] = function (module, filename) {
  const content = fs.readFileSync(filename, 'utf8');
  module._compile(content, filename);
}
Copy the code

As you can see, the loading method of JS is very simple, just read out the contents of the file, and then call another instance method _compile to execute it. See the corresponding source here: github.com/nodejs/node…

Compile execute js file: mymodule.prototype._compile

Mymodule.prototype. _compile is the core method for loading JS files, and it is also the most commonly used method. This method requires the object file to be taken out and executed, and the entire code to be wrapped before execution. To inject exports, require, module, __dirname, __filename, which is why we can use these variables directly in JS files. It’s not hard to do this either, if the file we require is a simple Hello World that looks like this:

module.exports = "hello world";
Copy the code

How do we inject the module variable into it? The answer is to add a layer of functions on top of it to make it look like this:

function (module) { // Inject the module variable
  module.exports = "hello world";
}
Copy the code

So if we take the contents of the file as a string, in order for it to look like this, we need to concatenate the beginning and end of the file. We put the beginning and end in an array:

MyModule.wrapper = [
  '(function (exports, require, module, __filename, __dirname) { '.'\n}); '
];
Copy the code

Note that we have an extra () wrap at the beginning and end of the concatenation, so that we can get the anonymous function later, and then add another () to pass the argument to execute. Then concatenate the function to be executed into the middle of the method:

MyModule.wrap = function (script) {
  return MyModule.wrapper[0] + script + MyModule.wrapper[1];
};
Copy the code

The mymodule. wrap wrapper gets exports, require, module, __filename, __dirname. Mymodule.prototype. _compile: mymodule.prototype._compile

MyModule.prototype._compile = function (content, filename) {
  const wrapper = Module.wrap(content);    // Get the wrapped function body

  // THE VM is the NodeJS virtual machine sandbox module, and the runInThisContext method can take a string and convert it to a function
  // The return value is the converted function, so compiledWrapper is a function
  const compiledWrapper = vm.runInThisContext(wrapper, {
    filename,
    lineOffset: 0.displayErrors: true});// exports, require, module, __filename, __dirname
  // exports can be used directly with module.exports, i.e. This. exports
  // call this.require; // call this.require
  // module is this
  // __filename directly uses the filename argument passed in
  // Get the __dirname from filename
  const dirname = path.dirname(filename);

  compiledWrapper.call(this.exports, this.exports, this.require, this,
    filename, dirname);
}
Copy the code

Note the several arguments we inject and this passed through call:

this:compiledWrapperIs through thecallCall, the first argument is insidethisSo here we pass in thetathis.exports, that is,module.exportsThat is to say, wejsInside the filethisIs themodule.exportsA reference to.

exports: compiledWrapperThe first parameter formally accepted isexportsSo do wethis.exports, sojsin-fileexportsIs themodule.exportsA reference to.

require: This method we pass isthis.requireIn fact, it isMyModule.prototype.require, that is,MyModule._load.

module: What we pass in isthis, which is an instance of the current module.

__filename: indicates the absolute path of the file.

__dirname: absolute path of the file folder.

Here, our JS file has actually recorded the end, the corresponding source see here :github.com/nodejs/node…

Load json file: myModule._extensions [‘.json’]

Loading a JSON file is much easier, just read the file and parse it into JSON:

MyModule._extensions['.json'] = function (module, filename) {
  const content = fs.readFileSync(filename, 'utf8');
  module.exports = JSONParse(content);
}
Copy the code

`exports`and`module.exports`The difference between

Node.js exports: Module. Exports: Module. Exports: module. Exports and module.exports are both injected with this line of code.

compiledWrapper.call(this.exports, this.exports, this.require, this,
    filename, dirname);
Copy the code

Exports === module.exports === {}, exports is a reference to module.exports if you use it all the time:

exports.a = 1;
module.exports.b = 2;

console.log(exports= = =module.exports);   // true
Copy the code

Exports === module.exports === module.exports === module.exports === module.exports === module.exports === module.

But if you ever use it like this:

exports = {
  a: 1
}
Copy the code

Or used like this:

module.exports = {
	b: 2
}
Copy the code

If you reassign exports or module.exports and change their reference address, the link between the two properties is broken and they are no longer equal. Module. exports is the export of the module, but your reassignment of module.exports does not change the export of the module, just changes the exports variable, because the module is still module and the export is module.exports.

A circular reference

Node.js handles circular references. Here’s an official example:

a.js:

console.log('a beginning');
exports.done = false;
const b = require('./b.js');
console.log('在 a 中，b.done = %j', b.done);
exports.done = true;
console.log(End of the 'a');
Copy the code

b.js:

console.log('b start');
exports.done = false;
const a = require('./a.js');
console.log('in b, a. tone = %j', a.done);
exports.done = true;
console.log(End of the 'b');
Copy the code

main.js:

console.log('the main start');
const a = require('./a.js');
const b = require('./b.js');
console.log('In main, a.tone =%j, b.tone =%j', a.done, b.done);
Copy the code

When main.js loads a.js, a.js loads b.js. At this point, B.js will try to load A.js. To prevent infinite loops, an unfinished copy of the A. js exports object is returned to the B. js module. Then B. Js completes the load and provides the EXPORTS object to the A. js module.

So how does this effect work? The answer is in our myModule. _load source code, notice the order of these two lines:

MyModule._cache[filename] = module;

module.load(filename);
Copy the code

In the above code, we set the cache first, and then perform the actual load.

mainloadinga.aTake a place in the cache before actually loading

aIt was loaded when it was officially loadedb

bLoad it againaIs already in the cacheaSo I’m going to go straight backa.exportsEven at this timeexportsIs incomplete.

conclusion

requireNot black magic, the entire Node.js module loading mechanism isJSThe implementation.
In each moduleexports, require, module, __filename, __dirnameNone of the five parameters are global variables, but are injected when the module is loaded.
To inject these variables, we need to wrap the user’s code in a function that forms a string and calls the sandbox modulevmTo implement.
In the initial state of the modulethis, exports, module.exportsThey all point to the same object, and if you reassign them, the connection breaks.
rightmodule.exportsThe reassignment will be exported as part of the module, but you have toexportsThe reassignment of does not change what the module exports, but doesexportsThis is just a variable, because modules are alwaysmodule, the exported content ismodule.exports.
To eliminate circular references, modules are added to the cache before they are loaded, and are returned to the cache the next time they are loaded. If the module is not fully loaded at this point, you may get incompleteexports.
Node.js implements this loading mechanism called CommonJS.

The full code for this article has been uploaded to GitHub:Github.com/dennis-jian…

The resources

Node.js module loading source: github.com/nodejs/node…

Node.js module official documentation: nodejs.cn/api/modules…

At the end of this article, thank you for your precious time to read this article. If this article gives you a little help or inspiration, please do not spare your thumbs up and GitHub stars. Your support is the motivation of the author’s continuous creation.

Welcome to follow my public numberThe big front end of the attackThe first time to obtain high quality original ~

“Front-end Advanced Knowledge” series:Juejin. Cn/post / 684490…

“Front-end advanced knowledge” series article source code GitHub address:Github.com/dennis-jian…

mo4tech.com (Moment For Technology) is a global community with thousands techies from across the global hang out!Passionate technologists, be it gadget freaks, tech enthusiasts, coders, technopreneurs, or CIOs, you would find them all here.

Drill into node.js’s module loading mechanism by writing the require function

A simple example

Require will run the target file first

`require`and`module.exports`Not black magic

Module type and loading sequence

Module type

Load order

Load folders

Supported file types

handwritten`require`

The Module class

Method the require

MyModule._load

`MyModule._resolveFilename`

`MyModule.prototype.load`

Load js files: myModule._extensions [‘.js’]

Compile execute js file: mymodule.prototype._compile

Load json file: myModule._extensions [‘.json’]

`exports`and`module.exports`The difference between

A circular reference

conclusion

The resources

Drill into node.js’s module loading mechanism by writing the require function

A simple example

Require will run the target file first

requireandmodule.exportsNot black magic

Module type and loading sequence

Module type

Load order

Load folders

Supported file types

handwrittenrequire

The Module class

Method the require

MyModule._load

MyModule._resolveFilename

MyModule.prototype.load

Load js files: myModule._extensions [‘.js’]

Compile execute js file: mymodule.prototype._compile

Load json file: myModule._extensions [‘.json’]

exportsandmodule.exportsThe difference between

A circular reference

conclusion

The resources

Related Posts

Get started with vue3.0 Composition API. Use a routing loop to create a navigation menu

Parse the nodeJs file stream

Right-most JS2Flutter Framework — Implementation of Animation and mini-game (IV)

`require`and`module.exports`Not black magic

handwritten`require`

`MyModule._resolveFilename`

`MyModule.prototype.load`

`exports`and`module.exports`The difference between