I. Main process of Webpack compilation

Compiler’s flow:
  1. Pass webpack.config.js as a parameter to the Compiler class (Entry-options)
  2. Creating Compiler instances
  3. Call Compiler.run to start compiling (make)
  4. Create Compilation (Compiler creates the Compilation object in the compiler, passes this in, and the Compilation contains a reference to the compiler)
  5. Start Chunk creation based on configuration (read file, convert to AST)
  6. Using Parser to parse dependencies from Chunk (find dependencies)
  7. Using Modules and dependencies to manage code Module dependencies (build-Modules)
  8. Generate the resulting code using the Template Compilation based data
  • There are three simple stages

Second, preparation

Let’s create a project with the following directory:

  selfWebpack
    - src
      - data.js
      - index.js
      - random.js
Copy the code
// index.js
import data from './data.js'
import random from './random.js'

console.log('🐻 I am data file -->', data)
console.log('🦁 I'm a random number -->', random)
console.log('🐺 I am index. Js)
Copy the code
// data.js
const result = 'I'm data in a file.'

export default result
Copy the code
// random.js
const random = Math.random()

export default random

Copy the code

Then we use Webpack to do a packaging first, analyze what we need to do

// Basic installation
npm init -y
npm install webpack@4.442. webpack-cli@4.2. 0 --save-dev
Copy the code
// package.json
/ / modify
"scripts": {
  "test": "echo \"Error: no test specified\" && exit 1"."build": "webpack --mode development"
},
Copy the code

Clean up the packaged code

(function(modules) {
  var installedModules = {};
  function __webpack_require__(moduleId) {
    if(installedModules[moduleId]) {
      return installedModules[moduleId].exports;
    }
    var module = installedModules[moduleId] = { i: moduleId, l: false.exports: {}}; modules[moduleId].call(module.exports, module.module.exports, __webpack_require__);
    module.l = true;
    return module.exports;
  }
  // Load entry module and return exports
  return __webpack_require__(__webpack_require__.s = "./src/index.js"); ({})"./src/data.js": function(module, __webpack_exports__, __webpack_require__) {
    "use strict";
    const result = 'I'm data in a file.'
    __webpack_exports__["default"] = (result);

  },
  "./src/index.js": function(module, __webpack_exports__, __webpack_require__) {
    "use strict";
    var _random_js__WEBPACK_IMPORTED_MODULE_1__ = __webpack_require__("./src/random.js");
    console.log('🐻 I am data file -->', _data_js__WEBPACK_IMPORTED_MODULE_0__["default"])
    console.log('🦁 I'm a random number -->', _random_js__WEBPACK_IMPORTED_MODULE_1__["default"])
    console.log('🐺 I am index. Js)},"./src/random.js": function(module, __webpack_exports__, __webpack_require__) {
    "use strict";
    const random = Math.random()
    __webpack_exports__["default"] = (random); }});Copy the code

The outermost layer is an immediate function that takes all modules (modules) lists. The modules argument passed in is an object.

  • The format of the object is filename: method.
  • Key is the relative path to the index.js file, value is an anonymous function, and the function body is the code we wrote in index.js. (This is how WebPack loads modules.)
We want to implement two functions
  1. importbecome__webpack_require__
  2. Read all dependencies in the module to generate a Template

Start building your own Selfpack

  • Implement packaged compiled code, in front of the SRC selfpack directory, at the same level to add a configuration file (selfpack. Config. Js), as follows:
  selfWbpack
    + src
    / / new
    - selfpack
      - compilation.js
      - compiler.js
      - index.js
      - Parser.js
    - selfpack.config.js
Copy the code
// selfpack.config.js
const { join } = require('path')
module.exports = {
  entry: join(__dirname, './src/index.js'),
  output: {
    path: join(__dirname, './dist'),
    filename: 'main.js'}}Copy the code

Four, to achieve transformation AST

  • Why convert to AST? Because we have import, we’re going to replace it with webpack_require.
  • How to do? Walk through the AST to collect the file paths introduced by the import statement.
  1. The first step is to find the entry file and get the file content through parameters
  2. Step 2, convert to AST
  3. Third, resolve the main module file dependencies
  4. Step 4, convert the AST back to JS code
  5. Fifth, analyze the dependencies between modules and replace import with webpack_require
4.1 Obtaining import Files
// selfpack/index.js
const Compiler = require('./Compiler')
const options = require('.. /selfpack.config.js')
const compiler = new Compiler(options)
compiler.run()
Copy the code
// selfpack/compilation.js
const fs = require('fs')

class Compilation {
  constructor(compiler) {
    const { options } = compiler
    this.options = options
  }compiler
  static ast(path){
    const content = fs.readFileSync(path, 'utf-8') // Read the file
    console.log('Get file', content)
  }
  buildModule(absolutePath, isEntry) {
    this.ast(absolutePath)
  }
}
module.exports = Compilation
Copy the code

npm install tapable

// selfpack/compiler.js
const { SyncHook } = require('tapable')
const Compilation = require('./Compilation')

class Compiler {
  constructor(options) {
    this.options = options
    this.hooks = {
      run: new SyncHook()
    }
  }
  run() {
    this.compile()
  }
  compile() {
    const compilation = new Compilation(this)
     // Find the entry file through entry
     const entryModule = compilation.buildModule(this.options.entry, true)}}module.exports = Compiler
Copy the code

Static method MDN

// selfpack/Parser.js
const fs = require('fs')
class Parser{
  static ast(path) {
    const content = fs.readFileSync(path, 'utf-8') // Read the file
    console.log('Read file', content)
  }
}
module.exports = Parser
Copy the code

Pass selfpack.config.js as an argument to the Compiler class and execute the run method. Call buildModule() with a new Compilation instance

  • buildModule( absolutePath, isEntry )
    • AbsolutePath: absolutePath to the entry file
    • IsEntry: Indicates whether it is the master module

Get the result of the import file:

The first step was successfully implemented, and now the second step is transformed into the AST

4.2 Converting to the AST

This step requires using @babel/ Parser to convert the code into an AST syntax tree. NPM install@babel /parser sourceType indicates that we are parsing the ES module

  • Invoke the Parser. Ast ()
  • Read the contents of the file with readFileSync and pass it to parser.parse() to get the AST.
// selfpack/Parser.js
const fs = require('fs')
const parser = require('@babel/parser')

class Parser{
  static ast(path) {
    const content = fs.readFileSync(path, 'utf-8') // Read the file
    console.log('Read file', content)
    const _ast = parser.parse(content, {
      sourceType: 'module' // indicates that we are parsing the ES module
    })
    console.log(_ast)
    console.log('I'm body content', _ast.program.body)
    return _ast
  }
}
module.exports = Parser
Copy the code

We’re on a roll here! This is the information for the entire file, and the content of the file we need is in the body of its property program. Take a look at the body content

This is an import Node attribute of SRC /index.js of type ImportDeclaration.

4.3 Resolving main module file dependencies

Next, parse the main module.

Traverse the AST using @babel/traverse NPM install @babel/traverse traverse() : the first argument is the AST, and the second argument is the configuration object

// selfpack/Parser.js
const traverse  = require('@babel-traverse').default
const fs = require('fs')
const parser = require('@babel/parser')
const path = require('path')

class Parser{
  static ast(path){
    const content = fs.readFileSync(path, 'utf-8') // Read the file
    const _ast = parser.parse(content, {
      sourceType: 'module' // indicates that we are parsing the ES module
    })
    console.log(_ast)
    console.log('I'm body content', _ast.program.body)
    return _ast
  }
  static getDependecy(ast, file) {
    const dependecies = {}
    traverse(ast, {
      ImportDeclaration: ({node}) = > {
        const oldValue = node.source.value
        const dirname = path.dirname(file)
        const relativepath = ". /" + path.join(dirname, oldValue) 
        dependecies[oldValue] = relativepath
        node.source.value = relativepath // convert./data.js to./ SRC /data.js}})return dependecies
  }
}
module.exports = Parser
Copy the code
  • Call parser. getDependecy method, obtain the dependent path of the main module, modify the source code.
  • GetDependecy () : static method on a node whose type is ImportDeclaration.
  • Node.source. value: is the value of import.
  • Because in our packaged code, the key in the input part becomes./src/data.jsSo there needs to be a corresponding change here

import data from './data.js' ==> require('./data.js') ==> require('./src/data.js')

Relativepath: Indicates the dependent file path. Dependecies: indicates the collected dependent object. The key is Node.source. value, and the value is the converted path.

import data from './data.js'
import random from './random.js'
Copy the code

Node.source. value: indicates from ‘./data.js’, ‘./random.js’

Path.relative (from, to) : Method returns (relative path) from (from) to (to) based on the current working directory

Process.cwd () : Returns the current working directory of the Node.js process (path.resolve())

// selfpack/compilation.js
const Parser = require('./Parser')
const path = require('path')

class Compilation {
  constructor(compiler) {
    const { options } = compiler
    this.options = options
    this.entryId
    / / add
    this.root = process.cwd() // The current directory to execute the command
  }
  buildModule(absolutePath, isEntry) {
    let ast = ' '
    ast = Parser.ast(absolutePath)
    const relativePath = '/' + path.relative(this.root, absolutePath)
    if(isEntry){
      this.entryId = relativePath
    }
    const dependecies = Parser.getDependecy(ast, relativePath)
    console.log("Dependencies", dependecies)
  }
}
module.exports = Compilation
Copy the code

After iterating through the ast, find the node type in the AST, and obtain the dependency of the index.js file (i.e. Data.js, random.js) from the AST of index.js.

The main module dependency path has been found! At this point, success is not far away.

4.4 Conversion Code

Next comes the conversion code, which converts the modified AST into JS code. Use @babel/core transformFromAst and @babel/preset-env. NPM install @babel/ core@babel /preset-env

  • TransformFromAst: converts the AST we pass in to our third parameter (@babel/preset-env), returns the converted code

@babel/preset-env is a way to turn the new JS feature we use into compatible code.

Now parser.js looks like this

/ / selfpack/Parser. Js complete
const traverse  = require('@babel-traverse').default
const fs = require('fs')
const parser = require('@babel/parser')
const path = require('path')
/ / add
const { transformFromAst } = require('@babel/core')

class Parser{
  static ast(path){
    const content = fs.readFileSync(path, 'utf-8') // Read the file
    const _ast = parser.parse(content, {
      sourceType: 'module' // indicates that we are parsing the ES module
    })
    console.log(_ast)
    console.log('I'm body content', _ast.program.body)
    return _ast
  }
  static getDependecy(ast, file) {
    const dependecies = {}
    traverse(ast, {
      ImportDeclaration: ({node}) = > {
        const oldValue = node.source.value
        const dirname = path.dirname(file)
        const relativepath = ". /" + path.join(dirname, oldValue) 
        dependecies[oldValue] = relativepath
        node.source.value = relativepath // convert./data.js to./ SRC /data.js}})return dependecies
  }
  / / add
  static transform(ast) {
    const { code } = transformFromAst(ast, null, {
        presets: ['@babel/preset-env']})return code
  }
}
module.exports = Parser
Copy the code
// selfpack/compilation.js.buildModule(absolutePath, isEntry) {
    let ast = ' '
    ast = Parser.ast(absolutePath)
    const relativePath = '/' + path.relative(this.root, absolutePath)
    if(isEntry){
      this.entryId = relativePath  // Save the file path to the main entry
    }
    const dependecies = Parser.getDependecy(ast, relativePath)
    / / add
    const transformCode = Parser.transform(ast)
    console.log("Transformed code", transformCode)
    return {
      relativePath,
      dependecies,
      transformCode 
    }
  }
}
  ...
Copy the code

Here are the results:

As you can see, const is successfully converted to var, but the path referenced by require(“./data.js”) is not yet consistent with modules’ key.

4.5 Recursive collection of dependencies

How do we determine what information a module should contain? First of all, we need to make sure that this file is unique, so we need the file path we want, because this one is unique. Then analyze the contents of the file:

  • Whether additional files have been introduced
  • Your own main content

So the module information we need is as follows:

  • Path to the module
  • Dependencies of this module
  • The converted code for this module

Here we get the transformed code and return an object in buildModule with the following structure:

// Get module information
  {
    relativePath: './src/xxx'.dependecies: {
      './data.js': './src/data.js'.'./random.js': './src/random.js'
    },
    transformCode: {... }}Copy the code

But buildModule can only collect one module’s dependencies, and our ultimate goal is to collect all dependencies, so we’ll do a recursive process. Modify compiler.js

// selfpack/compiler.js.compile() {
    const compilation = new Compilation(this)
     // Find the entry file through entry
    const entryModule = compilation.buildModule(this.options.entry, true)

    / / add
    this.modules.push(entryModule)
    this.modules.map((_module) = > {
      const deps = _module.dependecies
      for (const key in deps){
        if (deps.hasOwnProperty(key)){
          this.modules.push(compilation.buildModule(deps[key], false}}})))console.log('Final modules'.this.modules)
  }
  ...
Copy the code

Let’s take a look at the recursion in compile:

  1. Pass in the main entry filebuildModule , get the file module of the main entrance
  2. Outermost traversal of the main entry file module
  3. Then get the main module’s dependencies on all modules
  4. Push dependent modules into this.modules

Take a look at the final modules

We successfully get all the modules: paths, dependencies, transformed code.

Generate the Webpack template file

The last step in compiling is to generate the template file and place it in the output directory. Let’s just take the dist/main.js file we packaged at the beginning of this article and make some changes. Take a look at the modified compiler.js

/ / selfpack compilation. Js complete
const path = require('path')
const Parser = require('./Parser')
const fs = require('fs')


class Compilation {
  constructor(compiler) {
    / / modify
    const { options, modules } = compiler
    this.options = options
    this.root = process.cwd() // The current directory to execute the command
    this.entryId
    / / add
    this.modules = modules
  }
  buildModule(absolutePath, isEntry) {
    let ast = ' '
    ast = Parser.ast(absolutePath)
    const relativePath = '/' + path.relative(this.root, absolutePath)
    if(isEntry){
      this.entryId = relativePath
    }
    const dependecies = Parser.getDependecy(ast, relativePath)
    const transformCode = Parser.transform(ast)
    // console.log(" dependencies ", dependecies)
    // console.log(" Converted code ", transformCode)
    return {
      relativePath,
      dependecies,
      transformCode 
    }
  }
  / / add
  emitFiles(){
    let _modules = ' '
    const outputPath = path.join(
      this.options.output.path,
      this.options.output.filename
    )
    this.modules.map((_module) = > {
      // Remember to use quotation marks
      _modules += ` '${_module.relativePath}': function(module, exports, require){
        ${_module.transformCode}}, `
    })
    const template = ` (function(modules) { var installedModules = {}; function __webpack_require__(moduleId) { // Check if module is in cache if(installedModules[moduleId]) { return installedModules[moduleId].exports; } var module = installedModules[moduleId] = { exports: {} }; modules[moduleId].call(module.exports, module, module.exports, __webpack_require__); return module.exports; } // Return __webpack_require__('The ${this.entryId}'); ({})${_modules}
    })
    `
    const dist = path.dirname(outputPath)
    fs.mkdirSync(dist)
    fs.writeFileSync(outputPath, template, 'utf-8')}}module.exports = Compilation
Copy the code

The contents of the packaged file, roughly, look like this, with a few minor flaws. Take a look at the emitFiles function in action

  1. Get the path and filename of the output object in selfpack.config.js
  2. Iterate through all modules and place them in the input position of the template
  3. Create a new file and write the compiled code

Complete the compiler

/ / selfpack compiler. Js complete
const { SyncHook } = require('tapable')
const Compilation = require('./Compilation')

class Compiler {
  constructor(options) {
    this.modules = []
    this.options = options
    this.hooks = {
      run: new SyncHook()
    }
  }
  run() {
    this.compile()
  }
  compile() {
    const compilation = new Compilation(this)
    const entryModule = compilation.buildModule(this.options.entry, true)
    this.modules.push(entryModule)
    this.modules.map((_module) = > {
      const deps = _module.dependecies
      for (const key in deps){
        if (deps.hasOwnProperty(key)){
          this.modules.push(compilation.buildModule(deps[key], false}}})))/ / add
    compilation.emitFiles()
  }
}
module.exports = Compiler
Copy the code

The compiled code looks like this:

// dist/main.js
(function (modules) {
  var installedModules = {};
  function __webpack_require__(moduleId) {
    // Check if module is in cache
    if (installedModules[moduleId]) {
      return installedModules[moduleId].exports;
    }
    var module = installedModules[moduleId] = {
      exports: {}}; modules[moduleId].call(module.exports, module.module.exports, __webpack_require__);
    return module.exports;
  }
  // The entry function to execute
  return __webpack_require__('./src/index.js'); ({})'./src/index.js': function (module.exports.require) {
    "use strict";

    var _data = _interopRequireDefault(require("./src/data.js"));

    var _random = _interopRequireDefault(require("./src/random.js"));

    function _interopRequireDefault(obj) { return obj && obj.__esModule ? obj : { "default": obj }; }

    console.log('🐻 I am data file -->', _data["default"]);
    console.log('🦁 I'm a random number -->', _random["default"]);
    console.log('🐺 I am index. Js);
  }, './src/data.js': function (module.exports.require) {
    "use strict";

    Object.defineProperty(exports."__esModule", {
      value: true
    });
    exports["default"] = void 0;
    var result = 'I'm data in a file.';
    var _default = result;
    exports["default"] = _default;
  }, './src/random.js': function (module.exports.require) {
    "use strict";

    Object.defineProperty(exports."__esModule", {
      value: true
    });
    exports["default"] = void 0;
    var random = Math.random();
    var _default = random;
    exports["default"] = _default; }})Copy the code

At this point, a simple Webpack compilation process code is written. Copy the code to your browser to test it

Implement Plugins function of Webpack

How to develop a custom plugins? Webpack implements its own set of life cycles internally, and plugins use Apply to invoke the life cycles provided in WebPack. The life cycle of Webpack is mainly implemented by Tapable. Only SyncHook is used here, see this Tapable for more details.

We revise the website ConsoleLogOnBuildWebpackPlugin. Js example. Create a new plugins in the SRC sibling directory

  + src
  - plugins
    - ConsoleLogOnBuildWebpackPlugin.js
Copy the code

Write a simple plugins

// ConsoleLogOnBuildWebpackPlugin.js
const pluginName = 'ConsoleLogOnBuildWebpackPlugin';

class ConsoleLogOnBuildWebpackPlugin {
  apply(compiler) {
    compiler.hooks.run.tap(pluginName, compilation= > {
      console.log('The webpack build process is starting!!! ');
    });
    // Execute after the file is packaged
    compiler.hooks.done.tap(pluginName,(compilation) = > {
      console.log("The whole Webpack pack is finished.")})// execute when webpack outputs the file
    compiler.hooks.emit.tap(pluginName,(compilation) = > {
        console.log("File initiation.")}}}module.exports = ConsoleLogOnBuildWebpackPlugin;
Copy the code

The configuration file then imports the plugins

// selfpack.config.js
const { join } = require('path')
const ConsoleLogOnBuildWebpackPlugin = require('./plugins/ConsoleLogOnBuildWebpackPlugin')

module.exports = {
  entry: join(__dirname, './src/index.js'),
  output: {
    path: join(__dirname, './dist'),
    filename: 'main.js'
  },
  plugins: [new ConsoleLogOnBuildWebpackPlugin()],
}
Copy the code

For our SelfWebPack to support plugins, we need to make some changes.

// selfpack/index.js
const Compiler = require('./Compiler')
const options = require('.. /selfpack.config.js')
const compiler = new Compiler(options)
const plugins = options.plugins
for (let plugin of plugins) {
    plugin.apply(compiler)
}
compiler.run()
Copy the code
// selfpack/compiler.js
const { SyncHook } = require('tapable')
const Compilation = require('./Compilation')

class Compiler {
  constructor(options) {
    this.modules = []
    this.options = options
    this.hooks = {
      run: new SyncHook(),
      / / add
      emit: new SyncHook(),
      done: new SyncHook()
    }
  }
  run() {
    this.compile()
  }
  compile() {
    const compilation = new Compilation(this)
    / / add
    this.hooks.run.call()
     // Find the entry file through entry
    const entryModule = compilation.buildModule(this.options.entry, true)
    this.modules.push(entryModule)
    this.modules.map((_module) = > {
      const deps = _module.dependecies
      for (const key in deps){
        if (deps.hasOwnProperty(key)){
          this.modules.push(compilation.buildModule(deps[key], false}}})))// console.log(' final modules', this.modules)
    compilation.emitFiles()
    / / add
    this.hooks.emit.call()
    this.hooks.done.call()

  }
}
module.exports = Compiler
Copy the code

We implemented the Compiler lifecycle by defining the webpack lifecycle as soon as the compiler function was initialized and calling it during run.

The print result is as follows:

This article only implements the simple compilation principle, see webapck-Github for more implementation

The corresponding code is here on Github

Reference article: Handwritten WebPack core principles