preface

Webpack proficiency has become an essential survival skill for modern front-end engineers. Without a doubt, WebPack has become the leading front-end build tool, and technical documentation on how to use WebPack has proliferated on the Web. But there is little that can be said about the WebPack build process. This article attempts to explore how WebPack builds resources step by step through source code interpretation and breakpoint debugging.

As of this publication, the latest version of WebPack is WebPack 5.0.0-beta.1, meaning that the source code for this article is from the latest WebPack V5.

In particular, the source code listed in this article has been simplified, if you want to see the specific code you can access the official Webpack library according to the source file name I identified. Brief part of this article:

  • Removed module introduction, i.econst xxx = require('XXX');
  • Exception code, although exception handling is also very important, but this paper mainly analyzes the main process of webpack normal work, if the exception handling can not be ignored, I will specially explain;

How do I debug WebPack

I have always believed that learning source code is not a matter of reading line by line of code. For a mature open source project, there are many complicated branches. Trying to trace the path of a program by debugging the code step by step is the fastest way to quickly understand the basic architecture of a project.

The full Debugger feature in the VS Code editor is a great tool for debugging Node applications.

  1. First, in order to learn the webpack source code, you must first clone the source code from the WebPack library to the local:
git clone https://github.com/webpack/webpack.git
Copy the code
  1. Installation project dependencies; VS Code opens the local Webpack repository
npm install
cd webpack/
code .
Copy the code
  1. To avoid contaminating the project root directory, create a new one under the root directorydebugFolder for debugging code,debugThe folder structure is as follows:
debug-|
      |--dist    // Package and output the file
      |--src
         |--index.js   // Source code entry file
      |--package.json  // Some Loaders and plugins need to be installed during debug
      |--start.js      // debug Startup file
      |--webpack.config.js  // Webpack configuration file
Copy the code

The debug code is as follows:

//***** debug/src/index.js *****
import is from 'object.is'  // Introduce a small but beautiful third-party library to see how WebPack handles third-party packages
console.log('Nice to meet you, Webpack.')
console.log(is(1.1))


//***** debug/start.js *****
const webpack = require('.. /lib/index.js')  // Use the webpack function directly from the source
const config = require('./webpack.config')
const compiler = webpack(config)
compiler.run((err, stats) = >{
    if(err){
        console.error(err)
    }else{
        console.log(stats)
    }
})


//***** debug/webpack.config.js *****
const path = require('path')
module.exports = {
    context: __dirname,
    mode: 'development'.devtool: 'source-map'.entry: './src/index.js'.output: {
        path: path.join(__dirname, './dist'),},module: {
        rules: [{test: /\.js$/.use: ['babel-loader'].exclude: /node_modules/,}]}}Copy the code
  1. In VS CodeDebugBar to add debug configuration:
{
    "configurations": [{"type": "node"."request": "launch"."name": "Start the Webpack debugger"."program": "${workspaceFolder}/debug/start.js"}}]Copy the code

Once the configuration is complete, try clicking on the ► (launch) to see if the debugger is running properly (if it is, a main.js file will be packaged in debug/dist).

If you have time, I hope you can personally complete a Webpack debugging process, I believe you will have a harvest. It’s human nature to want to explore.

Next, take a step by step look at how WebPack works through breakpoint debugging.

The source code interpretation

Webpack startup mode

Webpack can be launched in two ways:

  1. throughwebpack-cliScaffolding to start, i.e. can be inTerminalTerminal direct operation;
webpack ./debug/index.js --config ./debug/webpack.config.js
Copy the code

This method is the most common and the fastest way, out of the box.

  1. throughrequire('webpack')Importing package execution;

/bin/webpack.js.

Starting point for WebPack compilation

Everything starts with const Compiler = webpack(config).

Webpack (./lib/webpack.js) :

const webpack = (options, callback) = > {
    let compiler = createCompiler(options)
    // If the callback function is passed, it will start automatically
    if(callback){
        compiler.run((err, states) = > {
            compiler.close((err2) = >{
                callbacl(err || err2, states)
            })
        })
    }
    return compiler
}
Copy the code

The Compiler object is returned after the webpack function is executed. There are two very important core objects in Webpack, compiler and Compilation, which are widely used in the whole compilation process.

  • CompilerClass (./lib/Compiler.js) : the main engine of Webpack, in compiler object records the complete webPack environment information, in Webpack from start to end,compilerIt only gets generated once. You can be incompilerObject read fromwebpack configInformation,outputPathAnd so on;
  • CompilationClass (./lib/Compilation.js) : represents a single build and build resource.compilationCompilation jobs can be executed multiple times, such as webpack work inwatchIn this mode, one is re-instantiated each time a change to the source file is detectedcompilationObject. acompilationObject represents the current module resource, compile-build resource, changing file, and state information that is being traced.

The difference between the two? Compiler represents the immutable WebPack environment; Compilation represents a compilation job, and each compilation may be different.

For example 🌰 : Compiler is like a mobile phone production line. It can start to work after being powered on and wait for instructions to produce mobile phones. Compliation is like the production of a mobile phone, the production process is basically the same, but the phone may be Xiaomi phone or Meizu phone. Different materials, different output.

Compiler class instantiated in createCompiler (./lib/index.js) :

const createCompiler = options= > {
    const compiler = new Compiler(options.context)
    // Register all custom plug-ins
    if(Array.isArray(options.plugins)){
        for(const plugin of options.plugins){
            if(typeof plugin === 'function'){
                plugin.call(compiler, compiler)
            }else{
                plugin.apply(compiler)
            }
        }
    }
    compiler.hooks.environment.call()
    compiler.hooks.afterEnvironment.call()
    compiler.options = new WebpackOptionsApply().process(options, compiler)  // Register all webPack built-in plug-ins in process
    return compiler
}
Copy the code

After the Compiler class is instantiated, if the Webpack function receives the callback, the compiler.run() method is executed directly, and WebPack automatically starts the compilation journey. If no callback is specified, the user needs to invoke the run method himself to initiate compilation.

From the above source, you can get some information:

  • Compiler is instantiated by Compiler, with properties and methods described in a later section, the most important of which is the compiler.run() method;

  • Iterate through the plugins array in WebPack Config. Here I have bolded the plugins array, so plugins should not be configured as objects. (In fact, the Webpack function will do the object schema validation for options).

  • Plugin: If plugin is a function, call it directly; If plugin is of any other type (mainly object), the Apply method of the Plugin object is executed. Apply function signature :(compiler) => {}.

    Webpack is very strict that our plugins array elements must be functions, or an object with an Apply field and apply is a function, for this reason.

    {
plugins: [ new HtmlWebpackPlugin() ]
}
Copy the code

  • Call hook: Compiler. Hooks. The environment. The call (), and the compiler. The hooks. AfterEnvironment. The call () is the source code to read the hooks at this point we met first call, in the after reading, you will meet more hooks registration and call. To understand the use of WebPack hooks, you need to understand Tapable, which is the foundation for writing plug-ins.

    As for Tapable, I will “deal with it differently”. –> I have documented it, go to “Write a custom Webpack plug-in from understanding Tapable”

  • Process (Options) : In Webpack Config, there are many other fields besides plugins, so the purpose of process(options) is to process these fields one by one.

At this point, we’ve seen what webPack does to prepare for the initialization phase. Webpack is truly powerful when the fuse compiler.run() is lit.” New WebpackOptionsApply().process(options, compiler) provides important logistical support for subsequent compilation.

process(options, compiler)

The job of the WebpackOptionsApply class is to initialize webPack Options. Open source file lib/WebpackOptionsApply js, you will find the top 50 lines is the introduction of various webpack built-in Plugin, you can guess the process approach should be a variety of new SomePlugin (). The apply (), That’s the way it is.

Compact source (lib/WebpackOptionsApply. Js) :

class WebpackOptionsApply extends OptionsApply {
    constructor() {
        super(a); } process(options, compiler){// When the incoming configuration information meets the requirements, the logic related to the configuration item is processed
        if(options.target) {
            new OnePlugin().apply(compiler)
        }
        if(options.devtool) {
            new AnotherPlugin().apply(compiler)
        }
        if. new JavascriptModulesPlugin().apply(compiler);new JsonModulesPlugin().apply(compiler);
        new. compiler.hooks.afterResolvers.call(compiler); }}Copy the code

In the source code… The ellipsis omits a lot of similar operations. The process function is long, close to 500 lines of code, and does two main things:

  1. New a lot of plugins and apply them.

    In the previous section, we learned that the WebPack plug-in is really just a class that provides apply methods, which are instantiated by WebPack and execute apply methods when appropriate. The Apply method receives the Compiler object, making it easier to listen for messages on hooks. Also, each Plugin instantiated in the process function is maintained by WebPack itself, so you’ll find a lot of plugin-ending files in the webPack project root directory. The user – defined plug-ins have been registered before. Each plug-in has its own mission. Their job is to hook a message on Compiler.hooks. Once a message is triggered, the callbacks registered on the message are called according to the hook type. There are three ways to hook: Tap tapAsync tapPromise, and you need to know how Tapable works.

  2. Do the initialization work according to the configuration items in options. XXX, and most of the initialization work is still done above 👆

To sum up: Once the process function is executed, WebPack registers all the hook messages it cares about and waits for them to be fired one by one during subsequent compilations.

Execute the process method to load the ammo and wait for the battle.

compiler.run()

/lib/ compiler.js) :

class Compiler {
    constructor(context){
    // All hooks are provided by 'Tapable'. Different hook types invoke different timings when triggered
    this.hooks = {
            beforeRun: new AsyncSeriesHook(["compiler"]),
            run: new AsyncSeriesHook(["compiler"]),
            done: new AsyncSeriesHook(["stats"]),
            // ...}}// ...
	
    run(callback){
        const onCompiled = (err, compilation) = > {
            if(err) return
            const stats = new Stats(compilation);
            this.hooks.done.callAsync(stats, err => {
                if(err) return
                callback(err, stats)
                this.hooks.afterDone.call(stats)
            })
        }
        this.hooks.beforeRun.callAsync(this, err => {
            if(err) return
            this.hooks.run.callAsync(this, err => {
                if(err) return
                this.compile(onCompiled)
            })
        })
    }
}
Copy the code

Read through the run function and you’ll see that it hooks several stages of the compilation process and calls the pre-registered hook function (this.links.xxxx.call (this)) at the corresponding stages, just like the React lifecycle function. BeforeRun –> run –> done –> afterDone. Third-party plug-ins can hook into different life cycles, receive compiler objects, and handle different logic.

The run function hooks the early and late stages of webpack compilation, leaving this.compile() to do the most critical code compilation in the middle. In This.ille (), programmer, another main character, was introduced.

compiler.compile()

The compiler.compile function is the main venue for module compiler.compile without further details, we’ll paste the truncated pseudocode first:

compile(callback){
    const params = this.newCompilationParams()  // Initialize the module factory object
    this.hooks.beforeCompile.callAsync(params, err => {
        this.hooks.compile.call(params)
        // compilation Records the compilation environment information
        const compilation = new Compilation(this)
        this.hooks.make.callAsync(compilation, err => {
            compilation.finish(err= > {
                compilation.seal(err= >{
                    this.hooks.afterCompile.callAsync(compilation, err => {
                        return callback(null, compilation)
                    })
                })
            })
        })
    })
}
Copy the code

The compile function, like run, triggers a series of hooks. BeforeCompile –> compile –> make –> afterCompile.

Make is the compilation process we care about. But in this case it is just a hook trigger, and it is clear that the actual compilation execution is registered on the hook callback.

Webpack is very flexible thanks to Tapable, and the popular Callback mechanism in Node (callback hell) is so sophisticated that it might not be easy to catch with breakpoints. Here I use a keyword search to find in reverse where the make hook is registered.

We found it in lib/ entryplugin.js by searching for hooks. Make. TapAsync.

Depending on your search terms, you’ll list a lot of distractions, and you’ll be smart enough to figure out which one is closest to the actual situation.

The EntryPlugin plugin is called in lib/ entryoptionplugin.js, and you’ll find something familiar in it:

if(typeof entry === "string" || Array.isArray(entry)){
	applyEntryPlugins(entry, "main")}else if (typeof entry === "object") {
  for (const name of Object.keys(entry)) { applyEntryPlugins(entry[name], name); }}else if (typeof entry === "function") {
  new DynamicEntryPlugin(context, entry).apply(compiler);
}
Copy the code

Remember how the entry field was configured in webpack.config.js? You can see that when entry is a string or array, the packaged resource is called main.js.

Back is not over yet, we continue to search keywords new EntryOptionPlugin, Oops, search the file’s lib/WebpackOptionsApply. Js. The make hook is already registered in the process function and is waiting for you to call it.

Back to the lib/EntryPlugin. Js see compiler. Hooks. Make. TapAsync did what. In fact, it was to run the method of compiliation. AddEntry and continue to explore the engineering of compiliation.

addEntry(context, entry, name, callback) {
    this.hooks.addEntry.call(entry, name);
    // Each entry in entryDependencies represents an entry point, and packaged output has multiple files
    let entriesArray = this.entryDependencies.get(name)
	entriesArray.push(entry)
    this.addModuleChain(context, entry, (err, module) = > {this.hooks.succeedEntry.call(entry, name, module);
        return callback(null.module); })}Copy the code

Is the role of addEntry module entry information is passed to the module in the chain, namely addModuleChain, then continue to invoke compiliation… factorizeModule, these calls will last entry entry information “translation” into a module (strictly, A module is an instantiated object of a NormalModule. It was a little hard to understand when reading the source code. Due to the trap hell of Node callback, I thought entry processing would be synchronous until I realized that the use of process.nextTick made many callbacks called asynchronously. More breakpoints and more debugging are recommended here to understand the meandering asynchronous callbacks.

Here I list the order in which the related functions are called: this.addEntry –> this.addModuleChain –> this.handleModuleCreation –> this.addModule –> this.buildModule –> Building_buildmodule –> module. Build (this refers to the engineering of engineering).

You will eventually go to the NormalModule object (./lib/ normalModule.js) and execute the build method.

The normalModule.build method calls its own doBuild method first:

const { runLoaders } = require("loader-runner");
doBuild(options, compilation, resolver, fs, callback){
    // runLoaders method imported from package 'loader-runner'
    runLoaders({
        resource: this.resource,  // The resource may be a js file, a CSS file, or an img file
        loaders: this.loaders,
    }, (err, result) => {
        const source = result[0];
        const sourceMap = result.length >= 1 ? result[1] : null;
        const extraInfo = result.length >= 2 ? result[2] : null;
        // ...})}Copy the code

DoBuild simply selects the appropriate loader to load a resource in order to convert the resource into a JS module (webPack only recognizes JS modules). Finally, the loaded source file is returned for further processing.

Webpack is good at handling standard JS modules, but not other types of files (CSS, SCSS, JSON, JPG), and so on, where it needs loader’s help. The loader’s job is to convert the source code to JS modules so that WebPack can recognize them correctly. Loader functions like a Linux information flow pipeline. It receives the source string stream, processes it, and returns the processed source string to the next loader. Loader basic paradigm :(code, sourceMap, meta) => string

After doBuild, any modules are converted to standard JS modules.

Try adding CSS code to your JS code and see the data structure of the converted standard JS module.

The next step is to compile the standard JS code. Source is handled like this in the callback passed to doBuild:

const result = this.parser.parse(source)
Copy the code

In this case, this.parser is actually an instance object of JavascriptParser. Finally, JavascriptParser calls the parse method provided by the third-party package Acorn to parse the JS source code.

parse(code, options){
    // Call the third-party plugin 'Acorn' to parse the JS module
    let ast = acorn.parse(code)
    // Omit some code
    if (this.hooks.program.call(ast, comments) === undefined) {
        this.detectStrictMode(ast.body)
        this.prewalkStatements(ast.body)
        this.blockPrewalkStatements(ast.body)
        // Webpack walks through the ast.body once, collecting all dependencies for the module and writing them to 'module.dependencies'
        this.walkStatements(ast.body)
    }
}
Copy the code

There is an online widget called AST Explorer that can convert JS code into a syntax tree AST online. Select Acorn as the parser. /debug/ SRC /index.js; /debug/ SRC /index.js

What does webpack parse do here? We use babel-loader to parse source files. What parse does best is collect module dependencies, such as module import statements import {is} from ‘object-is’ or const XXX = require(‘XXX’), that appear in debugging code. Webpack records these dependencies. Record it in the Module. dependencies array.

compilation.seal()

At this point, webPack has collected all the information and dependencies for the module, starting with the entry file, and it’s time to package the module further.

Before executing compilation.seal(./lib/Compliation), you can set a breakpoint to see what’s going on in compilation.modules. Compilation at this time. There are three sub modules modules, respectively. / SRC/index, js node_modules/object is/index, js and node_modules/object is/is. Is

Compilation. seal has several steps. It closes the module and generates resources, which are stored in compilation.assets and compilation.chunks.

You’ll see compilation.assets and compilation.chunks in most third-party Webpack plugins.

Then calls the compilation. CreateChunkAssets method all dependencies by corresponding template to render a stitching good string:

createChunkAssets(callback){
    asyncLib.forEach(
        this.chunks,
        (chunk, callback) => {
            // Manifest is an array structure, and each manifest element provides a 'render' method that provides subsequent source string generation services. When the render method is initialized is in './lib/ maintemplate.js'
            let manifest = this.getRenderManifest()
            asyncLib.forEach(
                manifest,
                (fileManifest, callback) => {
                    ...
                    source = fileManifest.render()
                    this.emitAsset(file, source, assetInfo)
                },
                callback
            )
        },
        callback
    )
}
Copy the code

You can set a breakpoint on the this.emitAsset(File, source, assetInfo) line in the createChunkAssets body to observe the data structure in the source at this point. In the source._source field, you can see the source code:

It is worth mentioning that during the execution of createChunkAssets, it will first read whether the same hash resource exists in the cache. If so, it will directly return the content. Otherwise, it will continue to execute the logic generated by the module and store it in the cache.

compiler.hooks.emit.callAsync()

After seal execution, all the information about the module and the packaged source code is stored in memory, and it is time to output it as a file. This is followed by a series of callback callbacks, culminating in the compiler.emitassets method body. The this.hooks. Emit life cycle is called in Compiler.emitAssets, and the file is then output to the specified folder according to the path property of the Output configuration of the Webpack Config file. At this point, you can see the debug code packaged in./debug/dist.

this.hooks.emit.callAsync(compilation, () => {
    outputPath = compilation.getPath(this.outputPath, {})
    mkdirp(this.outputFileSystem, outputPath, emitFiles)
 })
Copy the code

conclusion

Thank you very much for reading to the end. This article is a long one, and briefly summarizes the basic process of webPack compilation module:

  1. callwebpackThe function to receiveconfigConfigure information and initializecompilerIn the meantimeapplyAll plug-ins built into WebPack;
  2. callcompiler.runEnter the module compilation stage;
  3. Each new compilation instantiates onecompilationObject, record the basic information of the compilation;
  4. Enter themakePhase, or triggercompilation.hooks.makeThe hook,entryFor entry: a. Call appropriateloaderPreprocess the source code of the module and convert it to the standard JS module; B. Invoke a third-party plug-inacornAnalyze the standard JS module and collect module dependencies. At the same time, it will continue to recurse for each dependency, collecting dependency information for the dependency, and recurse again and again. You end up with a dependency tree 🌲;
  5. The last callcompilation.sealRender module, integrate various dependencies, finally output one or more chunks;

Here is a simple sequence diagram:

The above process cannot completely summarize the whole process of Webpack. As webpack.config configuration becomes more and more complex, WebPack will derive more processes to deal with different situations.

Is Webpack complex? Complex, Tabable and Node callbacks allow the process to go in many directions, and webPack is highly configurable because of its plug-in system. Is Webpack easy? Also easy, it does only one thing, compile and package JS modules, and do it perfectly.

The last

Code words are not easy if:

  • This article is useful to you, please don’t be stingy your little hands for me;
  • If you don’t understand or incorrect, please comment. I will reply or erratum actively.
  • Expect to continue to learn front-end technology knowledge with me, please pay attention to me;
  • Reprint please indicate the source;

Your support and attention is the biggest motivation for my continuous creation!