preface

Webpack is an indispensable tool in front-end programming. Its function is to pack multiple modules into one or more bundles. As a front-end novice, I always think Webpack is as mysterious as magic. I found that the principle of Webpack is not complicated, but this article is not to analyze the source code, in fact, before understanding the principle to read the source code is a very inefficient thing, we can try to implement a low version of Webpack

Since this article uses a lot of code, I will present the project code first

Let’s make a simple packer together

Let’s start with Babel

If you’re familiar with the front end, Babel is a Javascript compiler that compiles new js syntax into old browser-executable syntax.

The principle of Babel

Babel compiles code in three parts

  1. Parse: Converts code into an AST
  2. Traverse: traverse the AST for modification
  3. Generate: Convert AST to code2

Know the AST

AST Abstract Syntax Tree, which is an Abstract representation of the syntactic structure of source code. It represents the syntactic structure of a programming language as a tree, with each node in the tree representing a structure in the source code.

Let’s use the Babel tool to manually convert the let syntax to var syntax and observe the structure of the AST

Complete repository connection added

import traverse from '@babel/traverse'
import {parse} from '@babel/parser'
import generate from '@babel/generator'

const code = `let a = 1; let b = 2`
const ast = parse(code, {sourceType: 'module'})
console.log(ast, 'ast');
traverse(ast, {
  enter: item= > {
    if (item.node.type === 'VariableDeclaration') {
      if(item.node.kind === 'let') {
        item.node.kind = 'var'}}}})const result = generate(ast, {}, code)
console.log(result.code);
Copy the code

The above code only runs in Node. For debugging purposes, use node -r ts-node/register –inspect-brk let_to_var.ts to debug in the browser console

We can see the structure of the AST by looking at breakpoints

The result of running nodeJS

It is recommended that an online AST analyzer, AstExplorer, be used to explore the structures in the AST

es6 to es5

Es6 -> es5. It seems a bit impractical to use if for every syntax, but Babel /core provides a function to convert

import {parse} from '@babel/parser'
import * as babel from '@babel/core'

const code = `let a = 1; const b = 2`
const ast = parse(code, {sourceType: 'module'})
const result = babel.transformFromAstSync(ast, code, {
  presets: ['@babel/preset-env']})console.log(result.code);
Copy the code

Rely on the analysis of

What else can we do with Babel except convert JS syntax? Let’s try analyzing JS dependencies

First we create three JS files

index.js

import a from './a.js'
import b from './b.js'
console.log(a.value + b.value)
Copy the code

a.js

const a = {
  value: 1
}

export default a
Copy the code

b.js

const b = {
  value: 1
}

export default b
Copy the code

Dependency analysis code

import * as fs from 'fs';
import {parse} from '@babel/parser';
import {relative, resolve, dirname} from 'path';
import traverse from '@babel/traverse';

// Set the root directory
const projectRoot = resolve(__dirname, 'project1');
// Type declaration
type DepRelation = {
  [key: string]: { deps: string[], code: string }
}
// Initialize an empty depRelation for collecting data
const depRelation:DepRelation = {};

const collectCodeAndDeps = (filePath: string) = > {
  // The project path of the file is index.js
  const key = getProjectPath(filePath);
  // Get the content of the file and put the content in depRelation
  const code = fs.readFileSync(filePath).toString();
  depRelation[key] = {deps: [], code};
  // Convert the code to bit AST
  const ast = parse(code, {sourceType: 'module'});
  traverse(ast, {
    enter: item= > {
      if (item.node.type === 'ImportDeclaration') {
        // path.node.source.value is usually a relative directory, such as./a3.js, which needs to be converted to an absolute path first
        const depAbsolutePath = resolve(dirname(filePath), item.node.source.value);
        // Then turn to the project path
        const depProjectPath = getProjectPath(depAbsolutePath)
        // Write dependencies in depRelation
        depRelation[key].deps.push(depProjectPath)
      }
    }
  });
};

const getProjectPath = (filePath: string) = > {
  return relative(projectRoot, filePath);
};

collectCodeAndDeps(resolve(projectRoot, 'index.js'))

console.log(depRelation);
Copy the code

Code thinking

  1. callcollectCodeAndDeps('index.js')
  2. The firstdepRelation['index.js']Initialized to{deps: [], code}
  3. Convert the index.js code into an AST
  4. Let’s go through the AST and seeimportWhat dependencies are there, suppose a.js and B.js are relied on
  5. Write the paths of a.js and b.jsdepRelation['index.js'].deps 里

Finally printed by depRelation

Return to link

The above code can analyze only one dependency. What if it’s multiple?

Multilayer rely on

Resolving multiple levels of dependency is actually easy to solve by using recursion. Of course, using recursion is risky. If the dependency level is too deep, you risk calling stack overflow

Circular dependencies

What about ring dependence?

For example, import B.js from a.js and import A.js from b.js

If the loop dependencies are handled directly with the above code, the recursion will continue and end up calling stack Overflow

The solution is to add a conditional judgment to determine whether the file has already been recorded in depRelation[‘index.js’].deps before calling the parse function, and if so terminate the recursion

conclusion

The principle of bebel

Graph TD code --> parse --> AST --> traverse --> AST2 --> generate --> code2

Analysis depends on the process of the first is to put the code into the ast, then traverse the ast, whenever found the import statement, we rely on record, for multilayer dependencies, the way we can use recursive processing, if it is a circular dependencies, need to rely on the inspection, if it is already record dependence is not

Webpack core bundler

A bundle is produced by a Bundler. What is a bundle?

Here is the breakdown in the official documentation

It is generated by many different modules and contains the final version of the source file that has been loaded and compiled.

That is, a bundle is a file that contains all the modules and can execute all the modules, and it can be run directly in the browser

So the question we’re going to solve in this video is

  • Make the code in the module executable
  • Multiple modules are packaged into a single module

Prepare before you start

index.js

import a from './a.js'
import b from './b.js'
console.log(a.getB())
console.log(b.getA())
Copy the code

a.js

import b from './b.js'
const a = {
  value: 'a'.getB() {
    return b.value + ' from b.js'}}export default a
Copy the code

b.js

import a from './a.js'
const b = {
  value: 'b'.getA() {
    return a.value + ' from a.js'}}export default b
Copy the code

Obtained using the dependent code analyzed in the previous section

{ 'index.js': { deps: [ 'a.js', 'b.js' ], code: "import a from './a.js'\r\n" + "import b from './b.js'\r\n" + 'console.log(a.getB())\r\n' + 'console.log(b.getA())\r\n' }, 'a.js': { deps: [ 'b.js' ], code: "import b from './b.js'\r\n" + 'const a = {\r\n' + " value: 'a',\r\n" + ' getB() {\r\n' + " return b.value + ' from b.js'\r\n" ' }\r\n' + '}\r\n' + '\r\n' + 'export default a\r\n' }, 'b.js': { deps: [ 'a.js' ], code: "import a from './a.js'\r\n" + 'const b = {\r\n' + " value: 'b',\r\n" + ' getA() {\r\n' + " return a.value + ' from a.js'\r\n" ' }\r\n' + '}\r\n' + '\r\n' + 'export default b\r\n' }}Copy the code

Make the code in the module executable

In the above code, import/export is not run directly by the browser and needs to be converted to a function

We need to use bable/core to convert es6’s import/export syntax into ES5’s require function and exports object

const { code: es5Code } = babel.transform(code, {
    presets: ['@babel/preset-env']})Copy the code

The transformed code looks like this

Code,

Let’s take a look at the code of A. js and see what the code looks like after webpack is compiled

Doubt a

Object.defineProperties(exports.'__esModule', {value: true})
Copy the code

What this code does is

  • Add one to the current module__esModule: true, easy to separate from CommonJS module
  • andexports.__esMoudle = trueThe same effect, stronger compatibility

Doubt 2

exports["default"] = void 0;
Copy the code

Exports [“default”] = undefined; exports[“default”] = undefined

Details a

// import b from './b.js' becomes
var _b = _interopRequireDefault(require("./b.js"))
// b.value becomes
_b['default'].value
Copy the code

Parse the _interopRequireDefault function

  • This function is used to add to the moduledefualt, since commonJS does not export by default, add to ‘default’ for compatibility
  • _Underscores are used to avoid having the same name as other functions
  • _interopMost of the prefixed functions are intended to be compatible with older code

Details of the second

var _default = a
exports['default'] = _default
Copy the code

Equivalent to exports. Default = a

summary

By Babel conversion

  • willimportThe keyword becomesrequirefunction
  • willexportThe keyword becomesexportsobject

Multiple modules are packaged into a single module

To do that, we need to write a Bundler.

The first thing we need to know is what does the packaged code look like

var depRelation = [
  {key: 'index.js' , deps: ['a.js' , 'b.js'].code: function. }, {key: 'a.js' , deps: ['b.js'].code: function. }, {key: 'b.js' , deps: ['a.js'].code: function. }]// Why change depRelation from object to array?
// Because the first item of an array is an entry, objects have no concept of a first item
execute(depRelation[0].key) // Execute the entry file
function execute(key){
  var item = depRelation.find(i= > i.key === key)
  item.code(???) 
  // The code that executes item, so code better be a function, easy to execute
  // What parameters should be passed to code
  // The code needs to be perfected
}
Copy the code

There are three problems to be solved

  • depRelationIt’s an object. It needs to be an array
  • codeIt’s a string. How do I change it to a function
  • executeThe function needs to be perfected

DepRelation into an array

depRelation[key] = {deps: [], code};
Copy the code

Changed to

const item = { key, deps: [].code: es5Code }
depRelation.push(item);
// For other code changes, see the last implementation
Copy the code

Code is a string. How can I change it to a function

steps

  1. Package the code string in a functionfunction(require, module, exports), includingrequire module exportsThree parameters are specified by the CommonJS 2 specification
  2. Finally, write code into the generated file, and the code quotes will disappear, which can be interpreted as changing from a string to a code
code = ` var b = 1 b += 1 exports.defult = b `

code2 = ` function(require, module, exports) { $(code) } `
Copy the code

Perfecting the Execute function

Principal thinking

const modules = {} // Used to cache all modules
function execute(key) {
  if (modules[key]) {return modules[key]} // If the module is cached, return directly
  var item = depRelation.find(i= > i.key === key) // Find the module to execute
  var require = (path) = > { // Define the require function, which is executed by the require module
    return execute(pathToKey(path))
  }
  modules[key] = { __esModule: true } // Define an __esModule attribute to distinguish it from CommonJS
  var module = { exports: module[key] }
  // Executing this module will mount the exported content to exports.default, as shown in the compiled Babel code above
  item.code(require.module.module.exports) 
  return module.exports // {default: [exported object], __esModule: true}
}
Copy the code

Simple packer

What does a packed file look like

var depRelation = [ {key: 'index.js' , deps: ['a.js' , 'b.js'], code: function... }, {key: 'a.js' , deps: ['b.js'], code: function... }, {key: 'b.js' , deps: ['a.js'], code: Var modules = {} // modules is used to cache all modules. Execute (depRelation[0].key) function execute(key){var require = . var module = ... item.code(require, module, module.exports) ... }Copy the code

How do I generate this file?

The answer is simple: piece together a string and write it to a file

var dist = ''
dist += content
writeFileSync('dist.js', dist)
Copy the code

The dist file is too long, so I won’t put it here

Run the dist file and get

The packaged file is running successfully

summary

The packaged file is actually a collection of multiple modules and stored through the depRelation array. DeRelation [0] is the inlet file. Each element of the deRelation array is a module. A single element contains the module name key, the module’s dependency on deps, and the module’s executable code function. This function has three arguments: Require Module exports (CommonJS

We’ve seen how WebPack packaging works and made a simple wrapper, but it still has a lot of problems

  • The generated code has multiple duplicate __interopXXX functions
  • You can only import and run JS files
  • You can only understand import, you can’t understand require
  • Plug-ins not supported
  • Configuration entry files and dist file names are not supported

. So what’s the next step?

Loader

What is loader and why is loader needed

Looking back at the simple packager we made, this packager can only load JS, not CSS, what the hell

No, you need to write a CSS loader to support CSS

CSS – loader homemade version

Three-step logic

  • Our Bundler can only load JS
  • We want to load CSS

Corollary: If we can turn CSS into JS, then we can load CSS

How to convert to JS, var STR = [CSS code] stored in a variable

To make CSS work, create a new style tag, write the CSS code inside the 

// css-loader
const cssLoader = (code) = > { // Accept the code
    return `
      const str = The ${JSON.stringify(code)}
      if (document) {
        const style = document.createElement('style')
        style.innerHTML = str
        document.head.appendChild(style)
      }
    `
}

module.exports = cssLoader
Copy the code

Loader has been written, how to use it?

When the depRelation object was created, the code was processed using CSS-Loader

The complete code

The single responsibility principle for Webpack

Each loader in webpack does only one thing

Currently our loader does two things

  • Convert CSS to JS strings
  • Put the JS string inside the style tag

Modify the target to make two consecutive calls to the Loader

failed

Follow the goals above

Css-loader converts CSS to JS string

// css-loader 
const cssLoader = (code) = > {
    return `
      const str = The ${JSON.stringify(code)}
      module.exports str
    `
}

module.exports = cssLoader
Copy the code

Style-loader inserts JS strings into the style tag

const styleLoader = (code) = > {
    return `
      if (document) {
        const style = document.createElement('style')
        style.innerHTML = The ${JSON.stringify(code)} 
        document.head.appendChild(style)
      }
    `
}

module.exports = styleLoader
Copy the code

And then pack it again…

The result is this

Yi? Why are there weird strings

In fact, the string added to style-loader is not the code exported by CSS-loader, but the CSS code in the STR variable, so this cannot be implemented

How is webpack style-loader implemented

Webpack style – loader source code

Injectstylesheet into styleTag (content,… code

Webpack’s implementation differs from ours in that webPack can get the required code via request

summary

This time we tried to write the loader ourselves, but ran into a pit for this reason

Because style-loader doesn’t just translate code

  • Loaders like Saas-Loader and less-loader translate code from one language to another
  • This loader can be called in chain mode
  • But style-loader is inserting code, not translating it, so you need to find when and where to insert it
  • The timing of style-loader insertion is after csS-loader gets the result

Webpack is implemented as follows:

Injectstylesheet into styleTag (content,… code

The final summary

After many attempts, we have implemented a simple packer and loader. Although it is far from webPack, we are all doing the same thing, that is, analyzing dependencies, generating bundles, and finally exporting dist files. Loader’s function is to convert non-JS modules, Convert to JS modules, because WebPack only recognizes JS

This article is quite long, thank you to see the officer here, if you feel useful, trouble to move your little hands point a thumbs-up, thank you

If you are interested in the source code, you can go to this blog to analyze the Webpack source code