Full text 5000 words, in-depth analysis of Webpack Loader features, operating mechanism, development skills, welcome to like attention. Writing is not easy, without the consent of the author, prohibit any form of reprint!!

There’s a lot of stuff out there about Webpack Loader, and it’s hard to get around it, but writing Webpack blog series can’t get around it, so I’ve read more than 20 open source projects and tried to summarize some of the things you need to know when writing a Loader. Contains:

So, let’s get started.

Welcome to follow the public account [Tecvan] and reply [1] to get the brain map of Webpack knowledge system

Know the Loader

If I had to conclude, I think Loader is a content translator with side effects!

At its core, Webpack Loader can only implement content converters — converting a wide variety of resources into standard JavaScript content formats, such as:

css-loaderConvert the CSS to__WEBPACK_DEFAULT_EXPORT__ = ".a{ xxx }"format
html-loaderConvert HTML to__WEBPACK_DEFAULT_EXPORT__ = "<! DOCTYPE xxx"format
vue-loaderIt’s a little bit more complicated.vueThe file is converted to multiple JavaScript functions, corresponding to template, JS, CSS, and Custom Block respectively

So why do we need to do this conversion? Essentially, Webpack only recognizes text that conforms to the JavaScript specification (other parsers were added after Webpack 5) : In the build (make) stage, acorn will be called to convert the text into AST objects when parsing the content of the module, so as to analyze the code structure and module dependence. This logic does not work for images, JSON, Vue SFC and other scenes, so Loader needs to intervene to convert resources into content forms that Webpack can understand.

Plugin is another set of extension mechanism of Webpack, with stronger functions. It can insert specialized processing logic in the hooks of each object. It can cover the whole life process of Webpack, and its ability, flexibility and complexity will be much stronger than Loader.

Loader basis

At the code level, Loader is usually a function with the following structure:

module.exports = function(source, sourceMap? , data?) {// source indicates the input of the loader, either the file content or the processing result of the previous loader. };Copy the code

The Loader function accepts three arguments:

source: resource input, the contents of the resource file for the first loader to execute; The subsequent loaders are the execution results of the previous loader
sourceMap: Optional parameter, codesourcemapstructure
data: Optional. Other information that needs to be passed in the Loader chain, such asposthtml/posthtml-loaderThe AST object of the parameter is passed through this parameter

Source is the most important parameter, and most loaders need to translate source into another form of output, such as webpack-contrib/raw-loader.

/ /...
export default function rawLoader(source) {
  // ...

  const json = JSON.stringify(source)
    .replace(/\u2028/g.'\\u2028')
    .replace(/\u2029/g.'\\u2029');

  const esModule =
    typeofoptions.esModule ! = ='undefined' ? options.esModule : true;

  return `${esModule ? 'export default' : 'module.exports ='} ${json}; `;
}
Copy the code

This code wraps the text content into JavaScript modules, for example:

// source
I am Tecvan

// output
module.exports = "I am Tecvan"
Copy the code

Once modularized, this text content turns into a resource module that Webpack can handle and that other modules can reference and use.

Return multiple results

In the preceding example, a return statement is used to return the result of the processing. The Loader can also return more information as a callback, which can be used by downstream loaders or Webpack itself, for example, in webpack-contrib/eslint-loader:

export default function loader(content, map) {
  // ...
  linter.printOutput(linter.lint(content));
  this.callback(null, content, map);
}
Copy the code

Callback (null, Content, map) returns both the translated content and the sourcemap content. Callback’s full signature is as follows:

this.callback(
    // The null value can be passed when the Loader runs normally
    err: Error | null.// Translation result
    content: string | Buffer,
    // Source code sourcemap informationsourceMap? : SourceMap,// Any value to be passed between loaders
    // Often used to pass ast objects to avoid repeated parsingdata? : any );Copy the code

Asynchronous processing

In cases where asynchronous or CPU-intensive operations are involved, the Loader can also return processing results asynchronously, such as the core logic in webpack-contrib/less-loader:

import less from "less";

async function lessLoader(source) {
  // 1. Get the asynchronous callback function
  const callback = this.async();
  // ...

  let result;

  try {
    // 2. Call less to translate the contents of the module into CSS
    result = await (options.implementation || less).render(data, lessOptions);
  } catch (error) {
    // ...
  }

  const { css, imports } = result;

  // ...

  // 3. The translation is complete and the result is returned
  callback(null, css, map);
}

export default lessLoader;
Copy the code

In less-loader, the logic is divided into three steps:

callthis.asyncGets an asynchronous callback function, in which case Webpack marks the Loader as an asynchronous Loader and suspends the current execution queue untilcallbackBe triggered
calllessThe library translates less resources into standard CSS
Invoke an asynchronous callbackcallbackReturn processing result

This. async returns the same asynchronous callback signature as this.callback in the previous section, so I won’t repeat it here.

The cache

Loaders provide a convenient way for developers to scale, but the various resource content translation operations performed in Loaders are usually CPU-intensive — which can cause performance problems in a single-threaded Node scenario; Alternatively, asynchronous loaders may suspend subsequent Loader queues until the asynchronous Loader triggers a callback, which can cause the entire Loader chain to take too long.

To do this, Webpack caches Loader’s execution results by default until the resource or resource dependency changes. Developers need to have a basic understanding of this, and can explicitly declare no caching if necessary via this.cachable, for example:

module.exports = function(source) {
  this.cacheable(false);
  // ...
  return output;
};
Copy the code

Context and Side Effect

In addition to acting as a content translator, the Loader runtime can have side effects beyond content conversion by having a limited impact on the Webpack compilation process through contextual interfaces.

Context information can be obtained through this, this object by NormolModule. CreateLoaderContext function before calling Loader created, commonly used interface include:

const loaderContext = {
    // Obtain the configuration information of the Loader
    getOptions: schema= > {},
    // Add a warning
    emitWarning: warning= > {},
    // Add an error message, noting that this does not interrupt Webpack running
    emitError: error= > {},
    // Parse the specific path to the resource file
    resolve(context, request, callback) {},
    // Submit files directly. The submitted files are directly output to FS without subsequent chunk and module processing
    emitFile: (name, content, sourceMap, assetInfo) = > {},
    // Add additional dependency files
    // In watch mode, resource recompilation is triggered when dependent files change
    addDependency(dep){}};Copy the code

AddDependency, emitFile, emitError, and emitWarning all have side effects on subsequent compilation processes, such as less-loader containing the following code:

  try {
    result = await (options.implementation || less).render(data, lessOptions);
  } catch (error) {
    // ...
  }

  const { css, imports } = result;

  imports.forEach((item) = > {
    // ...
    this.addDependency(path.normalize(item));
  });
Copy the code

Explain, the code in the first call less compiled file contents, then iterate through all the import statement, that is the case result. Imports arrays, each invocation enclosing addDependency function to import other resources are registered as dependence, Any subsequent changes to these other resource files trigger a recompilation.

Loader chain call

In practice, multiple Loaders can be configured for a certain resource file. The loaders are executed from front to back (pitch) and then from back to front (pitch) in order to form a set of content translation workflow. For example, for the following configuration:

module.exports = {
  module: {
    rules: [{test: /\.less$/i,
        use: [
          "style-loader"."css-loader"."less-loader",],},],},};Copy the code

This is a typical less processing scenario. For files with the suffix. Less, CSS, and style, three Loaders cooperate to process resource files. According to the defined sequence, Webpack parses the contents of less files and passes them to less-Loader first. The result returned by less-loader is forwarded to CSS-Loader for processing. The result of csS-loader is passed to style-loader; The final processing result of style-loader shall prevail. The simplified process is as follows:

In the preceding example, the three Loaders play the following roles:

less-loaderLess => CSS conversion, output CSS content, can not be directly applied in the Webpack system
css-loader: Wraps CSS content like thismodule.exports = "${css}"The wrapped content complies with JavaScript syntax
style-loaderTo do this, you simply pack the CSS module into the require statement and at runtime call injectStyle and other functions to inject the content into the page’s style tag

The three Loaders each perform part of the content transformation work, forming a call chain from right to left. The design of chain call has two advantages. One is to keep the single responsibility of a single Loader and reduce the complexity of the code to a certain extent. Second, fine-grained functions can be assembled into complex and flexible processing chains to improve the reusability of a single Loader.

However, this is only part of the chain call, and there are two problems:

Once the Loader chain is started, it will not end until all loaders have been executed, with no chance of interruption unless an explicit exception is thrown
Some scenarios do not care about the content of the resource, but the Loader does not execute until the source content has been read

To solve these two problems, Webpack overlays the concept of pitch on top of Loader.

Loader Pitch

There are many articles about Loader on the Internet, but most of them do not give enough in-depth introduction to pitch, an important feature, and do not explain clearly why pitch is designed and what common use cases pitch has.

In this section, I will discuss the feature of Loader pitch from the three dimensions of What, how and why.

What is the pitch

Webpack allows a function called pitch to be mounted on this function, which is executed earlier than the Loader itself at runtime, for example:

const loader = function (source){
    console.log('After execution')
    return source;
}

loader.pitch = function(requestString) {
    console.log('Execute first')}module.exports = loader
Copy the code

Full signature of Pitch function:

function pitch(
    remainingRequest: string, previousRequest: string, data = {}
) :void {}Copy the code

Contains three parameters:

remainingRequest: Resource request string after the current loader
previousRequest: List of loaders experienced before executing the current loader
data: corresponds to the Loader functiondataThe same is used to pass information that needs to be propagated in the Loader

These parameters are not complex, but are closely related to requestString, so let’s look at an example to understand:

module.exports = {
  module: {
    rules: [{test: /\.less$/i,
        use: [
          "style-loader"."css-loader"."less-loader"],},],},};Copy the code

The parameters in csS-loader. pitch are as follows:

// Loader list and resource path after CSS-loaderremainingRequest = less-loader! ./xxx.less// css-loader Specifies the previous loader list
previousRequest = style-loader
/ / the default value
data = {}
Copy the code

Scheduling logic

Pitch in Chinese refers to Pitch, Pitch, strength, and the highest point of things, etc. I think Pitch is completely ignored because of this name, which reflects a whole set of life cycle concepts implemented by Loader.

In terms of implementation, the Loader chain execution process is divided into three stages: pitch, resource parsing and execution. In terms of design, it is very similar to the DOM event model, and PITCH corresponds to the capture stage. Execution corresponds to the bubbling phase; Between the two phases, Webpack will read and parse the resource contents, corresponding to the AT_TARGET phase of DOM event model:

The pitch phase executes the loader.pitch function (if any) from left to right in the configuration order. The developer can interrupt the execution of subsequent links when the pitch returns any value:

So why design pitch? After analyzing style-loader, VUe-loader, to-string-loader and other open source projects, I personally summed up two words: block!

Example: style – loader

Let’s review the less loading chain mentioned earlier:

less-loader: Converts the contents of less specifications to standard CSS
css-loader: Wraps CSS content as JavaScript modules
style-loader: Export the result of the JavaScript module tolink 、styleTags, etc., are attached to HTML to make CSS code run correctly in the browser

In fact, style-loader is only responsible for making CSS work in a browser environment. It doesn’t really care about the content, per se.

// ...
// The Loader itself does no processing
const loaderApi = () = > {};

// Concatenate module code according to parameters in pitch
loaderApi.pitch = function loader(remainingRequest) {
  / /...

  switch (injectType) {
    case 'linkTag': {
      return `${
        esModule
          ? `... `
          //Introducing the Runtime module:`var api = require(${loaderUtils.stringifyRequest(
              this.`!${path.join(__dirname, 'runtime/injectStylesIntoLinkTag.js')}`
            )}); Var content = require(${loaderUtils.stringifyRequest(
              this.`!!!!!${remainingRequest}`
            )}); content = content.__esModule ? content.default : content; `
      }/ /... `;
    }

    case 'lazyStyleTag':
    case 'lazySingletonStyleTag': {
        / /...
    }

    case 'styleTag':
    case 'singletonStyleTag':
    default: {
        // ...}}};export default loaderApi;
Copy the code

Key points:

loaderApiIs an empty function and does no processing
loaderApi.pitchThe exported code contains:
- Introduce runtime modulesruntime/injectStylesIntoLinkTag.js
- reuseremainingRequestParameter to reintroduce the CSS file

The running results are as follows:

var api = require('xxx/style-loader/lib/runtime/injectStylesIntoLinkTag.js')
var content = require('!!!!! css-loader! less-loader! ./xxx.less');
Copy the code

Note that the style-loader pitch function returns this paragraph, and the subsequent loader will not continue to execute, and the current call chain is broken:

After that, Webpack continues to parse and build the result returned by style-loader and encounters an inline loader statement:

var content = require('!!!!! css-loader! less-loader! ./xxx.less');
Copy the code

So from the Webpack point of view, the loader chain is actually called twice on the same file, the first time breaking in the style-loader pitch and the second time skipping the style-loader based on the contents of the inline loader.

Similar skills also appear in other warehouses, such as VUe-loader. Interested students can check my article “Webpack Case — Vue-Loader Principle Analysis” posted on ByteFE official account, which will not be expanded here.

Advanced skills

The development tools

Webpack provides two utilities for Loader developers that appear frequently in many open source Loaders:

Webpack /loader-utils: provides a series of utility functions such as reading configuration, requestString serialization and deserialization, and calculating hash values
Webpack /schema-utils: parameter verification tool

The specific interfaces of these tools are clearly described in the corresponding readme. Here are some common Loader examples: How to obtain and verify user configurations. How to concatenate output file names.

Get and verify the configuration

Loaders usually provide some configuration items for developers to customize the runtime behavior. Users can use the use.options property of the Webpack configuration file to set the configuration, for example:

module.exports = {
  module: {
    rules: [{
      test: /\.less$/i,
      use: [
        {
          loader: "less-loader".options: {
            cacheDirectory: false}},],}],},};Copy the code

In Loader, the getOptions function of loader-utils library is used to obtain user configurations and the validate function of schema-utils library is used to verify the validity of parameters, for example, CSS-loader:

// css-loader/src/index.js
import { getOptions } from "loader-utils";
import { validate } from "schema-utils";
import schema from "./options.json";


export default async function loader(content, map, meta) {
  const rawOptions = getOptions(this);

  validate(schema, rawOptions, {
    name: "CSS Loader".baseDataPath: "options"});// ...
}
Copy the code

When using schema-utils for validation, you need to declare the configuration template in advance, which is usually processed into an additional JSON file, such as “./options.json” in the example above.

Splicing output file name

Webpack supports setting output.filename, the name of the output file, in a manner similar to [path]/[name]-[hash].js. However, some scenarios such as webpack-contrib/file-loader need to concatenate the result based on the file name of the asset.

File-loader can import text or binary files such as PNG, JPG, SVG in JS module and write the file to the output directory. InterpolateName interpolates the name and path of a resource in file-loader using interpolateName provided by loader-utils.

import { getOptions, interpolateName } from 'loader-utils';

export default function loader(content) {
  const context = options.context || this.rootContext;
  const name = options.name || '[contenthash].[ext]';

  // Splice the name of the final output
  const url = interpolateName(this, name, {
    context,
    content,
    regExp: options.regExp,
  });

  let outputPath = url;
  // ...

  let publicPath = `__webpack_public_path__ + The ${JSON.stringify(outputPath)}`;
  // ...

  if (typeof options.emitFile === 'undefined' || options.emitFile) {
    // ...

    // Write a file
    this.emitFile(outputPath, content, null, assetInfo);
  }
  // ...

  const esModule =
    typeofoptions.esModule ! = ='undefined' ? options.esModule : true;

  // Return to modular content
  return `${esModule ? 'export default' : 'module.exports ='} ${publicPath}; `;
}

export const raw = true;
Copy the code

Core logic of the code:

According to the Loader configuration, invokeinterpolateNameMethod to concatenate the full path to the object file
Call contextthis.emitFileInterface, write out the file
returnmodule.exports = ${publicPath}Other modules can refer to this file path

In addition to file-loader, csS-loader and ESlint-loader are all useful to this interface. If you are interested, please go to the source code.

Unit testing

Writing unit tests in Loader is very profitable. On the one hand, developers don’t have to write demo and build test environment. On the one hand, projects with some level of test coverage usually mean higher and more consistent quality for end users.

After reading more than 20 open source projects, I have summarized a common unit Testing process for Webpack Loader scenarios, using Jest · 🃏 Delightful JavaScript Testing as an example:

Create the Webpack instance and run the Loader
Obtain Loader execution results, compare and analyze them to determine whether they meet expectations
Determine whether errors occur during execution

How to run Loader

There are two ways to do this. One is to run the Webpack interface in the Node environment and use code instead of the command line to compile. Many frameworks use this method, such as VUe-loader, stylus-Loader, babel-loader, etc. The disadvantage is that the operation efficiency is relatively low (can be ignored).

Take posthTML/posthML-loader as an example, which creates and runs the Webpack instance before launching the test:

/ / posthtml - loader/test/helpers/compiler. The js file
module.exports = function (fixture, config, options) {
  config = { / *... * / }

  options = Object.assign({ output: false }, options)

  // Create a Webpack instance
  const compiler = webpack(config)

  // Output the build results as MemoryFS to avoid disk writes
  if(! options.output) compiler.outputFileSystem =new MemoryFS()

  // execute, and return the result as a promise
  return new Promise((resolve, reject) = > compiler.run((err, stats) = > {
    if (err) reject(err)
    // Asynchronously returns the execution result
    resolve(stats)
  }))
}
Copy the code

Tip: as shown in example above, use the compiler. OutputFileSystem = new MemoryFS () statement to set the Webpack into output to the memory, can avoid to write operation, improve compilation speed.

The other method is to write a series of mock methods to build a mock Webpack runtime environment, such as emaphp/underscore-template-loader.

The comparison results

The stats object contains almost all information about the compile process, including time, artifacts, modules, chunks, errors, warnings, and so on. I shared several Webpack utility analysis tools that have been covered in depth in previous articles for those interested.

In a test scenario, the compiled final output can be read from a STATS object, such as the style-loader implementation:

/ / style - loader/SRC/test/helpers/readAsset js file
function readAsset(compiler, stats, assets) = >{
  const usedFs = compiler.outputFileSystem
  const outputPath = stats.compilation.outputOptions.path
  const queryStringIdx = targetFile.indexOf('? ')

  if (queryStringIdx >= 0) {
    // Parse out the output file path
    asset = asset.substr(0, queryStringIdx)
  }

  // Read the contents of the file
  return usedFs.readFileSync(path.join(outputPath, targetFile)).toString()
}
Copy the code

To illustrate, this code first calculates the file path of the asset output and then calls the readFile method of outputFileSystem to read the file contents.

Next, there are two ways to analyze content:

Call the Jestexpect(xxx).toMatchSnapshot()Assertions determine whether the current run results are consistent with previous runs to ensure consistency over multiple changes, and are used extensively by many frameworks
Interpret the resource content and determine whether it meets expectations. For example, in the unit test of less-Loader, the same code will be compiled twice by Less, once by Webpack and once directly calledlessAnd then analyze whether the results of the two runs are the same

If you are interested in this, you are strongly advised to check out the test directory of less-Loader.

Abnormal judgment

Finally, we need to determine if there is an exception during compilation, again from the stats object:

export default getErrors = (stats) = > {
  const errors = stats.compilation.errors.sort()
  return errors.map(
    e= > e.toString()
  )
}
Copy the code

In most cases, you want to compile error-free, and you just need to determine if the resulting array is empty. In some cases, you can expect(XXX).tomatchSnapshot () assertions to compare the results before and after the update with the snapshot if you need to determine whether a specific exception is thrown.

debugging

During Loader development, there are some tips to improve debugging efficiency, including:

Use NDB tools for breakpoint debugging
usenpm linkLink the Loader module to the test project
useresolveLoaderAdd the Loader directory to the test project, for example:

// webpack.config.js
module.exports = {
  resolveLoader: {modules: ['node_modules'.'./loaders/'].}}Copy the code

Irrelevant summary

This is the seventh article in the Webpack principle analysis series. To tell the truth, I did not expect to write so much at the beginning, and I will continue to focus on this field of front-end engineering in the future. My goal is to accumulate a book of my own, interested students are welcome to like and pay attention to it.

The articles

Share several Webpack utility analysis tools

Webpack 4+ Collection of excellent learning materials

[10,000 words summary] One article thoroughly understand the core principle of Webpack

Webpack plug-in architecture in-depth explanation

10 minutes introduction to Webpack: module.issuer properties

Dependency Graph: Dependency Graph (Dependency Graph

The Webpack Chunk subcontracting rules

Webpack Principles Series 6: Thoroughly understand the Webpack runtime

mo4tech.com (Moment For Technology) is a global community with thousands techies from across the global hang out!Passionate technologists, be it gadget freaks, tech enthusiasts, coders, technopreneurs, or CIOs, you would find them all here.

Webpack Principles series 7: How to develop Loader

Know the Loader

Loader basis

Return multiple results

Asynchronous processing

The cache

Context and Side Effect

Loader chain call

Loader Pitch

What is the pitch

Scheduling logic

Example: style – loader

Advanced skills

The development tools

Get and verify the configuration

Splicing output file name

Unit testing

How to run Loader

The comparison results

Abnormal judgment

debugging

Irrelevant summary

Webpack Principles series 7: How to develop Loader

Know the Loader

Loader basis

Return multiple results

Asynchronous processing

The cache

Context and Side Effect

Loader chain call

Loader Pitch

What is the pitch

Scheduling logic

Example: style – loader

Advanced skills

The development tools

Get and verify the configuration

Splicing output file name

Unit testing

How to run Loader

The comparison results

Abnormal judgment

debugging

Irrelevant summary

Related Posts

Programmers usually work overtime seriously, how to effectively promote their own?

TypeScript implements AVL and red-black trees

How to ensure iOS code # high quality # warning Settings