preface

When it comes to lexical analysis, I think many students may think of Babel, Acorn and other tools immediately. There is no denying that they are all powerful 😶.

However, es-module-lexer trumps them when it comes to analyzing ES Module statements on today’s topic.

So, today we will focus on the following 2 points, a simple es-module-lexer:

  • Es – the module – the lexer
  • How to apply ES-module-Lexer in actual scenarios

1 know es – the module – the lexer

Es – Module-Lexer is a toolkit for lexical analysis of ES Module statements. It is compressed to just 4 KiB and uses Inline WebAssembly for fast lexical analysis of ES Module statements.

1 kib = 1024 Byte

So, exactly how soon? According to the official example, Angular1 (720 KiB) uses Acorn parsing in 100 ms, while es-Module-Lexer parsing takes 5 ms, or 1/20 😵.

The use of es-module-lexer is also very simple. It provides an init Promise object and a parse method.

1.1 Init (Promise object)

Init must Resolve (Resolve) before the parse() method, which can be implemented in 3 steps:

  • First, callWebAssembly.compile()Method to compile the WebAssembly binary toWebAssembly.Module 的 Promiseobject
  • Then, callWebAssembly.Instantiate()Method to create an instance
  • Finally, it can be accessed on the instanceexportsProperty to get the methods provided by the calling module

The code for this procedure is:

let wasm;
const init = WebAssembly.compile(
  (binary= > typeof window! = ='undefined' && typeof atob === 'function' ? Uint8Array.from(atob(binary), x= > x.charCodeAt(0)) : Buffer.from(binary, 'base64'(a))'WASM_BINARY')
).then(WebAssembly.instantiate)
 .then(({ exports }) = > { wasm = exports; });
Copy the code

The binary code here is compiled from code that implements lexical analysis of ES Module statements implemented by C. Also, you can see that the instance exports is assigned to wASM.

1.2 the parse () method

The parse() method implements lexical analysis of ES Module syntax using the methods provided by the Webassembly.module shown above (that is, WASM).

The code for this procedure (pseudocode) :

function parse (source, name = The '@') {
  if(! wasm)return init.then(() = > parse(source));
  // Call a method on wASM to perform the corresponding operation
  return [imports, exports,!!!!! wasm.f()]; }Copy the code

Note that the methods provided on WASM are not analyzed here, so you can find out for yourself

As you can see, if we do not have Resolve (parse) init before calling the parse() method, the parse() method will Resolve (parse) init itself first. The parse() method is then called and returned in.then, so in this case the parse() method returns a Promise object.

Of course, in any case, the nature of the parse() method is to return an array of length 3. And the main ones we use are imports and exports.

Imports and exports are arrays where each element (object) represents the parsed result of an import statement, including the name of the imported or exported module, its location in the source code, and so on.

Next, let’s look at the basic use of ES-Module-Lexer through a simple example.

1.3 Basic Usage

First, we define a parseImportSyntax() method based on es-module-lexer:

const { init, parse } = require("es-module-lexer")

async function parseImportSyntax(code = "") {
  try {
    await init
    
    const importSpecifier = parse(code)
    return importSpecifier
  } catch(e) {
    console.error(e)
  }
}
Copy the code

You can see that the parseImportSyntax() method returns the parse result. Suppose we need to parse the statement importing ant-Design-Vue’s Button component:

const code = `import { Button } from 'ant-design-vue'`
parseImportSyntax(code).then(importSpecifier= > {
  console.log(importSpecifier)
})
Copy the code

Corresponding output:

[[{n: 'ant-design-vue'.s: 24.e: 38.ss: 0.se: 39.d: -1}], [],true 
]
Copy the code

Since we declared only the import statements, the final result of the parse is that the imports have elements with meanings for each attribute:

  • nRepresents the name of the module
  • sRepresents the starting position of the module name in the import statement
  • eRepresents where the module name ends in an import statement
  • ssRepresents the start of the import statement in the source code
  • seRepresents where the import statement ends in the source code
  • dIndicates whether the import statement is a dynamic import. If so, it is the corresponding start position. Otherwise, the default value is -1

So, after a brief understanding of the implementation principle and use of ES-Module-Lexer, I think students may think about how to use it in the actual scene? (Read more at 😎)

2 How to apply ES-module-Lexer in actual scenarios

While students may not realize where the ES-Module-Lexer is used, it has already entered our normal development.

Let’s take the viet-plugin-style-import plug-in as an example to see how it uses es-module-lexer. (Don’t go away, it’s going to be fun 😋)

2.1 Brief analysis of the principle of viet-plugin-style-import

Before we talk about the use of es-module-lexer in vite-plugin-style-import, we need to know what vite-plugin-style-import does.

It solves the problem of manually importing the corresponding component style when we import components on demand. For example, when using ant-design-vue, introducing a Button on demand simply declares:

import { Button } from "ant-design-vue"
Copy the code

Then, the corresponding code fragment is processed by viet-plugin-style-import:

import { Button } from 'ant-design-vue';
import 'ant-design-vue/es/button/style/index.js';
Copy the code

The realization of this process can be divided into the following three steps:

  • Use es-module-lexer for lexical analysis of source code import statements

  • The import statement of the style file is constructed according to the configuration item of the viet-plugin-style-import in the configuration file viet.config.js

  • Optionally inject specific code into source code, depending on the environment (Dev or Prod can be distinguished)

2.2 Use the Dark Magic of ES-Module-Lexer

In the basic usage section of 1.3, we explained that es-module-lexer parsing an import statement only returns information about the imported module. This is not sufficient in the viet-plugin-style-import section.

This is because viet-plugin-style-import also needs to know which components of the module are imported at this time, so that it can concatenate and generate the import statement of the corresponding style file.

This is where the dark magic of es-module-lexer comes in. We can replace the import statement with export, and es-module-Lexer will parse the exported component information (surprise 😲)!

For example, in the same ant-design-vue Button example above, replacing import would look like this:

export { Button } from "ant-design-vue"
Copy the code

The result of the es-module-lexer parsing is as follows:

[[{n: 'ant-design-vue'.s: 24.e: 38.ss: 0.se: 39.d: -1}], ['Button'].true
]
Copy the code

As you can see, Button is placed in exports of the parsed result (the second element of the array), so we know which components use the imported module 😎.

This process, in viet-plugin-style-import, is accomplished by the transformImportVar() method:

function transformImportVar(importStr: string) {
  if(! importStr) {return [];
  }

  const exportStr = importStr.replace('import'.'export').replace(/\s+as\s+\w+,? /g.', ');
  let importVariables: readonly string[] = [];
  try {
    importVariables = parse(exportStr)[1];
    debug('importVariables:', importVariables);
  } catch (error) {
    debug('transformImportVar:', error);
  }
  return importVariables;
}
Copy the code

conclusion

It is interesting to note that awesome-Vite has two plugins that support importing component style files on demand. Through reading, I think the students should know which to use 😎! Finally, summarize the advantages of ES-Module-Lexer in one sentence, that is: “Fast enough to fly.” If there are any improper expressions or mistakes in the article, students are welcome to Issue ~

Thumb up 👍

If you get something from this post, please give me a “like”. This will be my motivation to keep sharing. Thank you