AST details and application

Before we get to the AST, let’s make a brief statement about how the JavaScript engine works:

As you can see from the figure above, the first thing a JavaScript engine does is compile JavaScript code into an abstract syntax tree, hence this brief overview of the AST abstract syntax tree.

What is AST Abstract syntax tree

As we all know, in a traditional compilation language, a piece of program source code goes through three steps, collectively called “compilation”, before execution:

• Word segmentation/lexical analysis

  This process breaks down a string of characters into meaningful blocks of code, collectively known as lexical units (tokens).

  For example: let a = 1. This program is usually broken down into the following lexical units: let, a, =, 1; Whether whitespace is considered a unit of this method depends on what whitespace means in the language.

• Parsing/parsing

  The process is to convert a lexical unit stream into a nested tree of elements that represents the syntax structure of the program. This tree is called an abstract syntax code (AST).

• Code generation

  The process of converting an AST into executable code is called code generation.

Abstract syntax code (AST) is a tree representation of the abstract syntax structure of the source code. Every node on the tree represents a structure in the source code. Abstract representation transforms THE JS code into a structured data structure. This data structure is essentially a big JSON object, which we’re all familiar with, like a tree with many leaves. With roots, trunks, branches and leaves, no matter how small or big, it is a complete tree.

Simply understood, we write code according to certain rules into a tree structure.

Second, the use of AST

AST is used not only for compiling JavaScript engines, but also for actual development. For example, we commonly use Babel plug-in to transform ES6 into ES5, use UglifyJS to compress the code, CSS preprocessor, development of WebPack plug-in, VUE-CLI front-end automation tools and so on. These underlying principles are realized based on AST, AST is very powerful. Helps developers understand the essence of JavaScript.

3. Structure of AST

Let’s start with a simple set of AST trees:

Const team = 'big round FE'Copy the code

After transformation, the AST tree structure is output as follows:

{ "type": "Program", "start": 0, "end": 18, "body": [ { "type": "VariableDeclaration", "start": 0, "end": 18, "declarations": [ { "type": "VariableDeclarator", "start": 6, "end": 18, "id": { "type": "Identifier", "start": 6, "the end" : 8, "name" : "team"}, "init" : {" type ":" Literal ", "start" : 11, "the end" : 18, "value" : "around the big FE", "raw" : "' around large FE '}}]", "kind" : "const"}], "sourceType" : "the module"}Copy the code

We can see that a standard AST structure can be understood as a JSON object, so we can use some methods to parse and manipulate it. Here we first provide an online detection tool, you can experience: esprima.org/demo/parse.

4. AST compilation process

AST compilation flowchart:

As we can see, the AST tool transforms the source code in four stages:

1. Lexical analysis scanner

var company = 'zhuanzhuan'
Copy the code

If the above codes exist, the whole code will be scanned to generate tokens flow in the lexical analysis stage. The scanning process is as follows:

• We can judge by conditional statement, this character is a letter “/”, “digital”, space, “(“,”) “, “;” And so on.

• If it’s a letter or a number, the process continues until it’s not, at which point the string found is a “var”, is a Keyword, and the next character is a “space”, generating {“type” : “Keyword”, “value” : “var”} into the array.

• It goes down and finds the letter ‘company'(because the last value it found was “var”, it may return an error if it finds the next character is not a letter) followed by a space, generating {“type” : “Identifier”, “value” : “Company”} in an array.

• {“Punctuator” : “Punctuator”, “value” : “=”}} {“Punctuator” : “Punctuator”, “value” : “=”}}

• ‘zhuanzhuan’ is found and {“type” : “String”, “value” : “zhuanzhuan”} is generated and placed in an array.

  The analysis is as follows:

  

  The code generates ToKens flows

2. Parser generates an AST tree

Here we use esprima to generate and install the dependencies NPM I esprima –save

Take the following code for example:

const company = 'zhuanzhuan'
Copy the code

To get its corresponding AST, we do the following for it:

const esprima = require('esprima');
let code = 'const company = "zhuanzhuan" ';
const ast = esprima.parseScript(code);
console.log(ast);
Copy the code

The running results are as follows:

$ node test.js
Script {
  type: 'Program',
  body: [
    VariableDeclaration {
      type: 'VariableDeclaration',
      declarations: [Array],
      kind: 'const'
    }
  ],
  sourceType: 'script'
}
Copy the code

So we have an AST tree

3. Traverse AST tree, add, delete, change and check

Here we use estraverse to do this, and install related dependencies NPM I estraverse –save

Same code as above, let’s change it to const team = ‘big round FE’

const esprima = require('esprima'); const estraverse = require('estraverse'); let code = 'const company = "zhuanzhuan" '; const ast = esprima.parseScript(code); estraverse.traverse(ast, { enter: function (node) { node.name = 'team'; Node. value = "turn around FE"; }}); console.log(ast);Copy the code

The running results are as follows:

$ node test.js Script { type: 'Program', body: [ VariableDeclaration { type: 'VariableDeclaration', declarations: [Array], kind: 'const', name: 'team', value: 'team'}, sourceType: 'script', name: 'team', value: 'team'}Copy the code

This completes our traversal of the AST update.

4. The generator converts the updated AST into code

Here we use escodeGen to generate and install the dependencies NPM I escodeGen –save

The overall code structure is as follows:

const esprima = require('esprima'); const estraverse = require('estraverse'); const escodegen = require('escodegen'); let code = 'const company = "zhuanzhuan" '; const ast = esprima.parseScript(code); estraverse.traverse(ast, { enter: function (node) { node.name = 'team'; Node. value = "turn around FE"; }}); const transformCode = escodegen.generate(ast); console.log(transformCode);Copy the code

You get the following result:

$node test.js const team = 'test.js ';Copy the code

This completes the AST compilation of a simple piece of code.

5. A brief analysis of Babel principle

The Babel plug-in works on abstract syntax trees.

The three main processing steps of Babel are: parse, transform and generate.

• parsing

  To parse code into abstract syntax trees (AST), each JS engine (such as V8 in Chrome) has its own AST parser, whereas Babel uses Babylon(github.com/babel/babyl… This phase converts the information in the token into the AST representation structure.

• conversion

  The transformation step receives the AST and traverses it, adding, updating, and removing nodes during this process. Babel maintains the overall state of the AST tree through depth-first traversal with babel-traverse, and can complete the replacement, deletion, or addition of nodes. This method takes the original AST and custom conversion rules and returns the converted AST.

• generate

  Code generation step to the final (AST) after a series of transformation converts a string of code, but also creates source mapping (source maps) (www.html5rocks.com/en/tutorial… .

  Code generation is simple: Depth-first traverses the AST and builds a string that represents the transformed code.

  Babel is then converted to JS code via babel-Generator by depth-first traversing the AST and building a string that represents the transformed code.

6. The application of AST abstract syntax tree in VUE;

In VUE, AST is mainly used in template compilation.

Let’s first look at the overall flow chart of vue template compilation:

Compiling templates in VUE consists of three main steps:

1. Parser phase: Parse the code inside the template into an AST abstract syntax tree;
2. Optimizer stage: Tag AST abstract syntax tree static tags to prevent repeated rendering (optimized diff algorithm);
3. Code generator stage: the optimized AST abstract syntax tree generates the render function string through generate function;

Let’s look at the overall implementation process of vue source code:

export const createCompiler = createCompilerCreator(function baseCompile ( template: string, options: CompilerOptions ): CompiledResult {// Generate ast const ast = parse(template.trim(), Options) // The ast abstract syntax tree static tag is tagged to prevent the recursive rendering if (options.optimize! > > optimize(ast, options)} const code = generate(ast, options) return {ast, optimize(ast, options)} render: code.render, staticRenderFns: code.staticRenderFns } })Copy the code

Parser in order to realize the function is the template parsing into AST, we here is mainly to analyze the code parsing stage, here is the main use of the parse () the function, in fact, the parser is divided into several internal parser, such as HTML parser, text parser, and filtering the parser, is one of the main HTML parser. HTML parser is used to parse HTML, it will trigger various hook functions in the process of parsing HTML, let’s look at the code implementation:

ParseHTML (template, {// Parse the start tag start (tag, attrs, unary, start, end) {}, // parse the end tag (tag, start, end) {}, Chars (text: string, start: number, end: number) {},Copy the code

Here’s an example:

<div> we are big round FE</div>Copy the code

When the template is parsed by the HTML parser, the hook functions are start, chars, and end.

So the HTML parser in implementation is a function, it has two parameters —- template and options, our template is a small section to intercept and parse, so we need to constantly loop intercept, let’s look at vue internal implementation principle:

Function parseHTML (HTML, options) {while (HTML) {if (! lastTag || ! IsPlainTextElement (lastTag) {// In vue to judge is text, comment, condition comment, DOCTYPE, end, start tag Var textEnd = html.indexof ('<'); Conditionalcomment.test (HTML) {// conditionalcomment.test (HTML) {// conditionalcomment.test (HTML) {} Var doctypeMatch = html.match(doctype); If (doctypeMatch) {} var endTagMatch = html.match(endTag); If (endTagMatch) {} var startTagMatch = parseStartTag(); if (startTagMatch) {} } var text = (void 0), rest = (void 0), next = (void 0); If (textEnd < 0) {text = HTML HTML = ""} if (textEnd < 0) {text = HTML HTML =" "} // If text text exists // call Options. Chars callback, Chars (text)}}else{// The parent element is script, style, textarea}}}Copy the code

The AST syntax tree is generated by the vUE parser in the following way :\vue\packages\ Weex-template-compiler \build.js

Conclusion:

The AST abstract syntax tree is one of the most basic things in JavaScript (the AST is a concept in any programming language, so I won’t go into details here), and the possibilities are endless. It’s like a screwdriver that can take apart the JavaScript machine. It allows us to see the essence of something, but it also allows us to bulk build any javascript code.

When we were young, we dreamed of changing the world. Now we can use our own programs to build the network world we live in, which is rich and colorful.

To borrow a lyric: I’m still the same boy I was, not a bit changed. Time is just a test, in the heart of faith has not diminished. I hope you can firm belief in the way of software development, go further and further…..

(To be continued)

reference

[1] Zhu Yongsheng. WebKit Technology Insider [M]. Publishing House of Electronics Industry

[2] KYLE SIMPSON. JavaScript you Don’t Know (Vol. 1)[M]. Posts and Telecommunications Press

[3] Yue Ying. Simple And Profound Vue. Js [M]. Posts and Telecommunications Press

[4] V8 engine details -AST juejin.cn/post/684490…

[5] Babel Plugin manual github.com/jamiebuilds…