The profile
This article will study AST through the following aspects
- Why do you want to know AST and briefly explain its importance in development
- What is AST and have an intuitive understanding of AST
- How is an AST generated and how does it parse code into an AST
- The concrete application of AST is analyzed by interpreting the principle of Babel, the compilation process of VUE template and the realization principle of Prettier.
- What else can AST do, combine the work and think about what AST can do for us
Why learn AST
AST (Abstract Syntax tree) plays a very important role in the development process, but we rarely touch it directly.
Whether it is code compilation (Babel), packaging (Webpack), code compression, CSS preprocessing, code validation (ESLint), code beautification (Pretiier), compilation of templates in Vue, all of these can be implemented without the AST.
Learning AST can help us better understand the principles of these tools, and at the same time, we can use it to develop some tools to optimize our development process and improve development efficiency.
What is the AST
The AST is a tree-like representation of the abstract syntactic structure of source code.
In different scenarios, different parsers parse the source code into abstract syntax trees.
So let’s see visually what an AST looks like
code
let answer = 2 * 3;
Copy the codeCorresponding AST syntax tree
{
"type": "Program"."body": [{"type": "VariableDeclaration"."declarations": [{"type": "VariableDeclarator"."id": {
"type": "Identifier"."name": "answer"
},
"init": {
"type": "BinaryExpression"."operator": "*"."left": {
"type": "Literal"."value": 2."raw": "2"
},
"right": {
"type": "Literal"."value": 3."raw": "3"}}}]."kind": "let"}]."sourceType": "script"
}
Copy the codeSo how is an AST generated?
How is AST generated
The AST transforms THE JS source code into an abstract syntax tree through the JS Parser. It is divided into two steps
1. The word segmentation
Split the entire code string into an array of syntax units (tokens). JS syntax unit (token) function, return, operators, parentheses, numbers, strings, etc. the smallest unit that can be parsed. There are mainly the following types:
-
Identifier A string of consecutive characters that are not enclosed in quotation marks and can contain letters, digits, underscores (_), and $. Identifiers can be keywords such as var, return, function, built-in constants such as true, false, or a variable. Specific which kind of semantic, word segmentation stage does not distinguish, as long as the correct separation can be.
-
Digits hexadecimal, decimal, octal, and scientific expressions are all minimal units
-
Operators: +, -, *, /, etc
-
String to the computer, string will only participate in the calculation and display, there is no need to analyze the specific subdivision
-
Comments, whether line comments or block comments, do not care about their contents to the computer, so they can be used as the smallest unit that cannot be split again
-
Spaces Successive Spaces, newlines, indentation, etc. have no actual logical meaning as long as they are not in a string, so successive Spaces can be used as a syntax unit.
-
Other things, braces, brackets, braces, colons, etc.
Again using the above code as an example, the syntax unit array generated after word segmentation is as follows
[{"type": "Keyword"."value": "var"."range": [0, 3]}, {"type": "Identifier"."value": "answer"."range": [4, 10]}, {"type": "Punctuator"."value": "="."range": [11, 12]}, {"type": "Numeric"."value": "2"."range": [13, 14]}, {"type": "Punctuator"."value": "*"."range": [15, 16]}, {"type": "Numeric"."value": "3"."range": [17, 18]}, {"type": "Punctuator"."value": ";"."range": [18, 19]}]Copy the code2. Semantic analysis
The purpose of semantic analysis is to combine the grammatical units from word segmentation and analyze and determine the relations among them.
In simple terms, semantic analysis can be understood as the identification of statements and expressions.
- Statement, an area of code with boundaries. Two adjacent statements do not syntactically affect each other. Such as:
var a = 1; if(xxx){xxx} - An expression is a piece of code that eventually produces a result, which can be embedded in another expression and contained in an expression. Such as:
a++.i > 0 && i< 6
Semantic analysis is a recursive process that converts an array of segmented words into a tree representation. It also validates the syntax and throws syntax errors if there are any.
Specific applications of AST
Babel, Webpack, CSS preprocessing, ESLint, etc., all apply to AST trees. What role does AST play? ? Let’s take a look.
Let’s start with an implementation of how Babel works.
Implementation principles of Babel
Babel is a javascript compiler that compiles ES6 syntax into ES5
Babel’s work can be divided into three phases:
Step 1 Parse (Parse) The code is parsed into an abstract syntax tree using the parser Babylon
The babel-traverse plugin performs depth-first traversal of the abstract syntax tree. If the nodes need to be traversed, they can be added, updated or removed directly on AST objects. For example, arrow functions can be converted to normal functions to obtain a new AST tree.
Generate es5 code from the AST tree via babel-Generator
Vue template compilation process
Vue provides two versions, one is Runtime + Compiler, the other is Runtime only, the former contains the compiled code, the compilation process is put at Runtime, the latter does not contain the compiled code, Use webpack’s vue-loader to compile the render function from the template. Regardless of which version you use, there is always a step to compiling the template into the Render function.
Let’s analyze the compilation process of the VUE template, which is also a very important module in the vUE source code implementation. The compilation process of vUE templates is divided into three stages
Step 1 Parse
const ast = parse(template.trim(), options)
Copy the codeThe template string is parsed into AST. The parser is implemented by VUE itself. During the parsing process, regular expressions are used to parse the template sequentially.
The generated AST element nodes are of three types: 1 is a normal element, 2 is an expression, and 3 is plain text. Let’s look at an example
<ul :class="bindCls" class="list" v-if="isShow">
<li v-for="(item,index) in data" @click="clickItem(index)">{{item}}:{{index}}</li>
</ul>
Copy the codeThe AST tree generated from template parsing is as follows:
ast = {
'type': 1,
'tag': 'ul'.'attrsList': [].'attrsMap': {
':class': 'bindCls'.'class': 'list'.'v-if': 'isShow'
},
'if': 'isShow'.'ifConditions': [{
'exp': 'isShow'.'block': // ul ast element
}],
'parent': undefined,
'plain': false.'staticClass': 'list'.'classBinding': 'bindCls'.'children': [{
'type': 1,
'tag': 'li'.'attrsList': [{
'name': '@click'.'value': 'clickItem(index)'}].'attrsMap': {
'@click': 'clickItem(index)'.'v-for': '(item,index) in data'
},
'parent': // ul ast element
'plain': false.'events': {
'click': {
'value': 'clickItem(index)'}},'hasBindings': true.'for': 'data'.'alias': 'item'.'iterator1': 'index'.'children': [
'type': 2.'expression': '_s(item)+":"+_s(index)'
'text': '{{item}}:{{index}}'.'tokens': [{'@binding':'item'},
':',
{'@binding':'index'}]]}Copy the codeStep 2 Optimize the syntax tree
optimize(ast, options)
Copy the codeNot all data in the Vue template is responsive, and many data will never change after the first rendering, so the DOM generated by this part of data will not change, so we can skip the comparison of them in the process of patch. In this stage, the generated AST tree will be deeply traversed to detect whether each of its sub-trees is a static node. If it is a static node, they will never need to change the DOM generation, which will greatly optimize the update of the template at runtime.
During traversal, static and staticRoot are marked for each AST element node in the entire AST tree (recurse all children of that node, false if they are not static, true otherwise).
After this stage, the AST in the above example becomes
ast = {
'type': 1,
'tag': 'ul'.'attrsList': [].'attrsMap': {
':class': 'bindCls'.'class': 'list'.'v-if': 'isShow'
},
'if': 'isShow'.'ifConditions': [{
'exp': 'isShow'.'block': // ul ast element
}],
'parent': undefined,
'plain': false.'staticClass': 'list'.'classBinding': 'bindCls'.'static': false.'staticRoot': false.'children': [{
'type': 1,
'tag': 'li'.'attrsList': [{
'name': '@click'.'value': 'clickItem(index)'}].'attrsMap': {
'@click': 'clickItem(index)'.'v-for': '(item,index) in data'
},
'parent': // ul ast element
'plain': false.'events': {
'click': {
'value': 'clickItem(index)'}},'hasBindings': true.'for': 'data'.'alias': 'item'.'iterator1': 'index'.'static': false.'staticRoot': false.'children': [
'type': 2.'expression': '_s(item)+":"+_s(index)'
'text': '{{item}}:{{index}}'.'tokens': [{'@binding':'item'},
':',
{'@binding':'index'}].'static': false]]}}Copy the codeStep 3 generates the code
const code = generate(ast, options)
Copy the codeThe AST generates the render function using the generate method
with(this){
return (isShow) ?
_c('ul', {
staticClass: "list",
class: bindCls
},
_l((data), function(item, index) {
return _c('li', {
on: {
"click": function($event) {
clickItem(index)
}
}
},
[_v(_s(item) + ":" + _s(index))])
})
) : _e()
}
Copy the codeThe implementation principle of Prettier
As you can see from the Babel implementation principle and vue template compilation principle above, their implementation has a lot in common, is to first parse the source code into an AST tree, then the AST tree on the line processing, and finally generate what you want.
Prettier also does the same thing when parsing code into an AST tree, iterating through the AST, adjusting long sentences, whitespace, parentheses, and so on, and printing code.
summary
We have analyzed the Babel principle, vue template compilation process, Prettier principle, here we briefly summarize. If the source code is compared to a machine, then the word segmentation process is to split the machine into parts, semantic analysis process is to analyze the position and role of each part, and then according to the need for parts processing, and finally assembled into a new machine.
What else can AST do
So what can we do with AST at work? !
It’s time to imagine what needs we have in our daily work that can be addressed by developing a tool for AST. For example, an AST can be used to automatically turn code into a flowchart; Or according to the custom annotation specification, through the tool automatically generated documents; Or through the tool automatically generate skeleton screen file.
Do you have any other good ideas?
Welcome to my public account “front-end xiaoyuan”, I will update the original articles on it regularly. You can also add our wechat yu_shihu_ common communication technology.
