AST: Abstract Syntax Tree, which is an Abstract representation of the Syntax structure of the source code.

AST is a very basic but very important knowledge. We know TypeScript, Babel, Webpack, and Vue-CLI all rely on AST for development. This article will show the strength and importance of AST through AST and front-end engineering.

Live sharing video address: AST and front-end engineering actual combat

First, get to know AST

1, the demo – 1

The first time I saw the concept of an AST was in the book JavaScript You Don’t Know. Let’s start with an example

const a = 1
Copy the code

In traditional compiled languages, source code execution goes through three phases

  • Lexical analysis: The string of characters is broken up into code blocks (lexical units). In this example, the code is parsed into const, a, =, and 1.

  • Syntax analysis phase: transform the lexical unit flow into a syntax structure tree composed of hierarchical nested elements, which is called abstract syntax tree. The lexical unit stream of const, a, =, and 1 that is parsed in the example is converted to the following tree

  • Code generation phase: convert the AST into a series of executable machine instruction codes. For example, the machine creates a variable a in memory by executing the instruction and assigns the value 1 to it.

2, demo – 2

Let’s break down an official Recast example, which is also a little more complicated

function add (a, b) {
  return a + b
}
Copy the code
  • First of all, we go to the lexical analysis phase, and we getFunction, add, (, a,, b,), {, return, a, +, b,}13 code blocks
  • Then enter the syntax analysis phase, as shown in the figure

For details on the types of code blocks such as FunctionDeclaration, Identifier, and BlockStatement, click the link: AST Object Document

Second, recast

Since the AST dependency used in this article is Recast, and since recast itself is not documented except for a very short readme.md file, here is a separate introduction to some of its common apis. Before we begin, I recommend a platform for viewing the AST structure online, which is very useful

  • AST Explorer

If you know anything about Babel, you’ll know that Babel has a series of packages that encapsulate the AST to handle compilation. Recast is also developed based on packages such as @babel/core, @babel/ Parser, and @babel/types.

The introduction of

There are two ways to introduce Recast, import and CommonJs, as follows

  • importIn the form of
import { parse, print } from 'recast'
console.log(print(parse(source)).code)

import * as recast from 'recast'
console.log(recast.print(recast.parse(source)).code)
Copy the code
  • CommonJsIn the form of
const { parse, print } = require('recast')
console.log(print(parse(source)).code)

const recast = require('recast')
console.log(recast.print(recast.parse(source)).code)
Copy the code

Now that we’ve introduced Recast, let’s see what we can do with recast

1, recast. Parse

Let’s go back to our example and parse it directly to see what the AST structure looks like after parse

// parse.js
const recast = require('recast')

const code = `function add (a, b) { return a + b }`

const ast = recast.parse(code)
// Get the first body of the code block ast, our add function
const add = ast.program.body[0]
console.log(add)
Copy the code

Execute node parse.js to see the structure of the add function on our terminal

FunctionDeclaration {
  type: 'FunctionDeclaration'.id: Identifier... .params: [Identifier...] .body: BlockStatement...
}
Copy the code

Of course, if you want to see more content, you can go to the AST Explorer platform and set the mode to Recast mode to see the whole AST view, which is basically the same as what we analyzed above.

2, recast. Print

So far, we’ve just taken it apart, but what if we put the AST together into code that we can execute? OK, this is where recast.print comes in, and we assemble the code as it is

// print.js
const recast = require('recast')

const code = `function add (a, b) { return a + b }`

const ast = recast.parse(code)

console.log(recast.print(ast).code)
Copy the code

Then run node print.js and you can see that we print it out

function add (a, b) {
  return a + b
}
Copy the code

The official explanation is that this is just a reverse process, i.e

recast.print(recast.parse(source)).code === source
Copy the code

3, recast. PrettyPrint

In addition to the recast.print we mentioned above, Recast also provides a code beautification API called Recast.prettyPrint

// prettyPrint.js
const recast = require('recast')

const code = `function add (a, b) { return a + b }`

const ast = recast.parse(code)

console.log(recast.prettyPrint(ast, { tabWidth: 2 }).code)
Copy the code

If you run node prettyprint.js, you will find that N Spaces can be formatted out of the code, and the output is as follows

function add(a, b) {
  return a + b;
}
Copy the code

For details, see prettyPrint

4, recast. Types. Builders

i. API

As for the Builder API, don’t worry, I’m definitely not going to talk about it because it’s too much.

To see exactly what each API does, look directly at Parser APi-Builders, or recast Builders definition

Ii. Actual combat stage

OK, finally into the core of recast operations. If we want to change our code, recast.types.builders is our most important tool. Here we continue to learn about the Recast.types. Builders build tool by modifying the recast official case.

Function add (a, b) {… function add (a, b) {… } const add = function (a, b) {… }.

We know from Section 1 that if we want to make it const, we need a VariableDeclaration and a VariableDeclarator, Then we declare a function and we need to create a FunctionDeclaration. The rest is to fill in the parameters and body of the expression. The specific operations are as follows

// builder1.js
const recast = require('recast')
const {
  variableDeclaration,
  variableDeclarator,
  functionExpression
} = recast.types.builders

const code = `function add (a, b) { return a + b }`

const ast = recast.parse(code)
const add = ast.program.body[0]

ast.program.body[0] = variableDeclaration('const', [
  variableDeclarator(add.id, functionExpression(
    null.// Make the function anonymous
    add.params,
    add.body
  ))
])

const output = recast.print(ast).code

console.log(output)
Copy the code

Run node Builder1.js. The following output is displayed

const add = function(a, b) {
  return a + b
};
Copy the code

Look at this, don’t you think it’s funny. This is where the fun begins. Now, based on this example, let’s make a small extension. Const add = (a, b) => {… }.

A new concept is the arrow function, and of course, the recast.type.builders provide arrowFunctionExpression to allow us to create an arrow function. So our first step is to create an arrow function

const arrow = arrowFunctionExpression([], blockStatement([])
Copy the code

Print console.log(recast.print(arrow)). The following output is displayed

() = > {}Copy the code

OK, we’ve got an empty arrow function. Next, we need to make further modifications based on the above modifications. In fact, we just need to replace the functionExpression with arrowFunctionExpression.

ast.program.body[0] = variableDeclaration('const', [
  variableDeclarator(add.id, b.arrowFunctionExpression(
    add.params,
    add.body
  ))
])
Copy the code

The print result is as follows

const add = (a, b) = > {
  return a + b
};
Copy the code

OK, so far we’ve seen that recast.types.builders can provide us with a set of apis that let us output like crazy.

5, recast. Run

Read file command line. First, I create a new read.js with the following content

// read.js
recast.run((ast, printSource) = > {
  printSource(ast)
})
Copy the code

Then I’ll create a new demo.js with the following content

// demo.js
function add (a, b) {
  return a + b
}
Copy the code

Then run node read demo.js and the output is as follows

function add (a, b) {
  return a + b
}
Copy the code

As you can see, we are directly reading the contents of the demo.js code in read.js. So how does that work?

Read the file directly from fs.readFile and parse the code. The printSource we saw provides a default printSource function process.stdout.write(output)

import fs from "fs";

export function run(transformer: Transformer, options? : RunOptions) {
  return runFile(process.argv[2], transformer, options);
}

function runFile(path: any, transformer: Transformer, options? : RunOptions) {
  fs.readFile(path, "utf-8".function(err, code) {
    if (err) {
      console.error(err);
      return;
    }

    runString(code, transformer, options);
  });
}

function defaultWriteback(output: string) {
  process.stdout.write(output);
}

function runString(code: string, transformer: Transformer, options? : RunOptions) {
  const writeback = options && options.writeback || defaultWriteback;
  transformer(parse(code, options), function(node: any) {
    writeback(print(node, options).code);
  });
}
Copy the code

6, recast. Visit

This is an API for traversing AST nodes. If you want to traverse some types in the AST, you need to use recast.visit. The types that can be traversed are the same as those that can be constructed in recast.types. Builders. What builders do is build types, and what Recast.visit does is traverse the types in the AST. However, you need to pay attention to the following points when using

  • Each visit must be addedreturn falseOr,this.traverse(path)Otherwise, an error is reported.
if (this.needToCallTraverse ! = =false) {
  throw new Error(
    "Must either call this.traverse or return false in " + methodName
  );
}
Copy the code
  • You can iterate by adding visit to the type you want to iterate over. If you want to iterate over arrow functions in the AST, you can do this
recast.run((ast, printSource) = > {
  recast.visit(ast, {
    visitArrowFunctionExpression (path) {
      printSource(path.node)
      return false}})})Copy the code

7, recast. Types. NamedTypes

An API used to determine whether an AST object is of the specified type.

NamedTypes has two apis. One is namedtypes.node. assert: When a type is not configured, an error exits. The other is namedtypes.node. check: Checks whether the types are consistent and prints true or false.

Where Node is any AST object, for example, I make a function type determination against the arrow function, the code is as follows

// namedTypes1.js
const recast = require('recast')
const t = recast.types.namedTypes

const arrowNoop = () = > {}

const ast = recast.parse(arrowNoop)

recast.visit(ast, {
  visitArrowFunctionExpression ({ node }) {
    console.log(t.ArrowFunctionExpression.check(node))
    return false}})Copy the code

Execute node namedtypes1.js, and you can see that the printer outputs true.

The same applies to assert.

const recast = require('recast')
const t = recast.types.namedTypes

const arrowNoop = () = > {}

const ast = recast.parse(arrowNoop)

recast.visit(ast, {
  visitArrowFunctionExpression ({ node }) {
    t.ArrowFunctionExpression.assert(node)
    return false}})Copy the code

If you want to determine more AST types, simply replace Node with another AST type.

Third, front-end engineering

Now, let’s talk about front-end engineering.

The front section of engineering can be divided into four blocks, respectively

  • Modularity: Splitting a file into multiple interdependent files that are packaged and loaded uniformly to ensure efficient collaboration between multiple people. contains

    1. JS modularity: CommonJS, AMD, CMD, and ES6 Modules.
    2. CSS Modules: Sass, Less, Stylus, BEM, AND CSS Modules. One problem that both the preprocessor and the BEM have is style override. CSS Modules manage dependencies through JS, maximizing the combination of JS modularity and CSS ecology, such as style scoped in Vue.
    3. Modular resources: any resources can be loaded in the form of modules, most of the current project files, CSS, pictures and so on can be directly done through JS unified dependency processing.

  • Componentization: Unlike modularity, which is splitting files, code, and resources, componentization is splitting the UI level.

    1. Usually, we need to split the page, break it down into parts, and then implement the parts separately, and then assemble them.
    2. In our actual business development, we need to do different degrees of consideration for the component separation, which mainly includes the consideration of fineness and generality.
    3. For business components, you need to think more about an applicability to your line of business, that is, whether the business component you are designing is a “common” component to your current business, such as the permission validation component I analyzed earlier, which is a typical business component. Those who are interested can click on the portal to read for themselves.

  • Standardization: as the so-called no rules no circumference, some good norms can be very good to help us to carry out good development and management of the project. Standardization refers to the series of specifications that we developed at the beginning and during the development of the project

    1. Project directory structure
    2. Code specification: For the code section of the constraint, we generally use some enforcement measures, such as ESLint, StyleLint, etc.
    3. Connection specification: this can refer to my previous zhihu answer, the front and back end separation, the data returned by the background front end can not be written, how to do?
    4. File Naming convention
    5. Style management specifications: Popular style management methods include BEM, Sass, Less, Stylus, CSS Modules and so on.
    6. Git Flow workflow: this includes branch naming conventions, code merging conventions, and so on.
    7. Code review on a regular basis
    8. … , etc.

    All of this, and I’ve written a few more details about it in a previous post, TypeScript + large-scale projects in action

  • Automation: from the earliest grunt, gulp, and then to the current Webpack, parcel. These automated tools can save us a lot of work in automating merge, build, and package. As a part of this front-end automation, front-end automation also includes continuous integration, automated testing, and so on.

Being in any of these blocks is front-end engineering.

Four, practical: AST & Webpack Loader

The actual practice mentioned in this article is to write our own WebPack Loader through AST modification to automatically inject catch operations into the promise in our project, avoiding the need for us to manually write those common catch operations.

1. AST modification

Having talked so much, we finally entered our actual combat link. So what are we going to build in real time?

Scenario: In daily middle platform projects, there are often some requirements for form submission, so the submission needs to be limited to prevent the request from being repeated because of the shaking of the hands. There are many solutions to this scenario, but in my opinion, the best interaction is to add loading state to the submit button after clicking, then disable it, and then remove loading and disabled after the request succeeds. The detailed operations are as follows

this.axiosFetch(this.formData).then(res= > {
  this.loading = false
  this.handleClose()
}).catch(() = > {
  this.loading = false
})
Copy the code

This may seem OK, but if doing something like this makes your overall project code look more or less repetitive, what can you do about it?

Let’s write a WebPack Loader using the AST to automatically inject some code. If we have the following code in our project, we will automatically add the catch part of the processing, and actively use the first part of the THEN statement as the processing logic of catch

this.axiosFetch(this.formData).then(res= > {
  this.loading = false
  this.handleClose()
})
Copy the code

So let’s see what the AST structure looks like in this code without the catch, right

Its MemberExpression for

this.axiosFetch(this.formData).then
Copy the code

The arguments for

res => {
  this.loading = false
  this.handleClose()
}
Copy the code

OK, let’s take a look at the AST structure of code that has catch handling, as shown here

Its MemberExpression for

this.axiosFetch(this.formData).then(res= > {
  this.loading = false
  this.handleClose()
}).catch
Copy the code

There are two ArrowFunctionExpressions, which are

// ArrowFunctionExpression 1
res => {
  this.loading = false
  this.handleClose()
}
// ArrowFunctionExpression 2() = > {this.loading = false
}
Copy the code

So, what we need to do is roughly divided into the following steps

  1. The ArrowFunctionExpression type is iterated to obtain the first ExpressionStatement in its BlockStatement and saved as firstExp
  2. Use builders to create an empty arrow function and assign the saved firstExp to the BlockStatement of the empty arrow function
  3. Traverse the CallExpression type and change the AST’s MemberExpression to a format with catch fragments
  4. Returns the modified AST

Now, according to our thinking, we will do the AST modification step by step

First of all, we need to obtain the first ExpressionStatement in the existing arrow function. When obtaining, we need to ensure that the parent node of the current ArrowFunctionExpression type is a CallExpression type. And make sure its property is a PROMISE then function

// promise-catch.js
const recast = require('recast')
const {
  identifier: id,
  memberExpression,
  callExpression,
  blockStatement,
  arrowFunctionExpression
} = recast.types.builders
const t = recast.types.namedTypes

const code = `this.axiosFetch(this.formData).then(res => { this.loading = false this.handleClose() })`
const ast = recast.parse(code)
let firstExp

recast.visit(ast, {
  visitArrowFunctionExpression ({ node, parentPath }) {
    const parentNode = parentPath.node
    if (
      t.CallExpression.check(parentNode) &&
      t.Identifier.check(parentNode.callee.property) &&
      parentNode.callee.property.name === 'then'
    ) {
      firstExp = node.body.body[0]}return false}})Copy the code

Next, we need to create an empty arrow function and assign firstExp to it

const arrowFunc = arrowFunctionExpression([], blockStatement([firstExp]))
Copy the code

Then we need to iterate through the AST objects of CallExpression type and do the final MemberExpression transformation

recast.visit(ast, {
  visitCallExpression (path) {
    const { node } = path

    const arrowFunc = arrowFunctionExpression([], blockStatement([firstExp]))
    const originFunc = callExpression(node.callee, node.arguments)
    const catchFunc = callExpression(id('catch'), [arrowFunc])
    const newFunc = memberExpression(originFunc, catchFunc)

    return false}})Copy the code

Finally we replace it with CallExpression traversal

path.replace(newFunc)
Copy the code

The full code for the first version is as follows

// promise-catch.js
const recast = require('recast')
const {
  identifier: id,
  memberExpression,
  callExpression,
  blockStatement,
  arrowFunctionExpression
} = recast.types.builders
const t = recast.types.namedTypes

const code = `this.axiosFetch(this.formData).then(res => { this.loading = false this.handleClose() })`
const ast = recast.parse(code)
let firstExp

recast.visit(ast, {
  visitArrowFunctionExpression ({ node, parentPath }) {
    const parentNode = parentPath.node
    if (
      t.CallExpression.check(parentNode) &&
      t.Identifier.check(parentNode.callee.property) &&
      parentNode.callee.property.name === 'then'
    ) {
      firstExp = node.body.body[0]}return false
  }
})

recast.visit(ast, {
  visitCallExpression (path) {
    const { node } = path

    const arrowFunc = arrowFunctionExpression([], blockStatement([firstExp]))
    const originFunc = callExpression(node.callee, node.arguments)
    const catchFunc = callExpression(id('catch'), [arrowFunc])
    const newFunc = memberExpression(originFunc, catchFunc)

    path.replace(newFunc)

    return false}})const output = recast.print(ast).code
console.log(output)
Copy the code

Execute node promise-cat.js and the printer outputs the result

this.axiosFetch(this.formData).then(res= > {
  this.loading = false
  this.handleClose()
}).catch(() = > {
  this.loading = false
})
Copy the code

So you can see that we’ve done what we wanted to do

  1. But there are a few things we need to do. The first thing we need to do is make sure that arguments are arrow functions when CallExpression traversal.

  2. Next, we need to determine if the firstExp we obtained exists, because our THEN processing can be an empty arrow function.

  3. Then to prevent a promise from having any custom catch operations, you need to ensure that its property is then.

  4. Finally, in order to prevent a situation where multiple Callexpressions need to be injected automatically and then operate differently, an ArrowFunctionExpression traversal needs to be performed within them

After the compatibility of these common cases, the specific code is as follows

recast.visit(ast, {
  visitCallExpression (path) {
    const { node } = path
    const arguments = node.arguments

    let firstExp

    arguments.forEach(item= > {
      if (t.ArrowFunctionExpression.check(item)) {
        firstExp = item.body.body[0]

        if (
          t.ExpressionStatement.check(firstExp) &&
          t.Identifier.check(node.callee.property) &&
          node.callee.property.name === 'then'
        ) {
          const arrowFunc = arrowFunctionExpression([], blockStatement([firstExp]))
          const originFunc = callExpression(node.callee, node.arguments)
          const catchFunc = callExpression(id('catch'), [arrowFunc])
          const newFunc = memberExpression(originFunc, catchFunc)
  
          path.replace(newFunc)
        }
      }
    })

    return false}})Copy the code

Then we need to make a Webpack-loader to use in our actual project. The default parser for Parse is Recast /parsers/esprima. Babel-loader is used in most projects, so we need to change the parser to recast/parsers/ Babel

const ast = recast.parse(code, {
  parser: require('recast/parsers/babel')})Copy the code

2, webpack loader

At this point, our AST modification of the code is complete, but how do we apply it to our actual project?

OK, at this point we need to write our own WebPack Loader.

In fact, how to develop a WebPack Loader, webPack official documentation has been very clear, I would like to make a small summary for friends.

I. Develop the Loader locally

First, you need to create a new file locally for your development loader. For example, we will throw it under SRC /index.js, as shown in the webpack.config.js configuration

const path = require('path')

module.exports = {
  // ...
  module: {
    rules: [{test: /\.js$/,
        use: [
          / /... Other loaders you need
          { loader: path.resolve(__dirname, 'src/index.js')}]}]}Copy the code

The SRC /index.js content is as follows

const recast = require('recast')
const {
  identifier: id,
  memberExpression,
  callExpression,
  blockStatement,
  arrowFunctionExpression
} = recast.types.builders
const t = recast.types.namedTypes

module.exports = function (source) {
  const ast = recast.parse(source, {
    parser: require('recast/parsers/babel')
  })

  recast.visit(ast, {
    visitCallExpression (path) {
      const { node } = path
      const arguments = node.arguments

      let firstExp

      arguments.forEach(item= > {
        if (t.ArrowFunctionExpression.check(item)) {
          firstExp = item.body.body[0]

          if (
            t.ExpressionStatement.check(firstExp) &&
            t.Identifier.check(node.callee.property) &&
            node.callee.property.name === 'then'
          ) {
            const arrowFunc = arrowFunctionExpression([], blockStatement([firstExp]))
            const originFunc = callExpression(node.callee, node.arguments)
            const catchFunc = callExpression(id('catch'), [arrowFunc])
            const newFunc = memberExpression(originFunc, catchFunc)
    
            path.replace(newFunc)
          }
        }
      })

      return false}})return recast.print(ast).code
}
Copy the code

Then, it’s done.

Ii. The NPM contract

I mentioned this in a previous post, but I won’t talk about it here. If you haven’t done NPM, you can click the link below to check it out

Unraveling component Libraries (releasing NPM package snippets)

OK, at this point, my Promise-catch-loader has been developed. Next, just use it in your project

npm i promise-catch-loader -D
Copy the code

Since my project is built on vue-cli3.x, I need to configure this in my vue.config.js

/ / js version
module.exports = {
  // ...
  chainWebpack: config= > {
    config.module
      .rule('js')
      .test(/\.js$/)
      .use('babel-loader').loader('babel-loader').end()
      .use('promise-catch-loader').loader('promise-catch-loader').end()
  }
}
/ / ts version
module.exports = {
  // ...
  chainWebpack: config= > {
    config.module
      .rule('ts')
      .test(/\.ts$/)
      .use('cache-loader').loader('cache-loader').end()
      .use('babel-loader').loader('babel-loader').end()
      .use('ts-loader').loader('ts-loader').end()
      .use('promise-catch-loader').loader('promise-catch-loader').end()
  }
}
Copy the code

Then I have the following promise operations in my project

<script lang="ts">
import { Component, Vue } from 'vue-property-decorator'
import { Action } from 'vuex-class'

@Component
export default class HelloWorld extends Vue {
  loading: boolean = false
  city: string = 'Shanghai'

  @Action('getTodayWeather') getTodayWeather: Function

  getCityWeather (city: string) {
    this.loading = true
    this.getTodayWeather({ city: city }).then((res: Ajax.AjaxResponse) = > {
      this.loading = false
      const { low, high, type } = res.data.forecast[0]
      this.$message.success(`${city}Today:${type} ${low} - ${high}`)}}}</script>
Copy the code

Then look at source in your browser and see the following result

As for the code, I’ve hosted it on GitHub, promise-catch-loader

conclusion

At this point, we’re done in the field. Of course, the article is only a first guide, more types also have to explore their own partners.

AST is also very useful. For example, we are familiar with Vue. Its SFC(.vue) file is also parsed automatically based on AST, namely VUe-Loader, which ensures that we can normally use Vue for business development. The WebPack build tool, for example, is also based on AST to provide us with very useful capabilities for merging, packaging, building optimization, and so on.

In short, get the AST right and you really can do a lot.

Finally, I hope the content of this article can help partners understand: What is AST? How to use AST to make our work more efficient? What can an AST do for front-end engineering?

If you think the article is good, then hope you can move your little hand, help point a like, thank you ~

Front-end AC group: 731175396