It’s time to meet again, doupi fans. In today’s issue, Taro Sauce from Bytedance data platform brings you into the world of AST.

Author: Taro sauce

What is the AST

Abstract Syntax Tree (AST) is a Tree representation of the Abstract Syntax structure of source code, corresponding to which is concrete Syntax Tree. Abstract because the abstract syntax tree does not represent every detail that occurs in real grammar and is grammar-independent and language-independent. Can imagine the AST as a standardized programming interface definition language, only a set of specifications, is for the programming language itself, small to variable declarations, big to the complex modules, can be described by a set of specifications, interested students can understand the concept and principle of AST, the focus of this article focused on the application of JavaScript AST.

Why AST

For front-end students, AST related scenarios are everywhere in daily development; Examples: Webpack, Babel, Lint, prettier, Codemod, etc., are all AST based; Mastering AST is the same as mastering the ability to control code, which helps us to broaden our thinking and vision. Whether it is writing framework, tools and logic, AST can be a great help to you.

AST Parsing Process

First of all, I recommend an AST conversion site: AstExplorer.net. It is very important to bookmark it. Besides JS, there are many AST libraries for other languages; It can be used as a playground without doing any configuration.

Before explaining the case, let’s first understand the parsing process, which is divided into three steps:

  1. Source code –> ast

  2. Traverse Ast (traverse ast, visit nodes in the tree, do various operations on nodes)

  3. Ast –> code

There are many AST engines for source code parsing, and the AST transformed is much the same;

Use Cases

Start with a variable declaration like this:


const dpf = 'DouPiFan';

Copy the code

Copy the code into AstExplorer and you get the following (simplified) result, which illustrates the process from source to AST;

Select different third-party libraries to generate AST, the results will be different. Here uses Babel/Parse as an example. Babel is a javascript Compiler. It supports ES2015+ code in the browser. This is just one of the application scenarios of Babel.

Going back to Babel-Parser, it uses Babylon as the parsing engine. It is an AST to AST operation. Babel encapsulates the parsing and generating steps from Babylon. Because every operation does these two steps; For applications, the focus is on traversing and updating AST nodes;

The first Babel plug-in

Let’s take the simplest Babel plug-in as an example to see how it works.

When we were developing the Babel-Plugin, we just needed a visitor to describe how to convert the AST. Add it to your list of Babel plugins and you’re ready to go. Our first Babel plugin is complete.

How is babel-plugin-import implemented?

If you have used ANTD before, you know that the babel-plugin-import plug-in is used to load ANTD components on demand. After configuration, the effect is as follows:

Import {Button} from 'antd' ↓ ↓ ↓ ↓ ↓ import Button from 'antd/lib/ Button 'Copy the code

This article aims to introduce the introduction of the plug-in for the implementation details and various boundary conditions, refer to the plug-in source code;

To think in the way of AST, the steps are as follows:

  1. Import {XXX} from ‘antd’

  2. Import Button from ‘antd/lib/ Button ‘import Button from ‘antd/lib/ Button’

Implementation steps

  1. Open artifact: AST Explorer, copy the first line of code into artifact

  2. Click the import keyword in the code, and the corresponding node will be automatically located as follows:

ImportDeclaration {type: "ImportDeclaration", speciFIERS: [{// "ImportSpecifier", imported: { type: "Identifier", name: "Button" } } }] source: { type: "StringLiteral", value: "antd" }, ... }Copy the code

Source code is converted into objects with types and attributes, whether keywords, variable declarations, or literal values, that have corresponding types;

  1. The import statement corresponds to the type ImportDeclaration

  2. {Button} corresponds to the specifiers array. The example only introduced “Button”, so there is only one element in the specifiers array

  3. The element in specifiers, known as buttons, is of type ImportSpecifier;

  4. ‘antd’ in the source node, type: StringLiteral, value antd

Again: the example is not a complete logical implementation, details and boundary conditions, can refer to the source code or improve their own;

The operation for the AST is similar to the DOM API of the browser. First determine the type of node to search, and then according to specific conditions, reduce the search scope, and finally to find the node, add, delete, change, check;

// Export default function({types: t}) {return {// Visitor receives two parameters for each function: path and state Visitor: { ImportDeclaration(path, state) { const { node } = path; // Source = antd if(node.source.value === 'antd'){const speciFIERS = node.speciFIERS Local // construct source const source = T.literal (' ${node.source.value}/lib/${local.name} ') // Construct import statement return t.importDeclaration([t.importDefaultSpecifier(local)], source) }) console.log(result) path.replaceWithMultiple(result) } } } } }Copy the code

Validation is also simple: copy this code into the AST Explorer and view the output. At this point, the “easy” plug-in implementation is complete;

Let’s review the implementation idea again:

  1. Compare the source code for differences in the syntax tree to determine which conversions and changes to make

  2. Analysis of the type, you can find the type description in the Babel official

  3. In the plug-in template, the corresponding type node is accessed by visitor for adding, deleting, modifying and checking

Codemod

Now that you’ve covered the basics of ast in Babel, take a look at Codemod.

If you use ANTD3, you have come into contact with ANTD3 to ANTD4 Codemod, which is a tool library to help us automate the conversion of ANTD3 code to ANTD4; Because its nature is transcoding, implementing Codemod based on Babel is perfectly ok. But in addition to code conversion, also need command line operation, source code reading, batch execution conversion, log output and other functions, he is a function set, code conversion is a very important part of it; So, another tool, jscodeshift, is recommended. He is a Transform runner, so our core job is to define a set of transform rules, the rest of the command line, source reading, batch execution, log output can be handed to jscodeshift.

The preparatory work

Start by defining a transform, much like the Babel plug-in


import { Transform } from "jscodeshift";

const transform: Transform = (file, api, options) => {

return null;

};

export default transform;

Copy the code
Hands-on practice

We tried replacing the Button component’s “type” property with “status” and adding the width property to the style:

// Input const Component = () => {return (<Button type="dange" width="20" />)} // Output const Component = () => {return ()  <Button statue="dange" style={{ width: 20 }} /> ) }Copy the code
Differences in contrast

  1. React component property type: JSXIdentifier; property “type” changed to “status”

  2. If the component has a “width” property, move that property to the “style” property

The code for finding a Button component is as follows:

import { Transform } from "jscodeshift"; const transform = (file, api, options) => { const j = api.jscodeshift; Return j(file.source).find(j.sxopeningElement, {name: {type: 'JSXIdentifier', name: 'Button' } }) }; export default transform;Copy the code
Attribute to replace

Next, add attribute substitution logic to replace type with status

export default function transformer(file, api) { const j = api.jscodeshift; return j(file.source) .find(j.JSXOpeningElement, { name: { type: 'JSXIdentifier', name: 'Button' } }).forEach(function(path){ var attributes = path.value.attributes; attributes.forEach(function(node, index){ const attr = node.name.name; If (attr === 'type'){// if attr is type, replace the attribute name with status node.name.name = 'status'}})}).tosource (); }Copy the code

Jscodeshift can get: j(file.source).findjsxElements () instead of find.

JSXNamespaceName = “status”; JSXNamespaceName = “status”; JSXNamespaceName = “status”;

Processing width

If width is present, get the value of width and then delete the node;

Next, create a style node of type jsxAttribute and set width back to style

. attributes.forEach(function(node, index){ const attr = node.name.name; If (attr === 'width'){// Get the value of width; width = node.value.value; Attributes. Splice (index, 1)} let width; If (width){// construct the style node var node = j.jsxattribute (// set the name of attr to: style j.jsxIdentifier('style'), // construct jsxExpressionContainer {} // construct objectExpression j.sxexpressionContainer (j.objectProperty([j.objectProperty( J.dentifier ('width'), j.singliteral (width),),])),) // Insert style node attributes. Splice (index, 0, node)}}...Copy the code

conclusion

Jscodeshift implementation is simpler than Babel implementation, but it takes extra time and effort to achieve a more perfect state, especially in the face of large-scale code processing, there are many boundary conditions, need to consider very comprehensive; But it’s worth the investment to automate most of the work;

Jscodeshift is more like a fully functional set of tools, you can focus on the implementation of the converter, please choose the right tool according to the actual scenario.

Front-end team of Bytedance data platform, responsible for r&d of big data-related products within the company. We have a strong passion in front-end technology. In addition to research and development related to data products, we have a lot of exploration and accumulation in data visualization, mass data processing optimization, Web Excel, WebIDE, private deployment and engineering tools. If you are interested, please feel free to contact us.