In addition to realizing business functions in daily work, there are often some scenarios that need to be automated. For the most part, we’ll do it in a scripted way. As a front-end engineer, if you’re going to use Node.js to do any automated work, you’re going to need some text processing skills. The following article describes some of the techniques used in developing an automation tool.

Processing user input

There are usually two ways to handle the differences in the types of command-line tools we develop:

Pure command-line tools

First complete a welcome screen:

const chalk = require('chalk');
const boxen = require('boxen');
const yargs = require('yargs');

const greeting = chalk.white.bold('Welcome to the XXX tool');
const boxenOptions = {
  padding: 1.margin: 1.borderStyle: 'round'.borderColor: 'green'.backgroundColor: '# 555555'};const msgBox = boxen(greeting, boxenOptions);
console.log(msgBox);
Copy the code

Use yargs, a tool that automatically parses user input from the command line:

const options = yargs
  .usage('Usage: --inject-janus|--inject-kani')
  .option('inject-janus', {
    describe: 'injection janus'.type: 'boolean',
    demandOption: false,
  })
  .option('inject-kani', {
    describe: 'injection kani'.type: 'boolean',
  }).argv;
Copy the code

To parse a menu of commands such as the following

./cli --inject-janus
Copy the code

Interactive command line tool

The Nodejs command-line tool handles user input using the Inquirer library:

import inquirer from 'inquirer';

await inquirer.prompt([
  {
    name: 'repoName'.type: 'input',
    message:
      'Please enter a project name :',
  },
  {
    name: 'repoNamespace'.type: 'input',
    message: 'Please enter the gitlab namespace, such as gFE',
    default: 'gfe',}]);Copy the code

Inline script

The Node command line often needs to use some system native tools, for Linux, OSX, etc., can use shellJS this package to call shell scripts, so as to extend the capabilities of our automation tools.

const shell = require('shelljs');
 
shell.exec('git commit -am "Auto-commit"'
Copy the code

File to read and write

The configuration information of the project is basically in a separate file, so we need to use the interface related to processing files to process. Common file processing interfaces are:

Fs. access: file access
Fs. chmod: changes the read and write permissions of files
Fs. copyFile: copies files
Fs. link: link file
Fs. watch: listens for file changes
Fs. readFile: Read files (high frequency)
Fs. mkdir: creates a folder
Fs. writeFile: Write files (high frequency)

import {promises as fs} from 'fs';

async function readJson() {
    return fs.readFile('./snowflake.txt'.'utf8');
}

async function saveFile() {
    await fs.mkdir('./saved/snowfalkes', {recursive: true});
    await fs.writeFile('./saved/snowflakes/xx.txt', data);
}

console.log(await readSnowflake());
Copy the code

JSON

JSON files often exist as configuration files in front-end projects, and the most common way to manipulate configuration files is to work with JSON text. The code looks like this:

const data = require('.. /test.json');

data['xxx'] = 'a';
Copy the code

The second argument of the json. stringify interface (used for formatting) is also commonly used when serializing:

// Keep two Spaces indented
JSON.stringify(a, null.2)
Copy the code

The path

Path parsing is also often required during file reading and writing. Resolve and path.parse are used to handle relative and absolute paths. The former is used to do the transformation of the path, the latter is mainly used to get more detailed information on the path.

import * as path from 'path';

const relativeToThisFile = path.resolve(__dirname, './test.txt');
const parsed = path.parse(relativeToThisFile);

// interface ParsedPath {
// root: string;
// dir: string;
// base: string;
// ext: string;
// name: string;
// }
Copy the code

__dirname: path of the current file
Process. CWD: indicates the path where the command is executed.

It is important to distinguish between the path of the file you are working on and the relative path of the current command line. The former is usually the project path address, and the latter is usually the current working path.

Text processing

Once you have the ability to read and write files, you need to replace the text as you develop the automation tool. In the actual development process, text processing usually adopts two ways: regular substitution and abstract syntax tree transformation.

Regular replacement

For simple text, we usually use regular substitutions. The benefits are that the code is relatively clean and can be implemented using an interface built into the language without the need for additional libraries. The most common interface handling in JS is string.replace or shell script execution with shellJS module. The former is mainly for regular processing, while the latter can use shell script powerful text processing tools such as SED, AWK, etc. Look at this code:

import { promises as fs } from 'fs';

const code = await fs.readFile('./test-code.js');

code = code.replace(/Hello/.'World');
code = code.replace(/console.log\\((.*)\\)/.'console.log($1.toUpperCase())');

await fs.writeFile('./test-new-code.js', code); 
Copy the code

AST (Abstract syntax tree)

Using the regular approach is sufficient for regular file modifications, but there is a problem with using the regular approach: strings are often unstructured, so readability is not very good. At the same time, for complex scenarios, such as the need for some logical judgment, it is difficult to use re to cover well.

Then we have another solution, that is, we can directly parse the source code into structured data (AST), and directly add, delete, modify and check in the abstract syntax tree, and replace it with the results we want. Finally, write the transcoded AST back to the file. This whole process is a bit like what Babel translator does.

Learning how to operate the AST not only helps us develop automation tools, but also enables the following functions:

JS code syntax style check, (see esLint)
Error messages, auto-completion, refactoring in the IDE
Code compression and obfuscation, code conversion (see prettier, Babel)

To learn how to use aN AST for text transformation, you first need to understand the common structure of abstract syntax trees. It is essentially a tree structure with language programming information attached. The nodes in it are the products of lexical parsing, such as literals, identifiers and methods, call declarations, and so on. Here are some common syntax node information (tokens):

Literal: Literal
Identifier: Identifier
CallExpression: Method invocation
VariableDeclaration: VariableDeclaration

To see an abstract syntax book after code parsing, you can use the tool AST EXplorer.net astexplorer.net/.

esprima + esquery + escodegen

The combination of Esprima + EsQuery + EsCodeGen is a common tool for manipulating AST. The esprima esprima.org/ library is mainly used to parse the JS syntax tree, as shown in the following code:

import { parseScript } from 'esprima';

const code = `let total = sum(1 + 1); `
const ast = parseScript(code);
console.log(ast)
Copy the code

The parseScript interface allows you to extract the syntax tree structure from the source file. The following structure is obtained:

Is a nested tree structure that can obtain all node information through deep traversal. After parsing the source code and getting the syntax tree, we can manipulate the node structures just as we would the DOM structure. Here we use the EsQuery tool to find the nodes that need to be modified:

import { parseScript } from 'esprima';
import { query } from 'esquery';

const code = 'let total = say("hello world")';
const ast = parseScript(code);
const nodes = query(ast, 'CallExpression:has(Identifier[name="say"]) > Literal');

console.log(nodes);
Copy the code

Finally, we can get the parameter values of the say method call:

[
  Literal {
    type: 'Literal',
    value: 'hello world',
    raw: '"hello world"'}]Copy the code

Then, we can try to modify the AST node information ourselves, such as here I want to change the parameter in the code to “Hello bytedance”, and finally generate the code. The code is as follows:

import { parseScript } from 'esprima';
import { query } from 'esquery';
import { generate } from 'escodegen';

const code = 'let total = say("hello world")';
const ast = parseScript(code);
const [literal] = query(ast, 'CallExpression:has(Identifier[name="say"]) > Literal');
literal.value = 'hello bytedance';

// Generate the final code with esCodeGen. Escodegen: accepts a valid AST and generates JS code
const result = generate(ast);
console.log(result);
// Final result: let total = say("hello bytedance");
Copy the code

Of course, sometimes you need to replace the whole syntax tree, so you can use the Estemplate library to quickly generate the corresponding AST information and assemble it onto the original AST. Take the following code for example:

var ast = estemplate('var <%= varName %> = <%= value %> + 1; ', {
  varName: {type: 'Identifier'.name: 'myVar'},
  value: {type: 'Literal'.value: 123}});console.log(escodegen.generate(ast));
// > var myVar = 123 + 1;
Copy the code

The AST can be generated in a templatable language, making it easy to modify the old AST structure when adding or replacing nodes.

example

Let’s use the above knowledge to achieve a few interesting little functions:

Implement a custom ESLint rule

import { parseScript } from 'esprima';
import { query } from 'esquery';

const code = `Object.freeze()`;
const ast = parseScript(code);
const queryStatement = 
  'CallExpression:has(MemberExpression[object.name="Object"][property.name="freeze"])';
const nodes = query(ast, queryStatement);

if(nodes.length ! = =0) {
  throw new Error('Do not use object.freeze! `);
}
Copy the code

To put this into practice, I prefer to build a JScodeshift, a Codemode tool provided by Facebook. The bottom layer encapsulates recast github.com/benjamn/rec… This library.

In this file the entire process, the principle is the same as above. It also involves parsing the syntax tree, modifying the syntax tree, and finally generating code. Moreover, the transform function is used to expose the interface. The advantage of the transform function is that the interface is very concise and the programming style of the original code can be retained in the final output code, so it is very suitable for code refactoring, configuration file modification and other scenarios.

How it works is shown below:

AST == DOM树 AST-EXPLORER == 浏览器 JSCODESHIFT == Jquery

Find => Find the operation

Node lookup is the most central step for AST operations, and we can often visualize node information using the AST-Explorer platform. Then use the query statement to locate the desired node path.

Look at this code:

find(j.Property, {value: { type: 'literal'.raw: 'xxx'}})Copy the code

Replace => Replace operation

Replacing nodes is also a very common feature in the actual development process, and new nodes are constructed in accordance with ast-types github.com/benjamn/ast… Type definition of:

node.replaceWith(j.literal('test'));// Replace with a string node

node.insertBefore(j.literal("test")); // Insert the newly constructed AST after this node

node.insertAfter(j.literal()); // Insert the newly constructed AST in front of the node
Copy the code

Here’s a trick to remember the API: “Find things in upper case, create nodes in lower case.”

Create => Create a node

j.template.statements`var a = 1 + 1`;


j.template.expression`{a: 1}`;
Copy the code

export default function transformer(file, api) {

  // import jscodeshift
    const j = api.jscodeshift;
    // get source code

    const root = j(file.source);
    // find a node

    return root.find(j.VariableDeclarator, {id: {name: 'list'}})
    .find(j.ArrayExpression)
    .forEach(p= > p.get('elements').push(j.template.expression`x`))
    .toSource();
};
Copy the code

J (file.source).tosource ({quote:'single'}); J (file.source).tosource ({quote:'double'});
Copy the code

Print => Print the final output

The printing part of the code is relatively simple and can be done directly using the toSource method. Sometimes we also need to control some code output format (such as quotation marks), you can use attributes such as quote to handle this.

test

When writing CoDemod code, testing is essential. Since it involves file modification, using tests can greatly simplify our development work.

Jscodeshift officially provides some test tool functions that can be used directly to write our test code quickly. First, you need to create two directories:

Testfixtures: This directory is mainly used to store test files to be modified, with the input.js end representing the files to be converted and the output.js end representing the expected converted files.
Tests: This directory is used to store all test case code

const { defineTest } = require('jscodeshift/dist/testUtils');
const transform = require('.. /index');
const jscodeshift = require('jscodeshift');
const fs = require('fs');
const path = require('path');

jest.autoMockOff();

defineTest(__dirname, 'bff');

describe('config'.function () {
  it('should work correctly'.function () {
    const source = fs.readFileSync(
      path.resolve(__dirname, '.. /__testfixtures__/config.output.ts'),
      'utf8'
    );
    const dest = fs.readFileSync(
      path.resolve(__dirname, '.. /__testfixtures__/config.output.ts'),
      'utf8'
    );
    const result = transform.config({ source, path }, { jscodeshift });
    expect(result).toEqual(dest);
  });
});
Copy the code

// The second argument is used to specify the scope. If not specified, it takes effect globally
jscodeshift.registerMethods({
    log: function() {
        return this.forEach(path= > console.log(path.node.name));
    }
}, jscodeshift.Identifier);

jscodeshift.registerMethods({
    log: function() {
        return this.forEach(path= > console.log(path.node.name)); }});// Then you can use custom methods directly in the syntax tree
jscodeshift(ast).log();
Copy the code

The extend expansion

Apart from the basic jscodeshift interface, jscodeshift provides an extension interface that allows you to bind custom utility functions to the Jscodeshift namespace using registerMethods.

example

Code refactoring tool: github.com/reactjs/rea… This is the official code migration tool provided by React, which can greatly reduce the human cost when refactoring code for large projects.
Arco Design Migration Tool: [email protected] Migration Arco Design Guide

Reference documentation

AST parser:

Github.com/acornjs/aco…
www.npmjs.com/package/est…
Github.com/facebook/js…

mo4tech.com (Moment For Technology) is a global community with thousands techies from across the global hang out!Passionate technologists, be it gadget freaks, tech enthusiasts, coders, technopreneurs, or CIOs, you would find them all here.

Play with automated tool development

Processing user input

Pure command-line tools

Interactive command line tool

Inline script

File to read and write

JSON

The path

Text processing

Regular replacement

AST (Abstract syntax tree)

esprima + esquery + escodegen

example

Implement a custom ESLint rule

Find => Find the operation

Replace => Replace operation

Create => Create a node

Print => Print the final output

test

The extend expansion

example

Reference documentation

Play with automated tool development

Processing user input

Pure command-line tools

Interactive command line tool

Inline script

File to read and write

JSON

The path

Text processing

Regular replacement

AST (Abstract syntax tree)

esprima + esquery + escodegen

example

Implement a custom ESLint rule

Find => Find the operation

Replace => Replace operation

Create => Create a node

Print => Print the final output

test

The extend expansion

example

Reference documentation

Related Posts

CSS classic layout – The Holy Grail layout

Keep-alive Cache (Advanced)

The fondly used JS extension operator 13