SSH, member of bytedance Web Infra team, welcome to contact Sshsunlight

This article is a reflection on the deprecated code removal tool I recently wrote within the company, which has so far removed about 6W lines of code across multiple projects.

First in the front end of the public account from advanced to hospital, welcome to pay attention.

The cause of

Many projects have a long history, and many of the files or exported variables are no longer used, affecting maintenance iterations. For example, if the backend asks you, “Interface x counts whether interface X is still in use?” You look in the project and say, oh boy, there are several other uses, and it turns out that the definition or document was never introduced, which leads you to continue to maintain the document or interface, which affects iterative efficiency.

We will start by deleting obsolete exports, and we will continue to delete obsolete files.

Exports, there are several difficulties:

  1. How to find variables that are exported but not imported by other files stably?

  2. How do I determine that the variables in Step 1 are not used within this file (scoping)?

  3. How to delete these variables stably?

The overall train of thought

To give the overall idea, our partner in the company recommended the open source library Pzavolinsky/TS-unused-exports, which has been used steadily in the project for a period of time. This library can solve the appeal of the first step above, that is, to find out the export. But other files do not import variables. But the next two steps are still tricky, and I’ll start with my conclusion:

  1. How do I determine that the variables in Step 1 are not used within this file (scoping)?

Call the ESLint API on the parsed file,no-unused-varsBy default, ESLint does not support analysis of exported variables, because if you export the variable, ESLint assumes that it will be used externally. For this limitation, it’s just a simple rewrite of fork.

  1. How to delete these variables stably?

Write your own rule fixer delete useless variable analysis, followed by formatting, because ESLint after removing code format will go bad, so call manually prettier API to restore code is beautiful.

I’ll go over each of these steps in detail.

Export and Import analysis

After testing, pzavolinsky/ TS-unused-exports can reliably analyze unused export variables, but this tool for analyzing the relationship between import and export is limited to this. We will not analyze whether the exported variable is used inside the code.

Internal file usage analysis

In the second step, the problem is more complicated. In this step, ESLint is used to write the no-unused-vars rule and provide a fixer for the rule.

Why ESLint?

  1. Widely used by the community and validated by numerous projects.

  2. Based on scope analysis, find out exactly the unused variables.

  3. The AST provided conforms to estREE/ESTREE general standards and is easy to maintain and extend.

  4. ESLint can solve the problem of introducing new useless variables after a deletion, typically when a function is removed, and a function within that function may also become invalid code. ESLint executes the fix function repeatedly until there are no more new fixable errors.

Why fork it and rewrite it?

  1. The official no-unused-vars default does not consider exported variables. After reading the source code, I found that only a small amount of code changes can break this restriction, so that exported variables can also be analyzed for use within the module.

  2. After the first rewriting, many exported variables are referenced by other modules, but because they are not used inside the module, they will also be analyzed as unused variables. So I need to provide a varsPattern rule option, the analysis scope is limited to ts – unused – exports give the export of unused variables, such as varsPattern: ‘^ foo $| ^ bar $’.

  3. The official no-unused-Vars only provide hints and do not provide automatic repair solutions. You need to write your own, which will be explained in detail below.

How to delete variables

When writing code in the IDE, we sometimes find that some parts of ESLint that float red are automatically fixed after saving, but others don’t. This is what ESLint’s Rule Fixer does. By referring to the Apply Fixer section of the official documentation, each ESLint Rule writer can decide if and how their Rule can be automatically fixed. ESLint provides a fixer kit that encapsulates many useful node operations, such as fixer.remove(), Fixer. ReplaceText (). The official no-unused-Vars does not provide an automatic fix for the code due to stability. You need to write a fixer for this rule yourself. Add fix/suggestions to no-unused-vars rule · Issue #14585 · eslint/ esLint

The core changes

Split the ESLint Plugin into a separate directory with the following structure:

.js.js.js.js.js.js.js.js.js.js.js.js.js.js.js.js.js.js.js.js.js.js.js.js.js.js.js.js.js.js.js.js.js.js Unused-vars.js ├── eslint-rule-js ├─ package.jsonCopy the code

  • Eslint-plugin. js: a plug-in entry. Rule can only be used if it is imported from outside

  • Eslint-rule-unused-vars. js: esLint’s official ESLint /no-unused-vars code, where the main core code is located.

  • eslint-rule-typescript-unused-vars : The code inside typescript-eslint/no-unused-vars inherits eslint/no-unused-vars and adds some typescript AST node analysis.

  • Eslint-rule-js: a rule entry that introduces typescript rule and adds auto-fix logic to the rule using ESlint-rule-Composer.

ESLint Rule changes

Our analysis is related to deletion, so there must be a strict scope for export variables that are deemed to be external unused by TS-unused -exports. For this reason, consider adding a varsPattern configuration that imports the name of the unused ts-unused-exports variable. The main change logic is in the collectUnusedVariables function, which is used to collect variables that are not used in the scope. Exports are skipped here.

else if (
  config.varsIgnorePattern &&
  config.varsIgnorePattern.test(def.name.name)
) {
  // skip ignored variables
  continue;
+ } else if (
+ isExported(variable) &&
+ config.varsPattern &&
+! config.varsPattern.test(def.name.name)
+) {
+ // Conforms to varsPattern
+ continue;
+}
Copy the code

The outside can then limit the scope of the analysis in such a way as:

rules: {
  '@deadvars/no-unused-vars': [
    'error',
    { varsPattern: '^foo$|^bar$']}},Copy the code

Next, remove the “isExported” judgment that you collected unused variables in the original version, and collect all exported variables that are not used within the file. }}}}}}}}}}}}}}}}}}}}}}}}}}

if ( ! isUsedVariable(variable) &&- !isExported(variable) &&! hasRestSpreadSibling(variable) ) { unusedVars.push(variable); }Copy the code

ESLint Rule Fixer

The next step is to add automatic fixes. This part of the logic in eslint-rule-js is simply to judge and delete the AST nodes of various unused variables analyzed in the previous step. Here is the simplified function handling code:

module.exports = ruleComposer.mapReports(rule, (problem, context) = > {
  problem.fix = fixer= > {
    const { node } = problem;
    const { parent } = node;

    // Function node
    switch (parent.type) {
      case 'FunctionExpression':
      case 'FunctionDeclaration':
      case 'ArrowFunctionExpression':
        // Call fixer to delete
        returnfixer.remove(parent); . .default:
        return null; }};return problem;
});
Copy the code

Currently, the following node types are deleted:

  • FunctionExpression

  • FunctionDeclaration

  • ArrowFunctionExpression

  • ImportSpecifier

  • ImportDefaultSpecifier

  • ImportNamespaceSpecifier

  • VariableDeclarator

  • TSEnumDeclaration

You only need to maintain this file for the subsequent deletion logic of new nodes.

Delete unnecessary files

I did a version of useless code deletion based on webpack-Deadcode-plugin before, but some problems were found during the actual use.

The first is that it is too slow. The plug-in analyzes which files are useless based on the results of the WebPack compilation, and needs to compile the project every time it is used.

In addition, after the fork-ts-checker-webpack-plugin was added a few days ago for type checking, the deletion scheme suddenly failed, and only useless files of type.less were detected. After checking and checking, it was found that they were the pot of this plugin. It will take all of the ts files under the SRC directory to join webpack dependence, namely compilation. FileDependencies (can try to open the plug-in, in the development environment to try to move to a completely not import the ts file, as will trigger a recompile)

And deadcode – the plugin is dependent on compilation. FileDependencies this variable to determine which files are not used, all ts file, in the variable scan out useless files only naturally other types.

This action should be intentional on the part of the official plug-in, considering the following:

// Import a TS type directly
import { IProps } from "./type.ts";

// use IProps
Copy the code

When using an older version of the fork-ts-checker-webpack-plugin, if you change the IProps to cause a type error, the webPack compilation will not trigger an error.

Tsconfig includes all ts files, including [” SRC /**/*.ts”], and includes [” SRC /**/*.ts”].

Files that provide only type dependencies for main entry and unused Files are not being checked for detailed discussion

plan

First, try manually deleting the fork-ts-checker-webpack-plugin in Deadcode mode, which will scan for unwanted dependencies, but importing only types from files as described above will still be considered useless and deleted by mistake.

Considering that there are many situations in real world where a type.ts file writing interface or type is created, we have to give up this scheme first.

If pzavolinsky/ TS-unused -exports can analyze the dependencies of all imported and exported files, it should also be easy to analyze unused files.

After debugging the source code, the principle of this tool is roughly teased out:

  1. Built-in in TypeScriptts.parseJsonConfigFileContentThe API scans the complete TS file path in the project.
 {
    "path": "src/component/A"."fullPath": "/Users/admin/works/test/src/component/A.tsx",
  {
    "path": "src/component/B"."fullPath": "/Users/admin/works/test/apps/app/src/component/B.tsx",}...Copy the code
  1. Exports and imports are analyzed using TypeScript’s built-in compile API.
{
  "path": "src/component/A"."fullPath": "/Users/admin/works/test/src/component/A.tsx"."imports": {
    "styled-components": ["default"]."react": ["default"]."src/components/B": ["TestComponentB"]},"exports": ["TestComponentA"]}Copy the code
  1. According to the above information, analyze the use times of each variable in each file, screen out unused variables and output them.

The idea is to collect all the imports from the files and find the files that are not included in the imports from the first step.

Some noteworthy changes

Cyclic deletion of files

After the first useless file is detected and deleted, it is likely that some new useless files will be exposed. Here’s an example:

[{"path": "a"."imports": "b"
  },
  {
    "path": "b"."imports": "c"
  },
  {
    "path": "c"}]Copy the code

File A imports file B, and file B imports file C.

In the first round of scanning, no file was introduced into A, so A was regarded as useless file.

Since A introduces B, B is not treated as a useless file, and c is also not treated as a useless file.

So the first delete is just going to delete file A.

As long as the scope of files is narrowed after each deletion, for example, after a is deleted for the first time, only files are left:

[{path: "b".imports: "c"}, {path: "c",},];Copy the code

B is added to the list of useless files. Repeat this step to delete c files.

Support Monorepo

Monorepo is now very popular. Each project in Monorepo has its own tsConfig and forms its own project. It is often the case that files or variables in project A are used by dependencies in Project B.

If you scan files within a single project, you will mistakenly delete many files used by the quilt project.

The idea here is simple:

  1. Added the –deps parameter to allow passing tsConfig paths for multiple subprojects.

  2. Filter the imports section scanned by the subproject to find dependencies imported from the main project with the alias @main (for example, import {Button} from ‘@main/components’).

  3. Merge the imports into the main project’s dependency set for the next scanning step.

Supports custom file scanning

The TypeScript API only scans for.ts and.tsx files by default. AllowJS and.jsx files are also scanned with allowJS enabled. Many of the.less,.svg files in the project were not used, but they were ignored.

Here I breakpoint follow up ts parseJsonConfigFileContent function inside, found that there are some more hidden parameters and logic, use of hack ways to support the custom suffix.

Of course, there are some troublesome changes involved, such as the library originally did not consider index.ts, index.less exists at the same time, through some changes in the source code eventually bypassed this restriction.

Less,.sass, and.scss are supported by default. You can add custom suffixes by adding extraFileExtensions configurations, as long as you make sure the suffixes are imported using import syntax.

import * as ts from "typescript";

const result = ts.parseJsonConfigFileContent(
  parseJsonResult.config,
  ts.sys,
  basePath,
  undefined.undefined.undefined, extraFileExtensions? .map((extension) = > ({
    extension,
    isMixedContent: false.// hack ways to scan all files
    scriptKind: ts.ScriptKind.Deferred,
  }))
);
Copy the code

Other options: TS-prune

Ts-prune is a dead exports inspection solution implemented entirely based on TypeScript services.

background

The TypeScript service provides a useful API: findAllReferences. When we right-click on a variable in VSCode and select “Find AllReferences”, we call this low-level API to Find AllReferences.

Ts-morph is a library that encapsulates some low-level apis including findAllReferences, providing a more concise and easy to use invocation method.

Ts-prune is a package based on TS-Morph.

A simplified t-MORph-based code for dead exports looks like this:

// this could be improved... (ex. ignore interfaces/type aliases that describe a parameter type in the same file)
import { Project, TypeGuards, Node } from "ts-morph";

const project = new Project({ tsConfigFilePath: "tsconfig.json" });

for (const file of project.getSourceFiles()) {
  file.forEachChild((child) = > {
    if (TypeGuards.isVariableStatement(child)) {
      if (isExported(child)) child.getDeclarations().forEach(checkNode);
    } else if (isExported(child)) checkNode(child);
  });
}

function isExported(node: Node) {
  return TypeGuards.isExportableNode(node) && node.isExported();
}

function checkNode(node: Node) {
  if(! TypeGuards.isReferenceFindableNode(node))return;

  const file = node.getSourceFile();
  if (
    node.findReferencesAsNodes().filter((n) = >n.getSourceFile() ! == file) .length ===0
  )
    console.log(
      ` [${file.getFilePath()}:${node.getStartLineNumber()}: ${ TypeGuards.hasName(node) ? node.getName() : node.getText() }`
    );
}
Copy the code

advantages

  1. TS services are integrated with a variety of ides, tested by numerous large projects, and reliability goes without saying.

  2. Instead of checking for extra variables to be used inside the file, as in ESLint, findAllReferences checks inside the file, right out of the box.

disadvantages

  1. Slow, TSProgram initialization, and findAllReferences call, in a large project speed is still a little slow.

  2. Documentation and specifications are poor, ts-MORPH’s documentation is still too crude, many core methods are not documented, which is not good for maintenance.

  3. Module syntax is inconsistent. TypeScript findAllReferences does not recognize the Dynamic Import syntax and requires additional processing for modules imported in the form of Import ().

  4. T-prune encapsulates a relatively perfect dead exports test solution, but the author doesn’t seem to want to do it automatically. This is where the second disadvantage comes in. It is very difficult to follow the document to explore the deletion solution. Looks like a German guy managed to convince the author to mention an auto-deleted MR: Add a fix mode that automatically fixes unused exports (revival), but it doesn’t pass GithubCI because of memory overflow. I personally forked the code down and ran it on a large internal project, and it was indeed a memory overflow, and looked at the auto-fix code, which was also a very routine TS-Morph-based API call, and guessed that it was a performance problem with the underlying API?

After a comprehensive evaluation, ts-unused-exports + ESLint was finally chosen.

The last

We are bytedance’s Web Infrastructure Team. As the company’s basic technology Team, our goal is to provide excellent technical solutions to help the company grow, and at the same time to create an open technology ecosystem to promote the development of the company and the industry’s front-end technology. At present, the main focus of the team includes modern Web development solutions, low code building, Serverless, cross-end solutions, terminal basic experience, ToB, etc., and has set up research and development teams in many places, including Beijing, Shanghai, Hangzhou, Guangzhou, Shenzhen, Singapore.

Team column: zhuanlan.zhihu.com/bytedancer

Drop wX: Sshsunlight

Some team members:

  • github.com/leeight (Team Leader)
  • github.com/dexteryy (JS Hacker, SF/F Nerd)
  • github.com/underfin (Vue.js Core Contributor)
  • github.com/Amour1688 (Vue.js Contributor)
  • github.com/oyyd (Node.js Core Contributor)
  • github.com/theanarkh (Node.js Advocate)
  • github.com/leizongmin (Node.js Advocate)
  • github.com/losfair (WebAssembly)
  • Github.com/Brooooookly… (Rust & napi.rs/)
  • github.com/amio
  • Github.com/niudai & www.zhihu.com/people/niu-…
  • github.com/protoman92