Picographer.co/Rocket-lift…

By Pan Wanqiang

A tease.

The opposite of “incremental” here is “total.” On Linux, when you need to back up data or synchronize files across servers, you use a tool called rsync, which is faster than the SCP /cp command because it determines the difference between the existing data and the new data and transfers only the different parts, known as “incremental” synchronization.

In the area of engineering front-end development, this article introduces the use of “incremental” thinking to speed up the code review, packaging and build processes, and thus improve the efficiency of the development process.

2. Incremental code checks

The front-end uses ESLint for static code specification checking. As front-end engineering evolves, we’ll integrate code reviews with the development workflow, automatically doing ESLint checks before and after code is submitted. Code commit checks trigger ESLint checks via Git hooks when a developer commits. When a project has a lot of code, it can take a couple of minutes to complete each commit. Code delivery checks are made with the help of continuous integration processes, such as triggering code checks at MR, which can block MR’s flow. It is often the case that a MR changes only one line of code, but scans the entire project, which can seriously affect the efficiency of continuous integration. So in most cases we don’t need to do a full scan of ESLint, we’re more concerned with whether there’s a problem with the new code.

Next we implement incremental code commit checking for a project by customizing Git’s pre-commit hook script.

2.1 Looking for the Modified file

In this script, ESLint checks execution to file granularity. The first step in implementing incremental code checking is to be able to find the incremental code, that is, which files have been modified. We used the Git version management tool to look for the difference between the staging area and the HEAD at commit time and to find the list of files that were modified.

  1. usegit diffFind the file that you want to modify this time, and add--diff-filter=ACMRTo remove the deleted file, the deleted file does not need to be checked again;
  2. Use the exec function of the child_process module to execute git commands in node.
  3. The output is a string composed of modified files, do a simple string processing to extract the list of files to check;
const exec = require('child_process').exec;
const GITDIFF = 'git diff --cached --diff-filter=ACMR --name-only';
// Execute git command
exec(GITDIFF, (error, stdout) = > {
    if (error) {
        console.error(`exec error: ${error}`);
    }
    // Process the returned result and get the list of files to check
    const diffFileArray = stdout.split('\n').filter((diffFile) = > (
        /(\.js|\.jsx)(\n|$)/gi.test(diffFile)
    ));
    console.log('Files to check:', diffFileArray);
});
Copy the code

2.2 Check the code of the modified file

ESLint provides a function of the same name (ESLint) as a call to the Node.js API (earlier versions of 7.0.0 use the CLIEngine class) so we can perform code checks in the Node script and get the results.

  1. Use ESLint’s lintFiles function to code check the list of files;
  2. The result returned is an array of check results for each file. The array is processed to get the check results and output a prompt.
const { ESLint } = require('eslint');
const linter = new ESLint();

// After obtaining the list of documents to be checked above

let errorCount = 0;
let warningCount = 0;
if (diffFileArray.length > 0) {
    // Perform ESLint code checks
    const eslintResults = linter.lintFiles(diffFileArray).results;
    // Process the check results and extract the number of error reports and warning numbers
    eslintResults.forEach((result) = > {
        // The data structure of result is as follows:
        / / {
        // filePath: "xxx/index.js",
        // messages: [{
        // ruleId: "semi",
        // severity: 2,
        // message: "Missing semicolon.",
        // line: 1,
        // column: 13,
        // nodeType: "ExpressionStatement",
        // fix: { range: [12, 12], text: ";" }
        / /}],
        // errorCount: 1,
        // warningCount: 1,
        // fixableErrorCount: 1,
        // fixableWarningCount: 0,
        // source: "\"use strict\"\n"
        // }
        errorCount += result.errorCount;
        warningCount += result.warningCount;
        if (result.messages && result.messages.length) {
            console.log(`ESLint has found problems in file: ${result.filePath}`);
            result.messages.forEach((msg) = > {
                if (msg.severity === 2) {
                    console.log(`Error : ${msg.message} in Line ${msg.line} Column ${msg.column}`);
                } else {
                    console.log(`Warning : ${msg.message} in Line ${msg.line} Column ${msg.column}`); }}); }}); }Copy the code

2.3 Friendly Prompt and error handling

  1. Make a friendly output prompt on the command line interface to check whether the code is passed.
  2. If there are any errors in the check result, git will quit with a non-zero value and abandon the commit.
if (errorCount >= 1) {
    console.log('\x1b[31m'.`ESLint failed`);
    console.log('\x1b[31m'.` ✖${errorCount + warningCount} problems(${errorCount} error, ${warningCount} warning)`);
    process.exit(1);
} else if (warningCount >= 1) {
    console.log('\x1b[32m'.'ESLint passed, but need to be improved.');
    process.exit(0);
} else {
    console.log('\x1b[32m'.'ESLint passed');
    process.exit(0);
}
Copy the code

At this point, the pre-commit hook script is complete, and you just need to configure the execution of the script in the package.json file to implement incremental code checking. The end result is that developers no longer have to wait for the full code review to complete before submitting their code, and the script will quickly find the changed file and review it.

If you want to implement this in your own projects, you can use the open source library Lint-Staged directly in conjunction with Husky.

The incremental check for code delivery is implemented in a similar way to the above steps. The key is to find the incremental part.

2.4 Comparison of results

Take a medium scale project with 460 JS files as an example. The left side shows the time of full code review, and the right side shows the time of incremental code review:

If the developer changed only one file, the full review at commit took 38 seconds, while the incremental review only took 2 seconds.

2.5 More granular incremental checks

Given that large projects can have many large files, ESLint is still inefficient to check the entire file if only a few lines of code have been modified. We can try to find incremental lines of code to check.

First of all, use git diff command to find the modified part, here you need to do some string processing to extract the code block; The code block is then checked using the lintText method in the ESLint Node.js API. If you are interested in it, you can try it yourself.

Three. Incremental packaging build

Consider a business scenario in which a large multi-page Web application (MPA) with hundreds of pages takes tens of minutes for each fully packaged build. Sometimes developers only change a page or a common component, but they have to wait a long time to release it, seriously affecting the efficiency of continuous integration and online problem solving.

Using the example of a packaged build with WebPack, we also try to optimize the problem with “incremental” thinking.

3.1 Searching for the Modified File

As before, the first step is to find the incremental code, which files have been modified in this release. The easiest is still to choose to use the Git diff command. Unlike the incremental code check, here you compare the integration branches to be released to the trunk and find a list of different files between them.

const execSync = require('child_process').execSync;
const path = require('path').posix;
const GITDIFF = 'git diff origin/master --name-only';
// Execute git command
const diffFiles = execSync(GITDIFF, {
        encoding: 'utf8',
    })
    .split('\n')
    .filter((item) = > item)
    .map((filePath) = > path.normalize(filePath));
Copy the code

After obtaining the modified file list, webPack packaging cannot be triggered directly. You need to find the entry file according to the reference relationship between the files, and pass the page entry that needs to be repackaged to WebPack.

3.2 Calculating incremental entries

The idea is to build a dependency tree for each page entry file, and if the tree contains the modified file mentioned above, the page needs to be repackaged. As shown in the figure:

The modified files are already available, and the next step is to build the dependency tree for each entry file. There are many front-end modularized specifications, to implement the dependency analysis of each file needs to take into account various situations, here recommended an open source library Madge, it will be code into an abstract syntax tree for analysis, and finally return a dependency tree.

For example, the two entry files in the figure above have the following dependency tree:

// List of files to be modified
const diffFiles = ['util/fetch.js'];

// Use madge library to calculate the dependency tree example code, can see the official documentation
// Promise.all([madge('./demo/index.js'), madge('./demo/buy.js')]).then((result) => {
// result.forEach((e) => {
// console.log(e.obj());
/ /})
// });
// The resulting dependency tree is as follows
const relyTree = {
    // Demo /index.js file dependency tree
    {
        'demo/a.jsx': ['util/fetch.js'].'demo/b.js': [].'demo/index.js': ['demo/a.jsx'.'demo/b.js'].'util/fetch.js': []},// Demo /buy.js file dependency tree
    {
        'util/env.js': [].'demo/buy.js': ['demo/c.js'.'demo/d.js'].'demo/c.js': ['util/env.js'].'demo/d.js': []}};Copy the code

The dependency tree of each entry file is deeply traversed to determine whether the files in the list of modified files need to be repackaged and built. The example code is as follows:

Example code for calculating incremental entry */
// Full page entry
const entries = [
    'demo/index.js'.'demo/buy.js',];// Check whether two arrays intersect
function intersection(arr1, arr2) {
    let flag = false;
    arr1.forEach((ele) = > {
        if (arr2.includes(ele)) {
            flag = true; }});return flag;
}

// Calculate the incremental entry
const incrementEntries = [];
for (const i in relyTree) {
    for (const j in relyTree[i]) {
        if(intersection(relyTree[i][j], diffFiles)) { incrementEntries.push(i); }}}Copy the code

For example, we know that the util/fetch. Js file is modified in this release, and we know that only demo/index page is affected by the above two dependency trees. Changing the webPack configuration to trigger packaging only with this file as the entry parameter can greatly improve the speed of packaging build.

3.3 Boundary Cases

Some front-end engineering dependencies are the NPM packages described in package.json, installed in the node_modules folder, and the dependencies between modules are quite complex. For simplicity’s sake, when git Diff finds a module upgrade in package.json in the first step, it can trigger full packaging, given that this is not a very frequent event.

3.4 Comparison of results

Take an MPA project with 50 pages as an example. In the following figure, the left side shows the time of the full packaging build, and the right side shows the time of the incremental packaging build:Assuming that the developer makes changes to two pages and incremental packaging calculates that only these two page entries are sent to WebPack, the entire packaging build process will be reduced from 7 minutes to 50 seconds, greatly improving continuous integration.

summary

This article uses two specific business scenarios, incremental code checking and incremental packaging building, as examples to show how incremental thinking can improve efficiency in front-end development engineering. These two cases may not be directly transferable to your front-end engineering practice, but they are intended to introduce a programming design idea that you can use in more places with your own imagination.

The resources

  • IncrementLint
  • ESLint Node.js API
  • Use Husky, commitLint, and Lint-Staged to build your front-end workflow
  • How to make your Lint checks more efficient?
  • Use Madge to generate engineering dependency diagrams

This article is published by netease Cloud Music front end team. Any unauthorized reprint of the article is prohibited. We hire front end, iOS, Android all the year around. If you’re ready for a career change and you love cloud music, join us GRP. Music-fe (at) Corp.netease.com!