Old engineering package speed up (1) – Roadhog package speed up

background

In 2019, the build of the front-end framework of the project often reached more than 900s, and the code recompilation of the development environment also required about 10-20s, which seriously affected the development efficiency and made optimization urgent.

The environment

Hardware: I5-6200U 4-core @ 2.3ghz, 8G RAM, SSD.

Software: Win10, Node V8.12.0, Webpack v3.12.0, NPM v6.2.0.

Front-end frameworks: DVA + ANTD, DVA V1.2.1, ANTD V2.13.13, Roadhog v1.3.4.

The status quo

It involves 108 modules, first build: 87.1s, second build: 43.3s, package: 760.8s, package file size: 90.7m.

Improve the process

Upgrading related build tools, such as Node, Webpack, or Roadhog and DVA, will increase the speed somewhat. However, it is very troublesome to blindly upgrade the subsequent dependency error check, or even related component functions are not available. Therefore, when the system is stable and there are many business modules, We are not considering that option.

Remove redundant modules and invalid references

The current project was reconstructed from another old project, and the function modules were 80% similar at the beginning, so the front-end framework of the old project was directly used at the beginning of the project. Therefore, many modules of the old project are actually useless under the current project. In addition, the code is riddled with invalid references as shown below

import xx from 'xxx';
Copy the code

In fact, the business implementation does not use XX, and the related libraries that have been discontinued must be removed in time to remove the related dependencies from the source code.

By removing redundant modules and invalid references, the packaging speed is over 100 seconds faster. The above situation is actually the result of this part of the optimization work.

However, with subsequent business expansion and iteration, there will still be more and more modules, so the optimization of this way is still relatively limited.

Webpack visualizer – analysis of the plugin

A very useful plugin that allows you to clearly see the components of the code, down to each file, the percentage of each module, and the multiple version references that may exist in the project.

The roadhog plug-in is built in, and when you add the following code to the Webpack, a stats.html file will be generated at build time.

const Visualizer = require('webpack-visualizer-plugin');
  
webpackConfig.plugins.push(
    new Visualizer()
);
Copy the code

Stats.html looks like this:

Echarts takes up more than 2M, tF-AG-Grid takes up 1.4m, csS-Loader, Lodash, moment, ANTD take up 1.4m.

externals

External simply declares our dependency as an external dependency, and the external dependency passes

As for Echarts, which is the biggest package size, we only use it on the front page, it can be separated from the project and introduced externally.

In webpack.config.js, add the following code:

webpackConfig.externals = {
    'echarts': 'echarts'
};
Copy the code

So when we import echarts from ‘echarts’ in our code, the echarts are actually taken from the window global and not packaged in the build. Of course, the corresponding Entry file must be imported through script.

Package: 711.6s, package file size: 90M. Not much has changed.

DllPlugin and DllReferencePlugin

Public libraries like Lodash and Moment can also be handled as external, but we tried it out in our project and it was still listed in stats.html. After analysis, the moment is taken as an example, because ANTD relies on moment, so it is still included in construction. At this point, the DllPlugin can be used to handle this.

Those of you who have used Windows systems will often see files with the.dll suffix. These files are called dynamic link libraries, and a dynamic link library can contain functions and data for other modules to call. Why does applying this approach to Web builds speed things up so much? The reason is that a dynamically linked library that contains a large number of reusable modules only needs to be compiled once, and modules that are included in the dynamically linked library will not be recompiled later in the build process, but will use the code in the dynamically linked library directly. Since most of the DLC contains common third-party modules, such as React and react-dom, the DLC does not need to be recompiled unless the versions of these modules are upgraded.

Webpack already has built-in support for dynamically linked libraries, which need to be accessed through two built-in plug-ins:

1. DllPlugin plugin: used to package individual dynamic link library files.
2. DllReferencePlugin: Used to introduce the DllPlugin packed dynamic link library file into the main configuration file.

The configuration of this content can refer to here, the configuration is not cumbersome. To put it simply, the DllPlugin is used to pre-compile modules, and the DllReferencePlugin is used to reference these pre-compiled modules. Note that the DllPlugin must be executed once before the DllReferencePlugin is executed.

Fortunately, the earlier version of Roadhog already supports the DllPlugin model, but only for the development environment. Packaging for the official environment is still not supported. The configuration here mainly refers to this issue.

1. Off by default
2. The.roadhogrc dllPlugin configuration enables the configuration of exclude and include packages
3. The roadhog buildDll command is provided to manually buildDll bundles
4. After dllPlugin is enabled, roadhog buildDll is reminded to execute Roadhog buildDll when manifest.json is detected in Roadhog Server
5. The user then needs to manually import Roadhogs. DLL. Js into the debugged HTML

Roadhogrc.js: DllPlugin: true; roadhogrc.js: DllPlugin: true; You also need to run the Roadhog buildDll script to pre-generate Roadhog.dll.

So add the following command to script in package.json:

"scripts": {
    "dll": "roadhog buildDll"
}
Copy the code

Execute NPM run DLL to generate Roadhog.dl.js.

We can see roadhogs. DLL. Js and Roadhogs. Json in node_modules\ roadhogs – DLLS, which describes which modules are in the dynamic link library file, including Lodash and moment.

And introduced in the entry file index.ejs, the first build 62.3s, build 27.5s again. Using the Webpack-Visualizer-plugin, Lodash and Moment are no longer involved in the build, but ANTD will still be involved in the build, as roadhog’s authors explained. The DLL approach does not affect production packaging, so we only use this approach for development environments, and the speed increase is significant. For production packaging, we use the commonChunkPlugin for several packaged bulk files.

commonChunkPlugin

Take a look at the current package file size, packaged JS files 110, nearly 90M, there are 49 JS files reached more than 1M.

According to the analysis of stats.html, it can be seen that the excessive volume is mainly caused by repeated packaging of TF-Ag-grid, moment and ANTD. Therefore, our idea is to extract the common module to Vann.js through CommonsChunkPlugin. We added the following code in webpack.config.js for the online environment (the development environment has been optimized with DllPlugin) :

if (env === 'production') {
    webpackConfig.entry = {
        index: './src/index.js'.vendor: [
            'moment'.'lodash'.'react'.'react-dom'.'tf-ag-grid'
        ]
    }
    webpackConfig.plugins.push(
        // Extract the common parts
        new webpack.optimize.CommonsChunkPlugin({
            name: ['vendor'].filename: '[name].[hash].js'.minChunks: Infinity,}}))Copy the code

The packing test results are as follows:

Tf-ag-grid does not participate in public extraction and packaging: the packaging time is 656.4s, the size of the package file is 66.6m, and the volume of most files is reduced by 300-400KB;

Tf-ag-grid participates in public extraction and packaging: the packaging time is 355.0s, the size of packaged files is 34.0m, and the volume of some files is reduced to more than 1M.

It can be seen that after TF-AG-Grid participates in public packaging, the packaging efficiency has made a huge leap and the result is satisfactory. Of course, if the frequently used modules in ANTD in the project are separately removed and packaged into ANTD.js, the effect will be better. We will not expand them here and leave them for subsequent optimization. We can refer to this issue to solve the problem of roadhog single page application extracting public modules.

The test results

Final plan

1. Echarts is configured with externals and uses script to import the address.
2. The development environment configures DllPlugin and precompiles common modules;
3. The online environment extracts common modules through the commonChunkPlugin to speed up packaging.

Follow-up studies

1. In this packaging speed study, happyPack, PrefetchPlugin and Webpack-Ugli-fi – Parallel have also been tried, but they are either limited by Roadhog or the optimization speed is not obvious. We do not know whether there is a configuration problem or other problems, as a follow-up study.
2. For ANTD, the commonChunkPlugin can then be used to extract high-frequency modules and continue to improve packaging speed.

The last

In fact, there is another culprit that slows down the packaging speed. This culprit not only slows down the packaging speed, but also affects the SVN update, NPM install and even file replication speed, which is McAfee installed by IT for us. Of course, we have no way to do this, after all, the company’s security needs. At the end of the day, we need to optimize from our framework level.

reference

• [1] the Boost webpack build performance | Optimising webpack build performance | webpack build performance optimization
• [2] Roadhog Build Optimization – Crazy Flow – Blog Garden
• [3] sorrycc/roadhog/GitHub
• [4] 4-2 Using DllPlugin · Simple Webpack
• [5] Solving the problem of roadhog single page application extracting public modules