background

One day, I suddenly found that the build project would fail regularly. CALL_AND_RETRY_LAST Allocation failed – JavaScript heap out of memory Since the project was built on CI/CD, and the operation and maintenance adjusted the resource ceiling during this period, what caused this need to be further investigated, is it due to real memory shortage or memory leak?

Memory problems

Break through the limit

On 64-bit computers, the default memory limit for the V8 engine is about 1.5GB, which doesn’t matter if you have more RAM, but NodeJs allows you to set the memory limit for the node process by using the max_old_space_size parameter. At least let the program build first.

// Increase the limit to 4096 MB, as long as the computer supports it.
node --max_old_space_size=4096 build.js
/ / or
// Increase the upper limit to 4194304 KB
node --max_new_space_size=4194304 build.js
Copy the code

Survey questions

After adjustment, it was found that there was no problem any more, but inexplicably why there was insufficient memory? However, it is understandable that as the project gets bigger, it will inevitably run out of memory and increase the CPU. Now let’s use Chrome DevTools to troubleshoot our builder.

The process of screening tests patience, because the computer configuration is slow to run. Now that we’re talking about using Chrome DevTools, we need to build evidence. It is recommended to use node-nightly or Node-heapdump with memwatch-Next. Here we use Node-nightly. The installation and usage link is available, so I won’t go into details.

I collected a heap allocation sample and a heap dynamic allocation timeline. It turns out that there is no abnormal memory continued to grow. There are a few references that are not collected, but not memory leaks. There are two reasons for the sudden increase: 1. Instantiating Compiler, which inherits Tapable plug-in framework to register and call a series of plug-ins; 2. Instantiate a plug-in such as UglifyJsPlugin, read the source file, compile and print it, in this case sourcemap (needed for special reasons).

Sample heap memory allocation

Dynamic heap memory allocation timeline

The resources

  • Debugging Memory Leaks and Memory Bloat in Node.js – Tech @ Side
  • Leokongwq. Making. IO / 2016/11/08 /…
  • Developers.google.com/web/tools/c…

New problem – packing speed

After the memory problem is resolved, it is found that the local packaging speed is also unusually slow (note: the build environment can affect the packaging speed, but the online build environment resources are shared, so take the local computer to test, build time varies from person to person). The current packaging diagram is as follows:

My colleague’s (high end programmer) computer looks like this before optimization:

Without further ado, I have the desire to optimize because of configuration problems. The low-end configuration is as follows:

  • Model: MacBook Air(13-inch, 2017);
  • Processor: 1.8 GHz Intel Core I5
  • Memory: 8 GB 1600 MHz DDR3

Packaging related as follows:

  • Scaffolding:create-react-app v1;
  • Technology stack:React / Typescript / Antd / Less;
  • Package optimization also deals with Code Splitting, ExtractText, UglifyJs and so on.

To do a good job, he must sharpen his tools

Now is the time to select tools to analyze our project. The candidate tools are progress-bar-webpack-plugin/webpackbar/speed-measure-webpack-plugin. What we want is to be able to best analyze which phase takes time. So let’s compare these tools to see if they match our needs. PS: Webpack — Progress does not meet our needs, because the information is too simple and we have no place to investigate problems.

progress-bar-webpack-plugin

As you can see from the figure below, the Progress-bar-webpack-plugin, like webpack-Progress, does not meet our needs. It simply shows packaged progress information.

webpackbar

Webpackbar is also better than progress-bar-webpack-plugin without doing any configuration, at least knowing where the card is, loading node_modules dependent processes.

We can set profiles to get more information, of course, display information is only loaders, and we often need the time of plugins, of course, you can also customize the output information, we will not discuss here, interested friends can try.

// Display detailed information by configuring profiles
plugins: [
  new WebpackBar({
    profile: true.reporters: ['profile'].// Note that the configuration is critical, otherwise there is no information})]Copy the code

Speed-measure-webpack-plugin (recommended)

The speed-measure-webpack-plugin allows you to obtain the elapsed time of plugins and loaders with a very simple configuration.

const SpeedMeasurePlugin = require('speed-measure-webpack-plugin');
const smp = new SpeedMeasurePlugin();
const smpWrapperConfig = smp.wrap({
  // Pass the webPack configuration as a parameter to SpeedMeasurePlugin. webpackConfig, });module.exports = smpWrapperConfig;
Copy the code

test

We use speed-measure-webpack-plugin to detect the time of each stage, but it is worth noting that we only need to pay attention to which stage takes the longest time, rather than how long it runs. The addition of the speed-measure-webpack-plugin also slowed down our build time. (This is the result of my repeated testing, if there is a problem, please point out 😂)

We used speed-measure-webpack-plugin to test it and found that UglifyJsPlugin occupied the longest time. After investigation, we found that there were many such problems on Github issue, even FATAL ERROR as we mentioned above: CALL_AND_RETRY_LAST Allocation failed – JavaScript heap out of memory

Optimization of trying

As mentioned above, the construction speed of the project is slow, and UglifyJsPlugin accounts for half. Of course, there are many methods of packaging speed optimization on the Internet, but I will not expand it here. One is because the effect is not obvious, and the other is because the project itself has been dealt with in the early stage, so we will optimize it specifically here.

webpack-parallel-uglify-plugin

The uglifyjs-webpack-plugin used in CRA can also be used with parallel: true. I have tested that there is not much difference in speed between the two. More importantly, the plugin has not been updated for a long time, so this is skipped and not recommended.

happypack

As for Happypack, I’m sure there are a lot of rumors on the web about it, ranging from a single process build mode to a multi-process build mode to speed up code building. Happypack loaders are compatible with loaders. Some configurations are as follows:

const HappyPack = require('happypack');
const os = require('os');
const happyThreadPool = HappyPack.ThreadPool({ size: os.cpus().length });

// module
{
  test: /\.(ts|tsx)$/.include: resolveApp('src'),
  exclude: /node_modules/.use:'happypack/loader? id=tsx',}< span style = "box-sizing: border-box;

// plugins
new HappyPack({
  id: 'tsx'.threadPool: happyThreadPool,
  loaders: [{loader: require.resolve('ts-loader'),
      options: {
        happyPackMode: true.transpileOnly: true.getCustomTransformers: (a)= > ({
          before: [
            tsImportPluginFactory([
              {
                libraryName: 'antd'.libraryDirectory: 'es'.style: true,},]),],}),},}]}),Copy the code

For some reason, my computer was faster before Happypack than it was the first time after happypack, but the cache builds were neck and neck. This is because happypack has certain requirements on the computer’s kernel. If the computer’s kernel is low and you start multi-threading, it will make the CPU full and slow down. Therefore, this solution is not the best choice (my computer sucks anyway).

terser-webpack-plugin

Terser-webpack-plugin is a compression plugin used by Webpack4 to replace Uglifyjs-webpack-plugin. If we simply combine webpack3 and Terser-webpack-plugin, I wonder if I can solve the problem of compression speed.

In Webpack 3, the official plug-in provided is Terser-Webpack-plugin-Legacy (which looks like a compromise). As you can see from the picture below, oh my God, this is amazing, it is a qualitative leap (I can’t believe I tried several times).

The configuration is as follows:

new TerserPlugin({
  parallel: true.cache: true.terserOptions: {
    parse: {
      ecma: 8,},compress: {
      ecma: 5.warnings: false.comparisons: false.inline: 2,}}}),Copy the code

summary

After a while of (unspecified) observation, there are several ways to speed up builds in WebPack 3:

  • Set cache, can effectively reduce the speed of re-build;
  • useterser-webpack-pluginreplaceuglifyjs-webpack-plugin;
  • If you are sure that some modules have no dependencies, you can set themnoParse;
  • usealiasThis can improve the development efficiency.
  • usewebpack-bundle-analyzerEliminate irrelevant dependencies;
  • It can be configured if the module is identifiedresolve.modules, such asresolve.modules = ['node_modules'], can reduce the search scope;
  • loadersYou can usetest/include/excludeTo reduce unnecessary traversal;
  • Try it if your build environment allowshappypack.

Last Egg

If your project uses something like react-loadable to load on demand, please note that react-loadable can help load on demand depending on the route. It uses import() instead of import because import is statically compiled, whereas import(), like require, can be loaded dynamically. However, never use variables in the reference process, as this can result in a compile that passes but takes an obscene amount of time or runs out of memory. – ES6 DYNAMIC IMPORT AND WEBPACK MEMORY LEAKS – Adrian Oprea – Medium

Upgrade Webpack 4

So finally, let’s try this “dark tech” that claims to improve compilation speed by 60% to 98%. Since we are using create-React-app, there are more or less a lot of problems during the upgrade process. Here I will record the problems I encountered during the upgrade process.

Because create-react-app is already eject in the project, the official recommended quick update to React-scripts cannot be used.

The preparatory work

Yarn add -d webpack webpack-cli webpack-dev-server, upgrade WebPackage 4. One of the three sets is required. Don’t panic. There’s only so much preparation. The panic is how to deal with the compatibility issues after the upgrade 😂😂😂.

The obstacles

All things are hard at first, and then harder and harder. Note: Each time the problem is resolved, execute the program directly, i.e. Yarn Start/Build, which will not be described below.

_this.compiler.applyPluginsAsync is not a function

👉🏻 Upgrade fork-ts-checker-webpack-plugin.

Plugin could not be registered at ‘html-webpack-plugin-before-html-processing’. Hook was not found. BREAKING CHANGE: There need to exist a hook at ‘this.hooks’. To create a compatibility layer for this hook, Hook into ‘enclosing _pluginCompat’.

👉🏻 updates html-webpack-plugin@next and react-dev-utils; 👉🏻 Also make the following optimizations for the configuration file dev/prod:

// plugins
[
  new HtmlWebpackPlugin({
    ... // dev and prod keep their original configuration
  }),
  new InterpolateHtmlPlugin(HtmlWebpackPlugin, env.raw)
]
Copy the code

webpack is not a function

👉🏻 make the following optimizations for start.js:

// Adjust to the object structure
const compiler = createCompiler({ webpack, config, appName, urls, useYarn });
Copy the code

When specified, “proxy” in package.json must be a string. Instead, the type of “proxy” was “object”. Either remove “proxy” from package.json, or make it a string.

👉🏻 Install/upgrade HTTP-proxy-Middleware; 👉🏻 delete proxy from package.json and add SRC/setupproxy.js and add it to paths.js; 👉 🏻 modify webpackDevServer. Config. Js

Note:

Why delete proxy in package.json? Since proxy exists as a string in package.json, the proxy field in package.json is read first by default, followed by setupproxy.js.

// paths.js
module.exports = { ... .proxySetup: resolveApp('src/setupProxy.js'),}// webpackDevServer.config.js
before(app, server) {
  if (fs.existsSync(paths.proxySetup)) {
    require(paths.proxySetup)(app); }}// src/setupProxy.js
const proxy = require('http-proxy-middleware');

module.exports = function(app) {
  app.use(proxy('/api', {
    target: 'https://xxx.xx.com'.changeOrigin: true,})); };Copy the code

This. HtmlWebpackPlugin. GetHooks is not a function if this error, you can try the following:

👉🏻 delete node_modules and reinstall them; 👉🏻 reinstall html-webpack-plugin@next; 👉🏻 ensure new InterpolateHtmlPlugin(env.raw) -> new InterpolateHtmlPlugin(HtmlWebpackPlugin, env.raw)

DeprecationWarning: Pass resolveContext instead and use createInnerContext DeprecationWarning: Resolver: The callback argument was splitted into resolveContext and callback DeprecationWarning: Resolver#doResolve: The type arguments (string) is now a hook argument (Hook). Pass a reference to the hook instead.

This is not an error, you can either ignore it or do the following: 👉🏻 update tsconfig-paths-webpack-plugin

Tapable.plugin is deprecated. Use new API on .hooks instead

👉🏻 Upgrade the extract-text-webpack-plugin, but the plugin is no longer recommended in Webpack4. Use the mini-CSs-extract-plugin instead. Note that you can use the mini-CSs-extract-plugin without using the style-loader — Advanced configuration example

The rest of the problem is to encounter any plug-in incompatible directly upgrade can be, for example:

TypeError: Cannot read property ‘ts’ of undefined URIError: Failed to decode param ‘/%PUBLIC_URL%/favicon.ico’ TypeError: Cannot read property ‘ts’ of undefined URIError: Failed to decode param ‘/%PUBLIC_URL%/favicon.ico’

👉🏻 Upgrade TS-loader and file-loader

extracting one single css file

How to use the mini-CSs-extract-plugin to package all CSS files into one CSS file? There are many ways to fully implement all CSS in a single file using the officially recommended method, but Conflicting order between warnings may be required during this process. We can turn off Remove Order Warnings. RIP CommonsChunkPlugin. Md · GitHub

// See more splitChunks
// https://webpack.docschina.org/plugins/split-chunks-plugin/
module.exports = {
  optimization: {
    splitChunks: {
      cacheGroups: {
        styles: { // Entry Entry name
          name: 'styles'.// Extract the name of chunk
          test: /\.css$/.chunks: 'all'./ / initial | | all async, async by default
          enforce: true,},},},},... }Copy the code

Replace dependency packages

After the configuration is complete, replace the following dependencies with those recommended by WebPack4.

  • extract-text-webpack-plugin -> mini-css-extract-plugin;
  • uglifyjs-webpack-plugin -> terser-webpack-plugin.

summary

Up to this webpackage 4 has basically been solved, the rest of the problem, are based on personal needs to deal with. The process of upgrading to Webpack 4 is not so smooth, but it is a big version of Webpack, try it and maybe it will be successful, after all, Webpack 4 has been optimized in many places, some of the dependency packages that have security problems have been solved, and finally, after the upgrade, MY local and my colleagues build time.

My computer

Someone else’s computer