Recently I learned the use of Webpack4, and tried to upgrade and optimize the project Webpack, and record some practice process of this upgrade.
Painful development experience
A long wait
The project introduced Webpack as a packaging tool in 2016, and used VUe-CLI to build build-related code. Since then, there has been no major update. With the iteration of the project up to now, the amount of code is no longer a few thousand lines, and the local startup development environment has increased from ten seconds to more than 200s now. Every run dev or rebuild is accompanied by a long time of waiting with dull eyes
The cure of chaos
Over a span of more than two years, the project’s build code was rewritten by countless people, riddled with redundancy, clutter, and logic that only God could understand. My colleague had to start all over again when doing the theme function. He opened a small project built with WebPack4 and hid it in a corner of CSS folder, waiting for the project to be rediscovered with the integration of WebPack4 in the future. The co-existence of Webpack2 and Webpack4 forces everyone on the team to have two terminals, one for projects and the other for styles
In a nutshell: Build is too slow, build code is messy, and your development is inefficient
How to solve it? Less nonsense, quick upgrade webpack4
Improve construction efficiency
To improve packaging efficiency, there are two simple ways:
- Improve packing speed per unit time
- Clean up unnecessary packing
Multi-pronged: happyPack
Just like the name of this plug-in, it really makes people happy when it is used up. The packaging speed is not improved by a little bit. The principle is to open multiple node child processes and use various Loaders in parallel to process the source files to be packed, in other words, to improve the packaging speed in unit time
To quote happyPack:
HappyPack sits between webpack and your primary source files (like JS sources) where the bulk of loader transformations happen. Every time webpack resolves a module, HappyPack will take it and all its dependencies and distributes those files to multiple worker “threads”.
Those threads are actually simple node processes that invoke your transformer. When the compiled version is retrieved, HappyPack serves it to its loader and eventually your chunk.
The performance of my computer is better than that of the company’s computer. The company’s computer runs more than 200s according to webpack2 configuration, and the first build of the computer even lasts 5 minutes after restarting
- It takes time for the project to start locally using Webpack2
- Local startup using WebPack4 takes time
- Webpack4 + happyPack(babel-loader) Local startup time
- Webpack4 + happyPack(babel-loader + eslint-loader) Local startup time
From the experimental results, it can be seen that the compilation speed is significantly improved after using happyPack, and the time is shortened by nearly 55%. The optimization effect is significant
HappyPack supports many common loaders (happyPack compatibility list). You can use multiple happyPack instances in webPack configuration and separate them with different loaders. For example, eslint-loader and babel-loader can be used for.js files, and happyPack can be used to create threadpools so that these happyPack instances share a ThreadPool to improve resource utilization.
HappyPack configuration and use, the official documentation is very clear, Baidu also have a lot of tutorial articles for reference, here is no more details
Strip third-party libraries with DLLS
The project will inevitably use some third-party libraries, unless the version is upgraded, the code of these libraries will not change significantly, which means that these libraries do not need to participate in the build and rebuild process every time. If you can extract this code and build it ahead of time, you can skip third-party libraries when building your project, which increases efficiency even further — in other words, clean up unnecessary packaging.
dllPlugin+dllReferencePlugin
DLL is a way for Microsoft to realize the concept of a shared function library (baidu Baike said), itself cannot be executed, for other programs to call. DllPlugin +dllReferencePlugin webPack provides a built-in plugin for DLL referencePlugin, which can easily do this, just need to do the following things:
- Separate webpack configuration webpack.dll. Conf and use dllPlugin to define the DLL files to be packaged
- Run webpack.dll. Conf to generate
xxx.dll.jsAnd the corresponding manifest filemanifest-xxx.jsonAnd introduce each in the project template index.htmlxxx.dll.js- In the webpack configuration of the project, the dllReferencePlugin and
manifest-xxx.jsonTell WebPack which packages have been built in advance and do not need to be re-built
Webpack4 + happyPack(XXX-loader) + DLL Local startup time
From 71s to 45s, this is another significant improvement, further reducing the time by nearly 40% and increasing the efficiency by 71% compared to the original Webpack2 compilation time, even with company’s book, the efficiency can increase by at least 50%. My first thought when I saw this was: Oh my God!! Well, this lament contains not only surprise at the results, but also the outrageous inefficiencies of previous builds.
Optimize performance a little bit
In addition to reducing code packaging time, using DLLS also helps optimize web page performance. It is common to extract third-party libraries into a block of code named vendors. This has the benefit of preventing common dependencies from being repackaged, changing less frequently, having relatively stable hashes in production environments, and taking full advantage of the browser’s caching strategy to reduce requests for vendors files. However, the size of a single JS file may be too large, and there will be obvious blocking when re-requesting resources. With DLLS, the vendors’ size is reduced, and the network overhead of requesting the vendors files is reduced, because a large number of third-party libraries are extracted ahead of time
Do not use DLLS, vendors by volume
Some of you might be confused, though, that the vendors’ size is reduced, but the reduced volume is simply abstracted in a different location, and the requests are still made, so the total cost is not reduced. In fact, THE DLL itself can be divided by configuring multiple entries to further optimize the performance of the request DLL file through the concurrent request of the browser.
{
entry: {
vue: ['vue'.'vuex'.'vue-router'].// Vue family bucket DLL: vue.dll
ec: ['echarts'.'echarts-wordcloud'].// Echarts related DLL: ec.dll.js
commons: [
// Other third-party libraries: commons.dll.js]}}Copy the codeOf course, even in a development environment, the 3.88m Vendors package is still large, and this is simply to show the effect of stripping third-party libraries through DLLS, not to discuss code splitting and related optimizations in detail here.
A few small pit
In webpack.dll. Conf, the library name exposed by output must be the same as the name of the DllPlugin, as noted in the official documentation
{
output: {
filename: '[name].dll.js'.path: path.resolve(__dirname, '.. '.'lib/dll'),
library: '[name]_[hash]'
// the name of the global variable exposed in vendor.dll.js, which is used as the manifest name in DllPlugin,
// This must be consistent with the name: '[name]_[hash]' in webpack.DllPlugin.
},
plugins: [
new webpack.DllPlugin({
path: utils.resolve('lib/dll/manifest-[name].json'),
name: "[name]_[hash]" // Be consistent with library}})]Copy the codeIn addition, vue uses the Runtime package by default. In a development environment, if you need vUE to compile a template, use it like this:
new Vue({
template: '<div>{{ hi }}</div>'
})
Copy the codeYou must import the full vue package, which is written in the Webpack alias configuration (see the Vue documentation) :
module.exports = {
// ...
resolve: {
alias: {
'vue$': 'vue/dist/vue.esm.js' // to use webpack 1, use 'vue/dist/vue.com.js'}}}Copy the codeThis means that references to vUE in webpack.dll.conf should be consistent with those in the project, otherwise the packaging of vUE will not be skipped when building the project
As for the specific configuration and use of dllPlugin and dllReferencePlugin, the official documents give examples, and there are a lot of tutorial articles on Baidu for reference, which will not be introduced in detail here
DLL acceleration with manifest exploration
If you just want to know how to build more efficiently, can you skip this part
After the author completed the configuration, it did not reach the level of 45s immediately. The effect was not very good when it was started for the first time. There was no obvious improvement in efficiency. The addition of DLL does not significantly shorten the packaging time, indicating that there are still third-party libraries into the packaging process. There is a concept of manifest in Webpack. The author only knows that it is related to the mapping and loading of modules, but does not know the specific content. Therefore, I just guess that it is related to this at that time, and continue to investigate along this road. Sure enough, I omitted several manifest.json files when using the dllReferencePlugin. This was purely due to my carelessness in not looking at the documentation (so it’s important to look at the documentation), but I took the opportunity to briefly explore what the manifest is and why using DLLS can speed things up.
What comes out of a packaged DLL
Take a look at what files are generated after running webpack.dll.conf
In the case of multiple entries, each entry file generates a DLL file and a JSON file. For example, vue. DLL. Js and manifest-vue
Vue. DLL. Js:
var vue_01cf92ee1ec06f1bc497 =
(function(modules) { // webpackBootstrap
var installedModules = {};
function __webpack_require__(moduleId) {
// __webpack_require__ source code
}
return __webpack_require__(__webpack_require__.s = 0) ({})"./node_modules/vue/dist/vue.esm.js":
(function (module, __webpack_exports__, __webpack_require__)) {
"use strict";
eval("xxx"); // webpack require vue
}),
// Other modules...
// ...
0: (function (module, exports, __webpack_require__) {
eval("module.exports = __webpack_require__; \n\n//# sourceURL=webpack://%5Bname%5D_%5Bhash%5D/dll_vue?"); })})Copy the codeThe immediate function above seems a bit cumbersome, so let’s write it differently and preserve some of the details
var requireModules = function(modules) { // webpackBootstrap
var installedModules = {};
function __webpack_require__(moduleId) {
if (installedModules[moduleId]) { // Check whether the module is loaded
return installedModules[moduleId].exports;
}
var module = installedModules[moduleId] = { // Create a module
i: moduleId,
l: false.exports: {}};// Load the module
modules[moduleId].call(module.exports, module.module.exports, __webpack_require__);
// The flag module has been loaded
module.l = true;
// Returns the content exported by the module
return module.exports;
}
// Define attributes and methods for __webpack_require__
// __webpack_require__.xxx = xxx
// ...
return __webpack_require__(__webpack_require__.s = 0); // Execute modules[0] to expose the loader for the vue.dll.js internal module
}
var modules = {
"./node_modules/vue/dist/vue.esm.js": / / module id
function (module, __webpack_exports__, __webpack_require__)) { // Module loading function
eval("xxx"); // Webpack loads vue
},
// Other modules
// ...
// Expose the loader
0: function (module, exports, __webpack_require__) { // The entire vue.dll. Js module
// Expose the internal module loader of vue.dll.js for external calls and loading of VUe-related modules
eval("module.exports = __webpack_require__; \n\n//# sourceURL=webpack://%5Bname%5D_%5Bhash%5D/dll_vue?"); }}var vue_01cf92ee1ec06f1bc497 = requireModules(modules);
Copy the codeThe DLL file does several things:
- A mapping table of submodule loading functions, called literal objects, is defined
modules - Defines an internal loader
__webpack_require__And module cacheinstalledModules - through
requireModuleThe function exposes the internal loader to the global variablevue_01cf92ee1ec06f1bc497To be called when an external module is loaded
When vue.dll.js is introduced in index. HTML, the loader of DLL’s internal module is exposed under global, and vue_01CF92EE1EC06f1bc497 can be directly called when webpack loads the module. The final result is equivalent to:
var vue_01cf92ee1ec06f1bc497 = __webpack_require__; // The loader inside the closure
Copy the codeSo vue.dll.js itself cannot execute the code of internal modules, but only provides external calls, which is exactly the definition of DLL files
Having a DLL alone is not enough. The webpack of the project needs to know that the DLL exposes a loader called vue_01CF92EE1EC06f1BC497 and what modules this loader contains. The manifest file contains this information.
The manifest – vue. Json:
{
"name": "vue_01cf92ee1ec06f1bc497"."content": {
"./node_modules/vue/dist/vue.esm.js": {
"id": "./node_modules/vue/dist/vue.esm.js"."buildMeta": {
"exportsType": "namespace"."providedExports": ["default"]}}}}Copy the codeThe manifest contains details of the module’s origin as the id to retrieve the module, and also specifies which __webpack_require__ loader is used to load these modules. When the program is run, __webpack_require__ can load the corresponding module using the module ID. Refer to webpack’s official explanation:
As the compiler enters, resolves, and maps out your application, it keeps detailed notes on all your modules. This collection of data is called the “Manifest” and it’s what the runtime will use to resolve and load modules once they’ve been bundled and shipped to the browser. No matter which module syntax you have chosen, those import or require statements have now become webpack_require methods that point to module identifiers. Using the data in the manifest, the runtime will be able to find out where to retrieve the modules behind the identifiers.
Tell the project where the DLL is and what’s in it
With the MANIFEST, how do I tell a project that I have a DLL and don’t need to repackage it? The DLLReferencePlugin passes the manifest file to the project’s Webpack to tell it which modules can be referenced directly and the packaging process can be skipped. Dllreferenceplugin.js reads the manifest file and mounts the DLL exposed loader as an external dependency to the module factory of Webpack.
Read the manifest:
compiler.hooks.beforeCompile.tapAsync( // Webpack creates a compilation hook to read the module manifest in the DLL.
"DllReferencePlugin",
(params, callback) => {
if ("manifest" in this.options) {
const manifest = this.options.manifest;
if (typeof manifest === "string") {
params.compilationDependencies.add(manifest);
compiler.inputFileSystem.readFile(manifest, (err, result) => { // Read the manifest file
params["dll reference " + manifest] = parseJson(result.toString("utf-8"));
return callback();
});
return; }}returncallback(); });Copy the codeCreate external dependencies:
// WebPack creates compilation hooks that tell WebPack that I have a DLL and what modules are in the DLL
compiler.hooks.compile.tap("DllReferencePlugin", params => {
// Read the configuration in the manifest
let manifest = this.options.manifest;
if (typeof manifest === 'string') {
manifest = params["dll reference " + manifest];
}
let name = this.options.name || manifest.name;
let sourceType = this.options.sourceType || manifest.sourceType;
let content = this.options.content || manifest.content;
// Create external dependencies
const externals = {};
const source = "dll-reference " + name; // Tells Webpack to expose global variables, prefixed with DLL-reference to indicate that this is a DLL resource
externals[source] = name; // Resource name: vue_01CF92EE1ec06f1bc497
const normalModuleFactory = params.normalModuleFactory;
// Introduce an external module factory plug-in to mount DLLS in an externally dependent manner
new ExternalModuleFactoryPlugin(sourceType || "var", externals).apply(
normalModuleFactory
);
// Introduce the proxy module factory plug-in to create proxies for each module in the DLL
new DelegatedModuleFactoryPlugin({
source: source,
type: this.options.type,
scope: this.options.scope,
context: this.options.context || compiler.options.context,
content,
extensions: this.options.extensions
}).apply(normalModuleFactory);
});
Copy the codeIn webpack.dll. Conf, the name attribute of the DllPlugin must match the library attribute of the output
The webpack module creation process is done in normalModuleFactory, which contains built-in hook functions for adding processing logic when a module is parsed and created. Here introduces two key plug-in ExternalModuleFactoryPlugin and DelegatedModuleFactoryPlugin, what did they in normalModuleFactory hooks?
Create DLL modules to speed up packaging
Before the compilation of the project webpack actually starts, all the DLL information is already available and the normalModuleFactory of WebPack will take care of the rest. ExternalModuleFactoryPlugin and DelegatedModuleFactoryPlugin these two plug-ins in factory hooks (build a module factory), the module hooks (create a module) added their own callback function, Let Webpack first look for external dependencies when parsing modules. If it finds one, create a module proxy object directly. Otherwise, create a normal module object and use loader to load resources during build.
Combined with DllReferencePlugin, the overall process is as follows:
Entering normalModuleFactory process after the first access to create an external module factory in factory hook function, ExternalModuleFactoryPlugin plug-in defines the factory in factory hooks functions:
// ExternalModuleFactoryPlugin.js
normalModuleFactory.hooks.factory.tap( / / factory hooks
"ExternalModuleFactoryPlugin",
factory => (data, callback) = > { // Returns a factory function that creates an external module
const context = data.context;
const dependency = data.dependencies[0];
const handleExternal = (value, type, callback) = > {
// Input parameter collation
// ...
callback(
null.new ExternalModule(value, type || globalType, dependency.request) // Create an external module for the DLL
);
return true;
};
const handleExternals = (externals, callback) = > {
// Process externals of different types, such as Array and Object
// ...
if (
typeof externals === "object" &&
Object.prototype.hasOwnProperty.call(externals, dependency.request)
) {
return handleExternal(externals[dependency.request], callback); // Create an external module object if the requested resource is an external resource
}
callback();
};
handleExternals(this.externals, (err, module) = > {if (err) return callback(err);
if (!module) return handleExternal(false, callback);
return callback(null.module); // The newly created external module is passed back to the module-building process in WebPack via the incoming callback}); });Copy the codeThe Factory hook returns the factory function, which is immediately called by normalModuleFactory, and vue_01CF92EE1EC06f1BC497 is mounted as an external module to normalModuleFactory
Once the factory is set up, the normalModuleFactory enters the module resolution process (resolver). After the resolution, a NormalModule object is created for the resolution result by default and passed as a parameter to the Module hook function. In the module hooks, DelegatedModuleFactoryPlugin will determine whether the incoming NormalModule exists in a DLL, if there is to create a proxy object and returns, otherwise NormalModule returned directly
normalModuleFactory.hooks.module.tap(
"DelegatedModuleFactoryPlugin".module= > {if (module.libIdent) {
const request = module.libIdent(this.options);
if (request && request in this.options.content) { // option.content is the content in the manifest
const resolved = this.options.content[request];
return new DelegatedModule( // Create an agent for the module in the DLL
this.options.source, // vue_01cf92ee1ec06f1bc497
resolved,
this.options.type,
request,
module); }}return module; });Copy the codeLooking at the Definition of the DelegatedModule class, you can see that needRebuild directly returns false, build directly marks the module as built and adds dependencies, no loader is executed, so modules in the DLL are skipped during code build and do not participate in the packaging process
class DelegatedModule extends Module {
constructor(request, type, userRequest) {}
needRebuild(fileTimestamps, contextTimestamps) {
return false; // Skip the rebuild process
}
build(options, compilation, resolver, fs, callback) {
this.built = true; // Mark the module as "built"
this.buildMeta = Object.assign({}, this.delegateData.buildMeta);
this.buildInfo = {};
this.delegatedSourceDependency = new DelegatedSourceDependency(
this.sourceRequest
);
this.addDependency(this.delegatedSourceDependency); // Add proxy dependencies
this.addDependency(
new DelegatedExportsDependency(this.this.delegateData.exports || true)); callback(); }// Other methods
// ...
}
Copy the codeIn contrast, regular modules participate in the packaging and rebuild process
class NormalModule extends Module {
constructor(request, type, userRequest) {}
needRebuild(fileTimestamps, contextTimestamps) {
// Rebuild check code
// ...
}
build(options, compilation, resolver, fs, callback) {
return this.doBuild();
}
doBuild(options, compilation, resolver, fs, callback) {
runLoaders(); // Run loaders to build the module
}
// Other methods
// ...
}
Copy the codeAt this point, the Manifest has done its job, and the DLL waits quietly for the Runtime to call
conclusion
Through the upgrade of Webpack, the project Webpack4 was unified, various headaches of partners were solved, and positive feedback of partners was obtained. The construction process was much cleaner than before, and the construction efficiency was also greatly improved. In the process of upgrading, also take a look at the DLL working process, harvest a lot of knowledge. In this summary came out to record the process of unification.