Use Yarn(V1.12 +)’s Plug’n’Play mechanism instead of node_modules. This is an experimental feature for now.

Yarn Plug’n’Play can help you get out of node_modules?


background

Node_modules has long been the object of national ridicule, the developers of other languages see node_modules on Node look and quit, with a word to describe it is’ heavy! ‘.

If you are not familiar with the Node module lookup mechanism, please click on the source code of require()

The size and number of pieces of a simple front end project (create-React-app) :

And macOS /Library directory size files:

A single line of Hello World requires more than 130MB of dependency modules, and the number of files is 32,313. By comparison, macOS /Library takes up 9.02GB of space and has twice as many files (67,890). Node_modules has the following characteristics:

  • The directory tree structure is complex
  • The number of files is large and small
  • Many dependencies, a simple project will install several tons of dependencies

So node_modules is a nightmare for mechanical hard drives, and I remember one time a colleague took an afternoon to delete node_modules. For front-end developers, we have N projects that require NPM install 😹.

In addition, The Node module mechanism has the following disadvantages:

  • Node itself has no concept of a module; it finds and loads at run time. This disadvantage is similar to ‘Dynamic vs. static languages’. You might run fine in your development environment, but you might not run online because a module was not added to package.json

  • The Node module’s lookup strategy is wasteful. This disadvantage can be optimized in most front-end projects, such as Webpack, which restricts the lookup to node_modules in the project root directory, but nested dependencies still require more than two lookups

  • Node_modules does not efficiently handle duplicate packages. Two packages with the same name but different versions cannot coexist in the same directory. This results in nested node_modules, and these project ‘dependent dependencies’ cannot be shared with projects or other dependencies:

    #① Assume that the project depends on modules A, B and C, and the dependency tree is:
#  +- a
#    +- react@15
#  +- b
#    +- react@16
#  +- c
#    +- react@16
#During YARN installation, the power of dependency promotion (as follows) is based on the number of times the project is dependent.
#The node_modules structure is as follows:. └ ─ ─ node_modules ├ ─ ─ a │ ├ ─ ─ index. The js │ ├ ─ ─ node_modules │ │ └ ─ ─ the react # @ 15 │ └ ─ ─ package. The json ├ ─ ─ b │ ├ ─ ─ index. The js │ ├─ ├─ download.txt ├─ ├─ download.txt ├─ download.txt
#(2) Now assume that the root project depends on react@15, and the dependencies for the project must be placed in the node_modules root directory.
#Because a directory cannot have the same name, react@16 has no promotion opportunity.
#The node_moduels structure after installation is. └ ─ ─ node_modules ├ ─ ─ a │ ├ ─ ─ index. The js │ └ ─ ─ package. The json react @ # 15 ascension ├ ─ ─ b │ ├ ─ ─ index. The js │ ├ ─ ─ node_modules │ │ └ ─ ─ The react # @ 16 │ └ ─ ─ package. The json ├ ─ ─ c │ ├ ─ ─ index. The js │ ├ ─ ─ node_modules │ │ └ ─ ─ the react # @ 16 │ └ ─ ─ package. The json └ ─ ─ the react # @ 15#As can be seen from the above results, there is duplication at react@16
Copy the code

For this, Yarn integrates Plug’n’Play(PNP for short) to solve node_modules’ hell ‘.


The basic principle of

As usual, Yarn generates a node_modules directory, and Node looks in the node_modules directory according to its module lookup rules. Node doesn’t actually know what the module is, so it looks in node_modules, and if it doesn’t find it, it looks in node_modules, and so on. This is very inefficient.

But Yarn, as a package manager, knows the dependency tree for your project. Can Yarn tell Node? Tell it to go directly to a directory to load modules. This improves the efficiency of Node module lookups and reduces copying of node_modules files. This is thePlug'n'PlayThe basic principle of.

In PNP mode, Yarn does not create node_modules. Instead, it creates a.png.js file, which is a Node program. This file contains the dependency tree information for the project, the module lookup algorithm, It also contains the patch code for the Module finder (overriding the module._load method in the Node environment).


Using the PNP mechanism has the following advantages:

  • Get rid of the node_modules.
    • Time: Compared with running in a hot cache environmentyarn installSave 70% of your time
    • Spatial: In PNP mode, all NPM modules are stored in the global cache directory, which relies on tree flattening to avoid copy and duplication
  • Improve module loading efficiency. Node invoks a large number of stat and readdir system calls to locate modules. PNP obtains or locates modules through Yarn
  • It is no longer limited that different versions of the node_modules module with the same name cannot reside in the same directory

The installation of Yarn on a Mac is fast and takes only a few seconds under hot cache. The reason is the copy-on-write mechanism of SSD + APFS. This allows a copy of the file to take up no space, equivalent to creating a link. So copy and delete is very fast. But node_modules’ complex directory structure and large number of files still require a lot of system calls, which can slow down the installation process. 💡 If PNP is tedious or unreliable, use SSDS with copy-on-write file systems.


Risks of using PNP:

Various tools in the front-end community today rely on the node_modules module lookup mechanism. For example,

  • Node
  • Electron, Electron – builder and so on
  • Webpack
  • Typescript: Locates type declaration files
  • Babel: Preset plug-in and Preset
  • Eslint: Locates plug-ins and preset, rules
  • Jest
  • Editor, such as VsCode
  • . 😿

PNP is a very new thing, coming out last September (2018). Integrating these tools with the PNP is a challenge, and these tools and PNP are constantly iterating, the PNP is unstable and can change in the future, which also introduces some maintenance burdens.

In addition to the module lookup mechanism, there are some tools that do other things directly in node_modules, such as caching and storing temporary certificates. The cache – loader, for example, webpack – dev – server


Open the PNP

If it were a pure Node project, the migration process would be relatively simple. First, enable PNP install mode in package.json:

{
  "installConfig": {
    "pnp": true}}Copy the code

Next install dependencies:

yarn add express
Copy the code

After installation, a.pnp.js file will appear in the project root directory. Next write the code:

// index.js
const express = require('express');
const app = express();
const port = 3000;

app.get('/', (req, res) => res.send('Hello World! '));

app.listen(port, () => console.log(`Example app listening on port ${port}! `));
Copy the code

Error: Cannot find module ‘express’ Error: Cannot find module ‘express’ Error: Cannot find module ‘express’ This is because there is no patch Node module finder yet. You can run the following command:

yarn node

#or

node --require="./.pnp.js" index.js
Copy the code

The.pnp.js file should not be submitted to the version library, which contains the hard-coded cache directory. Data reconstruction is performed on Yarn V2


How does it integrate into existing projects?

PNP integration is nothing more than a re-implementation of the module lookup mechanism of existing tools. With the development of front-end engineering, a front-end project can integrate many tools, and if these tools do not fit, it can be said that PNP can not move forward. However, this is beyond the control of the PNP and requires the cooperation of the tool developers.


Several projects in the community have integrated PNP:

  • create-react-app
  • gastby


Node

Js file, which will patch the Module object of Node and re-implement the Module lookup mechanism

Webpack

The module finder used by Webpack is enhanced- Resolve, which can be extended to support PNP through the PNP-webpack-plugin plug-in.

const PnpWebpackPlugin = require(`pnp-webpack-plugin`);

module.exports = {
  resolve: {
    // Extend module finder
    plugins: [PnpWebpackPlugin],
  },
  resolveLoader: {
    // Extend the Loader module finder.
    plugins: [PnpWebpackPlugin.moduleLoader(module)],}};Copy the code

jest

Jest supports configuration of finders via resolver:

module.exports = {
  resolver: require.resolve(`jest-pnp-resolver`),};Copy the code

Typescript

Typescript also uses its own module finder, and the TS team has not allowed third-party tools to extend the finder for performance reasons. Which means it’s not working yet.

In this issue, someone suggested using “moduleResolution”: “yarnpNP” or using a ts-loader-like resolveModuleName to support PNP module lookup.

The RESPONSE from the TS team is that PNP (or NPM tink) is still in its early stages and there may be changes in the future, such as.pNP. Js files, which are clearly not suitable for pit entry that early. In addition to optimizing and controlling compiler performance, TS has no plans to expose the interface to third party code execution during compilation.

So there is no babel-like plugin mechanism for Typescript today. Ts-loader is now PNP compliant unless it implements a ‘TS compiler host’. For example, TS-Loader extends the plugin mechanism and module lookup mechanism to support plug-ins such as TS-import-plugin.

const PnpWebpackPlugin = require(`pnp-webpack-plugin`);

module.exports = {
  module: {
    rules: [{test: /\.ts$/.loader: require.resolve('ts-loader'),
        options: PnpWebpackPlugin.tsLoaderOptions(),
      },
    ],
  },
};
Copy the code


In summary, Typescript is not supported and there are no plans for development in the near future, so VsCode is out of the question. Fork-ts-checker-webpack-plugin is not up to speed. Typescript is clearly the first obstacle to PNP


Other tools

  • rollup-plugin-pnp-resolve
  • resolve: babel, gulp
  • Eslint: not fully supported until V6.
  • flow
  • Create-react-app supports PNP, but Typescript mode does not
  • Electron: There’s no news yet. For a electron application, the dependency is self-contained, so PNP may not be suitable for this scenario


conclusion

To sum up, PNP is a good solution to solve the problem of space and time efficiency of Node module mechanism. However, at this stage, it is still immature, has a lot of holes to tread, and has a lot of problems integrating with various tools in the community. Therefore, it is not recommended for use in production environments.

So for the average developer, if you want to improve the speed of NPM installation, you still need to use SSD+ copy-on-write! 😂

Below is the integration of various project (✅ (support) | 🚧 (plan or imperfect) | ❌ (support)) :

project
Webpack ✅
rollup ✅
browserify ✅
gulp ✅
jest ✅
Node ✅
Typescript/VScode IntelliSense ❌
eslint 🚧
flow 🚧
create-react-app 🚧
ts-loader ✅
fork-ts-checker-webpack-plugin 🚧


reference

  • Plug’n’Play Whitepaper PNP paper
  • How to get rid of node_modules with Yarn Plug ‘n’ Play
  • Yarn Official Documents
  • Pnp-sample-app Official PNP example
  • Yarn’s Future – v2 and beyond
  • Hacker News Discussion

Related issues:

  • Yarn Plug ‘N Play should generate a static manifest file, not .pnp.js
  • Typescript: Add new moduleResolution option: yarn-pnp
  • fork-ts-checker-webpack-plugin: Custom resolveModuleName

Other options

  • npm tink: a dependency unwinder for javascript
  • pnpm Fast, disk space efficient package manager
  • Yarn Workspaces Multiple projects share dependencies