preface

The current team of the author uses Monorepo to manage all business projects. As the number of projects increases, stability and development experience are challenged, and many problems are exposed, it is obvious that the existing Monorepo architecture is not enough to support the increasingly large business projects.

The existing Monorepo is implemented based on yarn Workspace and can be reused across projects through various packages in the LINK repository. Package Manager also naturally chooses YARN. Although it relies on Lerna, it is rarely used due to the rarity of packet sending scenarios.

It can be summarized as the following three points:

  • Use the package in the Yarn Workspace link repository
  • Use YARN as package Manager to manage dependencies in your project
  • Lerna builds dependent packages according to dependencies before the application is built

Existing problems

Command inconsistency

There are three kinds of commands

  1. yarn
  2. yarn workspace
  3. lerna

Beginners are prone to misunderstanding, and some commands overlap.

Slow release

If we need to publish App1, yes

  1. Full install dependencies, app1, APP2, app3, and Package1 through Package6 dependencies are installed.
  2. Package is built entirely, not just package1 and package2, which app1 depends on.

Phantom dependencies

A library that uses a Package that is not part of its dependencies is called Phantom dependencies, and this problem is magnified in the existing Monorepo architecture (dependency promotion).

Because the version correctness of phantom dependence cannot be guaranteed, it brings uncontrollable risks to the program operation. App depends on lib-a, lib-A depends on Lib-x, because of the dependency promotion, we can directly reference lib-x in app, which is not reliable, whether we can reference lib-x, and what version of lib-X we reference is entirely up to the developer of lib-A.

NPM doppelgnger

Multiple copies of the same version of Package may be installed and packaged.

Assume the following dependencies exist

There are two possible outcomes of the final dependent installation:

  1. Lib-x @^1 * 1, lib-x@^2 * 2
  2. Lib-x @^2 * 1, lib-x@^1 * 2

Eventually three copies of Lib-X will be installed locally, and three instances will exist when packaging, which can cause problems if lib-X requires singletons.

Yarn duplicate

Yarn Duplicate and solution

Assume the following dependencies exist

When (p) NPM is installed on the same module, it checks whether the installed module version conforms to the version range of the new module. If yes, it will skip. If no, it will install the module under the node_modules of the current module. That is, lib-a will reuse [email protected], which the app depends on.

However, using Yarn V1 as the package manager, lib-A installs a separate copy of [email protected].

  • difference between npm and yarn behavior with nested dependencies #3951
  • Yarn installing multiple versions of the same package
  • yarn-deduplicate-Cleans up yarn.lock by removing duplicates.
  • Yarn v2 supports package deduplication natively

PeerDependencies risk

The Yarn dependency promotion may cause a BUG in the peerDependencies scenario.

  1. App1 rely on [email protected]
  2. App2 rely on [email protected]
  3. [email protected] has [email protected] as peerDependency, so app2 should also have [email protected] installed

If app2 forgot to install [email protected], the structure is as follows

--apps
    --app1
    --app2
--node_modules
    [email protected]
    [email protected]
Copy the code

[email protected] is misquoted as [email protected].

Package reference specification missing

There are currently three types of references in the project:

  1. Source code references: use the package name to reference. The host project’s build script needs to be configured to incorporate this Package into the build process. Similar to publishing a TypeScript source package directly, projects that reference the package require some adaptation.
  2. Source code references: Use file path references. You can understand “source files hosted outside of their own SRC” as part of the host project’s source code rather than the Package. The host needs to provide all these dependencies, which can be reused across projects on the premise of improving Yarn dependencies, but there are high risks.
  3. Product references. The package is complete, and the product is referenced directly by the package name.

Package reference version uncertainty

Assuming a Package1 in Monorepo is published to the NPM repository, how should App1 in Monorepo write a version number in package.json that references package1?

package1/packag.json

{
  "name": "package1"."version": "1.0.0"
}
Copy the code

app1/package.json

{
  "name": "app1"."dependencies": {
    "package-1": "?" // What is the version number here? ` * ` or ` 1.0.0 `}}Copy the code

When dealing with cross-references of items in Monorepo, Yarn makes the following judgments:

  1. Check whether package1 matching the required version of APP1 exists in Monorepo.
  2. If yes, perform the link operation and app1 directly uses the local package1.
  3. If no, pull package1 matching the version from the remote NPM repository for APP1 to use.

Note that: * Cannot match prerelease version ๐Ÿ‘‰ Workspace package with prerelease version and wildcard DEP version #6719.

Assume the following scenario:

  1. Package1 was released earlier1.0.0Version, at this time the remote warehouse and local Monorepo code consistent;
  2. The product student raised a demand that only serves Monorepo internal application;
  3. Package1 in1.0.0Iteration under version, no need to change version number release;
  4. Yarn Determines that package1 in Monorepo meets the requirements of APP1 (* ๆˆ– 1.0.0);
  5. App1 successfully uses the latest features of Package1.

Until, at some point, the requirement feature needs to be made available to external business parties.

  1. Pacakge1 changes the version to1.0.0 - beta. 0And release;
  2. Yarn Determines that the package1 version in Monorepo does not meet the requirements of APP1.
  3. Pull it from the far end[email protected]For app1;
  4. The distal[email protected]Local that has fallen behind app1’s previous use[email protected]Too much;
  5. Prepare accident notification and review.

This uncertainty leads to a common question when referring to such packages: am I referring to a local version or a remote version? Why is it sometimes a local version and sometimes a remote version? I want to use the latest package1 content and always need to keep up with the package1 version number, so why do I use Monorepo?

Yarn. The lock conflict

(p) NPM supports automatic resolution of lockfile conflicts. Yarn needs to be handled manually. In large Monorepo scenarios, almost every branch merge will encounter yarn.lock conflicts.

  • No conflict resolution brainlessyarn.yarn.lockAll versions will be updated topackage.jsonLatest, too risky to loselockfileThe meaning of;
  • Manual conflict resolution tends to occurGit conflict with binary files, can only be usedmasterSubmit again and againyarn, the process is tedious.

Automatically resolve conflicts in lockfile ยท Issue #2036 ยท PNPM/PNPM

It can be found that the existing Monorepo management mode has too many defects. With the continuous increase of projects in Monorepo, the construction speed will be slower and slower, and the robustness of the program cannot be guaranteed. Developer self-awareness alone is not reliable, we need a solution.

Recommended reading: node_modules dilemma

The solution

pnpm

Fast, disk space efficient package packageManager

Before npm@3, the structure of node_modules was clean and predictable, because each dependency in node_modules had its own node_modules folder, all of which were specified in package.json.

โ”œโ”€ trash โ”œโ”€ โ”œโ”€ trash โ”œโ”€ trash โ”œโ”€ trash โ”œโ”€ trash โ”œโ”€ trash โ”œโ”€ trash โ”œโ”€ trash โ”œโ”€ trash โ”œโ”€ trash โ”œโ”€ trash โ”œโ”€ trash โ”œโ”€ trash โ”œโ”€ trash โ”œโ”€ trash โ”œโ”€ trash โ”œโ”€ trash โ”œโ”€ trash โ”œโ”€ trash โ”œโ”€ trash โ”œโ”€ trash โ”œโ”€ trash โ”œโ”€ trash โ”œโ”€ trash โ”œโ”€ trash โ”œโ”€ trashCopy the code

But this creates two serious problems:

  1. Too deep a dependency level can cause problems under Windows;
  2. The same Package is copied many times when it is a dependency of several other different packages.

To solve these two problems, npm@3 rethinks the node_modules structure and introduces a tiling scheme. This leads to the following familiar structure.

Node_modules โ”œ โ”€ foo | โ”œ โ”€ index. Js | โ”” โ”€ package. The json โ”” โ”€ bar โ”œ โ”€ index. The js โ”” โ”€ package. The jsonCopy the code

Unlike npm@3, PNPM solves the problem encountered by npm@2 in a different way than tiling node_modules.

In the node_modules folder created by PNPM, all packages are grouped with their own dependencies (isolated), but the dependency hierarchy is not too deep (soft links to real addresses outside).

- > - a symlink (or junction on Windows) node_modules โ”œ โ”€ foo - >. Registry.npmjs.org/foo/1.0.0/node_modules/foo โ”” โ”€ Registry.npmjs.org โ”œ โ”€ foo / 1.0.0 / node_modules | โ”œ โ”€ bar - >.. /.. / bar / 2.0.0 / node_modules/bar | โ”” โ”€ foo | โ”œ โ”€ index. The js | โ”” โ”€ package. The json โ”” โ”€ bar / 2.0.0 / node_modules โ”” โ”€ bar โ”œ โ”€ index. The js โ”” โ”€ package.jsonCopy the code
  1. Resolve Phantom dependencies based on the non-flattened node_modules directory structure. Package only touches its own dependencies.
  2. Avoid repeated packaging (same version) by reusing the same version of Package by soft chain, solve NPM doppelgnger (incidentally solve disk occupancy).

As you can see, many of the package manager-related problems are solved.

  • Why should we use pnpm?
  • Tiling node_modules is not the only way

Rush

a scalable monorepo manager for the web

  1. Command unification.

Rush (x) XXX a shuttle to reduce the cost of beginners. Meanwhile, Rush commands except Rush add and rushx XXX need to be run under the specified project, other commands are global commands and can be executed in any directory within the project, avoiding the problem of frequently switching project paths on the terminal.

  1. Strong dependency analysis skills.

Many of the commands in Rush support parsing dependencies, such as the -t(to) argument:

$ rush install -t @monorepo/app1
Copy the code

This command installs only app1’s dependencies and the dependencies of the package that APP1 depends on, that is, on demand.

$ rush build -t @monorepo/app1
Copy the code

This command executes the build scripts for App1 and the package that App1 depends on.

Similarly, the -f(from) argument allows the command to apply only to the current package and packages that depend on it.

  1. Ensure consistency of dependent versions

Projects in Monorepo should be as consistent as possible with dependent versions, otherwise there is a high risk of double packaging and other problems.

Rush provides a number of capabilities to ensure this, such as Rush Check, Rush Add -p package-name -m, and ensureConsistentVersions.

Interested students can browse Rush’s official documentation on their own, which is full of details and provides answers to frequently asked questions.

Package reference specification

Product reference

In traditional reference mode, app directly references the build product of package after the build is completed. The development phase can enable real-time builds with the capabilities provided by build tools (e.g. TSC — Watch)

  • Pros: Standard, app friendly.
  • Cons: As more modules grow, the package hot update rate can become unbearable.

Source reference

The main field in package.json is configured as the entry file of the source file, and the app that references this package needs to include this package in the compilation process.

  • Advantages: With the help of the thermal update capability of the APP, there is no process of generating construction products, and the thermal update speed is fast
  • Disadvantages: App adaptation is required,aliasAdaptation cumbersome;

Reference standard

  1. Packages for internal use in projects, called features, should not be released externally and will be used directlymainField set to source file entry and configure app project webpack, go compiled form.
  2. For packages that need to be distributed, features should not and should not be referenced. There must be a build process. If source code development is needed to increase the speed of hot updates, a custom entry field can be added.

Add: The rush build command supports build product caches. Incremental build apps can be achieved if the app split granularity is small, the number of reusable packages is large enough, and the image package supports set and GET of build product caches.

Workspace protocol (workspace:)

Rush was born before PNPM and Yarn supported Workspace capabilities. Rush’s approach was to centrally install all the packages in the Common/Temp folder, and Rush then created symbolic links from each project to Common/Temp. PNPM Workspace is essentially equivalent.

Enable the PNPM workspace capability to use the Workspace: protocol to ensure that referenced versions are deterministic, and packages referenced by this protocol will only use the contents in Monorepo.

{
  "dependencies": {
    "foo": "workspace:*"."bar": "workspace:~"."qar": "workspace:^"."zoo": "The workspace: ^ 1.5.0." "}}Copy the code

It is recommended to use this protocol when referring to the package in Monorepo, referring to the latest local version of the content, ensuring that changes can be diffused and synchronized to other projects in a timely manner, which is also an advantage of Monorepo.

If you must use the remote version, you need to configure the project (cyclicDependencyProjects configuration) in rush.json, see rush_json.

PNPM Workspace :* matches prerelease version ๐Ÿ‘‰ Local prerelease version of packages should be linked only if the range is *

The problem record

Monorepo Project Dependencies Duplicate

This problem is similar to Yarn Duplicate mentioned earlier, but is not unique to Yarn.

Assume the following dependencies (transform the Yarn Duplicate example into the Monorepo scenario)

App1 and Package1 both belong to Monorepo internal project.

In the Rush(PNPM)/Yarn project, the installation is carried out in strict accordance with the version declared in project package.json in Monorepo, that is, app1 is installed [email protected] and package1 is installed [email protected].

When app1 is packaged, both [email protected] and [email protected] are packaged.

This may surprise you, but it’s natural when you think about it.

In another way, the whole Monorepo is a large virtual project, and all of our projects exist as direct dependencies of this virtual project.

{
  "name": "fake-project"."version": "1.0.0"."dependencies": {
    "@fake-project/app1": "1.0.0"."@fake-project/package1": "1.0.0"}}Copy the code

When installing dependencies, (p) NPM downloads direct dependencies first and then indirect dependencies, and determines whether the installed module version (direct dependency) conforms to the version range of the new module (indirect dependency) when installing the same module. If so, it skips. If not, install the module under node_modules of the current module.

However, lib-a, the direct dependency between App1 and Package1, is the indirect dependency of fake-project. Therefore, install the fake-project according to the version described in package.json.

Solution: Rush: Preferred versions

Rush can avoid duplicating two compatible versions by manually specifying preferredVersions. Here, preferredVersions of lib-A in Monorepo is specified as 1.2.0, which is equivalent to directly installing the specified version of the module under the virtual project as a direct dependency.

{
  "name": "fake-project"."version": "1.0.0"."dependencies": {
    "@fake-project/app1": "1.0.0"."@fake-project/package1": "1.0.0"."lib-a": "1.1.0"}}Copy the code

For Yarn, the installation of a certain version of lib-a in the root directory is invalid because of Yarn Duplicate. But there are still two ways to deal with it:

  1. throughyarn-deduplicateTargeted modificationyarn.lock;
  2. useresolutionsField. Too rough, not likepreferredVersionsIncompatible versions can be allowed and are not recommended.

Keep in mind that in Yarn, duplicate dependencies should also be eliminated Package by Package.

  1. For common libraries with side effects, versioning should be consistent;
  2. For other common libraries that are small (or support loading on demand) and have no side effects, repackaging is acceptable to a certain extent.

prettier

Since node_modules no longer exists in the root directory, each project needs to install a prettier as devDependency and write the.prettierrc.js file.

Prettier is installed globally in the root directory where.prettierrc.js is created (without relying on any third party package) as a lazy rule.

eslint

To use eslint-config-react-app in a project, install peerDependencies as well as eslint-config-react-app.

Why does eslint-config-react-app not have this set of plug-ins built in as dependencies, but as peerDependencies? The consumer does not need to care which plug-ins are referenced in the default configuration.

Support having plugins as dependencies in shareable config #3458

In summary: Depending on how the ESLint plugin is found, if dependency promotion fails (multiple versions conflict) and the required plugin is installed in a non-root directory node_modules, there may be a problem. Installing peerDependencies ensures that this problem does not occur.

PeerDependencies are also installed in some open source esLint configurations that do not require peerDependencies. These configurations take advantage of the flat node_modules structure of YARN and NPM, where packages are promoted to the root node_modules. Therefore, it can work normally. Even so, in Yarn.-based Monorepo, once dependencies get complicated, it’s possible that plug-ins can’t be found, and it’s an interesting coincidence that they work.

In Rush, there was no reliance on upgrades (and upgrades were not always guaranteed), and installing a series of mods was too cumbersome, so you could get around it by patching.

git hooks

Husky is typically used in projects to register pre-commit and COMMIT-msg hooks to validate code style and COMMIT information.

Obviously, with the Rush project’s structure, the root directory has no node_modules, and husky cannot be used directly.

We can achieve the same effect with the rush init-Autoinstaller ability. This section mainly refers to the installation Git hooks and Enabling Prettier documentation.

# Initialize a rush-lint autoinstaller

$ rush init-autoinstaller --name rush-lint

$ cd common/autoinstallers/rush-lint

Install lint dependencies

$ pnpm i @commitlint/cli @commitlint/config-conventional @microsoft/rush-lib eslint execa prettier lint-staged

# Update pnpm-lock.yaml for rush-Lint

$ rush update-autoinstaller --name rush-lint
Copy the code

Add commit-lint.js and commitlint.config.js to the rush-lint directory as follows

commit-lint.js

const path = require('path');
const fs = require('fs');
const execa = require('execa');

const gitPath = path.resolve(__dirname, '.. /.. /.. /.git');
const configPath = path.resolve(__dirname, './commitlint.config.js');
const commitlintBinPath = path.resolve(__dirname, './node_modules/.bin/commitlint');

if(! fs.existsSync(gitPath)) {console.error('no valid .git path');
    process.exit(1);
}

main();

async function main() {
    try {
        await execa('bash', [commitlintBinPath, '--config', configPath, '--cwd', path.dirname(gitPath), '--edit'] and {stdio: 'inherit'}); }catch (\_e) {
        process.exit(1); }}Copy the code

commitlint.config.js

const rushLib = require("@microsoft/rush-lib");

const rushConfiguration = rushLib.RushConfiguration.loadFromDefaultLocation();

const packageNames = [];
const packageDirNames = [];

rushConfiguration.projects.forEach((project) = > {
  packageNames.push(project.packageName);
  const temp = project.projectFolder.split("/");
  const dirName = temp[temp.length - 1];
  packageDirNames.push(dirName);
});
Scope must be all/package name/package dir name
const allScope = ["all". packageDirNames, ... packageNames];module.exports = {
  extends: ["@commitlint/config-conventional"].rules: {
    "scope-enum": [2."always", allScope],
  },
};
Copy the code

Note: there is no need to add prettierrc.js (the root directory already exists) and eslintrc.js (the projects already exist).

The.lintStageDRC file was added to the root directory

.lintstagedrc

{
  "{apps,packages,features}/**/*.{js,jsx,ts,tsx}": [
    "eslint --fix --color"."prettier --write"]."{apps,packages,features}/**/*.{css,less,md}": ["prettier --write"]}Copy the code

With the dependencies installed and configured, we registered the command execution in Rush.

Modify the common/config/rush/command – line. Json file commands in the field.

{
  "commands": [{"name": "commitlint"."commandKind": "global"."summary": "Used by the commit-msg Git hook. This command invokes commitlint to lint commit message."."autoinstallerName": "rush-lint"."shellCommand": "node common/autoinstallers/rush-lint/commit-lint.js"
    },
    {
      "name": "lint"."commandKind": "global"."summary": "Used by the pre-commit Git hook. This command invokes eslint to lint staged changes."."autoinstallerName": "rush-lint"."shellCommand": "lint-staged"}}]Copy the code

Finally, bind rush commitlint and rush lint to commit-msg and pre-commit hooks, respectively. Add commit- MSG and pre-commit script to common/git-hooks directory.

commit-msg

#! /bin/sh

node common/scripts/install-run-rush.js commitlint || exit $? # + +
Copy the code

pre-commit

#! /bin/sh

node common/scripts/install-run-rush.js lint || exit $? # + +
Copy the code

This completes the requirement.

Avoid installing ESLint and Prettier globally

After installing ESLint and Prettier in the rush-Lint directory as we did in the previous section, we didn’t need a global installation, just need to configure VSCode.

{
  // ...
  "npm.packageManager": "pnpm"."eslint.packageManager": "pnpm"."eslint.nodePath": "common/autoinstallers/rush-lint/node_modules/eslint"."prettier.prettierPath": "common/autoinstallers/rush-lint/node_modules/prettier"
  // ...
}
Copy the code

The appendix

Common commands

yarn rush(x) detail
yarn install rush install Install dependencies
yarn upgrade rush update Rush Update install dependencies, based on lock files

Rush Update — Full full update to the latest version conforming to package.json
yarn add package-name rush add -p package-name Yarn add By default, the installation version starts with ^. Minor version updates can be accepted

The default installation version of Rush Add starts with ~ and only patch updates are accepted

You can add –caret to rush Add to achieve the same effect as YARN Add

Rush Add cannot install multiple packages at once
yarn add package-name –dev rush add -p package-name –dev –
yarn remove package-name – Rush does not provide the remove command
– rush build Execute the build script for the project with the change (git-based) in the file

Rush build -t @monorepo/app1 indicates that only @monorepo/app1 and its dependent packages are built

Rush build -t @monorepo/app1 indicates that only the @monorepo/app1 dependent package is built, not including itself
– rush rebuild Build scripts for all projects are executed by default
Yarn XXX (Custom script) Rushx XXX (custom script) Yarn XXX Run the XXX script in package.json in the current directory (NPM scripts).

Rushx XXX did the same. You can run rushx directly to view the scripts supported by the current project.

workflow

# pull the latest changes from Git
$ git pull

# Update NPM dependencies
$ rush update

# Repackage @monorepo/app1 dependent projects (excluding the package itself)
$ rush rebuild -T @monorepo/app1

Enter the specified project directory
$ cd ./apps/app1

# Start a project
$ rushx start # or rushx dev
Copy the code

Refer to the article

  • Rush.js
  • Node_modules dilemma
  • Why should we use pnpm?
  • Tiling node_modules is not the only way