preface

With the successful MERGE of VUe3 and Vite warehouse that Utah university switched package management to PNPM recently, and some projects in vue ecosystem such as VueUse also switched to PNPM, It announces that the project warehouse in vUE ecosystem has completed the migration from the original YARN Workspace Monorepo to PNPM workspace Monorepo.

Patak (github.com/patak-js), a core Vite contributor and a member of the VUE team, tweeted a vivid description of the project’s migration: “The project has fallen like dominoes to PNPM.”

For details on PNPM, see my previous article: PNPM: The Most advanced package management tool, which I won’t cover more in this article.

Vue migration project

For the specific process of project migration in the VUE ecosystem, please refer to some of these PR’s:

  • Github.com/vuejs/vue-n…
  • Github.com/vitejs/vite…
  • Github.com/vueuse/vueu…

This includes the source repository of the current VUe3.0 project:

As well as the current community inside the hot Bundleless tool vite source repository:

It can be seen that the two pr migrations were completed by Utah University. Meanwhile, Zkochan (github: github.com/zkochan), the author of PNPM, also made a code review for the PR migrations of Vite.

The above projects are based on MonorePO to do warehouse management. The WORKSPACE of PNPM has excellent support in monorePO scenarios. Of course, there are also migrations of non-MonorePO projects. For example, the package-in-project management tool for the naive- UI repository, which was migrated by the authors, is PNPM for improving the speed of dependent installation under CI (see PR: github.com/TuSimple/na…). .

Here are some of the reasons for the migration and the source of the problem. Note that the following is based on some community discussions and may not be complete or accurate, but I will try to get the details right and I will not read too much into it. If wrong, welcome to correct.

The migration reasons

There was a poll on the package manager in a Twitter post in September. I was in the PNPM community at the time and heard about it. Here are the results:

Zkochan, the author of PNPM, was satisfied with the results, as PNPM had not reached such a high usage rate due to some trends in the community.

That tweet was followed by an update from the university:

Esbuild 0.13 now uses optionalDependencies to install platform-specific binaries. Yarn 1/2 will download all binaries before picking the right one. Other (update to date) package managers only downloads the matching one.

This may be the thing that pushes me away from Yarn 1 :/

OptionalDependencies esBuild uses optionalDependencies after V0.13 to install dependencies for different platforms (github.com/evanw/esbui…). However, YARN 1/2 does not download the packages for the corresponding platform according to the optional rule. Instead, yarn 1/2 chooses to download all the packages.

So why did a tweak to esbuild come to mind for Utah? Vite currently uses the esbuild library in a number of scenarios: For example, in the current development phase, Vite uses esbuild dependency prepackaging to convert third-party dependencies into ESM bundle artifacts.

This relationship makes esBuild an underlying dependency of Vite. As mentioned earlier, the Vite repository is built based on yarn Workspace Monorepo. Therefore, every time you install dependencies using YARN during Vite development, Install esbuild and its associated packages.

Here’s how this change in EsBuild works and why it allowed Vite to migrate the Monorepo shelf directly.

Dependency distribution mechanism

As mentioned in the previous section, EsBuild uses optionalDependenceis as its current install dependency strategy. This section describes the process of distributing packages such as these cross-platform dependencies.

For details about this section, see the first part of the community article: Developing high Performance Node.js extensions with Rust and N-API.

Here I take nodeJS native Addon code distribution as an example to give an introduction:

For nodejs extension development, please refer to my previous article: nodejs C++ extension development.

There are two mainstream distribution methods:

Distribute the JS code and Postinstall downloads the corresponding artifacts

Typically, addon’s and the like, developed in other languages, package the artifacts into an executable binary (such as C++ extensions that end in.node).

The postinstall script is installed in a way that is common in the community. For example, node-sass is installed as follows:

Node-sass puts native Addon precompiled products in a CDN address. Then, when users install Node-sass using NPM install, The Addon production file is downloaded from the CDN using the postinstall script.

Esbuilds prior to V0.13 were also distributed in this way.

There is a drawback to this approach, as you can see that the downloaded binary address is a Github release address, which is often due to not being able to accommodate domestic/overseas users. However, it is generally possible to solve this problem by building a relevant download image in the country, but the problem of unsynchronization of the mirror is also often occurs.

cross-platformnative addonDistribute it through different NPM packages

Two of the most popular build tools on the market right now, SWC and ESBuild, take this approach. Each native Addon corresponds to an NPM package. Then, all NPM packages corresponding to native Addon are counted as optionalDependencies, and the OS and CPU fields in package.json of these NPM packages are added. Let the corresponding package management tool automatically select which native package to install for packages of different platforms during installation, for example, esbuild’s current NPM package structure:

{
  "name": "esbuild"."version": "0.14.1"."optionalDependencies": {
    "esbuild-android-arm64": "0.14.1"."esbuild-darwin-64": "0.14.1"."esbuild-darwin-arm64": "0.14.1"."esbuild-freebsd-64": "0.14.1"."esbuild-freebsd-arm64": "0.14.1"."esbuild-linux-32": "0.14.1"."esbuild-linux-64": "0.14.1"."esbuild-linux-arm": "0.14.1"."esbuild-linux-arm64": "0.14.1"."esbuild-linux-mips64le": "0.14.1"."esbuild-linux-ppc64le": "0.14.1"."esbuild-netbsd-64": "0.14.1"."esbuild-openbsd-64": "0.14.1"."esbuild-sunos-64": "0.14.1"."esbuild-windows-32": "0.14.1"."esbuild-windows-64": "0.14.1"."esbuild-windows-arm64": "0.14.1"}}Copy the code

Esbuild-android-arm64 package.json esbuild-android-arm64 package.json esbuild-android-arm64 package

{
  "name": "esbuild-android-arm64"."version": "0.14.1"."os": ["android"]."cpu": ["arm64"]}Copy the code

This approach can be considered the least influential distribution method for native Addon users, including esbuild, SWC, and NAPI-RS mentioned here.

The downside of this approach is that it can be a heavy burden on the developer: multiple systems and CPU architecture packages need to be maintained simultaneously. At the same time development/debugging also needs to consume a lot of work.

The previously mentioned underlying Vite dependencies, esBuild, were migrated to this optionalDependencies mode by postinstall Script after v0.13:

Package manager support

In the previous section, we introduced some of the common dependency distribution mechanisms of Native Addon, as well as the optionalDependencies mechanism that esBuild now uses after V0.13. As mentioned above, due to the dependency installation mechanism of Yarn1, all the cross-platform packages in ESbuild will be downloaded every time during vite development (for example, ios packages will be downloaded on Android platform), which will result in the dependency installation of vite warehouse every time. It takes a long time. See the issue below YARN (github.com/yarnpkg/ber…

Here’s an example (explained by esbuild author evanw):

As PNPM v6.14 currently behaves, there are various cross-platform and CPU architecture packages for optionalDep dependencies under ESBuild, but only those packages that fit the current platform architecture will actually be installed. Other packages that are not in this category simply generate meta data in the lock file.

In fact, the size of the package is much larger than the size of the data (for example, the data is about 0.5MB, and a package is about 8MB), so assume that there are 13 packages under optionalDep, Then PNPM will install packages of about 0.5MB * 13 + 8MB = 14.5MB, while YARnV1 will install packages of about 0.5MB * 12 + 8MB * 12 = 102MB, which will make different package management tools, Yarn installs far more things than PNPM, resulting in slow dependency installs.

For the problem mentioned above, please refer to the Downloads Extra Data column in the following table. Currently, yarnv3.1.0 is the only version of YARN that fixes this problem. PNPM, NPM V7, and [email protected] all fix this problem.

Vue Ecology project completed migration

Uvu did a community consultation with developers and conducted a poll on package manager, and around 90% of twitter responses recommended PNPM, Including the current Vue Core Team antfu(github.com/antfu) has also been working on their own open source project Slidev (github.com/slidevjs/sl…) PNPM is used in Chinese practice, and some functions of PNPM are also full of praise.

After a few days, vite started migrating from YARN workspace to PNPM workspace:

Although there were some problems in the process of migration, it was basically completed successfully with the help of PNPM authors and the community, and the actual migration cost was not particularly large, you can refer to the previous PR.

After Vite migrated, other VUE ecosystem projects followed, and while these other projects may not have the same underlying esbuild issues as Vite, some of PNPM’s other advantages (such as tight management of dependencies, fast dependency installation, Natural Monoreo Workspace support, etc.) has also attracted vue’s ecological migration of package management tools. Because of the vite migration experience, other projects were also quickly completed, basically a vue3 migration related Mr Was completed in a day, and gradually almost all the projects in the Vue ecosystem were completed migration.

The practice of migrating PNPM

If you want to know how to migrate from a complete Yarn Workspace project to PNPM workspace, there is no need to study how to migrate from Vite or VUe3 PR. PNPM has an article from the community: Replacing Lerna + Yarn with PNPM Workspaces.

The author introduces the file changes and project changes that need to be made if migrating from YARN Workspace (the project is based on LERNA, but there is not much difference) to PNPM workspace. This is probably a process:

  • Replace the script command withyarnThe related commands are replaced with:pnpm <command>orpnpm run <command>
  • Delete the toppackage.jsonIn theyarn workspaceconfiguration
  • Replace the workspaceConfiguration withpnpm-workspace.yamlFile to replace
  • Adjust thepipeline, as well asDockfileOr otherCI/CDConfiguration file in the dependency installation command
  • Delete theyarn.lockFile (here can also use the author developed perfectpnpm importCommand to completeyarn.lockFile conversion/laugh)
  • Adjust build related scripts (if any)lernaThe relevantbuildScript)
  • Add a.npmrcFiles are used to customize somepnpm 的 CLIBehavior (or not)

Interested students can refer to it, or directly communicate with the author can also (the author has migrated more of this type of project in byte, it also has some experience, I will not do too much introduction here).

conclusion

In fact, I have already noticed that vUE started the migration of PNPM when uVU launched the vote on package management tools, but I did not pay close attention to the underlying reasons.

Previous Yarn authors released Yarnv3.1 (documentation: dev.to/arcanis/yar… Feature is: Yarn supports PNPM mode dependent installation (i.e., content-Addressable Store) in this version, However, yarn’s authors say that the most complex feature implemented in this release is the support for the dependency packages mentioned in this article for installing different platforms and CPU architectures on demand:

I noticed this during my recent study of SWC, and took a heart-warming look at the reasons behind this series of migration problems.

Finally, I wrote such an article that is not very practical, which can be used as a record if there are relevant requirements for development later (such as the development of native Addon in other languages). Hopefully PNPM will become a popular package management tool in the community in the future