background

When using NPM or YARN to manage project dependencies, the following questions may occur:

  1. What if there is a problem with project dependencies? If node_modules is deleted and then installed again, is there any risk?
  2. Is it a problem to install all dependencies into Dependencies without distinguishing devDependencies?
  3. Our application relies on PKG-A and PKG-B, and PKG-A relies on PKG-B. Will PKG-B be installed or repackaged multiple times?
  4. In one project, I use NPM and others use YARN. What problems can this cause?
  5. Do we want to commit lockfile(package-lock.json/yarn.lock) to the project repository?
  6. There are a lot of conflicts in the lockfile operation with Git. How do you solve them?

The internal mechanism and thinking behind NPM

Let’s start with the first question. “Remove node_modules and reinstall NPM” is always a good way to solve dependency installation problems. What are the risks? Let’s explore it together.

The installation mechanism of NPM is well worth exploring. PIP is a global installation, but NPM’s installation mechanism has a different design philosophy.

NPM will preferentially install dependencies into the project directory. This has the advantage of keeping dependencies separate for different projects and reducing the API burden on package authors; The disadvantage is also obvious, if both our REPO_A and REPO_B have the same dependency pkG_c, then this common dependency will be installed once in each project. That is, the same dependency may be installed multiple times on our computers.

npm install

Here is an overview of the NPM installation dependency process, with the following steps to focus on:

  1. Check the configuration.Including project level, user level, global level, built-in.npmrcFile.
  2. Determine the dependency version and build the dependency tree. There are two sources to determine the project dependency version: package.json file and lockfile file. The two sources for confirming the version and building the dependency tree are mutually complementary. If package-lock.json file exists and conforms to package.json declaration, read directly; Otherwise, reconfirm the dependent version.
  3. Download package resources. If yes, use the cache file directly. If no, download and add it to the cache. Then unpack the package to node_modules according to the dependency tree.
  4. Generate the lockfile file.

Several logics can be identified:

  1. Json and package-lock.json files are required for version validation when building the dependency tree. Package-lock. json installation version is confirmed first, if it meets the rules, it will prevail, otherwise the version range declared by package.json will be reconfirmed. In particular, manually changing package information during development will result in abnormal lockfile version information, which may also be confirmed by package.json. The confirmed dependency tree is saved in package-lock.json file, which is different from yarn.lock.
  2. A higher version of the same dependency is installed in the top-level directory, node_modules. Otherwise, they are scattered in some dependent node_modules directories, such as node_modules/expect- JSX /node_modules/react.
  3. If version incompatibility is caused by relying on upgrade and multiple versions need to coexist, the higher version is still installed to the top level, and the lower version is distributed to different directories.
  4. The existence of lockfile ensures the certainty of project dependency structure and the stability of project operation in multiple environments.
  5. .

Yarn installation concepts and overcoming dependency management difficulties

As a dependency management tool different from NPM, YARN was born to solve some problems of NPM in history, such as the lack of integrity and consistency guarantee for dependencies, and the slow installation speed of NPM. Although NPM has been developed so far, it is similar to YARN in many aspects. However, the installation concept of YARN still needs our attention. The installation concept proposed by YARN solved the dependency management problem of NPM at that time:

  1. Certainty. The yarn.lock mechanism ensures certainty, including but not limited to a clear dependent version and a clear dependent installation structure. That is, it can be installed the same way on any machine and in any environment.
  2. The modules are installed flat. Different versions of dependent packages are grouped into a single version according to certain policies to avoid redundancy by creating multiple copies. (NPM has the same optimization)
  3. Better network performance. Yarn adopts the concept of request queuing, similar to concurrent connection pooling, to better utilize network resources. A better retry mechanism has also been introduced for installation failures. (Earlier versions of NPM downloaded sequentially, handing over download control to the next package after the first package was fully downloaded.)
  4. Introduce cache mechanism to realize offline policy. (NPM has similar optimizations)

Yarn. lock File structure

Use dependencies such as react as an example to learn about the structure of the yarn.lock file and how to determine the dependency version.

# THIS IS AN AUTOGENERATED FILE. DO NOT EDIT THIS FILE DIRECTLY. 
# yarn lockfile v1 
expect-jsx@^5.0. 0: 
 version "5.0.0" 
 resolved "[http://registry.npmjs.org/expect-jsx/-/expect-jsx-5.0.0.tgz#61761b43365f285a80eb280c785e0783bbe362c7] (HTTP: / / http://registry. Npmjs.org/expect-jsx/-/expect-jsx-5.0.0.tgz#61761b43365f285a80eb280c785e0783bbe362c7)" 
 integrity sha1-YXYbQzZfKFqA6ygMeF4Hg7vjYsc= 
 dependencies: 
 collapse-white-space "^ 1.0.0" 
react "^ 16.0.0"
 react-element-to-jsx-string "^ 13.0.0" 
react-rater@^6.0. 0: 
 version "6.0.0" 
 resolved "[http://registry.npmjs.org/react-rater/-/react-rater-6.0.0.tgz#2e666b6e5e5c33b622541df6a7124f6c99606927] (HTTP: / / http://registr Y.npmjs.org/react-rater/-/react-rater-6.0.0.tgz#2e666b6e5e5c33b622541df6a7124f6c99606927)" 
 integrity sha512-NP1+rEeL3LyJqA5xF7U2fSHpISMcVeMgbQ0u/P1WmayiHccI7Ixx5GohygmJY82g7SxdJnIun2OOB6z8WTExmg== 
 dependencies: 
 prop-types "^ 15.7.2" 
react "^ 16.8.0"
 react-dom "^ 16.8.0" 
// One or more dependency declarations with the same version range determine which version is available. This is the certainty of lockfile.
react@^16.0. 0, react@^16.8. 0:
version "16.14.0"
 resolved "[http://registry.npmjs.org/react/-/react-16.14.0.tgz#94d776ddd0aaa37da3eda8fc5b6b18a4c9a3114d] (HTTP: / / http://registry.npmjs.or G/react / - / react - 16.14.0. Tgz# d776ddd0aaa37da3eda8fc5b6b18a4c9a3114d 94)" 
 integrity sha512-0X2CImDkJGApiAlcf0ODKIneSwBPhqJawOa5wCtKbu7ZECrmS26NvtSILynQ66cgkT/RJ4LidJOc3bUESwmU8g== 
 dependencies: 
 loose-envify "^ 1.1.0." " 
 object-assign "^ 4.4.1" 
 prop-types "^ 15.6.2" 
// If there are multiple versions of the same dependency, the highest version is installed in the top-level directory, node_modules.
react@^17.01.:
version "17.0.2"
 resolved "[http://registry.npmjs.org/react/-/react-17.0.2.tgz#d0b5cc516d29eb3eee383f75b62864cfb6800037] (HTTP: / / http://registry.npmjs.org . / react / - / react - 17.0.2 tgz# d0b5cc516d29eb3eee383f75b62864cfb6800037)" 
 integrity sha512-gnhPt75i/dq/z3/6q/0asP78D0u592D5L1pd7M8P+dck6Fu/jJeL6iVVK23fptSUZj8Vjf++7wXA8UNclGQcbA== 
 dependencies: 
 loose-envify "^ 1.1.0." " 
 object-assign "^ 4.4.1"
Copy the code

From the information that depends on the version description above, the following points can be determined:

  1. All dependencies, whether project declaration dependencies or dependency dependencies, are managed flat.
  2. The version of a dependency is determined by the version declaration scope of all dependencies. Dependencies that have the same version declaration scope are grouped into a category, and a version of the dependency within that scope is determined. If more than one version of the same dependency exists, it will be juxtaposed.
  3. Each version of a dependency determination consists of the following:
    1. Declared versions of multiple dependencies that conform to the Semver specification;
    2. The determined version number version field;
    3. Version integrity verification field
    4. Depend on the list
  4. One significant difference between NPM and Yarn is that yarn.lock does not rely on a fixed version. That is to say, yarn.lock alone cannot determine the node_modules directory structure, and it needs to work with package.json.

yarn install

Here are the steps to install dependencies in YARN: Checking checking checking checkingCheck whether there are any nPM-related configuration files in the project, such as package-lock.json, etc. If they exist, they may be warned because they may conflict. During this phase, the system OS and CPU information are also checked.

This step is to resolve the dependency tree, resolving packages, and resolving version information. Json dependencies, including dependencies, devDependencies, and optionalDependencies. Then, the version information of dependent packages is obtained by traversing the first-layer dependencies, and the version information of nested dependencies under each dependency is searched recursively. The parsed and parsed packages are stored in a Set data structure, which can ensure that packages within the same version range will not be parsed repeatedly.

  • If the package has not been parsed, obtain the version information from yarn.lock for the first time and mark it as parsed.
  • If no package is found in yarn.lock, a request is made to Registry to obtain information about the highest known package that meets the version range, and the current package is marked as resolved once obtained.

In short, after a complex parsing algorithm, we determined the specific version information and download location of all dependencies.

This step involves locating the specific package resource from the system cache. First an attempt is made to find the dependent package in the cache, and if the cache is not hit, the dependent package is downloaded into the cache. For packets that do not hit the cache, Yarn maintains a FETCH queue for network requests based on rules. This is an optimization point to solve the problem of slow NPM V3 installation when YARN was first created. Parallel download is supported.

How do I know if I hit the cache?

Determine that there is a match in the system.”cachefolder+slug+node_modules+pkg.name“The path of the rule, if it exists, is judged to have hit the cache, otherwise it will be re-downloaded. It is worth noting that different versions of packages are managed flat in the cache. The following is the dependency cache for Webpack in the cache, which can be viewed through yarn Cache dir. Linking dependenciesThis step is to copy the cached dependencies into the project directory, while following the flattening principle. As mentioned earlier, NPM installs dependencies into the project directory first, so dependencies in the global cache need to be copied to the project. Before copying dependencies, Yarn parses peerDependencies. If it does not find a dependency that matches the peerDependencies statement, Yarn will warn you (this does not affect command execution), and copy dependencies to the project.

If there are binary packages in the dependent package that need to be compiled, this step will be done.

How to crack the dependency management dilemma

In the NPM V2 era, installed dependencies would reside in the node_modules directory of reference dependencies, and if there were too many dependencies, they would form a huge dependency tree. This structure, while straightforward, is unfriendly to large projects. Deep dependency hierarchies are bad for development, and reuse of dependencies is also a problem. Introduced the concept of flattening in NPM V3. Look at some examples of scenarios 🌰 :

Scenario 1: Different NPM versions install dependent structures

[email protected] dependency [email protected], NPM V3 is flat management dependency.

Scenario 2: Multiple NPM versions coexist

Based on scenario 1, install [email protected], which relies on another version of [email protected]. Since [email protected] already exists in the root directory, NPM V3 installs [email protected] into the [email protected] dependent node_modules directory.

Pretty boy doubt: why [email protected] in the top, and [email protected] in the sub-level?

Scenario 3: The number of dependent versions and the distribution of dependent versions

Based on scenario 2, install [email protected], which also relies on [email protected]. Similarly, NPM V3 installs [email protected] into the [email protected] dependent node_modules directory because [email protected] already exists in the root directory.

Beautiful boy doubt: you may question, at this time there are [email protected] [email protected]’t version V2 appear in the top-level installation directory instead of version V1?

This depends on the order in which the dependencies are installed. If a version of a dependency appears at the right time, it will be installed into the top-level node_modules directory. NPM V3 is not meant to be a stable package management tool, as the order in which different versions appear leads to differences in dependency structure. As in life, the order of introduction is important. It determines where and what you do.

Scenario 4: The dependent version is duplicate and available

Based on scenario 3, install [email protected], which relies on [email protected]. NPM V3 skips the installation of this dependency because the target version already exists in the top-level directory.

Scenario 5: Version Upgrade

Based on scenario 3, if [email protected] is updated and its dependency is [email protected]. So NPM v3 is executed in the following order: remove [email protected], install [email protected], install [email protected], leaving [email protected] in the top-level directory, So [email protected] will be installed into its parent’s dependent node_modules directory.

Scenario 6: Dependent Version Multiple directories exist and reuse conditions are met

Based on scenario 5, update [email protected], which relies on [email protected]. Then NPM V3 is executed in the following order: delete [email protected], install [email protected], delete [email protected], install [email protected], resulting in the following structure.

At this point, you’ll find that there are multiple of [email protected] in different node_modules directories, do they only need one copy in the top-level directory? That’s right, we deleted node_modules and reinstalled it to get the clean structure you want.In fact, a more elegant way to achieve this structure is to use the NPM dedupe command. Yarn automatically runs the dedupe command when installing dependencies. Because of some of the above holes in NPM history, it is recommended to use YARN as a package management tool for project collaboration. Of course, many problems have been optimized since the development of NPM. Yarn and NPM are two dependency management tools that keep up with each other and get inspiration from each other.

NPM vs. yarn 💪

Here is a simple comparison between NPM v6 and YARN V1. This is the version we use in production and development.

As two similar package management tools, NPM and YARN take inspiration from each other in the implementation of some functions.

Similarities:

  1. Package. json as the project dependency description file
  2. Node_modules is used as the dependent storage directory. Yarn V2 is no longer like this
  3. Lockfile Locks version dependency, which is called yarn.lock in yarn and package-lock.json in NPM. Yarn. lock is also supported in NPM V7. It ensures a stable node_modules directory structure for different machines or environments.

Differences:

  1. Dependency management policies.
  2. Lockfile. Package-lock. json comes with version-locking + dependencies. If you want to change some dependencies, the impact may be more complex than it seems. Yarn. lock has version locking and no dependency structure. To manage project dependencies using YARN, package.json + yarn.lock must determine the dependency structure.
  3. Performance. (Comparing NPM V6 with YARN V1) Currently NPM V7 optimizes caching and download network policies, narrowing the performance difference.

NPM enterprise deployment private server principle

The source registry in NPM is actually a query service. In the case of npmjs.org, the query service url is registry.npmjs.org/. Append the name of the dependency to this url and you get a JSON object that contains all the information about the dependency. Such as:

  • registry.npmjs.org/react
  • registry.npm.taobao.org/react

We can set the installation source by using the NPM config set Registry command. Do you know why our company deploys private NPM images? Although NPM is not blocked, downloads of third-party dependencies are still slow, which can seriously affect the CI/CD process or local development efficiency. In general, we see the following advantages in deploying NPM private:

  1. Ensure high speed and stable NPM service
  2. Ensure the security of publishing private modules
  3. Audit mechanism can guarantee the quality and safety of NPM module on private server

Enterprise private server deployment, can obtain security, stability, high-speed protection.

Tips for managing project dependencies (brainstorming…)

  1. It is recommended to use YARN as the team package management tool instead of NPM. Although versions after NPM V6 tend to be stable and secure, it is still recommended to use YARN as a unified package management tool for teams due to historical reasons and team management compatibility.
  2. The lockfile file must exist in the project, and manual modification is prohibited, because this ensures stable operation of the project.
  3. If yarn.lock has problems during code merging, try using YARN Install to resolve the problem.
  4. .

The resources

  • npm install
  • yarn install
  • yarn.lock
  • NPM vs. Yarn: Which Package Manager Should You Choose?
  • Lockfiles should be committed on all projects

❤️ Thank you

That is all the content of this sharing. I hope it will help you

Don’t forget to share, like and bookmark your favorite things.

Welcome to pay attention to the public number ELab team receiving factory good article ~

We are from the front end department of Bytedance, responsible for the front end development of all bytedance education products.

We focus on product quality improvement, development efficiency, creativity and cutting-edge technology and other aspects of precipitation and dissemination of professional knowledge and cases, to contribute experience value to the industry. Including but not limited to performance monitoring, component library, multi-terminal technology, Serverless, visual construction, audio and video, artificial intelligence, product design and marketing, etc.

Interested students are welcome to post in the comments section or use the internal tweet code to the author’s section at 🤪

Bytedance calibration/social recruitment promotion code: WKVVX3V

Post links: jobs.toutiao.com/s/dLLnGvr