What was the last time you messed with dependencies and build configurations, and how long did it take? One design of NPM is devDependencies, which is often blamed for development environment stability problems in front-end projects.

It all started with this famous command:

npm install
Copy the code

It was a great order, no doubt. Without NPM install, it is estimated that 80% of today’s front-end work can be eliminated. Its job is to recursively install dependencies starting with dependencies and devDependencies in the front-end project into node_modules. To this day, when you get a front-end project, your subconscious mind is “put everything together with one command, and then start the development environment with one command.” This is clearly a huge improvement over the slash-and-burn era.

However, behind this easy-to-use tool at first glance, there have been many slots. For example, many of you may have seen this vivid contrast:

How big is node_modules for your project? For serious on-line projects, the contents of this directory are not 500 MB, I’m afraid you are embarrassed to say hello to people. However, relying on package size is not the focus of this article — after all, it is the best evidence of the rapid development of front-end engineering. The first thing I want to point out is a special aspect of front-end projects.

The specificity of front-end projects

The unique nature of the front-end domain is that the resources required to build a project are not only diverse, but also highly fragmented.

What does it mean to have a great variety of resources? For common programming languages, their package managers are tailored specifically for that language, from Python’s PIP to Java’s Maven to Rust’s Cargo. The file formats of these languages are also uniquely determined. However, the resource types that need to be hosted in a front-end project basically include CSS, HTML, SVG, fonts, images, in addition to JavaScript… The construction, translation, packaging, optimization of each of these resources… And other tools, almost all written JS, through NPM to install. So today’s front-end projects have a wide range of resources to handle through NPM installation dependencies.

So what is the concept of “highly fragmented”? The amazing thing about the front-end community is that there are a bunch of blooming solutions for each of the resources mentioned above:

  • You want to write JS? The official standard alone is ES2015/2017/2018/2019… So much, and the various stages of malformed syntax, not to mention the various xxscripts represented in TypeScript.
  • You want to write CSS? I heard that Less, Sass, and Stylus are all out of style. Is it PostCSS or CSS Modules? Styled Components oh, there is also a Styled component, as if it also works?
  • You want to write HTML? Let’s see if you like JSX from the React party or Vue gang.vue? Don’t forget EJS, Pug, and countless other template syntax options to choose from.

The trouble here is not so much picking, comparing or using a specific tool (which for many of you has the pleasure of a woman picking clothes), but that you have to make highly fragmented choices. Keep in mind that for almost every solution (or standard) above, there are one or more build tools, each of which may have its own plug-in architecture and versions updated constantly. Your build selection is almost always a small part of a larger set of possibilities — don’t you need to include them in devDependencies? Isn’t it a very suitable design?

In theory, yes. DevDependencies shares the same node_modules directory as dependencies.

Vulnerability to devDependencies

Think about how you update dependencies when writing other languages: When updating a Python crawler package, do you want to update the version of the Python interpreter along with it? Who doesn’t have to do this to themselves — the actual business logic code and its dependencies are almost always two separate things from the tools used to build the project. However, for the front-end domain, because a large portion of the front-end toolchain is in node_modules, it is very easy to affect your toolchain when updating business logic dependencies.

Many of you may not realize how much of a dependency node_modules has on the toolchain type. React and Vue, for example, community scaffolding tools, how many dependencies and devDependencies do they generate? The author made a simple attempt as follows:

  • Install React and ReactDOM separately, which takes only 3.9 MB of space.
  • Installing Vue separately also takes up only 3.6m space.
  • Create a blank React project with create-react-app and use 189.6m space.
  • Use vue-CLI to create a blank VUE project that takes up 164.5m space.

“DevDependencies” is one of the most important dependencies in the framework. While there will certainly be more business dependencies in real projects, the browser-side business base libraries rarely have complex heavy dependency structures, and it’s easy to increase the overall size of the toolchain dramatically as you improve your build configuration based on your project needs. Therefore, it is a reasonable assumption that more than half of the actual front-end project dependencies belong to devDependencies. The problem with this is not absolute size, but that the high fragmentation of build resources makes build tools need to iterate quickly, bringing in a large number of dependent versions. With so many dependent versions, the stability of the assembly may not be reassuring if it accidentally drifts. This is not a JS language level problem, but a system level problem for any large software. Anyone who has struggled with graphical interface dependencies on radical Linux distributions will understand this.

Create-react-app adds all dependencies after eject to dependencies. This makes sense from the point of view of the application and library, but it does not affect our results.

With the exception of NPM update, it is possible (but not certain) for each NPM and YARN install to thoughtfully update the toolchain version based on semantic version specifications, thus introducing potential instability issues. If you reload the same package.json file a few days later, the build tool version in devDependencies will almost certainly be different. Who do you think updates GCC/XCode/Android Studio diligently on a daily basis? This is exactly what happens in many front-end projects, because node_modules change frequently, and the core tools for building many resources are in there.

See here certainly many students will not sit still: there is not a special version of drift Lock file? Yes, Lock can Lock versions, but remember that Lock is inherently conflict-prone. Delete the reload – congratulations on the full update package again. In fact, the tool chain for devDependencies is easily implicated in the following scenarios:

  • Projects rely on internal NPM packages and often need to be updated in parallel on different development branches
  • The project needs to checkout to one of the older versions to fix problems and resynchronize business dependencies
  • When special private repository configurations need to be used within the team
  • When the package management tool versions used by team members are not identical

To go too far, ensuring that any two installations in a real world generate the same Lock is almost as difficult as ensuring that two phones of the same model get the same running score. It’s not that Lock isn’t well designed, but the way it’s being used now, there’s a lot of potential for surprises, even when it comes to tool chain stability.

So we already know that devDependencies, which are mixed with business dependencies, are liable to be accidentally updated, with or without Lock, introducing instability. However, the “vulnerability” here is not only the toolchain version is prone to fluctuations, there are other troublesome issues.

For example, devDependencies can affect the development experience of macro repositories. A mono-repo is a repository that manages a bunch of packages together in this form:

My - mono - repo ├ ─ ─ package. The json └ ─ ─ packages ├ ─ ─ A │ ├ ─ ─ package. The json │ ├ ─ ─ node_modules │ └ ─ ─ the SRC ├ ─ ─ B │ ├ ─ ─ ├── ├─ ├─ download. TXT ├─ ├─ download. TXT ├─ SRCCopy the code

Suppose we maintain packages A, B, and C ourselves. These packages also have dependencies on each other. Generally speaking, based on NPM link command or more automatic Lerna tool, we can maintain the dependency between them through soft link. If you need to maintain these packages yourself, it is natural to think that A, B, and C should have their own dependencies and devDependencies.

We can actually do that. The problem is that as long as each package in the macro repository introduces its own distinct devDependencies, each of these packages introduces a huge node_modules, which not only introduces a lot of redundancy, but also slows down the initialization of the repository, making links more fragile, like flying wires between several huge machines. If packages in the warehouse also need to be linked to other projects, it’s even more troublesome. In our practice, if each package in the macro repository individually relies on a large build tool like Babel, the subtle differences between them will often result in reconfiguration of link relationships, resulting in a lot of pointless Lock file changes. Don’t do this unless you’re doing it temporarily to integrate old projects.

In a sense, the Link operation in the macro warehouse is the price NPM pays for simplicity. NPM maps dependencies based on the directory structure of the file system. You can directly modify the base library Webpack and Vue in node_modules to see the effect, so that you can contribute PR to the community. A Link is a soft Link that creates a dependency between any two directories. However, dependencies that need to be managed by their own Link are mostly private and business dependencies that reuse the same node_modules “container” as a large number of dependencies used for building, which is inherently unstable.

In addition to macro repositories and links, expansive devDependencies have an impact on CI building. For teams that maintain their own business NPM packages, business dependencies are likely to be updated quickly. This means that when implementing NPM install on CI, you often have to alert an entire chaotic node_modules for trivial business dependencies on bugfix upgrades, affecting build speed and stability. Of course, mainstream build systems have build caches, but we’ve had some accidents in production where Yarn dependencies used the wrong version of the cache for dependencies, causing problems on the line. What to do? Clear the build cache and start from scratch… By the way, on a dev machine we built a while back, the inode index on the disk was even brushed to zero. Of course, you can say that these problems are the cauldron for building cache, Lock, and Linux, but for devDependencies, which bring tens of thousands of lines of Lock files, you equal… You must be responsible, too. Why beat the whole egg into water and then pick out the shell?

The current popularity of scaffolding tools also contributes to the instability of this environment configuration. Since create-React-App took the lead in demonstrating eject’s ruthless gameplay, the mainstream scaffolding tools have been largely untouchable. Many internal “unified scaffolding” tools are still “copy a mess -> install -> run” quality three. This can be handy when there are fewer projects, but dependency management after a project is built can be tricky.

What can we do

At this point, we’ve talked a lot about the problem with devDependencies behavior. So what can we do? Add devDependencies to a separate node_modules module. In this regard, it is easy to think of a number of simple and convenient practices, such as:

  • For simple front-end application projects, a global Parcel or even the browser’s native ES Module is sufficient, without the need to configure special packaging tools.
  • Common front-end application projects can be distinguishedbuild 和 srcTwo paths, each with its own package.json file, so thatnode_modulesIsolated. That way they don’t need eitherdevDependencies.
  • For macro warehouse type front-end projects, each package can be heavydevDependenciesExtract it and create a top-level module specifically for building. This top-level module can be packaged as a single repository for official distribution, with each package distributing the source code directly.
  • In addition to scaffolding, consider encapsulating a specialized and stable for multiple front-end applications where there is some specificationbuildTools to global, similar to very specialized parcels — just install it globally and you don’t even have to install the Babel plug-in in each project, just a fewdepencenciesIt is enough. Doesn’t it feel better to build a wheel than a quality three
  • Isolate specifically for projects that require CIbuildType dependence also helps speed up builds and improves the stability of incremental builds.
  • Oh, and don’t forget the test kit. They really fit indevDependenciesThere are also relatively few annoying build problems. Of course, it’s easy to isolate its bags, which is a matter of opinion.

To sum up, this article seems to cover a lot of ground, but the really important points are these:

  • Because of the complexity of the resources used to builddevDependenciesDependence can easily swell.
  • The large number of dependencies used by these builds will share Lock and with the actual dependencies of the projectnode_modulesAnd become more vulnerable and less efficient.
  • As long as this part of the bloated but stable dependence fromnode_modulesIt is easy to improve the stability of the build. This separation is also fairly easy to achieve.

At the end of the day, devDependencies do exist. But the complexities we face today make it easy to exceed the design load. Therefore, the stability of the front-end development environment is largely related to the inherent nature of the project and our current usage of toolchains, not to the weak types of JS. Hopefully this article will reduce some of the daily hassles and give you more confidence in the community 😀