Content introduction (convenience for students who want to quickly understand the content and conclusion of the article)

1. Node.js divides dependencies into dependency and devDependency, but sharing the same node_modules folder is no longer appropriate in the development of increasingly complex front-end projects.
2. Tool dependencies and business dependencies share the same node_modules folder, which makes the development and build process inefficient and fragile, including the following problems: node_modules bloated and inefficient, version drift, Monorepo architecture mismatch, CI building process fragile, etc.
3. The solution is to separate concerns and place business and tool dependencies separately. One idea is for business dependencies to be compiled into a version management system, and tool dependencies to be precompile and released by a dedicated build team, allowing developers to select versions for global installation.

Node_modules status quo

Node_modules dependency problem is not solved in 2020, but more and more obvious. We look at a lot of packages like, “WTF, when did I install this dependency?” Node_modules is not 500 MB and you are embarrassed to say that you are doing front-end. How many of these dependencies are really used in the final product? How much of the development process, how much of the build process, is dependent on tools?

The author made a simple experiment:

• Install React and ReactDOM separately, which takes only 3.9 MB of space.
• Installing Vue separately also takes up only 3.6m space.
• Create a blank React project with create-react-app and use 189.6m space.
• Use vue-CLI to create a blank VUE project that takes up 164.5m space.

Not the whole picture, but certainly a few, as you can recall from your development process what tools you used: packaging tools, development servers, test frameworks, various Linters, TypeScript compilers, Babel, and more. These tools have their own dependencies. Children have children, and grandchildren have children, and grandchildren are endless, and soon your hard drive won’t fit. These tool dependencies, on the other hand, are only used during development and build, and even at different stages, such as many unit tests, which are actually run through the CI process online, but are all bundled into the node_modules folder with the business dependencies.

Problems with DevDependency

Node_modules shared by utility dependencies and business dependencies not only causes the folder to grow inexplicablly, NPM installation is slow, but also causes the dependency version drift, causing all kinds of puzzling bugs.

Today, front-end engineering has entered an era of skyrocketing complexity, which is caused by the skyrocketing types of resources to be processed at the front-end. Different resources require different tools for processing, and the high-speed iteration of front-end technology is superimposed. In 5 years, Build tools have changed from Grunt and Gulp to Webpack, Parcel, Rollup, Vite, Snowpack, Esbuild, etc. In the future, this rapid upgrade of tools, combined with the growth of resources, has brought about a rapid increase in tool complexity. It also brings a strong reliance on tool versioning. However, under the current Semantic version management method, even a small business dependent NPM install may cause tool dependency drift of unknown versions, which brings great challenges to the stability of the whole construction process. Delete reload, version is not crematorium.

I would like to say two more words about Lock. The original intention of Lock is good, hoping to solve the problem of inconsistency between the dependency version and the Lock file, but we have encountered in the process of using NPM install new package, and Lock file conflict situation, this time how to do? Delete node_modules and reinstall it. Congratulations

On the other hand, as front-end projects become more and more complex, more and more front-end projects adopt Monorepo architecture and need to go through online CI process for release. However, the current design method of devDependency cannot adapt to such construction method.

My - mono - repo ├ ─ ─ package. The json └ ─ ─ packages ├ ─ ─ A │ ├ ─ ─ package. The json │ ├ ─ ─ node_modules │ └ ─ ─ the SRC ├ ─ ─ B │ ├ ─ ─ ├── ├─ ├─ download. TXT ├─ ├─ download. TXT ├─ SRCCopy the code

A, B, and C each have their own node_modules. This raises the question of where devDependency will be installed. If installed on their own node_modules, So a lot of space is actually occupied by redundant tool dependencies. If you want to uniformly install node_modules in the parent directory, then you need to solve the problem of dependency versions of different subdirectories. Even if lerna and other tools can be used for automatic management, NPM install in a subdirectory may also cause some co-dependent version drifts in the parent directory, introducing unknown changes to the development and build of other subdirectories.

In addition to not being suitable for the Monorepo architecture, the design of devDependency causes problems during online CI. First of all, redundant node_modules leads to higher space and network overhead, which makes the environment initialization process longer during CI. In fact, all the tool dependencies in devDependency, such as packaging, Lint, testing, etc., are not used during CI. The dependency tools are different, but the full devDependency needs to be downloaded at each step; On the other hand, the upgrade of business dependence often leads to passive upgrade of tool dependency unknowingly, which leads to the invalidation of the previously cached devDependency. If the cache is not cleared in time and the version is updated, it is easy to lead to the inconsistency between the build and the development environment, resulting in unknown version problems. The source of all these vulnerabilities lies in the complexity of the front-end project, which has exceeded the load of devDependency designed at the beginning. DevDependency and dependency are placed indifferently in node_modules, just like when cracking eggs. And then you have to pick the shell out of the beaten egg

Unimagined roads

So far, we have talked a lot about the problems of devDependency. How can we solve them?

First, we need to define what the root cause of the problem is, and here I’ll go straight to my conclusion that the root cause of all the problems is the failure to separate concerns between tool dependency and business dependency. If you have some experience in using other programming languages, can recall that in Python, or Java, c + +, never rely on conventional tools rely on with business together at that moment, this is because both the role of, update frequency, use requirement, is different, for the business relies on, we finally is to be integrated into the products, Is with business attributes, need to be able to timely solve business problems, update frequency will be more frequent, especially in the current popular Monorepo and private NPM; As for tool dependency, our requirements are stable, uniform and efficient, and there is no need for frequent changes, or even if changes are made, they should be insensitive and transparent to business developers, let alone unstable tools due to changes in business dependency.

Now that we have identified the root of the problem, our solution is obvious:

On the one hand, devDependency’s tool dependencies can be removed from node_modules. Further, we can package these tool dependencies into a team-specific build tool, which can be installed globally by each business developer. It’s not a bad development experience to install a few business dependencies in your own project without even needing Babel. For packaged tools, can be given to a special build team for maintenance, or even packaged into a two-level package, such as PKG, deno compile to further improve efficiency.

On the other hand, business dependencies on dependency can remain in node_modules, and further, node_modules can be included in Git version control. Since the tool dependencies have been removed, the rest are business dependencies, which should be built into the final product. We need to ensure strong consistency in various environments. Meanwhile, node_modules without tool dependencies will be reduced to a reasonable level and brought under git’s control. There’s not much extra overhead.

If node_modules is added to Git’s management overhead, we can assume that in any long-running project, the ratio of business dependency to its own code is at most 1: 1. It is impossible to appear in a mature commercial project. The code written by ourselves is not as big as the dependency introduced. In summary, the cost of versioning business dependencies is acceptable relative to the benefits of greater consistency.

Now that we have the direction in hand, we can start to make specific changes:

First, the easiest and fastest way is to split Dependency and devDependency into two package.json files, and then upgrade the directory structure of devDependency to one level, using node.js’s module layer lookup feature. The dependency and devDependency can be split without any code changes. The directory structure is as follows

| - node_modules # installation devDependency rely on | -- package. # json record devDependency dependent on | - myApp | - node_modules # installation The dependency of relying on business generation | | - SRC # -- package. # json record the dependency dependence. | - gitignoreCopy the code

Next, in the.gitignore file, the node_modules of the devDependency installation will be removed, and the node_modules of the devDependency installation will be kept in the Git repository as follows

node_modules ! myApp/node_modulesCopy the code

Git clone will be able to download large files. Git can be used to ensure consistency of dependencies, as long as the code from the same branch checkout is used. Business rely on must be exactly the same, on the other hand, if there is a classmate of new or modified business relies on, can also be recorded git, make changes to the traceability, moreover, can also according to this kind of situation, to rely on the reviews, this before just change package. The json can change business relies on, It’s easy to overlook, so the benefits of incorporating Git for stability and consistency seem to me to be much greater than the size issue.

Finally, it is recommended to lock the version of the dependent library in the outermost package.json, or entrust it to specialized students for unified management. Business students only need to care about their own business dependencies.

, of course, the above scheme is the most simple transformation, mainly in order to give you a way of thinking, you can refer to the basic idea is the separation of concerns, of tools, business to business, to the actual situation of different projects, you can on the basis of the above ideas, further, to find the accord with his team of maintenance mode.

A little perspective on the future

Node.js has played an important role in the development of front-end engineering. It can even be said that it is node.js that truly leads the front-end to the field of engineering. Although there were applications to deal with front-end code through Java or Python in the past, for front-end programmers, The need to master another language always feels like there’s a window between them, and Node.js is the one that breaks the window.

While we should not ignore node.js’ contribution to front-end engineering, we should also be aware of the design limitations of Node.js. After all, the original design purpose of Node.js, whether the module specification of common.js, In the initial design, it is more aimed at using Node.js for server-side programming services. The dependency and devDependency installation methods are not specifically designed for front-end engineering. One of the problems with this is that while we enjoy the engineering power of Node.js, the dependency management approach directly adopted by Node.js becomes fragile and unreliable due to the nature of the front-end project itself.

Today, front-end engineering is facing more and more challenges:

1. What about front-end engineering under the Trend of Bundless led by Vite?
2. How to ensure the efficiency of development and construction under monorepo architecture?
3. Under the micro-front-end architecture, how to develop and build?

These new situations are new situations that people have never faced when designing Node.js. We cannot ask the designers of Node.js to consider all kinds of situations at the very beginning, because that is unrealistic. What we should do is to analyze the essence of the problem and make further progress on the shoulders of predecessors. To find a solution that is more appropriate for the situation.

This article will try to analyze some problems of node.js front-end engineering from dependency and devDependency, hoping to bring some different perspectives to readers. If you have any questions, please leave a comment in the comments section and discuss together

Author: ES2049 | magic circle

The article can be reproduced at will, but please keep this link to the original text.

You are welcome to join ES2049 Studio. Please send your resume to [email protected]