As the scale of front-end engineering increases, the relationship between the third party and its own dependency packages becomes more and more complicated. What problems might arise and how to solve them? Here are some practices from our front-end team.

What are complex dependencies

Installing dependencies is simply a matter of NPM install XXX for front-end developers. So, does this simply install a lot of dependencies on a project, even complex dependencies? Here we define “complex” as follows:

  • You need to maintain multiple different packages yourself for use in the most downstream business projects.
  • In addition to being depended on by downstream businesses, these packages may have dependencies on each other, and they may also depend on upstream packages.
  • Different packages may reside in different Git repositories and have separate testing, build, and release processes.

If you rely purely on NPM install, then all packages must be published to NPM before they can be updated by other packages. In the process of “co-tuning” these packages, going through the formal release process every time there is a slight change is undoubtedly tedious and inefficient. What tools do we have at our disposal to solve this problem?

Community tool Takeaway

When it comes to managing dependencies between multiple packages, there are a number of tools that come to mind immediately, such as:

  • Link command of NPM
  • The workspace command of Yarn
  • Lerna tools

The root of all evil here is the NPM link command. While those of you familiar with it probably know that it has a lot of problems, it does solve basic linking problems. A quick review of usage: Suppose you maintain a downstream business project called APP, and upstream dependencies called DEP. To achieve “app updates synchronously when DEP changes”, just do this:

# 1. Execute in the deP path
npm link

# 2. Execute in the app path
npm link dep
Copy the code

This forms the basic “link” relationship between app and DEP. If you go to node_modules in your app, you can see that NPM actually creates a “shortcut” (soft link) for your operating system to jump to the DEP. Manually maintaining this link relationship is cumbersome and error-prone when there are multiple interdependent packages. You can use the community yarn Workspace or Lerna to automatically manage these packages for you. Since the two are so close, we will only introduce the Lerna tool used in our production environment.

Lerna is also very silly to use, you can just put the dependency packages in the same directory in the following style, without making any changes to their specific build configuration:

my-lerna-repo/
  package.json
  packages/
    dep-1/
      package.json
    dep-2/
      package.json
    dep-3/
      package.json
    ...
Copy the code

Then a lerna bootstrap automatically handles their dependencies — package.json for each package can be written with the name of the other package (note that this is based on the name field in package.json, not the directory name). This way, you can safely manage these packages in the same Git repository without having to worry about the tedious initialization process, which is what Babel and React do today.

Of course, the actual scenario doesn’t have a command or tool ready to go. Here are some lessons from dependency management in practice:

Generation and release of cyclic dependencies

When you first start using dependency management tools like Lerna, some students may be tempted to break up dependencies very piecemically. It is possible to have cyclic dependencies — package A depends on PACKAGE B, which in turn depends on A. How did this happen? Here’s an example:

  1. Suppose you are maintaining a reusable editor package, the Editor. For better UI componentization, you split the UI part of it into the Editor-UI package.
  2. Editor-ui components require an instance of Editor, so you list Editor as a dependency on editor-ui.
  3. The Demo page for Editor wants to show an application with a full UI, so you list editor-ui as an Editor dependency.

This is where circular dependencies come in. NPM supports dependency installation in this scenario, but its presence can make dependencies difficult to understand, so we want to avoid it as much as possible. The good news here is that circular dependencies are mostly related to a less intuitive need. In the example above, the upstream Editor package relies on the downstream Editor-UI package, which can be clearly pointed out during solution review. Simply show the Demo page in the Editor-UI package instead — and if you have a circular dependency, don’t be afraid to use the “requirement is unreasonable” veto.

Multi-dependent package initialization and synchronization

We have already mentioned that Lerna Boostrap can correctly perform dependency installation and linking of multiple packages. But does this mean that a Lerna warehouse loaded with multiple packages will be able to run properly with just this command? There’s a little detail here.

If you manage multiple packages that are configured with their own build and publish commands and then merged with Lerna, you can have problems like this: The entry they specify under package.main is a build file in the form of dist/index.js, but the resulting code is generally not committed to Git these days. When new code is pulled down to run, even if the tool handles the linking correctly, there is still a chance that one of the subpackages will not be packaged successfully — in this case, manually NPM run build in the dependent package directory. Of course, in this case, after updating the source code for one package, you also need to do a build operation on that package to generate the product before the other packages can be synchronized. Although this is not difficult to understand, it often causes some unnecessary confusion, so it is specially mentioned here.

There is upstream and downstream dependency management

In real scenarios, dependency can not be completely managed by Lerna and other tools, but there is a distinction between upstream and downstream. What is this concept? The diagram below:

In general, upstream base libraries (such as Vue/Lodash, etc.) are not suitable for direct maintenance into their own macro repositories, and downstream specific business projects are mostly independent of these own dependencies, which are also outside the control of Lerna tools. At this point, we still need to go back to the basic NPM link command to establish local links. But that could lead to more problems. For example, if you manage the Editor and Editor-UI dependencies in Lerna, and the business project app depends on them, it is not difficult to link both editor and Editor-UI to the app. However, links can easily be broken. Consider the following workflow:

  1. In order to fix some problems with the editor in your app, you have updated the code of the Editor and verified it locally.
  2. younpm publishEditor and new versions of editor-UI.
  3. In your appnpm install editor editor-uiAnd commit the changes accordingly.

Boom! After this last step, not only the link between app and Editor is broken, but also the link between Editor and Editor-UI is broken. This is the downside of soft links: changes made downstream also affect upstream. In this case, you need to re-do the Lerna Bootstrap and NPM link to re-establish these dependencies, which can be quite tricky for business projects with frequent iterations. Our proposed workaround to this problem consists of two parts:

  • You can deploy a business project environment dedicated to dependency installations.
  • You can write your own link command insteadnpm link.

The former sounds like a hassle, but in fact you just need to make a copy of the app directory. Assuming that the app-deps directory is obtained after replication, then:

  • Link editor-UI and Editor to the app directory and use them to develop locally.
  • When you need to update a dependent version, do it in the app-deps directorynpm install editorCan. This does not break existing links in app projects.

Of course, the dependencies between app and app-DePs may not be completely synchronized — a problem that can be solved with the habit of pulling code. Another problem scenario is that if a downstream business project uses a non-NPM package manager such as CNPM to install dependencies, then the native Link command is likely to fail. Using the previous example, we can create the link command in the Editor project instead of NPM link:

// link.js
const path = require('path');
const { exec } = require('./utils'); // It is recommended to encapsulate childprocess. exec as a Promise

const target = process.argv[2];
console.log('the Begin linking... ');

if(! target) {console.warn('Invalid link target');
    return;
}

const baseDir = path.join(__dirname, '.. / ');
// Distinguish between relative and absolute paths
const targetDepsDir = target[0= = ='/'
    ? path.join(target, 'node_modules/my-editor')
    : path.join(__dirname, '.. / ', target, 'node_modules/my-editor');

console.log(`${baseDir} → ${targetDepsDir}`);

exec(`rm -rf ${targetDepsDir} && ln -s ${baseDir} ${targetDepsDir}`)
.then((a)= > {
    console.log('🌈 Link done! ');
})
.catch(err= > {
    console.error(err);
    process.exit(1);
});
Copy the code

Add a “link”: “node./link.js” configuration to the package.json file in the editor, and you can use the NPM link path/to/app. This link operation skips many intermediate steps and is therefore much faster than NPM’s native link and can be adapted to CNPM installed business projects.

In the case of “own-dependence → downstream business”, these two methods can basically ensure a smooth development pace. There is a problem, however, that “upstream dependencies → own dependencies” may still require some fiddling. What does that correspond to?

In general, the most upstream base library should be fairly stable. But you may also need to modify or even maintain such base libraries. For example, our Editor relies on our open source history state management library StateShot, so we need to link StateShot locally into the Editor.

Can’t this scenario continue the previous NPM Link routine? Sure, but when the upstream base library doesn’t need frequent iterations to synchronize, we recommend using the NPM pack command instead of Link to keep the dependency structure stable. How do I use this command? Just do this:

  1. Suppose you have the upstream Base package, then build it in its directory and run itnpm pack.
  2. Pack to generatebase.tgzAfter that, it runs under the Editor package managed by Lernanpm install path/to/base.tgz.
  3. lerna bootstrapEnsure that the link relationship is correct.

The benefit of Pack is that it avoids the soft link pit and can more realistically simulate a package’s release to installation process, which is useful for ensuring that a released package will be installed and used properly.

conclusion

Front-end engineering is still evolving, from the simplest NPM install to various commands and tools, I believe that the trend of the future will be to make it easier for us to maintain larger projects, and I also hope that some of the practices in this paper can be helpful to front-end students.