Cross-compile the Monorepo project with NPM7 and TypeScript

Monorepo

Monorepo is a way to manage project code by managing multiple modules/packages in a project repository (REPO), as opposed to the usual repO for each module.

There are many large open source projects using this approach, such as Babel,React, etc.

Why Monorepo

In large class library development, an ecosystem is formed, which contains many sub-packages. For example, in the React ecosystem, there are many NPM packages such as react, react-DOM, and React-server. To the user, each package is an independent NPM package, but to the developer of the package, the packages are designed to be independent of each other, but the development and testing process, there are many links. Such as:

Code specifications and code styles should be unified.
When A package depends on B Package, if B has an upgrade, it should update A’s dependency file and test whether A still meets expectations.
When A package relies on B Package, if A function of A needs to be modified and B needs to be supported, B needs to be modified at the same time. If A and B are in different warehouses, modification and debugging will be extremely troublesome.
.

Based on the above factors, monorePO will be your ultimate destination if you develop a package that is not a completely isolated package, but rather a cluster of packages with dependencies.

How Monorepo

Monorepo is essentially an exercise in an idea of how code is organized. So there are best practices in many dimensions, such as third party dependency management, version number management, DevOps, testing, and so on, all with different solutions that can’t be covered in a single article. This article focuses on how to organize dependencies between packages using Workspaces and automatically compile cross-dependency packages with typescript to achieve a minimally-usable Monorepo solution.

Structure of the sample

As shown below, we build a Monorepo with only x-CLI and X-core packages. In x-CLI, we use x-core functions to generate a dependency.

. ├ ─ ─ node_modules / ├ ─ ─ the README. Md ├ ─ ─ package - lock. Json ├ ─ ─ package. The json ├ ─ ─ packages │ ├ ─ ─ x - cli │ │ ├ ─ ─ lib │ │ │ ├ ─ ─ Cli. Which s │ │ │ ├ ─ ─ cli. Js │ │ │ ├ ─ ─ cli. Js. Map │ │ │ ├ ─ ─ the main, which s │ │ │ ├ ─ ─ the main, js │ │ │ └ ─ ─ the main, js. Map │ │ ├ ─ ─ Package. The json │ │ ├ ─ ─ the SRC │ │ │ ├ ─ ─ cli. Ts │ │ │ └ ─ ─ main. Ts │ │ └ ─ ─ tsconfig. Json │ └ ─ ─ x - core │ ├ ─ ─ lib │ │ ├ ─ ─ The index, which s │ │ ├ ─ ─ index. The js │ │ └ ─ ─ index. The js. Map │ ├ ─ ─ package. The json │ ├ ─ ─ the SRC │ │ └ ─ ─ index. The ts │ └ ─ ─ tsconfig. Json ├ ─ ─ Tsconfig. Build. Json └ ─ ─ tsconfig. JsonCopy the code

At the beginning of the project, x-CLI and X-Core had not yet been released. How did X-CLI reference X-Core?

Use relative paths? If a relative path is used to reference the X-core in the X-CLI, the relative paths are inconsistent after the X-CLI is released, which may cause errors.
The use of NPM link? It’s achievable, but not elegant enough.

The industry already has a mature solution for such scenarios using Workspaces. Yarn was the first to support workspaces capabilities, so in many articles on Monorepo, yarn is recommended as a package management tool. However, starting with NPM 7, NPM also supports workspaces, which, although not as powerful as YARN, is almost sufficient. This article will use NPM as a package management tool.

Workspaces

This is a new property added in NPM 7. Workspaces is an optional field with an array of file Pattern values. When you execute install, NPM will go through all the matching addresses and soft connect the following packages to the project’s top-level node_modules folder, as described in this article.

In the following example, any subfolder under the./ Packages folder that contains a package.json file is considered a workspace.

We open the package.json file and add the workspaces field:

/* package.json */

{
  "name": "npm-ts-workspaces-example"."private": true."workspaces": ["packages/*"]}Copy the code

NPM install is installed in the root directory. After this, we can see that all dependencies are installed in the root directory, including the x-CLI and x-core dependencies, and x-CLI and X-core themselves are soft connected to node_modules.

At this point, we can use x-Core in x-CLI just like any other dependency package.

Cross dependencies

I refer to cross-references between packages as cross dependencies.

Runtime cross dependency issues

Using Workspaces solved many of our runtime package reference problems. Because of the cross-dependencies referenced with soft links, you can refer to the correct DIST file either at compile time or at run time. We can analyze it:

X-cli references @example/x-core

import { awesomeFn } from "@example/x-core";
Copy the code

@example/ X-core soft links to Packages/X-core

@ example/[email protected] - > / Users/study/NPM - ts - machine-specific - example/packages/x - the coreCopy the code

The x-core exported file is the TS build product lib/index.js

{
  "name": "@example/x-core"."main": "lib/index.js"."types": "lib/index.d.ts"
}
Copy the code

So all you need is for the x-core build product lib/index.js to be correct and the entire dependency to be correct.

Cross dependencies at compile time

It looks great, but if you try it, you’ll always find something you don’t like.

If I modify the X-Core, introduce an incompatible change, and forget to run the X-CLI before sending the package, it’s easy to cause bugs in the X-CLI. After all, the X-CLI relies on x-Core, but doesn’t know when x-core changes and doesn’t show dependencies during compilation.
During debugging, x-core modifications cannot be synchronized to the X-CLI in a timely manner. Or build x-CLI manually, very primitive; Either watch the entire X-Core folder in the X-CLI, which on the one hand is extremely performance intensive, and on the other hand, gets out of control as cross-dependencies increase.

The ideal state of our imagination would be:

When I was developing the X-CLI, there was a feature that had to change the x-core source code. At this point, I changed the x-Core source code, and x-CLI should listen for the changes in real time, recompile the code, and take effect in the development environment in real time.

Compile cross dependency

After some checking, we can use references and Composite configurations in TypeScript to solve the PROBLEM of TS cross-dependency compilation.

First of all, let’s clarify a few concepts:

TypeScript project

Any folder that contains a valid tsconfig.json file is a TypeScript project.

references

References takes an array of objects, and each referenced path property can point to the directory containing the tsconfig.json file or directly to the configuration file itself.

Here’s what happens when you reference a project:

The module in the import reference project actually loads itThe outputDeclaration document (.d.ts).
If the referenced project generates oneoutFile, so the output file.d.tsThe declaration in the file is visible to the current project.
The build pattern (below) automatically builds the referenced project as needed.

When you break it down into multiple projects, it significantly speeds up type checking and compilation, reduces the memory footprint of the editor, and improves how programs are grouped logically.

composite

The referenced project must enable the new Composite setting. This option is used to help TypeScript quickly determine the location of the output file for the reference project. If the Composite tag is enabled, the following changes occur:

forrootDirSet to include if not explicitly specifiedtsconfigDirectory of files
All implementation files must match a certainincludeMode or infilesList in the array. If this restriction is violated,tscYou will be prompted which files are not specified.
Must be opendeclarationOptions.

conclusion

The directory that contains the tsconfig.json file is a TypeScript project.
References can refer to other TypeScript projects, which are built automatically at build time.
The referenced TypeScript project is setcompositeField, indicating that references are allowed.

implementation

Since x-core is the referenced package, we add the configuration to tsconfig.json:

/* tsconfig.json */

{
  "compilerOptions": {
    ...
    "composite": true
  }
}

Copy the code

Add dependency configuration in X-CLI:

/* packages/x-cli/tsconfig.json */

{
  "compilerOptions": {
    "rootDir": "src"."outDir": "lib"
  },
  "references": [{ "path": ".. /x-core"}}]Copy the code

Compilation.

An error occurs when TSC is compiled in the X-CLI folder.

I looked into this mistake for a long time and finally found a sentence:

To be compatible with existing build processes, TSC does not automatically build dependencies unless the — Build option is enabled.

That is to say, even though TSC is a compilation of TS, the effect of adding –build option is still different from that of not adding –build option, 🤦♀️.

Let’s try again:

tsc --build --verbose
Copy the code

As you can see, since the DIST file in x-core does not exist, TSC automatically does x-core compilation first and x-CLI compilation later, exactly as expected.

Watch. Listen for dependent changes and compile automatically.

We would prefer to have x-CLI detect and automatically compile changes to X-Core during x-CLI development. Let’s see if this works by adding the –watch field.
```
tsc --build --verbose --watch
Copy the code
```
As you can see, x-CLI TSC compiles automatically when x-core is updated, exactly as expected.

! [](/Users/pengchaoyang/Library/Application Support/typora-user-images/image-20210523170113922.png)

At this point, the monorePO development workflow that we want to support cross-dependencies is set up.

To optimize the

Incremental compilation

In listening mode, every change causes the associated TypeScript project to be completely recompiled, which in many cases is not only necessary, but also leads to CPU and memory overruns. You can enable TypeScript incremental compilation. The first compilation generates a tsconfig.tsbuildinfo file that records the signature information of each TS file, and subsequent updates only update the changed files, greatly improving compilation efficiency.

{
  "compilerOptions": {
    "incremental": true}}Copy the code

Common configuration

One of the problems Monorepo resolves itself is to reuse various configuration files throughout the repository, including ESLint, tsconfig, etc., on the one hand to reduce redundancy and on the other hand to eliminate inconsistencies and ensure that the code style is consistent throughout the repository.

Here we can promote the common configuration in the TSconfig file of x-CLI and X-core to the root directory of the project. Each sub-project only inherits the configuration of the root directory, which can ensure the uniqueness of configuration items.

/* /tsconfig-base.json */
{
  "compilerOptions": {
    "incremental": true."target": "es2019"."module": "commonjs"."declaration": true."sourceMap": true."composite": true."strict": true."moduleResolution": "node"."baseUrl": "./packages"."esModuleInterop": true."skipLibCheck": true."forceConsistentCasingInFileNames": true}}Copy the code

/* /packages/x-cli/tsconfig.json */
{
  "extends": ".. /.. /tsconfig.json"."compilerOptions": {
    "rootDir": "src"."outDir": "lib"
  },
  "references": [{ "path": ".. /x-core"}}]Copy the code

conclusion

In the process of developing the NPM package of jsonschema ecology, we encountered problems related to multiple package cross-reference, cross-compilation, inconsistent configuration, version number, and third-party dependence. We are also gradually exploring the solution of Monorepo. In the recent trial process, we have obtained good benefits. For students with similar scenarios, it is worth trying. At the same time, we still have a lot of room for optimization in the whole monorepo workflow, welcome to share and exchange more.

Reference documentation

www.tslang.cn/docs/handbo…
Stackoverflow.com/questions/5…

Join us

We have toutiao, Douyin, Watermelon video, Huoshan Video, TopBuzz, Mantis Shrimp, education, security, gaming and many more business lines to choose from.

Push through: [email protected]

Internal push code: RWQMBNH

Title format: school recruit/internship/clubs recruit – name – intention post – city query page: job.toutiao.com/s/eC53ysY

Job Details:

Participated in the company’s search business research and development Hybrid/Wap/Web;
Participate in the construction of search cloud platform, provide full-link configuration, Centralization and content introduction;
Participate in the construction, operation and maintenance of high-concurrency SSR service, and conduct in-depth search for performance optimization and stability construction;
Participate in the construction of search micro-front-end system and develop tool chain to create DevOps solutions;
Participate in the development of visual editor and other tools, and use technical means to improve the efficiency of search research and development;
Participated in the search of node.js basic function abstraction, and developed basic library and OpenAPI.

Cross-compile the Monorepo project with NPM7 and TypeScript

Monorepo

Why Monorepo

How Monorepo

Structure of the sample

Workspaces

Cross dependencies

Runtime cross dependency issues

Cross dependencies at compile time

Compile cross dependency

TypeScript project

references

composite

conclusion

implementation

To optimize the

Incremental compilation

Common configuration

conclusion

Reference documentation

Join us

Related Posts

How to do intelligent prompts and validation of interfaces based on TypeScript and Swagger backend documents

2021 Conclusion: My ordinary road

React learn how to use React and its advanced features