Dubbo-go V3 Go Module step pit

Dong Jianhui Sheng Aofei

1. Background

The problem stems from the fact that we want to make a change to dubbo-go’s Module path, using dubbo.apache.org/dubbo-go/v3 instead of github.com/apache/dubbo-go.

First of all, we did a path mapping and put a dubbogo/v3 file under dubbo.apache.org as follows:

<html>
  <head>
    <meta name="go-import" content="dubbo.apache.org/dubbo-go/v3 git <https://github.com/apache/dubbo-go>">
    <meta name="go-source" content="Dubbo.apache.org/dubbo-go/v3 git {/ dir} < https://github.com/apache/dubbo-go/tree/3.0 > {< https://github.com/apache/dubbo-go/blob/3.0 / dir} / {file} {line} # L >">
    <meta http-equiv="refresh" content="0; url=https://pkg.go.dev/dubbo.apache.org/dubbo-go/v3">
  </head>
  <body>
    <p>Redirecting to <a href="<https://pkg.go.dev/dubbo.apache.org/dubbo-go/v3>">pkg.go.dev/dubbo.apache.org/dubbo-go/v3</a>.</p>
  </body>
</html>
Copy the code

Second, we changed the module of go.mod and all the corresponding imports, and changed the module path used by all submodules to refer to Dubbo-go.

2. Problem analysis

After the above modifications, we found that CI failed when we introduced PR. After further log investigation, we determined that an error occurred when CI was running the integration test. The specific error message is as follows:

The execution logic of this section is to use Docker to build an image of the integration test content in Dubbo-Go, and start the container for testing. The images used in the packaging Dockerfile path in github.com/apache/dubbo-go/test/integrate/dubbo/go-server directory, controlled STEP identification error log, We can locate specific errors that occur in the following two steps:

#...
# STEP 9
RUN test ${PR_ORIGIN_REPO} && go mod edit -replace=dubbo.apache.org/dubbo-go/v3=github.com/{PR_ORIGIN_REPO}@${PR_ORIGIN_COMMITID} || go get -u dubbo.apache.org/dubbo-go/v3@develop
#...
# STEP 11
RUN go mod tidy && go install github.com/apache/dubbo-go/test/integrate/dubbo/go-server
Copy the code

In STEP 9, we replace dubbogo’s dependency path with go mod edit-replace, replacing it with the repository address and commit ID where the PR request was made. On top of this, an error occurred when the mirror build went to STEP11 and tried to use the go mod tidy package.

Looking back at the error log, we can see that:

Since we only specify the COMMIT ID, when the go mod actually runs, it will generate an assumed version number for us (see the problem extension at the end of this article for more information on the assumed version number) that will capture the latest valid tag for the remote repository is V1.4.1. Here the remote repository is github.com/Mulavar/dubbo-go, this is my own dubbo-go branch, which pulled the fork earlier, and its last tag was v1.4.1, and incretracted to V1.4.2, and the main version is V1. But in previous, our dubbogo module path is declared as dubbo.apache.org/dubbogo/v3, major version is v3, this led to go mod edit – before and after the replace rely on major version is v1 and v3 respectively, An inconsistency occurred and an error occurred while actually pulling the Dubbogo dependency.

3. Problem solving

In the problem Analysis section we address this problem in STEP 9 of the mirror build, i.e

# STEP 9
RUN test ${PR_ORIGIN_REPO} && go mod edit -replace=dubbo.apache.org/dubbo-go/v3=github.com/{PR_ORIGIN_REPO}@${PR_ORIGIN_COMMITID} || go get -u dubbo.apache.org/dubbo-go/v3@develop
Copy the code

In this step, when we use go mod Edit-replace, we replace a v3 go module with a V1 version of the module, resulting in a major version inconsistency error. So we only need to specify that the module main version is v3 when we replace the dependency path. We also need to add v3 to the module dependency path after go mod edit-replace.

Modify before:

go mod edit -replace=dubbo.apache.org/dubbo-go/v3=github.com/{PR_ORIGIN_REPO}@${PR_ORIGIN_COMMITID}

Tail After modification:

go mod edit -replace=dubbo.apache.org/dubbo-go/v3=github.com/${PR_ORIGIN_REPO}/v3@${PR_ORIGIN_COMMITID}

After fixing this problem, we submitted the code to check CI. In the log information printed in STEP 8, it can be seen that the dubbogo path after replacement is v3, and the version number (V3.0.0-20210509140455-2574eAB5AD0b) when pulling the package is also V3. Successfully pulled the dependency.

4. Problem expansion

4.1 Semantic Import Versioning

When we use Go Modules to build the dependencies of the project, we need to declare the version number we use for the dependencies of the Go project. Considering that compatibility has always been the most important and headache problem for developers when releasing new versions, Therefore, GO uses Semantic Import Versioning to establish standards for version numbering to ensure that each developer can specify the dependent versions to use according to their project requirements. According to go’s semantic import versioning guidelines, the version number consists of three parts:

Note: The above image and some of the content in section 4.1 are from reference 4, Go Modules In Detail.

Where Major Version indicates that this is a new, larger version, and even that this new version may not be compatible with the older version.

While Minor version indicates that this is an iteration in a large version, which is mainly used for incrementing when features are added.

Patch version is the fine-grained version update, indicating the repair of some bugs.

There are two ways to specify the major version. One is to declare directly as above, and the other is to add the version suffix at the end of the project path. For example, dubbogo 3.0 declared module is dubbo.apache.org/dubbo-go/v3, where v3 is a major version declaration.

The pseudo – version 4.2

Go encourages us to specify explicit version numbers when using dependencies, such as the dependency on cloud.google.com/go declared in Dubbogo’s go.mod:

But considering that in some cases we want to use an unreleased version of a module that only has a known COMMIT ID, such as the github.com/Microsoft/go-winio dependency declared by Dubbogo, Instead of the real version, you can use a pseudo version number. Assume that the format of the version number is

Vx.y.(Z+1)-0. Yyyymmddhhmss - abcDEF123456 // (3) vx.y.(Z+1)-0. vX.Y.(Z+1)-0.yyyymmddhhmmss-abcdef123456+incompatible // (4) vX.Y.Z-pre.0.yyyymmddhhmmss-abcdef123456 // (5) vX.Y.Z-pre.0.yyyymmddhhmmss-abcdef123456+incompatibleCopy the code

You can see that the pseudo-version is divided into three parts by a short slash. The first part, X.Y.Z, is the same as the one mentioned in Section 4.1: major version, Minor Version, and Patch version. The second part is a timestamp in the format YYYYMMDDHHMMSS, and the third part is a 12-bit commit hash that can be specified manually or, if not, generated automatically by GO.

The rules for generating pseudo-version go are as follows:

Query the latest tag of the main version of the project (if the project dependency path does not have an explicit v2/v3 suffix, the default version is V1)
If there is a tag, the patch version is incremented by the latest tag
No tag, automatically generated according to the project path with the suffix version number (such as V3 automatically generated a 3.0.0pseudo-version)

Following this rule, looking back at the previous problem analysis and problem solving, we can see why the pseudo-version of Dubbogo was V1.4.2 in the first place, precisely because Go thought that dubbogo after Replace depended on the main version being V1. To find the latest tag under the major version, and incremented the patch version, resulting in inconsistency between the previous and the previous major versions. When we added v3 to the version path, GO could not find the tag under the corresponding major version, so it automatically generated V3.0.0 for us, which passed CI.

4.3 Go module nesting

Unlike Java, GO actually has no concept of submodules. Even sometimes, we’ll see multiple Go modules in a repo, such as a project’s root directory with a go.mod, and subdirectories with go.mod.

This is called a nested module in Go Modules, rather than a parent module, where two modules have no relationship and are independent of each other.

So when does a single repo need multiple modules? In general, there are two situations in which we would consider using a single-repo, multi-module form of development.

A nested module changes frequently and requires frequent releases.

The nested modules in the root module only depend on a fixed version of the root module.

In both cases, the essence is that there is no strong version binding between two modules, but they need to be placed under one RPEO for some other reason, resulting in a single-repo multi-module situation.

4.4 Parsing dubbogo Static Mapping files

Dubbo-go uses static file mapping to realize module redirection. The contents of the static file are as follows:

The core parts are the meta tags go-import and go-source.

<html>
  <head>
    <meta name="go-import" content="dubbo.apache.org/dubbo-go/v3 git <https://github.com/apache/dubbo-go>">
    <meta name="go-source" content="Dubbo.apache.org/dubbo-go/v3 git {/ dir} < https://github.com/apache/dubbo-go/tree/3.0 > {< https://github.com/apache/dubbo-go/blob/3.0 / dir} / {file} {line} # L >">
    <meta http-equiv="refresh" content="0; url=https://pkg.go.dev/dubbo.apache.org/dubbo-go/v3">
  </head>
  <body>
    <p>Redirecting to <a href="<https://pkg.go.dev/dubbo.apache.org/dubbo-go/v3>">pkg.go.dev/dubbo.apache.org/dubbo-go/v3</a>.</p>
  </body>
</html>
Copy the code

【go-import】 Content is divided into three parts:

dubbo.apache.org/dubbo-go/v3: The module declaration for this project.

Git: the version control tool you use.

Github.com/apache/dubb… : Tell Go Get where to find the source code for this project.

【go-source】 The go-source is used to generate a specific go doc (now pkg.go.dev) document for the project. The first two parts are the same as the go-import, which is the module declaration and version control tool for the project. The latter two parts respectively play the following roles:

Github.com/apache/dubb… : Declares the location of the project’s source code.

Github.com/apache/dubb… : Maps documents and code to help us click on the document’s directory tree and jump to the corresponding content.

Such as in the PKG. Go. Dev/dubbo apach…

It will jump to the corresponding document for the corresponding file.

Welcome students who are interested in the Apache/Dubbo-GO project to join the exchange group by scanning the code [or search the nail group number 31363295] :

At the same time, please scan the following WECHAT QR code and follow the Dubbogo community wechat official account:

5 the author

Dong Jianhui (github @Mulavar), a recent graduate of VLIS laboratory at Zhejiang University, is currently working as a server development engineer at Ape Tutoring. Aofei (github @Aofei) is the author of the Goproxy. cn project and a contributor to go source code (such as mod tool).

The resources

1 “The Go Blog – Using Go Modules” blog.golang.org/using-go-mo…
2 Go Modules Reference golang.org/ref/mod
3 “Go how Module launch v2 and above version” blog.cyeam.com/go/2019/03/…
4 “Go Modules” www.sulinehk.com/post/go-mod…