Wechat search [brain into fried fish] follow this fried fish with fried liver. In this paper, making github.com/eddycjy/blo… Already included, there are my series of articles, materials and open source Go books.

Hello everyone, I am an ordinary decocted fish. I was very lucky to invite @Aofei, the author of Goproxy.cn, to Go night on Thursday to read to us the technology sharing of the 61st issue of Go Modules, Go Module Proxy and Goproxy.cn.

This night reading sharing by @Sheng Aofei is a good interpretation of Go Modules, which is close to engineering practice. I certainly hope to share more of this knowledge with you. Therefore, with this article, you can also pay more attention to Go night reading. Every week, I will share technical topics related to Go through zoom online live broadcast, hoping to be helpful to you.

preface

Modules released in Go 1.11 open a new door for Go developers. The idealized dependency management solution makes Go move towards the first dependency utopia (Deptopia) in the history of computer programming. Along with the Module comes the Module Proxy Protocol, through which we can implement the Go Module proxy, or dependency mirror.

The Go 1.13 release brought a lot of improvements to the module, so the module alignment is the biggest change developers will feel directly in the Go 1.13 release. The problem is that the GOPROXY environment variable in Go 1.13 has a default value of proxy.golang.org that is not accessible in mainland China. After a heated discussion in Golang/Go #31755 (some even took the term to “free world” levels), Ultimately, the Go core team was unable to provide Chinese developers with an official module agent that could be accessed in mainland China.

To help Chinese Go developers develop better in the future, Quniuyun launched a non-profit project, GoProxy.cn, whose goal is to provide a free, reliable, continuously online, CDN-accelerated module proxy for Gopher in China and elsewhere in the world. The future is expected to be modular, so the sooner Go developers can get into modules the sooner they can get into the future.

If you were reluctant to cut into Go 1.11 and Go 1.12 due to the lack of modules, Go 1.13 is a safe place to start. This post will discuss how to use modules and module proxies, and the pitfalls you will encounter in their use. It will also explain how to quickly set up your own private module proxies, and briefly introduce Goproxy.cn launched by Qiuniuyun and why its emergence is important for Chinese Go developers.

directory

  • Go Modules introduction
  • Quickly migrate projects to Go Modules
  • Common pit with Go Modules
    • Pit 1: Check whether the project has Go Modules enabled
    • Pit 2: Manage environment variables for Go
    • Pit 3: Migrate to Go Modules from DEP, Glide, etc
    • Pit 4: Pull private modules
    • Pit 5: Update existing modules
    • Pit 6: Major version number
  • Introduction to Go Module Proxy
  • Goproxy China (Goproxy. Cn)

Go Modules introduction

Go Modules (formerly VGO) is an idealized language-like dependency management solution pushed by the Go Team (Russ Cox). It was released with Go1.11 and has been optimized and adjusted a lot in Go1.13. If you want to use Go Modules but are still stuck in 1.11/1.12, upgrading is highly recommended.

Three key words

Strong push

First of all, it’s not nonsense, because Go modules are really forced, as follows:

  • Before: We all know that before Go Modules, there was a project called DEP, which was an official pilot project for Go, and also aimed to address the dependency management shortcomings of Go. Before Russ Cox introduced Go Modules, almost everyone in the community thought THAT DEP was definitely the official dependency management solution for Go in the future.
  • Then: Russ Cox went ahead and launched Go Modules, which no one thought would Go overboard, and the community exploded in an instant. Everyone agreed that the Go Team was too domineering and dictatorial to even say hello. I remember a lot of people talking to Russ Cox on the Internet, and all kinds of experts who relied on management solutions came up to give their opinions, and at one point the discussion went beyond Go to other languages.

idealized

It’s ideal in that it mandates semantic version control, as follows:

  • The Go modules are so malicious that if your Tag doesn’t follow semantic versioning it ignores your Tag and generates an assumed semantic versioning version number for you based on your Commit time and hash value.
  • Go Modules also defaults to the idea that as long as your main version number stays the same, there must be no Breaking changes in the module version, because that’s how semantic version control works. Isn’t that idealistic?

Language level:

I actually made this up myself. I just think that Go Modules is designed as a language-level feature. For example, if your main version number changes, the import path in your code changes too. It considers two modules with different major versions to be completely different modules. Also, Go Moduels is designed so tightly into the entire Go command that it’s ubiquitous, which is why I’d say it’s a bit of a language-level feature, though not too rigorous.

Who’s pushing the Go Modules

What about Russ Cox, whom many of you probably know, who is by far the most committed person on the Go project, more than twice the number two.

Russ Cox is still at the helm of Go. (You should know that The former head of Go was Rob Pike, but I heard he left the COUNTRY because he didn’t like The Trump administration. He’s getting old, so he’s stepping down, but he’s still involved with Go.)

Russ Cox has a very strong personality and a very unique way of looking at things, which is why the concept of Go Modules was so widely responded to as soon as he proposed it. It’s been pushed, but it turns out that the current Go Modules do really well, so this shows that a certain degree of “dictatorship” is acceptable, at least to ensure that a project is more focused in one direction.

Anyway, Go Modules are now an integral part of the Go language.

GOPATH

One of the purposes of Go Modules is to solve GOPATH’s problems, which is to say, get rid of GOPATH.

Opt-in

Go modules are still in the opt-in phase, which means you can use them if you want, and you can’t use them if you don’t. But Go2 is likely to be mandatory in the future.

“module” ! = “package”

One thing that needs to be corrected is that the terms “module” and “package”, that is, the terms “module” and “package” are not equivalent. They are “collection” and “element”. “module” contains “package”, and “package” belongs to “module”. A module is a collection of zero, one, or more packages.

Go Modules Related attributes

go.mod

Module example.com/foobar go 1.13 require (example.com/apple v0.1.2 example.com/banana v1.2.3 example.com/banana/v2 V2.3.4 example.com/pineapple v0.0.0-20190924185754-1b0db40DF49a) exclude example.com/banana v1.2.4 replace example.com/apple v0.1.2 => example.com/rda v0.1.0 replace example.com/banana => example.com/hugebananaCopy the code

Go.mod is the most important file required for a Go Moduels-enabled project. It describes the meta information for the current project (that is, the current module), and each line begins with a verb, which currently has the following five verbs:

  • Module: Defines the module path for the current project.
  • Go: Used to set the expected GO version.
  • Require: Used to set a specific module version.
  • Exclude: Used to exclude a particular module version from use.
  • Replace: Used to replace one module version with another.

The filling format here is basically package reference path + version number. In addition, the special one is Go $version. At present, from the code of Go1.13, it is only used as an identifier.

go.sum

Go.sum is a file similar to the gopkg.lock file of DEP. It lists all the module versions that the current project relies on directly or indirectly, and specifies the SHA-256 hash values of those module versions to ensure that go does not tamper with those module versions that the project relies on in future operations.

Example.com/apple v0.1.2 h1: WXkYYl6Yr3qBf1K79EBnL4mak0OimBfB0XUf9Vl28OQ = example.com/apple v0.1.2 / go mod H1: xHWCNGjB5oqiDr8zfno3MHue2Ht5sIBksp03qcyfWMU = example.com/banana v1.2.3 H1: qHgHjyoNFV7jgucU8QZUuU4gcdhfs8QW1kw68OD2Lag = example.com/banana v1.2.3 / go mod H1: HSdplMjZKSmBqAxg5vPj2TmRDmfkzw + cTzAElWljhcU = example.com/banana/v2 v2.3.4 H1: zl/OfRA6nftbBK9qTohYBJ5xvw6C oNKizR7cZGl3cI = example.com/banana/v2 v2.3.4 / go mod h1:eZbhyaAYD41SGSSsnmcpxVoRiQ/MPUTjUdIIOT9Um7Q= ...Copy the code

We can see that there are two possible module paths:

Example.com/apple v0.1.2 h1: WXkYYl6Yr3qBf1K79EBnL4mak0OimBfB0XUf9Vl28OQ = example.com/apple v0.1.2 / go mod h1:xHWCNGjB5oqiDr8zfno3MHue2Ht5sIBksp03qcyfWMU=Copy the code

The former is the hash value for the Go Modules package file zip, while the latter is the hash value for go.mod. Either they exist at the same time, or just go.mod hash.

When there is no Zip hash, Go omits the zip hash when it decides that a module version is definitely not needed, and there is no Zip hash, just go.mod hash.

GO111MODULE

This environment variable is the Go modules switch and has the following parameters:

  • Auto: Enable go Modules only if the project includes the go.mod file, which is still the default in Go 1.13, see golang.org/issue/31857.

  • On: Mindlessly enable Go Modules, recommended Settings, default in future versions, making GOPATH history.

  • Off: Disables the Go Modules.

GOPROXY

This environment variable is used to set the Go module proxy as follows:

  • Its value is a list of Go Module proxies separated by commas (,) (explained later)

    • Function: Enables Go to quickly pull modules from mirror sites without traditional VCS mode. It has a default: https://proxy.golang.org, direct, but it is a pity that proxy.golang.org in China can’t access, so it is recommended to use goproxy. Cn as an alternative, you can perform statements: Go env – w GOPROXY = https://goproxy.cn, direct.

    • Off: Forbid Go to use any Go Module proxy in subsequent operations.

Above, we can see “direct” in the value list. What does it do? In fact, the “direct” in the value list is a special indicator, which is used to tell Go to fetch the source address of the module version (such as GitHub). When the previous Go Module proxy in the value list returns a 404 or 410 error, Go automatically tries the next one in the list. If the value is direct, the source is returned. If the value is EOF, the source is terminated and invalid version: Unknown Revision is thrown. The error.

GOSUMDB

The value is a Go Checksum database that ensures that the module version data is not tampered with when Go pulls the module version, either from the source site or through the Go Module proxy. The value can also be off. That is, Go cannot verify the module version in subsequent operations

  • Format 1:

    + .

  • Format 2:

    +
    .

  • Have a default value: sum.golang.org (it doesn’t follow the above format because Go does something special with the defaults).

  • Can be proxyed by the Go Module Proxy (see Proxying a Checksum Database).

  • Sum.golang.org cannot be accessed in China. Therefore, it is recommended to set GOPROXY to goproxy.cn because goproxy.cn supports the sum.golang.org proxy.

Go Checksum Database

The Go Checksum database is used to protect Go from being pulled from any source to an illegal version of the Go module. The Go Checksum database is used to protect Go from being pulled from any source to an illegal version of the Go module.

If you are interested, you can check out Proposal: Secure the Public Go Module Ecosystem, which describes the algorithm mechanism in detail. If you want to simplify things, you can also check out Go Help Module-Auth.

GONOPROXY/GONOSUMDB/GOPRIVATE

These three environment variables are used when the current project relies on a private module, that is, a module specified by GOPROXY, the Go Module Proxy, or a module specified by GOSUMDB, which the Go Checksum Database cannot access

  • All three values are a module path prefix separated by a comma (,), matching the same rules as path.Match.
  • GOPRIVATE is the special one, and its value will be the default for GONOPROXY and GONOSUMDB, so the recommended best posture is to just use GOPRIVATE.

In terms of use, For example, GOPRIVATE=*.corp.example.com indicates that all module versions whose paths are prefixed with the next level domain name of corp.example.com (such as team1.corp.example.com) will not pass through the Go Module The proxy and Go checksum database do not include corp.example.com itself.

Global Caching

The global cache for Go Modules is as follows:

  • Only one copy of data of the same module version is cached and used by all other modules.
  • Currently, all module version data is cached$GOPATH/pkg/modand$GOPATH/pkg/sumUnder, the future may move to$GOCACHE/mod and$GOCACHE/sumUnder (may be in when$GOPATHAfter elimination).
  • You can usego clean -modcacheClear all cached module version data.

In addition, GOCACHE is no longer allowed to be set to off after Go1.11. I think this is also in preparation for moving the module data cache, so you should adapt as soon as possible.

Quickly migrate projects to Go Modules

  • Step 1: Upgrade to Go 1.13.
  • Step two: Get GOPATH out of your mind completely and step into the future early.
    • Change the GOBIN path (optional) :go env -w GOBIN=$HOME/bin.
    • Open Go Modules:go env -w GO111MODULE=on.
    • Set GOPROXY:go env -w GOPROXY=https://goproxy.cn,direct# is required in China because its default value is blocked.
  • Step 3 (optional): Reorganize all your projects according to your preferred directory structure.
  • Step 4: Execute at the root of your projectgo mod init <OPTIONAL_MODULE_PATH>To generate the go.mod file.
  • Step 5: Find a way to convince everyone around you to do the first four steps.

Change of go Get behavior after migration

  • withgo help module-get 和 go help gopath-getTo understand the behavior of Go Get in the Go Modules enabled and unenabled states respectively
  • withgo getPull new dependencies
    • Pull the latest version (tag is preferred) :go get golang.org/x/text@latest
    • pullmasterBranch’s latest COMMIT:go get golang.org/x/text@master
    • Pull commit with tag v0.3.2:Go get golang.org/x/[email protected]
    • Pull commit with hash 342b231, which will be converted to v0.3.2:go get golang.org/x/text@342b2e
    • withgo get -uUpdate existing dependencies
    • withgo mod downloadDownload all dependencies specified in the go.mod file
    • withgo mod tidySort out existing dependencies
    • withgo mod graphLook at existing dependency structures
    • withgo mod initGenerate go.mod files (the only subcommand in Go 1.13 that can generate go.mod files)
  • withgo mod editEdit the go.mod file
  • withgo mod vendorExport all existing dependencies (in fact, Go Modules are de-emphasizing the concept of Vendor)
  • withgo mod verifyVerifies whether a module has been tampered with

Here we notice two special points:

  • First point: why “a COMMIT with a hash of 342b231 will eventually be converted to V0.3.2”. This is because the @342b2e commit is set, but because the Go modules compare to the tag, if the corresponding COMMIT is found to be associated with the tag, it will be converted.
  • Second point: why notgo mod vendorBecause Go Modules are downplaying the concept of Vendor, it’s likely that Go2 will be removed.

Common pit with Go Modules

Pit 1: Check whether the project has Go Modules enabled

Pit 2: Manage environment variables for Go

Go1.13 added go env -w to write environment variables to the path returned by os.UserConfigDir.

Pit 3: Migrate to Go Modules from DEP, Glide, etc

This is mainly refers to the old dependency package management tool (DEP/Glide, etc.) migration, because of the BUG will not go through the proxy, there are two solutions:

  • Manually create a go.mod file and execute Go mod Tidy to supplement it.
  • GOPROXY is not used.

Pit 4: Pull private modules

Here I mainly want to involve two knowledge points, as follows:

  • GOPROXY doesn’t have access to anyone’s private modules, so you can rest assured that security is fine.
  • In addition to setting the address of the module proxy, GOPROXY also needs to add a special identifier “Direct” to successfully pull the private library.

Pit 5: Update existing modules

Pit 6: Major version number

Introduction to Go Module Proxy

Here, the function of Go Module Proxy (left) and its corresponding protocol interaction process (right) are emphasized again. Interested partners can take a look.

Goproxy China (Goproxy. Cn)

This section mainly introduces the practical operation of Goproxy and some Q&A and recent situation of Goproxy.cn, as follows:

Q&A

Q: What will happen if the Chinese Go language community doesn’t have our own Go Module Proxy?

* * A: ** In Go 1.13 both environment variables GOPROXY and GOSUMDB have default values that are inaccessible in China, despite my efforts at golang.org/issue/31755, But in the end, we still could not find a perfect solution for Chinese Go language developers. So from now on, all Chinese developers of Go language, as long as they use Go Modules, must first modify GOPROXY and GOSUMDB before they can normally use Go for development. Otherwise, even the simplest program may not be able to run (as long as it relies on third party modules).

Q: The main reason why I set up Goproxy China (goproxy.cn)?

* * A: ** In fact, earlier this year, I also tried to request Go Team in golang.org/issue/31020 to find a way to prevent the default value that GOPROXY would have at that time from being accessed normally in China, but Go Team seemed unable to do anything about it. That’s why I decided to set up Goproxy.cn. Since no one else can help, we’ll have to do it ourselves, for no other reason than to make it easier for everyone to use Go modules in the future.

At first, I proposed the idea of setting up Goproxy.cn with Xu Shu (founder and CEO of Qiniuyun Xu Shiwei) of Qiniuyun. The purpose was to try it out, but Unexpectedly, Xu Shu recognized my idea and expressed his willingness to work together with me without more than a minute of consideration. At that time, I happened to be writing my graduation thesis, so AFTER the development of the project, I did not do the handover with Qiniuyun, but kept running on my personal server. It wasn’t until a goproxy.cn attack cost me more than $100 in an afternoon that I realized I really couldn’t do this kind of project alone. Individuals to do not rely on the project, in case of more people rely on this project, the project and then anything, it will give everyone 􏰁 into unnecessary loss. So I hurriedly and seven niuyun made the handover, the goproxy.cn completely handed over seven niuyun, even the domain name transfer to go.

Recent situation

  • Goproxy China (goproxy.cn) is currently the most reliable Go Module proxy in China (not to brag about it).
  • For Chinese Go language developers tailored 􏰁, support agent GOSUMDB default value, after global CDN acceleration, high availability, can be applied into the company’s complex development environment, can also be used as upstream agent.
  • An open source, non-profit project supported by infrastructure provided free of charge by Qiniuyun, China’s trusted cloud service provider.
  • The goal is to provide a free, reliable, continuously online Go Module proxy via CDN plus 􏰀 for Go language developers in China and the world.
  • The domain name has been put on record by seven niuyun (Shanghai ICP 11037377-56).

situation

Here is the storage size, mainly for module package code, and generally speaking, the code will not be large, 0-10MB, 10-50MB accounted for the largest, is understandable, but more than 100MB module package code is exaggerated.

At this time, I mainly show the recent network data of Goproxy. cn. I believe that the future will be higher and higher, which is worth looking forward to.

Q&A

Q: How to solve the problem that the final target module fails to be pulled when Go 1.13 pulls the module version from GitLab because Go mistakenly looks for the target module version in an unexpected path?

* * A: ** The
set with Goget in GitLab had some problems, causing Go 1.13 to incorrectly identify the module’s path. This is a Bug that has been fixed in the new version of GitLab. Details can be found at github.com/golang/go/i… This Issue. Then the current solution in addition to upgrading the version of GitLab, you can also refer to github.com/developer-l… This reply.

Q: Can you use Go Modules to rely on two or more minor versions of the same module at the same time?

**A: ** No, Go modules can only rely on two or more major versions of A module at the same time (with different major versions). For example, you can rely on both example.com/[email protected] and example.com/foobar/[email protected] because their module paths are different. Go Modules Specifies that when the major version number is not v0 or v1, the major version number must appear explicitly at the end of the module path. However, relying on two or more smaller versions at the same time is not supported. For example, if module A is directly dependent on both module B and C, and module A is directly dependent on V1.0.0 of module C, and module B is directly dependent on V1.0.1 of module C, So eventually Go Modules selects v1.0.1 of module C for module A instead of v1.0.0 as specified in module A’s go.mod file.

This is because Go Modules assumes that as long as the main version number stays the same, everything else can be upgraded to the latest version. However, if the project Break falls due to the latest result, the Go modules will Fallback to the previous older version, such as v1.0.0 in the previous example.

Q: When does the Hash check for a module version in the go.sum file come in pairs, and when does it only have one line?

**A: ** Normally, the Hash check for A module’s version in the go.sum file will have two lines, the first line is the Hash check for the module’s ZIP file and the second line is the Hash check for the module’s go.mod file. However, there are cases where only one line of the module’s go.mod file Hash check will appear, but not the module’s ZIP file Hash check. This happens when the Go Modules decides that you don’t need the module at all for your current project. The module’s ZIP file is never downloaded, so there’s no need to Hash it, just the module’s go.mod file, because the go.mod file is needed when digging into project dependencies.

Q: Can you explain the behavior and usage of the replace verb in the go.mod file in more detail?

* * A: ** The replace verb replaces one module version with another. This is a substitution between the module version and the module path. The “=>” identifier precedes the “module path” of the module version to be replaced. At the back of the content is the goal to replace the module version, namely path, the path can be a relative path to the local disk, can also is the absolute path to a local disk, also can be a network path, but the target path will not be in the future as you in your project code “import path (import path)”, The “import path” in the code is prefixed by the “module path” you replaced with the target “module version”.

Also note that Go Modules do not support relative paths in the import path. For example, if project A depends on module B, for example, module B’s “module path” is example.com/b, and its disk path is ~/ B, in project A’s go.mod file you have A line replace example.com/b=>~/ B, Then “import subgrade” in the code in project A is import”example.com/b” instead of import”~/b”, and the rest is done automatically by Go modules for you.

And then, as I mentioned in the share, the verbs exclude and replace only apply to the current main module, the current project, those other module versions that it depends on and if there’s a version of the module that you’re replacing, Go modules will still pull the version of your module to be replaced for the version of the module you depend on.

For example, if project A is directly dependent on modules B and C, and module B is directly dependent on module C, then your replace C =>~/some/path/ C in project A’s go.mod file will only affect the code written in project A. Module B uses the same C you replaced with before, not ~/some/path/c.

conclusion

After the release of Go1.13, more and more people come into contact with Go Modules and Go Module Proxy, and I often see various partners consulting in various groups, including myself who also contributed several “pits”. Therefore, I think the technology sharing of Go Modules, Go Module Proxy and Goproxy.cn by AOfei is of great practical significance. If you have any further suggestions or questions, please feel free to discuss them.

Finally, I would like to thank the people behind Goproxy.cn (@qiniuyun and @Sheng Aofei) for their selfless contribution to the Go language community in China.

My official account

Further reading

  • night-reading-go/issues/468

  • 第 61 期 Go Modules, Go Module Proxy and goproxy.cn

  • YouTube: 【Go 】第 61 期 Go Modules, Go Module Proxy and goproxy.cn