Brief comment: There hasn’t been a good solution for Go to rely on package versioning, and although the community has produced more than a dozen tools to solve this problem, there hasn’t been any official support. This proposal is expected to see official package versioning in the next version of Go, removing GOPATH dependencies and introducing the concept of Modules, recompiling in a real sense, a major change
It’s time to add package version control for Go!
More specifically, we need to push the concept of package versions into the common vocabulary of Go developers and tools so that they can communicate with each other about exactly what program code needs to be compiled, run, and parsed. Similarly, the go command needs to tell the developer exactly which version of which package is being used in the build.
Version control allows us to recompile. When I let you try out the latest version of my program, I knew that you were not only getting the code for my latest program, but also the same version of the package that my code relied on to compile the exact same binary package.
Version control also allows us to compile the same way from stage to stage, even though our dependencies may have a new version, so long as our configuration does not allow it, the go command will not use the new version of the package.
While adding version control is a necessary feature, the go command line is simple, efficient, and easy to understand. For now, many programmers don’t pay much attention to version control, and for the most part, it doesn’t matter. Just think if we have a reasonable mode of design and the default configuration, let developers don’t need too much attention to version control, program still can be a very good job, and have little impact on the existing engineering, released the new version is simple enough, even can ignore version control in the development of daily work, such a version control model is what we want.
In short, versioning is a must, but it should be transparent enough not to break the go Get functionality. This article explores a proposal that fully implements these, and provides a prototype demo that is currently available, which hopefully will lay the groundwork for integration into the GO command. In this article I intend to discuss what needs to be done and what does not need to be done during transition, and based on this discussion I will make further adjustments to the proposal and prototype, and submit an official proposal that will be integrated as an optional feature into Go 1.11.
This proposal retains the essence of Go Get, increases repeated construction, adopts semantic version control, abandonsVendor, abandonsgopath when foundation engineering is created, and provides a smooth migration method for old projects. At present, this proposal is still in the initial stage, if there is any problem in details, We will fix it before the Go major release
background
Before we discuss the proposal, let’s take a look at the current situation. It may be a long story, but the lessons of history are important for the present, and make it clear what this bill has changed. If you’re bored, skip to the proposal, or go to the prototype demo
Makefiles, goinstall, andgo get
In November 2009, Go released its first version with a compiler, linker, and some built-in libraries. At the time, you had to compile and link your programs by running 6G and 6L, as well as some simple makefiles. In most cases, a simple wrapped GoBuild can compile a single package and generate the corresponding Makefile. At the time, there was no proper way to share code with others. Although many features are not yet available, they are still being released, and Go plans to put some of the remaining features into the community.
In February 2010, GoInstall appeared, a new zero-configuration command line designed to download packages from source management libraries like Bitbucket and GitHub. Goinstall introduced the path convention that is common among Go developers today. Because there was no code to follow this convention, Goinstall was initially limited to standard library imports, but developers quickly migrated their own naming conventions to the unified convention we know today. These published Go packages have evolved into a coherent ecosystem.
Goinstall also eliminates makefiles, eliminating the complexity of building configurations for users. Although it is occasionally inconvenient for package authors not to be able to generate code every time they build, this simplification is important for package consumers: The user doesn’t have to worry about the compilations of the toolset in the installed package being inconsistent with the compilations of the package author; simplicity makes a lot of sense for a tool. Makefiles are necessary for step-by-step compilation of packages; Reverse engineering how to make the same package use different tools (like go Vet or code completion) makefiles is difficult in this area. Even if compile dependencies are maintained correctly, it is difficult for any one makefiles to recompile if necessary. Although some people felt that removing makefiles lost flexibility, in retrospect the benefits far outweighed the inconvenience.
In December 2011, as part of the pre-release of Go 1, we introduced the use of Go Get to replace Goinstall in the Go command.
Overall, Go Get is transformative in that it allows GO developers to share code and build with each other, and tools isolate the details of the GO command build system. However, Go Get lacks the concept of version control, The need for version-related functionality was actually clear from the first discussion of GoInstall. Unfortunately, at least in our Go team, it wasn’t clear what to do. When Go Get needs a package, it always downloads the latest copy from a remote version control system like Git or Mercurial, and the lack of package version management leads to at least two major flaws.
Versioning and API stability
The first major drawback of Go Get without version control is that there is no way to know if a given update is what users expect.
In November 2013, Go 1.2 added a FAQ for basic advice on package versioning:
Publicly distributed packages should maintain backward compatibility.
Go 1 compatibility guidelines
A good reference is provided: do not delete exported names, encourage use of conformance semantics (composite literals
Tag named version, etc. If you need different functionality, add a new name instead of changing the old one. If you need completely separate functionality, create a new package under a new import path
In March 2014, Gustavo Niemeyer founded Gopkg. in as a “Go language stability API”. In /yaml.v1 and gopkg.in/yaml.v2, you can point to different committed versions of your Git repository (and possibly in different branches). The idea is that after a major change in dependency library functionality, you can use the old v1 version import path as a backup, and then create a new version v2 that introduces a completely different API through the v2 import path.
In August 2015, Dave Cheney presented a proposal to Adopt Semantic versioning, which sparked an interesting discussion in the following months, with everyone agreeing that semantic versioning was a good idea, But no one knows what to do next: what tools should semantic versions use?
Any discussion of semantic versioning will inevitably be countered with Hiram’s law:
When an API has enough users, it doesn’t matter what you promise in the contract, and all observed behavior in your system will be directly dependent on some users.
While Hiram’s law holds true empirically, semantic versioning is still an effective way to establish a reasonable expectation relationship between different releases. Generally speaking, upgrading from 1.2.3 to 1.2.4 should not break your original code, whereas upgrading from 1.2.3 to 2.0.2 May break the original code. If you upgrade to 1.2.4 and have a problem, the authors will usually fix the problem in 1.2.5 according to bug reports. If you upgrade to 2.0.0 and have a problem, it is probably the result of a major upgrade.
Vendoring and repeatable builds
The second major drawback of Go Get without the concept of version control is that you probably won’t be able to implement the idea of recompiling. There is no way to make sure that users of your program rely on the same version of the package you compiled. In November 2013, the Go 1.2 FAQ also added the following basic recommendations:
If you’re using an externally supplied package and you’re worried that it might change unexpectedly, the easiest way to do this is to copy it to your local library (which is what Google does internally), mark it as a local library and put it in a new import path, such as”
original.com/pkg”Copy to”
You.com/external/or…”. Goven by Keith Rarick is a tool that automates this functionality
Goven, released by Keith Rarick in March 2012, copies all the packages you depend on to your local repository and updates all the import paths to the new local path. Modifying source code dependencies in this way works for compilation, but there are some problems. This modification makes it difficult for the local package to compare the changes with the new copy to incorporate the required updates.
In September 2013, Keith released GoDEp, “a new tool for freezing package dependencies.” The most important improvement in Godep was the addition of what we now know as Go Vendoring — copying dependencies into projects without modifying source files — and somehow setting up the GOPATH implementation without the support of a toolchain
In October 2014, Keith suggested adding support for the concept of “external packages” to the Go toolchain so that tools could better analyze engineering by convention. At that time, many tools similar to GoDEP also appeared, and Matt Farina wrote an article, “Glide in the Sea of Go Package Managers,” comparing GoDEP with many of the later tools, the most prominent of which was Glide.
In April 2015, Dave Cheney introduced GB, an “engineering-based build tool… Repeatable builds from vendor source code “without re-importing. (Another purpose of GB is to avoid placing required code in a specially designated GOPATH directory, which is not a good development process for many developers.)
That spring, Jason Buberel investigated the Go package management tool to see how the efforts of these different people could be combined to avoid duplication and waste. His investigation made it clear to the Go team that the Go command needed to support Vendoring directly without reimporting packages. At the same time, Daniel Theophanes began to develop a file format specification to describe the exact source and version information of the code in the Vendor directory. In June 2015, we accepted Keith’s proposal vendor as an experimental feature in Go 1.5, which was optional in Go 1.5 and default in Go 1.6. Authors of all vendoring tools are encouraged to work with Daniel to develop a file specification for a unified metadata format.
Incorporating vendoring concepts into Go’s toolchain allows program analysis tools like Go Vet to better understand engineering. To date there are dozens of Go package management tools or Vendoring tools to manage vendor directories. On the other hand, because these tools use different metadata file format specifications, they cannot easily share dependent information between them.
More importantly, vendoring is not a complete solution; it provides a recompilable implementation without addressing package versioning, and it does not help the project understand the package version to decide which version to use. Package managers such as Glide and DEP implicitly introduce the concept of version control into Go compilation through some setting of the vendor directory, and do not require direct toolchain support. In fact, with many tools in the Go ecosystem not version-aware, it is clear that Go needs to provide direct toolchain support for package versions.
An experiment in official package management
At GopherCon 2016, a group of Gophers interested in Go gathered at Hack Day for a discussion around Go package management. One outcome was the formation of a committee and package management advisory group with the goal of creating a new Go package management tool with the vision of unifying existing tools, The implementation method is still through the vendor directory. Initiated by Peter Bourgon and co-drafted by committee members Andrew Gerrand, Ed Muller, Jessie Frazelle, and Sam Boyer, the deP was implemented under Sam’s leadership, For background information see Sam’s article “So You Want to Write a Package Manager” (February 2016) and “The Saga of Go Dependency Management” (December 2016). And “The New Era of Go Package Management” at GopherCon in July 2017.
Dep serves a variety of purposes: it is an improvement on current best practices, an important step towards a successful solution, and an “official experiment” that gives us a deeper understanding of what we should be doing and what we are not doing right for Go developers. But DEP is not the final prototype for go command integration package versioning. It explores the design space in a powerful and flexible way, so to speak, acting as makefiles when we compile the Go program. Once we understand the design and focus on a few key features that must be supported, You’ll find that this design removes a lot of other functionality from the Go ecosystem and reduces the cost of presentation. At the same time, the mandatory convention makes the Go code look more uniform and understandable, making it easier for building tools.
Let’s look at the next phase of DEP’s goal: to complete the first draft of a final prototype for go command integration, similar to package management for GoInstall. This prototype is a separate command called VGO, which you can think of as a go command that supports package versioning. This is a new experimental feature, and as with the introduction of GoInstall, some code and projects already support VGo, while others need to be adapted. As with makefiles, some control and presentation layers were removed, simplifying the system and reducing user costs.
The vGO experiment does not mean that we will stop supporting DEP, we will continue to ensure that DEP is available until the go command integration path is determined, implemented and available. Of course, we also try our best to ensure a smooth transition from DEP to GO command integration. If the project is not using DEP (note that GODEP and Glide have stopped updating, it is suggested to migrate to DEP), then it can directly migrate to VGO
The proposal
The proposal for adding version control to the GO command consists of four steps. The first is to be compatible with the import rules in Go FAQ and Gopkg. in, that is, to establish an expectation that the package import path of the new version should be backward compatible with the old version. Second, a simple new algorithm (called minimal version selection) is used to filter out which package version is used at compile time. Third, introduce the concept of the Go module, which is a set of packages containing a single version and declaring the minimum version dependencies they require. Fourth, define how these changes will be integrated into the GO command so that the basic workflow of the GO command cannot change much from now on. Let’s take a closer look at each one, and I’ll cover it in more detail in other posts this week
Importing compatibility Rules
The biggest pain in package management systems is resolving compatibility issues. For example, on most systems package B states that the required package D versions are 6 or higher, and package C then states that the required package D versions are 2,3 and 4, but no higher than 5. If you’re writing package A and you want to introduce packages B and C at the same time, you’re out of luck: there is no separate version of D that B and C can choose to compile into A at the same time. B and C are doing something reasonable, and there’s nothing you can do to change it, so you’re stuck.
To avoid a dominant designer designing a system that makes existing large programs fail to compile, the proposal requires package authors to follow the following import compatibility principles:
If an old package and a new package have the same import path, the new package must be backward compatible with the old package
This rule is a restatement of the previous Go FAQ, quoting the last part of the FAQ: “If a complete change is required, create a package with a new import path”. The developers wanted to express such a change through semantic versioning, so we added semantic versioning to our proposal. Specifically, major version 2 and later versions can be distinguished by including version information in the path, such as:
import "github.com/go-yaml/yaml/v2"
Copy the codeV2.0.0 was created, which means a major change in semantic versioning, creating a package with a new import path in accordance with import compatibility principles. Because each major version has a different import path, a given Go executable might contain any of the major versions, which is exactly what we would expect.
Package authors following the import compatibility principle allows us to reduce adaptation efforts, make the system simpler and reduce fragmentation of the package ecosystem. Of course, the reality is that despite the authors’ best efforts, there will inevitably be user breakages. Therefore, it is important to use an upgrade mechanism that does not upgrade frequently, which is what we will cover next.
Minimum version selection
Almost all package management today, including DEP and Cargo, builds with the latest package version, which I think is the wrong convention for two important reasons. First, the “last available version” may change due to external events, such as a new release. Maybe tonight someone in the package you depend on will release a new version, and the next morning you might compile it and have different results. Second, to override this default, developers spend a lot of time telling the package manager which version of the package not to use.
In the proposal we used a different approach called minimal version selection. By default, each package builds with the oldest available version, which keeps yesterday’s and today’s compilations unchanged, since you don’t want to release an older version today. Even better, developers can simply tell the package manager which version is least available, and the package manager can quickly decide which version is available. We call it minimal version selection partly because we choose minimal version, and partly because it minimizes the overall system, avoiding the complexity of the existing system.
Minimum version selection specifies the minimum version requirements for modules that depend on it, making it a good choice for subsequent upgrade and downgrade operations. At the same time, it can also complete compilation by excluding version-specific dependencies or specifying version-specific dependencies.
The minimal version selection completes a repeatable build by default without locking the file.
Minimal version selection is the key to import compatibility. Instead of saying, “No, the version is too new,” users are more likely to say, “No, the version is too old,” in which case the solution is clear: just upgrade to the new version.
Go Modules
The Go module is a collection of packages that share the prefix of an import path, known as a module path. A module is a unit of version control. The version of a module is represented by a semantic version string. When Git is used in development, the developer defines a new semantic version by adding a new tag to the Git repository of the module. Although the semantic version approach is strongly recommended, pointing to specific Commit is also supported.
The module is defined in a new file called go.mod, which contains the smallest version of the package the module depends on. Here’s a simple go.mod file:
// My hello, Module "rsc. IO /hello" require ("golang.org/x/text" v0.0.0-20180208041248-4e4a3210bb54 "rsc. IO /quote" v1.5.2)Copy the codeIO /hello defines a module by path identifying rsc. IO /hello, which itself depends on two other modules: golang.org/x/text and rsc. IO /quote. The module itself is compiled using the version of the dependency list specified in the go.mod file. For higher levels of compilation, other places where the module is imported will compile with a newer version of it.
Package publishers are better off using semantic tag releases, and VGO encourages version tags rather than arbitrary submissions. The rsc. IO /quote module uses a tag version, while the golang.org/x/text module does not provide a tag version. For unnamed commits, v0.0.0-YYYYMMDDHHMmsS-COMMIT indicates a commit on a specified date. In semantic versioning, the string V0.0.0 indicates the pre-release version number and YYYYMMDDHHMmsS-commit indicates the pre-release version identifier.
In addition to specifying required dependencies, the go.mod file can also implement the exclusion and replacement versions mentioned in the previous section, but these only work if the module is compiled directly, not if the module is compiled as part of an overall project. See examples for details.
Goinstall and old Go Get download code directly through version control tools like git and HG, which has a number of problems, including fragmentation: users can’t download code hosted in the Bazaar repository without BZR. Modules, by contrast, download zip packages over HTTP. Previously, go Get used a command line tool for version control to download packages from mainstream code hosting sites. Now, VGo directly downloads packages from the API provided by the site.
Module is given in the form of unified by zip can download agreement is more simple, let companies or individuals can be in any reasons (cached copy to prevent source security or want to be deleted) do your own download agent, using a proxy to ensure usability and through go. Mod defines what code to use, vendor directory is no longer needed.
goThe command
The go command must be updated to use module functionality. One important change is that common build commands like goBuild, Go Install, Go Run, and Go Test will now need to resolve their dependencies for specific requirements, When using golang.org/x/text in a new module, you can simply import and compile the Go source without worrying about versioning separately.
However, the most important changes or put an end to the GOPATH workspace Settings, as the Go code due to Go. The mod file contains the complete module path dependence and defines each use version, therefore includes. The mod file directory can be regarded as a directory tree root directory, the directory tree on its working space, And it is isolated from other similar directories. Now you just need git Clone and CD to copy code directly, no need for GOPATH
What’s next?
I also published an article called “A Tour of Versioned Go” about what it’s like to use VGO. This article will give you an idea of how to download and play vGO now, and I will post more articles later this week to fill in some of the content skipped in this article. I hope you have some suggestions for this article and I will try to read the Go Subreddit and Golang-Nuts email feedback. On Friday I will post a closing chapter of the FAQ, and next week I will submit a formal Go proposal.
Try vgo (vgo) and start tagging versions in your repository, creating and checking in go.mod files. Note that if your repository has an empty go.mod but dep, Glide, glock, Godep, godeps, govend,govendor, or GVT configuration files, vGO will use them to populate the go.mod file.
The original:
Go += Package Versioning
Translation: Muse’s Lover