• A new Go API for Protocol Buffers
  • By Joe Tsai, Damien Neil, and Herbie Ong
  • The Nuggets translation Project
  • Permanent link to this article: github.com/xitu/gold-m…
  • Translator: Prince Stuart
  • Proofread by Quzhen and Chauncey Chen

Go releases new version of the Protobuf API

introduce

We are pleased to announce the release of a major revision of the Go API for Protocol Buffers – Google’s programming language-independent data Exchange Interface format.

Motivation to build a new API

The first version of the Protocol Buffer for Go was released by Rob Pike in March 2010, and the first official version of Go was released two years later.

In the decades since the first release, the Package has grown and grown as Go has grown. The demand of users is also growing.

Many people want to use reflection Package to write programs that check protocol buffer messages. Reflect Package provides a view of the Go types and values, However, information about the protocol buffer system is ignored. For example, we might want to write a function that iterates through log entries and removes any data that is labeled as sensitive and not part of the Go type system.

Another common requirement is to use the Protocol Buffer compiler to generate other data structures, such as dynamic message types, which can represent messages whose type is unknown at compile time.

We have also observed that the root cause of frequent problems is the proto.message interface, which identifies the generated values of the Message types and does little to describe the behavior of those types. Programs crash or behave unpredictably when users create types that implement the interface (often inadvertently embedding message in other structures) and pass values of these types to functions that expect to generate message values.

All three problems have a common cause, and the usual solution is that the Message interface should fully specify the behavior of Message, functions that operate on Message values should be free to accept any type, and interfaces of these types should be properly implemented.

Since it was not possible to change the existing definition of the Message type while maintaining package API compatibility, we decided it was time to start developing a new major version of the non-compatible Protobuf module.

Today, we are happy to release this new module and hope you enjoy it.

No Reflection.

Reflection is the flagship feature of the new implementation. Similar to the Reflect package that provides a view of the Go type and values, the ProtoReflect package systematically provides a view of the values based on the Protocol buffer type.

The full description of the ProtoReflect package is too long for this article, however, we can take a look at how to write the log cleaning functions mentioned earlier.

First, we will write. Proto file to define its protobuf. FieldOptions types of extension, so that we can will comment field as sensitive information or not.

syntax = "proto3";
import "google/protobuf/descriptor.proto";
package golang.example.policy;
extend google.protobuf.FieldOptions {
    bool non_sensitive = 50000;
}
Copy the code

We can use this option to identify certain fields as non-sensitive.

message MyMessage {
    string public_name = 1 [(golang.example.policy.non_sensitive) = true];
}
Copy the code

Next, we’ll write a Go function that accepts any message value and removes all sensitive fields.

// Clear all sensitive fields in pb
func Redact(pb proto.Message) {
   // ...
}
Copy the code

The function takes the proto.Message argument, which is the interface type implemented by all the generated Message types. This type is an alias defined in the ProtoReflect package:

type ProtoMessage interface{
    ProtoReflect() Message
}
Copy the code

To avoid populating the namespace that generates the message, the interface contains only a method that returns ProtoReflect.message, which provides access to the contents of the message.

Why an alias? Since Protoreflect. Message has a method for returning the original proto.Message, we need to avoid looping imports between the two packages.

Protoreflect. Message. Range method for each filler fields of the Message calls a function.

m := pb.ProtoReflect()
m.Range(func(fd protoreflect.FieldDescriptor, v protoreflect.Value) bool {
    // ...
    return true
})
Copy the code

Used to describe protocol buffer type protoreflect. FieldDescriptor fields and containing protoreflect field values. The Value field to invoke the range function.

Protoreflect. FieldDescriptor. The Options method to Google. Protobuf. FieldOptions message returns the field option.

opts := fd.Options().(*descriptorpb.FieldOptions)
Copy the code

Why use type assertion? Because the generated descriptorpb package depends on protoReflect, the ProtoReflect package cannot return the correct option type otherwise it will cause circular import problems.)

We can then examine the options to see values extended to Boolean:

if proto.GetExtension(opts, policypb.E_NonSensitive).(bool) {
    return true // Do not delete non-sensitive fields
}
Copy the code

Note that we are looking at field descriptors here, not field values, and that the information we are interested in is in the Protocol Buffer type system, not the Go language.

This is also an example of how we have simplified the Proto Package API, where the original proto.GetExtension returns a value and an error message. The new proto. Extended decoding error reported at Unmarshal.

Once we have identified the fields that need to be modified, it is easy to clear them:

m.Clear(fd)
Copy the code

To sum up, our complete modification function is as follows:

// Clear all sensitive fields in pb
func Redact(pb proto.Message) {
    m := pb.ProtoReflect()
    m.Range(func(fd protoreflect.FieldDescriptor, v protoreflect.Value) bool {
        opts := fd.Options().(*descriptorpb.FieldOptions)
        if proto.GetExtension(opts, policypb.E_NonSensitive).(bool) {
            return true
        }
        m.Clear(fd)
        return true})}Copy the code

A more complete implementation would drill down recursively into these message value fields. We hope these simple examples have given you a better understanding of Protocol Buffer Reflection and how it can be used.

version

We called the original version of the Go Protocol Buffer APIv1 and the new version APIv2. Since APIv2 does not support forward compatibility with APIv1, we need to use a different path for each module.

(These API versions are different from the Protocol Buffer language versions: Proto1, Proto2, proto3. APIv1 and APIv2 are implementations in Go, and both support the Proto2 and Proto3 language versions.)

Is APIv1 github.com/golang/protobuf module.

Is APIv2 google.golang.org/protobuf module. We took advantage of the need to change the import path to switch versions and bind them to different host providers. (we are considering the google.golang.org/protobuf/v2, speak more clearly, this is the second major version of the API, but in the long run, we believe that the shorter path name is a better choice.)

We know that not all users migrate to the new version of the package at the same rate, some migrate quickly, and others may stay with the old version indefinitely. Even within a program, it is possible to use different versions of the API, which is critical. So, we continue to support programs that use APIv1.

  • github.com/golang/[email protected]Is the latest pre-APIv2 version of APIv1.
  • github.com/golang/[email protected]Is a version of APIv1 implemented by APIv2. The API is the same, but the underlying implementation is supported by the new API. This version contains conversion functions between APIv1 and APIv2,proto.MessageInterface to simplify the conversion between the two.
  • google.golang.org/[email protected]Is APIv2, the module depends ongithub.com/golang/[email protected], so any application that uses APIv2 will automatically choose a corresponding version that integrates APIv1.

Why start with V1.20.0? For clear service delivery, we do not expect APIv1 to reach V1.20.0. Therefore, the version number is sufficient to distinguish APIv1 from APIv2.

We intend to maintain APIv1 support for a long time.

Regardless of which VERSION of the API is used, the organization ensures that any given program is implemented using only a single Protocol Buffer. It allows applications to gradually adopt the new API or not adopt it at all while still taking advantage of the new implementation. The minimum version selection principle means that the program needs to retain the original implementation method until the maintainer chooses to update to a new version (either directly or by updating dependencies).

Note the other features

Google.golang.org/protobuf/encoding/protojson package using standard JSON mapping will protocol buffer message into JSON, And fixed some issues with the old JSONPB package that were difficult to change without affecting existing users.

Google.golang.org/protobuf/types/dynamicpb package provides the proto in the message. The message, Used to derive messages of type Protocol Buffer at run time.

Google.golang.org/protobuf/testing/protocmp package provides the use github.com/google/cmp package to compare protocol buffer function of the message.

Google.golang.org/protobuf/compiler/protogen package provides support for writing protocol compiler plugin.

conclusion

Google.golang.org/protobuf module is to Go protocol buffer support, significant improvements for reflection (reflection), a custom message implementation and neat API provide priority support surface. We intend to permanently maintain the old API in a new API wrapper so that users can adopt the new API at their own pace.

Our goal with this update is to amplify the benefits of the old API while addressing its problems. As we complete each newly implemented component, we will use it in Google’s code base, and this progressive approach gives us confidence in the usability, performance, and correctness of the new API. I believe it’s ready for use in a production environment.

We are excited to see this release out and hope it will continue to serve the Go ecosystem for the next decade and beyond.

Related articles

  • Working with Errors in Go 1.13
  • Debugging what you deploy in Go 1.12
  • HTTP/2 Server Push
  • Introducing HTTP Tracing
  • Generating code
  • Introducing the Go Race Detector
  • Go maps in action
  • go fmt your code
  • Organizing Go code
  • Debugging Go programs with the GNU Debugger
  • The Go image/draw package
  • The Go image package
  • The Laws of Reflection
  • Error handling and Go
  • “First Class Functions in Go”
  • Profiling Go Programs
  • A GIF decoder: an exercise in Go interfaces
  • Introducing Gofix
  • Godoc: documenting Go code
  • Gobs of data
  • C? Go? Cgo!
  • JSON and Go
  • Go Slices: usage and internals
  • Go Concurrency Patterns: Timing out, moving on
  • Defer, Panic, and Recover
  • Share Memory By Communicating
  • JSON-RPC: a tale of interfaces
  • Third-party libraries: goprotobuf and beyond

If you find any mistakes in your translation or other areas that need to be improved, you are welcome to the Nuggets Translation Program to revise and PR your translation, and you can also get the corresponding reward points. The permanent link to this article at the beginning of this article is the MarkDown link to this article on GitHub.

The Nuggets Translation Project is a community that translates quality Internet technical articles from English sharing articles on nuggets. The content covers Android, iOS, front-end, back-end, blockchain, products, design, artificial intelligence and other fields. If you want to see more high-quality translation, please continue to pay attention to the Translation plan of Digging Gold, the official Weibo, Zhihu column.