This tutorial provides a basic introduction to using protocol Buffers in Go applications, using the Protocol Buffer language with the Proto3 version. By creating a simple sample application to show you how

  • Define the message format in the.proto file.

  • Use the Protoc compiler to compile and generate Go code.

  • Read and write messages using Go’s Protocol Buffer API.

It is not a comprehensive guide to using the Protocol Buffer in Go, see the previous two tutorials for more detailed reference information.

Guide to the Protobuf language

Protobuf generate Go code guidelines

Why use the Protocol Buffer

The example we will use is a very simple “address book” application that reads and writes a person’s contact details in a file. Each person in an address book has a name, ID, email address, and contact phone number.

How do you serialize and retrieve such structured data? There are several ways to solve this problem:

  • Serialize Go data structures using GOBS (a custom serialization encoding in Go). This is a good solution for Go specific environments, but it won’t work if you need to share data with applications written for other platforms.

  • A special way to encode data items as a single string can be invented – for example, encoding four integers as “12:3:-23:67”. This is a simple and flexible approach, although it does require writing one-time coding and parsing code, and parsing has a small run-time cost. This is best for encoding very simple data.

  • Serialize the data to XML. This approach is very attractive because XML is (sort of) human-readable, and there are libraries for many languages. This might be a good choice if you want to share data with other applications/projects. However, XML is notoriously space intensive, and encoding/decoding it can cause a significant performance penalty to your application. In addition, navigating an XML DOM tree is much more complicated than navigating simple fields in a class normally.

The Protocol Buffer is a flexible, efficient, automated solution to this problem. Using the Protocol Buffer, you can write a.proto description of the data structure to be stored. From this, the Protocol Buffer compiler creates a class that implements automatic encoding and parsing of the protocol buffer data using a valid binary format. The generated class provides getters and setters for the fields that make up the Protocol Buffer and takes care of the details of reading and writing the Protocol Buffer as a unit. Importantly, the Protocol Buffer format supports the idea of extending the format over time so that code can still read data encoded in the old format.

Get the sample program

The example is a set of command-line applications for managing address book data files, encoded using the Protocol Buffer. The add_person_go command adds a new entry to the data file. The list_people_go command parses the data file and prints the data to the console.

PS: wechat does not allow you to click on the original link to download the program code on Github.

Download these files into your project directory:

  • Addressbook.proto file that describes the protocol buffer message format

  • Command line program addPerson. go, listpeople.go

Defining the protocol Format

To create an address book application, you need to start with the.proto file. The definition in the.proto file is simple: Define a message for each data structure to serialize, and then specify a name and type for each field in the message. In our example, the.proto file that defines the message is addressbook.proto.

.proto files start with a package declaration, which helps prevent naming conflicts between different projects.


     
    
      
    syntax = "proto3";
    
      
    package tutorial;
    
      
    import "google/protobuf/timestamp.proto";
    
     
Copy the code

In Go, the package name of the Protocol buffer is used as the Go package unless you specify gopackage. Even if you do provide gopackage, you should still define a package name in the.proto file to avoid name collisions in the Protocol Buffers namespace and non-GO languages.

Next, the message definition. A message is simply an aggregation containing a set of type fields. Many standard simple data types can be used as field types, including bool, INT32, float, double, and String. You can also use other message types as field types to add more structure to the message.


     
    
      
    message Person {
    
      
      string name = 1;
    
      
      int32 id = 2;  // Unique ID number for this person.
    
      
      string email = 3;
    
      
      enum PhoneType {
    
      
        MOBILE = 0;
    
      
        HOME = 1;
    
      
        WORK = 2;
    
      
      }
    
      
      message PhoneNumber {
    
      
        string number = 1;
    
      
        PhoneType type = 2;
    
      
      }
    
      
      repeated PhoneNumber phones = 4;
    
      
      google.protobuf.Timestamp last_updated = 5;
    
      
    }
    
      
    // Our address book file is just one of these.
    
      
    message AddressBook {
    
      
      repeated Person people = 1;
    
      
    }
    
     
Copy the code

In the example above, the Person message contains the PhoneNumber message, and the AddressBook message contains the Person message. You can even define message types nested within other messages – as you can see, the PhoneNumber type is defined in Person. If you want one of the field values to have a range of values from a predefined list of values, you can also define an enumeration type – where you specify a phone number that can be MOBILE, HOME, or WORK.

The “= 1”, “= 2” tags on each element identify the unique “tag” used in binary encoding for that field. Tags 1-15 require one less byte to encode than larger numbers, so as an optimization, you can decide to use these tags for frequently used or repetitive elements, leaving tags 16 and higher for optional elements that are less frequently used. Each element in a repeating field needs to be re-encoded with a tag number, so repeating fields are particularly good for this optimization.

If field values are not set, the default values are used: numeric type zero, string empty string, and BOols false. For embedded messages, the default value is always the “default instance” or “prototype” of the message, where no fields are set. Calling the accessor to get the value of a field that has not been explicitly set always returns the default value for that field.

If a field is repeatable, the field can be repeated any number of times (including zero). The order of repeated values is preserved in the Protocol buffer. Think of repeatable fields as variable-length arrays.

You will find complete instructions for writing.proto files – including all possible field types – in the Protobuf language guide. Don’t look for class inheritance like this, the Protocol Buffer does not support this.

Compile the protocol buffers

With.proto, the next thing you need to do is generate the classes (structs and struct methods in Go) that you need to read and write AddressBook (and Person and PhoneNumber) messages. To do this, you need to run the Protocol Buffer interpreter protoc on.proto:

  1. Make sure the compiler Protoc is installed first

  2. The protoc plug-in can be installed to compile and generate Go code. You can run the following command to install the plug-in

    go get -u github.com/golang/protobuf/protoc-gen-goCopy the code

  3. Now run the compiler, specifying the source directory (where the application’s source code is located – use the current directory if no value is provided), the target directory (where you want the generated code to be; Usually the same as $) SRC_DIR), and.proto’s path. In this case, you… :

    protoc -I=$SRC_DIR --go_out=$DST_DIR $SRC_DIR/addressbook.protoCopy the code

    4. We use the sample go into compiled code pb. Go file path is pb github.com/protocolbuffers/protobuf/examples/tutorial with protoc compile time so the use of the target path should be


     
    
      
    protoc --go_out=$GOPATH/src/github.com/protocolbuffers/protobuf/examples/tutorial ./addressbook.proto
    
     
Copy the code

$GOPATH/src/github.com/protocolbuffers/protobuf/examples/tutorial directories need to create good in advance.

Protocol buffer API

Generating addressbook.pb.go provides the following useful types:

  • You have the AddressBook structure with the People field.

  • A Person structure with Name, Id, Email, and PHONE fields.

  • Person_PhoneNumber structure containing the Number and Type fields.

  • Type Person_PhoneType and constants defined for each value in the Person.phoneType enumeration.

You can read more about what’s generated in the “Generate Code” guide, but for the most part, you can think of these as perfectly normal Go types.

Action speaks a thousand words, download the code provided in the tutorial, run the compile command above, and see what is generatedaddressbook.pb.goThe code in.

Here’s an example of how to create a Person instance:


     
    
      
    p := pb.Person{
    
      
            Id:    1234,
    
      
            Name:  "John Doe",
    
      
            Email: "[email protected]",
    
      
            Phones: []*pb.Person_PhoneNumber{
    
      
                    {Number: "555-4321", Type: pb.Person_HOME},
    
      
            },
    
      
    }
    
     
Copy the code

Serialize protocol buffer data in Go

The purpose of using a Protocl buffer is to serialize your structured data so that it can be parsed elsewhere. In Go, the Marshal function of the Proto library is used to serialize the Protocol Buffer data. Pointers to the structure of a Message implement the proto.message interface. Calling Proto.Marshal returns the Protocol buffer encoded in its wired format. For example, we use this function in the add_person command:


     
    
      
    book := &pb.AddressBook{}
    
      
    // ...
    
      
    // Write the new address book back to disk.
    
      
    out, err := proto.Marshal(book)
    
      
    if err ! = nil {
    
      
            log.Fatalln("Failed to encode address book:", err)
    
      
    }
    
      
    if err := ioutil.WriteFile(fname, out, 0644); err ! = nil {
    
      
            log.Fatalln("Failed to write address book:", err)
    
      
    }
    
     
Copy the code

Parse the Protocol buffer in Go

To parse coded messages, use the Unmarshal function of the Proto library. It is called to parse the data in the BUF into a Protocol buffer and place the results in a structure. Therefore, to parse a file in the list_people command, we use:


     
    
      
    // Read the existing address book.
    
      
    in, err := ioutil.ReadFile(fname)
    
      
    if err ! = nil {
    
      
            log.Fatalln("Error reading file:", err)
    
      
    }
    
      
    book := &pb.AddressBook{}
    
      
    if err := proto.Unmarshal(in, book); err ! = nil {
    
      
            log.Fatalln("Failed to parse address book:", err)
    
      
    }
    
     
Copy the code

Run the Go application

  • Run go build add_person. go and go build list_people. go on the command line to generate two binary files, add_person and list_people.

  • Run the./ add_person ADDRESS_BOOK program on the command line to prompt you for input. Use the input from the command line to build the address book data and then serialize the data into the protocol buffer stored in the ADDRESS_BOOK file.

  • Running the./ list_people command line reads the Protocol buffer data from the ADDRESS_BOOK file, parses it into the structure, and then prints the Person data from the structure.