Recently, I was in charge of the development of feature platform service. The initial version of feature data access was using JSON String, which was a relatively simple and commonly used data exchange format. This is feasible when the data structure is simple and the performance is not high, but as the call volume of the service increases, the performance is also required, this approach is not feasible, encounter the following main problems:

  • JSON strings are slow to parse, and can often take tens of milliseconds when calling heavily, resulting in call timeouts.
  • JSON takes up storage space, does not compress, and takes more time to transmit over the network.
  • The GC problem

The middleware team suggested using Protocol Buffer, and decided to use Protocol Buffer to optimize the service after investigation. The following is a brief introduction to its use, which mainly includes three steps:

  1. in.protoData formats are defined in the file
  2. According to the.protoFile generates class files
  3. use

1. Define the data format

Create a new.proto file, such as mq_message.proto:

// To specify which version of the syntax to use, specify syntax = on the first line"proto3"; package example; // Specify javA_package, use package if not specified. The generated classes will be placed under the package option JAVA_Package ="com.example.example"; // The generated classname option JAVA_outer_classname ="MQMessageProtos"; Message MQMessage {// The required field must be provided, otherwise Build throws RuntimeException, and Parse throws IOException. // Declare an enumeration type enum OperationType {UPDATE = 0; DELETE = 1; } // Declare a field of enumeration type. The number following the equals sign is the identifying number OperationType OperationType = 1; // Declare a field of type string name = 2; // Declare a List of repeated Item items = 3; } message Item {// Declare a string key = 1; // Declare a map field map<string, string> fields = 2; }Copy the code
  1. First specify the syntax version,proto2 和 proto3There are some differences, please refer to official documentation for details (ladders may be required,proto2.proto3), used in this articleproto3.
  2. Package and JAVA_Package specify under which package the Java classes to be generated, avoiding naming conflicts.
  3. Java_outer_classname Specifies the generated classname. If this parameter is not specified, the generated classname is based on the file name. For example, my_proto.
  4. Message is used for data format definition.
    • a.protoMultiple messages can be defined in a file
    • Fields defined in message support string, Byte, bool, MAP, enum, numeric type, and user-defined message
    • Defining fields requires specifying unique identifying numbers that are used to identify fields in the binary format message once you start using this (Numbers 1-15 are encoded in one byte, and numbers greater than 15 are encoded in more bytes, so try to leave smaller numbers as the most common fields. Do not specify the identity number of a field in order.
    • If you need to define a List, add repeated to the field.
    • If you already use the message-generated class and want to add fields, just add them. When the class of the new field parses the old data, the new field is set to the default value. New fields are ignored when the old class parses new data.

2. According to.protoFile generates class files

After defining the.proto file, compile the.proto file using the Protocol Buffer compiler. To install the Protocol Buffer compiler locally, use BREW install Protobuf for Mac OS. For other systems, see the following:

protoc --java_out=./ ./mq_message.proto
Copy the code

Java_out specifies the directory in which the compiled files will be generated, and finally the files you want to compile.

Use 3.

Here is an example of a class generated from the mq_message.proto file above:

public class TestPB {
    public static void main(String[] args) {
        MQMessageProtos.Item item1 = MQMessageProtos.Item.newBuilder()
                .setKey("1")
                .putFields("field1"."value1")
                .putFields("field2"."value2")
                .build();
        MQMessageProtos.Item item2 = MQMessageProtos.Item.newBuilder()
                .setKey("2")
                .putFields("field3"."value3")
                .putFields("field4"."value4")
                .build();
        MQMessageProtos.MQMessage mqMessage = MQMessageProtos.MQMessage.newBuilder()
                .setName("testName") .setOperationType(MQMessageProtos.MQMessage.OperationType.UPDATE) .addItems(item1) .addItems(item2) .build(); byte[] byteArray = mqMessage.toByteArray(); try { MQMessageProtos.MQMessage message = MQMessageProtos.MQMessage.parseFrom(byteArray); System.out.println(message.getName()); } catch (InvalidProtocolBufferException e) { e.printStackTrace(); }}}Copy the code

Finally, welcome to my personal blog: Chen Minjie