preface

In an API design, there is the following design, which is also commonly seen on the web.

@Data
public class ApiResult {
    private int code;
    private String error;
    private Object data;
}
Copy the code

If equivalent substitution is required, the following design can be used:

message ApiResult {
    int32 code = 1;
    string error = 2;
    google.protobuf.Any data = 3;
}
Copy the code

Google.protobuf. Any can be understood as Java Object, but it is different from Object. Any is not the parent of all messages, whereas Object is the parent of all classes. In some cases the use is not so convenient, hope to have a more convenient design. Any is a proto class and can be replaced with a proto class.

We customize a donespeak. Protobuf. AnyData, then you can have the following structure:

message ApiResult {
    int32 code = 1;
    string error = 2;
    donespeak.protobuf.AnyData data = 3;
}
Copy the code

Any Protobuf: Google. Protobuf. Any)

Google.protobuf. Any is also defined by the proto file

Remove all comments, Google/protobuf/any proto only the following content, can customize a completely.

syntax = "proto3";

package google.protobuf;

option csharp_namespace = "Google.Protobuf.WellKnownTypes";
option go_package = "github.com/golang/protobuf/ptypes/any";
option java_package = "com.google.protobuf";
option java_outer_classname = "AnyProto";
option java_multiple_files = true;
option objc_class_prefix = "GPB";

message Any {
    string type_url = 1;
    bytes value = 2;
}
Copy the code

Any. Proto is compiled to produce a Message class, and Protobuf also adds the necessary methods for any. We can see how any.java differs from other Message classes in the source code for the any.proto compiled class below.

Google.protobuf. Any is itself a GeneratedMessageV3

GeneratedMessageV3 and Builder code:

public  final class Any 
    extends GeneratedMessageV3 implements AnyOrBuilder {

    // typeUrl_ will be a java.lang.String value
    private volatile Object typeUrl_;
    private ByteString value_;
    
    private static String getTypeUrl(String typeUrlPrefix, Descriptors.Descriptor descriptor) {
        return typeUrlPrefix.endsWith("/")? typeUrlPrefix + descriptor.getFullName() : typeUrlPrefix +"/" + descriptor.getFullName();
    }

    public static <T extends com.google.protobuf.Message> Any pack(T message) {
        return Any.newBuilder()
            .setTypeUrl(getTypeUrl("type.googleapis.com",
                                message.getDescriptorForType()))
            .setValue(message.toByteString())
            .build();
    }

    public static <T extends Message> Any pack(T message, String typeUrlPrefix) {
        return Any.newBuilder()
            .setTypeUrl(getTypeUrl(typeUrlPrefix,
                                message.getDescriptorForType()))
            .setValue(message.toByteString())
            .build();
    }

    public <T extends Message> boolean is(Class<T> clazz) {
        T defaultInstance = com.google.protobuf.Internal.getDefaultInstance(clazz);
            return getTypeNameFromTypeUrl(getTypeUrl()).equals(
                defaultInstance.getDescriptorForType().getFullName());
    }

    private volatile Message cachedUnpackValue;

    @java.lang.SuppressWarnings("unchecked")
    public <T extends Message> T unpack(Class<T> clazz) throws InvalidProtocolBufferException {
        if(! is(clazz)) {throw new InvalidProtocolBufferException("Type of the Any message does not match the given class.");
        }
        if(cachedUnpackValue ! =null) {
            return (T) cachedUnpackValue;
        }
        T defaultInstance = com.google.protobuf.Internal.getDefaultInstance(clazz);
        T result = (T) defaultInstance.getParserForType().parseFrom(getValue());
        cachedUnpackValue = result;
        returnresult; }... }Copy the code

Any has two fields: typeUrl_ and value_.

TypeUrl_ save values for the description of the Message class types, the original proto file Message to bring the value of the package, such as any typeUrl type.googleapis.com/google.protobuf.Any. Value_ is the ByteString of the Message object saved in the Any object, obtained by calling the toByteString() method. With this information, you can order a new one yourself.

Custom AnyData

common/any_data.proto

syntax = "proto3";

package donespeak.protobuf;

option java_package = "io.gitlab.donespeak.proto.common";
option java_outer_classname = "AnyDataProto";

// https://github.com/protocolbuffers/protobuf/blob/master/src/google/protobuf/any.proto
message AnyData {
    / / value for the < package >. < messageName >, such as API. Donespeak. Cn/data. The proto. DataTypeProto
    string type_url = 1;
    / / value for the message. ToByteString ();
    bytes value = 2;
}
Copy the code

Encoding and parsing of AnyData

Custom AnyData is just a plain Message class that requires a separate utility class to implement Pack and Unpack.

package io.gitlab.donespeak.javatool.toolprotobuf.anydata;

import com.google.protobuf.Descriptors;
import com.google.protobuf.InvalidProtocolBufferException;
import com.google.protobuf.Message;
import io.gitlab.donespeak.proto.common.AnyDataProto;

public class AnyDataPacker {
    private static final String COMPANY_TYPE_URL_PREFIX = "type.donespeakapi.cn";

    private final AnyDataProto.AnyData anyData;

    public AnyDataPacker(AnyDataProto.AnyData anyData) {
        this.anyData = anyData;
    }

    public static <T extends com.google.protobuf.Message> AnyDataProto.AnyData pack(T message) {
        final String typeUrl = getTypeUrl(message.getDescriptorForType());

        return AnyDataProto.AnyData.newBuilder()
            .setTypeUrl(typeUrl)
            .setValue(message.toByteString())
            .build();
    }

    public static <T extends Message> AnyDataProto.AnyData pack(T message, String typeUrlPrefix) {
        String typeUrl = getTypeUrl(typeUrlPrefix, message.getDescriptorForType());

        return AnyDataProto.AnyData.newBuilder()
            .setTypeUrl(typeUrl)
            .setValue(message.toByteString())
            .build();
    }

    public <T extends Message> boolean is(Class<T> clazz) {
        T defaultInstance = com.google.protobuf.Internal.getDefaultInstance(clazz);
        return getTypeNameFromTypeUrl(anyData.getTypeUrl()).equals(
            defaultInstance.getDescriptorForType().getFullName());
    }

    private static String getTypeNameFromTypeUrl(String typeUrl) {
        int pos = typeUrl.lastIndexOf('/');
        return pos == -1 ? "" : typeUrl.substring(pos + 1);
    }

    private volatile Message cachedUnpackValue;

    public <T extends Message> T unpack(Class<T> clazz) throws InvalidProtocolBufferException {
        if(! is(clazz)) {throw new InvalidProtocolBufferException("Type of the Any message does not match the given class.");
        }
        if(cachedUnpackValue ! =null) {
            return (T) cachedUnpackValue;
        }
        T defaultInstance = com.google.protobuf.Internal.getDefaultInstance(clazz);
        T result = (T) defaultInstance.getParserForType().parseFrom(anyData.getValue());
        cachedUnpackValue = result;
        return result;
    }

    private static String getTypeUrl(final Descriptors.Descriptor descriptor) {
        return getTypeUrl(COMPANY_TYPE_URL_PREFIX, descriptor);
    }

    private static String getTypeUrl(String typeUrlPrefix, Descriptors.Descriptor descriptor) {
        return typeUrlPrefix.endsWith("/")? typeUrlPrefix + descriptor.getFullName() : typeUrlPrefix +"/"+ descriptor.getFullName(); }}Copy the code

You can easily see that this class is basically the same as the implementation in Google.protobuf.any. Yes, this class is extracted directly from the Any class. You can also make the unpack method static, in which case the utility class is completely static. The original implementation is kept here so that it can be cached when unpacked. Since the Message classes are immutable, this strategy works well for multiple unpacks.

Define a Lookup utility Class that maps typeUrl to Class

As described above, a separate unpacking tool is provided here, which provides more unpacking methods. The utility class has a static unpack method that is called directly without instantiation. The other method uses the MessageTypeLookup class. The MessageTypeLookup Class is a registered Class that stores the mappings between Class Descriptors and Message descriptors. The existence of this class allows all possible Message classes to be registered and then unpackaged in general without having to try to find the class that anyData.Value’s data corresponds to.

MessageTypeUnpacker.java

package io.gitlab.donespeak.javatool.toolprotobuf.anydata;

import com.google.protobuf.InvalidProtocolBufferException;
import com.google.protobuf.Message;
import io.gitlab.donespeak.proto.common.AnyDataProto;

public class MessageTypeUnpacker {
    private final MessageTypeLookup messageTypeLookup;

    public MessageTypeUnpacker(MessageTypeLookup messageTypeLookup) {
        this.messageTypeLookup = messageTypeLookup;
    }

    public Message unpack(AnyDataProto.AnyData anyData) throws InvalidProtocolBufferException {
        AnyDataPacker anyDataPacker = new AnyDataPacker(anyData);
        Class<? extends Message> messageClass = messageTypeLookup.lookup(anyData.getTypeUrl());
        return anyDataPacker.unpack(messageClass);
    }

    public static <T extends Message> T unpack(AnyDataProto.AnyData anyData, Class<T> messageClass)
        throws InvalidProtocolBufferException {
        AnyDataPacker anyDataPacker = new AnyDataPacker(anyData);
        returnanyDataPacker.unpack(messageClass); }}Copy the code

MessageTypeLookup registers the mapping between typeUrl and Message’s classes to facilitate typeUrl lookup.

MessageTypeLookup.java

package io.gitlab.donespeak.javatool.toolprotobuf.anydata;import com.google.protobuf.Descriptors;import com.google.protobuf.Message;import java.util.HashMap;import java.util.Map;public class MessageTypeLookup {    private final Map<String, Class<? extends Message>> TYPE_MESSAGE_CLASS_MAP;    private MessageTypeLookup(Map<String, Class<? extends Message>> typeMessageClassMap) {        this.TYPE_MESSAGE_CLASS_MAP = typeMessageClassMap;    }    public Class<? extends Message> lookup(final String typeUrl) {        String type = typeUrl;        if(type.contains("/")) {            type = getTypeUrlSuffix(type);        }        return TYPE_MESSAGE_CLASS_MAP.get(type);    }    public static Builder newBuilder(a) {        return new Builder();    }    private static String getTypeUrlSuffix(String fullTypeUrl) {        String[] parts = fullTypeUrl.split("/");        return parts[parts.length - 1];    }    public static class Builder {        private final Map<String, Class<? extends Message>> TYPE_MESSAGE_CLASS_BUILDER_MAP;        public Builder(a) {            TYPE_MESSAGE_CLASS_BUILDER_MAP = new HashMap<>();        }        public Builder addMessageTypeMapping(final Descriptors.Descriptor descriptor,            final Class<? extends Message> messageClass) {            TYPE_MESSAGE_CLASS_BUILDER_MAP.put(descriptor.getFullName(), messageClass);            return this;        }        public MessageTypeLookup build(a) {            return newMessageTypeLookup(TYPE_MESSAGE_CLASS_BUILDER_MAP); }}}Copy the code

With MessageTypeLookup, you can pre-register all possible messages into this class, and then unpack them using this class so that you can basically implement a generic AnyData unpack implementation. However, the registration of this class will be very troublesome, and all messages need to be added manually, which is laborious and error-prone, and will be added every time a new class is added, which is very troublesome.

Finds the class under the specified path and its inner class

To address the shortcomings of MessageTypeLookup above, you can add a method to find qualifying classes by package path. In development, it is common to put all Proto under a single package name, so you just need to know the package name and then scan all classes under the package to find subclasses of GeneratedMessage 3. Register the result with MessageTypeLookup. This way, even if new Message classes are added, they can be automatically registered without having to be manually added to MessageTypeLookup.

Find all classes under a package

To achieve this, we use the Reflection library, which provides many useful Reflection methods. If you want to implement such a reflection method, actually quite troublesome, and there will be a lot of pits. It would be fun to talk more about reflection and class loading later.

This section is inspired by Spring’s @ComponentScan annotation. Similarly, there are two ways to scan, one by prefixing the package name and the other by specifying the package of the class as the package to scan. Both methods allow multiple paths to be provided.

<! -- https://mvnrepository.com/artifact/org.reflections/reflections --><dependency>    <groupId>org.reflections</groupId>    <artifactId>reflections</artifactId>    <version>0.9.11</version></dependency>
Copy the code

ClassScanner.java

package io.gitlab.donespeak.javatool.toolprotobuf.anydata;import java.util.Set;import com.google.protobuf.GeneratedMessageV3;import org.reflections.Reflections;public class ClassScanner {    public static <T> Set<Class<? extends T>> lookupClasses(Class<T> subType, String... basePackages) {        Reflections reflections = new Reflections(basePackages);        return reflections.getSubTypesOf(subType);    }    public static<T> Set<Class<? extends T>> lookupClasses(Class<T> subType, Class<? >... basePackageClasses) { String[] basePackages =new String[basePackageClasses.length];        for(int i = 0; i < basePackageClasses.length; i ++) {            basePackages[i] = basePackageClasses[i].getPackage().getName();        }        return lookupClasses(subType, basePackages);    }}
Copy the code

Register a subclass of GeneratedMessageV3 from a package into MessageTypeLookup

Once we have a scanning utility class for our classes, the requirement to “register a subclass of GeneratedMessageV3 in a package into MessageTypeLookup” becomes very easy.

With ClassScanner, we can get all the class objects of the GeneratedMessageV3 class, and we need to get typeurls. Because the Message#getDescriptorForType() method is an object method, it is necessary to use the reflected method to get an instance of the desired class object, and then call getDescriptorForType() to get the typeUrl. You also know that the Message classes are immutable and that all constructors are private and can only be created from the Builder class. The static method Message#newBuilder() is called by reflection to create a Builder, which is then used to get the Message instance. At this point, all the necessary work has been done.

MessageTypeLookupUtil.java

package io.gitlab.donespeak.javatool.toolprotobuf.anydata;

import com.google.protobuf.GeneratedMessageV3;
import com.google.protobuf.Message;

import java.lang.reflect.InvocationTargetException;
import java.util.Set;

public class MessageTypeLookupUtil {

    public static MessageTypeLookup getMessageTypeLookup(String... messageBasePackages) {

        / / used here GeneratedMessageV3 as the parent class lookup, prevent similar com. Google. Protobuf. AbstractMessage classes appear
        Set<Class<? extends GeneratedMessageV3>>
            klasses = ClassScanner.lookupClasses(GeneratedMessageV3.class, messageBasePackages);

        return generateMessageTypeLookup(klasses);
    }

    private static MessageTypeLookup generateMessageTypeLookup(Set<Class<? extends GeneratedMessageV3>> klasses) {
        MessageTypeLookup.Builder messageTypeLookupBuilder = MessageTypeLookup.newBuilder();
        try {
            for (Class<? extends GeneratedMessageV3> klass : klasses) {
                Message.Builder builder = (Message.Builder)klass.getMethod("newBuilder").invoke(null); Message messageV3 = builder.build(); messageTypeLookupBuilder.addMessageTypeMapping(messageV3.getDescriptorForType(), klass); }}catch (NoSuchMethodException | IllegalAccessException | InvocationTargetException e) {
            // will never happen
            throw new RuntimeException(e.getMessage(), e);
        }
        return messageTypeLookupBuilder.build();
    }

    public static MessageTypeLookup getMessageTypeLookup(Class
       ... messageBasePackageClasses) {

        / / used here GeneratedMessageV3 as the parent class lookup, prevent similar com. Google. Protobuf. AbstractMessage classes appear
        Set<Class<? extends GeneratedMessageV3>>
            klasses = ClassScanner.lookupClasses(GeneratedMessageV3.class, messageBasePackageClasses);
        returngenerateMessageTypeLookup(klasses); }}Copy the code

A unit test is added here to provide usage of the MessageTypeLookupUtil class.

Here we add a number of different Proto classes, and the generated code location looks something like this, where the $represents the inner class.

io.gitlab.donespeak.proto.common
    .AnyDataProto.class$AnyData.class
    .ApiResultProto.class$ApiResult.class

io.gitlab.donespeak.javatool.toolprotobuf.proto
    .DataTypeProto.class$BaseData.class
    .StudentProto.class$Student.class
Copy the code

Test class implementation: MessageTypeLookupUtilTest. Java

package io.gitlab.donespeak.javatool.toolprotobuf.anydata;

import com.google.protobuf.Message;
import io.gitlab.donespeak.javatool.toolprotobuf.proto.DataTypeProto;
import io.gitlab.donespeak.javatool.toolprotobuf.proto.StudentProto;
import io.gitlab.donespeak.proto.common.AnyDataProto;
import io.gitlab.donespeak.proto.common.ApiResultProto;
import org.junit.Test;

import static org.junit.Assert.*;

public class MessageTypeLookupUtilTest {

    @Test
    public void getMessageTypeLookup1(a) {
        MessageTypeLookup messageTypeLookup = MessageTypeLookupUtil.getMessageTypeLookup(
            "io.gitlab.donespeak.proto.common");

        Class<? extends Message> anyDataMessage =
            messageTypeLookup.lookup(AnyDataProto.AnyData.getDescriptor().getFullName());
        // AnyDataProto is under the package
        assertNotNull(anyDataMessage);
        assertTrue(AnyDataProto.AnyData.class.equals(anyDataMessage));

        Class<? extends Message> studentMessage =
            messageTypeLookup.lookup(StudentProto.Student.getDescriptor().getFullName());
        // StudentProto is not in the specified package
        assertNull(studentMessage);
    }

    @Test
    public void getMessageTypeLookup2(a) {
        MessageTypeLookup messageTypeLookup = MessageTypeLookupUtil.getMessageTypeLookup(
            "io.gitlab.donespeak.proto.common"."io.gitlab.donespeak.javatool.toolprotobuf.proto");

        Class<? extends Message> anyDataMessage =
            messageTypeLookup.lookup(AnyDataProto.AnyData.getDescriptor().getFullName());
        / / AnyDataProto under io.gitlab.donespeak.proto.com mon
        assertNotNull(anyDataMessage);
        assertTrue(AnyDataProto.AnyData.class.equals(anyDataMessage));

        Class<? extends Message> studentMessage =
            messageTypeLookup.lookup(StudentProto.Student.getDescriptor().getFullName());
        / / StudentProto in IO. Gitlab. Donespeak. Javatool. Toolprotobuf. Under the proto
        assertNotNull(studentMessage);
        assertTrue(StudentProto.Student.class.equals(studentMessage));
    }

    @Test
    public void getMessageTypeLookup3(a) {
        MessageTypeLookup messageTypeLookup =
            MessageTypeLookupUtil.getMessageTypeLookup(ApiResultProto.ApiResult.class, DataTypeProto.BaseData.class);

        Class<? extends Message> anyDataMessage =
            messageTypeLookup.lookup(AnyDataProto.AnyData.getDescriptor().getFullName());
        AnyDataProto is packaged with ApiResultProto
        assertNotNull(anyDataMessage);
        assertTrue(AnyDataProto.AnyData.class.equals(anyDataMessage));

        Class<? extends Message> studentMessage =
            messageTypeLookup.lookup(StudentProto.Student.getDescriptor().getFullName());
        // StudentProto and DataTypeProto are the same packageassertNotNull(studentMessage); assertTrue(StudentProto.Student.class.equals(studentMessage)); }}Copy the code

reference

  • protocolbuffers/protobuf/src/google/protobuf/any.proto @Github
  • ronmamo/reflections @Github
  • ronmamo/reflections#UseCases.md @Github
  • Protocol Buffers, Part 3 — JSON Format @codeBurst

Related articles

  • Protobuf and POJO interconversion – via Json
  • Protobuf and Json conversion