Protobuf/POJO conversion - via Json - Moment For Technology

preface

This article is a follow-up to Protobuf and Json interconversion, which addresses the interconversion of different Protobeans and PoJos in the system hierarchy. The converted Protobuf and Pojo should have the same name and type of property (when the Proto property is of type Message, the corresponding property is of type Pojo’s Object, so both should have the same property).

The basic idea of transformation

The following protobuf file is used for this test:

StudentProto.proto

syntax = "proto3";

option java_package = "io.gitlab.donespeak.javatool.toolprotobuf.proto";

message Student {
    string name = 1;
    int32 age = 2;
    Student deskmate = 3;
}
Copy the code

DataTypeProto.proto

syntax = "proto3";

import "google/protobuf/any.proto";

option java_package = "io.gitlab.donespeak.javatool.toolprotobuf.proto";
package data.proto;

enum Color {
    NONE = 0;
    RED = 1;
    GREEN = 2;
    BLUE = 3;
}

message BaseData {
    double double_val = 1;
    float float_val = 2;
    int32 int32_val = 3;
    int64 int64_val = 4;
    uint32 uint32_val = 5;
    uint64 uint64_val = 6;
    sint32 sint32_val = 7;
    sint64 sint64_val = 8;
    fixed32 fixed32_val = 9;
    fixed64 fixed64_val = 10;
    sfixed32 sfixed32_val = 11;
    sfixed64 sfixed64_val = 12;
    bool bool_val = 13;
    string string_val = 14;
    bytes bytes_val = 15;

    Color enum_val = 16;

    repeated string re_str_val = 17;
    map<string, BaseData> map_val = 18;
}
Copy the code

Direct conversion of

By mapping, the properties with the same name and the same class are copied directly. This implementation is mainly implemented through the reflection mechanism.

[ A ] <--> [ B ]
Copy the code

– direct conversion will be implemented using protobuf’s reflection mechanism, it will be difficult, it is being tried Another way is to try using Apache Common BeanUtils or Spring BeanUtils to copy properties. Spring BeanUtils:

public class ProtoPojoUtilWithBeanUtils {

    public static void toProto(Message.Builder destProtoBuilder, Object srcPojo) throws ProtoPojoConversionException {
        // Message is an immutable class. There are no setter methods
        try {
            BeanUtils.copyProperties(srcPojo, destProtoBuilder);
        } catch (Exception e) {
            throw newProtoPojoConversionException(e.getMessage(), e); }}public static <PojoType> PojoType toPojo(Class<PojoType> destPojoKlass, Message srcMessage)
        throws ProtoPojoConversionException {
        try {
            PojoType destPojo = destPojoKlass.newInstance();
            BeanUtils.copyProperties(srcMessage, destPojo);
            return destPojo;
        } catch (Exception e) {
            throw newProtoPojoConversionException(e.getMessage(), e); }}}Copy the code

This implementation is bound to be problematic for several reasons

A ProtoBean does not allow null values, and a Pojo does. Copying from a Pojo to a Proto must have a non-null exception
BeanUtils matches the method name and getter/setter type. Nested types cannot be copied properly because of type mismatch
The Java generated by the Proto properties of Map and List adds Map and List to the property names, respectively. If you want to be able to copy, you need to follow this rule to specify the property names of Projo
Enum The type does not match the type of the ProtoBean. If you want to copy the type, you can use the Enum domain of the ProtoBeanget**Value()Method and name the Pojo property name accordingly

In general, BeanUtils is not suitable for this task. Only Protobuf reflection can be considered for implementation later. This is not the focus of this article, so let’s move on to another implementation.

Indirect conversion (currency exchange)

The conversion is done through a unified medium, just like money. For example, if a currency wants to be converted into yen, the bank will first convert the currency into dollars, then dollars into yen, and vice versa.

[ A ] <--> [ C ] <--> [ B ]
Copy the code

In the implementation, we can use platform-independent, language-independent Json as an intermediate C, converting ProtoBean A to Json C, and then converting Json C to ProtoBean B objects. This method will be explained in detail below.

Code implementation

ProtoBean can be transformed to Json tool has two, one is com. Google. Protobuf/protobuf – Java – util, The other is com.googlecode.protobuf-java-format/protobuf-java-format. The performance and effectiveness of the two are yet to be compared. Here are using com. Google. Protobuf/protobuf – Java – util, the reason is that protobuf – Java – the format of JsonFormat will Map format {” key “:” “, “value” : “”}, and the JsonFormat in protobuf-java-util can be serialized to the desired key-value structure, which also conforms to the POJO-to-JSON format.

<! -- https://mvnrepository.com/artifact/com.google.protobuf/protobuf-java-util -->
<dependency>
    <groupId>com.google.protobuf</groupId>
    <artifactId>protobuf-java-util</artifactId>
    <version>3.7.1</version>
</dependency>

<! -- https://mvnrepository.com/artifact/com.googlecode.protobuf-java-format/protobuf-java-format -->
<dependency>
    <groupId>com.googlecode.protobuf-java-format</groupId>
    <artifactId>protobuf-java-format</artifactId>
    <version>1.4</version>
</dependency>
Copy the code

For Pojo and Json conversion, Gson is used here because both Protobuf and Pojo are from Google home.

The complete implementation is as follows: Protobeanutils.jave

import java.io.IOException;

import com.google.gson.Gson;
import com.google.protobuf.Message;
import com.google.protobuf.util.JsonFormat;

The getter and setter fields of the two objects converted to each other must match exactly. * In addition, for ProtoBean enum and bytes, follow the following rules when converting to POJO:  * <ol> * <li>enum -> String</li> * <li>bytes -> base64 String</li> * </ol> *@author Yang Guanrong
 * @date2019/08/18 declared * /
public class ProtoBeanUtils {

    /** * Convert ProtoBean objects to POJO objects **@paramDestPojoClass Specifies the class type of the target POJO object@paramSourceMessage ProtoBean instance * that contains data@param<PojoType> Class type paradigm * for the target POJO object@return
     * @throws IOException
     */
    public static <PojoType> PojoType toPojoBean(Class<PojoType> destPojoClass, Message sourceMessage)
        throws IOException {
        if (destPojoClass == null) {
            throw new IllegalArgumentException
                ("No destination pojo class specified");
        }
        if (sourceMessage == null) {
            throw new IllegalArgumentException("No source message specified");
        }
        String json = JsonFormat.printer().print(sourceMessage);
        return new Gson().fromJson(json, destPojoClass);
    }

    /** * Convert POJO objects to ProtoBean objects **@paramDestBuilder Target Message object Builder class *@paramSourcePojoBean POJO object * that contains data@return
     * @throws IOException
     */
    public static void toProtoBean(Message.Builder destBuilder, Object sourcePojoBean) throws IOException {
        if (destBuilder == null) {
            throw new IllegalArgumentException
                ("No destination message builder specified");
        }
        if (sourcePojoBean == null) {
            throw new IllegalArgumentException("No source pojo specified");
        }
        String json = newGson().toJson(sourcePojoBean); JsonFormat.parser().merge(json, destBuilder); }}Copy the code

Like Protobuf/Json conversion, the above implementation cannot handle Any data. You need to add your own TypeRegirstry to do the conversion.

A TypeRegistry is used to resolve Any messages in the JSON conversion. You must provide a TypeRegistry containing all message types used in Any message fields, Distinctive Or the JSON conversion will fail because data in Any message fields is undistinctive. You don’t need to supply a TypeRegistry if you don’t use Any message fields.

Class JsonFormat.TypeRegistry @JavaDoc

The way to add TypeRegistry is as follows:

// https://codeburst.io/protocol-buffers-part-3-json-format-e1ca0af27774
final var typeRegistry = JsonFormat.TypeRegistry.newBuilder()
        .add(ProvisionVmCommand.getDescriptor())
        .build();
final var jsonParser = JsonFormat.parser()
        .usingTypeRegistry(typeRegistry);

final var envelopeBuilder = VmCommandEnvelope.newBuilder();
jsonParser.merge(json, envelopeBuilder);
Copy the code

test

A Pojo class that matches the Proto file, baseDatapojo.java

import lombok.*;import java.util.List;import java.util.Map;/ * * *@author Yang Guanrong * @date2019/09/03 who fell * /@Getter@Setter@ToString@NoArgsConstructor@AllArgsConstructor(access = AccessLevel.PRIVATE)@Builderpublic class BaseDataPojo {    private double doubleVal;    private float floatVal;    private int int32Val;    private long int64Val;    private int uint32Val;    private long uint64Val;    private int sint32Val;    private long sint64Val;    private int fixed32Val;    private long fixed64Val;    private int sfixed32Val;    private long sfixed64Val;    private boolean boolVal;    private String stringVal;    private String bytesVal;    private String enumVal;    private List<String> reStrVal;    privateMap<String, BaseDataPojo> mapVal; }Copy the code

The test class ProtoBeanUtilsTest. Java

package io.gitlab.donespeak.javatool.toolprotobuf.withjsonformat;

import static org.junit.Assert.*;

import java.io.IOException;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.HashMap;
import java.util.Map;

import org.junit.Test;

import com.google.common.io.BaseEncoding;
import com.google.protobuf.ByteString;

import io.gitlab.donespeak.javatool.toolprotobuf.bean.BaseDataPojo;
import io.gitlab.donespeak.javatool.toolprotobuf.proto.DataTypeProto;

/ * * *@author Yang Guanrong
 * @date2019/09/04 14:05 * /
public class ProtoBeanUtilsTest {

    private DataTypeProto.BaseData getBaseDataProto(a) {
        DataTypeProto.BaseData baseData = DataTypeProto.BaseData.newBuilder()
            .setDoubleVal(100.123 D)
            .setFloatVal(12.3 F)
            .setInt32Val(32)
            .setInt64Val(64)
            .setUint32Val(132)
            .setUint64Val(164)
            .setSint32Val(232)
            .setSint64Val(264)
            .setFixed32Val(332)
            .setFixed64Val(364)
            .setSfixed32Val(432)
            .setSfixed64Val(464)
            .setBoolVal(true)
            .setStringVal("ssss.. tring")
            .setBytesVal(ByteString.copyFromUtf8("itsbytes"))
            .setEnumVal(DataTypeProto.Color.BLUE)
            .addReStrVal("re-item-0")
            .addReIntVal(33)
            .putMapVal("m-key", DataTypeProto.BaseData.newBuilder()
                .setStringVal("base-data")
                .build())
            .build();

        return baseData;
    }

    public BaseDataPojo getBaseDataPojo(a) {
        Map<String, BaseDataPojo> map = new HashMap<>();
        map.put("m-key", BaseDataPojo.builder().stringVal("base-data").build());

        BaseDataPojo baseDataPojo = BaseDataPojo.builder()
            .doubleVal(100.123 D)
            .floatVal(12.3 F)
            .int32Val(32)
            .int64Val(64)
            .uint32Val(132)
            .uint64Val(164)
            .sint32Val(232)
            .sint64Val(264)
            .fixed32Val(332)
            .fixed64Val(364)
            .sfixed32Val(432)
            .sfixed64Val(464)
            .boolVal(true)
            .stringVal("ssss.. tring")
            .bytesVal("itsbytes")
            .enumVal(DataTypeProto.Color.BLUE.toString())
            .reStrVal(Arrays.asList("re-item-0"))
            .reIntVal(new int[] {33})
            .mapVal(map)
            .build();

        return baseDataPojo;
    }

    @Test
    public void toPojoBean(a) throws IOException {
        DataTypeProto.BaseData baseDataProto = getBaseDataProto();
        BaseDataPojo baseDataPojo = ProtoBeanUtils.toPojoBean(BaseDataPojo.class, baseDataProto);

        // System.out.println(new GsonBuilder().setPrettyPrinting().create().toJson(baseDataPojo));

        asserEqualsVerify(baseDataPojo, baseDataProto);
    }

    @Test
    public void toProtoBean(a) throws IOException {
        BaseDataPojo baseDataPojo = getBaseDataPojo();

        DataTypeProto.BaseData.Builder builder = DataTypeProto.BaseData.newBuilder();
        ProtoBeanUtils.toProtoBean(builder, baseDataPojo);
        DataTypeProto.BaseData baseDataProto = builder.build();

        // System.out.println(new GsonBuilder().setPrettyPrinting().create().toJson(baseDataPojo));
        // Gson cannot be used to convert messages (which contain nested structures, and nested messages contain nested structures), the stack will overflow
        // Because Protobuf has no null value
        // System.out.println(JsonFormat.printer().print(baseDataProto));

        asserEqualsVerify(baseDataPojo, baseDataProto);
    }

    private void asserEqualsVerify(BaseDataPojo baseDataPojo, DataTypeProto.BaseData baseDataProto) {
        assertTrue((baseDataPojo == null) = = (! baseDataProto.isInitialized()));if(baseDataPojo == null) {
            return;
        }
        assertEquals(baseDataPojo.getDoubleVal(), baseDataProto.getDoubleVal(), 0.0000001 D);
        assertEquals(baseDataPojo.getFloatVal(), baseDataProto.getFloatVal(), 0.00000001 D);
        assertEquals(baseDataPojo.getInt32Val(), baseDataProto.getInt32Val());
        assertEquals(baseDataPojo.getInt64Val(), baseDataProto.getInt64Val());
        assertEquals(baseDataPojo.getUint32Val(), baseDataProto.getUint32Val());
        assertEquals(baseDataPojo.getUint64Val(), baseDataProto.getUint64Val());
        assertEquals(baseDataPojo.getSint32Val(), baseDataProto.getSint32Val());
        assertEquals(baseDataPojo.getSint64Val(), baseDataProto.getSint64Val());
        assertEquals(baseDataPojo.getFixed32Val(), baseDataProto.getFixed32Val());
        assertEquals(baseDataPojo.getInt64Val(), baseDataProto.getInt64Val());
        assertEquals(baseDataPojo.isBoolVal(), baseDataProto.getBoolVal());
        assertEquals(baseDataPojo.isBoolVal(), baseDataProto.getBoolVal());
        assertEquals(baseDataPojo.getStringVal(), baseDataProto.getStringVal());
        // Convert ByteString to base64 Strings
        if(baseDataPojo.getBytesVal() == null) {
            // The default value is ""
            assertTrue(baseDataProto.getBytesVal().isEmpty());
        } else {
            assertEquals(baseDataPojo.getBytesVal(), BaseEncoding.base64().encode(baseDataProto.getBytesVal().toByteArray()));
        }
        / / Enum String
        if(baseDataPojo.getEnumVal() == null) {
            // The default is 0
            assertEquals(DataTypeProto.Color.forNumber(0), baseDataProto.getEnumVal());
        } else {
            assertEquals(baseDataPojo.getEnumVal(), baseDataProto.getEnumVal().toString());
        }
        if(baseDataPojo.getReStrVal() == null) {
            // The default list is empty
            assertEquals(0, baseDataProto.getReStrValList().size());
        } else {
            assertEquals(baseDataPojo.getReStrVal().size(), baseDataProto.getReStrValList().size());
            for(int i = 0; i < baseDataPojo.getReStrVal().size(); i ++) { assertEquals(baseDataPojo.getReStrVal().get(i), baseDataProto.getReStrValList().get(i)); }}if(baseDataPojo.getReIntVal() == null) {
            // The default list is empty
            assertEquals(0, baseDataProto.getReIntValList().size());
        } else {
            assertEquals(baseDataPojo.getReIntVal().length, baseDataProto.getReIntValList().size());
            for(int i = 0; i < baseDataPojo.getReIntVal().length; i ++) {
                int v1 = baseDataPojo.getReIntVal()[i];
                intv2 = baseDataProto.getReIntValList().get(i); assertEquals(v1, v2); }}if(baseDataPojo.getMapVal() == null) {
            // The default is empty
            assertEquals(0, baseDataProto.getMapValMap().size());
        } else {
            assertEquals(baseDataPojo.getMapVal().size(), baseDataProto.getMapValMap().size());
            for(Map.Entry<String, DataTypeProto.BaseData> entry: baseDataProto.getMapValMap().entrySet()) { asserEqualsVerify(baseDataPojo.getMapVal().get(entry.getKey()), entry.getValue()); }}}@Test
    public void testDefaultValue(a) {
        DataTypeProto.BaseData baseData = DataTypeProto.BaseData.newBuilder()
            .setInt32Val(0)
            .setStringVal("")
            .addAllReStrVal(new ArrayList<>())
            .setBoolVal(false)
            .setDoubleVal(3.14 D)
            .build();
        // Default values are not printed
        / / double_val: 3.14System.out.println(baseData); }}Copy the code

The above tests can be completed and passed, with special attention to the default values for the properties of the class type. There is no null value in Protobuf, so the default value for classtype attributes will not be null. But when you map to a Pojo, the default value for the ProtoBean is converted to the default value for the Pojo, which is the default value for the data type in Java.

Default value list

type	Proto default values	Pojo default values
int	0	0
long	0L	0L
float	0F	0F
double	0D	0D
boolean	false	false
string	“”	null
BytesString	“”	(string) null
enum	0	(string) null
message	{}	(object) null
repeated	[]	(List/Array) null
map	[]	(Map) null

This list is only a brief list, for more detailed information, see protobuf’s official documentation. Or, as an alternative, create a ProtoBean with all the data types, such as DataTypeProto.BaseData, and look at the no-argument constructor in that class to get a sense of what the default value is.

.private static final DataTypeProto.BaseData DEFAULT_INSTANCE;
static {
    DEFAULT_INSTANCE = new DataTypeProto.BaseData();
}
private BaseData(a) {
    stringVal_ = "";
    bytesVal_ = com.google.protobuf.ByteString.EMPTY;
    enumVal_ = 0;
    reStrVal_ = com.google.protobuf.LazyStringArrayList.EMPTY;
    reIntVal_ = emptyIntList();
}
public static iDataTypeProto.BaseData getDefaultInstance(a) {
    returnDEFAULT_INSTANCE; }...Copy the code

Again, protobuf has no null value, you can’t set a null value, you can’t get a null value.

Protobuf support Java data types to see: com. Google. Protobuf. Descriptors. FieldDescriptor. JavaType

Reference and recommended reading

Translate Protobuf and Json to @Donespeak
Protocol Buffers @Google Developers
Com. Google. Protobuf/protobuf – ja…
Com. Googlecode. Protobuf – Java – format/protobuf – ja…
Protocol Buffers, Part 3 — JSON Format
Converting Protocol Buffers data to Json and back with Gson Type Adapters
Any source @ dead simple
Any Official document @office

mo4tech.com (Moment For Technology) is a global community with thousands techies from across the global hang out!Passionate technologists, be it gadget freaks, tech enthusiasts, coders, technopreneurs, or CIOs, you would find them all here.

Protobuf/POJO conversion – via Json

preface

The basic idea of transformation

Direct conversion of

Indirect conversion (currency exchange)

Code implementation

test

Reference and recommended reading

Protobuf/POJO conversion – via Json

preface

The basic idea of transformation

Direct conversion of

Indirect conversion (currency exchange)

Code implementation

test

Reference and recommended reading

Related Posts

Golang10-go Language Web server

The end of the year is near. Goodbye 2021

Java based _04 | formally to the world of Java (classes and objects, classes and objects of memory allocation, rounding, constructor)