0 foreword
When interacting with data within the Java team, there are several commonly used methods, such as Java native serialization method, JSON serialization method, ProtoBuf serialization method, etc. For comparative experiments, please refer to Performance Comparison of Several Serialization Methods.
It can be seen from the comparative experiment that ProtoBuf has the best performance, because of its efficient serialization performance and excellent cross-platform capability, which is widely used in cross-platform data interaction. But when you don’t need to think about cross-platform data interaction, ProtoBuf’s cumbersome process becomes an annoying “stain.”
The general process of data transfer based on Ptoro is to first write the Proto file, then manually compile the Proto file to generate Java classes, then instantiate the Java classes and populate the data, and finally compile the data into binary data for transmission. Among them, writing and manually compiling Proto files is a tedious process (such as referencing Proto to each other in complex projects), and Java classes generated by Proto use creator mode (namely Bulider mode). In fact, the assignment process after instantiation is inconsistent with the style of the project history code, which brings about major changes.
In order to solve this problem, this paper will introduce the JProtoBuf tool developed by the Baidu team, and summarize the matters needing attention in the process of using it, as well as verify its consistency with the ProtoBuf function developed by the Google team through experiments.
Introduction to Jprotobuf
1.1 define
Protobuf is a Java program for the development of a set of simple class library, the purpose is to simplify the use of Java language Protobuf class library. With JProtoBuf, you can define field types using Java annotations without having to understand proto file manipulation and syntax.
Making address:
https://github.com/jhunters/j…
1.2 the principle
Here’s how JProtoBuf works:
- Scan annotations on the class for analysis (similar to ProtoBuf reading Proto files for analysis)
- According to the result of annotation analysis, dynamically generate Java code for ProtoBuf serialization and deserialization
- Using JDK6 and above Code Compile API to compile and load into ClassLoader
1.3 performance
The main performance of JProtoBuf is to scan the annotations on the class, dynamically generating the code compilation process. It is almost as efficient as the code ProtoBuf generates during serialization and deserialization. If you use a precompiled plug-in, you don’t need to generate and compile code at run time, which is more efficient.
1.4 the characteristics of
- No need to write proto files and tedious manual compilation process, support annotation method based on POJO objects, convenient and quick. ProtoBuf supports all types, including object nesting, array, and enumeration types
- Proxy objects are dynamically generated according to proto files, which can save POJO object writing work. Full support for all functions of proto files, including inline objects, anonymous objects, enumerated types
- It provides the function of automatically generating proto files from POJO object annotation, so as to facilitate the management and maintenance of proto description files
- Provide pre-compiled Maven plug-ins to further improve performance
- Added precompiled Gradle plugin
- 2. X version. Support TimeStamp type, consistent with native ProtoBuf. Date type is supported and docs are passed using long type
2 Jprotobuf installation
The Maven code is as follows:
<! -- jProtobuf --> <dependency> <groupId>com.baidu</groupId> <artifactId> jProtobuf </artifactId> <version>2.4.5</version> </dependency> <dependency> <groupId>com.baidu</groupId> <artifactId>jprotobuf-precompile-plugin</artifactId> The < version > 2.2.2 < / version > < / dependency >3 Case data
Test idea: build the same test data through their respective methods, serialize and de-sequence the differences between the two. Note: Try to design case data to include all data types.
3.1 ProtoBuf data definition
The organizational structure is shown in the figure, in which Data relies on Student, Student relies on Person, and Teacher as the third-party extension package for testing the extension function.
The data.proto code is as follows:
// Google Protocol Buffers Version 3.
syntax = "proto3";
// Package name.
package prcticeProto.messages;
// Options for code generation.
option java_package = "learnProto.practiceTest.protoModel";
option java_outer_classname = "SchoolModel";
option java_multiple_files = true;
// import packages
import "google/protobuf/any.proto";
import "practiceProto/categories/student.proto";
message School {
message Location{
string name=1;
uint32 id=2;
}
Location schoolLocation = 1;
bool isOpen =2;
repeated categories.Student allStudents = 3;
google.protobuf.Any extend =4;
}Student. Proto code is as follows:
// Google Protocol Buffers Version 3.
syntax = "proto3";
// Package name.
package prcticeProto.categories;
// Options for code generation.
option java_package = "learnProto.practiceTest.protoModel";
option java_outer_classname = "StudentModel";
option java_multiple_files = true;
// import packages
import "practiceProto/base/person.proto";
message Student {
base.Person baseInfo = 1;
fixed32 calssId = 2;
sint32 score = 3;
}The person.proto code is as follows:
// Google Protocol Buffers Version 3.
syntax = "proto3";
// Package name.
package prcticeProto.base;
// Options for code generation.
option java_package = "learnProto.practiceTest.protoModel";
option java_outer_classname = "PersonModel";
option java_multiple_files = true;
message Person{
message Location{
string placeName=1;
uint64 placeId=2;
}
enum Gender{
man=0;
woman=1;
}
string name = 1;
int32 age=2;
Gender gender=3;
float height=4;
double weight=5;
Location location=6;
}The code of teacher.proto is as follows:
// Google Protocol Buffers Version 3.
syntax = "proto3";
// Options for code generation.
option java_package = "learnProto.practiceTest.protoModel";
option java_outer_classname = "TeacherModel";
option java_multiple_files = true;
message Teacher{
string name = 1;
int32 age=2;
}3.2 JProtoBuf data definition
The organizational structure is shown in the figure.
The syntax for the definition of Jprotobuf is as follows:
https://github.com/jhunters/j…
The school code is as follows:
package learnProto.jprotobuf.model;
import com.baidu.bjf.remoting.protobuf.Any;
import com.baidu.bjf.remoting.protobuf.FieldType;
import com.baidu.bjf.remoting.protobuf.annotation.Protobuf;
import com.baidu.bjf.remoting.protobuf.annotation.ProtobufClass;
import java.util.List;
@ProtobufClass
public class School {
@ProtobufClass
public static class Location{
@Protobuf(fieldType= FieldType.STRING, order=1)
private String name;
@Protobuf(fieldType= FieldType.UINT32, order=2)
private Integer id;
public String getName() {
return name;
}
public Location setName(String name) {
this.name = name;
return this;
}
public Integer getId() {
return id;
}
public Location setId(Integer id) {
this.id = id;
return this;
}
}
@Protobuf(fieldType= FieldType.OBJECT, order=1)
private Location schoolLocation;
@Protobuf(fieldType= FieldType.BOOL, order=2)
private Boolean isOpen;
@Protobuf(fieldType= FieldType.OBJECT, order=3)
private List<Student> allStudents;
@Protobuf(fieldType = FieldType.OBJECT,order = 4)
private Any extend;
public Any getExtend() {
return extend;
}
public School setExtend(Any extend) {
this.extend = extend;
return this;
}
public Location getSchoolLocation() {
return schoolLocation;
}
public School setSchoolLocation(Location schoolLocation) {
this.schoolLocation = schoolLocation;
return this;
}
public Boolean getOpen() {
return isOpen;
}
public School setOpen(Boolean open) {
isOpen = open;
return this;
}
public List<Student> getAllStudents() {
return allStudents;
}
public School setAllStudents(List<Student> allStudents) {
this.allStudents = allStudents;
return this;
}
}Student code is as follows:
package learnProto.jprotobuf.model; import com.baidu.bjf.remoting.protobuf.FieldType; import com.baidu.bjf.remoting.protobuf.annotation.Protobuf; import com.baidu.bjf.remoting.protobuf.annotation.ProtobufClass; @ProtobufClass public class Student{ @Protobuf(fieldType=FieldType.OBJECT, order=1) private Person baseInfo; @Protobuf(fieldType=FieldType.FIXED32, order=2) private Integer calssId; @Protobuf(fieldType=FieldType.SINT32, order=3) private Integer score; public Person getBaseInfo() { return baseInfo; } public Student setBaseInfo(Person baseInfo) { this.baseInfo = baseInfo; return this; } public Integer getCalssId() { return calssId; } public Student setCalssId(Integer calssId) { this.calssId = calssId; return this; } public Integer getScore() { return score; } public Student setScore(Integer score) { this.score = score; return this; }}The Person code is as follows:
package learnProto.jprotobuf.model; import com.baidu.bjf.remoting.protobuf.FieldType; import com.baidu.bjf.remoting.protobuf.annotation.Protobuf; import com.baidu.bjf.remoting.protobuf.annotation.ProtobufClass; @ProtobufClass public class Person { @ProtobufClass public enum Gender { MAN(0),WOMAN(1); @Protobuf(fieldType= FieldType.INT32, order=1) private final Integer value; Gender(Integer value) { this.value = value; } public Integer toValue() { return this.value; } public Integer value() { return toValue(); } } @ProtobufClass public static class Location{ @Protobuf(fieldType= FieldType.STRING, order=1) private String placeName; @Protobuf(fieldType= FieldType.UINT64, order=2) private Long placeId; public String getPlaceName() { return placeName; } public Location setPlaceName(String placeName) { this.placeName = placeName; return this; } public Long getPlaceId() { return placeId; } public Location setPlaceId(Long placeId) { this.placeId = placeId; return this; } } @Protobuf(fieldType= FieldType.STRING, order=1) private String name; @Protobuf(fieldType= FieldType.INT32, order=2) private Integer age; @Protobuf(fieldType= FieldType.ENUM, order=3) private Gender gender; @Protobuf(fieldType= FieldType.FLOAT, order=4) private Float height; @Protobuf(fieldType= FieldType.DOUBLE, order=5) private Double weight; @Protobuf(fieldType= FieldType.OBJECT, order=6) private Location personLocation; public String getName() { return name; } public Person setName(String name) { this.name = name; return this; } public Integer getAge() { return age; } public Person setAge(Integer age) { this.age = age; return this; } public Gender getGender() { return gender; } public Person setGender(Gender gender) { this.gender = gender; return this; } public Float getHeight() { return height; } public Person setHeight(Float height) { this.height = height; return this; } public Double getWeight() { return weight; } public Person setWeight(Double weight) { this.weight = weight; return this; } public Location getPersonLocation() { return personLocation; } public Person setPersonLocation(Location personLocation) { this.personLocation = personLocation; return this; }}TEACHER code is as follows:
package learnProto.jprotobuf.model; import com.baidu.bjf.remoting.protobuf.annotation.ProtobufClass; @ProtobufClass public class Teacher { private Integer id; private String name; public Integer getId() { return id; } public Teacher setId(Integer id) { this.id = id; return this; } public String getName() { return name; } public Teacher setName(String name) { this.name = name; return this; }}4 Case Tests
4.1 Basic data type testing
public class jprotobufTest { public static void writeFileByte(byte[] b,String path) { File file = new File(path); long size = file.length(); try (FileOutputStream fos = new FileOutputStream(file)) { fos.write(b); fos.flush(); } catch (IOException e) { e.printStackTrace(); } } public static byte[] readFileByte(String path) { File file = new File(path); long size = file.length(); byte[] b = new byte[(int) size]; try (InputStream fis = new FileInputStream(file)) { if (fis.read(b) < 0) { return null; } } catch (IOException e) { e.printStackTrace(); } return b; {} public void protobuff (String path). / / instantiate learnProto practiceTest. ProtoModel. Person. The Builder builderPerson = learnProto.practiceTest.protoModel.Person.newBuilder().setAge(10).setGender(learnProto.practiceTest.protoModel.Person.Ge Nder. Woman). Elegantly-named setName (" Tom "). SetHeight (100.00 f) setWeight (100.00 d) setLocation ( learnProto.practiceTest.protoModel.Person.Location.newBuilder().setPlaceId(123l).setPlaceName("hubei") ); learnProto.practiceTest.protoModel.School school = learnProto.practiceTest.protoModel.School.newBuilder() .setIsOpen(true) .setSchoolLocation(learnProto.practiceTest.protoModel.School.Location.newBuilder().setId(123).setName("hubei")) .addAllStudents(Student.newBuilder().setBaseInfo(builderPerson).setCalssId(10).setScore(-1)) .addAllStudents(Student.newBuilder().setBaseInfo(builderPerson).setCalssId(10).setScore(1)) .build(); Try {// serialize byte[] bytes = school.toByteArray(); writeFileByte(bytes,path); Println ("protobuf: "+ Arrays.toString(bytes)+", number of bytes: "+ Arrays.toString(bytes) "); // Deserialize Byte [] Bytes1 = ReadFileByte (Path); learnProto.practiceTest.protoModel.School parseFrom = learnProto.practiceTest.protoModel.School.parseFrom(bytes1); } catch (InvalidProtocolBufferException e) { e.printStackTrace(); } public void jProtoBuf (String Path){// Person Person = new Person (). SetAge (10). SetGender (Person. Gender. MAN). Elegantly-named setName (" Tom "). SetHeight (100.00 f) setWeight (100.00 d) setPersonLocation ( new Person.Location().setPlaceId(123l).setPlaceName("hubei") ); ArrayList<learnProto.jprotobuf.model.Student> studentArrayList = new ArrayList<>(); studentArrayList.add(new learnProto.jprotobuf.model.Student().setCalssId(10).setScore(-1).setBaseInfo(person)); studentArrayList.add(new learnProto.jprotobuf.model.Student().setCalssId(10).setScore(1).setBaseInfo(person)); School sch = new School().setOpen(true).setAllStudents(studentArrayList).setSchoolLocation(new School.Location().setId(123).setName("hubei")); Try {// serialize Codec<School> simplePedec = protoBufProxy.create (school.class); byte[] b = simpleTypeCodec.encode(sch); writeFileByte(b,path); Println (" + Arrays.toString(b)+", "+b ", "+b "); // deserialize byte[] bytes = ReadFileByte (Path); School newStt = simpleTypeCodec.decode(bytes); } catch (IOException e) { e.printStackTrace(); } } @Test public void test(){ protobuff("C:UsersadminDesktopProtodemo.pack"); jprotobuf("C:UsersadminDesktopJprotodemo.pack"); }}Running results:
Protobuf data after serialization: [10, 9, 10, 5, 104, 117, 98, 101, 105, 16, 123, 16, 1, 26, 43, 10, 34, 10, 3, 84, 111, 109, 16, 10, 24, 1, 37, 0, 0, -56, 66, 41, 0, 0, 0, 0, 0, 89, 64, 50, 9, 10, 5, 104, 117, 98, 101, 105, 16, 123, 21, 10, 0, 0, 0, 24, 1, 26, 43, 10, 34, 10, 3, 84, 111, 109, 16, 10, 24, 1, 37, 0, 0, -56, 66, 41, 0, 0, 0, 0, 0, 0, 89, 64, 50, 9, 10, 5, 104, 117, 98, 101, 105, 16, 123, 21, 10, 0, 0, 0, 24, 2] Number of bytes: 103 [10, 9, 10, 5, 104, 117, 98, 101, 105, 16, 123, 16, 1, 26, 43, 10, 34, 10, 3, 84, 111, 109, 16, 10, 24, 0, 37, 0, 0, 0, -56, 66, 41, 0, 0, 0, 0, 0, 89, 64, 50, 9, 10, 5, 104, 117, 98, 101, 105, 16, 123, 21, 10, 0, 0, 0, 24, 1, 26, 43, 10, 34, 10, 3, 84, 111, 109, 16, 10, 24, 0, 37, 0, 0, -56, 66, 41, 0, 0, 0, 0, 0, 0, 89, 64, 50, 9, 10, 5, 104, 117, 98, 101, 105, 16, 123, 21, 10, 0, 0, 0, 24, 2], number of bytes: 103Experiments show that the serialization method of jProtoBuf is consistent with that of ProtoBuf and can be used safely.
4.2 Third party extension testing
@Test public void schooltest() throws IOException { // jproto Any any = Any.pack(new Teacher()); School person = new School().setExtend(any); Codec<School> simpleTypeCodec = ProtobufProxy.create(School.class); Try {// Serialized Byte [] b = SimpleType Dec.Encode (Person); writeFileByte(b,"C:UsersadminDesktopjproto.pack"); Println (" + Arrays.toString(b)+", "+b ", "+b "); / / deserialization byte [] bytes = readFileByte (" C: UsersadminDesktopjproto. Pack "); School newStt = simpleTypeCodec.decode(bytes); } catch (IOException e) { e.printStackTrace(); } // proto com.google.protobuf.Any any1 = com.google.protobuf.Any.pack(learnProto.practiceTest.protoModel.Teacher.newBuilder().build()); learnProto.practiceTest.protoModel.School builderPerson = learnProto.practiceTest.protoModel.School.newBuilder() .setExtend(any1) .build(); Try {/ / serialized String path = "C: UsersadminDesktopProtodemo. Pack"; byte[] bytes = builderPerson.toByteArray(); writeFileByte(bytes,path); Println ("protobuf: "+ Arrays.toString(bytes)+", number of bytes: "+ Arrays.toString(bytes) "); // Deserialize Byte [] Bytes1 = ReadFileByte (Path); learnProto.practiceTest.protoModel.School parseFrom = learnProto.practiceTest.protoModel.School.parseFrom(bytes1); } catch (InvalidProtocolBufferException e) { e.printStackTrace(); }}Running results:
JProto serialized data: [34, 58, 10, 54, 116, 121, 112, 101, 46, 103, 111, 111, 103, 108, 101, 97, 112, 105, 115, 46, 99, 111, 109, 47, 108, 101, 97, 114, 110, 80, 114, 111, 116, 111, 46, 106, 114, 111, 116, 111, 98, 117, 102, 46, 109, 111, 100, 101, 108, Number of bytes: 60 Protobuf data after serialization: [34, 29, 10, 27, 116, 121, 112, 101, 46, 103, 111, 111, 103, 108, 101, 97, 112, 105, 115, 46, 99, 111, 109, 47, 84, 101, 97, 99, 104, 101, 114], number of bytes: 31Experiments show that the extension function of JProtoBuf can be used normally, but its mechanism may be different from ProtoBuf.
5 Usage summary
1, field annotation @ProtoBuf, can not write, do not write the case according to the default type, order according to the writing order; You can also write, custom type and order. But do not write some fields, some fields do not write, the order will be messy!
2. Build @ProtoBufClass without an argument constructor, and instantiate it with no arguments. It can only be constructed without arguments and assigned by set or add (in line with the instantiation specification of proto).
You can call _getIDL_ to get the IDL, or proto file, but all the nested structures are broken down into separate structures.
4. The return type of the setter can be set to the class itself to achieve chaining. As shown in figure:
5. When defining data for basic data types, use wrapper classes. For example, for integers, do not write int, but INTGER, as shown in the figure above. If set to int, the system initializes id=0 by default. Even if no value is assigned, the value will be written to serialization. If set to INTGER, no value is assigned and will not be written to the serialized byte order. (This is different from proto. In proto, if a field is set to a default value, the data will not be written during serialization. After deserialization, all unassigned fields will be assigned to the default value of the corresponding type. If the field is not assigned, it will be null after deserialization, and enum will be the first enumerated value).
Note: Unassigned fields are null after deserialization, not the default value.
7, Set multiple fields, some of which are not assigned, and it will not affect the size of the serialized data.
6 References
[1] [2] by comparison of several methods of serialized properties https://github.com/jhunters/j… [3] https://github.com/jhunters/j…