preface

ProtoBuf

Protocol Buffers is a language-independent, platform-independent, extensible method of serializing structured data for use with communication protocols, data stores, and so on. Protocol Buffers is a flexible, efficient, automated method of serializing structured data – comparable to XML, but smaller (3 to 10 times), faster (20 to 100 times) and simpler than XML. You can define the structure of the data and then use specially generated source code to easily write and read the structure in various data streams in various languages. You can even update data structures without breaking deployed programs compiled from older data structures.

In brief, ProtoBuf is a structured data serialization [1] method, which is a simple analogy to XML[2] and has the following characteristics:

  • Language independent, platform independent. ProtoBuf supports multiple languages such as Java, C++ and Python, and supports multiple platforms
  • Efficient. That is, smaller (3 to 10 times), faster (20 to 100 times), and simpler than XML
  • Good scalability and compatibility. You can update data structures without affecting or breaking old programs

protostuff

Protostuff is a serialization method based on protobuf implementation. The most obvious advantage over Protobuf is that it does not require us to write a.proto file to implement serialization with almost no performance loss

I. Pressure measuring tools

Contains manometer tools and manometer results

Github.com/eishay/jvm-…

Two, contrast test

In testing, ProtoStuff + Compress is a better option

The json structure ProtoStuff + compress is 77% of the original solution (Gson+Snappy)

The XML structure ProtoStuff + compress is 84% of the original solution (Gson+Snappy)

Three test www.cnblogs.com/xiaoMzjm/p/…

2.1 Tool Code

import com.google.common.collect.Maps; import io.protostuff.LinkedBuffer; import io.protostuff.ProtostuffIOUtil; import io.protostuff.Schema; import io.protostuff.runtime.RuntimeSchema; import java.util.Map; / * * * * * / pb serialization tool public class ProtoStuffSerializerUtils {private static Map < class <? >, Schema<? >> schemaCacheMap = Maps.newConcurrentMap(); /** * Serialize * @param <T> Serialize object type * @return Serialized data */ @suppresswarnings (" Unchecked ") public static <T> byte[] serialize(T object) { Class<T> cls = (Class<T>) object.getClass(); LinkedBuffer buffer = LinkedBuffer.allocate(LinkedBuffer.DEFAULT_BUFFER_SIZE * 2 * 20); byte[] bytes; try { Schema<T> schema = getSchema(cls); bytes = serializeInner(object, schema, buffer); } catch (Exception e) { throw new IllegalStateException(e.getMessage(), e); } finally { buffer.clear(); } return bytes; } / deserialization * * * * @ param data serialized data * @ param CLS serialized object type * @ param < T > object serialization generic * @ return after deserialization of the object * / public static < T > T deserialize(byte[] data, Class<T> cls) { T deserialize; try { Schema<T> schema = getSchema(cls); deserialize = deserializeInner(data, schema.newMessage(), schema); } catch (Exception e) { throw new IllegalStateException(e.getMessage(), e); } return deserialize; } /** * serialization ** @param SRC * @param schema * @param buffer * @param <T> * @return */ private static <T> byte[] serializeInner(final T src, final Schema<T> schema, final LinkedBuffer buffer) { return ProtostuffIOUtil.toByteArray(src, schema, buffer); } /** * deserialization ** @param data * @param result * @param schema * @param <T> * @return */ private static <T> T deserializeInner(final byte[] data, final T result, final Schema<T> schema) { ProtostuffIOUtil.mergeFrom(data, result, schema); return result; } /** * Obtain schema ** @param CLS data class * @param <T> generic * @return schema */ @suppresswarnings ("unchecked") private static <T> Schema<T> getSchema(Class<T> CLS) {Schema<T> schemaCache = (Schema<T>) schemacachemap.get (CLS);  if (null ! = schemaCache) { return schemaCache; } else { RuntimeSchema<T> schemaNew = RuntimeSchema.createFrom(cls); schemaCacheMap.put(cls, schemaNew); return schemaNew; }}}Copy the code

2.2 Test Code

package utils; import com.jd.finance.jrpayorder.rest.vo.QueryOrderResponse; import com.jd.finance.jrpaypassword.common.utils.GsonUtils; import com.jd.finance.jrpaypassword.domain.vo.OrderPinCacheVo; import com.jd.finance.jrpaypassword.rest.vo.OrderPinCheckPwdCacheRequest; import com.jd.finance.jrpaypassword.utils.CompressUtils; import com.jd.jr.mall.order.service.utils.ConverterUtils; import com.jdd.mall.order.utils.MallOrderXmlUtils; import com.jdd.mall.order.utils.ProtoStuffSerializerUtils; import org.apache.commons.lang.StringUtils; import org.junit.Test; import org.slf4j.Logger; import org.slf4j.LoggerFactory; public class JvmSerializerTest { private static final Logger LOGGER = LoggerFactory.getLogger(JvmSerializerTest.class); @Test public void testProtoBuff() throws Exception { int count = 20000; testJson(count); testXml(count); LOGGER.error("-------------------------"); testJson(count); testXml(count); } private void testJson(int count) throws Exception { long size = 0; OrderPinCheckPwdCacheRequest request = MallOrderXmlUtils.genOrderPinCheckPwdCacheReq("1", "testApp2016#$%", StringUtils.EMPTY); OrderPinCacheVo orderPinCacheVo = ConverterUtils.writeRequest2XmlVo(request); QueryOrderResponse orderResponse = new QueryOrderResponse(); orderResponse = ConverterUtils.xmlVo2Json(orderPinCacheVo, orderResponse); long start = System.nanoTime(); for (int i = 0; i < count; i++) { byte[] serialize = ProtoStuffSerializerUtils.serialize(orderResponse); byte[] compress = CompressUtils.compressByte(serialize); size = compress.length; byte[] decompressByte = CompressUtils.decompressByte(compress); QueryOrderResponse deserialize = ProtoStuffSerializerUtils.deserialize(decompressByte, QueryOrderResponse.class); } long end = System.nanoTime(); long use = end - start; LOGGER.info("pb j time(ns):{}, count:{}, size:{}", use, count, size); start = System.nanoTime(); for (int i = 0; i < count; i++) { String value = GsonUtils.toJson(orderResponse); byte[] compress = CompressUtils.compress(value, CompressUtils.CPMPRESS_TYPE_SNAPPY); size = compress.length; String decompress = CompressUtils.decompress(compress); QueryOrderResponse deserialize = GsonUtils.fromJson(decompress, QueryOrderResponse.class); } end = System.nanoTime(); use = end - start; LOGGER.info("gc j time(ns):{}, count:{}, size:{}", use, count, size); } private void testXml(int count) throws Exception { long size = 0; OrderPinCheckPwdCacheRequest request = MallOrderXmlUtils.genOrderPinCheckPwdCacheReq("1", "testApp2016#$%", StringUtils.EMPTY); OrderPinCacheVo orderPinCacheVo = ConverterUtils.writeRequest2XmlVo(request); long start = System.nanoTime(); for (int i = 0; i < count; i++) { byte[] serialize = ProtoStuffSerializerUtils.serialize(orderPinCacheVo); byte[] compress = CompressUtils.compressByte(serialize); size = compress.length; byte[] decompressByte = CompressUtils.decompressByte(compress); OrderPinCacheVo deserialize = ProtoStuffSerializerUtils.deserialize(decompressByte, OrderPinCacheVo.class); } long end = System.nanoTime(); long use = end - start; LOGGER.info("pb x time(ns):{}, count:{}, size:{}", use, count, size); start = System.nanoTime(); for (int i = 0; i < count; i++) { String value = GsonUtils.toJson(orderPinCacheVo); byte[] compress = CompressUtils.compress(value, CompressUtils.CPMPRESS_TYPE_SNAPPY); size = compress.length; String decompress = CompressUtils.decompress(compress); OrderPinCacheVo deserialize = GsonUtils.fromJson(decompress, OrderPinCacheVo.class); } end = System.nanoTime(); use = end - start; LOGGER.info("gc x time(ns):{}, count:{}, size:{}", use, count, size); }}Copy the code

2.3 Test Data

// LinkedBuffer.default_buffer_size * 2 * 20 20-11-22.00:37:05.965 [main] INFO JvmSerializerTest -pb j time(ns):580574613, count:20000, Size :4222 20-11-22.00:37:09.719 [main] INFO jVMSerializertest-gc j time(ns):3753833917, count:20000, Size :3597 time GC /pb = 6.46 SIZE GC /pb = 0.85 20-11-22.00:37:10.393 [main] INFO JvmSerializerTest -pb x time(ns):673694103, count:20000, Size :13919 20-11-22.00:37:18.991 [main] INFO jVMSerializertest-gc x time(ns):8597234849, count:20000, Size :6769 time GC /pb = 12.76 size GC /pb = 0.48Copy the code
// LinkedBuffer.default_buffer_size * 2 * 20 PB add snappy compress 20-11-22.00:57:11.856 [main] INFO JvmSerializerTest  - pb j time(ns):1131198655, count:20000, Size :2755 20-11-22.00:57:15.618 [main] INFO jVMSerializertest-gc j time(ns):3761834247, count:20000, Size :3597 time GC/PBC = 3.32 size GC/PBC = 1.30 time PBC /pb = 1.94 size PBC/PB = 0.65 20-11-22.00:57:18.455 [main] INFO  JvmSerializerTest - pb x time(ns):2835673303, count:20000, Size :5668 20-11-22.00:57:27.005 [main] INFO JvmSerializerTest -gc x time(ns):8550253723, count:20000, Size :6769 time GC/PBC = 3.01 size GC/PBC = 1.19 time PBC /pb = 4.2 size PBC /pb = 0.40Copy the code

Serialization [1] : Conversion of structured data or objects into a format that can be stored and transferred (e.g. over a network) with the assurance that the serialization can be reconstructed back to the original structured data or object later (possibly in another computing environment). For more information, see wikipedia.

Analogy with XML[2] : here it mainly refers to the analogy of serialization in data communication and data storage application scenarios, but personally, XML as an extended markup language is fundamentally different from ProtoBuf.