This is the 18th day of my participation in the August Genwen Challenge.More challenges in August

Introduction to the

Netty is powerful, because it built a lot of very useful codecs, through the use of these codecs can be very convenient to build a very powerful application, today to tell you about netty’s most basic built-in codecs.

Built-in encoder in Netty

When importing netty packages, we can see that Netty has a number of artifactids starting with netty-codec:

netty-codec
netty-codec-http
netty-codec-http2
netty-codec-memcache
netty-codec-redis
netty-codec-socks
netty-codec-stomp
netty-codec-mqtt
netty-codec-haproxy
netty-codec-dns
Copy the code

There are 10 codec packages in total, among which netty-Codec is the most basic one, and the other nine are extensions and adaptations of different protocol packages. Netty supports common and popular protocol formats, which is very powerful. Because codec is so extensive, it is not easy to explain them. This article will use Netty-Codec as an example to explain the most basic and versatile codec.

Issues to be aware of using CODEC

Although Netty provides convenient codec codecs, some coDecs need to work with Frame Detection, as we mentioned in a previous article. Frame Detection is first used to divide ByteBuf into each ByteBuf representing real data, and then it is sent to Netty’s built-in CODEC or custom CODEC for processing, so as to achieve the desired effect.

Basic CODEC built into Netty

The basic codec in Netty is base64, bytes, Compression, JSON, marshalling, Protobuf, serialization, String, and XML.

The following will explain them one by one.

base64

This codec is responsible for the conversion between ByteBuf and base64 ByteBuf. It’s all from ByteBuf to ByteBuf, but the content has changed.

There are two key classes, Base64Encoder and Base64Decoder. Because Base64Decoder is a MessageToMessageDecoder, so need to use a DelimiterBasedFrameDecoder processed in advance, common examples are as follows:

ChannelPipeline pipeline = ... ; // Decoders pipeline.addLast("frameDecoder", new DelimiterBasedFrameDecoder(80, Delimiters.nulDelimiter())); pipeline.addLast("base64Decoder", new Base64Decoder()); // Encoder pipeline.addLast("base64Encoder", new Base64Encoder());Copy the code

bytes

Bytes is used to convert an array of bytes to ByteBuf. FrameDecoder is also used before decode. It is usually used as follows:

ChannelPipeline pipeline = ... ; // Decoders pipeline.addLast("frameDecoder", new LengthFieldBasedFrameDecoder(1048576, 0, 4, 0, 4)); pipeline.addLast("bytesDecoder", new ByteArrayDecoder()); // Encoder pipeline.addLast("frameEncoder", new LengthFieldPrepender(4)); pipeline.addLast("bytesEncoder", new ByteArrayEncoder());Copy the code

compression

The content of compression package is relatively rich, mainly on the data compression and decompression services. It supports the following algorithms:

brotli
Bzip2
FastLZ
JdkZlib
Lz4
Lzf
Snappy
Zlib
Zstandard
Copy the code

Compression is especially helpful for the transmission of large data volumes. By compression, the amount of data transmitted can be saved and the transmission speed can be improved.

But compression is calculated using a specific algorithm, so it is a high CPU operation, and we need to balance network speed and CPU performance when using it.

json

The json package contains only one JsonObjectDecoder class, which is responsible for converting json objects or arrays from Byte streams into JSON objects and arrays.

JsonObjectDecoder is a direct subclass of ByteToMessageDecoder, so it does not need FrameDecoder. JsonObjectDecoder determines the starting position of the Byte array based on the matching of parentheses to distinguish which Byte data belongs to the same Json object or array.

This class is useful if you want to use JSON to transfer data.

marshalling

JBoss Marshalling is JBoss Marshalling. It is a serialization method for objects from JBoss. However, the latest API for JBoss Marshalling has not been updated for 10 years since April 27, 2011.

So we will not go into the details of this serialization content, interested partners can explore.

protobuf

Protobuf is a google-produced information exchange format that can be thought of as a serialization method. It is a language-neutral, platform-neutral, extensible mechanism for serializing structured data, similar to XML but smaller, faster, and simpler than XML.

Netty support for Protobuf is the ability to convert message and MessageLite objects from Protobuf.

Protobuf’s two encoders are also message-to-message conversions, so frame Detection is also required. Of course, you can also use other frame detection such as LengthFieldPrepender and LengthFieldBasedFrameDecoder as shown below:

ChannelPipeline pipeline = ... ; // Decoders pipeline.addLast("frameDecoder", new LengthFieldBasedFrameDecoder(1048576, 0, 4, 0, 4)); pipeline.addLast("protobufDecoder", new ProtobufDecoder(MyMessage.getDefaultInstance())); // Encoder pipeline.addLast("frameEncoder", new LengthFieldPrepender(4)); pipeline.addLast("protobufEncoder", new ProtobufEncoder());Copy the code

LengthFieldPrepender automatically precedes the field with a length field:

Before: + -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- + | "HELLO, WORLD" | + -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- - + after:  +--------+----------------+ + 0x000C | "HELLO, WORLD" | +--------+----------------+Copy the code

Of course netty prepared two special for protobuf frame detection, they are ProtobufVarint32FrameDecoder and ProtobufVarint32LengthFieldPrepender. Before we look at these two classes, we need to look at the Base 128 Varints in Protobuf.

What are Varints? A small integer takes up a small amount of space while a large integer takes up a large amount of space. In this way, there is no need to fix a specific length, which can reduce the length of data, but brings about complexity of parsing.

So how do you know how many bytes are needed for this data? In a protobuf, the highest bit of each byte is a judgment bit. If this bit is set to 1, it indicates that the next byte and this byte are the same number. If this bit is set to 0, it indicates that the next byte is unrelated to this byte and data ends at this byte.

For example, a byte is 8 bits. If the byte represents the integer 1, it can be represented by the following byte:

0000, 0001,Copy the code

If an integer does not fit into a single byte, then multiple bytes are required for concatenation, such as 300:

1010 1100 0000 0010
Copy the code

Why 300? First, look at the first byte, whose prefix is 1, indicating that there is another byte following it. Look at the second byte, which starts with 0, indicating that we are done. Let’s get rid of the judgment bits and replace them with the following numbers:

010 1100 000 0010
Copy the code

For a protobuf, the byte number is reversed, so we need to swap the two bytes:

000 0010 010 1100 
Copy the code

That is:

10, 010, 1100Copy the code

=256 + 32 + 8 + 4 = 300

Are generally used in protobuf Varint as the length of the field, so the netty provides ProtobufVarint32LengthFieldPrepender and ProtobufVarint32FrameDecoder ByteBuf for conversion.

For example, add varint to ByteBuf:

   BEFORE ENCODE (300 bytes)       AFTER ENCODE (302 bytes)
   +---------------+               +--------+---------------+
   | Protobuf Data |-------------->| Length | Protobuf Data |
   |  (300 bytes)  |               | 0xAC02 |  (300 bytes)  |
   +---------------+               +--------+---------------+
Copy the code

Delete varint’s length field when decoding:

BEFORE DECODE (302 bytes) AFTER DECODE (300 bytes) +--------+---------------+ +---------------+ | Length | Protobuf Data  |----->| Protobuf Data | | 0xAC02 | (300 bytes) | | (300 bytes) | +--------+---------------+ +---------------+Copy the code

serialization

Serialization is converting objects to binary data, and virtually all codec can be serialized. They provide conversion methods between objects and bytes.

Netty also provides two object conversion methods: ObjectDecoder and ObjectEncoder.

Note that these two objects are incompatible with the JDK’s ObjectInputStream and ObjectOutputStream. Can use CompactObjectInputStream, CompactObjectOutputStream and CompatibleObjectEncoder.

string

Strings are the most commonly used object, and Netty provides StringDecoder and StringEncoder for Strings.

Again, before using these two classes, the message needs to be converted, usually using the LineBasedFrameDecoder line by line:

ChannelPipeline pipeline = ... ; // Decoders pipeline.addLast("frameDecoder", new LineBasedFrameDecoder(80)); pipeline.addLast("stringDecoder", new StringDecoder(CharsetUtil.UTF_8)); // Encoder pipeline.addLast("stringEncoder", new StringEncoder(CharsetUtil.UTF_8));Copy the code

xml

XML is also a very common format, but it can be bulky and should be used less now. Netty provides an XmlFrameDecoder for parsing.

Since XML has its own start and end characters, it does not need to do frame detection. Instead, it can be converted directly, such as:

+ -- -- -- -- -- + + -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- + | < an | | Xml Element / > | + -- -- -- -- -- + + -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- + into:  +-----------------+ | <anXmlElement/> | +-----------------+Copy the code

+-----+-----+-----------+-----+----------------------------------+ | <an | Xml | Element/> | <ro | Ot > < child > content < / child > < / root > | + -- -- -- -- -- + + -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- + + -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- + into:  +-----------------+-------------------------------------+ | <anXmlElement/> | <root><child>content</child></root> | +-----------------+-------------------------------------+Copy the code

It’s all ok.

conclusion

Netty provides many excellent codecs for various application protocols, so you can use them to find out the differences between different protocols.

This article is available at www.flydean.com/16-netty-bu…

The most popular interpretation, the most profound dry goods, the most concise tutorial, many tips you didn’t know waiting for you to discover!

Welcome to pay attention to my public number: “procedures those things”, understand technology, more understand you!