This is the 17th day of my participation in the August Text Challenge.More challenges in August

preface

Encoding and decoding: The conversion of data from one particular protocol format to another.

Programs that deal with encoding and decoding are often called encoders and decoders. Netty provides components that make it easy to write codecs for a variety of different protocols.

Introduction to codec

Encoding and decoding can be divided into two parts, namely encoding and decoding. We should know that in the network, data is transmitted in the form of bytecode, and we can only recognize text, pictures and other formats, so writing network applications inevitably need to manipulate bytes to convert the data that we can recognize into the program that the network can recognize, this process is called encoding and decoding.

Overview of codecs

Encoding, also known as serialization, serializes objects into arrays of bytes for network transport, data persistence, or other purposes.

Decoding, called deserialization, restores byte arrays read from the network, disk, and so on to the original object (usually a copy of the original object) for subsequent business logic operations.

The program that implements the codec function is also called a codec. The purpose of a codec is to intertranslate the raw byte array with the target program data format.

A codec consists of two parts: a decoder and an encoder.

You can imagine sending a message. Messages are data in a structured application. The encoder converts the message format to the data format suitable for transmission, while the corresponding decoder converts the transmitted data back to the message format in the program. Logically, converting a message format to a data format suitable for transmission is treated as operating outbound data, while converting the transmission data back into a message format that handles inbound data.

Netty embedded encoder

Netty has a number of codecs built into it to simplify development. The following figure shows Netty’s embedded codec.

As you can see, Netty’s embedded codec basically covers the codec work that may be involved in network programming, including the following:

  • Support byte and message conversion, Base64 conversion, decompression files.
  • rightHTTP,HTTP2,DNS,SMTP,STOMP,MQTT,SocksAnd so on.
  • rightXML,JSON 、Redis,Memcached,ProtobufPopular formats such as support.

The encoder and decoder structure is simple, the message is automatically through ReferenceCountUtil after encoding and decoding. The release (the message). If you don’t want to release the news can use ReferenceCountUtil. Retain (the message), the main difference is that retain can make reference number and would not happen, most of the time don’t need to do so.

decoder

The decoder’s primary responsibility is to convert inbound data from one format to another. Netty provides a rich abstract base class for decoders. Easy for developers to customize the decoder.

These base classes fall into the following two main categories:

  • Decode from byte to message (ByteToMessageDecoder and ReplayingDecoder).
  • Decoder from message to message (Message message Decoder).

Netty’s decoder is an abstract implementation of ChannelInboundHandler. Using a decoder in practice is as simple as formatting the inbound data and passing it to the next ChannelInboundHandler in the ChannelPipeline for processing. Putting the decoder in the ChannelPipeline makes the entire program flexible, while also making it easy to reuse logic.

ByteToMessageDecoder abstract class

The ByteToMessageDecoder abstract class is used to convert bytes into messages (or other sequences of bytes). ByteToMessageDecoder inherited from ChannelInboundHandlerAdapter. ChannelInboundHandlerAdapter in parallel flow methods will bytes from ByteBuf decoding for another message type.

1. Common methods

When processing network data, sometimes the data is too large to be sent all at once. So how do you know when the data has been sent? The ByteToMessageDecoder abstract class caches data to the station and provides the following methods for developers to use.

The core source code for these methods is as follows:

protected abstract void decode(ChannelHandlerContext ctx, ByteBuf in, List<Object> out) throws Exception;

protected void decodeLast(ChannelHandlerContext ctx, ByteBuf in, List<Object> out) throws Exception {
    if (in.isReadable()) {
        // Only call decode() if there is something left in the buffer to decode.
        // See https://github.com/netty/netty/issues/4386decodeRemovalReentryProtection(ctx, in, out); }}final void decodeRemovalReentryProtection(ChannelHandlerContext ctx, ByteBuf in, List<Object> out)
    throws Exception {
    decodeState = STATE_CALLING_CHILD_DECODE;
    try {
        decode(ctx, in, out);
    } finally {
        boolean removePending = decodeState == STATE_HANDLER_REMOVED_PENDING;
        decodeState = STATE_INIT;
        if(removePending) { fireChannelRead(ctx, out, out.size()); out.clear(); handlerRemoved(ctx); }}}Copy the code

The above methods are described as follows:

  • Decode () : This is the only abstract method that must be implemented.decode()The method is called with an object containing the data passed inByteBufAnd one for adding decoded messagesList. The call to this method is repeated until it is determined that no new elements have been added to itList, or theByteBufUntil there are no more bytes to read in. And then, ifListNot empty, then its contents will be passed toChannelPipelineThe next one inChannelInboundHandler.
  • DecodeLast () : The default implementation provided by Netty is simply calleddecode()Methods. whenChannelThis method is called once when the state of the. This method can be overridden to provide special handling.

2. Example of a decoder that converts bytes into integers

In this example, four bytes at a time are read from the inbound ByteBuf, decoded into an integer, and added to a List. When no more data can be added to the List, its contents are sent to the next ChannelInboundHandler.

public class ToIntegerDecoder extends ByteToMessageDecoder {
    @Override
   public void decode(ChannelHandlerContext ctx, ByteBuf in, List<Object> out) throws Exception {
    if (in.readableBytes() >= 4) { out.add(in.readInt()); }}}Copy the code

In the above code, the steps are as follows:

  1. Implement inheritanceByteToMessageDecoderIs used to decode bytes into messages.
  2. Check if there are at least four bytes readable (an int is four bytes long).
  3. From the inboundByteBufRead int, add to decode messageListIn the.

The process flow of the whole example is shown below:

The whole process is very simple for encoders and decoders. Once a message is encoding or decoding, it automatically be ReferenceCountUtil. Release (the message) calls. If you don’t want to release the news can use ReferenceCountUtil. Retain (message).

ReplayingDecoder abstract class

ReplayingDecoder abstract class is a subclass of ByteToMessageDecoder, ByteToMessageDecoder decoder decoding read buffer before the need to check whether the buffer has enough bytes, the use of ReplayingDecoder without their own check; If there are enough bytes in ByteBuf, it is read normally. Decoding stops if there are not enough bytes.

Because of this packaging, ReplayingDecode has some limitations.

  • Not all of themByteBufOperations are supported and thrown if an unsupported operation is calledUnReplayableOperationException.
  • Performance, useReplayingDecodeSlightly slow andByteToMessageDecoder.

If you can live with the restrictions listed above, you may prefer ReplayingDecoder to ByteToMessageDecoder. ByteToMessageDecoder is recommended to meet the needs of the situation, because its processing is relatively simple, not as complex as ReplayingDecoder implementation.

The following code is the implementation of ReplayingDecoder:

/** * Integer decoder */
public class ToIntegerReplayingDecoder extends ReplayingDecoder<Void> {

    @Override
    protected void decode(ChannelHandlerContext ctx, ByteBuf in, List<Object> out) throws Exception { out.add(in.readInt()); }}Copy the code

MessageToMessageDecoder abstract class

The MessageToMessageDecoder abstract class is used to decode one message into another.

The core source code is as follows:

public abstract class MessageToMessageDecoder<I> extends ChannelInboundHandlerAdapter {

    private final TypeParameterMatcher matcher;

   
    protected MessageToMessageDecoder(a) {
        matcher = TypeParameterMatcher.find(this, MessageToMessageDecoder.class, "I");
    }


    protected MessageToMessageDecoder(Class<? extends I> inboundMessageType) {
        matcher = TypeParameterMatcher.get(inboundMessageType);
    }


    public boolean acceptInboundMessage(Object msg) throws Exception {
        return matcher.match(msg);
    }

    @Override
    public void channelRead(ChannelHandlerContext ctx, Object msg) throws Exception {
        CodecOutputList out = CodecOutputList.newInstance();
        try {
            if (acceptInboundMessage(msg)) {
                @SuppressWarnings("unchecked")
                I cast = (I) msg;
                try {
                    decode(ctx, cast, out);
                } finally{ ReferenceCountUtil.release(cast); }}else{ out.add(msg); }}catch (DecoderException e) {
            throw e;
        } catch (Exception e) {
            throw new DecoderException(e);
        } finally {
            try {
                int size = out.size();
                for (int i = 0; i < size; i++) { ctx.fireChannelRead(out.getUnsafe(i)); }}finally{ out.recycle(); }}}protected abstract void decode(ChannelHandlerContext ctx, I msg, List<Object> out) throws Exception;
}
Copy the code

Decode of MessageToMessageDecoder is the only abstract method that needs to be implemented. Each inbound message is decoded to a different format, and the decoded message is passed to the next ChannelInboundHandler in the pipeline.

Here is an example of a MessageToMessageDecoder:

public class IntegerToStringDecoder extends MessageToMessageDecoder<Integer> {
    @Override
    public void decode(ChannelHandlerContext ctx, Integer msg List<Object> out) throws Exception { out.add(String.valueOf(msg)); }}Copy the code

In the code above, IntegerToStringDecoder inherits from MessageToMessageDecoder to convert an Integer to a String. There are two steps:

  • IntegerToStringDecoderInherited fromMessageToMessageDecoder.
  • throughtring.valueOf()Converts the string of an Integer message.

Inbound messages are parsed according to the parameters declared in the class definition (Integer in this case) rather than ByteBuf. In the example, the decoded message (String in this case) will be added to the Listand passed to the next ChannelInboundHandler.

The process flow chart of the whole example is as follows:

conclusion

Above we focus on Netty decoder related knowledge. In the next section we will look at encoders.