preface

This section is presented to the user to see Netty out-of-the-box decoder: LengthFieldBasedFrameDecoder.

Note that reading this article requires some knowledge of Netty, such as ByteBuf’s readIndex and writeIndex, and some basic knowledge of other decoders.

But once you get the hang of it, it’s still pretty simple, but the official documentation examples are too “hardcore”.

Netty Version: 4.1.6 Migrating to my own blog: Portal

The experimental code

With same Netty unit tests for example (I have done some changes), and code comments is written by myself, now if you can’t understand the code is also normal, believe in the end will be suddenly enlightened: LengthFieldBasedFrameDecoderTest. Java

import io.netty.buffer.ByteBuf;
import io.netty.buffer.Unpooled;
import io.netty.channel.embedded.EmbeddedChannel;
import org.junit.Assert;
import org.junit.Test;

import static io.netty.util.ReferenceCountUtil.releaseLater;

public class LengthFieldBasedFrameDecoderTest {

    @Test
    public void testDiscardTooLongFrame1(a) {
        ByteBuf buf = Unpooled.buffer();
        /* * | 4 byte| * +-------+ * | 32 | length = 4 bytes * +-------+ */
        buf.writeInt(32);

        /* * | 4 bytes| 32bytes | * +--------+---*---+-----------+----+ * | 32 | 1 | 2 | ... | 32 | length = 36 bytes * +--------+---+---+-----------+----+ */
        for (int i = 0; i < 32; i++) {
            buf.writeByte(i);
        }

        /* * | 4 bytes| 32bytes | 4 bytes| * +--------+---*---+-----------+----+--------+ * | 32 | 1 | 2 | ......... | 32 | 1 | length = 40 bytes * +--------+---+---+-----------+----+--------+ */
        buf.writeInt(1);

        /* * | 4 bytes| 32bytes | 4 bytes|1 bytes| * +--------+---*---+-----------+----+--------+-------+ * | 32 | 1 | 2 | . | 32 | 1 | a | length = 41 bytes * +--------+---+---+-----------+----+--------+-------+ */
        buf.writeByte('a');
        EmbeddedChannel channel = new EmbeddedChannel(new LengthFieldBasedFrameDecoder(36.0.4));
        try {
            channel.writeInbound(buf);
            Assert.fail();
        } catch (TooLongFrameException e) {
            // expected
        }
        Assert.assertTrue(channel.finish());

        ByteBuf b = channel.readInbound();
        int readableBytes = b.readableBytes();
        System.out.println(readableBytes);
        Assert.assertEquals(5, readableBytes);
        // Read 4 bytes of data
        int readInt = b.readInt();
        System.out.println(readInt);
        Assert.assertEquals(1, readInt);
        // Read 1 byte of data
        byte readByte = b.readByte();
        System.out.println((char)readByte);
        Assert.assertEquals('a', readByte);
        b.release();

        Assert.assertNull(channel.readInbound());
        channel.finish();
    }
Copy the code

The output of the experimental code is as follows:

5
1
a
Copy the code

As for how these output results come out, I believe you can understand after reading this blog.

Follow up the source code

LengthFieldBasedFrameDecoder inheritance

Let’s look at LengthFieldBasedFrameDecoder inheritance relationship diagram:

There are several classes in the diagram that I mentioned in The Difference between Inbound and Oubound events.
ByteToMessageDecoder was mentioned in another blog post.

From the class diagram above, coupled with some relevant knowledge of the pipeline, it is not hard to find, we can actually use LengthFieldBasedFrameDecoder as ChannelHandler, that is to say, decoding is event propagation, processing in one of the ring. In code, it would look something like this:

    / / a little
    .childHandler(new ChannelInitializer<SocketChannel>() {
        @Override
        public void initChannel(SocketChannel ch) {
            ch.pipeline().addLast(new Base64Decoder());
            ch.pipeline().addLast(new DelimiterBasedFrameDecoder());
            ch.pipeline().addLast(new FixedLengthFrameDecoder(3));
            ch.pipeline().addLast(new LineBasedFrameDecoder(10.false.false));

       }
Copy the code

(This paragraph has been used in the previous section, but it is used for emphasis.)

LengthFieldBasedFrameDecoder attribute analysis

I say LengthFieldBasedFrameDecoder difficult, really difficult to understand its properties, may my brain is stupid, it took nearly two hours is a preliminary understanding on the interpretation of attribute in the official document.

Let’s take a look at its properties, which must be very desperate the first time:

public class LengthFieldBasedFrameDecoder extends ByteToMessageDecoder {

   /** Sort enumeration */
    private final ByteOrder byteOrder;
    /** Maximum packet length */
    private final int maxFrameLength;
    /** Number of bytes offset by the length field */
    private final int lengthFieldOffset;
    /** Number of bytes occupied by the length field */
    private final int lengthFieldLength;
    /** lengthFieldEndOffset = lengthFieldOffset + lengthFieldLength ** /
    private final int lengthFieldEndOffset;
    /** Fixes the value of the length field */
    private final int lengthAdjustment;
    /** Specifies the number of bytes */
    private final int initialBytesToStrip;
    /** Whether to raise an exception when the packet length is larger than maxFrameLength */
    private final boolean failFast;
    /** Whether you are in discard mode */
    private boolean discardingTooLongFrame;
    /** Discarded length */
    private long tooLongFrameLength;
    /** The remaining bytes to be discarded */
    private longbytesToDiscard; . (Non-attribute code omitted)Copy the code

It may take a little more time on your own to fully understand these attributes. My blog (below) won’t necessarily help you understand these attributes either, but it’s a good reference.

Here is an example of an official document to understand slowly, very long, I hope you have patience to read through:

lengthFieldLength

LengthFieldOffset = 0 lengthFieldLength = 2 lengthAdjustment = 0 InitialbyStrip = 0 (= do not strip header Bytes) after decoding (14 bytes) | 2 bytes 12 bytes | | | 2 bytes 12 bytes | | + -- -- -- -- -- -- -- - + -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- - + +--------+----------------+ | length | Actual Content |----->| Length | Actual Content | | 0x000C | "HELLO, WORLD" | | 0x000C | "HELLO, WORLD" | +--------+----------------+ +--------+----------------+Copy the code

LengthFieldLength =2 lengthFieldLength=2 lengthFieldLength=2 bytes lengthFieldLength=2 bytes lengthFieldLength=2 bytes lengthFieldLength=2 bytes lengthFieldLength=2 bytes lengthFieldLength=2 bytes lengthFieldLength=2 bytes lengthFieldLength=2 bytes lengthFieldLength=2 bytes lengthFieldLength=2 bytes lengthFieldLength=2 bytes lengthFieldLength=2 bytes
The decoder then reads a further 12 bytes, and the first two bytes together form a packet (unchanged before and after decoding).

initialBytesToStrip

lengthFieldOffset = 0 lengthFieldLength = 2 lengthAdjustment = 0 initialBytesToStrip = 2 (= the length of the Length Field) decoding (14 bytes) after decoding (12 bytes) | 2 bytes 12 bytes | | | 12 bytes | + -- -- -- -- -- -- -- - + -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- - + + -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- + |  Length | Actual Content |----->| Actual Content | | 0x000C | "HELLO, WORLD" | | "HELLO, WORLD" | +--------+----------------+ +----------------+Copy the code

LengthFieldLength =2 lengthFieldLength=2 lengthFieldLength=2 bytes lengthFieldLength=2 bytes lengthFieldLength=2 bytes lengthFieldLength=2 bytes lengthFieldLength=2 bytes lengthFieldLength=2 bytes lengthFieldLength=2 bytes lengthFieldLength=2 bytes lengthFieldLength=2 bytes lengthFieldLength=2 bytes
Initialbyteststrip =2 tells the decoder that the first two bytes do not need to be read. So, the encoder moves the readerIndex to the second byte, which is to discard the HDR1 field and the length field.
Finally, the encoder is able to start reading the data, reading 12 bytes from the first 2 bytes, the packet consisting of “HELLO, WORLD”.

lengthAdjustment

lengthFieldOffset = 0 lengthFieldLength = 2 lengthAdjustment = -2 (= the length of the Length field) initialBytesToStrip = 0 (= do not strip the header) before decoding (14 bytes) after decoding (14 bytes) | 2 bytes 12 bytes | | | 2 bytes 12 bytes | | +--------+----------------+ +--------+----------------+ | Length | Actual Content |----->| Length | Actual Content | | 0x000C | "HELLO, WORLD" | | 0x000C | "HELLO, WORLD" | +--------+----------------+ +--------+----------------+Copy the code

LengthFieldLength =2 lengthFieldLength=2 lengthFieldLength=2 bytes lengthFieldLength=2 bytes lengthFieldLength=2 bytes lengthFieldLength=2 bytes lengthFieldLength=2 bytes lengthFieldLength=2 bytes lengthFieldLength=2 bytes lengthFieldLength=2 bytes lengthFieldLength=2 bytes lengthFieldLength=2 bytes lengthFieldLength=2 bytes lengthFieldLength=2 bytes
If the logic is followed, the decoder will read 14 bytes later, but only 12 bytes later, where lengthAdjustment comes in handy.
To read the actual number of bytes = initial length of domain values (14) + lengthAdjustment (2), and then the initialBytesToStrip = 0 decoder, said don’t need to discard bytes, will read back “the real number of bytes to read” bytes and read in front of the 2 bytes of data packets.
Parameter Meaning: lengthAdjustment can be used to correct the number of bytes to be read when the value of the length field means something else.

lengthFieldOffset

LengthFieldOffset = 2 lengthFieldLength = 3 lengthAdjustment = 0 InitialbyStrip = 0 (= do not strip header Bytes) after decoding (17 bytes) 2 bytes | | 3 bytes 12 bytes | | | 2 bytes 3 bytes | | 12 bytes | +----------+----------+----------------+ +----------+----------+----------------+ | Header 1 | Length | Actual Content |----->| Header 1 | Length | Actual Content | | 0xC AFE | 0x00000C | "HELLO, WORLD" | | 0xCAFE | 0x00000C | "HELLO, WORLD" | +----------+----------+----------------+ +----------+----------+----------------+Copy the code

This time is a little different, because lengthFieldOffset=2, this parameter tells the decoder that the value of the length field is not read from the beginning byte, but from the last two bits of the first byte, because the first two bits are other data fields that do not belong to the length field.
Ok, now the decoder reads 3 bytes from the third byte and succeeds in getting the initial value of the length field, which is 12.
In the end, lengthAdjustment and InitialbytestStrip are both equal to 0. No additional operation is required. The data packet is composed of 12 bytes read later and 5 bytes read earlier.

It’s still lengthAdjustment, but this time it’s a little bit different than before

Testtest2: lengthFieldOffset = 0 lengthFieldLength = 3 lengthAdjustment = 2 (= the length of Header 1) testtest2: testtest2 = 0 (17 bytes) after decoding 3 bytes (17 bytes) | | 2 bytes 12 bytes | | | 3 bytes | 2 bytes 12 bytes | | +----------+----------+----------------+ +----------+----------+----------------+ | Length | Header 1 | Actual Content |----->| Length | Header 1 | Actual Content | | 0x00000C | 0xCAFE | "HELLO, WORLD" | | 0x00000C | 0xCAFE | "HELLO, WORLD" | +----------+----------+----------------+ +----------+----------+----------------+Copy the code

LengthFieldLength =3 lengthFieldLength=3 lengthFieldLength=3 bytes lengthFieldLength=3 bytes lengthFieldLength=3 bytes lengthFieldLength=3 bytes lengthFieldLength=3 bytes lengthFieldLength=3 bytes lengthFieldLength=3 bytes lengthFieldLength=3 bytes lengthFieldLength=3 bytes lengthFieldLength=3 bytes lengthFieldLength=3 bytes lengthFieldLength=3 bytes
However, the decoder finds that lengthAdjustment=2, i.e. lengthFieldLength needs to be corrected to 14 (=12+2), otherwise it will not get the full data after reading only 12 bytes.
In the end, found lengthAdjustment, initialBytesToStrip = 0, do not need additional operations, read 14 bytes plus the front to the back of 3 bytes of data packets.

Here are two examples that, not surprisingly, may refresh your “think it’s right” thoughts:

lengthFieldOffset = 1 (= the length of HDR1) lengthFieldLength = 2 lengthAdjustment = 1 (= the length of HDR2) InitialBytesToStrip = 3 (= the length of HDR1 + LEN) before decoding (16 bytes) after decoding (13 bytes) | 1 bytes 2 bytes | | 1 bytes 12 bytes | |  |1 bytes| 12 bytes | +-------+--------+-------+----------------+ +-------+----------------+ | HDR1 | Length | HDR2 | Actual Content |----->| HDR2 | Actual Content | | 0xCA | 0x000 C| 0xFE | "HELLO, WORLD" | | 0xFE | "HELLO, WORLD" | +-------+--------+-------+----------------+ +-------+----------------+Copy the code

LengthFieldOffset = 1 and! =0, this tells the decoder that the value of the length field is not read from the beginning byte, but from the first bit of the first byte, because the first bit is a different data field and does not belong to the length field.
Ok, now the decoder reads 2 bytes from the second byte and succeeds in getting the initial value of the length field, which is 12.
At this point, the decoder also finds that lengthAdjustment=1, i.e. lengthFieldLength needs to be corrected to 13 (=12+1), otherwise it will not be able to read the full data after only 12 bytes.
Initialbyteststrip =3 tells the decoder that the first three bytes do not need to be read. So, the encoder moves the readerIndex to the third byte, which is to discard the HDR1 field and the length field.
Finally, the encoder is able to read the data, 13 bytes from the beginning of the third byte, the packet consisting of the HDR2 field and the “HELLO, WORLD”.

lengthFieldOffset = 1 lengthFieldLength = 2 lengthAdjustment = -3 (= the length of HDR1 + LEN, Negative) initialBytesToStrip = 3 before decoding (16 bytes) after decoding (13 bytes) | 1 bytes 2 bytes | | 1 bytes 12 bytes | | | 1 bytes | 12 bytes  | +-------+--------+-------+----------------+ +-------+----------------+ | HDR1 | Length | HDR2 | Actual Content |----->| HDR2 | Actual Content | | 0xC A | 0x0010 | 0xFE | "HELLO, WORLD" | | 0xFE | "HELLO, WORLD" | +-------+--------+-------+----------------+ +-------+----------------+Copy the code

LengthFieldOffset = 1 and! =0, this tells the decoder that the value of the length field is not read from the beginning byte, but from the first bit of the first byte, because the first bit is a different data field and does not belong to the length field.
Ok, now the decoder reads 2 bytes from the second byte and succeeds in getting the initial value of the length field, which is 16.
At this point the decoder also finds that lengthAdjustment=-3, that lengthFieldLength needs to be corrected to 13 (=16-3), since the next 13 bytes are now readable.
Initialbyteststrip =3 tells the decoder that the first three bytes do not need to be read. So, the encoder moves the readerIndex to the third byte, which is to discard the HDR1 field and the length field.
Finally, the encoder is able to read the data, 13 bytes from the beginning of the third byte, the packet consisting of the HDR2 field and the “HELLO, WORLD”.

If the above analysis is not correct, I recommend reading it a few times, because I myself have to roll back and forth to the official document several times to understand it.

Decode implementation

The next is the core part, that is, the ByteToMessageDecoder passed data decoding.

In addition, it is recommended to decode with an understanding of the attributes mentioned above, otherwise you will suffer the consequences: io.netty.handler.codec.LengthFieldBasedFrameDecoder#decode(io.netty.channel.ChannelHandlerContext, io.netty.buffer.ByteBuf, java.util.List

)

    @Override
    protected final void decode(ChannelHandlerContext ctx, ByteBuf in, List<Object> out) throws Exception {
        Object decoded = decode(ctx, in);
        if(decoded ! =null) {
            // Add the decoded object to out and pass it to the ByteToMessageDecoder for propagationout.add(decoded); }}Copy the code

Follow up the decode method: io.netty.handler.codec.LengthFieldBasedFrameDecoder#decode(io.netty.channel.ChannelHandlerContext, io.netty.buffer.ByteBuf)

    protected Object decode(ChannelHandlerContext ctx, ByteBuf in) throws Exception {
        // Processing in discard mode
        if (discardingTooLongFrame) {
            // The number of bytes left to be discarded
            long bytesToDiscard = this.bytesToDiscard;
            // Fetch the smallest of the number of bytes that can be read and the number of bytes that remain to be discarded
            int localBytesToDiscard = (int) Math.min(bytesToDiscard, in.readableBytes());
            / / throw away
            in.skipBytes(localBytesToDiscard);
            // Update the number of bytes remaining to be discarded
            bytesToDiscard -= localBytesToDiscard;
            // Save the remaining number of bytes to be discarded
            this.bytesToDiscard = bytesToDiscard;

            // There is no need to throw an exception quickly
            failIfNecessary(false);
        }

        // If the number of bytes of readable data is smaller than the user-defined packet size, the incomplete packet is incomplete and will not be decoded
        if (in.readableBytes() < lengthFieldEndOffset) {
            return null;
        }

        // The pointer position after the offset of the Length field
        // Combined with the experimental code =0
        int actualLengthFieldOffset = in.readerIndex() + lengthFieldOffset;

        FrameLength is the value of the length field
        LengthFieldLength (frameLength=32)
        /* * | 4 bytes| 32bytes | 4 bytes|1 bytes| * +--------+---*---+-----------+----+--------+-------+ * | 32 | 1 | 2 | . | 32 | 1 | a | length = 41 bytes * +--------+---+---+-----------+----+--------+-------+ */
        long frameLength = getUnadjustedFrameLength(in, actualLengthFieldOffset, lengthFieldLength, byteOrder);

        if (frameLength < 0) {
            in.skipBytes(lengthFieldEndOffset);
            throw new CorruptedFrameException(
                    "negative pre-adjustment length field: " + frameLength);
        }

        // Total number of bytes to read = Length field point value + correction + (Length itself in bytes + Length offset)
        FrameLength = 36(=32+4)
        / / in addition, to remind here has not been calculated to abandon the number of bytes (initialBytesToStrip)
        frameLength += lengthAdjustment + lengthFieldEndOffset;

        // The length of the packet to be read is less than the length of the packet to be read (one packet length)
        if (frameLength < lengthFieldEndOffset) {
            in.skipBytes(lengthFieldEndOffset);
            throw new CorruptedFrameException(
                    "Adjusted frame length (" + frameLength + ") is less " +
                    "than lengthFieldEndOffset: " + lengthFieldEndOffset);
        }

        // The length to be read > the maximum length
        if (frameLength > maxFrameLength) {
            // The number of bytes left to be discarded
            long discard = frameLength - in.readableBytes();
            // Total number of bytes discarded
            tooLongFrameLength = frameLength;

            if (discard < 0) {
                // buffer contains more bytes then the frameLength so we can discard all now
                in.skipBytes((int) frameLength);
            } else {
                // Enable the discard mode
                discardingTooLongFrame = true;
                // Record the number of bytes remaining to be discarded
                bytesToDiscard = discard;
                // Discard all currently readable bytes and move the read pointer
                in.skipBytes(in.readableBytes());
            }
            // Fast exception
            failIfNecessary(true);
            return null;
        }

        // never overflows because it's less than maxFrameLength
        int frameLengthInt = (int) frameLength;
        // If the number of bytes that can be read is smaller than the number of bytes intended to be read, the byte stream is not large enough to assemble a packet
        if (in.readableBytes() < frameLengthInt) {
            return null;
        }

        // If the number of bytes to be thrown is greater than the number of bytes to be read, an exception is thrown
        if (initialBytesToStrip > frameLengthInt) {
            in.skipBytes(frameLengthInt);
            throw new CorruptedFrameException(
                    "Adjusted frame length (" + frameLength + ") is less " +
                    "than initialBytesToStrip: " + initialBytesToStrip);
        }
        / / pointer backward initialBytesToStrip bytes
        // With the experimental code, we did not set the discarding byte, so the readerIndex stays where it is.
        in.skipBytes(initialBytesToStrip);

        // extract frame
        int readerIndex = in.readerIndex();
        // Number of bytes actually read = Number of bytes originally intended to read - Number of bytes to discard
        Initialbyteststrip =0, so actualFrameLength = 36
        int actualFrameLength = frameLengthInt - initialBytesToStrip;
        // Start reading data and encapsulate it into a packet (decode) based on the length of the read pointer that has been moved to the "discarded byte"
        ByteBuf frame = extractFrame(ctx, in, readerIndex, actualFrameLength);
        // Move the read pointer behind the data that has been read
        in.readerIndex(readerIndex + actualFrameLength);
        // Return the packet
        return frame;
    }
Copy the code

Combined with the experimental code, the process “diagram” is shown as follows:


      /** * Before decoding */

      | 4 bytes|          32bytes       | 4 bytes|1 bytes|
      +--------+---*---+-----------+----+--------+-------+
      |   32   | 1 | 2|... |32 |    1   |   a   |        length = 41Bytes + -- -- -- -- -- -- -- - + - + - + -- -- -- -- -- -- -- -- -- -- - + - + + -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- + write write | | readerIndex writerIndex | | | | | | | | | | | | | | | | | | | | ||||||| ||| |/** * After decoding */

      | 4 bytes|          32bytes       | 4 bytes|1 bytes|
      +--------+---*---+-----------+----+--------+-------+
      |   32   | 1 | 2|... |32 |    1   |   a   |        length = 41Bytes + -- -- -- -- -- -- -- - + - + - + -- -- -- -- -- -- -- -- -- -- - + - + + -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- + write write | | readerIndex writerIndexCopy the code

According to the final result of the above flowchart, I believe that the output result of the experimental code should be understood in seconds:

5  // b.readableBytes(): obtains the number of remaining readable bytes in the accumulator, that is, writerIndex-readerIndex
1  // b.readadint (): the value of the last 4 bytes is 1 after the readerIndex is read.After reading into the following "chart" : |4 bytes|          32bytes       | 4 bytes|1 bytes|
      +--------+---*---+-----------+----+--------+-------+
      |   32   | 1 | 2|... |32 |    1   |   a   |        length = 41Bytes + -- -- -- -- -- -- -- - + - + - + -- -- -- -- -- -- -- -- -- -- - + - + + -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- + write write | | readerIndex | writerIndex a// b.readadbyte (): the value of the last byte is the letter a after the readerIndex is read (the readerIndex should also be moved by one byte).After reading into the following "chart" : |4 bytes|          32bytes       | 4 bytes|1 bytes|
      +--------+---*---+-----------+----+--------+-------+
      |   32   | 1 | 2|... |32 |    1   |   a   |        length = 41Bytes + -- -- -- -- -- -- -- - + - + - + -- -- -- -- -- -- -- -- -- -- - + - + + -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- + write | writerIndex = readerIndexCopy the code

Ok, LengthFieldBasedFrameDecoder decode method and cases of successful commissioning at this point, if you are interested in some bad case, discard mode, etc, please Netty unit testing debugging grope for yourself. But before this, still suggest you to fully understand the meaning of the attribute of LengthFieldBasedFrameDecoder first, otherwise can’t read the code.

mo4tech.com (Moment For Technology) is a global community with thousands techies from across the global hang out!Passionate technologists, be it gadget freaks, tech enthusiasts, coders, technopreneurs, or CIOs, you would find them all here.

Thoroughly understand the length of the Netty domain Netty 】【 decoder: LengthFieldBasedFrameDecoder

preface

The experimental code

Follow up the source code

LengthFieldBasedFrameDecoder inheritance

LengthFieldBasedFrameDecoder attribute analysis

Decode implementation

Thoroughly understand the length of the Netty domain Netty 】 【 decoder: LengthFieldBasedFrameDecoder

preface

The experimental code

Follow up the source code

LengthFieldBasedFrameDecoder inheritance

LengthFieldBasedFrameDecoder attribute analysis

Decode implementation

Related Posts

SQL8 select current salary of all employees

LeetCode (Palindrome number)

[Thread pool] Where do threads in the thread pool go if they encounter an exception? How to deal with it?

Thoroughly understand the length of the Netty domain Netty 】【 decoder: LengthFieldBasedFrameDecoder