This section will focus on discussing a very common protocol design methodology in network programming: protocol header + message body.

The so-called communication protocol is a kind of “convention” followed by both communication parties. It is used for the communication sender to assemble the content into “binary stream” in accordance with the format stipulated by the “communication protocol”, and the communication receiver to decode the original request from the binary stream in accordance with the format stipulated by the “communication protocol”.

How to design the communication protocol?

Warm tips: this article follows the directory structure: first refine the general methodology of communication protocol design, and then source analysis Netty provides solutions, finally gives the best practices, we were missed the final practice part oh…

1. General protocol design methodology

In network programming, popular this kind of classical protocol design methodology: protocol header + message body.

The key points of its design are as follows:

  • The length of the protocol header is fixed, usually the minimum length to identify a service.

  • The protocol header contains a length field that identifies the length of a complete packet. The number of bytes used to indicate the length field directly determines the maximum length of a packet. The length field is usually designed to be 4 bytes.

  • The message body stores business data. For example, if it is a Dubbo protocol, the message body may contain request parameters, the name of the called service, etc., and the storage of string classes is usually organized in terms of field length and field content.

For a more intuitive demonstration, I use a simple RPC communication scenario to implement a remote service invocation similar to the Dubbo service, whose communication protocol can simply be set up as shown below:

After the design mode of the communication protocol based on Header + Boby, the communication receiver can decode one original request packet from the binary stream very easily. The basic routine of decoding is as follows (the solution to the “sticky packet” question that the interviewer likes to ask in the interview)

  • First, check whether there is a complete Head Header in the cumulative cache. For example, if the length of the Header of a packet is 6 bytes, check whether the number of readable bytes in the cumulative cache is greater than or equal to 6 bytes. If the number is less than 6 bytes, skip this processing and wait for more data to arrive in the cumulative cache.

  • Try to read the first 6 bytes and extract the value stored in the length field, that is, the packet length. Then judge that the number of readable bytes in the accumulative cache is greater than or equal to the length of the entire packet. If the accumulative cache does not contain a complete packet, skip this processing and wait for more data to reach the accumulative cache.

  • If a complete package is included, the contents are read sequentially in the format of the communication protocol.

Because of the design concept is very general, Netty to unified encapsulation: the deal design LengthFieldBasedFrameDecoder, making its debut next let’s look at how Netty encapsulate, reveal more implementation details, let everybody to combine theory with practice.

2, LengthFieldBasedFrameDecoder explanation

2.1 an overview of the

The following is a detailed interpretation of its core attributes:

  • ByteOrder ByteOrder sequence of bytes, Netty default use big-endian sequence (mainly for numeric types like int, long), the so-called big end sequence, usually it can be understand that the receiver to receive the order of the byte stream from high byte value type.

  • Int maxFrameLength Maximum length of a message.

  • Int lengthFieldOffset specifies the starting offset of the length field.

  • Int lengthFieldLength Specifies the length of the length field in bytes.

  • Int lengthFieldEndOffset Specifies the end offset of the length field, equal to lengthFieldOffset + lengthFieldLength.

  • Int lengthAdjustment lengthAdjustment value. This value represents the distance between the length field in the protocol and the message body field directly.

  • Int initialBytesToStrip front how many bytes don’t skip the one package processing, usually will deal skip head, only to transmit content in the body of the message to the downstream.

  • Boolean Whether failFast fails quickly.

  • Boolean discardingTooLongFrame Whether to engulf (skip) a large frame packet.

  • Long tooLongFrameLength is currently processing the actual size of the engulfed packet.

  • Long bytesToDiscard before the next decoding, need to ignore the number of bytes, used when encounter more than maxFrameLength package.

If the above attributes are a little confusing, it doesn’t matter, because there are two diagrams at the end of this section that outline the protocol (graphically delineating the location and meaning of each attribute).

2.2 Decode method

Let’s take a look at the decode method to understand its inner workings by reading the source code.

LengthFieldBasedFrameDecoder#decode

Step1: skip the processing logic for invalid packets. If discardingTooLongFrame to true, said it was processing more than * * * * maxFrameLength bag, need to skip the long bag, not its decoding, because the data is arriving cumulative buffer zone, and can’t skip the invalid unit at a time, For introducing bytesToDiscard variables, can skip the number of bytes used to record the, when bytesToDiscard 0 said after an invalid package already all over, need to deal with the normal packet, discardingTooLongFrame at this time will be reset to false.

LengthFieldBasedFrameDecoder#decode

Step2: if the readable byte size of the accumulative buffer is smaller than the end offset of the length field, return null to end the decoding, indicating that the data in the accumulative buffer is incomplete.

Step3: try to get the length of the package from the cumulative cache. LengthFiedlOffset specifies the actual offset of the length field. The lengthFiedlOffset specifies the actual offset of the length field, and the lengthFieldLength of the length field is combined with the byte sequence ** (large-ended sequence, small-ended sequence) **.

Step4: here is the processing logic when the packet length exceeds the maximum packet length allowed by the protocol, and here we will skip the lengthAdjustment attribute for a moment.

  • If the readable bytes in the current cumulative cache are greater than frameLength, greater than the length of the current packet, you can skip the packet by calling the skipBytes method.

  • If the current accumulation of cache OuDeKe read less than frmaeLength, needs several times over, so the first data from the cumulative area all over, and then through bytesToDiscard records need to skip the number of bytes.

Step5: if the data in the accumulative cache does not contain a complete packet, return null to end the decoding and wait for more packets to arrive.

Step6: extract a complete packet length through the slince method of ByteBuf, decode a complete packet, and complete a packet decoding.

2.3 Graphic Lengthfield Basic Frame protocol

In Netty LengthFieldBasedFrameDecoder a lengthAdjustment attribute, to use the code snippet below:

frameLength += lengthAdjustment + lengthFieldEndOffset

Copy the code

LengthAdjustment field, which can be positive or negative, is mainly used for packet lengthAdjustment. For details, see the following analysis.

LengthAdjustment > 0

LengthAdjustment < 0 In most cases, the length field indicates the length of the body of the message, but in some protocols, the length field indicates the length of the entire message, so Netty can adjust the length adjustment to a negative value. To adjust the size of the data frame.

Conclusion: LengthAdjustment is Netty designed for adaptation of existing agreement fields, namely Netty LengthFieldBasedFrameDecoder is for I to the header + body, And length field-based protocols are a common solution to accurately represent data frames (lengths of business data) through lengthAdjustment, which is a reverse thinking here.

3. Best practices of protocol design subclasses

Best practices: LengthFieldBasedFrameDecoder decode method of the duties of the decoded from binary stream a complete packet, its return type or ByteBuf, Therefore, the decode method of the custom codec is to call the decode method of the parent class to get ByteBuf, and then decode the object of the data in ByteBuf.

Namely LengthFieldBasedFrameDecoder object is not responsible for converting ByteBuf agreement, but from the binary stream decoding a data frame, object and converts ByteBuf agreement duties by its subclass implementation, usually coding style is as follows:

This article is introduced here, your likes, forwarding, message is the biggest encouragement to me.