This is the 26th day of my participation in the November Gwen Challenge. Check out the event details: The last Gwen Challenge 2021
Juejin. Cn introduces the concept of sticky packages and half packages. This blog will continue to describe how Netty handles sticky half packages.
Short link
The idea of a short link is that the client disconnects from the server each time it sends data to the server. The message boundary at this point is between the connection established and the connection disconnected. At this time, there is no need to use sliding Windows and other technologies to buffer data, so there will be no sticky packet phenomenon. But if one-time send too much data, the receiver can’t hold all the data once, or half a pack happens, so short link cannot solve the phenomenon of half a pack, short link artificially set up this way message boundaries, but obviously this method is of low efficiency, and it can solve the problem of glue bag but can’t solve the problem of half a pack.
The client code needs to modify the channelActive method to close the connection each time it sends data:
public void channelActive(ChannelHandlerContext ctx) throws Exception {
log.debug("sending...");
ByteBuf buffer = ctx.alloc().buffer(16);
buffer.writeBytes(new byte[] {0.1.2.3.4.5.6.7.8.9.10.11.12.13.14.15});
ctx.writeAndFlush(buffer);
// Use short links and disconnect after each send
ctx.channel().close();
}
Copy the codeThe client sends messages to the server in short link mode for 10 consecutive times. The running result is as follows, and the sticky packet problem does not occur:
Fixed length decoder
The idea behind the fixed-length decoder is that the client and the server agree on a maximum length to ensure that the client will not send more than this length each time. If the length of the sent data is insufficient, the length needs to be replenished
When the server receives data, it will split the received data according to the agreed maximum length. Even if sticky packets are generated in the process of sending, the data can be split correctly through the fixed-length decoder. The server needs to use FixedLengthFrameDecoder to decode data of fixed length as follows
ch.pipeline().addLast(new FixedLengthFrameDecoder(16))
Ensure that the length of data to be sent each time is the specified size. The code for sending data on the client is as follows
// The maximum length is 16
final int maxLength = 16;
// The data to be sent
char c = 'a';
Sends 10 packets to the server
for (int i = 0; i < 10; i++) {
ByteBuf buffer = ctx.alloc().buffer(maxLength);
// A fixed-length byte array. Unused parts are filled with 0
byte[] bytes = new byte[maxLength];
// Generate data that contains 0 to 15 characters
for (int j = 0; j < (int)(Math.random()*(maxLength-1)); j++) {
bytes[j] = (byte) c;
}
buffer.writeBytes(bytes);
c++;
// Send the data to the server
ctx.writeAndFlush(buffer);
}
Copy the codeThe server needs to use FixedLengthFrameDecoder to split sticky data. This handler needs to be added before the LoggingHandler to ensure that the data has been split when it is printed
// Use fixed length decoder to split sticky packet data
ch.pipeline().addLast(new FixedLengthFrameDecoder(16));
ch.pipeline().addLast(new LoggingHandler(LogLevel.DEBUG));
Copy the codeThe running results are as follows:
Line decoder
LineBasedFrameDecoder(int maxLength) can be used to split data with newline (\n) delimiters. But can be by DelimiterBasedFrameDecoder (int maxFrameLength ByteBuf… Delimiters) to specify which delimiters to split the data with (multiple delimiters can be passed in). Both decoders require a maximum length of the data passed in. If the maximum length is exceeded, TooLongFrameException will be thrown
The following example code uses the newline character \n as the separator. The client code needs to use the newline character \n as the separator at the end of the sent data
// The maximum length is 64
final int maxLength = 64;
// The data to be sent
char c = 'a';
for (int i = 0; i < 10; i++) {
ByteBuf buffer = ctx.alloc().buffer(maxLength);
// Generate data whose length ranges from 0 to 62
Random random = new Random();
StringBuilder sb = new StringBuilder();
for (int j = 0; j < (int)(random.nextInt(maxLength-2)); j++) {
sb.append(c);
}
// Data ends with \n
sb.append("\n");
buffer.writeBytes(sb.toString().getBytes(StandardCharsets.UTF_8));
c++;
// Send the data to the server
ctx.writeAndFlush(buffer);
}
Copy the codeServer code you need to use DelimiterBasedFrameDecoder to take apart the data, the handler needs to be added before LoggingHandler, ensure that the data is printed has been broken up
// Split sticky packet data with line decoder with \n delimiter
// The maximum length needs to be specified
ch.pipeline().addLast(new DelimiterBasedFrameDecoder(64));
ch.pipeline().addLast(new LoggingHandler(LogLevel.DEBUG));
Copy the codeThe following example code uses the custom delimiter \c as the delimiter:
Client code
.// Data ends with \c
sb.append("\\c"); buffer.writeBytes(sb.toString().getBytes(StandardCharsets.UTF_8)); .Copy the codeThe server code when using DelimiterBasedFrameDecoder need to specify a delimiter
// Place the delimiter in ByteBuf
ByteBuf bufSet = ch.alloc().buffer().writeBytes("\\c".getBytes(StandardCharsets.UTF_8));
// Split sticky packet data with line decoder with \c delimiter
ch.pipeline().addLast(new DelimiterBasedFrameDecoder(64, ch.alloc().buffer().writeBytes(bufSet)));
ch.pipeline().addLast(new LoggingHandler(LogLevel.DEBUG));
Copy the codeRunning results:
Length field decoder
When transmitting data, a field can be added to the data to indicate the length of useful data. When decoding, this field can be read to indicate the length of the field, and other relevant parameters can be read at the same time, so as to know what the final data is like
LengthFieldBasedFrameDecoder decoder can provide more abundant split method, its construction method has five parameters
public LengthFieldBasedFrameDecoder(
int maxFrameLength,
int lengthFieldOffset, int lengthFieldLength,
int lengthAdjustment, int initialBytesToStrip )
Copy the codeArgument parsing
- MaxFrameLength Maximum data length
- Represents the maximum length of data (including additional information, length identifier, etc.)
- lengthFieldOffset The starting offset of the data length identifier
- The byte used to indicate the number of bytes of data is initially used to identify the length of useful bytes, since additional information may precede it
- lengthFieldLength Data length Indicates the number of bytes(Number of bytes used to indicate the length of useful data)
- The number of bytes in the data used to indicate the length of useful data, == Note not the length of the content, but the length of the length field ==
- lengthAdjustment The length represents the offset from useful data
- Used to indicate the distance between a data length identifier and useful data, since additional information may exist between the two
- initialBytesToStrip Data read starting point
-
Data between 0 and Initialbytestteststrip is read, but data between 0 and Initialbytestteststrip is not read
-
Parameters of the graphic
example
lengthFieldOffset = 0
lengthFieldLength = 2
lengthAdjustment = 0
initialBytesToStrip = 0 (= do not strip header)
BEFORE DECODE (14 bytes) AFTER DECODE (14 bytes)
+--------+----------------+ +--------+----------------+
| Length | Actual Content |----->| Length | Actual Content |
| 0x000C | "HELLO, WORLD" | | 0x000C | "HELLO, WORLD" |
+--------+----------------+ +--------+----------------+
Copy the codeThe length identifier starts from 0. The length identifier contains 2 bytes
0x000C is the length of HELLO, WORLD
lengthFieldOffset = 0 lengthFieldLength = 2 lengthAdjustment = 0 initialBytesToStrip = 2 (= the length of the Length field) BEFORE DECODE (14 bytes) AFTER DECODE (12 bytes) +--------+----------------+ +----------------+ | Length | Actual Content |----->| Actual Content | | 0x000C | "HELLO, WORLD" | | "HELLO, WORLD" | +--------+----------------+ +----------------+Copy the codeThe length identifier starts from 0 and is 2 bytes long. The second byte is read from the beginning (the length identifier is skipped here), which can be interpreted as removing the first two bytes when decoding
Because we skipped two bytes for length, we read HELLO, WORLD
lengthFieldOffset = 2 (= the length of Header 1)
lengthFieldLength = 3
lengthAdjustment = 0
initialBytesToStrip = 0
BEFORE DECODE (17 bytes) AFTER DECODE (17 bytes)
+----------+----------+----------------+ +----------+----------+----------------+
| Header 1 | Length | Actual Content |----->| Header 1 | Length | Actual Content |
| 0xCAFE | 0x00000C | "HELLO, WORLD" | | 0xCAFE | 0x00000C | "HELLO, WORLD" |
+----------+----------+----------------+ +----------+----------+----------------+
Copy the codeThe length identifier is preceded by 2 bytes of additional content (0xCAFE). The third byte is the beginning of the length identifier, which means 3 bytes of length (0x00000C).
Header1 has additional information that needs to be skipped when reading the length identifier to get the length
lengthFieldOffset = 0
lengthFieldLength = 3
lengthAdjustment = 2 (= the length of Header 1)
initialBytesToStrip = 0
BEFORE DECODE (17 bytes) AFTER DECODE (17 bytes)
+----------+----------+----------------+ +----------+----------+----------------+
| Length | Header 1 | Actual Content |----->| Length | Header 1 | Actual Content |
| 0x00000C | 0xCAFE | "HELLO, WORLD" | | 0x00000C | 0xCAFE | "HELLO, WORLD" |
+----------+----------+----------------+ +----------+----------+----------------+
Copy the codeThe length identifier starts at 0 and is 3 bytes long, followed by 2 bytes of other content (0xCAFE)
The length identifier (0x00000C) represents the length of the data from the following lengthAdjustment (2 bytes), that is, HELLO, WORLD, not including 0xCAFE
lengthFieldOffset = 1 (= the length of HDR1)
lengthFieldLength = 2
lengthAdjustment = 1 (= the length of HDR2)
initialBytesToStrip = 3 (= the length of HDR1 + LEN)
BEFORE DECODE (16 bytes) AFTER DECODE (13 bytes)
+------+--------+------+----------------+ +------+----------------+
| HDR1 | Length | HDR2 | Actual Content |----->| HDR2 | Actual Content |
| 0xCA | 0x000C | 0xFE | "HELLO, WORLD" | | 0xFE | "HELLO, WORLD" |
+------+--------+------+----------------+ +------+----------------+
Copy the codeThe length identifier is preceded by 1 byte of other content, followed by 1 byte of other content, and the reading starts three bytes after the length identifier, that is, 0xFE HELLO, WORLD
use
The handler is tested through the EmbeddedChannel
public class EncoderStudy {
public static void main(String[] args) {
// Simulate the server
// Test the handler using EmbeddedChannel
EmbeddedChannel channel = new EmbeddedChannel(
// The maximum length of the data is 1KB, with 1 byte of additional information before and after the length identifier. The length identifier is 4 bytes (int).
new LengthFieldBasedFrameDecoder(1024.1.4.1.0),
new LoggingHandler(LogLevel.DEBUG)
);
// Simulate the client to write data
ByteBuf buffer = ByteBufAllocator.DEFAULT.buffer();
send(buffer, "Hello");
channel.writeInbound(buffer);
send(buffer, "World");
channel.writeInbound(buffer);
}
private static void send(ByteBuf buf, String msg) {
// Get the length of the data
int length = msg.length();
byte[] bytes = msg.getBytes(StandardCharsets.UTF_8);
// Write data information to buF
// Write other information before the length identifier
buf.writeByte(0xCA);
// Write the length of the data
buf.writeInt(length);
// Write additional information after the length identifier
buf.writeByte(0xFE);
// Write specific databuf.writeBytes(bytes); }}Copy the codeThe results
