This is the 13th day of my participation in the August More Text Challenge

One, foreword

The client and server interact with data and need to define protocols for parsing.

For example, the front and back ends typically interact via HTTP, using JSON:

  • You can seeHTTP headerWill be should beConten - Type: application/json
$curl - H "content-type: application/json" - X POST - data '{}' http://127.0.0.1:8080/file/post - v Note: Unnecessary use of -x or --request, POST is already implied. * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 7890 (# 0) > POST HTTP / 1.1 > http://127.0.0.1:8080/file/post Host: 127.0.0.1:8080 > user-agent: curl/7.58.0 > Accept: */* > proxy-connection: Keep-Alive > Content-Type:application/json > Content-Length: 2 > * upload completely sent off: 2 out of 2 bytes < HTTP/1.1 200 < Content-Length: 4 < Connection: keep-alive < content-type: text/plain; 2 out of 2 bytes < HTTP/1.1 200 < Content-Length: 4 < Connection: keep-alive < content-type: text/plain; charset=UTF-8 < Date: Fri, 13 Aug 2021 08:23:11 GMT < Keep-Alive: timeout=4 < Proxy-Connection: Keep-alive < * Connection #0 to host 127.0.0.1 left intact postCopy the code

The following protocols are used in IM: Custom protocol + Protobuf:

  • Custom protocol: Prevents external cracking

  • Protobuf: More efficient data, smaller data volume

Serialization using protobuf, that is, the request body is a complex Java object, serialized into a binary byte array.


(1) User-defined protocols

At present, there are many common protocols in the market, such as HTTP, HTTPS, JSON-RPC, FTP, IMAP, Protobuf, etc.

User-defined protocols have the following advantages:

  • Performance: Compared to common protocols
  • Scalability: Compared with common protocols, user-defined protocols can be extended to meet service requirements.
  • Security: Common protocols are public, and many vulnerabilities have been hacked. Custom protocols are more secure because the hacker needs to break your protocol first.

Netty, as an excellent network communication framework, has provided a very rich abstract base classes for encoding and decoding. It is more convenient to extend and implement custom protocols based on these abstract base classes.

Common Netty encoders:

  • MessageToByteEncoderObject encoded as a byte stream;
  • MessageToMessageEncoderOne message type is encoded into another message type.

Netty common decoder types:

  • ByteToMessageDecoder/ReplayingDecoderDecode the byte stream into a message object;
  • MessageToMessageDecoderDecode one message type into another message type.

Codecs can be divided into primary decoders and secondary decoders.

A decoder is used to solve TCP unpacking/sticky packet problem, according to the protocol after parsing byte data.

If you need to convert parsed bytes to the object model, you need to use a secondary decoder, and the encoder works the other way around.

  • Primary codec:MessageToByteEncoder/ByteToMessageDecoder.
  • Secondary codec:MessageToMessageEncoder/MessageToMessageDecoder.

How to determineByteBufIs there a complete packet?

The most common way to do this is by reading the message length dataLength. If the length of the readable data of ByteBuf is smaller than that of dataLength, it indicates that ByteBuf is not enough to obtain a complete packet.


(2)Protobufuse

Protobuf: official website

  1. Download:Protobuf - Java - 3.9.1. Tar. Gz
  2. Installation (Environment)Ubuntu 18)
CD protobuf-3.9.1. /autogen.sh./configure make make check sudo make install protoc --versionCopy the code
  1. Compile generated files: The corresponding files are generated in the corresponding directories
Protoc - java_out = im/SRC im/proto/AuthenticateRequest protoCopy the code

Problem, if present:

protoc: error while loading shared libraries: libprotoc.so.8: cannot open shared object file: No such file or directory
Copy the code

Solution:

#Method 1. 
sudo ldconfig


#Method 2. Seems to be globally invalid, even if placed in /etc/profile
export LD_LIBRARY_PATH=/usr/local/lib
Copy the code

Ii. Cases:RPCagreement

The structure of the agreement is as follows:

@Data
public class MiniRpcProtocol<T> implements Serializable {
    private MsgHeader header;
    private T body;
}
Copy the code

The specific headers are as follows:

@Data
public class MsgHeader implements Serializable {
    / * + -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- + | magic number 2 byte | protocol version number 1 byte | serialization algorithm 1 byte | 1 byte message type | + -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- + | state 1 byte | message ID 8 byte data length 4 byte | | + -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- + * /
    private short magic;        / / the magic number
    private byte version;       // Protocol version number
    private byte serialization; // Serialization algorithm
    private byte msgType;       // Packet type
    private byte status;        / / state
    private long requestId;     / / message ID
    private int msgLen;         // Data length
}
Copy the code