This is the 13th day of my participation in the August More Text Challenge
One, foreword
The client and server interact with data and need to define protocols for parsing.
For example, the front and back ends typically interact via HTTP, using JSON:
- You can see
HTTP header
Will be should beConten - Type: application/json
$curl - H "content-type: application/json" - X POST - data '{}' http://127.0.0.1:8080/file/post - v Note: Unnecessary use of -x or --request, POST is already implied. * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 7890 (# 0) > POST HTTP / 1.1 > http://127.0.0.1:8080/file/post Host: 127.0.0.1:8080 > user-agent: curl/7.58.0 > Accept: */* > proxy-connection: Keep-Alive > Content-Type:application/json > Content-Length: 2 > * upload completely sent off: 2 out of 2 bytes < HTTP/1.1 200 < Content-Length: 4 < Connection: keep-alive < content-type: text/plain; 2 out of 2 bytes < HTTP/1.1 200 < Content-Length: 4 < Connection: keep-alive < content-type: text/plain; charset=UTF-8 < Date: Fri, 13 Aug 2021 08:23:11 GMT < Keep-Alive: timeout=4 < Proxy-Connection: Keep-alive < * Connection #0 to host 127.0.0.1 left intact postCopy the code
The following protocols are used in IM: Custom protocol + Protobuf:
-
Custom protocol: Prevents external cracking
-
Protobuf: More efficient data, smaller data volume
Serialization using protobuf, that is, the request body is a complex Java object, serialized into a binary byte array.
(1) User-defined protocols
At present, there are many common protocols in the market, such as HTTP, HTTPS, JSON-RPC, FTP, IMAP, Protobuf, etc.
User-defined protocols have the following advantages:
- Performance: Compared to common protocols
- Scalability: Compared with common protocols, user-defined protocols can be extended to meet service requirements.
- Security: Common protocols are public, and many vulnerabilities have been hacked. Custom protocols are more secure because the hacker needs to break your protocol first.
Netty, as an excellent network communication framework, has provided a very rich abstract base classes for encoding and decoding. It is more convenient to extend and implement custom protocols based on these abstract base classes.
Common Netty encoders:
MessageToByteEncoder
Object encoded as a byte stream;MessageToMessageEncoder
One message type is encoded into another message type.
Netty common decoder types:
ByteToMessageDecoder/ReplayingDecoder
Decode the byte stream into a message object;MessageToMessageDecoder
Decode one message type into another message type.
Codecs can be divided into primary decoders and secondary decoders.
A decoder is used to solve TCP unpacking/sticky packet problem, according to the protocol after parsing byte data.
If you need to convert parsed bytes to the object model, you need to use a secondary decoder, and the encoder works the other way around.
- Primary codec:
MessageToByteEncoder/ByteToMessageDecoder
. - Secondary codec:
MessageToMessageEncoder/MessageToMessageDecoder
.
How to determineByteBuf
Is there a complete packet?
The most common way to do this is by reading the message length dataLength. If the length of the readable data of ByteBuf is smaller than that of dataLength, it indicates that ByteBuf is not enough to obtain a complete packet.
(2)Protobuf
use
Protobuf: official website
- Download:
Protobuf - Java - 3.9.1. Tar. Gz
- Installation (Environment)
Ubuntu 18
)
CD protobuf-3.9.1. /autogen.sh./configure make make check sudo make install protoc --versionCopy the code
- Compile generated files: The corresponding files are generated in the corresponding directories
Protoc - java_out = im/SRC im/proto/AuthenticateRequest protoCopy the code
Problem, if present:
protoc: error while loading shared libraries: libprotoc.so.8: cannot open shared object file: No such file or directory
Copy the code
Solution:
#Method 1.
sudo ldconfig
#Method 2. Seems to be globally invalid, even if placed in /etc/profile
export LD_LIBRARY_PATH=/usr/local/lib
Copy the code
Ii. Cases:RPC
agreement
The structure of the agreement is as follows:
@Data
public class MiniRpcProtocol<T> implements Serializable {
private MsgHeader header;
private T body;
}
Copy the code
The specific headers are as follows:
@Data
public class MsgHeader implements Serializable {
/ * + -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- + | magic number 2 byte | protocol version number 1 byte | serialization algorithm 1 byte | 1 byte message type | + -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- + | state 1 byte | message ID 8 byte data length 4 byte | | + -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- + * /
private short magic; / / the magic number
private byte version; // Protocol version number
private byte serialization; // Serialization algorithm
private byte msgType; // Packet type
private byte status; / / state
private long requestId; / / message ID
private int msgLen; // Data length
}
Copy the code