The original author of this article is “Crystal shrimp Dumpling”. The original article is sponsored by “Yu Gang Shuo” writing platform. The copyright of the original article belongs to “Yu Gang Shuo” wechat public account.
1, the introduction
Quick Understanding of Network Communication Protocols (Part 1)
A Quick Understanding of Network Protocols (Part 2)
Network programming lazy introduction (three) : a quick understanding of the TCP protocol is enough
Just one entry for beginners: Developing mobile IM from scratch
Tip:
At the end of the article in attachment
2. Series of articles
This is the eighth article in a series that Outlines the following:
- Network Programming Lazy Introduction part 1: Quick Understanding of Network Communication Protocols part 1
- “Network Programming Lazy Introduction ii: A Quick Understanding of Network Communication Protocols (Part II)”
- “Network programming lazy Introduction (3) : A quick understanding of TCP protocol is enough.”
- Network Programming Slacker’s Guide (part 4) : Quickly Understand the Difference between TCP and UDP
- A Quick Look at why UDP Sometimes Has an Advantage over TCP
- “Network programming lazy introduction (six) : The history of the most popular hub, switch, Router function principle introduction”
- Web Programming Slacker’s Guide (7) : Understand HTTP in a Nutshell
- “Network programming lazy introduction (8) : How to write tcp-based Socket long connection” (this article)
If you find this series too basic, you can go straight to the Unknown Network Programming series, which contains the following table of contents:
- “Unknown Network Programming (1) : A Simple Analysis of the DIFFICULT Problems in TCP Protocol (part 1)”
- “Unknown Network Programming (II) : A Brief Analysis of the DIFFICULT Problems in TCP Protocol (Part II)”
- TIME_WAIT and CLOSE_WAIT when closing TCP connections
- “Network Programming under the radar (Part 4) : A Deep Dive into TCP’s Abnormal Shutdown.”
- Hidden Network Programming part 5: UDP Connectivity and Load Balancing
- Unknown Network Programming (6) : Understanding UDP in Depth and Using it Well
If you are interested in server-side high performance network programming, you can read the following series of articles:
- “High Performance Network Programming (1) : How many Concurrent TCP connections can a Single server Have?”
- “High Performance Network Programming part II: The Famous C10K Concurrent Connection Problem in the Last Decade”
- High Performance Network Programming (PART 3) : The Next 10 Years, It’s Time to Consider C10M Concurrency
- “High Performance Network Programming (IV) : Theoretical Exploration of High Performance Network Applications from C10K to C10M”
Please refer to the summary article on the characteristics and optimization methods of mobile terminal network:
- Summary of Optimization Methods for Short Connections in Modern Mobile Networks: Request Speed, Weak Network Adaptation, and Security Guarantee
- Mobile IM Developers must Read (1) : Easy to Understand the “weak” and “Slow” mobile Web
- Mobile IM Developers must Read (ii) : Summary of the Most Complete Mobile Weak Network Optimization Methods ever
3. Reference materials
TCP/IP, rounding
Chapter 11 ·UDP: User datagram protocol
TCP/IP, rounding
Chapter 17: TCP: Transmission control Protocol
TCP/IP, rounding
Chapter 18: Establishment and termination of TCP connections
TCP/IP, rounding
Chapter 21. TCP Timeouts and retransmission
Easy to Understand – In-depth understanding of TCP (part 1) : Theoretical basis
Easy to Understand – In-depth understanding of TCP (part 2) : RTT, sliding window, and congestion handling
Theory classic: TCP protocol three handshake and four wave process in detail
Connection between theory and practice: Wireshark captures packets to analyze TCP three-way handshake and four-way wave
Protocol Diagram of Computer Network Communication (Chinese Edition)
High performance network programming (a) : the number of concurrent TCP connections can be a single server
High performance Network programming (II) : The last 10 years, the famous C10K concurrent connection problem
High performance Network Programming (III) : In the next 10 years, it is time to consider C10M concurrency
High performance Network programming (IV) : Theoretical exploration of high performance network applications from C10K to C10M
This section describes the differences between TRANSPORT layer protocols TCP and UDP
Why does QQ use UDP instead of TCP?
Mobile im protocol selection: UDP or TCP?
4. Introduction to TCP/IP
Once upon a time in technology: TCP/IP changed the world
4.1 IP
4.2 the TCP protocol
Here’s a quick look at the three-way handshake:
- First, the client sends a SYN to the server, assuming the sequence number is X. This x is generated by the operating system according to certain rules, so it can be considered a random number;
- After receiving a SYN, the server sends another SYN to the client. In this case, the seq number of the server is y. At the same time, an ACK x+1 tells the client that “SYN has been received and it is ready to send data.”
- When the client receives a SYN from the server, it replies with an ACK y+1. This ACK tells the server that the SYN has been received and that the server is ready to send data.
After these three steps, the TCP connection is established. Note the following three points:
- The connection is initiated by the client;
- TCP allows us to carry data when the client replies with an ACK to the server in step 3. It can’t be done because of the limitations of the API;
- TCP also allows for a “four-way handshake”, which again, due to API restrictions, does not happen.
TCP/IP: 3 handshakes and 4 waves
- “Understanding TCP in Depth (PART 1) : Fundamentals”
- In – Depth understanding of TCP protocol (part 2) : RTT, sliding Windows, congestion Handling
- Classic Theory: TCP three-way handshake and Four-way Wave
- Wireshark Packet Capture Analyzing TCP Three-Way Handshake and Four-Way Wave
TCP/IP Details – Volume 1
The online reading version can be accessed here
Protocol Diagram of Computer Network Communication (Chinese Edition)
Click here to enter the original hd image
5. Basic Socket usage
ServerSocket
Socket
The procedure for using the socket is as follows:
- 1) Create ServerSocket and listen for client connections;
- 2) Connect to the server using Socket;
- 3) Use socket.getinputStream ()/getOutputStream() to get input and output streams for communication.
5.1 Step 1: Create ServerSocket and listen for client connections
public class EchoServer { private final ServerSocket mServerSocket; public EchoServer(int port) throws IOException { // 1. Create a ServerSocket and listen on the port. Port mServerSocket = new ServerSocket(port); } public void run() throws IOException { // 2. Socket client = mServersocket.accept (); handleClient(client); } private void handleClient(Socket socket) { // 3. Using sockets for communication... } public static void main(String[] argv) { try { EchoServer server = new EchoServer(9877); server.run(); } catch (IOException e) { e.printStackTrace(); }}}Copy the code
5.2 Step 2: Connect to the Server using Socket
public class EchoClient { private final Socket mSocket; Public EchoClient(String host, int port) throws IOException {// Create a socket and connect to the server mSocket = new socket (host, port); } public void run() {public static void main(String[] argv) {try { Localhost EchoClient client = new EchoClient("localhost", 9877); client.run(); } catch (IOException e) { e.printStackTrace(); }}}Copy the code
5.3 Step 3: Get input/output streams through socket.getinputStream ()/getOutputStream() for communication
First, let’s implement the server:
public class EchoServer { // ... private void handleClient(Socket socket) throws IOException { InputStream in = socket.getInputStream(); OutputStream out = socket.getOutputStream(); byte[] buffer = new byte[1024]; int n; while ((n = in.read(buffer)) > 0) { out.write(buffer, 0, n); }}}Copy the code
Let’s look at the client side:
public class EchoClient { // ... public void run() throws IOException { Thread readerThread = new Thread(this::readResponse); readerThread.start(); OutputStream out = mSocket.getOutputStream(); byte[] buffer = new byte[1024]; int n; while ((n = System.in.read(buffer)) > 0) { out.write(buffer, 0, n); } } private void readResponse() { try { InputStream in = mSocket.getInputStream(); byte[] buffer = new byte[1024]; int n; while ((n = in.read(buffer)) > 0) { System.out.write(buffer, 0, n); } } catch (IOException e) { e.printStackTrace(); }}}Copy the code
For those unfamiliar with lambda, replace Thread readerThread = new Thread(this::readResponse) with the following code:
Thread readerThread = new Thread(new Runnable() { @Override public void run() { readResponse(); }});Copy the code
Open the two terminals and run the following commands respectively:
$ javac EchoServer.java
$ java EchoServer
$ javac EchoClient.java
$ java EchoClient
hello Server
hello Server
foo
fooCopy the code
5.4 There are a few final points to note
- 1) In the above code, all of our exceptions are not handled. In actual applications, when exceptions occur, you need to close the socket and handle errors according to actual services.
- 2) On the client side, we did not stop readThread. In practice, we can make the thread return from a blocked read by closing the socket. Recommended reading “Java Concurrent Programming practice”;
- 3) Our server only handles one client connection. If you need to process multiple clients at the same time, you can create threads to process requests. This is left as an exercise for the reader to complete.
6, Socket, ServerSocket silly points not clear
Before getting into the subject of this section, consider this question:
The answer is 3 sockets:
Sharp-eyed readers may have noticed that in the last section I described them this way:
In the Java SDK, sockets have two interfaces: ServerSocket for listening to customer connections and Socket for communication.
Note:
Note:
First of all,
The following
now
So said.
Volume 1
7, Socket “long” connection implementation
7.1 Background
For those familiar with sockets, there is an API like this:
socket.setKeepAlive(true);Copy the code
Note:
Let’s talk about long connections in iOS network programming
Why does TCP – based mobile IM still need the heartbeat keepalive mechanism?
Wechat team original sharing: Android version of wechat background to live combat sharing (network to live)
Summary of Message push on Android: implementation principle, heartbeat survival, problems encountered, etc
Given that there is a pair of connected sockets, the socket will no longer be available if:
- 1) One end of a closed TCP connection is a socket: the party that closes the TCP connection sends a FIN to notify the other party to close the TCP connection. In this case, if the other End reads the socket, it will read EoF (End of File). So we know that the socket is closed;
- 2) Application crash: The socket will be closed by the kernel, as in case 1;
- 3) System crash: it is too late for the system to send the FIN because it is already on its knees. At this point, the other party cannot know this information. When the other party attempts to read the data, it finally returns read time out. If data is written, it is an error such as host unreachable.
- 4) The cable is cut and the network cable is pulled out: Similar to situation 3, if the socket is not read or written, both sides do not know that the accident occurred. Unlike case 3, if we connect the network cable back, the socket will still work.
The heartbeat
For example, if we use JSON to communicate, we can add a type field to the protocol packet to indicate whether the JSON is heartbeat or business data:
{"type": 0, // 0 indicates heartbeat //... }Copy the code
7.2 Example
First, the interface part:
Public final class LongLiveSocket {/** * ErrorCallback */ public interface ErrorCallback {/** * Returns true */ Boolean onError(); } /** * DataCallback public interface DataCallback {void onData(byte[] data, int offset, int len); } /** * public interface WritingCallback {void onSuccess(); void onFail(byte[] data, int offset, int len); } public LongLiveSocket(String host, int port, DataCallback dataCallback, ErrorCallback errorCallback) { } public void write(byte[] data, WritingCallback callback) { } public void write(byte[] data, int offset, int len, WritingCallback callback) { } public void close() { } }Copy the code
Let’s get straight to the implementation:
public final class LongLiveSocket {
private static final String TAG = "LongLiveSocket";
private static final long RETRY_INTERVAL_MILLIS = 3 * 1000;
private static final long HEART_BEAT_INTERVAL_MILLIS = 5 * 1000;
private static final long HEART_BEAT_TIMEOUT_MILLIS = 2 * 1000;
/**
* 错误回调
*/
public interface ErrorCallback {
/**
* 如果需要重连，返回 true
*/
boolean onError();
}
/**
* 读数据回调
*/
public interface DataCallback {
void onData(byte[] data, int offset, int len);
}
/**
* 写数据回调
*/
public interface WritingCallback {
void onSuccess();
void onFail(byte[] data, int offset, int len);
}
private final String mHost;
private final int mPort;
private final DataCallback mDataCallback;
private final ErrorCallback mErrorCallback;
private final HandlerThread mWriterThread;
private final Handler mWriterHandler;
private final Handler mUIHandler = new Handler(Looper.getMainLooper());
private final Object mLock = new Object();
private Socket mSocket; // guarded by mLock
private boolean mClosed; // guarded by mLock
private final Runnable mHeartBeatTask = new Runnable() {
private byte[] mHeartBeat = new byte[0];
@Override
public void run() {
// 我们使用长度为 0 的数据作为 heart beat
write(mHeartBeat, new WritingCallback() {
@Override
public void onSuccess() {
// 每隔 HEART_BEAT_INTERVAL_MILLIS 发送一次
mWriterHandler.postDelayed(mHeartBeatTask, HEART_BEAT_INTERVAL_MILLIS);
mUIHandler.postDelayed(mHeartBeatTimeoutTask, HEART_BEAT_TIMEOUT_MILLIS);
}
@Override
public void onFail(byte[] data, int offset, int len) {
// nop
// write() 方法会处理失败
}
});
}
};
private final Runnable mHeartBeatTimeoutTask = () -> {
Log.e(TAG, "mHeartBeatTimeoutTask#run: heart beat timeout");
closeSocket();
};
public LongLiveSocket(String host, int port,
DataCallback dataCallback, ErrorCallback errorCallback) {
mHost = host;
mPort = port;
mDataCallback = dataCallback;
mErrorCallback = errorCallback;
mWriterThread = new HandlerThread("socket-writer");
mWriterThread.start();
mWriterHandler = new Handler(mWriterThread.getLooper());
mWriterHandler.post(this::initSocket);
}
private void initSocket() {
while (true) {
if (closed()) return;
try {
Socket socket = new Socket(mHost, mPort);
synchronized (mLock) {
// 在我们创建 socket 的时候，客户可能就调用了 close()
if (mClosed) {
silentlyClose(socket);
return;
}
mSocket = socket;
// 每次创建新的 socket，会开一个线程来读数据
Thread reader = new Thread(new ReaderTask(socket), "socket-reader");
reader.start();
mWriterHandler.post(mHeartBeatTask);
}
break;
} catch (IOException e) {
Log.e(TAG, "initSocket: ", e);
if (closed() || !mErrorCallback.onError()) {
break;
}
try {
TimeUnit.MILLISECONDS.sleep(RETRY_INTERVAL_MILLIS);
} catch (InterruptedException e1) {
// interrupt writer-thread to quit
break;
}
}
}
}
public void write(byte[] data, WritingCallback callback) {
write(data, 0, data.length, callback);
}
public void write(byte[] data, int offset, int len, WritingCallback callback) {
mWriterHandler.post(() -> {
Socket socket = getSocket();
if (socket == null) {
// initSocket 失败而客户说不需要重连，但客户又叫我们给他发送数据
throw new IllegalStateException("Socket not initialized");
}
try {
OutputStream outputStream = socket.getOutputStream();
DataOutputStream out = new DataOutputStream(outputStream);
out.writeInt(len);
out.write(data, offset, len);
callback.onSuccess();
} catch (IOException e) {
Log.e(TAG, "write: ", e);
closeSocket();
callback.onFail(data, offset, len);
if (!closed() && mErrorCallback.onError()) {
initSocket();
}
}
});
}
private boolean closed() {
synchronized (mLock) {
return mClosed;
}
}
private Socket getSocket() {
synchronized (mLock) {
return mSocket;
}
}
private void closeSocket() {
synchronized (mLock) {
closeSocketLocked();
}
}
private void closeSocketLocked() {
if (mSocket == null) return;
silentlyClose(mSocket);
mSocket = null;
mWriterHandler.removeCallbacks(mHeartBeatTask);
}
public void close() {
if (Looper.getMainLooper() == Looper.myLooper()) {
new Thread() {
@Override
public void run() {
doClose();
}
}.start();
} else {
doClose();
}
}
private void doClose() {
synchronized (mLock) {
mClosed = true;
// 关闭 socket，从而使得阻塞在 socket 上的线程返回
closeSocketLocked();
}
mWriterThread.quit();
// 在重连的时候，有个 sleep
mWriterThread.interrupt();
}
private static void silentlyClose(Closeable closeable) {
if (closeable != null) {
try {
closeable.close();
} catch (IOException e) {
Log.e(TAG, "silentlyClose: ", e);
// error ignored
}
}
}
private class ReaderTask implements Runnable {
private final Socket mSocket;
public ReaderTask(Socket socket) {
mSocket = socket;
}
@Override
public void run() {
try {
readResponse();
} catch (IOException e) {
Log.e(TAG, "ReaderTask#run: ", e);
}
}
private void readResponse() throws IOException {
// For simplicity, assume that a msg will not exceed 1024-byte
byte[] buffer = new byte[1024];
InputStream inputStream = mSocket.getInputStream();
DataInputStream in = new DataInputStream(inputStream);
while (true) {
int nbyte = in.readInt();
if (nbyte == 0) {
Log.i(TAG, "readResponse: heart beat received");
mUIHandler.removeCallbacks(mHeartBeatTimeoutTask);
continue;
}
if (nbyte > buffer.length) {
throw new IllegalStateException("Receive message with len " + nbyte +
" which exceeds limit " + buffer.length);
}
if (readn(in, buffer, nbyte) != 0) {
// Socket might be closed twice but it does no harm
silentlyClose(mSocket);
// Socket will be re-connected by writer-thread if you want
break;
}
mDataCallback.onData(buffer, 0, nbyte);
}
}
private int readn(InputStream in, byte[] buffer, int n) throws IOException {
int offset = 0;
while (n > 0) {
int readBytes = in.read(buffer, offset, n);
if (readBytes < 0) {
// EoF
break;
}
n -= readBytes;
offset += readBytes;
}
return n;
}
}
}Copy the code
Here is our new implementation of EchoClient:
public class EchoClient { private static final String TAG = "EchoClient"; private final LongLiveSocket mLongLiveSocket; public EchoClient(String host, int port) { mLongLiveSocket = new LongLiveSocket( host, port, (data, offset, Len) -> log. I (TAG, "EchoClient: received: "+ new String(data, offset, len)), // return true; } public void send(String msg) { mLongLiveSocket.write(msg.getBytes(), new LongLiveSocket.WritingCallback() { @Override public void onSuccess() { Log.d(TAG, "onSuccess: "); } @Override public void onFail(byte[] data, int offset, int len) { Log.w(TAG, "onFail: fail to write: " + new String(data, offset, len)); MLongLiveSocket. Write (data, offset, len, this); }}); }}Copy the code
Here are some examples of output:
03:54:55. 583, 12691-12713 / com. Example. Echo I/LongLiveSocket: readResponse: Heart beat received 03:55:00.588 12691-12713/com.example.echo I/LongLiveSocket: readResponse: Heart beat received 03:55:05.594 12691-12713/com.example.echo I/LongLiveSocket: readResponse: Heart beat received 03:55:09.638 12691-12710/com.example.echo D/EchoClient: onSuccess: 03:55:09.639 12691-12713/com.example. Echo I/EchoClient: EchoClient: received: Hello 03:55:10.595 12691-12713/com.example.echo I/LongLiveSocket: readResponse: Heart Beat received 03:55:14.652 12691-12710/com.example.echo D/EchoClient: onSuccess: 03:55:14.654 12691-12713/com.example. Echo I/EchoClient: EchoClient: received: Echo 03:55:15.596 12691-12713/com.example.echo I/LongLiveSocket: readResponse: Heart beat received 03:55:20.597 12691-12713/com.example.echo I/LongLiveSocket: readResponse: Heart Beat received 03:55:25.602 12691-12713/com.example.echo I/LongLiveSocket: readResponse: Heart beat receivedCopy the code
8, with TCP/IP protocol design
TCP/IP, rounding
8.1 How can I Upgrade the Protocol Version?
8.2 How Can I Send Data Packets of Variable Length?
8.3 Uploading Multiple Files Is successful only when all files are successfully uploaded
8.4 How to Ensure the order of Data?
There are two possibilities for the worker thread to put the results in:
- 1) The task just completed happens to be next, and the result is queued. Then, starting at the head of the buffer, all data that can be put into the result queue is put in;
- 2) The completed task cannot be placed in the result queue, so the result queue is inserted. Then, as in the previous case, you need to check the buffer.
8.5 How can I Ensure that the Peer Party receives the MESSAGE?
Data service guarantee and responder mechanism are discussed in detail in the following articles:
- Realization of IM Message Delivery Guarantee Mechanism (I) : Ensuring reliable Delivery of Online Real-time Messages
- Realization of IM Message Delivery Guarantee Mechanism (II) : Ensuring reliable Delivery of offline Messages
- IM group chat messages are so complex, how to ensure that not lost and not heavy?
- “Message Reliability and Delivery Mechanism of Mobile IM from client’s Perspective”
9. Download the source code attachment
Teach you to write based on TCP Socket long connection – source code (52im.net).zip
(142.48KB, download times: 0, price: 1 gold)
Jekton. Making. IO / 2018/06/23 /…
Appendix: More network programming materials
Once upon a time in technology: TCP/IP changed the world
What is the maximum size of a packet in UDP?
The principle of Java new generation network programming model AIO and the introduction of Linux system AIO
NIO framework introduction (a) : server based on Netty4 UDP bidirectional communication Demo
NIO framework introduction (ii) : Server based on MINA2 UDP bidirectional communication Demo
NIO framework introduction (3) : iOS and MINA2, Netty4 cross-platform UDP two-way communication actual combat
NIO framework introduction (four) : Android and MINA2, Netty4 cross-platform UDP two-way communication actual combat
P2P technology details (a) : NAT details – detailed principle, P2P introduction
(2) : P2P NAT traversal (hole) solution
P2P technology details (three) : P2P technology STUN, TURN, ICE details
Easy to Understand: Quickly understand the PRINCIPLE of NAT penetration in P2P technology
More of the same…