The Internet is arguably the greatest design ever devised by mankind. It connects hundreds of millions of computers and communications devices across the globe, enabling users on opposite sides of the globe to communicate instantly.
The “protocol” is the core of such a large and complex system, and the “protocol” is the rules that the communication parties must follow, under which different datagrams may be resolved into different response actions.
In short, “protocol” means that if the sender and receiver follow this rule to send data packets, they can get some special functions or services on top of the basic data transmission. Otherwise, your data will not be recognized by others. For example, by following the TCP protocol at both ends, you can obtain reliable data transmission capability over unreliable network transmission.
The whole computer network is layered, there are seven layers of models, there are five layers of models, I think the five layer model is more easy to understand. We introduce these five layers from top to bottom, they are respectively, application layer, transport layer, network layer, data link layer and physical layer.
The application layer
The “application layer” is the closest layer to the user, and the processes on the host make up the “application layer”. For example, if you type “www.baidu.com” into your browser’s address bar, what does your browser do at the application layer?
First, the browser returns the IP address corresponding to the domain name www.baidu.com using the DNS protocol, which we will discuss in more detail.
Next, the application layer decides to create a “TCP socket” and then encapsulate the request action into an Http datagram and push it into the socket.
There are two types of sockets: TCP socket and UDP socket. The application layer may send dozens of datagrams at the same time, and the transport layer may receive all the response packets. How does the application layer distinguish the response packets?
Sockets are used to distinguish applications at the application layer and are often identified by port numbers and IP addresses. The transport layer can know which socket to push the packet to by checking the source port number and IP address of the response packet.
When an application-layer datagram is pushed into a socket, all the application-layer work is done and it doesn’t have to worry about where subsequent packets go.
Also shows some here, “TCP socket” and “UDP socket” the essence of the two difference is that the former ensure a datagram to the destination, but necessarily time-consuming, while the latter does not guarantee that the datagram will be able to reach the destination, but the speed is quick, this also is the application layer protocol to consider in choosing a transport layer protocol.
We’ll talk more about TCP and UDP, but let’s look at how DNS works, how it resolves a domain name back to its IP address.
Principle of DNS
To be clear, DNS is an application-layer protocol, and its transport protocol of choice is UDP, so your domain name resolution process is usually fast, but there are also frequent resolution failures, but it is better to refresh.
On the DNS server, the domain name and its corresponding IP address are stored as a record, and all records cannot be stored on only one server. I believe that no matter how powerful the server is, it cannot withstand the global concurrency of hundreds of millions of times.
Roughly speaking, there are three types of DNS server, root DNS server, top-level DOMAIN DNS server, and authoritative DNS server.
The top-level domain DNS server is responsible for top-level domain names such as com, org, net, edu, and gov.
The root DNS server stores the IP addresses of all top-level DNS servers, meaning that you can find top-level DNS servers through the root server. For example, www.baidu.com, the root server returns the IP addresses of all top-level domain servers that maintain com.
Then you choose any one of the top-level domain servers and ask the top-level domain server. The top-level domain server should be able to make a judgment and give the address of the authoritative server responsible for the current domain. In the case of Baidu, the top-level domain server will return all the authoritative server addresses responsible for the baidu domain.
Therefore, you can choose any authoritative server address and query the specific IP address of www.baidu.com. The authoritative server will return the specific IP address to you.
So far, we have briefly described the general process of domain name resolution. There are some details that are not covered. We will look at them in full with an example later.
In the whole DNS resolution process, there is a very core character we have not introduced it, it is like the “assistant” of the host, help the host to query the IP address of the domain name. It’s called a “local DNS server.”
We’ll talk about this later every time you dynamically obtain an IP address through DHCP. In fact, the router will not only return you the IP address, but also tell you a DNS server address, this is your local DNS server address, that is, all your domain name resolution requests just tell it, it will look up for you and return the results to you.
In addition, the local DNS server usually has cache function, usually within two days of the records are cached, so most of the time you will not feel the domain name resolution process, because it is usually taken from the cache, very fast.
Let’s look at a simple case:
I found a picture on the Internet. It takes too much time to draw it myself, but suffice to say, now suppose the request is “www.xx.com”.
- 1: The host sends a query packet to the local DNS. If the query packet exists in the cache of the local DNS server, the host directly returns the query packet
- ② the local server finds that there is no root server in the cache and sends a query packet from the internal root server list
- ③ the root server resolves the suffix and tells the local server the list of all the top servers responsible for.com
- (4) : local server select a top-level domain server to continue to query. Com domain server to get the domain name to continue to resolve, return to the list of all authoritative servers responsible for. Xx domain
- ⑥ : The local server sends the query packet again from one of the returned authoritative servers, and finally obtains the specific IP address from one of the authoritative servers
- ⑧ : Returns a result to the host
In fact, the whole DNS packet sending and response process is to follow our layer 5 protocol, but the focus here is to understand the DNS protocol itself, so it does not mention the specific details of other layers, the emphasis here is to remind you that DNS is only an application layer protocol.
Transport layer
The task of the transport layer is to collect all the datagrams that are pushed out of the socket by the application layer, repackage the application layer datagrams according to the transport layer protocol specified by the application layer, TCP or UDP, and push them to the network layer to be sent.
TCP and UDP are two transport layer protocols. The former is a reliable transport protocol based on connections, while the latter is an unreliable transport protocol without connections. Therefore, the former is more suitable for scenarios that require high data integrity, while the latter is suitable for scenarios that allow data loss but require high transmission rates, such as: Voice calls, videos, and so on, lose one or two bags at most stuck once, not a big deal.
UDP
UDP is different from TCP in complexity. It does not guarantee the reliable transmission of data to the destination, nor does it guarantee the data to the destination in sequence. It only provides simple error checking. The packet format is as follows:
The source port number is used to respond to packet delivery, the destination port number is used to deliver data to the destination process, and the checksum is used to check whether data is damaged during transmission. If the data is damaged, UDP directly dismisses the packet.
TCP
TCP is slightly more complex. It is connection-oriented and provides reliable data transmission services based on connections. Its data packet format is as follows:
Simply explaining the meaning of each field in the packet format is not very practical, and you will find it difficult to understand. When we describe TCP actions such as “three-way handshake”, “four-way wave”, and “packet loss and retransmission”, we will continuously explain how these actions use the relevant fields in the packet.
First of all, let’s look at the familiar “three-way handshake”, which is basically synonymous with TCP, regardless of whether people understand the specific principle, referring to TCP, basically know “three-way handshake”.
However, the purpose of TCP three-way handshake is to ensure that the two parties can establish a stable connection and complete the request and response of data packets. As for why the three-way handshake is rather than four or five times, this is a philosophical question, which will not be discussed here.
The first step:
The client sends a special TCP packet to the server. The packet does not contain data at the application layer. The SYN field in the TCP header is 1 (see packet format above).
In addition, the client will randomly generate an initial sequence number and fill it in the “Sequence number” field of the packet, indicating that the current sequence number of the packet is this, and my subsequent groups will increase based on this sequence number.
The packet is then sent to the server through the network layer, link layer, and physical layer.
The second step:
If the packet is lost, the client tries to send again after a certain interval.
If the packet reaches the server correctly, the server disconnects the TCP header to see a special SYN handshake packet and allocates resources such as cache for the connection.
The server then builds the response message. SYN is a field used for synchronization. The response message will still be set to 1, and the server will randomly generate an initial sequence number in the sequence number field of the response message.
Finally, the server assigns a value to the acknowledgement field in the response message. This value is the serial number value sent by the client plus one.
The overall meaning is, “I agree to your connection request, my initial number is XXX, I have received your initial number, and I am waiting for your next group to arrive.”
Step 3:
After receiving the response packet from the server, the client allocates resources such as cache for the TCP connection to the client. The connection is established.
In fact, from the third step, the client can carry the application layer data to exchange packets with the server. In each packet, the SYN is 0, because it is only used to synchronize the initial sequence number.
In general, the whole “handshake” process looks like this:
Let’s take a look at the “four waves” of dismantling a TCP connection.
Because a TCP connection consumes a large amount of host resources, not only the server needs to allocate various cache resources, but also the client needs to allocate corresponding resources. Because TCP is “full-duplex communication”, the server and client are actually the same, who is the client and who is the server is relative.
This is important to note that a TCP connection can be disconnected not only by the client, but also by the server.
Here we assume that the client initiates the disconnection request actively:
The first step:
The client constructs a special TCP packet, sets the FIN header field to 1, and sends the packet.
The second step:
The server receives the special FIN packet and responds with an ACK packet to tell the client that the packet requesting closure has been received and I am processing it.
Step 3:
The server sends a FIN packet telling the client that I’m closing the connection.
Step 4:
The client returns an ACK message telling the server, I received the message you just sent, I have confirmed, and you can close the connection.
After receiving the ACK packet from the client, the server releases all the resources used for the TCP connection. At the same time, the client releases all the resources used for the TCP connection after a certain period of time.
Here’s a picture to make it more intuitive:
Let’s go over some of these details in conjunction with the graph and the associated numbering information.
First, the client sends a special group with serial number U. After the packet is sent, the client enters the fin-WaIT-1 state. In this state, the client of the TCP connection can no longer send datagrams, but can accept datagrams. The client waits for the response packet from the server.
Then, the server receives the request packet to terminate the connection sent by the client. The server constructs a response packet to tell the client that “I have received all the packets before u+1” and enters close-wait state, which lasts for a short time.
The server then sends its FIN datagram to inform the client that the server is closing the connection and enters the LAST_ACK state waiting for the client to respond.
Once the client receives the FIN packet, the client returns a FIN acknowledgement packet and enters the time-wait state. Resources occupied by the client TCP connection will be released after 2MSL interval.
At the same time, when the server receives the final acknowledgement packet from the client, it disconnects the server and releases related resources.
Why does the client need to wait 2MSL to release TCP resources?
That’s because 2 MSL is the longest time a message is present in the network, more than the time the message will be discarded, and if the client final confirmation packet loss in the network, the server will be launched overtime request, resend the third wave, at this time of waiting for the client can then resend a confirmation request.
This is why the client waits for a maximum transmission time. One might wonder why none of the previous requests timed out but only the last data send timed out?
In fact, the reason is very simple, I believe you can also imagine, is that TCP has its own timing ability, after a certain period of time does not receive a certain packet confirmation message, it will automatically resend, and if I do not do wait and directly close the connection, then how do I know whether the server has received my confirmation message?
After waiting for a maximum period, if no packet request is received from the server within this period, the acknowledgement packet must reach the server. Otherwise, the acknowledgement packet can be sent again.
Now that the “three-way handshake” and “four-way wave” of TCP have been briefly described, let’s look at some other features of TCP, such as reliable transmission, congestion control, and so on
First, let’s look at how TCP achieves reliable transmission, that is, how to solve the problem of packet loss in network transmission.
TCP uses the “back N step” protocol to implement reliable transport, to be precise, TCP is partly optimized on this basis.
The “Rollback N-step” protocol is also called the “sliding window” protocol, which allows a maximum of N sent but unacknowledged packets to the sender. As shown in the figure, the length from P1 to P3 is N, and the window here refers to the range from P1 to P3.
The whole window can slide forward only when the sender receives the p1 confirmation message. In fact, before receiving the P1 confirmation message, the server will only cache these “unexpected packets” even if the subsequent packets have been received.
After the server receives the least expected packet, it retrieves the subsequent packets from the cache, merges them and delivers them upwards, and then returns an acknowledgement packet to the sender.
When multiple consecutive packets have been acknowledged in the sender window from left to right, the entire window slides forward by multiple lengths.
Here’s an example:
This is a sender window. Gray indicates the packets that have been confirmed, yellow indicates the packets that have been sent but not confirmed, green indicates the packets to be sent next, and white indicates the packets that are unavailable.
This assumes that the server has received packets 6 and 7, but last time it sent packet 4 up to the application layer, that is, it is waiting for packet 5, so it temporarily caches the packets 6 and 7, and delivers the packets to the application layer when packet 5 arrives.
Now packet 5 is retransmitted due to timeout and finally reaches its destination. The server delivers packets 5, 6, and 7 and returns an acknowledgement message. ACK = 8 indicates that all packets before the serial number 8 have been received.
When the sender receives the confirmation message, 5, 6, and 7 turn gray and the window moves forward by 3 units.
In addition, I would like to emphasize that TCP does not negate acknowledgments, so if a server responds to multiple packets with the same serial number, it is very likely that a subsequent packet with that serial number will be lost.
For example, if a server sends multiple ACK acknowledgments for packet 5, what does that mean? At present, the complete upward delivery number of my server is no.5. I have not received the subsequent message. You had better send it again and not wait for timeout.
This is at the heart of fast retransmission.
So TCP reliable transmission we are also basically introduced, let’s see if the network congestion, TCP is how to control the sending of traffic?
TCP considers that packet loss is congestion, and the transmission efficiency needs to be reduced. The network is considered normal every time an acknowledgment packet is received, which increases the transmission efficiency.
TCP congestion control algorithm consists of three parts: slow start, congestion avoidance and fast recovery.
The idea of slow start is to send a datagram slowly at first, for example, only once in a certain period of time. After receiving an acknowledgement packet, two datagrams at twice the rate will be sent next time in the same period of time, and so on.
Therefore, in a short period of time, the sender of a TCP connection will increase exponentially, but once packet loss occurs, that is, redundant ACK acknowledgments are received, or the acknowledgement ACK of a packet is never received and has to start a timeout retransmission, then the sender considers “the network is congested”.
Set the rate directly to one, i.e., a round-trip period, send only one packet, and set a variable ssTHRESH to express the concept of a threshold value, which is half the rate of the sender when the packet was dropped last time.
The sending efficiency of subsequent senders also increases exponentially. However, unlike the first time, once the threshold is reached, TCP enters the “congestion avoidance” mode. In this mode, the sending efficiency increases cautiously instead of exponentially.
The idea of congestion avoidance is that after all datagrams sent in each round trip period have been confirmed, another packet is sent next time, so the slow increase in efficiency is prudent.
If the sending end loses packets due to timeout, notice that this is timeout. Set the sending rate to slow start, and the threshold is half of the current sending efficiency.
If the server returns multiple redundant ACKS to make sure that you have lost packets, TCP considers this not serious. In this case, TCP halves the current sending efficiency and enters a fast recovery phase.
The basic idea of fast recovery is to increase the sending efficiency of several packets after receiving several redundant ACKS. That is to say, your server did not receive several of my packets, so I will increase the rate of sending these two packets to you quickly.
During this period, if packet loss occurs due to timeout of the sender, in the same processing mode, the initial sending rate is reduced to half the current sending efficiency as the threshold, and the sender enters the slow start phase.
Of course, if the acknowledgement of lost packets is received during this period, the transmission efficiency is reduced and congestion avoidance occurs.
In this way, the core of the entire TCP several ideas have been introduced, the entire transport layer is basically clear. Now that you have some understanding of the transport layer, let me summarize.
The transport layer’s job is to retrieve all the data that needs to be sent from the sockets of each process in the application layer, and then select TCP or UDP to encapsulate the data and push it to the network layer below for sending.
To be continued…
All the code, images and files in this article are stored in the cloud on my GitHub:
(https://github.com/SingleYam/overview_java)
Welcome to wechat public number: jump on the code of Gorky, all articles will be synchronized in the public number.
