This article is participating in the “Network protocol must know must know” essay contest.
To learn more about HTTPS, go to the Network Protocol Security section here.
For additional information about network protocols, go to the Network Protocol Supplement here.
HTTP
This article explains HTTP in more detail
HTTP is short for Hyper Text Transfer Protocol, which is used to Transfer hypertext from the World Wide Web server to the local browser. HTTP is based on TCP/IP protocol communication protocol to transfer data (HTML files, image files, query results, etc.). It does not involve packet transmission and mainly defines the communication format between the client and the server. Port 80 is used by default.
The characteristics of Http
1. Simple and fast: when customers request services to the server, they only need to send the request method and path. The commonly used request methods are GET, HEAD, PUT, DELETE, and POST. Each method specifies a different type of contact between the client and the server. Because HTTP protocol is simple, the HTTP server program size is small, so the communication speed is very fast.
2. Flexibility: HTTP allows the transfer of any type of data object.
3. Connectionless: Connectionless means that only one request can be processed per connection. The server disconnects from the customer after processing the request and receiving the reply from the customer. In this way, transmission time can be saved.
4. Stateless: HTTP is stateless and does not save the communication status between the request and response. There is no dependency between any two requests. Intuitively speaking, each request is independent and has no direct connection to the previous or subsequent request. The protocol itself does not retain information about all previous request or response messages. The HTTP protocol is designed to be so simple in order to process a large number of transactions more quickly and ensure protocol scalability.
HTTP / 0.9
HTTP/0.9 was created in 1991
HTTP is the original version of the HTTP protocol. It only supports GET and requests to access resources in HTML format.
HTTP / 1.0
HTTP/1.0 was created in 1996
- Added request mode POST and HEAD
- The introduction of request and response headers, based on the Content-Type, supports a variety of data formats, not limited to HTML version 0.9
- Browser cache support
- Status codes are introduced
HTTP / 1.1
HTTP/1.1 was created in 1999
New features
-
Added persistent connections
-
Cookies were introduced to solve the HTTP stateless problem.
-
Provide virtual host support
- In HTTP/1.0, each domain name was bound to a unique IP address, so only one domain name could be supported by a server. However, with the development of virtual host technology, it is necessary to bind multiple virtual hosts on a physical host. Each virtual host has its own separate domain name, and these separate domain names all share the same IP address. Therefore, the HTTP/1.1 request header added a Host field to represent the current domain name address, so that the server can do different processing based on different Host values.
-
For dynamically generated content, the browser can know in real time if the file has been received
-
When DESIGNING HTTP/1.0, you need to set a full data size in the response header, such as Content-Length: 901, so that the browser can receive data based on the set data size. However, with the development of server-side technology, the content of many pages is generated dynamically, so the final data size is not known until the data is transferred, which causes the browser to not know when all the file data will be received.
HTTP/1.1 solves this problem by introducing the Chunk transfer mechanism. The server will divide the data into several data blocks of arbitrary size, and each data block will be sent with the length of the previous data block. Finally, a zero-length block is used as a symbol of the completion of data transmission. This provides support for dynamic content.
-
advantages
Added a persistent connection method, keep-alive
It features the ability to transmit multiple HTTP requests over a single TCP connection that remains open as long as the browser or server is not explicitly disconnected. Http1.1 Default Connection: close.
disadvantages
The network usage is not high
1.TCP slow start policy: The TCP slowly starts requests and checks whether the current network environment is stable. However, this process is too slow, resulting in low network utilization.
2.TCP connection appears to compete for resources. When our bandwidth is insufficient, the key resources cannot be quickly replied by the server, and the browser will wait for a long time, or even get stuck.
TCP can only handle one request at a time and needs to wait for the current request to complete before processing the next request, so when TCP packet loss, waiting, retransmission, or other reasons. This process can lead to problems. Keep subsequent requests waiting. This will also lead to low network utilization.
How do I optimize http1.1 connections?
TCP slow starts and TCP connections compete for resource bandwidth due to TCP itself, and queue blocking due to HTTP/1.1 mechanism, so we can only optimize the queue blocking problem, the rest can be left to the next generation of HTTP2.0 protocol optimization.
The browser maintains a maximum of six TCP persistent connections for each domain name. We can use this advantage to use reasonable CDN resources. For example, if two DIFFERENT CDN names are used, a maximum of three * six TCP connections can be sent at the same time.
HTTP / 2.0
HTTP/2.0 was created in 2015
TCP is slow to start, TCP is competing for resources, and HTTP1.1 is not good enough. So http2.0 can perfectly solve this problem, or long connection, only maintain a TCP connection, so that TCP slow start only once, TCP connection before there is no competition.
As far as queue congestion is concerned, HTTP2.0 only alleviates the problem of queue congestion. That leaves http3.0.
New features
- multiplexing
- Sets the priority of the request
- Server push
- The head of compression
advantages
multiplexing
Multiplexing is the most important feature in HTTP2.0, allowing concurrent requests to be orders of magnitude larger than HTTP1.1, increasing HTTP transmission efficiency by 20-60%.
The process is that the client can request it concurrently, the server can receive it concurrently, process it according to priority, and send it back to the client.
Implementation process of multiplexing
Http2.0 adds binary framing layer processing.
- The browser will request the information ready.
- The binary framing layer divides the data into frames, which are converted into frames with request ID numbers and sent to the server through the protocol stack “called streams”.
- After receiving the frame data, the server will convert the frame into data according to the ID and hand it to the server for processing.
- After processing, the server passes the data of the response to the binary framing layer.
- Similarly, the binary framing layer converts the data into ID numbered frames that are sent to the browser through the protocol stack.
- The browser receives the frame data, converts the frame into data, and passes the data to the appropriate request.
The above is the detailed process of multiplexing, because these frames can be concurrent, randomly sent, is the meaning of multiplexing, and the binary frame layer will convert data to binary has better compatibility, the calculator can better identify and improve efficiency.
Binary layered detailed solution
The frame structure is shown above. It contains the frame length, frame type, flag bit, stream identifier, and the composition of the data actually transmitted, and a total of 9 bytes.
The length of the frame
Frame length records the length of the actual transmitted data (frame load) for this frame. It is represented by 24 bits, meaning that a frame can transmit up to 16 megabytes of data.
The frame type
Frame types are represented by 8 bits and can represent up to 256 types. HTTP/2 defines a total of 10 types. It can be divided into data frame and control frame. Data frames are used to store HTTP packets, and control frames are used to manage the transmission of streams.
Sign a
The flag bit is represented by eight bits, which can be stored and carry control information.
The flow identifier
Stream identifiers are represented by 32 bits, but the highest bit is reserved, so up to 31 bits can be used. The upper limit is 2^31, which is about 2.1 billion. With stream identifiers, the receiver can identify frames with the same stream identifier from out-of-order frames and combine them in order.
Priority of the request
The browser prioritises resources based on type/context, and the binary framing layer further optimizes performance by optimizing the interlacing and transmission order of these frames after dividing the data into many frames. Each stream can have a request priority, and the server can control resource allocation (CPU, memory, bandwidth) based on the priority of the stream, and send the highest priority frame to the client first when the response data is ready. High-priority streams should be sent first, but not absolutely. The problem of first-queue blocking may be introduced again, with slow high-priority requests blocking the delivery of other resources.
● Highest priority: primary HTML — Highest
● High priority: CSS file –High
● Priority: JS file —-High/Low
● Low priority: Picture — Low
Server push
Server push function, that is, the server to send data to the client without request.
When a user requests an HTML page, the server knows that the HTML page will reference several important JavaScript files and CSS files, so after receiving the HTML request, the CSS files and JavaScript files to be used will be attached to the browser. After the browser parses the HTML file, the CSS and JavaScript files can be retrieved directly. The server can cache the client’s files, which can speed up the first time the page is opened.
The head of compression
HTTP requests and responses, status lines and request/response headers are information fields, there is no real data, so HTTP2.0 creates a table of all information fields, and creates an index for each field in the table. Clients and servers share this table, and they use index numbers to represent information fields. This avoids the repetitive, cumbersome fields of pre-HTTP2.0 and delivers them in a compressed way to improve utilization, using the HPACK algorithm.
HPACK algorithm
The first is to create a hash table between the server and the client. The fields to be used are stored in this table. Then during the transmission, only the index (such as 0,1,2…) is needed. Pass to the other side, the other side to get the index table on the line. In this way, the request header field can be simplified and reused to a great extent.
Huffman encoding, the principle of Huffman encoding is to establish an index table for all the characters that appear first, and then make the index corresponding to the characters that appear more frequently as short as possible. When transmitting, such index sequence is also transmitted, which can achieve a very high compression rate.
disadvantages
Or queue head congestion problem, because HTTP2.0 is based on TCP/IP, then the transmission of only one after another transmission, so queue head congestion problem, will exist, can not be solved, can only try to slow down.
As long as there is packet loss in the transmission process, then it will block the following transmission, waiting for retransmission, according to statistics, when the packet loss rate reaches 2%, http2.0 transmission efficiency is not as efficient as http1.1.
HTTP / 3.0
Http3.0 is a generational product that addresses the root causes of TCP’s shortcomings and integrates TCP’s strengths into HTTP3.0. Although it is still in draft status until June 2021, I believe that the future must be worth our expectation.
TCP shortcomings
Team head congestion
Http1.1 multiple TCP connections, but one of the connections blocked, the rest of the five connections can continue, and http2.0 only one TCP connection, if there is a blockage, It may also be less efficient than Http1.1.
TCP connection establishment delay
The network delay, also known as Round Trip Time (RTT), is the Time required for the client to send a packet to the server and the server to return the packet.
When establishing a TCP connection, three handshakes are required to confirm the connection with the server, that is, 1.5 RTT is consumed before data is transmitted.
If HTTPS is configured, a TLS connection is also required. There are two versions of TLS, TLS1.2 and TLS1.3. Each version takes a different time to establish a connection.
It takes about 2.5-3.5 RTT. “One RTT is about 10ms ~ 100ms”. In fact, sometimes users can feel the lag of such time consumption.
TCP advantages
Reliable transport
Packet loss and retransmission occur.
Request-reply (stable)
When the client transmits/receives data, the server responds.
QUIC agreement
Because network links go through many layers, Wi-Fi, routers, switches, and so on, but they all use TCP or UDP for transmission, there is not much room for innovation in HTTP3.0. It has to discard TCP in favor of UDP.
Quick handshake: Realizes the quick handshake function. Since QUIC is udP-based, it can use 0-RTT or 1-RTT to establish connections, which means that QUIC can send and receive data as quickly as possible, which can greatly improve the speed of first page opening.
Integrated TLS1.3: CURRENTLY, TLS1.3 is the most advanced security protocol and the fastest.
Multiplexing: The transfer of data over multiple paths over a single connection.
Http3.0 shortcomings
1. The packet loss rate is too high. Because UDP optimization is not as good as TCP optimization, there is a long way to go.
2. Current browser vendors do not support browsers well.
With all that said, this is a lot to parse.
TCP three-way handshake
HTTP is currently based on TCP connections with the exception of version 3.0.
Ack confirmation number: indicates the number of the first byte of the packet that is expected to be received next time. If ACK = N, it indicates that all data up to n-1 have been correctly received. The confirmation sequence number field is valid only when the ACK flag bit is 1. ACK =seq+1. After a connection is established, ACK must be set to 1 for all transmitted packet segments. ACK is used to confirm the packets received from the sender and prevent packet loss.
-
Sign a
- ACK: Confirmation number ACK is a confirmation of data packets, when the right after receiving the packet, the receiver sends an ACK to the sender, it will show to confirm the data packets, each packet will have a serial number, if you receive the packet is wrong, or wrong order, also can apply for resend, NAK is a negative answer, ACK is an affirmative reply to ensure the correct transmission of data. This is the transmission mechanism of TCP protocol, which is called connection-oriented.
- RST: resets the connection
- SYN: Initiates a new connection. When two computers talk over a TCP connection, the connection is initialized first. The package that does this is called SYN.
- FIN: Releases a packet with the flag set to end a TCP return call, but the port is still open for receiving subsequent data
Three-way handshake process
The first handshake client sends a request to the server
SYN packets are initialized, SYN=1 indicates that the connection is ready, and the sequence number of the first byte of the packet to be sent, seq=x.
The second handshake is processed by the server and returns to the client
After receiving a valid connection request, the server also initializes SYN=1 and confirms ACK=1, as well as ACK= x+1, expecting the sequence number of x+1 for the next handshake. Seq =y is the sequence number of the first byte of the sent packet.
Third handshake The client sends data packets to the server
ACK=1 and seq = x+1 are sent to the server. After the TCP connection is completed, the client and the server enter the ESTABLISHED state and complete the three-way handshake. Then you can pass the data.
If it’s two handshakes?
The server cannot confirm whether the client has received the message, so it cannot establish the channel.
If it’s four handshakes?
If the server is shaking hands for the fourth time, there will be no information to confirm, because the serial number to confirm has been confirmed, no additional operations.
TCP waved four times
Connection: keep-alive http1.0 needs to be set. Http1.1 defaults to long links.
We can also use keep-alive:timeout=15 or Connection:close.
- However, after receiving a FIN packet, the server may not close the connection immediately. Therefore, the server can only reply with an ACK packet telling the client: “I received the FIN packet you sent.” I can send FIN packets only after all the packets on the server are sent. Therefore, I cannot send FIN packets at the same time.
Four wave flow
Note: When the communication is complete, both the client and the server can disconnect. For example, if the client does not disconnect or send data, the server is still waiting to receive data, which occupies the resources of the server. The server can also disconnect.
First wave The client initiates a disconnection from the server
At this time, both the client and server are in ESTABLISHED state. When the client wants to disconnect, FIN=1(1 indicates the connection request) and seq= U. After the connection is sent, the client enters the FIN-wait state to terminate the WAIT.
Second wave the server back to the browser
After receiving the request from the client, the server replies with ACK=1, SEq = V, ACK= U +1. After B sends the request, the server enters close-wait to CLOSE the WAIT state
After receiving the reply from the server, the client enters fin-WaIT-2 and terminates the WAIT state. Because the server only replies, it is processing the disconnection related work and waiting to see whether the data has been transferred. Therefore, the server does not agree to disconnection and the client still needs to WAIT.
Third wave the server agrees to disconnect the notification client
After the server processes and sends the information, the server sends FIN=1(sending an interrupt request, that is, agreeing to the interrupt),ACK=1 (confirming receipt), SEq = W,ACK= U +1. After the server sends the information, the server enters the list-ACK state.
Wave the client for the fourth time to confirm and notify the server
After receiving the interrupt request from the server, the client replies with ACK=1 (confirming receipt), SEq = U +1, ACK= W +1. After sending the interrupt request, USER A enters the time-wait state.
After receiving the confirmation from the client, the server enters the CLOSED state and disconnects from A. Then, it immediately enters the LISTEN state and continues to accept the connection request from the next client. When the client waits for 2ms (2 minutes is recommended by RFC 1122), the client enters the CLOSED state and is disconnected from server B.
Why does TCP need four waves instead of two?
Because when the client sends the disconnect request, the server needs to immediately respond to a client, but the server at this time need to do some more time-consuming, so immediately returned to the client first ‘ok, I was disconnected operation’ subsequent handling, finished processing, such as to go back to the client, then client received, reply the service side, can disconnect.
If you don’t understand anything, or if there are any inadequacies or mistakes in my article, please point them out in the comments section. Thank you for reading.
“Browser Working Principles and Practices”
Perspective HTTP protocol — Chrono
MDN