Introduction: The figure above is the table of contents of illustrated HTTP. Recently, I re-read this book and decided to write a summary to deepen my understanding. However, when making the summary, I thought it would be better to summarize the knowledge related to the network comprehensively, so I searched a lot of materials on the Internet and finally finished this paper.

Solemnly declare: this article is just my learning summary, I dare not mistake people’s children, if you see this article, feel there is any wrong place please point out. This article refers to a large number of blogs, articles, such as infringement, please contact me. thank you

OSI vs TCP/IP

OSI

OSI stands for the Open System Interconnection Reference Model. OSI divides a computer network architecture into seven layers.

1. Physical layer: transmits data frames over the local area network (LAN). It is responsible for managing computers, communication devices and the communication between network media. Including pins, voltage, cable voltage, hub, repeater, network card, host interface, etc.
2. Data link layer: responsible for network addressing, error detection, and error correction. The frame is formed when the header and tail are added to the packet. Data link headers (DLH) are methods that contain physical addresses and error detection and correction. A data linked list tail (DLT) is a string indicating the end of a packet. Examples are Ethernet, wireless Local area Network (Wi-Fi) and General Packet Radio Service (GPRS).
3. Network layer: determines the path selection and forwarding of data, and adds network header (NH) to packets to form packets. The network header contains network data, such as Internet protocol (IP) data.
4. Transport layer: The transport header (TH) is appended to the data to form packets. The transport header contains the protocol used to send information, such as transmission control Protocol (TCP).
5. The session layer is responsible for setting up and maintaining communication connections between two computers in the computer network during data transfer.
6. Presentation layer: Converts data into a format that is compatible with the receiver’s system format and suitable for transmission.
7. Application layer: Provides an interface for application software to set up communication with another software. For example, HTTP, HTTPS, FTP, TELNET, POP3, SMTP, and SSH.

OSI model is a standard rather than an implementation, TCP/IP four layer model is simplified from OSI 7 layer model, is a concrete implementation of an application.

Here are the four layers of models, starting from the bottom up:

TCP/IP

The link layer

Also known as data link layer, network interface layer. The part of the hardware used to handle network connections.

• Ethernet protocol

Ethernet states that a group of electrical signals constitutes a packet, called a Frame. Each frame is divided into two parts: header and data. The length of the header is fixed at 14 bytes, and the length of the data ranges from 46 to 1500 bytes. If the length of the data is insufficient, the data must be filled (for example, ARP and RARP). If the length is too long, the data must be divided into multiple frames for transmission.

The fixed 14-byte header is: destination address (6 bytes), source address (6 bytes), and type (2 bytes). The destination address or source address is a MAC address. 0x0800 is an IP datagram, 0x0806 is an ARP request, and 0x8035 is an RARP. The CRC at the end is the check code.

Ethernet uses broadcast to send packets. There are 3 computers, for example, a child in the network, computer 1 want to send a packet to computer 2, no. 2 and no. 3, the computer will receive the package, and then read the header of the packet, access to the destination MAC address, and its own MAC address match, if the same is to receive the package, or discard this package.

As you can see from the broadcast above, Ethernet packets must know the MAC address of each other to transmit data. How does one network adapter know the MAC address of another network adapter? It is actually obtained through THE ARP protocol, which will be explained later.

The network layer

Also known as the network interconnection layer. Used to process packets of data as they flow across the network.

TCP/IP protocol

The Internet is a huge network composed of numerous sub-networks. MAC addresses broadcast data between subnetworks. What about between different subnetworks?

The birth of the network layer is to introduce a new set of addresses that allow us to distinguish between different computers and whether they share the same subnetwork.

The protocol that defines the network address is called the IP protocol, and the address it defines is called the IP address. At present, the IP protocol version 4, or IPv4, is widely used. This version states that the network address consists of 32 bits. Traditionally, four decimal numbers represent IP addresses, ranging from 0.0.0.0 to 255.255.255.255.

The IP address is divided into two parts. The first part represents the network and the second part represents the host. However, from the IP address alone, we can not determine whether the network part is the first 24 bits or the first 16 bits and so on. In this case, we need to use a subnet mask. A subnet mask is a parameter that represents the characteristics of a subnetwork. Formally equivalent to an IP address, it is also a 32-bit binary number with all the network parts being 1’s and all the host parts being 0’s. IP address, for example, 192.168.1.1, if known network part is the first 24 bits, the host part is eight, after the subnet mask is 11111111.11111111.11111111.00000000, corresponding decimal is 255.255.255.0. The two IP addresses AND the subnet mask are respectively calculated AND compared to determine whether the two IP addresses belong to the same subnetwork.

IP datagram format

The header length and data length of an IP datagram are variously long, but are always multiples of 4 bytes. For IPv4, the 4-bit version field is 4. The length of the 4-bit header is 4 bytes, the minimum value is 5, that is, the minimum length of the header is 4*5=20 bytes, that is, the IP header without any options. The maximum value of the 4-bit energy is 15, that is, the maximum length of the header is 60 bytes.

The 8-bit TOS field has three bits to specify the priority of IP datagrams (now obsolete), four bits to indicate optional service types (minimum latency, maximum throughput, minimum cost), and one bit that is always 0. The 16-bit total length is the number of bytes in the entire datagram (including the IP header and the PAYLOAD at the IP layer). The maximum length is 65535 bytes. As mentioned above, the maximum size of the Ethernet data frame is 1500 bytes.

Each time an IP datagram is transmitted, the 16-bit identifier increases by 1, which can be used for sharding and reassembling datagrams. The 3-bit flag and 13-bit slice offset are used for sharding.

TTL (time to live) is this: the source host for packet set a survival time, 64, for example, every router has put the value minus 1, if the route has been reduced to 0 means too long still can’t find the destination network, just throw away the package, so the survival time of the unit is not seconds, but jump (hop). The protocol field indicates whether the upper-layer protocol is TCP, UDP, ICMP, or IGMP. Then there is the checksum, where only the IP header is checked and the data is checked by a higher protocol. Next comes the 32-bit IP address.

ARP protocol

If you want to send data to the other machine, you need both an IP address and a MAC address. The IP address can be obtained through DNS.

There are two cases:

In the first case, if two hosts are not on the same subnetwork, there is no way to obtain the MAC address of the other host. They can only send the data packets to the getaway gateway at the junction of the two subnetworks for processing.

In the second case, if two hosts are on the same subnetwork, we can obtain the MAC address of the other host through ARP. FF:FF:FF:FF:FF:FF :FF:FF indicates a broadcast address. The host receiving the broadcast matches its OWN IP address, responds with its own MAC address if it does, and otherwise discards the packet.

The transport layer

The application layer provides data transfer between two computers in a network connection. The network layer is host-to-host communication, while the transport layer establishes port-to-port communication. Indicates UDP and TCP.

UDP

The UDP header is relatively simple, totaling 8 bytes. See the above figure for details.

TCP

Like UDP, both the active and destination ports are identified by IP addresses and port numbers. 32-bit serial number, 32-bit confirmation number, and window size are explained later. The length of the 4-bit header is similar to that of the IP header. It indicates the length of the TCP header. If there is no option field, the minimum length of the TCP header is 20 bytes. URG, ACK, PSH, RST, SYN, and FIN are the six control bits. SYN, ACK, FIN, and RST will be explained later.

Look at a simple question and answer example:

Three-way handshake

1. The Client sends segment 1 (Client:SYN_SEND) with the flag bit SYN and the serial number 1000. This serial number is used as a temporary address in network communication. Every data byte sent, this serial number is added by 1, so that the receiving end can deduce the correct sequence of packets according to the serial number and also find the situation of packet loss. The SYN bit and the FIN bit must have the same sequence number. Although the SYN bit was not sent this time, the sequence number 1001 should be used for the next time. MSS indicates the maximum size of a segment. If a segment is too large and the length of the encapsulated frame exceeds the maximum frame length of the link layer, the segment must be fragmented at the IP layer. To avoid this situation, the client must specify the maximum size of the segment sent by the server.

2. The Server sends segment 2 (Server:SYN_RECV), which also has the SYN bit, and ACK bit indicates confirmation. The confirmation number is 1001, which means “I have received segment 1000 and all segments before it, please send segment 1001 next time”, that is, the connection request of the client is answered. It also sends a connection request to the client with a maximum size of 1024.

3. The client sends section 3 (ESTABLISHED) to answer the connection request from the server and confirms that the serial number is 8001.

In TCP communication, if one party receives a segment sent by the other party, reads the destination port number and finds that no process on the host is using this port, the other party will reply with a segment containing RST bits.

The data transfer

1. The client emits segment 4, containing 20 bytes of data starting with serial number 1001.
2. The server sends segment 5 and confirms that the serial number is 1021. It acknowledges receipt of data with serial number 1001-1020 and requests to send data starting from serial number 1021. When the server responds, it also sends 10 bytes of data starting from serial number 8001 to the client, which is called piggyback.
3. The client sends segment 6 to acknowledge receipt of data numbered 8001-8010 from the server and requests to send data starting from 8001.

In the process of data transmission, ACK and confirmation serial number are very important. The data sent by the application layer to TCP will be temporarily stored in the TCP layer send buffer. After sending data, only after receiving the ACK segment from the other party, confirm that the data is successfully sent and removed from the cache. If the wait times out, a resend is performed.

Four times to wave

1. The Client sends segment 7 (Client:FIN_WAIT_1), where the FIN bit indicates the request to close the connection.
2. The Server issues segment 8 (Server:COSE_WAIT, Client:FIN_WAIT_2) to answer the Client’s request to close the connection.
3. The Server sends section 9 (Server:LAST_ACK), which also contains the FIN bit, to the client to close the connection.
4. The Client issues segment 10 (Client:TIME_WAIT) to answer the server’s request to close the connection.

Connect is a three-way handshake, and close connection usually requires four (three times), the server’s response and close the connection request is usually not merged in a paragraph, because of the connection and a half closed, in this case the client after close the connection will not be able to send data to the server again, but the server can also send data to the client, Until the server also closes the connection.

Instead of immediately closing the connection after issuing segment 10, the client enters a TIME_WAIT state and waits for 2MSL. MSL indicates the Maximum Segment Lifetime. If Segment 10 fails to be sent due to network reasons, the server will resend Segment 9 after 1MSL. Segment 9 takes 1MSL for the slowest transmission, so you need to wait 2MSL to ensure that the last ACK is successfully sent.

Sliding window:

We know that UDP protocol is unreliable. When the sending speed is too fast and the processing speed of the receiving end is slow, data will be lost due to the fixed buffer size. TCP solves this problem through the Sliding Window mechanism.

1. The sender initiates the connection, stating that the maximum size is 1460, the initial serial number is 0, and the window size is 4K, which means “I have 4K bytes free in the receive buffer, you should not send more than 4K”. The receiver responds with a connection request stating that the maximum segment size is 1024, the initial sequence number is 8000, and the window size is 6K. The sender replies, and the three-way handshake ends.

2. The sender sends segments 4-9, each with 1K of data, and the sender knows from the window size that the buffer on the receiver is full, so it stops sending data.

3. The application on the receiving end removes 2K data, and the receiving buffer is idle for 2K more. The receiver sends segment 10 and declares the window size of 2K when it replies that it has received 6K data.

4. The application on the receiving end removes 2K data again, the receiving buffer is idle in 4K, the receiver sends out segment 11, and redeclares the window size to be 4K.

5. The sender sends segments 12-13 with 2K data each, and segment 13 also contains FIN bits.

6. The receiver responds to the received 2K data (6145-8192), plus the FIN bit occupies a serial number of 8193, so the reply serial number is 8194, the connection is half-closed, and the receiver also declares that the window size is 2K.

7. The application on the receiving end removes 2K data, and the receiver redeclares the window size to be 4K.

8. The receiver application removes the remaining 2K data, the receive buffer is empty, and the receiver redeclares the window size to be 6K.

9. The application on the receiving end decides to close the connection after all the data has been withdrawn. The sending segment 17 contains the FIN bit. The sending end responds and the connection is completely closed.

The application layer

It is the layer with the most contact, such as HTTP, DNS, FTP.

Taking HTTP as an example, data received from the application layer (HTTP packets) is divided at the transport layer (TCP) for convenient transmission, and the packets are marked with serial numbers and port numbers before being forwarded to the network layer.

At the network layer (IP protocol), add the MAC address as the communication destination and forward the MAC address to the link layer.

The server at the receiving end receives data at the link layer and sends the data to the upper layer in sequence, all the way to the application layer.

The HTTP status code

The table below at 👇 lists common HTTP status codes

The story of HTTP status codes 302, 303, and 307

HTTP/2

HTTP/2, or H2 (encrypted connections based on TLS/1.2 or above) or H2C (unencrypted connections), is the second major version of the HTTP protocol, based primarily on the SPDY protocol.

SPDY, developed by Google, is an application layer protocol based on transmission Control Protocol (TCP) designed to reduce web page load times. Through priority and multiplexing, SPDY enables the creation of a single TCP connection to transport network content and image resources. TLS encryption is widely used in SPDY, and transmitted content is compressed in GZIP or DEFLATE format. In addition, SPDY also supports the server to actively push data to the client.

Differences between HTTP/2 and SPDY

1. HTTP/2 supports plaintext transport, while SPDY enforces the use of TLS encryption.
2. HTTP/2 header compression uses HPACK, while SPDY uses DEFLATE.

The advantage of HTTP / 2

1. Frames, messages, and flows

HTTP/2 uses binary rather than text to transmit data, which is the biggest difference from HTTP/1.x. HTTP/2 divides a TCP connection into streams. Each Stream can transmit several messages, and each Message consists of several minimal binary frames.

Stream (Stream)

As shown in the figure above, streams typically represent each request and response, and multiple streams can be transmitted simultaneously in HTTP/2.

Message

A message is a set of frames that represent a specific request or response.

The Frame (Frame)

A frame is the smallest unit in HTTP/2 data transmission. Each frame contains a header and data. The header (fixed 9 bytes) is formatted as follows:

+-----------------------------------------------+ | Length (24) | +---------------+---------------+---------------+ | Type (8) | Flags (8) | +-+-------------+---------------+-------------------------------+ |R| Stream Identifier (31) | +=+=============================================================+ | Frame Payload (0...) . +---------------------------------------------------------------+Copy the code

• Length in 3 bytes represents the size of the entire frame.

• Type indicates the type. Currently, 10 types are specified as follows:

  • 0x0: DATA Indicates the DATA frame type
  • 0x1: Type of HEADERS request header
  • 0x2: PRIORITY Sets the PRIORITY of the flow
  • 0x3: RST_STREAM terminates the stream
  • 0x4: SETTINGS Sets the parameters of this connection
  • 0x5: PUSH_PROMISE Server push
  • 0x6: PING tests RTT
  • 0x7: GOAWAY Terminates the connection
  • 0x8: WINDOW_UPDATE Flow control
  • 0x9: CONTINUATION continues the transmission of header data

• Flag is a flag bit. Common ones are END_STREAM, END_HEADERS, and PADDED. Detailed participation here

• R is the reserved bit

• Identifier An identifier used to track the frame membership of a logical stream. Simply put, it is the stream marked.

2. Multiplexing

With the above flow concept in mind, multiplexing is easy to understand. HTTP/1.0 requires three handshakes each time to transmit data. Although keep-alive is added in HTTP/1.1, it does not need to establish a connection each time, but it still needs to wait for the return of the last response before sending a new request. In HTTP/2.0, only a request can be added by adding a stream. The data stream is sent in the form of a message, and the message consists of one or more frames, which can be sent out of order.

3. Prioritize

Since it can be multiplexed, the server and client frames are sent interleaved. For frames sent to the server, the priority of data flow is introduced to solve the problem of priority processing. Streams can be prioritized by a frame of type PRIORITY or header frame. You can also define dependencies that allow one resource to be loaded before another. You can also group the priorities into a dependency tree, giving the developer control over the importance of each flow.

In the figure above 👆, letters identify stream identifiers and numbers indicate the weight assigned to each stream. The root of the tree is stream A, to which resources are allocated first, and only then to the dependent streams B and C. Streams B are assigned 40% of the weight, and streams C 60% of the weight, meaning that streams B and C occupy 40% and 60% of the available resources, respectively. So let’s go down.

4. The header compression

The HPACK algorithm, tailored specifically for HTTP/2, provides good support for header compression. It uses an index table to define common HTTP headers, and stores common HTTP headers in the table. When requesting HTTP headers, you only need to send the corresponding index. For example, 🌰 :method=GET has the corresponding index value of 2, and path=/index.html has the index value of 5. The payload of this Frame is 0x8285, and the type of the Frame is set to HEADERS. This is a GET /index.html request header. It saves traffic and improves transmission efficiency.

0x8285 instead of 0x0205? This is because the high level is set to 1 to indicate that the byte is a fully indexed value (both key and value are in the index). Similarly, the high flag bit can tell whether the byte belongs to a fully indexed value, whether only key is indexed, or whether neither key nor value is indexed. Because the size of the index table is limited, it only holds some commonly used HTTP headers, and a new HTTP Header cache can be dynamically appended to the end of the table on each request. The Dynamic part is called the Dynamic Table. Static tables and Dynamic tables together form index tables.

<---------- Index Address Space ----------> <-- Static Table --> <-- Dynamic Table --> +---+-----------+---+ + + -- -- -- -- -- -- -- -- -- -- -- -- -- - + -- -- -- + | 1 |... | s | |s+1| ... |s+k| +---+-----------+---+ +---+-----------+---+ ^ | | V Insertion Point Dropping PointCopy the code

Not only does HPACK reduce the amount of data by indexing values, but it also compresses the string size by Huffman encoding.

5. Server push

One of the key features of HTTP/2 is that the server can’t push it if it wants to. The server follows the request-response model, but the server can push multiple responses to the same request. When exchanging the SETTINGS Frame, the client sets the SETTINGS_ENABLE_PUSH (0x2) field to 1 to indicate that the server is allowed to push.

When a server wants to PUSH a resource, it sends a Frame of type PUSH_PROMISE with the Stream ID that PUSH will create. Tell the client “I’m going to use this ID to send you something next, you can catch it.”

Reference:

The illustration of HTTP

The OSI model

HTTP/2

Basic TCP/IP protocol

SPDY

Principles behind HTTP/2

A brief analysis of HTTP/2 new features