preface

If you don’t want to spend too much time reading a long theoretical book and want to learn the basics and advanced content of THE HTTP protocol in a simple language, then this blog is for you.

In addition to sorting out the basic concepts of HTTP, common fields, cookies, caching and other basic to advanced content, this paper also appropriately adds some answers to interview questions. Examples include the difference between HTTP2.0 and 1.1, what happens to the display from URL input to page, browser caching, HTTP status codes, and more.

Because of the complexity and complexity of HTTP knowledge, I will only put the more common knowledge THAT I can understand in colloquical terms to make sure that I do not teach others how to use their knowledge, and also to be responsible to my friends who read my blog.

The main references for this article are from Perspective HTTP protocol and Illustrated HTTP, and some references are from MDN. By definition, it is more like a book note, but I have removed a lot of the tedious knowledge from the book and made such a compilation.

If you think the knowledge I sorted out is useful to you, it is a great encouragement to give the author a thumbs up 👍🏻👍🏻.

Finally, I am maintaining a Github blog address, which is mainly used to record important knowledge points in the front end from learning to summarization, involving in-depth Javascript, HTTP protocol, data structure and algorithm, browser principle, ES6 and other content. It is currently updating, welcome everyone 🌟🌟star🌟🌟

Github address: My Github

The following text begins:

The origin of the HTTP

Prehistoric times

In the 1960s, the ADVANCED Research Projects Agency of Defense (ARPA) established the ARPA network, which has four nodes scattered across the country and is considered to be the “ancestor” of today’s Internet.

Then in the 1970s, based on the practice and thinking of ARPA network, researchers invented the famous TCP/IP protocol. TCP/IP quickly gained popularity over its competitors because of its good hierarchical structure and stable performance.

creation

In 1989, a Doctor named Tim Berners-Lee published a paper outlining an idea for a hyperlinked document system on the Internet.

In this paper he identified three key techniques.

1. URI: Uniform Resource Identifier (URI), the unique identity of a resource on the Internet.
2. HTML: Hypertext markup Language, describing hypertext documents;
3. HTTP: Hypertext transfer protocol, used to transfer hypertext.

Based on this idea, Dr. Lee called the system the “World Wide Web,” which is now the Web. This was the year HTTP was born.

HTTP / 0.9

Due to the multiple influences of memory, CPU computing power and network speed in the Internet world at that time, HTTP was simple in structure when it was involved in the beginning, and it was also in plain text. Only GET requests are allowed, which fetch the HTML document from the server and close the connection immediately after the request.

Although simple, as a prototype, it successfully fulfilled its historic mission, proving the web to be viable and laying the foundation for further expansion.

HTTP / 1.0

In 1993, Mosaic, actually the first web browser that could mix text and text, was created, followed by Apache, a server software, in 1995, which made it easier to build HTTP servers.

At the same time, the birth of calculator multimedia such as JPEG and MP3 led to the development of HTTP. This was the 1.0 era, and the version differed very little in form from current HTTP. Such as:

1. New methods such as HEAD and POST have been added;
2. The response status code is added to mark the possible cause of the error.
3. The concept of protocol version number is introduced.
4. The concept of HTTP headers is introduced to make HTTP more flexible in handling requests and responses.
5. Data transmission is no longer limited to text.

HTTP / 1.1

In 1995, Netscape Navigator and Microsoft’s Internet Explorer began what became known as the “Browser Wars,” which greatly promoted the development of the Web.

In 1999, HTTP/1.1 released the RFC document, which continued for more than a decade. Compared to 1.0, it changes the following:

1. Add new methods such as PUT and DELETE;
2. Added cache management and control;
3. Clear connection management to allow persistent connections;
4. Allows response data to be chunked, facilitating large file transfers;
5. Mandating the Host header makes Internet hosting possible.

Differences between HTTP2.0 and 1.1

In response to the slow connection and inability to keep up with the rapid growth of the Internet in version 1.1, Google has introduced its own server on its popular Chrome browser and the new SPDY protocol. At this point, the Internet Standardization Organization finally released version 2.0 based on this.

HTTP2.0 takes into account current broadband, security, and mobile Internet conditions, and complements the performance shortcomings of version 1.1:

1. Binary protocol, no longer plain text;
2. Can initiate multiple requests;
3. Compress the request header to reduce the amount of data transmission;
4. Allow the server to actively push data to the client;
5. Enhanced security, “de facto” requires encrypted communications.

Version 2.0, however, has not been widely available to this day.

The extension of HTTP3.0

Google has released a more optimized version of the QUIC protocol again, experimenting with its own Chrome browser, and continuing to push the development of the QUIC protocol with the support of a large number of users and data.

In 2018, the International Standards Organization released version 3.0 of HTTP3.0 as a standard. It’s possible we’ll just skip 2.0 and adopt 3.0.

What is the HTTP

HTTP stands for HyperTextTransfer Protocol. From its name, the Protocol can be broken into three parts:

• hypertext
• transmission
• agreement

agreement

What is an agreement? This is just like the agreement in daily life: labor agreement, rental agreement and so on is a concept, the essence of HTTP protocol is similar to such agreement.

Let’s start with the literal meaning of the agreement. First of all, collaboration means collaboration, which means there are multiple parties involved. Secondly, it is with the coordination of many parties, just need some basic communication etiquette and behavior agreement, this is discussion.

An agreement means that multiple participants stand together for the same purpose. In order to ensure cooperation, a behavioral agreement must be made on the responsibilities and rights of each party.

So here comes the first meaning of HTTP:

HTTP is a protocol used in the computer world. It uses a language that computers can understand to establish a specification for communication between computers, as well as related controls and error handling.

transmission

HTTP is A protocol for transfer, “transfer,” which means to move something from A to B. It contains the following two pieces of information:

1.HTTP is a two-way protocol

HTTP transmission requires at least two participants, A and B. Data needs to flow in both directions between A and B. Usually, A who initiates the action first is called the requester, and the receiving and transmitting party is called the responder.

The browser, for example, is often the requester, and the responder is the server, and they rely on HTTP to communicate. The browser sends the data request to the server, the server returns some form of data, and the browser gets the data and renders it.

2. Although two-way data transmission is allowed, there is transfer.

The original A<==>B becomes A<==> transfer station 1<==> Transfer station 2, etc. <==>B. As long as the transmission from A to B is not affected, there are multiple middlemen in the transmission process in principle. Middlemen also follow HTTP, but they can do data forwarding, security authentication, data compression, and other functions to optimize the transmission process.

From this, we know the second meaning of HTTP:

HTTP is a convention and specification for transferring data between two points in the computer world.

hypertext

Since it is called Text, it means that HTTP transmits a complete and meaningful piece of data, which can be processed by browsers and servers. It is not a binary package.

In the early days, the transmission of text than ordinary words, but now, we can transfer audio, images, video and other data stream, so the meaning of hypertext is beyond ordinary text, is a combination of a variety of data text, it can also include hyperlinks, can from a hypertext to jump to another hypertext, form a reticular structure.

HTML is a type of hypertext. It is simply a text file, but its tags can define links between images and videos, which can be interpreted by the browser to form a page containing a variety of audio and visual information.

So the hypertext interpretation is

A collection of hypertext data, such as text, image, audio and video

summary

HTTP is a communication convention and specification for the transmission of video, audio, pictures, text and other hypertext data between two points in the computer world.

Although HTTP has no entities, its dependencies require technical implementation, and it is an important foundation and technology for building the Internet

HTTP related applications

The Web server

The browser is the requestor in HTTP, and the other side of the protocol is the responder ==> Server,Web Server

Web servers have two layers of meaning: hardware and software.

Hardware is a machine in physical or cloud form, not necessarily a server in most cases, but a cluster of technologies such as reverse proxies and load balancing.

Software is an application program that provides Web services, usually running on a server with hardware meaning. It uses hardware capabilities to respond to massive HTTP requests, process various Web pages, pictures and other files, or forward requests and so on.

At present, the two most popular Web servers in the world are the old Apache server (features: complete functions, low learning threshold) and the rising Nginx (features: high performance, high stability).

CDN

The full name of CDN is Content Delivery Network. It is a Content establishment between browser and server. It applies caching and proxy technology in HTTP protocol and can proxy source site to respond to client request.

What is it good for?

It caches source data so that the browser doesn’t have to travel thousands of miles to the source server to get the response. If the scheduling algorithm of CDN is excellent, it can find the node that is far from the user’s follow-up and greatly shorten the response time.

The crawler

A browser is a user agent that accesses the Internet on our behalf.

But this agent can also be a robot, and these robots are called crawlers. Essentially an application that automatically accesses Web resources.

How did reptiles come about? Most of them are put out by search engines, which can capture web pages and store them in a huge database, and then build a keyword index, so that we can search the various pages of the Internet in search engines.

Crawlers also have some disadvantages, such as the disclosure of sensitive information. Therefore, now there is an anti-crawler technology, through various means to restrict crawlers.

Web Service

Web Service and Web Server have similar names, but are completely different things.

It is a Web (HTTP) based service architecture technology that can run either on the Intranet or, with proper protection, on the extranet.

Because HTTP protocol is used to transmit data, the server and client can be developed using different operating systems or programming languages in the Web Service architecture. For example, Linux+Java is used on the server and Windows+C# is used on the client, which has the advantage of cross-platform and cross-language.

summary

• The vast majority of resources on the Internet are transmitted through HTTP. The browser is the requestor of HTTP, namely the User Agent. Server is HTTP protocol in the response, commonly used Apache and Nginx;

• CDN is located between the browser and the server, mainly playing the role of cache acceleration.

• Crawler is another kind of User Agent, which can access network resources automatically.

• Web Service is a Service architecture technology with cross-platform and cross-language characteristics.

HTTP related protocols

TCP/IP

TCP/IP protocol is the “de facto” standard communication protocol in the network world at present. It is a general name of a series of network communication protocols, among which the two most core protocols are TCP and IP.

It also has other protocols such as UDP, ICMP, ARP and so on. Together they form a complex protocol stack.

There are four layers to this agreement

1. The application layer
2. Transport Layer (TCP home Layer)
3. Internet Layer (IP home layer)
4. The link layer

IP is short for Internet Protocol, which is mainly used to solve the problems of addressing and routing, and how to transmit data packets.

IP systems use the concept of an IP address to locate each computer. Corresponding to the telephone system, to make a call, you must connect to the telephone network. The communication company assigns a number, which is equivalent to an IP address.

TCP is the abbreviation of Transmission Control Protocol, meaning “Transmission Control Protocol”. It is based on IP Protocol to provide reliable and byte stream communication, which is the basis of HTTP.

Reliability refers to the assurance of data loss, byte stream to ensure data integrity.

TCP/IP is a reliable, complete protocol

DNS

TCP/IP uses IP addresses to identify computers. Since computers themselves process data, such addresses are convenient for computers, but they are not good for human memory.

The DNS domain name system can solve this problem. It replaces IP addresses with a set of meaningful names and establishes mappings. For example: visit www.baidu.com to access Baidu’s IP address 14.215.177.38 (there is more than this one)

In DNS, a domain name, also known as a host name, is designed to be a very hierarchical structure. The ones on the right, such as.com and.cn, are top-level domains

Below the top-level domain is the second-level domain, which is located to the left of the top-level domain. For example, in zh.wikipedia.org, Wikipedia is a secondary domain name. In w3.org, w3 is also a secondary domain name.

Below the level 2 domain name is the level 3 domain name, which is located to the left of the level 2 domain name. For example, in zh.wikipedia.org, zh is a level 3 domain name

To use TCP/IP to communicate, you still need to use IP addresses, so you need to do a translation of the domain name, “mapping” to its real IP address, this is called “domain name resolution”.

The URI and URL

DNS and IP addresses mark hosts on the Internet, but there is a lot of text, web pages and so on on the host. Which one do I need to find?

This is where the URI, the full Uniform Resource Locator identifier, comes in. You can use it to uniquely tag resources on the Internet.

Another representation of a URI is a URL (uniform resource Locator), commonly known as a web address. It is a subset of urIs, but not very different.

What does a URI look like?

http://nginx.org/en/download.html
Copy the code

The URI above consists of three basic parts:

1. Protocol name: HTTP

2. Host name: indicates the location of a host. It can be a domain name or AN IP address. Here is nginx.org

3. The path, that is, the location of resources on the host, is separated into multiple directories using slashes (/). This is/en/download. HTML

HTTPS

This is the secure version of HTTP, because HTTP is not secure enough, so on top of TCP/IP, set up a layer of SSL/TLS protocol, and HTTPS means HTTP running on SSL/TLS protocol.

HTTP over SSL/TLS

SSL/TLS is a secure and reliable transport protocol responsible for encrypting communications

SSL/TLS was originally called Secure Socket Layer (SSL) and later changed to Transport Layer Security (TLS). For historical reasons, many people called SSL/TLS.

It uses many cryptography research achievements, integrated symmetric encryption, asymmetric encryption, summary algorithm, digital signature, digital certificate and other technologies, can create a secret, secure transmission channel for both sides of communication in an insecure environment.

If the protocol name of a URL is HTTPS, it indicates that the URL is HTTPS enabled.

The agent

A proxy is an environment between the requestor and the responder in THE HTTP protocol. It serves as a forwarding station for forwarding requests and responses.

The following lists the types of agents

1. Anonymous proxy: completely “hide” the proxy machine, the outside world only see the proxy server;

2. Transparent proxy: As the name implies, it is “transparent and open” in the transmission process, the outside world knows both the proxy and the client;

3. Forward proxy: close to the client and send requests to the server on behalf of the client.

4. Reverse proxy: close to the server and responds to client requests on behalf of the server.

A CDN is actually a proxy that responds to client requests instead of a server, usually acting as a transparent proxy and a reverse proxy.

As an intermediate layer of transport, a proxy can do many things, such as:

1. Load balancing: Evenly distribute access requests to multiple machines to achieve access clustering.

2. Content cache: temporarily store up and down data to reduce the pressure on the back end;

3. Security protection: hide IP, use WAF and other tools to resist network attacks, protect the proxy machine;

4. Data processing: provides additional functions such as compression and encryption.

There is also a special proxy protocol for HTTP proxies.

summary

TCP/IP is the most commonly used protocol in the world. It has the characteristics of reliability and integrity. HTTP runs on top of TCP/IP.

A DNS domain name is an equivalent substitute for an IP address. You need to use domain name resolution to map an IP address

A URI consists of a protocol, host name, and path.

HTTPS consists of HTTP, SSL, TLS, and TCP/IP.

The proxy is a transfer station for HTTP transmission and can realize functions such as cache acceleration and load balancing.

TCP/IP network layered model and OSI network layered model

TCP/IP network layered model

TCP/IP takes a divide-and-conquer approach, dividing complex network communication into four layers

The bottom is the first layer, and it goes up.

Layer 1: Link Layer

This layer is mainly responsible for sending raw packets in Ethernet, wifi and other underlying networks. It works at the nic level and uses MAC addresses to mark network devices, so it is also called the MAC layer.

A MAC address, also called a LAN address, uniquely identifies a NIC and the device to which the NIC belongs.

The second layer: The IP protocol is at this layer. The IP protocol defines the CONCEPT of IP addresses. Therefore, based on the link layer, IP addresses are used instead of MAC addresses. Translate the IP address into a MAC address.

The transport layer is responsible for ensuring the reliable transfer of data between two points. This layer is where TCP is located, along with UDP.

TCP is a reliable, stateful, and ordered protocol. Data can be sent only after a connection is established between the two parties. Data loss and duplication must be ensured. Hence the classic interview question: three handshakes and four waves.

UDP is simpler. It has no state and does not need to establish a connection to send data at will, but there is no guarantee that the data will be sent to the other party.

TCP sends data sequentially, while UDP sends data sequentially and receives data out of order.

This layer is the layer where HTTP protocol is located. There are various application-oriented protocols. In addition to HTTP, Telnet and SSH encryption are also responsible for remote terminal protocols.

At the MAC layer, the transmission unit is frame, IP packet, TCP segment, and HTTP message. Collectively referred to as packets

OSI network layered model

OSI (Open System Interconnect Communication Reference Model)

In the era of TCP/IP, when there were many other protocols, the International Organization for Standardization came up with a unified specification, so the new network layer model OSI was designed.

The international Organization for Standardization is ISO for short, or OSI for short.

The OSI model has seven layers, some of which are similar to TCP/IP, from bottom to top

1. Layer 1: physical layer, the physical form of the network, such as cable, optical fiber, network card, hub and so on;
2. The second layer: data link layer, which is basically equivalent to the TCP/IP link layer;
3. Layer 3: network layer, equivalent to TCP/IP in the Internet layer;
4. Layer 4: transport layer, equivalent to TCP/IP transport layer;
5. The fifth layer is the session layer, which maintains the connection state in the network, that is, maintains the session and synchronization.
6. The sixth layer is the presentation layer, which translates the data into appropriate and understandable syntax and semantics.
7. The seventh layer is the application layer, which transmits data for specific applications

In terms of features, OSI is more complete than TCP/IP, in addition to the TCP/IP protocol details, but also specifically added to the physical device level.

The mapping relationship

Because OSI references TCP/IP’s layered architecture from the beginning of its design, it is relatively easy but imprecise to represent the correspondence between the two. The following figure shows

Because OSI’s seven-layer model is too detailed, TCP/IP’s actual application of session management, coding conversion, compression and other applications are often closely linked, it is difficult to separate. So this leads to a result:

OSI is the standard protocol, but TCP/IP can still be used in practical applications, so the comparison between layer 5 session layer and layer 6 presentation layer disappears.

Layer 4 load balancing and Layer 7 load balancing

Now that we know the concept of layer XX, we can see that layer four refers to the transport layer and layer seven refers to the application layer.

Layer 4 load balancing refers to the load balancing of back-end servers at the transport layer based on IP addresses and port numbers.

Layer 7 load balancing refers to the forwarding of data, such as HTTP urls, host names, and resource types, to back-end servers through appropriate policies at the application layer.

How does TCP/IP work

When sending data:

1. First the HTTP protocol wraps up what we need to transport – adding HTTP dedicated additional connections.

This is just like when we need to send express, we need to take a plastic bag and wrap it by ourselves.

2. After passing through the TCP layer, the TCP layer repackages the data and adds the TCP header

This is like the Courier gets our express, then will compress, continue to pack.

3. Add an IP header and a MAC header to the TCP data

It’s like a Courier putting a package into a tricycle and then into a big truck.

When the other party receives data:

1. Data goes to the MAC layer and IP layer of the peer party, and the peer party unpacks the data respectively

Couriers unload packages from big trucks and load them onto tricycles to each other’s homes

2. Remove the TCP header and HTTP header to obtain data

The other party takes down the package and your plastic bag and takes the contents.

This is how HTTP is transmitted, layer by layer, up the protocol stack, adding data specific to each layer, packaging it, and sending it out. There is an idiom that means to surround one layer by another

Receiving data, on the other hand, works its way up through the protocol stack, removing each layer’s proprietary header, and finally retrieving the data. To use an idiom to describe it is to pluck the silk from the cocoon

It can be represented by the following figure

summary

TCP/IP has four layers, with layer 2 IP and layer 3 TCP at the core, and HTTP at the top.

OSI has a seven-layer model, in which the data link layer is the link layer of TCP, the network layer is the network interconnection layer of TCP, the transport layer is the transmission layer of TCP, and the seventh application layer corresponds to the fourth layer of TCP.

OSI layer 1 TCP does not exist, layer 5 and layer 6 are in TCP application layer, by contrast, these layers have disappeared.

HTTP uses TCP/IP protocol stack layer by layer packaging, and then layer by layer unpacking to achieve data transmission.

Generally speaking, the application layer is written, and the operating system deals with four or fewer layers.

The domain name system

We now know the key between DNS and IP, and the process by which DNS servers convert domain names into IP addresses is called domain name resolution.

At present, there are hundreds of millions of sites in the world, and the HTTP traffic is astronomical every day, and most of these requests are accessed by domain names, so DNS has become an important infrastructure of the Internet, it is necessary to ensure the accuracy and efficiency of domain name resolution.

The core system of DNS is a three-layer tree and distributed service, which basically corresponds to the structure of domain name:

1. Root DNS Server: Manages the TOP-LEVEL DNS Server and returns the IP addresses of such top-level DNS servers as com, net, and CN

2. Top-level DNS Server: An authoritative DNS Server that manages its own domain name. For example, the COM top-level DNS Server can return the IP address of the apple.com DNS Server.

Authoritative DNS Server: manages the IP address of the host with its own domain name. For example, the Apple.com Authoritative DNS Server can send back the IP address of www.apple.com

Here the root domain name server is the key, there are currently 13 groups of root domain name servers in the world, and there are hundreds of mirrors, to ensure that they can be accessed.

With this system, any domain name can be queried from top to bottom in this tree structure, as if the domain name from right to left, and finally get the corresponding IP address of the domain name.

For example, if you want to access www.apple.com, perform the following three queries:

1. Visit the root DNS server and it will tell you the address of the “com” top-level DNS server;
2. Visit the “com” top-level domain name server, and it tells you the address of the “apple.com” domain name server;
3. Finally, go to the Apple.com domain name server and you get itwww.apple.comThe address.

Although the core DNS system is strong and stable all over the world, if the Internet users all over the world crowded into the system, even if not crowded down, the access speed will be slow

In addition to the core DNS system, there are two ways to reduce the stress of domain name resolution and get results faster. The basic idea is “caching”.

First, many large companies and network operators set up their own DNS servers to act as proxies for users’ DNS queries and access the core DNS system on behalf of users.

These “wild” servers are called “non-authoritative DNS servers” and can cache previous query results. If there are already records, the root server does not need to initiate a query and directly returns the corresponding IP address.

Second, the operating system also caches DNS resolution results. If you’ve visited www.apple.com before, the next time you type in the url in your browser, you won’t have to ask DNS. The operating system will get the IP address directly.

In addition, the operating system has a special “host mapping” file, which is usually an editable text. On Linux, it is /etc/hosts, and on Windows, it is C:\ Windows \ System32 \drivers\etc\hosts, If the operating system can’t find DNS records in the cache, it looks for this file.

summary

Domain names use strings instead of IP addresses to facilitate user memory, essentially a namespace system;

Domain name resolution converts domain names into IP addresses. DNS is like the telephone directory directory of our real world. It controls all websites on the Internet, acting as a “super steward”.

There are 13 important root DNS servers around the world that hold the IP addresses of top-level domain names, which look up the IP address from the right forward, The corresponding DNS server is the root DNS server. ==> Top-level DNS server (apple.com) ==> Authoritative DNS server (apple.com www.apple.com)

DNS is a tree-like distributed query system with multiple levels of caches to improve query efficiency

The two common caching methods are

• Network providers and carriers provide non-authoritative DNS servers to cache IP records without going through the root DNS server.
• The operating system itself caches IP addresses, and the host file is also used locally to cache IP addresses.

What does an HTTP packet look like

Message structure

TCP/UDP packets are transmitted with header data, which has been discussed above. HTTP, too, wraps a layer of data, and since HTTP is a hypertext protocol, much of what is actually being transmitted is readable without parsing.

HTTP request packets and response packets have the same structure and consist of three parts

1. Start line: Describes basic information about a request or response.

2. Request headers: Use key-value pairs to represent table text

Entity: The actual transmitted data, which may be plain text or binary data such as pictures and videos.

The request line and the request header are often referred to collectively as the request header or response header. The body of the message is called “body”.

The HTTP protocol stipulates that a packet must have a header, but a body can be omitted, and an empty line must be added after the header.

In the browser it looks like this

This is a request message:

Its first line, the request line, represents the GET mode, based on HTTP1.1

The second line is the request header, which describes the host name, link status, receive method, and so on.

The last line is a blank line. There’s no body behind it

It is allowed to send a GET request without a BODY.

Although the HTTP protocol does not explicitly limit header size, each browser and Web server limits request headers so that they are not too large for efficiency.

The request line

The request line consists of three parts:

1. Request method: is a verb, such as GET/POST, to operate on a resource.

2. Request target: Usually a URI that marks the resource that the request method will operate on.

3. Version: indicates the HTTP version

These three sections are usually separated by Spaces and end with newline CRLF.

This is true in the browser

GET / HTTP/1.1 
Copy the code

GET means send by the requester. The/is immediately followed by the root directory of the target, which means I want access to the default resource. Finally, the HTTP version.

The status line

A status line is run inside a response message, or you can call it a response line, but it also has a standard name — a status line. This means the status of the server response.

The status line is divided into three parts:

1. Version: indicates the HTTP version used by the packet

2. Status code: a three-digit number, which represents the result of processing in the form of numbers. For example, 200 is success and 500 is server error.

3. Reasons: To complement numeric states are more detailed explanations, such as helping others understand why.

In a browser, it might look like this

HTTP / 1.1 200 OK
Copy the code

HTTP1.1 based request, status code 200, all ok.

It could be something like this

HTTP / 1.1 404 Not Found
Copy the code

404 status code not found.

Header fields

The collection of request or status lines plus header fields constitutes the complete HTTP request or response header

The structure of the request header is the same as that of the response header, with the only difference being the start line.

Header fields are divided into key-values, separated by:, and ended with a newline. For example, in “Host: 127.0.0.1”, key is “Host” and value is “127.0.0.1”.

HTTP fields are flexible enough to use not only standard fields such as Host, but also arbitrary fields. So it’s very scalable.

But there are a few details to note:

1. The field name is case-insensitive. For example, you can write Host as Host, but it is more readable to capitalize the first letter.

2. No Spaces are allowed in the field name. Use hyphens (-) but not underscores (_). For example, “test-name” is a valid field name, and “test name” and “test_name” are incorrect field names.

3. The field name must be followed by a colon (:) and cannot contain Spaces. The field value after a colon (:) can contain multiple Spaces.

4. The order of fields is meaningless and can be arbitrarily arranged without affecting semantics;

5. A field cannot be repeated in principle unless the semantics of the field itself allow it, such as set-cookie.

Common header field

HTTP protocol has a lot of header fields, can achieve a variety of functions, generally divided into four categories:

1. Common fields: Either request headers or response headers can appear.

2. Request field: Can only appear in the header of the request to further explain the request information or additional conditions.

3. Response field: Appears only in the response header, which adds information about the response packet.

4. Entity field: Also a generic field that describes additional information about the body.

Here are some common fields

• Host field: a mandatory field specified in HTTP1.1 that tells the server which Host the request should be processed.

• User-agent field: A request field that appears only in the request header and represents the client that initiates the request.

• Date field: A generic field that usually appears in the response header and indicates when the HTTP was created.

• Server field: The response field, which appears only in the response header and tells the client the name and version number of the software that the current re provides the Web service. This field exposes part of the server’s information to the outside world, so it can be hacked, so it is allowed to have none, or to give a completely unrelated description.

On Github, this field doesn’t tell us whether to use Apache or Nginx.

• Content-length field: Represents the Length of the Body, which is an entity field. Tells the server how much data needs to be received.

summary

An HTTP packet consists of a start line, a header, a blank line, and an entity. Simply put, an HTTP packet is header and body

HTTP packets can have no body, but must have headers and empty lines.

The request header consists of a request line + header field, and the response header consists of a status line + header field

The request line consists of the request method, the request target, and the version number

The status line consists of the status code, additional information, and version number

The header field adopts the form of key:value, which is case-insensitive and does not have order problems. You can also customize the field to achieve arbitrary expansion.

In HTTP1.1, the HOST field must appear. It must appear in the request header and represent the HOST name of the other party

Request method

In the Request Header, there are eight Request methods and the target of the Request.

1.GET: Reads or downloads data

2.HEAD: Obtains the meta information of the resource

3.POST: Submits data, which is equivalent to writing data

4.PUT: Modifies data

5.DELETE: Deletes resources

6.CONNECT: Establish a special connection tunnel

7.OPTIONS: Lists the methods applied to the resource

8.TRACE: TRACE the request – the corresponding transmission path

All of these methods are capitalized.

GET/HEAD

GET is the oldest request method in HTTP and the most used.

The semantics are to fetch resources from the server, which can be static text, pages, videos, etc. Generally, if a GET request has parameters, its parameters need to be added to the query string in the URI.

The HEAD request is similar to the GET request method, but the server does not return the requested entity data. Instead, the server returns the response header, which is the meta information about the resource.

The HEAD method can be thought of as a simplified or lighter version of the GET method. It can be used in situations where resources are not needed to avoid the waste of transmitted data.

For example, to check if a file exists, simply send a HEAD request.

POST/PUT

Both methods are very similar in that they specify to send data to the server, usually in the body.

For example, if we send a request to the server to add to the shopping cart, the item you like will be sent to the server as data in the body.

PUT is the same, submitting data to the server, but semantically, PUT is more like Update, while POST is more like CREATE.

Unusual method

The DELETE: method instructs the server to DELETE the resource, which is so dangerous that the server usually does not actually DELETE the resource, but marks it with a DELETE mark.

CONNECT is a special method that requires the server to establish a special connection tunnel between a client and another remote server, with the Web server acting as a proxy.

The OPTIONS method asks the server to list the available operations on the resource, returned in the Allow field of the response header. It has limited functionality and is not very useful, and some servers (such as Nginx) simply don’t support it.

The TRACE method is used to test or diagnose HTTP links and display the request-response transmission path. It means well, but it has bugs that can leak information about a website, so Web servers are usually forbidden.

Security and idempotence

In practical interviews, there are two important concepts, security and idempotence.

So secure, that is, there is no risk of modifying the server resources, only GET and HEAD requests are safe, because they are read-only operations.

Idempotent is a mathematical concept which means that no matter how many times you operate, the result is the same. So the powers are equal.

Obviously, GET and HEAD are idempotent, but POST is not, because the semantics of POST represent creation, which of course creates new results. PUT is special in that although it also modiates data, its semantics modify rather than add or subtract, and it updates only one resource no matter how many times it does so, so it is idempotent as well.

We can assume that the result here refers to the amount of data. Multiple posts increase the amount of data, but multiple changes do not, which is the popular meaning of idempotent and non-idempotent.

summary

A request is an operation on a resource made by a client that requires the server to perform.

The request is just an indication, and you need to negotiate with the server how to do it.

The common request methods are GET and POST. They are getting data and sending data.

HEAD is a lightweight GET that fetches only meta information, the response header, at a time.

Basically the same as POST, PUT is used to update data. PUT is idempotent and POST is non-idempotent.

Security and idempotence are two important attributes that describe request methods.

URI

Request method is to send instructions to the server through instructions to guide the server to complete a certain action, commonly used is to obtain resources, so how to distinguish resources?

The answer is to use a URI, the Uniform Resource Identifier (URI). Because it often appears in the browser’s address bar, it is colloquially known as a web address.

Strictly speaking, A URI is not exactly the same as a web address. It contains two parts: the URl and the URN. In HTTP, the URl is the Uniform Resource Locator (URIS), and because urls are so common, urIs are often equated with URLS.

Uris are very important, and to understand HTTP, you have to understand URIs.

The format of the URI

URI is essentially a string, the string is the only location or name tag resources, it can not only mark the world wide web resources, also can mark the mail system, the local file system, such as any resources, the resource can be either a static text, page data, also can be provided by the Java dynamic service.

A URI consists of scheme, host: POST, Path, and Query.

scheme

The first component of a URI is called a Scheme, or protocol name, which indicates which protocol should be used to access resources.

The most common protocol is HTTP, followed by the secure and encrypted HTTPS protocol, and so on.

In the browser, if the browser sees Scheme on the address bar, it calls the corresponding underlying API to handle the URI.

After Scheme, there are always three characters: ‘://’, which separates scheme from the rest.

authority

After ://, authority indicates the host name of the resource. It is usually in the form of host:port, that is, the host name and port number.

The host can be an IP address or a domain name, which must be present otherwise the browser will not find the server. The default HTTP port number is 80, and the default HTTPS port number is 443.

path

With authority, the following path section marks which path the resource is under. With Scheme, host name, port number, and PATH, the service area can access the resource.

The path in the URI is similar to the directory path of the file system. The file directory of the early UNIX system was divided by /.

The path section must start with a /.

The instance

Let’s examine a few examples

http://nginx.org
http://www.chrono.com:8080/11-1
https://tools.ietf.org/html/rfc7230
file:///D:/http_study/www/
Copy the code

The first URI is simple: the protocol is HTTP, the host name is nginx.org, and the port number is the default 80. The path is ignored and defaults to /. That’s the root directory.

The second URI has the full path and port number.

The third URI is HTTPS, the default port number is 443, and the path is/HTML/rfC7230

The fourth URI is the file protocol, which represents the local file, followed by three slashes :// represents the split protocol and authority part, and the remaining/represents the beginning of the path, which is the /http_study/ WWW/path of drive D in the root directory. Since file is a special case of URI, it allows the host name to be ignored and defaults to localhost.

The URI you see on the browser is different from the URI you see on the server. For example, in my gold digging home page, open the F12 developer tool and click on View Source to see the original request header like this

The URI the server sees is /v1/list, which corresponds to the path I requested. This is because both the protocol name and Host name already appear in the request line and Host fields, so the server only needs to fetch the URI with the protocol name and Host name removed.

Query parameters

The protocol name + host name + path method can be used to pinpoint resources on the network, but we also want to add some additional decoration parameters for specific scenarios, such as to get a list of items, paging and sorting according to some rules, query parameters used in the field.

The URI is followed by a query section that takes? Start indicates additional requirements.

The query parameter has its own format, usually multiple key=value strings. The key-value pair is not a Javascript: concatenation, but an & concatenation. Both browsers and clients use this format to parse a long string of query parameters into an understandable data structure.

https://www.youtube.com/watch?v=FUN5rfoqLLA&t=116s
Copy the code

In the url above? V = fun5rFOQLla&t =116s is the query parameter query.

URI encoding

Only ASCII codes can be used in URIs. If you want to use a language other than English, such as Chinese or Japanese, some special URIs will appear “@&?” on path and query. These characters, and these characters have special uses on urIs, how do you distinguish them?

This brings us to the encoding rules for URIs, which introduce an encoding mechanism that performs special operations on character sets or special characters other than ASCII encoding.

The escape rules for URIs are a bit crude, converting non-ASCII or special characters directly to hexadecimal byte values preceded by %.

For example, the Milky Way will be escaped as %E9%93%B6%E6%B2%B3″

encodeURI('the Milky Way')
//"%E9%93%B6%E6%B2%B3"
decodeURI('%E9%93%B6%E6%B2%B3')
/ / "Milky Way"
encodeURIComponent('hello')
//"%E4%BD%A0%E5%A5%BD"
decodeURIComponent("%E4%BD%A0%E5%A5%BD")
/ / "hello"
Copy the code

The above two JavaScript functions encode and decode urIs.

fragment

Fragments are fragment identifiers, which are quick anchors. This anchor is used by the browser to retrieve the resource and jump directly to its indicated location, but the fragment identifier is only useful to the browser because it is not sent to the server for processing like a query parameter.

https://github.com/sudheerj/reactjs-interview-questions#table-of-contents
Copy the code

So the #table-of-contents here is the anchor point.

summary

A URI is a string that uniquely identifies a resource on a server, often called a URL.

A URI usually consists of scheme, Host: POST, PATH, and Query.

The URI to @ &? These special characters with specific usage and non-ASCII characters such as Chinese characters are encoded and escaped.

The HTTP status code

The response message consists of response header and response body data, and the response header consists of status lines and header fields.

Here is the structure of the status row

In the status line, the protocol version and Reason are not important. The most important thing is the status code. It is a decimal number representing the result of the server’s processing in code form. Its purpose is to express the state of HTTP data processing, and the client can change the state of processing according to the code.

The current RFC standard stipulates that the status code is 3-digit and can be divided into five categories in the range of 100-599, which are identified by a percentage sign:

1xx: indicates that the protocol processing is in the middle state and subsequent operations are required

2xx: Yes, the packet is received and processed correctly

3xx: redirection. The resource location changes and the client needs to resend the request

4xx: An error occurs on the client. The request packet is incorrect, and the server cannot process it

5xx: Server error. An internal error occurred while the server was processing the request

In HTTP, understanding and applying these status codes correctly is not the sole responsibility of the client or server, but the responsibility of both.

As the initiator of the request, the client needs to know whether the request is correctly processed and whether to send the request again through the status code after receiving the response packet. If an error occurs, the cause is the reason.

The server, as the receiver of the request, should also make good use of the status code. When processing the request, choose the most appropriate status code to reply to the client, inform the client of the result of processing, instruct the client to take the next action, especially when there is an error, try not to simply reply to 400, 500 such ambiguous status code.

Currently, RFC has 41 status codes, but the definition of status codes is open to automatic expansion. The following describes the common status codes.

1xx

Class 1XX status codes are prompts and are intermediate states for protocol processing, which are rarely used in practice. Occasionally we encounter ‘101 Switching Protocols’, which require clients to continue communication using other Protocols on top of HTTP.

2xx

2xx indicates that the server successfully receives and processes the request from the client.

200 OK: All is normal and the server returns the result.

204 No Content: The status is successful, but the response has No body data.

3xx

The 3xx class status code indicates that the resource requested by the client has changed, and the client must re-send the request to obtain the resource with the new URI, i.e., redirect.

• 301 Moved Permanently: Permanently redirects, which means that the requested resource no longer exists and needs to be accessed by a new URI.

• 302 Found: Temporary redirection, meaning the requested resource is still there but needs to be accessed by another URI for the time being.

Both 301 and 302 use the Location field after the response header to indicate the URI to jump to. The end result is similar, but the semantics are very different.

For example, if HTTP is upgraded to HTTPS and the original HTTP is no longer used, a permanent redirect to 301 is required.

When the server is upgraded and the service is temporarily unavailable, you can configure 302 temporary redirection. When the browser sees 302, it will know that the situation is temporary and will not do cache optimization. The next day, it will still access the original address.

• 304 Not Modifield: This status code is used for cache control. It does Not have jump meaning and can be interpreted as redirecting to a cached file. In the HTTP comparison cache, if the server has not modified the resource, the browser accesses the cached file and returns this status code.

4xx

The 4XX status code indicates that the request packet sent by the client is incorrect and the server cannot process it. This is the true meaning of the error code.

• 400 Bad Request: Common error code, which indicates that the Request is wrong. However, the error is not specified. Generally speaking, it is better to use other status codes with clearer meanings

• 403 Forvidden: Indicates that the server forbids access to the resource, not because the client fails to request the resource. The request may have no permission or no login.

• 404 Not Found: Indicates that the requested resource is Not Found on the server, but the server is abusing the status code. If the server is Not happy, it will return 404.

• 414 request-uri Too Long: The requested URL is Too Long.

5xx

5XX status code Indicates that the client requests correctly, but the server fails to return response data due to an internal error.

• 500 Internal Server Error: indicates a common Error code that indicates that the Server is faulty. This process is actually good for the server because it prevents hackers from prying or analyzing it.

• Implemented for client requests is Not yet supported, so stay tuned.

• 502 Bad Geteway: Error code returned when the server acts as a gateway or proxy, indicating that the server is normal and an error occurs when accessing the back-end server.

• 503 Service Unavailable: The server is busy and cannot respond to services temporarily. 503 is temporary and may become inactive After a period of time. Therefore, a retry-after field is displayed in 503 response messages, indicating that you will be ready to try again.

summary

The status line represents the result of the server’s processing of the request in the response message

The cause phrase after the status code is a simple text description that can be customized

The status code is a three-digit decimal number divided into five categories from 100 to 599

1xx: not commonly used, indicating that subsequent operations are required

2xx: The request is successfully processed. Common values are 200 and 204

3xx: request redirection. Common uses are 301, 302, and 304

4xx: indicates a client error. The common values are 400, 403, 404, or 414

5xx: server error, common 500, 501, 502, 503

The characteristics of HTTP

Now that I’ve covered the basics of HTTP, here’s a summary of the features of HTTP in terms of the basics

Flexible and extensible

HTTP is a flexible and extensible transport protocol.

When HTTP was first born, it only stipulated the basic format of the packet in an open attitude, such as the request line, space, line feed, header+body, etc., and the components of the packet were not strictly specified, which could be customized by the developer.

With the growth of the Internet, HTTP protocol gradually adds methods, version numbers, status codes, header fields and so on. Body is no longer limited to TXT or HTML in the form of text, but also adds arbitrary data such as pictures, audio and video, which also comes from its flexible and extensible characteristics.

RFC documents, however, can be more understood as the recognition and standardization of existing extensions, realizing a virtuous cycle from practice to practice.

Reliable transport

HTTP is a reliable transport protocol.

This is because HTTP is based on TCP/IP protocol, and TCP protocol itself is a reliable, complete protocol, so HTTP inherits this feature, can achieve reliable transmission between the requestor and the responder.

It works in much the same way as TCP/IP, wrapping the actual transmitted data, adding a header, and then sending or receiving the data through the TCP/IP stack.

We must correctly understand the meaning of reliable, reliable transmission means that under the normal network environment, the sending and receiving of information must be successful.

HTTP does not guarantee 100% transmission from one side to the other, reliability only provides a “promise” to the user that the data will be delivered “as far as possible” underneath.

If the network environment is very bad, such as poor connection environment or busy network, transmission may fail.

Application layer protocol

HTTP is an application-layer protocol.

In TCP/IP after the birth of more than ten years, there are a lot of application layer protocols, such as SSH protocol responsible for remote login, FTP protocol responsible for file transfer and so on, but they are known to focus on a very small application field, in the general data transmission field can not play at all.

HTTP with the packet structure that can carry any header field and entity data, as well as connection control, cache proxy and other features, as long as the performance is not too demanding, HTTP can deliver almost everything, meet various needs, is called universal protocol.

Request-reply

HTTP uses the request-response communication mode.

In layman’s terms, this pattern is called round and round, like a function call, where you fill in the header field and receive a response.

Request-response mode also makes clear the positioning of both sides in the HTTP protocol. The requestor always sends the request first and is active, while the responder can only reply after receiving the request. It is passive, if there is no request, there will be no action.

In browser-server, the browser is the requester and the server is the responder. However, if the server connects to the back-end server as an intermediate proxy, it may also play a request-reply role.

The traditional C/S system architecture is Client/Server, that is, Client/Server. In the HTTP protocol mode, the requester is the Client and the responder is the Server. This response mode is very consistent with the traditional architecture.

With the development of the Internet, the Browser/Server (B/S) architecture also emerged, which uses lightweight browsers as client applications to achieve thin client, while the Server abandons the private protocol and adopts the general HTTP protocol.

In addition, the request-response mode also fully conforms to the working mode of Remote Procedure Call (RPC), which can encapsulate HTTP requests into Remote function calls, leading to the emergence of WebService.

stateless

The HTTP protocol is stateless.

What is statelessness? Status is actually some data or marks saved by the client or server to record changes in the communication process.

As a comparison, TCP is a stateful protocol. It is in the CLOSED state at the beginning, ESTABLISHED after a successful connection, fin-wait after a disconnection, and finally CLOSED.

These states need to be maintained by some data structure within TCP. Simply put, it is a marker quantity, marking the current state. For example, 0 indicates CLOSED, 2 indicates ESTABLISHED, and so on.

Statelessness can be described figuratively as having no memory. For example, if the browser sends a request and carries its own identity Token, such as Token, the server will check the permissions and send the data back. After a while, the browser sends another request. Since the server does not record the request status, it does not know that the request is from the same browser.

In the HTTP protocol, there are no specified states, and the client and server are always in a state of ignorance. The two are not aware of the connection before the connection, and the receiving and sending messages are independent without any connection. The receiving and sending messages will not affect the server or browser, and no information will be saved after the connection.

UDP is also a stateless protocol. However, UDP is also connectionless. Packets are sent in order and received in disorder. HTTP has connections, sends and receives packets sequentially, and manages packets according to the order in which they are sent and received.

However, because HTTP is also a flexible and extensible protocol, although there is no state specified in the standard, it is possible to add this feature in a patch under the framework of the protocol.

Other features

The entity data transmitted through HTTP can be cached and compressed, can be segmented to obtain, support for identity authentication, international language and so on are the characteristics of HTTP.

summary

HTTP is flexible and extensible, and you can add any header field to achieve a variety of functions

HTTP is a reliable transport protocol based on TCP/IP to ensure complete delivery of data

HTTP is a universal application layer protocol. It is more versatile than FTP and SSH and can transmit arbitrary data

HTTP uses the request-reply mode, in which the client actively sends a request and the server passively replies to the request

HTTP is stateless in nature. Stateless means that it has no memory. The protocol does not require the recording of connection information, and each request is independent of the browser and server.

The Cookie mechanism

HTTP is stateless, meaning it has no memory. The good news is that HTTP can be extended, and with extension, it has memory. Here’s Cookie technology

What is a Cookie

Cookie is equivalent to a small piece of paper the server gives to the browser, which writes data that only the server can understand. The client needs to send this information to the server. When the server sees this piece of paper, it can recognize the client who sends the request.

Cookie transmission process

Two HTTP header fields are required: response header set-cookie and request header Cookie.

1. When the user accesses the server for the first time, the server must not know its identity, so the server first creates a key=value format of identity identification data, which needs to use set-cookie, and then sends the key-value data to the browser through the response header field.

HTTP / 1.0 200 OK
Content-type: text/html
Set-Cookie: theme=light
Set-Cookie: sessionToken=abc123; Expires=Wed, 09 Jun 2020 10:18:14 GMT
...
Copy the code

2. After receiving the response packet, the browser finds the set-cookie field in the response packet and saves it. The browser automatically adds the value to the Cookie header field and sends it to the server in the next request.

GET /spec.html HTTP / 1.1
Host: www.example.org
Cookie: theme=light; SessionToken = abc123...Copy the code

3. When the server sees the Cookie field in the request field, it will recognize the browser that has visited before and provide personalized service after identifying the browser’s identity.

As we said earlier, most HTTP header fields are not repeatable, with the exception of set-cookies, which can be Set to multiple fields at once. This allows you to store multiple key=value values. When the browser sends it, it only uses one Cookie field, but it uses; Separated.

Since cookies are saved by the browser, you have to go through the Cookie process all over again.

The attribute of the Cookie

Cookies are data that the server entrusts the browser to store on the client, and these data will identify the key information of the user, so some other attributes besides key=value need to be added to protect them.

The life cycle

The Cookie’s life cycle is colloquially known as an expiration date, and there are two attributes you can use here: max-age and Expires.

Max-age is the relative time, which is the maximum time in seconds. It is the time when the browser receives the response packet after the server receives the Cookie and returns it, and the max-age time is the validity time of the Cookie.

Expires is an absolute time, which translates as an expiration date.

They’re written like this

Set-Cookie:Max-Age=100; Expires=Fri,07-Jun-19 08:19:00 GMT;Copy the code

This means the expiration time is 100 seconds, and the deadline is 8:19, June 7, 2019, Friday.

Both can exist at the same time, and the browser preferentially selects max-age

scope

The scope is to tell the browser not to send cookies, but to a specific server and URI.

Set-Cookie:Domain=www.baidu.com; path=/;Copy the code

This setting is relatively simple. When the browser carries cookies, it compares the Domain name with the path part. If the criteria are not met, the Cookie is not sent in the request header.

You can use these two attributes to send cookies in different paths, such as /index and /users, but generally to save trouble, a /in the path part of the Cookie to indicate that the root directory.

security

In JavaScript, there is a method that reads cookies, document.cookie, which can be a security risk and cause XSS attacks (cross-site scripting attacks).

You can use the HttpOnly attribute to tell the browser that Javascript access is not allowed.

Another property that prevents XSRF attacks (cross-site request forgeries) is SameSite=Strict, which allows cookies to not be sent across sites following jump links. SameSite=Lax allows GET/HEAD to send cookies, However, POST is not allowed.

How to view the Cookie of a browser

The Cookie itself is not encrypted and can be viewed in a browser.

Cookies can be seen through both network-cookies and Application browsers. Application can see all cookies of the site.

The application of the Cookie

The basic function of Cookie is identity identification, saving user login information and realizing session transaction.

For example, if you log in to an e-commerce company with your account and password, the website server will send the browser a Cookie with the content of “Name = Yourid”, which successfully attaches your identity tag to you.

After you casually in a site visit which item page, the browser will automatically send cookies to the server, the server will always know your identity, dispense with the trouble that repeat login on one hand, on the other hand also can automatically record your browsing and shopping order (database or also use cookies) in the background, implement the “state”.

Another common use of cookies is AD tracking.

When you’re on the Internet, you’ll see a lot of advertising images behind advertising sites (like Google),

It secretly attaches a Cookie to you so that when you go to other sites, other ads can use the Cookie to read your identity and then do behavioral analysis and send you ads.

summary

Cookie is the server entrusts the browser to store some data, so that the server has the “memory ability”;

The response packet uses the set-cookie field to send the Cookie value in the form of key=value.

Cookie field in the request message to send multiple Cookie values;

In order to protect cookies, it is necessary to set the validity period, scope and other attributes, commonly used are max-age, Expires, Domain, HttpOnly, etc.

The most basic use of cookies is identity recognition and the realization of stateful session transactions.

Content negotiation of HTTP Body data

Data type and encoding compression

In the TCP/IP stack, data is transmitted basically in the format of Header+body, and they add their own headers during transmission. They don’t care what the body data is, as long as the data is sent out.

HTTP is not like that. It is an application-layer protocol where the work is only half done once the data arrives. It must also tell the upper-layer application what data it is.

Suppose HTTP does not have the ability to tell the data type, and the server sends the data to the browser, what should the browser do?

It can be guesswork. A lot of the data is in a fixed format, so checking the first few bytes of the data might tell you that it’s a GIF or an MP3 file, but that’s inefficient.

A solution to this problem existed long before HTTP was invented: MIME (Multipurpose Internet Mail Extension) for E-mail systems, which allowed E-mail to send data in addition to ASCII.

HTTP borrows part of the data type used to mark up the body, which is often referred to as MIME Type

MIME divides the data into eight classes, each subclass in the form of type/subtype strings, which fit neatly into HTTP header fields.

Common categories:

• Text: readable data in text format. We usually use text/ HTML for hypertext documents. There are also text/plain and Text/CSS.

• Image: an image file, such as image/ GIF, image/ JPEG, and image/ PNG.

• Audio/Video: Audio and video data, such as Audio/MPEG and video/ MP4.

• Application: Data format is not fixed and is interpreted by upper-layer applications. Common examples are Application/JSON, Application /JavaScript, application/ PDF, etc. If the implementation does not know what the data is, it will be application/ OCtet-stream, which is opaque binary data.

Encoding Type Is an Encoding type that tells the Encoding of the data, so that the original data can be unzipped correctly.

Encoding Types are much less common than MIME types, and only three are commonly used:

1. Gzip: GNU Zip compression format, also a popular compression format on the Internet.

2. Deflate: Zlib (Deflate) compression format, second only to Gzip in popularity

Br: New compression algorithm optimized for HTTP.

The header field used by the data type

With Encoding Type and MIME Type, both the server and the browser can know the type of the body data, so can there be a field that allows both parties to negotiate with each other to pass the desired data?

There is. HTTP defines two Accept request header fields and two Content entity header fields for “Content negotiation” between the client and server.

Accept means that the browser tells the server what kind of data it wants to receive, and Content means that the server tells the browser what type of data it wants to receive.

The header field of the request

POST /api/base/facade/file/file? Upload HTTP/1.1 Accept: */* Accept-encoding: gzip, deflate, brCopy the code

The header field of the response

HTTP/1.1 200 OK Content-type: Application /json; charset=UTF-8 Content-Encoding: gzipCopy the code

The Accept field specifies MIME types that the client understands, and the */* above means that all are understood. Use between multiple formats, split.

Accept:text/html,application/json
Copy the code

The Content-type field indicates the Type of entity data specified by the server. This makes it easier for the browser to manipulate the data type, such as text/ HTML, to render it.

Accept-encoding and Content-encoding are the supported decompression format of the browser and the actual compression format of the server, respectively.

This process can be compared to sending and receiving an express, for example, I send an express to my friend and tell him that I want to get something of XXX type (Accept), but I know that my friend will definitely pack it and send it to me, so I tell him that I support the accept-encoding. My friend received it and returned it to me with a content-type and told me to unwrap it in a content-encoding way.

Language type and encoding

MIME Type and Encoding Type solve the problem of computers understanding body data, so how to solve the problem of national languages?

HTTP introduced language types and character sets.

The so-called language type is the Chinese, English, Japanese, etc., which may also have subordinate local languages, so it also adopts the form of type-subtype. The only difference is that the language type is separated by -.

For example, en stands for any English, en-us for American English, en-GB for British English, and we use zh-cn for Chinese.

Why was the character set created? In the early days of computer, many character sets in language environments appeared, such as ASCII code in English, GBK in Chinese, etc. In order to achieve the unification of characters, Unicode was born. Later, with the popularization of the Internet, UTF-8, one of the implementations of Unicode, became widely popular. Utf-8 is now the standard character set on the Internet.

The header field used by the language type

HTTP also supports Accept and Content entity header fields of the language type.

The browser header field is Accept-language, which allows splitting.

Accept-Language:zh-CN,zh,en
Copy the code

The header field above means give me zh-CN Chinese characters, if not, give me any Chinese, if not, give me English.

Accept-charset is used when the request is short.

Accept-Charset:utf-8,gbk
Copy the code

The server uses a header field called Content-Language, which means the server tells the browser what type of text I’m passing.

Content-Language:zh-CN
Copy the code

The character set returned by the server is placed after the Content-Type field, denoted by ‘charset= XXX’.

Content-Type:text/html; charset=utf-8Copy the code

Today’s browsers are too powerful to send accept-Charset because it supports multiple character sets, and the server does not send Content-language, but simply tells the browser what Charset to use.

summary

The data type indicates what the Content of the entity data is, using MIME Type, and the associated header fields are Accept and Content-Type

The data Encoding represents how the entity data is compressed, and the related fields are accept-Encoding and Content-Encoding

The Language type represents the Language of the entity data, with the relevant fields being Accept-language and Content-Language

Character set is the encoding of entity data. The related fields are Accept-Charset and Content-Type

Method of transferring large files

Data compression

We’ve already seen that when HTTP transmits data, there is an Accept-encoding field, which represents the client-supported compression format, so the server can select one of them, put it in content-Encoding, and send the original compression to the browser.

If the compression rate is 50%, the original 100K data can be compressed into 50K, greatly improving the transmission efficiency.

This compression is great for text such as text/ HTML, but not for transferring images, audio and video, which are already highly compressed.

Block transportation

In addition to data compression, you can make large files smaller and break them into smaller pieces that can be distributed to browsers for restoration.

This method has the advantage of only sending and receiving a small number of files at a time, the network will not be occupied by large files for a long time, which can save bandwidth resources.

This method is called chunked in HTTP and is represented in the response message by the header field Transfer-Encoding:chunked. This means that the body of a message can be sent multiple times.

Chunked transfer can be used for streaming data, such as dynamically generated form pages in a database, where the Length of the body data is unknown and the exact Length cannot be given in the content-Length header field, so chunked sending is also possible.

Transfer-encoding: chunked and Content Length are mutually exclusive, that is, the two fields cannot appear at the same time in a response message. The transmission Length of a response message is either known or chunked.

Scope of the request

Chunking is dividing a large file into many smaller pieces, so if I want to get a particular piece of data from a large file, obviously I can’t do that with chunking.

HTTP also has the concept of range requests, which allows clients to use specialized fields in request headers to indicate that only a portion of a file is being retrieved.

Before making a range request, the Web server needs to explicitly inform the client that range requests are supported by using the accept-Ranges :bytes field in the response header. This field can be left unpassed or set to None if not supported

The request header Range is a special field for HTTP Range requests in the format of bytes=x-y, where x and y are data ranges in bytes.

Note that x and y represent offsets that range from 0, for example, 0 to 9 for the first 10 bytes.

Where x and y can be omitted, 0- indicates the start and end of the document, -1 indicates the last byte of the document, and -10 indicates the countdown of 10 bytes from the end of the document.

Once the server receives the Range field, it needs to do four things

• Check whether the scope is valid. Invalid return 416 code.

• If the Range is correct, the server calculates the offset from the Range header, reads the fragment of the file, and returns 206 Partial Content indicating that the body is only part of the data

• The server needs to add the content-range response header, which tells the actual offset of the fragment and the total size of the resource in the format of bytes x-y/ Length, which differs from the Range field in the request header with no = sign and the total length after the Range.

• Finally sent the data

The common applications of scope request are video drag-and-drop progress, multi-segment download, breakpoint continuation, etc.

Many piece of data

Range requests also support fetching more than one piece of data at a time, and can use more than one X-Y in a Range.

This requires a special MIME type: multipart/byteranges, indicating that the body of the message is composed of a sequence of multiple bytes, and a parameter boundary= XXX is given to indicate the division between the segments.

summary

Compressing files such as HTML is the basic method for transferring large files

Block transmission can send and receive data streaming, saving memory and bandwidth. Use the response header transfer-encoding :chunked

A range request can fetch only part of the data. It is mainly used for breakpoint continuation and video drag and drop. Use request header fields Range and response header fields Content-range. Correlation response status codes 416 and 206

May also request a one-time multiple range, and the response message is the content-type multipart/byteranges, body section parts will use a boundary string.

HTTP cache control

Cache (cache) is to optimize the performance of system, because the link length, network delay is not controllable, through HTTP access to network resources cost is higher, so the browser USES a caching mechanism, will get the data cached, next time again request, reuse as much as possible, this is clear: the benefits of fast response speed, save network bandwidth.

Cache processes

The overall browser cache looks like this:

• First the browser discovers and sends a request to fetch the server resource
• The server responds to the request, returns the resource, and marks the expiration date of the resource
• The browser caches resources for reuse

Mandatory cache

The header field that the server uses to mark the expiration date of a resource is cache-control, and a common attribute is max-age=xx, which means the maximum duration in seconds. This property is similar to the Cookie section, except that the header field is different.

The Cookie max-age is computed at the time the browser receives the Cookie, whereas the cache-control max-age is computed at the time the server sends the response header field.

There are other attributes such as:

• No_store: cache is not allowed
• No_cache: Can be cached, but before using it, the server must check whether it has expired and the latest version
• Must -revalidate: this means that the cache can be used once it has expired, but must be verified by the server.

Client cache control

After the server sends the cache command, the browser saves the cached data locally and reads the server resources in the local cache when needed, improving network efficiency.

Not only the browser can issue the cache-control header, but the browser can also issue the cache-control header, which means that both request-responder parties can use the cache-Control field for Cache Control.

When we hit the refresh button, the browser adds cache-Control :max-age=0 to the request header, which means I want the latest resource. Instead of reading the browser cache, a request is sent to the server.

Here is the header I sent when I hit Refresh in Chrome, with cache-Control :no-cache, which has the same effect as max-age.

When does the browser cache take effect? When we click on the forward and back buttons, we seefrom disk cache “, which triggers the browser cache.

Because of the forward, backward, and other operations, the browser only adds the most basic headers and does not add fields like cache-control, triggering local caching.

Negotiate the cache

Using cache-control to Control the Cache can only refresh the data, but does not make good use of the cached data, so there is a concept of negotiated Cache, also known as conditional request.

It uses two consecutive requests to form a validation action, and it works like this:

Step 1: Send a HEAD request to get the modification time or other meta information of the resource, and compare it with the cached data. If it hasn’t changed, use the cached data. If it has changed, go to step 2.

Step 2: Send a GET request to GET the latest version.

The HTTP protocol also defines a conditional request field starting with If for this purpose. The server provides last-Modified and ETag for the first response, and the second request verifies that the resource is up to date with the original value in the cache.

If the resource has Not changed, the server responds with a 304 Not Modified status code indicating that the cache is still valid, and the browser can update the expiration date and use the cache.

Last-modified means when a file was Last modified.

ETag is a unique resource identifier, which is used to solve the problem that file changes cannot be accurately identified by modification time.

For example, if a file is updated regularly and nothing changes, last-Modified means that it has changed and wastes bandwidth by passing it to the browser.

Using ETag accurately identifies resource changes, allowing the browser to make efficient use of the cache.

Last-modified is if-modified-since, and ETag is if-none-match.

When the server sends last-Modified to the browser for the first time, the browser caches the resource, and then the next time the request is made, puts if-Modified-since on the HEAD of the request. Tell the browser that my files are older than that, and the browser loads the resource using cache.

ETag is the same, except that the request header field is if-none-match instead.

summary

Caching is an important means to optimize system performance, and can be used in every link of HTTP transmission

The server uses cache-control to set the Cache policy. Max-age is commonly used to indicate the validity period of resources

The browser receives the data and stores it in the cache. If it doesn’t expire, it can be used directly. After expiration, the server must verify that it is still available

To verify that a resource is invalid, a conditional request is used, usually if-modified-since and if-none-match

Two conditions, ETag and last-Modified, are involved in verifying whether a resource has been Modified. The server must set ETag and last-Modified in the response message in advance and use them with the condition request

The browser can also send cache-control, such as max-age=0 for a refresh operation

A central tenet of browser caching is that a request without a request is the fastest request

HTTP proxy

HTTP adopts the request-response mode, which means that at least two parties will negotiate with each other. As mentioned in the previous chapter, we can also include various proxies in the middle of HTTP, equivalent to intermediaries, which is HTTP proxy.

In this hierarchy, the starting point is the browser, the role in the middle is called the proxy server, and the destination is the source server.

The proxy service does not generate any data content of its own, but merely acts as an intermediary to forward requests and responses upstream and downstream, that is, it can be either the requester or the responder of information.

The role of agency

Why add a layer of proxies in a simple request-reply mode, and what does it do?

The most basic function of proxy is load balancing. The so-called load balancing means that the proxy server controls request distribution and sends different request traffic to different source servers as far as possible to avoid excessive pressure on source servers and improve overall resource utilization.

In addition to load balancing, the intermediate agent can also do more functions:

• Security protection: Protects against network attacks or overload
• Encryption: Encrypts communication through SSL or TLS
• Data filtering: Intercepts upstream and downstream data
• Content cache: The response of the reuse server is temporarily stored

Proxy-related header fields

The proxy server identifies the proxy using the field Via.

Via is a generic field that can be used in response or request. Each time a message passes through a proxy node, the proxy server appends its own information to the end of the field.

If there are many agents in the communication link, a linked list is formed in the Via so that it is possible to know exactly how many steps a message has traveled before reaching its destination.

There are two agents in the figure:proxy1 和 proxy2, the client sends requests through these two proxies, which are added one by oneVia: proxy1, proxy2When the server returns a response, the header field isVia: proxy2, proxy1 。

The Via field only solves the problem of the client and the source server determining the existence of a proxy, and does not know the real information about the other side.

However, the IP address of the server is confidential because it is related to the Intranet security of the enterprise. Therefore, the IP address of the server is not known to the client. However, the server usually needs to know the real IP address of the client for access control, user portrait, and statistical analysis.

Unfortunately, the HTTP standard doesn’t define header fields For this purpose, but there are many de facto standards. The two most common header fields are X-Forwarded-For and X-real-IP.

• X-forwarded-for: This item is served by a chain

Each time it passes through a proxy node, a message is appended to the field, but Via adds the proxy host name (or domain name) to it. X-forwarded-for adds the IP address of the requestor. Therefore, the IP address at the left of the field is the address of the client.

• X-real-ip: indicates only the client IP address

Is another method to obtain the real IP address of the client. It simply records the IP address of the client without intermediate proxy information.

If there is only one proxy between the client and the source server, the values of the two fields are the same.

The agency agreement

With header fields like X-Forwarded-For, the source server gets the exact client information. But it is not an optimal solution for proxy servers.

X-forwarded-for forwarders have to parse HTTP headers. This is costly For forwarders. Originally, they only forward messages.

Another problem is that headers such as X-Forwarded-for must modify the original message, which in some cases is not allowed or even possible (For example, if the communication is encrypted using HTTPS).

As a result, The PROXY Protocol, defined by The well-known PROXY software HAProxy, is a de facto standard that is widely adopted (note that it is not RFC).

The proxy protocol has two versions, v1 and V2. V1 is similar to HTTP and is also plaintext, while V2 is binary. Today, I’ll just introduce v1, which is easier to understand. V1 adds a line of ASCII text to the HTTP message, which is equivalent to adding another header.

This line of text is very simple. It must start with five uppercase letters PROXY, then TCP4 or TCP6, representing the IP address type of the client, followed by the address of the requester, the address of the responder, the port number of the requester, and the port number of the responder, and then end with a carriage return (\r\n).

In the following example, the PROXY information line is added before the GET request line. The real IP address of the client is 1.1.1.1, the IP address of the responder is 2.2.2.2, and the port number is 55555.

PROXY TCP4 1.1.1.1 2.2.2.2 55555 80\r\n GET/HTTP/1.1\r\nHost: www.xxx.com\r\n
\r\n
Copy the code

When the server sees such a packet, it can get the address of the client as long as it parses the first line. It does not need to pay attention to the FOLLOWING HTTP data, saving a lot of things.

However, the forwarded-for IP address is not supported by the forwarded-For IP address. Therefore, the forwarded-for IP address must be forwarded-to the forwarded-for IP address.

summary

• An HTTP proxy is an intermediate link in the communication link between a client and a server, providing proxy services for both ends
• The proxy is in the middle layer, which adds more flexibility to HTTP processing and can realize load balancing, security protection, data filtering and other functions
• Proxy servers need to identify themselves with the Via field, and multiple proxies form a list
• If you want to know the Real IP address of the client, you can use the fields X-Forwarded-for and X-real-IP
• A special proxy protocol can transmit the real IP address of the client without changing the original packet

The latter

At this point, the basic and advanced knowledge of HTTP has been written. If you have not read enough, we recommend that you take a look at “Perspective HTTP protocol” and “Diagram HTTP” for further reading.

If you think it is helpful to you, please click 👍🏻, which is a great encouragement to me. Meanwhile, I may update the chapter later, and it will be posted on Github at the first time.

If you find the blog longer and would like to read it one by one, you can also check out my Github, star 🌟🌟

Github address: My Github