0, preface

As a starting point for the networking section, start with HTTP!

Full text is very long, suggest collection, prevent getting lost! Read one section at a time.

1. HTTP Overview

HyperText Transfer Protocol (HTTP) is an application-layer Protocol used to Transfer hypermedia documents (SUCH as HTML and images). The original purpose of HTTP is to publish and receive HTML documents. With the rich content of web pages, HTTP messages also support binary content such as images and audio, becoming a protocol for transferring computer resources.

Up to now, the main versions of HTTP are HTTP/0.9, HTTP/1.0, HTTP/1.1 and HTTP/2. A series of RFCS have been published, the most famous of which is RFC 2616, which standardizes HTTP/1.1.

As a transport protocol, HTTP has many characteristics:

  • Application layer protocol. This relates to the concept of a hierarchical model of computer networks, in which HTTP is the topmost application-layer protocol.

  • Client-server model. Also known as C/S model, HTTP communication process is a request-response model, the host that initiates the request is the client, and the host that responds to the request is called the server.

  • TCP/IP channels are usually used. In fact, HTTP can be built on top of any reliable transmission channel.

  • The default port is 80.

  • Stateless connection. HTTP is a connection protocol based on TCP (most of the time). In HTTP/0.9 and HTTP/1.0, a new connection is established during each HTTP communication, and the connection is disconnected after each communication is completed. This connection is called a non-continuous connection (short connection). Because of the high cost of establishing and disconnecting a connection, HTTP/1.1 and HTTP/2 use persistent connections by default, allowing multiple communications between the same client and server to take place over the same HTTP connection.

  • Plaintext protocol. This is no longer true for HTTP/2, but prior to that, HTTP was a text-based plaintext protocol.

  • Simple and flexible. HTTP is designed to be very simple and extensible, making it very useful. And very flexible, can transfer any type of file.

A typical HTTP communication process:

  1. The client makes an HTTP request through a browser application for the document it wants to retrieve.
  2. The request is processed through TCP/IP, through one intermediate proxy after another, and finally reaches the server host.
  3. The server host receives the request, processes it, checks out the requested document or data, puts it in the response, and sends the response to the client.
  4. The response content is processed by the TCP/IP layer and sent back to the server through intermediate proxies.
  5. The server receives the response and sends it to a browser, which renders the document.

Ignoring the routers in the network, the main components of an HTTP communication system are:

  • Client: the host that initiates the request.
  • User AgentThe program that initiates a request in a client, usually a browser, HTTP is the most commonB/S model. But it could also be any program that initiates a request, a command line, a crawler.
  • Server: a host that receives a request and sends a response.

2, URL

A URL is used in HTTP to identify the requested resource.

URL (Uniform Resource Location) is a string of characters that uniquely identifies a computer Resource and specifies the Uniform Resource Location protocol.

2.1. URL and URI

Before I go to URL, I have to say URI, and I often hear these two words used interchangeably, someone says URL, and someone jumps up and says isn’t it URI?

Well, it’s both.

Uniform Resource Identifier (URI) A Uniform Resource Identifier (URI) uniquely identifies resources on a computer network.

????? Is this different from the previous URL? B: Yes.

Uris are used to uniquely identify computer resources. The point is that there are more than one way to uniquely identify computer resources. Urls are one of them, and the most common.

If we assume that an address only houses one person, without considering the same name, a URL is like the person’s address and URN is like the person’s name. Both of these uniquely identify a person, and both serve as identifiers of a person, but the address tells you where to find the person, and the name doesn’t. Again, the URL specifies how to get the resource.

So you see, when we say URI, it could be a URL, and it usually is, so a lot of times, they’re interchangeable, unless you’re talking about urNs.

Example is URN URN: isbn: 0451450523, or URN: mpeg: mpeg7: schema: 2001, regardless of their naming rules, at ordinary times is rarely used.

So, we’re basically talking about urls, and since a URL is a URI, both urls and URIs are true in this case. It’s like a positive integer, somebody calls it a positive integer, somebody calls it an integer, it’s all right. Urls are more accurate, but URIs are more fault-tolerant. There’s no need to argue about who’s right, but it’s worth understanding the difference. The URL term is used below, although the standard uses urIs.

2.2. URL Format

The full format of a URL is as follows:

Type [agreement] : / / [confidential information] @ [server address] : [port] / [resource hierarchy UNIX file path] [name]? [query]#[fragment ID]Copy the code

[Access credential information], [port number], [query] and [fragment ID] are all optional items, so the general format we see is:

[Protocol type]://[server address]/[resource-level UNIX file path][file name]? [query]Copy the code

The default port number is 80. If other ports are used, the port number cannot be omitted.

Query using key1=value1&key2=value2 format.

Such as URL:https://zh.wikipedia.org: 443 / w/index. PHP? Page = 2 & title = Special: random page, can be obtained:

  • Protocol:https.
  • Host:zh.wikipedia.org.
  • Port:443.
  • File path:/w/index.php.
  • Query:Page =2&title=Special: random page, there are two query parameters.

3. Format of HTTP packets

HTTP has only two types of packets (messages) : request packets and response packets.

HTTP is a plaintext packet, which is easy to understand. This is deliberately designed to be simple. Even if HTTP/2 is encapsulated with “binary frames”, the message can still be understood in the same way as before.

Generally speaking, the format of HTTP packets (request packets and response packets) is as follows: start line + several headers + Spaces + body part.

  • Start line: the first line of the packet<CR><LF>A newline.
  • Header: key-value pair information that controls the communication process and behavior. Use between heads<CR><LF>Line breaks, that is, a header line.
  • Empty lines:Must be theIs used to distinguish the header line from the body. There can only be<CR><LF>Cannot contain other characters.
  • Body: Also called entity data, the data part of the message. Optionally, GET requests often have no body.

Packets are classified into request packets and response packets. The specific format depends on the packet type.

3.1 request message

< request method > <URL> < protocol version > < header 1> < header 2> <.... > < body >Copy the code

The first line of a request packet is also called the request line. It contains the request method, URL, and protocol version, separated by Spaces.

  • Request methodHTTP supports nine request methods, the most common of which areGET.POST.PUT.DELETE, etc.case-sensitive.
  • URL: does not include protocol (must be HTTP), host, and port (by header line)hostGive) part, e.g./index.html.
  • Protocol version: Indicates the HTTP version number, for exampleHTTP / 1.1.

Several header lines, also known as request headers.

Specify that only the Host header is required, otherwise there is no way to know the requested Host. So, a simple request is as follows:

GET/HTTP/1.1 Host: www.google.comCopy the code

Notice the blank lines.

3.2 response message

< protocol version > < status code > < status phrase > < header 1> < header 2> <.... > < body >Copy the code

The first line of the response packet is called the status line, which indicates the status information of the response. It consists of the protocol version, status code, and status phrase.

  • Version of the agreement: is the same as the request message, but is the first position in the response message.
  • Status code: INDICATES the HTTP response status code, indicating the processing status of the request200.404.500.
  • State of the phrase: a phrase used to describe the status of a response. Each status code has a default phrase that can be customized, but most use the default.

Header and entity data are similar to request messages.

Unlike request messages, the body of a response message is often used.

An example of a response message from Wikipedia:

HTTP/1.1 200 OK Content-Length: 3059 Server: GWS/2.0 Date: Sat, 11 Jan 2003 02:44:04 GMT Content-Type: text/html Cache-control: private Set-Cookie: PREF=ID=73d4aef52e57bae9:TM=1042253044:LM=1042253044:S=SMCc_HRPCQiqyX9j; expires=Sun, 17-Jan-2038 19:14:07 GMT; path=/; domain=.google.com Connection: keep-alive <html>... HTML text content... <html>Copy the code

4. Request method

HTTP defines a set of request methods that represent operations on a given resource.

Before introducing methods, take a look at some of the features of the request method:

  • Secure: A request method is considered secure if it does not change the state of the server, that is, the resources of the server.
  • Idempotent: A request is considered idempotent if it is submitted many times and results in the same result.
  • Cacheable: A request method is considered cacheable if its response can be cached for use on the next request.

GET

Requests a representation of the specified resource. Requests using GET should only be used to GET data.

The GET method is secure, idempotent, and cacheable.

GET is one of the most commonly used methods. The client requests to obtain content from the server are basically GET requests. GET can also be used to submit simple information, but this is not a canonical use.

HEAD

The same as GET is used to GET resources, but only the header information is retrieved, that is, the response has no body compared to GET.

HEAD is also secure, idempotent, and cacheable.

HEAD is used to retrieve information about resources, provided that the information is present in the header. For example, you can save bandwidth by getting the header to check the size of the file before deciding whether to download it.

POST

Used to submit information to a specified resource, often resulting in state changes or side effects on the server.

The POST method is insecure, non-idempotent, and most of the time not cacheable.

POST is used to send information to the server, and the main usage scenario is HTML form submission.

PUT

Creates or replaces the specified resource with the submitted information.

The PUT method is insecure, idempotent, and not cacheable.

PUT is also used for information submission. The biggest difference between PUT and POST is that it is idempotent. For example, when an order is submitted, if the POST method is used to submit an order for several times, multiple orders will be generated. If the PUT method is used, only one order will be generated. In this case, PUT is more suitable.

DELETE

Deletes a specified resource.

The DELETE method is insecure, idempotent, and not cacheable.

The DELETE method should be used to DELETE resources on the server.

OPTIONS

Communication options used to get the support of the destination resource.

The OPTION method is secure, idempotent, and not cacheable.

OPTIONS is often used to pre-check requests, asking the server what OPTIONS it can support before deciding whether to make a formal request.

The latter request method is rarely used, so be aware:

CONNECT

Used to establish a communication tunnel to a specified third party server, with the server receiving the request as an intermediate proxy.

TRACE

A loopback test is initiated to the target server, and the HTTP intermediate proxy may modify the request. After receiving the TRACE request, the server will return a TRACE response that is the same as the request, so that it can know the request content that finally reaches the server, and the Via field records the proxy server that passes through the middle.

TACE provides a debugging mechanism, but it has security holes and the server does not enable the TACE method by default.

PATCH

PATCH is used to modify some resources by submitting patches to specific resources. Unlike the overwriting operation of PUT, PATCH is non-idempotent.

If you have doubts about the above methods, you can click on the corresponding MDN document to view the application scenarios and cases.

There are nine HTTP methods, and for the most part, we’ll just use GET, POST, PUT, and DELETE.

Some sites only use GET and POST methods to GET, submit, delete, and modify resources.

Not surprisingly, the HTTP request method is semantic, and the actual operation on the resource depends on how the server handles the request.

For an extreme example, there is an article resource named 3. We can:

GET /getPosts/3 / GET posts /3 /editPosts/ GET /deletePosts/3 /deletePosts/ addPosts/3 / replace postsCopy the code

It is perfectly fine, and in fact quite a few are designed to do so, for the server to handle requests and operate on the database based on the URL. Even a single request method can be used.

HTTP provides these methods and uses without requiring us to do so.

As a complement, there is a RESTful style specification that states:

  1. Each URL should represent a resource.
  2. Client useGET.POST.PUT.DELETEOperations on the resources on the server represent the acquisition, update, replacement and deletion of resources.

Using RESTful style, the above example would look like this:

GET /posts/3
POST /posts/3
DELETE /posts/3
PUT /posts/3
Copy the code

Even if we don’t know what the server is doing, we know what the request is doing, which is semantic.

RESTful is a style specification and not mandatory. However, it is recommended to use RESTful style as much as possible.

5. Status code

The status code identifies the response status of an HTTP request.

Responses fall into five categories:

  1. 1XX— Message response
  2. 2XX— Successful response
  3. 3XX– redirect
  4. 4XX— Client error
  5. 5XX— Server error

The meanings, phrases, and possible scenarios of status codes are as follows:

100 Continue

Meaning: So far the request is ok, the client can continue to send the request.

Scenario: The client wants to send a request with a large body, but the server may not accept it. In this case, the client first issues a request with an Expect:100-continue and Content-Length headers, and then sends the request body after receiving a response with 100 continue from the server.

101 Switching Protocol

Description: The server is switching protocols at the request of the client.

Scenario: The client requires to upgrade the application protocol. For example, with the WebSocket Protocol, the client sends an HTTP request asking if the server supports upgrading to the WebSocket Protocol, and if the server does, replies with the 101 Switching Protocol.

200 OK

Meaning: The response is OK

The scene: Everywhere

201 Created

Description: Succeeded and created. This is usually used in POST or PUT response messages to indicate that the response succeeded and a new resource was created. The address of the new resource is the value of the first line of Location or the REQUESTED URL.

Scenario: As above, a POST request returns a 201 Created response with the address of the new resource in the response Location header line.

202 Accepted

The request has been accepted, but has not been successfully processed, and the result is unknown.

Scenario: After the request is transferred to another application or server for processing, or batch processing is performed later, a 202 Accept message is sent to the client to inform the client that the request is accepted, but the processing result is unknown.

204 No Content

The request was successful, but there is no response content. The client is told that the request has been processed, but there is no response content, and the client does nothing.

Scenario: When a client clicks on a link, the default behavior is to send a page jump. The client can reply with a 204 No Content response telling the browser not to jump.

206 Partial Content

Meaning: partial response. The response packet contains only part of the data, which is used to transfer part of a large file.

Scenario: Return part of a large file using 206 Partial Content and specify the data range and total file size in the response header (see scope request section below).

300 Multiple Choice

Meaning: The requested resource has multiple possible responses, and the user or client agent needs to select one of them for redirection.

Scenario: Check out this page. This is rarely used.

301 Moved Permanently

Meaning: Permanent redirect. Indicates that the requested resource has been permanently moved to another URL, and the new URL is in the Location of the response.

Scenario: When the requested resource is moved to another URL, the server usually gives a 301 to tell the user agent to use the new URL later.

302 Found

Meaning: Temporary redirection. The resource is temporarily moved to a new URL, this time retrieved from the new URL (given by Location), but the next time the request is made to the old URL, this is the meaning highlighted by Found.

Scenario: Configured response after server resources are temporarily moved.

Added: Redirection process. A redirection request will go through two requests before getting the destination resource. After the first request is made, the server responds with a 3XX redirect response, giving the new URL in Location. If it is permanent (301 or 308), the browser saves the new URL and automatically uses the new URL every time the old URL is requested in the future, and the search engine updates accordingly. In the case of temporary redirects (302 and 307), the browser will not save the new URL and will request from the old URL again.

304 Not Modified

Description: Not modified. You can cache the requested content instead of transferring it.

Scenario: Occurs when caching is used. If the request is cached, the cache server issues a conditional GET request to the target server with an if-modified-since or if-none-match header, and the server queries whether the resource has changed Since the cache. If not, return a response with 304 (see the caching validation section).

400 Bad Request

Error request. The semantics of the request are wrong and the server cannot understand it. The client should not repeat the request.

Scenario: An incorrect request is sent. The syntax of the request is incorrect. The message is too large for the client to process.

403 Forbidden

Meaning: Denial of service. The server has the capability but rejects the client’s request, often because the client has limited permissions.

Scenario: The IP address is blacklisted by the website. Your account does not have enough permissions to view some pages. Rejection is often part of a website’s business logic.

404 Not Found

Description: Failed to find resources.

Scenario: Making a request to a non-existent URL.

500 Internal Server Error

Description: An internal exception occurs on the server. The server fails to process requests properly.

Scenario: An exception occurs when the server processes a request.

Solution: Debug and modify the website code.

501 Not Implemented

Description: Requested method is not supported. This is server-based, and some servers only support GET and POST requests.

Scenario: Send a DELETE request to a server that does not support a DELETE request.

Solution: Contact the server administrator.

502 Bad GetWay

Description: The gateway is incorrect. The gateway or proxy server received an invalid response from an upstream server such as Tomcat. The error is due to timeout.

Scenario: Too many requests arrive at the same time and the server cannot process them, resulting in a timeout.

Solution: Client users can try to refresh. The server improves server efficiency.

503 Service Unavailable

Description: The service is unavailable. Indicates that the server is down for maintenance or overloaded. Requests will not be accepted at this time.

Scenario: Shutdown for maintenance.

The most common status codes are 200, 204, 206, 301, 302, 400, 403, 404, 500, 501, 502, 503.

Do you understand what they mean and what happens? I was dizzy anyway.

HTTP header

HTTP headers, also known as HTTP headers, are optional parts of HTTP messages that are used to deliver additional messages. They are important for clients and servers to understand HTTP messages.

According to where headers can appear and their meaning, they can be divided into four types:

  • Gm’s first: headers that can appear in requests and responses and are independent of the message entity. Such asDate.Cache-Control.
  • Request header (request header): only appears at the head of the request, which has nothing to do with the physical content. Such asCookie.User-Agent.
  • Response header (response header): the part that only appears in the response and is independent of the entity. Such asAge.Location.
  • Entity first: can appear in requests and responses forDescription Message entity informationSuch headers may also appear in the entity part of the request (the body). Such asContent-Length.Content-Language.

Tip: It is also common to refer to headers in requests as request headers and headers in responses as response headers, regardless of whether they are relevant to the content.

Some headers that begin with X- are custom headers that are not specified in the standard.

Multiple headers are separated by a


newline, which means each header line, no matter how long, is also called the header line.

Each header is represented as a header name: the key-value pair form of the header value. The header name is case – insensitive with a hyphen. The form of the header value depends on the Content of the header. Some headers may be a number, such as Content-Length:2000. Some headers may be strings, such as Cookie:id=22. Option =3, and some use instructions with specific meanings, such as Connection:keep-alive.

Supplementary: On the directive. The value of the header line can consist of one or more directives. Use between instructions; Separated. I can’t find an explicit definition of the directive, but personally I understand it to be a set of specified strings representing a specified meaning, such as keep-alive,close, and a key-value pair with a specified key, such as max-age=60. Strings that allow any combination of valid characters are not directives. For example, Cookie: useID =347; Option =2, where userID and option are not the properties specified by the Cookie, but are set by the user himself. Instruction is just a name. Don’t worry too much about it.

Here is the first part of a request:

GET /home. HTML HTTP/1.1 Host: developer.mozilla.org User-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; The rv: 50.0) Gecko / 20100101 Firefox 50.0 / Accept: text/HTML, application/XHTML + XML, application/XML. Q = 0.9 * / *; Q = 0.8 Accept - Language: en - US, en. Q = 0.5 Accept - Encoding: gzip, deflate, br Referer: HTTPs://developer.mozilla.org/testpage.html Connection: keep-alive Upgrade-Insecure-Requests: 1 If-Modified-Since: Mon, 18 Jul 2016 02:36:04 GMT If-None-Match: "c561c68d0ba92bbeb8b0fff2a9199f722e3a621a" Cache-Control: max-age=0Copy the code

Also, an example of a response header:

200 OK
Access-Control-Allow-Origin: *
Connection: Keep-Alive
Content-Encoding: gzip
Content-Type: text/html; charset=utf-8
Date: Mon, 18 Jul 2016 16:06:00 GMT
Etag: "c561c68d0ba92bbeb8b0f612a9199f722e3a621a"
Keep-Alive: timeout=5, max=997
Last-Modified: Mon, 18 Jul 2016 02:36:04 GMT
Server: Apache
Set-Cookie: mykey=myvalue; expires=Mon, 17-Jul-2017 16:06:00 GMT; Max-Age=31449600; Path=/; secure
Transfer-Encoding: chunked
Vary: Cookie, Accept-Encoding
X-Backend-Server: developer2.webapp.scl3.mozilla.com
X-Cache-Info: not cacheable; meta data too large
X-kuma-revision: 1085259
x-frame-options: DENY
Copy the code

The above example comes from MDN.

You probably don’t know much about these heads by now. Let’s start with some of the simplest and most commonly used trainers.

Host — destination Host

Request header. Port number Specifies the destination host and port number of the request. The port number is optional. By default, the port number of the request service is used, for example, port 80 for HTTP URL and port 443 for HTTPS URL.

Host: developer.cdn.mozilla.net
Copy the code
Host: developer.cdn.mozilla.net:8080
Copy the code

The Host header is required for the Request, and the default or multiple hosts may result in 400 Bad Request responses.

Referer — Address of source

Request header. Represents the source address of the request, and if you click on a link to send the request, its value is the URL of your current page.

Referer: https://developer.mozilla.org/en-US/docs/Web/JavaScript
Copy the code

Hint: Referer is a misspelling of the word Refferer

User-agent — User Agent

Request header. Contains a character string that allows the network protocol peer to identify the application type, operating system, software developer, and version number of the user agent making the request.

The user-agent: Mozilla / 5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36Copy the code
The user-agent: Googlebot/(+) HTTP://www.google.com/bot.html / 2.1 / crawler robotCopy the code

.

There are many others, but, before understanding the relevant principles and mechanisms, it is impossible to understand, so interspersed in the following related content.

7. HTTP entity data

Now, let’s focus on the other part — the body, the entity data part.

7.1 MIME types

There are various types of files on the computer, such as.txt text files,.jpg picture files,.mp3 audio files. File classification enables the computer to operate the files according to their types, and open and process the files correctly.

Similarly, data transmitted over the Internet is divided into multiple types, making it easier for communicators to understand and process the data. However, files transferred over the Internet are classified according to a different standard than those on computers, using a classification system called THE MIME Type Standard, which stands for Internet Media Type, or media type.

Multipurpose Internet Mail Extensions are also used for MIME types, but they are not appropriate to use media types or MIME types.

Media class representation format:

Type name/type name [; Optional]Copy the code

The representation of a type is case insensitive and is often lowercase.

The type name can be one of the following six types, each of which has a corresponding multiple subtype:

  • Text – Human readable text type. Such as text/plain, text/ HTML, text/ CSS, text/javascript.

  • Image – Image type. Such as image/ GIF, image/ PNG, image/ JPEG, image/ BMP, image/webp, image/ X-icon, image/vnd.microsoft.icon.

  • Audio — Audio files. Such as Audio/MIDI, audio/ MPEG, audio/ webM, audio/ OGG, audio/wav

  • Video — Video files. Such as video/webm, video/ogg.

  • Application — Binary data. Such as application/json, application/octet stream, application/PDF

  • Multipart — Multipart data type. Such as multipart/form – data, multipart/byteranges

Note: Multipart is not a MIME type. It is an HTTP specific multipart data type, because it is used to describe entity data types, just like MIME types.

Here are some common formats:

Text /plain — Common text format

The default text format, which means that it is a text format file, but do not know the specific type, the browser can only interpret as text, can not be further identified as CSS, javascript and other types.

Application /json — Json format

A data format widely used to represent lightweight data, often used in Web development.

Application /octet-steam — Unknown application file

Default value for the application file. The receiver treats it as a binary file, does not know how to execute it, and generally saves it locally to the user.

Multipart /form-data — Form data format

HTML form submission data format.

Multipart/Byteranges – a specific range of multipart files

Indicates that the file consists of several parts, each of which is a scope.

Add: You can learn more about MIME types here.

7.2. Subject compression

In order to reduce the size of the message body, HTTP usually compresses the body content when transmitting a message, similar to the principle of file compression.

The main compression formats, also called encoding methods: **gzip, Deflate, BR,compress, will be used later.

To distinguish uncompressed entity content, use **identity** to indicate uncompressed data.

7.3 entity header

Now, we can look at some headers that have to do with entities. They generally appear as Content-. Both requests and responses can have entity headers.

  • Content-type :

    — The MIME Type of the entity, followed by an optional character set Type, such as UTF-8, BGK, ISO-8859-15.

    Content-Type: text/html; charset=utf-8
Copy the code

  • Content-length :< Length > — The number of bytes in the entity.

    Content-Length: 39749
Copy the code

  • Content-encoding :< compressed-type > — The Encoding type (compression mode) of the entity, as mentioned in the previous section.

    Content-Encoding: br
Copy the code

  • Content-language :< Language > — The natural Language of physical Content. Such as zh-CN, en, and en-us.

    Content-Language: de, en
Copy the code

The above are just a few generic and common ones, and there are many more entity headers.

7.4 Content negotiation mechanism

According to the RESTful specification, a URL should correspond to a computer resource, but a resource can be represented in many ways. For example: English and Chinese versions of the same page; The contents of the same page obtained on the mobile and desktop terminals are different. The server selects the version of the resource variant that is most appropriate for the current user agent.

This is the result of content negotiation. The browser sends a request with an Accept-* line that indicates what form of content it wants to receive. This hope is lax because the user agent does not know the form of resources on the server. After receiving the request, the server selects the appropriate form according to the Accept-* header and clearly indicates it in the response, telling the user agent that I finally sent this form of resource, so that the user agent can know the type of resource it finally received.

Accept-* is a list of request headers that only appear in a request, including:

  • Accept:

    — What MIME type of response the client wants to receive. Content-type of the corresponding response

    Accept: text/html, application/xhtml+xml, application/xml; Q = 0.9 * / *; Q = 0.8Copy the code

  • Accept-encoding :< compression mode > -Encoding mode that the client can understand. Content-encoding of the corresponding response

    Accept-Encoding: deflate, gzip; Q = 1.0 *; Q = 0.5Copy the code

  • Accept-language :< Language > — a natural Language that clients can understand. Content-language of the corresponding response

    Accept-Language: fr-CH, fr; Q = 0.9, en. Q = 0.8, DE; Q = 0.7 *; Q = 0.5Copy the code

  • Accept-charset :< Charset > — The type of character set that the client can handle. Corresponding to the charset attribute in the response Content-Type.

    Accept-Charset: utf-8, iso-8859-1; Q = 0.5Copy the code

These headers are of the same form:

[;q=

,

[;q=

], which is used to separate multiple types. Each type is represented by the type name and corresponding quality factor. The wildcard * represents other types, that is, types that are not listed. The quality factor represents the priority of the type, the highest is 1, the lowest is 0.01,0 means not accepted, and the default is 1. Traditionally, the highest priority comes first, but the order does not affect the priority.



Note: This representation can be misleading. We traditionally think that; Is greater than, but here the opposite is true. Remember that the quality factor represents the priority of the type that precedes it.

In addition, user-agent also affects the type of resources actually sent.

The value of the content negotiation header is determined by the browser itself. Users are not aware of it and some operating system information or browser Settings may affect it.

Although the client will use these request headers to express the type of resource it wants to receive, the server can choose to ignore it, depending on whether the site developer ADAPTS this.

Finally, a Vary header is also used in the response, whose value is the header name that appears in the content negotiation. Used to indicate which headers the server refers to when selecting a resource. It is mainly used for cache differentiation, which is explained in the cache section.

Vary: User-Agent, Accept-Language
Copy the code

7.5. Block transmission

In some scenarios, a large amount of data needs to be generated dynamically. If the server can process the data and transfer the completed part, the response speed can be improved. HTTP/1.1 supports this technique, allowing entity data in a response to be transmitted in pieces and then assembled on the client side to form a complete message. This technology suitable for streaming is called chunking.

Chunking response requires transfer-encoding: chunked header to indicate that the entity part is chunked, because the data is dynamically generated, otherwise the size is determined, so content-Length cannot appear.

The packet entity in block transmission consists of multiple blocks. The first line of each block is a table length number, indicating the number of bytes in the data block. Length and data are separated by line breaks, and each block is separated by line breaks.

Finally, a data block with a length of 0 is used as the signal that the data is sent.

As an example of a block transfer on MDN:

HTTP/1.1 200 OK 
Content-Type: text/plain 
Transfer-Encoding: chunked

7\r\n
Mozilla\r\n 
9\r\n
Developer\r\n
7\r\n
Network\r\n
0\r\n 
\r\n
Copy the code

There is only one response, but the entity data is transferred in several batches, and finally, the client merges and extracts the data content as MozillaDeveloperNetwork.

The chunking technique is only available over HTTP/1.1, and HTTP/2 uses a more powerful stream structure.

7.6. Range requests and multi-segment data

For some large data, we may only want to fetch a certain range. For a two-hour movie, if you just want to watch the final battle, you should request the last 20 minutes or so of video data directly from the server. This is thanks to the range request feature. The breakpoint transfer principle often used to download files is also scope request.

A server that supports Range requests sends a response using the accept-range :bytes header to tell the client that Range requests are acceptable. Accept-rang: None or no accept-range header is not supported.

On the first request, the client can make a HEAD request to get the header information so that it knows whether the server supports scope requests.

If the server supports Range requests, the client can make a request with a Range:bytes=x-y request header that represents the data Range from x to Y (both x and Y) with the request offset.

The representation of the data range is very flexible. For a 100-byte data, you can use:

  • Range: bytes=0-10— 11 bytes from the first byte to the 11th byte.
  • Range:bytes=10-— from bytes with offset 10 to the end, equivalent to10-99..
  • Range:bytes=-10— in tens of bytes, equivalent to90-99..

After receiving the request, the client needs to perform the following operations:

  1. Verify that the request scope is legal, that is, out of bounds, etc. If the scope is wrong, return416 Range Not SatisfiableResponse indicates an end range error.
  2. If the range is valid, the data content of the specified range is read.
  3. Put the data in a206 Partial ContentTo the response message and setContent-Range: bytes x-y/length, indicating that the file is sentx-yRange, the total file size islength.
  4. Send a response packet.

If you set multiple ranges in the Range request header, the client can also request multiple ranges of data at once. Such as:

Range: bytes=0-499, -500 
Copy the code

Retrieves the first 500 bytes and the last 500 bytes.

This time, the corresponding response will set the content-type: multipart/beteranges; Boundary =boundary to indicate that this is a multi-segment response and that each segment of data is separated by –boundary, note the preceding –.

Each segment of data is represented by the ENTITY headers Content-Type and Content-range for the MIME Type and Range of the data segment, followed by an empty line, followed by the data segment.

I’m going to use a –boundary– after the last number to indicate that there’s no more data, notice both before and after.

According to the request above, one possible response part:

(the rest of the Content of the first) the content-type: multipart/beteranges; boundary=ThisISABoundary --ThisISABoundary Content-Type: text/plain Content-Range: Bytes 0-499/2000 (first 500 characters) --ThisISABoundary Content-type: text/plain content-range: Bytes 1500-1999/2000 (last 500 characters) --ThisISABoundary--Copy the code

8. HTTP connection management

8.1 short connection and long connection

In the DAYS of HTTP/0.9 and HTTP/1.0, a client could only make one request-reply communication per TCP connection. This connection is called a short, or non-continuous, connection. From the point of view of the whole process, each communication is independent, so there is a kind of disconnection.

This design made sense in the early days, when pages often consisted of a single HTML document and users typically took a long time to request again.

However, with the rich content of the page, an HTML document often references multiple images, styles, scripts, etc., which means that once a client initiates a request, there is a high probability that it will be requested again within a short time. The establishment and disconnection of TCP connections requires three handshakes and four waves of the hand. Therefore, establishing and disconnecting TCP connections frequently consumes a lot of costs.

Thus, in late HTTP/1.0, the long connection (persistent connection) model was proposed to allow the communication parties to keep the connection for the next communication.

Long connections are controlled by the header Connection, which has two values as follows:

Connection: keep-alive // The client or server wants to keep the Connection alive. HTTP/1.1 default Connection: close // The client or server actively requests to close the Connection after this communication. HTTP/1.0 defaultCopy the code

HTTP/1.1 enables long connections by default due to performance improvements, but generally uses Conneciton: keep-alive.

However, a long connection also has disadvantages. The connection consumes server resources. Even if users do not send requests for a short period of time, the server resources cannot be released in time and the concurrent processing capability decreases.

The keep-alive header field is an effective way to ensure that a long connection is broken in a timely manner. It limits the maximum time (in seconds) that a connection can be held or the maximum number of communications. Such as:

Keep-Alive: timeout=5, max=1000
Copy the code

8.2 assembly line

By default, a request in a long connection is made on the first request — response received — second request… This sequence, in which the next request must wait until the response from the previous request arrives, is another drawback of long connections. Subsequent requests are blocked until all previous requests are complete, a condition called queue head blocking.

To solve the problem of queue head blocking, HTTP/1.1 also introduced pipelining, whereby a request can be sent after the previous request but without waiting for the response to arrive. The pipeline can only support idempotent requests (figure below from MDN).

Pipelining significantly reduces latency for requests, but has proved difficult to implement. Although RFC2616 states that all devices that support HTTP/1.1 should support pipelining, none of the major browsers implement it, so pipelining is virtually unnecessary. HTTP2 uses a better alternative, multiplexing.

8.3. Concurrent Connection and Domain name Sharding

Just because the assembly line is unrealistic doesn’t mean there isn’t room for improvement. Instead, developers use concurrent connections and domain sharding to alleviate queue congestion.

Concurrent connections are the behavior of the client, that is, the client browser sends multiple connections to the same domain name. As a result, the queue for each connection becomes shorter and the blocking is less obvious. However, an excessive number of connections can be considered a suspect for a DDoS attack by the server, limiting requests from that client (403 warning). Generally, the maximum number of concurrent requests is 6.

Domain name sharding is an improved behavior on the server side. Since the number of concurrent connections is limited by the number of clients to the same domain name, it is possible to split a domain name into multiple domains that point to the same server (DNS knowledge). If one is split into three such domains, it means that the maximum number of concurrent requests between each client and server is (6 * 3 =) 18.

While both of these approaches work, they are additional improvements made by developers outside of the HTTP specification. Now, if there is a need, you should embrace HTTP/2 (HTTP/2 below).

9, Cookie,

9.1 What is Cookie

Cookie is a small piece of text data sent by the server to the client and stored locally on the client. Cookie can be used to realize identity authentication, personalized service, behavior tracking and other functions. Cookie technology has been applied to many websites.

Cookies were invented to compensate for the stateless connection characteristic of HTTP. A stateless connection means that the communication between two parties is not recorded. For example, client A communicates with server B once and then disconnects. Later, A makes A request to B again, but HTTP is stateless and it is not known whether A and B have communicated before.

Therefore, the server cannot determine the identity of the requester through HTTP. User login is the most typical scenario. If a user submits the login operation, the back end of the website is authenticated, but because of the stateless characteristics, the back end cannot know which request comes from the user who has logged in, so the user login function is out of the question naturally.

A simple solution is to attach a password to the response after the user logs in, and ask the client to carry the password with each request so that the server can know who is requesting the user. This password is Cookie.

9.2 Principle of Cookie

Cookie implementation requires three components:

  1. Cookie database at the back of the website. The website needs to remember the Cookie set for each user, so as to determine the corresponding user according to the Cookie value.

  2. Related header field of an HTTP packet. Set-cookie and Cookie header provide support for cookies.

  3. Cookies on the client store files locally. The client needs to save the Cookie sent by the server and bring the Cookie value with the next request, which is generally saved by the browser. A Cookie usually corresponds to a website, and Cookie files are mostly stored in the form of key-value pairs.

    Because the Cookie file is saved by the browser, the object of Cookie recognition is actually the browser. If multiple browsers are installed on one client, the Cookie on one browser will not be used by another browser. Similarly, if you share a computer, different people using the same browser will use the same Cookie, unless you change the Cookie.

On the website server using Cookie service, if a request does not have the Cookie header line, the client sending the request will be considered as a new user, and the server will assign and record a Cookie value for the user, and Set a set-cookie header line when sending the response. When the user agent (browser) receives the response, it extracts the value of set-cookie in the first line and saves it to a local Cookie file. The browser will query the local Cookie file before each request is sent. If there is a Cookie of the target website, it will be added to the first line of the Cookie of the request. The server receives the request and finds that the request has a Cookie, and it can recognize the user.

Tip: The following is the advanced content of cookies. Optionally, you can skip to the limitations of cookies section.

The entire syntax for the first line of set-cookie is:

Set-Cookie:name=value[; expires=<date>][; max-age=<second>][; domain=<domain>][; path=<path>][; secure]
Copy the code

Braces like [; Expires =

] are meant to be optional.

The following is a screenshot of a Cookie received using Baidu:

The first semicolon is preceded by the content of the Cookie, which is in the form of name=value. I don’t know what it means.

Domain=.baidu.com, path=/ does not belong to the cookie value, but is the attribute of the cookie. They tell the browser which domain name and path to carry the cookie.

Only one set-cookie can be Set. However, multiple set-cookies can be Set in a packet containing multiple set-cookies.

Then, here is the Cookie carried by a request:

You can see that some of these cookies are from above, and some are from other responses or saved by the browser.

In short, the browser sends the Cookie with the following header format:

Cookie: name1=value1; name2=value2; name3=value3;
Copy the code

9.3 Cookie properties

Cookies have some properties of their own. For example, validity period, scope, security, etc. These properties are typically set together when the server sets cookies.

In the domain mentioned above, path represents the scope of the Cookie, which specifies the domain name and the path under which the Cookie will be sent.

  • Domain Indicates the domain name for cookie delivery. If this parameter is not specified, it is the default domain name of the current access path, excluding subdomain names. If specified, cookies are applied to the specified domain name and its subdomains.

    In the preceding example, domain=.baidu.com, according to the specification, before the domain name. Be ignored, it will work on baidu.com, news.baidu.com/, zhidao.baidu.com/, etc. If domain attributes are ignored, only Baidu.com will be affected.

    The domain value set by the server must include the current host to be valid; otherwise, cookies are ignored. It would have been a mess.

  • Path indicates the path where the cookie is used. Generally, it is set to the root directory path=/, indicating that any path under the domain name can be used.

The validity period is a very important attribute of cookies, because expired cookies will be gradually cleared, cookie files are generally not too large, reducing the browser’s cookie management pressure, more importantly, from the point of view of security, public computers should clear cookies every time they close the browser. Websites should also re-authenticate users after a certain period of time.

Cookie uses Expires =

and max-age=

to set the Cookie validity period.

  • expires=<date>Said the cookieDue to the time.dateRepresents a point in time.
  • max-age=<second>Said the cookieThe effective time(in seconds) from the time the browser receives the response.

If the preceding two parameters are not set, this is a Session Cookie by default and will be cleared after the browser is closed. If both occur at the same time, max-age is preferred.

Set-Cookie: id=a3fWa; Expires=Wed, 21 Oct 2015 07:28:00 GMT; Max-Age=36000
Copy the code

One of the big issues with cookies is security. To make cookies as Secure as possible, HttpOnly, Secure, and SameSite directives are provided.

  • HttpOnlyIndicates that the cookie can only be used by HTTP browsers and cannot be obtained using scripts and various apis to prevent cross-site scripting attacks (XSS).
  • SecureIndicates that the cookie is valid only in HTTPS. Otherwise, the cookie is ignored. The main thing is to prevent cookie theft. If HTTPS is used, even if someone gets the request, the Cookie content cannot be known because the content is encrypted.
  • SameSiteCross-site request policy is set.SameSite=StrictIndicates that all cross-site requests cannot send cookies.SameSite=LaxIs more lenient, allowing security methods such as GET/HEAD, but not POST.

Despite the best efforts of cookies to ensure their security, sensitive information should not be stored in cookies because the whole mechanism is insecure.

9.4. Use cookies

Most backend frameworks have HTTP-related apis that use cookies, but I won’t mention them (mostly I don’t either). This is mainly about the front end.

Cookies are mainly handled by the front end through documnet. Cookie property.

1. Get a cookie

Document. Cookie retrieves all cookie values in the current path, using; Separated by Spaces. Of course, cookies with the HttpOnly directive do not appear. A simple split processing:

const cookies = document.cookie.split("; ");
Copy the code

2. Write a Cookie

Assignment to document.cookie performs the equivalent of set-cookie.

document.cookie = "user=czpcalm; max-age=60"    // Add a cookie that expires after 60 seconds
Copy the code

Note that this assignment is an appending operation, with no Spaces around the equals sign in the string.

Note: Spaces, :,; Such characters appear in the cookie must be transcoded, you can use the encodeURIComponent() method to encode the cookie content.

3. Delete the Cookie

Cookies can be deleted using the set-cookie feature. Set a past expires for the cookie, or max-age=0.

document.cookie = "user=czpcalm; max-age=0";   // Delete the cookie if it exists.
Copy the code

Pretty simple, right.

As an aside: Cookie was once used to save information on the client side, such as the name of a client logged in to this website. Since every web page can obtain this information through document. Cookie, it becomes a way to transfer information between different pages on the client side. But this content can also be sent along with the request, wasting bandwidth, and cookies are limited in number and size. This is highly deprecated and should now be done using methods such as localStorage.

9.5 Limitations of Cookies

Cookies provide us with a lot of convenience, such as identity authentication, customized services, session transactions and so on, but there are also many problems.

The cookie size of each domain name cannot exceed 4K. Most browsers also have restrictions on the number of cookies. The maximum number of cookies allowed by different browsers under the same domain name varies from 20 to 60.

According to the principle of Cookie, it identifies the browser and the server, and cannot realize the identification between the user and the server, which is a less accurate identification method.

Cookies need to be sent during transmission, increasing network transmission pressure, and are transmitted in plaintext, which may lead to Cookie theft.

The biggest problem is the security of cookies. A hacker may try to steal a user’s Cookie through a cross-site scripting attack to achieve a legitimate login. Users themselves may modify content through cookies to fool the server.

Finally, the function of identity authentication provides convenience for us, but there is also the problem of privacy disclosure, your behavior will be recorded by the server, so as to put targeted advertising. Some websites even put their own cookies in search engines and other websites through third-party cooperation, and record them in their own servers. When users visit their own websites, they push customized advertisements to users. These cookies are called third-party cookies, and if a company distributes them widely, it can track almost everything you do.

10. HTTP caching

The cache mechanism is an important part of HTTP. Using HTTP cache can effectively improve the response speed of HTTP requests.

10.1 overview of HTTP caching

HTTP cache, also known as Web cache, is similar to computer cache. It caches HTTP responses to nodes closer to the user and sends the cached content to the user when the user requests it again. The benefits are obvious: faster response times, less network traffic, less server stress.

The easiest one to understand is the client cache, also known as the browser cache. Everyone should know that the browser will temporarily save the page that you have just visited, and the page will load quickly when you step back. Many people think that the browser has reverted to the previous state. In fact, the HTTP request process is also sent. However, the browser will check the local cache before sending the request, and if the cache hits, it will use it directly. An HTTP request occurs, but is not sent, and the process does not even require a network, and the response is the fastest.

In addition, in the request-response process, there are many proxy servers on the network, proxy server is set up in the client and the source server (destination server) between the intermediate server. Client requests are first sent to the proxy server, which, depending on the request content, may forward the request to the source server for processing, may use caching to respond to the request, or may even discard it. The response sent from the server is also forwarded to the user by multiple agents. Agents with caching capability cache the content, and some agents may also make certain optimization changes to the response content. Many proxy nodes have the capability to cache responses, also known as caching proxies.

Internet Service providers (ISPs) or enterprises often configure local Web proxies to cache user responses on a specific LAN and improve the Web response speed.

Some websites also use a proxy server to improve response time and reduce the pressure on the source server. This is called a reverse proxy. Reverse proxy can implement load balancing, health checks, data filtering, etc., but we are only concerned with its content caching capabilities here.

When a request is sent to the caching proxy, it checks for cache first, uses cache if it exists, and otherwise sends the request to the next generation until it reaches the source server. As the response is sent, the caching proxy will cache it as it is forwarded.

Proxy caches are public caches, and the caches you use may come from other users. The client cache is a private cache.

When used in the following sections, the term cache refers to both client and server caches.

A cache resource may become invalid after a period of time because the resources on the server may have changed, in which case the cache needs to be revalidated for availability. When the request reaches a cache node, it is found that the cache exists but the cache has expired. In this case, the cache node will send the request to the server with if-modified-since: the cache time header. After receiving the request, the server will check whether the resource has changed during the cache time. Reply with a 200 OK response with the new resource. If there is no change, reply with 304 Not Modified without the resource, indicating that no changes have been made and the cache can be used, and the cache will be updated to fresh.

However, the whole detailed process is much more complex than that, mainly due to inconsistent freshness of resources. The response to some requests cannot be cached, such as most POST requests. It is understandable that only security requests (requests that do not cause server state changes) can be cached. Resources that change frequently are cached for a short time, while static resources such as images can be cached for a long time. We must use caching policies to manage the entire caching process effectively. Again, I put it in the advanced content.

Below belongs to the advanced content, choose to see.

10.2 Cache Control

HTTP Cache Control is almost entirely implemented by cache-Control common header lines. When present in a request, it represents what caches the client can accept; when present in a response, it represents which cache nodes the server wants cached and for how long.

Generic header: the header line that can appear in both the request and the response.

The cache-control header has a number of instructions. Some are generic, some are client-specific, and some are server-specific.

The no-store and no-cache directives are general directives that control how the cache is used:

  • no-store— Do not use caching. When in response, indicates that any node isDon’t saveThe response; On a request, indicates that the request does not accept any caching.
  • no-cache— Cache but revalidate. On the response, the representation can be saved, but you must verify that the cache is fresh before each use. On the request, indicates that the cache is acceptable, but you must verify that the cache content is fresh.

The other two attributes, public and private, are for the response only and control which cache nodes can cache the response:

  • privateOnly private caches (browser caches) are allowed to store the response. The default value.
  • publicIntermediate proxies can also be cached.

To ensure the cache is effective, max-age and s-maxage directives are also set to indicate the cache life:

  • Max-age =

    — The duration of the cache, from the time the request was sent. When it appears on the response, it represents the freshness time that can be cached, beyond which the cache is considered expired; On a request, indicates how many seconds prior to the request the cached response is acceptable.

    Header line Expires:

    You can also set the Cache expiration time, but it sets the expiration time, max-age is the expiration time, and the former is a header line, the latter is just an instruction in cache-control header line.

  • S-maxage =

    — only valid for proxy cache, overrides max-age, same effect, but with a higher priority.

In general, fresh caches are considered available, and stale caches need to be revalidated or updated before they can be used.

However, understand that freshness is only an estimate of how valid a copy of a resource is (as it is on the source server), and we cannot predict when the resource on the server will be modified.

So, there are some other instructions that govern how the cache is used. As a result, fresh caches may not be available, and stale caches may not be unavailable, depending on our requirements for cache resources.

For example, the no-store directive mentioned earlier requires that the cache must be validated every time, even if it is fresh. It has the same effect as max-age=0.

Tip: A browser refresh actually sends a request with cache-control: max-age=0.

In addition, there is a must-revalidate directive, which says that after the cache expires, validation must be performed, not if the cache is fresh. Notice the difference between this and no-store. There is also a proxy-revalidate directive that is valid only for proxy caches.

When buying things over time, they have high requirements on freshness of food items. They may not buy food that has expired within two months, while garbage bags that have expired for one month are acceptable. The client side has a similar requirement for caching, which is where the Min-fresh and max-stale directives come in.

  • Min-fresh =

    — The cache must be fresh after the second second.

  • Max-stale =

    — Cache stale within second can be used.

  • Only -if-cache – only cache and no need to verify freshness.

Finally, look at two examples of cache control:

Cache-control: no-store // No Cache is used. It is suitable for resources that change frequently, such as seckill interfaceCopy the code
Cache-control :public, max-age=31536000 // All nodes can Cache, and the validity period is long, applicable to static resourcesCopy the code

10.3. Cache validation

As we know, before using a cache, we often need to verify that resources on the server have not changed. For example, a cache that uses the no-cache directive, a cache with the must-revalidate directive expires, and so on.

The easiest way to determine whether files are consistent is to compare their last modification time. Add if-modified-since :< last Modified time of cached copy > to the first line of the validation request. After the request is received by the source server, the cache copy is compared with the last Modified time of the server resource to determine whether the cache is fresh.

These validations use timestamp comparisons and are only accurate to the second, depending on the HTTP time format. There is no difference between multiple changes in a second. For example, if a resource is modified to get version 1, it is denoted as version 1- last modified time 1. Within a second, it is modified again to get version 2. Since the time is only accurate to the second, it is denoted as version 2- last modified time 1. By comparing the last modification time, we get the wrong result. So timestamp validation is a weak form of validation.

To compensate for this, HTTP provides a stronger form of validation, using a resource identifier called ETag, adding ETag and implementing if-none-match headers.

ETag is the identification of a specific version of a server resource. Different versions of the same resource have different ETAGS. In other words, when a resource changes, its ETag also changes. This allows you to determine if the resources on the server have changed based on the ETag. Etags are strong and weak. Strong Etags require every byte to be the same, while weak Etags require semantic consistency, such as a few extra Spaces on an HTML document that are considered unchanged. Weak ETag starts with W/.

A server that uses an ETag validator sends a cacheable response with the ETag header line that identifies the resource’s ETag, such as:

ETag: "33 a64df551425fcc55e4d42a148795d9f25f89d4" / / a strong ETagCopy the code
ETag: W/"0815" // A weak ETagCopy the code

Note: ETag content is enclosed in “”.

In this way, when verifying the cache, the cache server sends the if-none-match :

header line. After the server receives the request, If no resource is matched according to the Etag, the resource has changed and the cache needs to be updated.

So, a cache validation request may have an if-modified-since or if-none-match header, or both (ETag validation is used in this case). These validation requests are called conditional GET requests.

After receiving a conditional GET request, the server uses ETag or timestamp validation. If the server resource has Not changed, a 304 Not Modified response is returned to notify the cache node that the cache can be used. In this case, the resource does Not need to be re-sent, reducing bandwidth. A 200 OK response message is sent only when the resource changes. The cache node receives it, updates the cache, and sends the response to the client.

10.4, Vary header line

Due to the user negotiation mechanism, requests for the same resource may obtain different resources due to different request headers. For example, the mobile browser and desktop can obtain the home page of Taobao by using the same GET method and URL. If the mobile side accidentally uses a desktop side cache, the experience is not good.

The first line of Vary is used to distinguish this matching scenario, which can only occur in the response. The Vary value is a list of the first lines that appear as part of the request match.

This can be hard to understand. Take the user agent (browser) as an example. Suppose a browser issues a request with the following request header:

GET/HTTP/1.1 Host: example.com User-agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64; The rv: 47.0) Gecko / 20100101 Firefox 47.0Copy the code

Its response comes with a Vary: user-agent header line, the response is cached, and then another request comes to the cache node:

GET/HTTP/1.1 Host: example.com User-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X x.y; The rv: 42.0) Gecko / 20100101 Firefox 42.0Copy the code

It does not work because the User-Agent appears in the Vary header of the cache and the two requests are different.

11, HTTP / 2

Since I’ve mentioned HTTP/2’s connectivity improvements many times before, I thought it was time to introduce HTTP/2.

The HTTP/2 standard was officially released in May 2015 as RFC 7540. According to w3TECHS, as of September 30, 2020, 48.4% of websites worldwide use HTTP/2.

11.1 compatible with HTTP/1

HTTP/2 is backward compatible, and upgrading from HTTP/1 to HTTP/2 requires no additional work on the part of clients and web applications, that is, transparency to users, a feature that has greatly accelerated the adoption of HTTP/2.

Secondly, from the perspective of protocol, HTTP/2 inherits the semantics of HTTP/1. The request method, status code, header line and so on used before are all valid.

However, HTTP/2 takes a radical change in packet syntax. Instead of maintaining plaintext packets, HTTP/2 uses binary encapsulation and frames for fragmentation. However, we still understand HTTP/2 semantically, and the protocol parser parses from the syntax layer.

Many of the improvements in HTTP/2 are based on this change in the binary format.

Binary framing, streaming, multiplexing, priority

HTTP/2 uses a smaller binary frame structure instead of the traditional plaintext format. No longer semantically readable to humans, but more computationally analytic, reducing a lot of parsing.

After binary framing, the frame becomes the unit of message transmission. How does the receiver determine which frames are coming from the same message?

HTTP introduction and a virtual structure called “streams”. A connection is divided into streams, which are a bi-directional sequence of binary frames. Frames from the same message are assigned the same stream ID, and the receiver combines frames with the same stream ID to form a single message. Simply put, a flow is a transfer of messages.

A connection is divided into multiple streams, which can transmit multiple messages at the same time. Multiplexing is realized and the problem of queue head blocking is solved. The key to understanding this is to understand that “streams” are actually virtual, just frame by frame, from the point of view of the connection, and do not require the frames to come from the same message. The purpose of the flow ID is to allow the receiver to receive the message correctly, and the connection is unaware of its existence.

With multiplexing, one client-server communication requires only one long connection, eliminating the resource waste of establishing multiple connections.

To make important messages faster, HTTP/2 also introduced the concept of stream priority for the first time, with higher-priority streams being sent first.

11.3. Head compression

An HTTP message is often accompanied by a dozen headers, which increase the size of the message to some extent, especially when there is no message body. Also, headers with fixed communication processes such as user-Agent need to be repeated with each request. To further shrink the message, HTTP/2 compresses the message header.

Header compression uses a special HPACK algorithm. The general principle is to mark all possible combinations of header and header values with index numbers. For example, Connection:close and Connection:keep-alive have different index numbers. Maintain a common static and dynamic table on both the client and server as a dictionary for indexes and headers. The sender looks up the dictionary to find the index number for each header to be set, and sends only the index number in the message. After receiving the message, the recipient can find the corresponding header based on the index number and know what header was sent and what value it corresponds to. In addition, numbers and characters are further compressed using Huffman encoding.

11.4. Server push

HTTP/2 also changes the request-response model to some extent, introducing a server push mechanism: under certain conditions, the server does not need to wait for the request to arrive, but can actively send resources to the client. For example, if a client requests an HTML page, it will immediately request references to scripts, CSS, images, etc. At this time, the server can actively send these contents to the client to speed up the response.

11.5. Security enhancement

In terms of HTTP design, there is no security at all. HTTP/2 used to be plaintext. HTTP/2 only uses binary format and does not encrypt the content. For security purposes, we actually use the HTTPS protocol in most cases. It encrypts HTTP messages using the security protocol SSL/TLS before they are transmitted (before entering TCP).

Due to the popularity of HTTPS, the HTTP/2 Working group also incorporated some security into the specification, mandating the use of TLS 1.2 or later, since earlier versions had been found to be vulnerable.

However, the HTTP/2 Working group was divided on encryption. Some argue that encryption should not be part of HTTP, and that it is wasteful for communications that do not need encryption. So, HTTP/2 actually has two versions, secure H2 and unencrypted H2C.

Aside: The evolution of HTTP

Finally, let’s conclude by reviewing several versions of HTTP and how they have evolved.

HTTP/0.9 — Single-line protocol

Released in 1991, it is extremely simple, with only the GET method, for requesting HTML documents. Resource requests are made using the GET URL command, and the response is an HTML document. Has been out of date.

HTTP/1.0 — Extensible protocols

Released in 1996, with much improvement.

  • Add POST and HEAD methods.

  • Add the version number, status code, and header line to the message, using the same starting line + header + entity format as now.

  • Added cache, content encoding, multi – segment data transmission and other functions.

HTTP/1.0 has no official standards and standardization is confusing. A few websites are still in use.

HTTP/1.1 — Standardized protocols

Released in 1997, eliminating clutter and making many improvements to HTTP1.0.

  • Add methods such as PUT and DELETE.

  • Long connections are used by default to reduce connection setup disconnection overhead, and pipeline mechanisms are proposed, though rarely used.

  • Introduce content negotiation.

  • Introduce cache control.

  • Supports response chunking. (HTTP/2) not supported.

  • Add the Host header so that different domain names can reside on the same server.

A very powerful protocol, still the most widely used protocol today.

HTTP/2 — Binary protocol

Released in 2015, it uses a binary message structure with many optimizations.

  • Breaking up messages into smaller binary structures called frames

  • Virtual “stream” structure is introduced to realize multiplexing, which basically solves the problem of queue head blocking and eliminates the need to establish multiple connections.

  • Compress the header to shrink the message.

  • Introduce server active push mechanism.

  • A later version of TLS is required.

HTTP/3 – the future of HTTP

Not released. As of now, HTTP/3 is still in draft form, and it is a major improvement on HTTP from the transport layer.

  • Instead of using TCP, use one customized for HTTPQUIC agreementTo solve the problem of slow TCP establishment and packet block.
  • QUIC built-in STL1.3, can only encrypt transmission, to ensure security.
  • No need to specify ports, more flexible service discovery mechanism.

Afterword.

At last. Even so, HTTP is too much content to be comprehensive and in-depth. I read a lot of documents and blogs, and READ a lot of HTTP messages, which is completely different from what I knew before. A lot of content, may forget later, make a good note, forget to look over.