TCP/IP network layered model	instructions	OSI network layered model	instructions
Application Layer	Because the bottom three layers lay the groundwork so well, a thousand flowers bloom in this layer, with various application-specific protocols. Examples include Telnet, SSH, FTP, SMTP, and of course our HTTP. The transmission unit is message or message.	Application Layer	Transfer data for specific applications.
–	–	Presentation Layer	Convert data into appropriate, understandable syntax and semantics.
–	–	Session Layer	Maintain the connection state in the network, that is, maintain session and synchronization.
Transport Layer	This layer of protocol, which ensures the reliable transfer of data between two points marked by IP addresses, is the layer at which TCP works, along with its sister protocol, UDP. The transmission unit is segment.	Transport Layer	Equivalent to the transport layer in TCP/IP.
Internet Layer	The IP protocol is at this layer. Because IP protocol is definedThe IP addressThe concept, so can be inThe link layerOn the basis of,Replace MAC addresses with IP addressesTo connect hundreds of lans and wans into a vast virtual network where devices can be found by “translating” IP addresses into MAC addresses. The transmission unit is packet.	Network Layer	The TCP/IP equivalent of the Internet layer.
Link Layer /MAC	Responsible for sending raw data packets on Ethernet, WiFi and other underlying networks, working at the network card level, usingMAC addresses to mark devices on the networkSo it’s sometimes called the MAC layer. The transmission unit is frame.	Data Link Layer	Equivalent to the TCP/IP connection layer.
		Physical Layer	The physical form of a network, such as cable, optical fiber, network card, hub, etc.

What is HTTP?

Hypertext Transfer Protocol (HTTP) is an application-layer protocol for transporting hypermedia documents such as HTML.

It is designed for communication between Web browsers and Web servers, but can also be used for other purposes.

HTTP follows the classic client-server model, where a client opens a connection to make a request and then waits until it receives a server-side response.

HTTP is a stateless protocol, which means that the server does not retain any data (state) between two requests.

The HTTP message

The structure of HTTP request packets and response packets is basically the same and consists of three parts:

1. Start line: Describes basic information about the request or response.
2. Header field set: describes the packet in more detail in key-value format.
3. Entity: Data that is actually transmitted. It may not be plain text but can be binary data such as pictures and videos.

The request message

Request Line

The beginning line of the request message is called the Request line, which briefly describes how the client wants to operate on the server’s resources.

Request Method (Method -> GET/POST) + request destination (Path -> URI) + Version of the protocol -> HTTP 1.0/1.1/2.0

These three sections are usually separated by a space and terminated with a CRLF newline.

Request method

Currently HTTP/1.1 specifies eight methods, all of which must be capitalized. I’ll list them briefly and explain them in more detail later.

1. GET: It means to request a resource from the server;
2. HEAD: Get meta information about the resource (response header);
3. POST: data is written or uploaded to the server.
4. PUT: submits data to the server for data update.
5. DELETE: deletes resources (rarely used).
6. CONNECT: requires the server to establish a special connection tunnel between the client and another remote server.
7. OPTIONS: Lists the operations that can be performed on the resource, returned in the Allow field of the response header;
8. TRACE: Traces the transmission path of the request and response.

GET and POST

role

GET is used to GET resources, and POST is used to transport entity bodies.

parameter

Both GET and POST requests can use additional parameters, but the parameters of GET appear in the URL as a query string, while the parameters of POST are stored in the entity body.

Because THE URL only supports ASCII codes, if there are Chinese characters in the parameters of GET, they need to be encoded first.

http://www.baidu.com/? Just search for it
http://www.baidu.com/?%E7%99%BE%E5%BA%A6%E4%B8%80%E4%B8%8B%EF%BC%8C%E4%BD%A0%E5%B0%B1%E7%9F%A5%E9%81%93
Copy the code

security

In THE HTTP protocol, security means that the request method does not “corrupt” the resources on the server, that is, does not substantially modify the resources on the server.

The GET method is secure, but the POST method is not, because the purpose of POST is to transmit the entity body content, which may be form data uploaded by the user. Once the upload is successful, the server may store this data in the database, thus changing the state.

idempotence

The Idempotent is a mathematical term that is borrowed from the HTTP protocol to mean that the same action is performed multiple times and the results are all the same.

GET /pageX HTTP/1.1 is idempotent, and the client receives the same result from multiple consecutive calls:

GET /pageX HTTP/1.1
GET /pageX HTTP/1.1
GET /pageX HTTP/1.1
GET /pageX HTTP/1.1
Copy the code

POST /add_row HTTP/1.1 is not idempotent, and if it is called more than once, multiple lines will be added:

POST /add_row HTTP/1.1   -> Adds a 1nd row
POST /add_row HTTP/1.1   -> Adds a 2nd row
POST /add_row HTTP/1.1   -> Adds a 3rd row
Copy the code

The cache

GET is cacheable, POST is not.

XMLHttpRequest

XMLHttpRequest is an API that provides the client with the ability to transfer data between the client and server. It provides an easy way to get data through a URL without having to refresh the entire page. This allows the web page to update only a portion of the page without disturbing the user. XMLHttpRequest is widely used in AJAX.

• When using the POST method of XMLHttpRequest, the browser sends the Header first and then the Data. But not all browsers do this. Firefox, for example, does not.
• The GET methods Header and Data are sent together.

URI

A Uniform Resource Identifier (URI) is essentially a string that uniquely identifies the location or name of a Resource.

• scheme Which protocol should the resource use http,https. It must be followed byThree specific characters: / / 。
• User :passwd@ Identity information Indicates the user name and password for logging in to the host. The information is displayed in plain text and is not recommended.
• authority Name of the host where the resource residesThe usual form ishost:port, the port can be omitted.
• path Marks the location of the resource. The PATH part of the URI must be/The beginning, that is, must contain/.
• The KV values are concatenated with the character &.
• It is an anchor point inside the resource located by the URI, and the browser jumps to the point indicated after retrieving the resource.

https://search.jd.com/Search?keyword=openresty&enc=utf-8&qrst=1&rt=1&stop=1&vt=2&wq=openresty&psort=3&click=0
// scheme -> https , authority -> search.jd.com , path -> /Search... , query -> keyword=openresty...
Copy the code

Uris support only ASCII characters. For character sets and special characters other than ASCII, URIs are escaped using Percent Encoding. The escape rule is to convert non-ASCII characters or special characters to hexadecimal byte values, followed by a %.

http://www.baidu.com/? Just search for it
http://www.baidu.com/?%E7%99%BE%E5%BA%A6%E4%B8%80%E4%B8%8B%EF%BC%8C%E4%BD%A0%E5%B0%B1%E7%9F%A5%E9%81%93
Copy the code

The response message

Status Line

The starting line of the response message is called the status line, which means the status of the server response.

Version of the protocol -> HTTP 1.0/1.1/2.0 + Status code -> 200/404 + Cause (Status message -> digital Status code)

These three sections are also separated by Spaces, and are ended with a CRLF newline.

Status code

Status code	category	meaning
1XX	Informational (Informational status code)	The received request is being processed
2XX	Success (Success Status code)	The request is successfully processed
3XX	Redirection (Redirection status code)	Additional action is required to complete the request
4XX	Client Error (Client Error status code)	The server cannot process the request
5XX	Server Error	The server failed to process the request

1 xx information

• 100 Continue: Indicates that everything is fine so far and the client can Continue sending the request or ignore the response.

2 xx success

• 200 OK: is the most common success status code, indicating that everything is normal. If you areHEADBoth the request and the response header returned by the server will have body data.
• 204 No Content: The request has been processed successfully, but the response message returned does not contain the body of the entity. Typically used when you only need to send information from the client to the server without returning data.
• 206 Partial Content: Indicates that the client sends a Range request. The response packet contains the entity Content specified by Content-range.

3 xx redirection

• 301 Moved Permanently: Permanently redirects. This means that the requested resource no longer exists and needs to be accessed again using a new URI.

• 302 Found: Temporary redirection, meaning the requested resource is still there but needs to be accessed temporarily with another URI.

  Both 301 and 302 use the Location field in the response header to indicate which URI to jump to next, and the browser redirects to the new URI.

• 303 See Other: Provides the same functions as 302, but 303 explicitly requires that the client use the GET method to obtain resources.

• Note: Most browsers will change POST to GET for 301, 302, and 303 redirects, although HTTP does not allow you to change POST to GET for 301, 302, and 303 redirects.

• 304 Not Modified: If the request header contains some conditions, for example: If-match, if-modified-since, if-none-match, if-range, if-unmodified-since, If the condition is not met, the server returns the 304 status code.

  • A request is made to the server, but the server responds with no changes to the file, no data content is returned, and the browser cache is used.

• 307 Temporary Redirect: indicates that the browser does not change the POST method to GET method for redirecting requests.

4XX Client error

• 400 Bad Request: Syntax errors exist in the Request packet.
• 401 Unauthorized: This status code indicates that the request to be sent requires authentication information (BASIC authentication and DIGEST authentication). If the request has been made before, the user authentication failed.
• 403 Forbidden: The request is rejected.
• 404 Not Found: The requested resource was Not Found on the server and therefore cannot be provided to the client.

The rest of the error code is more explicit about the cause of the error:

• 405 Method Not Allowed: Certain methods are Not Allowed to operate resources. For example, POST is Not Allowed. Only GET is Allowed.
• 406 Not Acceptable: The resource does Not meet the requirements of the client request, for example, the request is in Chinese but English only;
• 408 Request Timeout: The server waits too long for the Request Timeout.
• 409 Conflict: Multiple requests are in Conflict, which can be understood as a race when multiple threads are concurrent;
• 413 Request Entity Too Large: The body in the Request packet is Too Large.
• 414 request-uri Too Long: The URI in the Request line is Too large.
• 429 Too Many Requests: The client sent Too Many Requests, usually due to the server’s connection restriction policy;
• 431 Request Header Fields Too Large: A field or the total size of the Request Header is Too Large.

5XX Server error

• 500 Internal Server Error: An Error occurred while the Server was performing a request.
• 501 Not Implemented: The functions requested by the client are Not supported.
• 502 Bad Gateway: Error code returned when the server functions as a Gateway or proxy, indicating that the server is working properly and an error occurs when accessing the back-end server. The specific cause of the error is unknown.
• 503 Service Unavailable: The server is temporarily overloaded or is down for maintenance and cannot process requests now.

Headers

Request line + header field = request header; Status line + header field = response header.

The structure of the request header and response header is basically the same, the only difference is the start line, so the fields in the request header and response header can be introduced together.

The header field is in the form of key-value. The key and value are separated by:, and the field ends with a CRLF newline.

HTTP header field is very flexible, not only can use the standard Host, Connection and other existing headers, but also can add any custom header, which brings unlimited extension possibilities to HTTP protocol.

However, there are a few things to note when using header fields:

1. Field names are case insensitive, for exampleHostOr you could write it ashost, but capital letters are more readable;
2. Do not allow Spaces in field names. Use hyphens instead-, but cannot use underscores_. For example,test-nameIs a valid field name, andtest name,test_nameIs an incorrect field name;
3. The field name must be immediately followed:, cannot have Spaces, and:The following field value can be preceded by multiple Spaces;
4. The order of fields is meaningless and can be arbitrarily arranged without affecting semantics;
5. A field cannot be duplicated in principle unless the semantics of the field itself allow it, for exampleSet-Cookie.

Generic header field

Header field name	instructions
Cache-Control	Controls the behavior of caching
Connection	Controls header fields that are no longer forwarded to agents and manages persistent connections
Date	Date and time when the packet was created
Pragma	Packet instructions
Trailer	View the header of the packet end
Transfer-Encoding	Specifies the transmission code of the packet body
Upgrade	Upgrade to another protocol
Via	Proxy server information
Warning	Error notification

Request header field

Header field name	instructions
Accept	The type of media that the user agent can handle
Accept-Charset	Preferred character set
Accept-Encoding	Priority content encoding
Accept-Language	Preferred language (natural language)
Authorization	Web Authentication Information
Expect	Expect specific behavior from the server
From	Email address of the user
Host	Request the server where the resource resides
If-Match	Compare Entity Tag (ETag)
If-Modified-Since	Compares the update times of resources
If-None-Match	Compare entity tags (as opposed to if-match)
If-Range	Send scope requests for entity Byte when the resource is not updated
If-Unmodified-Since	Compare resource update times (as opposed to if-modified-since)
Max-Forwards	Maximum transmission hop by hop
Proxy-Authorization	The proxy server requires authentication information of the client
Range	Byte range request for the entity
Referer	The original acquirer of the URI in the request
TE	Priority of transmission encoding
User-Agent	HTTP client program information

Response header field

Header field name	instructions
Accept-Ranges	Whether to accept byte range requests
Age	Calculate the elapsed time of resource creation
ETag	Matching information of resources
Location	Causes the client to redirect to the specified URI
Proxy-Authenticate	The proxy server authenticates the client
Retry-After	Request the timing of the request to be made again
Server	HTTP server installation information
vary	Proxy server cache management information
WWW-Authenticate	Authentication information about the server to the client

Entity header field

Header field name	instructions
Allow	HTTP methods supported by the resource
Content-Encoding	The encoding method applicable to the entity body
Content-Language	The natural language of entity subjects
Content-Length	The size of the entity body
Content-Location	Replace the URI of the corresponding resource
Content-MD5	The packet digest of the entity body
Content-Range	The location range of the entity body
Content-Type	The media type of the entity body
Expires	The date and time when the entity body expires
Last-Modified	The last modified date and time of the resource

Blank line (CRLF)

Blank lines, that is, “CRLF”, the hexadecimal “0D0A”.

entity/body

The actual data transmitted may not be pure text, but binary data such as pictures and videos. Corresponding to the header, it’s called the body.

1. The data type indicates what the Content of the entity data is, using MIME Type, with the associated header fields Accept and Content-Type;
2. The data Encoding represents how the entity data is compressed. The related header fields are accept-encoding and Content-Encoding.
3. The Language type represents the natural Language of entity data, with the associated header fields being Accept-Language and Content-Language (not normally sent);
4. The character set represents the encoding of entity data. The associated header fields are Accept-Charset (not normally sent) and Content-Type;

MIME type

Multipurpose Internet Mail Extensions, or MIME for short. HTTP takes a portion of it and marks the body’s data type, which is the MIME type we hear all the time.

Common MIME types:

1. Text: readable data in text format,text/html(Hypertext documents),text/plain(plain text),text/css(Style sheets), etc.
2. Image: indicates an image fileimage/gif,image/jpeg,image/pngAnd so on.
3. audio/video: Audio and video data, for exampleaudio/mpeg,video/mp4And so on.
4. Application: The data format is not fixed, may be text or binary, must be interpreted by the upper-layer application. Common areapplication/json.application/javascript,application/pdfIn addition, if you really don’t know what type of data is, like the black box just said, it will beapplication/octet-stream.Opaque binary data 。

Accept: text/html,application/xml,image/webp,image/png The "," delimiter lists multiple types that the client can receive.
Content-Type: text/html // The actual type of entity data.
Copy the code

Encoding type

HTTP sometimes compresses data during transmission to save bandwidth, using Encoding Type.

There are only three commonly used Encoding types:

1. Gzip: GNU Zip compression format, is the most popular compression format on the Internet;
2. Deflate: Zlib (Deflate) compressed format, second in popularity to Gzip;
3. Br: A new compression algorithm (Brotli) optimized for HTTP.

Accept-Encoding: gzip, deflate, br // The compression format supported by the client. If not, the client does not support compression of data.
Content-Encoding: gzip // The compression format used by the entity data, if not, the corresponding data is not compressed.
Copy the code

Language types and character sets

In order to solve the problem of language internationalization, two concepts are introduced: language type and character set.

Accept-Language: zh-CN, zh, en // The request header field, representing the natural language understood by the client, lists multiple types with ', 'delimiters.
Content-Language: zh-CN // Entity header field, which tells the client the language type used for entity data. (Will not be sent)

Accept-Charset: gbk, utf-8 // Request header field, which represents the character set requested by the browser. (Will not be sent)
Content-Type: text/html; charset=utf-8 // The generic header field contains the entity data character set returned by the server.
Copy the code

Browsers support multiple character sets and typically do not send accept-charset, while the server does not send content-language. Because the Language used can be inferred from the character set, only the accept-language field will be displayed in the request header. The response header will only have the Content-Type field.

Quality values and outcomes of content negotiation

Accept: text/html,application/xml; q=0.9, */ *; Q = 0.8 than: Accept - Encoding, the user-agent, AcceptCopy the code

When HTTP header fields such as Accept, accept-encoding and accept-language are used for content negotiation, a special q parameter can be used to indicate the weight to set the priority, where q is the quality factor.

The maximum value of the weight is 1, the minimum value is 0.01, the default value is 1, and a value of 0 indicates rejection. This takes the form of a data type or language code followed by one; And then q=value.

A word of caution here; In most programming languages; Is stronger than, and in HTTP content negotiation is exactly the opposite,; The meaning of phi is less than phi.

The Vary field indicates that the server looks at the Accept-Encoding, user-Agent, and Accept header fields and decides which response to send back.

The Vary field can be thought of as a special version tag for the response message. Whenever a request header such as Accept changes, Vary varies with the response message. That is, there may be multiple “versions” of the same URI, mainly used by proxy servers in the middle of transport links to implement caching services, which will be covered later in HTTP caching.

HTTP method of transferring large files

Data compression

Accept-Encoding: gzip, deflate, br // The compression format supported by the client. If not, the client does not support compression of data.
Content-Encoding: gzip // The compression format used by the entity data, if not, the corresponding data is not compressed.
Copy the code

Block transmission

Transfer-Encoding: chunked // The generic header field, the body is broken up into chunks and sent one by one.
Copy the code

Transfer-encoding: The chunked and Content-Length fields are mutually exclusive, that is, the chunked and content-Length fields cannot appear at the same time in a response message. The transmission Length of a response message is either known or unknown.

1. Each block contains two parts, a length header and a data block;
2. Length head isCRLF(Enter newline, that is\r\n) ending a line in plain text with a hexadecimal number indicating the length;
3. The data block is followed by the length header and is also used lastCRLFEnd, but the data does not containCRLF;
4. The end is indicated by a block of length 0, i.e0\r\n\r\n.

HTTP/1.1 200 OK
Content-Type: text/plain
Transfer-Encoding: chunked

7\r\n // Length is in hexadecimal format. Is \ r \ n CRLF
Mozilla\r\n
9\r\n
Developer\r\n
7\r\n
Network\r\n
0\r\n
\r\n
Copy the code

Range requests

Dragging a progress bar through a video is actually a way to retrieve a piece of data from a large file, which chunking doesn’t have.

To meet these requirements, THE HTTP protocol introduced the concept of range requests, allowing clients to use special fields in the request headers to indicate that only a portion of a file is being retrieved.

Accept-Ranges: none // Range requests are not supported
Accept-Ranges: bytes = x - y // Range requests are supported. X and y are data ranges in bytes, representing offsets.

// Assume the file is 100 bytes
Accept-Ranges: bytes = 0- // Indicates from the beginning of the document to the end of the document, equivalent to 0-99, i.e. the entire file;
Accept-Ranges: bytes = 10- // from the 10th byte to the end of the document, equivalent to 10-99;
Accept-Ranges: bytes = -1 // Is the last byte of the document, equivalent to 99-99;
Accept-Ranges: bytes = -10 // Count down 10 bytes from the end of the document, equivalent to 90-99.
Copy the code

After the server receives the Range field, it needs to do four things:

1. Check whether the scope is valid. Scope out of bounds returns status code 416.
2. The Range is correct, the offset is calculated from the Range header, and the file fragment is read. Return status code 206.
3. The server adds the response header field content-range. The value is in bytes x-y/length format.
4. Send data.

HTTP/1.1 206 Partial Content
Date: Wed, 15 Nov 2015 06:25:24 GMT
Last-Modified: Wed, 15 Nov 2015 04:58:08 GMT
Content-Range: bytes 21010-47021/47022
Content-Length: 26012
Content-Type: image/gif

... 26012 bytes of partial image data ...
Copy the code

Many piece of data

Range: bytes=0-9.20-29.30-39 / / request header
Copy the code

The MIME used in the response header is multipart/byteranges and the parameter boundary= XXX is used to separate the sequence of bytes.

Each byte sequence also needs to be marked with content-Type and Content-range for the Type and Range of the data.

HTTP/1.1 206 Partial Content
Content-Type: multipart/byteranges; boundary=00001111
Content-Length: 189
Connection: keep-alive
Accept-Ranges: bytes
 
 
--00001111
Content-Type: text/plain
Content-Range: bytes 0-9/96
 
// this is
--00001111
Content-Type: text/plain
Content-Range: bytes 20-29/96
 
ext json d
--00001111--
Copy the code

HTTP Connection Management

Short connection and long connection

When a browser accesses an HTML page with multiple images, it requests image resources in addition to the HTML page resources requested. If you had to create a TCP connection for every HTTP communication, it would be expensive.

A long connection requires only one TCP connection to carry out multiple HTTP communications.

• Starting from HTTP/1.1, the default connection is long. To disconnect the connection, the client or server must request to disconnect the connectionConnection : close;
• Short connections were the default prior to HTTP/1.1, and long connections were used if neededConnection : Keep-Alive.

Queue Head blocking

Queue header blocking has nothing to do with short and long connections, but is caused by the basic REQUEST-reply model of HTTP.

Because HTTP dictates that packets must be received on a first-in, first-out (FIFO) serial queue. The requests in the queue have no priority, only the order in which they were queued. The first request is processed first. If the queue leader’s request is processed slowly, all subsequent requests must wait. This causes the head of the queue to be blocked.

Performance optimization

Concurrent connections

Initiate multiple long connections to a domain name at the same time, using quantity to solve quality problems. But there are drawbacks to this approach. If each client wanted to make many connections fast, the number of users multiplied by the number of concurrent connections would be astronomical. The server’s resources simply cannot support, or the server considers it a malicious attack, but will cause denial of service.

The HTTP protocol recommends that clients use concurrency, but not abuse it.

Domain sharding

Don’t HTTP and browsers limit the number of concurrent connections? Ok, then I will open several domain names, such as shard1.chrono.com, shard2.chrono.com, and these domain names all point to the same server www.chrono.com, so that the actual number of long connections will go up again, but also to solve the queue head blocking problem.

HTTP redirects and jumps

https://im.qq.com/download/
Copy the code

Click ** download ** link on QQ home page, what will happen?

The browser parses the URI in the text -> initiates a new HTTP request with the URI -> gets the response and renders the page to which the new URI points.

Such a forward initiated by the browser user is called an active forward, while a forward initiated by the server is called a Redirection.

Location:https://im.qq.com/download/ // Absolute URI.
Location:/index.html // Relative to the URI, Location is the response header field.Copy the code

Application scenarios of redirection

• The resource is unavailable and needs to be replaced with a new URI. For example, domain name change, server change, website revision, and system maintenance.
• To avoid duplication, let multiple urls jump to the same URI, adding access points does not add extra work. For example, some sites apply for multiple domain names with similar names and then redirect them to their main site.

Redirection related issues

• Performance consumes two request-replies, one more than normal access. Appropriate use, not abuse.

• Loop jump If the redirection policy is set inadvisedly, an infinite loop of A=>B=>C=>A may occur. The browser must be able to detect loop jumps and should stop sending requests with an error when it does.

HTTP Cookie

The HTTP protocol is stateless, mainly to keep HTTP as simple as possible so that it can handle a large number of transactions. HTTP/1.1 introduced cookies to store state information.

Cookie is a small piece of data that the server sends to the user’s browser and saves locally. It will be carried when the browser sends another request to the same server, which is used to tell the server whether the two requests come from the same browser. Since each subsequent request will need to carry Cookie data, there is an additional performance overhead (especially in a mobile environment).

Cookies were once used to store client data as the only storage method because there was no other suitable storage method, but now as modern browsers begin to support a variety of storage methods, cookies are gradually phased out. New browser apis already allow developers to store data directly locally, such as using the Web Storage API (local and session storage) or IndexedDB.

How cookies work

1. The client sends an HTTP request to the server
2. When the server receives an HTTP request, it adds a set-cookie field to the response header
3. The browser saves the Cookie after receiving the response
4. Cookie information is sent to the server through the Cookie field in each subsequent request to the server.

The attribute of the Cookie

Cookie: name=value; name2=value2; name3=value3 / / request header
Set-Cookie: <cookie-name>=<cookie-value>; Domain=<domain-value>; Secure; HttpOnly...
Copy the code

The Cookie life cycle

Set-Cookie:Expires=Wed, 21 Oct 2015 07:28:00 GMT; Max-Age=10; // max-age has a higher priority
Copy the code

• An Expires is an absolute point in time, which can be understood as a deadline.
• Max-age indicates the relative time, in seconds. To obtain the absolute expiration time, add the max-age to the time when the browser receives the packet.

Cookie scope

The Domain identifier specifies which hosts can accept cookies. If this parameter is not specified, it defaults to the host of the current document (excluding the subdomain name). If Domain is specified, subdomain names are generally included. For example, if you set Domain=mozilla.org, cookies are also included in the subdomain (e.g. Developer.mozilla.org).

Set-Cookie: Domain = mozilla.org;
Copy the code

The Path identifier specifies which paths under the host can accept cookies (the URL Path must exist in the request URL). Child paths are also matched with the character %x2F (“/”) as the path separator. For example, if Path=/docs is set, the following addresses will match:

/docs
/docs/Web/
/docs/Web/HTTP
Copy the code

In most cases, only one or no Cookie is used, indicating that cookies can be used in any path under the domain name.

Cookie security

• HttpOnly CookieIt can only be transmitted over HTTP,Other access modes are prohibited. The browser will disable itdocument.cookieAnd all the relevant apis. That will stop itCross-site scripting (XSS).
• SameSite can be set to Strict/Lax/None to strictly restrict Cookie cross-site/allow GET/HEAD cross-site/not restrict, respectively. Protects against cross-site request forgery (XSRF) attacks.
• SecureSo this CookieOnly HTTPS can be used to encrypt the transmissionThe plaintext HTTP protocol forbids sending. But the Cookie itself is not encrypted, the browser is still in clear text form.
  • ** Note: ** Insecure sites (HTTP:) can no longer set the Secure directive in cookies (a new restriction introduced in Chrome 52+ and Firefox 52+).

The application of the Cookie

Cookies are mainly used for the following three aspects:

• Session state management (such as user login status, shopping cart, game score, or other information that needs to be logged).
• Personalization (such as user – defined Settings, themes, etc.)
• Browser behavior tracking (e.g. tracking and analyzing user behavior, etc.)

Cookie shortcomings

• The length limit is only 4KB.
• Security Because cookies are transmitted in plaintext, they are easy to be intercepted or tampered with, and session information will be exposed after interception.
• Extra overhead Cookies are sent to the server each time an HTTP request is made, adding overhead.
• Some clients do not support cookies.

HTTP cache control

** Cache ** is a technique for saving a copy of a resource and using it directly on the next request. When the Web cache discovers that the requested resource has been stored, it intercepts the request and returns a copy of the resource rather than going to the source server to download it again.

Strong cache

No need to send a request to the server, directly read the browser’s local cache, in Chrome Network display HTTP status code 200, in Chrome, Strong caches are divided into Disk Cache and Memory Cache. The location of the Cache is controlled by the browser. Strong caching is controlled by a combination of the Header attributes Expires, cache-Control, and Pragma.

Pragma

Pragma:no-cache; // Disable caching, only for HTTP1.0. The response header field does not support this property.
Copy the code

Expires

Expires defines a cache time relative to the time on the server. It defines a resource “expire time.” If the time on the client doesn’t match the time on the server (especially if the user has changed the system time on his computer), the cache time may not be meaningful.

Expires:Wed, 21 Oct 2015 07:28:00 GMT;//http1.0 specifies the expiration time of the cache. Entity header field.
Copy the code

Cache-Control

HTTP1.1 adds cache-Control to define the expiration date of a Cache.

// Cacheability
Cache-Control:no-cache;// equivalent to 'max-age=0,must-revalidate', where the resource is cached but the cache expires immediately. At the same time, the resource validity is forcibly verified on the next access.
Cache-Control:no-store;// Requests and responses are not cached. Generic header fields.
Cache-Control:public;// Indicates that the response can be cached by the client that sent the request, proxy server, etc. Response header field.
Cache-Control:private;// Indicates that the response can only be cached by a single user, not as a shared cache (that is, the proxy server cannot cache it). Response header field.
/ / due
Cache-Control:max-age=<seconds>;// Cache resources, but expire after the specified time in seconds. Time is the time relative to the request. Generic header fields.
Cache-Control:s-maxage=<seconds>;// Overrides max-age or Expires headers, but only for shared caches (such as individual agents); private caches ignore them. Response header field.
Cache-Control:max-stale[=<seconds>];// Resources remain valid even if the cache expires within the specified time. Request header field.
Cache-Control:min-fresh=<seconds>;// Cached resources must remain fresh for at least a specified time. Request header field.
// revalidate and reload
Cache-Control:must-revalidate;// Once a resource has expired (e.g., beyond max-age), the cache cannot respond to subsequent requests with the resource until it is successfully validated to the original server. Response header field.
Cache-Control:proxy-revalidate;// It has the same effect as must-revalidate, but it only applies to shared caches (such as proxies) and is ignored by private caches. Response header field.
/ / other
Cache-Control:no-transform;// Force the proxy server not to convert resources, and forbid the proxy server to modify content-encoding, Content-range, and Content-Type fields (therefore proxy gzip compression will not be allowed). Request header field.
Cache-Control:only-if-cached;// Return only cached resources, no network access, if no cache return 504. Request header field.
Copy the code

Assume that the requested resource was cached on April 5 and expired on April 12.

When max-age and max-stale and min-fresh are used together, their Settings take effect independently of each other. The most conservative cache policy is always effective.

This means that if max-age = 10 days, max-stale = 2 days, min-fresh = 3 days, then:

• According to the max-age setting, the original cache period is overwritten, and the cache resource will expire on April 15 (5 + 10 = 15).

• According to the max-stale setting, the cache will remain valid for two days after expiration. In this case, the Response will return 110(Response is stale) status code, and the cache resource will expire on April 14 (12 + 2 = 14).

• According to the setting of Min-fresh, there should be at least 3 days of freshness, and the cache resources will expire on April 9 (12-3 = 9);

Since the client always adopts the most conservative caching policy, requests for this resource will be re-validated to the server after April 9.

Negotiate the cache

When the cache time of a resource on the client expires, but the server has not updated the resource, if the resource is large, do we need to send the resource again?

The answer is no. In order to enable the client and server to verify whether the cache file is updated and improve the reuse rate of the cache, HTTP1.1 added several header fields to do this.

Last-Modified / If-Modified-Since / If-Unmodified-Since

1. When the client sends the request, the server will indicate the last modification time of the resourceLast-Modifie:GMTIs returned to the client along with the entity header field.
2. The client marks this information for the resource and sends the information to the server for inspection in the request message when it requests the resource again.
   • If the transfer time is the same as the final modification time of the resource on the server, it indicates that the resource has not been modified, and the status code 304 is returned with empty content, saving bandwidth and time.
   • If the two times are inconsistent, the server sends back the resource with a status code of 200, similar to the first request.

Last-modified Indicates the time when the requested resource was Last Modified. The format is GMT.

If-modified-since A cache check field whose value is the last-Modified value of the Last response header. If the value is the same as the current last-Modified value of the requested resource, a 304 status code response will be returned; otherwise, a 200 status code response will be returned. Request header field. The most common application scenarios:

• The most common application scenario is to update without specificETagThe cached entity of the tag.

If-unmodified-since the cache validation field, whose value is the last-Modified value of the Last response header, represents the feed is normally updated If the resource is not Modified after the specified time, otherwise the response of the 412 Precondition Failed state code is returned. Request header field. It is used in the following scenarios:

• Unsafe requests, such as using POST to request updatesWikiThe update is performed only if the document has not been modified.
• withIf-RangeFields, when used together, can be used to ensure that new fragment requests come from an unmodified document.

Last-Modified: Wed, 21 Oct 2015 07:28:00 GMT;// Less accurate than ETag, alternate mechanism. HTTP1.0.
If-Modified-Since: Wed, 21 Oct 2015 07:28:00 GMT;// When used with if-none-match, it (if-modified-since) is ignored.
If-Unmodified-Since: Wed, 21 Oct 2015 07:28:00 GMT;
Copy the code

Problems with Last-Modified:

1. The last-Modified tag is only accurate to the second level. If a file has been Modified more than once in a second, it will not accurately indicate the freshness of the file.

2. Some files may change periodically, but their contents do not change (only the modification time), but last-modified changes make the file uncacheable.

3. The server may not obtain the correct file modification time or the time on the proxy server may be inconsistent with that on the proxy server.

ETag / If-None-Match / If-Match

To address these last-Modified issues, HTTP1.1 also introduces the ETag entity header field.

The server calculates a unique identifier for the resource through an algorithm. When sending the resource response to the client, it adds the ETag: unique identifier to the entity header field and returns it to the client. Such as:

ETag: "x234dff";
Copy the code

The client keeps the ETag field and brings it to the server on the next request. The server can determine If the resource has been modified relative to the client by comparing the ETag sent by the client (sent as the value of the if-none-match field) with the ETag of the resource on its own server.

• If the server finds that the ETag does not match, it sends the new resource (including the new ETag, of course) directly to the client in a regular GET200 packet return.

• If the ETag is consistent, 304 is returned to inform the client to use the local cache.

ETag: "33a64df551425fcc55e4d42a148795d9f25f89d4"// Strong cache ETag requires that resources match exactly at the byte level.
ETag: W/"0815"// Weak ETag only requires that the resources remain semantically unchanged, but may change internally (the order of the tags in the HTML is adjusted, with a few extra Spaces).

If-None-Match: "bfc13a64729c4290ef5b2c2730249c88ca92d82d"// Request header field.
If-None-Match: W/"67ab43"."54ed21"."7892dd"
If-None-Match: * // The asterisk is a special value that can represent any resource. It is used only when uploading resources, usually using the PUT method, to check whether resources with the same identification ID have been uploaded.
    
If-Match: "bfc13a64729c4290ef5b2c2730249c88ca92d82d"// Request header field.
If-Match: W/"67ab43"."54ed21"."7892dd"
If-Match: * // The asterisk refers to any resource.
Copy the code

If-None-Match

This is used to determine whether cached resources are valid

• In the case of request methods GET and HEAD

  • ETagThe list does not match and the server returns a response with status code 200.
  • ETagThe list matches, and the server returns a response with status code 304.

• Other methods, especially PUT, set if-none-match used to * to generate files that are not known to exist in advance. This ensures that no similar upload operation has been performed before and prevents data loss from previous operations.

When used with if-modified-since, if-none-match has a higher priority (If the server supports it).

If-Match

This is used to determine whether a condition is met

• forGET 和 HEADMethod, used with the Range header field, can be used to ensure that the scope of a new request is the same resource as the scope of the previous request. ifETagSelect * from ‘Range Not Satisfiable’ where response = 416 (Range Not Satisfiable)
• For other methods, especiallyPUT.If-MatchHeader fields can be used to avoid update loss problems. It can be used to detect that the user wants to upload updates that do not overwrite those made after the original resource was retrieved. If the conditions of the request are not met, you need to return feed Failed 412 response.

Cache header comparison

The head	Advantages and Features	Weaknesses and problems
Expires	1, HTTP 1.0 artifacts, can be used in HTTP 1.0 and 1.1, easy to use. 2. Identify the expiration time by time.	1. The time is sent by the server (UTC). If the server time is inconsistent with the client time, problems may occur. 2, there is a version problem, before the expiration of the modification client is unknown.
Cache-Control	1. HTTP 1.1 product, which identifies expiration time with time interval, solves the problem of the relative time between Expires server and client. 2. There are a lot more options than Expires.	1. HTTP 1.1 content does not apply to HTTP 1.0. 2, there is a version problem, before the expiration of the modification client is unknown.
Last-Modified	1, there is no version problem, each request will go to the server for verification. If the last modification time is the same, 304 will be returned. If the last modification time is different, 200 and the resource content will be returned.	1. Whenever a resource is modified, it will be returned to the client regardless of whether the content has changed substantially. Such as periodic rewriting, in which the resource contains virtually the same data. 2. The time is used as the mark, and it is impossible to identify the situation where multiple changes are made within one second. 3, some servers do not know exactly when a file was last modified.
ETag	1, can more accurately determine whether the resource is modified, can identify the situation of multiple modifications within a second. 2, there is no version problem, each request back to the server for verification.	1. Performance loss is required to calculate ETag value. 2. In the case of distributed server storage, if the algorithms used to calculate the ETag are different, the browser may find that the ETag does not match when it obtains the page content from one server and verifies it on another server.

Cache priority

Pragma > Cache-Control > Expires.

ETag > last-Modified (when co-existing, last-Modified is an ETag backup).

The browser can create a space in memory or hard disk to store a copy of the requested resource.

When requesting a resource, the Cache is searched in order of Service Worker -> Memory Cache -> Disk Cache -> Push Cache.

200 From Memory cache Reads the cache from the memory without accessing the server. Store smaller files, and the cache resource is destroyed when the process is closed.

200 From Disk cache Reads the cache from the hard disk without accessing the server. Store large files, and cache resources remain after the process is shut down.

200 From Prefetch Cache When a preload or prefetch resource is loaded, both of them are stored in the HTTP cache. After the load is complete, if the resource can be cached, It is stored in the HTTP cache for later use; If the resource cannot be cached, it is stored in memory cache until it is used.

Cache request flow

User behavior

• When refreshing a web page using F5, the strong cache is skipped, but the negotiated cache is checked.
• When using Ctrl+F5 to force a web page refresh, download new resources directly from the server, skipping strong and negotiated caching.

HTTP proxy service

Proxy An intermediary inserted in the original communication link between a client and a server that is also a server but provides a proxy service.

The so-called proxy service refers to that the service does not generate content itself, but is in the middle position to forward upstream and downstream requests and responses. It has dual identities: when facing downstream users, it acts as a server, representing the source server to respond to client requests; When facing the upstream source server, it represents the client and sends requests on behalf of the client.

This focuses on the most common reverse proxy.

The role of agency

• Load balancing: Distributes external traffic to multiple source servers using algorithms to improve the overall resource utilization and performance of the system.
• Health check: Use heartbeat mechanisms to monitor back-end servers and remove them from the cluster to ensure high service availability.
• Security protection: protects the proxy back-end server, limits IP addresses or traffic, and defends against network attacks and overload.
• Encryption uninstallation: SSL/TLS is used to encrypt communication authentication on the external network, but not on the secure internal network, eliminating the cost of encryption and decryption.
• Data filtering: Intercepts data in the upper and lower rows and optionally specifies policy modification requests or responses.
• Content caching: The staging and reuse of server responses, which is closely related to cache control, will be discussed later.

Proxy-related header fields

It is used by the Via proxy server to identify the proxy. Via is a generic header field that can appear in either the request header or the response header.

In the case of multiple proxy nodes, the via header fields sent by the client and the response returned by the server are not in the same order:

The Via field only solves the problem of the client and the source server determining the existence of a proxy, and does not know the real information about the other side.

However, the IP address of the server is confidential because it is related to the Intranet security of the enterprise. Therefore, the IP address of the server is not known to the client. However, the server usually needs to know the real IP address of the client for access control, user portrait, and statistical analysis.

Unfortunately, the HTTP standard doesn’t define header fields For this purpose, but there are many de facto standards. The two most common header fields are X-Forwarded-For and X-real-IP.

• X-forwarded-for: This item is served by a chain

  Each time it passes through a proxy node, a message is appended to the field, but Via adds the proxy host name (or domain name) to it. X-forwarded-for adds the IP address of the requestor. Therefore, the IP address at the left of the field is the address of the client.

• X-real-ip: indicates only the client IP address

  Is another method to obtain the real IP address of the client. It simply records the IP address of the client without intermediate proxy information.

  If there is only one proxy between the client and the source server, the values of the two fields are the same.

The agency agreement

X – Forwarded – For faults:

X-forwarded-for The forwarder parses the HTTP header, which degrades the forwarder performance.

Another problem is that headers such as X-Forwarded-for must modify the original message, which in some cases is not allowed or even possible (For example, if the communication is encrypted using HTTPS).

Hence The PROXY Protocol, defined by The well-known PROXY software HAProxy, which is also a de facto standard and widely adopted (note that it is not RFC).

The proxy protocol has two versions, v1 and V2. V1 is similar to HTTP and is also plaintext, while V2 is binary.

V1 adds a line of ASCII text before HTTP packets. This line of text is very simple, starting with five uppercase letters PROXY, followed by TCP4 or TCP6, indicating the IP address type of the client. This is followed by the requester address, the responder address, the requester port number, the responder port number, and is ended with a carriage return newline (\r\n).

PROXY TCP4 1.11.1. 2.22.2. 55555 80\r\n
GET / HTTP/1.1\r\n
Host: www.xxx.com\r\n
\r\n
Copy the code

What do you think are the disadvantages of agency? How to avoid it in practical application?

The disadvantage of the agent is that the link length increases, which increases the response time. Therefore, the complex and time-consuming operations unrelated to the service should be minimized.

HTTP cache proxy

Cache control + proxy service = Cache proxy

After joining the cache, the proxy service needs to do two things when it receives the response data from the source server:

• The first, of course, is to forward the packet to the client.
• The second is to save the message to its ownCacheIn the water.

The next time the same request is made, the proxy server can send the 304 or cached data directly instead of retrieving it from the source server. This reduces client wait time and saves network bandwidth for the source server.

Cache control for the source server

Whether proxy caching is allowed for data:

• private: indicates that the cache can be saved only by clientsprivate.
• publicThat would allow

Re-validate after cache expiration: proxy-revalidate The proxy cache expiration must be validated by the client (must-revalidate).

Cache lifetime: S-maxage limits how long you can cache on the proxy server.

private, max-age = 5;
public, max-age = 5, s-maxage = 10;
max-age = 30, proxy-revalidate, no-transform;
Copy the code

Client cache control

only-if-cached

Indicates that only proxy cached data is accepted, not the response from the source server. If there is no cache on the proxy or the cache expires, a 504 (Gateway Timeout) should be returned to the client.

Vary: Accept-Encoding

For the server, there may be more than one version of the resource file, such as compressed and uncompressed, and different versions of the resource are often returned for different clients. For example, older browsers may not support decompression, so return an uncompressed version. The new browser supports compression and returns a compressed version, which saves bandwidth and improves experience. So how do you distinguish between this version? Vary is needed at this point.

By specifying Vary: accept-encoding, the server tells the proxy server that two versions of this resource need to be cached: compressed and uncompressed. In this way, the old browser and the new browser, through the proxy, get the uncompressed and compressed versions of the resource respectively, avoiding the embarrassment of both getting the same resource.

Vary:Accept-Encoding,User-Agent;
Copy the code

The Age and Date

The presence of this field indicates a hit on the proxy server cache. It refers to the time cached by the proxy server for the requested resource, in seconds. As follows:

Age:2383321
Date:Wed, 08 Mar 2017 16:12:42 GMT
Copy the code

The proxy server sent a request for the resource to the source server at 16:12:42 on March 8, 2017. The resource has been cached for 2383321 seconds.

Date refers to the time when the response was generated. When a request passes through the proxy server, the Date returned may not be the latest, and the proxy server will usually add an Age field to tell you how long the resource has been cached.

Cache practice

Static resource

Content that will never be modified: JS and CSS files, images, and binaries of any type belong to this category.

Forever, and I do say forever. It is common practice to specify a version number for static resources. Whenever they change, their URLS change.

Here are some simple rules for static resources:

• Embed a fingerprint in a file or path. Avoid using query strings for fingerprints. Also, make sure the generated URL is longer than 8 different characters.

<link rel = "Stylesheet" href="http://static.xxx.com/a_f02bc2.css">
<script src = "http://static.xxx.com/a_13cfa51.js"   
Copy the code

• Using these HTTP headers:

Cache-Control: public, max-age=31536000
Expires: (One year later today)ETag: (based on content generation) last-modified: (some time in the past)vary: Accept-Encoding
Copy the code

Dynamic resource

Different cache control Settings should be used for different application privacy and freshness requirements.

For non-private and constantly changing resources (think stock information), we should use the following:

Cache-Control: public, max-age=0 
Expires: (Current time)ETag: (based on content generation) last-modified: (some time in the past)vary: Accept-Encoding
Copy the code

The effect of these Settings is:

• These resources can be cached publicly (through browsers and proxy servers). Each time the browser uses these resources, the browser or proxy server checks to see if they have a newer version and downloads them.
• It is important to note that the browser has some flexibility in rechecking resource timeliness. Typically, when the user clicks the “Back/Forward” button, the browser does not reexamine these resource files, but instead uses the cached version.

If you need more control and need to tell the browser to reexamine the resource files even when the user clicks the Back/forward button, you can use:

Cache-Control: public, no-cache, no-store
Copy the code

Not all dynamic resources will immediately become obsolete. If they last at least 5 minutes, use:

Cache-Control: public, max-age=300
Copy the code

With this setup, the browser only rechecks after five minutes. Before that, cached content is used directly. If the stale content needs to be tightly controlled after 5 minutes, you can add the must-revalidate field:

Cache-Control: public, max-age=300, must-revalidate
Copy the code

For private or user-specific content, replace public with private to prevent the content from being cached by the proxy.

Cache-control: private,...Copy the code

Advantages and disadvantages of HTTP

advantages

• Simple, flexible, and easily extensible;
  • simpleThe basic message format isheader+bodyThe header information is also in simple text format with common English words.
  • Each of the core components of a flexible and extensible protocol — request methods, URIs, status codes, reason phrases, header fields — is not written down, allowing developers to customize, expand, or interpret them at will.
• With mature software and hardware environment, the application is very wide, is the Internet infrastructure;
• Stateless, can easily achieve clustering, expand performance;
• Plaintext transfers can be easily viewed or modified without the help of any external tools.

disadvantages

• Stateless and without memory, it cannot support transactions that require multiple consecutive steps, such as online shopping.
• Plaintext transmission, data completely visible to the naked eye, easy to be eavesdropped;
• It is insecure and cannot verify the identities of communication parties or determine whether packets are tampered.

HTTPS

HTTPS is a “very simple” protocol, which specifies the new protocol name “HTTPS” and the default port number 443. The rest of the request-response mode, packet structure, request method, URI, header field, connection management, and so on all follow HTTP.

HTTPS = HTTP + SSL/TLS

Symmetric and asymmetric encryption

Symmetric encryption

Symmetry-key Encryption means that the Key used for both Encryption and decryption is Symmetric. As long as the security of the key is ensured, the whole communication process can be said to be confidential.

Currently commonly used are:

• AES: The key length can be 128, 192, or 256.

• ChaCha20 Another encryption algorithm designed by Google, the key length is fixed at 256 bits.

Encrypted packet mode

Symmetric algorithms also have the concept of a grouping pattern, which allows the algorithm to encrypt plaintext of any length with a fixed-length key, turning small secrets (i.e. keys) into large secrets (i.e. ciphertext).

AEAD (Authenticated Encryption with Associated Data) adds authentication functions at the same time of Encryption. GCM, CCM and Poly1305 are commonly used.

The advantages and disadvantages

• Advantages: fast operation speed;
• Disadvantages: The key cannot be securely transferred to the communicator, so key exchange is a problem.

Asymmetric encryption

It has two keys, a public key and a private key. The two keys are different (asymmetric) and the public key can be made public for anyone to use, while the private key must be kept strictly secret.

Public and private keys have a special unidirectional property. Although both can be used for encryption and decryption, the encrypted public key can only be decrypted with the private key, and vice versa.

Asymmetric encryption can solve the problem of key exchange.

Currently commonly used are:

• The security of RSA is based on the mathematical problem of integer decomposition. The product of two large prime numbers is used as the material to generate the key. It is very difficult to calculate the private key from the public key. Now the recommended key length is 2048.
• ECC (Elliptic Curve Cryptography) is based on the mathematical problem of discrete logarithms of Elliptic curves. It uses specific Curve equations and basis points to generate public and private keys. The subalgorithm ECDHE is used for key exchange, and ECDSA is used for digital signature.

The advantages and disadvantages

• Advantages: The public key can be transmitted to the communication sender more securely.
• Disadvantages: Slow operation speed.

Mixed encryption

Although asymmetric keys do not have key exchange problems, they are slow to run. If you use asymmetric encryption alone, the security is guaranteed, but the communication speed is snail’s pace, and the utility becomes zero.

Therefore, the hybrid encryption that combines symmetric encryption and asymmetric encryption is produced. They learn from each other and can encrypt and decrypt efficiently and exchange keys safely.

1. At the beginning of communication, asymmetric algorithms such as RSA and ECDHE are used to solve the key exchange problem first.
2. Then generate the session key used by the symmetric algorithm with random number, and encrypt it with public key. Because session keys are short, usually only 16 or 32 bytes, it doesn’t matter if they are slower.
3. The peer party decrypts the ciphertext with the private key and extracts the session key. In this way, the secure exchange of symmetric keys is realized, and asymmetric encryption is no longer used, but symmetric encryption is used.

Digital signature and certificate

Can we keep the message secure by encrypting it?

Consider the following:

Asymmetric encryption algorithms are public, and anyone can generate a pair of public and private keys.

When the server returns public key A1 to the client, the middleman replaces it with his own public key B1 and sends it to the browser.

At this time, the browser knows nothing about it. It uses public key B1 to encrypt key K and sends it out, which is intercepted by the middleman. The middleman decrypts the key WITH his private key B2, obtains the key K, and then encrypts the key A1 of the server and sends it to the server to complete the communication link.

To ensure security, we need not only encryption, but also message integrity and identity authentication.

The algorithm

A Digest Algorithm is also known as a Hash Function. Mapping a large space to a small space can be used to check the integrity of data due to the unidirectional and avalanche effect on input.

The digest algorithm recommended by TLS is SHA-2. Sha-2 is a series of digest algorithms, including SHA224, SHA256, and SHA384, which can generate 28-byte, 32-byte, and 48-byte digests respectively.

However, it does not have confidentiality and encrypts messages and digests with a session key in a hybrid encryption system, a term called hashed message Authentication code (HMAC).

A digital signature

Now there’s a loophole, the two endpoints of communication, which is how do you prove it’s you? How does the server prove to be a server?

The private key in asymmetric encryption, using the private key and the digest algorithm, can achieve digital signature, identity authentication and non-repudiation.

However, because of the low efficiency of asymmetric encryption, the private key encrypts only the abstract of the original text

The private key here is you have to have a private key, the server has to have a private key, you exchange public keys with each other, unless your private key is compromised, authentication and non-repudiation are guaranteed.

Digital certificates and CAS

By now, using a combination of symmetric encryption, asymmetric encryption and digest algorithms, we have achieved the four major features of security, is it not perfect?

No, there is also a public key trust issue. Since anyone can publish a public key, how do you ensure that it is not forged?

Find a trusted third party that is recognized as “the starting point of trust and the end point of recursion” to build the trust chain of the public key. This is a Certificate Authority (CA), which uses the CA’s private key to sign your public key (including the serial number, purpose, issuer, validity period, etc.) and your public key to package and sign again, to form a digital Certificate.

So how does CA prove itself? It’s all about the chain of trust. The smaller CA can have the larger CA sign authentication, but the Root CA at the end of the chain has to prove itself.

In other words, my public key is signed by the CA’s private key, so I need to get the CA’s public key to decrypt, decryption success can prove that it is not forged, then the trust chain problem, the final solution is Root CA. This is called a self-signed Certificate or Root Certificate. With this Certificate system, the Root Certificate of each CA is built into the operating system and browser

That is, if your public key is not issued by the CA, it must be installed in the root certificate store of your system for the browser to consider it secure.

TLS

TLS protocol components

Record deal

The Record Protocol specifies the basic unit for sending and receiving data in TLS: Record. It’s a bit like segment in TCP, where all the other subprotocols need to be sent via the logging protocol. However, multiple records can be sent at once in a TCP packet and do not need to return an ACK as TCP does.

Alarm agreement

The Alert Protocol’s job is to send an Alert message to the other party, sort of like the status code in HTTP. For example, protocol_version does not support older versions, bad_certificate is a certificate problem, and the other party can choose to continue or terminate the connection immediately after receiving an alert.

Handshake protocol

Handshake Protocol is the most complex subprotocol of TLS. It is more complex than TCP’S SYN/ACK. During Handshake, the browser and server negotiate TLS version number, random number, password suite, and other information, and then exchange certificate and key parameters. Finally, the two parties negotiate the session key, which is used in the subsequent hybrid encryption system.

Change password specification protocol

The last is the Change Cipher Spec Protocol, which is simply a “notification” that all subsequent data will be encrypted. So, on the other hand, before that, the data was clear text. (This protocol was removed in TLS 1.3 and may still exist for compatibility with older versions of TLS).

The heartbeat protocol

The Heartbeat extension provides a new protocol for TLS/DTLS, allowing keep-alive functionality to be used without the need for renegotiation. The Heartbeat extension also provides the basis for path MTU (PMTU) discovery (this is a new addition to TLS 1.3, which did not have this protocol prior to TLS 1.3). .

TLS 1.2 Connection process parsing

ECDHE handshake process

1. After a TCP connection is established, the browser sends a Client Hello message, which greets the server. It contains the TLS version number of the Client, the supported cipher suite, and a Client Random number for subsequent generation of the session key.

   • When establishing connections using TLS, browsers and servers need to select a set of encryption algorithms to secure communication. A combination of these algorithms is called a cipher suite.

2. The server receives this and immediately returns server_random to confirm the TLS version number and the password suite ECDHE used.

   • To prove its identity, the Server sends the Server Certificate to the client.
   • Because the Server has chosen the ECDHE algorithm, it will send the Server Key Exchange message after the certificate, which contains the elliptic curve public Key (Server Params) to implement the Key Exchange algorithm, plus its own private Key signature authentication.
   • Followed by a Server Hello Done message.

3. The browser receives the certificate and verifies the digital certificate and signature.

   • The Client also generates an elliptic curve public Key (Client Params) according to the requirements of the cipher suite and sends Client Key Exchange messages to the server.

   • Client Random + Server Random Obtains the pre-master through ECDHE algorithm.

   • Client Random, Server Random, and pre-Master generate Master Secret for the encrypted session.

   master_secret = PRF(pre_master_secret, "master secret",
                       ClientHello.random + ServerHello.random) / / Master Secret formula
   Copy the code

   The PRF here is the pseudo-random number function, which is based on the last parameter in the cipher suite, such as SHA384, and reinforces the randomness of Master Secret again through the digest algorithm.

   The master key has 48 bytes, but it is not the final session key for communication. PRF will also be used to extend more keys, such as client_write_key for client sending, server_write_key for server sending, and so on. Avoid security risks associated with using only one key.

   With the master key and the derived session key, the handshake is almost complete. The client sends a Change Cipher Spec message and a Finished message to summarize all data and encrypt it for authentication.

4. In the same way, the server sends Change Cipher Spec and Finished messages. Both parties verify that encryption and decryption are OK, and the handshake ends. After that, the server sends and receives encrypted HTTP requests and responses.

Cipher suite

TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 // Key exchange algorithm + signature algorithm + symmetric encryption algorithm + digest algorithm
Copy the code

• The key negotiation algorithm uses ECDHE.
• The signature algorithm uses RSA.
• The communication after handshake uses AES symmetric algorithm, the key length is 256 bits, and the packet mode is GCM.
• The algorithm uses SHA384.

RSA handshake procedure

The general process remains the same, except that the pre-master does not need to use an algorithm to generate a random number. Instead, the Client directly generates a random number, encrypts it with the server’s public Key, and sends a Client Key Exchange message to the server. The server uses the private key to decrypt, so that the two sides have realized the sharing of three random numbers, and the master key can be generated.

1. The client is connected to the server.

2. The server sends the CA certificate to the client.

   Send a CA chain without ROOT certificate.

3. The client verifies the reliability of the certificate.

   Parsing the CA chain, authenticating with the cas on the chain, layer by layer until you find the root certificate, you can determine that the certificate is trusted.

4. The client retrieves the public key from the CA certificate.

5. The client generates a random key K and encrypts it with the public key to get K *.

6. The client sends k* to the server.

7. The server receives k* and decrypts it with its own private key to obtain K.

8. At this point, both parties have obtained the key K, and the negotiation is completed.

9. Then use the key K * to encrypt the information.

The differences between RSA and ECDHE handshake processes:

• The RSA key negotiation algorithm does not support forward secrecy, and the ECDHE key negotiation algorithm does support forward secrecy.
• Using RSA key negotiation algorithm, TLS can transmit application data after completing four handshakes. For ECDHE algorithm, the client can send encrypted HTTP data in advance without waiting for the server’s last TLS handshake, saving the round-trip time of a message.
• If ECDHE is used, the Server Key Exchange message is displayed during the second TLS handshake, but the message is not displayed during the RSA handshake.

TLS 1.3 feature Analysis

Strengthen the security

Cipher suite name	code
TLS_AES_128_GCM_SHA256	{0 x13, 0 x01}
TLS_AES_256_GCM_SHA384	{0 x13, 0 x02}
TLS_CHACHA20_POLY1305_SHA256	{0 x13, 0 x03}
TLS_AES_128_GCM_SHA256	{0 x13, 0 x04}
TLS_AES_128_GCM_8_SHA256	{0 x13, 0 x05}

Handshake analysis

1. The browser still sends a Client Hello.

   • Because messages from 1.3 are compatible with 1.2, the leading version number, the supported cipher suite, and the Client Random structure are all the same (except that the Random number is 32 bytes in this case).

2. The Server also returns a Server Hello message after receiving Client Hello, and still provides a Server Random number and the selected password suite.

   • Verify that TLS1.3 is used in the SUPPORted_versions file, and add the corresponding public key parameter to the key_share extension.

With only two messages exchanged, the client and server get four shared messages: Client Random and Server Random, Client Params and Server Params, both sides can use ECDHE respectively to calculate the pre-master, HKDF is then used to generate Master Secret, which is much more efficient than TLS1.2.

In TLS1.3, a Certificate Verify message is added. The Certificate Verify message uses the server private key to add a signature to the handshake data, such as the curves, packages, and parameters, which is similar to the Finished message. But because it is a private key signature, it strengthens the identity authentication and anti – tampering.

After the two Hello messages, the client authenticates the server certificate and sends a Finished message to complete the handshake and send and receive HTTP packets.

HTTP/2

HTTP / 1 x defects

HTTP/1.x implementation simplicity comes at the expense of performance:

• Clients need to use multiple connections to achieve concurrency and reduce latency;
• Does not compress the request and response headers, resulting in unnecessary network traffic;
• No valid resource priority is supported, resulting in low utilization of underlying TCP connections.

The head of compression

HTTP/1.1 comes with a lot of information in the header and is sent repeatedly each time.

HTTP/2.0 requires both the client and server to maintain and update a table of previously seen header fields, thereby avoiding repeated transfers.

HTTP/2 does not use the traditional compression algorithm, but developed a special HPACK algorithm, in the client and server to establish a “dictionary”, index number to represent the repeated string, but also uses Huffman encoding to compress integers and strings, can reach 50%~90% high compression rate.

Binary frame

HTTP/2.0 divides packets into HEADERS frames and DATA frames, both of which are in binary format.

During communication, only one TCP connection exists, which hosts any number of two-way streams.

• A data Stream has a unique identifier and optional priority information for carrying two-way information.
• A Message is a complete set of frames corresponding to a logical request or response.
• Frames are the smallest unit of communication. Frames from different data streams can be sent interlaced and then reassembled according to the data stream identifier of each Frame header.

Server push

When a client requests a resource, HTTP/2.0 sends the related resource to the client, so that the client does not need to make the request again. For example, when a client requests a page.html page, the server sends script.js and style.css related resources to the client.

CORS cross-domain

When a resource requests a resource from a different domain or port than the server on which the resource itself resides, the resource makes a cross-domain HTTP request.

For security reasons, browsers restrict cross-source HTTP requests from within scripts. For example, the XMLHttpRequest and Fetch apis follow the same origin policy. This means that Web applications using these apis can only request HTTP resources from the same domain that loaded the application, unless CORS headers are used.

(Translator’s note: Cross-domain does not necessarily mean that the browser restricts cross-site requests. It is also possible that cross-site requests can be made normally, but the return result is blocked by the browser. The best example of this is the CSRF cross-site attack principle, where requests are sent to back-end servers regardless of whether they are cross-domain! Note: Some browsers do not allow cross-domain access to HTTP from HTTPS domains, such as Chrome and Firefox, and these browsers are a special case where they intercept requests before they are sent.

The Web Applications Working Group, part of the W3C, has recommended a new mechanism, Cross-Origin Resource Sharing (CORS). This mechanism enables Web application servers to support cross-site access control, making it possible to transfer data securely across sites. It is important to note that this specification is for API containers (such as XMLHttpReques or Fetch) to mitigate the risk of cross-domain HTTP requests. **CORS requires both client and server support. Currently, all browsers support this mechanism. 支那

The Cross-Origin Sharing Standard allows cross-domain HTTP requests to be used in the following scenarios:

• The above mentioned byXMLHttpRequestOr cross-domain HTTP requests initiated by Fetch.
• Web fonts (cross-domain font resources are used in CSS via @font-face), so Web sites can publish TrueType font resources and only allow authorized sites to make cross-site calls.
• WebGL map
• Using drawImage willImages/videoDraw the canvas
• Style sheets (using CSSOM)
• Scripts (unhandled exception)

CORS can be divided into simple request, pre-request and request with credential information.

1. Simple requests

Some requests do not trigger CORS precheck requests. This article refers to such a request as a “simple request,” noting that the term is not part of the Fetch (where CORS is defined) specification. A request is considered a “simple request” if all of the following conditions are met:

1. Use one of the following methods:

• GET
• HEAD
• POST

2. The Fetch specification defines a set of header fields that are CORS safe. No header fields other than this set may be set artificially. The set is:

• Accept
• Accept-Language
• Content-Language
• Content-type (Additional restrictions need to be noted)
• DPR
• Downlink
• Save-Data
• Viewport-Width
• Width

3. The value of the content-type is limited to one of the following:

• text/plain
• multipart/form-data
• application/x-www-form-urlencoded

4. None of the XMLHttpRequestUpload objects in the request have any event listeners registered; The XMLHttpRequestUpload object can be accessed using the xmlHttprequest.upload attribute.

5. No ReadableStream object is used in the request.

In short, there are two key points to remember:

1. Use only GET, HEAD, or POST request methods. If POST is used to send data to the server, the content-Type must beapplication/x-www-form-urlencoded.multipart/form-data 或 text/plainOne of them.
2. Custom request headers (such as x-Modified) are not used.

For example:

For example, suppose the web application at http://foo.example wants to access the resources at http://bar.other. The following JavaScript generation
// Code should be executed on foo.example:
var invocation = new XMLHttpRequest();
var url = 'http://bar.other/resources/public-data/';
function callOtherDomain() {
  if(invocation) {    
    invocation.open('GET', url, true); invocation.onreadystatechange = handler; invocation.send(); }}Copy the code

// Let's see what the browser sends to the server and what the server returns to the browser in this scenario:
GET /resources/public-data/ HTTP/1.1
Host: bar.other
User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9.1b3pre) Gecko/20081130 
Minefield/3.1b3pre
Accept: text/html,application/xhtml+xml,application/xml; q=0.9, */ *; Q = 0.8 Accept - Language: en - us, en. Q =0.5 Accept-encoding: Gzip, Deflate Accept-Charset: ISO-8859-1, UTF-8; Q = 0.7 *; Q = 0.7 Connection: keep - the alive Referer: http://foo.example/examples/access-control/simpleXSInvocation.html Origin: http://foo.example // This request comes from http://foo.exmaple. HTTP/1.1 200 OK Date: Mon, 01 Dec 2008 00:23:53 GMT Server: Apache/2.0.61 Access-Control-Allow-Origin: * // This indicates that the server accepts cross-site requests from any site. If the parameter is set to http://foo.example. Other sites will not be able to access http://bar.other's resources across the site. Keep-Alive: timeout=2, max=100 Connection: Keep-Alive Transfer-Encoding: chunked Content-Type: Application/XML // The server returns information to the browserCopy the code

The request returns relevant response information if:

• If the resource is publicly accessible (like any HTTP resource that allows GET Access), returning the Access-Control-Allow-Origin:* header is sufficient, except for some requests that require Cookies and HTTP authentication information.

• If resource Access is restricted based on the same domain name, or if the resource to be accessed requires credentials (or set credentials), then it is necessary to filter ORIGIN in the request header information, or at least respond to the source of the request (e.g. Access-control-allow-origin: http://arunranga.com).

  In addition, the access-Control-allow-Credentials: True header is sent, as discussed in a later section.

2. Advance request

Unlike the simple request described above, “Requests to Precheck” requires that you first make a precheck request to the server using the OPTIONS method to know whether the server will allow the actual request. The use of precheck requests prevents cross-domain requests from having unexpected impacts on the server’s user data.

Precheck requests should be sent first when any of the following conditions are met:

1. Any of the following HTTP methods are used:

• PUT
• DELETE
• CONNECT
• OPTIONS
• TRACE
• PATCH

2. A header field other than the header field set that is safe for CORS was set artificially. The set is:

• Accept
• Accept-Language
• Content-Language
• Content-Type (but note the additional requirements below)
• DPR
• Downlink
• Save-Data
• Viewport-Width
• Width

3. The value of the content-type is not one of the following:

• application/x-www-form-urlencoded
• multipart/form-data
• text/plain

4. The XMLHttpRequestUpload object in the request registers any number of event listeners.
5. A ReadableStream object is used in the request.

Unlike the simple request discussed above, a “pre-request” requires that an OPTIONS request be sent to the destination site to determine whether the cross-site request is safe and acceptable for the destination site. This is done because cross-site requests can cause damage to the data at the destination site. A request is treated as a pre-request when:

1. Requests are made in methods other than GET, HEAD, or POST. Alternatively, use POST but request data asapplication/x-www-form-urlencoded.multipart/form-dataortext/plainData types other than. For example, send data of type POSTapplication/xmlortext/xmlRequest for XML data.
2. Use custom request headers (such as adding something like X-Pingother).

Here’s an example:

var invocation = new XMLHttpRequest();
var url = 'http://bar.other/resources/post-here/';
var body = '{C}{C}{C}{C}{C}{C}{C}{C}{C}{C}Arun';
function callOtherDomain(){
  if(invocation){
    invocation.open('POST', url, true);
    invocation.setRequestHeader('X-PINGOTHER'.'pingpong');
    invocation.setRequestHeader('Content-Type'.'application/xml'); invocation.onreadystatechange = handler; invocation.send(body); }}Copy the code

As above, a POST request is created with XMLHttpRequest, a custom request header (X-Pingother: Pingpong) is added to the request, and the data type is specified as Application/XML. Therefore, the request is a cross-site request in the form of a “pre-request”. The browser sends a “pre-request” using an OPTIONS. Based on the request parameters, Firefox 3.1 decides that it needs to send a “pre-request” to see if the server accepts the actual subsequent request. OPTIONS are HTTP/1.1 methods used to obtain more information on the server side, which should not affect server data. Along with the OPTIONS request, the following two request headers are sent:

Access-Control-Request-Method: POST
Access-Control-Request-Headers: X-PINGOTHER
Copy the code

Assume that the server successfully responds with the following information:

Access-Control-Allow-Origin: http://foo.example // indicates that the server allows the request for http://foo.example
Access-Control-Allow-Methods: POST, GET, OPTIONS // Indicates that the server accepts POST, GET, and OPTIONS requests
Access-Control-Allow-Headers: X-PINGOTHER // Pass a list of acceptable custom headers. The server also needs to set one corresponding to the browser. Otherwise, a Request header field x-requested-with is not allowed by access-Control-allow-headers in preflight response is reported
Access-Control-Max-Age: 1728000 // Tells the browser how long the result of this "pre-request" will be valid. In the example above, 1728,000 seconds represents a period of 20 days in which the browser can process cross-site requests to the server without sending a "pre-request" and only need to make judgments based on the results.
Copy the code

3. Request with credential information

An interesting feature of Fetch with CORS is the ability to send identity credentials based on HTTP cookies and HTTP authentication information. In general, browsers do not send identity credentials for cross-domain XMLHttpRequest or Fetch requests. If you want to send credential information, you need to set some special flag bit for XMLHttpRequest.

In this case, one of the scripts in foo.example makes a GET request to bar.other and sets Cookies:

var invocation = new XMLHttpRequest(); var url = 'http://bar.other/resources/credentialed-content/'; function callOtherDomain(){ if(invocation) { invocation.open('GET', url, true); invocation.withCredentials = true; invocation.onreadystatechange = handler; invocation.send(); }}Copy the code

Line 7 sets the withCredentials flag of XMLHttpRequest to true to send Cookies to the server. Because this is a simple GET request, the browser does not make a “precheck request” for it.

However, if the response on the server side does not carry access-Control-allow-credentials: true, the browser will not return the response content to the sender of the request.

Assume that the server successfully responds with the following information:

Access-Control-Allow-Origin: http://foo.example
Access-Control-Allow-Credentials: true
Set-Cookie: pageAccess=3; expires=Wed, 31-Dec-2008 01:34:53 GMT
Copy the code

If the bar.other response header does not contain access-Control-allow-credentials: true, the response is ignored.

Note: When sending a response to a request with withCredentials, the server must specify the domain name to allow the request. Do not use *.

In the example above, if the response header is access-Control-allow-Origin: *, the response will fail. In this example, because the access-Control-allow-origin value is foo.example, the requested domain name, the client returns the content with the credential information. Also note that more cookie information is created as well.

CORS versus JSONP

• JSONP can only implement GET requests, whereas CORS supports all types of HTTP requests.
• With CORS, developers can make requests and get data using plain XMLHttpRequest, which has better error handling than JSONP.
• JSONP is mainly supported by older browsers, which tend not to support CORS, as most modern browsers already do.
• Compared with JSONP, CORS is undoubtedly more advanced, convenient and reliable.

reference

Detailed explanation of OSI seven layer model

Perspective HTTP protocol

MDN HTTP tutorial

Hardcore! 30 illustrations of common HTTP interview questions

HTTP

Graphic HTTP caching

Summary of HTTP cache control

Browser cache mechanism analysis

Understand browser caching strategies inside out

HTTP cache

Abnormal static resource cache and update

HTTPS,

TLS & DTLS Heartbeat Extension

HTTP Basics