HTTP protocol principles + practice

HTTP protocol and its history

Classical five-layer network model

The physical layer mainly defines how physical devices transmit data. Simply speaking, the physical layer is computer hardware, network port, network cable and optical cable.

The data link layer establishes data link connections between communicating entities.

The network layer creates logical links for data transfer between nodes, that is, the client accesses the logical relationship of how the server finds the address of the server.

The transport layer provides users with reliable end-to-end services. How clients and servers transfer data, transfer data, and assemble data are defined at the transport layer. There are two protocols: TCP and UDP.

The application layer provides a lot of services for the application software. HTTP protocol is implemented on this layer. As long as we use HTTP related tools, we can help us transfer data. Built on THE TCP protocol, so the transmission mode is to complete on the basis of TCP, IP protocol.

HTTP protocol history

HTTP / 0.9

There is only one command, GET, and no HEADER to describe the data. The server closes the TCP connection after sending.

HTTP / 1.0

Added commands such as POST and PUT.
Added status code(the status of the server to process the request) and header-related content.
Multi-character set support, multi-part sending, permissions, caching, and more.

HTTP / 1.1

Keep Alive supports persistent connections. In HTTP/1.0, an HTTP request creates a TCP connection between the client and the server, which closes the TCP connection when the server returns the content. Performance is much better if the TCP connection is not closed after being established. That’s what long connections do.
Added pipeline, can send multiple requests on the same TCP connection, but for the server incoming requests are in order to return the content of the request.
Host has been added so that you can run multiple Web services on the same physical server.

HTTPS

HTTPS is simply a secure version of the HTTP protocol. The actual usage is not that different from HTTP/1.1.

HTTP2

HTTP/1.1 HTTP/1.1 HTTP/1.1 HTTP/1.1 HTTP/1.1 HTTP/1.1 HTTP/1.1 HTTP/1.1 HTTP/1.1 HTTP/1.1 HTTP/1.1 HTTP/1.1 HTTP/1.1 All requests can be made concurrently on this TCP connection. Improve transmission efficiency.
Header compression. In HTTP/1.1, many HTTP headers must be sent and returned in a complete manner every time a request is sent or returned. However, in fact, much of this header information is stored in the form of a string, thus occupying a large amount of bandwidth, and header compression effectively reduces bandwidth usage.
Push function in HTTP/1.1 can only be initiated by the client (active party) request, server (passive party) response to the request, return content. HTTP2 adds push functionality that allows servers to initiate data transfers.

Principle of HTTP

HTTP is an application layer protocol, composed of requests and responses, is a standard client-server model, is a stateless protocol. HTTP is usually carried on top of TCP, and sometimes on top of TLS or SSL (HTTPS).

HTTP has two concepts: request and response. Request and response are packets that need to pass through a transport channel. This transport channel creates a connection in TCP between the client and the server. HTTP requests are sent on this connection. It is possible to send multiple HTTP requests over a TCP connection. In HTTP/1.0, a TCP connection was created when an HTTP request was created, and the server responded by closing the TCP connection. In HTTP/1.1, you can enable keep Alive to keep the connection alive. After the first request is completed, the second request can be sent over the same TCP connection. Requests over a TCP connection can be made concurrently in HTTP2, so you only need to create a TCP connection.

An HTTP request can be divided into four steps:

1) First, the client and server need to establish a TCP connection.

2) After establishing a TCP connection, the client sends a request to the server in the format of a uniform resource Identifier (URL), protocol version number, followed by MIME information including request modifiers, client information, and possible content.

3) After receiving the request, the server will give the corresponding response information in the form of a status line, including the protocol version number of the information, a successful or error code, followed by MIME information including server information, entity information and possible content.

4) The client receives the information returned by the server and displays it to the user through the browser. Then the client disconnects from the server.

The request process is followed by three handshakes.

First the client initiates a request to create a connection packet, which is sent to the server. The packet contains an identifying bit SYN=1, which indicates the packet to create the request, and Seq=X(a number, usually 1). After receiving the packet, the server knows that a client wants to establish a connection with it. The server opens a TCP socket port and returns a packet to the client. The packet also contains SYN=1, ACK=X+1, where X is Seq,Seq=Y, from the client. The client gets the packet and knows that the server has given it permission to create the TCP connection. The client sends ACK=Y+1,Y is the Seq of the server, and Seq=Z to the server. This is how you create a TCP connection. So why go through a three-way handshake? This is to prevent the server from opening useless connections. Because network transmission is delayed, it can be transmitted over very long distances via optical fibers and various proxy servers. If the client sends SYN=1,Seq=X during transmission, the server directly creates a TCP connection and returns the content to the client. When the packet is lost due to network transmission, the client cannot receive the message returned by the server. The client may set a timeout period. After the timeout period expires, the client closes the TCP connection request and initiates a new connection request. If there is no third handshake, the server does not know whether the client has received the message and whether to create or close the TCP connection. The port of the TCP connection is left open, waiting for the client to send the actual HTTP request. The connection to the server is wasted, so it takes three handshakes to confirm that the client and the server are aware in time that the packet has not been received due to network or other reasons, and the connection is closed without waiting. The three-way handshake is actually designed to avoid some of the server overhead associated with these network traffic delays. The following uses Wireshark to view the details about the three-way handshake packets used to create TCP connections.

In the figure above, you can clearly see the interaction between the client browser (IP: 180.97.163.244:13789) and the server (port 80). The three-way handshake between the client and the server is circled in red (13789->80,80->13789,13789->80) :

1) The client sends a connection request to the server. This is the first step of TCP three-way handshake (13789->80).

2) the server responds to the client’s request with the identifier :(SYN, ACK), Seq=Y (Y = 0), ACK=X+1 (1). This is the second step of the three-way handshake (80->13789).

3) The browser responds to the server’s confirmation and the connection is successful. Is ACK, Seq=Z(Z is 1), ACK=Y+1 (is 1). This is the third step of the three-way handshake (13789->80).

Once the handshake is complete, the client can send the actual HTTP request to the server.

URI – URL and URN

Uris — Uniform Resource Identifiers, containing urls and UrNs. Used to identify a fixed location on the Internet where resources are located. We can find this resource through a link. The main purpose of the HTTP protocol is to find a resource and get it somehow.

URL — Unified resource Locator user:[email protected]:80/path? Query =… Is used to locate a specific page of a Web site. The meaning of each part is as follows:

Http:// is scheme, which defines the protocol used to access the resource. When accessing services through different protocols, the resolution methods are different. It specifies how services and senders transfer and parse data.
User :pass@ : Indicates that resources can be accessed only by a specific identity. The user and password must be added for authentication. But you don’t really need it in Web development these days.
Host.com: Locate the server where resources reside (IP address). The domain name can be an IP address or a domain name. The domain name is resolved to the corresponding IP address through DNS.
80: ports, each server has many ports, can run many Web services,web services can listen to different ports. The port is used to locate one of the web services on the server that we found. Find the physical server first and then the Web server. Without a port, port 80 is accessed by default.
/path: routing, web services store a lot of content, routing to locate the content we are looking for.
Query =string: search parameter.
#hash: If the resource content is a document and the content is very large, hash represents a fragment of the document. It is time for development to use hashes as anchor points frequently.

URN — a permanent uniform resource locator that can still be found after the resource has been moved. There is no mature solution yet.

HTTP Packet Format

First line of request message:

HTTP methods: Used to define operations on resources. GET, POST, PUT, and DELETE are commonly used.
Url: The address of the resource to be requested.
Protocol version: HTTP/1.0

Request packet Header: Accept, accept-language See the details later.

Response message:

Protocol version: HTTP/1.0
HTTP code: defines the server’s response to the request. Each section of code has its own semantics.
1. 100 to 199 indicates that data is returned only when the operation continues.
2. 200 to 299 indicates that the operation is successful.
3. 300 to 399 indicates redirection.
4. The requests sent from 400 to 499 are faulty.
5. 500 to 599 indicates a server problem.
HTTP header: content-type, content-lengt see the details below.
Main part

Overview of HTTP features

Restriction and resolution of CORS cross-domain requests

Simulation of cross-domain

server.js

const http = require('http');
const fs = require('fs');
http.createServer((req,res) => {
    console.log('req come', req.url);
    const html = fs.readFileSync('test.html','utf8');
    res.writeHead(200,{
        'Content-type': 'text/html'
    })
   res.end(html);
}).listen(8888);
Copy the code

server2.js

const http = require('http');
http.createServer((req,res) => {
    console.log('req come', req.url);
    res.end('1122');
}).listen(8887);
Copy the code

test.html

<! DOCTYPE html> <html lang="en"> <head> <meta charset="UTF-8"> <meta name="viewport" content="width=device-width, <meta HTTP-equiv =" x-UA-compatible "content=" IE =edge"> <title>Document</title> </head> <body> </body>  <script> var xhr = new XMLHttpRequest(); XHR. Open (' GET 'and' http://127.0.0.1:8887/ '); xhr.send(); </script> </html>Copy the code

The access-Control-allow-Origin header is not set. The access-Control-allow-Origin header is not set. The access-Control-allow-Origin header is not set. Access-control-allow-origin is the header that the server allows for cross-domain requests.

Add set access-Control-Allow-Origin header to server2.js to resolve cross-domain issues.

res.writeHead(200,{
     'Access-Control-Allow-Origin': '*'
 })
Copy the code

Accessing localhost:8888 again will not return an error. The request is sent and the server returns the data.

If access-Control-allow-Origin is not available and is set to Allow cross-domain Origin, the browser intercepts the returned content. The above error is reported on the command line. Access-control-allow-origin is set to ‘*’, indicating that any domain name can Access the service. This is not secure. You can set it to Allow a domain name to cross the domain. Browsers, however, allow link, IMG, and script tags to write paths on the tags to load content across domains. So the principle that JSONP can achieve cross-domain is to load a link on the script tag, which accesses the server. Because the content returned by the server is controllable, the script tag code that can be written in the content returned by the server is an executable JS code. Then jSONP is called to initiate what was set for us earlier in the request. Note that the method defined by the callback parameter needs to be defined by the front and back end. In fact, the whole process of JSONP is similar to declaring a function on the front end and returning to execute the function on the back end. The process of executing a function with the required data in its parameters is actually quite straightforward. The following is an example of how jSONP can be implemented across domains.

<! DOCTYPE html> <html lang="en"> <head> <meta charset="UTF-8"> <meta name="viewport" content="width=<s>, <meta HTTP-equiv =" x-UA-compatible "content=" IE =edge"> <title>Document</title> </head> <body> <button Type = "submit" id = "BTN" > click me < / button > < script SRC = "https://cdn.bootcss.com/jquery/3.3.1/jquery.min.js" > < / script > <script> $('#btn').click(function(){ var frame = document.createElement('script'); frame.src = 'http://localhost:8887/user-info? callback=func'; $('body').append(frame); }); Function func(res){alert(res.message+res.name+' you have '+res.age+' old '); } </script> </body> </html> server.js const http = require('http'); const fs = require('fs'); http.createServer((req,res) => { console.log('req.url',req.url); const html = fs.readFileSync('test.html','utf8'); if (req.url === '/') { res.writeHead(200, { 'Content-Type': 'text/html', }) res.end(html); } }).listen(8888); server1.js const http = require('http'); const fs = require('fs'); const url = require('url'); http.createServer((req,res) => { var parseObj = url.parse(req.url, true); var callback = parseObj.query.callback; if (parseObj.pathname === '/user-info') { res.writeHead(200, { 'Content-Type': 'text/javascript', }) let data = { message: 'success! ', name: 'the question', age: 18 } data = JSON.stringify(data) res.end(`${callback}(` + data + `)`); } }).listen(8887);Copy the code

The server returns data:

Click the button to get the data returned by the server:

In this way, the localhost:8888 page will not be cross-domain access localhost:8887.

CORS cross-domain constraints and pre-request validation

We mentioned above that cross-domain problems can be solved by setting the Access-Control-Allow-Origin header, but not all cases can be solved by setting this header. Here’s a look at some of the other limitations of browser cross-domain requests. Let me give you an example

<! DOCTYPE html> <html lang="en"> <head> <meta charset="UTF-8"> <meta name="viewport" content="width=device-width, <meta HTTP-equiv =" x-UA-compatible "content=" IE =edge"> <title>Document</title> </head> <body> </body>  <script> fetch('http://localhost:8887', { method: 'POST', headers: { 'X-Test-Cors': '123', } }) </script> </html> server1.js const http = require('http'); const fs = require('fs'); http.createServer((req,res) => { console.log('req come', req.url); const html = fs.readFileSync('test.html','utf8'); res.writeHead(200,{ 'Content-type': 'text/html', }) res.end(html); }).listen(8888); server2.js const http = require('http'); http.createServer((req,res) => { console.log('req come', req.url); WriteHead (200,{' access-Control-allow-origin ': 'http://127.0.0.1:8888'}) res.end('1122'); writeHead(200,{' access-Control-allow-Origin ': 'http://127.0.0.1:8888'}) res.end('1122'); }).listen(8887);Copy the code

Set a custom request header to X-test-cors in the fetch. The following error will be reported when accessing localhost:8888. This means that our custom headers are not allowed in cross-domain requests. Before addressing this issue, I’ll talk about other CORS limitations and CORS pre-requests.

Other limitations of CORS:

Allowed methods: GET, HEAD, and POST. Other methods such as PUT and DELETE are not allowed and the browser will have a pre-validation request. And we’ll talk about what the pre-request does.
Allow the content-type: text/plain, multipart/form – data, application/x – WWW – form – urlencoded. Other Content-Types must be pre-authenticated before they can be sent.
Header restrictions: Custom headers are not allowed and require pre-request validation by the server. The heads below are allowed. Check out fetch.spec.whatwg.org/#cors-safel… .

The reason why the browser makes these restrictions is because it wants to keep the server safe when operating across domains. Do not allow random requests and methods to cross domains, and do not want a single cross-domain request to cause server data to be maliciously tampered with. Providing these limits allows you to determine whether the request should be responded to.

CORS pre-request:

In cross-domain situations, non-simple requests (such as request methods PUT or DELETE, content-Type fields of application/ JSON, and adding additional HTTP headers) will start with an empty body OPTIONS request, called a “precheck” request. It requests permission information from the server. After the precheck request is successfully responded to, a real HTTP request is initiated. From the network, the browser will first send a request (OPTIONS) to obtain the server’s approval to send the request, and then send the actual POST request. This is what browsers do with cross-domain request prerequests.

What exactly does the browser use to determine whether the request is allowed to return? To Allow custom Headers to be sent in a request, the server needs to return a new access-Control-allow-headers header to tell the browser that the custom Headers are allowed.

const http = require('http'); http.createServer((req,res) => { console.log('req come2', req.url); Res. WriteHead (200, {' Access - Control - Allow - Origin: 'http://127.0.0.1:8888', 'Access - Control - Allow - Headers' : 'X-Test-Cors', }) res.end('1122'); }).listen(8887);Copy the code

After setting access-Control-allow-headers to our custom Headers, the request can be sent normally.

Access-control-allow-methods can also be used to set the allowed request Methods. The PUT method is not allowed before setting.

Res. WriteHead (200, {' Access - Control - Allow - Origin: 'http://127.0.0.1:8888', 'Access - Control - Allow - Headers' : 'X-Test-Cors', 'Access-Control-Allow-Methods': 'POST,PUT,Delete' })Copy the code

The PUT method is allowed after setting

You can cache the pre-check request results of the browser by setting access-Control-max-age. Access-control-max-age indicates the maximum cross-domain duration, within which no pre-request verification is required. Formal requests can be made directly.

The Cache cache-control

Cache-control has the following features:

Cacheability: Specifies where caching can be performed

Public: indicates that the public value is set in cache-Control during the HTTP request return process. It indicates that any path through which the HTTP request returns, including the intermediate HTTP proxy server and the browser that sends the request, can Cache the returned content.
Private: Only the browser that initiated the request can cache.
No-cache: You can cache locally or on a proxy server, but you need to check with the server every time. If the server returns a message that the local cache can be used, the local cache can be used.

Expiration: cache expiration time.

The most common is max-age. Max-age = seconds(how many seconds are cached).
S-maxage =seconds Specifies the expiration time set for the proxy server.
Max-stale =seconds: if the returned resource has a max-stale setting after the expiration of a max-age cache, the stale cache can be used as long as the cache expires within the max-stale time.

revalidation

Must-revalidate: indicates that the cache with max-age set expires. You must send a request to the source server to obtain the resource again and then verify whether the resource has expired.
Proxy-revalidate: works in much the same way as must-revalidate, except for caching servers. After the cache expires, the cache server must fetch data from the source server.

No-store: no local or proxy server can store the cache, and the requesting server always requests data.
No-transform: used for proxy servers. Some proxy servers will transform resources. No-transform tells the proxy server not to change the returned content.

The following example demonstrates the effect of setting cache-control. The main demonstration is how the browser reads the cache locally with max-age set.

<! DOCTYPE html> <html lang="en"> <head> <meta charset="UTF-8"> <meta name="viewport" content="width=device-width, <meta HTTP-equiv =" x-UA-compatible "content=" IE =edge"> <title>Document</title> </head> <body> </body>  <script src="/script.js"></script> </html> server.js const http = require('http'); const fs = require('fs'); http.createServer((req,res) => { console.log('req come', req.url); if (req.url === '/') { const html = fs.readFileSync('test.html','utf8'); res.writeHead(200,{ 'Content-type': 'text/html' }) res.end(html); }else if (req.url === '/script.js') { res.writeHead(200,{ 'Content-type': 'text/javascript' }) res.end('console.log("script loaded")'); } }).listen(8888);Copy the code

It’s all over the network. A refresh will still request data from the server. Now set ‘cache-control ‘: ‘max-age=20’.

res.writeHead(200,{
     'Content-type': 'text/javascript',
     'Cache-Control': 'max-age=20',
})
Copy the code

You can see that the resource is fetched from the memory cache. At the same time, the acquisition time is 0, that is, there is no network delay, the speed is very fast. The background prints out the content.

To verify that the browser is fetching data from the cache, we fix the return from the server before the cache expires. Restart the service and the page console still outputs Script Loaded instead of Script Loaded Twice. Note The browser does not obtain data from the server, but cache it.

res.end('console.log("script loaded twice")');
Copy the code

Cache validation last-Modified and Etag use

The following figure shows the operation of caching. The browser creation request is first searched in the local cache, and if the local cache hits, it is returned directly to the browser. Transmission over no network. If the local cache is not hit, a request is sent over the Internet, through a proxy server, which looks up the relevant cache Settings and checks to see if the resource is cached. If hit, it is returned to the local cache and back to the server. If not, go to the source server and request the resource.

If no Cache is set to cache-control, the browser will verify the resource on the server every time it sends a request for a resource that has been configured with cache-Control. The browser will read the Cache of the resource only when it determines that the resource can be used by Cache. So how do you verify that? There are two main authenticated HTTP headers in HTTP: Last-Modified and Etag.

Last-modified: indicates the Last modification time. Used with if-modified-since or if-unmodified-since. If the last-Modified header in the header returned by the request resource specifies a time, then the next time the browser makes a request, the last-Modified value will be added. Go to the server via if-modified-since or if-unmodified-since. The server reads if-modified-since compared to the last time the resource was Modified. If the two times are the same, the resource has not been Modified. The server tells the browser to use the cache directly.

Etag: is a more stringent authentication that is performed through data signatures. The most typical method is to hash the resource contents. The contents of a resource generate unique signatures. If the content of a resource changes, its signature changes. This command is used with if-match or if-non-match. The browser sends the request with an if-match or if-non-match header, which is the value of the Etag returned by the server. The server takes the header sent by the browser and compares the resource’s signature to determine whether to use caching.

Let’s take an example to see the effect. Set headers cache-control and last-Modified and Etag

res.writeHead(200,{
    'Content-type': 'text/javascript',
    'Cache-Control': 'max-age=200000,no-cache',
    'Last-Modified': '123',
    'Etag': '777'
})
Copy the code

Cache-control Every request sent by the browser will be sent to the server even if max-age is set, as long as no-cache is set. As shown in the figure below, each request for a script resource is transmitted over the network rather than fetched from the cache.

You can see that the Response header has two authenticated HTTP headers last-Modified and Etag set inside.

When the browser requests again, it will carry the two headers of the corresponding validation cache.

The server then checks If the request header if-none-match matches its Etag. If they are the same, set the status code to 304(Not Modified resource has Not been Modified) to tell the browser to use the cache.

   const etag = req.headers['if-none-match'];
   if (etag === '777') {
        res.writeHead(304,{
            'Content-type': 'text/javascript',
            'Cache-Control': 'max-age=200000,no-cache',
            'Last-Modified': '123',
            'Etag': '777'
        })
        res.end('');
    }
Copy the code

If the browser is not using the cache then the return of the request should be empty. But now the request returns something. Note The browser obtains resources from the cache.

If no-cache is removed, the second request is fetched directly from the memory cache without validation.

If set to no-store, any cached information will be ignored and will be treated as a fresh request to fetch data from the server.

'Cache-Control': 'max-age=200000,no-store',
Copy the code

The Cookie and Session

Cookie

Cookie is a content saved to the browser through the set-cookie header when the server returns data. After the browser saves the cookie, it carries it with it on the next syndomain request. In this way, users can use cookies to ensure that the returned data is the user’s during the session when they visit the website. When the browser closes, the cookie is gone. Cookies are key-value pairs and can be set to multiple values. Cookies have the following properties.

Max-age and expires set the expiration time
Secure only sends cookies when HTTPS is used
HttpOnly cannot be used to access cookies through document.cookie to ensure user data security

Let’s demonstrate how cookies are used. The cookie is set on the server

const http = require('http');
const fs = require('fs');

http.createServer((req,res) => {
    console.log('request com', req.url);

    if (req.url === '/') {
        const html = fs.readFileSync('test.html','utf8');
        res.writeHead(200, {
            'Content-Type': 'text/html',
            'Set-Cookie': 'id=123'
        })
        res.end(html);
    }
}).listen(8888);
Copy the code

Revisiting the page the browser sends a request with a cookie on its head.

Set multiple cookies: set-cookie headers can be repeated in Response headers.

'Set-Cookie': ['id=123','abc=456']
Copy the code

Cookie set expiration time: set id=123 this cookie will expire in 2s

'Set-Cookie': ['id=123;max-age=2','abc=456']
Copy the code

After 2 seconds, the browser will only bring up the ABC cookie in the request header.

Set the cookie HttpOnly

'Set-Cookie': ['id=123;max-age=20','abc=456;HttpOnly']
Copy the code

You can only get id=123 from document.cookie.

Set the cookie domain

Cookies can set the permission to access the domain. Generally speaking, cookies written by the current domain cannot be obtained by other domains. For example, a cookie is written under the website of a.com, which cannot be accessed under the website of B.com. There are more restrictive schemes for cookies for the same domain name. For example, a.com has a secondary domain name test.a.com, which allows test.a.com to access the cookies set by a.com. This is done by setting the domain.

http.createServer((req,res) => {
    const host = req.headers.host;
    if (req.url === '/') {
        const html = fs.readFileSync('test.html','utf8');
        if (host === 'a.test.com:8888') {
            res.writeHead(200, {
                'Content-Type': 'text/html',
                'Set-Cookie': ['id=123;max-age=20','abc=456']
            })
        } else {
           res.writeHead(200, {
                'Content-Type': 'text/html',
            })
        }
        res.end(html);
    }
}).listen(8888);
Copy the code

Map A.test.com and B.test.com to 127.0.0.1 through hosts

Visit a.test.com:8888 and set cookies.

Visit b.test.com:8888 without cookies.

This means that cookies cannot be shared between different domain names.

Both a.test.com and b.test.com are test.com secondary domain names. If you want to obtain cookies from both the secondary domain names of test.com, set domain.

http.createServer((req,res) => {
const host = req.headers.host;
if (req.url === '/') {
    const html = fs.readFileSync('test.html','utf8');
    if (host === 'test.com:8888') {
        res.writeHead(200, {
            'Content-Type': 'text/html',
            'Set-Cookie': ['id=123;max-age=20','abc=456;domain=test.com']
        })
    } else {
       res.writeHead(200, {
            'Content-Type': 'text/html',
        })
    }
    res.end(html);
}
Copy the code

}).listen(8888);

To map test.com to 127.0.0.1, visit test.com:8888. You can see that the cookie was set successfully.

Now access a.test.8888 and b.test.8888, you can see that there are cookies, ABC =456. This is because ABC is the domain of test.com. Therefore, only ABC cookies can be shared. Therefore, both a.test.com and b.test.com can only get the cookie value ABC set by test.com.

session

HTTP is a stateless protocol, so when the server needs to record the user status, it needs to use some mechanism to identify the specific user, which is Session. In a typical scenario like a shopping cart, when you click the order button, HTTP is stateless, so you don’t know which user is doing it, so the server creates a special Session for a particular user, identifies that user, and keeps track of that user, so it knows how many books are in the shopping cart. This Session is stored on the server and has a unique identifier. There are many ways to save sessions on the server, including memory, databases, and files. Session transfer should also be considered when clustering. In large websites, there is usually a special cluster of Session servers to store user sessions. In this case, Session information is stored in memory, and some caching services such as Memcached are used to store sessions. How does the server identify a particular client? This is where Cookie comes in. Each TIME an HTTP request is made, the client sends a Cookie to the server. In fact, most applications use cookies to realize Session tracking. When a Session is created for the first time, the server tells the client in the HTTP protocol that it needs to record a Session ID in the Cookie, and sends this Session ID to the server for each subsequent request. I knew who you were. Someone asked, what if cookies are disabled on the client’s browser? In this case, session tracking is typically done using a technique called URL rewriting, where each HTTP interaction is followed by a parameter such as SID = XXXXX that the server identifies the user. Session is a data structure stored on the server to track user status. This data can be stored in clusters, databases, and files. Cookie is a mechanism for the client to save user information, which is used to record some user information and also a way to realize the Session.

HTTP Long connection

HTTP requests are sent over TCP connections, which are classified into long and short connections. To send an HTTP request, you need to create a TCP connection, then send the HTTP request over the TCP connection and receive the reply from the server. The HTTP request ends, and the server negotiates with the server whether to close the TCP connection. If the TCP connection is not closed, it will cost some money to keep it open, but subsequent HTTP requests can be sent directly over the TCP connection, eliminating the need for a three-way handshake. If closed, you can reduce the number of concurrent connections between the server and the browser. However, the next time you send an HTTP request and re-establish a TCP connection, there is an overhead in network latency. In reality, the concurrency of the website is large. If the TCP connection is re-created for each HTTP request, the cost of creating THE TCP connection may be greater than the cost of maintaining the long connection. And long connection can be set connection time, that is, too long time without HTTP request, then automatically closed connection. Therefore, long connections are generally maintained by default. Take a look at baidu’s web page, check Network, right click on Name and select Connection ID to check the Connection ID of each HTTP. Connection ID indicates the ID of the TCP Connection, so that we can distinguish whether it is the same TCP Connection.

You can see that the Connection ID is the same for many requests. For example, 116067, image requests are sent over this TCP connection.

However, there are many other connections such as 115692, 115695, etc., the domain name is not the same to create a new connection. However, they try to reuse the TCP connection created earlier.

This is because HTTP/1.1 connections send requests sequentially over TCP. If 10 requests cannot be sent concurrently over a TCP connection, requests sent over a TCP connection will only be sent one after the other. In fact, we want to do it concurrently, so it’s more efficient. So the browser allows concurrent TCP connections, and Chrome allows up to six at a time. When the limit of six concurrent requests is reached, incoming requests wait for the previous six to complete.

How can the service ensure that long connections are created instead of short ones? Connection: keep-alive indicates that the created TCP Connection is a long Connection.

Connection: keep-alive: Connection: Keep-alive: Connection: keep-alive: Connection: keep-alive: Connection: keep-alive: Connection: keep-alive: Connection: keep-alive However, when the server returns, it can choose to hold or not hold the long connection. If the server chooses not to hold the long connection, the browser will still shut it down.

Here’s how Connection: keep-alive works. Create a simple service with Node. By default, the server has keep-alive enabled.

server.js

http.createServer((req,res) => {
    console.log('request com', req.url);
    const img = fs.readFileSync('test.jpg');
    const html = fs.readFileSync('test.html','utf8');
    if (req.url === '/') {
        res.writeHead(200, {
            'Content-Type': 'text/html',
        })
        res.end(html);
    } else  {
        res.writeHead(200, {
            'Content-type': 'image/jpg'
        })
        res.end(img);
    }
}).listen(8881);
Copy the code

test.html

<! DOCTYPE html> <html lang="en"> <head> <meta charset="UTF-8"> <meta name="viewport" content="width=device-width, <meta HTTP-equiv =" x-UA-compatible "content=" IE =edge"> <title>Document</title> </head> <body> <img src="/test1.jpg" alt=''> <img src="/test2.jpg" alt=''> <img src="/test3.jpg" alt=''> <img src="/test4.jpg" alt=''> <img src="/test5.jpg" alt=''> <img src="/test6.jpg" alt=''> <img src="/test7.jpg" alt=''> </body> </html>Copy the code

Start the service and access localhost:8881. As you can see, 7 images are requested, the same image, but the requested URL is different, so the browser will load the image 7 times. Check the ConnectionId

You can see that some connectionids are the same, that is, they reuse the TCP connection you created earlier. There are also different connectionids because the browser allows six TCP connections to be created concurrently, so the seventh image will wait for the first six image requests to end and reuse the previous TCP connection. We can switch the network to Fast 3G and check the Waterfall.

You can see that the first six images are all simultaneous requests, and each ConnectionId is different. There is a long wait for a new TCP connection to be available before the request is sent. The TCP connection 20868 established when the first picture is sent is reused. If you place the mouse on the Waterfall corresponding to 20868, you can see that the first 20868 has Initial Connection (the time consumed to create a TCP connection), while the second 20868 has no Initial connection. Note No TCP connection is created during the request for the seventh image, but the first image is reused.

What happens if we turn keep-alive off? Connection has two values, one keep-alive and the other close. Close indicates that the TCP connection will be closed after a request is completed. Modify server Response.

const http = require('http');
const fs = require('fs');

http.createServer((req,res) => {
    console.log('request com', req.url);
    const img = fs.readFileSync('test.jpg');
    const html = fs.readFileSync('test.html','utf8');
    if (req.url === '/') {
        res.writeHead(200, {
            'Content-Type': 'text/html',
            'Connection': 'close',
        })
        res.end(html);
    } else  {
        res.writeHead(200, {
            'Content-type': 'image/jpg',
            'Connection': 'close',
        })
        res.end(img);
    }
}).listen(8881);
Copy the code

You can see that after long connections are closed, the ConnectionId of each request is different. That is, the TCP connection is not reused. After a request is sent, the TCP connection is closed. A new request is sent and a new TCP connection is established. Normally, HTTP services developed today turn on long connections by default, but keep-alive can be set to shut down. Add channel multiplexing to HTTP2, which means sending HTTP requests concurrently over a TCP connection so that only one TCP connection is required. Greatly reduces the overhead of creating multiple TCP connections. Goole uses HTTP2, where a cross-domain request has only one TCP connection.

Data consultation

Data negotiation is when a client sends data to a server, the client declares the format of the data it wants and the restrictions associated with the data. The server may have many different types of data, and it will return the corresponding format based on the headers sent by the client. Specify the desired data TYPE by Accept in the request, which is limited by the MIME TYPE declaration. Accept-encoding Specifies the Encoding mode in which data is transmitted. It is used to limit how the server compresses data. Accept-language indicates the Language in which the data is returned. User-agent indicates browser information. You can determine whether the page returned is a PC or mobile page based on the user-agent. The server sets the Content-Type to correspond to Accept, which can Accept several different data formats, and the Content-Type can choose one of them as the data format that the server actually returns. Content-encoding corresponds to accept-encoding, which states which compression the server uses, such as gzip. Content-language Indicates whether the corresponding Language was returned based on the request. Create a simple service

const http = require('http');
const fs = require('fs');

http.createServer((req,res) => {
    console.log('request com', req.url);

    const html = fs.readFileSync('test.html','utf8');
    res.writeHead(200, {
        'Content-Type': 'text/html'
    })
    res.end(html);

}).listen(8888);
Copy the code

Go to localhost:8888 and check the Request Headers

You can see that there is a lot of information in the Accept declaration, so the browser knows that it can display data in these formats. Accept-encoding declares three popular Encoding methods. Accept-language is set by the browser according to the Language of your system, q represents the weight, the larger the Language will be preferentially selected for display.

User-agent also has a lot of information, including system – and browser-specific information. Since the browser was originally developed by Netscape, the default header is Mozilla/5.0. Many older servers only support Mozilla/5.0, so Mozilla/5.0 will be added to the browser by default to support older servers. AppleWetKit is the browser kernel, and modern browsers like Chrome and Safari use WebKit. KHTML is a version of the rendering engine, similar to Gecko for firefox. The Chrome version number and Safari version number follow. This information can then be used to determine whether the page returned to the user should be adapted to those browsers.

These are some of the information that the browser sends to the server for data negotiation. The server takes this information, makes a judgment, and then returns the desired information to the browser.

Let’s look at Response Headers

The server returns data in text/ HTML format. The server can also return a header ‘x-content-type-options ‘:’nosniff’. Since Internet Explorer has long since refused to accept the content-type declared by the server, or since the server has not declared the content-type, It will predict the format of the content returned by the server. This can lead to security issues, such as some code that should be displayed as text ends up running as a script, causing security information to be leaked. Therefore, x-Content-type-options is designed to prevent the browser from proactively predicting the data format returned by the server.

Let’s look at content-encoding.

Looking at the size column, 423B is the actual size of the data during the entire transmission. This size will include the HTTP headers and body as well as the first line information. The following 258B is the actual Content in the body, which is the data obtained and decompressed according to content-Encoding. In the following example, the size of the data transferred will change after using gzip compression, but the following 258B will not change.

const http = require('http');
const fs = require('fs');
const zlib = require('zlib');

http.createServer((req,res) => {
    console.log('request com', req.url);

    const html = fs.readFileSync('test.html');
    res.writeHead(200, {
        'Content-Type': 'text/html',
        'Content-Encoding': 'gzip'
    })
    res.end(zlib.gzipSync(html));

}).listen(8888);
Copy the code

You can see that the transfer size is 362B because the body part was compressed using GIZP during the transfer. You can see that the size of the data transferred is still larger than the size of the body part, because the data transferred is too small. If more than 1KB is compressed, the actual transfer size will be smaller than the size of the body content after decompression. This is the advantage of compression, which speeds up network transmission. The server can select the best compression method based on the accept-encoding sent by the browser.

Finally, the Content-Type for sending a request, sometimes sending a request sends some data, like submitting a form. There must be a format for this data. If you don’t declare content-Type, you don’t know what format to send it to the server.

<! DOCTYPE html> <html lang="en"> <head> <meta charset="UTF-8"> <meta name="viewport" content="width=device-width, <meta HTTP-equiv =" x-UA-compatible "content=" IE =edge"> <title>Document</title> </head> <body> <form action="/form" method="post" enctype="application/x-www-form-urlencoded"> <input type="text" name="name"> <input type="password" name="password"> <input type="submit"> </form> </body> </html>Copy the code

Application/X-www-form-urlencoded is the default encType for forms, where form form data is encoded as key/value format and sent to the server.

Multipart /form-data This format is used when you need to upload a file from a form. Multipart means that the request has multiple parts. This is because the file must be unpacked when it is uploaded from the form because it cannot be transmitted as a string. To transfer as binary data, we can’t transfer the file if we concatenate strings the way we did before. The string following boundary is used to separate each item in the submission form.

<! DOCTYPE html> <html lang="en"> <head> <meta charset="UTF-8"> <meta name="viewport" content="width=device-width, <meta HTTP-equiv =" x-UA-compatible "content=" IE =edge"> <title>Document</title> </head> <body> <form action="/form" id='form' method="post" enctype="multipart/form-data"> <input type="text" name="name"> <input type="password" name="password"> <input type="file" name="file"> <input type="submit"> </form> <script> var form = document.getElementById('form'); form.addEventListener('submit',(e) => { e.preventDefault(); var formData = new FormData(form); fetch('/form', { method: 'POST', body: formData }) }) </script> </body> </html>Copy the code

After receiving the submitted form, the server parses each item of the body in a different way, depending on the header information. For more content-Type values, see www.runoob.com/http/http-c…

Redirect

The resource is not located at the specified URL. The server tells the browser where the requested resource is, and the browser then requests a new URL for the resource. The server returns code 302 and adds the Location header to the header.

const http = require('http');
const fs = require('fs');
http.createServer((req,res) => {
    if (req.url === '/') {
        res.writeHead(302,{
            'Location': '/new'
        })
        res.end('');
    }
    if (req.url === '/new') {
        res.writeHead(200,{
            'Content-Type': 'text/html'
        })
        res.end('<div>this is content</div>');
    }
}).listen(8888);
Copy the code

Access localhost: 8888

The server first returns 302, indicating that the browser wants to jump (temporary jump, 301 permanent jump). Add Location: /new to the response header. The browser then requests /new for the resource.

302 Temporary redirect and 301 Permanent redirect:

(1) 302 Redirection Each access ‘/’ must go through the specified hop path of the server to jump to the new address. Call ‘/’ and then ‘/ new’ each time.

(2) 301 specifies that ‘/ ‘is permanently changed to ‘/new’, so it will tell the browser to change the browser to ‘/new’ the next time it visits ‘/ ‘, without going through the server to specify the redirect address. All subsequent calls to ‘/’ are direct calls to ‘/new ‘.

(3) The resource corresponding to 301 is from disk cache, which is read from the cache. The request for ‘/’ is already in the browser cache. Since 301 defines the link as permanently new, the browser cache 301 returns as long as possible. Even if we change 301 to 200, the browser will still redirect to ‘/new’ because it is fetched from the cache. Unless the user clears the cache himself.

CSP(Content-Security-Policy)

The HTTP protocol limits access to resources in order to make web sites more secure. CSP limits where resources are fetched and where requests are sent. CSP reports resource overreach if a resource is acquired that it should not be. It limits its scope by first limiting all link requests globally by default-src. You can then restrict it based on a particular resource type.

Anything related to the outer chain can be restricted by resource type. For example, connect-src: represents the target to which the request is directed. Img-src: The url from which images can be loaded. Style-src /script-src: How many urls can style /script resources be loaded from? Let’s see how this header works.

<! DOCTYPE html> <html lang="en"> <head> <meta charset="UTF-8"> <meta name="viewport" content="width=<s>, <meta HTTP-equiv =" x-UA-compatible "content=" IE =edge"> <title>Document</title> </head> <body> <script> console.log('inline js'); </script> </body> </html>Copy the code

Writing scripts directly in HTML, we don’t want to execute scripts directly on the page for security reasons. Because XSS attack is to inject some scripts into the website through some methods to cause problems on the page or even steal user information. The embedded script was probably inserted through a rich text editor in a web page. So we want to limit the ability to script directly on the page. Load the script only through the outer chain. You can write the following content-security-policy header on the server.

const http = require('http');
const fs = require('fs');

http.createServer((req,res) => {
    console.log('request com', req.url);
    const html = fs.readFileSync('test.html','utf8');
    res.writeHead(200, {
        'Content-Type': 'text/html',
        'Content-Security-Policy': 'default-src http: https:'
    })
    res.end(html);
}).listen(8888);
Copy the code

If you start the service to access localhost:8888, the following error message is displayed.

Load JS via external chain.

<! DOCTYPE html> <html lang="en"> <head> <meta charset="UTF-8"> <meta name="viewport" content="width=<s>, <meta HTTP-equiv =" x-UA-compatible "content=" IE =edge"> <title>Document</title> </head> <body> <script  src="/test.js"></script> </body> </html> const http = require('http'); const fs = require('fs'); http.createServer((req,res) => { console.log('request com', req.url); const html = fs.readFileSync('test.html','utf8'); if (req.url === '/') { res.writeHead(200, { 'Content-Type': 'text/html', 'Content-Security-Policy': 'default-src http: https:' }) res.end(html); } else { res.writeHead(200, { 'Content-Type': 'application/javascript', }) res.end('console.log("loading srcipt")'); } }).listen(8888);Copy the code

No errors are reported in the background and the script execution results are printed out.

Processing restrictions load scripts through the external chain, and you can also restrict the domain name of the external chain. For example, if you can only pretend to be js under this domain name, you can set default-src to self.

res.writeHead(200, {
    'Content-Type': 'text/html',
    'Content-Security-Policy': 'default-src \'self\''
})
Copy the code

An error is reported when loading a JS that is not in the domain, and the resource status is blocked: CSP. The browser side limits the request.

< script SRC = "https://cdn.bootcdn.net/ajax/libs/jquery/3.6.0/jquery.js" > < / script >Copy the code

You can also restrict the designation of a particular site. For example, the setting allows loading of the content below cdn.bootcdn.net/.

res.writeHead(200, {
    'Content-Type': 'text/html',
    'Content-Security-Policy': 'default-src \'self\' https://cdn.bootcdn.net/'
 })
 
Copy the code

Now loading cdn.bootcdn.net/ajax/libs/j… The background will not report an error.

Setting default-src does not work for form submission, so you can limit the action for form submission as follows.

server.js: res.writeHead(200, { 'Content-Type': 'text/html', 'Content-Security-Policy': 'default-src \'self\'; form-action \'self\'' }) html: <form action="http://www.baidu.com"> <button type="submit">click me</button> </form>Copy the code

Click the button background error.

For more information about CSP, see developer.mozilla.org/en-US/docs/…

If there is a resource that we do not want to load, we can have the CSP actively report a request to the server. Report URI /report(server path). You can see that the browser sends a report request to the server.

res.writeHead(200, { 'Content-Type': 'text/html', 'Content-Security-Policy': 'default-src \'self\'; form-action \'self\'; report-uri /report' })Copy the code

CSP can also be used by writing in HTML. The effect is the same as the server Settings.

<meta http-equiv="Content-Security-Policy" content="default-src 'self'; form-action 'self'; report-uri /report">Copy the code

HTTPS

HTTPS is HTTP+Security. HTTP is plaintext transmission, and at every layer of the Internet, if someone intercepts a packet sent, they can parse and read any information in the packet, which is the equivalent of running naked. Then use Wireshark to capture packets of HTTP websites.

You can see the Hypertext Transfer Protocol (HTTP). You can see all information in the Hypertext Transfer Protocol, including the address, Host, and header information of the GET request. You can even see the value of cookies, which are often used to store user session information. If the packet is intercepted, the cookie can be used to imitate the user’s login and request the user’s data.

Above is the server’s response, with the data compressed by Gzip. But we can take the Data in the Data chunk and extract it. So HTTP protocol has no security properties. So why is HTTPS secure? HTTPS has public and private keys, and a public key is an encrypted string that is available to anyone on the Internet. This encrypted string is used to encrypt the information we transmit. After the data encrypted with the public key is sent to the server, the server can decrypt the data only by using the private key. No one can get the private key because the private key is placed on the server. So even intercepted packets cannot be decrypted, making the process of sending the data secure.

Public and private keys are used to transfer information during a handshake. The HTTPS handshake process is as follows:

During the handshake, the Client will initiate a Client Hello, generate a random number and transmit it to the server, along with the encryption suite supported by the Client.

The server takes this random number and stores it, and the server generates a random number. At the same time, the server selects a Cipher Suite to return to the client.

The server also continues to send a return telling the client the public key (certificate).

Once the client gets the certificate, it generates a pre-master key, which is also an Encrypted random Handshake Message, and transmits it to the server.

This process is one that cannot be resolved by an intermediary. The server takes this random number and decrypts it with the private key to get something really useful. The client and server simultaneously use an encryption suite to generate the master secret key through algorithmic operations on these three random numbers. This master key is also impenetrable to the middleman. Once you have the master key, all data transmitted later is encrypted with this master key. Therefore, the security of data transmission is ensured. Use Wireshark to capture packets from HTTPS websites.

The scraping tool has no way of seeing what the requested address is. All I got was a Secure Sockets Layer, which contained encrypted data. There is no way to decrypt it because there is no public key. Only the client and server can access the transmitted data by decrypting it with the master key. This ensures data security.

Write in the last

These are some of the things I think are important about HTTP. Finally, a diagram summarizes the complete process of the HTTP request returning after the browser enters the URL.

Enter the URL in the address bar and press Enter. It is possible that the server has sent a 301 message to redirect the browser to a new page. The browser has recorded the message. So there is a Redirect at the beginning of the request, which is pure client-side behavior. So the first node in the figure is the Redirect. The second step is for the browser to determine whether the resource is cached or not. If it is cached, it applies the cache. Otherwise, it goes to step 3. The third step is DNS resolution to find the server IP address corresponding to the domain name. Step 4 Create a TCP link. Step 5 Send the request. Step 6 Receive the server response.

HTTP protocol and its history

Classical five-layer network model

HTTP protocol history

HTTP / 0.9

HTTP / 1.0

HTTP / 1.1

HTTPS

HTTP2

Principle of HTTP

URI – URL and URN

HTTP Packet Format

Overview of HTTP features

Restriction and resolution of CORS cross-domain requests

CORS cross-domain constraints and pre-request validation

The Cache cache-control

Cache validation last-Modified and Etag use

The Cookie and Session

Cookie

session

HTTP Long connection

Data consultation

Redirect

CSP(Content-Security-Policy)

HTTPS

Write in the last

Related Posts

ES chain judgment operator? . | open merge operator?? | CSS inset

An interview question that gets 90% of the answers wrong

Let’s start at 0. Handwritten promises from scratch.