Nginx Learning Notes series 3 – Understanding the request process in depth

This is the 21st day of my participation in Gwen Challenge

background

This is the third part of the Nginx study notes collection, which summarizes the process and underlying mechanism of Nginx request processing, as well as the corresponding DATA structure in THE C language

Request process

Request concept

Request is an HTTP request in Nginx. The data structure in Nginx is ngx_HTTP_request_T, which encapsulates an HTTP request, including the request line, request header, request body, response line, response header, and response body

HTTP request is a typical request-response type of network protocol, and HTTP is a text protocol, so in the analysis of the request line and request header, as well as the output response line and response header, often is a line for processing, reading a line of data, analysis of the request line contains method, URI, HTTP_version information, and then a Line processing request header, and according to the request method and request header information to decide whether the request body and the length of the request body, and then to read the request body, get request, we need the output data processing request, and then generating response line, response headers and response body, after the response is sent to the client, a complete request is processed

Nginx uses ngx_HTTP_request_T to store data related to parsing requests and output responses

Nginx complete request processing process

A request starts with ngx_HTTP_init_request. In this function, the read event is set to ngx_HTTP_PROCESS_request_line, that is, the subsequent network event is executed by ngX_HTTP_process_request_line, which is Ngx_http_read_request_header to read the request data, and then call ngx_HTTP_parse_request_line function to parse the request line. Nginx uses the state machine to parse the request line to improve efficiency, and metho D, instead of using string comparisons directly, converts four characters into an integer, and then compares them once to reduce the number of CPU instructions

A request line contains the request method, URI, version, and can also contain host information, such as a request GET www.taobao.com/uri Nginx ignores the host field in the request header and uses the host field in the request line to find the virtual host. The http0.9 version does not support headers, requiring special processing; Therefore, when the request header is parsed later, the protocol version is 1.0 or 1.1, and the parsed parameters for the entire request line are stored in the ngx_HTTP_REQUEST_T structure

After parsing the request line,Nginx sets the handler for the read event to ngx_HTTP_PROCESS_request_HEADERS, and subsequent requests are placed in The ngx_HTTP_process_request_HEADERS function is used to read and parse the request headers as well as the row Er reads the request headers and calls ngx_HTTP_parse_header_line to parse a row of headers. The parsed headers are stored in the ngX_HTTP_request_T domain headers_in, which is a linked list structure that holds all headers, and HTT Some requests in P need special processing. These request headers and request handler function are stored in a mapping table, namely ngx_HTTP_headers_in. During initialization, a hash table will be generated The handler for this request header, such as the Host header, is ngx_HTTP_process_host

When Nginx resolves to two return newlines, indicating the end of the request header, ngx_HTTP_process_REQUEST is called to process the request. Ngx_http_process_request sets the current connection’s read-write event handler to ngX_HTTP_reQU Est_handler, then call ngx_HTTP_handler to actually start processing a complete HTTP request; Note that the read and write event handlers are both ngx_HTTP_request_handler, because in this function read_event_handler or write_event_h in ngX_HTTP_request_T are called, respectively, depending on whether the current event is a read or write event Nginx does not read the body of the request, andler. Set read_event_handler to ngx_HTTP_block_reading

The data processing begins in the ngx_HTTP_handler function, which sets write_event_handler to ngX_HTTP_CORE_RUN_PHASES and executes the ngX_HTTP_CORE_RUN_PHASES function, ngx_HTt Ngx_http_core_run_phases (ngX_HTTP_CORE_RUN_PHASES, ngX_HTTP_CORE_RUN_Phases, ngX_HTTP_CORE_RUN_Phases, ngX_HTTP_CORE_RUN_Phases) ¶ For ngx_http_core_run_phases

Ngx_http_core_run_phases will be called to process the request. The resulting response headers will be placed in ngX_HTTP_REQUEST_T’s headers_OUT. The various Nginx phases will process the request and filter the data. Data processing, such as TRUNCked transmission, GZIP compression, etc., filter here includes header filter and body filter, that is, response header or response body is processed; Filter is a linked list structure, containing header filter and body filter respectively. All the filters in header filter are executed first, and then all the filters in body filter are executed in header filter The last filter in filter, ngx_HTTP_header_filter, will iterate through all of the response headers, and finally need to output the response headers in a continuous memory, then call ngx_HTTP_write_filter for output, ngx_HTTP_WRITe_fil Ter is the last one in the body filter, so Nginx will call ngx_HTTP_write_filter after passing through a series of body filters

Note that Nginx puts the entire request header into a buffer, the size of which is set by the configuration item client_header_buffer_size. If a user’s request header is too large to fit into the buffer,Nginx reallocates a new bu that is larger Ffer buffers can be set by using large_client_header_buffers. The large_buffers group, for example, will have four buffers of 8K size to store rows or headers Integrity, has a complete line or request, must be placed on a continuous memory, so a complete line request or request header will only be stored in a buffer, so, if the request is greater than the size of a buffer, it will return a 414 error, if a request header is greater than the size of a buffer size, will return to 4 00 error

In actual business application scenarios, developers need to adjust the above buffer parameters according to the actual requirements to optimize the program

Keepalive and keepalive_timeout

Keepalive is used to indicate whether to close a long connection in an HTTP connection. If a long connection is required,Nginx sets the Keepalive property of the current connection after output the response body and waits for the next request from the client

Nginx cannot wait forever. When Nginx sets keepalive to wait for the next request, it also sets a maximum wait time, which is specified by the keepalive_TIMEOUT option. If set to 0, keepalive is disabled

For Nginx with a large number of requests, disabling Keepalive will result in a large number of time-wait state sockets. When the client needs to access the same server for multiple times, enabling Keepalive has great advantages, such as the image server. Usually, a web page contains many images Opening Keepalive also significantly reduces the number of time-Waits

pipe

Http1.1 introduces a new feature: Pipeline,pipeline is actually pipelines-based, it can be seen as a refinement of keepalive, because pipeline is also based on long connections, the purpose is to use a connection to do multiple requests, if the client to submit multiple requests, for keepa Live, then the second request, must wait until the first response to the request to receive completely, to initiate, this and stop waiting TCP protocol is the same, get two response time for at least 2 RTT, and for pipeline, the client don’t have to wait until after the first request processing, can immediately launched the second request, get two The response time may be up to 1RTT

Nginx supports pipelines directly, but the processing of multiple requests in the pipeline is not parallel, is still a request after request processing, only in the first request, the client can initiate a second request, so Nginx use pipeline to reduce the processing of a The time to wait for the request header data for the second request after the first request

When Nginx reads data, it puts it into a buffer, so if Nginx finds any data in the buffer after processing the previous request, it considers the remaining data to be the start of the next request, and then proceeds to the next request. Otherwise, it sets the KE epalive