Access layer Nginx architecture and module introduction share

Course Objectives:

2) Familiar with the application scenarios of Nginx. 3) Familiar with Nginx features, architecture models and related processes. 4) Familiar with the module classification of Nginx custom development

Course Outline:

1. Introduction and features of Nginx
2. Nginx application scenarios
3. Nginx framework model introduction
4. Nginx internal process introduction
5. Nginx custom module development introduction
6. Nginx core time point module introduction
7. Introduction to the Nginx shunt module
8. Upstream (110: Upstream
9. Nginx Query_upstrem
10. Nginx Query_conf module
11. Nginx shared memory supports the Redis protocol module
12. This section describes the Nginx log playback pressure test tool

1. Introduction and features of Nginx

Nginx introduction:

Nginx (Engine X) is a high-performance Web server and reverse proxy server, as well as an IMAP/POP3/SMTP server

• Russian programmer Igor Sysoev started it in 2002
• Nginx is the fastest growing Web server with 33.3% market share
• In 2011, a commercial company was founded

Nginx community branch:

• Openresty author @agentzh(Zhang Yizhun) developed, the biggest feature is the introduction of ngx_Lua module, support the use of Lua plug-in development, and a collection of many rich modules, as well as the LuA library.
• Tengine is mainly developed by taobao team. The feature is to integrate the new functions brought by some of taobao’s own businesses.
• Nginx official version, update iteration is relatively fast, and available free version and commercial version.

Nginx source code structure:

• The code quantity is about 110,000 lines of C code
• Source directory structure
• Core (Trunk and Base)
• Event (event-driven model and different IO reuse modules)
• HTTP (HTTP server and modules)
• Mail (Mail proxy server and module)
• OS (Operating system-specific implementation)
• Misc (Miscellaneous)

Nginx features:

• Reverse proxy, load balancer
• High reliability, single-master, multi-worker mode
• High scalability and high modularity
• non-blocking
• event-driven
• Low memory consumption
• Hot deployment

2. Nginx application scenarios

The scenario is as follows:

• Static file server
• Reverse proxy, load balancer
• Security defense
• Intelligent routing (enterprise-level grayscale testing, map POI one-click cutting flow)
• Gray released
• static
• Being pushed
• Real time picture compression
• Preventing hotlinking

3. Nginx framework model introduction

Process component role:

• The master process
• Monitor the status of worker processes
• Restart a new worker process when it dies
• Process signals and notify worker processes

• Worker processes
• Process client requests
• Get a signal from the main process and do something about it

• Cache loader process
• Load the cache index file information and exit

• The cache manager process
• Manages the cache size of the disk. If the cache size exceeds the preset value, the least used data will be deleted

Framework model:

Framework model flow:

4. Nginx internal process introduction

4.1 Framework model process

4.2 Master Initialization Process

4.3 Worker initialization

4.4 Worker initialization Process

4.5 STATIC File Request I/O Process

4.6 HTTP Request Process

4.7 11 STAGES of AN HTTP Request

4.8 the upstream module

• Access the third-party Server
• The underlying HTTP communication is perfect
• Asynchronous nonblocking
• Zero copy of upstream and downstream memory, saving memory
• Support custom module development

4.8.1 Upstream Framework process

4.8.2 Upstream internal process

4.9 Reverse Proxy Process

5. Nginx custom module development

5.1 Modular design features of Nginx

• Highly abstract module interface
• The module interface is very simple and flexible
• Configuration module design
• Simplification of the core module interface
• Multi-level and multi-category module design

5.1 Internal core modules

5.2 handler module

• Receives the request from the client and builds the response header and body.

5.3 the filter module

• The filter module is a module that filters the header and content of the response, and can process the header and content of the response. Its processing time is after retrieving the reply content and before sending the response to the user.

5.4 the upstream module

• Enables NGINX to complete the receiving, processing and forwarding of network data across the limitation of single machine, and access back-end services in pure asynchronous mode.

Load_balance:

• Load balancing module, to achieve a specific algorithm, in many back-end servers, select a server out as a request forwarding server.

5.3 ngx_lua module

• Scripting language
• Low memory overhead
• Fast running speed
• Powerful Lua coroutines
• non-blocking
• Business logic is written in natural logic

5.4 Customize Demo development

Handler module:

• Writing the Config file
• Write modules to generate content response information

1. # config file:
2. server {
3. .
4. location test {
5. test_counter on;
6. }
7. }
8. #config
9. ngx_addon_name=ngx_http_test_module
10. HTTP_MODULES="$HTTP_MODULES ngx_http_test_module"
11. NGX_ADDON_SRCS="$NGX_ADDON_SRCS $ngx_addon_dir/ngx_http_test_module.c"
12. #ngx_http_test_module.c
13. staticngx_int_t
14. ngx_http_test_handler(ngx_http_request_t*r)
15. {
16. ngx_int_t rc;
17. ngx_buf_t*b;
18. ngx_chain_tout;
19. ngx_http_test_conf_t*lrcf;
20. ngx_str_t ngx_test_string = ngx_string("hello test");
21. lrcf = ngx_http_get_module_loc_conf(r, ngx_http_test_module);
22. if( lrcf->test_counter ==0){
23. return NGX_DECLINED;
24. }
25. /* we response to 'GET' and 'HEAD' requests only */
26. if(! (r->method &(NGX_HTTP_GET|NGX_HTTP_HEAD))){
27. return NGX_HTTP_NOT_ALLOWED;
28. }
29. /* discard request body, since we don't need it here */
30. rc = ngx_http_discard_request_body(r);
31. if( rc ! = NGX_OK ){
32. return rc;
33. }
34. /* set the 'Content-type' header */
35. / *
36. *r->headers_out.content_type.len = sizeof("text/html") - 1;
37. *r->headers_out.content_type.data = (u_char *)"text/html";
38. * /
39. ngx_str_set(&r->headers_out.content_type,"text/html");
40. /* send the header only, if the request type is http 'HEAD' */
41. if( r->method == NGX_HTTP_HEAD ){
42. r->headers_out.status = NGX_HTTP_OK;
43. r->headers_out.content_length_n = ngx_test_string.len;
44. return ngx_http_send_header(r);
45. }
46. /* set the status line */
47. r->headers_out.status = NGX_HTTP_OK;
48. r->headers_out.content_length_n = ngx_test_string.len;
49. /* send the headers of your response */
50. rc = ngx_http_send_header(r);
51. if( rc == NGX_ERROR || rc > NGX_OK || r->header_only ){
52. return rc;
53. }
54. /* allocate a buffer for your response body */
55. b = ngx_pcalloc(r->pool,sizeof(ngx_buf_t));
56. if( b == NULL ){
57. return NGX_HTTP_INTERNAL_SERVER_ERROR;
58. }
59. /* attach this buffer to the buffer chain */
60. out.buf = b;
61. out.next= NULL;
62. /* adjust the pointers of the buffer */
63. b->pos = ngx_test_string.data;
64. b->last= ngx_test_string.data + ngx_test_string.len;
65. b->memory =1;/* this buffer is in memory */
66. b->last_buf =1;/* this is the last buffer in the buffer chain */
67. /* send the buffer chain of your response */
68. return ngx_http_output_filter(r,&out);
69. }

6. Nginx core time point module introduction

Solve the problem of slow fault location at the access layer, help the OP quickly determine the root cause of the problem, give priority to self-verification, and improve the efficient productivity of the access layer.

7. Introduction to the Nginx shunt module

Features: Implement very flexible dynamic change policy to cut traffic. The flow can be switched smoothly and losslessly. By switching traffic at the second level, the scope of influence can be reduced, thus reducing losses. Switch traffic or disable traffic in seconds based on a certain city or feature. Capacity failure of single room level is tolerated, which shortens the stop loss time of single room failure. Quickly isolate or sample traffic. Efficient gray testing, improve productivity.

8. Introduction to Nginx dynamic upstream module

In this way, the access layer can adapt to the cloud environment with dynamic scheduling to achieve smooth online and offline services and flexible capacity expansion/reduction. This improves the productivity and stability of the access layer and ensures smooth and lossless traffic.

9. Introduction to Nginx query_upstream

Link tracing: Combs the status of links between interfaces and the back-end. Query information about the upstream server corresponding to the location interface.

Introduction to the Nginx query_conf module

Get nginx configuration file formatting information in JSON format.

11.Nginx shared memory support redis protocol module introduction

Dynamically add shared memory based on configuration files. Github.com/lidaohang/n…

• Ngx_shm_dict Shared memory core module (red-black tree, queue)
• Ngx_shm_dict_manager Adds timer events to periodically clear expired keys in the shared memory and adds read events. Redis supports the redis protocol by using the REDis – CLI get,set,del, TTL
• Ngx_shm_dict_view View shared memory

12. Nginx log playback pressure test tool

• Parse logs to perform a replay pressure test and simulate the backend server slowness github.com/lidaohang/p…

Solutions that

• The client parses the host,port, URL, and body of the access.log build request
• Put the backend response time, backend response status code, and backend response size into the header header
• The backend server obtains the corresponding header to simulate the response body size, response status code, and response time

use

• Copy the tested access.log logs to the logs folder
• Set up the nginx server to test and configure upstream to point to the back-end server cut-off port
• Start the backend server instance server/backserver/main. Go
• Bin/wrk-c30-T1-s conf/nginx_log.lua http://localhost:8095

Author: Li Hang