The authors introduce

Xu Tong, a member of the proprietary Nail front-end team, is responsible for the proprietary nail workbench & open platform related business modules.

background

Writing this article is the main reason himself responsible for business daily iteration updated front static resource, all online validation regression function is normal, but the next day some user feedback into the front end application for the first time, the page content display abnormal, the second or third to open the page properly, abnormal situation as shown in the diagram below client (android and ios) :

Faced with this situation, I decided to urgently roll back the front-end content to the previous version, and sure enough, everything is fine (the number of successful traffic visits remains the normal level of yesterday).

Then view the application monitoring logs at the point in time when the problem occurred:

All requests for front-end static resources are reported with status: 502 or 504. Considering that the front-end static resource publishing volume increased by about 700KB, there is a high probability that the so-called nginx traffic limiting (increased traffic, resulting in some connections not responding) was triggered.

Front-end Deployment History

  • In the era of jQuery, resources deployed online by front-end projects are basically deployed together with backend resources, which help configure forwarding rules.
  • Modern front-end engineering projects are deployed separately at the front and back ends, which are deployed through Docker. Docker uses nginx.conf to configure access to static resources.

The cause of the problem

The original nginx configuration was:

# proxy conf user admin admin; worker_processes 2; worker_rlimit_nofile 100000; events { xxxxx; } http { include mime.types; default_type application/octet-stream; root xxxx; sendfile on; tcp_nopush on; server_tokens off; keepalive_timeout 0; client_header_timeout 1m; send_timeout 1m; client_max_body_size 2048m; check_shm_size 20m; index index.html index.htm; log_format xxxxx; access_log xxxxx; log_not_found off; # log_empty_request off; #eagleeye_traceid_var $eagleeye_traceid; gzip on; Gzip_http_version 1.0; gzip_comp_level 6; gzip_min_length 1024; gzip_proxied any; gzip_vary on; gzip_disable msie6; gzip_buffers 96 8k; gzip_types text/xml text/javascript text/plain text/css application/javascript application/x-javascript application/rss+xml application/json; proxy_set_header Host $host; proxy_set_header Web-Server-Type nginx; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; #proxy_set_header EagleEye-TraceId $eagleeye_traceid; proxy_redirect off; proxy_buffers 128 8k; proxy_intercept_errors on; server_names_hash_bucket_size 512; # other omit XXX: XXXXX; }Copy the code

Here are some of the nginx parameters you need to know:

  • Worker_processes: Specifies the number of working processes. Generally, set this parameter to the number of CPU cores. A reasonable value should be set based on the actual performance and load of the server. If you do not know how to set a reasonable value, auto is best.
  • HTTP: can nest multiple servers, configure proxy, cache, log definition and most functions and third-party module configuration.
  • Server: Configures the parameters of the virtual host. One HTTP server can have multiple servers.
  • Gzip compression: gzip_comp_level Indicates the compression level, ranging from 1 to 9. The larger the number, the better the compression is and the more CPU time it consumes
  • Proxy_cache cache Settings: the project page needs to load a lot of data, which is not often changed, does not involve personalized customization, and dynamically generates data for each request. The performance is not as good as caching the results according to the request route and parameters. Using Nginx cache will greatly improve the request speed. To enable content caching, you only need to configure proxy_cache_path, which sets the path and configuration of the cache, and proxy_cache, which enables the cache.
  • Proxy_cache_valid: Configures the cache duration of files in the nginx cache. If the value is proxy_cache_VALID 200 304 2m; Note The cache duration of the cache file whose status is 200 and 304 is 2 minutes. If you access the cache file two minutes later, the file will expire and the source server will fetch data again
  • Open_file_cache module file descriptor cache: Only for open file handles and source information. Caching file handles means that you do not have to close and open a file each time, reducing the operation of system calls.
  • Location: Specifies the pattern to match the URI requested by the client
    • Matches the URI type with four parameters or no parameters.
    • Naming location, identified with @, is similar to defining a goto block.
The location [= | | | ~ ~ * ^ ~] / URI {... } location @/name/ {... }Copy the code

Note: Nginx is a two-tier instruction to match request URIs. The first is the server directive, which does the first-level matching by domain name, IP, and port. When a matching server is found, it enters the location matching of the server.

Locations are not matched exactly in the order in which they appear in the configuration file, request URIs are matched as follows:

  1. First, accurate matching =, the successful accurate matching will immediately stop other types of matching;
  2. If no accurate match succeeds, prefix match is performed. Prefix matches with ^~ are searched first. If prefix matches with ^~ are successful, other types of matches are stopped immediately. If common prefix matches (without the parameter ^~) are successful, regular matches are saved temporarily and the search for regular matches continues.
  3. If = and ^~ fail to match, search for regular matches ~ and ~*. If multiple re matches exist at the same time, they are matched in the sequence in the configuration file. If they are matched, other re matches are stopped immediately.
  4. If all re matches are unsuccessful, the results of the ordinary prefix matches (without arguments ^~) temporarily stored in 2 are returned
  • Proxy_pass: specifies the proxy_pass directive of the ngx_HTTP_proxy_module, which is used to configure forwarding. The url after proxy_pass is followed by /, indicating the absolute root path. If there is no slash, it indicates the relative path

Problem analysis:

Take a look at the nginx.conf file for the online environment configuration above.

  • Online Pod is 4-core, but write to 2 core only, and only 2 Pod
  • There is no cache
  • Gzip has a high compression level and consumes a lot of CPU time each time

These reasons combined with the heavy morning rush hour traffic, directly overturned the car.

The solution

Part of the configuration for the final version of Nginx looks like this:

server { listen 8080; server_name _; client_max_body_size 1024M; location / { root xxxxx; index index.html index.htm; } location =/nginx_status {allow 127.0.0.0/24; deny all; stub_status on; expires off; access_log off; } } server { listen 80; server_name _; client_max_body_size 1024M; Proxy_cache_path: XXXX. location / { proxy_cache static_cache; proxy_cache_valid 200 302 1h; add_header Nginx-Cache "$upstream_cache_status"; Proxy_pass http://127.0.0.1:8080; root xxxxx; index index.html index.htm; } location =/nginx_status {allow 127.0.0.0/24; deny all; stub_status on; expires off; access_log off; }}Copy the code
  • The front-end request 200 and 302 of the proxy was cached for 1 hour
  • Open_file_cache is set to further improve performance
  • Lower gzip_level
  • Add Pod machine
  • Configure worker_processes to maximize reuse of Pod specifications, or write a script to maximize reuse of Pod performance at initialization
let cpu_num=$(cat /sys/fs/cgroup/cpu/cpu.cfs_quota_us)/$(cat /sys/fs/cgroup/cpu/cpu.cfs_period_us) if [ ${cpu_num} -lt 2 ]; then cpu_num=2 fi sed -i '/worker_processes/s/worker_processes.*/worker_processes '"${cpu_num}"'; /g' $BASE_CONF_DIR/nginx-proxy.confCopy the code

Through the above optimization means, the effect of pressure measurement will improve the QPS index nearly 100 times, a one-time solution to the problem.

Conclusion:

Nginx configuration is more complex than nginx configuration. For simple front-end configuration, see the following:

location ~ .*.(html|htm)$ { add_header Cache-Control "public, no-cache"; # HTML file negotiation cache, which asks the server if the browser is locally up to date. The latest server will return the latest}. The location ~ *. (js | | CSS PNG | JPG | jpeg | SVG | eot | woff2 | the vera.ttf) ${add_header cache-control "public, max-age=xxxx"; # Non-HTML content cache normal iteration cycle time: a month or more}Copy the code