Nginx performance optimization practices

Nginx system architecture

Nginx contains a single master process and multiple worker processes. All of these processes are single-threaded and designed to handle thousands of connections simultaneously. The worker process is where connections are processed, and it is used to process client requests. The master process is responsible for reading configuration files, processing sockets, deriving worker processes, opening log files, etc. In summary, a master process is a process that can manage requests by processing signal responses. Check out moreNginx Core Modules and Configuration Practices.

Tuning Nginx performance parameters

General parameter explanation

Go to /etc/nginx and edit nginx.conf. You can see the following parameters. A brief introduction:

# number of nginx processes. It is recommended to specify the number of CPU cores, usually the same as or multiple of the number of CPU cores.
worker_processes 8;

# Maximum number of connections per worker process
worker_connections 1024;

Allocate CPU to each process. In this example, 8 processes are allocated to 8 cpus. You can write more than one process, or allocate one process to multiple cpus.
worker_cpu_affinity 00000001 00000010 00000100 00001000 00010000 00100000 01000000 10000000;

# I/O multiplexing model for events
use epoll;

Accept as many connections as possible after receiving a new connection notification
multi_accept on;
Copy the code

The ePoll interface was introduced in Linux kernel 2.5 as a variant of the Poll interface. Compared to the multiplexing I/O model implemented by SELECT, ePoll’s greatest benefit is that it does not dramatically decrease in efficiency as the number of monitored descriptors increases.

worker_processes number;

Each worker process is a single-threaded process that calls various modules to achieve a variety of functions. If these modules confirm that no blocking calls will occur, then there should be as many processes as there are CPU cores; On the other hand, if blocking calls are possible, a slightly larger number of worker processes need to be configured. For example, if the business aspect causes users to request a large number of reads from static resource files on local disk, and memory on the server is so small that most requests to access static resource files must read from disk (the head addressing is slow) rather than from the disk cache in memory, Then disk I/O calls may block worker processes for a small amount of time, resulting in overall performance degradation of the service.

Maximum number of connections per worker process

Syntax: worker_connections number;

Default: worker_connections 1024;

Worker_cpu_affinity cpumask] [cpumask…

Binds the Nginx worker process to the specified CPU kernel

Why bind the worker process to the specified CPU kernel? Assuming that each worker process is very busy, if multiple worker processes are competing for the same CPU, synchronization problems can occur. On the other hand, if each worker process has an exclusive CPU, complete concurrency will be realized in the kernel scheduling strategy.

For example, if you have four CPU cores, you can do the following configuration:

worker_processes 4;
worker_cpu_affinity 1000 0100 0010 0001;
Copy the code

Notice The worker_CPU_affinity configuration is valid only for Linux operating systems.

View the current machine CPU

# check the number of physical CPU machine cat/proc/cpuinfo | grep "physical id" | sort | uniq | wc - l # 1 # view the current number of physical CPU core machine cat/proc/cpuinfo | grep "CPU cores"|uniq #cpu cores : 2Copy the code

Nginx worker process priority setting

Syntax: worker_priority nice;

Default: worker_priority 0;

The priority is determined both by the static priority and the dynamic adjustment (currently only ±5) that the kernel makes based on process execution. The value of NICE ranges from -20 to +19. -20 is the highest priority and +19 is the lowest priority. Therefore, if users want Nginx to consume more system resources, nice can be configured to be smaller, but it is not recommended to be smaller than the kernel process nice value (typically -5)

Maximum number of handle descriptors that can be opened by Nginx worker processes

Syntax: worker_rlimit_nofile limit;

Default: empty

Change the maximum number of open files for worker processes. If not set, this value is the operating system limit. Your operating system and Nginx can handle more files than “ulimit -a”, so set this value high so Nginx doesn’t have “too many open files” problems.

Whether to open the Accept lock

Grammar: accept_mutex [on | off]

Default: accept_mutext on;

Accept_mutex is a load balancing lock for Nginx. When the number of connections established by a worker process reaches 7/8 of the maximum number of connections specified by worker_connections, it greatly reduces the chance that the worker process will attempt to establish a new TCP connection. The Accept lock is turned on by default. If turned off, it takes less time to establish TCP connections, but the load between worker processes is very uneven, so it is not recommended to turn it off.

The delay between using the ACCEPT lock and actually establishing a connection

Syntax: accept_mutex_delay Nms;

Default: accept_mutex_delay 500ms;

After using the Accept lock, only one worker process can get the Accept lock at the same time. This accept lock is not a blocking lock and will return immediately if it is not retrievable. If only one worker process attempts to obtain the lock and fails to obtain it, it must wait at least the time defined by accept_mutex_delay before attempting to obtain the lock again.

Nginx cache combat

Case analysis

An e-commerce platform needs to achieve 700+ QPS on the commodity details page, how to do it?

The product details page covers the following main services:

• Product details page HTML page rendering
• Price of service
• Promotional services
• Inventory status/delivery to service
• Advertising word service
• Pre-sale/SEC kill service
• Evaluation of the service
• Trial service
• Recommended service
• Product Introduction Service
• Some special services related to each category

Solution:

1. Use Ajax to dynamically load price, advertising, inventory and other services

2. Use key value to cache the main HTML of the detail page.

Solution architecture:

Question:

At 500QPS it is difficult to continue to pressure up.

Analyze the reasons:

A detail page HTML body up to an average of 150 KB then at 500QPS has approached the kilo-megabyte LAN broadband limit. Intranet traffic must be reduced.

Nginx static cache based solution:

Cache configuration

The configuration steps

• Client and proxy request caching
• Set the cache space to store cache files
• Use cache space in location
• Open the file cache configuration

The buffer size of the client request body
client_body_buffer_size 512k;
The buffer size of the client request header, which can be set based on the system paging size
client_header_buffer_size 4k;
client_max_body_size 512k;
large_client_header_buffers 2 8k;
proxy_buffer_size 16k;
proxy_buffers 4 64k;
proxy_busy_buffers_size 128k;
proxy_temp_file_write_size 128k;
# specify under which circumstances a failed request should be sent to the next backend server
proxy_next_upstream http_502 http_504 http_404 error timeout invalid_header;

Set cache space to store cache files
proxy_cache_path /usr/local/nginx/cache levels=1:2 keys_zone=nginx-cache:20m max_size=50g inactive=168h;

# use cache space in location, pathName is the project directory, please customize
location /pathname { 
	proxy_set_header X-Real-IP $remote_addr;
	proxy_cache nginx-cache;
	proxy_cache_valid 200 304 302 5d;
	proxy_cache_valid any 5d;
	proxy_cache_key '$host:$server_port$request_uri';
	add_header X-Cache '$upstream_cache_status from $host';
	proxy_set_header X-Real-IP $remote_addr;
	proxy_pass http://localhost/pathname;
}

Open file cache configuration
Max specifies the number of cached files, which is recommended to be the same as the number of open files. Inactive means how long it will take before the cache is deleted before the file is requested.
open_file_cache max=65535 inactive=60s;

The # file descriptor is always open in the cache, and if a file is not used once in inactive time, it will be removed.
open_file_cache_min_uses 1;

# specify how often to check the cache for valid information.
open_file_cache_valid 80s;
Copy the code

Description of cache parameters

The parent element	The name of the	describe
http	proxy_cache_path	Specifies the root path for the cache
	levels	The cache directory level contains the highest three layers, and each layer contains 1 to 2 characters. For example, 1:1:2 represents three layers.
	keys_zone	Cache block name and memory block size. Such as cache_item: 500 m. Cache_item specifies a cache_item whose size is 500m. The earliest data that exceeds the size will be cleared.
	inactive	Maximum idle time :10d If a data has been idle for 10 days, it will be cleared
	max_size	Maximum number of hard disks in the cache. Excess idle data will be cleared
location	proxy_cache	Specifies the cache area, corresponding to the value set in keys_zone
	proxy_cache_key	Assemble cache keys using parameters such as: $host$uri $is_args$Args will use the full path MD5 value as the Key
	proxy_cache_valid	Set the cache validity period for different status codes

 

View the cache directory

As a result of the above configuration, after the restart, we can see that many cache files are generated in the cache storage path: /usr/local/nginx/cache, levels=1:2, level 2, level 2, level 2, level 2, level 2 During the test, you can go to this directory to check whether the cache file is generated.

If we open one of the files, we’ll notice something strange. It contains the request header information. When we process an HTTP request, it reads from here first.

Cache clearing

This functionality can be implemented using the third-party module ngx_CACHE_purge. Add the ngx_cache_purge module to nginx

#Download the ngx_cache_purge module package, where nginx version 1.6.2 purge corresponds to version 2.0Wget HTTP: / / http://labs.frickle.com/files/ngx_cache_purge-2.3.tar.gz#Check the installed modules
./sbin/nginx -V
#Go to the nginx installation package directory and reinstall --add-module full path to decompress the module./configure --prefix=/root/svr/nginx --with-http_stub_status_module --with-http_ssl_module - add - the module = / root/SVR/nginx/models/ngx_cache_purge - 2.0#recompile
make
#Copy the installation directory /objs/nginx file to replace the original nginx file
#Check Whether the installation is successful
nginx -t 
Copy the code

Clear configuration:

location ~ /clear(/.*) {
  The IP address allowed to access
   allow           127.0.0.1;
   allow           192.168.0.193;
   Disable IP addresses
   deny            all;
   # configure to clear specified cache and path (with proxy_cache_key 1 to)
   proxy_cache_purge    cache_item '$host:$server_port$request_uri';
}    
Copy the code

Direct access after the configuration: 192.168.0.193 domain name is set to www.test.com

# access build cache file
http://www.test.com/?a=1
Clear the generated cache. If the specified cache does not exist, a 404 error will be reported.
http://www.test.com/clear/?a=1
Copy the code

Specifies that pages are not cached

Configuration syntax:

proxy_no_cache
grammar	proxy_no_cache string …
The default	—
scope	http,server,location
note

Example:

.Check whether the current path is the specified path
if($request_uri ~ ^/(url3|login|register|password\/reset)) {
		# Set a variable to store whether a cache is needed
  	set $cookie_nocache 1;
}

location/ {...proxy_no_cache$cookie $arg_nocache $arg_comment; . }...Copy the code

Cache hit analysis

Add hit display to HTTP headers

Nginx provides the variable $upstream_cache_status to display the status of the cache. We can add an HTTP header to the configuration to display the status, similar to squid.

 location  / {
        proxy_redirect          off;
        proxy_set_header        Host            $host;
        proxy_set_header        X-Real-IP       $remote_addr;
        proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_connect_timeout   180;
        proxy_send_timeout      180;
        proxy_read_timeout      180;
        proxy_buffer_size       128k;
        proxy_buffers           4 128k;
        proxy_busy_buffers_size 128k;
        proxy_temp_file_write_size 128k;
        proxy_cache cache;
        proxy_cache_valid 200 304 1h;
        proxy_cache_valid 404 1m;
        proxy_cache_key '$host:$server_port$request_uri';
        add_header  Nginx-Cache "$upstream_cache_status";
        proxy_pass http://backend;
    }
Copy the code

Using curl or a browser, you can view the following headers:

HTTP / 1.1 200OK
Date: Mon.22 Apr 2013 02:10:02 GMT
Server: nginx
Content-Type: image/jpeg
Content-Length: 23560
Last-Modified: Thu, 18 Apr 2013 11:05:43 GMT
Nginx-Cache: HIT
Accept-Ranges: bytes
Vary: User-Agent
Copy the code

$upstream_CACHE_STATUS contains the following states:

• MISS is missed and the request is sent to the back end
• HIT cache HIT
• EXPIRED cache EXPIRED requests are passed to the back end
• UPDATING the cache and the old reply will be used
• STALE back end will get STALE reply

Nginx cache hit ratio statistics

Nginx provides us with the $upstream_cache_status function, which naturally writes hit status to the log. The log format can be defined as follows:

log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                  '$status $body_bytes_sent "$http_referer" '
                  '"$http_user_agent" "$http_x_forwarded_for"'
                  '"$upstream_cache_status"';
Copy the code

HIT ratio statistics method: Divide the number of hits by the total number of logs to get the HIT ratio of cache:

After understanding the working principle, you can also use the crontab script to collect daily hit ratio statistics in a log for future reference.

# crontab -l1 0 * * * /opt/< a href="http://www.test.com/" title="shell"target="_blank"> shell< /a> /nginx_cache_hit > > /usr/local/nginx/logs/hitCopy the code

The contents of the visit script are as follows:

#! /bin/bashLOG_FILE='/usr/local/nginx/logs/access.log.1' LAST_DAY=$(date +%F -d "-1 day") awk '{if($NF==""HIT"") hit++} END {printf  "'$LAST_DAY': %d %d %.2f%n", hit,NR,hit/NR}' $LOG_FILECopy the code

Nginx other optimization measures

Dynamic and static separation

Save static files to the /usr/share/nginx/html folder and configure static file blocking rules. A single project can be placed directly in the /usr/share/nginx/html root directory. If multiple projects are involved, create folders based on the project root directory to store static files.

Configure the interception rule as follows:

location ~ .*\.(eot|svg|ttf|woff|jpg|jpeg|gif|png|ico|cur|gz|svgz|mp4|ogg|ogv|webm) {
	# All static files read directly from hard disk
	root /usr/share/nginx/html;
	expires 30d; Cache for 30 days
}

location ~ .*\.(js|css)?/usr/share/nginx/ HTML; expires 12h; }
Copy the code

The result: The entire site will be slower to access and most of the content will be read from static file directories or hard drives.

Connection timeout

If the connection resources are held for a long time, the requests will accumulate and Nginx will not be able to handle the requests efficiently. Therefore, we should pay attention to setting timeout time for connection control, and automatically recover resources through the timeout mechanism to avoid resource waste.

Client and server Settings
server_names_hash_bucket_size 128;
server_names_hash_max_size 512;
# Long connection timeout configuration
keepalive_timeout  65;
client_header_timeout 15s;
client_body_timeout 15s;
send_timeout 60s;

# Proxy Settings
Timeout for establishing a connection to the backend server. Note that this should not be longer than 75 seconds
proxy_connect_timeout 30s;
proxy_send_timeout 120s;
# timeout for reading responses from back-end servers
proxy_read_timeout 120s;
Copy the code

GZIP compression

Before we do the gzip package, it is best to compress some static files into min files and merge them into the same file when requested. After gzip compression, you will find that the site loads faster again.

# Enable gzip to reduce the amount of data we send
gzip on;
# Minimum number of bytes allowed for compression
gzip_min_length 1k;
#4 units of 16K memory for the compressed result stream cache
gzip_buffers 4 16k;
The default is 1.1
gzip_http_version 1.1;
# Gzip compression ratio, can be set in 1~9, 1 is the minimum compression ratio, the fastest, 9 is the maximum compression ratio, the slowest, CPU consumption
gzip_comp_level 4;
Type of compression
gzip_types text/plain text/css application/json application/x-javascript text/xml application/xml application/xml+rss text/javascript; 
Some browsers support compression, some do not support compression, so avoid waste does not support compression, so the client HTTP header to determine whether to use compression
gzip_vary on;
Gzip compression below IE6 is poorly supported by some versions of IE6
gzip_disable "MSIE [1-6].";
Copy the code

Access to the current limit

We build sites so users can access them, and we want to use them for legitimate access. So some measures have to be taken to limit users who abuse access. This abuse refers to the number of connections requested to the server per second from the same IP address. Because it could be multiple attempts by crawler robots on multiple machines around the world to crawl the site’s content at the same time.

Limit the number of connections to prevent DOS attacks
limit_conn_zone $binary_remote_addr zone=perip:10m;
limit_conn_zone $server_name zone=perserver:10m;
# limit the maximum number of concurrent connections for the same client IP address
limit_conn perip 2;
# limit the maximum number of concurrent connections to a server
limit_conn perserver 20;
Limit the download speed according to your server bandwidth configuration
limit_rate 300k; 
Copy the code

Efficient data transfer configuration

Enable efficient file transfer mode. Tcp_nopush and tcp_nodelay prevent network and disk I/O congestion and improve nginx working efficiency.
sendfile on;
# Packets are not immediately sent out. When the maximum number of packets is reached, they are sent out at once, which helps to solve network congestion.
tcp_nopush on;
# As soon as a packet is generated, no matter how small, send it as soon as possible
tcp_nodelay on;
Copy the code

Optimization of kernel parameters

Edit the /etc/sysctl.conf file and adjust parameter Settings as required.

Set tcp_max_TW_BUCKETS to 180000 if timewait is lowered
net.ipv4.tcp_max_tw_buckets = 5000

# Enable reuse, allowing time-wait state sockets to be reused for new TCP connections
net.ipv4.tcp_tw_reuse = 1

# The maximum number of TCP sockets in the system that are not associated with any user file handle. If this number is exceeded, the orphan connection is immediately reset and a warning message is printed. This restriction is intended only to prevent simple DoS attacks
net.ipv4.tcp_max_orphans = 262144

# How often TCP sends keepalive messages when keepalive is in use. The default value is 2 hours. We can shorten the time span
net.ipv4.tcp_keepalive_time = 30
Copy the code

The log configuration

Log files are critical to our daily operations and maintenance. Without logs, it would be difficult to identify anomalies, and solving problems would be like finding a needle in a haystack. Log storage is necessary, indispensable, we look at how to configure conducive to troubleshooting problems?

• Keyword: The keywords access_log and error_log cannot be changed
• Error_log error log level: [the debug | info | notice | warn | error | crit | alert | emerg], the higher the level the less information. Do not set the info level lower than this level, which can cause a lot of disk I/O consumption.

Error_log production scenarios are generally warn | error | crit one of these three levels

# define log template
log_formatLog template Name Log format;Access log
access_logpath format gzip[=level] [buffer=size] [flush=time]; Log format template Gzip compression. Level Indicates the compression level. Size of the cache for storing cached logs# error log
error_log<FILE> <LEVEL>; Keyword log file error log level# sample
log_format main '$remote_addr - $remote_user [$time_local] "$request"'
                        '$status $body_bytes_sent "$http_referer"'
                        '"$http_user_agent" "$http_x_forwarded_for"';
Copy the code

Some pictures from the network, copyright to the original author, delete.Copy the code

Thank you for the above, please, please, please!

💗 big guys with close attention to my WX public number [dime money small assistant] and nuggets column [dime money] more dry goods waiting for you to ~ ~