preface

It is well known that Linux does not support high concurrency well under default parameters, mainly limited by the maximum number of open files in a single process, kernel TCP parameters and IO event allocation mechanism, etc. Here are a few ways to tune A Linux system to support a high-concurrency environment.

1. The Iptables

If not necessary, disable or uninstall the iptables firewall and prevent the kernel from loading the iptables module. These modules affect concurrency performance.

2. Maximum number of open files in a single process

The normal distribution limits the maximum number of files that can be opened in a single process to 1024, which is far from meeting the high concurrency requirements. The adjustment process is as follows:

At the # prompt type:

# ulimit - n 65535Copy the code

Set the maximum number of files that can be opened for a single process started by root to 65535. If the command output similar to Operationnotpermitted is displayed, the above limit fails to be modified because the value specified in Operationnotpermitted exceeded the soft or hard limit for the number of files opened by the user. Therefore, you need to change the soft and hard limits that Linux imposes on users on the number of open files.

First, modify the limits.conf file and add:

# vim /etc/security/limits.conf
 
* softnofile 65536
* hard nofile65536
Copy the code

‘*’ indicates that the limit of all users is changed. Soft or hard specifies whether to modify soft or hard limits. 65536 specifies the new limit value that you want to change, the maximum number of open files (note that the soft limit value is less than or equal to the hard limit). Save the file after modification.

Step 2 modify the /etc/pam.d/login file and add the following line to the file:

vim /etc/pam.d/login
 
sessionrequired /lib/security/pam_limits.so
Copy the code

This tells Linux to call the pam_limits.so module after the user has logged in to the system to set the maximum limit on the number of resources that the user can use (including the maximum number of files that the user can open). The pam_limits. So the module from the/etc/security/limits the conf file reading configuration to set the limit. Save the file after modification.

Step 3 to check the maximum number of open files at the Linux system level, run the following command:

cat/proc/sys/fs/file-max
32568
Copy the code

This indicates that the Linux system can open a maximum of 32,568 files at the same time (that is, the total number of files opened by all users), which is the Linux system hard limit and should not be exceeded by any user. In general, this system-level hard limit is the best maximum number of open files that Linux can calculate at startup based on the state of the system’s hardware resources, and should not be modified without special need, unless you want to set a value beyond this limit for user-level open files. To modify this hard limit, modify fs.file-max= 131072 in /etc/sysctl.conf

This is to force Linux to set the hard limit on the number of open files at the system level to 131072 after the startup is complete. Save the file after modification.

After the above steps are complete, restart the system and you can generally set the maximum number of files that can be opened simultaneously on Linux to a specified value for a single process of a specified user. If you run the ulimit-n command after the restart to check that the maximum number of files that can be opened by a user is still lower than the maximum value set in the preceding step, it may be because the number of files that can be opened by a user has been limited by using the ulimit-n command in the /etc/profile login script. When you run the ulimit-n command to change the maximum number of files that can be opened at the same time, the new value must be less than or equal to the previous value set by the ulimit-n command. Therefore, it is impossible to use this command to increase the value. If so, open the /etc/profile script and look for ulimit-n to limit the maximum number of files a user can open at the same time. If so, delete the command, or change its value to a suitable value, and save the file. The user logs out and logs in to the system again.

By doing so, you remove system restrictions on the number of open files for communication handlers that support high-concurrency TCP connection processing.

3. Kernel TCP parameters

In Linux, after the TCP connection is disconnected, the port is released in TIME_WAIT state for a certain period of time. When a large number of concurrent requests are made, a large number of TIME_WAIT connections will be generated. If the connections cannot be disconnected in time, a large number of port and server resources will be consumed. At this point we can optimize TCP kernel parameters to clean up TIME_WAIT ports in time.

The methods described below are only effective for connections with a large number of TIME_WAIT states that consume system resources, but otherwise the effect may not be significant. You can use the netstat command to check the connection status of the TIME_WAIT state by typing the following combination command,

View the current TCP connection status and the number of connections:

# netstat - n | awk '/ ^ TCP / {+ + S [$NF]} END {for (S) in a print a, S [a]}' # this command will output similar to the following results:  LAST_ACK16 SYN_RECV348 ESTABLISHED70 FIN_WAIT1229 FIN_WAIT230 CLOSING33 TIME_WAIT18098Copy the code

We only care about the number of time_waits. As you can see here, there are more than 18,000 time_waits, which takes up more than 18,000 ports. Note that the number of ports is only 65535, occupying one or less will seriously affect the subsequent new connections. In this case, it is necessary to adjust the Linux TCP kernel parameters to make the system release TIME_WAIT connections faster.

Edit the configuration file /etc/sysctl.conf and add the following lines to the file:

Vim /etc/sysctl.conf net.ipv4.tcp_syncookies= 1# SYNCookies are enabled. When SYN overflow occurs, cookies are enabled to prevent a small number of SYN attacks. The default value is 0, indicating that the SYN wait queue is disabled. Net.ipv4. tcp_tw_reuse= 1# Indicates that reuse is enabled. Allows time-WaitSockets to be re-used for new TCP connections. Default is 0, indicating closure. Net.ipv4. tcp_tw_recycle= 1# Enables the fast recovery of time-waitSockets from TCP connections. The default value is 0, which indicates that the TCP connection is disabled. Net.ipv4. tcp_fin_timeout= 30# Change the system default TIMEOUT time. # Enter the following command for the kernel parameters to take effect: sysctl-pCopy the code

After such adjustment, besides further improving the load capacity of the server, it can also defend against DoS, CC and SYN attacks with small traffic.

In addition, if you have a large number of connections, we can optimize the TCP port range to further improve the concurrency of the server. Again, in the above parameter file, add the following configuration:

Net.ipv4. tcp_keepalive_time= 1200# Indicates the frequency at which TCP sends keepalive messages when Keepalive is enabled. The default value is 2 hours. The default value is 20 minutes. Net.ipv4. ip_local_port_range= 1024 65535# Indicates the range of ports used for outbound connections. The default value is small, ranging from 1024 to 65535. Net.ipv4. tcp_max_syn_backlog= 8192# Indicates the length of the SYN queue. The default is 1024. Increase the queue length to 8192 to accommodate more network connections waiting for connections. Net.ipv4. tcp_max_TW_BUCKETS = 5000# Indicates that the system keeps the maximum number of time_waits at the same time. If this number is exceeded, TIME_WAIT will be cleared immediately and a warning message will be printed. The default value is 180000. This parameter controls the maximum number of time_waits, as long as the number is exceeded.Copy the code

You are advised to enable these parameters only on servers with heavy traffic. It is not necessary to set these parameters on servers with low traffic.

4. Other TCP parameters of the kernel

Net.ipv4. tcp_max_syn_backlog= 65536# Maximum number of connection requests that have not yet received client confirmation. The default value is 1024 for a system with 128 MB of memory and 128 for a system with small memory. Net.core.net dev_max_backlog= 32768# The maximum number of packets allowed to be sent to the queue if each network interface receives packets faster than the kernel can process them. Net.core. somaxconn= 32768# For example, the backlog for the Listen function in web applications is limited to 128 for kernel parameters, while the NGX_LISTEN_BACKLOG defined by Nginx is limited to 511. So it is necessary to adjust this value. net.core.wmem_default= 8388608 net.core.rmem_default= 8388608 net.core.rmem_max= 16777216 Wmem_max = 873200 net.core. Wmem_max = 16777216 The 0# timestamp avoids the winding of the serial number. A 1Gbps link is bound to encounter a previously used serial number. Timestamps enable the kernel to accept such "abnormal" packets. I need to turn it off here. Net.ipv4. tcp_synack_retries= 2# To open the peer connection, the kernel sends a SYN with an ACK that responds to the previous SYN. The second of the three handshakes. This setting determines how many SYN+ACK packets the kernel sends before abandoning the connection. Net.ipv4. tcp_syn_retries= 2# Number of SYN packets to be sent before the kernel distries the connection. Tcp_tw_len = 1 net.ipv4.tcp_tw_reuse= 1 Allows time-WaitSockets to be reused for new TCP connections. Net.ipv4. tcp_wmem= 8192 436600 873200#TCP write buffer :8192 436600 873200 net.ipv4.tcp_rmem = 32768 436600 Tcp_mem = 94500000 91500000 92700000 # tcp_mem= 94500000 91500000 #net.ipv4.tcp_mem[0]: Below this value, TCP has no memory pressure. #net.ipv4.tcp_mem[1]: At this value, the memory stress phase is entered. #net.ipv4.tcp_mem[2]: TCP refuses to allocate sockets higher than this value. The above memory units are pages, not bytes. Aliases = 3276800 # The maximum number of TCP sockets in a system that are not associated with any user file handles. If these numbers are exceeded, The connection will immediately reset and print a warning message, this limit is only to prevent simple DoS attacks, do not rely on it too much or artificially reduce this value, should increase this value (if increased memory). Net.ipv4. tcp_fin_TIMEOUT = 30 # This parameter determines how long a socket remains in fin-WaIT-2 state if it is closed at the request of the local end. The peer end can go wrong and never close the connection, or even go down unexpectedly. By default, it is 60 seconds. The usual value for a 2.2 kernel is 180 seconds. You can press this setting, but keep in mind that even if your machine is a lightweight WEB server, there is a risk of running out of memory due to a large number of dead sockets. Fin-wait-2 is less dangerous than Fin-WaIT-1 because it can only eat 1.5K of memory at most. But they live longer.Copy the code

It also involves a TCP congestion algorithm problem, you can use the following command to view the local provided by the congestion algorithm control module:

$sysctl net.ipv4. tcp_available_CONGestion_control # The advantages and disadvantages of TCP congestion control algorithms, applicable environment, and performance analysis. For example, hyBLA can be used for high delay and HTCP algorithm can be used for medium delay. Net.ipv4. tcp_congestion_control=hybla# If you want to set TCP congestion algorithm to hybla net.ipv4.tcp_fastopen= 3# Additional, for kernel versions higher than 3.7.1 we can enable tcp_fastopenCopy the code

5. I/O event allocation mechanism

To enable high-concurrent TCP connections on Linux, you must verify that your application uses the appropriate network I/O technology and I/O event dispatch mechanism. The available I/O technologies are synchronous I/O, non-blocking synchronous I/O, and asynchronous I/O. In the case of high TCP concurrency, if you use synchronous I/O, it will seriously block your program unless you create a thread for each TCP connection I/O. However, too many threads will cause huge overhead due to the scheduling of threads.

Therefore, the use of synchronous I/O in high TCP concurrency situations is not desirable, so consider using non-blocking synchronous I/O or asynchronous I/O. Techniques for non-blocking synchronous I/O include the use of select(), poll(), epoll, and other mechanisms. The technique for asynchronous I/O is to use AIO.

From the perspective of the I/O event dispatch mechanism, the use of SELECT () is not appropriate because it supports a limited number of concurrent connections (typically less than 1024). In terms of performance, poll() is also inappropriate. Although poll() can support a high number of TCP concurrency, due to its “polling” mechanism, when the number of concurrency is high, its operation efficiency is quite low and there may be uneven ALLOCATION of I/O events, resulting in “hunger” phenomenon of I/O on some TCP connections.

This is not the case with epoll or AIO. (Earlier Linux kernels implemented AIO by creating a thread for each I/O request in the kernel, which also had serious performance problems with high-concurrency TCP connections. But AIO has been improved in the latest Linux kernels).

To sum up, when developing Linux applications that support high concurrent TCP connections, epoll or AIO technology should be used as far as possible to realize I/O control over concurrent TCP connections, which will provide effective I/O guarantee for the support of high concurrent TCP connections by the upgrade program.

After such an optimized configuration, the server’s TCP concurrent processing capability is significantly improved. The preceding configurations are for reference only. For the production environment, adjust the configurations based on the actual situation.

Original text: blog.csdn.net/qq\_2967786…