Blog.csdn.net/lijinqi1987…
github.com/yedf/handy\
Github.com/ideawu/c100… — code
www.ideawu.net/blog/ — c1000k\
www.ideawu.net/blog/archiv… Build C1000K server (1) – Basics \
www.ideawu.net/blog/archiv… Build C1000K server (2) – Comet server with millions of connections
\
Application scenarios
Chat rooms or instant message push systems, because many messages need to be pushed to the client when they are generated, so when no message is generated, you need to hold the client connection, so when there are a large number of clients, to hold a large number of long connections.
Server Configuration
Here we are targeting a 10M concurrent connection.
General server default limit 1024 file handle, which is the most support 1024 concurrent connections, long under the root user to edit the/etc/security/limits file, modify:
[plain] view plain copy
- * soft nofile 1048576
- * hard nofile 1048576
\
· Soft is a warning value, while hard is a true threshold, beyond which an error is reported.
· Soft refers to the Settings that are currently in effect for the system. Hard indicates the maximum value that can be set in the system
·nofile — Maximum number of open files in a single process
· The asterisk is for all users. If only one user login ID is used, replace the asterisk
In theory, 10 processes can achieve 10m concurrent long connections. However, when testing, the maximum number of concurrent connections can only reach 28200. In this case, you need to change the default local port range.
In Linux, the port range ranges from 0 to 65536. The system provides the default port range:
[plain] view plain copy
- cat /proc/sys/net/ipv4/ip_local_port_range
- 32768, 61000,
\
Therefore, the current port range is about 28200 from 61000-32768. To expand the port range, modify /etc/sysctl.conf and add a line:
[plain] view plain copy
- net.ipv4.ip_local_port_range= 1024 65535
\
Save to take effect
[plain] view plain copy
- Sysctl -p
\
Then when the number of connections issued by the test program is greater than a certain value (about 400,000), the dmesg command will give a large number of warnings: [warn]socket: Too ManyOpenFiles in system
In this case, you need to change file-max, which indicates the maximum number of file handles that can be opened at the same time for all processes in the system. Add fs.file-max = 1048576 to /etc/sysctl.conf, save sysctl -p and make it take effect.
For example, if you have a certain number of server connections, see the dmesg command for a large number of TCP: toomany of orphsockets errors.
[plain] view plain copy
- net.ipv4.tcp_mem= 786432 2097152 3145728
- net.ipv4.tcp_rmem= 4096 4096 16777216
- net.ipv4.tcp_wmem= 4096 4096 16777216
- net.ipv4.tcp_max_orphans= 131072
\
Net.ipv4. tcp_rmem is used to set the size of the read buffer. Three values, the first is the minimum value of the read buffer, the third is the maximum value, and the middle is the default value. We can change the size of the read buffer in the program, but cannot exceed the minimum and maximum. To minimize the amount of memory used per socket, I set the default value to 4096.
Net.ipv4. tcp_wmem is used to set the size of the write buffer.
The size of read buffer and write buffer directly affects the memory usage of sockets in the kernel.
Net.ipv4. tcp_mem is the size of TCP memory, which is expressed in pages. A page equals 4096 bytes. The three values are low, pressure, and high
· Low: TCP does not consider releasing memory when the number of memory pages used by TCP is lower than this value.
· Pressure: When THE number of memory pages used by TCP exceeds this value, TCP tries to stabilize its memory usage and enters pressure mode. When the memory consumption is lower than low, TCP exits pressure state.
· High: number of pages that allow all TCP sockets to be queued to buffer datagrams. If memory usage exceeds this value, the system refuses to allocate sockets, and the background log reads “TCP: too many of orphsockets”.
Typically these values are calculated at system startup based on the amount of system memory. According to the current tcp_MEm maximum number of memory pages is 1864896. When the memory is (1864896 x 4)/1024K=7284.75 MB, the system cannot allocate memory for new socket connections, that is, TCP connections are rejected. In the actual test environment, it was observed that at around 990,000 connections, the process was killed, triggering an outof socket memory error (seen with the dmesg command). Each connection occupies approximately 7.5K memory (the calculation method is given below). The approximate memory usage can be calculated (990000 x 7.5/1024K = 7251M). This matches the tcp_MEm maximum number of page values, so this value also needs to be modified.
Orphans also need to be set. Net.ipv4. tcp_MAX_orphans, which indicates the number of sockets that the system can handle that are not of any process and may need to be concerned when setting up a large number of connections quickly. As the number of sockets that don’t belong to any process exceeds this value, DMESG sees “too many of orphsockets”.
In summary, the server needs to be configured as follows:
The/etc/sysctl. Conf to add:
[plain] view plain copy
- fs.file-max= 10485760
- net.ipv4.ip_local_port_range= 1024 65535
- net.ipv4.tcp_mem= 786432 2097152 3145728
- net.ipv4.tcp_rmem= 4096 4096 16777216
- net.ipv4.tcp_wmem= 4096 4096 16777216
- net.ipv4.tcp_max_orphans= 131072
The/etc/security/limits. Conf modification:
[plain] view plain copy
- * soft nofile 1048576
- * hardnofile 1048576
\
The online test
Ucloud cloud host for testing, server configuration:
\
System resources before starting pressure test:
\
\
\
Stand-alone concurrent long connection tests are performed using the test program that comes with the github.com/yedf/handy library, which uses epoll on Linux systems and kqueue on MacOSX.
Select a host S as the server and run the server
[plain] view plain copy
- 10m/10m-svr100 300 10 301 # Start 10 sub-processes, each process listens on ports 100-300.
\
Select another host C as the client and run the client (S is the internal IP address of the server).
Start 10 client terminal processes, connect to ports 100-300 of S, initiate 10m connections, create all connections in 500 seconds, send a heartbeat every 600 seconds, 64 bytes of heartbeat data, multi-process management port 301.
[plain] view plain copy
- 10m/10m-cli<S> 100 300 10000000 500 10 600 64 301
\
On the server side, watch SS — S was used to find that the number of TCP connections continued to rise and finally stabilized at about 4m
\
\
Consumption of resources:
\
The system uses about 20 GB of memory, but very little CPU
Client error:
[plain] view plain copy
- 0m-cli:handy/logging.cc:164: void handy::Logger::logv(int, const char*, int, constchar*, const char*, …) : Assertion `0′ failed.
- 2017/05/03-15:54:58.4048281a46 FATAL Handy /poller.cc:35 epoll_ctl add failed 28 No space left on device
- 2017/05/03-15:54:58.4048121a40 FATAL Handy /poller.cc:35 epoll_ctl add failed 28 No space left on device
- 2017/05/03-15:54:58.4048251A45 FATAL Handy /poller.cc:35 epoll_ctl add failed 28 No space left on device
\
This error is typically a disk full cause, but here is a client in the epoll_ctl, memory is full cause the failure of registered epoll events, so the client stopped creating a connection at this point, visible bottlenecks in the pressure test of the client at this time, if the client memory enough, theoretically the service side 10 m concurrent long connection should be able to achieve.
Description:
Maximum number of files per process: ulimit -n can change this number up to 1048575, so a single process can open up to a maximum of one million files. Ten million concurrent connections require ten million file descriptors, so we use multiple processes to support ten million files.
Load balancing between multiple processes: Nginx uses multiple processes to increase its throughput. Originally, it used shared locks to balance the load. For servers with more cores, more processes do not achieve linear performance improvement. The latest Linux kernel introduces the SO_REUSEPORT option, which automatically balances multiple processes listening on the same port and is a kernel level solution. Handy adopted this scheme, which is better than nginx’s older approach (the latest NGINx also supports SO_REUSEPORT).
Not enough local client ports in the test: let the server listen on 200 ports, so that each port the client connects to the server only has 50K connections, then increase the default local port range to meet the requirements (see server system parameters above).
In the test, if tens of thousands of connections are created at once, the majority of connection creation will fail. Therefore, let the client create 2000 connections every 100ms to improve the success rate of connection creation.
\