In the development of the network, I found that many students did not understand a basic problem thoroughly. How many network connections can a machine support? I think IT is necessary for me to write a separate article on this issue. Many students’ first reaction to this question is 65535. The reason is: “It is said that the maximum number of ports is 65535, so the maximum number of long connections is 65535.” Is that true? Others say it is limited by the size of a TCP connection’s quad, which is a very, very large number. Both answers are right and neither is right.

The key to understanding this problem is to separate the roles at both ends of the TCP connection – the client and the server. Any server you have is usually both a server and a client. For example, for the back-end interface of your development, for the user you are the server side. But you have to ask Redis, Mysql to get the data, and then it becomes the client. If you don’t separate the machine as a client and a server, you’ll be stuck with this question forever. So this article respectively from the client side, the server side two to launch a chat.

The client

Now let’s talk about clients separately. How many TCP connections can a machine support when it acts as a client? There’s no point in saying that. Let’s try it in code.

1. Test the cat

Before we start the experiment, let’s check the configuration of the number of ports on the machine at hand

$ sysctl -a | grep ip_local_port_range
net.ipv4.ip_local_port_range = 15000	65000
Copy the code

From the output of the kernel parameters above, you can see that the kernel has 50000 ports open for TCP connections. What follows is a seemingly long, but actually very simple, TCP client connection. Use it to connect to any of your TCP servers, such as Nginx, Redis, etc.

I use this code to initiate a connection to Nginx on one of my machines

$PHP client1.php Specifies the IP address of a serverCopy the code

Using the netstat command, we saw a steady increase in the number of connections, but when it reached 5W, we got an error

Socket_connect () failed because :Cannot assign requested addressCopy the code

If you think back to our ip_local_port_range parameter values, 65000 to 15000 only open 50,000. Actually, it’s over the limit. At this time we seem to be able to draw a preliminary conclusion. When Linux establishes connections as a client, the maximum number of connections is limited by the kernel parameter net.ipv4.ip_local_port_range. The ip_local_port_range is configurable and the theoretical range is 0-65535

The advanced

If you believe my conclusion at this point, I will take you to the ditch again. Why? Let’s look at the following experiment.

First, I run the ifconfig command to see that there are two network adapters on my machine, and each network adapter has been configured with an IP address.

Next, we changed the code in the first experiment to allow socket_bind to bind the IP before initiating the connection.

Next we launch the two consoles separately and execute the code separately. X and 10.153.x are the IP addresses of two network adapters on the experimental machine.

X IP address of a server 80 $PHP client1.php 10.143.x.x IP address of a server 80Copy the code

At this time, monitor the ESTABLISH connection of the machine through ss command, and find that it has exceeded 50,000 and is approaching 100,000.

$ss - n | grep ESTAB | wc - 90005 lCopy the code

Now we can finally reach a more accurate conclusion, limited to the ip_local_port_range parameter and limited to 65535 for clients with one Ip. But Linux alone can be configured with multiple IP addresses, and with a few IP addresses, the maximum theoretical value can be multiplied several times

Multiple network cards are not necessary. Multiple IP addresses can be configured even if there is only one network adapter. This is done with k8S, where multiple pods can be deployed on a single physical machine. But each POD is assigned a separate IP address, so you don’t have to worry about having too many pods deployed on your physical machine and affecting the number of TCP connections in your POD. The moment the IP is given to you, your POD is isolated from other applications.

The service side

Let’s now go back to the server side. What is the maximum number of concurrent connections supported by the server? Someone starts to get delightfully confused: “Is server-side theory also a port limit?” . Well, suppose our Nginx server only listens on port 80 if it is affected. Nginx can only accept one TCP connection. This is obviously ridiculous.

Ok, let’s look at another answer that’s a little bit more plausible. A TCP connection consists of a quad. Regardless of address reuse (Unix’s SO_REUSEADDR option), for our Nginx Server, its IP and port are fixed. Only remote IP (also known as client IP) and remote port (client port) in cp connection 4-tuple are mutable. The maximum number of connections it can make is 2 to the 32 power (IP number) x 2 to the 16 power (port number). That’s a big number 2.8*10 to the 14th, 200 trillion!!

On Linux, you can listen on other ports besides 80, such as 3306 for Mysql, 6339 for Redis, and of course, you can listen on all 65535 ports. So the theory goes to 2 to the 32nd power of IP addresses x 2 to the 16th power of ports x 2 to the 16th power of ports. And if you do the math, this is essentially an infinite number.

However, there is always a gap between ideal and reality, because Linux costs resources to maintain each TCP connection. Processing connection requests, keeping alive, and sending and receiving data consumes some CPU, while maintaining TCP connections mainly consumes memory. Our problem is to consider the maximum number of connections, so we don’t worry about sending and receiving data. So TCP, when it’s at rest, doesn’t consume much CPU, it consumes much memory. On Linux, memory is limited. A TCP connection that does not send data consumes around 3.3K of memory. If data is sent, you need to allocate a send cache for each TCP, depending on the size of your net.ipv4.tcp_wmem parameter. By default, the minimum is 4K. If the sending is complete, the memory consumed by the cache will be reclaimed.

Assuming you only keep connections and don’t send data, the maximum number of connections your server can make = your memory /3.3K. Given 4GB of memory, the acceptable number of TCP connections is around 1 million.

In this example, we consider the premise of holding all server-side connections in one process. In real projects, many network IO models also create a thread or coroutine for the TCP connection for the convenience of sending and receiving data. For the lightest Golang, a coroutine stack requires 2KB of memory overhead.

conclusion

How many network connections can a machine support? This simple question actually buried a hole, resulting in countless heroes and heroines confused. It’s the same as the problem of how many birds are left after nine birds have killed one in the tree, without telling you whether the tree is real or fake. It didn’t say if the gun was loud or silent. Through today’s analysis, I believe you can finally hold your head high and rub this problem under your feet.

• TCP connection client: The TCP connection theory for each IP address is limited by the IP_local_port_range parameter, as well as by 65535. However, you can configure multiple IP addresses to enhance your ability to establish connections.
• **TCP connected server machine: each listening port is theoretically large, but this number has no practical significance. The maximum number of concurrent connections depends on the size of your memory, so each static TCP connection takes about 3

.3K memory. **

My public account is “developing internal Skills and Practicing”. Here I am not simply introducing technical theories, nor only introducing practical experience. But to combine theory with practice, with practice to deepen the understanding of theory, with theory to improve your technical practice ability. Welcome you to follow my public number, also please share with your friends ~~~

The development of internal training of hard disk album:

• Illustrates the Linux network packet receiving process
• Monitoring and tuning the receiving process of Linux network packet
• Talk about TCP connection time