(1) Cannot assign requested address

There was an uproar in the wechat group.

It was Friday, I thought, and it was the end of the day, but it wasn’t that exciting.

Open wechat a look, the heart is cool, all reported system 403 error message. Don’t say off duty, I’m afraid the boss will let me off duty forever.

Don’t panic, in the long-term team cooperation training, I understand a truth: steady we can win.

After calming down, I analyzed the cause of the problem carefully.

403 Indicates insufficient permissions, which means that our subsystem failed to pull permissions from the authentication center. Login directly to the subsystem server, manually execute the pull permission program, it is indeed unable to pull.

My guess is good. Is the Internet down? Telnet target port try, and Linux returns this error message:

Cannot assign requested address

why

The cause of this error is that the client connection ports allocated by Linux are exhausted and the socket connection cannot be established.

As we all know, establishing a connection requires four parts: destination IP, destination port, client IP and client port. The first three of these are constant, only the client port is constantly changing.

Therefore, when a large number of frequent connections are established and the port is not released immediately, the default value is 60 seconds, the client port will be insufficient.

That’s the essence of the problem.

Verify with two commands:

Check the number of connections:

# netstat -ae | wc -l
# netstat -ae | grep TIME_WAIT | wc -l
Copy the code

View the range of available ports:

# sysctl -a | grep port_range
net.ipv4.ip_local_port_range = 50000    65000
Copy the code

The result is that the number of connections is much larger than the number of ports available.

To solve

How do you solve it? There are two options:

  1. Reduce TIME_WAIT
  2. To increase the range of available ports

Reduce TIME_WAIT

Edit the kernel file /etc/sysctl.conf and add the following contents:

// Enable SYN Cookies. When SYN wait queue overflows, enable cookies to handle SYN attacks. // To prevent a small number of SYN attacks, the default value is 0, indicating that the SYN is disabled. Net.ipv4. tcp_syncookies = 1 // Enable reuse. Allow time-Wait Sockets to be re-used for new TCP connections. Default is 0, indicating closure. Net.ipv4.tcp_tw_reuse = 1 // Fast recovery of time-wait Sockets in TCP connections is enabled. The default value is 0, indicating that fast recovery of time-wait sockets is disabled. Net.ipv4. tcp_tw_recycle = 1 // Modify the default TIMEOUT period of the system. The default TIMEOUT period is 60s net.ipv4.tcp_fin_timeout = 30Copy the code

To increase the range of available ports

Edit the kernel file /etc/sysctl.conf and add the following contents:

// Represents the range of ports used to connect outward. The value is 1024 to 65535. net.ipv4.ip_local_port_range = 1024 65535Copy the code

Finally, run sysctl -p for the parameters to take effect.

analyse

I solved the problem by increasing the range of available ports. It seems that I can leave work normally.

But that’s not all. Why are there so many connections all of a sudden? Through the analysis of the log, it was found that one of my colleagues requested the system interface crazily in the past period of time, which should be caused by this operation. I asked if it was actually crawling data. Good guy directly to the hard, looking for me to provide an API is not fragrant?

But it also reflects our system is too fragile, a little reptile to fail. After analyzing the online code, THERE are three areas that should be optimized:

  1. The authentication center should not establish a new connection with each request, but should reuse the previous connection, such as the singleton pattern;
  2. Permissions are relatively infrequent, and subsystems should build local caches rather than live requests;
  3. More than toPOSTInterface set frequency limit,GETInterfaces should also be limited.

All that’s left is to optimize the code, but have a good weekend first.

The brain map and source code in the article have been uploaded to GitHub for students who need to download.

Address: github.com/yongxinz/te…

List of Go columns:

  1. Go | Development environment setup and development tools VS Code configuration
  2. Declaration and assignment of variables and constants
  3. Basic data types: integers, floating-point numbers, complex numbers, Booleans, and strings
  4. Go column | Compound data types: Arrays and slicing
  5. Go column | Compound data types: dictionary maps and structs
  6. Go column | process control, catch all
  7. Go column | function stuff
  8. Go column | Error handling: defer, Panic, and Recover
  9. Go column | talk about how
  10. Go | interface