🔥Redis collapsed. I managed to dump the pot on the next team

Record a Redis connection overload troubleshooting process.

The article addresses: blog.piaoruiqing.com/2019/12/15/…

The project won’t get up!

. The project won’t get up again!

. 叒 yi !!!!!!!!!

Last week, I heard a lot of colleagues say that the project has lost 叒 yi, and Redis can’t connect to it. I am busy developing new requirements in another normal environment, so I don’t have time to worry about it.

So, after looking at the log, the number of connections is too many… Emmm, I added 0 to the upper limit of connection number in Redis configuration for my colleague, troubleshooting and so on.

ERR max number of clients reached
Copy the code

Finally… The total can not escape, the author with the environment was also done collapse, and is inexplicably cold.

But… Blow stine!

After all, modifying the upper limit of the number of connections is a palliative rather than a cure. Originally, the upper limit of the number of connections is 10000, and there are only dozens of micro-services in total.

Someone must have left the connection open. Look into him.

Known:

1. After the Redis service restarts, it takes a while for the connection to be full again.
2. There are dozens of services in total. The configured number of 10,000 connections is unlikely to be used up under normal circumstances.

The first step is to restart and preempt a connection

After rebooting Redis, I immediately connected to Redis and checked the number of clients.

$ docker exec -it $(docker ps | grep redis | awk '{print $1}') redis-cli -a {pwd}
127.0.0.1:6379> info
...
# Clients
connected_clients:391
...
Copy the code

tips:

The info command displays various information and statistics about the Redis server.

Step 2 Record all clients

Check again a few minutes later:

127.0.0.1:6379> info
...
# Clients
connected_clients:10002
...
Copy the code

At this point, the connection is fully occupied.

Save all client info in a file and get ready to catch this guy who’s doing this.

127.0.0.1:6379> Client list ID =7863 addr=172.18.0.104:56836 fd=6150 Name = age=72 IDLE =72 FLAGS =N DB =0 sub=0 psub=0 Multi =-1 qbuf=0 qbuf-free=0 obl=0 OLL =0 OMem =0 events=r CMD =ping ID =7864 addr= 172.18.0.50/56262 fd=6151 name= age=72 idle=72 flags=N db=9 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping id=7865 Addr =172.18.0.104:56840 fd=6152 Name = age=72 IDLE =72 FLAGS =N DB =0 sub=0 psub=0 multi=-1 qBUf =0 qBUf free=0 OBL =0 OLL =0 omem=0 events=r cmd=ping ...Copy the code

tips:

Client list: lists all client information.

Step 3 Find the abnormal IP address

With all the client connection information, you can find out whose pot it is. Run the following command to output the top five IP addresses:

$ cat client-list | awk '{print $2}' | awk -F "[= :]." '{print $2}'| sort | uniq -c | sort - k1, 1 nr | head 4244-5 5432 172.18.0.50 172.18.0.104 43 172.18.0.59 40 172.18.0.54 32 172.18.0.55Copy the code

So far, 172.18.0.50 and 172.18.0.104 are locked. These are both addresses for the Docker internal network.

tips:

Awk ‘{print $2}: print IP. Addr =172.18.0.104:56836.

Awk -f “[=:]” ‘{print $2}’: split addr=172.18.0.104:56836 with colons

Sort: sorting.

Uniq -c: Counts the quantity and displays the number of repeats of the row next to each column.

Step 4 Locate the service and throw the pot

Get the IP is not far from the target, through docker inspect can output docker instance information, including IP.

$ docker inspect --format='{{.Name}} - {{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}'$(docker ps - aq) | grep 172.18.0.50 / docker_xxxxx - service - 172.18.0.50 $docker inspect - format ='{{.Name}} - {{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}'$(docker ps - aq) | grep 172.18.0.104 / docker_yyyyy - service - 172.18.0.104Copy the code

tips:

Docker inspect: fetch metadata of container/image. –format: format output with template.

XXXXX and YYyyy, MMMM… Service for the Py (thon) group next door. Here, here’s the pot.

If this article is helpful to you, please give a thumbs up (~ ▽ ~)”

Recommended reading

• MySQL high frequency interview questions soul torture
• How do I block risky IP addresses in real time via Nginx logs
• Repeatedly suit! How to migrate data
• How to Learn to program
• Open API Gateway Practice # 1 — Design an API gateway
• Open API Gateway Practice ii – Replay attack and Defense
• Build K8S from scratch with official documentation

Welcome to pay attention to the public account (code such as poem)