Why does the container stop automatically?
Why does the server stall?
Where does the process’s mysterious connection point?
Discover – a container that automatically stops
One day, I found that a container deployed on the server was stopped. I thought it was stopped or deleted by a colleague’s misoperation.
But when I log in to the server and restart the container, I notice a strange phenomenon: the container automatically stops after a few seconds.
In general, this could be a problem with the container itself.
However, checking the container logs did not get any error messages, and the container image has been deployed and running stably on other servers, so there should be no bugs.
So the guess is that the system resources are insufficient, such as disk, memory, CPU.
However, when I run the top command to check the CPU and memory, an exception is found: the CPU usage of a process reaches 99%.
Of course, this is not a big surprise for our servers, where we usually perform some computing tasks and use a lot of CPU.
But since this server is used by almost no colleagues other than me, and the process command line is not visible, it aroused my suspicion.
Validation – There is more than one exception
Mining process identification
With such high CPU utilization, I’m reminded of the recent mining virus.
Run the netstat -anp command to check whether the process has established an external network connection.
Sure enough, a connection was established to the IP address 5.196.26.96. At https://www.ipip.net/ip.html to the IP address to a foreign place.
Further confirms my suspicions. Because domestic servers have strict record management mechanism, many attackers will deploy servers abroad.
In order to further confirmation, again to threat information platform for query https://x.threatbook.cn/ip/5.196.26.96.
The platform also issued a threat warning, and it is safe to assume that this is a mining process.
Of course, if you want to further confirm, you can extract the MD5 value of the execution file to the relevant website for identification.
Where does the mining program come from?
Mining programs typically have scripts downloaded and executed by trojans, so use the history command to check the download behavior.
No suspicious downloads were found, so it’s likely that hackers cleaned up the operation logs or downloaded the files from other sources.
To further rule out the possibility that other viruses may be used as daemons to start mining processes periodically or at startup, check crontab configuration information.
No new suspicious files were found, so the hacker probably didn’t set a scheduled task.
No suspicious boot option configuration was found.
Suspicious image and container
At this point, the trail stops. Can only think of another Angle ~
According to the administrator, this server is rarely used at ordinary times, and the use of a strong password, the possibility of password leakage is very small.
Combined with the container stop time I deployed, the server was hacked within a few hours of my deployment.
So suspicion probably has something to do with my operation.
A new situation was found when using the docker command to search.
Some containers use unknown images (Heybb /theimg2) or unofficial ones (Zoolu/Ubuntu).
You can’t find a Dockerfile or readme for any of these images in the Docker Hub. And they’re all new, but downloads are growing fast.
It is strange that this unexplained, oddly named image can be downloaded multiple times, so it can be inferred that the hacker uploaded the image carrying the Trojan.
Then the docker inspect command is used to check these containers, and it is found that the container does not write system files through the mount directory, but will execute a script file of mac.sh.
To view the file with the cat command, there is only one line
This is clearly a Monro coin mining operation.
summary
Now it turns out that more than one hacker has penetrated the server, either by deploying mining containers, or by deploying mining processes and deleting records.
Processing – Clean up the process and close the bug
The first task, of course, is to clean up the mining process and container, along with the corresponding execution files and images.
This, of course, is a palliative.
To fundamentally solve the problem, you need to analyze the source to prevent the server from being invaded again.
Combined with the above clues and personal experience analysis, it is likely to use the vulnerability of Docker to invade.
I started the Docker remote API service when deploying the container. It is likely that the service was exposed to the public network, so I immediately entered the server IP address and corresponding port in the browser, and sure enough, I could access it!
The server operator does not provide the default firewall service, and the ports on the server are directly exposed to the public network.
The way of hacking can also be basically guessed, through the Docker remote API server operation container, container with mining process deployed on the server.
Or copy the mining program to the server as a directory mount to be triggered and executed in some way.
Fixing this vulnerability is as simple as shutting down the external exposure service.
advice
Network security is a very important topic, but developers often do not pay enough attention to it.
In view of this incident, several experiences have been summarized:
Do not use the default port except for some Web services (HTTP/HTTPS).
Hacker’s invasion operations are generally automated and batch.
The action is to use the port scan tool to scan for a specific default port.
For example, in this case, port 2375 of the server must be detected (2375 is the default port of the Docker Remote API).
The idea is a bit like phishing, which uses low-level trickery to screen out gullible people.
So we usually try to avoid using the default port when writing programs.
Do not expose services that do not need to be accessed externally by binding 0.0.0.0.
Before starting Docker remote API service listening 0.0.0.0 IP, because I saw many online tutorials are configured like this, but in fact there are great security risks. (There’s a big difference between getting something done and getting it done!)
In fact, the service does not need to be provided to the external network, in fact, as long as the listening subnet IP is enough. For example, 127.0.0.1 and 172.17.0.1.
Consider the abnormal as much as possible
In addition to considering the normal input and output of the program during development, we also need to assume some special cases to test.
Here’s how developers and hackers think differently:
Developer: A – program – B
Hacker: S – program -?
The developer is thinking about making sure that if you type A, you get B. Hackers will often find bugs or vulnerabilities by typing in S that the developer did not consider.
Use a firewall to restrict port access.
Network services, firewalls are important.
This intrusion has something to do with the mindset that cloud server vendors have their own firewalls.
Authenticate visitors with certificates.
For services that need to provide external access, authentication is also an effective way to avoid attacks.
Docker, for example, supports TLS certificates to authenticate server and client identities.
conclusion
Searching for a Trojan horse is a lot like playing detective, looking at the clues at the crime scene to find out who did it and how.
Fortunately, when the problem was discovered, the system was not immediately reinstalled in such a simple and crude way to solve the problem, otherwise the vulnerability still exists, the server will still be attacked.
For more authoritative knowledge on network security, you can refer to OWASP TOP10 2017, which contains the 10 most common vulnerabilities and defense measures.
The Docker Remote Unauthorized vulnerability in this article and similar Redis unauthorized vulnerabilities are OWASP TOP 10 vulnerabilities.