First, the cause of the incident

  • Some time ago to the Docker environment teaching video recording, links to reference www.bilibili.com/video/BV1BK… . , in order to better demonstration, in ali cloud made a cloud server, their local connection.
  • After the cloud server in hand, in order to allow their local can be connected to ali cloud server to do the following operations:
    • Open SSH (port:22) port: for remote connection.
    • All other ports are opened: relevant containers are started when teaching videos are recorded. Some fixed ports of containers, such as port 6379, are mapped to random ports of the host computer. In order to access the contents of containers from the public network, these ports are opened (this is a hidden danger).

​

2. Cause investigation

  • During the recording of the teaching video, I made some relevant demonstration cases, for example, the container containing Redis/Nginx and other services was built in the cloud server (but the Redis service did not set the password), as shown in the figure below

​

  • Boy, right after I set up these containers, the cloud servers in the back started calling me. The first is CPU inflation (which is a basic feature of mining), as shown in the figure below

​

  • Then there are the safety warnings

​

  • When you open the Docker 3 network, yes, Redis has been hacked.

​

  • Possible cause: Malicious attacks are launched to access local services and upload malicious scripts.
  • I called the police for a day or two, and at first I didn’t care, but then I couldn’t help it, and I could let you play in front of me?? Then turn it on and fuck you.

3. Find out the destination of the mining machine (Singapore)

  • CPU skyrocketed, it must be the background malicious program in the run, first top command to check, sure enough, two common mining virus, crypto and Pnscan
top
Copy the code

​

  • These malicious programs usually reside in /usr/share. Check them out and run the following command to remove them
# to check the ls -l/usr/share | gre cry # delete sudo rm crypto - rf/usr/share / * * # to confirm whether delete completed under the ls -l/usr/share | gre cry # to check the ls -l / usr/share | gre scan # delete sudo rm - rf/usr/share / * scan *Copy the code

​

  • Then check the network situation, the program must be some of the local data to the outside, the result found that this guy changed a lot of my local command all, annoyed me, such as the following netstate command to delete my executable permission.
Netstat -aptn sudo netstat -aptn # where to run netstat whereis netstat /bin/netstatCopy the code

​

  • Netstat does not have x permissions, chmod does not have x permissions for netstat, chmod does not have x permissions for netstat, chmod does not have x permissions for netstat, chmod does not have x permissions for netstat.
sudo chmod +x /bin/netstat
lsattr /bin/netstat
Copy the code

​

  • As you can see from the lsattr command above, /bin/netstat has an I permission, which means that the file cannot be modified, that is, cannot be given executable permission. Here are two commands:
    • Lsattr: displays the file properties.
    • Chattr: corresponds to lsattr and is used to set file properties.
  • The solution is to use the chattr command to remove the I attribute. The result is as follows: the chattr command cannot be executed because chattr does not have executable permissions
Sudo chattr -i /usr/bin/netstat -l /usr/bin/chattrCopy the code

​

  • Chattr has an I attribute and cannot be assigned permissions.
Sudo chmod +x /usr/bin/chattrCopy the code

​

  • Oh, ho, what do we do now? Chattr can neither assign permissions nor change its own properties. Then an idea occurred to me:
    • I have the chattr command on my other machine, so copy the chattr command on the other machine to the machine being attacked.
    • Because the chattr command on the other machine is good, it can be used directly.
  • Type the following command to copy the chattr command from another machine to the local directory, save it as ~/chattr_tmp, and remove the I attribute of the problematic chattr command with chattr_tmp, as shown below:
# will 0.0.0.0 machine/usr/bin/chattr copy to the local, saved as a ~ / chattr_tmp sudo SCP ubuntu: 0.0.0.0: / usr/bin/chattr ~ / chattr_tmp # check, Sudo /usr/bin/chattr /home/dongyusheng/chattr_tmp -i /usr/bin/chattrCopy the code

​

  • After the above execution is complete, you can give chattr the executable permission as follows:
Sudo chmod +x /usr/bin/chattr +x /usr/bin/chattr +x /usr/bin/chattrCopy the code

​

  • Use chattr to remove the I attribute of netstate, and then grant it executable permission, as shown below:
sudo chattr -i /bin/netstat
sudo chmod +x /bin/netstat
ls -l /bin/netstat
Copy the code

​

  • Then check the network information, you can see that Crypto is transmitting data to a machine with IP 172.104.165.191, the IP check is Singapore machine.
sudo netstat -aptnl
Copy the code

​

​

Delete malicious processes

  • The kill command does not have the executable permission, so we have to use the same method to grant the executable permission to the kill command

​​

whereis kill
sudo ls -l /bin/kill
sudo chmod +x /bin/kill
lasttr /bin/kill
sudo chattr -i /bin/kill
lsattr /bin/kill
sudo chmod +x /bin/kill
sudo ls -l /bin/kill
Copy the code

​

  • The kill command can be used to kill both processes
sudo kill -9 657 604
Copy the code

​

  • After kill view view process again, you can see the CPU back to normal, the program is gone
top
Copy the code

​

Delete the container image

  • Then checking the local container (the container has been stopped by me, and this is the screenshot below), the guy builds me a container that used to run the script.
# check whether sudo docker ps-a is running or notCopy the code

​

  • And if I look at the mirror, this guy built me a mirror
Sudo Docker images sudo Docker ImagesCopy the code

​

  • I tried to delete the container, but it failed because some content was linked to it.
Sudo docker rm 5e3defeb058cCopy the code

​

  • Go to the container directory and find the contents of the container
cd /var/lib/docker/containers
sudo find . | grep 5e3defeb058c
Copy the code

​

  • Delete them when you find them, no mercy.
sudo rm -rf 5e3defeb058cd75b39740cd1584945f9a822a022b2a7c3bf025ee027017edb0f
sudo rm -find . | grep 5e3defeb058c
Copy the code

​

  • And then deleting the container again is still not ok.

​

  • There is no choice but to restart the Docker process. Docker: “systemctl” can’t execute systemctl: “systemctl” can’t execute systemctl: “systemctl” can’t execute systemctl: “systemctl” can’t execute systemctl”
sudo service docker restart
whereis service
sudo chattr -i /usr/sbin/service
sudo chmod +x /usr/sbin/service
sudosudo service docker restart
Copy the code

​

  • Then decisively to systemctl program to grant executable permission, command to restart the Docker service, success.
whereis systemctl
ls -l /bin/systemctl
sudo chattr -i /bin/systemctl
sudo chmod +x /bin/systemctl
sudosudo service docker restart
Copy the code

​

  • Check the container again, hahahahahahaha, the container is gone.
Sudo Docker ps-aCopy the code

​

  • If the container is gone, the image can be deleted, as shown below.
Sudo docker iamges sudo docker iamges rmi b4a24c6be971Copy the code

​

Sixth, concluding remarks

  • Make me a long time this thing, here to give you a general process:
    • Step 1: The cloud server is under attack.
    • Step 2: Enter the server investigation, found malicious programs.
    • Step 3: delete malicious programs, although in the deletion of a lot of commands were found to be tampered with, but in my clever solution to these commands.
    • Step 4: Check that the container is also attacked and delete the container and associated images.
  • Here are some tips for using cloud servers:
    • Do not open some important ports.
    • If you are forced to open the port, please configure the relevant password. The password should not be too simple, otherwise it is easy to be cracked by violence.
    • If the cloud server is no longer in use and has no functions to use, try to shut it down.
  • Here I closed the cloud server related important port, first temporarily develop SSH:

​

  • In addition, after the Docker teaching video recording is completed, I plan to restart and install the cloud server, and I will not open the port at will later.
  • Finally to give yourself a advertising, video speak well of www.bilibili.com/video/BV1BK… .