PK creative Spring Festival, I am participating in the “Spring Festival creative submission contest”, please see: Spring Festival creative submission Contest

1. Accident scene

One day, I logged in to the server and wanted to do some tests. When I was using curl command, I found that the prompt command did not exist. Then I tested the wget command, which was the same thing. Other commands such as netstat and ps are also disabled, but top is still available.

I took a look at the load of the system (some restart processing has been done when the screenshot is taken) and found that the average load is 19.40, which is ridiculously high. There is definitely something wrong.

Run the top command, press 1 to expand the multi-core view, and press C to sort the list by CPU usage. High CPU usage, but no high CPU usage programs appear.

The top command is suspected to have been tampered with. Copy a top command from the same distribution and run it again.

At this point, the server is frantically sending out packets, which can be seen through the visual window of shell software. For a more intuitive view, use the following command to find that the server is frantically sending packets to port 6379 of 146.165.159.*.

tcpdump -nn
Copy the code

The first idea is to restrict IP addresses through the outbound rules of the firewall, but then create new links, treating the symptoms rather than the root cause. It appears to be a mining program that takes advantage of this time to back up important files to local, and then pack the less important files after they are disconnected from the network.

Because two of my student computers logged in without secret through the private key, the other one was also infected. Due to work reasons, I did not have time to deal with it and received the alarm message.

2. To deal with

Important files have been backed up to the local, cut off the network!! After the VNC login, run the command to stop the network service. (VNC here can still be used after the network is stopped)

systemctl stop network
Copy the code

Don’t worry about the problem for now, put the less important files into a package backup, and then deal with the network problems after downloading to the local. To check for scheduled tasks, use the following command:

crontab -l
Copy the code

If yes, run the following command to check:

Cat /var/spool/cron/root # Or use the following command: more /var/log/cron logCopy the code

Found that a flock -xn was executed every minute… The command. Flock hasn’t come across any. Ask Baidu.

flock

Flock — File lock under Linux

When multiple processes may be performing operations on the same data, these processes need to ensure that other processes are not also performing operations to avoid data corruption.

Typically, such processes use a “lock file”, which means that they create a file to tell other processes that they are running, and if they detect that file, they assume that another process is working on the same data. The problem is that if the process accidentally dies and does not clean up the lock file, the user will have to do it manually.

-s,--shared: Attempts by other processes to set an exclusive lock on the FD that is directed to the file fail, while attempts by other processes to set a shared lock on the FD that is directed to the file succeed. -x, -e, -- EXCLUSIVE: obtains an exclusive lock, or write lock. -u is the default. -- UNLOCK: releases the lock manually. -n, --nb, --nonblock: Non-blocking mode that returns 1 when the lock fails to be acquired instead of waiting -w, --wait, --timeout seconds: -o, --close: closes the FD that sets the lock before executing command, so that the command child does not hold the lock. -c, --command Command: Executes the following statement in the shellCopy the code

It doesn’t matter that much, just know that the script is executed at the end.

You can see from the log that the source of the script is

http://107.189.3.150/b2f628/cronb.sh
Copy the code

And downloaded it and it was blocked by the virus. Give it a thumbs up for tinder. Go back to notepad and look at the script.

Should not do things it all dry, interested friends can download samples to study, by all means avoid running in disorder!

If your machine is Aliyun, it will also remove the protection service or cloud alarm in the system.

According to the contents of the script, reverse process:

chattr -iea /var/tmp/* rm -rf /var/tmp/* rm -rf /var/spool/cron/* rm -rf /etc/cron.d/* rm -rf /var/spool/cron/crontabs Rm -rf /etc/crontab systemctl stop contab # Delete the hidden key chattr -iea /home/hilde/rm -rf /home/hilde/sudo systemctl Disable kswapd0. Service sudo systemctl stop kswapd0. Service rm/etc/systemd/system/kswapd0. Service # delete execution script chattr - ia /etc/newsvc.sh chattr -ia /etc/svc* chattr -ia /etc/phpupdate chattr -ia /etc/phpguard chattr -ia /etc/networkmanager chattr -ia /etc/newdat.sh chattr -iea /etc/ld.so.preload rm -rf /etc/ld.so.preload rm -rf /etc/newsvc.sh rm -rf /etc/svc* rm -rf /etc/phpupdate rm -rf /etc/phpguard rm -rf /etc/networkmanager rm -rf /etc/newdat.sh chattr -i /usr/lib/systemd/systemd-update-daily rm -rf /usr/lib/systemd/systemd-update-daily chattr -ia /etc/zzh chattr -ia /etc/newinitCopy the code

After clearing the ld.so.preload file, you can find the mining process in top and kill it.

Then analyze the reasons for entering. Save the files that need to be backed up and reinstall the system.

cat /var/log/secure
Copy the code

3. Checking

To facilitate analysis, you can also limit the rate on the server.

To begin, clear all queue rules for eth0

tc qdisc del dev eth0 root 2> /dev/null > /dev/null
Copy the code

Define the top-level (root) queue rule and specify the default category number. So the speed of outsourcing slowed down.

tc qdisc add dev eth0 root handle 1: htb default 20 tc class add dev eth0 parent 1: Classid 1:20 HTB Rate 2000kbit # (1KB/s = 8KBit/s)Copy the code

Checking tc Status

tc -s -d qdisc show dev eth0
tc -s -d class show dev eth0
Copy the code

Deleting TC Rules

tc qdisc del dev eth0 root
Copy the code

After limiting the rate, you can also run the following command to check the IP address and port number of outbound packets. The IP address and port number of outbound packets are blocked by the firewall or further check the process.

tcpdump -nn
Copy the code

You can also install the Nethogs program to observe the program that sends the package.

nethogs
Copy the code

[root@localhost ~]# nethogs --help nethogs: invalid option -- '-' usage: Nethogs [-v] [-b] [-d seconds] [-t] [-p] [device [device [device...]]] -v: displays the version information, in uppercase letters V. -d: delays the refresh rate, in seconds. The default value is 1. -t: tracking mode -b: bug hunting mode - - Indicates tracking mode. -p: mixed mode (not recommended). Device: Name of the device to be monitored. By default eth0Copy the code

For example, to find port 2375, run the netstat command

netstat -antpu | grep 2375
Copy the code

Find the PID scheduler for the process.

lsof -i :2375
Copy the code

kill -9 4049
Copy the code

After clearing the boot option, the mining program does not run automatically after restarting the computer.

vim /etc/rc.d/rc.local
Copy the code

Finally, remember never to take chances and reinstall the system, or you won’t know what the back door is. After investigation, my server is probably caused by redis vulnerability, and some cases are caused by the opening of DOCker API interface.