Linux server mining Trojan is cleared

Abnormal phenomenon

1. The total CPU usage of the server is 50%, and some processes occupy about 800% (16 cores in total).

top
Copy the code

positioning

1. Data transmission

netstat -lntupa
Copy the code

2. Scheduled tasks

[root@bogon .new]# cat /etc/passwd | cut -f 1 -d : |xargs -I {} crontab -l -u {}
* * * * * /tmp/.ICE-unix/.new/-bash > /dev/null 2>&1;
no crontab for bin
no crontab for daemon
no crontab for adm
no crontab for sync
no crontab for mail
no crontab for ftp
no crontab for nobody
no crontab for avahi-autoipd
no crontab for dbus
no crontab for polkitd
no crontab for tss
no crontab for postfix
no crontab for ntp
no crontab for sshd
no crontab for mysql
no crontab for redis
no crontab for tcpdump
no crontab for dockerroot
no crontab for systemd-network
no crontab for xdja
no crontab for ansible
Copy the code

3. Locate the Trojan horse file

[root@bogon tmp]# pwd
/tmp
[root@bogon tmp]# ls -la
total 8
drwxrwxrwt.  8 root  root  4096 Jan 22 10:31 .
dr-xr-xr-x. 18 root  root  4096 Sep 16 15:37 ..
drwxrwxrwt.  2 root  root     6 Aug 26  2016 .font-unix
drwxr-xr-x   2 root  root    30 Jan 12 16:44 hsperfdata_root
drwxrwxrwt.  3 root  root    17 Nov 13 08:17 .ICE-unix
-rw-r--r--   1 root  root     0 Jan 17 16:01 .lock
srwxrwxrwx   1 mysql mysql    0 Oct 20 13:59 mysql.sock
drwxrwxrwt.  2 root  root     6 Aug 26  2016 .Test-unix
drwxrwxrwt.  2 root  root     6 Jul 25  2019 .X11-unix
drwxrwxrwt.  2 root  root     6 Aug 26  2016 .XIM-unix
Copy the code

[root@bogon tmp]# cd .ICE-unix/
[root@bogon .ICE-unix]# ls -la
total 4
drwxrwxrwt. 3 root root   17 Nov 13 08:17 .
drwxrwxrwt. 8 root root 4096 Jan 22 10:31 ..
drwxr-xr-x  2 root root   31 Jan 22 12:29 .new

[root@bogon .ICE-unix]# cd .new/
[root@bogon .new]# ls -la
total 1840
drwxr-xr-x  2 root root      31 Jan 22 12:29 .
drwxrwxrwt. 3 root root      17 Nov 13 08:17 ..
-rwxr-xr-x  1 root root     119 Nov 13 08:18 -bash
-rwxr-xr-x  1 root root 1878432 Sep 17 01:52 x86_64
Copy the code

Processing method

1. Kill the -bash process and find that it will start automatically soon

kill -9 28332
Copy the code

2. Delete a scheduled task

[root@bogon .new]# crontab -l
* * * * * /tmp/.ICE-unix/.new/-bash > /dev/null 2>&1;
[root@bogon .new]# crontab -r
[root@bogon .new]# 
[root@bogon .new]# 
[root@bogon .new]# 
[root@bogon .new]# 
[root@bogon .new]# crontab -l
no crontab for root
Copy the code

3. Delete the Trojan horse program, or download and save the Trojan horse program for subsequent analysis and research

[root@bogon tmp]# ls -la
total 8
drwxrwxrwt.  8 root  root  4096 Jan 22 10:31 .
dr-xr-xr-x. 18 root  root  4096 Sep 16 15:37 ..
drwxrwxrwt.  2 root  root     6 Aug 26  2016 .font-unix
drwxr-xr-x   2 root  root    30 Jan 12 16:44 hsperfdata_root
drwxrwxrwt.  3 root  root    17 Nov 13 08:17 .ICE-unix
-rw-r--r--   1 root  root     0 Jan 17 16:01 .lock
srwxrwxrwx   1 mysql mysql    0 Oct 20 13:59 mysql.sock
drwxrwxrwt.  2 root  root     6 Aug 26  2016 .Test-unix
drwxrwxrwt.  2 root  root     6 Jul 25  2019 .X11-unix
drwxrwxrwt.  2 root  root     6 Aug 26  2016 .XIM-unix
[root@bogon tmp]# rm -rf .font-unix .ICE-unix .Test-unix .X11-unix .XIM-unix
[root@bogon tmp]# ls -la
total 4
drwxrwxrwt.  3 root  root    57 Jan 22 12:55 .
dr-xr-xr-x. 18 root  root  4096 Sep 16 15:37 ..
drwxr-xr-x   2 root  root    30 Jan 12 16:44 hsperfdata_root
-rw-r--r--   1 root  root     0 Jan 17 16:01 .lock
srwxrwxrwx   1 mysql mysql    0 Oct 20 13:59 mysql.sock
[root@bogon tmp]# 
Copy the code

4. Kill the Trojan horse process. After the Trojan horse process is killed, it may automatically start again

kill -9 22590
Copy the code

5. The Trojan horse is restarted and its PID is 30382

ls -l /proc/30382
Copy the code

6. View the scheduled task logs. Exceptions occur every hour

tail -2000f /var/log/cron
Copy the code

7. Check the scheduled task information. Copy /bin/sysdrr to /usr/bin/-bash every hour, start the script, and delete -bash

[root@localhost tmp]# cat /etc/cron.daily/ logrotate man-db.cron mlocate sync [root@localhost tmp]# cat /etc/cron.daily/sync #! /bin/bash # # Start/Stop the pwnrig clock daemon # # chkconfig 2345 90 60 # description: sync clock (GNU System) cp -f -r -- /bin/sysdrr /usr/bin/-bash 2>/dev/null cd /usr/bin/ 2>/dev/null ./-bash -c >/dev/null rm -rf -- -bash 2>/dev/null [root@localhost tmp]# cat /etc/cron.hourly/sync #! /bin/bash # # Start/Stop the pwnrig clock daemon # # chkconfig 2345 90 60 # description: sync clock (GNU System) cp -f -r -- /bin/sysdrr /usr/bin/-bash 2>/dev/null cd /usr/bin/ 2>/dev/null ./-bash -c >/dev/null rm -rf -- -bash 2>/dev/nullCopy the code

8. Clear the scheduled tasks again

[root@localhost etc]# cd /etc/cron.weekly/ [root@localhost cron.weekly]# ll total 4 -rwxr-xr-x 1 root root 246 May 5 2015 sync [root@localhost cron.weekly]# rm -rf sync rm: cannot remove 'sync' : Operation not permitted [root@localhost cron.weekly]# laattr sync -bash: laattr: command not found [root@localhost cron.weekly]# lsattr sync ----i----------- sync [root@localhost cron.weekly]# rm -rf Sync rm: cannot remove 'sync' : Operation not permitted [root@localhost cron.weekly]# chattr -R -i sync [root@localhost cron.weekly]# lsattr sync ---------------- sync [root@localhost cron.weekly]# rm -rf sync [root@localhost cron.weekly]# [root@localhost cron.weekly]# [root@localhost cron.weekly]# [root@localhost cron.weekly]# ll total 0 [root@localhost cron.weekly]#Copy the code

Run the following command to clear sync files or other suspicious files in other directories

chattr -R -i sync
lsattr sync
rm -rf sync
Copy the code

9. Delete the Trojan horse program, or download and save the Trojan horse program for subsequent analysis and research

[root@localhost etc]# cd /bin/
[root@localhost bin]# ls -la sysdrr
-rwxr-xr-x 1 root root 1878432 May  5  2015 sysdrr
[root@localhost bin]# 
[root@localhost bin]# 
[root@localhost bin]# 
[root@localhost bin]# 
[root@localhost bin]# chattr -R -i sysdrr
[root@localhost bin]# lsattr sysdrr
---------------- sysdrr
[root@localhost bin]# rm -rf sysdrr
Copy the code

10. Clean up the SSH

rm -rf /root/.ssh
Copy the code

11. Check all servers as far as possible

Safety recommendations

1. Log in with a key, not a password
2. Use secure password policies and strong passwords. Do not use weak passwords to prevent hackers from cracking them by force
3. Redis better not open ports or enable TLS and password authentication or add IP whitelisting etc
4. Extranet remote connection 22 Whitelist or disable extranet direct connection 22
5. Upgrade the component version of the exposed vulnerability, such as OpenSSH
6. Firewall disables Trojan communication IP

Reference documentation

Cloud.tencent.com/developer/a…

Blog.csdn.net/whatday/art…

Blog.csdn.net/weixin_4528…