preface

According to the online article, and combined with their own server situation, that the invasion is due to redis port open to the outside, and empty password, resulting in Kworkerd, SustSE malicious intrusion LINUX mining, originally thought convenient debugging (are my pot), now only learn lessons. Let’s start!

To analyze problems

Things began with a ordinary but again great day-to-day coding in the afternoon, suddenly discovered that a particular service database connection refused, then check the mysql server on service status, unexpectedly is closed, when I open shortly after the service is closed automatically again, from the start, all of the service connection request block slow, the page cannot be accessed or load timeout, began Question (in a panic).

  1. The inputTOP -cCheck the current server situation, the strange thing is that the CPU runs about 99%, but there is no high CPU usage process, after looking at the Linux system status for a long time, found occasionallysustse,kworkerdSuch as unfamiliar process words, non-stop search.
  2. The discovery of this bookMining analysis”Article, with distance ideas, find/var/tmpDirectory and clean up these several virus scripts
  3. Return to the script after a while, and seeTiming taskUnknown file has been obtained in. The file address ishttp://192.99.142.246:8220/mr.sh, the inputcrontab -eModified deletes this script, but requests for scheduled tasks are still written
  4. Continue to inputnetstat -anpWhen you view the current network connection, multiple abnormal connections are found. The IP address of the scheduled task is the same as that of the scheduled taskps aux|grep 8220Find andkillAfter dropping the process, the process will still be written, it seems that there are multiple scripts to write scheduled tasks
  5. Finally find this kworkerd analysis article, want to clear the code through the above scheme, found that it is not general, can not be completely removed, and the malicious script on the article is probably in the middle of this year, now the script is updated, but also have to rely on their own, the analysis comes to an end, to start to solve the problem.

The solution

Although the script can not be completely fixed for a time, unlimited brush, and files have been requested, but the day will always go on is not, so while submitting the work order to Ali Cloud, while analyzing the current malicious script.

Aliyun suggestion

A snapshot backup is made based on the engineers’ suggestions for future rollback.

  1. Reset the system and create a snapshot to create a pay-as-you-go cloud disk for data copy
  2. Install ali cloud some security software, screening protection
  3. If there are malicious brush site, upload some malicious scripts, look at the access log, can be blocked by the security group, or learn about the Web application firewall can effectively prevent someone brush site

Clearing Malicious Scripts

Script, and then through the analysis of http://192.99.142.246:8220/mr.shSHELL found that script will be deleted and new process and the script, through the record the script content, will ‘delete the script’ one-time execution, that execution success and no malicious processes.

chattr +i /usr/bin/wget
chmod 700 /usr/bin/wget
chattr +i /usr/bin/curl
chmod 700 /usr/bin/curl
/etc/init.d/iptables stop
service iptables stop
SuSEfirewall2 stop
reSuSEfirewall2 stop
pkill -f sysxlj
pkill -f jourxlv
pkill -f sustes
rm -rf /etc/ld.so.preload
rm -rf /usr/lib/void.so
rm -rf /etc/voidonce.sh
rm -rf /usr/local/lib/libjdk.so
rm -rf /usr/local/lib/libntp.so
ps aux|grep "I2NvZGluZzogdXRmLTg"|grep -v grep|awk '{print $2}'|xargs kill -9
rm -rf /lib64/library1.so
rm -rf /usr/lib64/library1.so
rm -rf /lib64/library1.so
rm -rf /usr/lib64/library1.so
iptables -I OUTPUT -s 167.99.166.61 -j DROP
iptables -I INPUT -s 167.99.166.61 -j DROP
iptables -I OUTPUT -p tcp -m string --string "pastebin" --algo bm -j DROP
...
Copy the code

Related to the protection

After solving the problem, it is found that there is basically no abnormal situation. To really solve the problem, we need to do some work related to safety protection:

  • Disable the external ports of redis,mysql, etc., and add strong passwords.
  • According to the official Linux operating system hardening document of Aliyun, security hardening of PHP environment, MySQL service and other services is also indispensable

Afterword.

This is really an insignificant negligence, but also a fatal mistake, who can not think of my first contact with virtual currency was due to REDIS port intrusion, network security is so important, well-founded, convincing.

‘

Refer to the link

  1. Write down a mining virus analysis
  2. Kworkerd malicious mining analysis
  3. Hardening the Linux operating system
  4. MySQL service security hardening
  5. PHP environment security hardening
  6. Malicious Script Backup
  7. Delete script command