A server was hacked the whole process of investigation and thinking

This article has participated in the “Digitalstar Project” and won a creative gift package to challenge the creative incentive money.

Tencent cloud activities a while ago, HA C I bought a lightweight server, deployed their own website.

Everything is in order.

Until one morning, WHEN I opened my website as usual, it was not working.

So I ran a series of canvases.

1. Check logs

The first thing that comes to mind is to log in to the server and check the logs of abnormal login.

Boy, I found that the server can not log in!

1) Log in to the server using VNC

The first thing that comes to mind is that password login is disabled.

So I log in using VNC in the background of Tencent Cloud.

If you cannot remotely log in to the server using SSH from the client, you can log in to the server using VNC.

2) View the sshd_config file

/etc/ssh/sshd_config:

PasswordAuthentication no # Indicates that password login is not allowed

If password login is disabled, it should be logged in with the private key being used.

Change to yes and restart SSHD

systemctl restart sshd
Copy the code

3) Use the terminal to log in again

After the modification, use the terminal tool to log in to the VM again, because the VNC login tool is too difficult to use.

Found ready to log in.

Look at the authorized_keys file

 vi /root/.ssh/authorized_keys
Copy the code

Boy, don’t talk about code!

Added a secret key pair to my server:

4) View login logs

• Use the last and history commands to view login logs and operation logs

Last # View all login IP addresses

History # View the command record for the operation

It is not surprising to find that there is no abnormal IP, and if there is a login, there is a good chance that the login log will be deleted.

• Use the lastb command again to check:

Lastb # lists information about users who failed to log in

FIG. 1

Figure 2

Lastb results:

Column 1: user name column 2: terminal location column 3: login IP address or kernel column 4: Start time Column 5: End time (still login in down until normal shutdown crash until forced shutdown) Column 6: DurationCopy the code

The above results indicate that the server was hit by violence.

The IP should be through proxy. In the second picture, the other party directly uses root as the user name and keeps hitting the library. It seems that I have found the right user name, and finally I really logged in and changed my secret key pair.

Look up the IP:

The IP is from a foreign country, so it’s hard to find the location. It could also be a proxy IP.

2, find the Trojan horse file

1) Use the top command to see

The normal top command does not show the Trojan at all, and looks normal because the top command has probably been modified by an intruder:

Plain top command

2) busybox command

Run Busybox top to see hidden CPU-hogging processes. The original top has been modified to not show virus processes and must be executed in BusyBox

Download busyBox, the screening tool provided by Tencent Cloud,

[root@VM-8-8-centos ~]# wget https://tao-1257166515.cos.ap-chengdu.myqcloud.com/busybox --2020-12-14 15:12:59-- https://tao-1257166515.cos.ap-chengdu.myqcloud.com/busybox Resolving tao-1257166515.cos.ap-chengdu.myqcloud.com (tao-1257166515.cos.ap-chengdu.myqcloud.com)... 132.232.176.6, 132.232.176.7, 139.155.60.205,... Connecting to tao-1257166515.cos.ap-chengdu.myqcloud.com (tao-1257166515.cos.ap-chengdu.myqcloud.com) | 132.232.176.6 | : 443... connected. HTTP request sent, awaiting response... 200 OK Length: 1001112 (978K) [application/octet-stream] Saving to: 'busybox. 1' 100% [= = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = >] 1001112 1.36 MB/s in 0.7 s [root @ VM - 8-8 - centos ~] # cp busybox /usr/bin/ [root@VM-8-8-centos ~]# busybox top -bash: /usr/bin/busybox: Permission denied [root@VM-8-8-centos ~]# cd /usr/bin/ [root@VM-8-8-centos bin]# chmod 777 /usr/bin/busybox [root@VM-8-8-centos ~]# busybox topCopy the code

Trojan file caught:

Busybox top command

See above CPU usage reached nearly 100%, mining is no doubt.

Finally, we searched together with Tencent cloud technology for most of the day and finally found the following Trojan files, directories:

/tmp/.X25-unix/.rsync/c/tsm64
/tmp/.X25-unix/.rsync/c/tsm32
/tmp/.X25-unix/.rsync/a/kswapd0
/usr/bin/systemd-network
/usr/bin/kswaped
Copy the code

Finally, lock the process named pamdicks, kill the Trojan, and delete the Trojan file.

If you do not enter the full name, ls, ll, lsattr file view command will not display this Trojan file at all:

Unable to display Trojan file, full name required

Take a look at the mining process before deleting:

ls -lh /proc/5445/fd
Copy the code

 top -H -p 5445
Copy the code

The pamdicks process has six child threads:

Finally traced to a binary file, touched the knowledge range, can not open, just delete it.

3. Delete the Trojan horse file

1) Modify authorized_keys

Delete the public key of the authorized_keys file. When I ran rm, the intruder locked my authorized_keys file with + I lock, so sad:

Chattr + I /etc/authorized_keys Indicates that the file cannot be deleted, modified, or moved

I deleted the chattr command from my server. The server was hacked. Deleting the chattr command is a common operation.

Chattr command cannot be found:

You can only install chattr manually.

yum install e2fsprogs
Copy the code

Installation successful:

[root@VM-8-8-centos run]# which chattr
/usr/bin/chattr
Copy the code

Empty the authorized_keys file:

[root@VM-8-8-centos .ssh]# chattr -i authorized_keys
[root@VM-8-8-centos .ssh]# echo > authorized_keys
[root@VM-8-8-centos .ssh]# cat authorized_keys
Copy the code

2) Run the rm command to delete Trojan horse files

Kill and delete discovered Trojan files:

[root@VM-8-8-centos .ssh]# kill -9 5445 [root@VM-8-8-centos .ssh]# chattr -i /usr/bin/pamdicks [root@VM-8-8-centos .ssh]# rm /usr/bin/pamdicks rm: remove regular file '/usr/bin/pamdicks'? y [root@VM-8-8-centos .ssh]# rm /tmp/.X25-unix/.rsync/c/lib/64/tsm rm: Remove regular file '/ TMP /. X25 - Unix /. / lib / 64 / TSM rsync/c'? yCopy the code

The CPU usage drops after the delete:

Trojan file clean up, finally the server to disable password login, switch to generate good secret key login.

Come to an end for now.

4. Find the source of the attack — Redis

After a few days, when I opened Redis, I found a strange key value in Redis.

I didn’t notice this problem before.

Redis is black

The effect! Redis opens the port and does not set a password.

I don’t know how it got injected into my server. But this Crackit button is weird.

The following information was found online:

Redis Crackit Vulnerability:

The hacker accesses the redis service remotely, clears the REDis database, writes his SSH login public key, and then backs up the REDis database as /root/.ssh/ rules.

This successfully writes your public key to SSH authotrized_keys and directly root to the hacked host without password.

That said, it is highly likely that my server was not logged in by brute force, but was attacked using Redis as an entry point.

Use the top command to see, I go! The Trojan file will continue to download if the Redis key is not cleared.

Kswapd0 disguised Trojan horse

This kSWAPd0 is a little familiar.

The CPU usage of KSWAPd0 is too high because the physical memory is insufficient. Swap and memory page swapping are used to exchange data, resulting in high CPU usage.

/ TMP /.x25- Unix /.rsynckswapd0

So this Redis is not solved, the intruder will continue to use your Redis vulnerability to continue to invade your server.

The actual kSWAPd0 process looks like this:

That’s normal

To recreate the process, I run the Redis value command through:

[root@VM-8-8-centos ~]# ping d.powerofwish.com (193.160.32.164) 56(84) bytes of data.Copy the code

Download a pm.sh script. Open the script:

As you can see, the sh script essentially downloads a PNG file with executable permissions.

I directly download the PNG file, grant permissions, and execute./ PNG

See a disguised bin script, delete it and write it to /usr/bin.

Then the chattr command is deleted and the authorized_keys file is modified during the continuous brushing

The final script should be executed /usr/bin/kswaped, to start mining.

Look at the CPU again with Top, and it goes up to nearly 100%

Catch the bag and see:

Tcpdump -i eth0 '((not port 45695) and (not host 127.0.0.1) and (not host 183.60.83.19))'Copy the code

The CDN node 104.27.129.57 was found in the United States. Export the captured package to WirteShark to see:

The red part is the source IP, as well as the other party has requested the database operation (port 3306).

It’s also possible that my server is just a DDoS attack node. So in order to maintain network security, or timely processing Trojan files.

A Distributed Denial of Service attack (DDoS) is an attack launched by multiple attackers in different locations at the same time against one or more targets, or when an attacker takes control of multiple machines in different locations and uses them to attack the victim at the same time. Because the attack points are distributed in different places, this type of attack is called distributed denial of service attack, in which there can be more than one attacker.

DDos Attack Diagram

5, prevention,

Too many public network ports are exposed, especially Redis6789 and MySQL3306, and the server password is too simple.

Preventive measures:

1) Server

Modify the /etc/ssh/sshd_config file

• Turn off password login and allow only secret key pair login

• Change the SSH default port to prevent brute force database cracking

• Disable direct login as the root user and enable specific IP address access

Such as:

# Allow only users and IP addresses to access AllowUsers aliyun [email protected],[email protected].* deny SSH login to DenyUsers Zhangsan and aliyun Aliyun # authorizedkeysfile. SSH /authorized_keys PubkeyAuthentication yes RSAAuthentication yes # PasswordAuthentication noCopy the code

2) application

• Redis only allows local access, modifies default ports and does not expose all IP addresses, Redis defaults to bind 127.0.0.1 for a reason

• MySQL only allows access to IP addresses that are required

• Example Set firewall access rules for ports

If you want to be more secure, you can use jumpers, fortress access.

3) the backup

Periodically back up data and snapshots.

​