1. Introduction

The email received yesterday is as shown in the picture below:Oh, my God, for a front-end developer with only a passing knowledge of the server side, the hackers knew how to bully the little guy and break into my server.

I had no idea where the virus was or how the hacker was getting into my server. Next, let’s go through the troubleshooting step by step, first look at how to determine whether the Linux server is hacked?

2. Check whether the Linux server is invaded

When a server is hacked, or hacked, it means that unauthorized people or programs log on to the server and use its computing resources for their own purposes, usually with bad effects.

Disclaimer: If your server is hacked by a state agency like the NSA or a criminal group, you won’t notice any problems, nor will these technologies detect them.

However, most compromised servers are hacked by programs like Autobots or by cheap attackers like “Script Boys” and dumb criminals.

Such attackers abuse server resources while accessing them and do little to hide what they are doing.

Symptoms of a compromised server

When servers are invaded by inexperienced attackers or automated attackers, they tend to consume 100% of their resources. They may consume CPU resources to mine digital currency or send spam, or bandwidth to launch DoS attacks.

So the first sign of a problem is that the server is “slowing down.” This could mean that the page on your website is slow to open, or that emails take a long time to send.

So what should you check?

2.1 Who is currently logged in?

The first thing you need to look at is who is currently logged in to the server. It is not complicated to discover that an attacker has logged on to the server to perform operations.

The corresponding command is w. Running w produces the following output:

10:43:09 up 214 days,  1:20,  1 user,  load average: 0.00, 0.00, 0.00
USER     TTY      FROM              LOGIN@   IDLE   JCPU   PCPU WHAT
root     pts/1    183.15.179.82    09:49    0.00s  0.02s  0.00s w
Copy the code

Check the IP address, is oneself logged in, no one else logged in, if you check the IP address is not your OWN IP address, you should pay attention, this is not a good sign.

Stop and take a deep breath, don’t panic and just kill their SSH connection. Unless you can prevent them from entering the server again, they will quickly come in and kick you out to prevent you from going back.

See the “What To do after An intrusion” section at the end of this article to see what to do if you find evidence of an intrusion.

The whois command takes an IP address and tells you all about the organization registered with that IP, including the country.

2.2 Who has logged in?

The Linux server keeps track of which users, from which IP, when they logged in, and for how long. You can view this information using the last command.

The output looks like this:

Root PTS /1 78.31.109.1 Thu Nov 30 08:26 Still logged in root PTS /0 113.174.161.1 Thu Nov 30 08:26 Still logged in root PTS /1 78.31.109.1 Thu Nov 30 08:24-08:26 (00:01) Root PTS /0 113.174.161.1 Wed Nov 29 12:34-12:52 (00:18) Root PTS /0 Mon Nov 27 13:32-13:53 (00:21)Copy the code

Here you can see that the UK IP and Vietnam IP alternate, and the top two IP are still logged in. If you see any unauthorized IP, please refer to the last section.

The logon history is recorded in the binary /var/log/wtmp file (LCTT) and is therefore easily deleted. Often attackers simply delete the file to hide their attacks. So if you run the last command and only see your current login, it’s a bad sign.

If you don’t have a login history, be careful and keep an eye out for other clues to the intrusion.

2.3 Reviewing command History

Attackers at this level usually don’t pay attention to the history of the mask command, so running the history command shows everything they’ve ever done. Be sure to watch out for wget or curl commands to download unconventional software such as spambots or mining programs.

The command history is stored in the ~/.bash_history file, so some attackers delete this file to hide what they are doing. As with login history, if you run the history command and nothing is output, the history file has been deleted. This is also a bad sign and you need to check your server carefully. (If there is no command history, you may have misconfigured it.)

2.4 Which Processes are consuming CPU?

The kind of attackers you often encounter don’t do much to hide what they’re doing. They run special CPU-intensive processes. This makes it easy to spot these processes. Just run top and look at the first few processes.

This also shows attackers who are not logged in. For example, someone might be sending spam with an unprotected script.

If you don’t know the top process, you can Google the process name, or check losf and Strace to see what it does.

Using these tools, the first step is to copy the process PID from top and run:

strace -p PID
Copy the code

This displays all the system calls made by the process. It produces a lot of content, but this information tells you what the process is doing.

lsof  -p PID
Copy the code

This program lists the files that the process has opened. It’s easy to understand what it’s doing by looking at the files it accesses.

2.5 Checking All System processes

An unauthorized process that doesn’t consume a lot of CPU may not show up in top, but it can still be listed in Photoshop. The ps auxf command should display clear enough information.

You need to check every process you don’t know. Running Ps regularly (which is a good habit) can help you spot strange processes.

2.6 Checking the Network Usage of processes

Iftop functions like TOP. It displays the processes that send and receive network data, as well as their source and destination addresses. Processes such as DoS attacks or spambots easily show up at the top of the list.

2.7 Which Processes are Listening for Network connections?

Usually an attacker installs a backdoor program that listens specifically for network ports to receive instructions. The process does not consume CPU or bandwidth while waiting, so it is not easy to detect with commands like top.

Both the lsof and netstat commands list all networking processes. I usually give them the following parameters:

lsof -i
Copy the code
netstat -plunt
Copy the code

You need to keep an eye out for processes in the LISTEN and ESTABLISHED states that are either waiting to be connected or are already connected. If you encounter processes you don’t know, use Strace and lsof to see what they’re doing.

3. What to do after being hacked?

After the above steps, you can basically find out whether your server has been hacked. If there is a breach, first of all, don’t be nervous, especially if the attacker is logged in. You need to regain control of the machine before the attacker is alerted that you have discovered him. If he finds out you’re on to him, he’ll probably lock you out of the server and start destroying the body.

If you’re not good at it, just turn it off. You can run either shutdown -h now or systemctl poweroff on the server. You can also log in to the host provider’s control panel to shut down the server. Once shut down, you can start configuring firewalls or consult your vendor.

If you are confident about yourself and your host provider provides an upstream firewall, then you just need to create and enable the following two rules:

  • Only SSH login from your IP address is allowed.
  • Block anything else, not just SSH, but any protocol on any port.

This will immediately close the attacker’s SSH session, leaving you with access to the server.

If you cannot access upstream firewalls, you need to create and enable these firewall policies on the server itself, and then use the kill command to close the attacker’s SSH session after the firewall rules take effect. (Local firewall rules may not block an established SSH session, so you need to kill it manually to be on the safe side.)

A final method, if supported, is to log in to the server over an out-of-band connection such as a serial console and then stop the network functionality with systemctl stop network.service. This will shut down all network connections on all servers so you can slowly configure those firewall rules.

Once you’ve regained control of your server, don’t think you’re done.

Don’t try to fix the server and then use it again. You never know what an attacker has done, so you can never be sure that the server is still secure.

The best way is to copy all the data and reinstall the system. (YOUR program is not trusted at this point, but the data is generally fine.)

Process of 4.

After the above steps 2.1, 2.2 and 2.3, I did not find any abnormality or login of my server. However, in Step 2.4, I found that some unknown process was consuming CPU. Then I used Strace -p PID to find that this process was sending out a large number of packets. Use lsof in step 2.7 – I found that this process is in monitoring network connection, the process name Pnscan, after I use kill 9 PID, run the ps aux | grep pnscan found the process still exist, so the Internet and found this to be the JBoss loophole poisoning.

5. Check whether process ports are used by each other

Corresponding view under Linux process takes up port: look at the program process number: ps ef | grep process name

[root@iZ28ti4v56dZ ~]# ps -ef | grep nginx
root     17453     1  0 Feb27 ?        00:00:00 nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf
nginx    17455 17453  0 Feb27 ?        00:14:13 nginx: worker process
root     32251 32073  0 17:45 pts/0    00:00:00 grep nginx
Copy the code

REDHAT: the port number to check the process occupies: netstat – NLTP | grep process ubuntu: view the process load port: netstat anp | grep pid

/ root @ iZ28ti4v56dZ ~ # netstat anp | grep TCP 17453 0 0 0.0.0.0:0.0.0.0:80 * 17453 / nginx TCP 0 0 0.0.0.0: LISTEN. "443 0.0.0.0:* LISTEN 17453/nginx Unix 3 [] STREAM CONNECTED 11886123 17453/nginx Unix 3 [] STREAM CONNECTED 11886122 17453 / nginx/root @ iZ28ti4v56dZ ~ # netstat NLTP | grep TCP 17453 0 0 0.0.0.0:80 0.0.0.0: * LISTEN 17453 / nginx TCP 0 0 LISTEN 0.0.0.0:443 0.0.0.0: * 17453 / nginxCopy the code

In Linux, run the lsof command: lsof -i: indicates the port number

[root@iZ28ti4v56dZ ~]# lsof -i:443
COMMAND   PID  USER   FD   TYPE   DEVICE SIZE/OFF NODE NAME
nginx   17453  root    7u  IPv4 11886111      0t0  TCP *:https (LISTEN)
nginx   17455 nginx    7u  IPv4 11886111      0t0  TCP *:https (LISTEN)
Copy the code

6. Jboss vulnerability causes Linux server poisoning solution

6.1 Poisoning

  1. The network is congested and the access delay increases.
  2. Abnormal scheduled tasks exist in the system scheduled task list. Procedure
  3. An abnormal process has occurred.
  4. A large number of abnormal files are found in the $JBOSS_HOME/bin or /root directory.

6.2 Symptom Analysis

This is a kind of worm virus that is popular on the Internet recently. It uses jXM-Console and Web-console default account vulnerabilities of Jboss middleware programs to attack, infect Linux servers and become zombie proxies.

  1. The network congestion was caused by the worm using a name calledpnscanThe tool continuously performs port scans. Sends a large number of request packets, occupying network bandwidth.
  2. The following abnormal scheduled tasks (sometimes only two) are displayed in the periodic task list. Crontab “l

.sysync.pl and. Sysdbs are hidden files that can be viewed in the ls-la list.

  1. View the process, you can check the following abnormal processYou can also see some abnormal Javas processes on some servers. Please check whether these Javas processes are called by the application.

  2. The following abnormal files are found in the $JBOSS_HOME/bin or /root directoryGz is the installation package of virus source code, which generates the above files after installation.

6.3 Solutions

Step 1: Check the virus

Killall -9 javas Killall -9 pns Killall -9 perl

CD /root or CD $JBOSS_HOME/bin rm -rf bm rm -rf.pl rm -rf treat.sh rm -rf install-sh rm -rf version rm -RF kisses rm -RF PNS rm -RF Makefile rm -RF ipsort rm -RF kisses rm -rf.sysdbs rm -rf.sysync.pl

Sysdbs 1 1 24 perl ~/.sysync.pl 1 1 24 perl ~/.sysync.pl 1 1 10 ~/. Sysdbs Delete the service crond stop line

Step 2: Jboss security hardening, change the default passwords of JMX-Console and Web-Console

JMX Security Configuration:

Comment out the GET and POST lines, and leave the security-constraint section uncommented.Remove the security-domain commentChanging the Password of admin

Web-console security hardening

The modification method is the same as JMX security hardening.

Step 3: Test

After completing Jboss security hardening, perform HTTP access test to see whether the authentication window can be displayed normally and whether the user name and password can be accessed normally.

xxx.xxx.xxx.xxx/web-console

XXX, XXX, XXX, XXX/JMX – conslol…

6.4 Suggestions for Jboss Vulnerability Attack

For virus attacks generally or to prevent the main, once found that the server has been poisoned to solve the relevant intractable. In order to effectively prevent such virus attacks, the following suggestions are provided:

  1. Jboss applications should run as non-root users to prevent viruses from gaining superuser privileges, changing root passwords, and controlling the server.

  2. Enable authentication for the Jboss console and change the default password, which must be somewhat complex. If you don’t, you can even turn off the administrative port and related statistics and delete the Jboss home directory and files.

  3. Upgrade Jboss to the latest version, especially boss Ben’s Jboss has more bugs, and the new version of Jboss has higher security.

  4. The separation of WEB applications from receivers, for example, can be achieved through the integration of Apache and Jboss, which is more secure on the one hand and more suitable for access with high concurrent traffic on the other.

7. To summarize

From yesterday’s antivirus investigation to today, no warning email has been sent to me, indicating that the virus has been removed. From this incident, I found that there are inevitably some bad people in the world, to attack other people’s servers, specifically attacking people like me who only understand the surface of the server knowledge, but also want to thank them, let me decide to learn relevant knowledge to resist their attack.