preface

It has been about 10 days since the last DDOS attack. I can’t remember the exact day since the last attack. However, in the wake of a recent attack that was relatively prolonged, I felt compelled to take a moment and share my experience of being hacked.

Before describing the experience, let’s briefly introduce the server configuration:

  • ECS 1 core 2 GB memory 1MB bandwidth for Linux
  • RDS 2 core 240MB memory, maximum connection number 60
  • Redis 256MB shared instance, no use after moving
  • CND pays by volume and caches small files

The above configuration, for a daily page view of thousands of websites should be more than enough, and supporting a dozen or so, the following is a simple site deployment:

experience

Some time ago I heard that the Internet tycoon Ruan Yifeng blog was DDOS experience, it can be described as lasting ah, was eventually forced to transfer the server, is said to be blackmail. But I don’t know why is which fairy board board has targeted my small station? Am I handsome than Ruan God?

Well, the story begins. On June 14, 2018, AT 2:30 am, I received an alarm notification from Alibaba Cloud system informing me that the website was unavailable, while I was still asleep.

As usual, I woke up around six o ‘clock and checked my phone habitually, only to receive another SMS alert. Normally, I could have slept for another two hours, but then I got a jolt of energy and suddenly I was completely sleepy. After all, I was a blogger with thousands of page views.

Get up, turn on your computer, and try to visit blogs and forums, only to find that your browser keeps spinning around.

Troubleshoot problems

Try logging in to the server remotely:

  • View the Nginx and PHP – FPM, ps – ef | grep XXXX
  • Check the free memory free-m
  • View CPU usage top
  • View the Nginx error log tail -f error.log
  • Check log capacity lL-h
  • Check the number of simultaneous connections netstat NAT | grep ESTABLISHED | wc -l

After the operation, there are no exceptions, memory and CPU are stable, Nginx and PHP processes are fine. After rebooting PHP and Nginx, the site is still accessible.

Check error log, background hard brush log, casually check a few IP, there are India, the United States, the Philippines and so on, of course, most or domestic IP. In one night, I brushed hundreds of megabytes of logs (I cleaned it once last time by D). Anyway, I think it is quite a lot, compared with the page views of the website at ordinary times.

There have been several attacks before, but only in twos or threes, using Nginx to disable IP. This time, however, it is not clear that disabling IP addresses will solve the problem, so many IP collections are a problem (which can be obtained by regular matching of course), and there is also the possibility of friendly fire.

On the way to work

However, work is the business, the mind will not solve the problem, took a look at the error log, but also hard to brush, and then conveniently sent a circle of friends and then go to wash:

All the way down the road, wondering if it’s 9 o ‘clock, they leave the night shift on time and then they can go back and talk to themselves.

In the work

When I got to the company, the first thing I did was of course log on to the server remotely. Normally, this is a point in time that no user will access.

Restart the service for several times, visit the home page will be stuck, and then the instant crash, the whole site (community + blog) can not access. In that case, just go to work and wait for the attacks to stop.

During the group of small friends asked the website how, can not open coconut? Nearly noon, looked at the error log, there are so several IP to try to request a different address, a Chou is not what good things, a decisive deny. Now there are not so many requests, restart some Nginx and PHP processes, but access to the home page is still stuck? What a strange egg.

I checked the monitoring alarm panel and found that CPU and memory utilization rate and the total number of current connections were normal without any abnormality. There was indeed a fluctuation between 2:00 and 6:00 in the morning, but it would not be killed by D. Since you are logged in, why not restart both ECS and RDS?

Sure enough, restart the magic of the good, have lunch with a mobile phone access, normal, can eat at ease.

Problem solving

In fact, I am not sure how the final problem was solved. Let me tell you a few confusing points:

  • The CPU and memory of the ECS server are also below normal thresholds
  • Both Nginx and php-fpm processes have been restarted respectively
  • Although the number of RDS database connections fluctuates, it is not fully occupied
  • Look at the error log. Requests come from hundreds of different IP addresses, and most of them are visiting community urls
  • And why are these broilers always at night? Cheap at night? Or organizing attacks in the Western Hemisphere
  • Is this targeted or random? Hopefully it’s random
  • Stop the community once in the middle, blog can always be normal access, suspected to be the home database query problem, based on the number of connections should not be this problem, is it a Discuz Bug? However, after the database was restarted, it did work.

In fact, Ali Cloud has basic DDOS protection, cleaning trigger value:

  • Request traffic per second: 300M
  • Number of packets per second: 70,000

For the general small station, is absolutely impossible to reach 300M flow threshold, blog CND peak is only 3M.

So, these wavelet flow attacks can only bear their own silently, and the machine configuration is not high, can not afford bandwidth can only be free to attack the carefree, it is better to directly close the station, throw him a Nginx + static page let it go.

Offensive and defensive strategy

There’s nothing you can do about it if someone is actually blocking your site, and I’m talking about small and medium webmasters, you can’t even reach the cleaning threshold for DDOS basic protection.

If you are just an unknown station, you don’t have to think that much. Despite the low cost of DDOS these days, no one can afford to lose money unless you offend someone.

Of course, we can’t just sit back and wait for normal attacks. Here are a few tips to share with you. Reverse proxies use OpenResty.

Nginx optimization

Nginx claims a maximum concurrency of 5W, in fact, for small and medium sites dozens or hundreds of concurrent is good, the most basic parameters can meet the needs. But it’s best to hide the version number for security purposes.

Hide tokens off server_tokens off Configure in the HTTP moduleCopy the code

PHP optimization

In the PHP rendering header, the PHP version number is included, such as x-powered-by: PHP /5.6.30, this is somewhat insecure, and some hackers may use scanning methods to batch find lower versions of PHP servers and exploit PHP vulnerabilities (such as hash collisions) to attack the server.

Php_admin_flag [expose_php] = offCopy the code

IP blacklist

For the worst kind of attack, joining a blacklist is a good option, or someone else will crush you to death:

# add the following configuration to the HTTP module of Nginx to deny 61.136.197. XXX; # deny 61.136.197.0/24;Copy the code

Number of IP accesses per day

Limit the daily access times of a single IP address. Normally, the access depth of a user rarely exceeds 10, and the jump rate is generally between 50% and 70%. In fact, what we need to do is to limit the daily visits of a single IP to 100 or even 50.

Limit concurrency

Limiting the number of accesses is not enough. Attackers can flood with hundreds or thousands of requests at a moment, and if these requests reach the back-end service, they can overwhelm the database service, so we need to set the concurrency based on our own site visits.

  • Limit the number of concurrent IP addresses
  • Limit the total number of concurrent requests

You are advised to use the leaky bucket algorithm to limit traffic to shape traffic requests.

Configuration CND

Based on the bandwidth and the normal user access speed considerations, it is recommended to configure CND, the following is the blog traffic usage, peak 3MB, for my 1MB bandwidth server is certainly not able to resist ah, and there is also the community access.

Configure the cache

Database resources are valuable, so try not to let requests go straight to the back end. In fact, before moving, blog and community are configured with Redis cache. Because the Redis service purchased before is a private network, the new account could not be connected, and then gave up.

It seems that this time, the need to configure a idle server, anyway, idle is idle, can play a role is also good.

  • Ali Cloud Redis to accelerate Discuz forum access
  • Aliyun Redis accelerates Typecho blog access

conclusion

The front also said, for the attack, the station really no solution, can do a good job of basic protection can be. But for those of you who are or will be broilers:

  • Software vulnerabilities must be patched in time, always pay attention to the Internet related dynamics.
  • Hackers took control of dalian Chicken by using the compromised router to harvest network traffic.
  • Most of the chicken is not security awareness, and have been long-term use, by the discovery, many cloud service providers host, hosting server host, hackers use vulnerability control.
  • DDoS hacking attacks are turning into industrialization and platform services. If someone wants to harm you, they can attack you for a whole month with just a button and a few hundred yuan, and then give themselves a song called cool cool.

reference

Talk about limiting traffic effects from building distributed seckill systems