V Reading Directory V Concept of Service traffic limiting V Why service traffic limiting V How service traffic limiting V Precautions V Blog Summary V Blog Preface Traffic limiting can be regarded as a kind of service degradation. Traffic limiting protects the system by limiting the input and output traffic of the system. Generally speaking, the throughput of the system can be measured. In order to ensure the stable operation of the system, once the threshold of the need to limit is reached, it is necessary to limit the flow and take some measures to complete the purpose of limiting the flow. For example: delay processing, reject processing, or partial reject processing and so on.
Before introducing the concept of traffic limiting, let’s talk about the current limiting around us. If there are code farmers in The Capital estimate that the current limiting is the most deeply felt. If there is a XXX meeting in the capital, the major subway stations will limit the current.
Every year, Double 11 is the paradise of the people who cut their hands off. At the midnight of November 11, you may have encountered the following scenes.
Service limiting for architectural design
Service limiting for architectural design
Of course, in recent years, major e-commerce companies have done better and better support for concurrency. Here are just some problems that we often need to deal with when Double 11 was just launched.
Through these two scenarios, the function of service traffic limiting is basically understood:
Service traffic limiting actually refers to a method to limit the traffic or function of the system according to preset rules in order to ensure that the limited resources can be properly served when the system resources are insufficient to cope with a large number of requests, that is, when the system resources conflict with the access volume.
Let’s go back to the top. Why is V limiting traffic? Here’s another example from our lives: Some of the popular tourist attractions, tend to have strict limits on the number of people visit the travel daily, such as the Forbidden City in Beijing, happy valley, etc., every day will only sell a fixed number of tickets, if you go to the late, may the day tickets have been sold out, can’t go in for pleasure in the day, even if you go in, line up also can to you doubt life.
Why do tourist attractions have such restrictions? Wouldn’t it be better to sell more tickets and make more money?
Actually for tourist attractions, they are also very helpless, because of the scenic spot service resources are limited, daily service number is limited, once let go of the restrictions, the attractions of the staff will be not enough use, health is also not guaranteed, safe hidden trouble, super dense crowd will be seriously affected tourist experience. However, due to the popularity of the scenic spot, tourists came to play in an endless stream, far beyond the carrying capacity of the scenic spot, so the scenic spot had to make a daily limit on the flow of people.
The same is true for system services in the IT software industry.
If your system theory is that you can serve 100W users in a unit of time, but today 300W users suddenly come, due to the randomness of user flow, if you do not limit the flow, it is very likely that 300W users will overwhelm the system at a stroke, resulting in no service for everyone.
Therefore, in order to ensure that the system can provide normal services for at least 100W users, we need to design the system for traffic limiting.
One might wonder why the system is not designed to support a cluster of 300 million users.
That’s a good question. If the system is a long-term 300W users to access, is certainly to do the above upgrade, but often face the situation is that the daily access of the system is 100W, but occasionally there are some unpredictable specific reasons for a short period of traffic surge, at this time, the company is often out of the consideration of cost savings, We don’t scale up our system to maximum size for an unusual spike.
Back to top v Service traffic limiting Traffic limiting for system services can be performed in the following modes:
- Fuse:
This mode requires that the fuse breaker measures be taken into account at the beginning of the system design. If a fault occurs in the system and cannot be rectified within a short period of time, the system automatically checks and turns on the fuse breaker to deny traffic access and prevent overload requests from the backend due to heavy traffic. The system should also be able to dynamically monitor the repair status of the back-end program, and when the program has been stabilized, the fuse switch can be turned off to resume normal service.
- Service degradation:
All functions and services of the system are graded. When the system has a problem and needs emergency flow limiting, the less important functions can be degraded and the service can be stopped. In this way, more resources can be released to provide core functions.
In electric business platform, for example, if a sudden traffic surges, can temporarily to review, non-core functions such as integral to downgrade, stop these services, release the machines and CPU resources to safeguard the normal order, and the whole system such as the degradation function services can be back to normal after, to start again, to the single/compensation processing. In addition to functional degradation, you can also use the method of reading cache and writing cache without directly operating the database as a temporary degradation scheme.
- Delay processing:
This pattern requires a traffic buffer pool at the front end of the system to buffer all requests into this pool without immediate processing. The back-end real business handler then pulls the requests out of the pool and processes them in turn, often using the queue pattern. This is equivalent to using an asynchronous way to reduce the back-end processing pressure, but when the traffic is large, the back-end processing capacity is limited, the request in the buffer pool may not be processed in time, there will be a certain degree of delay.
- Privileged handling:
In this mode, users are classified. By preset classification, the system gives priority to the user groups that need high security, and the requests of other user groups are delayed or not directly processed.
In actual projects, the following technical methods can be used to limit access traffic:
♛ Circuit breaker technology The circuit breaker technology can mainly refer to the practice of Hystrix, the open source component of Netflix, which consists of three modules: circuit breaker request judgment algorithm, circuit breaker recovery mechanism and circuit breaker alarm.
♛ counter method The system maintains a counter that increments every incoming request by 1, decreases by 1 when the request is processed, and rejects any new request when the counter is greater than the specified threshold.
Based on this simple approach, some advanced features can be extended, such as the threshold value can not be fixed, but can be adjusted dynamically. In addition, there can also be multiple groups of counters to manage different services, to ensure that they do not affect each other.
The ♛ queue method is based on the FIFO queue, all requests are queued, and the back-end program takes out the requests to be processed from the queue.
The queue-based approach also allows for more gameplay, such as multiple queues with different priorities.
The ♛ token bucket method is first based on a queue in which requests are placed. But in addition to the queue, a token bucket is set up, and there is a script that places tokens in the token bucket at a constant rate. The backend processor must take a token out of the bucket for every request it processes, and if it runs out of tokens, it can’t process the request. We can control the rate at which the script places tokens to achieve the speed at which the back-end processing is controlled to achieve dynamic flow control.
Back to top V Precautions When we do service traffic limiting, we still have some principles and considerations to pay attention to:
Real-time monitoring: The system must perform real-time monitoring of the entire link to ensure timely detection and processing of traffic limiting. Manual switch: In addition to automatic current limiting, a switch that can be manually controlled is required to ensure manual intervention at any time. Performance of traffic limiting: In theory, traffic limiting affects the normal service performance to some extent. Therefore, you need to optimize and control the performance of traffic limiting. Back to the top v blog concludes that system failures are often unpredictable and unavoidable, so we, as system designers, must plan in advance to deal with possible system risks at any time.