Due to the limited load capacity of the business application system, each API interface is capped to prevent the business application system from being overwhelmed by unexpected requests. API interface traffic control policies include traffic diversion, degradation, and traffic limiting. This paper discusses the traffic limiting strategy, which reduces the access frequency and concurrency of service interfaces in exchange for high availability of service interfaces and business application systems.
The purpose of traffic limiting is to limit the speed of concurrent access/requests or requests within a time window to protect the system. Once the rate reaches the limit, service denial, queuing or waiting, and degradation can be processed.
Current limit algorithm
There are two commonly used traffic limiting algorithms: leaky bucket algorithm and token bucket algorithm.
Bucket algorithm
The Leaky Bucket algorithm is commonly used in Traffic Shaping or Rate Limiting on the network. It controls the Rate at which data is injected into the network and smoothen burst Traffic on the network. The leaky bucket algorithm provides a mechanism by which burst traffic can be shaped to provide a steady flow of traffic to the network.
Leaky bucket algorithm is simple. Water (request) enters the leaky bucket first, and the leaky bucket flows out of the water at a certain speed (the interface has a response rate). If the inflow speed is too high, the water will overflow directly (the access frequency exceeds the interface response rate), and then the request will be rejected. The schematic diagram is as follows:
Because the leaky bucket rate is a fixed parameter, the leaky bucket algorithm cannot burst traffic to the port rate even if there is no resource conflict (no congestion) in the network. Therefore, leaky bucket algorithm is inefficient for traffic with burst characteristics.
Token bucket algorithm
Token bucket algorithm is one of the most commonly used algorithms in Traffic Shaping and Rate Limiting. Typically, the token bucket algorithm is used to control the amount of data sent to the network and to allow the delivery of burst data.
The token bucket algorithm works by putting tokens into the bucket at a constant rate. If the request needs to be processed, a token needs to be fetched from the bucket first. When no token is available in the bucket, the service is denied. In principle, the token bucket algorithm and the leaky bucket algorithm are opposite, one is “water”, the other is “water”.
Another benefit of token buckets is that you can easily change the speed. As soon as the rate needs to be increased, the rate at which the tokens are put into the bucket is increased as needed. A certain number of tokens are added to the bucket at a regular time (say, 100 milliseconds), and some variations calculate the number of tokens that should be added in real time.
Go-kit microservice traffic limiting implementation
Combined with the above analysis, I will implement the flow limiting function of microservices based on Go-Kit. Through reviewing goKit/Kit/Ratelimit source code, it is found that GoKit is based on the go package golang.org/x/time/rate built-in an implementation; In addition, the juju/ Ratelimit implementation scheme used by GoKit by default (officially removed now), I will implement it in two ways respectively.
Unlike the previous two articles, this implementation will be based on goKit’s built-in type endpoint.Middleware, which is actually a function that encapsulates an endpoint using decorator mode. The definition is as follows:
# Go-kit Middleware Endpoint
type Middleware func(Endpoint) Endpoint
Copy the codeJuju/ratelimit scheme
The examples in this article will continue to improve on the code in the previous article (address at the end of the article).
Step-1: Creates a current limiter
First, install the latest version of the Juju/Ratelimit library using the following command:
go get github.com/juju/ratelimit
Copy the codeThen, create a go file named instrument.go to implement the flow limiting method: token bucket (BKT) returns endpoint.middleware. Use the TakeAvaiable method of the token bucket to obtain the token. If the token is obtained successfully, the execution continues. If the token fails to be obtained, an exception is returned (i.e., flow limiting). The code is as follows:
var ErrLimitExceed = errors.New("Rate limit exceed!") / / NewTokenBucketLimitterWithJuju use juju ratelimit create current-limiting middleware func NewTokenBucketLimitterWithJuju (BKT * ratelimit. Bucket) endpoint.Middleware {return func(next endpoint.Endpoint) endpoint.Endpoint {
return func(ctx context.Context, request interface{}) (response interface{}, err error) {
if bkt.TakeAvailable(1) == 0 {
return nil, ErrLimitExceed
}
return next(ctx, request)
}
}
}
Copy the codeStep-2: Modify main
The next Step is to use juju/ Ratelimit to create the token bucket (refreshed every second with a capacity of 3) and then to decorate the Endpoint with a step-1 implementation of limiting. Add the following code to the main method.
// add ratelimit,refill every second,set capacity 3
ratebucket := ratelimit.NewBucket(time.Second*1, 3)
endpoint = NewTokenBucketLimitterWithJuju(ratebucket)(endpoint)
Copy the codeAfter modification, the complete code is as follows:
func main() {
ctx := context.Background()
errChan := make(chan error)
var logger log.Logger
{
logger = log.NewLogfmtLogger(os.Stderr)
logger = log.With(logger, "ts", log.DefaultTimestampUTC)
logger = log.With(logger, "caller". log.DefaultCaller) } var svc Service svc = ArithmeticService{} // add logging middleware svc = LoggingMiddleware(logger)(svc) endpoint := MakeArithmeticEndpoint(svc) // add ratelimit,refill every second,set capacity 3
ratebucket := ratelimit.NewBucket(time.Second*1, 3)
endpoint = NewTokenBucketLimitterWithJuju(ratebucket)(endpoint)
r := MakeHttpHandler(ctx, endpoint, logger)
go func() {
fmt.Println("Http Server start at port:9000")
handler := r
errChan <- http.ListenAndServe(": 9000", handler)
}()
go func() {
c := make(chan os.Signal, 1)
signal.Notify(c, syscall.SIGINT, syscall.SIGTERM)
errChan <- fmt.Errorf("%s", <-c)
}()
fmt.Println(<-errChan)
}
Copy the codeStep-3: compile & run
Compile and run the application on the console, then test it through the Postman request interface, and you can see the output log message:
Ts = 2019-02-19 T03: were. 1908613 zcaller=logging.go:41 function=Subtract a=10 b=1 result=9 took=0s ts= 2019-02-19t03:20:13.7144627zcaller=logging.go:41 function=Subtract a=10 b=1 result=9 took=0s ts= 2019-02-19t03:20:14.2276079zcaller=logging.go:41 function=Subtract a=10 b=1 result=9 took=0s ts= 2019-02-19t03:20:14.741427zcaller=server.go:112 err="Rate limit exceed!"Ts = 2019-02-19 T03:20:15. 2091773 zcaller=logging.go:41 function=Subtract a=10 b=1 result=9 took=0s ts= 2019-02-19T03:20:16.0261559zcaller=logging.go:41 function=Subtract a=10 b=1 result=9 took=0s ts= 2019-02-19t03:20:16.6406654zcaller=server.go:112 err="Rate limit exceed!"Ts = 2019-02-19 T03: did. 1912533 zcaller=logging.go:41 function=Subtract a=10 b=1 result=9 took=0s ts= 2019-02-19t03:20:17.7828906zcaller=server.go:112 err="Rate limit exceed!"
Copy the codeAs you can see from the log, Rate limit exceed! , that is, after the current limiter has issued the token, the request will be interrupted and the service is unavailable; On subsequent access, the service resumes, i.e., the limiter resumes filling the token bucket.
Gokit built-in implementation scheme
Step-1: Creates a current limiter
First download the dependent go/ Time /rate package and install it as follows (go get instruction cannot be used directly) :
git clone https://github.com/golang/time.git [Your GOPATH]/src/golang.org/x
Copy the codeThen in instrument. Go NewTokenBucketLimitterWithBuildIn adding method, in which x/time/method to realize current limiting rate:
/ / NewTokenBucketLimitterWithBuildIn x/time/rate create current-limiting middleware func NewTokenBucketLimitterWithBuildIn (BKT * rate Limiter) endpoint.Middleware {return func(next endpoint.Endpoint) endpoint.Endpoint {
return func(ctx context.Context, request interface{}) (response interface{}, err error) {
if! bkt.Allow() {
return nil, ErrLimitExceed
}
return next(ctx, request)
}
}
}
Copy the codeStep-2: Modify main
Change the encapsulation of the current limiting method to the following implementation:
//add ratelimit,refill every second,set capacity 3
ratebucket := rate.NewLimiter(rate.Every(time.Second*1), 3)
endpoint = NewTokenBucketLimitterWithBuildIn(ratebucket)(endpoint)
Copy the codeStep-3: compile & run
Compile and run the application on the console, then test it through the Postman request interface, and you can see the output log message:
Ts = 2019-02-19 T06:03:26. 8650217 zcaller=logging.go:41 function=Subtract a=10 b=1 result=9 took=0s ts= 2019-02-19t06:03:27.574717zcaller=logging.go:41 function=Subtract a=10 b=1 result=9 took=0s ts= 2019-02-19T06:03:28.1274404zcaller=logging.go:41 function=Subtract a=10 b=1 result=9 took=0s ts= 2019-02-19T06:03:28.5892068zcaller=logging.go:41 function=Subtract a=10 b=1 result=9 took=0s ts= 2019-02-19T06:03:29.1327522zcaller=server.go:112 err="Rate limit exceed!"Ts = 2019-02-19 T06:03:29. 59453 zcaller=logging.go:41 function=Subtract a=10 b=1 result=9 took=0s ts= 2019-02-19T06:03:30.2138805zcaller=server.go:112 err="Rate limit exceed!"Ts = 2019-02-19 T06:03:30. 6257682 zcaller=logging.go:41 function=Subtract a=10 b=1 result=9 took=0s ts= 2019-02-19T06:03:31.2772011zcaller=server.go:112 err="Rate limit exceed!"
Copy the codeThe log shows the same effect as the Juju/Ratelimit scheme.
conclusion
This paper first introduces two common traffic limiting algorithms: leak-bucket algorithm and token bucket algorithm, and then realizes traffic limiting by two schemes (Juju/Ratelimit and GoKit built-in library).
In the process of service development, we need to give full consideration to the availability of services, especially those services that consume system resources, and increase the flow limiting mechanism for them to ensure the stable and reliable operation of services.
Image from the Internet.
- This article code address: github.com/raysonxin/g…
- Token Bucket:en.wikipedia.org/wiki/Token_…
- Juju/ratelimit:github.com/juju/rateli…
This article is first published in my wechat public account [Xi Yi Ang bar], welcome to scan code attention!
