1. Meet the Sentinel

1.1. Avalanche problems and solutions

1.1.1. Avalanche problem

In microservices, the invocation relationship between services is complicated, and one microservice often depends on several other microservices.

An avalanche occurs when a service in a microservice invocation link fails, causing all microservices in the entire link to become unavailable.

The server supports a limited number of threads and concurrence, and requests that are blocked will run out of server resources, causing all other services to become unavailable and the current service to become unavailable.

Then, other services that depend on the current service will eventually become unavailable over time, resulting in cascading failures and an avalanche of failures:

There are four common ways to solve an avalanche problem:

  1. Timeout handling
  2. The warehouse wall model
  3. Fusing the drop
  4. Flow control

1.1.2. Timeout handling

Scheme 1: Timeout processing

Set the timeout period. If there is no response after a certain time, an error message will be returned

1.1.3. Warehouse wall mode

Scheme 2: warehouse wall mode

The wall pattern is derived from the design of the cabin:

The cabin is divided into separate Spaces by bulkheads. When the hull is damaged, only part of the space will enter, keeping the fault within a certain range and preventing the whole hull from being flooded.

Similarly, we can limit the number of threads that each business can use to avoid exhausting the entire Tomcat resource, hence thread isolation.

1.1.4. Fuse downgrade

Plan 3: circuit breaker downgrade

The circuit breaker collects statistics on the proportion of abnormal service execution. If the proportion exceeds the threshold, the circuit breaker interrupts the service and blocks all requests to access the service.

The circuit breaker counts the number of requests to access a service.

If the proportion of abnormal requests to access service D is too high, service D is considered to have avalanches and will intercept all requests to access service D, resulting in a circuit breaker:

1.1.5. Flow control

Scheme 4: Flow control

Flow control: QPS that restricts service access to prevent service failures due to sudden traffic increase.

1.1.6. Summary

What is the avalanche problem?

  • Microservices call each other, because one service in the call chain fails, the whole link becomes inaccessible.

How to avoid service failures caused by instantaneous high concurrent traffic?

  • Flow control

How can you avoid an avalanche of problems caused by service failures?

  • Timeout handling
  • Thread isolation
  • The drop fuse

It can be considered that:

  • Traffic limiting protects services from service failures caused by instantaneous high concurrent traffic, thus avoiding avalanches. It’s a preventive measure.
  • Timeout processing, thread isolation, and degraded circuit breaker are used to control the fault within a certain range when some services fail to avoid an avalanche. It’s a remedy.

1.2. Comparison of service protection technologies

Support for multiple service protection technologies in SpringCloud:

  • Netfix Hystrix
  • Sentinel
  • Resilience4J

Hystrix framework was popular in the early stage, but the Sentinel framework of Alibaba is the most widely used in China. Here we make a comparison:

Sentinel Hystrix
Isolation strategy Semaphore isolation Thread pool isolation/semaphore isolation
Fuse downgrading strategy Based on the ratio of slow calls or exceptions Based on failure ratio
Real-time indicator realization The sliding window Sliding Windows (based on RxJava)
Rule configuration Support for multiple data sources Support for multiple data sources
scalability Multiple extension points Plug-in form
Annotation-based support support support
Current limiting Based on QPS, traffic limiting based on call relationships is supported Limited support
Traffic shaping Support slow start, uniform queuing mode Does not support
System adaptive protection support Does not support
The console Out of the box, you can configure rules, view second-level monitoring, machine discovery, etc imperfect
Common framework adaptation Servlet, Spring Cloud, Dubbo, gRPC, etc The Servlet, SpringCloud Netflix

1.3.Sentinel introduction and installation

1.3.1. Meet Sentinel

Sentinel is an open source micro service traffic control component of Alibaba. Sentinelguard. IO/zh-CN /index…

Sentinel has the following characteristics:

  1. Rich application scenarios: Sentinel has undertaken the core scenarios of Alibaba’s double Eleven traffic drive in the past 10 years, such as SEC killing (i.e. burst traffic control within the range of system capacity), message peaking and valley filling, cluster flow control, real-time fusing of unavailable downstream applications, etc.
  2. Complete real-time monitoring: Sentinel also provides real-time monitoring capabilities. From the console, you can see a summary of the performance of a single machine-by-second data, or even a cluster of less than 500 machines, for accessing the application.
  3. Extensive Open source ecosystem: Sentinel provides out-of-the-box integration modules with other open source frameworks/libraries, such as Spring Cloud, Dubbo, and gRPC. You can quickly access Sentinel by introducing the appropriate dependencies and simple configuration.
  4. Sophisticated SPI extension points: Sentinel provides an easy-to-use, sophisticated SPI extension interface. You can quickly customize the logic by implementing an extension interface. For example, customize rule management and adapt dynamic data sources.

1.3.2. Install Sentinel console

  1. download

Sentinel provides a UI console for limiting traffic on the system. You can download it on GitHub.

  1. run

Place the JAR package in any non-Chinese directory and run the following command:

Java jar sentinel - dashboard -, version 1.8.1. JarCopy the code

To change the default port, account, and password of Sentinel, perform the following operations:

Configuration items The default value instructions
server.port 8080 Service port
sentinel.dashboard.auth.username sentinel Default Username
sentinel.dashboard.auth.password sentinel The default password

For example, to modify a port:

Java - Dserver. Port = 8090 - jar sentinel - dashboard -, version 1.8.1. JarCopy the code
  1. access

accessThe sentinel console is available at http://localhost:8090:

You need to enter your account and password, both of which are sentinel by default

When I log in, I find nothing:That’s because we haven’t integrated with microservices yet. To use Sentinel, you must combine microservices, and here I have a cloud-Demo project. The project structure is as follows:

1.4. Sentinel microservices integration

We integrate sentinel in the order-service and connect the sentinel console as follows:

1) Sentinel dependency is introduced

<! --sentinel--> <dependency> <groupId>com.alibaba.cloud</groupId> <artifactId>spring-cloud-starter-alibaba-sentinel</artifactId> </dependency>Copy the code

2) Configure the console

Modify the application.yaml file to add the following:

server:
  port: 8080
spring:
  cloud: 
    sentinel:
      transport:
        dashboard: localhost:8090
Copy the code

3) Access any endpoint of the ORder-service to trigger sentinel monitoring

Open a browser, visit http://localhost:8090/order/101 to trigger a sentinel surveillance (create a demo test)

Then go to sentinel’s console and see what it looks like:

2. Flow control

Although there are four schemes for avalanche problem, traffic limiting is to avoid service failure due to sudden flow and to prevent micro-service avalanche problem. Let’s learn the model first.

2.1. Cluster point link

Cluster-point links: call links within a project. When a request enters a microservice, it first visits the DispatcherServlet, then the Controller, Service, and Mapper. Such a chain of calls is called a cluster point link. Each monitored interface in a cluster-point link is a resource.

By default sentinel monitors every Endpoint of SpringMVC (the method in Controller), so each Endpoint of SpringMVC is a resource in the invocation link.

For example, the endpoint in OrderController in the order-service we just visited: /order/{orderId}

Flow control and fusing are set for resources in cluster point links, so we can click the button behind the corresponding resource to set rules:

  • Flow control: Flow control
  • Downgrade: Downgrade circuit breaker
  • Hotspot: hotspot parameter traffic limiting, a type of traffic limiting
  • Authorization: Permission control for requests

2.1. Quick start

2.1.1. The sample

Click the flow control button after resource/Order /{orderId} to pop up the form.

You can fill in traffic limiting rules as follows:

Limit /order/{orderId} to a single QPS of 1, i.e., only one request per second is allowed. Any request exceeding this limit will be intercepted with an error.

2.1.2. Practice

Requirement: Set the flow control rule for the /order/{orderId} resource, QPS can’t exceed 5, then test.

1) First add flow limiting rules to sentinel console

2) Test with JMeter

If you haven’t used JMeter before, see JMeter Quick Start

Open JMeter and import a prewritten JMeter test sample:

Options:

20 users, run in 2 seconds, QPS 10, over 5.

Start the service:

Select Flow control start, QPS<5 right click to run:

Note that do not click the Execute button in the menu to run.

Results:

As you can see, there are only five successful requests at a time

2.2. Flow control mode

When adding flow limiting rules, click Advanced options to select three flow control modes:

  • Direct: Collects statistics on current resource requests and limits traffic for current resources when a threshold is triggered. This mode is also the default mode
  • Association: Collects statistics on other resources related to the current resource. When the threshold is triggered, traffic limiting for the current resource is performed
  • Link: collects statistics on the requests from the specified link to the resource and limits traffic on the specified link when a threshold is triggered

Quick start testing is in direct mode.

2.2.1. Association mode

Association mode: Collects statistics on other resources related to the current resource and limits traffic for the current resource when a threshold is triggered

Configuration rules:

Syntax Description: When a threshold is triggered for the number of visits to /write resources, traffic is restricted for/READ resources to avoid affecting/Write resources.

Usage scenario: For example, the user needs to modify the order status during payment and query the order. Query and modify operations compete for database locks. Service requirements are priority payment and order update services. Therefore, when the triggering threshold of the order service is changed, the flow of the query order service needs to be restricted.

Requirements:

  • Create two new endpoints in OrderController: / ORDER /query and/Order /update without implementing the business
  • Configure a flow control rule to limit the flow of/ORDER/Query requests when the number of QPS accessed by /order/update resources exceeds 5

1) Define/Order/Query endpoints to simulate order queries

@getMapping ("/query") public String queryOrder() {return "query successful "; }Copy the code

2) Define/Order /update endpoints to simulate order updates

@getMapping ("/update") public String updateOrder() {return "updateOrder succeeded "; }Copy the code

Restart the service, access both interfaces, and view the cluster point link of the Sentinel console:

3) Configure flow control rules

Click the button after the endpoint to limit the flow. We are limiting the flow of the order query/Order /query, so click the button after it:

Fill in flow control rules in the form:

4) Test in Jmeter

Select Flow Control Mode – Relevance:

You can see 1000 users, 100 seconds, so the QPS is 10, above the threshold we set: 5

To view HTTP requests:

The target of the request is /order/update, so that the breakpoint triggers the threshold.

/order/query = /order/query

The flow is restricted.

5) Summarize that the association pattern can be used if the following conditions are met:

  1. Two competing resources
  2. One has a higher priority and one has a lower priority

2.2.2. Link mode

Link mode: Collects statistics on the requests from the specified link to the resource and determines whether the number exceeds the threshold.

Configuration example:

For example, there are two request links:

  • /test1 –> /common

  • /test2 –> /common

If you only want to count requests coming into /common from /test2, you can configure it like this:

Practical cases

Requirements: There are order query and order creation services, both of which need to query for goods. For the query order into the query goods request statistics, and set the flow limit.

Steps:

  1. Add a queryGoods method to the OrderService without implementing the business

  2. In OrderController, modify the/Order/Query endpoint to call the queryGoods method in OrderService

  3. Add a/Order/Save endpoint to OrderController and call the queryGoods method of OrderService

  4. QueryGoods (/order/query)

Implementation:

1) Add commodity query methods

In the order-service service, add a queryGoods method to the OrderService class:

Public void queryGoods(){system.err. Println (" queryGoods "); }Copy the code

2) When querying orders, query goods

In the OrderController of the order-service, modify the business logic of the/Order/Query endpoint:

@getMapping ("/query") public String queryOrder() {// Query orderService.queryGoods(); System.out.println(" query order "); Return "Order query succeeded "; }Copy the code

3) New orders, query goods

In the OrderController of the order-service, modify the/Order /save endpoint to simulate the new order:

@GetMapping("/save")
public String saveOrder(a) {
    // Query the item
    orderService.queryGoods();
    // Query the order
    System.err.println("New Order");
    return "New order successfully";
}
Copy the code

4) Add resource tags to query items

By default, methods in OrderService are not monitored by Sentinel, requiring us to annotate the methods to be monitored ourselves.

Add @sentinelResource annotation to the queryGoods method of OrderService:

@SentinelResource("goods")
public void queryGoods(a){
    System.err.println("Commodity Inquiry");
}
Copy the code

In link mode, two links from different sources are monitored. But sentinel defaults to setting the same root resource for all requests going into SpringMVC, which invalidates link mode.

We need to turn off this resource aggregation for SpringMVC and modify the application-yml file of the Order-service service:

spring:
  cloud:
    sentinel:
      web-context-unify: false # Close context integration
Copy the code

/order/query /order/save

5) Add flow control rules

Click the flow control button behind the Goods resource and fill in the following information in the form that pops up:

Only the resources that enter /goods from /order/query are counted. The QPS threshold is 2.

6) Jmeter test

Select Flow Control Mode – Link:

You can see here 200 users, post in 50 seconds, QPS is 4, which is above our threshold of 2

One HTTP request is to access /order/save:

Results of the run:

Completely unaffected.

Another is to call /order/query:

Running results:

Only two pass at a time.

2.2.3. Summary

What are the flow control modes?

  • Direct: Limits the current resource flow
  • Association: Triggers the threshold for resources with a higher priority and limits traffic for resources with a lower priority.
  • Link: When threshold statistics are collected, only the requests from the specified resource to the current resource are collected