I complacently came to the boss’s office: “report the boss, high usable artifact has offered, you spend huge sums of marketing of high flow when to?” The boss hehe 1: “the high traffic next Monday report, you can prepare to do the handover with him.” A burst of cold wind blowing outside the window, the next is not what will happen.

1. Start from the high concurrency and high traffic of Alibaba Double 11

As we all know, the annual Double 11 is not only a shopping carnival, but also a big test and carnival of science and technology, allowing us to see so many high-end black technology full of imagination. In the whole process of high concurrency and high traffic, Sentinel undertook the core scene, which perfectly guaranteed the stability of Alibaba’s singles’ Day.

With the popularity of microservices, stability between services becomes increasingly important. Sentinel takes flow as the entry point to protect the stability of service from multiple dimensions such as flow control, fusing downgrading and system load protection.

  1. Continuous peak: traffic limiting, slow call degradation
  2. SEC kill (also known as pulse flow) : limiting, slow call degradation
  3. Peak clipping: A large number of messages may arrive at the consumer end at the same time. If all messages are requested instantaneously, the system load will be too high. Sentinel uses a constant velocity model to spread messages out over a period of time, allowing the system to handle as many messages as possible while keeping the load at the processing capacity level, thus “peak clipping and valley filling”.
  4. Cold start: when the flow increases suddenly, we want the system to switch from idle to busy time longer, or if the system is in a state of idle for a long time before this, we hope to process the request rate increase slowly, after the expected time, reach the system processes the request rate set point;
  5. Automatic detection and protection of hot commodities: automatic statistics of hotspot parameters with the highest frequency of access and flow control;
  6. Uneven traffic in a cluster: Traffic limiting in a cluster is used to solve the problem that the overall traffic limiting effect is not accurate due to uneven traffic among machines in a cluster.
  7. Complete real-time monitoring: Sentinel also provides real-time monitoring capabilities. From the console, you can see the second level data of a single machine connected to the application, or even the summary performance of a cluster of less than 500 machines.

As a fully functional and highly available flow control framework, the Core sentinel-Core package is only 286KB (latest version 1.7.2) and is lightweight enough to safely introduce Sentinel-Core without having to worry about dependency issues (face, no spotty is not possible). According to the official data, the performance loss brought by the introduction of Sentinel is very small, and the loss is almost negligible when the single QPS is not too large (the loss of single 4.3W QPS is about 2.36%).

Ii. Core functions and principles

2.1 Module Description

  • Sentinel-core core modules, current limiting, degradation, system protection and so on are implemented here
  • Sentinel – Dashboard console module, can be connected to the Sentinel client to achieve visual management
  • The Sentinel-Transport transport module provides basic monitoring server and client APIS, as well as some implementations based on different libraries
  • Sentinel -extension module, mainly for DataSource partial extension implementation
  • Sentinel-adapter adapter module, mainly to achieve some common framework adaptation
  • How to use Sentinel for traffic limiting and degradation
  • The Sentinel-benchmark module provides benchmarks for the accuracy of core code
  • Sentinel-logging module, which integrates Sentinel logs into the project through SLF4J
  • Sentinel-cluster cluster, which provides cluster flow control in version 1.4

In terms of usage, there are two main parts: Core library (Java client) : a runtime environment that does not depend on any framework/library, can run on Java 7 or above, and has good support for frameworks such as Dubbo/Spring Cloud (see mainstream framework adaptation). Console (Dashboard) : The Dashboard is mainly responsible for managing push rules, monitoring, and managing machine information.

2.2 General Framework

Sentinel is a linked list based on seven different slots. Each Slot does its job and passes the request to the next Slot until a BlockException is thrown in one of the slots. The first three slots collect statistics, and the latter slots control whether the request is blocked or allowed based on the statistics results and configured rules.

  • NodeSelectorSlot Collects the paths of resources and stores the call paths of these resources in a tree structure for traffic degradation based on the call paths.
  • ClusterBuilderSlot is used to store resource statistics and caller information, such as RT, QPS and Thread Count of the resource. These information will be used as the basis for multi-dimensional traffic limiting and degradation.
  • StatisticSlot is used to record and collect statistics about runtime indicators at different latitudes.
  • FlowSlot is used to control the flow according to preset flow limiting rules and slot statistics.
  • AuthoritySlot controls the blacklist and whitelist according to the configured blacklist and whitelist and the source information of the call.
  • Degrades a pack with statistics and preset rules.
  • SystemSlot controls the total incoming traffic based on the system state, such as load1.

    Sentinel also provides an easy-to-use, sophisticated SPI extension interface. Allows developers to quickly customize logic by implementing extended interfaces. For example, customize rule management and adapt dynamic data sources.

2.3 the core concept: the Resource/Entry/Context/Node

  • Resource refers to anything protected in Sentinel as a Resource. It can be a service, a method in a service, or even a piece of code. Any code defined through the Sentinel API is a resource that can be protected by Sentinel. In most cases, resources can be identified using method signatures, urls, and even service names as resource names.

  • Context Context is a class that holds metadata about the current state of the calling chain. Each time a resource is entered, a Context is created. Multiple contexts may be created for the same resource name. A Context contains three core objects: 1) the root node of the current calling chain: EntranceNode 2) the current Entry: Entry 3) the node associated with the current Entry: The Node Context holds only one Entry that is currently being processed, along with the root Node of the call chain. Note that every time a new resource is entered, a new Context is created.

  • Entry each call to SphU# Entry () generates an Entry, which stores the following data: the creation time of the Entry, the node associated with the current Entry, and the node corresponding to the call source associated with the current Entry. Entry is an abstract class that has only one implementation class, a static class in CtSph: CtEntry

  • The Node Node is used to store real-time statistics of a resource. It is an interface. You can access the Node to obtain the real-time status of the corresponding resource and perform traffic limiting and degradation based on it.

2.4 Easy access and Dashboard overview

The official demo code is relatively perfect:

/** * Configure rules */
private static void initFlowRules(a){
    List<FlowRule> rules = new ArrayList<>();
    FlowRule rule = new FlowRule();
    rule.setResource("HelloWorld");
    rule.setGrade(RuleConstant.FLOW_GRADE_QPS);
    // Set limit QPS to 20.
    rule.setCount(20);
    rules.add(rule);
    FlowRuleManager.loadRules(rules);
}

public static void main(String[] args) {
    // Configure rules.
    initFlowRules();

    while (true) {
        // Since version 1.5.0, you can use try-with-resources directly to automatically exit entry
        try (Entry entry = SphU.entry("HelloWorld")) {
            // Protected logic
            System.out.println("hello world");
	} catch (BlockException ex) {
            // Handle flow-controlled logic
	    System.out.println("blocked!"); }}}Copy the code

If it is SpringCloud project, https://github.com/sentinel-group/sentinel-guides/tree/master/sentinel-guide-spring-cloud for reference

Three, production environment under the use of some thinking

3.1 Traffic control rule management and push

As mentioned earlier, Sentinel allows you to configure rules dynamically in real time, rather than having them predefined in code that cannot be changed. In reality, traffic is dynamic. For example, a promotion may lead to a significant increase in traffic, and there may even be some unexpected traffic. For best protection, dynamic rules can pull a dying application back from the dying edge. However, by default, after the rules are set in Dashboard, the rules are pushed to the client through THE API and directly updated to the memory, extending the write datasource. This method is simple and has no dependency, but its disadvantages are also obvious: it does not guarantee consistency; The rules are saved in memory and disappear upon reboot. Therefore, the Push mode is recommended for the production line

1. Introduce Sentinel module and Nacos storage extension in Spring Cloud application:

implementation ('com. Alibaba. Cloud: spring -- cloud - starter - alibaba - sentinel: 2.1.2. RELEASE'){
    exclude group:'com.alibaba',module: 'fastjson'}
implementation 'com. Alibaba. CSP: sentinel - a datasource - nacos: 1.7.2. RELEASE'
Copy the code

2. Add the configuration information to the application

spring:
  cloud:
    sentinel:
      transport:
        # Sentinel Dashboard address
        dashboard: localhost:8080
      datasource:
        ds:
          nacos:
            # nacos address
            server-addr: localhost:8848
            GroupId for storing rules in nacOS
            dataId: madashu-test-sentinel
            DataId for storing rules in nacOS
            groupId: DEFAULT_GROUP
            Type of rule to store
            rule-type: flow
Copy the code

3. Set corresponding rules in Nacos

JSON: [{"resource": "/madashu/hello"."limitApp": "default"."grade": 1."count": 5."strategy": 0."controlBehavior": 0."clusterMode": false}]Copy the code
  • Resource: indicates the name of the resource
  • LimitApp: call source for flow control. Default does not distinguish the call source
  • Grade: Type of traffic limiting threshold (0- Traffic limiting based on the number of concurrent requests 1- Traffic control based on QPS)
  • Count: traffic limiting threshold
  • Strategy: Invokes the relational traffic limiting policy
  • ControlBehavior: Traffic control effect (direct reject, WarmUP, uniform queuing)
  • ClusterMode: indicates whether the clusterMode is used

The official community provides Demo for saving and subscribling traffic limiting rules. If you need to configure fusible downgrading, system protection, and gateway traffic limiting, you can refer to the configuration. Basic approach: Dashboard serializes and saves xxRuleEntityVO models to nacOS, and applications are generated from nacOS subscriptions into xxRule domain models.

3.2 How can I Set the Single-node Traffic Limiting Threshold

This must not copy the online configuration or head, otherwise it will lead to large-scale production line mistakenly killed or paralyzed, must be configured according to the capacity planning and water level setting. For details, you can adjust the weight and proportion of a node’s flow in soft load by referring to the single-node capacity planning idea until it approaches the limit. Record the QPS of the limit state and set the standard according to 70% water level of the single-machine room, so that the single-machine current limiting threshold of the resource can be calculated.

4. Migrate from Hystrix to Sentinel

Hystrix has been used in most microservices projects. Moving from Hystrix to Sentinel is not a line-by-line configuration. Don’t be misled by some tutorials online.

4.1 Why does Sentinel not support thread pool isolation

For example, Sentinel does not provide thread pool isolation, which is heavier, but semaphore isolation, which is lighter than Hystrix. The advantage of thread pool isolation is that it has a high degree of isolation, and can be processed for one resource in the thread pool without affecting other resources, but the cost is a large number of threads, thread context switching is relatively large, especially for low latency calls. In addition, managed thread switching can cause problems with lost context delivery based on ThreadLocal (such as Spring transaction management).

4.2. Migration scheme

Hystrix function The migration plan
Thread pool isolation/semaphore isolation Sentinel does not support thread pool isolation; Semaphore isolation corresponds to thread number limiting in Sentinel
fuse Sentinel supports fuse downgrading by average response time, outlier ratio, and outlier number.
The Command to create Resources can be defined directly using the Sentinel SphU API, which is separate from rule configuration, as described here
Rule configuration The configuration rules can be hardcoded through the API in Sentinel, and multiple dynamic rule sources are also supported
Annotation support Sentinel also provides annotation support for easy migration
Open source Framework support Sentinel provides adaptive modules of Dubbo, Spring Cloud and gRPC out of the box. If you used Spring Cloud Netflix before, you can migrate to Spring Cloud Alibaba

Ali open source project documents are still relatively perfect, if encountered problems in the process of migration, please refer to: github.com/alibaba/Sen…

Five, combat pit: Jar package compatible pit

5.1 FastJSON Serialization Exception

There was a Spring Cloud project that used FastJSON to print entry and exit parameters in an AOP, and it worked perfectly. When sentinel was introduced, the jar package Spring-Cloud-starter-Alibaba-Sentinel :2.1.2.RELEASE was added, and the fastJSON serialization failed miserably!

java.lang.NoSuchMethodError: com.alibaba.fastjson.serializer.JavaBeanSerializer.processValue (Lcom/alibaba/fastjson/serializer/JSONSerializer; Lcom/alibaba/fastjson/serializer/BeanContext; Ljava/lang/Object; Ljava/lang/String; Ljava/lang/Object;) Ljava/lang/Object; Ljava/lang/Integer; at com.alibaba.fastjson.serializer.ASMSerializer_12_XXX.writeNormal(Unknown Source) ~[?:?]Copy the code

The sentinel JAR package relies on Fastjson1.2.62, which is buggy. For details, see github.com/alibaba/fas…

Solution: just exclude the dependent jar package, fastjson1.2.60 does not have this problem. There is no verification of whether this issue has been fixed in the latest 1.2.68, you can check it if you are interested.

implementation ('com. Alibaba. Cloud: spring -- cloud - starter - alibaba - sentinel: 2.1.2. RELEASE'){
    exclude group:'com.alibaba',module: 'fastjson'}
Copy the code

5.2. Sleuth and Sentinel, let’s fight

In microservices, a request often goes through multiple business modules, such as user initiated payment, which may go through membership service, transaction service, payment service, account service, etc. At this time, once a problem occurs in a service, it will be very difficult to locate. On Spring Cloud projects, it is common to introduce the link call tracing framework Spring Cloud Sleuth+ Zipkin. It is important to note that if Feign is used in a project, the fuse may have failed. Sleuth extended Feign to transmit the information of link call tracking, and Sentinel extended Feign to achieve fusing current limiting, which led to the conflict between the two unrelated frameworks.

Solution: Plan 1: Make a decision, haha. Sleuth supports Hystrix, Hystrix has stopped maintenance, Sentinel is on fire, and maybe Sleuth will support Sentinel soon. Solution 2: Switch to another link call tracking framework. There are many open source products on the Internet. My previous company also developed and opened source such a system. Scheme 3: recently it was found that a big guy on the Internet provided a solution to modify the source code. The implementation is still very simple, integrating Sentinel into Sleuth

www.cnblogs.com/yinjihuan/p…

The above is just a superficial experience of Sentinel. There are many excellent design ideas, advanced algorithms, and code worth reading in Sentinel. We will taste it later.

Reference, acknowledgment:

Blog.csdn.net/g6U8W7p06dC… www.cnblogs.com/yinjihuan/p… Github.com/alibaba/Sen… www.jianshu.com/p/5a468b6a0…

Public number: Code uncle senior programmer, architect technical community. Micro service | | | large data architecture design technology management.