preface

A few days ago, a service in the production environment of the company triggered the flow control mechanism of Sentinel due to the increase of traffic. Then, users reported that the access was slow, and the positioning was found to be caused by a scheduled task. Later, the task was optimized and released, and the traffic returned to normal.

This is a perfectly normal production problem, probably most of the students have experienced, most of the experienced problems are solved after nothing, resulting in the possibility of accidents and recurrence, and ultimately caused a bad experience for users. So I think all production problems need to be reviewed, of course, the purpose of the review is not for accountability, but to prevent the same mistake from happening again. Then let’s briefly analyze this problem. Firstly, it must be the negligence of the business level that leads to the unreasonable large number of requests sent by task. Secondly, our flow control is just simple and crude flow control without better warning measures, so we don’t know about it until it affects users (i.e. flow control or fuse breaker has been triggered).

What about our solution? The first is certainly business-level prevention, but that is not the focus of this article and will not be discussed here. The second is early warning, that is, whether we can know before the flow control is about to be triggered, and then alarm the relevant person in charge to intervene in advance to prevent the flow control fuse from being triggered. It’s not completely avoidable, but it’s better than ringing the alarm after a flow control or fuse trigger.

Since the Sentinel of Ali was used for flow control before, the specific implementation introduced in this paper is to use Sentinel’s custom slot function. This custom slot slot is described in a sentence in Sentinel official document, and then a demo code is added. I also encountered a lot of pits in the process of using, so share the results to everyone.

If you’re not familiar with Sentinel, go to Github for a quick trial and read this article. Github address: github.com/alibaba/Sen…

If you want to familiarize yourself with the custom slot feature, take a look at how Sentinel works: github.com/alibaba/Sen…

There is also the source demo for custom slot writing: github.com/alibaba/Sen…

The specific implementation

Here are some implementations of Sentinel warning, if you already use Sentinel flow control or fuses.

  1. Custom CustomSlotChainBuilder implements SlotChainBuilder interface, where we add our custom Slot to SlotChain
import com.alibaba.csp.sentinel.slotchain.ProcessorSlotChain; import com.alibaba.csp.sentinel.slotchain.SlotChainBuilder; import com.alibaba.csp.sentinel.slots.DefaultSlotChainBuilder; import com.qiaofang.tortoise.gateway.component.ApplicationContextUtil; import com.qiaofang.tortoise.gateway.config.SentinelProperties; import org.springframework.stereotype.Component; import javax.annotation.Resource; /** * custom slot ** @author chenhao */ public class CustomSlotChainBuilder implements SlotChainBuilder {@override public ProcessorSlotChainbuild() {
        ProcessorSlotChain chain = new DefaultSlotChainBuilder().build();
        SentinelProperties sentinelProperties = (SentinelProperties) ApplicationContextUtil.getContext().getBean("sentinelProperties");
        chain.addLast(new FlowEarlyWarningSlot(sentinelProperties));
        chain.addLast(new DegradeEarlyWarningSlot(sentinelProperties));
        returnchain; }}Copy the code

2. Customize FlowEarlyWarningSlot and DegradeEarlyWarningSlot

Custom FlowEarlyWarningSlot

import com.alibaba.csp.sentinel.context.Context; import com.alibaba.csp.sentinel.node.DefaultNode; import com.alibaba.csp.sentinel.slotchain.AbstractLinkedProcessorSlot; import com.alibaba.csp.sentinel.slotchain.ResourceWrapper; import com.alibaba.csp.sentinel.slots.block.flow.FlowRule; import com.alibaba.csp.sentinel.slots.block.flow.FlowRuleChecker; import com.alibaba.csp.sentinel.slots.block.flow.FlowRuleManager; import com.alibaba.csp.sentinel.slots.block.flow.FlowRuleUtil; import com.alibaba.csp.sentinel.util.AssertUtil; import com.google.common.collect.Lists; import org.slf4j.Logger; import org.slf4j.LoggerFactory; import org.springframework.beans.BeanUtils; import org.springframework.util.CollectionUtils; import java.util.List; import java.util.Map; import java.util.stream.Collectors; /** * slot ** @author chenhao */ public class FlowEarlyWarningSlot2 extends AbstractLinkedProcessorSlot<DefaultNode> { /** *log
     */
    private Logger logger = LoggerFactory.getLogger(this.getClass());

    private final FlowRuleChecker checker;

    public FlowEarlyWarningSlot2() {
        this(new FlowRuleChecker());
    }

    /**
     * Package-private forTest. * * @param Checker flow rule checker * @since 1.6.1 */ FlowEarlyWarningSlot2(FlowRuleChecker checker checker) { AssertUtil.notNull(checker,"flow checker should not be null");
        this.checker = checker;
    }


    private List<FlowRule> getRuleProvider(String resource) {
        // Flow rule map should not be null.
        List<FlowRule> rules = FlowRuleManager.getRules();
        List<FlowRule> earlyWarningRuleList = Lists.newArrayList();
        for(FlowRule rule : rules) { FlowRule earlyWarningRule = new FlowRule(); BeanUtils.copyProperties(rule, earlyWarningRule); / * * * this is equivalent to 80% of the changing rules threshold to the original, to achieve the effect of early warning, * * make 0.8 a configuration is proposed, / earlyWarningRule setCount (rule. The getCount () * 0.8); earlyWarningRuleList.add(earlyWarningRule); } Map<String, List<FlowRule>> flowRules = FlowRuleUtil.buildFlowRuleMap(earlyWarningRuleList);return flowRules.get(resource);
    }

    /**
     * get origin rule
     *
     * @param resource
     * @return
     */
    private FlowRule getOriginRule(String resource) {
        List<FlowRule> originRule = FlowRuleManager.getRules().stream().filter(flowRule -> flowRule.getResource().equals(resource)).collect(Collectors.toList());
        if (CollectionUtils.isEmpty(originRule)) {
            return null;
        }
        return originRule.get(0);
    }

    @Override
    public void entry(Context context, ResourceWrapper resourceWrapper, DefaultNode node, int count, boolean prioritized, Object... args)
            throws Throwable {
        String resource = context.getCurEntry().getResourceWrapper().getName();
        List<FlowRule> rules = getRuleProvider(resource);
        if(rules ! = null) {for(FlowRule rule: rules) {// All the rules obtained here are 80% of the configured threshold. If the threshold is detected here, it means that the threshold is 80% of the real threshold. You can send an alarm to the responsible personif(! checker.canPassCheck(rule, context, node, count, prioritized)) { FlowRule originRule = getOriginRule(resource); String originRuleCount = originRule == null ?"Unknown" : String.valueOf(originRule.getCount());
                    logger.info("FlowEarlyWarning: The current traffic indicator of service {} has exceeded {} and is close to the configured flow control threshold :{},", resource, rule.getCount(), originRuleCount); // The TODO alarm function is self-implementedbreak;
                }
            }
        }
        fireEntry(context, resourceWrapper, node, count, prioritized, args);
    }

    @Override
    public void exit(Context context, ResourceWrapper resourceWrapper, int count, Object... args) { fireExit(context, resourceWrapper, count, args); }}Copy the code

DegradeEarlyWarningSlot

import com.alibaba.csp.sentinel.context.Context; import com.alibaba.csp.sentinel.node.DefaultNode; import com.alibaba.csp.sentinel.slotchain.AbstractLinkedProcessorSlot; import com.alibaba.csp.sentinel.slotchain.ResourceWrapper; import com.alibaba.csp.sentinel.slots.block.degrade.DegradeRule; import com.alibaba.csp.sentinel.slots.block.degrade.DegradeRuleManager; import com.alibaba.csp.sentinel.util.AssertUtil; import com.google.common.collect.Lists; import com.qiaofang.tortoise.gateway.config.SentinelProperties; import org.slf4j.Logger; import org.slf4j.LoggerFactory; import org.springframework.beans.BeanUtils; import org.springframework.util.CollectionUtils; import java.util.List; import java.util.stream.Collectors; /** * slot ** @author chenhao */ public class DegradeEarlyWarningSlot2 extends AbstractLinkedProcessorSlot<DefaultNode> { /** *log*/ private Logger logger = LoggerFactory.getLogger(this.getClass()); * @param resource * @param resource * @return
     */
    private List<DegradeRule> getRuleProvider(String resource) {
        // Flow rule map should not be null.
        List<DegradeRule> rules = DegradeRuleManager.getRules();
        List<DegradeRule> earlyWarningRuleList = Lists.newArrayList();
        for(DegradeRule rule : rules) { DegradeRule earlyWarningRule = new DegradeRule(); BeanUtils.copyProperties(rule, earlyWarningRule); EarlyWarningRule. SetCount (rule. GetCount () * 0.8); earlyWarningRuleList.add(earlyWarningRule); }return earlyWarningRuleList.stream().filter(rule -> resource.equals(rule.getResource())).collect(Collectors.toList());
    }

    /**
     * get origin rule
     *
     * @param resource
     * @return
     */
    private DegradeRule getOriginRule(String resource) {
        List<DegradeRule> originRule = DegradeRuleManager.getRules().stream().filter(rule -> rule.getResource().equals(resource)).collect(Collectors.toList());
        if (CollectionUtils.isEmpty(originRule)) {
            return null;
        }
        return originRule.get(0);
    }

    @Override
    public void entry(Context context, ResourceWrapper resourceWrapper, DefaultNode node, int count, boolean prioritized, Object... args)
            throws Throwable {
        String resource = context.getCurEntry().getResourceWrapper().getName();
        List<DegradeRule> rules = getRuleProvider(resource);
        if(rules ! = null) {for (DegradeRule rule : rules) {
                if(! rule.passCheck(context, node, count)) { DegradeRule originRule = getOriginRule(resource); String originRuleCount = originRule == null ?"Unknown" : String.valueOf(originRule.getCount());
                    logger.info("DegradeEarlyWarning: The current fuse indicator of service {} has exceeded {} and is close to the configured fuse threshold :{},", resource, rule.getCount(), originRuleCount);
                    break;
                }
            }
        }
        fireEntry(context, resourceWrapper, node, count, prioritized, args);
    }

    @Override
    public void exit(Context context, ResourceWrapper resourceWrapper, int count, Object... args) {
        fireExit(context, resourceWrapper, count, args);
    }
Copy the code

3. Under the resources folder of new meta-inf. Services folder, new file com. Alibaba. CSP. Sentinel. Slotchain. SlotChainBuilder (file name doesn’t matter) content is as follows

Write your complete package path for CustomSlotChainBuilder here
com.xxx.sentinel.CustomSlotChainBuilder
Copy the code

Here is basically ok, with the process or encountered a lot of pits, simply list a few

  • It is not possible to change the count property of the FlowRule directly, because the underlying validation rules use the Controller property of the FlowRule, which is private, so you can get the original configuration first and then regenerate it through FlowRuleUtil
  • DefaultNode: DefaultNode: DefaultNode: DefaultNode: DefaultNode: DefaultNode: DefaultNode: DefaultNode: DefaultNode

Write in the last

I rarely write this kind of technical blog, so what question, or not rigorous place, we can put forward, beg light spray me ha ha ha

PS: This article was written by a friend of mine. Welcome to submit good articles

If you are interested, you can pay attention to my wechat public number, Simtiandi, and read more technical articles for the first time. I also have some open source code on GitHub github.com/yinjihuan