Microservice governance Practices: How to automatically remove single point exceptions

• Some application grayscale release, first on a few machines, because the code logic to write a problem, causing the thread pool full, abnormal running.
• In the server cluster, the load of some servers is too high due to full disks or resource contention between hosts, and the client invocation times out.
• The thread pool of some machines in the server cluster is Full, causing a Full Garbage Collection.

Microservice Outlier Ejection

• What is outlier instance removal

• The ability to remove outlier instances

• And fuse the difference

• The outlier instance removed the supported version

www.aliyun.com/product/eda…

To prepare

Github.com/aliyun/alib…

Deploy the microservice Demo on EDAS

@Value("${exception.ip}")
    private String exceptionIp;

    @Override
    public List<CartItem> viewCart(String userID) {

        if(exceptionIp ! = null && exceptionIp.equals(getLocalIp())) { throw new RuntimeException("Runtime exception");
        }

        return cartStore.getOrDefault(userID, Collections.emptyList());
    }Copy the code

• Why is cartService expanded to two instances? As you may have guessed, the runtime assigns the IP address of one instance by configuring the ACM configuration center to simulate an instance exception scenario.

Simulating service Exceptions

while :
do
        result=`curl The $1 -s`
        if [[ "$result"= = *"500"*]];then
                echo `date +%F-%T` $result
        else
                echo `date +%F-%T` "success"
        fiSleep 0.1doneCopy the code

Enable the outlier removal policy

create

The basic information

Select effective Application

Allocation strategy

Has been created

strategy

Verify the effect of outlier removal

Modify/disable the outlier removal policy

• Modify the corresponding policy rules

• Deleting a Corresponding Policy

Very different monitoring

Before starting outlier removal

After starting outlier removal

Outlier instances remove specific control logic

• The default QPS lower limit is 1

• The default error rate lower limit is 50%

• By default, the maximum number of instances removed is 20%

• Exception types

• Description of the recovery detection unit time (30s by default) and the maximum number of unrecovered times (40 by default)

• out

• General best practices

Outlier instance removal technical details

Non-invasive technology

Outlier instance removal technique

Outlier Detection Outlier Detection

Outlier Ejection