In the last week’s review, we analyzed timeout retry scenarios. This time, we will focus on the available scenarios.

1. Background

Customers have high requirements for the availability of our system, which cannot be less than 99%. In order to monitor the availability of these systems without code intrusion of each sub-product, we adopted a separate development service for exploration. In fact, exploration is commonly said: regularly call various interfaces to check whether the service is available. Different system interface access protocols are mainly divided into:

  • websocket
  • http
  • git ssh
  • git http
  • Linux Shell (SSH)

In the K8S cluster environment, all kinds of proxy forwarding middleware are included, resulting in excessively long links. Once network jitter or request timeout occurs, the link will be misjudged as unavailability, which will eventually lead to ugly indicators. Therefore, you are advised to add a retry policy. If the retry exceeds three times, the service is considered to be unavailable.

2. Analysis of retry scheme implementation

The leader says to use the httpClient retry. I think Leader’s suggestion is quite good, but it only supports Http protocol. What about other protocols? And then you start thinking is there an easy way to do this? So I mentally described what I wanted:

  1. Is there a universal solution?
  2. Each protocol has to be developed and retried, which is a huge amount of work. I can’t get bogged down in details. I want to abstract the rules out of the details and then use thought classes like design patterns to solve problems.
  3. Business logic cannot be invaded.

Think more deeply:

  1. At the beginning of the system design, the policy pattern was used to carry on the abstraction.
  2. The proxy pattern is also used to avoid invoking policies between controller layers. The proxy model is essentially an agent, and the agent can do whatever he wants before implementing the strategy.

Okay, I think I’ve got a solution. I’m gonna go after the agent.

3. Collect retry scheme

Before doing things, I like to collect whether there are mature solutions. Because time is urgent and there is no time to repeat the wheel, I have collected two common solutions:

3.1, Spring – Retry

Spring Retry provides the ability to automatically reinvoke failed operations. This is helpful if the error may be temporary, such as a temporary network failure. Spring Retry provides declarative control over processes and policy-based behavior that is easy to extend and customize.

3.1.1. Introduce dependencies

<dependency>
    <groupId>org.springframework.retry</groupId>
    <artifactId>spring-retry</artifactId>
</dependency>

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-aop</artifactId>
</dependency>
Copy the code

3.1.2 Add comments to interfaces

@Service
@Slf4j
public class RemoteService {

    /** * Add a retry annotation to trigger a retry mechanism when an exception occurs. The default value is 3. The delay is 2000ms and the delay is increased by 1.5 times. When the return result does not meet the requirements, an error is reported to trigger retry. *@param count
     * @return
     * @throws Exception
     */
    @ Retryable (value = {RemoteAccessException. Class}, maxAttempts = 5, backoff = @ backoff (1.5) delay = 2000, multiplier =)
    public String call(Integer count) throws Exception {
        if(count == 10){
            log.info("Remote RPC call do something... {}",LocalTime.now());
            throw new RemoteAccessException("RPC call exception");
        }
        return "SUCCESS";
    }

    /** * Define the callback, noting that the exception type and method return value type are the same as the retry method *@param e
     * @return* /
    @Recover
    public String recover(RemoteAccessException e) {
        log.info("Remote RPC Call fail",e);
        return "recover SUCCESS"; }}Copy the code

There are three parameters in @retryable,

  • Value is a recoverable exception type, that is, a retry exception type.
  • The maxAttempts represent the maximum number of attempts, which is 3 by default.
  • Exclude: specifies that exceptions are not retried. The value is empty by default
  • Include, specifies exception retries. If null, exceptions are retried
  • By default, there is no delay, that is, try again immediately after the failure. Of course, a better solution is to add the delay time, depending on the service scenario, you can also not add the delay (delay = 3000L) in parentheses), the default delay is 1000ms.

@Backoff

  • Delay: Retry after the specified delay
  • “Multiplier” : indicates the multiple of delay. If the value of the delay is eg: delay=1000L, “multiplier” =2, the value of the first retry is 1 seconds, the value of the second retry is 2 seconds, and the value of the third retry is 4 seconds

Note:

Note that if the @Retryable methods are invoked at the Service layer and then at the Controller layer, the @Retryable does not work if the methods are invoked at this class. Because when @retryable is used, Spring creates a proxy around the original bean that can then be processed on an AD hoc basis, which is the principle of retry. So in this case, Spring recommends that we call an actual method, catch the exception we throw in value, and make the call based on the @Retryable configuration. With the @retryable method, you need to issue exceptions that are not sought by Retry

3.1.3, call

@RestController
@RequestMapping("/retry")
@Slf4j
public class RetryController {

    @Autowired
    private RemoteService remoteService;

    @RequestMapping("/show/{count}")
    public String show(@PathVariable Integer count){
        try {
            return remoteService.call(count);
        } catch (Exception e) {
            log.error("RetryController.show Exception",e);
            return "Hello SUCCESS"; }}}Copy the code

3.1.3. Enable retry

@SpringBootApplication
@EnableRetry // Enable retry
public class Application {

    public static void main(String[] args) { SpringApplication.run(Application.class,args); }}Copy the code

3.2, Guava Retrying

Guava-retrying is a thread-safe Java retry library that provides a common way to handle any code that needs to be retried. It can easily and flexibly control the number of retries, retry timing, retry frequency, stop timing, and exception handling.

3.2.1. Introduce dependencies

<dependency>
      <groupId>com.github.rholder</groupId>
      <artifactId>guava-retrying</artifactId>
      <version>2.0. 0</version>
</dependency>

Copy the code

3.2.2 Getting started Demo

Callable<Boolean> callable = new Callable<Boolean>() {
    public Boolean call(a) throws Exception {
        return true; // do something useful here}}; Retryer<Boolean> retryer = RetryerBuilder.<Boolean>newBuilder() .retryIfResult(Predicates.<Boolean>isNull())// Try again when callable returns null
    .retryIfExceptionOfType(IOException.class) // Callable throws IOException retry
    .retryIfRuntimeException() // Callable throws RuntimeException retry
    .withStopStrategy(StopStrategies.stopAfterAttempt(3)) // Retry three times and stop
    .build();
try {
    retryer.call(callable);
} catch (RetryException e) {
    e.printStackTrace();
} catch (ExecutionException e) {
    e.printStackTrace();
}

Copy the code

** Next to elaborate on it: **

  • RetryerBuilder is a factory creator. You can customize the retry source and support multiple retry sources. You can configure the retry times or retry timeout, and you can configure the waiting interval to create Retryer instances.
  • The retry source of RetryerBuilder supports Exception and custom assertion objects, which are set by retryIfException and retryIfResult. Multiple and compatible objects are supported simultaneously.
  • RetryIfException: Runtime and Checked exceptions will be retried, but error will not be retried.
  • RetryIfRuntimeException retries only when runtime exceptions are thrown. Checked and Error are not retried.
  • RetryIfExceptionOfType allows us to retry only when certain exceptions occur, such as NullPointerException and IllegalStateException, which are Runtime exceptions and include custom errors

​

3.2.3 WaitStrategies Retry wait policy

ExponentialWaitStrategy Indicates the wait strategy of the index

Exponential compensation algorithm Exponential Backoff

.withWaitStrategy(WaitStrategies.exponentialWait(100, 5, TimeUnit.MINUTES))
Copy the code

Create a perpetually retry retry and wait for each retry failure in an increasing exponential time, up to a maximum of 5 minutes. After 5 minutes, try again every 5 minutes. For this example:

Wait time after the first failure:2^1 * 100;2^2 * 100;2^3 * 100; .Copy the code

In ExponentialWaitStrategy, we can pay attention to the source code that calculates the wait time according to the number of retries:

@Override
public long computeSleepTime(Attempt failedAttempt) {
    double exp = Math.pow(2, failedAttempt.getAttemptNumber());
    long result = Math.round(multiplier * exp);
    if (result > maximumWait) {
        result = maximumWait;
    }
    return result >= 0L ? result : 0L;
}
Copy the code

If later have similar requirements, we can write down these algorithms, the index for more compensation algorithm an Exponential Backoff, can refer to: en.wikipedia.org/wiki/Expone…

FibonacciWaitStrategy FibonacciWaitStrategy

Fibonacci Backoff compensation algorithm

.withWaitStrategy(WaitStrategies.fibonacciWait(100, 2, TimeUnit.MINUTES))
Copy the code

Create a perpetually retry retry with Fibonacci numbers for each retry failure up to 2 minutes; After 2 minutes, try again every 2 minutes. For this example:

Waiting duration after the first failure: 1*100; 1 * 100; 2 * 100; 3 * 100; 5 * 100; .Copy the code

FixedWaitStrategy Fixed-duration wait strategy

withWaitStrategy(WaitStrategies.fixedWait(10,  TimeUnit.SECONDS))
Copy the code

Fixed waiting policy: If a failure occurs, retry is performed after a fixed waiting period.

RandomWaitStrategy RandomWaitStrategy

withWaitStrategy(WaitStrategies.randomWait(10,  TimeUnit.SECONDS));
withWaitStrategy(WaitStrategies.randomWait(1,  TimeUnit.SECONDS, 10, TimeUnit.SECONDS));
Copy the code

Random waiting policy: You can set a maximum random waiting period or a random waiting period.

IncrementingWaitStrategy IncrementingWaitStrategy

withWaitStrategy(WaitStrategies.incrementingWait(1,  TimeUnit.SECONDS, 5, TimeUnit.SECONDS))
Copy the code

Incremental wait strategy, according to the initial value and incremental value, the waiting time increases successively. For this example:

After the first failure, the system waits for 1s successively. 6 s (1 + 5); 11 (1 + 5 + 5) s; 16 (1 + 5 + 5 + 5) s; .

ExceptionWaitStrategy Abnormal wait strategy

withWaitStrategy(WaitStrategies.exceptionWait(ArithmeticException.class, e -> 1000L))
Copy the code

Specify the wait time for retries based on the exceptions that occur; If the exception does not match, the waiting duration is 0.

CompositeWaitStrategy CompositeWaitStrategy

.withWaitStrategy(WaitStrategies.join(WaitStrategies.exceptionWait(ArithmeticException.class, e -> 1000L),WaitStrategies.fixedWait(5, TimeUnit.SECONDS)))
Copy the code

Compound waiting strategy; If the program being executed satisfies one or more wait policies, the wait time is the sum of all wait policy times.

3.2.4 StopStrategies Retry stop policies

NeverStopStrategy

withStopStrategy(StopStrategies.neverStop())
Copy the code

It never stops. It always needs to be retried.

StopAfterAttemptStrategy

withStopStrategy(StopStrategies.stopAfterAttempt(3))
Copy the code

After the maximum number of retries is reached, the task is terminated.

StopAfterDelayStrategy

withStopStrategy(StopStrategies.stopAfterDelay(3, TimeUnit.MINUTES))
Copy the code

After the maximum retry duration is reached, the task is terminated regardless of the number of times the task has been executed.

BlockStrategies Blocking policies

By default, only one blocking strategy is provided: ThreadSleepStrategy, which is implemented through Thread.sleep(sleepTime). But it also gives us a lot of room to implement blocking strategies ourselves.

AttemptTimeLimiters Limit of the task execution duration

This indicates the time limit for executing a single task (if the execution of a single task times out, the current task is terminated).

** NoAttemptTimeLimit unlimited length limit **

.withAttemptTimeLimiter(AttemptTimeLimiters.noTimeLimit())
Copy the code

As the name implies, there is no limit to the execution time; Each time, after the execution of the task is completed, the subsequent retry policy is performed.

FixedAttemptTimeLimit

.withAttemptTimeLimiter(AttemptTimeLimiters.fixedTimeLimit(10, TimeUnit.SECONDS));
.withAttemptTimeLimiter(AttemptTimeLimiters.fixedTimeLimit(10, TimeUnit.SECONDS, Executors.newCachedThreadPool()));
Copy the code

You can specify duration limits for tasks, and in order to control thread management, it is best to specify the appropriate thread pool.

3.2.5 retry Listening

If you need to do something extra when retries occur, such as sending email notifications, you can use RetryListener. Guava Retryer automatically calls back listeners after each retry and supports registering multiple listeners.

@Slf4j
class DiyRetryListener<Boolean> implements RetryListener {
    @Override
    public <Boolean> void onRetry(Attempt<Boolean> attempt) {
        log.info("Retry times :{}",attempt.getAttemptNumber());
        log.info("Delay from first retry :{}",attempt.getDelaySinceFirstAttempt());
        if(attempt.hasException()){
            log.error("Abnormal cause :",attempt.getExceptionCause());
        }else {
            System.out.println("Normal processing result :{}"+ attempt.getResult()); }}}Copy the code

Once you have defined your listeners, you need to register them in Retryer.

        Retryer<Boolean> retryer = RetryerBuilder.<Boolean>newBuilder()
                .retryIfResult(Predicates.<Boolean>isNull()) // Try again when callable returns null
                .retryIfExceptionOfType(IOException.class) // Callable throws IOException retry
                .retryIfRuntimeException() // Callable throws RuntimeException retry
                .withStopStrategy(StopStrategies.stopAfterAttempt(3)) // Retry three times and stop
                .withRetryListener(new DiyRetryListener<Boolean>()) // Register the listener
                .build();

Copy the code

4, summarize

  • These two components decouple the retry logic from the system logic.
  • Think for yourself and stand on the shoulders of giants.

See you next time for spring Retry!

5, references,

Blog.csdn.net/chaoHappy/a… www.jianshu.com/p/0aca07c31… Rholder. Making. IO/guava – retry… www.jianshu.com/p/a289dde63… Juejin. Cn/post / 701210…