directory
- Retry usage scenarios
- How to elegantly design a retry implementation
- Guava-retrying basic usage
- Guava-retrying implementation principle
- Guava-retrying advanced usage
- Problems encountered in use
- Guava version conflict
- Dynamically adjust the retry policy
Retry usage scenarios
In many business scenarios, the retry mechanism is necessary to eliminate various instability factors and logical errors in the system and ensure the maximum probability of obtaining the desired result.
Especially when calling remote services, in high concurrency scenarios, we may not get the results we want, or not get the response at all, because of server response delays or network problems. At this point, an elegant retry call mechanism gives us a higher probability of getting the expected response.
Normally, we retry through a scheduled task. For example, if an operation fails, the system records it. When the scheduled task starts again, the system saves the data to the scheduled task method and runs the task again. Finally, until you get the result you want.
The disadvantage of both the retry mechanism based on scheduled tasks and the simple retries we wrote ourselves is that the retry mechanism is too simple and not elegant to implement.
How to elegantly design a retry implementation
A complete retry implementation would solve the following problems:
- Retry under what conditions
- Under what conditions stop
- How to Stop retry
- How long to wait for a stop retry
- How to wait
- Request time limit
- How to end
- How do I listen to the entire retry process
And, for better encapsulation, the implementation of retry is generally divided into two steps:
- Construct the retries using the factory pattern
- Execute the retry method and get the result
A complete retry process can be simply represented as:
Guava-retrying basic usage
Guava-retrying is an implementation of the retry mechanism based on Google’s core library guava, which can be described as a retry tool.
Here’s a quick look at how it’s used.
1. The Maven configurations
<! -- https://mvnrepository.com/artifact/com.github.rholder/guava-retrying -->
<dependency>
<groupId>com.github.rholder</groupId>
<artifactId>guava-retrying</artifactId>
<version>2.0.0</version>
</dependency>
Copy the codeIt is important to note that this release relies on the 27.0.1 version of Guava. If you have a few lower versions of Guava in your project, that’s fine, but too low and it’s not compatible. At this point you will need to update the Guava version of your project or simply remove your own Guava dependencies and use the guava dependencies passed over from Guava-Retrying.
2. Implement Callable
Callable<Boolean> callable = new Callable<Boolean>() {
public Boolean call(a) throws Exception {
return true; // do something useful here}};Copy the codeThe call method of Callable is your own actual business call.
- Construct Retryer with RetryerBuilder
Retryer<Boolean> retryer = RetryerBuilder.<Boolean>newBuilder()
.retryIfResult(Predicates.<Boolean>isNull())
.retryIfExceptionOfType(IOException.class)
.retryIfRuntimeException()
.withStopStrategy(StopStrategies.stopAfterAttempt(3))
.build();
Copy the code- Use retries to perform your business
retryer.call(callable);
Copy the codeBelow is the complete reference implementation.
public Boolean test(a) throws Exception {
// Define the retry mechanism
Retryer<Boolean> retryer = RetryerBuilder.<Boolean>newBuilder()
//retryIf Retry condition
.retryIfException()
.retryIfRuntimeException()
.retryIfExceptionOfType(Exception.class)
.retryIfException(Predicates.equalTo(new Exception()))
.retryIfResult(Predicates.equalTo(false))
// Wait policy: each request interval is 1s
.withWaitStrategy(WaitStrategies.fixedWait(1, TimeUnit.SECONDS))
// Stop policy: try the request 6 times
.withStopStrategy(StopStrategies.stopAfterAttempt(6))
TimeLimiter = new SimpleTimeLimiter(); TimeLimiter = new SimpleTimeLimiter();
.withAttemptTimeLimiter(AttemptTimeLimiters.fixedTimeLimit(2, TimeUnit.SECONDS))
.build();
// Define the request implementation
Callable<Boolean> callable = new Callable<Boolean>() {
int times = 1;
@Override
public Boolean call(a) throws Exception {
log.info("call times={}", times);
times++;
if (times == 2) {
throw new NullPointerException();
} else if (times == 3) {
throw new Exception();
} else if (times == 4) {
throw new RuntimeException();
} else if (times == 5) {
return false;
} else {
return true; }}};// Invoke the request with the retry
return retryer.call(callable);
}
Copy the codeGuava-retrying implementation principle
At the core of Guava-Retrying are the Attempt class, the Retryer class, and some Strategy-related classes.
- Attempt
Attempt is both a retry request and the result of the request. It records the number of times the request was made, whether it contains an exception, and the return value of the request.
/**
* An attempt of a call, which resulted either in a result returned by the call,
* or in a Throwable thrown by the call.
*
* @param <V> The type returned by the wrapped callable.
* @author JB
*/
public interface Attempt<V>
Copy the code- Retryer
Retryer is constructed through the RetryerBuilder factory class. The RetryerBuilder is responsible for assigning the defined retry policy to the Retryer object.
These retry policies are used individually when Retryer executes the call method.
Here’s a look at the Retryer call method’s implementation.
/**
* Executes the given callable. If the rejection predicate
* accepts the attempt, the stop strategy is used to decide if a new attempt
* must be made. Then the wait strategy is used to decide how much time to sleep
* and a new attempt is made.
*
* @param callable the callable task to be executed
* @return the computed result of the given callable
* @throws ExecutionException if the given callable throws an exception, and the
* rejection predicate considers the attempt as successful. The original exception
* is wrapped into an ExecutionException.
* @throws RetryException if all the attempts failed before the stop strategy decided
* to abort, or the thread was interrupted. Note that if the thread is interrupted,
* this exception is thrown and the thread's interrupt status is set.
*/
public V call(Callable<V> callable) throws ExecutionException, RetryException {
long startTime = System.nanoTime();
// Description: Loop based on attemptNumber -- that is, how many retries
for (int attemptNumber = 1; ; attemptNumber++) {
// Note: The entry method is executed immediately without waiting
Attempt<V> attempt;
try {
// Note: Perform a specific service in callable
//attemptTimeLimiter limits every attempt to wait often
V result = attemptTimeLimiter.call(callable);
Construct a new attempt using the result of the call
attempt = new ResultAttempt<V>(result, attemptNumber, TimeUnit.NANOSECONDS.toMillis(System.nanoTime() - startTime));
} catch (Throwable t) {
attempt = new ExceptionAttempt<V>(t, attemptNumber, TimeUnit.NANOSECONDS.toMillis(System.nanoTime() - startTime));
}
// Description: Iterate over custom listeners
for (RetryListener listener : listeners) {
listener.onRetry(attempt);
}
// Note: Determine whether the retry conditions are met to continue waiting and retry
if(! rejectionPredicate.apply(attempt)) {return attempt.get();
}
// Note: The stop policy is satisfied at this point, because the desired result has not been obtained, so an exception is thrown
if (stopStrategy.shouldStop(attempt)) {
throw new RetryException(attemptNumber, attempt);
} else {
// Instructions: Execute the default stop strategy -- thread sleep
long sleepTime = waitStrategy.computeSleepTime(attempt);
try {
// Note: You can also execute the defined stop policy
blockStrategy.block(sleepTime);
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
throw newRetryException(attemptNumber, attempt); }}}}Copy the codeRetryer is executed as follows.
Guava-retrying advanced usage
Based on the implementation principle of Guava-Retrying, you can determine your own retry strategy based on the actual business.
The following uses data synchronization as an example to customize a retry policy.
The following implementation is based on Spring Boot 2.1.2.RELEASE.
And use Lombok to simplify beans.
<dependency>
<groupId>org.projectlombok</groupId>
<artifactId>lombok</artifactId>
<optional>true</optional>
</dependency>
Copy the codeBusiness description
When the item is created, the price of the item needs to be set separately. Since two operations are being performed by two people, the problem is that the item is not created, but the price data is already built. In this case, the price data needs to wait until the item is properly created to continue the synchronization.
We create the item through an HTTP request and modify the price of the item through a timer.
When the item does not exist, or the quantity of the item is less than 1, the price of the item cannot be set. The price of the commodity can be set successfully only when the commodity is successfully created and the quantity is greater than 0.
The implementation process
- Custom retry blocking policy
The default blocking strategy is thread sleep, which is implemented using a spin lock and does not block the thread.
package net.ijiangtao.tech.framework.spring.ispringboot.demo.retryer.guava.strategy;
import com.github.rholder.retry.BlockStrategy;
import lombok.NoArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import java.time.Duration;
import java.time.LocalDateTime;
/** * The implementation of the spin lock does not respond to thread interrupts */
@Slf4j
@NoArgsConstructor
public class SpinBlockStrategy implements BlockStrategy {
@Override
public void block(long sleepTime) throws InterruptedException {
LocalDateTime startTime = LocalDateTime.now();
long start = System.currentTimeMillis();
long end = start;
log.info("[SpinBlockStrategy]... begin wait.");
while (end - start <= sleepTime) {
end = System.currentTimeMillis();
}
// Use the Duration new in Java8 to calculate the interval
Duration duration = Duration.between(startTime, LocalDateTime.now());
log.info("[SpinBlockStrategy]... end wait.duration={}", duration.toMillis()); }}Copy the code- Custom retry listener
RetryListener monitors multiple retry processes and can use Attempt to do additional things.
package net.ijiangtao.tech.framework.spring.ispringboot.demo.retryer.guava.listener;
import com.github.rholder.retry.Attempt;
import com.github.rholder.retry.RetryListener;
import lombok.extern.slf4j.Slf4j;
@Slf4j
public class RetryLogListener implements RetryListener {
@Override
public <V> void onRetry(Attempt<V> attempt) {
// The number of retries,(note: the first retry is actually the first call)
log.info("retry time : [{}]", attempt.getAttemptNumber());
// The delay from the first retry
log.info("retry delay : [{}]", attempt.getDelaySinceFirstAttempt());
// Retry result: Abnormal termination or normal return
log.info("hasException={}", attempt.hasException());
log.info("hasResult={}", attempt.hasResult());
// What causes the exception
if (attempt.hasException()) {
log.info("causeBy={}" , attempt.getExceptionCause().toString());
} else {
// The result of a normal return
log.info("result={}" , attempt.getResult());
}
log.info("log listen over."); }}Copy the code- The custom Exception
Some exceptions need to be retried, and some do not.
package net.ijiangtao.tech.framework.spring.ispringboot.demo.retryer.guava.exception;
/** * When this exception is thrown, a retry is required */
public class NeedRetryException extends Exception {
public NeedRetryException(String message) {
super("NeedRetryException can retry."+message); }}Copy the code- The interface between the retry service and Callable is realized
Use the Call method to invoke your own business.
package net.ijiangtao.tech.framework.spring.ispringboot.demo.retryer.guava.model;
import lombok.AllArgsConstructor;
import lombok.Data;
import java.math.BigDecimal;
/** * product model */
@Data
@AllArgsConstructor
public class Product {
private Long id;
private String name;
private Integer count;
private BigDecimal price;
}
Copy the codepackage net.ijiangtao.tech.framework.spring.ispringboot.demo.retryer.guava.repository;
import net.ijiangtao.tech.framework.spring.ispringboot.demo.retryer.guava.model.Product;
import org.springframework.stereotype.Repository;
import java.math.BigDecimal;
import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.atomic.AtomicLong;
/** * DAO */
@Repository
public class ProductRepository {
private static ConcurrentHashMap<Long,Product> products=new ConcurrentHashMap();
private static AtomicLong ids=new AtomicLong(0);
public List<Product> findAll(a){
return new ArrayList<>(products.values());
}
public Product findById(Long id){
return products.get(id);
}
public Product updatePrice(Long id, BigDecimal price){
Product p=products.get(id);
if (null==p){
return p;
}
p.setPrice(price);
return p;
}
public Product addProduct(Product product){
Long id=ids.addAndGet(1);
product.setId(id);
products.put(id,product);
returnproduct; }}Copy the codepackage net.ijiangtao.tech.framework.spring.ispringboot.demo.retryer.guava.service;
import lombok.extern.slf4j.Slf4j;
import net.ijiangtao.tech.framework.spring.ispringboot.demo.retryer.guava.exception.NeedRetryException;
import net.ijiangtao.tech.framework.spring.ispringboot.demo.retryer.guava.model.Product;
import net.ijiangtao.tech.framework.spring.ispringboot.demo.retryer.guava.repository.ProductRepository;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Component;
import java.math.BigDecimal;
import java.util.HashMap;
import java.util.Map;
import java.util.concurrent.Callable;
/** * business method implementation */
@Component
@Slf4j
public class ProductInformationHander implements Callable<Boolean> {
@Autowired
private ProductRepository pRepo;
private static Map<Long, BigDecimal> prices = new HashMap<>();
static {
prices.put(1L.new BigDecimal(100));
prices.put(2L.new BigDecimal(200));
prices.put(3L.new BigDecimal(300));
prices.put(4L.new BigDecimal(400));
prices.put(8L.new BigDecimal(800));
prices.put(9L.new BigDecimal(900));
}
@Override
public Boolean call(a) throws Exception {
log.info("sync price begin,prices size={}", prices.size());
for (Long id : prices.keySet()) {
Product product = pRepo.findById(id);
if (null == product) {
throw new NeedRetryException("can not find product by id=" + id);
}
if (null == product.getCount() || product.getCount() < 1) {
throw new NeedRetryException("product count is less than 1, id=" + id);
}
Product updatedP = pRepo.updatePrice(id, prices.get(id));
if (null == updatedP) {
return false;
}
prices.remove(id);
}
log.info("sync price over,prices size={}", prices.size());
return true; }}Copy the code- Construct Retryer
Take the above implementation as a parameter and construct Retryer.
package net.ijiangtao.tech.framework.spring.ispringboot.demo.retryer.guava.service;
import com.github.rholder.retry.*;
import net.ijiangtao.tech.framework.spring.ispringboot.demo.retryer.guava.exception.NeedRetryException;
import net.ijiangtao.tech.framework.spring.ispringboot.demo.retryer.guava.listener.RetryLogListener;
import net.ijiangtao.tech.framework.spring.ispringboot.demo.retryer.guava.strategy.SpinBlockStrategy;
import org.springframework.stereotype.Component;
import java.util.concurrent.TimeUnit;
/** * Construct retries */
@Component
public class ProductRetryerBuilder {
public Retryer build(a) {
// Define the retry mechanism
Retryer<Boolean> retryer = RetryerBuilder.<Boolean>newBuilder()
//retryIf Retry condition
//.retryIfException()
//.retryIfRuntimeException()
//.retryIfExceptionOfType(Exception.class)
//.retryIfException(Predicates.equalTo(new Exception()))
//.retryIfResult(Predicates.equalTo(false))
.retryIfExceptionOfType(NeedRetryException.class)
// Wait policy: each request interval is 1s
.withWaitStrategy(WaitStrategies.fixedWait(1, TimeUnit.SECONDS))
// Stop strategy: try the request 3 times
.withStopStrategy(StopStrategies.stopAfterAttempt(3))
TimeLimiter = new SimpleTimeLimiter(); TimeLimiter = new SimpleTimeLimiter();
.withAttemptTimeLimiter(AttemptTimeLimiters.fixedTimeLimit(2, TimeUnit.SECONDS))
// The default blocking strategy is thread sleep
//.withBlockStrategy(BlockStrategies.threadSleepStrategy())
// Custom blocking policy: spin lock
.withBlockStrategy(new SpinBlockStrategy())
// Customize the retry listener
.withRetryListener(new RetryLogListener())
.build();
returnretryer; }}Copy the code- Retryer is executed in combination with scheduled tasks
A scheduled task only needs to run once, but virtually all retry strategies are implemented. This greatly simplifies timer design.
Start by using @enablesCheduling to declare project support for timer annotations.
@SpringBootApplication
@EnableScheduling
public class DemoRetryerApplication {
public static void main(String[] args) { SpringApplication.run(DemoRetryerApplication.class, args); }}Copy the codepackage net.ijiangtao.tech.framework.spring.ispringboot.demo.retryer.guava.task;
import com.github.rholder.retry.Retryer;
import net.ijiangtao.tech.framework.spring.ispringboot.demo.retryer.guava.service.ProductInformationHander;
import net.ijiangtao.tech.framework.spring.ispringboot.demo.retryer.guava.service.ProductRetryerBuilder;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.scheduling.annotation.Scheduled;
import org.springframework.stereotype.Component;
/** * Commodity information timer */
@Component
public class ProductScheduledTasks {
@Autowired
private ProductRetryerBuilder builder;
@Autowired
private ProductInformationHander hander;
/** * Synchronizes commodity price timing task *@Scheduled(fixedDelay = 30000) : */ is executed 30 seconds after the last time
@Scheduled(fixedDelay = 30*1000)
public void syncPrice(a) throws Exception{ Retryer retryer=builder.build(); retryer.call(hander); }}Copy the codeResult: Since there is no item, retry and throw an exception.
2019- February -28 14:37:52.667 INFO [scheduling-1] n.i.t.f.s.i.d.r.g.l.RetryLogListener - log listen over.
2019- February -28 14:37:52.672 ERROR [scheduling-1] o.s.s.s.TaskUtils$LoggingErrorHandler - Unexpected error occurred in scheduled task.
com.github.rholder.retry.RetryException: Retrying failed to complete successfully after 3 attempts.
at com.github.rholder.retry.Retryer.call(Retryer.java:174)
Copy the codeYou can also add some item data to see the effect of a successful retry.
The full sample code is here.
Problems encountered in use
Guava version conflict
Because the version of Guava relied on in the project was too low, the following exception occurred when starting the project.
java.lang.NoSuchMethodError: com.google.common.util.concurrent.MoreExecutors.sameThreadExecutor()Lcom/google/common/util/concurrent/ListeningExecutor Service; at org.apache.curator.framework.listen.ListenerContainer.addListener(ListenerContainer.java:41)
at com.bzn.curator.ZkOperator.getZkClient(ZkOperator.java:207)
at com.bzn.curator.ZkOperator.checkExists(ZkOperator.java:346)
at com.bzn.curator.watcher.AbstractWatcher.initListen(AbstractWatcher.java:87)
at com.bzn.web.listener.NebulaSystemInitListener.initZkWatcher(NebulaSystemInitListener.java:84)
at com.bzn.web.listener.NebulaSystemInitListener.contextInitialized(NebulaSystemInitListener.java:33)
at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4939)
at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5434)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1559)
at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1549)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Copy the codeTherefore, exclude guava dependencies from the lower versions of the project.
<exclusion>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
</exclusion>
Copy the codeAlso, since Guava removed the Sametime Adexecutor method in the new release, but this method is currently required by ZK in the project, the appropriate Guava version needs to be manually set up.
Sure enough, the method used by MoreExecutors still exists in version 19.0, but is marked as expired.
@Deprecated
@GwtIncompatible("TODO")
public static ListeningExecutorService sameThreadExecutor(a) {
return new DirectExecutorService();
}
Copy the codeDeclare dependency guava version 19.0.
<! -- https://mvnrepository.com/artifact/com.google.guava/guava -->
<dependency>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
<version>19.0</version>
</dependency>
Copy the codeDynamically adjust the retry policy
In practice, retry policies, such as retry times and waiting time, need to be adjusted. Therefore, the configuration of retry policies can be parameterized and dynamically adjusted.
For example, the waiting time and retry times are increased during the second kill and Double 11 shopping festival to ensure peak error requests. In normal times, you can reduce the waiting time and retry times.
For system-critical services, if multiple retry steps succeed, you can use RetryListener to monitor and alarm.
A reference implementation for “dynamically adjusting retry policies” is provided below:
import com.github.rholder.retry.Attempt;
import com.github.rholder.retry.WaitStrategy;
/** * User-defined waiting policy: Dynamically adjusts the waiting time according to the retry times. The interval for the first request is 1s, the interval for the second request is 10s, and the interval for the third and subsequent requests is 20s. * * * put the wait strategy into effect withWaitStrategy when Retryer is created. * * RetryerBuilder.
newBuilder() *.withWaitStrategy(new AlipayWaitStrategy()) * * Similar effects can be achieved by customizing BlockStrategy So you can write it down. * * /
public class AlipayWaitStrategy implements WaitStrategy {
@Override
public long computeSleepTime(Attempt failedAttempt) {
long number = failedAttempt.getAttemptNumber();
if (number==1) {return 1*1000;
}
if (number==2) {return 10*1000;
}
return 20*1000; }}Copy the code