Taste in this article: Chilled bayberry Expected reading: 20 minutes
instructions
Company engaged in activities recently, need to rely on a third-party interface, no abnormalities in the testing phase, but found that rely on interface after online sometimes returns a system exception due to internal error, although the probability is not big, but always alarm always bad because of this, and the news of the dead-letter queue also need to trouble operations to deliver, So adding a retry mechanism is imperative.
Retry mechanism A protection mechanism that reduces the impact of network fluctuations and transient unavailability of dependent services on the system to ensure stable system operation. It makes your already stable system even more stable.
For the sake of illustration, suppose we want to retry as follows:
@Slf4j
@Component
public class HelloService {
private static AtomicLong helloTimes = new AtomicLong();
public String hello(a){
long times = helloTimes.incrementAndGet();
if (times % 4! =0){
log.warn(Exception occurred, time: {}, LocalTime.now() );
throw new HelloRetryException("Hello exception occurred");
}
return "hello"; }}Copy the codeCall:
@Slf4j
@Service
public class HelloRetryService implements IHelloService{
@Autowired
private HelloService helloService;
public String hello(a){
returnhelloService.hello(); }}Copy the codeIn other words, this interface will succeed only once in every four calls.
Manual try again
Let’s start with the simplest way, which is to retry the call directly:
// Manually retry
public String hello(a){
int maxRetryTimes = 4;
String s = "";
for (int retry = 1; retry <= maxRetryTimes; retry++) {
try {
s = helloService.hello();
log.info("HelloService returns: {}", s);
return s;
} catch (HelloRetryException e) {
log.info("Helloservice.hello () call failed, ready to retry"); }}throw new HelloRetryException("Retry times exhausted");
}
Copy the codeThe output is as follows:
Error: time: 10:17:21.079413300 HelloService.hello () call failed, ready to retry Error: time: 10:17:21.079413300 10:17:21.085861800 HelloService.hello () call failed, ready to retry error, time: 10:17:21.085861800 HelloService.hello () failed, ready to retry helloService Returns: Hello service.helloRetry() :helloCopy the codeThe program retries four times in a very short time and returns successfully.
This seems to solve the problem, but in practice, because there is no retry interval, it is likely that the dependent service has not yet recovered from the network exception, so it is highly likely that the next few calls will fail.
Also, this requires a lot of intrusive changes to the code, which is obviously not elegant.
The proxy pattern
Because the above processing method needs to modify a lot of business code, although it has realized the function, it is too intrusive to the original code and has poor maintainability.
So you need to take a more elegant approach and not modify the business code directly, so how do you do that?
It’s as simple as simply wrapping another layer around the business code, where the proxy pattern comes in handy.
@Slf4j
public class HelloRetryProxyService implements IHelloService{
@Autowired
private HelloRetryService helloRetryService;
@Override
public String hello(a) {
int maxRetryTimes = 4;
String s = "";
for (int retry = 1; retry <= maxRetryTimes; retry++) {
try {
s = helloRetryService.hello();
log.info("HelloRetryService returns: {}", s);
return s;
} catch (HelloRetryException e) {
log.info("Helloretryservice.hello () call failed, ready to retry"); }}throw new HelloRetryException("Retry times exhausted"); }}Copy the codeIn this way, the retry logic is completed by the proxy class, and the logic of the original business class does not need to be modified. If you want to modify the retry logic in the future, you only need to modify this class, and the division of labor is clear. For example, if you now want to add a delay between retries, you only need to make a few changes:
@Override
public String hello(a) {
int maxRetryTimes = 4;
String s = "";
for (int retry = 1; retry <= maxRetryTimes; retry++) {
try {
s = helloRetryService.hello();
log.info("HelloRetryService returns: {}", s);
return s;
} catch (HelloRetryException e) {
log.info("Helloretryservice.hello () call failed, ready to retry");
}
// Delay one second
try {
Thread.sleep(1000);
} catch(InterruptedException e) { e.printStackTrace(); }}throw new HelloRetryException("Retry times exhausted");
}
Copy the codeThe proxy model is more elegant, but if you rely on many services, creating a proxy class for each service is too cumbersome, and the retry logic is pretty much the same, just the number of retries and the latency. If every class wrote a long list of similar code, obviously, not elegant!
JDK dynamic proxy
At this point, dynamic proxies shine. Just write a proxy handler class to start a dog and get to ninety-nine.
@Slf4j
public class RetryInvocationHandler implements InvocationHandler {
private final Object subject;
public RetryInvocationHandler(Object subject) {
this.subject = subject;
}
@Override
public Object invoke(Object proxy, Method method, Object[] args) throws Throwable {
int times = 0;
while (times < RetryConstant.MAX_TIMES) {
try {
return method.invoke(subject, args);
} catch (Exception e) {
times++;
log.info("times:{},time:{}", times, LocalTime.now());
if (times >= RetryConstant.MAX_TIMES) {
throw newRuntimeException(e); }}// Delay one second
try {
Thread.sleep(1000);
} catch(InterruptedException e) { e.printStackTrace(); }}return null;
}
/** * get the dynamic proxy **@paramRealSubject Proxy object */
public static Object getProxy(Object realSubject) {
InvocationHandler handler = new RetryInvocationHandler(realSubject);
returnProxy.newProxyInstance(handler.getClass().getClassLoader(), realSubject.getClass().getInterfaces(), handler); }}Copy the codeUnit test:
@Test
public void helloDynamicProxy(a) {
IHelloService realService = new HelloService();
IHelloService proxyService = (IHelloService)RetryInvocationHandler.getProxy(realService);
String hello = proxyService.hello();
log.info("hello:{}", hello);
}
Copy the codeThe output is as follows:
Hello times: 1 an exception occurs, time: 11:22:20. 727586700 times: 1, time: 11:22:20. Hello 728083 times: 2 an exception occurs, the time: 11:22:21. 728858700 times: 2, time: 11:22:21. Hello 729343700 times: 3 an exception occurs, the time: 11:22:22. 729706600 times: 3, time: 11:22:22. Hello 729706600 times: hello: helloCopy the codeAfter four retries, Hello is printed, as expected.
Dynamic proxies can put all the retry logic together, which is obviously much more convenient and elegant than using proxy classes directly.
But don’t happy too early, because here is the agent of the HelloService class is a simple, not rely on other classes, so there is no question of created directly, but if the proxy class depends on the other by the Spring container management class, this way will throw an exception, because do not have to be dependent on the instance of the agent into creating instances.
In this case, it’s a bit more complicated. You need to take the already assembled instance that needs to be proxied from the Spring container, create the proxy class instance for it, and hand it over to the Spring container to manage, so that you don’t have to create a new proxy class instance every time.
Without further ado, the sleeves are rolled up.
Create a new utility class to get the proxy instance:
@Component
public class RetryProxyHandler {
@Autowired
private ConfigurableApplicationContext context;
public Object getProxy(Class clazz) {
// 1. Get the object from the Bean
DefaultListableBeanFactory beanFactory = (DefaultListableBeanFactory)context.getAutowireCapableBeanFactory();
Map<String, Object> beans = beanFactory.getBeansOfType(clazz);
Set<Map.Entry<String, Object>> entries = beans.entrySet();
if (entries.size() <= 0) {throw new ProxyBeanNotFoundException();
}
// If there are multiple candidate beans, determine if there are any proxy beans
Object bean = null;
if (entries.size() > 1) {for (Map.Entry<String, Object> entry : entries) {
if(entry.getKey().contains(PROXY_BEAN_SUFFIX)){ bean = entry.getValue(); }};if(bean ! =null) {return bean;
}
throw new ProxyBeanNotSingleException();
}
Object source = beans.entrySet().iterator().next().getValue();
Object source = beans.entrySet().iterator().next().getValue();
// 2. Check whether the proxy object for this object exists
String proxyBeanName = clazz.getSimpleName() + PROXY_BEAN_SUFFIX;
Boolean exist = beanFactory.containsBean(proxyBeanName);
if (exist) {
bean = beanFactory.getBean(proxyBeanName);
return bean;
}
// 3. Generate a proxy object if it does not exist
bean = RetryInvocationHandler.getProxy(source);
// 4. Inject the bean into the Spring container
beanFactory.registerSingleton(proxyBeanName, bean);
returnbean; }}Copy the codeJDK dynamic proxy is used:
@Slf4j
public class RetryInvocationHandler implements InvocationHandler {
private final Object subject;
public RetryInvocationHandler(Object subject) {
this.subject = subject;
}
@Override
public Object invoke(Object proxy, Method method, Object[] args) throws Throwable {
int times = 0;
while (times < RetryConstant.MAX_TIMES) {
try {
return method.invoke(subject, args);
} catch (Exception e) {
times++;
log.info("retry times:{},time:{}", times, LocalTime.now());
if (times >= RetryConstant.MAX_TIMES) {
throw newRuntimeException(e); }}// Delay one second
try {
Thread.sleep(1000);
} catch(InterruptedException e) { e.printStackTrace(); }}return null;
}
/** * get the dynamic proxy **@paramRealSubject Proxy object */
public static Object getProxy(Object realSubject) {
InvocationHandler handler = new RetryInvocationHandler(realSubject);
returnProxy.newProxyInstance(handler.getClass().getClassLoader(), realSubject.getClass().getInterfaces(), handler); }}Copy the codeNow that the main code is complete, modify the HelloService class and add a dependency:
@Slf4j
@Component
public class HelloService implements IHelloService{
private static AtomicLong helloTimes = new AtomicLong();
@Autowired
private NameService nameService;
public String hello(a){
long times = helloTimes.incrementAndGet();
log.info("hello times:{}", times);
if (times % 4! =0){
log.warn(Exception occurred, time: {}, LocalTime.now() );
throw new HelloRetryException("Hello exception occurred");
}
return "hello "+ nameService.getName(); }}Copy the codeThe NameService is actually quite simple and was created solely to test whether the dependency injection Bean works properly.
@Service
public class NameService {
public String getName(a){
return "Frank"; }}Copy the codeLet’s test it out:
@Test
public void helloJdkProxy(a) throws InvocationTargetException, NoSuchMethodException, InstantiationException, IllegalAccessException {
IHelloService proxy = (IHelloService) retryProxyHandler.getProxy(HelloService.class);
String hello = proxy.hello();
log.info("hello:{}", hello);
}
Copy the codeHello times: 1 an exception occurs, time: 14:40:27. Retry 540672200 times: 1, time: 14:40:27. Hello 541167400 times: 2 an exception occurs, the time: 14:40:28. 541584600 retry times: 2, time: 14:40:28. Hello 542033500 times: 3 an exception occurs, the time: 14:40:29. 542161500 retry times: 3, time: 14:40:29. Hello 542161500 times: hello: hello, FrankCopy the codePerfect, so you don’t have to worry about dependency injection because the Bean objects you get from the Spring container are already injected and configured. Of course, this is only for Singleton beans. You can make it a little more complete by determining whether the Bean type in the container is Singleton or Prototype. If Singleton is Singleton, do as above, and if Prototype, create a new proxy class object each time.
In addition, the JDK dynamic proxy is used here, so there is a natural disadvantage. If the class you want to proide does not implement any interface, you cannot create a proxy object for it, and this approach will not work.
CGLib dynamic proxy
Now that we’ve talked about JDK dynamic proxies, we have to mention CGLib dynamic proxies. Using JDK dynamic proxies has requirements on propped classes. Not all classes can be propped, and CGLib dynamic proxies solve this problem.
Create a CGLib dynamic proxy class:
@Slf4j
public class CGLibRetryProxyHandler implements MethodInterceptor {
private Object target;// Target object to be proxied
// Override the interceptor method
@Override
public Object intercept(Object obj, Method method, Object[] arr, MethodProxy proxy) throws Throwable {
int times = 0;
while (times < RetryConstant.MAX_TIMES) {
try {
return method.invoke(target, arr);
} catch (Exception e) {
times++;
log.info("cglib retry :{},time:{}", times, LocalTime.now());
if (times >= RetryConstant.MAX_TIMES) {
throw newRuntimeException(e); }}// Delay one second
try {
Thread.sleep(1000);
} catch(InterruptedException e) { e.printStackTrace(); }}return null;
}
// Define a method to get a proxy object
public Object getCglibProxy(Object objectTarget){
this.target = objectTarget;
Enhancer enhancer = new Enhancer();
enhancer.setSuperclass(objectTarget.getClass());
enhancer.setCallback(this);
Object result = enhancer.create();
returnresult; }}Copy the codeTo switch to CGLib dynamic proxies, replace these two lines of code:
// 3. Generate a proxy object if it does not exist
// bean = RetryInvocationHandler.getProxy(source);
CGLibRetryProxyHandler proxyHandler = new CGLibRetryProxyHandler();
bean = proxyHandler.getCglibProxy(source);
Copy the codeStart testing:
@Test
public void helloCGLibProxy(a) {
IHelloService proxy = (IHelloService) retryProxyHandler.getProxy(HelloService.class);
String hello = proxy.hello();
log.info("hello:{}", hello);
hello = proxy.hello();
log.info("hello:{}", hello);
}
Copy the codeHello times: 1 an exception occurs, time: 15:06:00. 799679100 additional retry: 1, time: 15:06:00. Hello 800175400 times: 2 an exception occurs, the time: 15:06:01. 800848600 additional retry: 2, time: 15:06:01. Hello 801343100 times: 3 an exception occurs, the time: 15:06:02.802180 cglib retry :3,time:15:06:02.802180 Hello times:4 Hello: Hello Frank Hello times:5 An exception occurs,time: 15:06:03. 803933800 additional retry: 1, time: 15:06:03. Hello 803933800 times: 6 an exception occurs, the time: 15:06:04.804945400 cglib retry :2,time:15:06:04.805442 Hello times:7 An exception occurs. Time: 15:06:05. 806886500 additional retry: 3, time: 15:06:05. Hello 807881300 times: hello: hello, FrankCopy the codeThis is great, a perfect solution to the pitfalls of JDK dynamic proxies. The elegance index has gone up quite a bit.
One problem with this solution, however, is that it requires intrusive changes to the original logic, tweaking it every time a proxy instance is invoked, which still makes a lot of changes to the original code.
Spring AOP
Want to make non-invasive changes to existing logic? Retry if you want an annotation? Can’t Spring AOP do it perfectly? Using AOP to set up the facets for the target invocation, you can add some additional logic before and after the target method invocation.
Create an annotation:
@Target(ElementType.METHOD)
@Retention(RetentionPolicy.RUNTIME)
@Documented
public @interface Retryable {
int retryTimes(a) default 3;
int retryInterval(a) default 1;
}
Copy the codeThere are two parameters: retryTimes indicates the maximum number of retries and retryInterval indicates the retryInterval.
Then comment on the methods that need to be retried:
@Retryable(retryTimes = 4, retryInterval = 2)
public String hello(a){
long times = helloTimes.incrementAndGet();
log.info("hello times:{}", times);
if (times % 4! =0){
log.warn(Exception occurred, time: {}, LocalTime.now() );
throw new HelloRetryException("Hello exception occurred");
}
return "hello " + nameService.getName();
}
Copy the codeNext, for the final step, write the AOP aspect:
@Slf4j
@Aspect
@Component
public class RetryAspect {
@Pointcut("@annotation(com.mfrank.springboot.retry.demo.annotation.Retryable)")
private void retryMethodCall(a){}
@Around("retryMethodCall()")
public Object retry(ProceedingJoinPoint joinPoint) throws InterruptedException {
// Get the number of retries and retry interval
Retryable retry = ((MethodSignature)joinPoint.getSignature()).getMethod().getAnnotation(Retryable.class);
int maxRetryTimes = retry.retryTimes();
int retryInterval = retry.retryInterval();
Throwable error = new RuntimeException();
for (int retryTimes = 1; retryTimes <= maxRetryTimes; retryTimes++){
try {
Object result = joinPoint.proceed();
return result;
} catch (Throwable throwable) {
error = throwable;
log.warn("Abnormal call, retry, retryTimes:{}", retryTimes);
}
Thread.sleep(retryInterval * 1000);
}
throw new RetryExhaustedException("Retry times exhausted", error); }}Copy the codeStart testing:
@Autowired
private HelloService helloService;
@Test
public void helloAOP(a){
String hello = helloService.hello();
log.info("hello:{}", hello);
}
Copy the codeThe output is as follows:
RetryTimes :1 Hello times:2 An exception occurs. Time: 16:49:30.224649800 The call is abnormal and retry is started. RetryTimes :1 Hello times:2 An exception occurs. 16:49:32.225230800 Abnormal call, retry, retryTimes:2 Hello times:3 Abnormal call, time: 16:49:34.225968900 The call is abnormal and retry. RetryTimes :3 Hello times:4 Hello :hello FrankCopy the codeThis is pretty elegant, a single comment can do the retries, just better.
Spring’s retry annotations
In fact, Spring has a pretty good retry mechanism, which is much better than the above section and doesn’t require you to rebuild the wheel yourself.
Let’s see if this wheel works.
Let’s start by introducing the jar packages required for retry:
<dependency>
<groupId>org.springframework.retry</groupId>
<artifactId>spring-retry</artifactId>
</dependency>
<dependency>
<groupId>org.aspectj</groupId>
<artifactId>aspectjweaver</artifactId>
</dependency>
Copy the codeThen add the @enableretry annotation to the launch or configuration classes, and the @retryable annotation to the methods that need to be retried (huh? It’s like my own custom annotation, right? Copying my notes! [Manual funny])
@Retryable
public String hello(a){
long times = helloTimes.incrementAndGet();
log.info("hello times:{}", times);
if (times % 4! =0){
log.warn(Exception occurred, time: {}, LocalTime.now() );
throw new HelloRetryException("Hello exception occurred");
}
return "hello " + nameService.getName();
}
Copy the codeBy default, three retries are performed, with a retry interval of one second. Of course, we can also customize the number of retries and interval. This is exactly what I did before.
But the retry mechanism in Spring also supports many useful features, such as more fine-grained control over retries by specifying that only certain types of exceptions will be retried, so that other types of exceptions will not be retried. The default is null and all exceptions are retried.
@Retryable{value = {HelloRetryException.class}}
public String hello(a){2. }Copy the codeYou can also use include and exclude to specify which exceptions to include or exclude for retries.
Maximum number of retries can be specified at maxAttemps. Default is 3.
You can use the interceptor setting to retry the interceptor bean name.
You can set a unique identifier for this retry through label for statistical output.
You can use exceptionExpression to add an exceptionExpression, which is executed after the exception is thrown to determine whether to retry later.
In addition, the retry mechanism in Spring supports setting the retry compensation mechanism using backoff, setting the retry interval, and setting the retry delay multiple.
Here’s an example:
@Retryable(value = {HelloRetryException.class}, maxAttempts = 5,
backoff = @Backoff(delay = 1000, multiplier = 2))
public String hello(a){... }Copy the codeThe method call will retry at a maximum of five times after the HelloRetryException is thrown, with the first retry interval of 1s and then increments of twice the size, second retry interval of 2s, third retry interval of 4s, and fourth retry interval of 8s.
The retry mechanism also supports the use of the @Recover annotation for cleanup, which is called after a specified number of retries and can be used for logging and other operations.
It is important to note that for the @Recover annotation to take effect, it must be in the same class as the methods that are labeled by @retryable and cannot have a return value.
And if the @RECOVER annotation is used, the original exception will not be thrown if no exception is thrown in the @RECOVER method after the maximum number of retries.
@Recover
public boolean recover(Exception e) {
log.error("Maximum retry times reached",e);
return false;
}
Copy the codeIn addition to using annotations, Spring Retry also supports retries using code directly on invocation:
@Test
public void normalSpringRetry(a) {
// Indicates which exceptions need to be retried. Key indicates the bytecode of the exception. Value true indicates that exceptions need to be retried
Map<Class<? extends Throwable>, Boolean> exceptionMap = new HashMap<>();
exceptionMap.put(HelloRetryException.class, true);
// Build a retry template instance
RetryTemplate retryTemplate = new RetryTemplate();
// Set the retry rollback policy, mainly setting the retry interval
FixedBackOffPolicy backOffPolicy = new FixedBackOffPolicy();
long fixedPeriodTime = 1000L;
backOffPolicy.setBackOffPeriod(fixedPeriodTime);
// Set the retry policy, mainly the retry times
int maxRetryTimes = 3;
SimpleRetryPolicy retryPolicy = new SimpleRetryPolicy(maxRetryTimes, exceptionMap);
retryTemplate.setRetryPolicy(retryPolicy);
retryTemplate.setBackOffPolicy(backOffPolicy);
Boolean execute = retryTemplate.execute(
//RetryCallback
retryContext -> {
String hello = helloService.hello();
log.info("Result of call :{}", hello);
return true;
},
// RecoverCallBack
retryContext -> {
//RecoveryCallback
log.info("Maximum retry times reached");
return false; }); }Copy the codeThe only advantage here is that multiple retry policies can be set:
NeverRetryPolicy: Allows RetryCallback to be called only once. No retries are allowed. AlwaysRetryPolicy: allows unlimited retries until the callback succeeds. Fixed retry policy. The default maximum retry times is3RetryTemplate Specifies the default policy. TimeoutRetryPolicy: Timeout Specifies the retry policy. The default timeout is1Seconds, within a specified timeout allows retry ExceptionClassifierRetryPolicy: set different abnormal retry strategy, similar to retry strategy, the difference is that here only to distinguish the various abnormal retry CircuitBreakerRetryPolicy: The retry policy with the fusing function needs to be set3OpenTimeout, resetTimeout, and delegate CompositeRetryPolicy: There are two types of combination retry policies. The optimistic combination retry policy allows you to retry as long as one policy allows you to retry. The pessimistic combination retry policy allows you to retry as long as one policy does not allow you to retryCopy the codeAs you can see, the retry mechanism in Spring is quite complete and more powerful than the AOP aspect I wrote above.
It is also important to note that because Spring Retry uses Aspect enhancements, there is an inevitable pitfall for Aspect use — method invocations that will expire Retry if the method annotated by @retryable is in the same class as the called.
However, there are some drawbacks. Spring’s retry mechanism only supports catching exceptions, not validating the return values.
Guava Retry
Finally, Guava Retry is another Retry tool.
Compared to Spring Retry, Guava Retry is more flexible and can be retried based on return value verification.
Let’s start with a little chestnut:
Import jar package first:
<dependency>
<groupId>com.github.rholder</groupId>
<artifactId>guava-retrying</artifactId>
<version>2.0.0</version>
</dependency>
Copy the codeThen use a small Demo to get a feel for it:
@Test
public void guavaRetry(a) {
Retryer<String> retryer = RetryerBuilder.<String>newBuilder()
.retryIfExceptionOfType(HelloRetryException.class)
.retryIfResult(StringUtils::isEmpty)
.withWaitStrategy(WaitStrategies.fixedWait(3, TimeUnit.SECONDS))
.withStopStrategy(StopStrategies.stopAfterAttempt(3))
.build();
try {
retryer.call(() -> helloService.hello());
} catch(Exception e){ e.printStackTrace(); }}Copy the codeCreate a Retryer instance and use it to call the method that needs to be retried. You can set up the retry mechanism in many ways, such as using retryIfException to retry all exceptions. Use the retryIfExceptionOfType method to set up retries for specified exceptions, and retryIfResult to retry returns that do not meet expectations, Use the retryIfRuntimeException method to retry all runtimeExceptions.
There are also five methods starting with with to set retry/wait/block policies/single-task execution time limits/custom listeners for more powerful exception handling.
Combined with Spring AOP, you can achieve more powerful Retry capabilities than Spring Retry.
In close comparison, Guava Retry offers the following features:
- You can set the time limit for a single execution of a task. If a timeout occurs, an exception is thrown.
- Retry listeners can be set up to perform additional processing.
- You can set the task blocking policy, that is, you can set what to do in the period between the completion of the current retry and the start of the next retry.
- You can combine the stop retry policy with the wait policy to set up more flexible policies, such as exponential wait times with a maximum of 10 calls, random wait times that never stop, and so on.
conclusion
From the simplest manual retry, to the use of static proxy, and then to JDK dynamic proxy and CGLib dynamic proxy, and then to Spring AOP, are the process of manually made wheels. Finally, two kinds of wheels are introduced. One is Spring Retry, which is easy to use and works naturally with the Spring framework, with one annotation doing everything, and the other is Guava Retry, which is more flexible and powerful without relying on the Spring framework.
In my opinion, the Retry mechanism provided by Spring Retry is powerful enough for most scenarios, and it is good to use Spring Retry without the additional flexibility provided by Guava Retry. Of course, it depends on the situation, but if it is not necessary, it is not encouraged to repeat the wheel, first research someone else’s wheel and then decide whether to do it yourself.
In this paper, to this end, and a day to finish the completed an article, the purpose of writing is to summarize and to share, I believe that the best practices can be summarized and accumulated in most scenarios are applicable, these best practices will be in the process of gradual accumulation, become more important than experience. Because experience is forgotten if it is not summed up, and what is summed up is not lost.
If you have a better idea about retry, welcome to exchange and discuss, also welcome to pay attention to my public number for message exchange.
