Automated test case failure reruns help improve the stability of automated test cases. Let’s take a look at the python and Java ecology.

How to do

If you’re using PyTest as a test driver in the Python ecosystem, you can use the PyTest-rerunfailures plug-in in two ways. One is to use the command line to specify pytest –reruns 2 –reruns-delay 1. Reruns indicates the number of repeated runs and reruns-delay indicates the delay of repeated runs. Another way is to use @pytest.mark.flaky(reruns=2, reruns_delay=1). This is generally used. You don’t want to rerun all test cases globally, but only specific test cases, so use this flag on specific test methods. In the Java ecosystem, where junit is the test driver, junit5 provides the @repeattest (2) annotation that allows you to test a class or method repeatedly, or can be customized to control the test case by implementing a TestRule interface.

public class MyTestClass {

    @Rule
    public RepeatRule repeatRule = new RepeatRule();

    @Test
    @Repeat(10)
    public void testMyCode() {
        //your test code goes here
    }
}

import static java.lang.annotation.ElementType.ANNOTATION_TYPE;
import static java.lang.annotation.ElementType.METHOD;
import java.lang.annotation.Retention;
import java.lang.annotation.RetentionPolicy;
import java.lang.annotation.Target;

@Retention( RetentionPolicy.RUNTIME )
@Target({ METHOD, ANNOTATION_TYPE })
public @interface Repeat {
    int value() default 1;
}

import org.junit.rules.TestRule;
import org.junit.runner.Description;
import org.junit.runners.model.Statement;

public class RepeatRule implements TestRule {

    private static class RepeatStatement extends Statement {
        private final Statement statement;
        private final int repeat;    

        public RepeatStatement(Statement statement, int repeat) {
            this.statement = statement;
            this.repeat = repeat;
        }

        @Override
        public void evaluate() throws Throwable {
            for (int i = 0; i < repeat; i++) {
                statement.evaluate();
            }
        }

    }

    @Override
    public Statement apply(Statement statement, Description description) {
        Statement result = statement;
        Repeat repeat = description.getAnnotation(Repeat.class);
        if (repeat != null) {
            int times = repeat.value();
            result = new RepeatStatement(statement, times);
        }
        return result;
    }
}Copy the code

You can also add a rerunFailingTestsCount parameter if you use Maven, but this controls all use cases.

Why should we let the failed use cases run again? Because automation is usually in the test environment or other off-line environment, the instability of the environment may cause the test cases to fail inexplicably, which greatly reduces the stability of the use cases. At this time, the failure replay mechanism can improve the stability of test cases to a certain extent and make more output.

What automated use cases should fail rerun

Interface test automation to suggest you can join this failure again, for the UI interface interface automation, failure to run, feel little sense, because often when the failure of the case, either due to interface elements didn’t load out, or is a use case is a problem with the logic, or accidental popup window, this time should let the errors thrown out as soon as possible, It’s better to fix it as soon as possible, rather than trying again where it doesn’t work. UI automation should be done with explicit and implicit waiting.

What failed use cases should be rerun

In the testing framework, it is best to distinguish what kind of exception is the service exception and what is the exception of the testing framework itself, for the service exception can be retried appropriately, for the framework exception does not run again, directly thrown. Assert failure certainly does not require a rerun. So in the control of test case execution, do not run all at once, selective, to ensure stability, but also to ensure efficiency, let automation play value.

conclusion

The value of automated testing is that it can quickly check the system. If the running time is multiplied by retries, there is no point. It is better to throw errors and fix them as soon as possible. In addition, the running sequence of automated test cases should also be controlled. Interfaces in front of the business should run first, and interfaces in the rear of the business should run after as much as possible. Landing interface and order interface, for example, belong to the business of the landing interface, after the order is to rely on, usually at the time of test order interface to initialize the login status, this time invokes the login interface, at the time of test cases to perform batch, can let landing interface test cases to run, if the interface has a problem, Then all the other use cases that need to work with the login interface will fail, and the subsequent use cases will not have to run, which will save a lot of time.

You are welcome to check out my blog, where there is more about the actual test content oh!!