introduce

The Go 1.7, testing package introduces a Run method on types T and B that allows the creation of subtests and subbenchmarks. The introduction of sub-tests and sub-benchmarks allows for better failures handling, refined control of tests run from the command line, parallel control, and often makes code simpler and easier to maintain.

Table – driven test

Before going into detail, let’s discuss common ways to write tests in Go. A series of related validations can be achieved by looping through a series of test cases:

1func TestTime(t *testing.T) { 2 testCases := []struct { 3 gmt string 4 loc string 5 want string 6 }{ 7 {"12:31", "Europe/Zuri", "13:31"}, // incorrect location name 8 {"12:31", "America/New_York", "7:31"}, // should be 07:31 9 {"08:08", "Australia/Sydney", "18:08"},10 }11 for _, tc := range testCases {12 loc, err := time.LoadLocation(tc.loc)13 if err ! = nil {14 t.Fatalf("could not load location %q", tc.loc)15 }16 gmt, _ := time.Parse("15:04", tc.gmt)17 if got := gmt.In(loc).Format("15:04"); got ! = tc.want {18 t.Errorf("In(%s, %s) = %s; want %s", tc.gmt, tc.loc, got, tc.want)19 }20 }21}Copy the code

Often referred to as table-driven tests, they reduce the amount of duplicate code and allow you to add more test cases directly than repeating the same code for each test.

Table-driven benchmark

Prior to Go 1.7, it was not possible to benchmark using the same table-driven approach. Benchmarks test the performance of entire functions, so iterative benchmarks only treat them as a benchmark in their entirety.

A common solution is to define separate top-level benchmarks, each calling a common function with different parameters. For example, before 1.7, the benchmark for AppendFloat of the Strconv package looked like this:

1func benchmarkAppendFloat(b *testing.B, f float64, fmt byte, prec, bitSize int) { 2 dst := make([]byte, 30) 3 b.ResetTimer() // Overkill here, but for illustrative purposes. 4 for i := 0; i < b.N; i++ { 5 AppendFloat(dst[:0], f, fmt, prec, bitSize) 6 } 7} 8func BenchmarkAppendFloatDecimal(b *testing.B) { benchmarkAppendFloat(b, 33909, 'g', -1, 64)} 9func BenchmarkAppendFloat(b *testing.B) {BenchmarkAppendFloat(b, 339.7784, 'g', -1, 64)}10func benchmarkAppendFloat(b *testing.B) {benchmarkAppendFloat(b, -5.09e75, 'g', -1, 64)} 11 func BenchmarkAppendFloatNegExp (b * testing. B) {benchmarkAppendFloat (b, 5.11 e-95, 'g', 1, 64) }12func BenchmarkAppendFloatBig(b *testing.B) { benchmarkAppendFloat(b, 123456789123456789123456789, 'g', -1, 64)} 13...Copy the code

Using the Run method provided in Go 1.7, the same set of benchmarks is now represented as a single top-level benchmark:

1func BenchmarkAppendFloat(b *testing.B) { 2 benchmarks := []struct{ 3 name string 4 float float64 5 fmt byte 6 prec int 7 bitSize int 8} {{9 "Decimal", 33909, 'g', 1, 64}, 10 {" Float ", 339.7784, 'g', 1, 64}, {11 "Exp", 5.09 e75, 'g', 1, {64}, 12 "NegExp", 5.11 e-95, 'g', 1, 64}, 13 {" Big ", 123456789123456789123456789, 'g', 1, 64}, 14... 15 }16 dst := make([]byte, 30)17 for _, bm := range benchmarks {18 b.Run(bm.name, func(b *testing.B) {19 for i := 0; i < b.N; i++ {20 AppendFloat(dst[:0], bm.float, bm.fmt, bm.prec, bm.bitSize)21 }22 })23 }24}Copy the code

A separate benchmark is created each time the Run method is called. The benchmark function that calls the Run method is Run only once and does not measure performance.

The new code has more lines of code, but is more maintainable, easier to read, and consistent with the table-driven approach typically used for testing. In addition, the common setup code is now shared between runs without the need to reset the timer.

Table-driven is used for sub-tests

Go 1.7 also introduces a Run method for creating child tests. This test is a rewritten version of our example above using child tests:

1func TestTime(t *testing.T) { 2 testCases := []struct { 3 gmt string 4 loc string 5 want string 6 }{ 7 {"12:31", "Europe/Zuri", "13:31"}, 8 {"12:31", "America/New_York", "7:31"}, 9 {"08:08", "Australia/Sydney", "18:08"},10 }11 for _, tc := range testCases {12 t.Run(fmt.Sprintf("%s in %s", tc.gmt, tc.loc), func(t *testing.T) {13 loc, err := time.LoadLocation(tc.loc)14 if err ! = nil {15 t.Fatal("could not load location")16 }17 gmt, _ := time.Parse("15:04", tc.gmt)18 if got := gmt.In(loc).Format("15:04"); got ! = tc.want {19 t.Errorf("got %s; want %s", got, tc.want)20 }21 })22 }23}Copy the code

The first thing to notice is the output difference between the two implementations. Original implementation print:

1-- FAIL: TestTime (0.00s)2 time_test.go:62: could not load location "Europe/Zuri "3 -- FAIL: TestTime (0.00s)2 time_test.go:62: could not load location "Europe/Zuri" 3Copy the code

Even with two errors, the test stops on the call to Fatal and the second test does not run.

The version that uses Run does both:

1--- FAIL: TestTime (0.00s)2    --- FAIL: TestTime/12:31_in_Europe/Zuri (0.00s)3        time_test.go:84: could not load location4    --- FAIL: TestTime/12:31_in_America/New_York (0.00s)5        time_test.go:88: got 07:31; want 7:31Copy the code

Fatal and its related methods cause child tests to be skipped, but not their parent tests other child tests.

Another point to note is the shorter error messages in the new implementation. Because subtest names can be uniquely identified, they do not need to be identified again in the error message.

There are other benefits to using sub-tests or sub-benchmarks, as described in the following sections.

Run specific tests or benchmarks

You can use the -run or -bench flags to identify child tests or benchmarks on the command line. Both flags take a slash-separated list of regular expressions that match the appropriate part of the full name of a child test or child benchmark.

The full name of a child test or child benchmark is a slash-separated list of names, starting with the top-level name. The name starts with the corresponding function name for the top-level test or benchmark, and the rest is the first argument to Run. To avoid display and parsing problems, names are cleaned up by replacing Spaces with underscores and escaping non-printable characters. The same cleanup rules apply to regular expressions passed to the -run or -bench flags.

Look at some examples:

Run tests using the European time zone:

$go test-run =TestTime/"in Europe"2-- FAIL: TestTime (0.00s)3 -- FAIL: TestTime/12:31_in_Europe/Zuri (0.00s)4 time_test.go:85: could not load locationCopy the code

Go test-run =Time/12:\[0-9\] -v

1$ go test -run=Time/12:[0-9] -v2=== RUN TestTime3=== RUN TestTime/12:31_in_Europe/Zuri4=== RUN TestTime/ 12:31_in_america /New_York5-- FAIL: TestTime (0.00s)6 -- FAIL: TestTime/12:31_in_Europe/Zuri (0.00s)7 time_test.go:85: could not load location8 -- FAIL: TestTime/ 12:31_in_america /New_York (0.00s)9 time_test.go:89: got 07:31; want 7:31Copy the code

Perhaps surprisingly, using -run = TestTime/New_York will not match any tests. This is because slashes present in location names are also treated as delimiters. Use it like this:

1$go test-run =Time//New_York2-- FAIL: TestTime (0.00s)3 -- FAIL: TestTime/ 12:31_in_america /New_York (0.00s)4 time_test.go:88: got 07:31; want 7:31Copy the code

Note that // in the string passed to -run, the/in the time zone name America/New_York is handled as if it were a child test separator. The first regular expression of the pattern (TestTime) matches the top-level test. The second regular expression (empty string) matches anything, in this case, the time and location of the continent. The third regular expression (New_York) matches the city portion of the location.

Treating the slash in the name as a delimiter lets the user refactor the hierarchy of the test without changing the name. It also simplifies escape rules. Users should avoid using slashes in names and replace them with backslashes if problems arise.

A unique sequence number is attached to a non-unique test name. Therefore, if no better subtest naming scheme is available, an empty string can be passed to Run, and the subtests can be easily identified by their serial numbers.

The Setup and Tear down

Sub-tests and sub-benchmarks can be used to manage common setup and tear-down code:

1func TestFoo(t *testing.T) { 2 // <setup code> 3 t.Run("A=1", func(t *testing.T) { ... }) 4 t.Run("A=2", func(t *testing.T) { ... }) 5 t.Run("B=1", func(t *testing.T) { 6 if ! test(foo{B:1}) { 7 t.Fail() 8 } 9 })10 // <tear-down code>11}Copy the code

When tests are run, the Setup and tear-down codes run at most once. This applies even if any of the subtests call Skip, Fail, or Fatal.

Parallelism control

Subtests allow fine-grained control over parallelism. To understand how to use subtests for parallel control, you need to understand the semantics of parallel testing.

Each test is associated with a test function. A test is called a Parallel test if the test function calls a Parallel method on its Testing.t instance. Parallel tests are never run at the same time as sequential tests until the sequential tests return. – Parallel flag defines the maximum number of parallel tests that can be run in parallel.

A test is blocked until all its child tests have completed. This means that in one test (TestXXX function), the sequential tests are not executed until the parallel tests are complete.

This behavior is the same for tests created by Run and top-level tests. In effect, a top-level test is a child of an implicit master test.

Run a set of tests in parallel

The semantics above allow one set of tests to run in parallel, but not others:

1func TestGroupedParallel(t *testing.T) { 2 for _, tc := range testCases { 3 tc := tc // capture range variable 4 t.Run(tc.Name, func(t *testing.T) { 5 t.Parallel() 6 if got := foo(tc.in); got ! = tc.out { 7 t.Errorf("got %v; want %v", got, tc.out) 8 } 9 ... 10 11} 12}})Copy the code

External tests will not complete until all parallel tests started by Run have completed. Therefore, no other parallel tests can run these parallel tests in parallel.

Note that we need to copy the range variable to ensure that the TC is bound to the correct instance. (Because range reuses TCS)

Clean up after parallel tests

In the previous example, depending on the semantics, wait for a set of parallel tests to complete before the other tests start. After a set of parallel tests that share a common resource, the same technique can be used to clean up:

 1func TestTeardownParallel(t *testing.T) { 2    // <setup code> 3    // This Run will not return until its parallel subtests complete. 4    t.Run("group", func(t *testing.T) { 5        t.Run("Test1", parallelTest1) 6        t.Run("Test2", parallelTest2) 7        t.Run("Test3", parallelTest3) 8    }) 9    // <tear-down code>10}Copy the code

The behavior of waiting for a set of parallel tests is the same as in the previous example.

conclusion

The Go 1.7 addition of subtests and subbenchmarks lets you write structured tests and benchmarks in a natural way that fits nicely into existing tools. Earlier versions of the Testing package had a 1-level hierarchy: package-level testing consisted of a separate set of tests and benchmarks. This structure has now been recursively extended to these individual tests and benchmarks. In effect, during implementation, top-level tests and benchmarks are tracked as if they were implicit children of master tests and benchmarks: the processing is the same at all levels.

The ability to define this structure for tests enables fine-grained execution of specific test cases, sharing setup and teardown, and better control over test parallelism. If you find any other uses, please share.

This article is translated by Xu Xinhua. From Go Language Chinese blog

译 文 : Using Subtests and sub-benchmarks

Like to scan attention to us