Note: This is really not clickbait. I’ve been writing code for 20+ years and I really think Go Fuzzing is the best code self-testing method I’ve ever seen. When I used AC automata algorithm to improve keyword filtering efficiency (increase ~50%) and mapReduce to improve the processing mechanism of panic, I found the edge case bug through Go Fuzzing. So deeply believe that this is the most awesome code I have ever tested method, no one!
Go Fuzzing has so far found over 200 bugs in the go standard library with very high quality code, see: github.com/dvyukov/go-…
Spring Festival programmers often wish you a bug-free code! Although joking, but for every programmer, we write bugs every day, this is a fact. The fact that the code has no bugs can only be falsified, not proved. The upcoming Go 1.18 release officially provides a great tool to help us prove the fraud – Go Fuzzing.
Go 1.18 is mostly about generics, but I really think Go Fuzzing is the most useful feature in Go 1.18, not one!
In this article, let’s take a closer look at Go Fuzzing:
- What is?
- How does it work?
- What are the best practices?
First, you need to upgrade to Go 1.18
Go 1.18 is not officially released yet, but you can download the RC version, and even if you are producing earlier versions of Go, you can develop environments that use Go Fuzzing to find bugs
What is go Fuzzing
According to the official documentation, Go Fuzzing automates testing by constantly giving different inputs to a program and intelligently looking for failed cases by analyzing code coverage. This method can find some edge cases as much as possible, and the real test does find some problems that are difficult to find at ordinary times.
How to use “go fuzzing”
Fuzz Tests:
-
The function must start with Fuzz and take only * testing.f, with no return value
-
Fuzz Tests must be in the *_test.go file
-
The fuzz target in the figure above is a method call (* testing.f).fuzz. The first argument is * testing.t, followed by arguments called fuzzing arguments, with no return value
-
There can only be one Fuzz target per Fuzz test
-
Call the f.A dd (…). The fuzzing arguments order and type are the same
-
Fuzzing Arguments only supports the following types:
string.[]byteint.int8.int16.int32/rune.int64uint.uint8/byte.uint16.uint32.uint64float32.float64bool
-
The fuzz target does not depend on global state and runs in parallel.
runfuzzing tests
If I write a Fuzzing test, for example:
Specific code / / https://github.com/zeromicro/go-zero/blob/master/core/mr/mapreduce_fuzz_test.go
func FuzzMapReduce(f *testing.F){... }Copy the codeSo we can do this:
go test -fuzz=MapReduce
Copy the codeWe get something like this:
fuzz: elapsed: 0s, gathering baseline coverage: 0/2 completed
fuzz: elapsed: 0s, gathering baseline coverage: 2/2 completed, now fuzzing with 10 workers
fuzz: elapsed: 3s, execs: 3338 (1112/sec), new interesting: 56 (total: 57)
fuzz: elapsed: 6s, execs: 6770 (1144/sec), new interesting: 62 (total: 63)
fuzz: elapsed: 9s, execs: 10157 (1129/sec), new interesting: 69 (total: 70)
fuzz: elapsed: 12s, execs: 13586 (1143/sec), new interesting: 72 (total: 73)
^Cfuzz: elapsed: 13s, execs: 14031 (1084/sec), new interesting: 72 (total: 73)
PASS
ok github.com/zeromicro/go-zero/core/mr 13.169s
Copy the codeI pressed Ctrl-C to terminate the test. Please refer to the official documentation for details.
Best practices for Go-Zero
According to my experience, THE best practices are preliminarily summarized as the following four steps:
- define
fuzzing argumentsFirst of all, how do you define itfuzzing argumentsAnd pass through the givenfuzzing arguments写fuzzing target - thinking
fuzzing targetHow do I write that? The point here is how do I verify that this is true, becausefuzzing argumentsIt was given “randomly”, so there had to be a universal way to verify the results - Think about how to print the result of a failed case so as to generate a new one
unit test - According to the failed
fuzzing testPrint the result and write a new oneUnit test, this new oneunit testWill be used for debugging solutionsfuzzing testFound problems and solidified down leftCI用
Next, we will use a simple array summation function to illustrate the above steps. The actual case of Go-Zero is a little complicated, and I will give the internal case of Go-Zero at the end of this article for your reference.
Here is a code implementation of the bug-injected summation:
func Sum(vals []int64) int64 {
var total int64
for _, val := range vals {
if val%1e5! =0 {
total += val
}
}
return total
}
Copy the codeDefinition 1.fuzzing arguments
You need to give at least one fuzzing argument, otherwise Go Fuzzing can’t generate the test code, so even if we don’t have good input, we need to define a Fuzzing argument that affects the results, Here we use the slice element count as the Fuzzing arguments, and Go Fuzzing automatically generates different arguments for running code Coverage to simulate the test.
func FuzzSum(f *testing.F) {
f.Add(10)
f.Fuzz(func(t *testing.T, n int) {
n %= 20. })}Copy the codeN here is to let Go Fuzzing simulate the number of slice elements. In order to ensure that the number of elements is not too large, we limit it to 20 (zero is ok), and we add a corpus with a value of 10 (called corpus in Go Fuzzing). This value is the one that makes Go Fuzzing cold start, and it doesn’t matter how much it is.
2. How do you writefuzzing target
This step focuses on writing verifiable Fuzzing Targets. While writing test code based on the given Fuzzing arguments, you also need to generate data to verify that the results are correct.
For our Sum function, it is actually relatively simple, which is to randomly generate a slice of N elements and Sum them to calculate the desired result. As follows:
func FuzzSum(f *testing.F) {
rand.Seed(time.Now().UnixNano())
f.Add(10)
f.Fuzz(func(t *testing.T, n int) {
n %= 20
var vals []int64
var expect int64
for i := 0; i < n; i++ {
val := rand.Int63() % 1e6
vals = append(vals, val)
expect += val
}
assert.Equal(t, expect, Sum(vals))
})
}
Copy the codeThis code is easy to understand, just to compare the Sum itself to the Sum Sum, so I won’t explain it in detail. But in complex scenarios you need to think carefully about how to write validation code, but it’s not too difficult, and if it’s too difficult, it’s probably because you don’t understand or simplify the test functions enough.
To run Fuzzing tests, run the following command:
$ go test -fuzz=Sum
fuzz: elapsed: 0s, gathering baseline coverage: 0/2 completed
fuzz: elapsed: 0s, gathering baseline coverage: 2/2 completed, now fuzzing with 10 workers
fuzz: elapsed: 0s, execs: 6672 (33646/sec), new interesting: 7 (total: 6)
--- FAIL: FuzzSum (0.21s)
--- FAIL: FuzzSum (0.00s)
sum_fuzz_test.go:34:
Error Trace: sum_fuzz_test.go:34
value.go:556
value.go:339
fuzz.go:334
Error: Not equal:
expected: 8736932
actual : 8636932
Test: FuzzSum
Failing input written to testdata/fuzz/FuzzSum/739002313aceff0ff5ef993030bbde9115541cabee2554e6c9f3faaf581f2004
To re-run:
go test -run=FuzzSum/739002313aceff0ff5ef993030bbde9115541cabee2554e6c9f3faaf581f2004
FAIL
exit status 1
FAIL github.com/kevwan/fuzzing 0.614s
Copy the codeSo here’s the problem! We see that the result is wrong, but it is difficult for us to analyze why it is wrong. Please carefully sample the output above. How do you analyze it?
3. How to print input for failed case
For the failed test above, if we could print out the input and form a simple test case, we could debug directly. It is best to copy/paste the printed input directly into the new test case. If the format is not correct, it will be too tiring to format the input line by line, and it does not have to be a single failed case.
So we changed our code to look like this:
func FuzzSum(f *testing.F) {
rand.Seed(time.Now().UnixNano())
f.Add(10)
f.Fuzz(func(t *testing.T, n int) {
n %= 20
var vals []int64
var expect int64
var buf strings.Builder
buf.WriteString("\n")
for i := 0; i < n; i++ {
val := rand.Int63() % 1e6
vals = append(vals, val)
expect += val
buf.WriteString(fmt.Sprintf("%d,\n", val))
}
assert.Equal(t, expect, Sum(vals), buf.String())
})
}
Copy the codeRun the command again and get the following result:
$ go test -fuzz=Sum
fuzz: elapsed: 0s, gathering baseline coverage: 0/2 completed
fuzz: elapsed: 0s, gathering baseline coverage: 2/2 completed, now fuzzing with 10 workers
fuzz: elapsed: 0s, execs: 1402 (10028/sec), new interesting: 10 (total: 8)
--- FAIL: FuzzSum (0.16s)
--- FAIL: FuzzSum (0.00s)
sum_fuzz_test.go:34:
Error Trace: sum_fuzz_test.go:34
value.go:556
value.go:339
fuzz.go:334
Error: Not equal:
expected: 5823336
actual : 5623336
Test: FuzzSum
Messages:
799023,
110387,
811082,
115543,
859422,
997646,
200000,
399008,
7905,
931332,
591988,
Failing input written to testdata/fuzz/FuzzSum/26d024acf85aae88f3291bf7e1c6f473eab8b051f2adb1bf05d4491bc49f5767
To re-run:
go test -run=FuzzSum/26d024acf85aae88f3291bf7e1c6f473eab8b051f2adb1bf05d4491bc49f5767
FAIL
exit status 1
FAIL github.com/kevwan/fuzzing 0.602s
Copy the code4. Write new test cases
Copy /paste generates the following code based on the failed case output. Of course, the framework is written by ourselves, and the input parameters can be directly copied into it.
func TestSumFuzzCase1(t *testing.T) {
vals := []int64{
799023.110387.811082.115543.859422.997646.200000.399008.7905.931332.591988,
}
assert.Equal(t, int64(5823336), Sum(vals))
}
Copy the codeThis will make it easy to debug and add a valid Unit test to ensure the bug never appears again.
go fuzzingMore experience
Go Version problems
I believe that most of the project line code will not be upgraded to 1.18 immediately after Go 1.18 is released, so what if the testing.f introduced by Go Fuzzing cannot be used?
Go (go.mod) does not upgrade to go 1.18, but it is fully recommended to upgrade to go 1.18, so we just need to put the above FuzzSum in a file with a filename like sum_fuzz_test.go and add the following command to the file header:
/ / go: build go1.18
/ / + build go1.18
Copy the codeNote: The third line must be an empty line, otherwise it will become a comment for the package.
In this way, no matter which version we use online, we will not report errors, and we generally run fuzz testing on the machine, which will not be affected.
Failure to go fuzzing
The steps above are for simple cases, but sometimes the problem becomes more complicated when a new Unit test is formed from input from a failed case and does not reproduce the problem (especially with goroutine deadlocks).
go test -fuzz=MapReduce fuzz: elapsed: 0s, gathering baseline coverage: 0/2 completed fuzz: elapsed: 0s, gathering baseline coverage: 2/2 completed, now fuzzing with 10 workers fuzz: elapsed: 3s, execs: 3681 (1227/sec), new interesting: 54 (total: 55) ... fuzz: elapsed: 1m21s, execs: 92705 (1101/sec), new interesting: 85 (total: 86) --- FAIL: FuzzMapReduce (80.96s) Fuzzing process hung or terminated continuously: exit status 2 Failing input written to testdata/fuzz/FuzzMapReduce/ee6a61e8c968adad2e629fba11984532cac5d177c4899d3e0b7c2949a0a3d840 To re-run: go test -run=FuzzMapReduce/ee6a61e8c968adad2e629fba11984532cac5d177c4899d3e0b7c2949a0a3d840 FAIL exit status 1 FAIL Github.com/zeromicro/go-zero/core/mr 81.471 sCopy the codeIn this case, it simply tells us that the Fuzzing process is stuck or ended abnormally, with a status code of 2. In this case, normally re-run will not reproduce. Why simply return error code 2? I carefully looked at the go Fuzzing source code, each Fuzzing test is run by a separate process, and then Go Fuzzing threw away the fuzzy test process output, just display the status code. So how do we solve this problem?
After careful analysis, I decided to write a regular unit test code like the Fuzzing test myself. This ensures that the failure is within the same process and prints the error message to the standard output. The code looks like this:
func TestSumFuzzRandom(t *testing.T) {
const times = 100000
rand.Seed(time.Now().UnixNano())
for i := 0; i < times; i++ {
n := rand.Intn(20)
var vals []int64
var expect int64
var buf strings.Builder
buf.WriteString("\n")
for i := 0; i < n; i++ {
val := rand.Int63() % 1e6
vals = append(vals, val)
expect += val
buf.WriteString(fmt.Sprintf("%d,\n", val))
}
assert.Equal(t, expect, Sum(vals), buf.String())
}
}
Copy the codeThis way we can simply simulate Go Fuzzing ourselves, but any errors we make we get clear output. Maybe I haven’t figured out go Fuzzing here, or there are other ways to control it. If you know, thank you for letting me know.
But for a simulation case that takes a long time to run, we don’t want it to be executed every time we run CI, so I put it in a separate file with a name like sum_fuzzcase_test.go and add the following instructions to the header:
//go:build fuzz
// +build fuzz
Copy the codeSo we need to run the simulation case with -tags fuzz, for example:
go test -tags fuzz ./...
Copy the codeComplex usage examples
The above is a simple example. If you don’t know how to write in a complex scenario, you can first see how Go-Zero landed in Go Fuzzing, as shown below:
- MapReduce – Github.com/zeromicro/g…
- Fuzzy testedA deadlock 和 goroutine leak, especially
chan + goroutineComplex scenarios can be used for reference
- Fuzzy testedA deadlock 和 goroutine leak, especially
- stringx – Github.com/zeromicro/g…
- Fuzzy test of the conventional algorithm implementation, algorithm class scenarios can be used for reference
The project address
Github.com/zeromicro/g…
Welcome to Go-Zero and star support us!
Wechat communication group
Pay attention to the public account of “micro-service Practice” and click on the exchange group to obtain the QR code of the community group.