Benchmark framework JMH quick start

JMH profile

JMH is a Java Microbenchmark Harness that is used to perform benchmark testing in Java. JMH is provided and maintained by the OpenJDK and provides reliable test results.

Benchmark A Benchmark is a test used to measure and evaluate software performance indicators. A quantitative and comparable test is performed on a specific performance indicator in a specific target scenario.

Add dependencies to the project

Create a benchmark project to introduce jar packages from JMH, which is currently in the latest version 1.23. For example, the dependency configuration for Maven is as follows.

<dependencies>
    <dependency>
        <groupId>org.openjdk.jmh</groupId>
        <artifactId>jmh-core</artifactId>
        <version>1.23</version>
    </dependency>

    <dependency>
        <groupId>org.openjdk.jmh</groupId>
        <artifactId>jmh-generator-annprocess</artifactId>
        <version>1.23</version>
    </dependency>
</dependencies>
Copy the code

Alternatively, we can use it directly in unit testing for projects that need to be benchmarked.

Annotated use

At run time, annotation configurations are used to parse and generate BenchmarkListEntry configuration class instances.

A method corresponds to an @benchmark annotation, and an @benchmark annotation corresponds to a Benchmark method.

Annotations on a class, or annotations on a field of a class, are configurations common to all benchmark methods in a class.

@Benchmark

Declare a public method as a benchmark method.

public class JsonBenchmark {
    
    @Benchmark
    @Test // Since this is a unit Test class, there is an @test annotation, which can be ignored
    public void testGson(a) {
        new GsonParser().fromJson("{\"startDate\":\"2020-04-01 16:00:00\",\"endDate\":\"2020-05-20 13:00:00\",\"flag\":true,\"threads\":5,\"shardingIndex\":0}", JsonTestModel.class); }}Copy the code

@BenchmarkMode

Through JMH we can easily test the throughput of an interface, the average execution time and other indicators of data.

Suppose I want to test the AverageTime taken by testGson’s methods, then I can specify the test dimension as mode.averagetime using the @benchmarkmode annotation.

public class JsonBenchmark {
    
    @BenchmarkMode(Mode.AverageTime) // Specify mode as mode. AverageTime
    @Benchmark
    @Test // Since this is a unit Test class, there is an @test annotation, which can be ignored
    public void testGson(a) {
        new GsonParser().fromJson("{\"startDate\":\"2020-04-01 16:00:00\",\"endDate\":\"2020-05-20 13:00:00\",\"flag\":true,\"threads\":5,\"shardingIndex\":0}", JsonTestModel.class); }}Copy the code

@Measurement

Suppose I want to measure the testGson method five times, then I can use the @measurement annotation.

public class JsonBenchmark {
    
    @BenchmarkMode(Mode.AverageTime) // Specify mode as mode. AverageTime
    @Benchmark
    @Measurement(iterations = 5, time = 1, timeUnit = TimeUnit.SECONDS)
    @Test // Since this is a unit Test class, there is an @test annotation, which can be ignored
    public void testGson(a) {
        new GsonParser().fromJson("{\"startDate\":\"2020-04-01 16:00:00\",\"endDate\":\"2020-05-20 13:00:00\",\"flag\":true,\"threads\":5,\"shardingIndex\":0}", JsonTestModel.class); }}Copy the code

The @measurement annotation has three configuration items:

• iterations: number of measurements;
• Time and timeUnit: Duration of each measurement,timeUnitSpecifies the unit of time, in this case: each measurement lasts 1 second and is performed within 1 secondtestGsonThe number of methods is not fixed and is determined by the method execution time and time.

@Warmup

To be accurate, we may need to warm up the testGson method. For example, if you create a GsonParser object in a method, preheating prevents the test results from being prepared when you first create GsonParser because it takes too long to load classes. The JVM uses a JIT just-in-time compiler, and a certain number of warm-up times allow the JIT to complete compilation of the testGson method call link, eliminating the effect of interpretation execution on test results.

The @warmUp annotation is used to configure the Warmup parameters.

public class JsonBenchmark {
    
    @BenchmarkMode(Mode.AverageTime) // Specify mode as mode. AverageTime
    @Benchmark
    @Warmup(iterations = 5, time = 1, timeUnit = TimeUnit.SECONDS)
    @Measurement(iterations = 5, time = 1, timeUnit = TimeUnit.SECONDS)
    @Test // Since this is a unit Test class, there is an @test annotation, which can be ignored
    public void testGson(a) {
        new GsonParser().fromJson("{\"startDate\":\"2020-04-01 16:00:00\",\"endDate\":\"2020-05-20 13:00:00\",\"flag\":true,\"threads\":5,\"shardingIndex\":0}", JsonTestModel.class); }}Copy the code

The @warmUp annotation has three configuration items:

• iterations: preheating times;
• Time and timeUnit: Duration of each preheating,timeUnitSpecifies the time unit.

Assume that @Measurement specifies 100 iterations and time is 10s, then: The number of times that each thread actually executes the benchmark method is equal to time divided by the single execution time of the benchmark method. Assuming that the execution time of the benchmark method is 1s, a measurement can only execute the benchmark method 10 times at most (time is 10s/method execution time is 1s). Iterations of 100 means testing 100 times (not executing the benchmark method 100 times).

@OutputTimeUnit

The OutputTimeUnit annotation is used to specify the unit of output method execution time. If the method takes seconds to execute, we can use @outputTimeUnit to specify the output time in seconds for easy observation. If the method takes milliseconds to execute, you can use @outputtimeUnit to specify the output time in milliseconds. Otherwise, if you use the default second, the output will be 10 to the minus something, which is not very intuitive.

public class JsonBenchmark {
    
    @BenchmarkMode(Mode.AverageTime) // Specify mode as mode. AverageTime
    @Benchmark
    @OutputTimeUnit(TimeUnit.NANOSECONDS) // Specify the unit of output elapsed time
    @Warmup(iterations = 5, time = 1, timeUnit = TimeUnit.SECONDS)
    @Measurement(iterations = 5, time = 1, timeUnit = TimeUnit.SECONDS)
    @Test // Since this is a unit Test class, there is an @test annotation, which can be ignored
    public void testGson(a) {
        new GsonParser().fromJson("{\"startDate\":\"2020-04-01 16:00:00\",\"endDate\":\"2020-05-20 13:00:00\",\"flag\":true,\"threads\":5,\"shardingIndex\":0}", JsonTestModel.class); }}Copy the code

@Fork

Fork is used to specify how many children are forked to execute the same benchmark method. Assuming we don’t need multiple processes, we can specify the number of processes to be 1 using @fork.

public class JsonBenchmark {
    
    @BenchmarkMode(Mode.AverageTime) // Specify mode as mode. AverageTime
    @Benchmark
    @Fork(1)
    @OutputTimeUnit(TimeUnit.NANOSECONDS) // Specify the unit of output elapsed time
    @Warmup(iterations = 5, time = 1, timeUnit = TimeUnit.SECONDS)
    @Measurement(iterations = 5, time = 1, timeUnit = TimeUnit.SECONDS)
    @Test // Since this is a unit Test class, there is an @test annotation, which can be ignored
    public void testGson(a) {
        new GsonParser().fromJson("{\"startDate\":\"2020-04-01 16:00:00\",\"endDate\":\"2020-05-20 13:00:00\",\"flag\":true,\"threads\":5,\"shardingIndex\":0}", JsonTestModel.class); }}Copy the code

@Threads

The @Threads annotation is used to specify how many Threads are used to execute the benchmark method. If @Threads is used to specify 2 Threads, then two Threads are created for each measurement to execute the benchmark method.

public class JsonBenchmark {
    
    @BenchmarkMode(Mode.AverageTime) // Specify mode as mode. AverageTime
    @Benchmark
    @Fork(1)
    @Threads(2)
    @OutputTimeUnit(TimeUnit.NANOSECONDS) // Specify the unit of output elapsed time
    @Measurement(iterations = 5, time = 1, timeUnit = TimeUnit.SECONDS)
    @Test // Since this is a unit Test class, there is an @test annotation, which can be ignored
    public void testGson(a) {
        new GsonParser().fromJson("{\"startDate\":\"2020-04-01 16:00:00\",\"endDate\":\"2020-05-20 13:00:00\",\"flag\":true,\"threads\":5,\"shardingIndex\":0}", JsonTestModel.class); }}Copy the code

If the @measurement annotation specifies a time of 1s and the benchmark method takes 1s to execute, then only one benchmark method will be executed in a single Measurement with a single thread, and 10 benchmark methods will be executed in a single Measurement with 10 threads.

Public comment

Suppose we need to create two benchmark methods in the JsonBenchmark class, testGson and testJackson, to compare the performance of the Gson and Jackson frameworks for parsing JSON. Then we can declare all annotations except the @benchmark annotation on the class, so that both Benchmark methods use the same configuration.

@BenchmarkMode(Mode.AverageTime)
@Fork(1)
@Threads(2)
@OutputTimeUnit(TimeUnit.NANOSECONDS)
@Warmup(iterations = 5, time = 1, timeUnit = TimeUnit.SECONDS)
@Measurement(iterations = 5, time = 1, timeUnit = TimeUnit.SECONDS)
public class JsonBenchmark {

    @Benchmark
    @Test
    public void testGson(a) {
        new GsonParser().fromJson("{\"startDate\":\"2020-04-01 16:00:00\",\"endDate\":\"2020-05-20 13:00:00\",\"flag\":true,\"threads\":5,\"shardingIndex\":0}", JsonTestModel.class);
    }

    @Benchmark
    @Test
    public void testJackson(a) {
        new JacksonParser().fromJson("{\"startDate\":\"2020-04-01 16:00:00\",\"endDate\":\"2020-05-20 13:00:00\",\"flag\":true,\"threads\":5,\"shardingIndex\":0}", JsonTestModel.class); }}Copy the code

Instead of creating a GsonParser or JacksonParser instance every time the method is executed, we can declare GsonParser and JacksonParser objects as fields of JsonBenchmark. (GsonParser and JacksonParser are tool classes encapsulated by the author, which are used with design mode to provide the function of switching parsing framework at any time for the project).

@BenchmarkMode(Mode.AverageTime)
@Fork(1)
@Threads(2)
@OutputTimeUnit(TimeUnit.NANOSECONDS)
@Warmup(iterations = 5, time = 1, timeUnit = TimeUnit.SECONDS)
@Measurement(iterations = 5, time = 1, timeUnit = TimeUnit.SECONDS)
@State(Scope.Benchmark)
public class JsonBenchmark {

     private GsonParser gsonParser = new GsonParser();
     private JacksonParser jacksonParser = new JacksonParser();

    @Benchmark
    @Test
    public void testGson(a) {
        gsonParser.fromJson("{\"startDate\":\"2020-04-01 16:00:00\",\"endDate\":\"2020-05-20 13:00:00\",\"flag\":true,\"threads\":5,\"shardingIndex\":0}", JsonTestModel.class);
    }

    @Benchmark
    @Test
    public void testJackson(a) {
        jacksonParser.fromJson("{\"startDate\":\"2020-04-01 16:00:00\",\"endDate\":\"2020-05-20 13:00:00\",\"flag\":true,\"threads\":5,\"shardingIndex\":0}", JsonTestModel.class); }}Copy the code

You also need to specify the shared domain of the field using the @state annotation. In this example, we use the @Threads annotation declaration to create two Threads to execute the benchmark method. Assuming we configure @state (scope.Thread), the fields gsonParser and jacksonParser are two different instances in different Threads.

Using the testGson method as an example, we can assume that JMH clones a gsonParser object for each thread. If you print the hashCode of the gsonParser object in the testGson method, you will see that the same thread prints the same result, but different threads print different results. Such as:

@BenchmarkMode(Mode.AverageTime)
@Fork(1)
@Threads(2)
@OutputTimeUnit(TimeUnit.NANOSECONDS)
@Warmup(iterations = 5, time = 1, timeUnit = TimeUnit.SECONDS)
@Measurement(iterations = 5, time = 1, timeUnit = TimeUnit.SECONDS)
@State(Scope.Benchmark)
public class JsonBenchmark {

     private GsonParser gsonParser = new GsonParser();
     private JacksonParser jacksonParser = new JacksonParser();

    @Benchmark
    @Test
    public void testGson(a) {
        System.out.println("current Thread:" + Thread.currentThread().getName() + "= = >" + gsonParser.hashCode());
        gsonParser.fromJson("{\"startDate\":\"2020-04-01 16:00:00\",\"endDate\":\"2020-05-20 13:00:00\",\"flag\":true,\"threads\":5,\"shardingIndex\":0}", JsonTestModel.class);
    }

    @Benchmark
    @Test
    public void testJackson(a) {
        jacksonParser.fromJson("{\"startDate\":\"2020-04-01 16:00:00\",\"endDate\":\"2020-05-20 13:00:00\",\"flag\":true,\"threads\":5,\"shardingIndex\":0}", JsonTestModel.class); }}Copy the code

The testGson method produces the following output:

current Thread:com.msyc.common.JsonBenchmark.testGson-jmh-worker-1==>2063684770
current Thread:com.msyc.common.JsonBenchmark.testGson-jmh-worker-2==>1629232880
current Thread:com.msyc.common.JsonBenchmark.testGson-jmh-worker-1==>2063684770
current Thread:com.msyc.common.JsonBenchmark.testGson-jmh-worker-2==>1629232880
current Thread:com.msyc.common.JsonBenchmark.testGson-jmh-worker-1==>2063684770
current Thread:com.msyc.common.JsonBenchmark.testGson-jmh-worker-2==>1629232880
current Thread:com.msyc.common.JsonBenchmark.testGson-jmh-worker-1==>2063684770
current Thread:com.msyc.common.JsonBenchmark.testGson-jmh-worker-2==>1629232880
......
Copy the code

@Param

The @param annotation can be used to specify the base method execution parameters. The @param annotation can only specify a String value, which can be an array, and the parameter values will be iterated at run time in the given order. If the @param annotation specifies multiple parameter values, JMH takes a measurement for each parameter value.

For example, we want to test the performance of json strings of different complexity parsed using the Gson framework versus the Jackson framework.

@BenchmarkMode(Mode.AverageTime)
@Fork(1)
@Threads(2)
@OutputTimeUnit(TimeUnit.NANOSECONDS)
@Warmup(iterations = 5, time = 1, timeUnit = TimeUnit.SECONDS)
@Measurement(iterations = 5, time = 1, timeUnit = TimeUnit.SECONDS)
@State(Scope.Thread)
public class JsonBenchmark {

    private GsonParser gsonParser = new GsonParser();
    private JacksonParser jacksonParser = new JacksonParser();

    // Specify that the parameter has three values
    @Param(value = {"{\"startDate\":\"2020-04-01 16:00:00\",\"endDate\":\"2020-05-20 13:00:00\",\"flag\":true,\"threads\":5,\"shardingIndex\":0}", "{\"startDate\":\"2020-04-01 16:00:00\",\"endDate\":\"2020-05-20 14:00:00\"}", "{\"flag\":true,\"threads\":5,\"shardingIndex\":0}"})
    private String jsonStr;

    @Benchmark
    @Test
    public void testGson(a) {
        gsonParser.fromJson(jsonStr, JsonTestModel.class);
    }

    @Benchmark
    @Test
    public void testJackson(a) { jacksonParser.fromJson(jsonStr, JsonTestModel.class); }}Copy the code

The test results are as follows:

Benchmark (jsonStr) Mode Cnt Score Error Units JsonBenchmark.testGson {"startDate":"2020-04-01 ","endDate":"2020-05-20 13:00:00","flag":true,"threads":5,"shardingIndex":0} avGT 5 12180.763 ± 2481.973 ns/op Jsonbenchmark. testGson {"startDate":"2020-04-01 16:00:00","endDate":"2020-05-20 14:00:00"} avGT 5 8154.709 ± 3393.881 Ns /op jsonBenchmark. testGson {"flag":true,"threads":5,"shardingIndex":0} avGT 5 9994.171 ± 5737.958 ns/op JsonBenchmark.testJackson {"startDate":"2020-04-01 16:00:00","endDate":"2020-05-20 13:00:00 ", "flag" : true, "threads" : 5, shardingIndex: 0} avgt 5 to 15663.060 + 9042.672 ns/op JsonBenchmark. TestJackson {"startDate":"2020-04-01 16:00:00","endDate":"2020-05-20 14:00:00"} avGT 5 13776.828 ± 11006.412 ns/op JsonBenchmark. TestJackson {" flag ": true," threads ": 5," shardingIndex ": 0} avgt 5 to 9824.283 + 311.555 ns/opCopy the code

Non-annotation use

Using annotations is the same as not using annotations, but annotations are more convenient. At run time, the annotation configuration is used to parse and generate BenchmarkListEntry configuration class instances, and the Options configuration in code is also parsed into BenchmarkListEntry configuration class instances (one for each method).

We can construct an Options using OptionsBuilder, for example, to implement the above example in a non-annotated manner.

public class BenchmarkTest{

    @Test
    public void test(a) throws RunnerException {
        Options options = new OptionsBuilder()
                .include(JsonBenchmark.class.getSimpleName())
                .exclude("testJackson")
                .forks(1)
                .threads(2)
                .timeUnit(TimeUnit.NANOSECONDS)
                .warmupIterations(5)
                .warmupTime(TimeValue.seconds(1))
                .measurementIterations(5)
                .measurementTime(TimeValue.seconds(1))
                .mode(Mode.AverageTime)
                .build();
        newRunner(options).run(); }}Copy the code

• include: Imports a benchmark class. The calling method passes the simple name of the class, not the package name.
• exclude: What methods to exclude. The defaultJMHforincludeEach of the imported classespublicMethods all produce oneBenchmarkListEntryConfigure class instances, that is, add eachpublicMethods are considered benchmark methods, which we can useexcludeExclude methods that do not require participation in benchmarking. For example, in this exampleexcludeRuled outtestJacksonMethods.

Jar package put on the server for execution

For large tests that take a long time and require a lot of threads, we can package the benchmark project into a JAR package and throw it into a Linux server for execution. For throughput benchmark testing, it is recommended to perform it on the server. The results will be more accurate. The hardware and system are close to the online environment, and it is not affected by the number of applications enabled on the machine, hardware configuration and other factors.

java -jar my-benchmarks.jar
Copy the code

Execute in IDEA

We don’t need to run tests on the server for common method execution time tests, such as testing the performance of several JSON parsing frameworks.

In the idea, we can write a unit test method, in the unit test method to create an org. Its. JMH. Runner. Runner, invoke the runner run method perform benchmarks. But JMH does not scan packages and does not execute every benchmark method, which requires us to tell JMH through configuration items which benchmarks need to be executed.

public class BenchmarkTest{

    @Test
    public void test(a) throws RunnerException {
        Options options = null; / / create the Options
        newRunner(options).run(); }}Copy the code

The complete example is as follows:

public class BenchmarkTest{
     @Test
     public void test(a) throws RunnerException {
        Options options = new OptionsBuilder()
                 .include(JsonBenchmark.class.getSimpleName())
                 // .output("/tmp/json_benchmark.log")
                 .build();
        newRunner(options).run(); }}Copy the code

Options was introduced earlier, and since the JsonBenchmark class in this example is already annotated, Options only needs to configure the class to perform the benchmark. If you need to execute more than one benchmark class, the include method can be called multiple times.

Call the Output method to configure the file path if you want to output the test results to a file, or to the console if you don’t.

Execute in IDEA using the JMH Plugin

Plug-in source address: https://github.com/artyushov/idea-jmh-plugin.

Install: Search for JMH Plugin in IDEA, and restart it to use it.

• 1. Perform a single executionBenchmarkmethods

In the line of the method name, IDEA will have an ▶️ execution symbol. Right-click to run. If you are writing a unit test method, IDEA will prompt you to choose whether to perform unit tests or benchmarks.

• 2. Execute all of a classBenchmarkmethods

In the line of the class name, IDEA will have an ▶️ execution symbol. Right-click to run, and all methods in the class annotated by the @benchmark annotation will be executed. If you are writing a unit test method, IDEA will prompt you to choose whether to perform unit tests or benchmarks.

Official JMH usage examples

• The officialdemo:

http://hg.openjdk.java.net/code-tools/jmh/file/tip/jmh-samples/src/main/java/org/openjdk/jmh/samples/
Copy the code

• The translation ofdemo:

https://github.com/Childe-Chen/goodGoodStudy/tree/master/src/main/java/com/cxd/benchmark
Copy the code