【Java】 Microbenchmark framework JMH

JMH is introduced

The JMH is a Java tool for building, running, and analyzing microbenchmarks written in Java and other JVM languages.

Depend on the package

• maven

  <dependency> <groupId>org.openjdk.jmh</groupId> <artifactId>jmh-core</artifactId> <version>1.32</version> </dependency> <dependency> <groupId>org.openjdk.jmh</groupId> <artifactId>jmh-generator-annprocess</artifactId> The < version > 1.32 < / version > < / dependency >Copy the code

• gradle

  implementation group: 'org.openjdk.jmh', name: 'jmh-core', version: '1.32'
  implementation group: 'org.openjdk.jmh', name: 'jmh-generator-annprocess', version: '1.32'
  Copy the code

use

Use the @Benchmark annotation to test

Use the @Benchmark annotation to specify the method you want to test. The Runner is the entry to the JMH, and Options is its entry parameter, which is generated by the OptionsBuilder. You can configure the test in Options. Specify the class name as an argument to OptionsBuilder::include to set the class you want to test.

package com.example;

import org.openjdk.jmh.annotations.Benchmark;
import org.openjdk.jmh.runner.Runner;
import org.openjdk.jmh.runner.RunnerException;
import org.openjdk.jmh.runner.options.Options;
import org.openjdk.jmh.runner.options.OptionsBuilder;

public class Example_01_HelloJMH {

    @Benchmark
    public String sayHello(a) {
        return "HELLO JMH!";
    }

    public static void main(String[] args) throws RunnerException {
        Options options = new OptionsBuilder()
                .include(Example_01_HelloJMH.class.getSimpleName())
                .build();
        newRunner(options).run(); }}Copy the code

Use @benchmarkMode to specify the test mode

Use the @benchmarkMode annotation to specify the test Mode, taking an array of Mode types.

@Benchmark
@BenchmarkMode(Mode.Throughput)
public void measureThroughput(a) throws InterruptedException {
    /* Only throughput is tested */
    TimeUnit.MILLISECONDS.sleep(100);
}

@Benchmark
@BenchmarkMode({Mode.Throughput, Mode.AverageTime, Mode.SampleTime})
public void measureMultiple(a) throws InterruptedException {
    /* Test throughput, average time and sampling time */
    TimeUnit.MILLISECONDS.sleep(100);
}

@Benchmark
@BenchmarkMode(Mode.All)
public void measureAll(a) throws InterruptedException {
    /* Test all, i.e. throughput, average time, sampling time and startup time */
    TimeUnit.MILLISECONDS.sleep(100);
}
Copy the code

The Mode enumeration class enumerates the JMH test modes, which are:

The OP in the unit stands for one operation, which by default means executing one test method. But we can specify how many times a test method is called to count as an operation. In the JMH, this is called the number of batches in an operation; for example, we can set five test methods to count as one operation.

Use @measurement to specify the number of tests

The @measurement annotation can be applied to a class or method to specify the number of tests, time, and number of batches. The parameters are:

• Iterations: Measurement times. The default value is 5 times.

• Time: indicates the duration of a single measurement. The default value is 10.

• TimeUnit: specifies the unit of time. The default is second.

• BatchSize: The number of times a batch is processed per operation. The default is 1, meaning that a call to a test method counts as one operation.

Each of the four arguments can be specified separately in Options, with the priority class < method < Options.

package com.example;

import org.openjdk.jmh.annotations.Benchmark;
import org.openjdk.jmh.annotations.Measurement;
import org.openjdk.jmh.runner.Runner;
import org.openjdk.jmh.runner.RunnerException;
import org.openjdk.jmh.runner.options.Options;
import org.openjdk.jmh.runner.options.OptionsBuilder;

import java.util.concurrent.TimeUnit;

@Measurement(iterations = 3, time = 500, timeUnit = TimeUnit.MILLISECONDS, batchSize = 1)
public class Example_03_Measurement {

    public static void main(String[] args) throws RunnerException {
        Options options = new OptionsBuilder()
                .include(Example_03_Measurement.class.getSimpleName())
                .build();
        new Runner(options).run();
    }

    @Benchmark
    public String hello01(a) {
        return "Java";
    }

    @Benchmark
    @Measurement(batchSize = 10)
    public String hello02(a) {
        return "Java";
    }

    @Benchmark
    @Measurement(iterations = 5, time = 1, timeUnit = TimeUnit.SECONDS, batchSize = 1)
    public String hello03(a) {
        return "Java"; }}Copy the code

Use @warmup to specify the number of warm-ups

Since the JVM uses the JIT just-in-time compiler to compile hot code, the same code may be executed too differently as the number of times increases, so we can let the code warm up a few times, without counting the warm-up time in the measurement timing. Use @warmup the same as @measurement.

Use @fork to set the number of test processes

Use @thread to set the number of threads used for execution

Avoid JIT optimization

The first is to use @warmup to WarmUp the code to prevent time differences between executions. But this only solves the timing problem. There are a lot of other optimizations that the JIT can do that are great for the JVM and terrible for the JMH.

List of JIT optimization techniques: PerformanceTacticIndex – PerformanceTacticIndex – OpenJDK Wiki (java.net)

Prevent unwanted code elimination

For some code that the JIT determines is not meaningful, the JIT will directly delete it during execution, such as:

@Benchmark
public void noCode(a) {}

@Benchmark
public void doSth(a) {
    double a = Math.sqrt(x);
}
Copy the code

The comparison between the two results is as follows:

Benchmark Mode Cnt Score Error Units Example_00_DeadCodeElimination. DoSth THRPT 15 4048.484 + / - 129.437 ops/us Example_00_DeadCodeElimination. NoCode THRPT 15 4117.765 + / - 94.406 ops/usCopy the code

Math:: SQRT is expensive, but the code in doSth is eliminated because the a variable has no effect. To keep the code from being erased, simply return a as a value or use the Blackhole object provided by the JMH:

@Benchmark
public void noCode(a) {}

@Benchmark
public void receiveByBlackHole(Blackhole blackhole) {
    blackhole.consume(Math.sqrt(x));
}

@Benchmark
public double returnRes(a) {returnMath.sqrt(x); }Copy the code

Execution result:

Benchmark Mode Cnt Score Error Units Example_00_DeadCodeElimination. NoCode THRPT 15 4099.014 + / - 110.712 ops/us Example_00_DeadCodeElimination. ReceiveByBlackHole THRPT 15 209.898 + / - 4.028 ops/us Example_00_DeadCodeElimination. ReturnRes THRPT 15 209.714 + / - 6.090 ops/usCopy the code

Test SpringBoot using the JMH

We want to use the JMH to test various Components of SpringBoot, such as the Controller or Service we wrote. Since Spring Beans automatically inject dependencies, we need to start SpringBoot in the test application.

package com.example.springbootjmhtest;

import com.example.springbootjmhtest.controller.TestController;
import org.openjdk.jmh.annotations.*;
import org.openjdk.jmh.runner.Runner;
import org.openjdk.jmh.runner.RunnerException;
import org.openjdk.jmh.runner.options.Options;
import org.openjdk.jmh.runner.options.OptionsBuilder;
import org.springframework.boot.SpringApplication;
import org.springframework.context.ConfigurableApplicationContext;

import java.util.concurrent.TimeUnit;

/**
 * <h3></h3>
 *
 * @author zohar
 * @version 1.0
 * 2021/8/4 16:37:23
 */
@BenchmarkMode(Mode.AverageTime)
@OutputTimeUnit(TimeUnit.MILLISECONDS)
@State(Scope.Benchmark)
public class SpringBootBenchMark {
    private ConfigurableApplicationContext springContext;
    private TestController testController;

    public static void main(String[] args) throws RunnerException {
        Options options = new OptionsBuilder()
                .include(SpringBootBenchMark.class.getSimpleName())
                .warmupIterations(3)
                .measurementIterations(3)
                .forks(3)
                .build();

        new Runner(options).run();
    }

    @Setup
    public void setUp(a) {
        / / start SpringBoot
        springContext = SpringApplication.run(SpringbootJmhTestApplication.class);
        // Load the Bean, which will automatically inject the AService and BService
        testController = springContext.getBean(TestController.class);
    }

    @TearDown
    public void tearDown(a) {
        springContext.close();
    }

    @Benchmark
    public void testStringBuffer(a) {
        // The controller calls the AService method
        testController.testAService();
    }

    @Benchmark
    public void testStringBuilder(a) {
        // The controller calls the BService methodtestController.testBService(); }}Copy the code

Test results:

Benchmark Mode Cnt Score Error Units SpringBootBenchMark. TestStringBuffer avgt 9 1.455 + / - 0.024 ms/op SpringBootBenchMark. TestStringBuilder avgt 9 1.358 + / - 0.029 ms/opCopy the code