Alibaba’s practice and optimization of open source pressure measurement tool JMeter

Authors: Ling Ran, Jian Quan

Apache JMeter [1] is an open source pressure measurement tool owned by Apach. It was founded in early 1999 and has a history of more than 20 years. JMeter is one of the mainstream open source pressure measurement tools with rich functions and a large community of users.

Performance tests are usually conducted before the launch of a new system or large-scale activities (such as e-commerce promotion and Spring Festival activities) to verify system capabilities and help troubleshoot performance bottlenecks.

A manometry activity can be roughly divided into several steps:

1. Scenario configuration. Configuring the pressure test scenario simulates the interaction between a user (service) and the system.
2. Pressure test execution. Start pressure test at specified pressure level.
3. Pressure monitoring analysis. Pressure measurement usually focuses on pressure RPS, success rate, service response time (RT), network bandwidth and other key indicators.
4. Report summary. Disclose whether the system capacity meets the requirements, and record the evolution and optimization process of the system performance.

Native JMeter implements pressure measurement

Edit the pressure test script on the GUI page of JMeter and click the Start button to debug the JMeter script. For detailed operations, please refer to JMeter official website [1].

For simple scenarios that require a small amount of concurrent testing, JMeter local testing can meet the requirements. However, with the increase of Internet users, the demand for the system to bear more concurrency is increasing day by day, and the pressure capacity of a single JMeter pressure machine has a certain limit, so it is necessary to use multiple pressure machines to improve the pressure capacity of JMeter, which requires the use of JMeter distributed pressure function.

JMeter distributed pressure measurement requires users to manage and maintain multiple machines. Note the following points when using JMeter:

• The firewall of the press is closed or the correct port is open. SSL is set or disabled for RMI.
• All the presses are on the same subnet. If IP addresses 192. XXX or 10. XXX are used, the servers reside on the same subnet.
• All pressers use the same version of JMeter and Java.
• All pressers have copied sliced CSV data files, dependent JAR packages, etc.
• During the pressure measurement, monitor the normal flow of the pressure machine and ensure that the pressure is consistent with the configuration.
• The collection of monitoring data before pressure is configured to facilitate the generation of reports after pressure measurement.

This shows that JMeter’s distributed pressure measurement requires coordination of resources, preparation and maintenance of the pressure engine, which is inefficient for the personnel performing the pressure measurement.

JMeter practices on the cloud

Alibaba has a variety of business forms, each of which is served by a series of distributed technical systems. With the rapid development of business, especially under the scenario of marketing activities such as Double 11, it has become a major technical problem to accurately evaluate the service capability of the whole business site.

In the process, we built our own full-link manometry system to address more complex and diverse manometry requirements and exported this technology to our performance test PTS, while supporting native JMeter manometry.

Practice JMeter through the console

Upload script

Open the home page of the PTS Console [2], and choose Test Center > Create Scenario > JMeter Test in the navigation tree to create a JMeter test scenario. Enter the scenario name, for example, jmeter-test. Scenario Configuration Page Click upload file to upload the test. JMX script that has passed the local test.

Pressure on the configuration

On the pressure configuration page, set the number of concurrent requests to 50 and the pressure test duration to 2 minutes.

Save the pressure measuring

Click Save to pressure, click OK in the pop-up prompt box, PTS starts to execute JMeter script in cloud engine to initiate pressure.

The page in the pressure test is as follows:

Note: Due to differences in machine configuration and network environment (the default value of the PTS press is 4-core 8G and BGP multi-line public network), the results of the PTS pressure test may be different from those of the local pressure test. In addition, the pressure configuration on the PTS overrides the configuration in the original script, which doesn’t matter whether it writes the fixed configuration or uses the JMeter attribute configuration.

Practice JMeter through OpenAPI

Cloud computing will evolve into a social infrastructure like hydropower and coal. OpenAPI is like a fast pipeline, connecting enterprises and Ali Cloud, delivering resources to enterprises continuously. IT has become a consensus that using cloud computing to build IT infrastructure is the future. OpenAPI is an important window for cloud service opening. Cloud services without OpenAPI will be difficult to be integrated by customers’ systems, which not only affects user experience but also restricts the development of cloud vendors. Similarly, in the field of pressure measurement, with the increasing diversification of pressure measurement requirements, more and more users want to inherit the cloud pressure measurement capabilities to their own systems, or according to their own business systems, customized pressure measurement platform, so as to achieve automatic customization of pressure measurement requirements.

The following code implements the One-click start of the JMeter pressure test scenario using PTS OpenAPI and viewing the pressure report after the pressure test is complete.

Introduce POM dependencies

<! -- Create the entity classes required by the PTS scenario, --> <dependency> <groupId>com.aliyun</groupId> <artifactId>pts-api-entity</artifactId> The < version > < / version 1.0.1 > < / dependency > <! --PTS Java SDK dependencies. --> <dependency> <groupId>com.aliyun</groupId> <artifactId>pts20201020</artifactId> </dependency> <! -- Ali Cloud core library. --> <dependency> <groupId>com.aliyun</groupId> <artifactId> aliyun-java-sdK-core </artifactId> <version>4.5.2</version> </dependency>Copy the code

Copy the following code

import com.aliyun.pts20201020.Client; import com.aliyun.pts20201020.models.*; import com.aliyun.teaopenapi.models.Config; import java.util.ArrayList; import java.util.List; import java.util.Map; public class StartingDemo { public static void main(String[] args) throws Exception { Client client = getClient(); String sceneId = createScene(Client); String reportId = startTesting(client, sceneId); // Start scenario String reportId = startTesting(client, sceneId); Int count = 0; // Check if a report has been generated while (! HasReport (client, reportId) && count++ < 20) {// if the report is not generated, wait (30s) before query. // wait for thread. sleep(30 * 1000) according to the test time. } // Check the report getJMeterReport(client, reportId); } private static boolean hasReport(Client client, String reportId) throws Exception { ListJMeterReportsRequest request = new ListJMeterReportsRequest(); Request. SetPageNumber (1); request.setPageSize(1); Request. SetReportId (reportId); ListJMeterReportsResponse response = client.listJMeterReports(request); return response.getBody().getReports().size() > 0; } private static void getJMeterReport(Client client, String reportId) throws Exception {// Viewing machine logs GetJMeterLogsResponse GetJMeterLogsResponse = getJMeterLogs(Client, reportId); List<Map<String, ? >> logs = getJMeterLogsResponse.getBody().getLogs(); / / see the sampler aggregated data GetJMeterSampleMetricsResponse getJMeterSampleMetrics = getJMeterSampleMetrics (client, reportId); List<String> sampleMetricList = getJMeterSampleMetrics.getBody().getSampleMetricList(); / / check sample log GetJMeterSamplingLogsResponse getJMeterSamplingLogs = getJMeterSamplingLogs (client, reportId); List<String> sampleResults = getJMeterSamplingLogs.getBody().getSampleResults(); } private static GetJMeterSamplingLogsResponse getJMeterSamplingLogs(Client client, String reportId) throws Exception { GetJMeterSamplingLogsRequest request = new GetJMeterSamplingLogsRequest(); Request. SetPageNumber (1); request.setPageSize(10); // Conditional setting request.setReportId(reportId); GetJMeterSamplingLogsResponse response = client.getJMeterSamplingLogs(request); return response; } private static GetJMeterSampleMetricsResponse getJMeterSampleMetrics(Client client, String reportId) throws Exception { GetJMeterSampleMetricsRequest request = new GetJMeterSampleMetricsRequest(); // set the reportId request.setReportId(reportId); GetJMeterSampleMetricsResponse response = client.getJMeterSampleMetrics(request); return response; } private static GetJMeterLogsResponse getJMeterLogs(Client client, String reportId) throws Exception { GetJMeterLogsRequest request = new GetJMeterLogsRequest(); Request. SetPageNumber (1); request.setPageSize(10); // Queried pressure test engine index request.setReportId(reportId); GetJMeterLogsResponse response = client.getJMeterLogs(request); return response; } private static String startTesting(Client client, String sceneId) throws Exception { StartTestingJMeterSceneResponse startTestingSceneResponse = startTestingScene(client,  sceneId); String reportId = startTestingSceneResponse.getBody().getReportId(); return reportId; } private static StartTestingJMeterSceneResponse startTestingScene(Client client, String sceneId) throws Exception { StartTestingJMeterSceneRequest request = new StartTestingJMeterSceneRequest(); request.setSceneId(sceneId); StartTestingJMeterSceneResponse response = client.startTestingJMeterScene(request); return response; } private static String createScene(Client client) throws Exception { SaveOpenJMeterSceneRequest request = new SaveOpenJMeterSceneRequest(); / / define scene SaveOpenJMeterSceneRequest. SaveOpenJMeterSceneRequestOpenJMeterScene scene = new SaveOpenJMeterSceneRequest.SaveOpenJMeterSceneRequestOpenJMeterScene(); SetSceneName ("test"); // Set the scene name scene.setscenename ("test"); // Set the file list, Including JMeter script lines, JMeter pressure dependent jar package, configuration data file List, such as < SaveOpenJMeterSceneRequest. SaveOpenJMeterSceneRequestOpenJMeterSceneFileList > fileList = new ArrayList<SaveOpenJMeterSceneRequest.SaveOpenJMeterSceneRequestOpenJMeterSceneFileList>(); / / Settings file attributes Need to set the file name and file public access to oss address SaveOpenJMeterSceneRequest. SaveOpenJMeterSceneRequestOpenJMeterSceneFileList testFile  = new SaveOpenJMeterSceneRequest.SaveOpenJMeterSceneRequestOpenJMeterSceneFileList(); testFile.setFileName("baidu.jmx"); testFile.setFileOssAddress("https://pts-openapi-test.oss-cn-shanghai.aliyuncs.com/baidu.jmx"); fileList.add(testFile); scene.setFileList(fileList); SetConcurrency to 1 million scene.setconcurrency (1000000); SetAgentCount (2000); setAgentCount(2000); setAgentCount(2000); setAgentCount(2000); setAgentCount(2000); SetDuration (60 seconds scene.setduration (60); // Set the name of the test file, which must be included in the file list scene.settestfile (" bidu.jmx "); request.setOpenJMeterScene(scene); SaveOpenJMeterSceneResponse response = client.saveOpenJMeterScene(request); return response.getBody().getSceneId(); } private static Client getClient() throws Exception {// Enter your own AK/SK String accessKeyId = "AK "; String accessKeySecret = "sk"; Config config = new Config(); config.setAccessKeyId(accessKeyId); config.setAccessKeySecret(accessKeySecret); Client client = new Client(config); return client; }}Copy the code

Fill in your AK/SK

Fill in the correct AK/SK in the getClient of the above code

Click the start

Click the Main method to launch

Practice JMeter through plug-ins

For long-time Users of JMeter, learning a new pressure tool can take some time. As a result, PTS developed a PTS-JMeter plug-in to help JMeter users directly use PTS pressure resources without changing their original pressure behavior. The pTS-JMeter plug-in is almost invisible to the user and is used in the same way as native JMeter. Save/open the JMeter script and click to start the pressure test.

Download and install

Click the link to download the latest version jar package [3]

Copy the JAR package to the Lib /ext extension directory in the JMeter home directory

Click on the pressure test

Create a New JMeter script or open an existing JMeter script and click the Pts-JMeter start button to start the pressure test

Check the report

During the pressure measurement, the JMeter GRAPHICAL interface displays some pressure indicators. Users can check the pressure process on the console at any time. After the pressure test is complete, PTS generates a more detailed pressure test report, which is kept for 30 days by default and can be viewed at the console at any time.

other

More details on how to use the PTS-JMeter plug-in can be found in the PTS help documentation [4].

Pressure monitoring analysis

Performance testing is not simply about initiating pressure, but monitoring and analysis of pressure loads (RPS, network bandwidth, etc.) and business performance (RT, success rate, etc.) is also an important part of pressure testing activities. Each request node (Sampler) in the JMeter script can have a name with business meaning (such as Home and Download Page), which we can call a business API. JMeter monitoring statistics are summarized by business API names. For example, two request nodes with the same name are summarized into one business API. Note that different service API nodes have different names when you configure scripts.

Business API stress load and performance

In practice, the statistics for different business apis can vary significantly (for example, viewing an ITEM RT is usually much faster than submitting an order), so PTS defaults to displaying separate statistics for each business API (as shown in the home and Download pages in the pressure test above).

The PTS data for each time point in the manslaughter is recorded in the background, resulting in a complete and intuitive manslaughter report. Click the service API real-time monitoring trend chart button to view the corresponding RPS, success rate, response time, network bandwidth and other monitoring data trend charts.

Service API sampling logs

Many times we also want to see details about the execution of a specific request. If 1% of the requests fail, you need to view the complete request and response information to locate the failure cause. When testing scripts in JMeter GUI, you can add the View Results Tree to View the details of a single request. However, when performing a stress test, recording the details of each request is unnecessary and consumes resources, which affects pressure performance.

Ali Cloud PTS takes a compromise approach, where the pressure engine samples each business API (pressure Sampler) at intervals to record details of a successful and failed request (if any). In the pressure test or the pressure test report page, click the “View Sampling log” button to query the recorded request sampling information, and support search and filtering based on service API (pressure test Sampler), response status (success), request time, etc.

Click View Details to see details for an individual request. At present, there are two display templates for detailed information: general and HTTP. The HTTP display template can display more friendly layout for HTTP requests, including request URL, request method, return code, complete request header, request body, response header, response body and so on.

Because only text content is displayed on the page, the request or response body may be displayed as garbled characters if it contains contents that cannot be recognized as text, such as images. In addition, when the request or response body is large, the corresponding content may be truncated.

JMeter log

When JMeter scripts are executed locally, logs are logged to the jmeter.log file by default. When JMeter scripts are executed on the PTS, JMeter logs can be queried on the JMeter log page in real time and filtered by log level, time, or thread name.

JMeter logs are used to locate errors reported during script execution. Some plug-ins may output important information through JMeter logs, or you can print logs directly in code such as Groovy scripts.

The report concludes

After the pressure test, PTS summarizes the monitoring data to form a pressure test report. The user analyzes and evaluates whether the system performance, such as RPS, success rate and RT (response time), meets expectations based on the pressure test report. In addition, it helps users troubleshoot and analyze performance bottlenecks in service systems.

You can query the list of historical manometry reports on the PTS manometry report page.

Click View Report to open the report details. The pressure test report is saved on PTS for 30 days by default. You can click the report Export button to export and save the PDF version of the pressure test report to your local PC. The pressure report summary includes the pressure test execution time, RPS, RT, and success rate. Scenario details include monitoring statistics of all scenario dimensions and service API dimensions.

PTS is much simpler to use than manually executing JMeter scripts from the command line, provides simple and intuitive monitoring, and provides massive pressure capabilities.

A link to the

[1] Apache JMeter 官网

jmeter.apache.org/

Pts.console.aliyun.com/?spm=5176.7…

Pts.console.aliyun.com/#/overviewp…

Jmeter-pts-testing-version.oss-cn-shanghai.aliyuncs.com/plugins/Ali…

[4] PTS Help Document: