Discuss the concurrency model differences between ab, WRK, JMeter and Locust

What are you choosing when you choose a manometry tool?

Stress test is a relatively “technical content” of test engineer’s daily work, many people are full of curiosity about this work. Except for a few special scenarios, you have to develop your own pressure testing scripts, most of the pressure testing work can be carried out with mature pressure testing tools. There are many tools available, both open source and commercial, but here are some common ones:

In addition to commercial pressure tools such as LoadRunner, familiarity is the most important factor for most testers when selecting a pressure tool. This familiarity is often due to past work experience, recommendations from colleagues, the number of online tutorials, scripting languages, etc. For example, when I started using Locust many years ago, it was because I was personally good at Python, even though there were few Chinese tutorials at the time.

However, after using Locust for some time, around the middle of 2015, I realized that Locust, as a pressure tool, seemed to generate far less pressure than JMeter and so BEGAN to focus on the concurrency model behind the pressure tools to understand the operation logic of different pressure tools and try to explain the performance differences I saw.

Synchronous, asynchronous, blocking, and non-blocking

To talk about the concurrency model, we cannot escape from the following four terms:

• Synchronous
• Asynchronous
• Obstruction (Blocking)
• Nonblocking

I would also like to point out that there are very few Chinese materials that you can find through search engines to explain these four concepts accurately.

I’m not going to try to explain the difference between these four words, just to mention some points that most sources ignore:

• To distinguish synchronous from asynchronous, it is important to be clear about the layers, such as framework, user space, kernel, IO model
• A synchronous call is blocked if it does not return a result
• The asynchronous call is initiated and returns directly, without a doubt, the process is not blocked

Interprocess communication is described in Operating System Concepts [9th Edition]

In other words, from the perspective of process communication latitude, blocking and non-blocking are synonyms with synchronous and asynchronous, but it is necessary to distinguish sender and receiver:

• Blocking the send
• Non-blocking send
• Blocking the accept
• Non-blocking acceptance

The above different types of sending methods and different types of receiving methods can be freely combined

In addition, we know that Linux has five I/O models:

1. Blocking I/O

2. Nonblocking I/O

3. I/O multiplexing

• select

• poll

• epoll

• AIO

All of the above are synchronous IO. Only the fifth model is asynchronous IO

With these concepts in mind, let’s move on to the concurrency model behind the pressure tools mentioned in the title of this article

Ab and JMeter based on multi-thread concurrency

Ab and JMeter, developed in C and Java respectively and based on multi-threaded concurrent model, are also the most popular open source pressure tools at present. Their working principles are similar, as shown below:

• Whether ab or JMeter, its so-called virtual users (vusers) correspond to one thread
• In a single thread, each query is invoked synchronously, with the next request waiting for the previous one to complete
• A request (query) is divided into three parts:
• Send-the pressure end sends data until the pressure end receives data
• Wait – The receiving end starts until the end of the processing
• Recv – The pressure end returns data until the pressure end receives it
• The concept of waiting time between two consecutive requests in the same thread is the blank space in the diagram

In a multi-threaded concurrency model, is it possible to produce more stress by increasing the number of threads?

The answer is no.

In fact, a process can only execute one thread at a time, and concurrency refers to switching threads in the process to achieve what looks like multiple task concurrency. However, thread context switching has a high cost, and too many threads can lead to a serious performance degradation.

From an application point of view, multi-threaded concurrency models often require maximum concurrency parameters, which can be difficult to handle if the pressure scenario is constantly pressurized.

Why is WRK faster than AB?

WRK is a pressure measuring tool similar to AB, also developed in C language, but more “modern” :

wrk is a modern HTTP benchmarking tool capable of generating significant load when run on a single multi-core CPU. It combines a multithreaded design with scalable event notification systems such as epoll and kqueue.

We explain the concurrency model of WRK through its execution parameters:

• connections: Connections to keep open
• threads: Number of threads to use

WRK connections close to ab concept of concurrency, see: https://github.com/wg/wrk/issues/205

Whereas in AB, concurrency is the number of threads, WRK has a separate threads parameter.

WRK uses the Epoll-based I/O reuse model to improve its throughput. See the following figure (select in the figure). To reduce context switching, WRK recommends that the number of threads be equal to the number of CPU cores, that is, each processor runs only one thread. This completely eliminates the consumption of thread switching.

Note that in this model, the pressure side is blocking when it initiates a request, waiting for multiple requests to be sent together, but is non-blocking when it receives them.

Is WRK based on this concurrency model necessarily better than ab, JMeter, etc.?

The answer is not necessarily. I/O multiplexing is not a silver bullet, and it is not necessarily suitable for all manometry scenarios. WRK and JMeter are compared on the external network. You can see which scenarios are more suitable for these two pressure measurements:

The reason WRK is not more popular than JMeter may be the lack of a GUI, as well as the fact that its scripts use Lua and are relatively niche.

Libuv behind Locust, Artillery

Locust is a distributed pressure measuring tool developed in Python, which is popular in China in recent years. Locust is not a Python based multithreading, but a Coroutine (provided by Gevent), which uses libev or libuv as eventloop.

The so-called evetloop can be better understood by looking at this diagram:

Less familiar is another pressure tool, Artillery, which is based on Node.js.

Why is it introduced here alongside Locust? Those of you who know Node.js already know why: the underlying Libuv provides very powerful asynchronous IO capabilities.

Its event loop logic:

Want to understand libuv can refer to the official document, not opened here: http://docs.libuv.org/en/v1.x/design.html

However, in my experience with Locust, the concurrency is incredibly low. Many people have reported this issue to the authorities (including myself), and I’m not sure if this issue has been eradicated by the current version. When in doubt, use Locust with caution, including its distributed execution mode.

Compare these manometry tools horizontally

While I was searching for information for this article, I came across a comprehensive comparison of known pressure measurement tools on the K6. IO blog. https://k6.io/blog/comparing-best-open-source-load-testing-tools

My view on these open source pressure tools is:

• JMeter can handle most pressure testing scenarios, even if the multithreaded concurrent model is not the most efficient
• If you are in a compression scenario that requires constant upward pressure, try a compression tool based on an asynchronous API
• If you do not meet the above tools, please read on

one more thing – Ultron

In addition to the concurrency models described above, I’m sure many of you have heard of Goroutine under Golang:

• A Goroutine can be understood as a user-mode thread, and switching a Goroutine has no kernel overhead
• Memory footprint is small, thread stack space is usually 2M, goroutine stack space minimum 2K
• G-M-P scheduling model

Golang was born for concurrency, and the GO project has a high concurrency capability without optimization. Someone once joked that the services written by CPP programmers with ten years of experience using various black magic might not have the high concurrency capability of Golang project. (Just a joke, don’t take it too seriously)

If Goroutine is so strong, are there any open source pressure tools based on Goroutine?

Start packing

I here recommend ultron under this project: https://github.com/qastub/ultron

• Based on the high concurrency capability of Goroutine, in the comparison test with WRK, Jmeter and other tools, the concurrency capability exceeds JMeter and is second only to WRK
• Goroutine provides the semantics of synchronization functions and is easy to understand
• Requests (or transactions) are highly abstracted for easy access to various protocols
• The Result and Report event monitoring interfaces are provided to facilitate expansion. For example, built-in support for Influxdb can be used to generate real-time reports
• Support distributed execution

This article is written fearfully, with my current knowledge reserve, it is difficult to explain such a big topic clearly, the article inevitably has mistakes and omissions, please see the officer not hesitate to point out.

References:

• How to understand the difference between blocking non-blocking and synchronous asynchronous?

• JMeter VS WRK

• Comparing the best open source load testing tools since 2017!