I have thought about the various sources of error in the two statistical models and how to avoid these possible errors. Some specific framework improvements have been made, as shown in the following articles:

  • Performance test error analysis text version – on
  • Performance test error analysis text version – next
  • Performance test error statistical practice

Today, I will share the influence of various parameters and performance indexes on performance test errors in performance test statistics, and the effect of various error reduction methods, so as to know how to improve performance test in the future.

Demonstrate the Demo

I wrote a Demo that simulated normal single-interface performance tests. I used the ThreadLimitTimesCount

model and used the sleep() method to simulate requests and responses as follows:


/** * Performance test statistical error test class * by:FunTester */
class DeviationTest extends SourceCode {


    static void main(String[] args) {
        def ts = []

        10.times {
            ts << new FunTester(StringUtil.getString(10), 60)}new Concurrent(ts, "FunTester error Analysis").start()

    }

    private static class FunTester extends ThreadLimitTimesCount<String> {


        FunTester(String s, int times) {
            super(s, times, null)}@Override
        protected void doing(a) throws Exception {
            sleep(0.1)}@Override
        ThreadBase clone(a) {
            new FunTester(StringUtil.getString(10), times)
        }
    }

}

Copy the code

The following is a comparative analysis of the error sources mentioned earlier.

Due to the short time of the simulation, the standard demonstration will not be performed here. It was also difficult to count code runs at the MS level during my own tests of idling, so I also dropped comparisons of code run times.

The number of threads

First, let’s study the impact of the number of threads on performance test errors. Let’s start with a set of simulations with 20 threads and 50 requests at a fixed request time to see the results.

~ ☢ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ~ JSON ~ ☢ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ~ > {> (1). "rt" : 103, > 1). The "failRate" : 0.0, "Deviation ":"3.29%", > ①. "errorRate":0.0, > ①. "executeTotal":1000 > ①. "qps2":187.79342723004694, > ①. "total":1000, > ①. > ①. "endTime":"2021-04-05 15:41:04", > ①. > 1). "table":"eJztjzEKwkAQRfvA3mEOkEC0sMgBPIDkAgEHXHBXySYQSxElpVVKj2BK7xOIeAxX7SzcFUQZmeHDm2r+PBGAx4xLnaIpML+0bX869vXufNjDMIZ ilmM29TkBIhCvuyZolgttEFKpMIEqMpjLbA66VCGsIoVTmWlXh/sPJTU8biWDOAZlQpVVdh3Z9QMWjumajY0DWxs/1Db2p65Z28AzPKreLr43fhs3w58UsyE BsiEb/hHYkD7YkD7YkD7YkD7YkD7YkD7YkD7YkD7YkD7YkD7YkD7YkD5EcAVmaHb1 ">} ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ~ ~ ☢ ~ JSON ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~Copy the code

The average measurement time is 103ms, and the total error is 3.29%, which is not very high. Let’s take a look at a simulation with 40 threads and 50 requests for a fixed request time to see what happens.

~ ☢ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ~ JSON ~ ☢ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ~ > {> (1). "rt" : 103, > 1). The "failRate" : 0.0, > 1). "threads" : 40, > 1). The "deviation" : "2.7%" > 1). The "errorRate" : 0.0, > 1). "executeTotal" : 2000, > ①. "qps2":377.8575477045154, > ①. "total":2000, > ①. > ①. "endTime":"2021-04-05 15:43:24", > ①. "mark":"FunTester error analysis ", > 1). "table":"eJztz08KgkAUBvC94B3eARQUooUH6ADhBQQHEpopHANbRv9ctnLZEXLZfQSjYzTVrmiUCsbFNzz43Pje97Mt6vBGCxEymbH0WlXN+dQUu8vxQAO PsknKorjLCrItW39rzOR8JiSjMOEsoNyVLE2iKYkFd2jpchYnkWi70d6DJ4KeuwLf84hLh0d54PtD9fkHxa+vLgs1nSS9vFKXKzWfQnf1wy9rNW+xVdMt9mq + iUdTXdhWXW7U0Gvo2rRu7VPchT2oASGEEJqvASGEEJqvASGEEJqvASGEEJqvASGEEJqvASGEEJqvASGEWuENvcRZ6Q = = ">} ~ ☢ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ~ JSON ~ ☢ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ~ ~ ☢ ☢ ~ ~ ~Copy the code

The average measurement time is 103ms, and the total error is 2.7%, which can be said to be very similar. Try a group of 100 threads again, and other conditions remain unchanged.

~ ☢ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ~ JSON ~ ☢ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ~ > {> (1). "rt" : 102, > 1). The "failRate" : 0.0, "Deviation ":"3.7%", > ①. "errorRate":0.0, > ①. "executeTotal":5000, > ①. "qps2":944.1087613293051, > ①. "total":5000, > ①. > ①. "endTime":"2021-04-05 15:44:38", > ①. "mark":"FunTester error analysis 051544" > 1). "table":"eJztzz0KwjAUwPE9kDu8A7RQF4cewANILxAwYMBEaVqoo4jS0amjR7Cj9ylUPIYpblXaih/w5IUHv0x5+XMGA84kNZG0iYyvZVmfT3W+vxwPMAo CSOaxFLMhb3DGu3dNpV0tjZUQKS1DyHwrYyUWYFLtwdrXcqaE6dvR/w+tDNzfCpsAbT0tMncdu+sHKrpOVWzdPLBz8xK5mxacVcXGDbR5urGLt77xRZrC326 KQiqkQir8P6gQP1SIHyrEDxXihwrxQ4X4oUL8UCF + qBA/VIgfKsQPFeKHsxsIDnrM ">} ~ ☢ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ JSON ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~Copy the code

This time the average response time is more accurate, but the error is higher, so let’s try another set of 200 threads.

~ ☢ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ~ JSON ~ ☢ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ~ > {> (1)."rt":103> 1)."failRate":0.0> 1)."threads":200> 1)."deviation":"3.2%"> 1)."errorRate":0.0> 1)."executeTotal":10000> 1)."qps2":1879.6992481203008> 1)."total":10000> 1)."qps":1941.7475728155339> 1)."startTime":"The 2021-04-05 15:45:27"> 1)."endTime":"The 2021-04-05 15:45:33"> 1)."mark":"FunTester error analysis 051545"> 1)."table":"eJztjz0KwkAQRvvA3mEOkEC0MwfwAJILBFww4K6STUBLxZ+UVik9gpbeR4h4DEctRDBmQVFGZ/ngTbPzzRMOWLx2pkNpUpkct9tytynzxWG9gqbvQ9pLZNS 12SEc8byrI81woI2EMFYygJFnZBJHfdCZcmHsKdmNI13XUX+HijVcdwUNFFDGVdEIxxaOb7B49e2LHGNlcvsywdRghqnCHGOHy233EM6+mGKgAvbLl5gqPCj +GM6G3+xnQzZkQzb8BbAhfbAhfbAhfbAhfbAhfbAhfbAhfbAhfbAhfbAhfbAhffyD4QlUrRlG">} ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ~ JSON ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~Copy the code

At present, the number of threads has little effect on the error and can be almost ignored.

Time & times

Let’s take a look at the influence of test time and number of requests. Since I choose a model with fixed number of requests this time, the main research parameter is still the number of requests.

First, a set of simulation results of 20 threads and 20 request times in fixed request time.

~ ☢ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ~ JSON ~ ☢ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ~ > {> (1). "rt" : 105, > 1). The "failRate" : 0.0, > 1). "threads" : 20, > 1). The "deviation" : "5.28%" > 1). The "errorRate" : 0.0, > 1). "executeTotal" : 400, > ①. "qps2":180.42399639152006, > ①. "total":400, > ①. > 1. "endTime":"2021-04-05 15:48:21", > 1. "mark":"FunTester error analysis ", > 1). "table":"eJztkksKwjAYhPdC7/AfwEIrrnoADyC9QKEBAyZKU0GX4nPpqkuPoEvvU6h4DOMDQbDaWjFdzM/AZDUzH8RqUIHrjKTPVMyi036fHXbZenncbqj lUNyLWBAWiSCrYb3v6jI1HEjFyOeCeTS2FYt40Cc5Ek2a2IKFPJCfOj7vEFzSLctzHYeEaopg7Lntln7+gKLqpclaqxDJ9xULrf80XSvQhKavmtJkqpVnM60 8 u / / w0rbSyrPn5Wky16KHlY57kVonuxDWYAYIQQhC8zNACEIQmp8BQhCC0PwMEIIQhOZngBCEIDQ A4SVCM8WpxSi ">} ~ ☢ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ~ JSON ~ ☢ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ~ ~ ☢ ☢ ~ ~ ~Copy the code

The average response time was 105ms and the error was 5.28%. Now increase the number of requests to 50.

~ ☢ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ~ JSON ~ ☢ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ~ > {> (1). "rt" : 104, > 1). The "failRate" : 0.0, "Deviation ":"2.79 ", > ①. "errorRate":0.0, > ①. "executeTotal":1000 > ①. "qps2":186.9508319312021, > ①. "total":1000, > ①. > ①. "endTime":"2021-04-05 15:49:48", > ①. "mark":"FunTester error analysis ", > 1). "table":"eJztz0sKwjAUBdB5oXt4C1BoxVEX4ALEDQQaMGCiNBbqUPx16MihS9Ch+xEUl2HUgeAvhRZi4YYHN4M27x7fowKnk6oe12OeXPb702F3ypfn7Zp aAY37CWdxkSfI9/zfu7pcj4ZKc+oJySPKmpongg1IpbJBk6bksWDKtsPeQwpFj7eiMAhI6oZkWRS2Q3OtQFH2HDe5mUISbMGWum05bqZm3mJmxhJzM5ZYmKk S7uU/hu + 9 frsy84zvf9ylbsi qAEhhBC6rwEhhBC6rwEhhBC6rwEhhBC6rwEhhBC6rwFhKeEVVbcQ1Q = = "/ >} ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ JSON ~ ☢ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~Copy the code

The average response time and error has gone down, now up to 100 requests, and look at the results.

~ ☢ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ~ JSON ~ ☢ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ~ > {> (1). "rt" : 103, > 1). The "failRate" : 0.0, "Deviation ":" 1.2px ", > ①. "errorRate":0.0, > ①. "executeTotal":2000, > ①. "qps2":191.2411550965768, > ①. "total":2000, > ①. > ①. "endTime":"2021-04-05 15:51:06", > ①. "mark":"FunTester error analysis 051550", > 1). "table":"eJztzzEKwjAUBuC9kDu8A7RQHRx6AA8gvUDAgAETpWmhjiJKR6eOHsGO3qdQ8RhG3EqxlQpx+MOD/015/8c8GvDmmY6FSUXyqKrmdm2K0/1ypml I6SoRfDnkC2Ie+3xrIcx2o42gWCoRUR4YkUi+Jp0pn3aBEkvJdd+N/h5Kanr/FU3CkJTxFc/tOrPrDxTdry4PdnriaOerKOy0gnl1ubdD7Rh5v+OUo3gJ/6A GhBBC6L4GhBBC6L4GhBBC6L4GhBBC6L4GhBBC6L4GhBBC6L4GhBBC6L4GhKOET / 9 XGKK = ">} ~ ☢ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ~ JSON ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~Copy the code

Average response times and errors were further reduced, and it seemed intuitive that increasing the number of requests would reduce errors. Let’s do another set of 200 request tests.

~ ☢ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ~ JSON ~ ☢ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ~ > {> (1). "rt" : 103, > 1). The "failRate" : 0.0, "Deviation ":"0.69%", > ①. "errorRate":0.0, > ①. "executeTotal":4000, > ①. "qps2":192.8268414963363, > ①. "total":4000, > ①. > ①. "endTime":"2021-04-05 15:52:44", > ①. "mark":"FunTester error analysis 051552" > 1). "table":"eJztzzEKwjAUBuC9kDu8A7RQHRx6AA8gvUDAgAETpWmhjiJKR6eOHsGO3qdQ8RhGRBBBW7EQhz88+N+U93/Mow5vnOlYmFQkl6pqTsem2J0Pexq GlM4SwaddviDmsc+3JsIsF9oIiqUSEeWBEYnkc9KZ8mkVKDGVXLfdaO+hpKb7X9EgDEkZX/HcriO79qB4++pyY6cltna+isLOI5hXl2s79Br9HH4+5Shuwj+ OASGEELqvASGEELqvASGEELqvASGEELqvASGEELqvASGEELqvASGEELqvAeFPwisab36D ">} ~ ☢ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ JSON ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~Copy the code

I believe the conclusion is obvious. Increasing the number of test requests can effectively reduce errors.

Request time

Now let’s simulate the effect of request time on error. Intuitively, the longer the request time, the smaller the error, because if the system error is constant, the longer the time, the smaller the percentage of error.

Start with a set of 50ms tests with 20 threads and 100 requests.

~ ☢ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ~ JSON ~ ☢ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ~ > {> (1). "rt" : 52, > 1). The "failRate" : 0.0, "Deviation ":"3.99 ", > ①. "errorRate":0.0, > ①. "executeTotal":2000, > ①. "qps2":369.2762186115214, > ①. "total":2000, > ①. > ①. "endTime":"2021-04-05 15:56:42", > ①. "mark":"FunTester error analysis 051556", > 1). "table":"eJztz08KgkAYBfC94B2+AyhIUAsP0AGiCwgOJDRTOAq27C8uW7nsCLnsPoLRMRpqV6QTSRP05MFzNd/72RZpfMNUjJlMWHwpy/p0rPPd+bCnnkf JJGZBqPME2ZbdfGvE5HwmJKNxxJlPmStZHAVTEil3aOFyFkaBaLuhMYRHgu6P+X2PuHR4kPn9gfrrQvHeVxVrFb3KVfSEDydWKq9qq9JSt7v69bSwKpYqetX 9 na/Ltqpio0LN9d1RnQt/YAaEEEJofgaEEEJofgaEEEJofgaEEEJofgaEEEJofgaEEEJofgaEEP658ApAgBUd ">} ~ ☢ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ~ JSON ~ ☢ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ~ ~ ☢ ☢ ~ ~ ~Copy the code

Let’s increase the response time to 100ms.

~ ☢ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ~ JSON ~ ☢ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ~ > {> (1). "rt" : 103, > 1). The "failRate" : 0.0, "Deviation ":"1.25%", > ①. "errorRate":0.0, > ①. "executeTotal":2000 > ①. "qps2":191.75455417066155, > ①. "total":2000, > ①. > ①. "endTime":"2021-04-05 15:58:10", > ①. "mark":"FunTester error analysis 051558", > 1). "table":"eJztzzEKwjAUgOE9kDu8A7RQHRx6AA8gvUCgAQMmStNCHUWUjk4dPYIdvU+h4jGMiOBSW6EoT1548GXKy88Z9DjTzETSpjK5VlVzPjXF/nI8wDi AdJ5IEfd5Ajjj73fNpF0tjZUQKS1DyH0rEyUWYDLtwdrXMlbCdO3o/odWBh5vhaMgAG09LXJ3nbjrABWtpy63bjrYufmIws0Tzupy4wZaGGb/68Zvcy/85X4 QpEIqpMJ/gArxQ4X4oUL8UCF + qBA/VIgfKsQPFeKHCvFDhfihQvxQIX44uwGetH51 ">} ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ JSON ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~Copy the code

The error goes down by half and says, again, the intuition is right. Now increase the response time to 200ms.

~ ☢ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ~ JSON ~ ☢ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ~ > {> (1). "rt" : 203, > 1). The "failRate" : 0.0, "Deviation ":"0.95%", > ①. "errorRate":0.0, > ①. "executeTotal":2000, > ①. "qps2":97.58477677482313, > ①. "total":2000, > ①. > ①. "endTime":"2021-04-05 15:59:46", > ①. "mark":"FunTester error analysis ", > 1). "table":"eJztzzEKwjAUgOE9kDu8A7RQOjj0AB5AeoFAHxgwUZoW6iha6ejk6BHs6H2Eiscw4CbW1sHhyQsPvpAhyS8FjFjT0qboCszvbdtdzl2zv50OEEd QzHNU2ZgrQAr5+a0ZutXSOoRUG0ygCh3mWi3AliaAdWgw08oOvTH8D6MtPO9K4igC4wKjKr+d+O2XFdfjxk8fWz997Pz0Ufv5LY2fF6R4e/xHcCF9uJA+XEg FLqQPF9KHC + nDhfThQvpwIX24kD5cSB8upA8X0ocL6cOF9JHiAVtAtsc = ">} ~ ☢ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ~ ~ ☢ ~ JSON ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~Copy the code

The error keeps going down, but it’s not going down as much as it was before, and it basically confirms what your intuition tells you, that the longer the response takes, the smaller the error.

Response time dispersion coefficient

In practice, the response time is not fixed, and for some results, the response time can vary greatly, even several times. Let’s look at the impact of response time dispersion on performance test errors.

I introduce a random (0,1) number to simulate the discrete coefficient of response time.

The following is a simulation of 20 threads with 100 requests, with a random sleep configuration of Sleep (0.1 + getRandomDouble() / 10) and a theoretical average sleep time of 150ms.

~ ☢ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ~ JSON ~ ☢ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ~ > {> (1). "rt" : 153, > 1). The "failRate" : 0.0, "Deviation ":"3.56 ", > ①. "errorRate":0.0, > ①. "executeTotal":2000, > ①. "qps2":126.07160867372667, > ①. "total":2000, > ①. > ①. "endTime":"2021-04-05 16:06:50", > ①. "mark":"FunTester error analysis 051606", > 1). "table":"eJztlU0KgkAYhveCd/gOoGDRygN0gOgCQgMJzRSOQS37d9nKZUfIZfcRjI7RB4EUWJo/afUNL3yzmvd5GH9UBTKs7lT0mXSZcwmC6HSMvO35sIe 2Ae7QYdYgyxGgKurrrh6Tk7GQDPo2ZybMdMkc2xqBmHIN5jpnA9sSaR3pHNwWcDvLbBkd4FLj1sxsGwZuS7DIv0J/gcGxweDwMJmM8lTddzyOUhtDf4nBscY k91XSHvc+96yUIvEu844CNKVylEAV+itMJTyF6GKgHaZKrpyU2V+gemkbwfkGdfw8fvbeC9M3njvFouD3uUk23+mQYPT2v/pbhqr8wh2lGTYAgwzJkAzrxyB DMiTD + jHIkAzJsH4MMiTDPze8AjKhbpE = ">} ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ~ JSON ~ ☢ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~Copy the code

Below, double the dispersion coefficient of random number, configure sleep(0.1 + getRandomDouble() / 5), and the theoretical average sleep time is 200ms.

~ ☢ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ~ JSON ~ ☢ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ~ > {> (1). "rt" : 202, > 1). The "failRate" : 0.0, > 1). "threads" : 20, > 1). The "deviation" : "5.3%" > 1). The "errorRate" : 0.0, > 1). "executeTotal" : 2000, > ①. "qps2":93.76465072667604, > ①. "total":2000, > ①. > ①. "endTime":"2021-04-05 16:10:54", > ①. "mark":"FunTester error analysis 051610", > 1). "table":"eJztlk0KwjAQRvdC7zAHUFAXQnsADyBeoGDAgonSKOjSf5euuvQIdul9CorHcEAUFKmxybQVEgaSVb730gmpUwGF0Z6ILpNjFl7j+Hw6nneby2E PzTqM+yHzeypbgFNx0rM6TI6GQjLoBpx5MK1JFgb+AMSEV2FW46wX+OJbxncOHgi47+U16i3gssr9qdd0XVwasNAaSbTAwmmHpWSTMWaN9YihS0uiJdZbDmF qEs2xPusRpyfRCkslk4RAXZmG4rVxNSc9Gs2jMEr0bMgtllmkzGQ/X5J8CY12Eg1l1rueJ+nzGMk6zyhxbt1oiLrYFtCiL+FRKxso/FCUcHo1oX8TcjX6z0+ SavUvN + QXszK9B1SGJcCwhtbQGhaPYQ2toTUsHsMaWsNUwxsm4B0O ">} ~ ☢ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ JSON ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~Copy the code

Sleep (0.1 + getRandomDouble() / 2) = 350ms

~ ☢ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ~ JSON ~ ☢ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ~ > {> (1). "rt" : 355, > 1). The "failRate" : 0.0, "Deviation ":"4.85%", > ①. "errorRate":0.0, > ①. "executeTotal":2000 > ①. "qps2":53.60493165371214, > ①. "total":2000, > ①. > ①. "endTime":"2021-04-05 16:13:51", > ①. "mark":"FunTester error analysis ", > 1). "table":"eJztlk1qwzAQhfcB32EOkEASSKE+QA5QcgFDBDFUSrESSJdtfpdZaZkjJMveR5DSY2Sg0EUJthyN5FlYPLBXet+T5GclHXAY46WaCL0Qxc/lcv0 6Xw+779MRhn1YzAqRTV2mgKSTlHu9CP02V1rAJJcihVVPiyLPXkEtZRfee1JM80xVeVRzyFzB71zpYDAEqbsyW6Wj5yd8JUjhNaz5QOHjgHJKQ2AT1M2aT9Q /q3CO1mxQ9+0CuVqzRZVahnB2t6V1t2aNquFLT+GJQEXicNZi0dQ+DAGJ/splj6JF8iEj3S1auhC7R0VY1t+MSO/+0XjSNruitYlpyjYqdezviYicL3YlPQu 8 bxowqa7 / JJxqxTMNw8rxSRTt5hEvFfuqeixZwDtr0 + lYYLQJ24RtwuYx2oReCW9YS + Wj ">} ~ ☢ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ~ JSON ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~Copy the code

Let’s double the dispersion coefficient and configure sleep(0.1 + getRandomDouble()) to see the effect.

~ ☢ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ~ JSON ~ ☢ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ~ > {> (1). "rt" : 590, > 1). The "failRate" : 0.0, "Deviation ":"10.51 ", > ①. "errorRate":0.0, > ①. "executeTotal":2000, > ①. "qps2":30.336584403961957, > ①. "total":2000, > ①. > ①. "endTime":"2021-04-05 16:21:47", > ①. "mark":"FunTester error analysis 051620", > 1). "table":"eJztl01qwkAYhvdC7vAdQMEfkJID9ADFCwQcMOBMS0bBLv136WqWHkGXvc+ApcfoQKkgtsn8fclEDC8kq3mfh8nMJFEDNK7nKRsQPiHZ1+l0/ji ed5vPwx66bZiMMpIMdYaAqBHld70Q/vbKOIFBSkkMsxYnWZqMgU1pE95blAzThBV1FGLQlMHPUHGn2wPKmzSZxZ32U189e7BwvaTYqeiYOFRsVdCbpFir/Na gtv1ZhdQoxVwltxKj+VL7fx9WuxQLFYNqvwTG5kgUdvUeKRwRvJA4ToZvIttXE5EKEcmFDB/Lmk6KlUoJaHaEZeNZUGocRpWT6p+bQRBLsVSpmNOMOkBkHfJ WsQvpK9oafBkEsRbdLK7 / Zepyuza57Ix1VLk1Ku1LozyruqxzQ7OarX8Tu/ubsVvDADAehg/DXMNv5swe1g = = ">} ~ ☢ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ~ JSON ~ ☢ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ~ ~ ☢ ☢ ~ ~ ~Copy the code

This increased the error so much that I began to wonder if the error did not increase in the last experiment by accident. But this has to do with the end of synchronization that we’ll talk about later. After I fix the error increase caused by the end of asynchrony, I will repeat the experiment of dispersion coefficient.

The basic conclusion is that the greater the discrete coefficient of interface request response time, the greater the error.

before & after

Next, the last link of this period is the influence of time-consuming operation in before and after methods on errors. You don’t actually have to do an experiment to get a conclusion, but the longer it takes, the bigger the error. The reason is that in the case of a certain request response error, the time consumed by the execution of before and after increases the total time T, and the number of requests is the same. The total time T divided by the total number of requests, and the QPS must be reduced. In addition, QPS=T/count(REq) itself is the calculation formula for low QPS, so the general conclusion is: When performing time-consuming operations in the before and after methods, the error is greater.

Set 20 threads, 100 request times, and fixed sleep time of 100ms.

~ ☢ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ~ JSON ~ ☢ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ~ > {> (1). "rt" : 103, > 1). The "failRate" : 0.0, "Deviation ":"9.82%", > ①. "errorRate":0.0, > ①. "executeTotal":2000 > ①. "qps2":175.1006828926633, > ①. "total":2000, > ①. > ①. "endTime":"2021-04-05 16:32:43", > ①. "mark":"FunTester error analysis 051632" > 1). "table":"eJztz0sKwjAQgOF9IXeYA7TQuuwBPID0AoUGDJgoTQt167NLV116BLv0PoWKxzA+ltUGtODAhIEvq0l+5oDFGecq4jrj6bWu2/OpLXeX4wFGPmT TlMeJzQpgDvv81oTrxVxpDpGQPITC0zwV8QxULl1YepInIlZ9b/T/QwoFz11h4PsgtSvjIgyC+/UHFd+epirNWJW83bAy08PGjB1bMx2/aqq1GTteKzrZm7H J8YMhYM6Ay / 8 cksqpfekhcvfdhfihqvxqix6oed9uib8qxa8v4ock8uof + KFC/FAhfphzAzQA + zM = ">} ~ ☢ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ~ JSON ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ~ ~ ☢ ~ ` ` ` now speak to increase ` 5 s ` sleep time, look at the results. ` ` ` shell ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ JSON ~ ☢ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ~ > {> (1). "rt" : 103, > 1). The "failRate" : 0.0, "Deviation ":"33.29%", > ①. "errorRate":0.0, > ①. "executeTotal":2000 > ①. "qps2":129.5336787564767, > ①. "total":2000, > ①. > ①. "endTime":"2021-04-05 16:33:25", > ①. "mark":"FunTester error analysis 051633", > 1). "table":"eJztz0EKgkAUgOG9MHd4B1DQVuEBOkB4AaGBBpopHAVbRlQuW7nsCLnsPoLRMRpqE4SOlIseveHB58aZ9zMHepxJpiKuU57cqqq5nJtifz0dYeR DOk94POtzBTCHdb815Xq1VJpDJCQPIfc0T0S8AJVJF9ae5DMRK9sb9j2kUPC8Kwx8H6R2ZZyHQTA2nwNUfHvqsjDTq+SfXqnLjRkLWzNvPFbo2KTlty52Ziw CzLTxutGnMGeYe34XKsQPFeKHCvFDhfihQvxQIX6oED9UiB8qxA8V4ocK8UOF + KFC/DDnDnuZ/C4 = ">} ~ ☢ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ JSON ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~ ~ ☢ ☢ ~ ~ ~Copy the code

Conclusion is consistent with the theoretical value, no further verification.

Next time, we will share how error is affected by exception handling, regex, asynchronous termination, and locking resources. I was gonna do a video presentation. Unfortunately, I don’t have much time to prepare. I will have a chance to have a live chat later.

FunTester.Tencent Cloud Author of the Year,Boss direct hire contract author.Official GDevOps media partner, non-famous test development.
  • Automated Testing trends in 2021
  • FunTester test framework architecture diagram
  • FunTester test project architecture diagram
  • Techniques for filtering automated test cases
  • Details in the test case
  • Probe into branch problems in link pressure measurement
  • Simplify test cases
  • JMeter parameter signature – Groovy script form
  • The thread synchronization CyclicBarrier class is applied at the performance test collection point
  • Performance test error analysis text version – on
  • Performance test error analysis text version – next