- 💜 Hello everyone, I am Chen Dasheng, from Chongqing, non-academic training background, halfway out of the programmer.
- 💜 advocates freedom, open source and sharing.
- 💜 has not yet nuggets any creative title, efforts to move forward, always become the light in their hearts, come on
Most testers are stressed out when it comes to performance testing. This is especially true for me. To do well in performance testing requires a vast body of knowledge, and the lessons learned from continuous practice are invaluable. And everyone has a unique understanding of performance testing, this time gradually uncover the mysterious veil of performance testing, combined with classroom learning and their own digestion and understanding, summed up some basic knowledge of performance testing, hoping to help you understand performance testing.
The basics of performance testing
On the premise of ensuring the correct implementation of functions, it evaluates potential performance problems through appropriate performance testing methods and strategies, and collects and inspeces various performance indicators of server applications and the use of server hardware resources.
Classification of performance indicators
From the Angle of measurement of performance test analysis, performance indicators can be collected and investigated from the following dimensions:
- System Performance Indicators
- Resource performance Indicators
- Middleware metrics
- Database Specifications
- Stability index
- Scalability metrics
- Reliability index
The following will introduce the common indicators of the above dimensions, their meanings and industry reference standards.
System Performance Indicators
Common system performance indicators refer to the following categories:
- The response time
- System processing capability
- throughput
- Number of concurrent users
- Error rate
The response time
Definition and explanation: Response time, abbreviated as RT. It is the time for the system to respond to a request. It can be understood as the time for the whole process from the time when the user initiates a request to the time when the client receives the response returned from the server. Intuitively, this metric is very consistent with people’s subjective perception of software performance, because it gives a complete record of how long the entire computer system takes to process requests.
General with pressure in the performance test by end of the server returns processing results to be pressure test for measuring time, unit for seconds or milliseconds, commonly due to a system usually provides many functions, and different function of processing logic also differ in thousands ways, so the response time of the different function also is not the same, even the same function in the case of different input data response time is not the same. So, when we talk about the response time of a system, we usually mean the average time of all functions of the system or the maximum response time of all functions of the system.
Industry reference standard: Different industries and different businesses have different acceptable response times. In general, for online real-time transactions:
- Internet enterprises: less than 500 milliseconds, such as Taobao business about 10 milliseconds.
- Financial enterprises: less than 1 seconds is better, some complex business less than 3 seconds.
- Insurance companies: under 3 seconds is better.
- Manufacturing: less than 5 seconds is preferred.
- Time window: different data volume results are different, in the case of large data volume, within 2 hours.
- Internet enterprises: less than 500 milliseconds, such as Taobao business about 10 milliseconds.
- Financial enterprises: less than 1 seconds is better, some complex business less than 3 seconds.
- Insurance companies: under 3 seconds is better.
- Manufacturing: less than 5 seconds is preferred.
- Time window: different data volume results are different, in the case of large data volume, within 2 hours.
It should be noted that the absolute value of the response time does not directly reflect the performance of the software. The performance of the software actually depends on the user’s acceptance of the response time.
System processing capability
Definition and explanation: System processing capability refers to the ability of system to process information by using system hardware platform and software platform. System processing capability is evaluated by the number of transactions that the system can process per second. Transactions can be understood in two ways: one is a business process from the perspective of business personnel; The second is a transaction application and response process from the system perspective. The former is called a business transaction process and the latter a transaction. Both transaction metrics can be used to evaluate the processing power of the application system.
In general, the system processing capacity is measured by the following indicators:
- HPS (Hits Per Second) : indicates the number of Hits Per Second, expressed in Hits Per Second.
- TPS (Transaction per Second) : The number of transactions processed by the system per Second.
- Query per Second (QPS) : indicates the number of queries processed by the system per Second. The unit is times per Second.
For Internet services, if some services have only one connection request, then TPS=QPS=HPS. In general, TPS is used to measure the entire business process, QPS is used to measure the number of interface queries, and HPS is used to represent the click request to the server.
Industry reference standard:
Whether TPS, QPS or HPS, this indicator is a very important indicator to measure the system’s processing capacity. The larger the better. According to experience, in general:
- Financial industry: 1000TPS~ 5000Tps, excluding internet-based activities
- Insurance industry: 100TPS~100000TPS, excluding internet-based activities
- Manufacturing industry: 10TPS~5000TPS
- Internet E-commerce: 10000TPS~1000000TPS
- Internet medium site: 1000TPS~ 5000tps
- Internet small websites: 500TPS~10000TPS
throughput
Definition and explanation: Throughput refers to the number of requests processed per unit of time by the system. For single-user systems, response time is a good measure of system performance, but for concurrent systems, throughput is often used as a performance indicator.
For a multi-user system, if the average response time for a single user is T, when you have n users, the response time per user is usually not N × T, but often much less than n× T (although in some special cases it can be larger or even much larger than N × T). Generally speaking, throughput is a relatively common indicator. If the maximum throughput of two systems with different user numbers and user usage patterns is basically the same, the processing capacity of the two systems can be judged to be basically the same.
Number of concurrent users
Definition and Explanation: The number of concurrent users refers to the number of users who log in to the system and perform service operations at a time.
Concurrent Users The maximum number of concurrent users for a long-connected system is the concurrent access capability of the system. For short-connected systems, the maximum number of concurrent users is not equal to the concurrent access capability of the system, but is related to the system architecture and system processing capability.
Compared to throughput, the number of concurrent users is a more intuitive but also more general performance metric. In fact, the number of concurrent users is a very inaccurate metric because different user usage patterns can result in different numbers of requests per unit of time from different users.
Error rate
Definition and Explanation: Error rate, abbreviated as FR, refers to the probability of failed transactions under system load. Error rate = (Number of failed trades/Total trades)*100%.
Industry reference standard:
Different systems have different requirements for error rates, but generally not more than six per thousand, that is, the success rate is not less than 99.4%
Resource performance Indicators
Common resource performance indicators are listed as follows:
- CPU
- memory
- Disk throughput
- Internet consumption
CPU
Definition and explanation: CPU, also known as central processing Unit (CPU), is a large scale integrated circuit, is the computing Core and Control Unit of a computer. Its main function is to interpret computer instructions and process data in computer software.
Industry reference standard:
CPU indicators refer to CPU usage, including the user, sys, wait, and Idle states.
- CPU utilization should be below the industry warning range, that is, less than or equal to 75%.
- CPU SYS % is less than or equal to 30%;
- CPU wait% is less than or equal to 5%;
memory
Definition and explanation: Memory is one of the most important parts in the computer. It is the bridge of communication with CPU. All programs in a computer are run in memory, so the performance of memory has a great impact on the computer.
Industry reference standard:
In order to maximize the use of memory, current operating systems store caches in the memory. Therefore, 100% memory utilization does not mean that there is a memory bottleneck. Whether the system has a memory bottleneck is mainly measured by SWAP (SWAP with virtual memory) SWAP space utilization. The SWAP space usage is less than 70%. Too many swaps may degrade the system performance.
Disk throughput
Definition and Explanation: Disk Throughput refers to the amount of data that can pass a Disk per unit of time without Disk faults.
Industry reference standard:
Disk indicators include read and write meGabits per second, disk busy rate, number of disk queues, average service time, average waiting time, and space usage. The disk busy rate directly reflects whether a disk has a bottleneck. Generally, the disk busy rate is lower than 70%.
Internet consumption
Network Throughput, referred to as Network Throughput, refers to the amount of data that can pass through the Network in a unit time without Network faults. The unit is Byte/s. The network throughput indicator measures the requirements of the system on the transmission capability of network devices or links. If the network throughput is close to the maximum transmission capability of the network device or link, upgrade the network device.
Industry reference standard: Network throughput refers to the number of Megabits of incoming and outgoing traffic per second. Generally, it cannot exceed 70% of the maximum transmission capacity of the device or link.
Middleware metrics
Commonly used middleware, such as Tomcat and Weblogic, mainly includes JVM, ThreadPool and JDBC, as follows:
| Level indicators | The secondary indicators | unit | explain |
|---|---|---|---|
| GC | The GC frequency | How many times per second | Garbage collection frequency of the Java VIRTUAL machine |
| GC | Full GC frequency | How many times per hour | Full Garbage collection frequency of the Java virtual machine |
| GC | Full Average GC duration | seconds | The average length of time used for complete garbage collection |
| GC | Full Maximum GC duration | seconds | Maximum length of time for complete garbage collection |
| GC | Heap usage | The percentage | Heap usage |
| ThreadPool | Active Thread Count | a | Number of active threads |
| ThreadPool | Pending User Request | a | Number of queued user requests |
| JDBC | JDBC Active Connection | a | Number of JDBC active connections |
Industry reference standard:
- The number of running threads cannot exceed the maximum value. Generally, when the system performance is good, the minimum number of threads is set to 50 and the maximum number is set to 200.
- The number of CURRENTLY running JDBC connections cannot exceed the maximum value. Generally, when the system performance is good, the minimum VALUE of JDBC is 50 and the maximum value is 200.
- GC frequency should not be frequent, especially for FULL GC. Under normal circumstances, when system performance is good, the MINIMUM heap size and maximum heap size of JVM should be set to 1024M respectively.
Database Specifications
Common database indicators such as MySQL mainly include SQL, throughput, cache hit ratio, connection number, etc., as follows:
| Level indicators | The secondary indicators | unit | explain |
|---|---|---|---|
| SQL | Time consuming | microseconds | SQL Execution Time |
| throughput | QPS | a | Number of queries per second |
| throughput | TPS | a | Transactions per second |
| shooting | Key Buffer hit ratio | Per cent | Index buffer hit ratio |
| shooting | InnoDB Buffer hit ratio | The percentage | InnoDB buffer hit ratio |
| shooting | Query Cache hit ratio | The percentage | Query cache hit ratio |
| shooting | Table Cache hit ratio | The percentage | Table cache hit ratio |
| shooting | Thread Cache Hit ratio | The percentage | Thread cache hit ratio |
| The lock | Waiting for the number of | time | Lock wait times |
| The lock | Waiting time | microseconds | Lock wait time |
Industry reference standard:
- SQL time as small as possible, generally microsecond level.
- The higher the hit rate, the better, generally not less than 95%.
- The lower the lock wait times, the better, the shorter the wait time, the better.
Stability index
Minimum stability time: The minimum time that a system can operate stably at 80% of its maximum capacity or at standard pressure (the expected daily pressure of the system).
Generally speaking, a system running on a normal working day (8 hours) should be able to ensure stable operation of the system for at least 8 hours.
For a 7 x 24 operating system, ensure that the system runs stably for at least 24 hours. If the system cannot run stably, the system performance may deteriorate or even crash as the service volume increases and the system runs for a long time.
Reference standard:
- The TPS curve is stable without large fluctuations.
- None resource indicators are leaked or abnormal.
Scalability metrics
Definition and Explanation: It refers to the relationship between increased hardware resources and increased processing capabilities when application software or operating systems are deployed in clusters.
The calculation formula is :(add performance/original performance)/(add resource/original resource) x 100%.
The trend of expansion index should be obtained through multiple rounds of testing. Generally, the expansion ability of application system is very good, and the expansion index should be linear or close to linear. Now many large-scale distributed systems have very good expansion ability.
Reference standard:
The ideal scaling capability is a multifold increase in resource and performance. The expansion capacity is at least 70%.
Reliability index
When measuring and analyzing system reliability indicators, three categories are commonly used for server performance testing:
- Double machine hot standby
- The cluster
- Backup and Restore
Double machine hot standby
For a system that uses dual-system hot backup as a means to ensure reliability, the following indicators can be measured:
- Node switchover is successful and consumes time.
- Check whether services are interrupted during the two-node cluster switchover.
- Whether a node is switched back and the elapsed time.
- Check whether services are interrupted during switchover.
- Data loss during node switchback during the two-node switchover, use the pressure generation tool to simulate the actual service occurrence and maintain certain performance pressure on applications to ensure that the test results meet the actual production conditions.
The cluster
For a system that uses a cluster, the following methods are used to evaluate cluster reliability:
- Check whether services are interrupted when a node in the cluster is faulty
- Whether to restart the system when a node is added to the cluster
- Whether to restart the system after the faulty node recovers and joins the cluster
- After the faulty node recovers and joins the cluster, check whether services are interrupted
- How long does node switchover Take? In addition to verifying cluster reliability, you need to use a pressure tool to simulate actual service occurrence and maintain performance pressure on applications to ensure that test results are consistent with actual production conditions.
Backup and Restore
This indicator is used to verify the reliability of the system backup/recovery mechanism, including system backup and recovery, database backup and recovery, and application backup and recovery. It includes the following tests:
- Whether the backup is successful and how long it takes.
- Whether to automate the backup using scripts.
- Whether recovery is successful and how long it takes.
- Restore the application principle of whether script automation is used to complete the indicator system.
- The adoption and inspection of indicators depend on the test purpose and test demand of the corresponding system. The system under test is different, the test purpose is different, the test requirements are different, and the index items examined are also very different.
- Some systems involving additional front-end user access capability need to investigate the index of user access concurrency capability.
- For the performance verification of batch processing, batch processing efficiency is mainly considered and batch processing time window is estimated.
- If the test target involves system performance and capacity, the test requirements must clearly describe the performance indicator requirements based on the definition of related indicators.
- After obtaining test indicators, describe related prerequisites (such as the number of service volumes and system resources).
Performance test scenario design
Scene classification
- Manual scenario
Manual scenarios can assign different scripts to different users in the same group, load generators.
- The target scene
A goal-oriented scenario is one that first defines a goal to be tested to achieve, and then loadRunner automatically creates a scenario based on that goal.
Scenario Design Strategy
- Fast growth
Usage: For example, the seckill function.
Problem: Loading method in loadRunner scenario: simultaneously loading. And Initialize the
What is the difference between the option to initialize all vuser users at once?
- Slow growth
Application scenario: A single scenario, such as opening a page, interface, and login.
- The number of users is stopped
Stopping a scenario Means that a user exits the current scenario after executing the scenario.
Q: In general, is it appropriate for the user to stop the scene in the same way as the user loads it, or is it appropriate for the user to exit all at once?
Question: User scenario execution time: Can it be understood that the overall execution time of the user scenario is equal to:
User loading time + user execution time + User exit scenario time?
Scenario Applicable Occasion
- A single scenario
For example, open a page or log in to a user.
- Mix scene
A mixed scenario is a scenario consisting of multiple services. For example, BBS forum Posting, with user login, Posting and reply services, these services can form a mixed scenario. When running the scenario, how many vusers can be specified to perform the operation of a single business.
Question: in the mixed scenario, dealing with a single business set up checkpoints, BBS BBS Posting checkpoint, for example, when the number of virtual users change, its entire Posting things significantly slower response time, is increased after the checkpoint, number of virtual users in performing the scene, will affect its transaction response time? Or should checkpoints not be added more than once in a mixed scene?
press
- Press definition
Press as the name implies is to increase the pressure of the machine, that is, the load machine, in the performance test process, can be assigned to multiple pressure machine for pressure.
- Procedure for adding load machines
1. Ensure that the federated machine has LRagent installed and enabled. (Status bar will have a small satellite)
2. The RPC service of the local system is enabled and changed to automatic.
3. Please log in to the federated machine from your Controller’s machine.
4, close the firewall + antivirus +360, etc. To have the permission, ensure that the load machines are in the same network segment and the network can communicate with each other.
The resources
So much for performance testing! If you need to know other content related to software testing, you can go to the “[Software Testing]” column to view and learn ~
At the same time, you are welcome to leave a message in the comment area or add my friends if you don’t understand or have mistakes. We can learn and exchange 🤗 together.
- ✔️ pay attention to the public number programmer, do not regularly share learning dry goods
- ✔️ If this article is useful to you, remember to like and follow again ~
- ✔️ we’ll see you next time! 👋 👋 👋
🌻 Recommended reading:
💜 After writing “proficient” on my resume, I was suffocated by an interviewer with work experience
💜 Bytedance test post interview failed, I reviewed the reasons for the failure, decided to fight again
💜 “My monthly salary has changed from 5K to 24K since I switched to test development for more than a year”, confessions of a liberal arts girl
💜 test engineers can easily break the monthly income of ten thousand? After looking at these 20 charts (market + learning skills), I fell to my knees!