At present, the platform will always do all kinds of screening and testing in advance, ready for fear of high concurrency, but it is often impossible to prevent. In fact, in the current mobile Internet popularity, ultra-high concurrency pressure not only exists in the field of e-commerce, online education, online office, social entertainment and other fields are also affected by it.

At high concurrency,
How can IT systems hold up? Is there a way to anticipate and build a “defensive fortress” in advance? On June 23, farce like cloud CTO He Yipeng an online already, “the outbreak era, enterprise performance evaluation of best practice” live, live the in-depth the performance high concurrency site common abnormal events, Shared how from 0 to 1 construct cloud pressure measuring system of standardization, standardization, looking for the best answer to that question.

Why do performance tests have to be done

“The importance of performance testing is self-evident, and if it is not done well, it can lead to catastrophic problems.”

As you know, performance exceptions include

· Instant surge of user visits;

· Full traffic on the server;

· System resources are occupied for a long time;

· The service access exceeds the maximum limit and the service is too narrow;

· Although the website is accessible, the latency is extremely high.

The first situation often occurs in the scene of buying tickets. The aggregation of a large amount of traffic five minutes before buying tickets often leads to insufficient bandwidth at the early stage of the server, which leads to poor user experience in the process of buying tickets, thus affecting the turnover of the platform.

The second type of server
CPU load is also very common, generally speaking, the business of complex system monitoring the performance of the single consumption reached 20%, the single remaining is usually only 70% to 80%, complex scenarios frequent access may lead to the CPU instantaneous as high as 90% above, based on this, if during the period of performance test did not test well in the scene, For the server is a relatively big disaster.

The third situation is the common load balancing device full load. Now most enterprises use load balancing devices of cloud manufacturers, which basically exist
The upper limit of PPS connections, which is not well estimated, can cause connection errors for subsequent users when the upper limit is full, typically HTTP 503 errors.

In the fourth case, the system is overloaded and the access limit is exceeded. The main problem during the test is that the capacity estimation is insufficient
Rapid service takeover can only be completed in the 90S. Therefore, it is necessary to perform hot switch and hot deployment scenarios during capacity assessment of the early performance test. After the scenario is set up, services can be quickly taken over through horizontal expansion, and some complex performance problems can be quickly solved.

In the fifth case, there is no problem with website access, but the website access delay is extremely high, and some service interfaces time out in large areas, affecting user experience.

In the development process, we will find that whether in development or testing, the common culprit is a small range of code errors, which will lead to some functional and performance problems, causing great losses. Therefore, strict needs assessment is very necessary, if can well analyze the common and abnormal business scenarios, once the online problems, can also be easily dealt with.



In the whole process of requirements, operations personnel do not need to be particularly anxious to do some coding operations. As shown in the above, prior to determine the design of the test scenario, test process of combing, test data management and execution sequence, then completed by performance test, an executive summary operation, summarizing the test results, by recording each node performance problems, formed the entire test analysis report, including the tuning data, parameter configuration data.

Generally, there are three methods for o&M personnel to configure indicator combing based on online performance data: parameter configuration for normal o&M, parameter adjustment for system anomalies, and adjustment method for emergency anomalies or catastrophic problems.

If the performance test is not in place, the direct economic loss will be inestimable. Take e-commerce enterprises as an example, the survey data from Amazon shows that when the access speed of e-commerce decreases every time

In 100 milliseconds, the turnover will be reduced by at least 1%. Compared with scenes like 618 and Double 11, if the user experience is poor and the payment cannot be made, the loss is conceivable.

Choose which test method is effective?

“In the era of mobile Internet, how can enterprises conduct effective and accurate performance tests for frequent marketing activities and rapid product iterations?”

With the rapid development of mobile Internet, the number of enterprise businesses such as e-commerce, online education and ticketing has risen sharply, and the value of ultra-high concurrency has been breaking through to the next level. At the same time, the business is complicated under the whole

The architecture of IT system is also evolving rapidly, with the transformation from single host to 1000 application hosts, more than 4000+ distributed CDN nodes, more than 10 device layers of link nodes, and the prevalence of distributed micro-service architecture. In this context, traditional performance testing faces many problems:

· 10 physical hosts are required to build a concurrent test environment with 10000 users;

· The deployment time of the test environment is more than 5 days and the environment reuse rate is low;

· The License authorization cost of 10,000 concurrent applications exceeds one million;

· The management of tool scripts, data and reports is scattered, causing great security risks;

· Test tools such as LoadRunner and Jmeter are complicated to operate and costly to learn, making it difficult for ordinary testers to master.

With traditional performance testing declining, cloud pressure testing rapidly absorbs nutrients to catch up, and performance testing has ushered in the spring of innovation and change.

The concept of cloud pressure measurement was born in 2005. With the rapid development of cloud computing technology, cloud resources are used to achieve elastic, scalable and freely scalable distributed pressure generation mode. Using cloud resources, cloud pressure testing realizes one-stop performance testing, which can simulate various abnormal scenarios of the system. Users do not need to purchase multiple resources, including servers and computer rooms, which can save a lot of resource costs and labor costs. At present, foreign such as Soasta, domestic such as Ruixiang Cloud, its cloud pressure measurement products have become the traditional performance testing platform of the most powerful rivals.

Compared to the traditional performance test solution, cloud pressure test has
Four advantages:

· Simple and easy to use: the script of cloud pressure test can be generated in 3 minutes, because all test resources are deployed in the cloud, which can realize the second-level startup, the second-level return of test data and the synchronous location of performance problems.

· Full stack monitoring: all cloud pressure measurement products are based on distributed cloud computing services, which can respond quickly based on location, and can also realize synchronous monitoring data backtracking to achieve full stack monitoring data collection, comprehensively covering the network layer, server layer, operating system layer and application layer.

· Large-scale deployment: The test nodes of most cloud pressure measurement manufacturers can cover the world, realize location-based on-demand customization, and realize full-link real nodes, reaching tens of millions of concurrent requests.

· High cost performance: SaaS services naturally have the advantage of flexibility. Cloud pressure measurement products can be charged on demand and do not need hardware deployment. It is easy to realize integrated test management services, and teams can also achieve teamwork, greatly improving work efficiency.

How to start a good performance test?

“What performance issues can the cloud stress test platform help users solve? How to solve it?”

Generally speaking, performance problems need to be analyzed from the network level, operating system level, application server, and server problems. At the network level, bandwidth is insufficient and network jitter is abnormal

In IDC deployment, the convergence ratio of switches needs to be considered. At the operating system level, the typical problem is the standardization of parameters, such as Sysctl and the configuration of some network parameters. On the server, you need to determine which processes occupy too much CPU during CPU monitoring. If the process usage is too high, you need to analyze the process usage. If the disk I/O usage is too high, you need to determine whether SSD disks are available.

If you want to more systematic analysis of performance testing problems, more comprehensive exploration of performance problems, then a set of systematic and complete testing process is indispensable.



The complete test process is shown in the figure above, starting from requirement analysis, test planning, script writing, test preparation, then a comprehensive analysis, and finally the evaluation report, which will include some indicators, such as the output of monitoring data and configuration data.

Cloud pressure measurement, the demand analysis of link need to pay attention to several important points, including the network information, information collection of firewall, anti-virus wall, load balance of the equipment, software and hardware encryption, the application of structured deployment, and user operation evaluation, etc., the use of habits when these points are analysis of relatively complete, you will make some more realistic scene.

In test planning, it is important to understand the time difference of accessing from different regions. For example, whether the response time of accessing from the northern region, the northern region, the southern region and the southern region is basically the same as that of some remote mountainous areas. If the response time is inconsistent, O&M personnel need to analyze the location of servers,
The CPN configuration is incorrect.

On the scripting side, the process needs to be simple enough that the business people can participate in the writing, and the advantage of this is that the business people can participate in the testing. When analyzing test scenarios, the business people closest to the market are more thorough than some regular technical people.

In the process of test preparation, the monitoring tool should be as comprehensive as possible. In addition to the typical five parts, it also needs to include some rapid output of dislocation prefabrication. The software and hardware of the monitoring machine must be equipped with functions similar to automatic alarm. Once a large area of problems occur, the operation and maintenance personnel can be prompted quickly for quick response.

In the comprehensive analysis, it should be noted that the estimation and production needs of basic data and test data are basically the same, so that the test results are basically not much different from the real online access results, and have very accurate reference value.

The analysis process can be completed by using tools. Only by analyzing the output content of each node in advance and organizing the whole test process can the final report or tuning indicator parameters have certain reference value, and the output result of the whole test is expected to serve as a reference for the optimal deployment of operation and maintenance in the later period.



Production and transaction log analysis, the importance of self-evident, from the point of the diagram above, the business distribution status, there is a lot of pin information, it is likely to be abnormal access scenario, need high response time on the request to do a complete analysis, including the basic amount, transaction peak, at ordinary times special session, the production problem, environmental load simulation, etc., If all of these simulations were in place, there wouldn’t be much of a problem.



In the production environment, the process of preparing test data is long, and data may be lost or omitted during data cleaning. To solve this problem,
Our firm like a cloud
In the long-term performance testing experience, there are four methods:

· Data preburial. That is, hang the test library under the production application, so that even if the test performance is slightly lower, it can basically measure the effect of the real access process, and the data is basically isolated and will not be polluted, which is convenient for later cleaning.

· Transformation of non-interface signs. Common examples are user-Agent field identifiers in HTTP request headers. You can select some unusual request headers in request identifiers, and perform service parsing at the back end to identify these data to improve data clearing speed.

· Bypass data routing. When the business flow is clear, the normal business data and the pressure data can be separated and processed, and then the pressure data table can be directed to track and clean. If only query transactions are conducted online, ruixiang Cloud’s main clean water meter and record table will not affect normal business.

· Interface field identification transformation. The identification bits of the pressure measurement field are reserved in the key data table, so that the information of the identification class can be directly filled in the pressure measurement stage, and the subsequent data cleaning can be directly based on this.



Based on the above analysis, I believe that you are right
Pressure test
We have had an in-depth understanding of all aspects and matters needing attention. Then, we will return to the exploration of the original problem. What performance problems can the cloud stress testing platform help enterprises solve? Lies mainly in
4 points:

· Real business flow simulation. Based on cloud pressure measurement, it can not only simulate the real access of hundreds of thousands of users, but also realize flexible user behavior simulation and rapid user scaling. In addition, network traffic quality can be quickly verified. Full network traffic can be verified based on normal traffic. If the enterprise uses a load balancer similar to F5 physical hardware, you can also verify that the PPS value of the physical hardware can meet high concurrency requirements.

· Resource monitoring. In addition to the rapid detection of CPU, memory and disk, it can also monitor the use of database resources, as well as some middleware resources.

· Operating system application optimization. The cloud pressure measurement platform can provide a very good test basis for Limit parameter configuration during the whole pressure measurement process, and can also conduct real-time tuning for Tomcat connection number and Jboss connection number.

· Locating performance problems. Combined with some common APM tools, we can quickly track some slow transactions, analyze some common problems in application and database, and simulate scenarios, such as slow transaction scenario simulation, network high throughput test scenario simulation, etc.

Write in the last

With the progress of science and technology, mobile Internet has realized the leap type development, software products have been applied to various fields, the outbreak, online model popular in all walks of life, but a traffic peak and frequency of high concurrency scene, in this context, how to guarantee system can bear high concurrent requests, bring the high quality service experience for users, has become the enterprise development
“The battlefield”, the performance test is the development of the market weapon.

The disadvantages of traditional pressure measurement have emerged, while the advantages of cloud pressure measurement have been highlighted, and the future development direction of performance testing has become clear. As a pioneer in the field of cloud pressure measurement in China, Ruixiang Cloud has accumulated a lot of practical experience and built a solid wall of performance test for enterprises in e-commerce, online education, online office and many other fields.