JB’s Reading Journey – Software Test 52 Talk (I)

instructions

Because digging gold an article can not exceed 2W words, so separate the upper and lower;

preface

Reading from a geek, the link is as follows: the course time.geekbang.org/column/103

Author: Ru Bingsheng

Personal feelings

First of all, let me list my personal feelings. The full text is very long, and it may take 20 minutes to read it patiently. Part of the content will be more detailed, and the reading experience may be poor, because it is mainly written.

In general, this course is worth the price. After reading it, I feel that it has a wide range of knowledge, which may not be useful for practical work, but it is a very good course to expand my knowledge. At the same time, I also know how other enterprises do it.

Through this course catalog, in fact, it can be seen that many large companies test engineers, not with business testing, with development ability is a necessary condition, because also want to give themselves a vigilance;

If you have to recommend it, chapter 5 on automated testing tools and Chapter 35 on data Building are worth a look.

I read this course for a long time, and I read it very seriously, including the comments, so when I see interesting comments, I will post them.

The following is all typed by hand, because JB still believes that what he writes/types will deepen his impression

The author’s way of software test cultivation

Changes experienced:

• Automated test case design and development
• Selection of test frame
• The testing framework is developed by ourselves
• Test framework architecture design
• Test servitization

Trend summary:

• The status of automated testing in software quality engineering has undergone a qualitative change, from the original automated testing as a supplement, but now to automated testing as the main
• The product iteration cycle is shortened, requiring a comprehensive and highly concurrent test execution infrastructure
• A qualified test engineer is concerned with pure testing, a good test engineer is concerned with the quality of the overall delivery
• How to build low maintenance, flexible assembly of automated scripts that involve layered design models

1) Do you really know tests? Start with user login tests

Test scenarios

Enter the user name and password and click the confirm button to verify the login

The most commonly used black box testing method

Equivalence division

All possible input data are divided into several subsets. In each subset, if any input data is equally effective in exposing potential errors in the program, such subsets constitute an equivalence class.

Boundary value analysis

Is to select input, output boundary value for testing; The simple description is exactly equal to, just greater than, just less than;

Boundary value is a supplement to equivalence class division

End test

A test method that includes all possible combinations of software input values and preconditions, so that no unknown software defects remain in a fully tested system;

However, this approach is not practical, because due to the time cost, the risk-driven model is generally adopted, which focuses on the selection of test scope and design of test cases;

Explicit functional test cases

Explicit functional requirements refer to the functions that the software itself needs to implement;

Based on equivalence class partition and boundary value analysis method:

Enter the registered user name and correct password to verify the login. Enter the registered user name and incorrect password to verify whether the login fails and the message is correct. Enter an unregistered user name and any password to verify whether the login fails and the information displayed is correct. If the user name and password are empty, the login fails and the correct information is displayed. If either of the user quota password is empty, the login fails and the correct information is displayed. If the verification code function is enabled for login, enter the correct verification code when the user name and password are correct. If the verification code function is enabled for the login function, if the user name and password are correct and you enter an incorrect verification code, the login fails and the message is correct.Copy the code

Other scenarios use cases:

Whether the user name and password are case-sensitive; Whether the password box on the page is displayed encrypted; Whether the system prompts you to change the password when a user created in the background system logs in successfully for the first time. Whether the functions of forgetting user name and password are available; Whether the length of user name and password is limited according to the design requirements; If the login function requires a verification code, click the verification code picture to check whether the verification code can be replaced and whether the new verification code is available. Whether the verification code will be refreshed when the web page is refreshed; If the captcha is time-sensitive, the validity of the in-time and out-of-time captcha needs to be verified respectively. If a user logs in successfully but the session times out, the user is redirected to the user login page if the user continues to perform operations. Whether different users, such as administrators and common users, have correct permissions after logging in to the system; Whether the default focus of the page is located in the user name input box; Whether shortcut keys such as Tab and Enter can be used normally; Normal login when the network is delayed or weak, or when the network is switched over or disconnected. Whether the password can be remembered and whether the remembered password is encrypted; Remember whether the password has a validity period and whether the password will be cleared after the expiration. Whether the user name and password support special characters and Chinese characters. Whether login apis can be used to send login requests and bypass captcha; Whether you can log in to the request packet captured by the packet capture tool directly; Whether the captured token information can be used directly on other terminals to bypass login; Whether the back end has the format length of the verification content; After login, enter the login URL to check whether you can log in again. Login error prompt whether there are security risks; Password strength verification; Whether null and input null string checks are consistent; Security remote login verification, replacement device login verification; Devices are mutually exclusive; Limit the number of password errors; Whether there is a requirement on the keyboard format when inputting the account password; Whether the password column needs to set the switch button of light and secret code; Whether to set the button to unclickable if the password format is not standard; Whether to set the quick delete button in the input field; Whether there is a kicking mechanism between multiple devices and platforms and account login; After the password is changed, the previous password does not take effect. User name rules and password rules; Front and background switching, horizontal and vertical screen switching;Copy the code

Nonfunctional requirements

Mainly involving security, performance and compatibility testing;

Security test cases:

Whether the user password is encrypted in background storage; Whether the user password is encrypted during network transmission; Whether the password has a validity period and whether a message is displayed asking you to change the password after the validity period expires. If you do not log in to the system, enter the URL in the browser to check whether the system is redirected to the login page. Whether the password input box does not support copy and paste; Whether passwords entered in the password input box can be viewed in page source mode; Enter the typical SQL injection attack string in the user name and password input box respectively to verify the system return page; Enter a typical XSS cross-site scripting attack string in the user name and password fields. Verify whether the system behavior has been tampered with; In the case of multiple login failures, whether the system will block subsequent attempts to deal with brute force cracking; If a user logs in to multiple browsers on the same terminal, check whether the mutual exclusion of the login function meets the design expectation. If a user logs in to browsers on multiple terminals, check whether the mutual exclusion is available. Whether the user's personal information stored in the database after the user login is encrypted; Whether personal information is printed in plaintext in the log during user login; Whether caching is used;Copy the code

Performance pressure test cases:

Check whether the response time of a single user login is less than 3 seconds. Whether the background request data is too much when a single user logs in; Check whether the response time of a user login is less than 5 seconds in a high-concurrency scenario. Whether server monitoring indicators meet expectations in high concurrency scenarios; In the concurrent scenario with high collection points, whether there are resource deadlocks and unreasonable resource waiting; If a large number of users log in and out continuously for a long time, whether the server has memory leaks. Ten users can log in at the same time. If nine or 11 users can log in at the same time, the prompt information is correct.Copy the code

Compatibility test cases:

Different browsers, validating the login page display and function under the different versions of the same browser, validating the login page display and function under different mobile terminals of different browsers, validating the login page display and function under the different resolution of interface, validating the login page display and functionCopy the code

summary

1) Use case design should consider explicit display functional requirements, as well as a series of non-functional requirements such as compatibility, security and performance;

2) Use case design is inexhaustible and limited by time cost, so a balance between defect risk and R&D results should be taken into account;

2) How to design a “good” test case

A use case that covers all equivalence classes and various boundary values is a useful case;

Characteristics that a good test case must have

Global completeness

Is a complete whole, is a set of effective test cases, can fully cover the test requirements;

The accuracy of equivalence class division

For each equivalence class, it can guarantee that as long as one of the input tests passes, the other inputs must also pass the test.

Completeness of the sum of equivalence classes

Ensure that all possible boundary values and boundary conditions have been correctly identified;

The three most commonly used test case design approaches

Equivalence division

Any input data in the equivalence class has the same effect on exposing potential errors in the program.

Grade system, integers ranging from 0 to 100, with a pass score of 60; Valid equivalence classes: any integer between 0 and 59; Any integer between 59 and 100; Invalid equivalence classes: negative numbers less than 0; An integer greater than 100; Any floating point number between 0 and 100; Other non - numeric characters;Copy the code

Boundary value analysis

Score system: -1, 0, 1, 59, 60, 61, 99, 100, 101Copy the code

Method of miscalculation

It refers to the method of designing test cases based on the understanding, past experience and personal intuition of the design of the software system to be tested to predict the possible defects of the software, which is similar to exploratory testing;Copy the code

How to design good test cases

In the specific use case design, first of all, it is necessary to make clear the multiple software functional requirement points corresponding to each business requirement, and then analyze the multiple test requirement points corresponding to each software functional requirement point, and then design test cases for each test requirement point.

Two key points:

• Starting from software functional requirements, identify test requirements comprehensively and without omission
• For each test requirement point, equivalence class division, boundary value analysis and error prediction methods should be used to comprehensively design the use case

Additional experience in use case design

Only by deeply understanding the architecture of the software being tested can we design better test cases to discover system boundary values and potential defects.

Must have an in-depth understanding of the design and implementation details of the software under test and the processing logic within the software;

Requirements coverage and code coverage should be introduced to measure test execution completeness.

summary

1) A good test case must be a complete set that can cover all equivalence classes and various boundary values; In most cases, use cases are evaluated from an ex post facto perspective in terms of missed detection rates and problem severity;

2) There are many ways to design but use cases, but the comprehensive application of equivalence class division, boundary value analysis and error speculation can meet most of the needs; Demand rationality test, namely user experience test;

3) In the redesign of good test cases, it is necessary to comprehensively identify test requirements without omission from the perspective of software functional requirements;

4) To design a good test case, it is necessary to deeply understand the architecture design of the software under test, go deep into the processing logic inside the software, and use requirement coverage and code coverage to measure the completeness of test execution;

3) What are unit tests? How to do unit testing well

What are unit tests

Unit testing refers to the work of checking and verifying the smallest testable units in the software in isolation from the rest of the program. Here, the minimum and test units usually refer to functions or classes.

Unit tests are typically performed in an automated manner and are more profitable in a scenario of extensive regression testing;

TV generation, testing, software development and testing comparison:

Electronic components are like units in software, usually functions or classes, and testing individual components is like unit testing in software testing; The assembled functional circuit board is like the module in the software, and the test of the circuit board is like the integration test in the software. When the TV set is fully assembled, it is like completing the pre-release version of the software. The boot test after the driver is fully assembled is like the system test in the software.Copy the code

How to do unit testing well

concept

The object of unit tests is code;

The main means of unit testing are driver code, pile code and mock code.

methods

Basic characteristics of the code and the cause of the error:

• Conditional branching, loop processing, and function calls
• If there is any classification omission, classification error, classification is not correct is not missing, but the processing logic error, will produce defects
• What kinds of inputs are normal if the proper functional logic is to be implemented
• Whether there are multiple boundary inputs that require special processing
• What are the possibilities of potential illegal inputs and how to deal with them

Unit test case details

Typically, a unit test use case is a collection of input data and expected output;

Based on the logic function that the code needs to implement, what input and what output should be produced;

Input data:

The input parameters of the function being tested are global static constants that need to be read from inside the function being tested are member variables that need to be read from inside the function The data obtained by calling the subfunction inside the function The data overwritten by calling the subfunction is embedded in the X system, the data overwritten when the call is interruptedCopy the code

Expected output:

The return value of the function being tested the output parameter of the function being tested the member variable being overwritten by the function being overwritten the global variable being overwritten by the function being updated by the file being updated by the database being updated by the function being updated by the message queue being updated by the function being testedCopy the code

Driver code, stub code, and mock code

The driver code is used to call the function under test;

The driver module usually includes three steps: data preparation before calling the function under test, calling the function under test and verifying the relevant results.

Stubs and mocks are used to replace real code for function calls under test;

Stub code is temporary code that replaces real code;

Internal implementation of pile function:

• The application of pile code first plays the role of isolation and completion, so that the code under test can compile, link, and allow independently
• The peg function should have exactly the same form as the original function, except for the internal implementation, so that the test code can link to the peg function correctly
• The peg functions used for isolation and completion are relatively simple, leaving the declaration of the original function and adding an empty implementation, with the goal of compiling links
• Pile function to achieve control function is the most widely used, according to the needs of test cases, output appropriate data as the internal input of the function under test

Mock code is very similar to stub code in that it is temporary code that replaces real code;

For mock code, the focus is on whether the mock method is called, with what parameters it is called, how many times it is called, and in what order multiple mock functions are called.Copy the code

For stub code, the focus is on using stubs to control the execution path of the function under test, not whether or how the stub is called.

How do unit tests work in real projects

Not all code should be unit tested, usually only for underlying or core modules;

Need to determine the selection of unit test framework:

• Java, junit and TestNG
• C/C + +, CppTestParasort C/C + + test

Introducing code coverage:

• Java, jacoco
• JavaScript, Istanbul

Perform unit testing, code coverage statistics, and continuous integration pipelining; Ensure that unit tests are triggered automatically with every code commit;

Possible problems with unit testing

Precision-coupled code is difficult to isolate; Compiling links after isolation is difficult to run The k testability of the code itself is poor, and the testability of the code is usually proportional to the size of the code. The call of the underlying functions of the system cannot be simulated directly by the pile code. The later code coverage becomes harder to improve;Copy the code

summary

1) Introduces the concept of unit test, discusses the composition of use cases, and the method of developing unit test in actual projects;

2) Three issues need to be paid attention to in the development process:

• To achieve the correct functional logic, the code must be classified correctly and complete without omission, and the processing logic of each classification must be correct;
• Unit testing is the level of code performed on the smallest and test units in the software in isolation from the rest of the software;
• Pile code plays the role of isolation and complement, so that the code under test can independently compile, link, and run;

4) Why do automated tests? What kind of project is suitable for automated testing

What is automated testing

The practice of transforming the testing behavior of human to software into the testing behavior of machine;

It’s essentially writing one piece of code and then testing another piece of code; The realization of automated test cases itself belongs to the development work, but also needs to constantly update with the change of the object under test, maintenance of use cases cost;

Automated testing loses value when the maintenance cost of automated use cases is higher than the test cost it saves.

Why do you need automated tests

advantage

• Automated testing can replace a lot of manual mechanical repetitive operations, test engineers can spend more time on more comprehensive use case design and new function testing;
• Automated testing can greatly improve the efficiency of regression testing and is very suitable for the agile development process.
• Automated testing can make better use of unattended time, to more frequent test execution, especially suitable for the current non-working time test execution, working time analysis of failure cases;
• Automated testing can efficiently realize some types of tests that cannot be completed or cost a lot of money, such as the system stability test with 7X24 hours continuous operation of key business and the stress test with high concurrency scenarios.
• Automated testing also ensures consistency and repeatability of operations and validation performed by each test, avoiding perceived omissions or omissions;

disadvantage

• Automated testing can not replace manual testing, only replace manual testing in the execution of high frequency, mechanical repeated steps;
• Automated testing is far more vulnerable than manual testing and cannot cope with changes in the system under test. Automated testing is not everything, it is just the execution of pre-defined test steps and verification of results;
• The development workload of automated test cases is much larger than that of simple manual tests. Only when the test cases are effectively executed more than or equal to 5 times, the cost of automated test can be recovered.
• Manual testing usually finds more defects than automated testing;
• The efficiency of testing largely depends on the design and implementation quality of automated test cases;
• In the early days of automated testing, use-case development is often inefficient;
• Business test experts and automated test experts are usually two groups of people, the former understand business but not technology, the latter understand technology but not business, need to cooperate with both;
• Automated test developers must have certain programming ability;

What projects are suitable for automated testing

• Demand is stable and will not change frequently;
• The development and maintenance cycle is long, requiring frequent implementation of regression use cases;
• Scenarios where the same tests need to be run repeatedly on multiple platforms;
• Some test items cannot be reproduced manually or are costly;
• The development of the tested software is more standardized, which can ensure the testability of the system.
• The tester already has some programming ability;

summary

1) Automated testing is a practice of converting manual testing of software into testing behavior performed by machines. It can free test engineers from mechanical repetitive testing work and put more experience in testing new functions and more comprehensive test case design.

2) Automated testing is a double-edged sword, which liberates the labor force of test engineers to some extent and completes some tests that cannot be achieved manually, but it is not a panacea; Once the maintenance costs outweigh the test savings, automation is not appropriate;

5) Do you know what automated testing techniques are used at each stage of software development

Unit test automation techniques

Unit test itself is an automated test, because it uses equivalence class partition and boundary value analysis methods to design test cases according to the detailed design of software, and then implements the test code in an automated way.

Broadly speaking, automation in the unit test phase not only refers to automation of test case execution, but also includes the following five aspects:

• Automation of use case framework code generation;
• Automatic generation of partial test input data;
• Automatic pile code generation;
• Automated static analysis of code under test;
• Automatic statistics and analysis of test coverage;

Automation of use case framework code generation

Some of the framework code is generated by automated tools rather than by hand, allowing developers to focus more on the coverage of test logic and the selection of test data.

For example, Selenium code can be recorded and TestNG framework code generated by automated tools.

Automatic generation of partial test input data

This part means that the automation tool can automatically generate test input data based on different variable types;

Automatic pile code generation

Automatic pile code generation means that the automatic tool can scan and analyze the tested code, automatically generate programmable pile code for other functions called by the function under test, and provide a pile code management mechanism based on test cases.

So what is pile pulling? In the unit test stage, if function B is called by function A internally is pile code, then in the code-level integration test stage, it is expected that function A will not call the fake function B, but call the real function B. The operation of replacing the original pile code function B with the real function B is called pile pulling.

Automated static analysis of code under test

Static analysis mainly refers to the static scanning of code, the purpose is to identify the code that violates the coding rules or coding style;

Usually this part of the work is combined with the specific coding rules and coding style of the project, by the automation tool through the built-in rules and user defined rules automation.

Common static code analysis tools are Sonar and Coverity;

Automatic statistics and analysis of test coverage

After the execution of the unit test case, the automation tool can automatically count various test coverage, including line of code coverage, branch coverage, MD/DC coverage, etc.

These automated metrics help measure the adequacy and completeness of the unit test case set, and can give you a basis for adding test cases to improve test coverage;

Automation techniques for code-level integration testing

To put it simply, code-level integration testing refers to testing software modules that have been developed together.

The biggest difference between the code-level integration test and the unit test is that the other functions called by the function under test in the code-level integration test must be real and cannot be replaced by pile code, while the unit test allows the use of pile code to simulate other functions called inside.

Automated techniques for Web Server testing

Web Server testing mainly refers to SOAP API testing and REST API testing. SoapUI and Postman are the most typical tools.

However, these testing tools are basically interface operation to manually initiate requests and verify response, so it is difficult to integrate with CI/CD, so API automation testing framework emerged.

If the API automation testing framework is used to develop test cases, these test cases are expressed in code as follows:

For code-based API test cases, there are usually three steps:

• Prepare test data for API calls;
• Prepare API call parameters and initiate API calls;
• Verify the return of the API call;

The most popular API automated testing framework is REST Assured, which makes it easy to initiate Restful API calls and verify the results returned.

Similarly, automated testing for Web Server includes not only automation of API test case execution, but also the following aspects:

• Automated generation of test scaffolding code;
• Automatic generation of partial test input data;
• Automation of response validation;
• Automatic script generation based on SoapUI or Postman;

Test automated generation of scaffolding code

In the process of API test development, more attention is paid to how to design the input parameters and combinations of test cases, as well as the verification of response in the case of different parameter combinations.

This is where you need to test the automatic generation of scaffolding code; The generated test scaffolding code contains the call to the tested API, the separation of test data from the script, and the empty implementation of response validation.

Automatic generation of partial test input data

It is similar to the automatic generation of input data for unit tests, except that the parameters for unit tests are function input parameters and function internal inputs, while the parameters for API tests are API parameters and the Payload of API calls.

The principle of data generation also follows the boundary value principle.

Automation of response validation

As for the verification of the result returned by API call, more attention is paid to the return status code, Scheme structure and specific field value.

The core idea is to automatically compare the return results of two same API calls, and automatically identify the different field values. Dynamic values such as timestamp and Session ID can be removed in the comparison process through rule configuration.

Automatic script generation based on SoapUI or Postman

SoapUI or Postman are single debuts that accumulate to the point where there is a large set of test cases. Introducing a code based API testing framework means that a large number of use cases need to be rewritten in code, which is disgusting.

The recommendation is to develop an automated code transformation generation tool that takes SoapUI or Postman test case metadata (JSON files) as input and outputs code based test cases that conform to API test framework specifications. The benefit is that the original accumulated use cases can be directly converted into automated test cases that can be accessed directly on CI/CD;

The newly added test cases can be verified by SoapUI or Postman tests. No problem, they can be directly converted into test cases that conform to the SPECIFICATIONS of the API testing framework.

Postman integrated CICD, postman + Newman + Jenkins, exported a JSON file in Postman, deployed Newman on another server, and executed the json file exported by Postman on the command line. You can then test and generate test reports directly in the server using the Newman tool.

Robot Framework + Requestslibrary to do API automation tests, identify teams with poor programming ability;

Automation techniques for GUI testing

The core idea is to automate the operation of page elements based on page element recognition technology to simulate the behavior of actual end users and verify the correctness of software functions.

• For GUI automated testing of traditional Web browsers, Selenium is the mainstream open source solution in the industry, while Micro Focus UFT (formerly HP QTP) is the commercial solution.
• For mobile native applications, the mainstream Appium is usually adopted, which integrates XCUITest for IOS environment and UIAutomator and Espresso for Android environment.

summary

This paper introduces the automation test technology of each stage of software development and development function cycle, including unit test, code level integration test, Web Service test and GUI test automation technology.

6) Do you really understand test coverage

Test coverage is usually used to measure the adequacy and integrity of tests. In a broad sense, test coverage can be divided into two categories: project-oriented requirements coverage and technical code coverage;

Demand coverage

Requirement coverage refers to the extent to which the test covers the requirements. The usual practice is to map each decomposed software requirement with the corresponding test resume one-to-many. The ultimate goal is to ensure that the test can cover every requirement to ensure the quality of software products.

Requirements management tools such as ALM, Doors, and TestLink are used to map requirements and tests to calculate test coverage.

The statistical method of demand coverage belongs to the software engineering practice under the traditional waterfall model. The traditional waterfall model pursues top-down planning, requirement analysis, software design, code writing, testing and operation and maintenance, which is a heavyweight process and has been difficult to adapt to the agile development in the era of Hurina.

So, general coverage, by default, means code coverage, not requirements coverage;

Code coverage

Is the percentage of the total number of items executed at least once;

If the number of items is a statement, it corresponds to line coverage; If the number of items is a function, the corresponding is the function coverage; If the number of items is paths, then path coverage is the corresponding;

Three code coverage metrics

• Line coverage, also known as statement coverage, refers to the percentage of statements that have been executed in the total number of executable statements (excluding code comments, empty lines, or header files). This is the most commonly used and least required coverage measure;
• Decision coverage, also known as branch coverage, is used to measure whether every decision branch in the program is tested, that is, whether the true and marry branches of each judgment in the code are covered only once.
• Condition coverage means that the possible value of each condition in the decision is satisfied at least once, and whether the TRUE and FALSE results of each condition in the decision are tested is measured.

Value of code coverage

The fundamental purpose of statistical code coverage is to identify potential missing test cases and make targeted additions, as well as to identify obsolete code that is unreachable due to requirements changes.

Higher code coverage indicates that your test case design is adequate and complete, but the cost of testing increases exponentially as coverage increases;

Limitations of code coverage

The code coverage feedback is just what logic of existing code is being executed and what logic is not being executed, but if other modules rely on this code, can you be sure that there is no problem?

The fundamental reason is that code coverage is calculated based on existing code;

High code coverage does not guarantee the quality of software, but low code coverage does not guarantee the quality of software.

Code coverage tool

JaCoCo is a mainstream open source coverage feature for Java code, which is easily embedded into Ant and Maven, and can be well integrated into Jenkins and other mainstream continuous integration tools.

What does a JaCoCo code coverage report look like?

As shown in the figure, the line coverage and branch coverage of each Java code file are included, and the number of lines, methods and classes of each Java code file is given.

The figure below shows the detailed code coverage within each Java file, with green lines indicating that it has been covered, red lines indicating that it has not been covered, and yellow lines indicating that it has been partially covered. The green diamond block on the left indicates that the branch has been completely covered, while the yellow diamond block indicates that the branch is only partially covered.

From this report, you can know how the code is actually performing, and then you can design test cases accordingly.

Consider using JEST for the front end;

How code coverage tools are implemented

To achieve code coverage statistics, the most basic method is to inject (Instrumentation); Injection is to automatically insert Probe code for coverage statistics into the code under test, and ensure that the inserted Probe code will not have any influence on the source code.

For Java Code, according to the different injection target, it can be divided into Source Code injection and Byte Code input. Based on the characteristics of JVM itself and execution efficiency, the current mainstream tools are basically using bytecode injection, the specific implementation of injection using ASM technology;

ASM is a Java bytecode manipulation framework that can be used to dynamically generate classes or enhance the functionality of existing classes, either directly generating class files or dynamically changing the behavior of classes before they are loaded into the JVM.

According to The time of injection, bytecode injection can be divided into two modes: on-the-fly injection mode and Offline injection mode.

On-the-fly injection mode

The feature is that there is no need to modify the source code, and there is no need to carry out bytecode piling in advance. It is suitable for the running environment supporting Java Agent.

The advantage is that code coverage information can be collected in real time without system downtime. The disadvantage is that the runtime environment must allow the use of Java Agent.

There are mainly two technical schemes to realize The on-the-fly mode:

• Develop a custom Class Loader to implement the Class loading policy. Before each Class loading, a probe needs to be inserted into the Class file. The early Emma implemented the probe insertion using this scheme.
• With Java Agent, the probe is inserted using premain(), the interceptor method that executes before main(). In practice, you need to add “-JavaAgent” to the JVM’s startup parameters and specify a proxy for real-time bytecode injection, so that the proxy is installed before each class file. Check whether the probe has been inserted. If not, insert the probe into the class file. JaCoCo uses this method.

Offline injection mode

In Offline mode, there is no need to modify the source code, but the file needs to be staked before the test starts, and the staked class file is generated in advance.

This method applies to scenarios where Java Agents are not supported and custom class loaders cannot be used.

The advantage is that the JVM does not need to use the Java Agent to open additional code during startup. The disadvantage is that the code coverage information cannot be obtained in real time, but only when the system is down.

Offlinne mode can be divided into Replace and Inject mode according to whether a new class file is generated or the original class file is directly modified.

Unlike The on-the-fly injection mode, Replace and Inject are implemented by using ASM to insert probes into The class file before The toilet runs, and no additional processing is required during The test run.

Cobertura is an example of using Offline mode;

summary

Test coverage is usually used to measure the adequacy and integrity of tests, including project-oriented requirements coverage and technology-oriented code coverage;

However, the statistical method of requirement coverage is no longer applicable to the current agile development model, and all the test coverage now refers to the code coverage;

High code coverage does not necessarily guarantee software quality, because code coverage is based on existing code and cannot detect defects that occur without considering certain inputs or processing certain situations;

7) How to fill in the software defect report efficiently

Defect report is an important bridge of communication between test engineer and development engineer, and also an important output of test engineer’s daily work. It is the most basic skill of test engineer to express the defects accurately and clearly without ambiguity.

Accurate and unambiguous expression means that development engineers can quickly understand defects and accurately locate problems based on defect reports. The development manager can accurately estimate the priority of defect repair; Product managers can understand the impact and severity of defects on users or business.

The quality of defect report itself is directly related to the speed of defect repair and the efficiency of the development engineer, and also affects the credit of the test engineer, the effectiveness of testing and the writing of the developer.

Defects in title

The defect title is usually the first part that others see. It is a general description of the defect, usually using the pattern of “what happened in what situation”.

Description should not only be clear and concise, the most important is to be specific enough, avoid too general description; At the same time, the description must clearly state the context of the problem, that is, the scene of the problem;

The title of the defect should not be too long. A more detailed description of the defect should be placed in the defect overview.

Summary of the defect

Defect overview usually provides more general description of defect nature and phenomenon, which is a refinement of defect title.

The purpose of defect overview is to describe the defect clearly and concisely so that the development engineer can focus on the essence of the defect;

Defects affect

The impact of defects describes the impact range and severity of problems caused by defects on users or services.

The defect impact determines the Priority and Severity of the defect. The development manager will decide the Priority of repairing the defect based on this. The product manager will use this as a basis to measure the severity of the defect and decide whether to release the product until the defect has been fixed.

To accurately describe the impact of defects, it is necessary to have a deep understanding of software application scenarios and requirements, which is also a test of the basic skills of test engineers.

Environment configuration

Environment configuration is used to describe the configuration details of the test environment in detail and provide necessary environment information for the recurrence of defects, such as operating system type and version, software package under test, browser type and version, software under test configuration information, cluster configuration parameters and so on.

It is important to note that the content of the environment configuration is described on demand, that is, only those environment-sensitive information that reproduce defects are described;

precondition

Preconditions refer to the state that the system should be in before the test steps start. The purpose is to reduce the description of defect recurrence steps, eliminate unnecessary interference, and make it more targeted.

Defect recurrence step

Defect reappearance step is the core content of the whole defect report, which aims to show the specific operation steps of defect reappearance to the developer in simple language.

It is important to note that the steps are usually described from the user’s point of view, and each step should be operable and consistent;

When writing defect recurrence steps, repeat these steps to confirm:

• Ensure reproducibility of defects;
• Find the shortest return path;

There are three scenarios that should be avoided when describing defect reoccurrence steps:

• General description, lack of actionable steps;
• Steps that are not related to defect recurrence occur;
• Lack of description of test data;

Expected results and actual results

The expected and actual results need to be clearly stated in describing the recurrence steps; The expected results come from an understanding of the requirements, while the actual results come from the results of test execution;

Priority and Severity

According to Baidu Baike, defect priority refers to the degree of urgency that a defect must be fixed, while defect severity refers to the degree of impact of faults caused by defects on software products.

The severity is the attribute of the defect itself, and usually will not change after confirmation, while the priority is the engineering attribute of the defect, which will change with the project progress, the cost of solving the defect and other factors.

What is the relationship between defect priority and severity?

• The more serious the defect, the higher the priority;
• The greater the scope of defects, the higher the priority will be;
• Defects that are not serious from a user impact perspective but hinder the execution of tests or automated tests are typically low-severity but high-priority problems.
• Although some defects are relatively serious, they may have lower priorities considering repair costs and technical difficulties.

Wordaround

Workarounds provide a way to temporarily work around the current defect without affecting product functionality, usually by or jointly decided by the test engineer or development engineer;

The availability of alternative schemes and the difficulty of implementation are the important basis for determining the priority and severity of defects.

Root cause analysis is as follows

If the root cause of the defect is identified, the cause of the defect is clearly described and the feedback is given to the development engineer, the efficiency of the development engineer to repair the defect will be greatly improved.

The attachment

Attachments usually provide necessary evidence to support the existence of defects. Common attachments include interface screenshots, test case logs, server-side logs, GUI test execution food, etc.

summary

An effective software defect report should include defect title, defect overview, defect impact, environment configuration, preconditions, defect recurrence steps, expected and actual results, priority and severity, workarounds, cause analysis, and attachments.

Other things I want to talk about

How to count, analyze, and report bugs in a phase?

This belongs to test defect statistics, bug trend analysis, bug convergence, bug module distribution, bug discovery stage statistics and so on. It belongs to the management team and quality process assurance team. Generally, this kind of report is generated by using the built-in function of defect management system.

In addition to what is required above, you also need the following:

• Defect module, convenient later statistics;
• Can accurately assign bugs to specific owners, in the simplest terms, first confirm whether the problem is back-end or front-end;
• Probability of occurrence;
• Comparative inflammation;

8) Starting with the end, how to do a good test plan

In the early software engineering practice, the formulation of software test plan is usually started after the completion of requirement analysis and test requirement analysis, and is an important link in the whole software development statement cycle.

What happens without a test plan?

What are the problems if there is no test plan?

• It is difficult to know exactly what the scope of testing is and what the specific testing strategy should be;
• It is difficult to estimate the specific workload and the number of test engineers needed, and it will also cause the unclear division of labor of each test engineer, resulting in the repeated execution of some test work and the omission of some tests;
• The overall progress of the test is completely uncontrollable, and it is even difficult to know exactly the completion of the mother’s test. It is even more difficult to estimate the exact time node for the completion of the test.
• The whole project has weak resistance to potential risks and is difficult to cope with demand changes and other emergencies;

From these problems, it can be deduced in reverse thinking that a good test plan should include the following points: test scope, test strategy, test resources, test progress and test risk estimation;

Test range

As the name implies, the test scope describes the object under test and the main test content;

The determination of the test scope is usually carried out after the completion of the test requirement analysis, so the process of determining the test scope is also the further verification of the test requirement analysis to some extent, which helps to find potential test omissions in the early stage.

The topic of testing strategy

Testing strategy is simply defined as “what to test first and what to test after” and “how to test” two problems;

The testing strategy requires a clear focus on testing and the sequencing of tests;

The testing strategy also needs to specify what type of testing and testing methods are used. Not only should you explain why this type of test was chosen, but you should also explain how to implement it.

A functional test

Design test cases according to the mind map of test requirement analysis;

It should be noted that the test automation of the trunk business process should be realized first.

For test points that require manual testing, decide what type of test case design approach to use and how to prepare relevant test data;

Also evaluate the testability of the software under test, and if testability is a problem, consider practical workarounds in advance, and even require developers to provide testability interfaces;

Compatibility test

The Web test needs to determine the browser type and version that is covered, and the mobile device test needs to determine the device type and IOS/Android version that is covered.

How do you determine the types of mobile devices you need to cover and the IOS/Android version list?

• For old products, obtain top 30% mobile device models and version information by analyzing historical data based on product statistics, which can be covered by this batch of firmware compatibility;
• If it is a new product, a website like TalkingData can check the current mainstream mobile devices, resolution size, IOS/Android and other information to determine the test scope.

Generally speaking, the compatibility test is the later stage of the function test, and the compatibility test will start after the function is basically stable.

If it is to access some new frameworks, at this time it is necessary to evaluate the information of access to the new framework, at this time it is necessary to conduct compatibility testing first, to ensure that the later will not introduce unsolvable compatibility problems;

The selection of compatibility use cases is basically from the automated test cases that have been realized.

The performance test

On the premise of clear performance requirements (concurrent users, response time, transaction throughput, CPU, memory, IO, bandwidth, transaction success rate, timeout error rate, etc.), combined with the characteristics of the system under test, design performance testing scenarios and determine the performance testing framework;

For example, do you initiate stress tests directly at the API level, or do you have to simulate end user behavior for protocol-based stress tests? Another example is whether to conduct a modular pressure test or launch a full link pressure test.

If performance is a background data sensitive scenario, you also need to determine the magnitude and distribution of the background data, and determine the technical solution for generating the background data, such as whether to generate the test data through concurrent API calls, do bulk INSERT and update operations directly on the database, or a combination of the two.

Either way, it is important to know the number of single-user scripts to be developed so that the pressure test scenarios can be assembled smoothly later;

The implementation of performance test is a complicated problem. Determine the type of performance test you want to solve based on the problem you want to solve, and then conduct tests based on the specific type of performance test;

• The implementation of performance testing, first according to the business scenario to determine the need to develop single-user scripts, script development will involve a lot of performance testing scripts, such as thinking time, collection points, dynamic association, etc.
• After the script development is completed, the test Scenario should be organized according to the unit of the script. The definition of the Scenario simply refers to the percentage of users who log in, the percentage of users who query, the waiting time between each user’s operation steps, and the growth rate of concurrent users in X seconds.
• Finally, it is the execution of specific test scenarios. Different from automated functional tests, the interpretation of performance test reports after the execution of performance tests is the most critical point in the whole test process.

Test resources

Testing resources usually include testers and test environments, which are limited.

The purpose of test planning is to ensure maximum output under effective resources. Therefore, test resources need to be clear about “who will test” and “where test” and “how test”.

Tester is the most important, which is directly related to the success or failure and efficiency of the whole test project. Tester resources are usually divided into two dimensions:

• Number of test engineers;
• Test engineer’s personal experience and ability;

Testing progress

The test schedule mainly describes the start time, required workload and expected completion time of various tests, and suggests the launch time of the final product based on this.

In the traditional waterfall model, the test schedule is entirely dependent on the time it takes to complete development and deliver the test version; If there is a delay in submitting the development case to beta, then the overall product launch time is also delayed without tailoring the testing requirements;

In agile mode, however, testing occurs throughout the development process, and much of the testing work is done simultaneously with the development work, so that the testing schedule is not entirely dependent on the time it takes to deliver a testable version;

Behavior-driven Development, also known as BDD, only allows non-programmer readable test cases to be written in natural language and associated with natural language-based step descriptions and specific business operations through StepDef. The most typical frame is Cucumber;

Test risk assessment

Schedule changes are often the main reasons for introducing project test risks, such as requirements sheet changes, development delays, discovery of major defects, and staff changes.

Therefore, in the development of the test plan, it is necessary to estimate the potential risks that may exist in the whole test process, and the coping strategy when these risks occur;

summary

A successful test plan must clearly describe the following five important aspects: test scope, test strategy, test resources, test schedule and test risk estimation;

• Test access needs to be clear about what to test and what to expect;
• The testing strategy needs to be clear about what to test first and how to test later.
• Testing resources need to be clear about who is testing and where;
• Test schedule is the need to clarify the start time of various tests, required workload and estimated completion time;
• Testing risk estimates requires figuring out how to respond effectively to potential changes;

Other want to say

In the process of rapid iteration, it is suggested to increase product requirement testing to ensure the consistency of everyone’s understanding of requirements before development and test development.

9) What are the core competencies of software test engineers

At present, test engineers are divided into two categories, one is to do business functional testing, the other is to do test development;

Core competencies that traditional test engineers should possess

In order of importance to test engineers, as follows:

• Test strategy design ability
• Test case design ability
• Quick learning ability
• Exploratory testing of thinking
• Defect analysis ability
• Automated test technology
• Good communication skills

Test strategy design ability

This is the core competitiveness;

Test strategy design ability refers to the ability to quickly and accurately understand the requirements of different software under test, and to define the test focus and the most suitable test method under limited time and resources.

Have excellent test strategy design ability, can very clearly answer these key questions encountered in the test process:

• To what extent the test should be performed;
• What tools are needed for testing;
• How to use automated testing and automated testing framework, and how to select;
• How to allocate testers’ volunteers reasonably;
• How to arrange the test schedule;
• Test how to deal with risks;

Excellent test strategy design ability is not a matter of a day, need to experience a lot of practical experience, but also maintain continuous thinking, take the initiative to refine the common content;

Test case design ability

Test case design ability refers to the ability to design excellent test cases that can efficiently find defects and ensure product quality for all types of tests.

To design test cases well, it is necessary not only to have a deep understanding of the business requirements of the software under test and the usage habits of the target users, but also to be familiar with the specific design and operating environment of the software, including technical architecture, caching mechanism, middleware technology, third-party service integration and so on.

In order to improve the ability of test case design, it is necessary to accumulate more at ordinary times. Only by constantly summarizing and summarizing common defect patterns, typical error types and encountered defects, can systematic use case design thinking be gradually formed.

You can also read some good test case design examples to broaden your mind;

Quick learning ability

There are two levels of meaning:

• Quick learning and understanding of different business requirements and functions;
• Ability to learn and apply new testing techniques and methods;

Learning tools, it is faster to read the official documents directly, because the content is definitely the latest and most authoritative;

When learning new content, we must understand its principle, rather than just stay on the surface, simple operation and use, keep this learning state for a long time, can improve logical thinking and understanding ability to a great extent;

Exploratory testing of thinking

Exploratory testing means that the test engineer keeps learning the system under test in the process of executing the test, at the same time, combing and analyzing more specific test concerns based on incorrect guesses and logical reasoning based on his own experience.

In essence, exploratory test thinking is the inevitable result of the combination of test case design ability and fast learning ability.

Excellent exploratory testing thinking can help achieve low-cost precision testing. The most popular understanding of precision testing can be summarized as the change of the development code, the change point and the change point are tested with clear goals and targeted, which is also one of the main test pushed by agile testing at present.

Defect analysis ability

There are three levels of meaning:

• For discovered defects, combined with the context of errors and background logs, the causes of defects can be predicted or located, and even specific lines of code can be clearly pointed out, which can greatly shorten the repair cycle of defects, and improve the recognition and trust of development engineers to test engineers;
• Based on the found defects, combined with exploratory testing thinking, deduce the possibility of similar defects, and thus find all related potential defects;
• It can reasonably analyze the defect types and defects occurring in a period of time, estimate the health status of the overall quality from point to point, and provide systematic detection and prevention measures for high-frequency defect types, and adjust the subsequent test strategy based on this;

These three levels are successively progressive, the more later the core competitiveness of test engineers can be drawn;

Automated test technology

The core value of automated testing or testing itself, automation is only a means;

Good communication skills

The test engineer’s role in the project is a bit like lubricant:

• On the one hand, it is necessary to coordinate with the product manager and project manager to ensure the normal realization of requirements and the overall quality of the project.
• On the other hand, it is necessary to constantly communicate and coordinate with developers to ensure timely repair and verification of defects;

Communication ability will directly affect the efficiency of business development, is a stepping stone;

Core competencies of test development engineers

Test system requirement analysis ability

In addition to code development ability, test development engineer should have test system requirement analysis ability; Be able to stand at the test architect’s level and identify test infrastructure requirements and application scenarios that improve efficiency;

Wider body of knowledge

Test developers need to have a very broad body of knowledge. You don’t just need to work with traditional test developers because they are users of the test tool or platform you are building; You also need to work closely with CI/CD and operations engineers, because the test tools or platforms you build need to be connected to the CI/CD pipeline and operations monitoring system.

In addition, you will need to understand the higher levels of test architecture deployment and production architecture deployment, and you will need to be familiar with the various technologies used for development.

Performance Test Engineer

The core value of a performance test engineer is not how many performance test tools he knows, but the ability to intuit and locate performance problems, which needs to be cultivated by solid basic theoretical knowledge and a lot of practice.

summary

According to the work content, test engineers are divided into functional test engineers and test development engineers;

For functional test engineer, its core competence include: test strategy design ability, test case design ability, fast learning ability, ability to exploratory testing thinking, defect analysis, test automation technology and good communication skill this seven part, you can be targeted to improve their ability in a particular field, to obtain a bigger development space of “ticket”.

For test development engineers, you need to have excellent test system requirements analysis ability and complete knowledge system, so as to ensure that you design the test work and platform, can better meet the requirements of improving test efficiency.

Why are tests confused

I have never seen the development will be confused, often is the test in confusion, after thinking for a long time, maybe it is these reasons:

• The development of learning routes and development routes are clear, most of the tests are feeling the stones across the river; Especially for functional testing entry, it is easier to confused;
• The test engineer is involved in so many things that he needs to know everything and master nothing, and then he doesn’t know how to develop his career and is overwhelmed by all kinds of chores every day.
• Performance test, automated test bottleneck, self-learning development is difficult;

10) What non-testing knowledge should software test engineers master

Development engineers typically do deep traversal, focusing on points; The test engineer is usually breadth traversal, is concerned with the surface;

• Basic knowledge of Linux/Unix/Windows operating system, traditional relational database technology such as Oracle/MySQL, NoSQL non-relational database technology, middleware technology, Shell/Python script development, version management tools and strategies, CI/CD pipeline design, F5 load balancing technology, Fiddler/Wireshark Tcpdump caught tools, such as browser Developer Tool, etc.;
• From web architecture design, container technology, microservices architecture, Service Mesh, DevOps, cloud computing, big data, artificial intelligence and blockchain technology.

Basically, everything is covered. But people’s time is limited, certainly not so one by one introduction, so the point accounted for more important;

Core knowledge of website architecture

In addition to functional tests, performance tests, stability tests, full-link pressure tests, failover tests, dynamic cluster capacity scaling tests, service degradation tests and security penetration tests are also required.

• For example, if you don’t know the application scenarios and fundamentals of distributed cache clusters like Memcached, if you don’t know the cache breakdown, cache avalanche, cache warm-up, and cache cluster expansion limitations, you can’t design test cases for specific cache system problems.
• Again, for example, if the scalability of the web site architecture design don’t understand, don’t know of the application server load balancing, the basic principle of separation technology without knowledge of the database, speaking, reading and writing, and will not be able to finish such as failover, dynamic cluster capacity expansion, service degradation and related test, at the same time for performance test and whole link may encounter all kinds of bottlenecks in the process of pressure test, It can also be difficult to locate and adjust.

Container technology

For test developers, there are many more scenarios that require application containers. For example, The mainstream Selenium Grid provides an official Docker version that can be directly used to create test execution environments in containers. It’s also easy to quickly set up test execution environments on Cloud computing platforms such as Pivotal Cloud Foundry and Google Cloud Platform.

Selenium Grid based on Docker greatly reduces the workload of upgrading and maintenance of Web Driver, browser version and guardian process version on batch VM nodes.

Test developers can also provide some test tools in the form of Docker images, rather than in the form of traditional installation packages or JAR files, so that test tools can be implemented out of the box.

Cloud computing technology

Learning about cloud computing should focus on how to use the infrastructure and services provided by the cloud;

You can try to use cloud services to deploy your application, but also can combine the various services provided by the cloud platform (configuration services, database services) and your application to do integration;

The conversation thinking

DevOps emphasizes that the development, testing, and operation teams can complete the full cycle management of software statements through the writing and communication of efficient automation tools, so as to achieve more frequent and continuous delivery of high-quality software. The fundamental purpose of DevOps is to improve the delivery ability of the business.

The specific forms of DevOps can be tools, methods and assembly lines, but the deeper connotation of DevOps lies in thinking and methods. With Agile and lean as the core, DevOps discovers problems and solves problems with systematic methods or tools, so as to realize continuous improvement.

My recommended learning path for DevOps is to start with a deep understanding of tools like Jenkins, to skillfully apply and combine plugins to build a flexible and efficient pipeline, and then gradually integrate more tools into the pipeline to accomplish more tasks.

Front-end development technology

From a test engineer’s perspective, mastering front-end development means you can do front-end testing more efficiently and find potential defects more easily. At the same time, you can also build your own test page, to complete the fine test of various front-end components, greatly improving test coverage and efficiency.

As for the learning path of front-end technology, you usually need to master the basic knowledge of JavaScript, CSS, JQuery and HTML5 first, and then learn some mainstream front-end development frameworks, such as angular. js and backbone. js. Of course, the Node.js ecosystem is so developed that if you can master Node.js, a lot of things can be implemented easily.

conclusion

Software test engineers need to master a great deal of non-test expertise, including: web architecture, container technology, cloud computing, DevOps thinking, and core knowledge and practices in front-end development technologies.

11) How should the test strategy of Internet products be designed

Different r&d processes determine different testing strategies

The biggest difference between the research and development of traditional products and Internet products is “fast”;

The great difference in the release cycle determines that the test strategy of traditional software products is not applicable to the test of Internet products, and the test strategy of the two must be greatly different in the test execution time and test execution environment.

The release process usually includes static code scanning, unit testing, compiling, packaging, uploading, downloading, deploying, and testing.

So how to effectively shorten test execution time while ensuring test quality and test coverage?

• The concurrent execution mechanism of test is introduced, and the test execution cluster containing a large number of test execution nodes is used to execute test cases concurrently. Test execution clusters, which can be understood simply as a group of machines dedicated to executing test cases concurrently;
• Secondly, we must find a breakthrough from the testing strategy.

Design of test strategy for traditional software products

The traditional test strategy for software products, such as the pyramid model shown in the figure below, has been regarded as the best practice for test strategy design for a long time.

Unit testing

At the bottom of the pyramid is unit testing, which belongs to the category of white-box testing and is generally completed by the development engineers themselves. Since the earlier defects are found, the lower the repair cost is, the testing strategy of all traditional software products advocates high investment in unit testing.

The API test

In the middle of the pyramid is API test, which is mainly aimed at the exposed interface of each module, usually using gray box test method.

Gray box test method is a test technology between white box test and black box test. Its core idea is to use the code coverage of test execution to know the design of test cases.

Take THE API interface test as an example, first design the test cases of how to call the API in the way of black box, and at the same time, calculate the code coverage in the test execution process, and then supplement more and more targeted test cases according to the code coverage.

GUI testing

At the top of the pyramid is GUI testing, also known as end-to-end (E2E) testing, which is the type of testing that most closely resembles the real user behavior of the software.

It is usually to simulate the behavior of real users using software, that is, to simulate the various operations of users on the software interface, and verify whether the corresponding results of these operations are correct;

The advantage of GUI testing is that it can actually simulate the behavior of real users and directly verify the commercial value of software. The disadvantage is that the code to execute is relatively large. Since automation is used, the use case maintenance and execution code is still large;

In addition, the stability of GUI testing is an important reason to hinder the development of GUI testing for a long time. The random failure rate of GUI tests is high even when many approaches such as retry mechanisms and exception scenario recovery mechanisms are used.

Design of test strategy for Internet products

For Internet products, the pyramid model no longer works;

GUI testing

The on-line cycle of Internet products determines that GUI testing cannot be carried out in a wide range;

• The iterative cycle of Internet products leaves very little time to develop GUI automation test cases;
• The client interface of Internet products changes frequently, and the efficiency of GUI automatic testing is very low.

Therefore, GUI testing of Internet products usually adopts manual testing strategy supplemented by automation.

Manual testing often uses the idea of exploratory testing to test newly developed or modified interface functions, while automatic testing mainly focuses on the basic function verification of relatively stable core business.

As a result, automated testing of guIs tends to cover only the most core scenarios that directly affect the main business process;

GUI testing of traditional software is heavyweight in terms of the number of use cases because of the long test cycle; GUI testing on the Internet is lightweight because the lead time is so short;

The API test

For Internet products, focusing on API testing is the wisest choice;

• The development and debugging efficiency of API test cases is much higher than that of GUI tests, and the code implementation of test cases is more standardized, which is usually to prepare test data, initiate request, verify response these several standard steps;
• API test cases are far more stable to execute than GUI tests;
• Individual API test cases tend to take much less time to execute than GUI tests;
• At present, many Internet products adopt microservice architecture, and the test of microservice is essentially the test of different Web services, that is, API test.
• Changes to API interfaces are generally rare, and even if there are changes, Backward Compatibility needs to be guaranteed in most cases. The basic requirement of backward compatibility is to ensure that the original API calls remain the same.

These characteristics of Internet products determine that API testing can achieve a good input-output ratio, so it should become the focus of the test of Internet products. This is why the testing strategy for Internet products is more like a diamond structure.

The following diagram shows a rhomboid testing strategy that follows the principle of “heavy API testing, lightweight GUI testing, lightweight unit testing”.

Unit testing

Internet products are usually divided into application layer and back-end services, which can be further subdivided into application services and basic services.

Back-end basic services and some common application services are relatively stable, and for the overall system is “affecting the whole”, so it is necessary to carry out a comprehensive unit test back-end services; For client applications that change a lot and non-common back-end application services, unit testing is rarely done.

In addition, for some core algorithms and key applications, such as bank gateway interface, third-party payment integration interface, etc., comprehensive unit tests should also be done.

To sum up, comprehensive unit testing of Internet products is only applied to those relatively stable and core modules and services, while large-scale unit testing of application layer or upper-layer business services is rarely carried out.

summary

Traditional software usually adopts pyramid model test strategy, while Internet products often adopt diamond model.

The rhombus model has the following four key points:

• Focus on the API testing of the middle layer to do a comprehensive test;
• Lightweight GUI testing that covers only the most core E2E scenarios that directly affect the main business processes;
• The highest level of GUI testing usually uses exploratory testing thinking to find as many potential problems as possible through manual testing;
• Downsizing testing takes a dial-and-conquer approach, with comprehensive unit testing for stable and core services and modules, and minimal unit testing for application or upper-layer businesses.

12) From 0 to 1: Your first GUI automated test

This chapter mainly introduces the idea of automation and the principle of Selenium. Interested students can read here.

13) Efficiency is king: decoupling of script and data + Page Object model

Problem: Test scripts have both test data and test operations, all in one script

Test script and data decoupling

The test input data is saved separately, and the input data of the test script is replaced by variables. A common one is CSV, read per line;Copy the code

This model is called data-driven;

The benefits are:

• Solved the problem of a large number of duplicate scripts;
• Other data can also be stored in the file for judgment;

Page Object model

The core idea of "Page object model" is to encapsulate the controls on the page and some operations of the controls in the unit of the page. Test cases use page objects to perform specific interface operations.Copy the code

14) Get closer to the business abstraction: Let automated test scripts better describe the business

How to control the granularity of operation functions

The granularity of operation function refers to how many operation steps an operation function should contain.Copy the code

• If the granularity is too large, the reusability of functions will be reduced.
• If the granularity is too small, it will lose the significance of function encapsulation.

Generally, it takes the completion of a business process as the main line, and extracts the set of high cohesion and low coupling operation steps;

Business process abstraction

Business process abstraction is a higher level of abstraction based on operational functions that are closer to the actual business. Test cases based on business process abstraction are often flexible and can be easily assembled according to the actual test requirements.Copy the code

The core idea of business process is to guide the encapsulation of testing business process from the business dimension.

15) Stumbling block: Talk about test data during GUI automation

From the technical means of creation, there are three main methods to create test data:

• API calls;
• Database operation;
• Integrated use of API calls and database operations;

In terms of creation timing, there are two main methods for creating test data:

• Create test data in real time during test case execution;
• Create test data “out of the box” prior to test case execution;

16) GUI tests can be played this way (Page Code Gen + Data Gen + Headless)?

• GUI test data automatic generation, mainly based on the type of test input data and the corresponding custom rule library implementation, and for multiple test input data, can be based on cartesian product to automatically combine a complete set of test cases;

• Headless browser, which can not see the running interface in the execution process, has the advantages of fast speed and simplified environment construction;

• Page object automatic generation technology, the basic idea is that there is no need to manually maintain Page Class, only need to provide the URL of the Web, it will automatically generate the location information of all the controls on the Page, and automatically generate Page Class, no open source, more commercial tools;