One, foreword
Code test coverage is a measure that describes how well a program’s source code is tested. It is a means of white-box testing that visually exposes blocks of code that cannot be covered by test cases. As a sharp tool to improve code quality, iQiyi development team and QA team have made some exploratory attempts and practices on how to access and use it.
Currently, code coverage is mostly at the overall level, not down to smaller granularity. In the collaborative development process, if the requirements can be refined to the level of granularity coverage, associated with a specific person in charge, the test efficiency and accuracy will be effectively improved.
This article describes how to refine code coverage statistics in combination with DevOps tools such as code management and project management to help improve code quality from a code coverage access and usage perspective.
Second, the background
Internet company software development mode is more agile and iterative development model, makes up the weaknesses of the traditional development mode, with a higher success rate and efficiency of development, can the product to the market as soon as possible first, again through the participation of users, the optimization of products, constant revision rapid iteration is achieved, and better adapt to market changes.
However, as the project iterates and the number of functional scenarios increases, the code engineering becomes larger and more complex, and code quality becomes critical. How to improve code quality in the development test cycle, code test coverage as a standard to measure code quality, how to access and use the coverage data, is a direction worthy of in-depth exploration and practice.
- define
> Codecoverage (English: Codecoverage) is a measure in software testing that describes the percentage and extent to which source code in a program is tested. The resulting percentage is called Codecoverage.
- Meaning of code coverage
1. Improve test cases: It is clear from the code coverage report which code was executed and which code was not. For code that has not been executed, the tester needs to think about whether it is a code logic design problem or a test case problem. If it is a code logic problem, then you need to communicate with developers and product people to achieve a consistent understanding of the requirements. If it is a test case problem, you need to fill in the missing test scenarios to ensure that all important scenarios are covered as much as possible to avoid service failures caused by untested code coming online.
2. Improve code quality: Coverage reports can help developers understand the logic of the code by providing insight into the execution flow of the code. Analyzing code coverage reports in production environments can also distinguish between actual user request execution coverage and “useful”/” useless “code, which helps optimize and simplify code logic.
3. Code quality standards: Code coverage is an exact value that quantifies vaguely defined code quality with accurate data, and is a measure of white box testing and code quality.
From the code coverage report, it can be clear which code has been executed and which code has not been executed. The code that has never been executed can be calculated to determine whether the test case is sufficient, helping testers to clarify the code logic and improve the test case. As an important standard to measure code quality, code test coverage can find hidden problems in the testing stage, eliminate hidden problems in the bud, and reduce the occurrence of online problems.
- The status quo
There are many open source coverage implementation tools, but in practical use, still need to do more preparatory work, often encountered the following problems:
- Incremental code cannot be distinguished
Incremental code refers to the newly modified code. The advantage of distinguishing incremental code from the full code is that users can only focus on the code they are responsible for, so as to remove interference and reduce the cost of manual filtering. Coverage tools are only tools for generating code coverage (full), not incremental code.
- High use cost
Coverage report is a comprehensive report, and users usually only care about their own code coverage. In the process of collaborative development, how to relate a certain code coverage to specific requirements (developers, testers) is a problem.
- The test environment is highly variable and costly to maintain
In the actual development and test cycle, there may be the phenomenon of modifying the code and then testing, and then modifying the code after testing, and the code coverage is a changing process. Considering that coverage tools can only collect fixed code, how to reduce the maintenance cost of this change process needs to be addressed.
Objectives and functions
Based on the above problems, iQiyi development team integrated DevOps tools on the basis of coverage generation tools to quickly access generated code coverage at a relatively small cost and support the calculation of fine-grained incremental code coverage reports at branch level and requirement level. The following uses Java as an example to briefly describe the implementation scheme and principle.
- Simplify access Process
Detailed coverage reports can be generated automatically by users concerned only with pushing produced Java files, class files, and exec files to specified locations.
- Incremental code reporting
Diff algorithm is used to distinguish incremental code and generate separate incremental report and full report.
- Code branch level code coverage
Calculate the code coverage in the test branch.
- Requirement level code coverage
Calculate code coverage for one of the requirements (issues) in Jira.
Fourth, technical implementation
The following uses Java as an example to briefly explain the implementation principle. Considering the current coverage tools with better maintainability, the JaCoCo open source tool was selected.
Operation principle
JaCoCo requires three dependencies to generate coverage reports: the source Java file, the compiled class file, and the resulting exec file for code probe piling execution.
1. Source Java files and compiled class files are pushed to the server by CI Job, and exec files are exported to the server by JVM through TCP connection.
2. Summarize the three-part files and use JaCoCo tool to generate the full code coverage report of the code.
3. Obtained diFF results of test branch and master branch code from Gitlab platform, namely incremental code, and combined with other three parts of data, generated incremental code coverage report.
4. Generally, the test code branch is the result of multi-person collaboration. Many feature code branches will be merged into the test branch, and then the initial source branch of incremental code can be obtained through git commit traceability relationship.
5. Finally, the Jira requirement is associated with the code branch to obtain the code coverage report covered by the requirement.
Realization of core technology
- JaCoCo transformation
JaCoCo coverage reports are generated based on Java source code, compiled Class files, and generated EXEC (JVMExecution Data, type.exec) coverage data. Exec coverage data is binary files generated by probe probe, including SessionInfo (source ID, start time, end time of the data) and ExecutionData (probe Boolean array corresponding to each class file). Jacoco officially provides three ways to export exec data: File System, TCP Socket Server and TCP SocketClient. The existing methods are limited by the need for flexible automatic reporting and can result in direct data loss if the JVM is abnormally shut down. Therefore, the TCP Socket Client is optimized and modified to support the scheduled reporting function. By configuring scheduled parameters, the scheduled reporting function can be flexibly configured.
- Generation of incremental code coverage
Use git tools to compare the test branch with the master branch and get incremental code (new and modified code). According to the incremental code, find the matching Jacoco exec file, and then use ASM framework to analyze and calculate the incremental coverage data.
- Code branch level coverage
After the incremental code coverage is obtained, the original commit branch of all code is traced through git commit information.
- Requirement level code coverage
In order to facilitate the association of code and requirements, and to quickly identify the relevant development and test leaders, there are some specifications for naming code branches.
Retrieve the code conforming to the specification from the code branch, query the Jira project management system, and make correlation. The code branch level coverage for the Jira requirement is the code test coverage for the requirement.
Requirement type | branch name prefix | example
—|—|—
With version requirements | feature/v {version} / {issue_id} _xxx | feature/v11.7.0 _xxx JIRA – 9443
Pure back-end demand | feature/backend / {issue_id} _xxx | feature/backend / _xxx JIRA – 9444
- Use requirement
In order to improve the code quality and reduce the occurrence of online problems, the coverage of incremental code is strictly required, and the reason should be noted if the coverage is below the standard.
| | | standard
—|—
| the deliverable standard incremental coverage > = 90% | | all code
| test via a standard branch incremental coverage > = 90% | | demand
| | code online standard code branch incremental coverage | > = 90%
High coverage is not the same as high code quality, but low coverage indicates the presence of untested code that can become a hidden problem with service stability when it goes live. But blindly pursuing high coverage is not desirable. On the one hand, invalid code (such as nulltype code) may not be overwritten; On the other hand, the human cost of pursuing high coverage will be relatively high.
- Requirements test launch process
[Requirements Test online process]
Testers develop test cases according to the requirements. After testing and verification in the test environment, they measure whether the test coverage is up to the standard according to the generated requirements test coverage report (check whether the coverage is up to the expectation, judge whether the coverage is up to the key branches, etc.). If the requirements are not up to the standard, supplementary use cases or code logic repair by developers should be carried out and repeated tests should be carried out. After reaching the standard, requirements can be approved and put into production. In this process, the platform will automatically generate the demand of the associated changes in code coverage, avoiding the complex collaboration environment affect demand for the other code, testers can quickly find uncovered code, and compared with on-line process through, reduce uncovered code hidden problem online, to a certain extent to ensure the quality of service.
V. Summary & planning
IQIYI were introduced in this paper the development team and QA team in test coverage of exploration and practice, in the further study of the base code test coverage tool, combining with other collaboration platform, support more rapid integration method, calculation shows the incremental code report, branch coverage report, demand coverage report, such as more dimension reports, It provides users with great convenience. After the establishment of the platform, the platform can quickly access more services and provide more dimensions of reference measurement standards for testers. The test coverage of key back-end application service code increases from less than 70% to 90%+, effectively ensuring the quality of service to a certain extent.
The current coverage data is derived from the test environment and is limited to the development test cycle. Subsequent code coverage in production course to make some attempts, for example, using the production environment of code coverage report automatic screening of some key interface request as dial test cases will produce environmental traffic requests, test environment contrast coverage report of production environment and test environment difference to measure the effect changes, etc.