At present, Maoyan has completed the construction of code coverage tools for various front-end projects and put them into practice in some projects. This article mainly introduces the front-end code coverage tool construction ideas and practice process, hope to relevant students for reference, welcome to exchange.

background

In Maoyan, there are various front-end projects, including PC terminal, mobile terminal, small program and client (MRN). Meanwhile, the business iteration speed is fast, and QA and RD do not have quantitative indexes when conducting tests and self-testing. To provide an automated way to quantify test results and drive test improvement, work began on developing front-end code coverage tools.

The goal is to build a multi-endpoint code coverage tool that supports incremental data statistics.

Tool structure

The tool is divided into five parts, including code staking, coverage data collection and reporting, coverage data storage, coverage report generation, and message notification.

  • Code staking: Different staking tools were selected according to different projects. Nyc was selected for PC, mobile, and client (MRN). Istanbul – Lib-Instrument for the secondary development of small program staking was used for the small program side, as described below.

  • Coverage Data collection and reporting: Coverage – Reporter is an NPM package that is embedded in the front end project and reports data when users visit the target page. The applet side reporting function is integrated into cat’s eye QA tool and will not be introduced in this article.

  • Coverage data store: Coverage-Admin is a back-end service that provides an interface for reporting data. S3 is a storage service that stores reported data.

  • Coverage report generation: Coverage-report is a Node project that uses toolkits such as Istanbul -lib-coverage, Istanbul – Lib-Report, and Istanbul – Reporters to generate reports, which are called by Jenkins through scripts.

  • Message notification: Coverage-report After the report is generated, maoyan internal message platform interface is invoked to send coverage related messages to users.

The details of each section are described below.

Code in the pile

To achieve code coverage, you first need to peg your code. The current popular piling tools in the industry are Babel-plugin-istanbul and NYC. Based on the characteristics of cat’s Eye project and considering the universality of the solution, we finally choose NYC as the piling tool.

Nyc uses the Istanbul lib-instrument to do this, and Istanbul lib-Instrument takes advantage of Babel’s ability to disassemble source code into an abstract syntax tree and reedit that abstract syntax tree. Add an abstract syntax tree of the probe code where necessary, and finally use Babel to convert the abstract syntax tree into code to accomplish code staking.

The whole piling process is integrated in the publishing platform, which provides the function of executing custom scripts and controlling the script execution process by passing different environment variables. The tool inserts are divided into two types:

  • Full piling: Piling for all files under the target folder.
  • Incremental piling: Piling files that change due to the development of new requirements under the target folder.

Insert provides two way of pile is in response to different scenarios: full amount when the pile is suitable for functional regression, for the whole project or part of the function test, the test focus on which part of the code is no longer the old function added, which part of the code to be adding new features, more attention is the function of the whole project is normal; Incremental staking is suitable for requirements testing, where the testing is more focused on code with new requirements changes.

Small program end pile

Maoyan small program users over 100 million, ensuring its quality is also a top priority. I ran into problems using NYC to pin applets.

Nyc peg is used. After the peg is inserted, the code has the following two changes compared with the original code:

  • Coverage data content at the front of the file.
  • A large amount of probe code inserted into the file.

This causes the code volume of small programs to become larger. Due to wechat’s size limit for small program subpackage (2MB), packages using NYC peg may not pack preview properly.

Taking a simple addition code as an example, the result is shown below:

It is these two additions that cause the size of the code to grow. If we can reduce these two parts, hopefully we can keep the package size under 2MB, so we made the following changes to Istanbul Lib-instrument:

  • Coverage data is no longer inserted into the code.
  • Modify the logic that records the number of times code is executed.

The details are as follows:

① The circled selection is to remove the coverage data content; ② ③ Circled the modified counting logic. In the process of staking, the logic of the codeVisitor was modified so that when the original codeVisitor encountered the statement, method, branch, etc., the location information of the content was recorded into the coverage data and a counter was added for the corresponding content. This part is removed from the revised logic as follows:

// istanbul-lib-instrument/src/source-coverage.js

newStatement(loc) {
    const s = this.meta.last.s;
    this.data.statementMap[s] = cloneLocation(loc);
    this.data.s[s] = 0;
    this.meta.last.s += 1;
    return s;
}

/ / modified
newStatementWithoutMap(loc) {
    const s = this.meta.last.s;
    this.meta.last.s += 1;
    return s;
}
Copy the code

The probe code execution method generated by the original piling logic is to use the object to directly perform self-addition. In this way, the name of the coverage object will repeatedly appear in the code file, such as COv_70RPPF3jl ().s[0]++ and cov_70rppF3jl ().s[1]++. In order to minimize the side effects of piling, Change the probe code to call uniform methods, while minimizing the length of code written when calling methods, as follows:

// Original pile logic pile code excerpt
cov_70rppf3jl();
cov_70rppf3jl().s[0] + +; add =(p1, p2) = > {
  cov_70rppf3jl().f[0] + +; cov_70rppf3jl().s[1] + +;return p1 + p2;
};

// New pile logic pile code excerpt
cov_70rppf3jl();
cadd("s".0);
add = (p1, p2) = > {
  cadd("f".0);
  cadd("s".1);
  return p1 + p2;
};
Copy the code

Cadd method is the extracted probe method. For details, please refer to the piling result diagram after the piling logic is modified above.

The data reported

Coverage data generated by front-end projects are stored in each end. To collect coverage data, each end needs to actively report it. Therefore, a toolkit (reporter) is developed for this purpose.

In order to ensure that data is not lost, on the basis of periodic reporting, the reporter adds page refresh and close event monitoring, and reports data once when the event is triggered. In addition, manual reporting is supported. Manual reporting depends on the debugging console. Reporter object is assigned to Window, and manual reporting can be called through the console when testing.

At the same time, to avoid excessive duplicate coverage data, each reporter generates an 8-bit ID when it is created, and each reporter retains only one copy of data per hour.

Data is stored

The coverage data generated by the front-end project is a JSON object, which is not suitable for storing in a traditional database. We chose S3 to store coverage data and store coverage data as files.

When reporter reports data, the project name, test branch, and commitHash value of the tested project are also reported. When stored in S3, the path is project name/test branch /commitHash. When generating the report, you can pull data from different dimensions as needed.

Report generation

The coverage report was generated using a series of kits provided by Istanbul, and the following flow chart is used:

Incremental report

When conducting requirements testing, QA students pay more attention to the coverage of the new code of the tested requirements. Incremental piling has been introduced in the previous article. Incremental piling is the incremental at the file level, which will be identified when the file is changed. The incremental code is the code level increment, can be accurate to the line, in every change file there are code changes, code changes are “new”, “change”, “delete”. Here, we think of “new” and “changed” code as incremental code.

The tool’s delta report currently supports only delta statements, which are calculated as follows:

The formula involves two indicators: 1. The number of overwritten new statements; 2.

The tool uses code DIff to analyze incremental code. Through the code DIFF, you can get the condition of each line in the file that changes between two branches. Through the condition of new lines, you can further analyze the condition of new statements.

New statement rule: If a statement contains new lines, it is considered new.

Overwritten new statement rule: If a statement is new and overwritten, it is considered overwritten.

/ / pseudo code
// Data is a coverageMap object with statementMap, branchMap, functionMap, s, b, and f
// lines is the new array of line numbers in the diff result of the file

const { statementMap, s } = data;
// Get the Map of the new statement
Object.entries(statementMap).forEach(([k, location]) = > {
  const { start, end } = location;
  const { line: startLine } = start;
  const { line: endLine } = end;
  for (let i = 0; i <= lines.length; i++) {
    // Determine if there are any new lines in this statement
    if (lines[i] >= startLine && lines[i] <= endLine) {
      // If so, load it into incrementStatementMap
      incrementStatementMap[k] = location;
      break; }}});// Get the index of the new statement
const incrementStatementNum = Object.keys(incrementStatementMap);

// Get the overridden new statement
Object.entries(s).forEach(([k, v]) = > {
  // Check whether the statement is overwritten
  if (v > 0) {
    // Determine whether the overwritten statement is a new statement
    if (incrementStatementNum.indexOf(k) > -1) {
      incrementStatementMap[k].covered = true; incrementCoveredS[k] = v; }}});// Add a Map to the statement, including location information
data.incrementStatementMap = incrementStatementMap;

// Add an overwritten statement counter
data.incrementCoveredS = incrementCoveredS;
Copy the code

After the above code processing, the coverageMap object has two new properties: incrementStatementMap and incrementCoverdS. The following Istanbul Reports are generated by parsing these two new properties to show the addition and coverage of rows in the report.

/ / pseudo code

// istanbul-reports/lib/html/annotator.js

// Parse the new statement
/ / fileCoverage: CoverageMap objects in the generating report process, the sub-objects are statementMap, branchMap, functionMap, S, B, F, incrementStatementMap, incrementCoveredS
// structuredText: the source code object in the report generation process, is an array, the elements are the contents of each line of code
function annotateIncrementStatements(fileCoverage, structuredText) {
  const { data } = fileCoverage;
  if (data.incrementStatementMap && data.incrementCoveredS) {
    const { incrementStatementMap } = data;
    Object.entries(incrementStatementMap).forEach(([k, location]) = > {
      const { start, end, covered } = location;
      const { line: startLine } = start;
      const { line: endLine } = end;
      structuredText.forEach(text= > {
        // A single statement in a front-end project may contain multiple lines. To make the report clearer, mark each line in the new statement
        if (text.line <= endLine && text.line >= startLine) {
          // mark as new
          text.addStatement = 'yes';
          // Calculate the number of new statements
          text.addStatementNum = Object.keys(incrementStatementMap).indexOf(k) + 1;
          // Determine whether to overwrite
          if (covered) {
            text.addStatementCovered = 'yes';
          } else {
            text.addStatementCovered = 'no'; }}}); }}})// istanbul-reports/lib/html/index.js
function detailTemplate(data) {
	// ...
  const statementAdd = line= > {
    // line.added: The added line in the diff result is set to yes
    if (line.addStatement === 'yes' && line.added === 'yes') {
      if (line.addStatementCovered === 'yes') {
        return '<span class="cline-any cline- addstatements-yes "> The new statement was overwritten${line.addStatementNum}</span>`;
      } else if (line.addStatementCovered === 'no')
        return '<span class="cline-any cline-addStatement"> No new statement was overwritten${line.addStatementNum}</span>`; }}// ...
}
Copy the code

The partial effect of a file in the incremental report is shown below:

The global effect of incremental reporting is shown below:

Data consolidation

In a real requirements test, a final coverage report is expected at the end of the test, documenting how much code was covered throughout the test process. Istanbul -lib-coverage provides a method to combine coverage data of the same version:

// istanbul-lib-coverage/lib/file-coverage.js
merge(other) {
    if (other.all === true) {
        return;
    }
    if (this.all === true) {
        this.data = other.data;
        return;
    }
    Object.entries(other.s).forEach(([k, v]) = > {
        this.data.s[k] += v;
    });
    Object.entries(other.f).forEach(([k, v]) = > {
        this.data.f[k] += v;
    });
    Object.entries(other.b).forEach(([k, v]) = > {
        let i;
        const retArray = this.data.b[k];
        if(! retArray) {this.data.b[k] = v;
            return;
        }
        for (i = 0; i < retArray.length; i += 1) { retArray[i] += v[i]; }}); }Copy the code

For different versions of coverage data, the direct sum policy will not be used. To obtain merge coverage results for a branch under test, merge coverage data from different versions.

Merge strategy: the results of coverage data of the same version are merged by the Istanbul lib-coverage method. When the coverage data of different versions are merged, the coverage data generated by the code of the latest version is used as the basis to merge the coverage data of the unchanged file and discard the coverage data of the changed file.

When merging coverage data of different versions, the first step is to determine whether the file has changed. At present, the tool compares whether statementMap, branchMap and functionMap serialized by two versions are equal. If all maps are unchanged, This indicates that there is no change in the position of statements, branches, or methods throughout the file.

alerts

The report was triggered by Jenkins, and the generated report was directly uploaded to S3 for storage. In order to let users know the test results, after the report was generated, the coverage-report (reporting tool) conducted a simple analysis on the coverage data and summarized the coverage data. When the user selects to generate a full report, a message body in the following format is generated:

[Git repository] Test branch: [current test branch] Test time: 2021/06/05-11:00 ~ 2021/06/07-23:59 Report address: #338 6/7/2021, 20:30:10 =============================== Coverage summary =============================== Statements : Functions: Branches: 21.39% (124/3091) Lines: 36.71% (1117/3043) = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = =Copy the code

When the user selects to generate an incremental report, a message body in the following format is generated:

[Test Item] Incremental code coverage report Updated Repository address: [Git repository address] Original branch: [original branch] --> Current testing branch: [Current testing branch] Test time: 2021/06/05-11:00 ~ 2021/06/07-23:59 Report address: #339 Click to view report time: 6/7/2021, 20:32:30 =============================== Coverage summary =============================== Statements : Functions: Branches: 29.29% (266/582) Lines: 25.19% (559/2219) IncrementStatements: 21.47% (292/1360) = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = =Copy the code

Subsequent planning

At present, maoyan has 7 internal projects to access the front-end code coverage tool, including PC, mobile, client and small program. Users also raise some problems, such as cumbersome configuration, lack of detailed data analysis, and trend change of no data. The coverage report was triggered by Jenkins, and the workflow was limited by Jenkins. In order to quickly analyze the data and generate the report, no more operations could be done.

There are still several areas of work that need to be done with front-end code coverage tools:

  • Incremental reports add more metrics, such as incremental branch coverage, incremental method coverage, and so on.
  • A more refined multi-version coverage data merge strategy.
  • Unified management of data and reports, a comprehensive display of code coverage at each test stage.

In the future, we will develop a code coverage platform based on the code coverage index, manage the code coverage of front and back end projects in a unified manner, analyze data from multiple dimensions, and produce more valuable reports to escort the products of Cat’s Eye.

Recruitment information

Cat’s eye quality team is continuing to recruit, welcome interested students to send resume: stamp here