Puppeteer-to-istanbul collection run time CUCUMBER automated test coverage

Automated testing is an important part of front-end engineering. Compared with unit testing, unit testing focuses on the general condition of every smallest unit in the code and the verification of edge conditions.

Automated testing, on the other hand, is more about checking and validating page elements from the UI level, or the user’s perspective. The existence of automatic test ensures the stability of production environment.

The automation framework described in this article uses CucumberJS and puppeteer uses headless to automate the process and puppeteer-to-istanbul to run the coverage collection for automated tests.

Start with an introduction to the automation framework used in front end engineering, CucumberJS.

cucumber

Cucumber is an open source tool based on the Executable Specifications of the BDD pattern. Behavior-driven development (BEHAVIor-driven Development) is an agile development technique that encourages collaboration between developers, testers, and non-technical people.

Apart from automated acceptance testing, Cucumber is more important than Cucumber to use a common language to build a bridge between the technical and non-technical members of the team.

• Executable: You can execute code like Java, Ruby… Run these specifications as well to validate and accept the target application. This is, of course, from a technical point of view;
• Specification: Triggered from a non-technical point of view, they are more concerned with a clear description of the system’s functionality: what the system can do under what circumstances than with validation itself.

By creating a non-developmental language that can be maintained by team members in a way that defines statements, the expectations of the developer and the acceptance team can be agreed upon.

Here is an example of a scenario use-case definition:

Function: Use ATM fixed amount withdrawal usually "withdrawal" menu contains several fixed amount, using these fixed amount withdrawal can avoid the keyboard input withdrawal amount, thus can accelerate the transaction, improve the efficiency of withdrawal. Scene outline: A Fixed Amount Withdrawal If I have a "<accountBalance>" balance in my account when I choose a "< 款 Amount>" withdrawal THEN I should receive a "<receivedAmount>" and My balance should be "<remainingBalance>" meta example: | accountBalance | withdrawAmount | receivedAmount | remainingBalance | | | | | 100.00 100.00 1000.00 900.00 | | 500.00 | | | | 0.00 500.00 500.00Copy the code

Cucumber is available in multiple languages and can be integrated into mainstream testing frameworks. We are using the JavaScript version here – CucumberJS.

In the process of writing test cases, there are three main steps. Non-technical personnel are responsible for the compilation of the above feature file. Technical personnel are responsible for the definition of specific statements, ensuring the feasibility, and finally running test cases to make the document/test case meet the expected execution.

1. Create feature file;
2. Generate test Step Definitions;
3. Run the test case.

English Keyword	Chinese simplified equivalent(s)
feature	function
background	background
scenario	Scene/script
scenarioOutline	Scene outline/script outline
examples	example
given	Suppose/suppose/assume
when	when
then	then
and	Besides/at the same time/at the same time
but	but

After executing the test case, you can use the open source Cucumber reporting tool to view the resulting report. Here are the common cucumber-reporter.

macaca

If you haven’t tried cucumberJS yet and are keen on its BDD approach to automated testing, try MacACA.

Macaca is an open source multi-terminal automation solution based on Cucumber of Ali. It offers a range of solutions, including computer vision (openCV, etc.), mock schemes, page element inspection tools, reporters, continuous integration, and more.

Macaca-reporter generates the following report results of brain map patterns, which are more concise and attractive than regular cucumber- Reporter.

Macaca is recommended mainly because it should be easier to find Chinese solutions than cucumberJS when you have deployment difficulties. And its ecological construction has provided a complete automation solution.

Cucumber Coverage

Whether Cucumber or Macaca, the presentations are different, but the data are basically the same. Then we’ll see that the percentage calculation in the report is usually the pass rate of the number of failed uses divided by the total number of uses. Why is there no overall coverage?

Sure, Cucumber can run out of the Istanbul report.

However, the one that can run out of coverage is actually cucumber for UT. In other words, you can use Cucumber to write unit tests and honestly report unit test coverage.

Unit tests using Cucumber are as follows:

// filename: rocket-launching.steps.js
const { Given, When, Then, And, But, Fusion } = require( 'jest-cucumber-fusion' )

const { Rocket } = require( '.. /.. /src/rocket' )
let rocket

Given( 'I am Elon Musk attempting to launch a rocket into space'.() = > {
    rocket = new Rocket()
} )

When( 'I launch the rocket'.() = > {
    rocket.launch()
} )
Copy the code

Here, cucumber is used for automatic testing, and cucumber is used in the front end to search and operate corresponding page elements with the headless browser, so as to get expected and actual results. You don’t actually run the front-end code, so you don’t get code coverage.

Then you start with a headless browser.

puppeteer

Puppeteer is a Node library that provides a high-level API for controlling Chromium or Chrome via the Devtools protocol. Puppeteer runs in Headless mode by default, but can be run in Headless mode by modifying the configuration file.

Headless Chrome allows you to run a test script on the command line without opening the browser. You can complete all user operations just like the real browser without worrying about external interference or using any display device. Make automated testing more stable.

Puppeteer has the following functions:

• Generate page PDF.
• Grab SPA (single-page application) and generate pre-rendered content (that is, “SSR” (server-side rendering)).
• Automatic form submission, UI testing, keyboard input, etc.
• Create an automated test environment that is constantly updated. Perform tests directly in the latest version of Chrome using the latest JavaScript and browser features.
• Capture the Timeline trace for the site to help analyze performance issues.
• Test the browser extension.

const puppeteer = require('puppeteer');
puppeteer.launch().then(async browser => {
  const page = await browser.newPage(); // Open the browser
  await page.goto('https://example.com'); // Open the sample page
  await page.screenshot({path: 'screenshot.png'}); / / screenshots
  await browser.close(); // Close the browser
});
Copy the code

Class: Coverage v0.9.0

Coverage collects information about the JavaScript and CSS sections used on the relevant page.

Examples of using JavaScript and CSS coverage to get the initial percentage:

// Enable JavaScript and CSS overrides
await Promise.all([  
  page.coverage.startJSCoverage(),  
  page.coverage.startCSSCoverage()
]);

// Navigate to the page
await page.goto('https://example.com');

// Disable JavaScript and CSS overrides
const [jsCoverage, cssCoverage] = await Promise.all([
  page.coverage.stopJSCoverage(),  
  page.coverage.stopCSSCoverage()
]);

let totalBytes = 0;
let usedBytes = 0;
const coverage = [...jsCoverage, ...cssCoverage];
for (const entry of coverage) {  
  totalBytes += entry.text.length;  
  for (const range of entry.ranges) usedBytes += range.end - range.start - 1;
}
console.log(`Bytes used: ${usedBytes / totalBytes * 100}% `);
Copy the code

useIstanbulTo output a coverage table, seepuppeteer-to-istanbul.

Methods

• Coverage. StartCSSCoverage v0.9.0 (options)
• Coverage. StartJSCoverage v0.9.0 (options)
• V0.9.0 coverage. StopCSSCoverage ()
• V0.9.0 coverage. StopJSCoverage ()

Use puppeteer-to-istanbul to collect run-time coverage

Check the Puppeteer API for Coverage related operations that allow us to gather runtime Coverage. Puppeteer-to-istanbul written in the corresponding report.

(async() = > {const pti = require('puppeteer-to-istanbul')
  const puppeteer = require('puppeteer')
  const browser = await puppeteer.launch()
  const page = await browser.newPage()
 
  // Enable both JavaScript and CSS coverage
  await Promise.all([
    page.coverage.startJSCoverage(),
    page.coverage.startCSSCoverage()
  ]);
  // Navigate to page
  await page.goto('https://www.google.com');
  // Disable both JavaScript and CSS coverage
  const [jsCoverage, cssCoverage] = await Promise.all([
    page.coverage.stopJSCoverage(),
    page.coverage.stopCSSCoverage(),
  ]);
  pti.write([...jsCoverage, ...cssCoverage], { includeHostname: true , storagePath: './.nyc_output' })
  await browser.close()
})()
Copy the code

Merge Coverage

However, the results are not perfect. Every use case execution will cover the results, so it is necessary to combine the coverage results of all features.

const pti = require('puppeteer-to-istanbul');
let featureCoverage = []; // Global collection coverage

Before(async function () {
  await featureScope.coverageStart();
});

After(async function () {
  await featureScope.coverageStop();
  featureCoverage = [...featureCoverage, ...featureScope.coverageJs, ...featureScope.coverageCss];
});

AfterAll(function() {
  pti.write(featureCoverage);
});
Copy the code

Ignore the files generated in Coverage

At this point, we basically get the result we want, but if we look closely at the generated runtime JS file, we will find that there are many common library files, which do not need to be overwritten, so we ignore them in the generation process:

let coverage = await page.coverage.stopJSCoverage();

// Exclude chai.js
coverage = coverage.filter(({url}) = >! url.includes('chai'));
puppeteerCoverage.write(coverage);
Copy the code

conclusion

From cucumber to Puppeteer, and finally to PTI, this paper explores the purpose scheme to be used step by step, and finally refines the code in the scheme to get the results we want.

During UI automation, we also need to set up coverage to verify the completeness of our test cases. Using macACA recommended in this article is a good solution for secondary development on the basis of MACACA.

reference

Cucumber uses advanced stages

CucumberJS

Puppeteer In Chinese

puppeteer-to-istanbul