background

Software testing is the process of using manual or automatic means to run or test a software system, its purpose is to check whether it meets the specified requirements or to find out the difference between the expected result and the actual result. [1]

From the point of view of whether or not to care about the internal structure of software and concrete implementation, testing methods are mainly divided into white box testing and black box testing. White box testing methods mainly include code inspection, static structure analysis, static quality measurement, logical coverage, basic path testing, domain testing, symbol testing, path coverage and program variation. Black box test methods mainly include equivalence class division method, boundary value analysis method, error conjecture method, causal diagram method, decision table driving method, orthogonal experimental design method, function diagram method, scene method and so on.

Software Testing classification

1. White box test

White box testing is also called structure testing, transparent box testing, logic driven testing or code based testing. White box testing is a test case design method where the box refers to the software being tested and the white box refers to the box being visible, meaning it is clear what is inside the box and how it works. The “white box” method has a comprehensive understanding of the internal logical structure of the program, and tests all logical paths. The “white box” method is an exhaustive path test. In using this scheme, the tester must examine the internal structure of the program, starting with the logic of the program, and derive test data. The number of independent paths through the program is astronomical

2. Black box test

Black box testing, also known as functional testing, tests each function to see if it works. In testing, the program as a can’t open the box, in a completely regardless of the program under the condition of internal structure and internal characteristics, in the test program interface, it only check whether the program function normal use according to the terms of the requirements specifications, whether the program can properly receive input data and produce the correct output information. Black box testing focuses on the external structure of the program, not considering the internal logical structure, and mainly tests the software interface and software function.

3. Exploratory testing

Exploratory testing can be said to be a testing thinking technique. It doesn’t have many practical testing methods, techniques, and tools, but it is a way of thinking about testing that all testers should master. Exploratory emphasizes the subjective initiative of testers, abandons the complicated test plan and test case design process, and emphasizes the timely change of test strategy when encountering problems.

The most straightforward definition of exploratory testing is: design tests and execute tests simultaneously. Exploratory testing is sometimes confused with AD hoc testing. Impromptu testing is usually an ad-hoc, impromptu bug-finding testing process. By definition, anyone can do an impromptu test. Exploratory testing, proposed by Cem Kaner, is a refined, thoughtful process compared to impromptu testing.

Types of front-end tests

Unit Test

Unit test, also known as module test, mainly aims at the test of the smallest testable unit in the program (generally referring to methods and classes). It has the characteristics of small investment and high yield. It can find the code defects early and is suitable for the test of public function library.

Unit tests are the easiest to implement: libraries of utility classes shared by multiple components in your code, subcomponents shared by multiple components, and so on.

In general, there must be unit tests in common functions/components to make sure the code works. Unit tests should also be the most numerous and have the highest coverage in your project

Integration testing

Integration testing is usually applied to functions/components with high coupling degree, functions/components with secondary encapsulation, functions/components composed of multiple functions/components, etc.

Integration test mainly includes module interface test, functional module test integrated by sub-functional modules, etc. The purpose is to verify whether the subsystems and sub-functions integrated by all modules still meet the quality objectives on the basis of unit test.

The purpose of integration testing is to test whether the modules combined together can work properly after unit testing. The combined code as a whole is tested against the exposed interfaces to see if the combined code works as expected.

E2E test

E2E(End to End) testing is used to verify software quality from the perspective of software users. For example, open a browser and perform a series of operations to verify whether the interface or function meets expectations. Is the user’s real use of the scenario simulation.

In the rapid iteration of the product now, there is an automated test, is an important guarantee of reconstruction, iteration. For the Web front end, the main test is whether forms, animations, page jumps, DOM rendering, Ajax, etc., work as expected.

Front-end automated testing – egg rolls and pyramids

Dynamic testing is a testing method that uses specific software to control the testing process and compare the actual results with the expected results. Through the test automation, the testing behavior of the software can be transformed into the automatic testing behavior of the machine, so as to replace a lot of manual testing operations, so that the test can be carried out quickly and repeatedly.

Cone model

A straightforward approach would be to write more high-level tests, such as lots of system tests, to cover more low-level tests, since higher-level tests cover more coverage. Of course, there were still some underlying tests that couldn’t be covered, so we wrote separate unit tests to supplement them, and we ended up with a model like this: an ice cream cone

However, this is time-consuming and laborious, and sometimes the test link of a complex system is extremely troublesome. At this time, we need to change the way of thinking.

Pyramid model

Pyramid model, which divides testing from bottom to top into unit testing, integration testing, and end-to-end testing (E2E testing /UI interface testing). As you go down the pyramid, testing becomes cheaper and more efficient, and as you go up the pyramid, testing becomes more expensive and less profitable.

Front-end automation test framework choice

1.1 Classification of test frameworks

1.1.1 Browser Control Framework

It is mainly used to control elements on the browser or app. Common frameworks include:

Web: Selenium/TestCafe/Cypress/Playwright/Puppeteer/Protractor/Nightmare

Mobile app: Appium/Robot Framework

1.1.2 Manage the test case design framework

It is mainly used to organize the design of test cases and the structure of test cases. Common frameworks include:

All: Cucumber

JS: Mocha/Jest/Jamine/TestCafe/Cypress/CodeceptJS

Ruby: Rspec

Java: TestNG/Junit

Python: pytest

1.1.3 Managing the Test Case execution framework

It is mainly used to organize the execution of test cases. Common frameworks include:

All: Cucumber

JS: Mocha/Jest/Jamine/TestCafe/Cypress/CodeceptJS

Ruby: Rspec

Java: TestNG/Junit

Python: pytest, unittest

1.1.4 Managing the test case reporting framework

Mainly used to organize test results and presentations, common frameworks are:

All: Cucumber/Allure

JS: Mocha/Jest/Jamine/TestCafe/Cypress/CodeceptJS

Ruby: Rspec

Java: TestNG/Junit

Python: pytest, unittest

1.1.5 Assertion Framework

These are mainly used to test assertions about results. Common frameworks include:

All: Selenium

JS: Chai/Jest/Jamine/TestCafe/Cypress/CodeceptJS

1.1.6 Coverage framework

Mainly used to collect code coverage, common frameworks are:

JS: Istanbul/Jest/blanket

Ruby: Rcov

Java: jacoo / cobertura

1.1.7 Visual test framework

It is mainly used for visual testing. Common frameworks include:

JS: Cypress/CodeceptJS/Applitools

1.1.8 support Mock/Spy/Stub

Mocks are used to mock requests. Common frameworks include:

Sinon/Cypress/CodeceptJS

1.1.9 Integrated Framework

Do not need to integrate with other frameworks to complete the entire test activity. Common frameworks include:

Cypress/CodeceptJS/TestCafe

1.2 Comparison of advantages and disadvantages of each testing framework

Tools/Frameworks

advantage

disadvantage

Selenium

• Free open source

• Support for multiple languages: Java/Python/Ruby/JS…

• Cross-platform support: Windows/Mac/Linux/Mobile (IOS/Android)

• Support multiple browsers: Chrome/Firefox/Safari/Edge

• Support the Headless, CI/CD

• Test case design: Support PageObject/ Snapshot

• Test case execution: Support concurrency

• Easy to get started

• Browser version compatibility needs to be considered

• Poor debugging and slow execution

• Stability is moderate and Retry is not supported

• Need to manage test design and execution with other testing tools: BDD, LLDB: Cucumber

• Needs to be used in conjunction with another test assertion framework: CHAI

• LLDB Test execution and report presentation: LLDB TestNG/ReportNG

• Visual testing is not supported

Cypress

• A complete framework that does not need to be used with other frameworks

• Open source free version and paid version, easy to get started

• Support cross-platform: Windows/Mac/Linux

• Support for Web/ unit/integration testing

• Browser: Chrome/Firefox/Edge/Electron

• Support for Headless, CI/CD(Edge, XX limits)

• Test case design: BDD/ snapshot/video recording

• Test case execution: fast execution, concurrent support, various execution modes, no need to integrate with third-party frameworks; Support element global automatic wait; Analog bandwidth

• Test case debugging: Visual debugging window

• Good test stability: Supports Retry

• Test results: Good report HTML page, support for Flaky Tests, support for Coverage report

• Support visual testing

• Only supports JS

• Safari testing is not supported (it may be later)

• Mobile testing is not supported and the test compatibility for Mobile Web apps is not very good

• Test case design: PageObject is not supported; Multiple tabs are not supported. Iframe can be problematic in some cases; Hover /scroll, file upload;

• API/ unit testing support is not very friendly and is not recommended if you are doing API testing alone

CodeceptJS

• A complete framework that does not need to be used with other frameworks

• Open source, free, easy to use

• Support cross-platform: Windows/Mac/Linux

• Support Web/Mobile(Android/IOS)/ integration testing

• Browser: Chrome/Firefox/Safari/Edge…

• Support the Headless, CI/CD

• Test case design: BDD/ snapshot

• Test case execution: fast execution, concurrent support, various execution modes, no need to integrate with third-party frameworks; Support element global automatic wait

• Good test stability: supports various levels of Retry

• Test result: good report HTML page

• Support visual testing

• Flexible switching various popular driver, test maintenance cost is low (webdriver/playwright/puppeteer/testcafe/appium/nightmare/protractor/API helpers, graphql)

• Good compatibility with third party frameworks and easy to expand

• Only supports JS

• API testing support is not very friendly and is not recommended for standalone API testing

• In some cases, video recording is supported: switch between different drivers, where LLDB: WebDriver

• Analog network speed is not supported

• Test case debugging does not have a visual window, but does have debugging commands

• Test case report: No coverage report currently available

TestCafe

• Open source, free, easy to use

• Support cross-platform: Windows/Mac/Linux

• Browser: Chrome/Firefox/Safari/Edge/mobile browsers…

• Support the Headless, CI/CD

• Test case design: PageObject/ Snapshot /Video

• Test case execution: fast execution speed, support concurrency, support element automatic waiting; Script changes are automatically executed

• Independently manage test case design and execution, assertions, and test reporting

• Only supports JS

• Compatibility: Mobile native apps are not supported

• Native support is not very friendly, it is a proxy server that communicates with the Web browser

• Does not support BDD and needs to be integrated with other tools

• Visual testing is not supported

Playwright

• Microsoft development, open source free, easy to use

• Support cross-platform: Windows/Mac/Linux

• Supports multiple languages: JS/Java/Python/C#

• Web/ Emulated Mobile(Android/IOS)

• Browser: Chrome/Firefox/Webkit based Apps

• Support the Headless, CI/CD

• Test case design: Built-in Inspector element

• Test case execution: fast execution speed, support concurrency, support element automatic waiting; Monitor simulated network requests; ** Supports native mouse/keyboard operation; Upload and download files; ** Support cross-page /frame/ TAB operation, dark mode operation

• API testing is not supported

• Test case debugging does not have a visual window, but does have debugging commands

• There is no way to independently manage test case design and execution, requirements and other frameworks

• Visual testing is not supported

Puppeteer

• Open source and free, higher than Cypress

• Support cross-platform: Windows/Mac/Linux

• Support Web/Mobile(Android/IOS)/ integration testing

• Browser: Chrome/Firefox, its own installation package and Chrome drive, support more native browser operations (native mouse/keyboard operation; Upload files)

• Support the Headless, CI/CD

• Test case design: Snapshot /BDD

• Test coverage: Collect front-end code coverage

• You can collect page performance data

• Lots of downloads

• JS support, poor compatibility with other languages

• Compatibility: Firefox and other browsers are not very friendly

• Community activity is moderate

• Just an API for Chrome Driver, there is no way to independently manage test case design and execution required with other frameworks

Protractor

• Open source and free, higher than Cypress; Setup is not convenient and needs to be used with Selenium

• Developed for AngularJS app

• Support cross-platform: Windows/Mac/Linux

• Support Web/Mobile (Android/IOS)

• Browser: various browsers, supporting more native browser operations (support native mouse/keyboard operations; Upload files)

• Support the Headless, CI/CD

• Only supports JS

• There is no way to manage test case design and execution independently. It needs to be integrated with other frameworks, where: Db: Jasmine/Mocha

Nightmare

• An API library for browser automation

• Open source and free, higher than Cypress

• Support cross-platform: Windows/Mac/Linux

• Browser: The electron Engine is used and does not need to be compatible with the browser

• Support the Headless, CI/CD

• It’s very fast and very stable

• Does not truly reflect compatibility

• Need to be used with other testing frameworks

• Only supports JS

Appium

• An automated testing framework for mobile Apps (Android/IOS)

• Multilingual support

• Open source free, general, easy to build

• Support image comparison

Robot Framework

• An automated testing framework for mobile Apps (Android/IOS)

• Multi-voice support

• Open source, free, relatively easy to build

Article 2 tools

Tools are generally more user-friendly and tester friendly than frameworks.

However, tools are rarely free, and the degree of customization is not very high, and sometimes cannot meet the needs of project customization.

Jest

Mocha

Selenium/WebDriver

Nightwatch

TestCafe

Cypress

Supported test types

Unit testing

Unit testing

E2E test

E2E testing, Node.js unit and integration testing

E2E test

Unit testing, integration testing, E2E testing

Support language

JS, Node

JS, Node

Java, Python, C#, JS

JS, Node

JS, TypeScript

JS

Whether visual testing is supported

no

no

no

no

no

is

Whether to come with an assertion library

is

no

no

is

no

is

Out of the box

is

no

no

Yes (Cumbersome installation and configuration)

is

is

Recording generation script

/

/

support

Does not support

Does not support

Does not support

Github Star

35.2 k.

20.5 k.

20.8 k.

10.7 k.

8.9 k.

31.3 k.

The document

Jest document

Mocha document

The Selenium document

Nightwatch document

Testcafe document

Cypress document

3, conclusion

With the ability to support unit testing, integration testing, and E2E testing at the same time, Cypress provided a complete test solution that met our needs. In addition, Cypress supports JS test case writing, Jquery element location selector, Headless and CI continuous integration, fast running, low cost to get started, and has a visual debugging interface for easy problem location

Cypress practice is the sole criterion for testing truth

1, install,

• yarn add cypressornpm install cypress
• After the installation,./node_modules/.bin/cypress installInstall the Cypress environment (including GUI tools)

2, configuration,

Cypress mainly includes the following two boot modes:

1) Command line execution of NPX cypress open: the test case set interface will be opened in the browser, which needs to be manually run.

2) Command line execution of NPX cypress run: all specified test cases will be run in headless browser mode without visual interface, but videos of the whole test process will be recorded during running, which can be viewed under cypress/ Videos directory.

Of course, in addition to running the above commands directly from the command line, you can also configure the scripts field of package.json to define the startup mode.

• Package. json: Configure both GUI and non-GUI (terminal) modes to run Cypress

  "scripts": {
      "cypress": "cypress run",
      "cypress-gui": "cypress open",
  Copy the code

  Copy the code

⚠️ After the configuration, run YARN Cypress [-gui] or NPM run cypress[-gui](parentheses mean optional) to initialize cypress and generate default configurations and directories

• Cypress. json(the same directory as package.json): Cypress provides a flexible configuration that allows you to customize your behavior to suit your needs, such as the following configuration I have for a project

• Visual interface operation

If we need to test in visual interface, after package.json is configured, we just need to run NPM run cypress:open to start Cypress and realize visual debugging. If we encounter the following error during startup, To restart cypress, run the NPX cypress install-force command.

If you successfully start Cypress, you will see the following screen. The examples directory contains Cypress’s own test case demo code (you can delete the test cases if you don’t need them later). Clicking on one of the test cases will automatically open the browser to run the test cases.

3. Introduction of project catalog

4. Run in headless browser mode

If we want to run in headless browser mode, after package.json is configured, we need to execute NPM run cypress:run, and cypress will run all of the specified test cases in headless browser mode

5. Write test cases

Next, using the example of verifying the search function of a Baidu page, I will show you how to write test cases that can end with.spec.js or.js and put into Cypress/Integration.

Create the test.js or test.spec.js test case file in cypress/ Integration as shown in the project directory below.

Next, you can start writing test cases in test.js. Cypress supports Jquery element selectors and Chinese character selectors, and also supports chain operations. Cypress automatically waits for an element to be operable before executing a command or assertion.

// <reference types="cypress" /> context() => {it(); Verify the search function ', () = > {cy. Visit (' https://www.baidu.com '). Then (() = > {/ / 1. Enter the search content cy.get(".s_ipt").should("exist").type("Cypress automated test "); / / 2. Click on the baidu the button cy. Get (". S_btn "). The contains (" baidu ") should (" exist "). Click (); Get ("#content_left").find("div").then(ele => {expect(ele. Length).gt(0); }); }); }); });Copy the code

npm install cypress-upload-file --save-dev
Copy the code

/ / / < reference types = "cypress" / > context (' file upload, () = > {it (' validation file upload function, () = > {/ / access page, Let file = "file/cover.jpg"; cy.get("input[type='file']").attachFile(file); }); });Copy the code

/// <reference types="cypress" /> const path = require("path"); The context (' file downloads, () = > {it (' validation file download function, () = > {/ / access page, download operation, Const downloadsFolder = cypress.config ("downloadsFolder"); Const downloadedFilename = pa. join(downloadsFolder, "download file.xls "); ReadFile (downloadedFilename).then(data => {// perform the assertion, step omitted}); }); });Copy the code

npm install mocha --save-dev
npm install mochawesome --save-dev
Copy the code