TLDR;

preface

Recently in the research of E2E automated testing, I found that the introduction of automated testing on the market is not good, so this paper will talk about UI automated testing from three aspects of concept introduction, technology selection, solution implementation.

The concept is introduced

Composition of automated tests

It was started by Mike Cohn (one of the founders of Agile development)

In his book “Succeeding With Agile” he first introduced the concept of a test pyramid.

As can be seen from the picture above, the whole pyramid model consists of three layers:

  • Unit Tests
  • Service Tests
  • UI Tests

The whole pyramid model represents the higher level of test integration, slower execution, and the lower level of test isolation, faster and lighter execution.

Here are some rules of thumb from the test pyramid:

  • Build tests of different granularity

  • The higher the level of testing, the less testing there should be, develop a reasonable testing strategy

To maintain the shape of the test pyramid, a fast, maintainable test combination with reasonable coverage should look like this:

  • Lots of small, fast unit tests
  • Comprehensive interface testing
  • A little UI testing

They say the best practices in the industry look at Google, and Google’s automated test layering ratio is:

  • Unit tests (70%)
  • Interface testing (20%)
  • UI testing (10%)

UI testing VS E2E testing

  • UI Test (User Interface Test)

    User interface testing

  • E2E (End-to-end Test)

    End-to-end testing, which simulates real user scenarios

The top of the testing pyramid (UI testing) is not “UI testing” in the literal sense here, which is misleading. For modern front-end applications, UI testing focuses on whether the product’s UI interaction is correct. It can be done by simulating the back end and doing it in unit tests.

E2E tests, on the other hand, are tests that need to simulate real user scenarios to check whether the whole system works in the right way.

So the broad definition of “UI Tests” (UI Tests that test the pyramid) can be thought of as E2E Tests.

What kind of scene is appropriate

  • Requirements are stable and do not change frequently
  • The UI interface is stable with little change
  • Long project cycle
  • Lots of regression testing

Technology selection

There are several common E2E testing solutions in the market

  • WebDriver based: Selenium, Appium
  • Based on the CDP: Puppeteer, offender
  • Inject Script: Cypress

Next, we will analyze and compare the implementation principles of different schemes one by one

WebDriver

WebDriver is a W3C standard browser Remote Control Protocol (Protocol).

www.w3.org/TR/webdrive…

Client is our test code, Browser is the Browser we want to control, the middle will have a WebDriver protocol implementation, is Browser Driver.

Using Language Binding on the Client side, you can implement multi-language test scripts (Java, Ruby, Python, C#, Javascript etc…).

Browser Driver side, is the implementation of each Browser based on WebDriver, used to remotely control the Browser

When the test script is executed: Initiates an HTTP request and generates a JSON request based on the JSON Wire Protocol, which is sent to the Browser Driver

The Browser Driver receives requests through the HTTP server, which then decides to execute instructions in the Browser

The execution status is sent back to the HTTP Server, which is then sent back to the automation script

CDP

Full name: Chrome Devtools Protocol

Namely: Chrome Developer Tools protocol

Chromedevtools. Making. IO/devtools – pr…

The Chrome Driver implementation in Selenium is also based on CDP secondary development to control the browser.

Compared with Webdriver, CDP can perform more low-level operations, such as checking, debugging, and listening network traffic.

Puppeteer is a CDP-based Nodejs implementation

Inject Script

Representative work: Cypress

Most testing tools, such as Selenium, run by running outside the browser and executing remote commands on the network. Cypress, on the other hand, executes in the same lifecycle as the application.

Technically, Cypress uses Webpack to pack test code into a bundle and run it in an iframe. When cypress is first loaded, the Cypress Web application hosts itself on a random port locally. After recognizing the first cy.visit() command, Cypress will change the local URL to match the origin of your remote application (to meet the same origin policy — same domain name, protocol, port) so that the test code and application run in the same Run loop.

Scheme comparison

WebDriver CDP Inject Script
Selenium Appium Puppeteer Playwright Cypress
advantages multilingual

cross-platform		 multilingual

Support native and Hybird		 Base CDP Base CDP

Support for multiple browsers

Perfect test plan		 Out of the box
disadvantages Selenium Appium Puppeteer Playwright Cypress
Suitable for the scene Standard automation scheme Mobile terminal real machine test Headless browser operation E2E test on PC Chrome E2E test

Plan be born

Finally, for common business scenarios, I chose Codeceptjs, an E2E test integration framework, which unified the API of the user layer and allowed me to choose a variety of different test schemes. The offender can be ourselves, WebDriver, Puppeteer, Protractor, TestCafe, and Nightmare on the Web side, and Appium and Detox on the mobile side.

  • Use the ourselves on the PC side
  • Use Appium on the APP side

Before the project lands, there is one more question to determine:

  • Where is the E2E test code? (There are two options)
    • Follow project code: Make it easy to iterate as the project changes
    • Independent warehouse maintenance: independent maintenance, decoupled from project warehouse

    Both options have advantages and disadvantages. The first option is suitable for project developers to maintain the test code, while the second option is suitable for testers to maintain the test code independently

Initialize the project

Install codeceptjs

npm install -D codeceptjs
Copy the code

Initialize the project

npx codeceptjs init
Copy the code

Select test framework

? What helpers do you want to use? ❯◉, the infection of WebDriver Protractor infection infection infection Puppeteer infection infection Appium Nightmare infection FileSystemCopy the code

After selecting ourselves, we may enter some custom configurations and start installing the appropriate test framework and creating a set of files. When the installation is complete, you will be prompted to create a test file, enter the name of the file, and have fun writing the test code.

/ / directory structure | - the output / / test report folders folder | | - specs / / test code - login_test. Js | - codecept. Conf., js / / codecept configuration file | - jsconfig. Json | - package. Json | - steps_file. Js / / binding instance object for some public methodsCopy the code

Based on the sample

Feature('login')

Scenario('Jump to home'.async ({ I }) => {
  await I.amOnPage('/')
  await I.seeInCurrentUrl('/login')
  I.fillField('[name="login-phone"]'.1234567890)
  I.fillField('[name="sms-code"]'.123)
  I.click('login'.'[type="submit"]')
  await I.see('managers'.'.role-item')
  I.click('text = "manager")
  I.click('text = "determine"')
  await I.seeInCurrentUrl('/home')})Copy the code

The code above is an E2E test of the base login process, and the offending API can be viewed here

Page Object Model

POM is the most basic design pattern in automated test code. The core idea is to decouple interaction from business, and extract the reusable interaction layer code.

Create a PageObject page

NPX codeceptjs gt = NPX codeceptjs generate: pageObjectCopy the code

POM code

// pages/User.js
const { I } = inject()

class User {
  async login (mobile, token) {
    await I.seeInCurrentUrl('/login')
    I.fillField('[name="login-phone"]', mobile)
    I.fillField('[name="sms-code"]', token)
    I.click('login'.'[type="submit"]')}async chooseRole (roleName, actionName = 'determine') {
    const hasRoleName = tryTo(() = > I.see(roleName, '.role-item'))
    if (hasRoleName) {
      I.click(`text="${roleName}"`)
      I.click(`text="${actionName}"`)}}}module.exports = new User()
module.exports.User = User
Copy the code

Business test code

// specs/login_test.js
Feature('login')

Scenario('Jump to home'.async ({ I, User }) => {
  await I.amOnPage('/')
  await User.login(1234567890.123)
  await User.chooseRole('managers')
  await I.seeInCurrentUrl('/home')})Copy the code

BDD

Full name: behavior-driven Development

English: Behavior driven development

BDD is an agile software development technology. The general process is shown as follows:

  • The business side, product managers, developers, and testers discuss and define requirements together
  • useGherkinTo write test cases that describe business scenarios
  • According to the developerGherkinWrite test cases to write code
  • The tester based onGherkinWrite test cases to accept tests

One of the most important features of BDD is that non-developers write test cases that are DSLS (Domain-specific languages) written in natural languages.

This helps to bridge the gap between developers’ understanding of the requirements for building the product and business people’s understanding of the technical difficulties that arise from the requirements.

Gherkin is a DSL originally proposed by Cucumber, a BDD testing framework.

Advantages:

  • Support for multiple languages
  • Business people can read it
  • Can describe software behavior

Gherkin takes files with the suffix. Feature as the carrier.

Each Gherkin scenario has a basic pattern that includes conditions (if), events (when), and consequences (then)

// features/login.feature
# language: zh-CNFunction: Login scenario: Jump to the home page If enter the project root path, then enter the login page when login account 1234567890 when select the manager role, then enter the home pageCopy the code

Write test code for the corresponding steps

// step_definitions/steps.js
const { I } = inject()
// Add in your custom step files

Given('Enter project root path'.() = > {
  I.amOnPage('/')
})

Then('Go to login page'.async() = > {await I.seeInCurrentUrl('/login')
})

When('Login account 1234567890'.async () => {
  I.fillField('[name="login-phone"]'.1234567890)
  I.fillField('[name="sms-code"]'.123)
  I.click('login'.'[type="submit"]')
})

When('Select the Role of manager'.async() = > {await I.see('managers'.'.role-item')
  I.click('text = "manager")
  I.click('text = "determine"')
})

Then('Go to home page'.async() = > {await I.seeInCurrentUrl('/home')})Copy the code

CI/CD

Implementation of CI/CD based on GitLab Pipline is mainly divided into two steps:

  • Smoke extraction test (manually trigger gitlab Pipline)
  • Pre-release (automatic trigger of gitLab API)

You just need to write.gitlab-ci.yml

e2e dev:
  stage: test
  image: mcr.microsoft.com/playwright:focal
  before_script:
    - yarn install
  script:
    - yarn e2e:dev
  only:
    - web // Only in gitlab Manually triggered on the Web
  artifacts:
      when: always
      reports:
        junit: 
          - output/result.xml

e2e pre:
  stage: test
  image: mcr.microsoft.com/playwright:focal
  before_script:
    - yarn install
  script:
    - yarn e2e:pre
  only:
    - triggers // Only in gitlab api Call time trigger
  artifacts:
      when: always
      reports:
        junit: 
          - e2e/output/result.xml
Copy the code

The test report

Finally, Codeceptjs supports many reporters, which can be configured according to individual needs

codecept.io/reports/

Since the Gitlab Pipline Tests show only support junit, you only need to configure junit for Codeceptjs by referring to the XML Report.

reference

  • Read the test pyramid
  • About Web -UI automation testing
  • The Selenium WebDriver architecture
  • Detailed introduction to the Cypress framework – Small Pineapple test notes
  • Application of UI automation test based on BDD concept in Ctrip vacation
  • Playwright
  • Codeceptjs