preface

Recently, I have been doing some front-end tests for more than a year, from small programs to shop decoration, which are basically pure front-end work. At the beginning of the transition from back-end testing to front-end testing, I was at a loss for front-end things, and no one in the team had done pure front-end testing, so I could only sum up experience while walking on holes. Then, the points prone to problems are formed into a system, and constantly summarize and explore, and finally form a set of front-end testing solutions. Here, there will be a good front-end quality assurance system to summarize, hope to communicate with you.

Let’s take a global look at the technical architecture of the awesome front end and what quality assurance things are done for each different level:

The awesome Node technology architecture is divided into business layer, basic framework layer, common components and basic services layer. We pay more attention to the basic framework, common components and business layer code. Node business layer does two things. One is the client layer that provides page rendering, which is used to interact with C-side users, including style, behavior JS, etc. The other is the Server layer that provides data services, which is used to assemble various interfaces provided by the background and complete the c-side interface encapsulation.

For each different layer, we did a few things to ensure quality, including:

  • For the entire business layer of the UI automation, core interface | page dial test;
  • Sentry alarm for client layer;
  • Interface test for the Server layer, business alarm;
  • Unit testing for infrastructure and common components;
  • Version change alarm for common component changes;
  • Maintenance of process specifications, use cases, etc. for online publication.

Let’s talk about the quality assurance work of these dimensions.

I. UI automation

Many people think that UI automation is expensive and cost-effective to maintain, but why is it at the top of the list of awesome front-end quality assurance systems?

Former server-assisted user interaction, simple interface testing, unit testing can’t reflect the user’s path, and concluded, from the past experience because of uncontrollable factors lead to the release of the functions of A and B cannot be used, especially the core simple scenario is unavailable from time to tome, so every time before an application, The core functions provided by this application will be executed once. With the continuous accumulation of business, more and more test scenarios need regression, resulting in a huge workload of regression. In order to reduce labor costs, we urgently need to release labor through automation, so UI automation of core process regression is put at the core.

UI automation, of course, the biggest pain points is indeed a maintenance cost, to reduce maintenance costs, we will page is divided into component dimensions, dimensions, and provide the unified package to deal with the general public components, special page logic, encapsulate common methods, such as initializing the browser information, choice of environment, login, network switch, click get element content, type, etc., The business regression use case only needs to focus on its own use case action steps.

1. Frame selection

Puppeteer [1] is a Node library maintained by Chrome. It drives Chrome or Chromium browsers based on DevTools and supports headless and non-headless modes. The official website provides very rich documentation, easy to learn.

There are many UI automation frameworks, including Selenium, Phantom; After comparison, puppeteer is relatively light, and only one NPM package is required to use it. It is event-driven and is more stable and performs better than Selenium’s wait polling. Plus, it’s Chrome native support, provides all of chrome’s supported apis, and our business scenarios only need to override Chrome, so it’s the best choice.

Mocha [2] + Mochawesome [3], Mocha is a mainstream testing framework, supporting beforeEach, before, afterEach, after hook functions, Assert assertions, test suites, use case orchestration, etc. Mochawesome is a third-party plugin for the Mocha testing framework that generates beautiful HTML/CSS reports.

There are also many js test frameworks to choose from, such as Mocha, AVA, Jtest, etc. Mocha is chosen because it is more flexible. Many configurations can be combined with third-party libraries, such as Report, which combines Mochawesome to generate good-looking HTML reports. Assertions can be replaced with powser-assert.

2. Scripting

  • Package base library
    • Encapsulates the initialization process of PC and H5 browsers
    • Encapsulates PC and H5 logins in a unified manner
    • Encapsulate the page model and component model
    • Unified operation method for encapsulating upload component, date component, Select component, and so on
    • Encapsulate input, click, hover, tap, scrollTo, hover, isElementShow, isElementExist, getElementVariable and other methods
    • Provides unified support for obtaining page elements and operation methods in the form of “HTML Tags >> Page text”
    • Encapsulate baseTest to add unified operations at the start and end of use cases
    • Encapsulate Assert and increase assertion logging
  • The business case
    • Installing the Base Library
    • Orchestrate business use cases

3. Execute logic

  • Execution by environment
    • Added pre-online environment code change trigger, online environment automatic execution
  • Monitor source code changes
    • Added GitLab Webhook to monitor source code merge master automatically in the pre-live environment
    • Added GitLab Webhook to monitor test case changes automatically executed in production
  • Daily scheduled execution
    • Added crontab to execute the online environment periodically every day

Two, interface test

Interface test is mainly aimed at the Server layer of Node. According to our development specification, Node does not do complex business logic, but it needs to provide dubbo interface for a transformation of service-oriented applications, or combine multiple Dubbo interfaces. Provide an HTTP interface for h5/ applets to render data, and the transformation process brings all kinds of data acquisition, combination, transformation, forming a new end-to-end interface. At this time, the automation of servitization interface alone could not guarantee the full coverage of the upper interface, so we also conducted automatic tests for Node interface. In order to use the unified testing framework within the test, we request the HTTP interface provided by Node through Java, so how to evaluate the quality of the interface test after the use cases are written? Does it fully cover all business logic? At this point, an effective method is needed to obtain the coverage of the test to check which scenarios are not covered in the interface test, so as to better detect the gaps.

Istanbul [4] is the industry’s easy-to-use JS coverage tool that uses module-loaded hooks to calculate statement, line, method, and branch coverage in order to transparently increase coverage when executing test cases. It supports all types of JS coverage, including unit testing, server-side functional testing, and browser testing.

However, our interface use cases are written in Java code and arrived at the Node server through Http requests. It is not js single test or browser function test. How can we obtain the coverage of the Node interface?

The solution is to add the cover parameter: –handle-sigint, which starts the service by adding the –handle-sigint parameter. When the service receives a sigINT signal, it tells Istanbul to generate coverage. This command works well for us and thus forms a model of our interface coverage:

1. Istanbule -- handle-SIGint Start service 2. Perform test case 3. Send SIGINT to end istanbule to obtain coverageCopy the code

Finally, we solved our Node interface coverage problem and built it automatically through Jenkins continuous integration

You can add the.istanbule.yml file to the root path to exclude or specify files that need to be counted

verbose: false
instrumentation:
    root: .
    extensions:
        - .js
    default-excludes: true
    excludes:['**/common/**'.'**/app/constants/**'.'**/lib/**']
    embed-source: false
    variable: __coverage__
    compact: true
    preserve-comments: false
    complete-copy: false
    save-baseline: false
    baseline-file: ./coverage/coverage-baseline.json
    include-all-sources: false
    include-pid: false
    es-modules: false
reporting:
    print: summary
    reports:
        - lcov
    dir: ./coverage
    watermarks:
        statements: [50, 80]
        lines: [50, 80]
        functions: [50, 80]
        branches: [50, 80]
    report-config:
        clover: {file: clover.xml}
        cobertura: {file: cobertura-coverage.xml}
        json: {file: coverage-final.json}
        json-summary: {file: coverage-summary.json}
        lcovonly: {file: lcov.info}
        teamcity: {file: null, blockName: Code Coverage Summary}
        text: {file: null, maxCols: 0}
        text-lcov: {file: lcov.info}
        text-summary: {file: null}
hooks:
    hook-run-in-context: false
    post-require-hook: null
    handle-sigint: false
check:
    global:
        statements: 0
        lines: 0
        branches: 0
        functions: 0
        excludes: []
    each:
        statements: 0
        lines: 0
        branches: 0
        functions: 0
        excludes: []
Copy the code

Unit testing

Unit testing is at the bottom of the pyramid in the testing layer. If unit testing is done properly, it can filter out most of the problems and find bugs in advance, which can also reduce the cost of bugs. After carrying out the single test for a period of time, we found that in the admirable Node framework, the server side of the business layer only does interface assembly, while the client side faces the browser, which is not suitable for unit test. Therefore, we only conducted single test for the basic framework and general components, ensuring that the basic service can eliminate most problems through single test. For example, general store information service in the basic framework, single test check store information acquisition; For example, a page-level commodity component can be tested to see if the HTML rendered by the commodity component is the same as the original.

The single test scheme has tried out two frameworks:

  • Jest[5]
  • ava[6]

The Jest scheme, which supports Matchers mode assertions, is recommended. Support for Snapshot Testing, which tests whether the HTML rendered by component class code is correct; Supports a variety of mocks, including mock method implementations, mock timers, mock dependent modules, and so on. Support Istanbule for easy access to coverage.

In short, front-end single test scheme is becoming more and more mature, need front-end developers pay more attention to JS single test, the bug in the cradle.

Four, the base database change alarm

Above, we have carried out unit tests on basic services and basic components, but single test cannot completely guarantee that the changes of the basic library are completely problem-free. With the introduction of the new version of the basic library in the business layer, bugs will be further brought into the business layer and ultimately affect the normal use of C-end users. So how to ensure that every time the business layer introduces a new version of the base library, there is a full regression? How do you make business test students more sensitive to base library changes? In response to this situation, we set out to build a base library version change widget. The implementation idea is as follows:

1. Compare a master code commit or merge request to determine if there is a specific base library version change in package.json 2. Send the code comparison of the two versions of the corresponding base library to the test leader 3. Judge the scope of use cases of this regression according to Changelog 4. Added GitLab Webhook so that only code merged into the merge publishing branch or master branch will trigger checksCopy the code

The introduction of this gadget allows testers to be informed of what needs are being changed to the base component and what aspects are most affected by the upgrade, thus avoiding relatively black box testing.

The first version of the implementation of the most simple function, the subsequent deep digging requirements, can do front-end code change precision test.

V. Sentry alarm

When I first got in touch with the front-end test, THERE was no tracking of any errors reported by JS, so I had a lot of trouble in troubleshooting and locating problems. Therefore, we started to introduce sentry alarm monitoring, which is used to monitor the running condition of js environment online.

Sentry [7] is an open source bug tracking tool that helps developers monitor and fix crashes in real time.

At the beginning, our access method was relatively simple and crude. Direct global access caused the problem that the alarm information was very large, and info and WARN information would be printed out after the global report.

After the change, the posture for using sentry is:

  • Sentry global information is reported and filtered
    • Error type: TypeError or ReferenceError
    • Error user > 1K
    • The error appears in the JS file
    • Error > 2 stores
  • Increase the active reporting of abnormal processes of core services

Finally, the filtered error information is sent to the alarm recipient by email, and the alarm is cleared in a fixed time.

Six, business alarm

In addition to the Sentry monitoring alarm, the service alarm of Node interface layer is also an essential part, which can detect service exceptions in the interface provided by Node in time. This part is the development and operation of the students to do, including Node framework access log system at the bottom; Correctly report the error level, error content, and error stack information at the service layer. Add a proper alarm policy to the log system. When the threshold is exceeded, alarms are generated by SMS or phone calls, facilitating timely fault discovery and troubleshooting.

Service alarms can quickly respond to production environment problems. If an alarm occurs after a release, we roll back the alarm immediately to ensure online stability.

Vii. Convention norms

In addition to the above testing and warning methods, we also did some basic construction such as process specification and use case maintenance, including:

  • Release the specification
    • Multiple daily branches are merged and published
    • Limit release time
    • Standardize the release process
    • Organize the core inspection points of self-test
  • Baseline use case library
    • The P0 core use cases for different businesses are updated regularly
    • Project use cases are regularly updated to the business regression use case library
    • Online problem scenarios are updated to the regression use case library

At present, the good front-end test routine is basically like this, of course, some usual efforts are not fully expanded, such as adding return value structure comparison in interface test; Add online interface or page dial test [8]; Conduct training on self-test case design for development, etc. There are also many new functions being explored, such as access to the traffic comparison engine, the online traffic to the pre-launch environment, the code before the launch of the comparison test; Add UI automation screenshot comparison; Explore UI automation of applets and more.

Reference links:

  • [1] github.com/GoogleChrom…
  • [2] www.npmjs.com/package/moc…
  • [3] www.npmjs.com/package/moc…
  • [4] github.com/gotwarlost/…
  • [5] github.com/facebook/je…
  • [6] github.com/avajs/ava
  • [7] docs.sentry.io
  • [8] tech.youzan.com/youzan-onli…