Shengcong Cheng – Product Manager of CODING

Times bring testing challenges

Testing has always been an important guarantee for the quality of the software development process, while in the traditional research and development mode, testing activities are always a relatively backward link in the software life cycle, I believe we are familiar with the following dialogue scenarios:

  1. After planning requirements:

    “This version of X is a key feature that must be tested!”

    “Okay, we’ve made a note on the requirements list.”

  2. During development:

    “Is the function done this time?”

    “It’s done, but it’s not finished yet.”

  3. Release date is approaching:

    “Is the feature live yet?”

    “In the pre-release environment, regression testing is ongoing and comparative analysis will take time.”

The questions mentioned above: “what is tested”, “is the test finished”, and “is the test fast”, can be said to be the team often faced with the “three soul torture”. With the popularization of Agile and DevOps mode in the software industry, more frequent delivery has aggravated the industry’s concern about testing. The lack of efficiency of testing is often the primary cause of delivery delay, and testing has become the biggest bottleneck for enterprises to implement DevOps transformation.

In response to these challenges, the concept of “continuous testing” (or “agile testing”) was introduced and slowly became an industry imperative. Continuous testing meets the following core requirements:

  1. What was tested: Subsets of tests based on business value without 100% regression test coverage.
  2. Finished testing: Test as needed and provide quick feedback during continuous delivery.
  3. Test fast: Make the test run fast enough.

What is continuous testing?

From Wikipedia definition: the process of performing automated testing in a software delivery pipeline in order to obtain immediate feedback on the business risks of pre-release software. Continuous delivery of mobius rings is shown below:

Admittedly, the above definition highlights the importance of automated testing as a basis for continuous testing. However, to return to the ultimate goal of “improving efficiency through continuous testing”, the single point of efficiency of improving the way test execution is not enough to reflect the nature of the change in testing philosophy brought about by continuous testing. From the perspective of overall test efficiency, “in-sprint testing,” or “Agile testing,” as described by another DevOps twin, the “Agile” model, is a better complement: continuous testing should be a fundamental and ongoing activity throughout the software delivery cycle. Images from the Jenkins community better illustrate this concept:

How do you implement continuous testing?

Continuous testing changes the way traditional testing works after testing, extending testing activities to every phase of the software development life cycle.

1. Requirements planning phase, plan tests early and strategically define test subsets.

First, start from the requirements analysis phase by planning tests in advance and writing test cases to achieve appropriate requirements coverage. This needs to be helpful practices including ATDD, BDD, especially for teams that have difficulty landing TDD. Second, have the awareness to optimize the test coverage. Testing should not blindly pursue 100% coverage, but rather risk-based Testing based on business Risk and value Testing strategies. “100% coverage of high priority requirements” is far more valuable than “80% coverage of all requirements”.

2. During the iteration, push the test to Shift left to realize the parallel work of test and development.

Test execution should come early in the software development lifecycle, and a variety of engineering practices can help teams move left: emphasis on test reviews, foundational assurance through unit testing, development and automated testing based on interface definitions, and the introduction of code scanning to determine compliance with coding specifications and engineering standards. In this way, integration test cases can continue to be written around the requirements during the iteration cycle, and work with the development to maintain progress, provide necessary testing support for the development, and make the testing and development work synchronized.

3. During the iteration, provide a complete test environment and correct test data in a convenient way.

Building a test environment close to production and quickly preparing desensitized data have always been two of the most important challenges facing the team. Now the maturity of cloud native technologies, especially Docker technology, makes it possible to build and destroy environments on demand. However, the management of test data is still a difficult problem. Basic data, such as account information and environmental information, which are easy to be standardized, have been better solutions in the industry, which has been a major progress. However, business data is still in the basic stage of process specification based on framework due to the lack of good business abstraction due to the variability of scenarios. Implementing Mock services based on interface definition can also improve process efficiency.

4. After the application is deployed, focus on shift-right testing.

In the traditional waterfall model, deployment is the next phase of testing, which means the end of testing after the application goes live and features are quickly verified. Continuous testing does not consider that the release of the test is finished, but emphasizes that after the release of the version online, continue to pay attention to the data monitoring and early warning of the production environment, timely find problems and follow-up solutions, to minimize the impact range. And use the data on the production can bring tangible value for the development process: such as replication to desensitization to prepare the test data, production data to the service to access data in the process of the analysis of the results can also be used for the development of test and provide optimization, so as to adjust the direction of forming better smoke and regression testing strategy and so on. Right shift practices include data analysis, gray/canary publishing, online real-time monitoring, tracking and processing of user feedback, and so on.

In addition, we practice continuous testing by focusing on the precipitation of data, and then constantly optimizing our behavior based on the metrics to achieve the continuous improvement team culture that DevOps espouses. With the spread of agile and DevOps culture in the software industry, r&d teams must expect to achieve shorter iterations, improve software development quality more effectively, and deliver business value more quickly.

CODING adheres to the concept of providing one-stop DevOps solutions for enterprise R&D teams. Through powerful test management functions, CODING helps R&D teams to take testing as a basic activity throughout the entire process of software delivery, greatly shortening the software delivery cycle, enabling simultaneous iteration of testing and R&D to achieve continuous testing. Help the team return its focus to quality delivery.