How to measure the validity of the test? What are the pits and key points in the construction of the evaluation system? This article shares the experience of capacity building of youku client test effectiveness evaluation and how to assist testers to adjust direction and strategy. (This series of articles will be released one after another, interested friends continue to pay attention to!)

background

In the wave of mobile Internet, the iteration speed of TOC mobile applications is very fast, and the application gameplay is emerging in endlessly. Therefore, the corresponding guarantee of application quality is also put forward higher requirements: rapid response, clear goal, effective means, etc. Quality assurance team in order to better achieve the above requirements, the need to develop a series of processes, systems or tools to assist in the channeling of the test, but no matter what is the test side to set up, there is a clear principle throughout, namely the validity (such as processes, systems or tools are used to test, also known as test validity). So how to measure the effectiveness of the test becomes particularly important, and the construction of its evaluation system is the most important. Based on the principle of “from practice to practice”, the effectiveness evaluation ability of Youku client testing is built to assist testers to adjust the testing direction and change the testing strategy.

Scheme selection

Different teams define their own rules to measure test effectiveness, which can be anything from Bug counts and trends, test case reviews, test strategy reviews, or code coverage data. Evaluation of the effectiveness of testing cannot be separated from the implementation of development and the context of the project, testing is never independent, testing and development are inseparable. Considering the current situation of Youku’s quality assurance system, we give priority to the most direct and most able to evaluate the effectiveness of tests from the perspective of data: code test coverage.

Code Test Coverage Tool (Android)

  • JaCoCo – Access to coverage via code injection
  • Youku internal coverage collection tool – youku self-developed class coverage statistics tool

[Tool comparison]

names Code into Performance loss Can be used for grayscale publishing The statistical range
JaCoCo You need to hack and package for each module Performance degradation is high and non-functional bugs can be introduced Cannot be used for grayscale publishing Module, class, method, and row levels
Youku internal coverage collection tool No code intrusion and no single module repackaging required There is almost no performance loss and no additional bugs are introduced Can be used for grayscale publishing, users have no perception Module and class levels

Through the collection of offline test coverage data of two versions, the coverage data of Youku client test in the latitude of class is not satisfactory at present, which is only about 40%. Combined with the tool comparison conclusion and client testing status, Youku internal coverage collection tool is implemented as the current preferred solution. Analyze test validity from macro to micro.

Overall structure and process

Referring to the figure above, the process for obtaining whether a class is loaded is summarized as follows:

  • Get all classLoaders’ ClassTable objects through the Native layer’s ClassTable
  • Get the list of class names from the ClassTable object
  • The class is loaded using the Lookup method of the ClassTable in libart.so

The design of the overall scheme is based on the following figure:

The sample results

Youku client test platform

  • Review the results

  • TOP Module class coverage comparison result

  • Module details compare data

The technical implementation

  1. The data download and analysis part was developed using Python3
  2. 300,000 ZIP * (100,000-120,000) classes, analysis and statistics in about 1 hour to complete.

Analysis machine for 8CPU, 16GRAM Ubuntu20.04LTS* 3. The main package of four parts

  • Data de-obfuscation logic
  • Class coverage statistical algorithm correlation
  • Download metadata, analyze and count the number of module and class calls based on PySpark

– Download logic based on Multiprocessing and Asyncio implementation – Interprocess (download & analysis) communication based on queue implementation

  • The task is triggered based on the task creation and timing mechanism of the test platform

Results the yield

  • The test effectiveness evaluation method of Android host and guest module/class latitude is established
  • Due to the use of online code heat analysis mechanism, based on its nondestructive performance, it can be directly used in daily automatic testing and gray testing
  • Effectively collected the test validity data of multiple versions of Youku host and guest
  • Effectively optimize test coverage for business modules

The follow-up plan

  • Promote the test effectiveness analysis ability to be implemented in the host and guest TOP module of Youku, and improve the coverage of single module not less than 10%.
  • Full-link automation and special tasks of automated testing based on Youku client test platform can carry out test coverage statistics.
  • Combined with youku host and customer technology map, the coverage rate is visualized to display the test link coverage more intuitively.
  • And more…