Bytedance Terminal Technology — Wu Sicheng

Background Automated precision testing refers to automatic and accurate testing of the modified part of the code in each MR, so as to improve the quality assurance of the code and reduce the cost of testing.

1.1 Existing Processes

The general development process is as follows:

To ensure that these changes do not introduce crashes and affect the online user experience, these changes need to be tested.

The test is generally divided into the development of classmate white box self-test and test classmate black box test.

The current testing process is as follows:

In addition to developing self-test, it is also necessary to test students. The conventional test means is to record test cases for each Activity dimension of the application. When submitting Mr Each time, the test students can run the test cases.

For each commit Mr, the changes we make to the code are abstracted as follows:

  1. New methods were added
  2. Change the original method
  3. Delete methods

In the first case, new logic may be added, or new functions may be added, so the existing test cases may not cover the new functions, and the test students need to record additional test cases.

In the latter two cases, the existing test cases can override the code change logic, whether it is a logical change to a method or a method deletion, so the test student only needs to execute the existing test case.

1.2 Automated test process

Manual execution of existing test cases is a repetitive and mechanical task, so we automate the process:

As shown in the figure above, we added automated tests to the CI process and relied on the “Build Package” task to get the APK package. We also use the company’s internal cloud presence platform, which means we can execute our automated test scripts directly through the HTTP request interface to the test machine to execute the test cases.

Cloud real machine platform interface

Soon after the actual process ran through, we encountered some new problems:

  • Teams like Tiktok and Toutiao have hundreds and thousands of Mr Every day, and if each Mr Runs a full test case, each test will be extremely time-consuming and expensive in cloud real machine resources
  • In addition, most Mr Code changes are small and may involve only a few function changes at a time, so using full test cases is obviously not reasonable

Therefore, we need to find suitable test cases for each Mr And accurately recommend them to the automated test process.

Two, accurate test scheme

Problem description

We want to automate testing by recommending test cases related to the M R code changes during each Mr.

Then we need to face the following problems:

  1. How do you associate test cases with code
  2. How to obtain the content of each Mr Change
  3. How to recommend test cases accurately

2.1 How do test cases correlate code

A test case is essentially a recording of a series of user actions, such as clicks and inputs, during a black box test, so how can we match these user actions to the actual code?

The connection point is actually on the Activity. As mentioned above, when recording the test case, the test student recorded with the Activity as the dimension, so if we can know which Activity the current code is associated with, we can only use the Activity test case to test the code. Can effectively reduce the number of test cases, improve test efficiency and accuracy.

So how do we know which Activity a piece of code is associated with?

This can be done by generating a chain of method calls.

2.1.1 What is a method call chain

It is to form a graph by connecting the call edges of all functions in a code, which is the method call chain graph:

2.1.2 Android Call Chain

If we can find the functions that are directly associated with the Activity and combine them with the method call chain, we can find the functions that are indirectly associated with the Activity.

As shown, function1 is directly associated with ActivityA, so all the other functions on function1’s call chain are indirectly associated with ActivityA.

We call this an Android call chain diagram with edges from activities to functions, and we’ll focus on how to generate an Android call chain diagram in the following sections.

The Android call chain should have the ability to:

  • Query all functions involved in the Activity from the Activity (no hierarchy)
  • Query all activities involving the function from the function
  • Query function call relationships, one hop, two hops, etc
  • Query the start function of an Activity
  • Query the next Activity of a Acitivity

2.2 Obtaining Mr Changes

One might wonder: isn’t it enough to get the diff before and after Mr Commit to get the change?

But it’s not that simple, because the diff we get is just a code segment that adds, deletes, and modifies, and there’s no way to associate an Activity with the Android call chain by a code segment alone. The node of the Android call chain is a method, so the ACTUAL Mr Change we need should be the method changed in Mr This time, the method change refers to here includes:

  1. Methods the new
  2. Methods changes
  3. Method of reducing

So how do we know which methods have changed in an Mr?

Here we use the static analysis technology, first of all to obtain the source files of all the changes in this Mr, as well as the corresponding source files before the change. Then intellij’s SDK converts the source file to PSI, and finally compares the methods that PSI can obtain changes.

PSI: Program structure interface, a semantic abstraction layer within the IntelliJ Platform, is responsible for parsing files and creating syntactical and semantic code models that support many of the Platform’s features. It can be thought of simply as an abstract syntax tree, but it does a finer level of parsing based on Java and Kotlin’s language features, identifying the semantics of classes, methods, parameters, and judgements in code.

Therefore, based on PSI, it is much easier for us to compare whether the methods in the two files have changed. The comparison rules are as follows:

  1. To add a method, compare the method names in the new file with those in the old file. If a method exists only in the new file but does not exist in the old file, it is a new method
  2. Delete method: compares method names in the new file with those in the old file. If a method exists only in the old file but does not exist in the new file, it indicates that the method is deleted
  3. Change the method. If the method exists in both the old and new files, then calculate the body size of the method in the new and old files respectively. If the size is inconsistent, it means that the method has been changed

2.3 Accurately recommend test cases

For the recommendation of test cases, it not only filters out the relevant Activity cases, but also combines the correlation between Activity and the change, whether the Activity is an online hotspot Activity, and the posterior probability of the Activity discovery associated with Crash, etc. To set the number of test steps for the Activity. In addition, the test models of target Activiy are assigned based on multi-dimensional data such as test coverage, crash rate and model distribution of online users. The specific recommendation algorithm process is not available to the public at present, please look forward to sharing in the future.

Iii. Android call chain construction process

3.1 Stage 1: Generate the global function call chain graph

A brief introduction to the background, call chains are implemented based on static analysis techniques, which can be simply divided into source analysis and product analysis. For example, Android provides Lint detection based on source analysis, while here call chains are generated based on APK analysis, also known as product analysis.

At present, there are mainly WALA and Soot static analysis frameworks for Java open source. Compared with WALA, Soot has more documents and a more active community, so we finally conducted customized development based on Soot.

The Android precision call chain generation tool — ByteRope developed by us is based on Soot customization development, Soot provides us with the generation ability of CallGraph, However, simple CallGraph cannot meet our demand for accurate association of activities, and further optimization and transformation are needed.

A brief introduction to the algorithm flow of call chain generation:

Call chain generation process:

  1. Parse apK to get all classes in APK
  2. Parse each class to get all methods in the class
  3. Parse all methods to get their body, which is made up of command statements such as copy, method call, and so on
  4. Parse the body of the method and, whenever a function call occurs, build an edge between the method and the method being called
  5. When we have traversed all method’s bodies in all classes, the call chain diagram is completed

The information we can get from building the call chain diagram:

  1. All classes in APK
  2. Methods in each class
  3. The body of each method
  4. Other methods called by each method (call edges)

3.2 Phase 2: Building the activity-method invocation link

3.1.1 Getting all activities in APK

As mentioned earlier, the purpose of the call chain is to find the activities associated with the method to recommend automated test cases, so we need to find all the activities as entry classes to the call chain.

Get the Activity method:

Mentioned above, in the process of generating invocation chain, we have got the apk in all kinds of information, only need to traverse all the classes, whether the class inheritance in android. The app. The Activity, androidx. Appcompat. App. AppCompatActivity, If inherited, the class is an Activity.

3.2.2 Generate a call chain with Activity as the entry point

In stage 1, the global call chain was generated. The purpose of generating the call chain with the Activity as the entry point is to ensure that the functions in the chain are linked from the Activity, thus ensuring that each method in the chain is associated with the Activity.

A call chain generated with the Activity as an entry point

4. Optimization of call chain: Associating Android native components

As mentioned previously, Soot provides us with the ability to generate CallGraph, but the simple CallGraph cannot meet the demand of accurate association Activity, so further optimization and transformation are needed.

4.1 background

Many components and controls in Android are invoked through layout files or asynchronous mechanisms, so it is difficult to associate these components and controls with the Activity they belong to, even if a global invocation chain is generated.

4.4.1 research

Components where this can happen are fragments, custom controls.

Fragment

Fragment is generally divided into static loading and dynamic loading.

  • Static loading is done in the activity’s layout file
  • Dynamic loading is typically loaded with fragmentTransaction.add (fragment).commit().

Custom control

Generally inherited from View or ViewGroup, loading mode is also divided into static loading and dynamic loading

  • Static loading uses the fully qualified name of the control as a Tag directly in a Layout file
  • Dynamic loading

The custom controls are typically loaded into the target ViewGroup via viewgroup.addView ().

4.2 plan

4.2.1 About explicitly Calling Fragment and Customizing View

Problems faced:

The call chain is not associated with system functions, so the Fragment or Override method in a custom View is not associated with the Fragment.

Modeling:

Associate an existing call chain with the Fragment’s system Override method.

However, since system functions cannot be directly associated with other methods, we manually added an edge to associate caller methods with override methods.

The Caller method is associated with the Override method

4.2.2 About Android Components and Fragments in the Static Call Layout File

Problems faced:

Because the Android components loaded through the layout file follow the internal logic of the Android system and are called asynchronously, the currently generated call chain does not have a path from the Activity to these Android components. In other words, These components are unable to find the Activity they are associated with, resulting in precision testing failing to recommend test cases.

modeling

Purpose: To build a path from an Activity to a component that is statically invoked. Find the connection point of static call, need to find all the connection points of the layout file to call Android components in the whole process, so as to connect the call chain path:

  1. The Activity sets the layout file with setContentView() :

    You can get the Activity’s layout file ID

  1. Find the mapping between layout file IDS and layout files:

    First, the layout files are stored in APK. When we unpack APK, we can find that the layout files are stored in the RES directory.

    Second, there is a binary file named resources.arcs in APK, which is the collection of all resource information in APK. The mapping between layout file ID and layout file is stored in resources.arcs.

  2. Parsing the layout file:

    First, understand how Activiy uses fragments and custom components in layout files:

    A. Invoke the Fragment in the layout file

    Use the tag and introduce the Fragment class name in the Android :name attribute:

    B. Call the custom View in the layout file directly call the custom control class as the tag of layout.xml

    C. Invoke other layout files in the layout file

    I. Introduce other layouts through labels

    Ii. Introduce other layouts using the Android: Layout property

Second, parse the layout file. Set the file parsing rule according to the above way that the Activity statically configures components through the layout file, so as to obtain the link between the Activity and the Fragment and the custom View:

At this point, the optimization of the Android call chain associated with Android native components is complete.

Five, the revenue

From May to June, the automated precise testing of the MR process connected to Douyin has achieved preliminary results:

  1. Tiktok Android: Toolline + basic business + social testing save 35%
  2. Compared with ordinary automated tasks (Activity coverage rate is about 0.5%), Activity coverage rate with accurate automation is increased by about 15 times on average, and about 3 crashes can be found on average

Sixth, summary and prospect

This paper first introduces the evolution of automated precision testing, and what problems we encountered in the process of automated precision testing, and their solutions; Secondly, this paper focuses on the role, nature and construction of The Android call chain in the process of automated precision testing, and introduces the optimization item of the Android call chain, that is, to customize the associated Activity based on Android features. The accuracy of Mr Change method associated with Activity is improved, so as to improve the recommendation accuracy of test cases, reduce unnecessary tests, and improve the test efficiency.

However, the application scenarios of Android call chain are far more than this. It can also be applied to sensitive method link tracing, API call combing and other scenarios. However, the requirements for the accuracy of the call chain are increased accordingly.

  • Improve the call chain algorithm. At present, the construction of the call chain is only limited to synchronous call, but there are many asynchronous call logic in Android, so we can cover these asynchronous call scenarios through modeling, so as to improve the accuracy of the call chain.
  • Generate more granular invocation chain, at present the method invocation chain is based on granularity build, but exist in the method of judging conditions lead to logic bifurcate, so if you can the method based on code semantic split into BasicBlock granularity, and make the invocation chain build, can make each invocation chain represents the logic of the information more accurate, Thus, it can also improve the accuracy of recommended test cases in the field of intelligent testing.

About bytedance terminal technology team

Bytedance Client Infrastructure is a global r&d team of big front-end Infrastructure technology (with r&d teams in Beijing, Shanghai, Hangzhou, Shenzhen, Guangzhou, Singapore and Mountain View), responsible for the construction of the whole big front-end Infrastructure of Bytedance. Improve the performance, stability and engineering efficiency of the company’s entire product line; The supported products include but are not limited to Douyin, Toutiao, Watermelon Video, Feishu, Kechedi, etc. We have in-depth research on mobile terminal, Web, Desktop and other terminals.

Now! Client/front-end/server/side intelligent algorithm/test development for global recruitment! Let’s change the world with technology. If you are interested, please contact [email protected], email subject resume – Name – Job intention – Desired city – Phone number.

The automated and accurate test mentioned in this paper will be launched in MARS, a bytedance application development kit. MARS is the result of the bytedance terminal technology team’s research and development practice in apps such as Douyin, Toutiao, Watermelon Video, Feishu and Tongchedi over the past nine years. For mobile R&D, front-end development, QA, operation and maintenance, product manager, project manager and operation roles, we provide one-stop overall R&D solutions to help enterprises upgrade their R&D mode and reduce their overall r&d costs. Click the link to enter the official website for more product information.