background

\

The automatic testing of AndroidAPP currently includes a lot of testing projects, and each project has a lot of innovation, including functional automation testing, stability automation testing and performance automation testing, etc. In the daily testing process, the above testing technologies or services will find some problems.

Traditional stability automated testing (Monkey testing) mainly simulates user behavior through arbitrary clicking, sliding, etc., but it has problems of low coverage and low execution efficiency. Automated performance testing is often limited to fixed test scenarios and fails to catch memory leaks outside of fixed scenarios. In addition, the compatibility test still relies on manual testing to cover multi-system versions and multi-models, and the functional availability of APP on different devices should be ensured as much as possible. However, such manual testing can hardly cover the mainstream Android models in the market.

For rapidly iterating new business teams, the maintenance cost of functional automated case scripts is high, and the investment of human resources is not much, although through the intervention of configuration or AI technology, automated test technology is still not ideal. Therefore, it is desirable to reduce script writing costs and script complexity, reduce script maintenance content, and allow as much traversal as possible to be performed by the service, while also finding more problems.

In order to achieve such a goal, iQiyi test team proposed a lightweight, simple and universal one-click APP physical examination scheme. Users only need to submit an APP address to give the APP a multi-dimensional health examination. Including [APP security check], [UI anomaly detection], [performance data analysis], [Crash and ANR detection] and other core content.

The solution

The solution mainly uses the open source tool ATX as the traversal engine, integrates the Recording and playback tool of the Web end to distribute the customized scripts to the real machine group in a wireless way, and performs the traversal of related controls on the basis of directional traversal. The task is completed in the period from the start of APP installation to the end of final traversal and automatic uninstallation at last.

The task is divided into four stages (installation, initialization, traversal, and uninstallation). The whole process includes log collection, picture collection, video recording, ANR and Crash listening collection and filtering, server log analysis, and data aggregation. For the process, the optimization of the success rate of APP automatic installation, the measurement scheme of startup time, the automatic reporting of Bug problems, UI anomaly verification and security scanning are added.

 

The overall framework

The overall function of one-click physical examination is composed of four modules: mobile cloud back-end service, scene service, Worker service and analysis and monitoring service. After the task is started, firstly lease some devices or specified devices from the mobile real device resource pool and remotely mount them to Worker. Automatically if the Worker is not enough use has extended to mount again, to conduct a task when the corresponding Worker received task, and assigned to the devices, distributed task according to the directional script number and equipment number matching, safety inspection, after the initialization opened APP general installation, implement traversal, data capture, image capture and video recording and other services, Related data files are transferred to the log service platform in structured format. \

One key physical examination overall frame

Driven by the kernel

Traditional functional automation tests typically use THE Uiautomator2 framework to write Android scripts, which are packaged as an APP and installed on the phone to implement the APP UI driver. The framework can traverse the control and also directionally run the specified function test. However, in the case of common unified device resource pool, multiple teams need to install the APP of integration test script on the mobile phone after renting the device, so the APP to be installed becomes more and more.

In addition, packaged and installed test APP to run tasks, the overall test development efficiency is very low. In order to overcome such problems, the team adopted the method of transmitting the executed action script to the corresponding mobile device through HTTP to achieve wireless traversal. The advantage is that directional traversal can be flexibly realized, and there is no need to pack and install the driver engine or driver test APP on the mobile end. The internal framework used by the team is mainly based on the secondary development of open source project ATX, including the stability upgrade of ATX-Agent on the device side, support for adding ATX screen recording, and internal log optimization processing, etc.

As shown in the figure below, py code can be converted into HTTP and transmitted to the mobile phone to form corresponding instructions to perform the relevant driver purpose.

* note:

[ATX-Agent] is an ARM program written by Go, which needs to be opened in ADB shell mode. It collects HTTP request instructions from clients and distributes them to different component modules for execution.

Android-uiautomator-server is a github open source project that integrates the ATX execution engine to automate operations such as clicking, typing, swiping, etc.

Minicap and Minitouch are two services for device real-time screen images and touch screen events, respectively, from the open source framework Open-STF. These components will be automatically pushed to the device if the ATX app is installed.

There is a UIAutomator2 in the Python framework library, which can be used to wirelessly request automated operations to devices using automated scripts as shown above.

Device Resource Pool

At present, the entire Android device resource pool has been successfully built and run effectively through the open-STF framework, serving the testing, development and debugging of the company’s multiple business lines. At present, nearly 100 devices are centrally managed in the laboratory, and multiple devices are managed by PC through Provider. Due to the technical development needs of regional device management and upper application, considering the special scene of one-click APP physical examination itself, iQiyi test team deusb the connection between the APP and the underlying device.

Here, the purpose of deusbization is to indicate that ATX first needs to obtain adBshell permission. If USB connection is removed, then adBConnect mode is required to wirelessly mount the device for ATX initialization. When ATX-Agent is started by ADB, the driver script can be directed to specify through HTTP. Dynamically delivered to the device, other logs such as performance logs can be retrieved using ADB shell directives. The Mobile Lab provides this basic leasing interface capability, assigning both the port and the IP of the provider machine.

In the figure below, of course, there are specific interfaces that support non-UI forms, which are highlighted here for intuitive purposes only. In this way, the device can be decoupled for upper-layer technologies to implement special application scenarios and remotely mount devices.

Universal APP Installation

The first step of one-click physical examination for Android applications is APP installation. However, the installation process of mobile phones on the market is not consistent at present, especially for vivo, OPPO and other brands, they must input the password and click the button successively to complete the operation.

As shown above, it is difficult to click the install button using traditional controls to find elements. Because the Install and cancel buttons are actually an image.

Through careful observation, it can be found that the key words involved in the installation process of all models are limited, mainly including: installation, determination, next step, confirmation, etc. In fact, as long as you can find these keywords and click the corresponding position can be achieved click, this time using OCR to find the key word location is the most appropriate. Here iQiyi test team proposed a universal APP installation solution based on OCR technology. Take screenshots during APP installation and upload the screenshots to the backend interface. The backend invokes the OCR service of IQiyi to obtain the text distribution and coordinate area in the screenshots. After comparing the screenshots with the general installation keyword database, the backend matches the successful keywords (LLDB). ) and its center point coordinates are returned to the mobile terminal, and the mobile device can click the coordinates.

One could argue that many companies do this, and some even use deep learning to do it. So why aren’t we using deep learning enablement? Here’s why:

  • 1) when multiple keywords at the same time appear in the screenshot, first click the on screen at the bottom of the text is considering the popup window can click the boot button appears at the bottom of the partial basic and on the right (of course, there are on the left of this kind of models, a filter can), to such a strategy selection uniqueness text coordinates can click step by step to realize installation steps.
  • 2) Password input timing of OPPP, VIVO (hereinafter referred to as OV) mobile phone system: For security reasons, when the password input box appears, the screenshot function is prohibited. Therefore, OCR text recognition will be ineffective, even ordinary control recognition is useless. Therefore, the team came up with such a solution: judging by the model, when the installation task is triggered, the unified user name and password of OV mobile phone can be authenticated by TAB and Enter keys (note: authorization is available only on THE OV test machine). The screenshots of subsequent operations are not affected, and the installation process is the same as 1).
  • 3) Screenshots are taken in Minicap mode to avoid the timeout requirement of countdown during installation of some brands such as Xiaomi mobile phones. The screenshots of ADB Shell are time-consuming and will exceed the timeout requirement of operation. Minicap screenshots also have their own problems, time interval between the need to control properly, such as into the APP business within the authorized box, bounced and overlapping play box appears also to be removed by a specific strategy, otherwise click when the failure, at present is by performing a click OCR copywriting matching approach to determine whether repeated again, In the later stage, the image similarity will be compared and removed.

At present, the success rate of APP installation through the above methods is higher than 95%, which basically meets the requirements of automatic APP installation of mainstream models. Some scenarios such as notification push interference, lock screen interference, OV user authentication network patency requirements and other abnormal situations occur, the solution in this area is still looking for. The following screenshots will also present a snapshot of the installation process to the user in the one-click health check, so that the user can temporarily intervene and adjust in case of the above abnormalities. At present, automatic APP installation service has been provided to some other technical special use.

The picture shows the automatic installation process of iQiyi gingerbread short video APP

APP Startup Time

Startup time data collection: The installation process of THE APP will be recorded, and the startup of the APP will also be recorded by using the screen recording function. Then, the waveform diagram drawn by the video parsing service platform for the RGB change rate of pictures will be found to match the calculation time of the eigenvalue provided by each line of business.

St (1,0,20) in the figure above represents the starting calculation point for APP opening, starting from the starting point on the left of the first waveform. 20 represents the threshold set by the business line. For example, data exceeding the threshold represents a waveform. Ed: the end point of opening the first screen of APP, and the eigenvalue has the same meaning as ST.

With such a back-end video parsing service platform, the startup time of an APP can be measured for the video provided by one-click physical examination. In addition, the recording function of the video needs to be explained. The team uses the traditional program AdbShell ScreenRecord, but Huawei mobile phones do not support this ADB command. Integrated MediaRecorder in ATX APP, setting encapsulates the video recording of APP UI start and stop operation.

Since similar waveforms above need to be output in a standard way, especially at ST point, the first starting point of the first waveform is the timing point of APP startup, it is necessary to use the tool FFMPEG to adapt and cut a section of white screen content for recorded video files. Because MediaRecorder will be called to start a blank Activity integrated in ATX, need to remove this part of the tool generated interference factors. This time consuming statistical testing is accurate, and the input automation method is more stable and universal.

UI exception check

The previous push of this official account

APP security verification

APP security verification is provided by iQiyi security cloud team with technical support. Currently, it is mainly completed by two types of analysis engines, namely, the vulnerability rule matching engine based on SMALI code and the static stain analysis engine based on FlowDroid. The former can quickly locate obvious vulnerabilities such as misuse of API parameters. The latter can detect deeper data breaches and permission leakage vulnerabilities through taint analysis techniques.

The static stain analysis engine is introduced here. On the basis of FlowDroid, a variety of general vulnerability rules are added to fix a large number of data flow processing bugs, expand the tracking range of data flow, and increase the accuracy of analysis. For the report output, detailed codes related to the location of vulnerabilities or data leakage paths are provided to facilitate users to troubleshoot related problems.

As shown in the figure above, there are basic security risks and Manifest permission analysis. For basic security risks, if you want to troubleshoot problems, you can click view to locate the specific APP code file path, which is very convenient.

Combined event-driven

Traversal operation events in one-click physical examination are mainly divided into basic random events and operation scenario events. These two types of events complement each other to ensure that traversal operation can cover normal operation behaviors of users. Click and slide events in basic random operation mainly include click, long press, double click, omnidirectional slide, etc., while the operation scene events add compost, click and wait, directional slide, identify slide and other operations to achieve different operations in different scenes. In addition to the ordinary text input function, the operation scenario event covers the text box recognition, keyboard call out and folding functions.

Mobile phone basic operations include equipment awaken, click the home button, click on the return key, etc., physics here in order to fit iQIYI player compatibility tests, joined the XuanBing, half a screen full screen switch, etc, sometimes in order to prevent traversal collapse or through out of the scope of business lines, and corresponding joined to restart the APP or return to the active intervention operation, It also supports APP switching between front and back.

Pictured above, specific traversal strategy for resolving the current control tree XML files, by the coordinates of the control tree to determine the current click control set, and then according to the steps to obtain the control action attribute characteristics, to determine the corresponding coordinates of the click, long press, text input and return possible operation, considering the need to control some traverse range, When the task is triggered, the solution will let the business line configure the Activity whitelist list of the business, so that the traversal will be monitored and returned to rectify in time.

In addition, in the traversal test process, it is inevitable to encounter some permission pop-ups, confirmation pop-ups, abnormal pop-ups, etc. In the traversal operation process, the physical examination scheme will monitor the pop-ups and perform corresponding click processing to avoid blocking the traversal operation. Based on the control tree itself has its own defects, such as RN or small program UI recognition, control tree loss, this part has already been upgraded and optimized.

 

Scene customization

One-click health check supports script execution of customized scenarios, such as configurable login support, specific vertical line services, and small program entry. Currently, each service line maintains the task script of the customized scenario. If you need to specify the operation of the pre-scenario before performing the one-click health check task, the customized script is automatically executed after the universal installation module runs, and the policy traversal is performed after the specified entry is reached.

In addition, for some lightweight business traversal requirements, iQiyi test team will improve script development efficiency to the level of recording and playback, reducing the technical input of business line personnel, as long as relevant recording and playback of the APP can realize the automatic driving of functions.

As shown in the figure above, after renting a device through the Web page, you can enter such webIDE to realize recording and playback for the mobile APP, a bit like Airtest tool. However, this is the IDE on the Web side. The iQiyi test team developed such tools mainly considering that users need to configure a large number of environment Settings or tool compatibility problems on the client side, so as to avoid high maintenance costs or slow development efficiency. Currently supports Python, scripts can be submitted to the cloud, and then integrated into the one-click checkup can realize the batch playback of multiple devices. You can select the pre-task name script in the following figure.

 

BUG submission report

After the traversal task is started, the physical examination solution will continuously monitor logCAT logs of Android devices and filter out key error information logs such as Crash and ANR related to APP. When keywords related to Crash and ANR are captured, the screenshots of a period of time before the occurrence of the problem will be taken out, and the behavior trajectory graph will be drawn on the screenshots, which will be regarded as part of the key information for Bug reporting. In addition to the behavior track chart, detailed device log information, download address of the current test package, mapping file corresponding to the current test package, and full logcat file are attached to the Bug submission list, and the title of the Bug contains key information about crash or ANR.

For the ANR error, if you have permission to pull the local files from the device, the local files will be added as attachments to the Bug list. However, many devices do not support pulling the local files directly, so we will get the BugReport file provided by Android and attach it to the Bug list. To provide the most effective information for the development of students.

Corresponding behavior trajectory in Bug information

In addition, it should be specially pointed out that in the process of raising bugs, repeated bugs may be found, such as the same crash for many times. Considering the problem of repeatedly raising bugs, a back-end filter is carried out during the process of raising bugs. For the bug with the same error reason, the back-end determines that it is a repeated bug and does not perform the subsequent bug reporting process after storing relevant error information in the library. At the same time, the previous bug list will be informed again in the appendix after the user task, indicating that the same problem occurred in traversal, please fix it in time.

The specific judgment rules are as follows: characters in logCAT need to be eliminated uniformly [marked in the red box in the log as shown in the following figure], and then the task is compared with the inventory Bug content in the dimension of unified version. If a match is found, it is considered that there is a duplicate Bug, and the same Bug will not be proposed again.

 

Log Analysis Service

The one-click health check task runs about 15 devices by default, and its duration is 30 minutes by default. It runs multiple service lines in parallel. Therefore, the amount of data to be processed is huge, including the processing of performance data, image data and log data, as well as the association between problem location data. Back-end log analysis is as follows:

Here are the parts:

  • Data source: Data of log service access, including but not limited to log, picture data, performance data, etc. At present, data output of one-click APP physical examination mainly includes the service forms of these three contents.
  • Data access: Uploads data to the log service according to the specified data format and communication protocol.
  • Data collection and storage: Collect, parse, and store data, in this case some atomic, structured data
  • Data calculation: Processes, calculates, and stores atomic data based on business scenarios
  • Data service: Provides an open interface to output data
  • Data application: develop front-end based on data for users to use

The actual rendering is shown below, which is a complete report, including some of the above. Of course, due to space requirements, part of the situation is not introduced. For example, the performance analysis data is collected by the back-end test team performance analysis platform for scientific denoising statistics, and integrated into the one-click physical examination to provide more detailed reference content.

conclusion

One-click physical examination has been applied to many new products and new business lines of iQiyi’s internal apps, which can help them find offline problems as soon as possible and comprehensively, reduce manpower input and reduce the complexity of docking. Multiple devices can be triggered with a few simple configurations and can be connected to continuous integration. In the future, in addition to optimizing coverage, other special indicators of physical examination will be enriched, and the diversification of scenes traversed will also be upgraded.

end

Maybe you’d like to see more

Scan the qr code below, more exciting content to accompany you!

Iqiyi technical product team

Think simply, act simply

\