background

After Apple restricted TestFlight, the total number of grayscale users of a single App through TestFlight channel was limited, and the ability of grayscale to expose stability problems was greatly limited. How to better expose more problems offline became the most intractable problem at present.

Fastbot is an intelligent testing service jointly developed by Quality Lab and GIP-ios platform architecture team. It makes use of machine learning and reinforcement learning for traversal testing, and provides basic stability testing service capabilities. In terms of code coverage and stability testing, Both have better results than traditional Monkey, more than doubling the number of problem finds and code coverage, and are comparable to manual regression in code coverage. [1]

Access to the history

The GIP-ios platform architecture team has been working on the Fastbot implementation since the second half of 2020. Based on the business testing pain points and testing problems encountered during the implementation, we mainly explored and practiced in the following three aspects:

  1. Fastbot general ability development, such as Fastbot operation visualization, Fastbot card screen detection;
  2. Fastbot capability expansion, which is mainly to do some business customization capabilities in the App under test, such as scene limitation, universal UI detection, keyboard scene optimization, etc.
  3. Fastbot data consumption, such as offline approval process construction, test machine allocation strategy exploration;

Based on the practices in the above three aspects, we have established a set of Fastbot based automated testing process, which forms a complete closed loop from problem discovery to consumption and continuously ensures the stability of the tested App. In addition, for each MR change, we access the precise directional testing process, which uses the code warehouse to build the code model, according to the CODE change of MR to obtain the affected associated VC, combined with Fastbot historical test experience set, directional test associated VC corresponding logic, hoping to truly achieve the “change where test where”.

Fastbot problem finding ability improved

One, scene limitation

In order to further improve Fastbot’s traversal ability and problem finding ability, so that it can cover more viewControllers, for VC with deep entry, due to the randomness of Fastbot test, it is difficult to traverse such VC at a time, and the test duration on such VC page cannot be guaranteed. In view of this kind of problem, we designed and implemented the scene-limited function, which can ensure its continuous test in a VC that supports scheme jump.

Its core logic is to build a WebServer service in the dynamic library for communication with Fastbot, and internal Hook UIViewController Present and Push related methods, monitoring VC into and out of the logic.

When Fastbot obtains the scheme to be directed through the configuration file, it will call the Checkin interface and pass the scheme to be transferred to the tested App. After the tested App obtains the qualified scheme, Complete the qualifying logic based on the current qualifying state (in the jump, jump success, exit orientation, etc.).

If the current Scheme can be defined, the App under test can complete the jump through the routing logic implemented by the business itself (or in the form of OpenUrl). When a restricted page is reached, Fastbot will trigger the isLimit request for each Action to obtain the current page stack in the App under test and determine whether the VC is in the restricted scene. If it has jumped out of the restricted scene, Fastbot will pull up the scheme to be restricted again.

Meanwhile, when App dismiss or pop out the current VC, the App under test will also trigger the detection logic. If there is no qualified VC in the current memory VC stack, the App under test will request the Jump_out service of Fastbot. When Fastbot receives this request, it triggers the Checkin request again, bringing up the qualified scenario.

After the implementation of the scene restriction function, it simplifies the process of specific VC restriction test, and is now widely used in the test of various activity scenarios, with the effective abnormal interception rate rising from less than 5% to more than 20%, ensuring the smooth online of each activity.

Second, barrier-free detection

At present, there are more than 17 million visually impaired people in China, among whom smart phone users account for 92%. All of them have strong demands for using mobile apps. However, when visually impaired users use mobile applications, their satisfaction is generally low, only 18% of them are very satisfied (data source: [2]).

In order to better serve the visually impaired, we have done a lot of work in the early stage, but with the continuous iteration of requirements, each new function will be launched with the birth of a new barrier-free adaptation, and even the previously completed adaptation will be damaged. Therefore, a complete barrier-free link is needed to ensure that the barrier-free adaptation of App remains highly available. Fastbot is used for automatic testing. When the App is in barrier-free mode, the problems that do not meet the barrier-free requirements in the current UI are tested, and related problems are recorded and reported to the server.

Its core principle is that in the process of Fastbot traversal, the UIAccessibility attribute is used to detect the current interface by Using Google’s GTXLib[3] rule during each step of operation. GTXLib is a accessibility testing tool for Google iOS that integrates XCTest internally and can be used with any XCTest-based framework.

Currently, GTXLib can detect the following seven types of problems:

  • Accessibility label punctuation:
  • Touch target size: control interaction size;
  • The org.eclipse.swt.accessibility label – traits overlap;
  • Accessibility label: Checks that all accessible elements must have an Accessibility label;
  • Supports Dynamic Type: text adjustsFontForContentSizeCategory attributes to YES, automatically zoom;
  • Contrast ratio: Contrast;
  • Accessibility traits: Accessibility features that tell ancillary applications how a Accessibility element should behave or be treated.
OBJECTIVE-C UIAccessibilityTraits UIAccessibilityTraitNone; / / the barrier-free elements without feature UIAccessibilityTraits UIAccessibilityTraitButton; // The barrier element should be treated as a button UIAccessibilityTraits UIAccessibilityTraitLink; / / the barrier-free elements should be regarded as a link UIAccessibilityTraits UIAccessibilityTraitSearchField; / / the barrier-free elements should be regarded as a search box UIAccessibilityTraits UIAccessibilityTraitImage; / / the current barrier-free elements has been selected UIAccessibilityTraits UIAccessibilityTraitSelected; / / the current barrier-free elements has been selected UIAccessibilityTraits UIAccessibilityTraitPlaysSound; / / the accessibility element is activated broadcast voice UIAccessibilityTraits UIAccessibilityTraitKeyboardKey; / / the barrier-free behavior of elements like the keyboard key UIAccessibilityTraits UIAccessibilityTraitStaticText; / / the barrier-free elements should be regarded as not change static text UIAccessibilityTraits UIAccessibilityTraitSummaryElement; / / when the application starts, the barrier-free elements provide information UIAccessibilityTraits UIAccessibilityTraitNotEnabled; / / the barrier-free elements is not available and cannot respond to user interaction UIAccessibilityTraits UIAccessibilityTraitUpdatesFrequently; // The barrier element updates its tag or value frequently. UIAccessibilityTraits UIAccessibilityTraitStartsMediaSession; / / when the accessibility element is activated, launched a media dialogue UIAccessibilityTraits UIAccessibilityTraitAdjustable; / / the value of the accessibility element within a certain range allows the continuous UIAccessibilityTraits UIAccessibilityTraitAllowsDirectInteraction; / / user interaction directly touch the barrier-free elements allow VoiceOver UIAccessibilityTraits UIAccessibilityTraitCausesPageTurn; / / the barrier-free elements should be a cause of auto flip, when VoiceOver read complete when pages of text UIAccessibilityTraits UIAccessibilityTraitHeader; // The accessibility element is a title that divides the content into chapters, such as the title of the navigation barCopy the code

Every time the user touchBegin screen is detected, GTXLib will traverse all Windows, encapsulate all cases that do not conform to the accessibility rules into GTXResult structure, and report to the server.

At present, Toutiao has established the Fastbot barrier-free testing task, which aims to help QA and RD to quickly find and modify cases that do not meet the barrier-free requirements in the current App. The detection effect is shown in the figure:

3. Abnormal task recovery

In analyzing the Daily task, we found that there were two common problems that affected Fastbot’s traversal ability: the App under test was stuck for a long time and the keyboard scene did not exit. In order to solve these two kinds of abnormal problems, we design a scheme of screen jamming detection and keyboard aggregation frequency limiting.

3.1 Fastbot card screen detection

During Fastbot testing, I often encountered situations where the App under test entered certain scenarios and failed to exit. A 10-hour Fastbot task can be stuck for the next 7-8 hours, which is a huge waste of time for a ByTest machine. As shown below, it is stuck after entering the video page.

Before solving this kind of stuck screen problem, the general is to find the corresponding scene, then analyze the cause of stuck, targeted treatment, but this treatment cycle will be relatively long, can not timely stop loss. Here, we implemented the Fastbot screeck-detection logic inside Fastbot, which will trigger the Fastbot screeck-detection logic if the page is detected to be stuck for more than a period of time.

Screen detection is based on the SID. Fastbot performs an MD5-like calculation on each page tree to generate a unique identifier for the scene, which is the SID. The easiest way to determine whether a screen is stuck is to compare historical SIDs at the end of an operation. For example, if the siDs have been the same ten times in a row, the page is stuck. Our solution to this problem is to count the sid of the last 30 times, and if the SID with the highest proportion exceeds 87.5% (empirical value), it is considered that the screen blocking process needs to be triggered.

The processing process of Fastbot’s internal screen stuck is as follows: forced return, handling system pop-ups, handling internal pop-ups of the tested App, and forced restart of the tested App.

When the logic is put into operation, the waste of testing time is reduced by 10%. Meanwhile, in order to analyze the specific cause of the blocked screen in the future, corresponding logs will be recorded synchronously when the App is forced to restart. At the end of the task, a task analysis program will be triggered, and specific blocked screen pages will be listed based on the logs for RD analysis and troubleshooting.

3.2 Keyboard scene optimization

In practical testing, through the analysis of test task, found that when Fastbot after entering into the keyboard, will stay in a long time on the keyboard input, a task and there are many times at the same time set up the keyboard operation, which can lead to test the keyboard is a high proportion, has affected the efficiency of Fastbot traversal, weakened front found problems.

To solve this problem, analyze the UI view tree structure in the keyboard scene and optimize the keyboard control. From the UI view tree, it can be analyzed that each key in the keyboard is an independent control, and the algorithm strategy is to traverse every control as much as possible, which will greatly increase the number of keyboard clicks. Therefore, when Fastbot obtains the page element tree, it will cluster the controls on the keyboard. When it obtains the page structure data, it will cluster the resource_id and other attributes of these controls according to the “KBKey” or “UIKeyboardEmoji” in the node information. All the controls on the keyboard are aggregated into one control to reduce the probability of clicking the keyboard.

In addition, in the headlines Feed comment scene, after clicking the full screen, the keyboard and input box fill the whole screen, as shown in the picture below. In this scenario, although the keyboard is aggregated into a single control, the operation path of clicking the keyboard is too deep to be completely solved. For this kind of case, we further optimize the keyboard scene from the perspective of the client, limit the frequency and duration of the keyboard, and run the keyboard for 5 times at most within a life cycle, and the duration of each keyboard is not more than 3 minutes. If the keyboard is not destroyed over time, the keyboard is directly destroyed. At the same time, the first responder of keyWindow is obtained according to the coordinates in the upper right corner. If there is Tap gesture in the first responder, the response logic of Tap is triggered to simulate the actual user operation to jump out of the scene.

After the optimization went online, in a 10-hour stand-alone test task, the proportion of keyboard appearance time was optimized from 20% to less than 5%, which effectively solved the impact of keyboard on the overall test task and improved the test time of non-keyboard scenes.

Fastbot results in consumption

Previously, we mainly described some exploration and practice schemes in the direction of improving Fastbot testing ability. For the intercepted problems, we also made some exploration to solve the problem of fast consumption stability, and provided standards and judgment basis to assist RD to make calibration decisions and optimize the efficiency of calibration.

In the early stage, we established a lot of indicators around Fastbot, such as recall rate, coverage rate, offline crash anomaly rate, machine completion rate, etc. Each indicator has its own scope of application, but these indicators are too scattered and have no primary or secondary, which cannot directly reflect the quality level of a certain version and provide a basis for version approval. For example, the low crash rate offline indicates that the version quality is good, but if Fastbot itself has problems and no crash is detected, the VC coverage of Fastbot should be considered comprehensively to reach a conclusion. Therefore, we made a set of offline identification process to solve the problem of difficult indicator judgment.

Now the offline approval process consists of three parts:

  • The first part is the quality assessment, which is also the core part to reflect the quality of App.
  • The second part is abnormal consumption, which is to identify and solve problems.
  • The third part is the accurate bayonet, used to confirm product quality.

The overall process is as follows:

After the overall launch of the process, it assisted the approval and launch of 10 versions of GIP side, and optimized the overall 2 person days.

The ground effect

Based on the stability test results and business benefits of Toutiao, relevant offline test procedures were constructed from the aspects of test strategy and R&D process. At present, we automatically build the trunk branch pack every day and use it for testing, and the recall rate of abnormal problems has increased from 19.1% initially to 45.76% now (number of fixed problems found by Fastbot bi-monthly/total number of fixed problems bi-monthly). The current VC coverage rate (VC number covered by Fastbot test/VC number touched by online users) is about 80%.

Because Fastbot precision directional test only tests Feature MR with VC change requirement on GIP side, its execution rate is 53.03% (number of tested MR executed/number of changed MR) from December 15, 2021.12.22. Affected by the test duration and whether VC is associated with scheme, the average associated VC coverage (the number of COVERED VC in the test/the number of associated VC in MR) of directional test is 42.95%, in which the residence time of covered associated VC takes up 46.69%. Bimonthly repair rate 15.38% (bimonthly fixed problems found by directional testing/total problems found by bimonthly directional testing). As can be seen from the current repair rate, precision targeting has been able to intercept some stability problems in the MR phase in advance, but there is still a lot of room for improvement.

Looking forward to

Fastbot’s automated task is already doing a good job of detecting stability issues within the App, helping Toutiao block around 50% of abnormal issues. In addition, we will continue to improve Fastbot traversal capability and improve Fastbot related metrics such as line coverage. At the same time, the offline recording and playback are linked, the core test case is recorded into the library, and the directional algorithm is combined to restore the recording operation behavior to the maximum extent, while conducting divergent testing and all-round testing within the scene. Especially for MR level changes, while testing stability, constantly improve the UI assertion and performance testing capabilities, complete the overall automation test, reduce the cost of human testing.

The appendix

[1] Run! Smart Monkey’s Fastbot cross-platform

[2] Report on The Basic Situation of Visually impaired Internet Users in China

[3] github.com/google/GTXi…

Join us

We as today’s headline iOS platform architecture team, performance optimization, basic components, business architecture, research and development system, security compliance, online and offline quality infrastructure positioning attribution of direction such as continuous deep platform, responsible for the security and improve product quality and development efficiency of today’s headlines, focusing on today’s headlines extends outward at the same time. If you’re passionate about technology, you love to push the envelope, and you want to change the experience of hundreds of millions of people with your own code, join us. We look forward to your growing with us. At present, we have recruitment requirements in Beijing and Shenzhen, resume email: [email protected]; Email subject: Name – Years of work – Today’s Headlines – Platform architecture – iOS/Android.