The introduction
O2O market and smart phones are developing rapidly. As a leading O2O mobile Internet company, Ele. me is also expanding its business. As App functions become more and more complex, coupled with the diversified technical solutions of mobile terminals and the complex network environment in China, there are many challenges to do App testing well. This paper mainly describes the challenges faced by Ele. me in App testing at present and the efforts made to cope with the challenges.
challenge
Currently, ELE. me has reached tens of millions of DAU. Under the background and requirements of rapidly adapting to market needs, catering to user needs and providing good user experience, mobile testers and developers are facing some new challenges.
Third party automation tools are not effective enough
With the rapid development of Ele. me business, the scene is becoming more and more complex and diversified, and the new functions and features of THE App are becoming more and more abundant. Therefore, the regression test requires more and more time and manpower, and there is an urgent need for automated solutions that can effectively improve the test efficiency.
For testers, in the case of rapid iteration and frequent UI changes, time and manpower are limited, and the inability to perform comprehensive regression testing is the biggest pain point. Using some third-party tools for script recording or script writing (such as Appium, etc.) has low ROI. Testers often need to modify scripts repeatedly according to CHANGES in UI, and at the same time, they also need to take into account the test tasks in iterations. The speed of script modification cannot keep pace with changes in UI, and maintenance costs are high. Finally, the problem of insufficient regression test coverage cannot be solved.
App performance is getting more and more attention
In a world where users are the most important and experience is the king, how to save electricity and flow and how to ensure the smooth operation of App is becoming more and more important. Accordingly, testers need to pay more attention to the performance of App in the testing process. Therefore, testers are required to collect real-time performance data (memory, CPU, flow, FPS, etc.) of App throughout the whole process, and make further comprehensive analysis on these data, such as the time point at which memory suddenly increases and does not fall back, the corresponding test steps, and the function page, etc. Multiple and scattered performance data and separate timelines for manual testing and data recording make it difficult for testers to find and locate problems.
Performance issues are not easy to solve for developers either. The performance data collected during the test can only reflect the external performance of the App running, corresponding to the internal code, involving a very wide range, in addition to the source code of the App itself, it will also involve the source code of the SDK and other third parties, which is very time-consuming to check.
summary
In the face of the above challenges, many large manufacturers & companies with pursuit of mobile technology will make efforts to develop their own automated test platform. The Ele. me mobile Infrastructure team is developing the Stellar testing platform to help testers perform mobile testing better and more efficiently.
The solution – Stellar testing platform
With the recording and playback SDK of mobile terminal, it is necessary to have an operating platform that can manage scripts, execute scripts and record results, and connect the process of recording, playback and recording results in series. Hence the Stellar mobile testing platform.
Architecture design
Below is a hierarchical architecture diagram of the platform:
The platform is mainly divided into three layers: front-end, service and mobile terminal. The front end is the Web page, which is the entrance to interact with users and provides functions such as script management, script debugging, and test report viewing. The service layer is the front-end and mobile terminal undertaking layer, responsible for script management, script execution task scheduling, test report generation, network request mock, etc. The mobile terminal accesses the App under test in the form of SDK, records UI operations and controls, sends proxy network requests to the mock Server, plays back scripts, and records logs and performance data.
process
After the App under test is connected to the SDK, the user can connect the test machine with the platform through the connection code, so as to realize script recording and playback. In addition to opening the platform and establishing the connection, script recording is carried out on the mobile terminal. After connecting the test machine to the platform, the tester can normally operate the test case on the mobile side. The SDK will record the whole test process and send the network request to the Mock Server. There is no additional operation for the tester, which improves the efficiency of script recording and reduces the operation complexity.
In order to facilitate the testers to quickly reach the functions under test, the SDK provides the scene selection function. During recording, the starting scene can be selected to directly reach the page under test, saving the testers’ time. Because network requests are stored in the Mock Server during recording, all requests are from the Mock Server during playback, ensuring the stability of playback. The tester can choose whether a request goes to the Mock Server or uses real data based on the actual situation. Network request editing will be provided later, allowing testers to define their own network requests.
Technical Solution Introduction
The front-end is mainly developed by combining the Element framework developed by Ele. me with WEditor. Elem. IO /#/ zh-cn
The Service layer is primarily a Java backend, with a small amount of secondary development based on STF. The Java backend includes script management, test reports, mock Server, and task scheduling is implemented by secondary development based on STF.
The mobile terminal uses the embedded SDK to proxy and record UI operations, and supports both iOS and Android terminals. The implementation idea of Android terminal is to proxy and record UI operations through embedded SDK. The basic idea is to add an intermediate layer on top of the App to capture and record on-screen UI actions. The diagram below:
The Implementation idea of Android SDK is to proxy and record UI operations, and play back based on Instrumentation. The implementation idea of iOS SDK is to listen to the events distributed in the Responder Chain, further analyze the events, record the path of the events and click areas. The same method is used for playback.
The above platform design requires almost no additional operations for the tester to record playback. You only need to turn on the recording switch and manually test the App according to the written test cases. After the test is completed, the script is recorded. The platform converts recorded scripts into readable steps that testers can use for secondary editing and script playback.
After the recording and playback function is realized, the support for the initial recording scene is added according to the usage scenario of the tester. The developer can customize the initial recording scene according to the requirements of the tester when accessing the SDK, so that the tester can quickly reach the functional module in his charge. There is no need to record the passing scene between the start page of App and the module under test.
There will be a follow-up article to introduce technical solutions, welcome to continue to pay attention to.
conclusion
Stellar’s testing platform is designed to address the pain points and challenges of mobile testing by automating script recording and playback of UI pages, and collecting performance data during script playback. In addition, the system snapshot corresponding to the bug can be recorded in the test process with one key and submitted to the bug management system. In the future, in-depth performance analysis will be added to identify performance problem spots at the code level to help developers quickly locate problems.
With the rapid development of mobile testing today, we hope to help more testers and developers improve efficiency through our exploration and efforts.