Tao Li joined Qunar in March 2015 as a client development engineer. He used to be in charge of the development of hotel business and self-service inbound and outbound projects. Now I am responsible for the development and maintenance of international hotel crawl system.

Android automated interactions can replace repetitive tasks performed by humans, including black box testing through automated apps and automated running of third-party apps. Common automated interactions include launching an App, clicking, dragging and dropping views, and text input. With the improvement of App security capability, it becomes more and more difficult to realize the automatic interaction of the complete process. This paper mainly discusses the current common automatic interaction solutions, the advantages and disadvantages of different solutions and application scenarios.

1 Traditional script execution scheme

ADB is a command line tool provided by Google to interact with Android devices. We can write scripts to execute each event one by one in a predetermined order. ADB performs an operation by getting the coordinates of the interface element (either uiautomator or dump XML, which is not the focus of this discussion) and passing the coordinates as a command-line argument.

adb shell input tap 100 500
Copy the code

The above command simulates clicking on the control at the screen coordinate (100,500).

adb shell input swipe 100 500 200 600
Copy the code

The above command simulates a finger swiping down and right on the screen.

adb shell input keyevent "KEYCODE_BACK"
Copy the code

The above command simulates the click of the return key.

A complete automated interaction process can be executed in the sequence of the above commands.

Advantages of the ADB script approach

  1. Simple implementation, only need to obtain the coordinates of the target element and other simple information to complete the related operations
  2. You can automate the interaction with the WebView

Disadvantages of the ADB script approach

  1. It is not flexible enough and depends on the written coordinates. The view position transformation caused by the App interface change will make the relevant commands in the script unable to be executed, so the page coordinates need to be analyzed again
  2. ADB links or socket links need to be set up, and changes in network conditions during the interaction can affect the automated interaction \

Application scenarios of ADB scripts

  1. App with simple interaction, low iteration frequency and low security level
  2. Webview page, App developed by Flutter

2 Android native methods for automatic interaction

We can use all kinds of plug-in framework to control App interface elements of the page, and is one of the ideas in the plug-in use ActivityLifecycleCallbacks to monitor each activity lifecycle.

public class MyApplication extends Application { private static final String TAG = "MyApplication"; / / declare an interface to monitor the Activity lifecycle private ActivityLifecycleCallbacks ActivityLifecycleCallbacks = new ActivityLifecycleCallbacks () {/ * * * under the application of each Activity statement cycle changes, will trigger the following function. */ @Override public void onActivityCreated(Activity activity, Bundle savedInstanceState) {// How to distinguish which activity in the parameter represents which activity you wrote. if (activity.getClass() == MainActivity.class) Log.d(TAG, "MainActivityCreated."); else if(activity.getClass()== SecondActivity.class) Log.d(TAG, "SecondActivityCreated."); } @Override public void onActivityStarted(Activity activity) { Log.d(TAG, "onActivityStarted."); } @Override public void onActivityResumed(Activity activity) { Log.d(TAG, "onActivityResumed."); } @Override public void onActivityPaused(Activity activity) { Log.d(TAG, "onActivityPaused."); } @Override public void onActivityStopped(Activity activity) { Log.d(TAG, "onActivityStopped."); } @Override public void onActivitySaveInstanceState(Activity activity, Bundle outState) { } @Override public void onActivityDestroyed(Activity activity) { Log.d(TAG, "onActivityDestroyed."); }}; @Override public void onCreate() { super.onCreate(); // Register your own Activity lifecycle callback interface. ! [Alt text](./WechatIMG59.png) registerActivityLifecycleCallbacks(activityLifecycleCallbacks); } @override public void onTerminate() {// Cancel the interface. unregisterActivityLifecycleCallbacks(activityLifecycleCallbacks); super.onTerminate(); }}Copy the code

After listening to the activity, you can analyze the viewId and attributes of the activity interface elements with the help of UIautomator. Different interface views can be automated in different ways.

2.1 Processing of simple View

The diagram below:

For a view like this, you can directly obtain the resource ID and confirm that the clickable property is true. The operation method is simple. You can perform the following operations in the activity lifecycle that you are listening to:

int fl_btn = activity.getResources().getIdentifier("dashboard_title",         "id", "com.android.settings");
View v =  activity.findViewById(fl_btn);

v.performClick();
Copy the code

2.2 Handling of views with hidden attributes

In some mixed development pages such as React Native, the above method no longer works, as shown in the view below:

As shown in the figure, the clickable property of the selected ViewGroup and its child view is false, and the resource ID of the view cannot be obtained. In this case, we can use the layout information dump from the graph. Using the Xpath element positioning tool to get the view of the interface, because the click property of these views is false, so calling performClick to implement the click method is no longer valid. Consider the more important class related to touch event passing below click: MotionEvent, MotionEvent can simulate almost all interactive events, including click, slide, two-finger operation, etc. Take clicking as an example:

private void simulateClick(View view, float x, float y) { long time = SystemClock.uptimeMillis(); // Must be systemclock.uptimemillis (). MotionEvent downEvent = MotionEvent.obtain(time, time, MotionEvent.ACTION_DOWN, x, y, 0); time += 500; MotionEvent upEvent = MotionEvent.obtain(time, time, MotionEvent.ACTION_UP, x, y, 0); view.onTouchEvent(downEvent); view.onTouchEvent(upEvent); }Copy the code

For sliding operations, add several ACTION_MOVE motionEvents in the middle of the starting position. To sum up, the advantages and disadvantages of time interaction automation with system native methods are as follows:

The benefits of automated interactions with the plug-in framework

  1. Maintainability is strong because you can get the view object of the interface directly, so even if the page layout changes, there is no need to change the code as long as the View exists
  2. Simulation results are better than the script approach is closer to the real operation

The drawbacks of automating interactions with plug-in frameworks

  1. Poor support for webView, Flutter framework App

Application scenarios

  1. App with frequent version iterations
  2. App developed without the Flutter framework

Analyses the two commonly used on the simulation method of automation of real users, script and call native methods, the two methods can basically complete the simple interaction process, based on this, we can also go to explore some deeper interactions, such as automation through various validation, etc., can also be based on these two methods.