Want to develop a web crawler and find it’s crawling backwards? Trying to capture packets on your App and find that the data is encrypted? Don’t worry, use Airtest to develop App crawler, as long as human eyes can see, you can catch, the fastest takes only 2 minutes, compatible with Unity3D, Cocos2dx-*, Android native App, iOS App, Windows Mobile… .

Airtest is a mobile PHONE UI automation test tool developed by netease. Its original purpose is to simplify the writing of mobile App GRAPHICAL interface test code through what you see is what you get, screenshot clicking and other functions.

Crawler development based on the belief that all the tools I can use to get data can be used to develop crawlers, I decided to use Airtest to develop mobile App crawlers.

Installation and use

Since the purpose of this article is to show you how to use Airtest to develop an App crawler, the introduction to Airtest as a test development tool will skip over and just show you how to install and perform basic operations.

Install Airtest

Download the Airtest from airtest.netease.com and install it as normal software. There is nothing special about the installation process. Airtest has packaged all the environments you need for your development, so once Airtest is installed, it’s ready to use.

The following figure shows the interface after the Airtest is run.

Connect the phone

In the case of an Android phone, since Airtest will install two auxiliary apps on the phone using ADB commands, and then control the phone by controlling the two auxiliary apps using ADB commands, you first need to make sure that adb debugging is enabled on the phone and that the App is installed on the phone using ADB commands.

Once Airtest has started, connect your Android phone to your COMPUTER and click Refresh ADB in the box below:

In this case, the phone information should be displayed in the upper right corner of the Airtest interface, as shown in the following figure.

Click the Connect button to see the phone’s interface on the screen, and when you manually manipulate the phone’s screen, the phone’s screen in Airtest updates in real time. See the figure below.

For some phones, such as Xiaomi, when you use Airtest for the first time, please note that the phone will pop up a prompt asking you whether you are allowed to install the App. At this time, you need to click the allow button.

Open the WeChat

Let’s take a look at a simple example of how to get started with Airtest quickly. More on this later.

For example, I now want to use the computer to control the phone and open wechat.

At this point, click the “Touch” button in the picture below:

At this point, move the mouse over the area of the phone screen to the right of the Airtest and the mouse will turn into a cross. Press the left mouse button in the upper left corner of the wechat icon, and drag it to the lower right corner of the wechat icon to release the mouse. Notice at this point what has changed in the middle code area, as shown in the figure below.

All right. That’s all you need to use your computer to open wechat.

Click on the triangle icon in the upper toolbar to run the code, as shown below.

After the code was run, wechat was opened.

Interface is introduced

Now that we have an intuitive use, let’s take a look at the Airtest interface, which will be more targeted.

The following figure shows the Airtest page.

Here, I’ve divided the Airtest into six zones, a-F6, and their functions are as follows:

  • Area A: Functions for common operations
  • B area: Python code writing area
  • Area C: Run log area
  • D area: mobile phone screen area
  • E zone: View the App page layout information
  • F area: toolbar

Area A is A commonly used screen operation function based on image recognition, such as:

  • touch: Click on the screen element
  • swipe: Slide screen
  • exists: Determines whether the screen element exists
  • text: Enter text in the input box
  • snashotScreenshots:
  • …

Generally speaking, click on A function in the A area, and then select the box on the D area screen, and the B area will automatically generate the corresponding operation code.

Section B is used to display and write Python code. In most cases, there’s no need to write code manually, because the code will be generated automatically based on what you do on the phone’s screen. Only actions that require special customization need to be modified.

The D-block shows the phone’s screen, which refreshes in real time when you operate it. You can also use the mouse to operate the phone directly from the D-block screen, and your actions will be automatically performed on the real phone.

The F block is a collection of commonly used tools, from left to right:

  1. New project
  2. Open the project
  3. Save the project
  4. Run the code
  5. Stop code
  6. View the run Report

The ones 1-5 are easy to understand, so what is viewing a run report?

After you’ve run it at least once, clicking on this feature will automatically open a web page for you. The page is shown in the following image, which is a running report of your code, down to the details of what elements were manipulated in each step.

Although it is convenient to operate the phone through the screenshot function, the screenshot involves the resolution problem, and the code cannot be used on different phones. So for the function of A area, do some simple operation, do not need to in-depth understanding.

More advanced functions need to be implemented through the E zone.

Operate the phone based on App layout information

Initialization code

The layout information of an App is like the HTML of a web page, saving the relative position and parameters of each element on the App. For an App, on mobile phones with different resolutions, the same element may have different coordinate points, but the attribute parameters of this element generally remain unchanged. Therefore, if you use the attribute parameters of the element to find and control the element, you can achieve precise positioning on the phone with different resolution.

The format of the App layout information depends on the development environment of the App. Click on the drop-down menu in the F section and you can see that you can specify different App development environments. Among them, Unity, Cocos-* and so on are generally used for games, Android is the native App of Android, iOS is the App of Apple… See the figure below.

Take the mobile version of Zhihu as an example. Since it is a native App of Android, select Android from the drop-down menu in the F area. At this time, pay attention to the pop-up prompt in the B area, asking you whether to insert the POCO initial code into the current input cursor position.

At this point, a section of code is automatically inserted in area B, as shown in the figure below.

Position and click

Now, click on the lock icon in section E, as shown in the image below.

After the lock icon is activated, you can operate the screen in the D area and click the word “Zhihu” under “Zhihu App”. You will find that the clicked App on the screen will not open. However, areas E and C have changed, as shown in the figure below.

The tree structure displayed in the E pane is the layout information of the current screen, which is similar to the HTML structure displayed in Chrome developer tools. The C block displays information about the element I’m currently clicking on.

Notice that in the element information, there is a text attribute whose value is known. Then, this attribute can be used as a positioning element, so we can write code in B:

poco(text="Zhihu").click()
Copy the code

After writing the code and running the program, you can see that the Zhihu App is opened. See the figure below.

Note that if you see a difference between the actual screen and the Airtest screen, you may have locked the Airtest screen. Click on the lock icon in the F area, unlock, and the phone screen in Airtest will be updated.

Locate and enter

After opening Zhihu, I want to use the search function of Zhihu, so go ahead and activate the lock icon, and then click the search box at the top of Zhihu, as shown in the picture below:

Continue to see the area C display attributes of the search box, you can see there is a name attribute, its value is com. Zhihu. The android: id/input, and a text attribute, its value for the ambassador CAI Xu Kunren New Year New Year in the NBA. Can I use the text property as I did with Zhihu? Yes, and no. It works because it actually works now; It can’t, because it is a popular search keyword on Zhihu and can change at any time. If you use this sentence successfully today, tomorrow the buzzword changes and your code won’t work. So you need to use the name attribute.

Common properties that are essentially unchanged include, but are not limited to: Name Type resourceId Package.

In addition, the search box on the home page of Zhihu is actually not able to input content. When you click on it, it will jump to another page, as shown in the picture below.

So you need to click on this input box to go to the actual search screen:

poco(name="com.zhihu.android:id/input").click()
Copy the code

The actual search interface is shown below.

As you can see, the name attribute’s value is still a com. Zhihu. The android: id/input, at this point you can input the content.

The set_text method is used to input the content.

poco(name="com.zhihu.android:id/input").set_text('Ancient Sword and Strange Tan Three')
Copy the code

Locate and screen

After entering the search terms and looking at the current page, the search results come up with three:

By comparing the attribute information of these three results, it is found that their name attribute is the same, while their text attribute is different. If you write the click action like this:

poco(name='com.zhihu.android:id/magi_title').click()
Copy the code

The default is to click on the first result.

What if I want to click on the second search result? You can write the code like this:

poco(name='com.zhihu.android:id/magi_title', text='Ancient Sword legend' (TV series)).click()
Copy the code

Or you can use index positioning like a list:

poco(name='com.zhihu.android:id/magi_title') [1].click()
Copy the code

The premise of both of these is that we already know what each result is. Suppose now I want to search for Gujian Qitan 3, but I don’t know what the search result is, what should I do? You can also use regular expressions:

poco(name='com.zhihu.android:id/magi_title', textMatches='^ Ancient Sword Legend iii.*$').click()
Copy the code

Sliding screen

Once you have entered the search results, you need to view the following questions, and you need to keep swiping up the screen. One important thing to note here is that Airtest only gets the layout information for elements on the current screen, not for content that is not on the screen. This is different from Selenium.

The command to swipe the screen is Swipe, which requires the use of coordinate information. But these coordinates are independent of the screen resolution. The coordinates here are defined as :(x, y), where x is the x-coordinate and y is the y-coordinate. The upper left corner of the screen is (0, 0), and the lower right corner of the screen is (1, 1). From left to right, the x-coordinate gradually increases from 0 to 1, and from top to bottom, the y-coordinate gradually increases from 0 to 1.

Now I’m going to swipe the screen up, so on the real machine, I’m holding down the bottom of the screen, and then I’m going to swipe the screen up, so the code looks like this:


Swipe (point of departure, left of destination)Poco. Swipe ([0.5, 0.8], [0.5, 0.2])Copy the code

The direction diagram is shown in the figure below:

In general:

  • If you slide up, you just change the ordinate, and the starting point is greater than the ending point
  • If you slide down, all you have to do is change the ordinate, and the starting point is less than the ending point
  • If you slide to the left, you just change the abscissa, and the starting value is greater than the ending value
  • If you slide to the right, all you need to do is change the abscess, and the starting value is less than the ending value

In the development of crawler, the Airtest operation involved has been basically introduced.

Use Python alone to control the phone

Airtest is easy to operate, but it’s not possible to install Airtest on every computer. So you need to find a way to separate the code from Airtest.

Airtest is based on Poco, an open source library for Python, and the Python code written in the B-block of Airtest is actually Poco code. So by installing the Poco library, you can control the phone directly from Python.

The Poco library installation command is:

pip install pocoui
Copy the code

This library relies on a lot of things and is slightly slower to install. Once the installation is complete, we copy the code into PyCharm, as shown in the figure below.

If you are a Linux or macOS user, please note that adb does not have permission to run. This is because adb installed with Poco does not have permission to run. You need to add permission to it to run the command on the terminal:

# chmod +x error message given adb addresschmod +x / Users/kingname/local/share/virtualenvs/ZhihuSpider/lib/python3.7 / site - packages/airtest/core/MAC/android/static/the adb/adb (Please change to your address for actual execution)Copy the code

After running the command, execute the code again. You can see that the code runs successfully and the phone is successfully controlled, as shown in the figure below.

How do I get screen text

Since the code in Airtest’s editor does not print Chinese properly, the rest of the code is executed directly in PyCharm.

To be a crawler, you need to get the text content on the phone. Back to the search page, I want to know how many results can be found by the keyword “Ancient sword and strange tan”, and how many discussions can be found in each result, as shown in the figure below:

At this point we need to do two things:

  1. Look at each search result separately
  2. Gets the text on the screen

The tree structure of area E is shown in the figure below:

The title of each search results as a text attribute’s value, in the name = ‘com. Zhihu. Android: id/magi_title’ corresponding elements; Each of the search results to discuss several as text attribute’s value, in the name = ‘com. Zhihu. Android: id/magi_count’ corresponding element.

The most straightforward way to do this is to take three headings and three discussion numbers and combine them together:

title_obj_list = poco(name='com.zhihu.android:id/magi_title')
title_list = [title.get_text() for title in title_obj_list]

discuss_obj_list = poco(name='com.zhihu.android:id/magi_count')
discuss_list = [discuss.get_text() for discuss in discuss_obj_list]

for title, discuss in zip(title_list, discuss_list):
    print(title, discuss)
Copy the code

The running effect is shown in the figure below:

But this practice is actually very dangerous, assuming that there will be a very rare search results, only the title does not discuss the number, so separate grab recombination, will lead to the final match dislocation. So it makes sense to focus on the big first and then the small. Each set of titles and discuss several, they all have their own parent node, three android. The arrow points to the diagram below widget. The LinearLayout:

So now, using the catch big and then catch small technique, first grab the parent node of each set of results, and then go to each result to get the title and the discussion number respectively.

However, how to obtain the parent node? As shown in the figure below, each property value of the parent node is not special, and writing any one of them may collide with another node.

In this case, the easiest thing to do is to double click on the parent node in the E area. The location code is automatically added, as shown in the figure below.

This location code looks very complicated, but the underlying logic is very simple, just looking from the top layer to the bottom layer.

The automatically generated location code is as follows:

poco("android.widget.LinearLayout").offspring("com.zhihu.android:id/action_bar_root").offspring("com.zhihu.android:id/parent_fragment_content_id").offspring("android.support.v7.widget.RecyclerView").child("android.widget.LinearLayout") [0]Copy the code

In this automatically generated location code, we see the offspring and child methods. Where child represents the child node, offspring represents the grandson node, the child node of the grandson node, the grandson node of the grandson node… . In short, using Child only searches the child nodes for what you want, while with Offspring, you iterate through all the nodes in the folder recursively until you find an attribute that matches the condition. Offspring are obviously slower than children.

In fact, we can simplify this location code a bit:

poco("com.zhihu.android:id/parent_fragment_content_id").offspring("android.support.v7.widget.RecyclerView").child("android.widget.LinearLayout") [0]Copy the code

This simplification method uses the same logic as the XPath copied from Chrome. The basic principle is to find the “unique” attribute value and then use that attribute value to locate.

Since I clicked on the first search result, there is a [0] at the end of the location code. Now, since we need to get all the content of the search results, we should remove [0] and use the for loop to expand and get the contents:

result_obj = poco("com.zhihu.android:id/parent_fragment_content_id").offspring("android.support.v7.widget.RecyclerView").child("android.widget.LinearLayout")
for result in result_obj:
    title = result.child(name='com.zhihu.android:id/magi_title').get_text()
    count = result.child(name='com.zhihu.android:id/magi_count').get_text()
    print(title, count)
Copy the code

The running effect is shown in the figure below.

Control multiple phones

When we plug in multiple Android phones on a computer, execute the command:

adb devices -l
Copy the code

The running effect is shown in the figure below.

Every phone will be listed. The number on the far left is the phone number. Use this string to specify multiple phones:

from airtest.core.api import auto_setup
from airtest.core.android import Android
from poco.drivers.android.uiautomation import AndroidUiautomationPoco
auto_setup(__file__)

device_1 = Android('76efadf3a7ce4')
device_2 = Android('adfasdfasf23')
device_3 = Android('adifu39ernla')

poco_1 = AndroidUiautomationPoco(device_1, use_airtest_input=True, screenshot_each_action=False)
poco_2 = AndroidUiautomationPoco(device_2, use_airtest_input=True, screenshot_each_action=False)
poco_3 = AndroidUiautomationPoco(device_3, use_airtest_input=True, screenshot_each_action=False)
Copy the code

In this way, using the USBHub on one computer and connecting 20 or 30 phones is perfectly fine.

Wireless mode

Airtest supports wireless mode, no USB is required, as long as the computer and the phone are connected to the same WIFI can be controlled:

If you’re interested in how to turn on wireless mode, leave a comment and I’ll keep writing.

Build a mobile crawler cluster

A computer can connect 30 mobile phones, so if there are many computers and many mobile phones, mobile phone crawler cluster can be realized. The operation effect is shown in the following figure.

How to build a crawler cluster is beyond the scope of this article. If you are interested, you can read my book: Python Crawler Development From The Beginning to the Practical chapter 10 has detailed instructions and cautions on how to build a mobile crawler cluster.

If you are interested in my book, please follow my wechat official account and communicate with me.