When designing automated test cases for Android phones, whether a test scenario can be automated depends on whether it needs human participation. For wifi can be automatically turned on and off, SMS can automatically send and receive such scenarios, without the participation of people can be judged through the program, so the test of wifi and SMS can be automated test through the program. But there are other test scenarios that require a human eye and are more difficult to automate.

Demand analysis

How do I determine whether a website has been loaded successfully when I visit a website using an Android browser? The target site did receive the request, and the HTML data was returned, and the phone received the data from the site, but for some reason, the browser didn’t display anything and the Log the browser sent out didn’t look anything at all. For such a scenario, in order to reduce the labor cost, how to let the test program automatically detect the web page failed to load, and notify the developer?

This requirement can be automated using computer image recognition.

Design ideas

Since this requirement only needs to judge whether the web page loads successfully, it does not need very sophisticated image recognition theory. For a web page, a successful load means that its contents appear normally in the browser. Therefore, you can design a special web page that is simple enough, but special enough to facilitate image recognition.

This article will be tested using a pure green web page. After the web page loads, most of the screen is green. At this point, take a screenshot and use the program to recognize the screenshot. If you see a large green area in the screenshot, it means the page has already loaded.

The Demo implementation

Pure green pages

Create a “greenpage.html” with the following code:

<html>
<head>
    <title>Green Page</title>
</head>
<body bgcolor="green">

</body>
</html>Copy the code

After the site loads successfully, the page is all green, as shown below.

Build a Web Server in the LAN, and let the LAN devices can be linked. Open the terminal, go to the folder where the HTML file is located, and use Python 3 to set up a simple WebServer on the LAN:

cd ~/Project/IdentifyWebpage
python -m http.serverCopy the code

Access computer IP:8000/greenpage.html from a mobile phone, and the result is as shown in the following figure.

Identify green screenshot

This Demo uses the Pillow image processing library to do image color recognition. Install the Pillow via PIP:

pip install pillowCopy the code

Once installed, use the following command in a Python program:

from PIL import ImageCopy the code

Import it image module.

The program uses the Image module to load the screenshot and read the RGB value of a certain color from the screenshot:

img = Image.open('snapshot.png')
color = img.getpixel((700.800))
print(color)Copy the code

In the code, (700, 800) is the coordinates of a point in the screenshot. The first parameter is the horizontal coordinate and the second parameter is the vertical coordinate. The upper left corner of the screenshot is (0, 0), and the further down, the bigger the vertical coordinates; The farther to the right, the greater the horizontal.

To be cautious, take 9 points in the screenshot and get their RGB values:

points = [(200.300), (455.678), (333.1200),
          (300.500), (888.678), (900.800),
          (400.600), (245.365), (799.777)]
img = Image.open('snapshot.png')
for point in self.points:
    color = img.getpixel(point)
    print(color)Copy the code

The result after running is as follows:

As can be seen from the figure, the RGB values of all 9 points are (0, 128, 0), which is exactly the green RGB value. At this point, you can assume that most of the picture is indeed green. If you think nine points is not enough, you can use code to generate hundreds of points to calculate.

Get screen shots

Android Debug Bridge (ADB) is a debugging tool for Android. You can use ADB commands to control mobile phones. To take a screenshot of your phone, just use the following two commands:

adb shell /system/bin/screencap -p /sdcard/screenshot.png
adb pull /sdcard/screenshot.png ~/Project/IndenfyWebpage/screenshot.pngCopy the code

The first command generates a screenshot and saves it to the phone’s built-in storage. Even though it says “sdcard,” this command still works on phones that don’t have an SD card.

The second command pulls the screenshot from the phone’s built-in storage and saves it to your computer. If you are running Windows, you can change the second command to:

adb pull /sdcard/screenshot.png D:/Project/IndenfyWebpage/screenshot.pngCopy the code

Save the file to drive D. Note the use of a slash (/) from the top right to the bottom left.

The full program can be found at -> github.com/kingname/Au…

More applications

Although this Demo is only for web testing. But this approach validates many other test scenarios besides web pages. For example, to verify that the video can play properly, to make a special video, video is several different solid color screen constantly switch. Each screen stays for a certain period of time, and the program periodically obtains screenshots and analyzes whether it is a solid color screen at this time, and whether the solid color screen is changing.

In theory, any test case that needs to display information on the screen could be automated using this method.