Small knowledge, big challenge! This article is participating in the creation activity of “Essential Tips for Programmers”.

I want to collect Kitty candy, but I don’t want to watch commercials. I want to write a script robot to help. I think of the script of wechat jumping through the level. This simple version of the robot should be able to achieve.

Brush shop set meow candy operation is very mechanical, divided into several steps:

  1. Click the “earn candy and get red envelope” button to expand the task panel
  2. Click the “Go Browse” button in the panel to start looking at the ads
  3. Commercial time 15 ~ 20 seconds
  4. Cat candy in hand, go back up to step 2

That is, if we can figure out where the button is, if we can trigger it, if we can go back one level, we can do it.

Implementation idea:

  1. adbI just want to take a screenshot of the phone and upload the picture to the PC
  2. PC side for image recognition, determine the location of the button
  3. adbPerform touch screen operations

Connect a mobile phone to a PC

Install ADB on PC

brew install --cask android-platform-tools
Copy the code

Verify the installation

adb version
Copy the code

Start debugging on mobile terminal, generally about mobile phone -> version number -> combo N times, after opening developer option, open USB debugging in other Settings.

Android 11 can also enable wireless debugging mode:

  • Developer options -> Wireless Debugging
  • Pair the device with the pairing code
  • PCadb pair ipaddr:portEnter the matching code to match
  • After the pair is paired successfully, the PC endsadb connect ipaddr:portThe connection is successful

See the official documentation for more details

adbTake a screenshot and upload it to the PC

import { promisify } from "util";
import { exec as nodeExec } from "child_process";
const exec = promisify(nodeExec);

async function pullCurrentScreenShot() {
  / / screenshots
  await exec("adb shell screencap /sdcard/meow_candy.png");
  / / upload
  await exec("adb pull /sdcard/meow_candy.png ./images");
}
Copy the code

Image recognition determines button position

There are several ways to identify elements from images:

  1. If the element has a fixed pixel distribution, the position is obtained by pixel matching
  2. If the element contains characteristic text, the position is obtained by text recognition
  3. Opencv accurately identifies elements as a whole

After a simple attempt at tesseract, exclude text recognition. Because the active page elements are too many, and the font is not uniform, OCR recognition accuracy is very low. Opencv involves image processing algorithms that take a while to get started. Implement the basic requirements first, try to use pixel matching simple implementation.

Pixel matching idea:

  1. Make sure you click on the target image and save the screenshot.
  2. Gets the pixel matrix of the source image and the target
  3. Search based on pixel matrix matching

It is too expensive to compare the full graph matrix when matching.

Since the accuracy requirement is not high, we first grayscale the whole image, then take the four endpoints of the target image to form a rectangular box scanning match, and finally calculate the pixel difference between the matched area of the original image and the target image.

Image processing is implemented using Jimp.

import Jimp from "jimp";

type TapTargetType = [number.number];
async function recognizeTapTarget(
  srcImgPath: string,
  targetImgPath: string
) :Promise<TapTargetType | null> {
  // Prepare the target image endpoint data
  const targetImage = await Jimp.read(targetImgPath);
  targetImage.greyscale();
  const tWidth = targetImage.bitmap.width;
  const tHeight = targetImage.bitmap.height;
  const rect = [
    targetImage.getPixelColor(0.0),
    targetImage.getPixelColor(tWidth, 0),
    targetImage.getPixelColor(tWidth, tHeight),
    targetImage.getPixelColor(0, tHeight),
  ];

  // Read the current screenshot
  const image = await Jimp.read(srcImgPath);
  image.greyscale();
  const sWidth = image.bitmap.width;
  const sHeight = image.bitmap.height;
  
  // Start scanning from the upper left corner
  let matchedPoint: TapTargetType | null = null;
  for (const { x, y } of image.scanIterator(
    0.0,
    sWidth - tWidth,
    sHeight - tHeight
  )) {
    // The first four items are endpoint comparison, and pixel difference calculation is carried out after layer upon layer verification
    if (
      image.getPixelColor(x, y) === rect[0] &&
      image.getPixelColor(x + tWidth, y + tHeight) === rect[3] &&
      image.getPixelColor(x + tWidth, y) === rect[1] &&
      image.getPixelColor(x, y + tHeight) === rect[2] &&
      Jimp.diff(image.clone().crop(x, y, tWidth, tHeight), targetImage, 0.4).percent <
        0.15
    ) {
      matchedPoint = [x, y];
      break; }}return matchedPoint;
}
Copy the code

adbTouch screen operation

See the description of adb input commands

adb shell input --help
Copy the code

Click on a certain point

adb shell input tap <x> <y>
Copy the code

Sliding screen

adb shell input swipe <x1> <y1> <x2> <y2> [duration]
Copy the code

Return to the superior

adb shell input keyevent 4
Copy the code

conclusion

Simple implementation, meet basic needs, point to point.

Cute Meow Candy Automatic Collector for 11.11 (github.com)