This article appeared in:Walker AI

With AI, what is the experience of playing a game with a normal player client?

Direct people do not have to play the game, put that, let the AI help you play, after a while to see their points up. How cool! Please save all the money. No problem. However, we are here today to discuss serious technical matters. How to use AI to realize our automated testing! That’s what we’re here for.

A competitive game, we’re going to break it down a little bit here, and we’re going to break it down into an outside interface and an inside combat. For example, like LOL, in the lobby, group, select hero interface these are the outside interface. When we wait for long bar, read into the game, hear the sound of the “product knocked” hero appeared to the front of us, after this time, you are already in the station’s part in the battle.

1. An overview of the

This paper mainly explains the automation of intra-office combat. It includes an introduction to the process of implementing combat automation and the framework for running the script itself. Readers should have some experience in Python programming and UI automation development. In particular, there is the use of AI provided by the company’s in-house team for decision-making operations. In a nutshell, we tell the AI what’s going on in the office, the AI thinks about it, returns the decision, and we execute the decision using automated scripts. But it is possible to do automated testing without AI. We need to build the desired process ourselves. Calling THE AI is like a reflex, and self-building the corresponding process script is like an unconditioned reflex.

UI automation for the game is implemented in the form of POCO’s UnitysDK.

A little bit about POCO. This is a client-side UI automation framework similar to Appium that utilizes interface elements for UI automation.

The official document: poco – Chinese. Readthedocs. IO/zh_CN/lates… *

Elements are derived from UITree, which is derived from the game client that connects to unitysDK. Through socket communication with the external mount agent, the UITree of the game using Unity engine is returned to the external for automatic correlation.

UITree contains all the attribute information of the elements in the corresponding Tree, including the position attribute (position information) of the element, which is also the core basis for us to lock the corresponding elements for operation.

Obtain the UI layer information of the game client through pocosDK, and handle the specific contents of the UI information accordingly.

Finally, the corresponding data is fed back to AI, and the AI makes the corresponding decision. After the decision information is returned to the local computer, the corresponding UI automation operation code is executed to complete the decision. Through the implementation of a large number of decisions, the real-time operation of the whole bureau is completed.

Then, the multi-round reopening detection mechanism is introduced to complete the automatic battle of intra-office combat.

  • Some simple demos:

In the Unity project, you need to install the Poco SDK in Unity to call Poco methods

import time
from poco.drivers.unity3d import  UnityPoco

poco = UnityPoco()

poco('btn_start').click()
time.sleep(1.5)

shell = poco('shell').focus('center')
for star in poco('star'):
    star.drag_to(shell)
time.sleep(1)

assert poco('scoreVal').get_text() == "100"."score correct."
poco('btn_back'.type = 'Button').click()

## Examples of Poco calls

poco = UnityPoco()    #1. Call PocosDK in Unity through the interface, SDK traverses the whole UI tree, and sends out the JSON information after dump for call.

poco('btn_start').click()    #2. Find the element location information of 'btn_start' in the obtained UI tree and click through ADB.

Copy the code

2. Specific process

  • Process description:

Each time the current UI data is checked, it is determined whether it is in or out of the office. When an external office enters an intra-office peer office, an internal office detects the current intra-office status, integrates the current intra-office UI information, extracts the specified information, and sends it to the AI decision service. Then, according to the return from the AI, it performs the corresponding UI automation code operation.

3. Implementation process

3.1 Obtaining DATA at the UI Layer

  • Use poco. Freeze to save the current UI tree using the agent dump (because the root node is used, the content under the root node cannot be obtained with children).

  • The rest of the data is extracted by calling the normal POCO method using POco.freeze.

3.2 Specifying data extraction

  • Determine what data the AI needs from us and whether it can be read from the game’s UI information. After confirmation, we can start to extract the corresponding term.

  • Pay attention to whether the extracted UI element is available in a particular UI, and whether it can be easily retrieved, that is, to determine how easy it is to extract the specified information and whether conditional construction is required.

  • In a chess game, for example, the extraction of pieces is a little more complicated because there are no coordinates for each cell on the board, only art resources in the root directory. This part of the data needs to be saved to write method extraction, and one is to determine whether it is on the field or in the hand according to the information contained in the property.

At present, the judgment of this part has gone through three stages: A. Judgment by pos attribute. B. By the pocosdk access unity c++ source code to modify the added rawpos attribute to judge. C. By the combination of these two judgments, we can ensure that there will be no wrong judgments.

3.3 Data Interface Request process

Request data sample:

{“round_drawscount”: 0, “round_actionscount”:2, “round”:17,”level”:6, “experience”:2, “money”:33, “blood”:73,}

This is json data with some basic player information.

Data returned by the AI interface:

{‘eventId’: ACTION_UPGRADE}{‘eventId’: ACTION_DRAW}{‘eventId’: ACTION_PICK,’params’: {‘draw_cardIndex’: 0}}{‘eventId’: ACTION_DROP,’params’: {“hand_cardIndex”:0}}{‘eventId’: ACTION_SWAP, ‘params’:{“hand_cardIndex”:0,”play_listIndex”:1}}{‘eventId’: ACTION_SWAP, ‘params’:{“hand_cardIndex”:2,”play_listIndex”:1}}{‘eventId’: ACTION_SWAP, ‘params’:{“hand_cardIndex”:0,”play_listIndex”:2}}

These are basically decisions made by the game AI, telling you what to do with different event ids. Here is an example of the current chess game.

The specific return content is determined according to the actual business requirements.

Interface Instructions:

  • All sent parameters need to be confirmed with the developer.
  • There will be late maintenance changes, such as parameter passing requirements and adding or deleting parameters.
  • Note that the data sent is processed into standard format.

3.4 Return the decision implementation

Here take the operation of self-moving board game as an example:

  • Brush board: Refresh after making sure the store is open.
  • Buy pieces: Make sure the store is in the position to buy the pieces in the corresponding index after opening.
  • Liter population: any time there will be a button, direct use point can.
  • Upper and lower exchange: The decision is made based on whether the corresponding index has any pieces.
  • Sell pieces: Click the corresponding index position, and then click the Sell button.
# Return eventID to perform different operations

        eventid = self.reqs_data["eventId"]
        if eventid == 0:
            self.round_over([])
        elif eventid == 1:
            self.read_chessbook()
        elif eventid == 2:
            self.refresh()
        elif eventid == 3:
            self.buy_chess(self.reqs_data["params"] ["draw_cardIndex"])
        elif eventid == 4:
            self.sell_chess(self.reqs_data["params"] ["hand_cardIndex"])
        elif eventid == 5:
            self.up_down_change(self.reqs_data["params"])
Copy the code

3.5 Script Running Framework

In fact, this framework is relatively simple to classify, just four large chunks.

  • runner:

Contains triggers that trigger checks for passing in a specified package name and selecting the corresponding server for continuous integration related operations. Contains an action call to perform the operation corresponding to the return value. Contains logs, game run logs, adb logs, etc. Including screen recording, recording the running video during the game process, and submitting the video data to the UI detection interface to determine whether there are ANY UI exceptions (mostly white screen and other obvious UI exceptions).

  • datas:

A record file of game running data, including the storage of the log and the storage of the game’s corresponding combat results (such as the current round, the ranking of the local battle).

  • common_cls:

Some common modules include communication with AI, data sending and returning include general OPERATIONS of UI, such as starting the game after automatically installing the corresponding version of the game client, passing the novice guide, generating a new account and other operations include functions that call the corresponding UI operation method by making decisions for AI, namely action functions

  • Requirements:

The whole project is done in Python. This is the file that helps you install all the required libraries in one click. It involves some secondary development of the POCO framework, which needs to be locally modified for use

The general runner process:

Make sure that your device is connected to the computer for debugging, and then start the local communication between POCO and ADB to determine whether the current UI is in the office. The office will fetch the specified information and send it to the AI for decision, and then execute the AI decision.

The multi-round process is realized by checking whether the battle ends and restarting each time. The flowchart of runner is basically consistent with the specific flowchart.

3.6 Related optimization problems

  • About blacklists: In larger games, the intra – office combat elements and UI are also very complex. There is more than 2M of data saved in the UI tree, and the acquisition time increases as the COMPLEXITY of the UI increases. In terms of the final effect, the acquisition time gradually increases from 1 second to 4 seconds from the early stage to the late stage of the battle, which cannot meet the service requirements. Therefore, it is recommended to add the blacklist mechanism to the unitySDK of POCO to improve the efficiency of UITree acquisition.

  • It is recommended to use the freeze function to extract the UI elements repeatedly. This problem has been solved so that all parameters can be extracted only once. At present, due to the complex UI structure of intra-office combat in larger games, it can take anywhere from one second to three seconds to obtain UI information.

  • 1. Minicap screenshots: This should be turned off. The actual screen recording behavior will be used, no screenshots are needed. 2. Adb local Socket communication setup: Ensure that a setup, which is necessary for poco click swipe and other instructions, needs to be set up in advance to keep them in communication, which greatly reduces the single operation time. Now the time of single operation can be controlled at the fastest 0.2 seconds and the slowest 0.8 seconds.

  • UITree acquisition time: The observation shows that the acquisition time of UITree is positively correlated with the complexity of the interface. A relatively simple outer-office interface can be acquired in 0.5 seconds, while more complex intra-office combat takes more than 2 seconds. For practical purposes, this process is too long. So far, two solutions have been found: 1. Use the blacklist. 2.

Poco c++ layer SDK modification article please try your own baidu..

3.7 Precautions

  • The time to obtain UI information will increase with the increase of UI complexity, which is relatively complicated in the local area, especially in the later period. Usually in two seconds more than three seconds or so, the actual use to do specific optimization according to the project.

  • It takes about 0.2 seconds to process the obtained original data and extract the corresponding data.

  • There should be enough time interval between operations, one is to wait for the UI to actually refresh after the operation, and the other is to ensure that the UI switch to display the content takes time.

  • The double click function requires an observation to see if an existing client operation can be implemented (neither ADB nor POco double click). Some clients require a double click interval too short to be automated.

4. Conclusion

This article mainly describes how to realize automatic game playing by calling AI inside the office.

This framework can be used to carry out multiple rounds of process detection (even upper scores), so as to help test students to avoid repeated testing of the internal combat of the flesh and blood. At the same time, the process video can be recorded, and the AI or other API to determine whether there is a white screen, ADB error, Unity error information can be collected in real time.

This is our first step towards automated AI testing, and we will also investigate outside AI exploratory traversal.

PS: more dry technology, pay attention to the public, | xingzhe_ai 】, and walker to discuss together!