This article was originally published in: Walker AI
During game project development, it is necessary to build an automated testing platform, in order to expect field combat to use automation to test, find intra-office bugs, avoid repetitive labor, improve testing efficiency and avoid human operation errors. Environment requirements Projects need to use Airtest and POCO to connect to the server of reinforcement learning, so that Airtest can send the status information to the server, and the server can return the decision for the next step.
1. Preparatory work
Understand the decision-making methods of Airtest, POCO and reinforcement learning Agent.
1.1 introduce Airtest
Airtest is an automated testing framework based on image recognition. The core of this framework is not the implementation method and technology, but the concept! The idea behind this framework was borrowed from Sikuli at MIT, who conceived a new UI testing mode based on image recognition controls rather than control objects in specific memory.
(1) Features of Airtest
- Support for programmable testing tools based on image recognition
- cross-platform
- Generate test report
- Support poCO and other SDK embedded, improve the ACCURACY of UI recognition
(2) Airtest interface (including click, slide, judge, screenshot and other interfaces)
(3) Airtest effect demonstration
The above shows Airtest interacting with the interface directly through image recognition. If you need to interact with specified elements on the interface, you need the methods provided by the Poco to operate on the elements on the interface
(4) Limitations of Airtest
However, in the actual project, the image will not remain unchanged. We need to capture the dynamic node of the project and click and move the dynamic node (such as the location node of store-bought flags).
We need another tool, Poco.
1.2 introduce Poco
Currently, Poco only supports native Android and ios interface invocation, and other platforms need to access the SDK of the corresponding platform
(1) The way Poco obtains THE UI tree
The child nodes are traversed from the root node down
In unity projects, you need to install the SDK for Poco in Unity
(2) Poco calls methods
(3) Examples of Poco invocation
poco = UnityPoco()
poco('btn_start').click()
Copy the codeAirtest calls PoCOSDK in Unity through the interface. The SDK traverses the entire UI tree and sends the DUMPED JSON information back to Airtest.
Airtest finds the element location information for ‘btn_start’ in the resulting UI tree and clicks through ADB.
1.3 Brief introduction to reinforcement learning
Environment is usually described by Markov process. Agent takes some Policy to generate Action and interacts with Environment to generate a Reward. Then Agent will adjust and optimize the current Policy according to Reward.
In the above illustration, state is a state of the environment, and Observation is the state of the environment observed by the Agent, where observation and state are the same. First of all, the Agent observes a state of the environment, such as a glass of water, and then takes an action that the Agent breaks the water in the cup. In this way, the state of the environment has changed. Then, the system gives a score to this behavior to tell the Agent whether such behavior is correct. Then the Agent takes further actions according to the newly changed environment state. The Agent’s goal is to make the Reward as big as possible.
2. Project execution Process:
2.1 background
One Agent is deployed on the server and others can access the server. The process is as follows:
A. The test end collects information -> the test end converts the information to the agreed state format -> the test end sends the state to the server -> The server returns an Agent’s decision -> The test end executes the decision after receiving the information ->
B. The test side collects information (the beginning of a new cycle)…
In this process, the test side took the most time to collect information, and decided to optimize part of it according to the project requirements.
2.2 Specific Problems
Poco’s first call to dump started a large number of mincap and other executables, resulting in a delay of about 7 seconds.
The Airtest operation encountered too much latency, resulting in only 4-5 actions per 30-second turn.
However, locally trained agents can have about 16 operands in each turn in the late stage, which results in agent actions that cannot be completely completed on the client.
2.3 Solutions
Preload the CLICK event for the POCO
It takes a long time to locate the dump. Therefore, you need to reduce the size of the JSON file dumped from the SDK interface.
A. Add tagfilter, blacklist, and propertyList parameters to unity interfaces to control the size of JSON files.
Where, TagFilter is used to filter unityGameObject for the specified tag, and can remove all objects except UI and Default.
A blacklist is used to filter the names of unityGameObject and improves the dump efficiency by 50%
Propertylist is used to reduce the number of parameters written to a single UnityGameObject. By default, a single object has more than 10 parameters. After filtering, about 6 parameters can be saved. The dump efficiency can be improved by 33%
B. Use the corresponding interface parameters in python interfaces
This method perfectly solves the problem of operation delay. At present, the client can complete about 20 actions in 30 seconds in a single turn.
2.4 Specific steps:
Layerfilter in this project, there are 13 layers. Only the UGO whose tag is UI and Default is recursively written to the child node information, eliminating scenes, special effects and other layers, which can greatly reduce the overhead.
Namefilter Not all UI node information is necessary for automated testing. Therefore, in recursive query of sub-nodes, the time cost of UGO written into the blacklist can be reduced by about 50%.
Main modification of dumpHierarchyImpl interface in C# poco AbstractDumper, as shown in the figure below:
private Dictionary<string.object> dumpHierarchyImpl (AbstractNode node, bool onlyVisibleNode, Dictionary<string.object> extrapar)
{
if (node == null)
{
return null;
}
Dictionary<string.object> payload = new Dictionary<string.object> ();if(extrapar ! =null && extrapar.ContainsKey("param4") && extrapar["param4"] != null)
{
payload = node.enumerateAttrs(extrapar["param4"].ToString());
}
else
{
payload = node.enumerateAttrs(null);
}
Dictionary<string.object> result = new Dictionary<string.object> ();
string name = (string)node.getAttr ("name");
result.Add ("name", name);
result.Add ("payload", payload);
List<object> children = new List<object> ();if(extrapar! =null)
{
if (extrapar.ContainsKey("param3") && extrapar["param3"] != null)
{
requirelayer = extrapar["param3"].ToString().Split('|').ToList();
string layer = (string)node.getAttr("layer");
if(! requirelayer.Contains(layer)) {//Debug.LogError("--dumpHierarchyImpl layer is not contains");
returnresult; }}if (extrapar.ContainsKey("param2") && extrapar["param2"] != null)
{
try
{
filterlist.Clear();
string str = extrapar["param2"].ToString();
filterlist = str.Split('|').ToList();
}
catch
{
Debug.LogError("~~~dumpHierarchy Implextrapar param2 error");
}
if (filterlist.Contains(name))
{
returnresult; }}}foreach (AbstractNode child in node.getChildren())
{
if(! onlyVisibleNode || (bool)child.getAttr ("visible")) { children.Add (dumpHierarchyImpl (child, onlyVisibleNode, extrapar)); }}if (children.Count > 0)
{
result.Add ("children", children);
}
return result;
}
Copy the codePropertyfilter JSON The default node parameters are:
Parameters such as name, payload, type, visible, pos, size, scale, anchorPoint, zOrders, Clickable, components, _ilayer, layer, _instanceId. We eliminated it
The visible | scale | anchorPoint | clickable | components | _ilayer | layer | _instanceId parameters not to use it, actually significantly reduce the file size, to dump out the json can reduce about 33% of the time overhead.
Modify the enumerateAttrs and GetPayload interfaces of UnityNode in C# poco as follows:
private Dictionary<string.object> GetPayload(string blackList)
{
Dictionary<string.object> all = new Dictionary<string.object>() {
{ "name", gameObject.name },
{ "type", GuessObjectTypeFromComponentNames (components) },
{ "visible", GameObjectVisible (renderer, components) },
{ "pos", GameObjectPosInScreen (objectPos, renderer, rectTransform, rect) },
{ "rawpos", GameObjectVec3Pos (objectRawPos) },
{ "rawrectpos", GameObjectVec3Pos (objectRectRawPos) },
{ "size", GameObjectSizeInScreen (rect, rectTransform) },
{ "scale".new List<float> () {1.0 f.1.0 f}}, {"anchorPoint", GameObjectAnchorInScreen (renderer, rect, objectPos) },
{ "zOrders", GameObjectzOrders () },
{ "clickable", GameObjectClickable (components) },
{ "text", GameObjectText () },
{ "components", components },
{ "texture", GetImageSourceTexture () },
{ "tag", GameObjectTag () },
{ "_ilayer", GameObjectLayer() },
{ "layer", GameObjectLayerName() },
{ "_instanceId", gameObject.GetInstanceID () },
};
Dictionary<string.object> payload = new Dictionary<string.object> ();if (!string.IsNullOrEmpty(blackList))
{
List<string> black_list = blackList.Split('|').ToList();
foreach(KeyValuePair<string.object> it in all)
{
if(black_list.Contains(it.Key))
{
continue; } payload.Add(it.Key,it.Value); }}else
{
payload = all;
}
return payload;
}
Copy the codePS: more dry technology, pay attention to the public, | xingzhe_ai 】, and walker to discuss together!