Figure 1: Simulated robot training using genetic algorithm (left) and evolutionary strategy (right).

Use the VINE

To utilize VINE, behavioral characteristics (BC) of each parent and all pseudooffspring were recorded during evaluation. Here, the BC can be any indicator of the agent’s behavior when interacting with its environment. For example, in Mujoco, we simply use the final position of the agent {x,y} as BC, so it indicates where the agent has been moved from the origin.

Visualization tools map parents and pseudo-descendants to 2D planes based on their BCS. To do this, it invokes a graphical user interface (GUI), whose main components consist of two interrelated graphics: one or more pseudo-child cloud maps (on different 2D planes) and a fitness map. As shown in Figure 2, the pseudoprogeny cloud map below shows the BC of each generation of parents and pseudoprogeny in the cloud, while the fitness graph, a key indicator of generational progress, shows the fitness score curve of parents.

Figure 2: examples of pseudodaughter cloud and fitness maps.

User interaction and the figure then, explore pseudo offspring cloud overall trend as well as any parent or pseudo offspring individual behavior during the process of evolution: (1) the user can visualize any generation of parents, the best cloud and (or) the pseudo posterity, and explore the different fitness pseudo offspring of BC in 2 d plane and spatial distribution; (2) Users can make generational comparisons and browse through generations to visualize how the parent and/or pseudoprogenies cloud moves on the 2D BC plane and how these moves relate to fitness scoring curves (as shown in Figure 3, the full movie clip of the mobile cloud can be automatically generated); (3) Click any point on the cloud map, and the behavior information and fitness score of the corresponding pseudo-offspring will be displayed.

Figure 3: Visual evolution of generational behavior. The color changes with each generation. Over the course of one generation, the color intensity of each pseudoprogeny was calculated based on the percentile of its fitness score within that generation (total five sub-boxes).

Additional use cases

The tool also supports advanced options and custom visualizations beyond the default. For example, as a replacement for the final single point {x, y}, BC can be replaced with the complete trace of each agent (for example, {x, y} connected by 1000 time point steps) instead of a single final {x, y} point. In this case, when the dimension of BC is greater than 2, reductive techniques (such as PCA or T-SNE) are used to reduce the dimension of BC data to 2D. Our tools automate these processes.

The GUI has the ability to load multiple sets of 2D BCS (possibly generated by different reduction techniques) and display them in a simultaneously connected cloud map, as shown in Figure 4. This feature provides a convenient way for users to explore different BC options and dimensionality reduction methods. In addition, users can extend the underlying visualization through customization capabilities. Figure 4 shows one such custom cloud map that shows some types of domain-specific high-dimensional BCS (in this case, the full trajectory of the broker) and the corresponding reduced 2D BCS. Another example of a custom cloud diagram in Figure 5 allows the user to replay the deterministic and random behavior of the agent when interacting with the environment.

Figure 4: Visualization of multiple 2D BCS and a high dimensional BC with a fitness graph.

Figure 5: VINE allows users to view videos of deterministic and random behavior of any agent.

The tool is also designed to handle domains outside of moving tasks. Figure 6 illustrates a cloud map that visualizes training agents to play Frostbit, one of the Atari 2600 games, using the final simulator RAM state (a vector of integer values of length 128 that captures all state variables in the game) as the BC. PCA is applied to map BC to 2D plane.

Figure 6: Visual proxy learning to demonstrate Frostbite.

In the image, we can observe that as evolution develops, the pseudoprogeny cloud moves to the left and toward the nebula there. By seeing the corresponding video of each agent playing the game, we can infer that each cluster corresponds to a semantically meaningful different end state.

VINE can also work seamlessly with other neuroevolutionary algorithms, such as GA, that allow generations to reproduce. In fact, the tool is independent of any particular neuroevolutionary algorithm. Users only need to modify their neuroevolutionary code slightly to save the BC they chose for a particular problem. In the code release, we provided an example of implementing such changes to ES and GA.

The next step

Because the evolutionary approach operates on a set of points, it provides opportunities for new types of visualization. After implementing a tool that provides visualization, we found it useful and wanted to share it with the machine learning community for the benefit of all. As neural evolution expands into neural networks with millions or more connections, gaining more insight through tools like VINE is becoming increasingly valuable and important for deepening progress.

You can find VINE here. It is lightweight, shiftable, and implemented in Python.

Thanks: We thank Uber AI Lab, especially Joel Lehman, Xingwen Zhang, Felipe Petroski Such and Vashisht Madhavan for their valuable advice and helpful discussion.

Figure 1, left source: For example, Felipe Petroski.

The Nuggets Translation Project is a community that translates quality Internet technical articles from English sharing articles on nuggets. The content covers Android, iOS, front-end, back-end, blockchain, products, design, artificial intelligence and other fields. If you want to see more high-quality translation, please continue to pay attention to the Translation plan of Digging Gold, the official Weibo, Zhihu column.