Dance Dance Revolution is a rhythm music game that has taken the world by storm. You’ve all played it before, or seen it in a arcade at the mall, and now it’s available to artificial intelligence!

According to Lei Feng, the rules of the game are that players adjust their steps according to the footstep indicator pattern on the dance stage and step on the designated stage button at a specific time according to the rhythm of the music. The final score depends on whether the right button is pressed at the right time. Different difficulty games have different step patterns, and the higher the difficulty, the more steps, and in a more complex order. Jump stage has four arrow buttons, each button has four states: on, off, hold, release. The four arrow keys can be independently activated or released, so there are 256 possible combinations of footsteps.

Step instructions on the screen are made according to the musical structure, with different musical rhythms or melodies corresponding to different steps. Despite the popularity of DDR, there were some complaints: the library was limited, players were sometimes unable to dance to their favorite music, or even if the music was available, players didn’t like the set dance steps. To solve this problem, DDR has opened source tools that allow players to choreograph their own music. But actually, the process of doing your own choreography is very painful and requires a lot of professional knowledge. The following is an example of the choreography process.

Researchers at the University of California, San Diego, have used a method to automatically generate steps, which can be used directly from raw audio to learn choreography. This method is learned by mimicking the semantic information of human-made step indicator patterns. This problem can be broken down into two sub-tasks:

  1. Step placement, placing a series of time stamps in the song where steps should be placed. This process produces different results depending on the difficulty the player chooses.

  2. Step selection, select which step to place on the timestamp (up, down, left and right).

Running these two steps sequentially produces a playable step indicator pattern. The process is shown below.

For Knife Party feat four seconds of the dance choreography process, from top to bottom are artificial choreographed steps, step selection prediction, step placement prediction, audio features

The main process of learning choreography is music Information Retrieval (MIR). The placement of steps is similar to onset detection. Onset Detection aims to identify the timing of musically prominent events, such as notes or drumbeats. In the task of choreography, not all onset in all music corresponds to the steps of DDR, but the steps of DDR must correspond to one onset.

In terms of data, the researchers used DDR’s labeled data: Stepmania Online, which is used for more than 100K songs, totaling 350Gb of data. In addition, the researchers collected and collated two high-quality data sets created by other authors.

In the step placement stage, the convolutional neural network is used. The whole model consists of two convolution layers and two full connection layers. In order to improve the performance of CNN, the researcher added c-LSTM model and combined convolutional coding with RNN to achieve the purpose of integrating longer window information. The network structure is shown in the figure below:

In the step selection stage, the researcher regarded step selection as a sequence generation problem. They used the RNN model, which is commonly used in processing language models, because RNN is better at capturing long-distance context. The network structure is shown in the figure below:


This paper provides a Demo for those interested in trying it out for themselves: deepx.ucsd.edu/ddc

You can also watch the official demo video on Youtube: Youtu.be /yUc3O237p9M

Please refer to the original paper: Dance Dance Convolution for more implementation details

Lei Feng net (public account: Lei Feng net)(public account: Lei Feng net) compiled

Lei Feng net copyright article, unauthorized reprint prohibited. See instructions for details.