Dance Dance Revolution, the music video game series, is massively popular. Over its nearly 20-year history, hundreds of songs have officially been included in the game as step charts — the sequences of instructions about when and where to put your feet on the dance platform.
But there’s also an open sourced DDR clone called StepMania, which allows fans to make and share their own step charts. Choreographing these charts can be tedious and slow, since it involves manually placing steps throughout a song, one at a time.
That’s why three researchers at the University of California, San Diego, are building a neural network that automatically makes DDR-style step charts from raw audio recordings. (Neural networks are computer systems that can learn certain tasks through exposure to stimuli, sort of like the human brain.) In a paper released Wednesday, the research team describes why its approach, called “Dance Dance Convolution,” is a faster and more flexible way to make new step charts.
The project began when Chris Donahue, longtime DDR player and machine learning researcher, was trying to solve one of the big problems in the field of music information retrieval (essentially, the extraction of data about music from recordings). It’s notoriously difficult to obtain large samples of music with metadata about beat and tempo. But it occurred to him that StepMania packs already contain tons of annotated musical data.
“It sort of dawned on me one day that I had somewhere buried deep in my hard drive, gigabytes and gigabytes of data from this game StepMania, from a folder I’d been transferring from computer to computer since I was a teenager. And I just had this idea to use that for tempo detection. I was playing with that for a while, and eventually it occurred to me that a much more interesting and novel problem might be to just try to recreate the game, rather than simply use the data as just sort of a convenient source.”
And that’s essentially what Dance Dance Convolution does. First, it determines step placement — points in time at which there should be a step. Second, step selection determines which steps (left, right, up, down, etc.) to place. The step placement part is especially interesting; the system skips the intermediate step of recognizing tempo or beat structure. Instead, it samples every 10 milliseconds or so to determine what moments should correspond to steps.
The system was trained on two sets of real-world data. The first and most useful is a set of 90 tracks choreographed by a prolific author who goes by the pseudonym Fraxtil. Fraxtil choreographed each track at all five difficulty levels, so there’s a lot of consistent data in that pool. The second source is a collection called In the Groove, and it contains 133 songs that are choreographed for all but the highest level by a variety of authors.
The results are encouraging, and the generated charts are playable. The team has released a video comparing one of the Fraxtil charts to a generated chart of the same track. They differ, for sure, but you can see that Donahue, who was dancing to each chart, has a similar success rate at performing each one. This demonstrates that the network is producing charts that are about as playable as human-choreographed ones.
This also demonstrates a key part of the neural network’s advantage: It can produce many different charts for the same song. One of the researchers, Zachary Lipton, explained that the concept of a “ground truth” is a little bit strange in the case of choreography. He compared it to language translation: “If I give you a passage of text in one language, and you had 10 high-quality translators, they’d come up with 10 different translations. On the other hand, there is some common structure in them.” Likewise, the Dance Dance Convolution system will produce a variety of charts for a given track, but those charts will tend to have some similar characteristics.
This could be great news for people who like dance step games. The neural network works best (for now) at the higher skill levels, where there are more steps in a chart, and therefore more data to train the system on. For experienced players who may get bored playing to the same tracks, the Dance Dance Convolution system could be a new way to change things up, and get new and varied choreography for a given song. Players could also theoretically feed the system any song from their own music collections and be able to play it in StepMania.
Right now, the team doesn’t have any plans to make a commercial run at the product. “StepMania is an open source game,” UCSD assistant professor Julian McAuley said. “We’re benefiting from the fact that it’s open source and that it’s possible for us to get so much high-quality data. We plan to contribute back in the sense that we want to make this a usable tool, and we already have people downloading charts that we’ve generated. But I don’t think we’re planning to make money from it. The scientific goals are enough for us.”
If you’d like to try Dance Dance Convolution for yourself, the team has made a demo of the tool available on its website.
