<p>Here is a paper about a very similar AI:
<a href="https://arxiv.org/pdf/1703.06891.pdf" rel="nofollow noreferrer">https://arxiv.org/pdf/1703.06891.pdf</a></p>
<p>They use <strong>two</strong> Neural Networks, the first one extracts audio features like Drums and decides when to place blocks <strong>(timing of blocks)</strong> and the other one decides how the blocks are arranged <strong>(direction of blocks)</strong>.</p>
Making an AI model that converts songs into a game level?
<p>I want to create an AI that coverts songs into beatsaber levels (It's a VR game). A beatsaber level can be stored as a array of &quot;blocks&quot; that looks like this:</p>
<pre><code>{
  &quot;time&quot;: 1.25,
  &quot;direction&quot;: &quot;up&quot;,
  &quot;hand&quot;: &quot;right&quot;,
  &quot;pos_x&quot;: 0,
  &quot;pos_y&quot;: 1
}
</code></pre>
<p>So the AI should convert songs into a level that has a lot of these blocks like the above. So far I have just done AIs that solve classification problems. But this problem has a generative nature. Does anyone know how to approch this kind of problem. Maybe with links that lead me in the right direction.</p>
<p>I have enough training data of already existing levels and songs. So that is no problem. I just need to know how the architecture of an AI like this looks like and how it should work.</p>
<p>Also I would like to make this with tensorflow if its possible. But other technologies are also fine.</p>
<p>Here is a link that should give you a better understanding of the game. The above JSON example resembles one block in the level. So the AI should generate an array of these blocks, and therefore generating a whole level.
<a href="https://www.youtube.com/watch?v=7JVXv2ySToU" rel="nofollow noreferrer">https://www.youtube.com/watch?v=7JVXv2ySToU</a></p>