title: Dataset doesn't fit in memory for LSTM training
body: <p>I am trying to create model that is trained on large music dataset. The midi files are converted into numpy arrays. Since LSTM requires sequential data, the dataset size becomes so huge on converting into a sequence for the LSTM.</p>
  
  <p>I convert the midi notes into index based on the keynote and duration, so I get 6 classes for C4 key. Likewise I get C3 to B5 so totally 288 classes along with classes for rest periods.</p>
  
  <p>The converted format of a single midi looks like this.</p>
  
  <p><code>midi = [0,23,54,180,23,45,34,.....];</code></p>
  
  <p>For training the LSTM, the x and y becomes</p>
  
  <p><code>x = [[0,23,54..45],[23,54,..,34],...];</code></p>
  
  <p><code>y=[[34],[76],...]</code></p>
  
  <p>The values in x and y are further transformed into one-hot encodings.
  Hence the size of the data becomes huge for just 60 small mid files, but I have 1700 files. How can I train the model with this amount of files. I checked ImageGenerator but it requires data to be in separate class directories. How to achieve this?</p>