title: Create generator that shuffle training data for Keras in R/train a Keras model with lowspec computer
body: <p>I have a dataset similar to MNIST (200.000 rows of 784 pixels + 1 categorical output (785 columns)) and I want to train a MLP and a CNN using the <code>Keras</code> library in R (in RStudio). I'm currently using a computer with 32 GB of RAM with an intel i7-8700 @3.2 GHz and I have not any problem when training this neural networks using the <code>fit()</code> function from Keras (training time ~4 minutes). However, when I execute the same script in my laptop (8 GB of RAM with intel i5-6300 @2.3 GHz) it can not even do an epoch in least than 10 minutes. </p>
  
  <p>I work as a laboratory professor in university and I'm worried that my students could not run the script with that database in their laptops due to lack of computation power. My idea was to fit the same models using a generator with the <code>fit_generator()</code> function, and load part of the dataset instead of the whole dataset in each call to the generator function (in order to use less memory than loading the entire dataset and resulting in a faster training). However, this produce some unexpected results. The accuracy reached by the <code>fit()</code> function is ~98.8% in training (120.000 rows) and ~98.4% in test (80.000 rows), but using the <code>fit_generator()</code> function is ~1.05% in the same training and ~1.01% in the same test. I have found related issues  <a href="https://github.com/keras-team/keras/issues/2389" rel="nofollow noreferrer">here</a>, <a href="https://stackoverflow.com/questions/52600127/keras-fit-generator-and-fit-results-are-different">here</a>, <a href="https://stackoverflow.com/questions/44785924/generating-data-for-training-in-keras">here</a> and <a href="https://stackoverflow.com/questions/44569938/memory-issues-using-keras-convolutional-network/44593430#44593430">here</a>, and it seems that the problem is that <code>fit_generator()</code> does not shuffle the training data and this produce that the network is always trained with the same batches (e.g. same gradient when computing backpropagation) and does not represent well the entire dataset, resulting in that poor accuracy. I've trained the model using <code>fit()</code> but setting the <code>shuffle</code> argument to <code>FALSE</code>, and the accuracy falls to 0.1% so it confirms that shuffling the training data is crucial to train the model.</p>
  
  <p>My questions are:</p>
  
  <ul>
  <li>Is it a good idea to use a generator to avoid problems/reduce training time when using a computer with lower specifications or is there a better solution? </li>
  <li>I'm training the models using the entire dataset by setting <code>steps_per_epoch</code> argument equal to <code>ceil(nrow(train_dataset)/batch_size)</code>, so it should use the same data when using <code>fit()</code> and <code>fit_generator()</code> except from the "shuffling" part right?</li>
  <li>In case that using a generator that loads part of the dataset is a good solution to train the models with low-spec computers, how can i shuffle the training data in an effective way using a generator?</li>
  </ul>
  
  <p>All the generators i've seen takes the entire dataset and produce a batch of the samples in each call or does not shuffle the data. I have created a generator with the code below. It takes as arguments <code>datafile</code> the file with the data (training data or test data), <code>batch_size</code> the size of the batch to produce in each call, <code>mlp</code> in order to process the data to train an MLP or a CNN, <code>val</code> in order to begin to produce batches in different index for validation data, and <code>shuffle</code> to indicate if we want to shuffle the data or not. My idea to shuffle the data was to create a random index and read only one row of the file for each number in the index (using the <code>skip</code> and <code>nrow</code> arguments in <code>read.table()</code>). This is extremely unefficient due to the several calls to <code>read.table()</code>:</p>
  
  <pre><code>data_generator &lt;- function(datafile, batch_size = 128, mlp = TRUE, val = TRUE, shuffle = TRUE) {
    nrow_file &lt;- R.utils::countLines(datafile) - 1
    if (!val) {
      skip &lt;- 0 
    } else {
      skip &lt;- nrow_file / 2
    }
    function() {
      # Calculate the rows to read in this epoch
      rows_to_read &lt;- batch_size
      if (skip + batch_size &gt; nrow_file) {
        rows_to_read &lt;- nrow_file - skip
      }
  
      if (shuffle) {
        index &lt;- sample (c(1:nrow_file), size=batch_size, replace =F)
      } else {
        index &lt;- (skip + 1):(skip + rows_to_read)
      }
  
      # Load only the rows that we want to use in training
      trData &lt;- as.list(numeric(batch_size))
      for(i in index) {
        ii &lt;- i - 1 
        trData[[which(i == index)]] &lt;- read.table(datafile, sep = ";", header = TRUE,
                                skip = ii, nrows = 1)
      }
      trData &lt;- do.call("rbind",trData)
      # Upload the rows to train
      skip &lt;&lt;- skip + batch_size
      if (skip &gt;= nrow_file) {
        skip &lt;&lt;- 0
      }
      # Build inputs and output
      y_train &lt;- trData[,1]
      x_train &lt;- trData[,-1]
      if (mlp) {
        # Return data as is for mlp
        list(data.matrix(x_train), data.matrix(y_train))  
      } else {
        # Return data reshaped for CNN
        list(array_reshape(data.matrix(x_train), c(nrow(x_train), 28, 28, 1)),
             data.matrix(y_train))
      }
    }
  }
  </code></pre>
  
  <p>The code I've used to train the MLP model (analogous to CNN) is:</p>
  
  <p>Without generator</p>
  
  <pre><code>MLP_model &lt;- keras_model_sequential()
  
  MLP_model %&gt;% 
    layer_dense(units = 500, activation = 'relu', input_shape = c(784),
                kernel_regularizer = regularizer_l2(l = 0.0001),
                bias_regularizer = regularizer_l2(l = 0.0001)) %&gt;% 
    layer_dropout(rate = 0.4, seed = 150)  %&gt;% 
    layer_batch_normalization() %&gt;%
    layer_dense(units = 300, activation = 'relu',
                kernel_regularizer = regularizer_l2(l = 0.001),
                bias_regularizer = regularizer_l2(l = 0.001)) %&gt;%
    layer_dropout(rate = 0.3, seed = 150) %&gt;%
    layer_batch_normalization() %&gt;%
    layer_dense(units = 10, activation = 'softmax',
                kernel_regularizer = regularizer_l2(l = 0.001),
                bias_regularizer = regularizer_l2(l = 0.001))
  
  
  MLP_model %&gt;% compile(
    loss = loss_categorical_crossentropy,
    optimizer = optimizer_adam(),
    metrics = c('accuracy')
  )
  
  history &lt;- MLP_model %&gt;% fit(
    x_train_mlp, y_train, 
    epochs = 20, batch_size = 124, 
    validation_split = 0.2, 
    shuffle = TRUE
  )
  </code></pre>
  
  <p>With generator:</p>
  
  <pre><code>MLP_model &lt;- keras_model_sequential()
  
  MLP_model %&gt;% 
    layer_dense(units = 500, activation = 'relu', input_shape = c(784),
                kernel_regularizer = regularizer_l2(l = 0.0001),
                bias_regularizer = regularizer_l2(l = 0.0001)) %&gt;% 
    layer_dropout(rate = 0.4, seed = 150)  %&gt;% 
    layer_batch_normalization() %&gt;%
    layer_dense(units = 300, activation = 'relu',
                kernel_regularizer = regularizer_l2(l = 0.001),
                bias_regularizer = regularizer_l2(l = 0.001)) %&gt;%
    layer_dropout(rate = 0.3, seed = 150) %&gt;%
    layer_batch_normalization() %&gt;%
    layer_dense(units = 10, activation = 'softmax',
                kernel_regularizer = regularizer_l2(l = 0.001),
                bias_regularizer = regularizer_l2(l = 0.001))
  
  MLP_model %&gt;% compile(
    loss = loss_categorical_crossentropy,
    optimizer = optimizer_adam(),
    metrics = c('accuracy')
  )
  
  history &lt;- MLP_model %&gt;% fit_generator(
    data_generator(traindatafile,
                   batch_size = 128, mlp = TRUE, val = FALSE), 
    steps_per_epoch = round((R.utils::countLines(traindatafile)-1) / (128)),
    epochs = 10)
  </code></pre>
  
  <p>Thanks in advance!</p>