title: How to create training and test DataSetIterators in deeplearning4j?
body: <p>I am building a recurrent neural network with deeplearning4j and I need to create the training and test data sets.</p>
  
  <p>All the examples provided in <a href="https://deeplearning4j.org/docs/latest/deeplearning4j-nn-recurrent" rel="nofollow noreferrer">the documentation</a> and the <a href="https://github.com/deeplearning4j/dl4j-examples/blob/master/dl4j-examples/src/main/java/org/deeplearning4j/examples/recurrent/seqclassification/UCISequenceClassificationExample.java" rel="nofollow noreferrer">example code</a>, use a <code>CSVSequenceRecordReader</code> to read CSV files.</p>
  
  <p>Then a <code>DataSetIterator</code> is created with the <code>SequenceRecordReaderDataSetIterator</code> constructor and fed into the <code>MultiLayerNetwork.fit()</code> or the <code>MultiLayerNetwork.evaluate()</code> method (depending if it's a training or test data set iterator).</p>
  
  <p>However, in my case, the data set I have is not stored in a CSV file. I access it online through a third-party library, pre-process it to obtain a <code>List&lt;Data&gt;</code> and a <code>List&lt;Labels&gt;</code> objects.</p>
  
  <p>How can I:</p>
  
  <p>1) create the <code>DataSetIterator</code> from my two lists?</p>
  
  <p>2) split the <code>DataSetIterator</code> in a training set and a test set?</p>
  
  <p><strong>Edit</strong>:</p>
  
  <p>I think my question is too broad. Let me try to narrow it down.</p>
  
  <p>I have started to read <a href="https://progur.com/2017/06/how-to-create-lstm-rnn-deeplearning4j.html" rel="nofollow noreferrer">this article</a> which uses a very simple approach to create a data set:</p>
  
  <p>It creates two INDArrays and builds a DataSet from them using the <code>DataSet(INDArray first, INDArray second)</code> constructor.</p>
  
  <p>Training the data works using <code>network.fit(dataSet);</code>, but I can't evaluate it while training, as the method <code>evaluate</code> requires an data set iterator, not a data set.</p>
  
  <p>Moreover, from what I understand, using this approach also means that there is only one huge data set, no mini batches.</p>
  
  <p>I also guess that I could create mini batches from this big data set by using the   <code>batchBy(int num)</code> method. But this method returns a list of data sets, and not an data set iterator... iterateWithMiniBatches() does return a data set iterator but when I looked at the source file, it returns null and is deprecated. Then I tried to see if there is an implementation of the DataSetIterator I could use, but there are a lot of them. I tried the BaseDataSetIterator but it does not take a DataSet as constructor parameter but a DataSetFetcher... Yet another layer.</p>
  
  <p>Is there somewhere an example that shows how to create a data set without using the default record readers? Or should I just create my how implementation of a record reader?</p>