title: Streaming large training and test files into Tensorflow's DNNClassifier
body: <p>I have a huge training CSV file (709M) and a large testing CSV file (125M) that I want to send into a <code>DNNClassifier</code> in the context of using the high-level Tensorflow API.</p>
  
  <p>It appears that the <code>input_fn</code> param accepted by <code>fit</code> and <code>evaluate</code> must hold all feature and label data in memory, but I currently would like to run this on my local machine, and thus expect it to run out of memory rather quickly if I read these files into memory and then process them.</p>
  
  <p>I skimmed the doc on <a href="https://www.tensorflow.org/programmers_guide/reading_data" rel="noreferrer">streamed-reading of data</a>, but the sample code for reading CSVs appears to be for the low-level Tensorflow API.  </p>
  
  <p>And - if you'll forgive a bit of whining - it seems overly-complex for the trivial use case of sending well-prepared files of training and test data into an <code>Estimator</code> ... although, perhaps that level of complexity is actually required for training and testing large volumes of data in Tensorflow?</p>
  
  <p>In any case, I'd really appreciate an example of using that approach with the high-level API, if it's even possible, which I'm beginning to doubt.</p>
  
  <p>After poking around, I did manage to find <a href="https://www.tensorflow.org/versions/master/api_docs/python/tf/contrib/learn/DNNClassifier#partial_fit" rel="noreferrer"><code>DNNClassifier#partial_fit</code></a>, and will attempt to use it for training.  </p>
  
  <p>Examples of how to use this method would save me some time, though hopefully I'll stumble into the correct usage in the next few hours.</p>
  
  <p>However, there doesn't seem to be a corresponding <code>DNNClassifier#partial_evaluate</code> ... though I suspect that I could break-up the testing data into smaller pieces and run <code>DNNClassifier#evaluate</code> successively on each batch, which might actually be a great way to do it since I could segment the testing data into cohorts, and thereby obtain per-cohort accuracy.</p>
  
  <p>==== Update ====</p>
  
  <p>Short version: </p>
  
  <ol>
  <li><p>DomJack's recommendation should be the accepted answer.</p></li>
  <li><p>However, my Mac's 16GB of RAM enough for it to hold the entire 709Mb training data set in memory without crashing. So, while I will use the DataSets feature when I eventually deploy the app, I'm not using it yet for local dev work.</p></li>
  </ol>
  
  <p>Longer version:</p>
  
  <p>I started by using the <code>partial_fit</code> API as described above, but upon every use it emitted a warning.  </p>
  
  <p>So, I went to look at the source for the method <a href="https://github.com/tensorflow/tensorflow/blob/master/tensorflow/contrib/learn/python/learn/estimators/estimator.py" rel="noreferrer">here</a>, and discovered that its complete implementation looks like this:</p>
  
  <pre><code>logging.warning('The current implementation of partial_fit is not optimized'
                  ' for use in a loop. Consider using fit() instead.')
  return self.fit(x=x, y=y, input_fn=input_fn, steps=steps,
                  batch_size=batch_size, monitors=monitors)
  </code></pre>
  
  <p>... which reminds me of this scene from Hitchhiker's Guide: </p>
  
  <p>Arthur Dent: What happens if I press this button?</p>
  
  <p>Ford Prefect: I wouldn't-</p>
  
  <p>Arthur Dent: Oh.</p>
  
  <p>Ford Prefect: What happened?</p>
  
  <p>Arthur Dent: A sign lit up, saying 'Please do not press this button again'.</p>
  
  <p>Which is to say: <code>partial_fit</code> seems to exist for the sole purpose of telling you not to use it.</p>
  
  <p>Furthermore, the model generated by using <code>partial_fit</code> iteratively on training file chunks was much smaller than the one generated by using <code>fit</code> on the whole training file, which strongly suggests that only the last <code>partial_fit</code> training chunk actually "took".</p>