<p>An interesting reading about such questions is Bengio's 2012 paper <a href="https://arxiv.org/pdf/1206.5533.pdf" rel="nofollow noreferrer">Practical Recommendations for Gradient-Based Training of Deep
Architectures</a></p>

<p>There is a section about online learning where the distribution of training data is unknown. I quote from the original paper</p>

<blockquote>
  <p>It
  means that online learners, when given a stream of
  non-repetitive training data, really optimize (maybe
  not in the optimal way, i.e., using a first-order gradient
  technique) what we really care about: generalization
  error.</p>
</blockquote>

<p>The best practice though to figure out how your dataset behaves under different testing scenarios would be to try them both and get experimental results of how the distribution of the training data affects your generalization error.  </p>
Must each tensorflow batch contain a uniform distribution of the inputs for all expected classifications?
<p>This is probably a newbie question but I'm trying to get my head around how training on small batches works.</p>

<p><strong>Scenario -</strong> </p>

<p>For the mnist classification problem, let's say that we have a model with appropriate hyerparameters that allow training on 0-9 digits. If we feed it with a small batches of uniform distribution of inputs (that have more or less same numbers of all digits in each batch), it'll learn to classify as expected.</p>

<p>Now, imagine that instead of a uniform distribution, we trained the model on images containing only 1s so that the weights are adjusted until it works perfectly for 1s. And then we start training on images that contain only 2s. Note that only the inputs have changed, the model and everything else has stayed the same.</p>

<p><strong>Question</strong> -</p>

<p>What does the training exclusively on 2s after the model was already trained exclusively on 1s do? Will it keep adjusting the weights till it has forgotten (so to say) all about 1s and is now classifying on 2s? Or will it still adjust the weights in a way that it remembers both 1s and 2s?</p>

<p>In other words, must each batch contain a uniform distribution of different classifications? Does retraining a trained model in Tensorflow overwrite previous trainings? If yes, if it is not possible to create small (&lt; 256) batches that are sufficiently uniform, does it make sense to train on very large (>= 500-2000) batch sizes?</p>