<p>From <a href="https://stats.stackexchange.com/q/164876/12359">Tradeoff batch size vs. number of iterations to train a neural network</a>:</p>

<p>From Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, Ping Tak Peter Tang. On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima. <a href="https://arxiv.org/abs/1609.04836" rel="nofollow noreferrer">https://arxiv.org/abs/1609.04836</a> :</p>

<blockquote>
  <p>The stochastic gradient descent method and its variants are algorithms of choice for many Deep Learning tasks. These methods operate in a small-batch regime wherein a fraction of the training data, usually 32--512 data points, is sampled to compute an approximation to the gradient. <strong>It has been observed in practice that when using a larger batch there is a significant degradation in the quality of the model, as measured by its ability to generalize.</strong> There have been some attempts to investigate the cause for this generalization drop in the large-batch regime, however the precise answer for this phenomenon is, hitherto unknown. In this paper, we present ample numerical evidence that supports the view that large-batch methods tend to converge to sharp minimizers of the training and testing functions -- and that sharp minima lead to poorer generalization. In contrast, small-batch methods consistently converge to flat minimizers, and our experiments support a commonly held view that this is due to the inherent noise in the gradient estimation. We also discuss several empirical strategies that help large-batch methods eliminate the generalization gap and conclude with a set of future research ideas and open questions.</p>
  
  <p>[…]</p>
  
  <p><strong>The lack of generalization ability is due to the fact that large-batch methods tend to converge to <em>sharp minimizers</em> of the training function</strong>.  These minimizers are characterized by large positive eigenvalues in $\nabla^2 f(x)$  and tend to generalize less well. In contrast, small-batch methods converge to flat minimizers characterized by small positive eigenvalues of $\nabla^2 f(x)$. We have observed that the loss function landscape of deep neural networks is such that large-batch methods are almost invariably attracted to regions with sharp minima and that, unlike small batch methods, are unable to escape basins of these minimizers.</p>
  
  <p>[…]</p>
  
  <p><a href="https://i.stack.imgur.com/30I6a.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/30I6a.png" alt="enter image description here"></a></p>
</blockquote>

<p>Also, some good insights from <a href="https://www.quora.com/profile/Ian-Goodfellow" rel="nofollow noreferrer">Ian Goodfellow</a>
answering to <a href="https://www.quora.com/Deep-learning-why-do-not-use-the-whole-training-set-to-compute-the-gradient" rel="nofollow noreferrer">why do not use the whole training set to compute the gradient?
</a> on Quora:</p>

<blockquote>
  <p>The size of the learning rate is limited mostly by factors like how
  curved the cost function is. You can think of gradient descent as
  making a linear approximation to the cost function, then moving
  downhill along that approximate cost. If the cost function is highly
  non-linear (highly curved) then the approximation will not be very
  good for very far, so only small step sizes are safe. You can read
  more about this in Chapter 4 of the deep learning textbook, on
  numerical computation:
  <a href="http://www.deeplearningbook.org/contents/numerical.html" rel="nofollow noreferrer">http://www.deeplearningbook.org/contents/numerical.html</a></p>
  
  <p>When you put
  m examples in a minibatch, you need to do O(m) computation and use
  O(m) memory, but you reduce the amount of uncertainty in the gradient
  by a factor of only O(sqrt(m)). In other words, there are diminishing
  marginal returns to putting more examples in the minibatch. You can
  read more about this in Chapter 8 of the deep learning textbook, on
  optimization algorithms for deep learning:
  <a href="http://www.deeplearningbook.org/contents/optimization.html" rel="nofollow noreferrer">http://www.deeplearningbook.org/contents/optimization.html</a></p>
  
  <p>Also, if
  you think about it, even using the entire training set doesn’t really
  give you the true gradient. The true gradient would be the expected
  gradient with the expectation taken over all possible examples,
  weighted by the data generating distribution. Using the entire
  training set is just using a very large minibatch size, where the size
  of your minibatch is limited by the amount you spend on data
  collection, rather than the amount you spend on computation.</p>
</blockquote>
Selection of Mini-batch Size for Neural Network Regression
<p>I am doing a neural network regression with 4 features. How do I determine the size of mini-batch for my problem? I see people use 100 ~ 1000 batch size for computer vision with 32*32*3 features for each image, does that mean I should use batch size of 1 million? I have billions of data and tens of GB of memory so there is no hard requirement for me not to do that.</p>

<p>I also observed using a mini-batch with size ~ 1000 makes the convergence much faster than batch size of 1 million. I thought it should be the other way around, since the gradient calculated with a larger batch size is most representative of the gradient of the whole sample? Why does using mini-batch make the convergence faster?</p>