<p>The <a href="https://twitter.com/ylecun/status/989610208497360896" rel="nofollow noreferrer">2018 opinion retweeted by Yann LeCun</a> is the paper <a href="https://arxiv.org/abs/1804.07612" rel="nofollow noreferrer">Revisiting Small Batch Training For Deep Neural Networks, Dominic Masters and Carlo Luschi</a> suggesting a good generic maximum batch size is:</p>
<blockquote>
<p>32</p>
</blockquote>
<p>With some interplay with choice of learning rate.</p>
<p>The earlier 2016 paper <a href="https://arxiv.org/abs/1609.04836" rel="nofollow noreferrer">On Large-batch Training For Deep Learning: Generalization Gap And Sharp Minima</a> gives some reason for not using big batches, which I paraphrase badly, as big batches are likely to get stuck in local (“sharp”) minima, small batches not.</p>
Will larger batch size make computation time less in machine learning?
<p>I am trying to tune the hyper parameter i.e <strong>batch size</strong> in CNN.I have a computer of corei7,RAM 12GB and i am training a CNN network with CIFAR-10 dataset which can be found in this <a href="http://torch.ch/blog/2015/07/30/cifar.html" rel="nofollow noreferrer">blog</a>.<br><br><em>Now At first what i have read and learnt about batch size in machine learning:</em></p>

<blockquote>
  <p>let's first suppose that we're doing online learning, i.e. that we're
  using a mini­batch size of 1. The obvious worry about online learning
  is that using mini­batches which contain just a single training
  example will cause significant errors in our estimate of the gradient.
  In fact, though, the errors turn out to not be such a problem. The
  reason is that the individual gradient estimates don't need to be
  super­accurate. All we need is an estimate accurate enough that our
  cost function tends to keep decreasing. It's as though you are trying
  to get to the North Magnetic Pole, but have a wonky compass that's
  10­-20 degrees off each time you look at it. Provided you stop to
  check the compass frequently, and the compass gets the direction right
  on average, you'll end up at the North Magnetic Pole just
  fine.<br><br></p>
  
  <p>Based on this argument, it sounds as though we should use online
  learning. In fact, the situation turns out to be more complicated than
  that.As we know we can use matrix techniques to compute the gradient
  update for all examples in a mini­batch simultaneously, rather than
  looping over them. Depending on the details of our hardware and linear
  algebra library this can make it quite a bit faster to compute the
  gradient estimate for a mini­batch of (for example) size 100 , rather
  than computing the mini­batch gradient estimate by looping over the
  100 training examples separately. It might take (say) only 50 times as
  long, rather than 100 times as long.Now, at first it seems as though
  this doesn't help us that much.<br><br></p>
  
  <p>With our mini­batch of size 100 the learning rule for the weights
  looks like:<a href="https://i.stack.imgur.com/QDhgB.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/QDhgB.png" alt="enter image description here"></a><br></p>
  
  <p>where the sum is over training examples in the mini­batch. This is
  versus<a href="https://i.stack.imgur.com/X4CkK.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/X4CkK.png" alt="enter image description here"></a><br> for online learning.
  Even if it only takes 50 times as long to do the mini­batch update, it
  still seems likely to be better to do online learning, because we'd be
  updating so much more frequently. Suppose, however, that in the
  mini­batch case we increase the learning rate by a factor 100, so the
  update rule becomes<br><a href="https://i.stack.imgur.com/KMnF1.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/KMnF1.png" alt="enter image description here"></a><br>
  That's a lot like doing separate instances of online learning with a
  learning rate of <code>η</code>. But it only takes 50 times as long as doing a
  single instance of online learning. Still, it seems distinctly
  possible that using the larger mini­batch would speed things up.</p>
</blockquote>

<p><br><br></p>

<p>Now i tried with <code>MNIST digit dataset</code> and ran a sample program and set the batch size <code>1</code> at first.I noted down the training time needed for the full dataset.Then i increased the batch size and i noticed that it became faster.
<br>
But in case of training with this  <a href="https://github.com/szagoruyko/cifar.torch/blob/master/models/vgg_bn_drop.lua" rel="nofollow noreferrer">code</a> and <a href="https://github.com/szagoruyko/cifar.torch" rel="nofollow noreferrer">github link</a> changing the batch size doesn't decrease the training time.It remained same if i use 30 or 128 or 64.They are saying that they got <code>92%</code> accuracy.After two or three epoch they have got above <code>40%</code> accuracy.But when i ran the code in my computer without changing anything other than the batch size i got worse result after 10 epoch like only 28% and test accuracy stuck there in the next epochs.Then i thought since they have used batch size of 128 i need to use that.Then i used the same but it became more worse only give 11% after 10 epoch and stuck in there.<strong>Why is that??</strong></p>