<p>Excellent Explanation by @majid ghafouri but I just want to add more details to make sure you got this and why we are using it or which advantages can we gain using it:</p>

<p>Stochastic Gradient Descent performs updates according to the following iterative process.
This type of learning, which performs updates a single example at a time is called <strong>online
learning.</strong></p>

<p>The Algorithm for it would looks like this:</p>

<pre><code>procedure Online
  for several epochs of training do
     for each training example in the data do
         Calculate gradients of the loss
         Update the parameters according to this gradient
     end for
  end for
end procedure
</code></pre>

<p>In contrast, we can also think of a batch learning algorithm, which treats the entire data set as a single unit, calculates the gradients for this unit, then only performs update after making a full pass through the data. These two update strategies have trade-offs.</p>

<p>• Online training algorithms usually find a relatively good solution more quickly, as they don’t need to make a full pass through the data before performing an update.</p>

<p>• However, at the end of training, batch learning algorithms can be more stable, as they
are not overly influenced by the most recently seen training examples.</p>

<p>The Algorithm for Batch would looks like this:</p>

<pre><code>procedure Batch
  for several epochs of training do
    for each training example in the data do
        Calculate and accumulate gradients of the loss
    end for
    Update the parameters according to the accumulated gradient
  end for
end procedure
</code></pre>

<p>• Batch training algorithms are also more prone to falling into local optima; the randomness in online training algorithms often allows them to bounce out of local optima and
find a better global solution.</p>

<p><strong>Minibatching</strong> is a happy medium between these two strategies. Basically, minibatched
training is similar to online training, but instead of processing a single training example at a
time, we calculate the gradient for n training examples at a time. In the extreme case of n = 1,
this is equivalent to standard online training, and in the other extreme where n equals the size of the data, this is equivalent to fully batched training.</p>

<p>As we increase the number of training examples, each parameter update becomes
more informative and stable, but the amount of time to perform one update increases, so it
is common to choose an n that allows for a good balance between the two.
One other major advantage of minibatching is that by using a few tricks, it is actually
possible to make the simultaneous processing of n training examples significantly faster than
processing n different examples separately. Specifically, by taking multiple training examples
and grouping similar operations together to be processed simultaneously, we can realize large
gains in computational efficiency due to the fact that modern hardware (particularly GPUs,
but also CPUs) have very efficient vector processing instructions that can be exploited with
appropriately structured inputs.</p>

<p>the Explanation is taken from this Excellent <a href="https://arxiv.org/pdf/1703.01619.pdf" rel="nofollow noreferrer">paper,</a> you can read further if you have time: </p>
What is the meaning of a 'mini-batch' in deep learning?
<p>I'm taking the fast-ai course, and in &quot;Lesson 2 - SGD&quot; it says:</p>
<blockquote>
<p>Mini-batch: a random bunch of points that you use to update your weights</p>
</blockquote>
<p>And it also says that gradient descent uses mini-batches.</p>
<p>What is a mini-batch?  What's the difference between a mini-batch and a regular batch?</p>