<p><strong>1. The difference between the two programs:</strong><br />
Conceptually, your two implementations are the same: you forward <code>gradient_accumulation_steps</code> batches for each weight update.<br />
As you already observed, the second method requires more memory resources than the first one.</p>
<p>There is, however, a slight difference: usually, loss functions implementation use <code>mean</code> to reduce the loss over the batch. When you use gradient accumulation (first implementation) you reduce using <code>mean</code> over each mini-batch, but using <em><code>sum</code></em> over the accumulated <code>gradient_accumulation_steps</code> mini-batches. To make sure the accumulated gradient implementation is <em>identical</em> to large batches implementation you need to be very careful in the way the loss function is reduced. In many cases you will need to divide the accumulated loss by <code>gradient_accumulation_steps</code>. See <a href="https://stackoverflow.com/a/65913698/1714410">this answer</a> for a detailed imlpementation.</p>
<hr />
<p><strong>2. Batch size and learning rate:</strong>
Learning rate and batch size are indeed related. When increasing the batch size one usually reduces the learning rate.<br />
See, e.g.:<br />
<em>Samuel L. Smith, Pieter-Jan Kindermans, Chris Ying, Quoc V. Le</em>, <a href="https://arxiv.org/abs/1711.00489" rel="nofollow noreferrer"><strong>Don't Decay the Learning Rate, Increase the Batch Size</strong></a> (ICLR 2018).</p>
Clarification about Gradient Accumulation
<p>I'm trying to get a better understanding of how Gradient Accumulation works and why it is useful. To this end, I wanted to ask what is the difference (if any) between these two possible PyTorch-like implementations of a custom training loop with gradient accumulation:</p>
<pre class="lang-py prettyprint-override"><code>gradient_accumulation_steps = 5
for batch_idx, batch in enumerate(dataset):
  x_batch, y_true_batch = batch
  y_pred_batch = model(x_batch)

  loss = loss_fn(y_true_batch, y_pred_batch)
  loss.backward()

  if (batch_idx + 1) % gradient_accumulation_steps == 0: # (assumption: the number of batches is a multiple of gradient_accumulation_steps)
    optimizer.step()
    optimizer.zero_grad()
</code></pre>
<pre class="lang-py prettyprint-override"><code>y_true_batches, y_pred_batches = [], []
gradient_accumulation_steps = 5
for batch_idx, batch in enumerate(dataset):
  x_batch, y_true_batch = batch
  y_pred_batch = model(x_batch)

  y_true_batches.append(y_true_batch)
  y_pred_batches.append(y_pred_batch)

  if (batch_idx + 1) % gradient_accumulation_steps == 0: # (assumption: the number of batches is a multiple of gradient_accumulation_steps)
    y_true = stack_vertically(y_true_batches)
    y_pred = stack_vertically(y_pred_batches)

    loss = loss_fn(y_true, y_pred)
    loss.backward()
    optimizer.step()
    optimizer.zero_grad()

    y_true_batches.clear()
    y_pred_batches.clear()
</code></pre>
<p>Also, kind of as an unrelated question: Since the purpose of gradient accumulation is to mimic a larger batch size in cases where you have memory constraints, does it mean that I should also increase the learning rate proportionally?</p>