title: Fluctuating loss during training for text binary classification
body: <p>I'm doing a finetuning of a <a href="https://arxiv.org/abs/2004.05150" rel="nofollow noreferrer">Longformer</a> on a <strong>document text binary classification task</strong> using <a href="https://huggingface.co/transformers/main_classes/trainer.html" rel="nofollow noreferrer">Huggingface Trainer class</a> and I'm monitoring the measures of some checkpoints with Tensorboard.</p>
  <p>Even if the F1 score and accuracy is quite high, I have perplexities about the <strong>fluctuations of training loss</strong>.</p>
  <p>I read online a reason for that can be:</p>
  <ul>
  <li>the <em>too high learning rate</em>, but I tried with 3 values (1e-4, 1e-5 and 1e-6) and all of them made the same effect</li>
  <li>a <em>small batch size</em>. I'm using a <strong>Sagemaker notebook p2.8xlarge</strong> which has 8xK80 GPUs. The batch size per GPU I can use to avoid the <strong>CUDA out of memory error</strong> is 1. So the total batch size is <strong>8</strong>. My intuition is that a bs of 8 is too small for a dataset containing 57K examples (7K steps per epoch). Unfortunately it's the highest value I can use.</li>
  </ul>
  <p>Here I have reported the trend of F1, accuracy, loss and smoothed loss. The grey line is with 1e-6 of learning rate while the pink one is 1e-5.</p>
  <img src="https://i.stack.imgur.com/KaM9D.png" width="400" />
  <img src="https://i.stack.imgur.com/aUCtr.png" width="400" />
  <img src="https://i.stack.imgur.com/u91dT.png" width="400" />
  <img src="https://i.stack.imgur.com/B0IMj.png" width="400" />
  <p>I reasume all the <strong>info</strong> of my training:</p>
  <ul>
  <li><strong>batch size</strong>: 1 x 8GPU = <strong>8</strong></li>
  <li><strong>learning rate</strong>: <strong>1e-4</strong>, <strong>1e-5</strong>, <strong>1e-6</strong> (all of them tested without improvement on loss)</li>
  <li><strong>model</strong>: <strong>Longformer</strong></li>
  <li><strong>dataset</strong>:
  <ul>
  <li><em>training set</em>: <strong>57K examples</strong></li>
  <li><em>dev set</em>: <strong>12K examples</strong></li>
  <li><em>test set</em>: <strong>12K examples</strong></li>
  </ul>
  </li>
  </ul>
  <p>Which could be the reason? Can this be considered a problem despite the quite good F1 and accuracy results?</p>