title: Loss value doesn't converging with Pytorch-Lightning
body: <p>I am quite new to deep learning and PyTorch lightning, and I have some issues with my loss of values while trying to pre-train BERT for a recommendation from scratch.</p>
  <p>I followed this tutorial <a href="https://towardsdatascience.com/build-your-own-movie-recommender-system-using-bert4rec-92e4e34938c5" rel="nofollow noreferrer">https://towardsdatascience.com/build-your-own-movie-recommender-system-using-bert4rec-92e4e34938c5</a> (and use the GitHub code as my starting point) for Bert4Rec.</p>
  <p>Here is the relevant snippet from my module implementation</p>
  <pre><code>class Recommender(pl.LightningModule):
      def __init__(self, vocabulary_size, features=128,
                   mask=1, dropout=0.4, lr=5e-5, iterations=[]):
          super().__init__()
          ...
          self.item_embeddings = torch.nn.Embedding(self.vocabulary_size, embedding_dim=features)
  
          self.input_pos_embedding = torch.nn.Embedding(512, embedding_dim=features)
  
          encoder_layer = nn.TransformerEncoderLayer(d_model=features, nhead=4, dropout=self.dropout)
  
          self.encoder = torch.nn.TransformerEncoder(encoder_layer, num_layers=6)
          self.linear_out = Linear(features, self.vocabulary_size)
  
      def encode_src(self, src_items):
          src_items = self.item_embeddings(src_items)
  
          batch_size, in_sequence_len = src_items.size(0), src_items.size(1)
          pos_encoder = (
              torch.arange(0, in_sequence_len, device=src_items.device)
                  .unsqueeze(0)
                  .repeat(batch_size, 1)
          )
          pos_encoder = self.input_pos_embedding(pos_encoder)
  
          src_items += pos_encoder
  
          src = src_items.permute(1, 0, 2)
  
          src = self.encoder(src)
  
          return src.permute(1, 0, 2)
  
  
      def forward(self, src_items):
          src = self.encode_src(src_items)
          out = self.linear_out(src)
          return out
  
      def training_step(self, batch, batch_idx):
          src_items, y_true = batch
  
          y_pred = self(src_items)
  
          y_pred = y_pred.view(-1, y_pred.size(2))
          y_true = y_true.view(-1)
  
          src_items = src_items.view(-1)
          mask = src_items == self.mask
  
          loss = masked_ce(y_pred=y_pred, y_true=y_true, mask=mask)
          accuracy = masked_accuracy(y_pred=y_pred, y_true=y_true, mask=mask)
          
          self.log(&quot;train_loss&quot;, loss)
          self.log(&quot;train_accuracy&quot;, accuracy)
          return loss
  
        ...
  
      def configure_optimizers(self):
          optimizer = torch.optim.Adam(self.parameters(), lr=self.lr)
          scheduler = torch.optim.lr_scheduler.ReduceLROnPlateau(
              optimizer, patience=10, factor=0.1
          )
  
      return {
          &quot;optimizer&quot;: optimizer,
          &quot;lr_scheduler&quot;: scheduler,
          &quot;monitor&quot;: &quot;valid_loss&quot;,
      }
  </code></pre>
  <p>For some reason, my learning rate stops decreasing after a certain point and stays pretty much the same.
  here is the average loss of each epoch for the 120 epochs:</p>
  <pre><code>[7.4491517169100865, 6.944184269394364, 6.700686190818046, 6.543606683656618, 6.390220044969438, 6.289553656115069, 6.232016379172141, 6.163771061806588, 6.099021504471849, 6.050294904737501, 5.98124076415588, 5.936885427545618, 5.917650453321211, 5.850849871401553, 5.8284001484050885, 5.8062581433667555, 5.756445451780363, 5.75059446653685, 5.702613687133407, 5.69182988902828, 5.6686874168651835, 5.619234044272621, 5.622982791713527, 5.603236912487744, 5.574997285703519, 5.554195235083411, 5.5423704346379, 5.526269711054362, 5.511078507095963, 5.514718792221329, 5.465791008016607, 5.466700993262015, 5.458291268324828, 5.45968119685237, 5.43866441748641, 5.433992590154852, 5.412535585798659, 5.422043593676837, 5.397110146683854, 5.380480324541843, 5.378754429869704, 5.375121655526224, 5.3922829947791415, 5.355316250889867, 5.342787119242045, 5.345856327671666, 5.339864837037431, 5.341125234588608, 5.3144074297762725, 5.305943638474137, 5.318321661428885, 5.3143898166335735, 5.294310178365316, 5.333440277311537, 5.317679136961669, 5.300545753778757, 5.288642429374717, 5.296188079797709, 5.258556879318513, 5.271586121262254, 5.256002846900168, 5.256834919626887, 5.256189991165329, 5.251029547031696, 5.258525119290815, 5.240669572436893, 5.229032474475819, 5.260464797626148, 5.227322119253653, 5.199392991500335, 5.227120114279701, 5.239981561809689, 5.231669370834534, 5.2572159003447725, 5.234966399791363, 5.195458581378391, 5.215773612290651, 5.203279604305615, 5.202362375813085, 5.203736131733006, 5.219805061399519, 5.201947963034904, 5.191771214192097, 5.203967503241232, 5.204905313533825, 5.187097654686318, 5.180448412537217, 5.194675721444406, 5.1764420959445925, 5.193467300098102, 5.159242201376486, 5.184554450862758, 5.160313559724046, 5.163822394830209, 5.178260997012332, 5.166500503713782, 5.173784165291695, 5.1573744650717614, 5.132783733927332, 5.162636699141921, 5.147606128448242, 5.136373396988985, 5.150685215378189, 5.1462944379678595, 5.1802691055370405, 5.143014167999482, 5.120888564918373, 5.1491153922763555, 5.148981760214041, 5.128810315518765, 5.1473502191576035, 5.122937468795089, 5.112937651119672, 5.1254263394349096, 5.149980151498163, 5.140357047110587, 5.109714733826387, 5.128054624324566, 5.132359328570667]
  </code></pre>
  <p>As you can see, it starts at ~7.5 decreases pretty monotonic to around 5.2, and then not really going anywhere, what could be the reason for it?
  My hyperparameters I use right now are:</p>
  <ul>
  <li>learning rate: 1e-4</li>
  <li>dropout: 0.3</li>
  <li>batch-size: 32
  And my training was 120 epochs for now.</li>
  </ul>
  <p>Some of the responses for similar issues suggest playing around with the hyperparameters, so I did try a few things:</p>
  <ul>
  <li><p>Since it could be stuck in local min, it could be that the learning rate needs to be changed in order to be able to go out of it (is that right?) - I tried to change my learning rate from 1-e4 to 5-e5, it didn’t help much</p>
  </li>
  <li><p>In order to check if the training work as expected, I tried to overfit my model on a small number of datasets (10) and the avg of the loss for the first 20 epochs look as follows:</p>
  <p>[0.0, 0.0, 0.0, 1.2672607898712158, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.9458177089691162, 0.0, 0.0, 0.0, 1.6752853393554688, 0.0, 0.0]</p>
  </li>
  </ul>
  <p>Any idea or suggestions would be very appreciated!</p>