title: why is my Neural Network stuck at high loss value after the first epochs
body: <p>I'm doing regression using Neural Networks. It should be a simple task for NN to do, I have 10 features and 1 output that I want to predict.I’m using pytorch for my project but my Model is not learning well. the loss start with a very high value (40000), then after the first 5-10 epochs the loss decrease rapidly to 6000-7000 and then it stuck there, no matter what I make. I tried even to change to skorch instead of pytorch so that I can use cross validation functionality but that also didn’t help. I tried different optimizers and added layers and neurons to the network but that didn’t help, it stuck at 6000 which is a very high loss value. I’m doing regression here, I have 10 features and I’m trying to predict one continuous value. that should be easy to do that’s why it is confusing me more.</p>
  
  <p>here is my network:
  I tried here all the possibilities from making more complex architectures like adding layers and units to batch normalization, changing activations etc.. nothing have worked</p>
  
  <pre><code>class BearingNetwork(nn.Module):
      def __init__(self, n_features=X.shape[1], n_out=1):
          super().__init__()
          self.model = nn.Sequential(
  
              nn.Linear(n_features, 512), 
              nn.BatchNorm1d(512),
              nn.LeakyReLU(),
              nn.Linear(512, 64),
              nn.BatchNorm1d(64),
              nn.LeakyReLU(),
              nn.Linear(64, n_out),
  #             nn.LeakyReLU(),
  #             nn.Linear(256, 128),
  #             nn.LeakyReLU(),
  #             nn.Linear(128, 64),
  #             nn.LeakyReLU(),
  #             nn.Linear(64, n_out)
          )
  
      def forward(self, x):
          out = self.model(x)
          return out
  </code></pre>
  
  <p>and here are my settings:
  using skorch is easier than pytorch. here I'm monitoring also the R2 metric and I made RMSE as a custom metric to also monitor the performance of my model. I also tried the amsgrad for Adam but that didn't help. </p>
  
  <pre><code>R2 = EpochScoring(r2_score, lower_is_better=False, name='R2')
  explained_var_score = EpochScoring(EVS, lower_is_better=False, name='EVS Metric')
  custom_score = make_scorer(RMSE)
  rmse = EpochScoring(custom_score, lower_is_better=True, name='rmse')
  
  bearing_nn = NeuralNetRegressor(
  
      BearingNetwork,
      criterion=nn.MSELoss,
      optimizer=optim.Adam,
      optimizer__amsgrad=True,
      max_epochs=5000,
      batch_size=128,
      lr=0.001,
      train_split=skorch.dataset.CVSplit(10),
      callbacks=[R2, explained_var_score, rmse, Checkpoint(), EarlyStopping(patience=100)],
      device=device
  
  )
  </code></pre>
  
  <p>I also standardize the Input values.</p>
  
  <p>my Input have the shape:</p>
  
  <pre><code>torch.Size([39006, 10])
  </code></pre>
  
  <p>and shape of output is:</p>
  
  <pre><code>torch.Size([39006, 1])
  </code></pre>
  
  <p>I’m using 128 as my Batch_size but I also tried other values like 32, 64, 512 and even 1024. Although normalizing output is not necessary but I also tried that and It didn’t work when I predict values, the loss is high. Please someone help me on this, I would appreciate every helpful advice. I ll also add a screenshot of my training and val losses and metrics over epochs to visualize how the loss is decreasing in the first 5 epochs and then it stays like forever at the value 6000 which is a very high value for a loss.</p>
  
  <p><a href="https://i.stack.imgur.com/cyEXP.png" rel="noreferrer"><img src="https://i.stack.imgur.com/cyEXP.png" alt="loss and metrics over epochs"></a></p>