title: pytorch LSTM Regression; the Loss function peaks up in every epoch
body: <p>I'm self learning the pytorch and trying out <strong>LSTM</strong> to apply in <strong>Remaining Useful Lifetime regression (RUL)</strong> problem.</p>
  
  <p>My input matrix consists of feature variables in vector stacking down along the rows [<strong>2D matrix of time-step by feature vector</strong>] I'm feeding the input with preprocessed <code>DataLoader</code> with the size of [batchsize, sequence length, number of featurs] for each <code>enumerate(train_loader)</code></p>
  
  <p><img src="https://i.stack.imgur.com/zybPL.png" alt="The training output of RUL for 50th epoch"></p>
  
  <p><img src="https://i.stack.imgur.com/Xt9zp.png" alt="RUL of the whole run"></p>
  
  <p>I was trying loss functions like MSELoss, L1Loss with RMSprop, SGD optimizer and the target is the simple declining RUL values along the timesteps.</p>
  
  <p>So the problem is, at every epochs, the loss calucation starts from the scratch as the new epoch proceeds. It could possibly due to the high integer values of the target, causing the high loss values. However, as the epoch repeats, the starting value of the loss were expected to be decreasing since the learning is somehow been done. but the result is not what i expected..</p>
  
  <p>Hopefully somebody could help me from this problem.
  Thank you in advance..!!!</p>
  
  <p>following the the model source:</p>
  
  <pre class="lang-py prettyprint-override"><code>class GRU(nn.Module):
      def __init__(self, input_dim, hidden_dim, batch_size, window_size, output_dim=1, num_layers=2):
          super(GRU, self).__init__()
          self.input_dim = input_dim
          self.hidden_dim = hidden_dim
          self.batch_size = batch_size
          self.window_size = window_size
  
          # Define the GRU layer
          self.gru1 = nn.GRU(
                  input_size=self.input_dim, 
                  hidden_size=hidden_dim,
                  bias=True)
  
          # Define the output layer
          self.linear1 = nn.Linear(hidden_dim, 500)
          self.linear2 = nn.Linear(500, 1)
  
      def forward(self, input, hidden):
          input=input.transpose(1,0)
          gru_out, self.hidden1 = self.gru1(input.view(self.window_size, self.batch_size, -1), hidden)
  
          y_pred = self.linear1(gru_out[-1].view(self.batch_size, -1))
          y_pred = self.linear2(y_pred)
  
          return y_pred.view(-1), self.hidden1.detach()
  </code></pre>
  
  <p>following is my train source:</p>
  
  <pre class="lang-py prettyprint-override"><code>#input shape: [sequence_length, batch_size, input_size]
  #target shape: [batch_size, 1]
  
  for j in range(epoch):
      for i, (dat, target) in enumerate(train_loader):
  
          out, hidden = model(dat, hidden)
          loss = criterion(out, target.view(-1))
          loss.backward(retain_graph=True)
  
          optimizer.step()
          optimizer.zero_grad()
  
          loss_history.append(loss.item())
  
          if i % 100 == 0:    
              print("{} epoch: {}/{}timestep, time: {}, loss {}:".format(j, i,len(train_loader), round(t2-t1,4), loss))
          RUL=torch.cat((RUL,out.data.view(-1)))
  
      plt.plot(RUL[-28000:].detach().numpy(), linewidth=0.5)
      plt.show()
  </code></pre>