title: Why do i get lagged results on my LSTM model
body: <p>I am new to machine learning and I am performing a Multivariate Time Series Forecast using LSTMs in Keras. I have a monthly timeseries dataset with 4 input variables (temperature, precipitation, Dew and wind_spreed) and 1 output variable (pollution). Using this data i framed a forecasting problem where, given the weather conditions and pollution for prior months, I forecast the pollution at the next month. Below is my code</p>
  <pre><code>X = df[['Temperature', 'Precipitation', 'Dew', 'Wind_speed' ,'Pollution (t_1)']].values
  y = df['Pollution (t)'].values
  y = y.reshape(-1,1)
  
  from sklearn.preprocessing import MinMaxScaler
  scaler = MinMaxScaler(feature_range=(0, 1))
  scaled = scaler.fit_transform(X)
  
  #dataset has 359 samples in total
  train_X, train_y = X[:278], y[:278]
  test_X, test_y = X[278:], y[278:]
  # reshape input to be 3D [samples, timesteps, features]
  train_X = train_X.reshape((train_X.shape[0], 1, train_X.shape[1]))
  test_X = test_X.reshape((test_X.shape[0], 1, test_X.shape[1]))
  print(train_X.shape, train_y.shape, test_X.shape, test_y.shape) 
  
  
  model = Sequential()
  model.add(LSTM(100, input_shape=(train_X.shape[1], train_X.shape[2])))
  model.add(Dropout(0.2))
  #    model.add(LSTM(70))
  #    model.add(Dropout(0.3))
  model.add(Dense(1))
  model.compile(loss='mean_squared_error', optimizer='adam')
  
  history = model.fit(train_X, train_y, epochs=700, batch_size=70, validation_data=(test_X, test_y), verbose=2, shuffle=False)
  
  # summarize history for loss
  plt.plot(history.history['loss'])
  plt.plot(history.history['val_loss'])
  plt.title('model loss')
  plt.ylabel('loss')
  plt.xlabel('epoch')
  plt.legend(['train', 'test'], loc='upper right')
  plt.show()
  </code></pre>
  <p>To do predictions i use the following code</p>
  <pre><code>from sklearn.metrics import mean_squared_error,r2_score
  yhat = model.predict(test_X)
  mse  = mean_squared_error(test_y, yhat)
  rmse = np.sqrt(mse)
  r2   = r2_score(test_y, yhat)
  
  print(&quot;test set performance&quot;)
  print(&quot;--------------------&quot;)
  print(&quot;MSE:&quot;,mse)
  print(&quot;RMSE:&quot;,rmse)
  print(&quot;R^2: &quot;,r2)
  
  fig, ax = plt.subplots(figsize=(10,5))
  ax.plot(range(len(test_y)), test_y, '-b',label='Actual')
  ax.plot(range(len(yhat)), yhat, 'r', label='Predicted')
  plt.legend()
  plt.show()
  </code></pre>
  <p><strong>Running this code i fell into the following issues:</strong></p>
  <ol>
  <li>For some reason am getting a lagged result for my test set which is not in my training data as shown on the below image. I do not understand why i have these lagged results (does it have something to do with including 'pollution (t_1)' as part of my inputs)?</li>
  </ol>
  <p>Graph Results:</p>
  <p><a href="https://i.stack.imgur.com/i2Alo.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/i2Alo.png" alt="enter image description here" /></a></p>
  <ol start="2">
  <li>By adding &quot;pollution (t_1)&quot; which is a shift by 1 lag of the polution variable as part of my inputs this variable now seems to dominate the prediction as removing the other varibales seems to have no influence on my results (r-squared and rmse) which is strange since all these variables do assit in pollution prediction.</li>
  </ol>
  <p>Is there something i am doing wrong in my code which is the reason for these issues? I am new to python so any help to answer the above 2 questions will be greatly appreaciated.</p>