title: Validation Loss Much Higher Than Training Loss
body: <p>I'm very new to deep learning models, and trying to train a multiple time series model using LSTM with Keras Sequential. There are 25 observations per year for 50 years = 1250 samples, so not sure if this is even possible to use LSTM for such small data. However, I have thousands of feature variables, not including time lags. I'm trying to predict a sequence of the next 25 time steps of data. The data is normalized between 0 and 1. My problem is that, despite trying many obvious adjustments, I cannot get the LSTM validation loss anywhere close to the training loss (overfitting dramatically, I think).</p>
  <p>I have tried adjusting number of nodes per hidden layer (25-375), number of hidden layers (1-3), dropout (0.2-0.8), batch_size (25-375), and train/ test split (90%:10% - 50%-50%). Nothing really makes much of a difference on the validation loss/ training loss disparity.</p>
  <pre><code># SPLIT INTO TRAIN AND TEST SETS
  # 25 observations per year; Allocate 5 years (2014-2018) for Testing
  n_test = 5 * 25
  test = values[:n_test, :]
  train = values[n_test:, :]
  
  # split into input and outputs
  train_X, train_y = train[:, :-25], train[:, -25:]
  test_X, test_y = test[:, :-25], test[:, -25:]
  
  # reshape input to be 3D [samples, timesteps, features]
  train_X = train_X.reshape((train_X.shape[0], 5, newdf.shape[1]))
  test_X = test_X.reshape((test_X.shape[0], 5, newdf.shape[1]))
  print(train_X.shape, train_y.shape, test_X.shape, test_y.shape)
  
  
  # design network
  model = Sequential()
  model.add(Masking(mask_value=-99, input_shape=(train_X.shape[1], train_X.shape[2])))
  model.add(LSTM(375, return_sequences=True))
  model.add(Dropout(0.8))
  model.add(LSTM(125, return_sequences=True))
  model.add(Dropout(0.8))
  model.add(LSTM(25))
  model.add(Dense(25))
  model.compile(loss='mse', optimizer='adam')
  # fit network
  history = model.fit(train_X, train_y, epochs=20, batch_size=25, validation_data=(test_X, test_y), verbose=2, shuffle=False)
  </code></pre>
  <p>Epoch 19/20</p>
  <ul>
  <li>14s - loss: 0.0512 - val_loss: 188.9568</li>
  </ul>
  <p>Epoch 20/20</p>
  <ul>
  <li>14s - loss: 0.0510 - val_loss: 188.9537</li>
  </ul>
  <p><img src="https://i.stack.imgur.com/VS2vs.png" alt="link to plot of Val Loss / Train Loss" /></p>
  <p>I assume I must be doing something obvious wrong, but can't realize it since I'm a newbie. I am hoping to either get some useful validation loss achieved (compared to training), or know that my data observations are simply not large enough for useful LSTM modeling. Any help or suggestions is much appreciated, thanks!</p>