title: Keras autoencoder : validation loss > training loss - but performing well on testing dataset
body: <p>IN SHORT:</p>
  
  <p>I have trained an Autoencoder whose validation loss is always higher than its training loss (see attached figure).<a href="https://i.stack.imgur.com/6hXFa.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/6hXFa.png" alt="as - Autoencoder"></a> I would think that this is a signal of overfitting. However, my Autoencoder performs well on the testing dataset. I was wondering if:</p>
  
  <p>1) with reference to the architecture of the network, provided below, anyone could provide insights on how to reduce the validation loss (and how it is possible that the validation loss is much higher than the training one, despite the performance of the Autoencoder being good on the testing dataset);</p>
  
  <p>2) if it is actually a problem that there is this gap between training and validation loss (when the performance on the testing dataset is actually good).</p>
  
  <p>DETAILS:</p>
  
  <p>I coded up my deep Autoencoder in Keras (code below). The architecture is 2001 (input layer) - 1000 - 500 - 200 - 50 - 200 - 500 - 1000 - 2001 (output layer). My samples are 1d functions of time. Each of them has 2001 time components. I have 2000 samples, which I split in 1500 for training, 500 for testing. Ot of the 1500 training samples, 20% of them (i.e. 300) are used as validation set. I normalize the training set removing the mean and dividing by the standard deviation. I use the mean and standard deviation of the training dataset to normalise the testing dataset as well. </p>
  
  <p>I train the Autoencoder using Adamax optimizer and mean squared error as loss function.</p>
  
  <pre><code>from tensorflow.keras.layers import Input, Dense, Dropout
  from tensorflow.keras.models import Model
  from tensorflow.keras import optimizers
  
  import numpy as np
  import copy
  
  
  # data
  data = # read my input samples. They are 1d functions of time and I have 2000 of them.
  # Each function has 2001 time components
  
  # shuffling data before training
  import random
  random.seed(4)
  random.shuffle(data)
  
  # split training (1500 samples) and testing (500 samples) dataset
  X_train = data[:1500]
  X_test = data[1500:]
  
  # normalize training and testing set using mean and std deviation of training set
  X_mean = X_train.mean()
  X_train -= X_mean
  X_std = X_train.std()
  X_train /= X_std
  
  X_test -= X_mean
  X_test /= X_std
  
  
  ### MODEL ###
  
  # Architecture
  
  # input layer
  input_shape = [X_train.shape[1]]
  X_input = Input(input_shape)
  
  # hidden layers
  
  x = Dense(1000, activation='tanh', name='enc0')(X_input)
  encoded = Dense(500, activation='tanh', name='enc1')(x)
  encoded_2 = Dense(200, activation='tanh', name='enc2')(encoded)
  encoded_3 = Dense(50, activation='tanh', name='enc3')(encoded_2)
  decoded_2 = Dense(200, activation='tanh', name='dec2')(encoded_3)
  decoded_1 = Dense(500, activation='tanh', name='dec1')(decoded_2)
  x2 = Dense(1000, activation='tanh', name='dec0')(decoded_1)
  
  # output layer
  decoded = Dense(input_shape[0], name='out')(x2)
  
  # the Model
  model = Model(inputs=X_input, outputs=decoded, name='autoencoder')
  
  # optimizer
  opt = optimizers.Adamax()
  model.compile(optimizer=opt, loss='mse', metrics=['acc'])
  print(model.summary())
  
  ###################
  
  ### TRAINING ###
  
  epochs = 1000
  # train the model
  history = model.fit(x = X_train, y = X_train,
                      epochs=epochs,
                      batch_size=100,
                      validation_split=0.2)  # using 20% of training samples for validation
  
  # Testing 
  prediction = model.predict(X_test)
  for i in range(len(prediction)):
      prediction[i] = np.multiply(prediction[i], X_std) + X_mean
  
  loss = history.history['loss']
  val_loss = history.history['val_loss']
  epochs = range(epochs)
  plt.figure()
  plt.plot(epochs, loss, 'bo', label='Training loss')
  plt.plot(epochs, val_loss, 'b', label='Validation loss')
  plt.title('Training and validation loss')
  plt.legend()
  plt.show()
  plt.close()
  </code></pre>