title: TensorFlow LSTM: Why does test accuracy become low, but not training one?
body: <p>I have tried to build LSTM model with TensorFlow. The training of the LSTM seem to work fine, getting more than 90% accuracy. A problem plagued me is “test accuracy” that is very low. So, I thought this was due to over-fitting? But the attempts such as increasing the training batch or reducing the element_size (from 10 to 5) were waste of my efforts, nor did the applying <em>"dropout"</em> solve it. I want some directions on how to improve my code to acquire the high test accuracy.
  The followings are summary of my data/parameters</p>
  
  <pre><code>Input variable is economic time series data standardized
  Output variable is categorical features (labels) converted by one-hot encoding
  
  Sequence_length : 20
  Element_size: 5
  Hidden_layer : 80
  Categories (labels): 30 
  Training batch : 924
  Test batch : 164
  Learn rate is 0.0005 (Is it low?)
  </code></pre>
  
  <p>Here is the code I build</p>
  
  <pre><code>#Split x_buch and y_batch
  train_x,test_x=np.split(x_batch,[int(batch_size*0.85)])
  train_y,test_y=np.split(y_batch,[int(batch_size*0.85)])
  print('train_x shape: {0} and test_x shape: {1}'.format(train_x.shape,test_x.shape))
  print('train_y shape: {0} and test_y shape: {1}'.format(train_y.shape,test_y.shape))
  
  #Create placehold for inpt, labels
  inputs=tf.placeholder(tf.float32,shape=[None,step_time,element_size],name='inputs')
  y=tf.placeholder(tf.float32,shape=[None,label_num],name='y')
  
  #Tensorflow  built-in functinon
  with tf.variable_scope('lstm'):
      lstm_cell=tf.contrib.rnn.LSTMCell(hidden_layer,forget_bias=1.0)
      cell_drop=tf.contrib.rnn.DropoutWrapper(lstm_cell, output_keep_prob=0.7)
      outputs,states=tf.nn.dynamic_rnn(cell_drop,inputs,dtype=tf.float32) 
      print('outputs shape: {0}'.format(outputs.shape))
  
  W1={'linear_layer':tf.Variable(tf.truncated_normal([hidden_layer,label_num],mean=0,stddev=.01))}
  b1={'linear_layer':tf.Variable(tf.truncated_normal([label_num],mean=0,stddev=.01))}
  
  #Extract the last relevant output and use in a linear layer
  final_output=tf.matmul(outputs[:,-1,:],W1['linear_layer'])+b1['linear_layer']
  
  with tf.name_scope('cross_entropy'):
      softmax=tf.nn.softmax_cross_entropy_with_logits(logits=final_output,labels=y)
      cross_entropy=tf.reduce_mean(softmax)
  
  with tf.name_scope('train'):
      train_step=tf.train.AdamOptimizer(learn_rate,0.9).minimize(cross_entropy)
  
  with tf.name_scope('accracy'):
      correct_prediction=tf.equal(tf.argmax(y,1),tf.argmax(final_output,1))
      accuracy=(tf.reduce_mean(tf.cast(correct_prediction,tf.float32)))*100
  
  #Training
  with tf.Session()as sess:
      sess.run(tf.global_variables_initializer())    
      for step in range(5000):
          sess.run(train_step,feed_dict={inputs:train_x,y:train_y})
          if step % 500 == 0:
              acc=sess.run(accuracy,feed_dict={inputs:train_x,y:train_y})
              loss=sess.run(cross_entropy,feed_dict={inputs:train_x,y:train_y})
              print('Inter'+str(step)+',Minibatch loss= '+'{:.6f}'.format(loss)+', Traning Accracy='+'{:.5f}'.format(acc))
  
  # Test
      test_acc=sess.run(accuracy,feed_dict={inputs:test_x,y:test_y})
      print("Test Accuracy is {0}".format(test_acc))
  </code></pre>
  
  <p>and its result is </p>
  
  <pre><code>Input Shape: (21760, 5)
  Output Shape: (21760, 30)
  x_batch shape: (1088, 20, 5)
  y_batch shape: (1088, 30)
  train_x shape: (924, 20, 5) and test_x shape: (164, 20, 5)
  train_y shape: (924, 30) and test_y shape: (164, 30)
  outputs shape: (?, 20, 80)
  Inter0,Minibatch loss= 3.398923, Traning Accracy=5.30303
  Inter500,Minibatch loss= 2.027734, Traning Accracy=38.09524
  Inter1000,Minibatch loss= 1.340760, Traning Accracy=61.79654
  Inter1500,Minibatch loss= 1.010518, Traning Accracy=72.83550
  Inter2000,Minibatch loss= 0.743997, Traning Accracy=79.76190
  Inter2500,Minibatch loss= 0.687736, Traning Accracy=79.76190
  Inter3000,Minibatch loss= 0.475408, Traning Accracy=85.17316
  Inter3500,Minibatch loss= 0.430477, Traning Accracy=87.22944
  Inter4000,Minibatch loss= 0.359262, Traning Accracy=89.17749
  Inter4500,Minibatch loss= 0.274463, Traning Accracy=90.69264
  Test Accuracy is 4.878048419952393
  </code></pre>
  
  <p>I’ve never used TensorFlow and LSTM model, so,this is the first time, hence I know I am doing something wrong but cannot put a finger on it</p>
  
  <p>So, Can someone provide help?</p>
  
  <p>　　　　</p>