title: TimeDistributed layer and return sequences etc for LSTM in Keras
body: <p>Sorry I am new to RNN. I have read  <a href="https://stackoverflow.com/questions/42755820/how-to-use-return-sequences-option-and-timedistributed-layer-in-keras/%22this%20post%22">this post</a> on TimeDistributed layer. </p>
  
  <p>I have reshaped my data in to Keras requried  <code>[samples, time_steps, features]</code>: <code>[140*50*19]</code>, which means I have 140 data points, each has 50 time steps, and 19 features. My output is shaped <code>[140*50*1]</code>. I care more about the last data point's accuracy. This is a regression problem. </p>
  
  <p>My current code is : </p>
  
  <pre><code>x = Input((None, X_train.shape[-1]) , name='input')
  lstm_kwargs = { 'dropout_W': 0.25, 'return_sequences': True, 'consume_less': 'gpu'} 
  lstm1 = LSTM(64, name='lstm1', **lstm_kwargs)(x)
  output = Dense(1, activation='relu', name='output')(lstm1)
  model = Model(input=x, output=output)
  sgd = SGD(lr=0.00006, momentum=0.8, decay=0, nesterov=False)
  optimizer = sgd
  model.compile(optimizer=optimizer, loss='mean_squared_error')
  </code></pre>
  
  <p>My questions are: </p>
  
  <ol>
  <li>My case is many-to-many, so I need to use <code>return_sequences=True</code>? How about if I only need the last time step's prediction, it would be many-to-one. So I need to my output to be <code>[140*1*1]</code> and <code>return_sequences=False</code>? </li>
  <li>Is there anyway to enhance my last time points accuracy if I use many-to-many? I care more about it than the other points accuracy. </li>
  <li><p>I have tried to use TimeDistributed layer as</p>
  
  <pre><code>output = TimeDistributed(Dense(1, activation='relu'), name='output')(lstm1)
  </code></pre>
  
  <p>the performance seems to be worse than without using TimeDistributed layer. Why is this so? </p></li>
  <li>I tried to use <code>optimizer=RMSprop(lr=0.001)</code>. I thought <code>RMSprop</code> is supposed to stabilize the NN. But I was never able to get good result using <code>RMSprop</code>.</li>
  <li>How do I choose a good <code>lr</code> and momentum for <code>SGD</code>? I have been testing on different combinations manually. Is there a cross validation method in keras? </li>
  </ol>