title: Confused about how to implement time-distributed LSTM + LSTM
body: <p>After much reading and diagramming, I think I've come up with a model that I can use to as the foundation for more testing on which parameters and features I need to tweak. However, I am confused about how to implement the following test case (all numbers are orders of magnitude smaller than final model, but I want to start small):</p>
  
  <ul>
  <li>Input data: 5000x1 time series vector, split into 5 epochs of 1000x1</li>
  <li>For each time step, 3 epochs worth of data will be put through 3 time-distributed copies of a bidirectional LSTM layer, and each of those will output a vector of 10x1 (10 features extracted), which will then be taken as the input for a second bidirectional LSTM layer.</li>
  <li>For each time step, the first and last labels are ignored, but the center one is what is desired.</li>
  </ul>
  
  <p><a href="https://i.stack.imgur.com/2dEeV.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/2dEeV.png" alt="example model"></a></p>
  
  <p>Here's what I've come up with, which does compile. However, looking at the model.summary, I think I'm  missing the fact that I want the first LSTM to be run on a 3 of the input sequences for each output time step. What am I doing wrong?</p>
  
  <pre><code>model = Sequential()
  model.add(TimeDistributed(Bidirectional(LSTM(11, return_sequences=True, recurrent_dropout=0.1, unit_forget_bias=True), input_shape=(3, 3, epoch_len), merge_mode='sum'), input_shape=(n_epochs, 3, epoch_len)))
  model.add(TimeDistributed(Dense(7)))
  model.add(TimeDistributed(Flatten()))
  model.add(Bidirectional(LSTM(12, return_sequences=True, recurrent_dropout=0.1, unit_forget_bias=True), merge_mode='sum'))
  model.add(TimeDistributed(Dense(n_classes, activation='softmax')))
  model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
  </code></pre>