title: Training RNN with error evaluation at every time step
body: <p>I have a simpleRNN / LSTM that I'm trying to train on a sequential classification task using tensorflow. There is a sequence of data (300 time steps) that predicts a label at t=300.  For my task I would like for the RNN to evaluate the error at every timestep (not just at the final time point) and propagate it backwards (as figure below).</p>
  <p><a href="https://i.stack.imgur.com/mtNqXm.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/mtNqXm.png" alt="enter image description here" /></a></p>
  <p>After some responses below it seems I need to do a few things: use return_sequences flag; use the TimeDistributed layer to access the output from the LSTM/RNN; and also defined a custom loss function.</p>
  <pre><code>model = Sequential()
  layer1 = LSTM(n_neurons, input_shape=(length, 1), return_sequences=True)
  model.add(layer1)
  layer2 = TimeDistributed(Dense(1))
  model.add(layer2)
      
  # Define custom loss
  def custom_loss(layer1):
          
      # Create a loss function
      def loss(y_true,y_pred):
         # access layer1 at every time point and compute mean error
         # UNCLEAR HOW TO RUN AT EVERY TIME STEP
         err = K.mean(layer1(X) - y_true, axis=-1)
         return err
             
      # Return a function
      return loss
  
  # Compile the model
  model.compile(optimizer='adam', loss=custom_loss(layer), metrics=['accuracy'])
      
  </code></pre>
  <p>For now I'm a bit confused of the custom_loss function as it's not clear that how I can pass in layer1 and compute the error inside the inner most loss function.</p>
  <p>Anyone has a suggestion or can point me to a more detailed answer?</p>