title: How to use AttentionMechanism with MultiRNNCell and dynamic_decode?
body: <p>I want to create a multi-layered dynamic RNN-based decoder that uses an attention mechanism. To do this, I first create an attention mechanism:</p>
  
  <pre><code>attention_mechanism = BahdanauAttention(num_units=ATTENTION_UNITS,
                                          memory=encoder_outputs,
                                          normalize=True)
  </code></pre>
  
  <p>Then I use the <code>AttentionWrapper</code> to wrap a LSTM cell with the attention mechanism:</p>
  
  <pre><code>attention_wrapper = AttentionWrapper(cell=self._create_lstm_cell(DECODER_SIZE),
                                               attention_mechanism=attention_mechanism,
                                               output_attention=False,
                                               alignment_history=True,
                                               attention_layer_size=ATTENTION_LAYER_SIZE)
  </code></pre>
  
  <p>where <code>self._create_lstm_cell</code> is defined as follows:</p>
  
  <pre><code>@staticmethod
  def _create_lstm_cell(cell_size):
      return BasicLSTMCell(cell_size)
  </code></pre>
  
  <p>I then do some bookkeeping (e.g. creating my <code>MultiRNNCell</code>, creating an initial state, creating a <code>TrainingHelper</code>, etc.)</p>
  
  <pre><code>        attention_zero = attention_wrapper.zero_state(batch_size=tf.flags.FLAGS.batch_size, dtype=tf.float32)
  
          # define initial state
          initial_state = attention_zero.clone(cell_state=encoder_final_states[0])
  
          training_helper = TrainingHelper(inputs=self.y,  # feed in ground truth
                                           sequence_length=self.y_lengths)  # feed in sequence lengths
  
          layered_cell = MultiRNNCell(
              [attention_wrapper] + [ResidualWrapper(self._create_lstm_cell(cell_size=DECODER_SIZE))
                                     for _ in range(NUMBER_OF_DECODER_LAYERS - 1)])
  
          decoder = BasicDecoder(cell=layered_cell,
                                 helper=training_helper,
                                 initial_state=initial_state)
  
          decoder_outputs, decoder_final_state, decoder_final_sequence_lengths = dynamic_decode(decoder=decoder,
                                                                                                maximum_iterations=tf.flags.FLAGS.max_number_of_scans // 12,
                                                                                                impute_finished=True)
  </code></pre>
  
  <p>But I receive the following error: <code>AttributeError: 'LSTMStateTuple' object has no attribute 'attention'</code>.</p>
  
  <p>What is the correct way to add an attention mechanism to a MultiRNNCell dynamic decoder?</p>