title: Multilayer Seq2Seq model with LSTM in Keras
body: <p>I was making a seq2seq model in keras. I had built single layer encoder and decoder  and they were working fine. But now I want to extend it to multi layer encoder and decoder.
  I am building it using Keras Functional API.</p>
  
  <h1>Training:-</h1>
  
  <h1>Code for encoder:-</h1>
  
  
  
  <pre class="lang-py prettyprint-override"><code>encoder_input=Input(shape=(None,vec_dimension))
  encoder_lstm=LSTM(vec_dimension,return_state=True,return_sequences=True)(encoder_input)
  encoder_lstm=LSTM(vec_dimension,return_state=True)(encoder_lstm)
  encoder_output,encoder_h,encoder_c=encoder_lstm
  </code></pre>
  
  <h1>Code for decoder:-</h1>
  
  <pre class="lang-py prettyprint-override"><code>encoder_state=[encoder_h,encoder_c]
  decoder_input=Input(shape=(None,vec_dimension))
  decoder_lstm= LSTM(vec_dimension,return_state=True,return_sequences=True (decoder_input,initial_state=encoder_state)
  decoder_lstm=LSTM(vec_dimension,return_state=True,return_sequences=True)(decoder_lstm)
  decoder_output,_,_=decoder_lstm
  </code></pre>
  
  <h1>For testing :-</h1>
  
  <pre class="lang-py prettyprint-override"><code>encoder_model=Model(inputs=encoder_input,outputs=encoder_state)
  decoder_state_input_h=Input(shape=(None,vec_dimension))
  decoder_state_input_c=Input(shape=(None,vec_dimension))
  decoder_states_input=[decoder_state_input_h,decoder_state_input_c]
  decoder_output,decoder_state_h,decoder_state_c =decoder_lstm #(decoder_input,initial_state=decoder_states_input)
  decoder_states=[decoder_state_h,decoder_state_c]
  decoder_model=Model(inputs=[decoder_input]+decoder_states_input,outputs=[decoder_output]+decoder_states)
  </code></pre>
  
  <p>Now when I try to increase the no. of layers in the decoder for training then training works fine but for testing it dosen't works and throws error.</p>
  
  <p>Actually the problem is when making it multi layer i had shifted the <code>initial_state</code> to a middle layer which used to be specified at the end.So
  when I am calling it during testing, it is throwing errors.</p>
  
  <pre class="lang-py prettyprint-override"><code>RuntimeError: Graph disconnected: cannot obtain value for tensor Tensor("input_64:0", shape=(?, ?, 150), dtype=float32) at layer "input_64".The following previous layers were accessed without issue: []
  </code></pre>
  
  <p>How should I pass the <code>initial_state=decoder_states_input</code> which is for the input layer so that it doesn't throws error.
  How should I pass the <code>initial_state=decoder_states_input</code> in the end layer for for the first Input layer??</p>
  
  <h1>EDIT:-</h1>
  
  <p>In that code I have tried to make multiple layers of decoder LSTM. But that's giving error.
  When working with single layer.The correct codes are:-</p>
  
  <h1>Encoder(Training):-</h1>
  
  <pre class="lang-py prettyprint-override"><code>encoder_input=Input(shape=(None,vec_dimension))
  encoder_lstm =LSTM(vec_dimension,return_state=True)(encoder_input)
  encoder_output,encoder_h,encoder_c=encoder_lstm
  </code></pre>
  
  <h1>Decoder(Training):-</h1>
  
  <pre class="lang-py prettyprint-override"><code>encoder_state=[encoder_h,encoder_c]
  decoder_input=Input(shape=(None,vec_dimension))
  decoder_lstm= LSTM(vec_dimension, return_state=True, return_sequences=True)
  decoder_output,_,_=decoder_lstm(decoder_input,initial_state=encoder_state)
  </code></pre>
  
  <h1>Decoder(Testing)</h1>
  
  <pre class="lang-py prettyprint-override"><code>decoder_output,decoder_state_h,decoder_state_c=decoder_lstm( decoder_input, initial_state=decoder_states_input)
  decoder_states=[decoder_state_h,decoder_state_c]
  decoder_output,decoder_state_h,decoder_state_c=decoder_lstm (decoder_input,initial_state=decoder_states_input)
  decoder_model=Model(inputs=[decoder_input]+decoder_states_input,outputs=[decoder_output]+decoder_states)
  </code></pre>