title: CNTK: How do I initialize LSTM hidden state?
body: <p>I'm trying convert a working image captioning CNN-LSTM network from TensorFlow to CNTK, and have what I think is a correctly trained model, but am having trouble figuring out how to extract predictions from the final trained CNTK model.</p>
  
  <p>This is the general architecture I'm working with:
  <a href="https://i.stack.imgur.com/P76mN.jpg" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/P76mN.jpg" alt="enter image description here"></a>
  This is my CNTK model:</p>
  
  <pre><code>def create_lstm_model(image_features, text_features):
      embedding_dim = 512
      hidden_dim = 512
      cell_dim = 512
      vocab_dim = 77
  
      image_embedding = Embedding(embedding_dim)
      text_embedding = Embedding(embedding_dim)
      lstm_classifier = Sequential([Stabilizer(),
                                    Recurrence(LSTM(hidden_dim)),
                                    Recurrence(LSTM(hidden_dim)),
                                    Stabilizer(),
                                    Dense(vocab_dim)]) 
  
      embedded_images = BatchNormalization()(image_embedding(image_features))
      embedded_text = text_embedding(text_features)
      lstm_input = C.plus(embedded_images, embedded_text)
      lstm_input = C.dropout(lstm_input, 0.5)
      output = lstm_classifier(lstm_input)    
  
      return output
  </code></pre>
  
  <p>I'm providing my data in CTF format, with fixed caption sequence sizes of 40, using this structure:</p>
  
  <pre><code>def create_reader(path, is_training):
      return MinibatchSource(CTFDeserializer(path, StreamDefs(
          target_tokens = StreamDef(field='target_tokens', shape=vocab_len, is_sparse=True),
          input_tokens = StreamDef(field='input_tokens', shape=vocab_len, is_sparse=True),
          image_features = StreamDef(field='image_features', shape=image_features_dim, is_sparse=False)
      )), randomize = is_training, max_sweeps = INFINITELY_REPEAT if is_training else 1)
  </code></pre>
  
  <p>Aside: the reason for three streams of data - I have an input image feature vector (last 2048-dim layer of a pre-trained ResNet), a sequence of input text tokens, and a sequence of output text tokens. So basically my CTF file, in terms of sequences, looks like:</p>
  
  <pre><code>0 | target_token_0  | input_token_0 | input_image_feature_vector (2048-dim)
  0 | target_token_1  | input_token_1 | empty array of 2048 zeros
  0 | target_token_2  | input_token_2 | empty array of 2048 zeros
  ...
  0 | target_token_40 | input_token_40 | empty array of 2048 zeros
  1 | target_token_0  | input_token_0 | input_image_feature_vector (2048-dim)
  1 | target_token_1  | input_token_1 | empty array of 2048 zeros
  1 | target_token_2  | input_token_2 | empty array of 2048 zeros
  ...
  1 | target_token_40 | input_token_40 | empty array of 2048 zeros
  </code></pre>
  
  <p>Basically, I couldn't figure out how to slice &amp; splice two sequences together in CNTK (even though you can splice two tensors easily), so I'm hacking around it by providing only the first element in a sequence with an input 2048-dim image feature vector, and the remaining elements in a sequence with an empty 2048-dim vector of zeros - setup for:</p>
  
  <p><code>C.plus(embedded_images, embedded_text)</code></p>
  
  <p>in the model above - where the goal is to essentially take the first element of a sequence of 40 <code>[2048]-&gt;[512]</code> image embeddings and <strong>hack-splice</strong>(TM) it in front of the last 39 elements of a sequence of 40 <code>[vocab_dim]-&gt;[512]</code> word embeddings. I'm counting on pretty empty <code>[2048]-&gt;[512]</code> image embeddings being learned for the empty image vectors (2048 zeros), so I'm taking my embedded image sequence and element-wise adding it to my embedded text sequence before all goes into the LSTM. Basically, this:</p>
  
  <pre><code>image embedding sequence: [-1, 40, 512]  (e.g., [-1, 0, 512])
  text embedding sequence:  [-1, 40, 512]  (e.g., [-1, 1:40, 512)
  +
  ---------------------------------------
  lstm input sequence:      [-1, 40, 512]
  </code></pre>
  
  <p>Which brings me to my actual question. Now that I have a model that trains decently well, I'd like to extract caption predictions from the model, basically doing something like this (from this <a href="https://github.com/yunjey/pytorch-tutorial/blob/master/tutorials/03-advanced/image_captioning/model.py" rel="nofollow noreferrer">PyTorch image captioning tutorial</a>):</p>
  
  <pre><code>def sample(self, features, states=None):
      """Samples captions for given image features (Greedy search)."""
      sampled_ids = []
      inputs = features.unsqueeze(1)
      for i in range(20):                                      # maximum sampling length
          hiddens, states = self.lstm(inputs, states)          # (batch_size, 1, hidden_size), 
          outputs = self.linear(hiddens.squeeze(1))            # (batch_size, vocab_size)
          predicted = outputs.max(1)[1]
          sampled_ids.append(predicted)
          inputs = self.embed(predicted)
          inputs = inputs.unsqueeze(1)                         # (batch_size, 1, embed_size)
      sampled_ids = torch.cat(sampled_ids, 1)                  # (batch_size, 20)
      return sampled_ids.squeeze()
  </code></pre>
  
  <p>The problem is, I can't figure out the CNTK equivalent for getting the hidden state out of an LSTM and pumping it back in next time step:</p>
  
  <pre><code>hiddens, states = self.lstm(inputs, states)
  </code></pre>
  
  <p>How does this work in CNTK?</p>