title: how to have a LSTM Autoencoder model over the whole vocab prediction while presenting words as embedding
body: <p>So I have been working on <code>LSTM Autoencoder model</code>. I have also created various version of this model.</p>
  
  <p><strong>1.</strong> create the model using the already trained word embedding:
  in this scenario, I used the weights of already trained Glove vector, as the weight of features(text data).
  This is the structure:</p>
  
  <pre><code>inputs = Input(shape=(SEQUENCE_LEN, EMBED_SIZE), name="input")
      encoded = Bidirectional(LSTM(LATENT_SIZE), merge_mode="sum", name="encoder_lstm")(inputs)
      encoded =Lambda(rev_entropy)(encoded)
      decoded = RepeatVector(SEQUENCE_LEN, name="repeater")(encoded)
      decoded = Bidirectional(LSTM(EMBED_SIZE, return_sequences=True), merge_mode="sum", name="decoder_lstm")(decoded)
      autoencoder = Model(inputs, decoded)
      autoencoder.compile(optimizer="sgd", loss='mse')
      autoencoder.summary()
      checkpoint = ModelCheckpoint(filepath='checkpoint/{epoch}.hdf5')
      history = autoencoder.fit_generator(train_gen, steps_per_epoch=num_train_steps, epochs=NUM_EPOCHS, validation_data=test_gen, validation_steps=num_test_steps, callbacks=[checkpoint])
  </code></pre>
  
  <ol start="2">
  <li>in the second scenario, I implemented the word embedding layer in the model itself:</li>
  </ol>
  
  <p>This is the structure:</p>
  
  <pre><code>inputs = Input(shape=(SEQUENCE_LEN, ), name="input")
  embedding = Embedding(input_dim=VOCAB_SIZE, output_dim=EMBED_SIZE, input_length=SEQUENCE_LEN,trainable=False)(inputs)
  encoded = Bidirectional(LSTM(LATENT_SIZE), merge_mode="sum", name="encoder_lstm")(embedding)
  decoded = RepeatVector(SEQUENCE_LEN, name="repeater")(encoded)
  decoded = LSTM(EMBED_SIZE, return_sequences=True)(decoded)
  autoencoder = Model(inputs, decoded)
  autoencoder.compile(optimizer="sgd", loss='categorical_crossentropy')
  autoencoder.summary()   
  checkpoint = ModelCheckpoint(filepath=os.path.join('Data/', "simple_ae_to_compare"))
  history = autoencoder.fit_generator(train_gen, steps_per_epoch=num_train_steps, epochs=NUM_EPOCHS,  validation_steps=num_test_steps)
  </code></pre>
  
  <ol start="3">
  <li><p>in the third scenario, I did not use any embedding techniques but used the <code>one hot encoding</code> for the features. and this is the structure of the model:</p>
  
  <pre><code>`inputs = Input(shape=(SEQUENCE_LEN, VOCAB_SIZE), name="input")
  encoded = Bidirectional(LSTM(LATENT_SIZE, kernel_initializer="glorot_normal",), merge_mode="sum", name="encoder_lstm")(inputs)
  encoded = Lambda(score_cooccurance,  name='Modified_layer')(encoded)
  decoded = RepeatVector(SEQUENCE_LEN, name="repeater")(encoded)
  decoded = LSTM(VOCAB_SIZE, return_sequences=True)(decoded)
  autoencoder = Model(inputs, decoded)
  sgd = optimizers.SGD(lr=0.1, decay=1e-6, momentum=0.9, nesterov=True)
  autoencoder.compile(optimizer=sgd, loss='categorical_crossentropy')
  autoencoder.summary()   
  checkpoint = ModelCheckpoint(filepath='checkpoint/50/{epoch}.hdf5')
  history = autoencoder.fit_generator(train_gen, steps_per_epoch=num_train_steps, epochs=NUM_EPOCHS, callbacks=[checkpoint])`
  </code></pre>
  
  <p>As you see, in the first and second model <code>Embed_size</code> in the <code>decoding</code> is the number of neurons in that layer. it causes the output shape of encoder layer becomes <code>[Latent_size, Embed_size]</code>. </p>
  
  <p>in the third model, the output shape of the encoder is <code>[Latent_size, Vocab_size]</code>.</p></li>
  </ol>
  
  <p><strong>Now my question</strong></p>
  
  <p>Is it doable to change the structure of the model in a way I have embedding for representing my words to the model, and at the same time having <code>vocab_size</code> in the decoder layer?</p>
  
  <p>I need to have <code>output_shape</code> of encoder layer be <code>[Latent_size, Vocab_size]</code> and at the same time I don't want to represent my features as the <code>one_hot encoding</code> for the obvious reason.</p>
  
  <p>I appreciate if you can share your idea with me.
  One idea could be adding more layers, consider that with any cost I don't want to have <code>Embed_size</code> in the last layer.</p>