<p>I've created a <a href="https://gist.github.com/maxim5/c35ef2238ae708ccb0e55624e9e0252b" rel="nofollow noreferrer">gist</a> with a simple generator that builds on top of your initial idea: it's an LSTM network wired to the pre-trained word2vec embeddings, trained to predict the next word in a sentence. The data is the <a href="https://raw.githubusercontent.com/maxim5/stanford-tensorflow-tutorials/master/data/arxiv_abstracts.txt" rel="nofollow noreferrer">list of abstracts from arXiv website</a>.</p>
<p>I'll highlight the most important parts here.</p>
<h2>Gensim Word2Vec</h2>
<p>Your code is fine, except for the number of iterations to train it. The default <code>iter=5</code> seems rather low. Besides, it's definitely not the bottleneck -- LSTM training takes much longer. <code>iter=100</code> looks better.</p>
<pre class="lang-py prettyprint-override"><code>word_model = gensim.models.Word2Vec(sentences, vector_size=100, min_count=1, 
                                    window=5, iter=100)
pretrained_weights = word_model.wv.syn0
vocab_size, emdedding_size = pretrained_weights.shape
print('Result embedding shape:', pretrained_weights.shape)
print('Checking similar words:')
for word in ['model', 'network', 'train', 'learn']:
  most_similar = ', '.join('%s (%.2f)' % (similar, dist) 
                           for similar, dist in word_model.most_similar(word)[:8])
  print('  %s -&gt; %s' % (word, most_similar))

def word2idx(word):
  return word_model.wv.vocab[word].index
def idx2word(idx):
  return word_model.wv.index2word[idx]
</code></pre>
<p>The result embedding matrix is saved into <code>pretrained_weights</code> array which has a shape <code>(vocab_size, emdedding_size)</code>.</p>
<h2>Keras model</h2>
<p>Your code is almost correct, except for the loss function. Since the model predicts the next word, it's a classification task, hence the loss should be <code>categorical_crossentropy</code> or <code>sparse_categorical_crossentropy</code>. I've chosen the latter for efficiency reasons: this way it avoids one-hot encoding, which is pretty expensive for a big vocabulary.</p>
<pre class="lang-py prettyprint-override"><code>model = Sequential()
model.add(Embedding(input_dim=vocab_size, output_dim=emdedding_size, 
                    weights=[pretrained_weights]))
model.add(LSTM(units=emdedding_size))
model.add(Dense(units=vocab_size))
model.add(Activation('softmax'))
model.compile(optimizer='adam', loss='sparse_categorical_crossentropy')
</code></pre>
<p>Note passing the pre-trained weights to <code>weights</code>.</p>
<h2>Data preparation</h2>
<p>In order to work with <code>sparse_categorical_crossentropy</code> loss, both sentences and labels must be word indices. Short sentences must be padded with zeros to the common length.</p>
<pre class="lang-py prettyprint-override"><code>train_x = np.zeros([len(sentences), max_sentence_len], dtype=np.int32)
train_y = np.zeros([len(sentences)], dtype=np.int32)
for i, sentence in enumerate(sentences):
  for t, word in enumerate(sentence[:-1]):
    train_x[i, t] = word2idx(word)
  train_y[i] = word2idx(sentence[-1])
</code></pre>
<h2>Sample generation</h2>
<p>This is pretty straight-forward: the model outputs the vector of probabilities, of which the next word is sampled and appended to the input. Note that the generated text would be better and more diverse if the next word is <em>sampled</em>, rather than <em>picked</em> as <code>argmax</code>. The temperature based random sampling I've used is <a href="https://medium.com/machine-learning-at-petiteprogrammer/sampling-strategies-for-recurrent-neural-networks-9aea02a6616f" rel="nofollow noreferrer">described here</a>.</p>
<pre class="lang-py prettyprint-override"><code>def sample(preds, temperature=1.0):
  if temperature &lt;= 0:
    return np.argmax(preds)
  preds = np.asarray(preds).astype('float64')
  preds = np.log(preds) / temperature
  exp_preds = np.exp(preds)
  preds = exp_preds / np.sum(exp_preds)
  probas = np.random.multinomial(1, preds, 1)
  return np.argmax(probas)

def generate_next(text, num_generated=10):
  word_idxs = [word2idx(word) for word in text.lower().split()]
  for i in range(num_generated):
    prediction = model.predict(x=np.array(word_idxs))
    idx = sample(prediction[-1], temperature=0.7)
    word_idxs.append(idx)
  return ' '.join(idx2word(idx) for idx in word_idxs)
</code></pre>
<h2>Examples of generated text</h2>
<pre class="lang-py prettyprint-override"><code>deep convolutional... -&gt; deep convolutional arithmetic initialization step unbiased effectiveness
simple and effective... -&gt; simple and effective family of variables preventing compute automatically
a nonconvex... -&gt; a nonconvex technique compared layer converges so independent onehidden markov
a... -&gt; a function parameterization necessary both both intuitions with technique valpola utilizes
</code></pre>
<p>Doesn't make too much sense, but is able to produce sentences that look at least grammatically sound (sometimes).</p>
<p>The link to the <a href="https://gist.github.com/maxim5/c35ef2238ae708ccb0e55624e9e0252b" rel="nofollow noreferrer">complete runnable script</a>.</p>
Using pre-trained word2vec with LSTM for word generation
<p>LSTM/RNN can be used for text generation.
<a href="https://blog.keras.io/using-pre-trained-word-embeddings-in-a-keras-model.html" rel="nofollow noreferrer">This</a> shows way to use pre-trained GloVe word embeddings for Keras model.</p>

<ol>
<li>How to use pre-trained Word2Vec word embeddings with Keras LSTM
model? <a href="https://codekansas.github.io/gensim" rel="nofollow noreferrer">This</a> post did help.</li>
<li>How to predict / generate next <em>word</em> when the model is provided with the sequence of words as its input?</li>
</ol>

<p>Sample approach tried:</p>

<pre><code># Sample code to prepare word2vec word embeddings    
import gensim
documents = ["Human machine interface for lab abc computer applications",
             "A survey of user opinion of computer system response time",
             "The EPS user interface management system",
             "System and human system engineering testing of EPS",
             "Relation of user perceived response time to error measurement",
             "The generation of random binary unordered trees",
             "The intersection graph of paths in trees",
             "Graph minors IV Widths of trees and well quasi ordering",
             "Graph minors A survey"]
sentences = [[word for word in document.lower().split()] for document in documents]

word_model = gensim.models.Word2Vec(sentences, size=200, min_count = 1, window = 5)

# Code tried to prepare LSTM model for word generation
from keras.layers.recurrent import LSTM
from keras.layers.embeddings import Embedding
from keras.models import Model, Sequential
from keras.layers import Dense, Activation

embedding_layer = Embedding(input_dim=word_model.syn0.shape[0], output_dim=word_model.syn0.shape[1], weights=[word_model.syn0])

model = Sequential()
model.add(embedding_layer)
model.add(LSTM(word_model.syn0.shape[1]))
model.add(Dense(word_model.syn0.shape[0]))   
model.add(Activation('softmax'))
model.compile(optimizer='sgd', loss='mse')
</code></pre>

<p>Sample code / psuedocode to train LSTM and predict will be appreciated. </p>