<p>I'm pretty sure, based on the age of the post, you've already solved your problem. But just in case, here is my 2 cents. </p>

<p>You end up predicting the most frequent word. So, if you remove the stopwords, you'll be predicting the next most frequent word. There are two ways, that I know of, to deal with this problem.</p>

<p>First, you can use a loss that emphasizes the less frequent classes, or words in your case. Here is a <a href="https://arxiv.org/pdf/1708.02002.pdf" rel="nofollow noreferrer">research paper</a> introducing focal loss, and conveniently, a <a href="https://github.com/umbertogriffo/focal-loss-keras" rel="nofollow noreferrer">github</a> implementation of it for keras.  </p>

<p>Another approach is to use class_weight in the fit function.</p>

<pre class="lang-py prettyprint-override"><code>model.fit(X, y, epochs=50, batch_size=128, callbacks=callbacks_list, class_weight=class_weight)
</code></pre>

<p>where you can set the weights for words with lower frequencies higher, e.g. inversely proportional to the frequencies. </p>
Keras Generative LSTM only predicting stop words
<p>I have created a model in keras using LSTM for predicting the next word given a sequence of words.Below is my code for the same:</p>

<pre><code>    # Small LSTM Network to Generate Text for Alice in Wonderland
# load ascii text and covert to lowercase
filename = "wonderland.txt"
raw_text = open(filename).read()
raw_text = raw_text.lower()
print raw_text
# create mapping of unique words to integers
print raw_text
raw_text = re.sub(r'[^\w\s]','',raw_text)
raw_text = re.sub('[^a-z\ \']+', " ", raw_text)
words_unsorted=list(raw_text.split())
words= sorted(list(set(raw_text.split())))
word_to_int = dict((w, i) for i, w in enumerate(words))
int_to_word = dict((i, w) for i, w in enumerate(words))
#print word_to_int

n_words = len(words_unsorted)
n_vocab = len(words)
print "Total Words: ", n_words
print "Total Vocab: ", n_vocab

# prepare the dataset of input to output pairs encoded as integers
seq_length = 7
dataX = []
dataY = []
for i in range(0, n_words - seq_length, 1):
    seq_in = words_unsorted[i:i + seq_length]
    seq_out = words_unsorted[i + seq_length]
    #print seq_in
    dataX.append([word_to_int[word] for word in seq_in])
    dataY.append(word_to_int[seq_out])


n_patterns = len(dataX)
print "Total Patterns: ", n_patterns

# reshape X to be [samples, time steps, features]
X = numpy.reshape(dataX, (n_patterns, seq_length, 1))
print X[0]
# normalize
X = X / float(n_vocab)
# one hot encode the output variable
y = np_utils.to_categorical(dataY)
# define the LSTM model
model = Sequential()
model.add(LSTM(256, input_shape=(X.shape[1], X.shape[2]), return_sequences=True))
model.add(Dropout(0.2))
model.add(LSTM(256))
model.add(Dropout(0.2))
model.add(Dense(y.shape[1], activation='softmax'))
model.compile(loss='categorical_crossentropy', optimizer='adam')
print model.summary()
# define the checkpoint
filepath="weights-improvement-{epoch:02d}-{loss:.4f}.hdf5"
checkpoint = ModelCheckpoint(filepath, monitor='loss', verbose=1, save_best_only=True, mode='min')
callbacks_list = [checkpoint]
# fit the model
model.fit(X, y, epochs=50, batch_size=128, callbacks=callbacks_list)
</code></pre>

<p>Issue is when I predict on a test sentence I always end up getting "and" as the next word prediction!Should I remove all stop words or something else?Further,I am training it for 20 epochs.</p>