<p>After poking around I found a <a href="https://arxiv.org/pdf/1708.02002.pdf" rel="nofollow noreferrer">research paper</a> introducing focal loss, and conveniently, a <a href="https://github.com/umbertogriffo/focal-loss-keras" rel="nofollow noreferrer">github</a> implementation of it for keras.</p>
<p>That, combined with @meowongac's suggestion ( I used Google word2vec embeddings) resulted in much better sampling of words with lower frequencies.</p>
<p>I also, separately, used <code>class_weight</code> :</p>
<pre class="lang-py prettyprint-override"><code>model.fit_generator(batch_sequence_data(prediction_sequences, 
                    BATCH_SIZE, SEQUENCE_LENGTH, VOCAB_SIZE), 
                    steps_per_epoch = steps_per_epoch, epochs=30, callbacks=[earlystop],
                    class_weight = class_weight)
</code></pre>
<p>which I set inversely proportional to the word frequency.
Again, combined with using Google word embeddings it worked, in some sense, even better, bringing up words with lower frequencies.</p>
<p>For example, for a 10 word sequence:</p>
<p><code>['two', 'three', 'marines', 'sort', 'charges', 'pending', 'another', 'fight', 'week', 'interesting']</code></p>
<p>the focal loss approach with gamma = 5 predicted the next word <code>people</code>, the class_weight approach predicted <code>attorney</code></p>
LSTM word prediction model predicts only the most frequent words, or which loss to use for imbalanced data
<p>I decided to try my hand at building a word prediction model using a recurrent neural network. There are a number of different examples online, including online courses, that make it sound that building such a model is fairly easy. Most of them use LSTM. Also, most, if not all, of them use a very small data set. I decided to try it with a larger data set, the 20 News Groups data set <code>from sklearn.datasets import fetch_20newsgroups</code>. I do some minimal preprocessing: removal of punctuation, stopwords and numbers.  </p>

<p>I'm predicting a word based on the 10 preceding words history. I only use the posts that have at least 11 words. For each post I build a training set by taking a sliding window of size 11 and sliding it along the post. For each position the first 10 words are predictors and the 11th word is the target. I put together a simple model: Embedding layer, LSTM layer, and the output Dense layer. Here is the code:</p>

<pre class="lang-py prettyprint-override"><code>def make_prediction_sequences(input_texts, max_nb_words, sequence_length = 10):
# input_texts is a list of strings/texts

# select top vocab_size words based on the word counts
# word_index is the dictionary used to transform the words into the tokens. 
    tokenizer = Tokenizer(oov_token='UNK',num_words=max_nb_words)
    tokenizer.fit_on_texts(input_texts)
    sequences = tokenizer.texts_to_sequences(input_texts)

    prediction_sequences = []
    for sequence in sequences:
        if len(sequence) &gt; sequence_length: # at least 1 for prediction
            for j in range(0,len(sequence) - sequence_length):
                prediction_sequences.append(sequence[j:sequence_length+j+1])

    word_index = {e:i-1 for e,i in tokenizer.word_index.items()  if i &lt;= max_nb_words} # i-1 because tokenizer is 1 indexed


    return (np.array(prediction_sequences) , word_index)

def batch_sequence_data(prediction_sequences, batch_size, sequence_length, vocab_size):
    number_batches = int(len(prediction_sequences)/batch_size)
    while True:
        for i in range(number_batches):
            X = prediction_sequences[i*batch_size:(i+1)*batch_size, 0:sequence_length]
            Y = to_categorical(prediction_sequences[i*batch_size:(i+1)*batch_size, sequence_length], num_classes=vocab_size)
            yield np.array(X),Y

VOCAB_SIZE = 15000
SEQUENCE_LENGTH = 10
BATCH_SIZE = 128
prediction_sequences, word_index = make_prediction_sequences(data, VOCAB_SIZE, sequence_length=SEQUENCE_LENGTH)

## define the model
EMBEDDING_DIM = 64
rnn_size = 32

sequence_input = Input(shape=(SEQUENCE_LENGTH,), dtype='int32', name='rnn_input')
embedding_layer = Embedding(len(word_index), EMBEDDING_DIM, input_length=SEQUENCE_LENGTH)
embedded_sequences = embedding_layer(sequence_input)
x = LSTM(rnn_size, use_bias=True)(embedded_sequences)
preds = Dense(VOCAB_SIZE, activation='softmax')(x)
model = Model(sequence_input, preds)
model.compile(loss='categorical_crossentropy',optimizer='adam',metrics=['categorical_accuracy'])

#train the model
steps_per_epoch = len(prediction_sequences)/(BATCH_SIZE * SEQUENCE_LENGTH)
earlystop = EarlyStopping(patience=3, restore_best_weights=True,monitor='loss')
history = model.fit_generator(batch_sequence_data(prediction_sequences, BATCH_SIZE, SEQUENCE_LENGTH, VOCAB_SIZE), 
                    steps_per_epoch = steps_per_epoch, epochs=30, callbacks=[earlystop])
</code></pre>

<p>The training achieves an accuracy of ~0.1. When I apply the model to predict words for the 10 word snippets from the training data, the output is overwhelmingly the most frequent word, 'one'. </p>

<p>I tried a more complex model with 2 LSTM layers, 2 Dense layers. I tried to use a pretrained word embedding using gensim word2vec model. Invariably the accuracy is ~0.1 and most of the time the prediction is 'one'. </p>

<p>When I thought about, it kind of made sense. Predicting the most frequent class for imbalanced data gives high accuracy 'for free'. This is clearly a local minimum, but one that is hard to escape. 
The thing is, the algorithm doesn't minimize the accuracy, it minimizes the loss, which is categoricall_crossentropy, and it is supposed to work just fine for imbalanced data. But, perhaps, it is not always true, and there a different loss that can be used to deal better with imbalanced data?</p>