are slower to train.

Usually word vocabularies are limited to a *shortlist* of the most frequent words. A special token such as [`<unk>` ]{}can be used in place of any out-of-shortlist (OOS) words, which is necessary with RNN language models in particular. The NNLM can be combined with a large-vocabulary n-gram model to obtain a probability for every training word. The [`<unk>` ]{}probability represents the total probability mass of all OOS words, which can be distributed according to n-gram language model probabilities. An n-gram language model is convenient to integrate with a feedforward NNLM, which is a particular kind of n-gram model itself, but less trivial in our RNN decoder. It also becomes computationally demanding to normalize the resulting probability distribution correctly using a large n-gram model [@Park:2010].

In our baseline shortlist NNLMs we distribute the OOS probability according to a unigram model. When rescoring lattices, the output does not have to be a proper probability distribution. Assuming that the probability mass that the NNLM and n-gram models allocate to the OOS words are close to each other, a reasonable approximation is to replace the OOS probabilities with n-gram probabilities [@Park:2010]. We tried this, but did not get good results 