hierarchical softmax, sampling-based softmax, class-based models, and subword models in speech recognition on languages that are known for very large vocabularies. Both data sets contain around two million unique word forms. In our experiments where the training time was limited to 15 days, class-based NNLMs clearly exceeded the performance of word-based NNLMs in terms of perplexity and recognition accuracy. The best results were from subword models. In the Estonian task, the best subword vocabularies were quite large, and the best result was from a class-based subword model. We also test two methods for weighting separate language modeling data sets: weighted sampling, which has already been introduced in [@Schwenk:2005] and update weighting, which is a novel method.

All the neural network language modeling techniques presented in this paper have been implemented in the open-source toolkit TheanoLM [@Enarvi:2016], which we hope to lower the threshold of using neural network language models in speech recognition research.[^4] We implemented hierarchical softmax [@Goodman:2001], noise-contrastive estimation [@Gutmann:2010], and BlackOut [@Ji:2016] training criteria, and a lattice decoder that takes advantage of parallel computation using a GPU.

We use a fairly complex recurrent model consisting of an LSTM layer and a highway network to obtain state-of-the-art results, and 