in perplexity for the exchange algorithm over Brown clustering, while clusterings created from distributed word representations have not worked as well [@Botros:2015; @Dehdari:2016; @Song:2017]. We present additionally a novel rule-based algorithm that clusters colloquial Finnish word forms, and also evaluate word error rate. Surprisingly, class-based n-gram models perform better than word models in terms of perplexity and speech recognition accuracy in conversational Finnish and Estonian.

Word classes and subword units are especially attractive in NNLMs, because the vocabulary size has a great impact on the memory consumption and computational complexity. The size of the input layer projection matrix and the output layer weight matrix, as well as the time required to normalize the output probabilities using softmax, have a linear dependency on the vocabulary size. The output normalization can also be made more efficient by using one of the several methods that try to approximate the full softmax, either by modifying the network structure or the training objective. So far the only comparison of these approximations for large-vocabulary NNLMs that we are aware of is in [@Chen:2016]. They found hierarchical softmax to perform best in terms of perplexity with a vocabulary of 800,000 words and a feedforward network.

We compare 