it very difficult to reliably estimate the probability of the rare words in new contexts. If we can cluster word forms into classes based on in which contexts they appear, we can get more reliable estimates for the class n-gram probabilities. In a class-based language model, the probability of a word within its class is usually modeled simply as the unigram probability of the word in the training data [@Brown:1992]:

$$\label{eq:class-ngram}
\begin{split}
&P(w_t \mid w_{t-n+1} \ldots w_{t-1}) = \\
&P(c(w_t) \mid c(w_{t-n+1}) \ldots c(w_{t-1})) P(w_t \mid c(w_t)),
\end{split}$$

where $c(w)$ is a function that maps a word to a class. This is also the model that we use in this article.

Statistical Methods for Clustering Words into Classes
-----------------------------------------------------

A common cost function for learning the word classes is the perplexity of a class bigram model, which is equivalent to using the log probability objective:

$$\label{eq:class-bigram}
\mathcal{L} = \sum_t [\log P(c(w_t) \mid c(w_{t-1})) + \log P(w_t \mid c(w_t))]$$

Finding the optimal clustering is computationally very challenging. Evaluating the cost involves summation over all adjacent classes in the training data [@Brown:1992]. The algorithms that have been proposed are suboptimal. Another approach that can be taken is to use knowledge about the 