because the assumption was too far from the truth. Our class NNLMs are similar to Equation \[eq:class-ngram\], except that RNNs do not fix the context to $n$ previous words—the length of the history used to predict the next word is limited only by the mini-batch size.

Memory consumption becomes a problem when using GPUs for training, since current GPU boards typically have no more than 12 GB of memory. Each layer learns a weight matrix whose dimensionality is input size by output size. For example, an NNLM with one hidden layer of size 1,000 and a vocabulary of size 100,000 requires a 1,000 by 100,000 matrix on the input and output layer. Assuming 32-bit floats are used, such a matrix uses 400 MB of memory. In addition, temporary matrices are needed when propagating the data through the network. Memory required to store the weight matrices can be reduced by using a small projection layer and a small layer before the output layer, or factorizing a large weight into the product of two smaller matrices [@Sainath:2013]. Another possibility is to divide weight matrices to multiple GPUs. The size of the temporary data depends also on the mini-batch size.

We did the 