experiments with quite a complex model to see how good speech recognition accuracy we are able to achieve in these tasks. The projection layer maps words to 500-dimensional embeddings. Both the LSTM and highway network layers have 1500 outputs. With vocabularies larger than 100,000 elements we added a 500-unit feedforward layer before the output layer to reduce memory consumption. With class models mini-batches were 32 sequences and with other models 24 sequences of length 25. We also explored the possibility of training models with larger vocabularies using hierarchical softmax and sampling-based softmax. These approximations are explained in more detail in the following sections.

Output Normalization
--------------------

The final layer normalizes the output to provide a valid probability distribution over the output classes. Normally the softmax function is used:

$$\label{eq:softmax}
y_{t,i} = \frac{\exp(x_{t,i})}{\sum_j \exp(x_{t,j})}$$

At each time step $t$, the cross-entropy cost function requires computing the conditional probability of the target word only, $P(w_{t+1} \mid w_0 \ldots w_t)
= y_{t,w_{t+1}}$. Still all the activations $x_{t,j}$ are needed to explicitly normalize the probability distribution. This becomes computationally expensive, because vocabularies can be very large, and the cost of computing the normalization term scales linearly with the vocabulary size.

There has been a 