“+” if and only if the previous morph ends with a “+”) are allowed [@Smit:2017]. After decoding the ASR results, the morphs are joined together to form words for scoring.

Neural Network Language Models
==============================

Recurrent neural networks are known to work well for modeling language, as they can capture the long-term dependencies neglected by n-gram models [@Mikolov:2010]. Especially the subword-based approach should benefit from this capability of modeling long contexts. In this article we experiment with language models that are based on LSTMs and highway networks. These are layer types that use *sigmoid gates* to control information flow. The gates are optimized along with the rest of the neural network parameters, and learn to pass the relevant activations over long distances.

LSTM [@Hochreiter:1997] is a recurrent layer type. Each gate can be seen as an RNN layer with two weight matrices, $W$ and $U$, a bias vector $b$, and sigmoid activation. The output of a gate at time step $t$ is

$$g(x_t, h_{t-1}) = \sigma(W x_t + U h_{t-1} + b),$$

where $x_t$ is the output vector of the previous layer and $h_{t-1}$ is the LSTM layer state vector from the previous time step. When a signal is multiplied 