by the output of a sigmoid gate, the system learns to discard unimportant elements of the vector depending on the gate’s input.

An LSTM layer uses three gates to select what information to pass from the previous time step to the next time step unmodified, and what information to modify. The same idea can be used to select what information to pass to the next layer. Highway networks [@Srivastava:2015] use gates to facilitate information flow across many layers. At its simplest, only one gate is needed. In the feedforward case, there is only one input, $x_t$, and the gate needs only one weight matrix, $W_\sigma$. The gate learns to select between the layer’s input and its activation:

$$\begin{aligned}
\label{eq:highway-network}
g(x_t) &= \sigma(W_\sigma x_t + b_\sigma) \\
y_t    &= g(x_t) \odot \tanh(W x_t + b) + (1 - g(x_t)) \odot x_t
\end{aligned}$$

While LSTM helps propagation of activations and gradients in recurrent networks, deep networks benefit from highway connections. We did not notice much improvement by stacking multiple LSTM layers on top of each other. While we did not have the possibility to systematically explore different network architectures, one LSTM layer followed by a highway network seemed to perform well. The 