its memory cell as ${\mathbf{c}}_t\in {{\mathbb R}}^D$, the input, output, forget gates as ${\mathbf{i}}_t,~{\mathbf{o}}_t,~{\mathbf{f}}_t \in {{\mathbb R}}^D$, [respectively.]{} [Let $k$ indicate which RNN stream it is,]{} the LSTM can be implemented as: $$\begin{aligned}
&\begin{pmatrix} {\mathbf{i}}_t^k \\ {\mathbf{f}}_t^k \\ {\mathbf{o}}_t^k \\ {\mathbf{g}}_t^k \end{pmatrix}  = 
\begin{pmatrix} \sigma \\ \sigma \\ \sigma \\\tanh \end{pmatrix} \mathlarger{{\mathbf{T}}}^k{}
\begin{pmatrix} {\mathbf{E}}^k{\mathbf{y}}_{t-1} \\ {\mathbf{h}}^k_{t-1} \end{pmatrix}
\label{eq:lstm}\\
&{\mathbf{c}}_t^k = {\mathbf{f}}_t^k \odot {\mathbf{c}}_{t-1}^k + {\mathbf{i}}_t^k \odot {\mathbf{g}}_t^k, \quad
{\mathbf{h}}_t^k = {\mathbf{o}}_t^k \odot {{\mathbf{c}}_t^k}. \nonumber
{\vspace{-0.mm}}\end{aligned}$$ Here $\sigma(\chi)$ is the sigmoid function $1/(1+e^{-\chi})$; $tanh$ is the hyperbolic tangent function; [${\mathbf{T}}^k\in{{\mathbb R}}^{4D\times2D}$]{} is a set of learned weights; ${\mathbf{g}}^k_t\in{{\mathbb R}}^D$ is the input to the memory cell; [${\mathbf{E}}^k \in {{\mathbb R}}^{D\times V}$]{} is a learned embedding matrix [in model $k$]{}, and ${\mathbf{E}}^k{\mathbf{y}}_t$ is the embedding vector of the word ${\mathbf{y}}_t$.

To incorporate image information, we use an image representation $\hat{\mathbf{x}}={\mathbf{W}}_x{\mathbf{x}}$ as the word embedding ${\mathbf{Ey}}_0$ when $t=1$, where ${\mathbf{x}}$ is a high-dimensional image feature extracted from a convolutional neural network [@Simonyan2015], and ${\mathbf{W}}_x$ is a learned embedding matrix. Note that the LSTM hidden state ${\mathbf{h}}_t^k$ summarizes ${\mathbf{y}}_{1:t-1}$ and ${\mathbf{x}}$. The conditional probability of the output caption words depends on the hidden state of the corresponding LSTM, $$\begin{aligned}
p({\mathbf{y}}_t | s_t=k, {\mathbf{x}}, {\mathbf{y}}_{1:t-1}) 