&\propto \exp({\mathbf{W}}_y^{k} {\mathbf{h}}_t^k) {\vspace{-0.mm}}\end{aligned}$$ where ${\mathbf{W}}_y^k \in {{\mathbb R}}^{D\times V}$ is a set of learned output weights.

[The]{} sentiment switching model generates the probability of switching [between the two RNN streams]{} at each time $t$, [with]{} a single layer network taking the hidden states of both RNNs as input: $$\begin{aligned}
p(s_t=1 | {\mathbf{x}}, {\mathbf{y}}_{1:t-1}) &= \sigma({\mathbf{W}}_s[{\mathbf{h}}^0_t;{\mathbf{h}}^1_t])\label{eq:switch}
{\vspace{-0.mm}}\end{aligned}$$ where ${\mathbf{W}}_s$ is the weight matrix for the hidden states.

[An illustration of this sentiment switching model is in Figure \[fig:rnn\].]{} In summary, the parameter set for each RNN ($k=\{0,1\}$) is [$\Theta^k=\{{\mathbf{T}}^k, {\mathbf{W}}_y^k, {\mathbf{E}}^k, {\mathbf{W}}_x^k\}$]{}, and that of the switching RNN is [$\Theta=\Theta^0\cup\Theta^1\cup {\mathbf{W}}_s$]{}. We have tried including ${\mathbf{x}}$ for learning $p(s_t|{\mathbf{x}}, {\mathbf{y}}_{1:t-1})$ but found no benefit.

![Illustration of the switching RNN model for captions with sentiment. Lines with diamonds denote projections with learned weights. LSTM cells are described in Eq \[eq:lstm\]. $\gamma_t^0$ and $\gamma_t^1$ are probabilities of sentiment switch defined in Eq (\[eq:switch\]) and act as gating functions for the two streams. []{data-label="fig:rnn"}](fig/parrallel_rnn){width=".35\textwidth"}

Learning the Switching RNN Model
--------------------------------

[One of the key challenges is to design a learning scheme for $p(s_t | {\mathbf{x}}, {\mathbf{y}}_{1:t-1})$ and two CNN+RNN components. We take a two-stage learning approach to estimate the parameters $\Theta$ in our 