probability of ${\mathbf{y}}_t$ and the current sentiment variable $s_t$ given the image feature ${\mathbf{x}}$ and the previous words ${\mathbf{y}}_{1:t-1}$, denoted by $p({\mathbf{y}}_t,s_t|{\mathbf{x}},\mathbf{y}_{1:t-1})$. We generate the word probability by marginalizing [over]{} the sentiment variable $s_t$: $$\resizebox{0.25\columnwidth}{!}{$
p({\mathbf{y}}_t|{\mathbf{x}},{\mathbf{y}}_{1:t-1})$}=\sum_{s_t}\resizebox{0.525\columnwidth}{!}{$p({\mathbf{y}}_t|s_t,{\mathbf{x}},{\mathbf{y}}_{1:t-1})p(s_t|{\mathbf{x}},{\mathbf{y}}_{1:t-1})
$}
\label{eq:bayesrule_s}
{\vspace{-0.mm}}$$ Here $p({\mathbf{y}}_t|s_t,\cdot)$ is the caption model conditioned on the sentiment variable and [$p(s_t|\cdot)$ is the probability of the word sentiment]{}. The rest of this section will introduce these components [and model learning]{} in detail.

Switching RNNs for Sentiment Captions {#ssec:sentimodel}
-------------------------------------

We adopt a joint CNN+RNN architecture [@vinyals2015show] in the conditional caption model. Our full model combines two CNN+RNNs running in parallel: one capturing the [factual word generation (referred to as the background language model)]{}, the other specializing in words with sentiment. [The full model is a switching RNN, in which the variable $s_t$ functions as a switching gate.]{} This model design aims to learn sentiments well, despite data sparsity – using only a small dataset [of]{} image description with sentiments (Section \[sec:mturk\]), with the help from millions of image-sentence pairs that factually describe pictures [@chen2015microsoft].

Each RNN stream [consists of]{} a series of LSTM units. Formally, we denote the $D$-dimensional hidden state of an LSTM as ${\mathbf{h}}_t \in {{\mathbb R}}^D$, 