switching RNN model based on a large dataset with factual captions and a small set with sentiment captions.]{}

**Learning a background multi-modal RNN.** We first train a CNN+RNN with a large dataset of image and caption pairs, denoted as $\mathcal{D}^0=\{({\mathbf{x}}_0^i,{\mathbf{y}}_0^i)\}_{i=1}^N$. $\Theta^0$ are learned by minimizing the negative log-likelihood of the caption words given images, [ $$\label{eqn:baselearn}
\resizebox{0.26\columnwidth}{!}{$   
    L^0(\Theta^0,\mathcal{D}^0)
= -$}\sum_i\sum_t \resizebox{0.5\columnwidth}{!}{$\log p({\mathbf{y}}^i_{0,t}|s_t=0,{\mathbf{x}}^i_0,{\mathbf{y}}^i_{0,1:t-1} ).$}
{\vspace{-0.mm}}$$ ]{}

**Learning from captions with sentiments.** [Based on the pre-trained CNN+RNN in Eq , we then learn the switching RNN using a small image caption dataset with a specific sentiment polarity, denoted as $\mathcal{D}=\{({\mathbf{x}}^i,{\mathbf{y}}^i,\eta^i)\}_{i=1}^M$, $M\ll N$. Here $\eta_t^i \in [0, 1]$ is the sentiment strength of the $t^{th}$ word in the $i$-th training sentence,]{} [being either positive or negative as specified in the training data.]{}

[We design a new training objective function to use word-level sentiment information for learning $\Theta^1$ and the switching weights ${\mathbf{W}}_s$, while keeping the pre-learned $\Theta^0$ fixed. For clarity, we denote the sentiment probability as: $$\begin{aligned}
\gamma^{0}_t = p(s_t = 0|{\mathbf{x}},{\mathbf{y}}_{1:t-1}), \quad\gamma^{1}_t = 1-\gamma^{0}_t;  \label{eq:switch}
{\vspace{-0.mm}}\end{aligned}$$ and the log likelihood of generating a new word ${\mathbf{y}}_t$ given image and word histories $({\mathbf{x}},{\mathbf{y}}_{1:t-1})$ as $L_t(\Theta,{\mathbf{x}},{\mathbf{y}})$, which can be written as (cf. Eq ), 