$\hat{\mathbf{x}}$ [for $t=1$]{} or the word embedding ${\mathbf{E}}^k{\mathbf{y}}_{t-1}$ and the hidden output ${\mathbf{h}}^k_{t-1}$ from time $t-1$, for both the background and sentiment streams $k=0,1$.

We learn models for positive and negative sentiments separately, due to the observation that either sentiment could be valid for the majority of images (Section \[sec:mturk\]). [We initialize $\Theta^1$ as $\Theta^0$ and use the following gradient of to minimize ${\cal L}(\Theta,\mathcal{D})$ with respect to $\Theta^1$ and ${\mathbf{W}}_s$, holding $\Theta^0$ fixed. $$\begin{aligned}
{\frac{\partial \cal L}{\partial \Theta}} = &-\sum_i\sum_t (1+ \lambda_\eta \eta^i_t ) [{\frac{\partial L_t}{\partial \Theta}} \nonumber\\&+ \lambda_\gamma ( \frac{\eta_t^i}{\gamma^{1,i}_t}{\frac{\partial \gamma^{1,i}_t}{\partial \Theta}} + \frac{1-\eta_t^i}{\gamma^{0,i}_t} {\frac{\partial \gamma^{0,i}_t}{\partial \Theta}}) ] +  {\frac{\partial R(\Theta)}{\partial \Theta}}
{\vspace{-0.mm}}\end{aligned}$$ ]{}Here ${\frac{\partial L_t}{\partial \Theta}},{\frac{\partial \gamma^{0,i}_t}{\partial \Theta}},\text{ and }{\frac{\partial \gamma^{1,i}_t}{\partial \Theta}}$ are computed through differentiating across Equations (\[eq:bayesrule\_s\])–(\[eq:switch\]). During training, we set $\eta_t=1$ when word ${\mathbf{y}}_t$ is part of an ANP with the target sentiment polarity, otherwise $\eta_t=0$. We also include a default L2-norm regularization for neural network tuning $|\Theta|^2$ with a small weight ($10^{-8}$). We automatically search for the hyperparameters $\lambda_\theta$, $\lambda_\eta$ and $\lambda_\gamma$ on a validation set using Whetlab [@snoek2012practical].

An Image Caption Dataset with Sentiments {#sec:mturk}
========================================

![image](fig/gt_rating){width=".9\textwidth"}

In order to learn the association between images and captions with sentiments, we build a 