$$\begin{aligned}
L_t&(\Theta,{\mathbf{x}},{\mathbf{y}})=  \log p({\mathbf{y}}_t | {\mathbf{x}}, {\mathbf{y}}_{1:t-1})= \\ &\log [\gamma^0_t p({\mathbf{y}}_t|s_t=0,{\mathbf{x}},{\mathbf{y}}_{-t} ) + 
 \gamma^1_t p({\mathbf{y}}_t|s_t=1,{\mathbf{x}},{\mathbf{y}}_{-t} )].\nonumber   \end{aligned}$$ The overall learning objective function for incorporating word sentiment is a combination of a weighted log likelihood and the cross-entropy between $\gamma_t$ and [$\eta_t$]{}, $$\begin{aligned}
{\cal L}(\Theta,\mathcal{D}) &= -\sum_i\sum_t (1+ \lambda_\eta \eta^i_t ) 
[L_t(\Theta,{\mathbf{x}}^i,{\mathbf{y}}^i)  \label{eq:giantr}
\\ &+ \lambda_\gamma (\eta_t^i \log\gamma^{1,i}_t + (1-\eta_t^i) \log\gamma^{0,i}_t)  ]  + R(\Theta),\nonumber\\
R(\Theta)=&\frac{\lambda_\theta}{2}\|\Theta^1 - \Theta^0\|^2 
\label{eq:modreg}\end{aligned}$$ where $\lambda_\eta$ and $\lambda_\gamma$ are weight parameters, and $R(\Theta)$ is the regularization term with weight parameter $\lambda_\theta$. Intuitively,]{} when $\eta_t > 0$, i.e. the training sentence encounters [a sentiment word]{}, the likelihood weighting factor $\lambda_\eta \eta^i_t$ [increases the importance of $L_t$ in the overall likelihood; at the same time, the cross-entropy term $\lambda_\gamma (\eta_t^i \log\gamma^{1,i}_t + (1-\eta_t^i) \log\gamma^{0,i}_t) $ encourage switching variable $\gamma^1_t$ to be $>0$, emphasizing the new model. The regularized training finds a trade-off between the [data]{} likelihood and L2 difference between the current and base RNN, [and is one of the most competitive approaches in domain transfer [@schweikert2008empirical]]{}.]{}

[[**[Settings for model learning.]{}**]{}]{} We use stochastic gradient descent with backpropagation on mini-batches to optimize the RNNs. We apply dropout to the input of each step, which is either the image embedding 