Each of the test images has three positive and/or three negative captions.

[[**[Systems for comparison.]{}**]{}]{} The starting point of our model is the RNN with LSTM units and CNN input [@vinyals2015show] learned on the MS COCO training set only, denoted as [*CNN+RNN*]{}. [Two simple baselines [*ANP-Replace*]{} and [*ANP-Scoring*]{} use sentences generated by [*CNN+RNN*]{} and then add an adjective with strong sentiment to a random noun. [*ANP-Replace*]{} adds the most common adjective, in the sentiment captions for the chosen noun. [*ANP-Scoring*]{} uses multi-class logistic regression to select the most likely adjective for the chosen noun, given the Oxford VGG features.]{} The next model, denoted as [*RNN-Transfer*]{}, learns a fine-tuned RNN [on the sentiment dataset]{} with additional regularization from [*CNN+RNN*]{} [@schweikert2008empirical], [as in $R(\Theta)$ (cf. Eq )]{}. We name the [full switching RNN system as]{} [*SentiCap*]{}, which jointly learns [the RNN and the switching probability with word-level sentiments from Equation (\[eq:giantr\]).]{}

![image](fig/3x4){width=".7\textwidth"}

[[**[Evaluation metrics.]{}**]{}]{} We evaluate our system both with automatic metrics and with crowd-sourced judgements through Amazon Mechanical Turk. Automatic evaluation uses the [Bleu]{}, [Rouge]{}$_L$, [Meteor]{}, [Cide]{}$_r$ metrics from the Microsoft COCO evaluation software [@chen2015microsoft].

In our crowd-sourced evaluation task AMT workers are given an image and two automatically generated sentences 