dog and the motocycle), or about their background settings (interaction with the dog at home, or how the motocycle came about).

![Example images with neural, positive () and negative () captions, by crowd workers in MSCOCO dataset [@chen2015microsoft] and this work (Section \[sec:mturk\]). []{data-label="fig:intro"}](fig/intro_example){width=".4\textwidth"}

In this paper, we describe a method, called SentiCap, to generate image captions with sentiments. We build upon the CNN+RNN (Convolution Neural Network + Recurrent Neural Network) recipe that has seen many recent successes [@donahue2015long; @Karpathy2015CVPR; @mao2014deep; @vinyals2015show; @xu2015show]. In particular, we propose a switching Recurrent Neural Network (RNN) model to represent sentiments. This model consists of two parallel RNNs – one represents a general background language model; another specialises in descriptions with sentiments. We design a novel word-level regularizer, so as to emphasize the sentiment words during training [and to optimally combine the two RNN streams]{} (Section \[sec:model\]). We have gathered a new dataset of several thousand captions with positive and negative sentiments by re-writing factual descriptions (Section \[sec:mturk\]). [Trained on 2000+ sentimental captions and 413K neutral captions, our switching RNN out-performs a range of heuristic and learned baselines in the number of emotional captions generated, and in a [[[variety]{}]{}]{} of subjective and human evaluation 