image representation and [such]{} language models. Fang [et al.]{}  used a cascade of word detectors from images and a [max-ent]{} model. The Show and Tell [@vinyals2015show] system used an RNN as the language model, seeded by CNN image features. Xu [et al.]{}  estimated spatial attention as a latent variable, to make the Show and Tell system aware of local image information. Karpathy and Li[ ]{} used an [RNN to generate a sentence from the alignment between objects and words]{}. [Other work has]{} [employed multi-layer RNNs [@Chen_2015_CVPR; @donahue2015long]]{} [for image captioning.]{} [Most]{} RNN-based multimodal language models [use]{} the Long Short Term Memory (LSTM) unit that preserves long-term information [and prevents]{} overfitting [@hochreiter1997long]. We adopt one of the competitive systems [@vinyals2015show] – CNN+RNN with LSTM units as our basic multimodal sentence generation engine, due to its simplicity and computational efficiency.

[Researchers have modeled]{} how [an]{} image is presented, and what kind of response it is likely to elicit from viewers, such as analyzing the aesthetics and emotion in images [@murray2012ava; @joshi2011aesthetics]. More recently, the Visual SentiBank [@borth2013sentibank] system constructed a catalogue of Adjective-Noun-Pairs (ANPs) that are frequently used to describe online images. We build upon Visual SentiBank to construct sentiment vocabulary, but 