to the best of our knowledge, no existing work tries to compose image descriptions with desired sentiments. [Identifying sentiment in text is an active area of research[ [@pang2008opinion; @socherrecursive]]{}. Several [teams]{} [@Nakagawa2010; @Mcdonald2011] [designed sentence models with latent variables representing the sentiment.]{} [Our work focuses on]{} generating sentences and not explicitly modelling sentiment using hidden variables.]{}

Describing an Image with Sentiments {#sec:model}
===================================

Given an image $I$ and its $D_x$-dimensional visual feature ${\mathbf{x}}\in{{\mathbb R}}^{D_x}$, our goal is to generate a sequence of words (i.e. a caption) ${\mathbf{Y}}=\{{\mathbf{y}}_1,\cdots,{\mathbf{y}}_T\}$ to describe the image with a specific style, such as expressing sentiment. Here ${\mathbf{y}}_t\in\{0,1\}^V$ is 1-of-V encoded indicator vector for the $t^{th}$ word; $V$ is the size of the vocabulary; and $T$ is the length of the caption.

We assume [[[that]{}]{}]{} sentence generation involves two underlying mechanisms, one of which focuses on the factual description of the image while the other describes the image content with sentiments. [We formulate such caption generation process using a switching multi-modal language model, which sequentially generates words in a sentence.]{} [Formally]{}, we introduce a binary [sentiment]{} variable $s_t\in\{0,1\}$ for every word ${\mathbf{y}}_t$ to indicate which mechanism is used. At each time step $t$, our model produces the 