128 are used with a fixed momentum of 0.99 and a fixed learning rate of 0.001. Gradients are clipped to the range $[-5, 5]$ for all weights during back-propagation. We use perplexity as our stopping criteria. The entire system has about 48 million parameters, and learning them on the sentiment dataset with our implementation takes about 20 minutes at 113 image-sentence pairs per second, while the original model on the MSCOCO dataset takes around 24 hours at 352 image-sentence pairs per second]{}. Given a new image, we predict the best caption by doing a beam-search with beam-size 5 for the best words at each position. We implementd the system on a multicore workstation with an Nvidia K40 GPU.

[[**[Dataset setup.]{}**]{}]{} The background RNN is learned on the MSCOCO training set [@chen2015microsoft] of 413K+ sentences on 82K+ images. We construct an additional set of caption with sentiments as described in Section \[sec:mturk\] using images from the MSCOCO validation partition. The [Pos]{} subset contains 2,873 positive sentences and 998 images for training, and another 2,019 sentences over 673 images for testing. The [Neg]{} subset contains 2,468 negative sentences and 997 images for training, and another 1,509 sentences over 503 images for testing. 