In this paper we study image captioning as a conditional GAN training, proposing both a context-aware LSTM captioner and co-attentive discriminator, which enforces semantic alignment between images and captions. We investigate the viability of two discrete GAN training methods: Self-critical Sequence Training (SCST) and Gumbel Straight-Through (ST) and demonstrate that SCST shows more stable gradient behavior and improved results over Gumbel ST. Significant progress has been made on the task of generating image descriptions using neural image captioning. Early systems were traditionally trained using cross-entropy (CE) loss minimization BID6 BID16 . Later, reinforcement learning techniques BID13 BID9 based on policy gradient methods were introduced to directly optimize metrics such as CIDEr or SPICE BID0 . Along a similar idea, BID14 introduced Self-critical Sequence Training (SCST), a light-weight variant of REINFORCE, which produced state of the art image captioning results using CIDEr as an optimization metric. To address the problem of sentence diversity and naturalness, image captioning has been explored in the framework of GANs. However, due to the discrete nature of text generation, GAN training remains challenging and has been generally tackled either with reinforcement learning techniques BID4 BID12 BID2 or by using Gumbel softmax relaxation BID5 , as in BID15 BID7 .Despite impressive advances, image captioning is far from being a solved task. It remains a challenge to satisfactorily bridge the semantic gap between image and captions to produce diverse, creative, and "human-like" captions. Although applying GANs to image captioning for promoting human-like captions is a very promising direction, the discrete nature of the text generation process makes it challenging to train such systems. The recent work of BID1 showed that the task of text generation for current discrete GAN models is difficult, often producing unsatisfactory results, and requires therefore new approaches and methods.In this paper, we propose a novel GAN-based framework for image captioning that enables better language composition, more accurate compositional alignment of image and text, and light-weight efficient training of discrete sequence GAN based on SCST. In summary, we demonstrated that SCST training for discrete GAN is a promissing new approach that outperforms the Gumbel relaxation in terms of training stability and the overall performance. Moreover, we showed that our context-aware attention gives larger gains as compared to the adaptive sentinel or the traditional visual attention. Finally, our co-attention model for discriminator compares favorably against the joint embedding architecture.
Image captioning as a conditional GAN training with novel architectures, also study two discrete GAN training methods.  An improved GAN model for image captioning that proposes a context-aware LSTM captioner, introduces a stronger co-attentive discriminator with better performance, and uses SCST for GAN training.