VITAL: A Visual Interpretation on Text with Adversarial Learning for Image Labeling
Tao Hu Chengjiang Long Leheng Zhang Chunxia Xiao
cs.CV

























1,3]Tao Hu
2]Chengjiang LongThis work was co-supervised by Chengjiang Long and Chunxia Xiao.
1]Leheng Zhang
1]Chunxia Xiao
[1]School of Computer Science, Wuhan University, Wuhan, Hubei, China
[2]Kitware Inc. Clifton Park, NY, USA
[3]School of Information Engineering, Hubei University for Nationalities, Enshi, Hubei, China
[ ]hutao_es@foxmail.com, chengjiang.long@kitware.com, 375602133@qq.com, cxxiao@whu.edu.cn
VITAL: A Visual Interpretation on Text with Adversarial Learning for Image Labeling
    [
    December 30, 2023
===================================================================================





   In this paper, we propose a novel way to interpret text information by extracting visual feature presentation from multiple high-resolution and photo-realistic synthetic images generated by Text-to-image Generative Adversarial Network (GAN) to improve the performance of image labeling. Firstly, we design a stacked Generative Multi-Adversarial Network (GMAN), StackGMAN++, a modified version of the current state-of-the-art Text-to-image GAN, StackGAN++,  
to generate multiple synthetic images with various prior noises conditioned on a text. And then we extract deep visual features from the generated synthetic images to explore the underlying visual concepts for text. Finally, we combine image-level visual feature, text-level feature and visual features based on synthetic images together to predict labels for images. We conduct experiments on two benchmark datasets, i.e., the Oxford 102 Category Flower Dataset  
and the Caltech-UCSD Birds-200-2011 Dataset，
and the experimental results clearly demonstrate the efficacy of our proposed approach.





§ INTRODUCTION


Nowadays, images are being taken and shared to be commented at an unprecedented rate among social networks like Facebook, Twitter, and Flickr. To help users efficiently organize and manage such media data from a very huge collection, it is necessary and practical to collect labeled visual datasets at large scale to develop automatic tools with robust machine learning approaches <cit.>. However, most of the current annotation platforms like Amazon Mechanical Turk <cit.> and LabelMe <cit.> do not make full use of the text information to aid the labeling problems, although most images on the web carry rich text information which includes informative and semantic signals like who took the photo and, where and with whom. 



Prior work takes advantage of text context information to improve image classification by various treatments like selecting top frequently used words and user-generated tags <cit.>, extracting text-level feature representation with Text Convolutional Neural Networks (CNNs) <cit.>, and exploring visual feature representation from a set of neighbor images <cit.> defined based on Jaccard similarity between image metadata especially tags extracted from a text. The intuition behind is that images with similar text context information tend to depict similar scenes.

Inspired by the development of Text-to-image Generative Adversarial Networks (GANs) <cit.>, which is able to generate high-resolution and photo-realistic synthetic images conditioned on a text, we propose a novel visual interpretation on text with adversarial learning, named as “VITAL", by interpreting text information with visual concepts extracted from a series of visually plausible synthetic images generated by Text-to-image GANs. 

Unlike Johnson et al.'s work <cit.> which uses a set of nearest neighbor images based on Jaccard similarity bewteen the text of query image and texts of training images, our proposed VITAL approach generates synthetic images conditioned on only the text of query image without using text from any other images.


As illustrated in Figure <ref>, the color information of nearest neighbor images to represent text are not well consistent with the real image, while synthetic images generated in our VITAL are able to not only cover most of the content in text information, but also provide much information about the background which is underlying in text information. This is also consistent with our human understanding to text information. 

“As there are 1000 Hamlets, there are 1000 readers." Usually, given a text that describes a specific scene, different readers can image different relevant visual scenes in their brains. Obviously, one synthetic image is not sufficient to simulate what multiple readers can visually interpret from a single text information. In principle, any text-to-
image GANs can be extended and incorporated into our VITAL framework and in this paper we just start to extend one current state-of-the-art Text-to-image GAN, StackGAN++ <cit.>, dubbed StackGMAN++ in brief with “M" indicating multiple generators, to extend the number of generators from the original 1 to K with different noise priors at each stage. The intuition behind is that each reader interprets Hamlets based on his/her own prior knowledge, and different prior knowledge leads to a different image of Hamlet in his/her mind. 

With K generators, we are able to generate K synthetic images. Since ResNet <cit.> has demonstrated successes in a vast of vision applications, we apply it to extract visual feature for each synthetic image. It worth emphasizing that we care much more about the common visual representation among these K synthetic images rather than each individual. Therefore, to obtain a compact visual feature representation, we first apply an affine transformation with a ReLU layer to adjust feature maps and reduce the channels before we apply an element-wise pooling to extract the common feature representation. 

Considering that feature fusion has been proved to be able to effectively improve the performance of image labeling, we combine the real image feature extracted from an image-level CNN, text feature extracted from a text-level CNN, and the common feature representation from K synthetic images by concatenation and feed them into a fully connected layer as a classification for image labeling <cit.>.



To sum up, our paper has three contributions:


  * We propose a novel way to interpret text information with K visually plausible synthetic images generated via our StackGMAN++ derived by modifying the number of generators at each stage from 1 to K on StackGAN++ associated with different noise priors.

  * We extract a common and compact visual feature representation from synthetic images conditioned on a text, and combine it with an image-level feature from real image, and a text-level feature together by conctatenatenation to boost the performance of image labeling. 

  * The experiments on two benchmark datasets have demonstrated that our proposed approach outperforms the state-of-art method <cit.>.





§ RELATED WORK

The related work can be divided into two categories: text information for image labeling and Text-to-image GANs.

Text information for image labeling. As an important source for multimedia, text information has been studied and combined with image content to improve the accuracy of image labeling, because it provides informative and semantic signals <cit.> like who took the photo and, where and with whom. McAuley et al. <cit.> selected top-1000 most frequently occurring words from the texts in the training set as tags <cit.>, explored pairwise social relations and applied a CRF-based structural learning approach for multi-label image annotation, which demonstrates impressive results although only metadata is used without any visual features. Following <cit.>, Johnson et al. <cit.> proposed a deep convolutional neural network to combine both the visual information of images and their neighbor images defined based on the shared similar metadata especially tags <cit.> from a text. With multiple text CNNs <cit.> emerged, Long et al. <cit.> proposed to extract deep text-level features rather than user-generated tags for text representation in image labeling. It worths mentioning that Johnson et al.'s work <cit.> is a visual interpretation of text, although neighbor images depend too much on the density distribution of training data. Different from <cit.>, we propose to use a Text-to-image GAN to generate high-resolution and photo-realistic synthetic images to extract visual feature representation for the text of query image only, which we expect to be complementary to both visual feature on real images and textual feature on a text.

Text-to-image GANs are proposed to effectively translate visual concepts from characters to pixels, with the aim to bridge these advances in text and image modeling <cit.>. Based on text descriptions, Reed et al. <cit.> was able to generate plausible 64 × 64 images for birds and flowers, and 128 × 128 images were successfully generated by utilizing additional annotations on object part locations <cit.>. To generate high-resolution (e.g., 227 × 227) and photo-realistic images, Nguyen et al. <cit.> proposed to generate images conditioned on a text using an approximate Langevin sampling approach with an iterative optimization process. Zhang et al. presented a Stacked GANs (StackGAN) <cit.> to generate 256 × 256 photo-realistic synthetic images with two stages, in which low-resolution images are generated at first stage, and then more details are added in the second stage to form high-resolution images with better quality by utilizing an encoder-decoder network before the upsampling layers. And its imporoved version StackGANN++ <cit.> extends to multi-stage GANs with multiple generators and multiple discriminators arranged in a tree-like structure. Based on StackGAN and StackGAN++, Xu et al. <cit.> adopted attention mechanism to generate synthetic images from fine-grained text with Attention Generative Adversarial Networks. We argue that high-resolution and photo-realistic synthetic images should be beneficial for us to extract visual representation for a text. Hence we modify the StackGAN++ to be a Generative Multi-Adverisal Network (GMAN), StackGMAN++, to generate K synthetic images with different generators. Different from Durugkar et al.'s  <cit.> GMAN with one generator and multiple discriminators, our StackGMAN++ uses multiple generators with various prior noise vectors and one discriminator at each stage to generate multiple synthetic images, to well represent visual concepts embedded in text information. 






§ PROPOSED METHOD


As illustrated in Figure <ref>, our proposed framework consists of three key components, i.e., StackGMAN++ to generate K synthetic images, extract visual feature representation from the synthetic images, and combine the synthetic image feature with real image feature and text feature to conduct a classification task for image labeling. We are going to discuss with details in the following subsections.



 §.§ StackGMAN++ to generate K synthetic images

Text-to-image Generative Adversarial Networks (GANs) <cit.> are proved to be able to generate visually-plausible images conditioned on a text. We start on StackGAN++ <cit.> because it is able to generate high-resolution and
photo-realistic images. 
Note there are two implementation versions for StackGAN++, i.e., StackGAN-v1 and StackGAN-v2. Our StackGMAN++ is derived from StackGAN-v2 which extends StackGAN-v1 from two stages to multiple stages and organizes generators and discriminators in a tree-like structure. 

Intuitively, given a text information, different people may imagine a different visual scene, and one text-based synthetic image is not sufficient to cover the underlying information behind the text itself. Therefore, to better explore the visual feature representation of any text information, we proposed to generate K synthetic images for each text information. 

Different from StackGAN-v2 which takes only one noise vector z as the input and has a generator G_i and a corresponding discriminator D_i for i-th branch with different scales, our StackGMAN++ takes a conditional vector c which is defined based on the text embedding φ_t in <cit.> and K different prior noise vectors z_1, …, z_K (we smaple values for each z_k from a normal distribution) as input, and at each stage i, we design one discriminator D_i and K generators  G_i^1, …, G_i^K to generate synthetic images at a certain scale. We pass the hidden feature h_i^k for each generator G_i^k by a non-linear transformation,

    h_i^k = {    F_i^k(c, z_k)                       i = 0
        F_i^k(h_i-1^k, c)       i = 1, …, m
    .

where h_i^k represents hidden features for the k-th generator G_i^k at the i-th branch, m is the total number of branches, and F_i^k is modeled as the corresponding neural network. In order to encourage the generators to draw images with more details according to the conditioning variables, c is concatenated to the hidden features h_i-1^k as the inputs of F_i^k for calculating h_i^k. Based on hidden features at different layers (h_1^k, …, h_m^k), generators G_1^k, …, G_m^k can generate synthetic images of small-to-large scales

    s_i^k = G_i^k(h_i^k), i = 1, …, m.


For the training purpose, we define the loss functions with joint conditional and unconditional distribution approximation at each stage i as following:

    ℒ_D_i   = K 𝔼_ x_i ∼ p_data_i [log D_i( x_i)] + 
       ∑_k=1^K 𝔼_s_i^k ∼ p_G_i^k [log(1 - D_i(s_i^k)] + 
        + K 𝔼_ x_i ∼ p_data_i [log D_i( x_i, c)] + 
       ∑_k=1^K 𝔼_s_i^k ∼ p_G_i^k [log(1 - D_i(s_i^k, c)],


    ℒ_G_i^1, …, G_i^K   = ∑_k=1^K 𝔼_s_i^k ∼ p_G_i^k [-log D_i(s_i^k)] 
       + ∑_k=1^K 𝔼_s_i^k ∼ p_G_i^k [-log D_i(s_i^k, c)]

where x_i is from the true image distribution p_data_i at the i-th scale, and s_i^k is from the model distribution p_G_i^k at the same scale. Then the discriminator D_i and generators G_i^1, …, G_i^K at i-th stage can be optimized in a joint form by alternatively maximizing ℒ_D_i and minimizing ℒ_G_i^1, …, G_i^K until convergence.


Based on StackGMAN++, 
we are able to generate K high-resolution and photo-realistic synthetic images s = {s^1, …, s^K} with s^k = G^k(s_m-1^k), and the size of each image is 256 × 256.





Discussion: we extend StackGAN++ to StackGMAN++ and joint learn the model with a united loss function to generate the diverse synthetic images. Note that the K generators in StackGMAN++ share weights. In this way, we can control the training cost when compared to train K StackGAN++ separately. We observe that training a StackGMANN++ with shared weights is less expensive than training K StackGAN++, and the K synthetic images generated by StackGMANN++ are more diverse. 



 §.§ Visual representation for text Information

Given a text t, we are able to generate K synthetic images s={s^1, …, s^K} with our StackGMAN++. Then we feed K generated synthetic images into ResNet <cit.> to extract visual feature. Note that we use ResNet as feature extractor ϕ in the pretrained ResNet model <cit.>, and get a feature map of the second last layer of ϕ(s^k) of 7 × 7 × 2048 size for each synthetic image s^k, respectively. 



In order to fuse visual features for these K synthetic image, we compute an h-dimensional hidden state for each image by applying an affine transformation and an element-wise ReLU nonlinearity σ(ε) = max(0, ε) to its feature. To let the model treat hidden states for each synthetic image differently, we apply distinct transformations to ϕ(s^k) with parameters W_k ∈ℝ^d× h and b_k ∈ℝ^h, and then we arrives at hidden states v_s^k∈ℝ^h for s^k ∈ s. To generate as single hidden state v_s ∈ℝ^h for all the synthetic images s, we apply an element-wise max-pooling on each v_s^k so that v_s = max_k  v_s^k, i.e., 

    v_s = max_k(σ(W_kϕ(s^k) + b_k))

and pass it to be included into the final combined feature representation with image feature and text feature. 



Discussion: we choose synthetic images rather than visual features because we want to visually interpret the text with high quality synthetic images so that the human can view directly, while visual features may miss some details.



 §.§ Feature fusion for image labeling


As shown in Figure <ref>, besides the visual feature representation v_s on K synthetic images, we also include the text-level feature v_t and image-level feature v_x together to better explore both image and text information to improve the quality for image labeling. In this paper, we choose to use the second last layer of ResNet <cit.> to extract the visual feature v_x for real image x because ResNet has been proved successful in most visual application tasks. 

Regarding text information, a standard deep learning model for text classification and sentiment analysis uses a word embedding layer and one-dimensional convolutional neural network <cit.>. The model can be expanded by using multiple parallel convolutional neural networks that read the source document using different kernel sizes. This, in effect, creates a multichannel convolutional neural network for text that reads text using different n-gram sizes (groups of words). We follow Kim's multi-channel model to implement a merged model with 3 text CNNs with kernels of different sizes, denoted as 3Text-CNNs, as illustrated in Figure <ref>,

to extract 512-dimensional text-level feature v_t from the second last layer. 

All these three kinds of features are combined to form the final feature representation by concatenation as ( v_x,  v_x̂,  v_t) and feed them into a fully connected layer to conduct a classification task as image labeling.





 §.§ Implementation details


The parameters need to be learned include the parameters in mK generators and m discriminators in StackGMAN++, parameters in ResNet-50s, 3Text-CNNs, the affine transformation parameter W_x̂ and b_x̂, and the parameters in the last fully connected layer. Note our training procedure is divided into two phases. At Phase I, we use pairs of text and its corresponding image to train our StackGMAN++ in an alternative optimization procedure until it convergence. Then at Phase II, we apply the trained StackGMAN++ to generate K high-resolution and photo-realistic synthetic images. Then we feed n real images, nK synthetic images and n text to learn the rest of parameters in the entire framework with cross-entropy loss function. 

Note that we follow the tricks in StackGAN-v2 to train StackGMAN++ at each stage at Phase I with a batch size of 12 for
350000 iterations. At Phase II, with a minibatch size of 50, we initialize all parameters with pre-trained models (ResNet and 3Text-CNNs) and use stochastic gradient descent with a fixed learning rate, RMSProp, as the optimization.




§ EXPERIMENTS




Our experiments are conducted on two datasets, i.e., the Oxford 102 Category Flower Dataset <cit.>,  and the Caltech-UCSD Birds-200-2011 Dataset <cit.>. 

We use accuracy as the metric to measure performance of image labeling.



 §.§ Experiments on the  Oxford 102 Category Flower Dataset


The Oxford 102 Category Flower Dataset <cit.> consists of 8,189 images from 102 categories of flowers which commonly occurs in the United Kingdom, and each category has 40 to 258 images. The images have large scales, pose and light variations. The text context information is provided by <cit.> with 10 descriptions for each image. Due to the limit space, we plot text using only 1 sentence and use the symbol “......" to represent the rest 9 sentences in this paper. We train our StackGMAN++ with text and the corresponding real images. With the learned StackGMAN++, we are able to generate K visual plausible synthetic images conditioned on a text for experiments. 





  §.§.§ Effectiveness of StackGMAN++


To verify the effectiveness of our StackGMAN++, we conduct experiments to check visual concept consistency between the generated synthetic images by StackGMAN++ and the corresponding real images, and the effectiveness of visual feature representation on K synthetic images. 

Visual concept consistency between synthetic and real images. We firstly visualize the generated synthetic images conditioned on a text and measure the correlation between our generated synthetic images with the corresponding images in a visual feature space.

As shown in Figure <ref>, our generated K synthetic images are able to not only cover main content elements in the text, but also provide underlying background and other rich visual information like size, shape and pose variations which are not mentioned in the text, due to various prior noise vector z_k. These observations are consistent with our human's behavior to interpret a text based on his/her prior knowledge. In other words, even given the same text, different people with different growing backgrounds will imagine different visual pictures in their brains. Such diverse interpretations are complementary to each other and can be merged to formulate a more representative format.

We also measure the correlation between our generated synthetic images conditioned on a text with the corresponding real images with two distance metrics, i.e., Hamming distance and Cosine similarity, between their visual feature vectors extracted by ResNet <cit.>. For Hamming distance in the range [0, 1], smaller value means the higher similarity between images.  
For Cosine similarity in the range [-1, 1], the value closer to 1.0 means two compared images are more correlative to each other in the given feature space. We plot both Hamming distance values and Cosine similarity value between our generated synthetic images and the corresponding real images in Figure <ref>. As we observe, none of Hamming distances is larger than 0.15 and all Cosine distance values are over 0.70, which indicates high correlation between each synthetic image and the corresponding real image in the visual feature space.







Effectiveness of visual feature representation on K synthetic images. 
As stated in Section <ref>, our VITAL uses the feature combinations ( v_x,  v_s,  v_t) where v_x and v_s indicate the visual feature representation extracted by ResNet <cit.> on real image, and synthetic images, respectively, and v_t represents the text-level feature extracted from 3Text-CNNs <cit.>. We then develop two different baseline algorithms with two different feature combinations, i.e., v_s only and ( v_x,  v_s). 
For notation simplification, we denote our proposed method to be VITAL-RST where “-RST" represents the feature combinations ( v_x,  v_s,  v_t). We further denote the first to second baseline algorithm to be VITAL-S and VITAL-RS, respectively. 

We conduct a group of experiments by setting K to be various values, i.e., 1, 2, 3 and 5. The results are summarized in the Figure <ref>. Apparently, for any algorithm among VITAL-S, VITAL-RS, and VITAL-RST, the performance accuracy goes up when the value of K increases. This indicates that multiple synthetic images are complementary to be used for extracting visual concepts embedded in text and boosting the accuracy for image labeling.  

Note that K is also the number of generators in each stage in our StackGMAN++. Therefore, when K=1, our StackGMAN++ is equivalent to StackGAN++ with its implementation version StackGAN-v2. Obviously, all these three algorithms with K (especially when K > 1) synthetic images generated by our StackGMAN++ always work better than using only synthetic images generated by StackGANN++. The observation strongly demonstrates the effectiveness and robustness of our StackGMAN++.




  §.§.§ Performance comparison






We compare our proposed VITAL-RST with Johnson et al.'s Convolutional Neural Network with Nearest Neighborhood <cit.>, denoted as “JCNN-NN", which explores the related neighbor images to interpret image metadata especially including tags from text information. To our best acknowledge, JCNN-NN is the most closely related work to our VITAL because it can be interpreted as a visual interpretation of image metadata with the related neighbor images. To make sure the comparison is fair, we utilize the same ResNet <cit.> as the visual feature extractor in JCNN-NN. In addition, we add two simple baseline algorithms, i.e., ResNet <cit.> and 3Text-CNNs <cit.>, which indicate using v_x only and using v_t only, respectively.

To clarify, we do not compare our VITAL with StackGMAN++ to other adversarial learning methods because this is not our focus and our focus is how to interpret text for image labeling by extending and applying the existing text-to-image GANs. We run experiments on the Oxford 102 Category Flower Dataset. Note that we also run JCNN-NN with five different number of neighbor images, as shown in Figure <ref>, from which we can find K=3 is the best choice for JCNN-NN.  

We conduct the experiments repeatedly for 10 times with 10 different random data split to form the training and testing sets and the results with all six algorithms are summarized in Table <ref>, from which we can observe: (1) 3Text-CNNs performs much worse than ResNet by a half and this conveys a clue that text on the dataset is a little weaker when compared with image content; (2) with single feature representation, our VITAL-S works much better than 3Text-CNNs and close to ResNet, 
which indicates visual feature extracted on synthetic images is more representative then text-feature; (3) combining with real image feature, VITAL-RS is able to improve the performance by 4.01% from ResNet, and works better than JCNN-NN <cit.> and ResNet <cit.>+3Text-CNNs <cit.> (i.e., a combination of RT without involving VITAL), which shows that our visual interpretation on text is more effective and robust than using a set of neighbor images defined based on the Jaccard similarity between image metadata especially tags extracted from text; 
and (4) combining with both real image feature and text feature, VITAL-RST performs the best. Apparently, the visual interpretation in our VITAL is robust and the visual synthetic image feature is complementary to both visual real image feature and text feature. 


















 §.§ Experiments on the Caltech-UCSD Birds-200-2011 Dataset


The Caltech-UCSD Birds-200-2011 Dataset consists of 11,169 
bird images from 200 categories and each each category has 60 images averagely. We randomly select 9,935 images for training, and use the resting 1,234 images for testing. The dataset is very chanllenging because it contains images with multiple objects and various backgrounds. We train our StackGMAN++ with K=5 and use it to generate synthetic images on the text with 10 descriptions for each real image to conduct the experimental evaluation. 













We repeat the experiments with 10 different random training/testing data splitting and summarize the results in Table <ref>, which can be observed from four aspects.  (1) ResNet performs much better than 3Text-CNNs, which indicates image is more representative than text content. (3) Using text information only, our VITAL-S still outperforms 3Text-CNNs. (3) VITAL-RST performs a little better than VITAL-RS which works much better than VITAL-S. Again, this observation confirms the complementary relationship between three kinds of feature representations. (4) Our VITAL-RS and VITAL-RST, performs better than JCNN-NN and ResNet+3Text-CNNs, which suggests that our proposed VITAL is good at visual interpretation on a text by extracting visual concepts from the text for boosting the labeling accuracy.



























 §.§ VITAL performance with GMAN vs. K GANs

Since we can train K GANs with K different noise levels seperately and then use the trained GANs to generate K synthetic images, we also can assess the performance of our VITAL with K GANs. For the purpose of fair comparison, we train K=5 StackGAN++ with different noise levels individually and use the K generated synthetic images to extract visual features for image labeling on the Oxford 102 Category Flower Dataset. The accuracy performance of VITAL-RST with K StackGAN++ is 90.59%, which is 3% lower than VITAL-RST with our StackGMAN++. 

We visualize the synthetic images generated by both StackGMAN++ and K StackGAN++ in Figure <ref>. Obviously, our proposed StackGMAN++ is able to generate more diverse images, and the diversity among the synthetic images benefits the performance of image labeling.





 §.§ Analysis on successful and failure cases














To further explain why our proposed VITAL works better than JCNN-NN, we conduct an analysis on the quality of the nearest neighbor (NN) images used in JCNN-NN. As shown in Figure <ref>, the color of neighbor images are not always consistent with real images, and the background of all the neighbor images are more complicated when compared with the generated synthetic images in Figure <ref>. Moreover, the Jarccard similarity between the query image and the k-th NNs always decreases when the value of k increases. 


As a non-parameter method, the quality of nearest neighbor images in JCNN-NN will be perfect if an infinite number of samples is available, but in practice the number of samples is limited. Therefore the quality of neighbor images depends on the density distribution in training data. If the distribution is dense enough, then the quality of neighbor images can be a guarantee. Obviously, in both the 102 Category Flower Dataset and Caltech-UCSD Birds-200-2011 Dataset, the observations indicate that the density is not sufficient to ensure the good quality of neighbor images for JCNN-NN.

On the contrary, our proposed VITAL generates K visual plausible synthetic images by StackGMAN++ conditioned on a text of query image only, without requiring text for any other images, which makes the generated synthetic images are closely correlated to the corresponding real images, displaying key visual elements embedded in text and even providing more details about the underlying background and variations. Therefore, our proposed VITAL is more robust to visually interpret text for improving the performance of image labeling.

We also visualize a successful case and a failure case in Figure <ref>. As we can observe, if text describes a flower with detailed information about colors hand shapes, then our StackGMAN++ is able to generate a bunch of informative synthetic images for leveraging the performance of image labeling. Otherwise, if a text is hard to understand, ambiguous and even misleading, then the generated synthetic images will not be consistent with each other to represent the visual concepts well.     












§ CONCLUSION


In this paper, we propose a novel way to visually interpret text with adversarial learning for image labeling. With our StackGMAN++, K high-resolution and photo-realistic synthetic images are generated conditioned on a text to represent the visual concepts. The visual synthetic image feature has been proved to be able to improve accuracy for image labeling, and it is complementary to real image feature and text feature. The experimental results conducted on two datasets well support our claims in the paper.

Our future work includes further exploring our GMAN to incorporate the state-of-the-art GANs like AttnGAN <cit.> to produce better quality of synthetic images, improving VITAL's performance on image labeling, and extending our VITAL to solve more general multimedia problems.

ieee



