Information Maximizing Visual Question Generation
Ranjay Krishna Michael Bernstein Li Fei-Fei







Information Maximizing Visual Question Generation
    Ranjay Krishna, Michael Bernstein, Li Fei-Fei

Stanford University

{ranjaykrishna, msb, feifeili}@stanford.edu

    May 5, 2019
====================================================================================================================






Though image-to-sequence generation models have become overwhelmingly popular in human-computer communications, they suffer from strongly favoring safe generic questions (“What is in this picture?”). Generating uninformative but relevant questions is not sufficient or useful. We argue that a good question is one that has a tightly focused purpose — one that is aimed at expecting a specific type of response. We build a model that maximizes mutual information between the image, the expected answer and the  generated question. To overcome the non-differentiability of discrete natural language tokens, we introduce a variational continuous latent space onto which the expected answers project. We regularize this latent space with a second latent space that ensures clustering of similar answers. Even when we don't know the expected answer, this second latent space can generate goal-driven questions specifically aimed at extracting objects (“what is the person throwing”),  attributes, (“What kind of shirt is the person wearing?”), color (“what color is the frisbee?”), material (“What material is the frisbee?”), etc. We quantitatively show that our model is able to retain information about an expected answer category, resulting in more diverse, goal-driven questions. We launch our model on a set of real world images and extract previously unseen visual concepts.





§ INTRODUCTION

The task of transforming visual scenes to language, questions <cit.>, answers <cit.> or captions <cit.>, has widely adopted an image-to-sequence architecture that encodes the image through a convolutional neural network (CNN) <cit.> and then decodes the language with a recurrent neural network <cit.>. The whole framework can be efficiently trained by maximum likelihood estimation (MLE) and has demonstrated state-of-the-art performance in various tasks <cit.>. However, this training procedure is not suitable for generating questions or enabling discovery of new concepts. In fact, most MLE-based training schemes have shown to produce generic questions that result in uninformative answers (e.g. “yes”) <cit.>, questions (e.g. “What is the person doing?”) <cit.>, captions (e.g. “A clear day with a blue sky” <cit.>) or dialogue (e.g. “I don't know”) <cit.>. Simply generating a generic question is not sufficient or useful for discovering new concepts.





Instead of generating generic questions, question generation models should be goal-driven — we show how they can be trained to ask questions aimed at extracting specific answer categories. Visual question generation is not a bijection, i.e. multiple correct questions can be generated from the same image. Previous research moved away from a supervised approach to question generation to a variational approach that can generate multiple questions by sampling a latent space <cit.> (see Figure <ref>). However, previous approaches are not goal-driven — they do not guarantee that the question will result in a specific type of answer. To remedy their problem, we could encode the answer along with the image before generating the question. While such an approach allows the model to condition its question on the answer, it is neither technically feasible nor practical. Technical infeasibility arises because variational models often lead to the posterior collapsing problem <cit.>, where the model can learn to ignore the answer when generating questions. Impracticality arises because the main purpose of asking questions is to attain an answer, implying that knowing the answer defeats the purpose of generating the question. 

To tackle the first challenge, we design a visual question generation architecture that maximizes the mutual information between the generated question with the image as well as with the expected answer (see Figure <ref>). We call our model Information Maximizing Visual Question Generator as it maximizes relevance with the image and expectation over the answer. Safe, generic questions that lead to uninformative answers are discouraged as they have low mutual information with either. However, optimizing for mutual information is often intractable and given the discrete tokens (words) we wish to generate, no unbiased, low variance gradient estimator exists <cit.>. We formulate our model as a variational auto-encoder that attempts to learn a joint continuous latent space between the image, question and the expected answer. Instead of directly optimizing discrete utterances, the question, image and expected answer are all trained to maximize the mutual information with this latent space. By reconstructing the image and expected answer representations, we can maximize the evidence lower bound (ELBO) and control what information the generated questions request.  

The second challenge arises from the lack of an expected answer in real world deployments. Since we require an answer to map the image into a latent space, it is not possible to generate questions in the absence of an answer. Enumerating all possible answers is infeasible. Instead, we propose creating a second latent space that is learned from the image and the answer category instead of the answer itself. Answer categories can be objects, attributes, colors, materials time, etc. During training, we minimize the KL-divergence between these two latent spaces. Not only does this allow us to generate visual questions that maximize mutual information with the expected answer, it also acts as a regularizer into the original latent space. It prevents the learned latent spaces from overfitting to specific answers in the training set and forces them to generalize to categories of questions.

We annotate the VQA dataset <cit.> with 15 categories for the top 500 answers and use it to train our model, which queries for specific answer categories. We evaluate our model on relevance to the image and on its ability to expect the answer type. Finally, we run our model on 1000 real world images and discover 80 new objects, 40 new attributes, 17 new colors, and 8 new materials.



§ RELATED WORK


Visual understanding has been studied vigorously through question answering with the availability of large scale visual question answering (VQA) datasets <cit.>. Current VQA approaches follow a traditional supervised MLE paradigm that typically relies on a CNN + RNN encoder-decoder formulation <cit.>. Successive models have improved performance by stacking attention <cit.>, modularizing components <cit.>, adding relation networks <cit.>, augmenting memory <cit.>, and adding proxy tasks <cit.>. While the performance of VQA models have been encouraging, they require a large labelled dataset with a predefined vocabulary. In contrast, we focus on the surrogate task of generating questions in the hopes of augmenting real world agents with the ability to expand it's visual knowledge by discovering new visual concepts.





In contrast to answering questions, generating questions has received little interest so far. In NLP, a few methods have attempted to automatically generate questions from knowledge bases using rule based <cit.> or deep learning based systems <cit.>. In computer vision, a few recent projects have explored the task of visual question generation to build curious visual agents <cit.>. These projects have also either followed an algorithmic rule-based <cit.> or learning-based <cit.> approach. Newer papers have treated the generation process as a variational process <cit.> or placed it in a active learning <cit.> or reinforcement learning setting <cit.>. Our work draws inspiration from these previous methods and extends them by treating question generation as a process that maximizes mutual information between not just the image but also considers the expected answer's category. We believe that a good question generator should be goal driven — it should generate questions to receive a particular answer category.



There is a large body of work exploring generative models and learning latent representation spaces. Early work focused primarily on stacked autoencoders and then on restricted boltzman machines <cit.>. Recent successes of these applications have primarily been a result of variational auto-encoders (VAEs) <cit.> and generative adversarial networks (GANs) <cit.>. With the reparameterization trick, VAEs can be trained to learn a semi-supervised latent space to generate images <cit.>. They have also been extended to continuous state space <cit.> and sequential models <cit.>. GANs, on the other hand, can learn image representations that support basic linear algebra <cit.> and even enable one-shot learning by using probabilistic inference over Bayesian programs <cit.>. Both VAEs and GANs have disentangled their representations based on class labels or other visual variations <cit.>. While we do not explicitly disentangle the representation, we will demonstrate later how the second latent space regularizes the original space and disentangles the representations of different answer categories. 

Generative models often require a series of tricks for successful training <cit.>. And even with these tricks, training them with discrete tokens is only possible by using gradient estimators. As we mentioned earlier, these estimators often suffer from one of two problems: high bias <cit.> or high variance <cit.>. Low variance methods like Gumbel-Softmax <cit.>, CONCRETE distribution <cit.>, semantic hashing <cit.> or vector quantization <cit.> result in biased estimators. Similarly, low bias methods like REINFORCE <cit.> with Monte Carlo rollouts, result in high variance <cit.>. We overcome this issue by introducing a continuous latent space that maximizes mutual information with encodings of the image, question and answer. This latent space can be trained using existing VAE training procedures that attempt to reconstruct the image and answer representations. We further extend this model with a second latent space conditioned on the answer category that removes the need for an actual answer when generating questions.



§ INFORMATION MAXIMIZING VISUAL QUESTION GENERATOR





Our aim is to generate questions that have a tightly focused purpose — questions with the aim to learn something specific about the image. Agents with the capability to request specific categories of information can extract new concepts more effectively from the real world. In this section, we detail how we design an Information Maximizing Visual Question Generator. Recall that the goal of our model is to generate questions given an image and an answer category. For example, if we want to understand materials or binary answers, our model should generate questions “What material is that desk made out of?” or “Is the desk on the right of the chair?”, respectively. Our two challenges are (1) technical infeasibility caused by non-differentiable discrete tokens and variational posterior collapse and (2) impracticality of requiring answers to generate questions. We start off with a formal definition of the problem, explain why current methods fail and then detail our training and inference process.




 §.§ Problem formulation

Let q denote the question we want to generate for an image i. This question should result in the an answer a of category c. For example, the question “What is the person in red doing with the ball?” should result in the answer “kicking”, which belong to category “activity”. Our final goal is to define a model p(q | i, c). But first, let's attempt to define a simpler model p(q | i, a) that maximizes the mutual information between the image and the question I(i, q) and between the expected answer and the question I(a, q). This objective can be written as:

    max I(i, q) + λ I(a, q)
    s. t.   q ∼ p(q | i, a)

where λ is a hyperparameter that adjusts for their relative importance in the optimization. 




 §.§ Continuous latent space

As already mentioned, directly optimizing this objective is infeasible because the exact computation of mutual information is intractable. Additionally, optimizing by estimating gradients between discrete steps is difficult as the estimator needs to have both low bias and low variance. To overcome this challenge, we introduce a continuous, dense, latent z-space. We learn a mapping p_θ(z | i, a), parameterized by θ, from the image and the expected answer to this latent space.

With this z-space, our new optimization becomes:

    max_θ I(q, z | a, i) + λ_1 I(a, z) + λ_2 I(i, z)
    s. t.   z ∼ p_θ(z | i, a)
     q ∼ p_θ(q | z)

where λ_1 and λ_2 are hyperparameters that relatively weight the mutual information terms in the optimization.




 §.§ Variational mutual information maximization

So far, we have avoided discrete tokens. However, this mutual information maximization is still intractable as it requires knowing the true posteriors p(z | i) and p(z | a). Fortunately, we can opt to maximize its ELBO:

    I(z, i)    = ℍ(i) - ℍ(i | z)
       =ℍ(i) + 𝔼_z ∼ p(z, i)[𝔼_î∼ p(i | z)[log p(î| z)]]
       =ℍ(i) + 𝔼_i ∼ p(i) [D_KL[p(î| z) || p_θ (î| z)] 
            + 𝔼_î∼ p(i | z)[log p_θ (î| z)]]
       ≥ℍ(i) + 𝔼_i ∼ p(i) [𝔼_î∼ p(i | z)[log p_θ (î| z)]]]

where ℍ(·) is the entropy function and 𝔼 is expectation. p_θ(·) is a function parameterized by θ. This optimization is often referred to as variational information maximization <cit.>. Similarly,

    I(z, a) ≥ℍ(a) + 𝔼_a ∼ p(a) [𝔼_â∼ p(a | z)[log p_θ (â| z)]]]


The third and final conditional mutual information term I(q, z | a, i) can also be bounded by:

    I(z, q | a, i)    ≥ℍ(q) + 
       𝔼_q ∼ p(q | i, a) [𝔼_q̂∼ p(q | z, a, i)[log p_θ (q̂| z, i, a)]]] 
       s. t.   p(q | z, a, i) = p(q | z)p(z | a, i)



Putting  Eq. <ref>, <ref> and <ref> together in Eq. <ref>:

    max_θ𝔼_p_θ (q,i,a) [   log p_θ (q | i,a, z) + λ_1 log p_θ (a | z) 
       + λ_2 log p_θ (i | z)]
    s. t.     p_θ (q,i,a) = p_θ (q | z) p_θ (z | i,a) p(i,a)

Note that we ignore the entropy terms associated with the training data as it doesn't involve the parameter θ we are trying to optimize. Therefore, optimizing Eq. <ref> can be accomplished by maximizing the reconstruction of the image and answer representations while maximizing the MLE objective of generating the question.



 §.§ Question generation by reconstructing image and answer representations

To functionalize the optimization presented above, we begin by first encoding the image using a CNN as a dense vector h_i (see Figure <ref>). Similarly, we encode the answer a using a long short term memory network (LSTM) <cit.>, which is a variant of RNNs, into another dense vector h_a. Next, we feed h_i and h_a into a VAE that embeds both into a latent z-space. In practice, we assume that z follows a multivariate Gaussian distribution with diagonal covariance. We use the reparameterization trick <cit.>, to generate means μ_z and standard deviations σ_z, combine it with a sampled unit Gaussian noise ϵ to generate z = μ_z + σ_z ϵ. 

From z, we reconstruct ĥ_i and ĥ_a and optimize the first two terms in Eq. <ref> by minimizing the following l_2 losses:

    L_i = || h_i - ĥ_i ||_2,      L_a = || h_a - ĥ_a ||_2

Next, we use a decoder LSTM to generate the question q̂ from z-space. We minimize the MLE objective L_MLE between q̂ and the true question in our training set q, which results in the third and final term in Eq. <ref>.




 §.§ Regularizing with a second latent space

So far, we have proposed building a model that maximizes the lower bound of mutual information between a latent space, the image and the expected answer. This allows us to generate questions if we know what the expected answer should be. This is not conducive to our original goal of deploying our model in real world situations where it does not know the answer a priori. If we already know the answer to a question, there is no point in generating a question.


To remedy this, we propose a second latent t-space. Instead of using both a and i to encode h_a and h_i into z-space, we discard the answer and instead only use its category c. We classify answers as being one of a few predefined categories, such as objects (e.g. “cat”), attributes (e.g. “cold”), color (e.g. “brown”), relationship (e.g. “ride”), counting (e.g. “1”), etc. These categories are cast as a one hot vector and encoded as h_c and used, along with h_i to embed into the variational t-space. We train t-space by minimizing the KL-divergence with z-space:

    L_t   = D_KL(p_θ(z| i, a), p_ϕ(t | i, c))
       = logσ_t - logσ_z + σ_z + (μ_t - μ_z)^2/2σ_t- 0.5

where ϕ are the parameters used to embed into t-space. This allows us to now utilize p_ϕ(t | i, c) to embed into a space that closely resembles z-space. Since we assume that both z-space and t-space follow a multivariate Gaussian with diagonal covariance, the KL term has the analytical form shown above. We no longer need to know the answer a to embed and generate questions. Intuitively, the t-space can be also thought of as a regularizer on z-space, preventing the model from overfitting to the answers in the training data and relying instead on utilizing the answer categories.

Putting them together, the final loss for our model is:

    L = L_MLE + λ_1 L_a + λ_2 L_i + λ_3 L_t

where λ_1 and λ_2 have already been introduced and λ_3 is a hyperparameter that controls the amount of regularization used in our model. Note that we are omitting the KL-loss with respect to a unit normal centered at zero that maintains the two latent spaces' priors.




 §.§ Inference

During inference, we are given an image i and answer category c and are expected to generate questions. We encode the inputs into the second latent t-space and sample from it to generate questions, as shown in Figure <ref>. This allows us to generate goal-driven questions for any image, focused towards extracting its objects, its attributes, etc. 




 §.§ Implementation details

We implement our model using PyTorch and plan on releasing all our code. We use ResNet18 <cit.> as our image encoder and do not fine-tune its weights. h_i, h_a and h_t are all 512 dimensional vectors. z-space and t-space are 100 dimensions. The encoders for the image and answer are trained only from L_MLE and not L_i, L_a or L_t to prevent the encoders from simply optimizing for the reconstruction loss at the cost of not being able to generate questions. We optimized the hyperparameters such that λ_1 = 0.01, λ_2 = 0.001, λ_3 = 0.005 with a learning rate of 0.001 that decays every 4 epochs for a total of 10 epochs.



§ EXPERIMENTS








To test our visual question generation model, we perform a series of experiments and evaluate the model along multiple dimensions. We start by discussing the dataset and evaluation metrics used. We then showcase examples of our model's generated questions when conditioned on the answer. Next, we demonstrate its ability when conditioned only on the answer category. We compare both these cases against a series of baselines and ablations. We analyze the diversity of questions produced within each answer category. Finally, we report a small proof of concept deployment of our model on real world images found online and show that it can learn new concepts.



 §.§ Experimental setup


Dataset. To enable the kind of interaction where we can specify input answer categories, we need a VQA dataset that categorizes its answers. The VQA dataset <cit.> has a few basic categorizations of questions but not their answers. We annotate the VQA <cit.> dataset answers with a set of 15 categories and label their top 500 answers. These categories include objects (e.g. “cat”, “person”), attributes (e.g. “cold”, “old”), color (e.g. “brown”, “red”), relationship (e.g. “ride”, “jump”), counting (e.g. “1”, “10”), etc. The top 500 answers make up the 82% of the VQA dataset, resulting in 367K training+validation examples. We treat their validation set as our test set as the answers in their test set are not publicly available. We break the training set up into a 80-20% train-validation split.




Evaluation metrics.
All past question generation papers have used a variety of evaluation metrics to calculate the quality of a question. While some have focused on maximizing diversity <cit.>, others have treated it as a proxy task to improve question answering <cit.>. Diversity measures have included using variants of beam search <cit.>, measuring novel questions or unique tri-grams <cit.> or creating rule-based datasets <cit.>. Proxy tasks have typically used accuracy of multiple-choice answers to measure the performance of question generation.

We too report a variety of different evaluation metrics to highlight different components of our model. First, we use language modeling evaluation metrics like BLEU, METEOR and CIDEr <cit.> to calculate how well our generated questions match the ground truth questions in our test set. Next, we measure the mutual information retained in the latent space by training a classifier to classify answer categories encoded in the latent space. This metric sheds light on how well our method retains information about the input answers or answer categories. Next, we measure relevance of the question, ensuring that the questions are valid for the given image and result in the expected answer category. Relevance results are calculated from majority vote conducted by hiring 3 crowd-workers that vote on whether a question can be answered given its corresponding image. Finally, we report diversity scores for each category, which measures the number of unique questions generated.






Baselines. We adapt a series of past CNN-RNN models to accept answer or answer types when generating questions. The first model  is a supervised, non-variational model that takes an image and answer as input and generates a question <cit.>. This model is reminiscent of the VQA models often used to answer questions <cit.>, except the answers are now inputs and the questions outputs <cit.>. Next,  is a variational version of , which embeds the answer and question to a latent space before generating the question <cit.>. We also train versions of these models that accept the answer categories instead of the answer:  and . When generating from a variational model, we set z = μ_z or t=μ_t to keep its outputs consistent for all measures except diversity.

We refer to our full model as  and can generate questions from either the answer latent space z or the category latent space t. We perform ablations on this model by removing specific components.  doesn't maximize mutual information with respect to the expected answer but can also generate questions from both the z and t spaces.  doesn't include the t- space and can only generate questions from answers. Finally,  doesn't train with the reconstruction loss nor does it have a second latent space t. Our evaluations  empirically demonstrate how these ablations justify our model designs. 




 §.§ Mutual information maximization

We check whether our model improves the mutual information retained in the latent space with the input answer. We freeze the weights of a trained model and embed input images, answers and categories into the latent z or t-space, depending on the model. We train a simple 3-layer MLP that attempts to classify the latent code as either one of the 15 answer categories or as one of the 500 answers. We evaluate our model on the test set with a random chance of 6.67% and 0.20%, respectively. Table <ref> shows that the baseline models do a poor job of actually remembering the answer or category, justifying the need for a mutual information maximization approach. Since these models are unable to retain information about the input answers, it also explains why they often generate safe, generic, uninformative questions. Since our model can embed into both the z as well as the t space, we report how well these two spaces retain information. We find that  retains near perfect information about the input answer category with an accuracy of 99.02% from t-space and 32.44% from the z-space. We find that when trained without the t-space,  retains more information as it no longer has to constrain the z-space to regularize answers of the same category. We also visualize a TSNE <cit.> representation of the two spaces in Figure <ref>. Models that don't reconstruct the answer (e.g. in ,  or any of the baselines) show visually inseparable categories.




 §.§ Generating questions given the answers

Since our model can produce questions from both answers as well as answer categories, we evaluate both scenarios individually. The language modeling section in Table <ref> showcases how the various models perform when generating questions from the z-space, i.e. generating questions from answers. We find that  performs the best over all the baselines and across all ablations of our model. This is likely because the latent space has more capacity when it is not also being regularized by the t-space. We find that  performs 6 METEOR points worse than  and  implying that forcing the model to reconstruct the answer does improve the quality of questions generated to better match the ground truth.    






 §.§ Generating questions with answer types

The lower half of Table <ref> evaluates how well our model and the baselines perform when generating questions in the absence of the actual answer and only in the presense of the answer categories. We find that overall, all the language metrics are slightly lower than when the questions were generated from the z-space. This is expected as now the questions need to be generated with only the answer category encoded in the t-space without knowing exactly what the answer is. Therefore, the models are penalized for asking an “object” question about the “horse” when the answer expects the question to focus on the “saddle” instead. We also qualitatively sample and report a random set of questions generated by our model in Figure <ref>. We see that our model often uses concepts in the image to ground the questions. It asks specific questions like “what is the bat made of?” or “is the man going to the right of the girl?”. However, there are categories like “time” that have a low diversity of training questions and result in the inevitable “what time of day is this?” question. The qualitative errors we have observed often occur when the model is forced to ask a question about a category that is not present in the image; it is hard to ask about “food” when no food is present.




 §.§ Measuring diversity of questions

For all the 177K images in our test set, we generated one question per answer category, resulting in a total of 2M questions. We report diversity in Table <ref> using two existing metrics: (1) Strength of generation: the percentage of unique generated questions normalized by the number of unique ground truth questions and (2) Inventiveness of generation: the percentage of unique questions unseen during training normalized by all unique questions generated. We compare our model with the baseline  which does not reconstruct the answer or the image. We find that our method results in more diverse set of questions across most categories. 
Questions asking for “shape” and “materials” tend to generate the most unseen questions as the model learns to generate questions like “what [shape/material] is the ____ [made out of]?” and injects objects in the given image into the missing blank. Answers agnostic to the image contents, such as “time”, result in the fewest number of novel questions. 




 §.§ Real world deployment of our model

To examine our model in a real world deployment, we generated 1 question each for 1000 images with hashtags , , ,  scraped from online public social media posts. Since our model needs an input answer category to ask a question, we trained a simple ResNet18 CNN <cit.> on the VQA images to output one of 4 categories (see Table <ref>). We generated answer categories using the CNN and fed it into our model to generate the questions. The questions were sent to two crowd workers: one answered the question and the other reported the relevance of the question with the image and the answer with the answer category. We found all the questions asked by both  and  to be relevant to the image while 97.2% and 56.8% were relevant to the answer category. Our methods questions led to more unseen concepts.







§ CONCLUSION



We believe that visual question generation should be a task that is aimed at extracting specific categories of concepts from an image. We define a good question to be one that is not only relevant to the image but is also designed to expect a specific answer category. We build Information Maximizing Visual Question Generator that maximizes the mutual information between the generated question, the input image and the expected answer. We extend this model to overcome technical challenges associated with maximizing mutual information with discrete tokens and collapsing posterior while also allowing it to generate questions when the expected answer is absent. We analyze the questions using language modeling, diversity, relevance and mutual information metrics. We further show that through a real world deployment of this system, it can discover new concepts.



  
Acknowledgements. We thank Justin Johnson, Andrey Kurenkov, Apoorva Dornadula and Vincent Chen for their helpful comments and edits. This work was partially funded by the Brown Institute of Media Innovation and by Toyota Research Institute (“TRI”) but this article solely reflects the opinions and conclusions of its authors and not TRI or any other Toyota entity.


ieee



