Discriminative  question  answering  models  can  overfit  to  superficial  biases  in datasets,  because their loss function saturates when any clue makes the answer likely.   We introduce generative models of the joint distribution of questions and answers, which are trained to explain the whole question, not just to answer it. Our  question  answering  (QA)  model  is  implemented  by  learning  a  prior  over answers,  and  a  conditional  language  model  to  generate  the  question  given  the answer—allowing scalable and interpretable many-hop reasoning as the question is generated word-by-word.   Our model achieves competitive performance with specialised discriminative models on the SQUAD and CLEVR benchmarks, indicating that it is a more general architecture for language understanding and reasoning than previous work. The model greatly improves generalisation both from biased training data and to adversarial testing data, achieving a new state-of-the-art on ADVERSARIAL SQUAD. We will release our code. Question answering tasks are widely used for training and testing machine comprehension and reasoning BID33 BID22 . However, high performance has been achieved with only superficial understanding, as models exploit simple correlations in the data BID48 Zhou et al., 2015) . For example, in Visual QA BID0 , the answer to What colour is the grass? can be memorised as green without considering the image (Figure 1 ). We argue that this over-fitting to biases is partly caused by discriminative loss functions, which saturate when simple correlations allow the question to be answered confidently, leaving no incentive for further learning on the example.We propose generative QA models, using Bayes' rule to reparameterise the distribution of answers given questions in terms of the distribution of questions given answers. We learn a prior over answers and a conditional language model for generating the question-reducing question answering to sequence-to-sequence learning BID40 , and allowing many-hop reasoning as the model explains the whole question word-by-word.Generative loss functions train the model to explain all question words, even if the answer is obvious. For example, a model cannot assign high probability to generating the question What colour is the grass? without learning a dependency between the image and the word grass. We show that this method allows much improved generalisation from biased training data and to adversarial test data, compared to state-of-the-art discriminative models.Word-by-word generative modelling of questions also supports chains of reasoning, as each subpart of the question is explained in turn. Existing methods use a pre-specified number of reasoning steps BID39 BID18 , which may be too many steps on easy cases, and too few on long and complex questions. We instead perform an interpretable reasoning step for each question word, and achieve 97.7% accuracy on the CLEVR benchmark BID21 .Our approach opens a promising new direction for question answering, with strong results in language understanding, reasoning and generalisation.Is the purple thing the same shape as the large gray rubber thing? Does the green rubber object have the same shape as the gray thing that is on the right side of the big purple object?(a) Two CLEVR questions. Both can be answered no using only subsets of the available information. A generative model must learn to perform additional reasoning to assign high likelihood to the complete question-answer pair. Word-by-word question generation allows a reasoning step to explain each word.Whilst filming in Mexico City, speculation in the media claimed that the script had been altered to accommodate the demands of Mexican authorities reportedly influencing details of the scene and characters, casting choices, and modifying the script in order to portray the country in a "positive light" in order to secure tax concessions and financial support worth up to $20 million for the film. This was denied by producer Michael G. Wilson.Which Bond producer would not confirm that the film had been changed to accommodate Mexican authorities?(b) A SQUAD question . A discriminative model can identify the only producer, and ignore the rest of the question. To generate the question and answer, our model needs coreference, negation and paraphrasing. These reasoning skills can improve generalisation on test examples with multiple plausible answers.Figure 1: Examples of questions that can be answered using only some question words (underlined). We introduced a generative model for question answering, which leverages the greater amount of information in questions than answers to achieve high performance in both language comprehension and reasoning. The approach demonstrates better robustness to biased training data and adversarial testing data than state-of-the-art discriminative models. There are numerous interesting directions for future work, such as combining information about an entity from multiple sources to generate questions. Given the rapid progress made on discriminative QA models in recent years, we believe there is significant potential for further improvements in generative question answering. A TRAINING DETAILS
Question answering models that model the joint distribution of questions and answers can learn more than discriminative models This paper proposes a generative approach to textual and visual QA, where a joint distribution over the question and answer space given the context is learned, which captures more complex relationships. This paper introduces a generative model for question answering and proposes to model p(q,a|c), factorized as p(a|c) * p(q|a,c).  The authors proposes a generative QA model, which optimizes jointly the distribution of questions and answering given a document/context. 