We introduce FigureQA, a visual reasoning corpus of over one million question-answer pairs grounded in over 100,000 images. The images are synthetic, scientific-style figures from five classes: line plots, dot-line plots, vertical and horizontal bar graphs, and pie charts. We formulate our reasoning task by generating questions from 15 templates; questions concern various relationships between plot elements and examine characteristics like the maximum, the minimum, area-under-the-curve, smoothness, and intersection. To resolve, such questions often require reference to multiple plot elements and synthesis of information distributed spatially throughout a figure. To facilitate the training of machine learning systems, the corpus also includes side data that can be used to formulate auxiliary objectives. In particular, we provide the numerical data used to generate each figure as well as bounding-box annotations for all plot elements. We study the proposed visual reasoning task by training several models, including the recently proposed Relation Network as strong baseline. Preliminary results indicate that the task poses a significant machine learning challenge. We envision FigureQA as a first step towards developing models that can intuitively recognize patterns from visual representations of data. Scientific figures compactly summarize valuable information. They depict patterns like trends, rates, and proportions, and enable humans to understand these concepts intuitively at a glance. Because of these useful properties, scientific papers and other documents often supplement textual information with figures. Machine understanding of this structured visual information could assist human analysts in extracting knowledge from the vast documentation produced by modern science. Besides immediate applications, machine understanding of plots is interesting from an artificial intelligence perspective, as most existing approaches simply revert to inverting the visualization pipeline (i.e. by reconstructing the source data). Mathematics exams, e.g. Graduate Records Examinations (GREs), often include questions about a plot regarding relations between the plot elements. When solving these exam questions, humans don't always build a table of coordinates for all data points, but judge by visual intuition.Thus motivated, and inspired by recent research in Visual Question Answering (VQA) BID1 Goyal et al., 2016) and relational reasoning BID6 BID20 , we introduce FigureQA. FigureQA is a corpus of over one million question-answer pairs grounded in over 100, 000 figures, devised to study aspects of comprehension and reasoning in machines. There are five common figure types represented in the corpus, which model both continuous and categorical information: line, dot-line, vertical and horizontal bar, and pie plots. Questions concern one-to-all and one-to-one relations among plot elements: for example, Is X the low median? , Does X intersect Y? . Their successful resolution requires inference over multiple plot elements. There are 15 question types in total, which address properties like magnitude, maximum, minimum, median, area-under-the-curve, smoothness, and intersections. Each question is posed such that its answer is either yes or no.FigureQA is a synthetic corpus, like the related CLEVR dataset for visual reasoning BID6 . While this means that the data may not exhibit the same richness as figures "in the wild", it permits greater control over the task's complexity, enables auxiliary supervision signals and most importantly provides reliable ground-truth answers. Further, by analyzing the performance on real figures of models trained on FigureQA it will be possible to extend the corpus to address limitations not considered during generation. The FigureQA corpus can be extended iteratively, each time raising the task complexity, as model performance increases. This is reminiscent of curriculum learning BID2 ) allowing iterative pretraining on increasingly challenging versions of the data. By releasing the data now, we want to gauge the interest in the research community and adapt future versions based on feedback, hopefully accelerating research in this direction. Additional annotation is provided to allow researchers to define other tasks than the one we introduce in this manuscript.The corpus is built using a two-stage generation process. First, we sample numerical data according to a carefully tuned set of constraints and heuristics designed to make sampled figures appear natural. Next we use the Bokeh open-source plotting library (Bokeh Development Team, 2014) to plot the data in an image. This process necessarily gives us access to the quantitative data presented in the figure. We also modify the Bokeh backend to output bounding boxes for all plot elements: data points, axes, axis labels and ticks, legend tokens, etc. We provide the underlying numerical data and the set of bounding boxes as supplementary information with each figure, which may be useful in formulating auxiliary tasks like reconstructing quantitative data given only a figure. Bounding box targets of plot elements relevant for answering a question might be useful for supervising an attention mechanism to ignore potential distractors. Experiments in that direction are outside of the scope of this manuscript, but we want to facilitate research of such approaches by releasing these annotations.As part of the generation process we balance the ratio of yes and no answers for each question type and each figure. This makes it more difficult for models to exploit biases in answer frequencies while ignoring visual content.We review related work in Section 2. In Section 3 we describe the FigureQA dataset and the visualreasoning task in detail. Section 4 describes and evaluates four neural baseline models trained on the corpus: a text-only Long Short-Term Memory (LSTM) model (Hochreiter & Schmidhuber, 1997) as a sanity check for biases, the same LSTM model with added Convolutional Neural Network (CNN) image features BID9 Fukushima, 1988) , and a Relation Network (RN) BID16 , a strong baseline model for relational reasoning.The RN respectively achieves accuracies of 72.40% and 76.52% on the FigureQA test set with alternated color scheme (described in Section 3.1) and the test set without swapping colors. An "official" version of the corpus is publicly available as a benchmark for future research. 1 We also provide our generation scripts, which are easily configurable, enabling researchers to tweak generation parameters to produce their own variations of the data.
We present a question-answering dataset, FigureQA, as a first step towards developing models that can intuitively recognize patterns from visual representations of data. This paper introduces a dataset of templated question answering on figures, involving reasoning about figure elements. The paper introduces a new visual reasoning dataset called Figure-QA which consists of 140K figure images and 1.55M QA pairs, which can help in developing models that can extract useful information from visual representations of data.