11institutetext: Brookhaven National Laboratory, Upton NY 11973, USA
11email<EMAIL_ADDRESS>
# An extensible point-based method for data chart value detection

Carlos Soto and Shinjae Yoo

###### Abstract

We present an extensible method for identifying semantic points to reverse
engineer (i.e. extract the values of) data charts, particularly those in
scientific articles. Our method uses a point proposal network (akin to region
proposal networks for object detection) to directly predict the position of
points of interest in a chart, and it is readily extensible to multiple chart
types and chart elements. We focus on complex bar charts in the scientific
literature, on which our model is able to detect salient points with an
accuracy of 0.8705 F1 (@1.5-cell max deviation); it achieves 0.9810 F1 on
synthetically-generated charts similar to those used in prior works. We also
explore training exclusively on synthetic data with novel augmentations,
reaching surprisingly competent performance in this way (0.6621 F1) on real
charts with widely varying appearance, and we further demonstrate our
unchanged method applied directly to synthetic pie charts (0.8343 F1).
Datasets, trained models, and evaluation code are available at
https://github.com/BNLNLP/PPN_model.

###### Keywords:

document analysis, chart extraction, value detection

##  1 Introduction

A long-standing challenge in document information mining is the general
inaccessibility to machine processing of data contained in embedded images.
Although established computer vision techniques (e.g. OCR) and more recent ML-
based ones – like image classification/captioning, object detection, and
similarity scores – have alleviated this limitation in some contexts (e.g. [2,
28, 32, 14]), identification and extraction of structured image data remains a
challenge. This is particular the case for scientific literature mining, in
which much of the new knowledge presented in each article is in the form of
data-filled charts.

Figure 1: Sample detection results of data points in scientific bar charts.
Images intentionally over-cropped, showing robustness to spurious artifacts.
Charts from [37, 4, 40, 42, 20].

There is of course immense scientific and practical value in collating
knowledge from vast and disparate document sources: enabling broad search,
powerful inference, new analyses, and a myriad other capabilities that are out
of reach when considering documents individually or in small sets as human
readers are limited to doing. Document mining has already impacted the state
of researcher access to knowledge found in the scientific literature: common
tools like keyword search and reference crawling may now be supplemented by
text mining of abstracts (or even full articles); scientific text corpora are
designed and published to enable such mining (e.g. [36, 45, 16]); and ML
models such as SciBERT [3] and BioBERT [18] have exploited such datasets to
give researchers new capabilities leveraging natural language processing
techniques. But thus far, automated and scalable information extraction has
been limited mainly to text, despite the wealth of information found in charts
and graphs.

The value of information found in data charts is certainly not limited to the
scientific literature; business, government, educational, historic, and many
other types of document sources could benefit from elucidation of their
graphically-embedded contents. But charts in published research are of
particular interest to us because they ostensibly form a part of the
scientific record, yet their format impedes access to the information they
contain. Presenting data graphically has obvious benefits, but if the
underlying data is not also accessible, reproducibility and comparison become
difficult. There are increasing efforts to encourage publication of research
data (including the raw data that comprise chart contents), but this practice
is not yet the norm. And there is certainly no expectation that the figures in
millions of published articles will be retroactively annotated with their
source data values. Even attempting this for small numbers of charts is a
daunting and tedious challenge.

It would therefore be highly desirable to be able to automatically reverse
engineer chart contents. Manual annotation tools such as WebPlotDigitizer [27]
and GraphReader [17] may be useful for single or small numbers of graphs, but
these of course do not scale beyond such use cases – and the manual
annotations they rely on mean they cannot be well adapted to automation. This
is further complicated by the tremendous variety of graph types, styles,
production tools, overlays, and other visualization effects that authors and
publishers employ.

In this paper, we address what we believe is the most pressing part of the
automated data extraction problem: accurate value detection in complex charts;
we focus our efforts on scientific bar charts. However, the method we propose
is designed to be readily extensible to other chart types (as we demonstrate
with pie charts) and to fit naturally in a full data extraction pipeline. The
approach we take is point-based, and focuses on the detection of salient data
points in chart images. Figure 1 shows sample results of our value detection
method on a number of scientific bar charts, demonstrating robustness to
varying styles and spurious artifacts. Note that this work does not include
the detection of text labels nor the alignment of these with detected data
points to complete the value extraction process.

Source code and trained models are available (link in abstract), and we also
contribute several datasets, including 500 manually-annotated bar charts from
scientific articles, which we used for our model fine-tuning and evaluation;
these are significantly more varied in appearance and the presence of
artifacts than previous synthetic and web-sourced datasets (e.g. those used in
[25, 48, 26, 8, 10]). Additionally: 1) we show that our approach significantly
outperforms a capable recent method [48], 2) we assess the use of data
augmentation to better exploit synthetic data, and 3) we demonstrate that our
method can be directly and successfully applied to other chart types – pie
charts in particular.

##  2 Related Work

Ours is not the first work to investigate information extraction from data
charts. ReVision [38] and ChartSense [15], for example, demonstrated accurate
classification of different types of charts images collected from the web, and
ReVision further used a set of hand-crafted features and rules to extract
salient marks and use these to infer chart values. ChartSense, on the other
hand, is an interactive extraction tool (similar to [27] and [17]), but
includes OCR for automatic text element labeling. Other works, such as REV
[31], ChartReader [33], and that by Al-Zaidy and Giles [1], have taken a
combined approach, using rule/heuristic-based element extraction supplemented
by OCR for text elements (ChartReader also uses ML-based chart
classification). The problem is of course that rules and hand-crafted features
are brittle and must be reproduced and tuned for each new chart type – or even
for different styles and contexts.

Deep learning approaches to chart value extraction have primarily leveraged
object detection methods. Liu et al. [25] and Zhao et al. [48], for example,
apply Faster R-CNN [35] to detect chart elements; Scatteract [7] uses an
object detector called ReInspect [41]; and Ma et al. [26] use Cascade R-CNN
[5] for bounding box detection, as well as a segmentation-based point
detector. With the notable exception of [33] (which reported detection
accuracies between 22.98% and 42.10% for data values in bar charts), prior
works have generally been limited to testing and demonstration on synthetic or
otherwise very clean chart images.

In contract to these and similar methods (see Davila et al. [9] for a recent
survey), we opt to exclusively use point-based value detection, as we believe
it is more readily adaptable to detecting the salient elements in different
chart types, is easier to annotate, can be readily extended to additional
element types (e.g. error bars and stacked charts), and better supports
integration with additional element labeling (e.g. OCR). Such point-based
detection has not been common in chart value extraction, but has seen used for
other purposes, such as radioastronomy and particle imaging [11, 43]. In this
work, we target complex scientific charts, and so we particularly aim to
reduce prior works’ assumptions about image simplicity, regularity of style
and appearance, and presence of artifacts.

##  3 Preface: Isolating Bar Charts in Documents

Visual value extraction of document-embedded charts could be attempted with
whole-page images; however in this work we separate the problem of chart
detection from value detection, and focus on the latter for several reasons:
1) There are already strong vision methods which can be adapted for detecting
charts of a particular class or recognizing charts among all extracted
document figures – as well as document layout analysis methods and datasets
for segmenting document pages into their several constituent regions [47, 39,
30, 21]. 2) In many cases, images can be extracted intact from documents (e.g.
those in HTML/XML formats as in PubMed Central, and supplementary material or
archived sources as found in arXiv), further simplifying that problem to a
classification one. And finally, 3) because image resolution must be preserved
at a quality that enables useful value extraction, doing so with whole-
document images would require significant (and largely wasted) computational
and memory expense.

Although detection of bar chart figures is not the focus of this work, we did
seek to verify for ourselves the performance and effects of applying existing
methods to this task. So in addition to the set of bar charts we value-
annotated, we also labeled a much larger dataset of isolated bar chart
bounding boxes (Sec. 5.1), and used it to train a standard YOLOv3 [34, 24]
object detection model. Our testing showed that the model (pre-trained on COCO
[22]) could be readily fine-tuned to detect bar charts with a mAP (mean
average precision) of 95.2% at 0.5 IoU (intersection-over-union), though
detection performance drops significantly at higher precision thresholds
(88.3% @0.75 IoU, 18.1% @0.9 IoU).

As higher IoU corresponds to better-fitting bounding boxes, our tentative
conclusion is that standard object detection models can accurately detect and
localize bar charts, but may be less capable of precisely determining the
chart bounding boxes (Note that we made no significant modifications to the
YOLOv3 model in our testing; there is likely plenty of room for improvement).
A value extraction method such as ours (Sec. 4), if used in a pipeline with
such chart detection methods, would therefore need to contend with spurious
artifacts from poor cropping of detected charts. One practical way to address
this is to intentionally over-crop the detected charts before sending them to
the value detection model, and training that model using data designed with
this expectation. We followed precisely this approach, as described in Sec.
5.2 \- 5.4.

##  4 A Point Proposal Network for Chart Value Detection

Figure 2: Architecture summary for our Point Proposal Network (PPN). Features
are extracted using a ResNet50 backbone; the main PPN constitutes a 4-layer
CNN with a classification and regression head, producing class and point-
tuning regression maps.

We designed a convolutional neural network model which predicts a set of
points in an image, and classifies them into a set of defined labels –
directly analogous to object detection, but restricting the problem to
predicting zero-size points rather than bounding boxes. We believe this is
better suited to the problem of value extraction in different chart and graph
types, whereas object detection models may not provide the same flexibility.
For example, an object detection model may be readily adapted to detect
elements in bar or pie charts (e.g. [25]), but it is not clear how it might be
applied to extracting values from line graphs, dense histograms, stacked
charts, etc. Furthermore, a point-based approach supports additional
granularity, such as identifying error bars. To support bar charts of
arbitrary appearance, we detect two element types: the peaks of each bar, and
the labeled tick marks along the value axis (but not the text labels
themselves). The chart origin is treated as a tick mark special case.

Our model (Figure 2) uses a ResNet50 backbone [13] for feature extraction at
an input image resolution of 224x224, with two feature maps extracted after
the 3rd (56x56, 256-channel) and final bottleneck blocks (7x7, 2048-channel).
A 1x1 convolutional layer is used to reduce the channel depth of the second
feature map to 256, and it is then up-sampled to 56x56 via bilinear
interpolation and added to the first feature map. This approach was taken to
preserve both deep semantic features and fine detail – necessary for data
value extraction. The 56x56 feature map resolution was chosen for its natural
fit within the existing ResNet50 model, and for consistency with the maximum
density of data points expected in scientific bar charts (denser charts may be
encountered or produced; we settled on a map size that covers a significant
majority of useful cases).

The feature map passes through a further 4-layer convolutional Point Proposal
Network (PPN) with two per-pixel prediction heads: a classifier and a fine-
tuning regression head. This is akin to region proposal networks used in e.g.
Faster R-CNN [35], except that the PPN can make point predictions directly,
without any need for ROI (region-of-interest) cropping to classify (and
further refine) bounding boxes. The classification head predicts one of three
classes: background, bar, and tick; the regression head predicts the point
offset. Figure 3 shows examples of the class maps produced for a sample chart.

Figure 3: An example of the class maps produced by our PPN. Left-to-right:
input image superimposed with our model’s bar and tick point predictions, the
background point map (negated), bar class map, and tick class map. The source
chart image is intentionally over-cropped. (Sample chart is from [29])

###  4.1 Predicting Data Points

After the PPN model produces 56x56 classification and regression maps, these
are converted to a list of predicted points for each bar and tick. This
prediction is done by first masking the class map at a threshold over the
background class (in practice, we found values between 0.9 and 0.99 work
well), and then selecting pixels that were classified as bars or ticks with
scores greater than some threshold (we found 0.75 works well). The
corresponding values in the regression map are used to tune the position of
each detected point, and the resulting points are passed through a non-maximum
suppression (NMS) function to remove duplicate detections. Our NMS
implementation sorts points in each class by confidence score and iteratively
removes low-scoring points within a threshold radius (we used a default
threshold of 1.5).

For the bar class, we made the NMS threshold ellipse anisotropic, giving it a
5x greater reach in the horizontal direction, as bar widths can vary
significantly from chart to chart. The final list of bar and tick point
predictions can then be visualized (as in Figures 1,3), used together with a
text label detector to extract numerical values (not covered in this work, see
Sec. 7.3), or compared to sets of ground truth points during training, as
discussed in the following section.

###  4.2 Loss Function and Training

During training, the model has access to the input image and to corresponding
ground truth labels for the lists of bar and tick points, and a class and
regression map generated from these lists. The PPN model, together with the
map-to-point conversion and NMS described in Sec. 4.1, generates predictions
for each of these, so the loss function consists of terms for each target as
well. In theory, we could derive a loss from just the lists of points, but we
suspected that this alone would make for a poor training signal – especially
at earlier epochs – and confirmed as much, finding that such an approach fails
to train effectively. Our loss function is then

$L=\lambda_{1}L_{cls}+\lambda_{2}L_{reg}+\lambda_{3}L_{pts}+\lambda_{4}L_{align}.$
(1)

The class map loss $L_{cls}$ is a regular cross-entropy loss, and the
regression map loss $L_{reg}$ uses a masked mean squared error:

$L_{reg}=\sum_{i,j}(\hat{r}_{i,j}-r_{i,j})^{2},\mathrm{if}\;\hat{c}_{i,j}\neq
background,$ (2)

where $\hat{r}_{i,j}$,$r_{i,j}$, and $\hat{c}_{i,j}$ are, respectively: the
predicted regression value, ground truth regression value, and predicted class
value at map position $i,j$. The points list loss is

$L_{pts}=\sum_{p}dist(\hat{p},p),\mathrm{if}\;dist(\hat{p},p)<T_{det},$ (3)

where $\hat{p}$ and $p$ are predicted and ground truth within-class points,
$dist$ is Euclidean distance, and $T_{det}$ is a max detection distance
threshold (default=1.5).

We followed a training schedule in which $\lambda_{3}$ starts at zero and is
“turned on” only when the training loss begins to plateau. At that point it is
weighted to $1/10$ relative to $\lambda_{1,2}$, due to scale differences. We
did not find training to be particularly sensitive to these other term
weights. When training had again plateaued, we added a fourth term to the loss
function to promote horizontal alignment of the predicted tick marks:

$\lambda_{4}L_{align}=\sum_{tp}abs(\widehat{tp}_{x}-mean(tp_{x}))$ (4)

This tick alignment loss is an example of a chart-type-specific loss term; we
expect analogues can be designed for other chart types as well. We set
$\lambda_{4}=1/1000$ relative to $\lambda_{1,2}$, again due to scale
differences. We trained our model with regular stochastic gradient descent,
using a learning rate of 1e-3, momentum = 0.9, and a batch size of 64.
Training took about 3-4 minutes per epoch (for 5000-image datasets) on two
NVIDIA 1080Ti GPUs.

##  5 Datasets

We annotated and generated five different datasets, described here. The first
is for bar chart detection, on which we evaluated baseline performance and
detection qualities of a standard object detection model (see Sec. 3). The
following three datasets are for the main problem this work addresses: data
point detection in bar charts to support automated value extraction. The last
dataset is for value detection in pie charts. All datasets and scripts are
available at https://github.com/BNLNLP/PPN_model.

Table 1:  Summary of dataset types, uses and statistics.  Dataset | Type | Use | Images | Labels
---|---|---|---|---
5.1 | Manual | Bar chart detection | 1592 | 3548
5.2 | Synthetic | Value detection (bar charts) | 5000 | 65618
5.3 | Manual | Value detection (bar charts) | 500 | 7109
5.4 | Augmented | Value detection (bar charts) | 5000 | 65618
5.5 | Synthetic | Value detection (pie charts) | 5000 | 32718

###  5.1 Bar chart detection

We collected 673 article PDFs from the PubMed Central Open Access Subset [6],
and rendered each page at 72dpi (using the pdftoppm utility from the standard
poppler-utils package), resulting in 1592 document page images with a typical
resolution of 612x792. We then manually labeled bounding boxes for each bar
chart in each page (using LabelImg [23]), whether they were standalone figures
or were embedded in part of another figure. The selection criteria we followed
excluded similar but distinct chart types such as histograms, and we avoided
including figures that contained multiple types of overlapping charts (e.g. a
line graph superimposed on a bar chart) or charts with multiple data points
per category (e.g. stacked charts or waterfall/floating charts). We did,
however, include bar charts in any orientation (horizontal, vertical, upside-
down), at any size, in any color, and with or without: error bars, 3D effects,
legends, category/data labels, or embedded titles/captions/comments or other
artifacts. The only strict criteria was that each chart must include a labeled
value axis.

In total, we annotated bounding boxes for 3548 scientific bar charts, of which
only 144 were horizontal (a 24:1 ratio). This observation informed our
decision to restrict our other datasets to vertical bar charts only, as there
would be disproportionately few horizontal ones to sample from, and we expect
our method and results to generalize without issue to those, with only minor
adaptation.

###  5.2 Synthetic bar charts for value detection

We used the Matplotlib library in Python to programmatically generate large
numbers of bar charts with randomly varying numbers of bars (2-10), bar widths
(0.1-0.9), colors (123 choices from the CSS4_COLORS space, excluding those
deemed too light), and values. We restricted our generation to plain bar
charts: no stacked charts, no grouped charts, and no error bars. This is due
to this dataset only being used for pre-training. Nonetheless, these synthetic
charts are similar in complexity to those used in previous works (e.g. [25,
48, 26]).

We generated 5000 charts (80:20 training:val split), and augmented these
simply by applying random padding (up to 50% per dimension) and cropping
(taking care to preserve the full chart within cropping bounds). Bar and tick
mark annotations were extracted programmatically during chart generation. We
also generated a separate test set of 1000 charts in the same manner.

###  5.3 Manually annotated real charts values

We randomly sampled 500 vertical-only bar charts from dataset 5.1 and re-
rendered the PDF pages containing them at 300dpi. We also used the bounding
box annotations from 5.1 to produce crops of the bar charts at high
resolution, with crops padded by 50% to intentionally incorporate parts of
surrounding text, nearby figures and other artifacts. This is to accommodate
the expectation that automatic bar chart extraction may yield imperfect crops
(see Sec. 3), so a value detection method should be robust to such artifacts.

We then used the online tool makesense.ai to manually label the bar, tick, and
origin points in each chart as precisely as practical (temporary image
recoloring and contrast adjustments made this process easier). We estimate
that our manual annotations are generally within 3 pixels distance of their
true value (median image size 663x475, range approximately $\pm$3X). We
converted these annotations to the simple label format we designed for 5.2.

###  5.4 Real-augmented synthetic charts

As it is much quicker to annotate whole chart bounding boxes than individual
data points in those charts (compare image and label numbers in datasets 5.1
and 5.3 in Table 1) we sought to exploit annotations of the former type to
augment our synthetically generated bar charts (dataset 5.2). A significant
amount of the variance in a published scientific chart is found in its
immediate surroundings (i.e. not only in its interior contents), so we decided
to use these surroundings to improve the local feature learning of our
convolutional PPN model.

We took random crops from the regions immediately adjacent to the bar chart
bounding boxes in dataset 5.1 (up to 50% beyond the chart bounds) and pasted
these to the padded surroundings of our synthetic charts. Before doing so, we
randomly applied morphological erosion (up to 4 pixels) and dilation (up to 1
pixel) to the synthetic charts in order to diversify their line thicknesses.
Finally, we randomly cropped the chart images (keeping the full chart in
frame) and rescaled along each dimension by up to 50%.

###  5.5 Synthetic pie charts

To demonstrate the extensibility of our method to other chart types, we also
generated a dataset of synthetic pie charts with Matplotlib. For these, we
produced two classes of point annotations: the boundaries between pie wedges,
and the pie chart centroid. From these points, the size/values of the pie
chart elements (and thus the chart’s original data) can be readily determined.

##  6 Experiments

We trained and evaluated our model on synthetic data 5.2, manually annotated
real data 5.3, and real-augmented synthetic data 5.4. Our primary evaluation
metric for bar charts is the accuracy of data point predictions for bar peaks
and value-axis tick marks. We also compare our method to [48] and demonstrate
easy extensibility of our approach to other chart types, particularly pie
charts.

###  6.1 Metrics

We evaluated our models’ data point predictions using F1 at a fixed precision
threshold measured in units of the PPN class map. As our model only predicts a
maximum of one data point per map cell, we chose a precision threshold of 1.5
cell widths to determine prediction recall and precision. The classification
map has a 56x56 resolution, so this corresponds to a maximum error of 1.5 / 56
= 2.68% (a stricter precision criterion than used in e.g. [48, 33]). At this
threshold, we can determine how accurately the model detects the location of
data points of interest, without setting an overly strict precision
requirement. Empirically, we found this threshold to be the coarsest whose
prediction results looked ‘close enough’ (all sample results in Figures 1,3,
and 5 were produced at this threshold), but we also evaluated performance at
other thresholds (see Figure 4D).

We chose to set the detection threshold with respect to the fixed-size feature
map, rather than in image space, in order to be representative of our chart
value extraction use case: error should be proportional to the size of the
chart. For the same reason, F1 is evaluated per image then averaged over the
test set. If we instead measured F1 over all target points in an image set,
results would appear artificially better ($\sim$5-10% uplift) because good
performance on charts with many points outweighs poor performance on charts
with few points.

###  6.2 Results

Table 2:  Experimental results for our model, trained and evaluated on our datasets (see Table 1). All results on test data unseen during training or hyperparameter optimization, measured at precision thresh=1.5. Real scientific charts in bold.  Trained on |  | Evaluated on |  | F1
---|---|---|---|---
5.2 (synthetic bar charts) |  | 5.2 |  | 0.9810
|  | 5.3 |  | 0.5549
5.3 (real bar charts) |  | 5.3 |  | 0.8705
5.4 (augmented synthetic bar charts) |  | 5.4 |  | 0.9307
|  | 5.3 |  | 0.6621

We first pre-trained our model on synthetic data only. As described in 5.2,
these synthetic charts were quite simplistic and so had a pronounced
‘artificial’ look to them (similar to those seen in prior works) that is not
necessarily representative of published scientific charts. As shown in Table 2
and Figure 4A, the pre-trained model achieves strong performance on synthetic
data (0.9810 F1), but is predictably less capable when evaluated on real data
(0.5549 F1). Note the effects of our training schedule (Sec. 4.2): $L_{pts}$
is added at epoch 1000 and has a pronounced effect on prediction in synthetic
data; $L_{align}$ is added at epoch 1500 and has a more significant impact on
real-chart performance.

We next continued training by fine-tuned on our manually annotated dataset.
Figure 4B shows evaluated performance results on a test-only holdout set of 54
charts. We again followed a multi-step training schedule: $L_{pts}$ was
enabled after epoch 5000 and $L_{align}$ was added after epoch 7000 (note that
detection performance on tick marks improves markedly after that point). Our
model ultimately achieved a mean F1 of 0.8705 on real test data (see Figure
1).

Figure 4:  A: Synthetic-trained model performance on synthetic and manually-
annotated real data. B: Model tuned on manually-annotated real data. C: Model
trained on real-augmented synthetic dataset. The horizontal line is the peak
performance (on real data) of model trained on regular synthetic data. D:
Impact of different precision thresholds on detection performance (same model
as B). Thresh=1.5 used for A-C. Figure 5: Sample results for model trained
only on augmented synthetic data. Left two: synthetic test samples, showing
real-document crops pasted at edges. Right two: real charts from scientific
articles (from [19, 44]). Note the increased detection errors. Figure 6:  A.
Sample results of our model on charts from [48, 12]. Note the similarity to
our synthetic samples. B. Sample results of our unmodified model trained on
synthetically generated pie charts, demonstrating direct method extensibility.

We also assessed the ability of our model to learn value detection for real
scientific bar charts by training only on synthetic data that was augmented
with fragments of unannotated real chart images (dataset 5.4). As shown in
Figure 4C, training the same model on this augmented data allows it to
significantly outperform a model trained on regular synthetic data (even with
5x less training). In this case, $L_{pts}$ was enabled after epoch 100 and
$L_{align}$ after epoch 200. This model achieved 0.6621 F1 on real data.
Figure 5 shows samples of this model’s detection performance on real-augmented
synthetic data and real data.

Table 3:  Comparison of our model to [48]. All results on [12] test set. Our evaluation criteria is 1.5 cell (2.7% deviation); the criteria used in [48] is 5% deviation (2.8 cell),  |  | Performance (F1) per eval criteria
---|---|---
Model |  | Ours (1.5 cell) |  |  [48] (5% deviation)
[48] |  |  |  | 0.91
Ours (unmodified) |  | 0.8959 |  | 0.9346
Ours (tuned on [12] training set) |  | 0.9718 |  | 0.9727

We compare our method to a recent work by Zhou et al. [48] which also aimed to
release their synthetic and real data. Unfortunately, in their GitHub [12],
they report they have lost their experimental data (presumably both types),
but reproduced the synthetic data from the same procedure as in the paper (on
which their method achieved 0.91 F1 score at a deviation criterion of 5%. We
note that the visual complexity of the charts in this dataset is similar to
those in other works (see Sec. 2). We evaluated our method against Zhou’s test
dataset, at the same 5% deviation criteria (equivalent to 2.8 cell widths on
our 56x56 classification grid), as well as at our own, stricter default
precision threshold of 1.5 cell widths (2.7% deviation). Quantitative results
are in Table 3, showing that both our unmodified method (i.e. trained on our
own data) and our method trained on their data significantly outperform their
results, even at a stricter precision threshold. Sample results for this
dataset are shown in Figure 6A; note the regularity/simplicity of these
charts, compared to those in Figure 1.

Finally, to demonstrate the ease and success of extending our method to
different chart types, we trained our unmodified model on synthetic pie charts
(dataset 5.5). No change was made to our method; only the annotations were
replaced. Our method achieved an F1 of 0.8319 (@1.5-cell thresh) on detection
of salient pie chart elements from a holdout test set; sample results are
shown in Figure 6B. Numerous additional sample results are provided in the
supplemental material for all models and datasets.

##  7 Discussion

This work shows that a point-based method can very effectively detect data
values in complex scientific bar charts with widely varying appearance, and
that synthetic data and simple augmentations can be well-leveraged to train
such a model with efficient use of manually labeled data. Our approach is
efficient (only 3% larger than a ResNet50 model), easily extensible (as we
demonstrate with pie charts) and designed to be readily incorporated in a data
extraction pipeline or as a component of an end-to-end model.

###  7.1 Extensibility

A key advantage we see in taking a point-based detection approach is the ready
extensibility of this method to additional chart elements and chart types. One
obvious example within the scope of bar charts is adding annotations for error
bars as an additional point class and training the same model to predict these
without modification. The same could be done with stacked charts, wherein
multiple points could be labeled for each bar. In either case, the same model
definition could be used, with the only change needed in the number of point
classes. As demonstrated in Sec. 4.2, custom loss terms could also be added if
desired and used to improve detection performance and consistency with
particular chart expectations. Even very simple synthetic data (e.g. dataset
5.2) can be used for model pre-training and augmented as discussed in Sec.
5.4.

For other chart types, the same approach holds. Histograms may be annotated
and detected just like bar charts, with the only major differences being that
the labeling of the data axis may be optional and the density of the data
points may be higher (histogram bins may also be labeled differently than
bars, but the alignment of such labels is independent of the detected data
points). Scatter plots, dot plots and other point-based charts have a direct
correspondence to a point-detection approach. We demonstrated that pie charts
can be well described by labeling points at their center and the division
between each wedge, and that our method can learn these without modification.
Even line graphs can be detected as a dense set of connected points. In each
case, salient data points can be detected with the same Point Proposal
Network, with only annotation changes needed (and optionally scaling up for
higher densities).

###  7.2 Impact

The aim of this work is to extend researchers’ ability to draw on information
that is at present inaccessible to machine processing. We believe this effort
and others like it will not only improve access to scientific knowledge, but
also enable new uses of that knowledge, particularly when applied at large
scales. Just as large-scale text mining has revolutionized natural language
processing, so too do we think mining of charts and graphics can change the
way the scientific literature is used as a resource for new research and
discoveries. We also believe this type of work is aligned with broader efforts
to improve scientific information access and usage (e.g. [46]), particularly
concerning the reusability of data. Outside the scientific literature, the
method we propose can of course be applied and have value for examination and
analysis of any source of charts (e.g. business, educational, government,
etc.).

One important risk worth noting is the potential for over-reliance on ML-
extracted information. As with other ML-based data mining techniques, the
value detection method we propose is hopefully useful, but necessarily
imperfect. Caution must of course be employed when using data extracted with
such techniques, especially when done at scales that restrict human oversight.
This is less of a concern if the method is used in an interactive setting.

###  7.3 Limitations

This work does not address the full problem of bar chart value extraction from
documents, but rather focuses on salient data point detection. As such,
practical end-user utilization of the method we present would require two
additional components: detection/cropping of charts in document images, and
label detection/recognition (e.g. via OCR techniques). As we showed in Sec. 3,
chart detection can be readily achieved with reasonably high accuracy using
commonplace object detection methods such as YOLOv3. Customization of such
models (e.g. to improve bounding box refinement) will certainly yield even
better performance, though we developed our PPN value detection method and
datasets with the expectation that some imperfection should be anticipated in
chart crops.

Concerning label recognition, we designed our point-based approach such that
its results can be incorporated with those of a text-detection model to
complete the value extraction task. The position of the detected bar peaks and
value-axis tick marks can be aligned with detected text to associate
corresponding text labels to detected data points. We expect this approach
will be particularly useful for detecting and extracting grouped bar charts,
and for extracting values with nonlinear (e.g. logarithmic) or
broken/segmented data axes as each tick mark can be labeled individually.
Alternatively, our method can be combined with a chart detector and text label
detector in a single model trained end-to-end, thus enabling direct chart
value extraction from full document images (or collections of them), without
the need for a pipeline approach.

Our decision to use a 56x56 feature map may prove a limitation for cases of
particularly large or dense charts, though the method may of course be scaled
up. The selection and generation criteria we used for our datasets –
particularly the exclusion of horizontal bar charts and omission of additional
data points such as error bars – is also not fully representative of the
charts found in the scientific literature. Again, however, we anticipate that
the point-based approach we pursued is particularly well suited to extending
this method to these and other use cases.

###  7.4 Future work

We believe this work presents several opportunities for further research.
Beyond extension to data extraction from other chart types (Sec. 7.1), we’ve
identified two particular research directions that merit investigation. The
first is to combine our value detection method with a chart detector and text
label detector in a single model, thus enabling end-to-end value extraction
directly from full document images (or collections of them), without the need
for a pipeline approach. The advantages of such an approach include reduction
of performance bottlenecks, as well as access to additional information which
may inform knowledge extraction (e.g. figure captions).

The second promising research avenue is the incorporation of an attention
mechanism in the value extraction data path. If used to attend over a chart
feature map, this could have particular value for identifying data groups,
better predicting the number of data points, and recognizing spurious
outliers. And if built into an end-to-end model as described above, attention
could be further used to refine connections between multiple information
classes (e.g. for knowledge graph generation informed by text, tables, and
graphics).

## References

  * [1] Al-Zaidy, R.A., Giles, C.L.: Automatic extraction of data from bar charts. In: Proceedings of the 8th international conference on knowledge capture. pp. 1–4 (2015)
  * [2] Algorri, M.E., Zimmermann, M., Hofmann-Apitius, M.: Automatic recognition of chemical images. In: Eighth Mexican International Conference on Current Trends in Computer Science (ENC 2007). pp. 41–46. IEEE (2007)
  * [3] Beltagy, I., Lo, K., Cohan, A.: Scibert: A pretrained language model for scientific text. arXiv preprint arXiv:1903.10676 (2019)
  * [4] Brackertz, M., Gong, Z., Leers, J., Renkawitz, R.: p66alpha and p66beta of the mi-2/nurd complex mediate mbd2 and histone interaction. Nucleic acids research 34(2), 397–406 (2006)
  * [5] Cai, Z., Vasconcelos, N.: Cascade r-cnn: Delving into high quality object detection. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 6154–6162 (2018)
  * [6] Central, P.: https://www.ncbi.nlm.nih.gov/pmc/tools/openftlist/. pubmed central open access subset.
  * [7] Cliche, M., Rosenberg, D., Madeka, D., Yee, C.: Scatteract: Automated extraction of data from scatter plots. In: Joint European Conference on Machine Learning and Knowledge Discovery in Databases. pp. 135–150. Springer (2017)
  * [8] Davila, K., Kota, B.U., Setlur, S., Govindaraju, V., Tensmeyer, C., Shekhar, S., Chaudhry, R.: Icdar 2019 competition on harvesting raw tables from infographics (chart-infographics). In: 2019 International Conference on Document Analysis and Recognition (ICDAR). pp. 1594–1599. IEEE (2019)
  * [9] Davila, K., Setlur, S., Doermann, D., Bhargava, U.K., Govindaraju, V.: Chart mining: a survey of methods for automated chart analysis. IEEE transactions on pattern analysis and machine intelligence (2020)
  * [10] Davila, K., Tensmeyer, C., Shekhar, S., Singh, H., Setlur, S., Govindaraju, V.: Icpr 2020-competition on harvesting raw tables from infographics. In: International Conference on Pattern Recognition. pp. 361–380. Springer (2021)
  * [11] Dominé, L., de Soux, P.C., Drielsma, F., Koh, D.H., Itay, R., Lin, Q., Terao, K., Tsang, K.V., Usher, T.L.: Point proposal network for reconstructing 3d particle endpoints with subpixel precision in liquid argon time projection chambers. Physical Review D 104(3) (Aug 2021). https://doi.org/10.1103/physrevd.104.032004, http://dx.doi.org/10.1103/PhysRevD.104.032004
  * [12] F. Zhou, e.a.: https://github.com/csuvis/barchartreverseengineering. barchartreverseengineering.
  * [13] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 770–778 (2016)
  * [14] Jain, R., Wigington, C.: Multimodal document image classification. In: 2019 International Conference on Document Analysis and Recognition (ICDAR). pp. 71–77. IEEE (2019)
  * [15] Jung, D., Kim, W., Song, H., Hwang, J.i., Lee, B., Kim, B., Seo, J.: Chartsense: Interactive data extraction from chart images. In: Proceedings of the 2017 chi conference on human factors in computing systems. pp. 6706–6717 (2017)
  * [16] Kononova, O., Huo, H., He, T., Rong, Z., Botari, T., Sun, W., Tshitoyan, V., Ceder, G.: Text-mined dataset of inorganic materials synthesis recipes. Scientific data 6(1), 1–11 (2019)
  * [17] Larsen, K.P.: http://www.graphreader.com/. online tool for reading graph image values.
  * [18] Lee, J., Yoon, W., Kim, S., Kim, D., Kim, S., So, C.H., Kang, J.: Biobert: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics 36(4), 1234–1240 (2020)
  * [19] Levine, L.: Tamoxifen and the rafoxifene analog ly117018: their effects on arachidonic acid release from cells in culture and on prostaglandin i 2 production by rat liver cells. BMC cancer 4(1), 1–8 (2004)
  * [20] Lewkowich, I.P., Herman, N.S., Schleifer, K.W., Dance, M.P., Chen, B.L., Dienger, K.M., Sproles, A.A., Shah, J.S., Kohl, J., Belkaid, Y., et al.: Cd4+ cd25+ t cells protect against experimentally induced asthma and alter pulmonary dendritic cell phenotype and function. Journal of Experimental Medicine 202(11), 1549–1561 (2005)
  * [21] Li, M., Xu, Y., Cui, L., Huang, S., Wei, F., Li, Z., Zhou, M.: Docbank: A benchmark dataset for document layout analysis. arXiv preprint arXiv:2006.01038 (2020)
  * [22] Lin, T.Y., Maire, M., Belongie, S., Bourdev, L., Girshick, R., Hays, J., Perona, P., Ramanan, D., Zitnick, C.L., Dollár, P.: Microsoft coco: Common objects in context (2015)
  * [23] Lin, T.: https://github.com/tzutalin/labelimg. labelimg.
  * [24] Linder-Noren, E.: https://github.com/eriklindernoren/pytorch-yolov3. a minimal pytorch implementation of yolov3. (2018)
  * [25] Liu, X., Klabjan, D., NBless, P.: Data extraction from charts via single deep neural network. arXiv preprint arXiv:1906.11906 (2019)
  * [26] Ma, W., Zhang, H., Yan, S., Yao, G., Huang, Y., Li, H., Wu, Y., Jin, L.: Towards an efficient framework for data extraction from chart images. arXiv preprint arXiv:2105.02039 (2021)
  * [27] Marin, F., Rohatgi, A., Charlot, S.: Webplotdigitizer, a polyvalent and free software to extract spectra from old astronomical publications: application to ultraviolet spectropolarimetry. arXiv preprint arXiv:1708.02025 (2017)
  * [28] Mondal, A., Lipps, P., Jawahar, C.: Iiit-ar-13k: a new dataset for graphical object detection in documents. In: International Workshop on Document Analysis Systems. pp. 216–230. Springer (2020)
  * [29] Nakamura, T.Y., Jeromin, A., Smith, G., Kurushima, H., Koga, H., Nakabeppu, Y., Wakabayashi, S., Nabekura, J.: Novel role of neuronal ca2+ sensor-1 as a survival factor up-regulated in injured neurons. The Journal of cell biology 172(7), 1081–1091 (2006)
  * [30] Oliveira, S.A., Seguin, B., Kaplan, F.: dhsegment: A generic deep-learning approach for document segmentation. In: 2018 16th International Conference on Frontiers in Handwriting Recognition (ICFHR). pp. 7–12. IEEE (2018)
  * [31] Poco, J., Heer, J.: Reverse-engineering visualizations: Recovering visual encodings from chart images. In: Computer Graphics Forum. vol. 36.3, pp. 353–363. Wiley Online Library (2017)
  * [32] Rajasekharan, J., Scharfenberger, U., Gonçalves, N., Vigário, R.: Image approach towards document mining in neuroscientific publications. In: International Symposium on Intelligent Data Analysis. pp. 147–158. Springer (2010)
  * [33] Rane, C., Subramanya, S.M., Endluri, D.S., Wu, J., Giles, C.L.: Chartreader: Automatic parsing of bar-plots. In: 2021 IEEE 22nd International Conference on Information Reuse and Integration for Data Science (IRI). pp. 318–325. IEEE (2021)
  * [34] Redmon, J., Farhadi, A.: Yolov3: An incremental improvement. arXiv preprint arXiv:1804.02767 (2018)
  * [35] Ren, S., He, K., Girshick, R., Sun, J.: Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in neural information processing systems 28, 91–99 (2015)
  * [36] Roberts, R.J.: Pubmed central: The genbank of the published literature (2001)
  * [37] Rocha, S., Campbell, K.J., Roche, K.C., Perkins, N.D.: The p53-inhibitor pifithrin-alpha inhibits firefly luciferase activity in vivo and in vitro. BMC molecular biology 4(1), 1–8 (2003)
  * [38] Savva, M., Kong, N., Chhajta, A., Fei-Fei, L., Agrawala, M., Heer, J.: Revision: Automated classification, analysis and redesign of chart images. In: Proceedings of the 24th annual ACM symposium on User interface software and technology. pp. 393–402 (2011)
  * [39] Soto, C., Yoo, S.: Visual detection with context for document layout analysis. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). pp. 3464–3470 (2019)
  * [40] Srahna, M., Leyssen, M., Choi, C.M., Fradkin, L.G., Noordermeer, J.N., Hassan, B.A.: A signaling network for patterning of neuronal connectivity in the drosophila brain. PLoS biology 4(11), e348 (2006)
  * [41] Stewart, R., Andriluka, M., Ng, A.Y.: End-to-end people detection in crowded scenes. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 2325–2333 (2016)
  * [42] Subbian, S., Mehta, P.K., Cirillo, S.L., Cirillo, J.D.: The mycobacterium marinum mel2 locus displays similarity to bacterial bioluminescence systems and plays a role in defense against reactive oxygen and nitrogen species. BMC microbiology 7(1), 1–13 (2007)
  * [43] Tilley, D., Cleghorn, C.W., Thorat, K., Deane, R.: Point proposal network: Accelerating point source detection through deep learning. arXiv preprint arXiv:2008.02093 (2020)
  * [44] Walker, W., Zhou, Z.Q., Ota, S., Wynshaw-Boris, A., Hurlin, P.J.: Mnt–max to myc–max complex switching regulates cell cycle entry. The Journal of cell biology 169(3), 405–413 (2005)
  * [45] Wang, L.L., Lo, K., Chandrasekhar, Y., Reas, R., Yang, J., Eide, D., Funk, K., Kinney, R., Liu, Z., Merrill, W., et al.: Cord-19: The covid-19 open research dataset. ArXiv (2020)
  * [46] Wilkinson, M.D., Dumontier, M., Aalbersberg, I.J., Appleton, G., Axton, M., Baak, A., Blomberg, N., Boiten, J.W., da Silva Santos, L.B., Bourne, P.E., et al.: The fair guiding principles for scientific data management and stewardship. Scientific data 3(1), 1–9 (2016)
  * [47] Zhong, X., Tang, J., Yepes, A.J.: Publaynet: largest dataset ever for document layout analysis. In: 2019 International Conference on Document Analysis and Recognition (ICDAR). pp. 1015–1022. IEEE (2019)
  * [48] Zhou, F., Zhao, Y., Chen, W., Tan, Y., Xu, Y., Chen, Y., Liu, C., Zhao, Y.: Reverse-engineering bar charts using neural networks. Journal of Visualization 24(2), 419–435 (2021)