Andrew Davidson, Arthur Morley-Bunker, George Wiggin, Logan Walker, Gavin
Harris, Ramakrishnan Mukundan, kConFab Investigators 11institutetext:
Department of Computer Science and Software Engineering, University of
Canterbury, Christchurch, New Zealand
11email<EMAIL_ADDRESS>22institutetext: Department of
Pathology and Biomedical Science, University of Otago, Christchurch, New
Zealand 33institutetext: Canterbury Health Laboratories, Christchurch, New
Zealand 44institutetext: Sir Peter MacCallum Department of Oncology, The
University of Melbourne, Melbourne, Victoria, Australia 55institutetext: Peter
MacCallum Cancer Center, Melbourne, Victoria, Australia

# Grey Level Texture Features for Segmentation of Chromogenic Dye RNAscope
From Breast Cancer Tissue

Andrew Davidson* 11 Arthur Morley-Bunker 22 George Wiggins 22 Logan Walker 22
Gavin Harris 33 Ramakrishnan Mukundan 11 kConFab Investigators 4455

###### Abstract

Chromogenic RNAscope dye and haematoxylin staining of cancer tissue
facilitates diagnosis of the cancer type and subsequent treatment, and fits
well into existing pathology workflows. However, manual quantification of the
RNAscope transcripts (dots), which signify gene expression, is prohibitively
time consuming. In addition, there is a lack of verified supporting methods
for quantification and analysis. This paper investigates the usefulness of
gray level texture features for automatically segmenting and classifying the
positions of RNAscope transcripts from breast cancer tissue. Feature analysis
showed that a small set of gray level features, including Gray Level
Dependence Matrix and Neighbouring Gray Tone Difference Matrix features, were
well suited for the task. The automated method performed similarly to expert
annotators at identifying the positions of RNAscope transcripts, with an
$F_{1}$-score of 0.571 compared to the expert inter-rater $F_{1}$-score of
0.596. These results demonstrate the potential of gray level texture features
for automated quantification of RNAscope in the pathology workflow.

###### keywords:

Biomedical image processing · Image segmentation · Cancer · RNAscope · Grey
level features

##  1 Introduction

In 2020, there were over 2.3 million new cases of breast cancer, and 685,000
deaths caused by breast cancer [2]. The effectiveness of breast cancer
treatments vary based on some qualities of the cancer, therefore, it is
crucial that these qualities are classified accurately and consistently. Over
the last several years the number of cancer cases has increased, and the
number of cases is projected to continue increasing [2]. Together with a
shortage in the number of pathologists available [10], these trends have led
to an increased pathology workload.

Within the last decade whole slide imaging has become common, which is the
practice of using advanced digital scanners to produce high resolution images
of patient tissue [6, 5]. There is an emerging opportunity to apply image
processing methods to whole slide images, to produce consistent and exhaustive
quantification of features relevant to breast cancer diagnosis.

###  1.1 RNAscope Staining

RNAscope is an in situ hybridization assay that is used to detect the presence
of certain RNA sequences in tissue [15]. It allows the expression of specific
gene sequences in a tissue sample to be visually quantified as stained dots
(transcripts). The density of RNAscope transcripts is an indicator of the
level of gene expression. This technique is applicable to the problem of
breast cancer diagnosis because the ideal treatment differs based on some
genetic characteristics of the tumour tissue, such as the level of expression
of the Erb-B2 Receptor Tyrosine Kinase 2 (ERBB2) gene, also referred to as
HER2 status.

To fit most easily into existing pathology workflows, a chromogenic RNAscope
dye can be applied to tissue alongside haematoxylin, which is a stain that
gives good definition to the nuclei. A stained sample can then be scanned on a
slide in normal lab conditions to produce a single image. Fluorescent RNAscope
dyes are also available and can be used to produce spectral image stacks
(multiple images) that are much simpler to quantify RNAscope from. However,
the fluorescent dyed samples deteriorate quickly, require a dark room to scan,
and are more difficult to prepare. Therefore, this paper will solely
investigate the segmentation of chromogenic RNAscope.

###  1.2 RNAscope Segmentation

The cases that are important to accurately segment are those with low or non-
existent RNAscope staining. If there is heavy RNAscope staining, the staining
is readily evident and gene expression can be deduced without having to count
the individual transcripts. However, there are some challenges involved with
designing a robust chromogenic RNAscope segmentation method. Since the
RNAscope transcripts present at varying hue, stain intensity, and shape
depending on stain preparation and tissue characteristics, segmentation is not
straightforward; simple colour filtering is inadequate unless the preparation
is consistently impeccable. This variability can be seen in figure 1. There is
also a lack of available data with this stain type that also has annotations
for the position of each RNAscope transcript.

Figure 1: A small patch of haematoxylin and chromogenic RNAscope stained
tissue, showing RNAscope transcripts with varying shape, size, and hue. Since
the level of RNAscope staining is low, an accurate RNAscope segmentation is
useful to determine the level of gene expression.

This paper details the development and testing of a gray level texture feature
based RNAscope segmentation method, which operates on data from whole slide
images. The images contain FFPE (formalin fixed paraffin embedded) breast
cancer tissue that has been stained with haematoxylin and chromogenic
RNAscope.

##  2 Related Work

Gray level texture features are useful descriptors that are commonly used to
extract information from medical images [3]. Gray level texture feature
extraction techniques include the gray level co-occurrence matrix [8] (GLCM),
which assesses the co-occurence of gray values with a given offset; the gray
level run length matrix [4] (GLRLM), which assesses runs of same valued
consecutive pixels, and the gray level size zone matrix [14] (GLSZM), which
assesses connected zones of same valued pixels. Further methods include the
gray level dependence matrix [12] (GLDM), which assesses the frequency of
near, similar valued pixels at each gray level, and the neighbouring gray tone
difference matrix [1] (NGTDM), which assesses the variability of gray values
from the average value of their nearby pixels.

A recent study [9] compared existing chromogenic RNAscope transcript counting
methods on FFPE haematoxylin stained colorectal cancer tissues. Although it
does not directly compare segmentation performance, this study indicates that
currently available open source methods for RNAscope segmentation are slow due
to requiring user input to function; by manually selecting the RNAscope
positions or validating each RNAscope candidate. The only listed open-source
method that does not require this level of user input (Trainable WEKA
Segmentation) is not able to differentiate between single RNAscope transcripts
and clusters of them, and is not considered fully automated due to the number
of user steps required. Even the commercial methods considered (SpotStudio and
Aperio RNA ISH) required some configuration to run, and did not perform better
than the open source methods by most metrics assessed.

Although there are existing solutions, the methods published in academic
literature focus on the clinical side of the problem and have not published
accuracy metrics for their solutions. There are also few existing solutions
that use haematoxylin and chromogenic RNAscope stained tissue, which is much
more convenient to fit into the pathology workflow. Some commercial methods
exist, but they do not have a verified accuracy, and the mechanisms by which
they operate are not in the public domain.

##  3 Method

A texture feature-based method for RNAscope transcript segmentation was
explored to assess the viability of this approach. It was implemented in
Python, making use of the pyradiomics [7] library.

###  3.1 Data Acquisition

Fourty whole slide images were obtained from the University of Otago,
Christchurch. Each of these whole slide images contain a scan of a tissue
microarray of haematoxylin and brown chromogenic RNAscope stained FFPE breast
cancer tissues. The tissue was scanned at 40x magnification (0.25 microns per
pixel).

###  3.2 Expert Annotation of RNAscope Transcripts

Tissue areas with light RNAscope staining (indicated by small brown/yellow
regions in the tissue) were identified for potential use as training data. The
RNAscope transcripts on a total of 144 480x480 pixel non-overlapping patches
were annotated by a trained pathology scientist and an anatomical pathologist,
with an overlap of 19 patches annotated by both. The experts annotated 113 and
50 patches respectively.

An agreement test was conducted on the 19 patches of tissue that were
annotated with RNAscope transcript positions by both experts. This was done to
provide a baseline evaluation metric. Since each set of expert annotations
were represented by a set of coordinates, a method was needed to provide a
reasonable agreement metric. Simply assessing exact matches of coordinates
would lead to very low agreement. Therefore, pairs of annotations from each
expert that were less than 5 pixels from each other were counted as true
positives (as shown in figure 2). Each annotation could only be paired to one
annotation from the other expert. 5 pixels was chosen as the threshold value
because the visibly stained portion of an RNAscope transcript generally has a
diameter of up to 5 pixels. The $F_{1}$-score could then be calculated to
evaluate the inter-rater agreement.

Figure 2: A visualization of the area for a potential true positive match
(within 5 pixels). A clean image of the area is shown on the left, with an
annotated version of the same image on the right. The first set of annotations
are shown in red, with a circular boundary showing the maximum distance that
could be considered a match. The second set of annotations are shown as black
diamonds.

The $F_{1}$-score, representing inter-rater agreement, was found to be 0.596,
with the recall of one annotator being 0.530 and the other being 0.682. It is
important to note that a relatively low score is expected for this problem
because each RNAscope transcript needs to not only be correctly recognized,
but also detected at nearly the same position. The type of tissue staining
used also causes additional difficulties, as discussed in section 1.2.

###  3.3 Candidate Selection

To reduce computational load, a pre-processing step is taken to reduce the
number of candidate pixels. It makes use of the colour information in the
image, but cannot be made very sensitive due to the inconsistent nature of
chromogenic RNAscope staining alongside haematoxylin.

Firstly, colour deconvolution (as described in [11]) is used to separate the
RNAscope (brown) stain colour into its own image. Then, the RNAscope image is
Gaussian blurred using a 5x5 pixel circular kernel. Next, it is thresholded
using a histogram-based method to keep only dark areas that are likely to
represent RNAscope staining. A histogram for the intensities up to 250 (out of
255) is constructed. Values above 250 are excluded to remove data from the
white background (non-tissue) areas. Then, the peak value is found and used as
the threshold value, as this corresponds to the base tissue intensity. The
remaining regions after thresholding are of lower value (indicating darker
staining) than the threshold value. If more than 50% of the tissue still
remains, the threshold value is decreased to make it more selective. A
threshold decrease of 8 was found to work well. The detected RNAscope regions
are separated using the Suzuki contour detection algorithm [13] to find each
region contour. Using the grayscale representation of the original image with
the same 5x5 pixel Gaussian blur applied, very dark (heavily stained) areas
are found by applying a binary threshold to keep only areas with lower than
100 value. The Suzuki contour detection algorithm is again used to convert
these regions into contour lists.

Both sets of contours are drawn onto a new, blank mask with width and height
dimensions matching the original image. The background colour is the
previously selected RNAscope threshold value. The RNAscope contours are filled
with their original intensity values from the RNAscope image (which are darker
than the threshold value). The dark region contours are filled with the
RNAscope threshold value minus a small value. A value of 11 was chosen, but
any value from 1 - 32 would function identically, resulting in the same
candidate density of 50%. In cases where contours overlap, the lowest value is
taken. This mask is used to generate a list of candidate coordinates that may
be RNAscope transcripts (dots). The list of candidates is created by
repeatedly adding the coordinates of the lowest remaining value on the mask to
the list of candidates. A circle centred on these coordinates (with radius
according to equation 1) is then drawn with its value set to the RNAscope
threshold value, which prevents the area from being selected again. intensity
in the equation refers to the mask value at the candidate location, and thresh
refers to the previously selected RNAscope threshold value. This means that
the radius will be 0 (producing a dot and allowing for more adjacent,
clustered detections), unless the difference between the candidate location
value and the RNAscope threshold value is less or equal to 32 (indicating a
weaker detection). This process ends when there are no remaining values darker
than the RNAscope threshold value. An example is shown in figure 3.

$radius=floor(\frac{max(intensity-thresh+64,0)}{32})$ (1)

Figure 3: An example of the candidates (red) selected from tissue, alongside
their corresponding original images.

###  3.4 Feature Extraction

A vector of texture features is extracted for each of the candidate
coordinates. The image is firstly separated into three single-channel images:
grayscale, haematoxylin, and RNAscope. The haematoxylin and RNAscope channels
are extracted using the previously mentioned colour deconvolution method [11].
A large set of first order, GLRLM, GLSZM, GLCM, GLDM, and NGTDM features are
extracted. For the features that can be assessed at different distances,
distances of 1, 2, and 3 pixels are used. GLDM cut-offs of 0, 1, and 2 pixels
are used. Each feature is assessed in both a 7x7 pixel and an 11x11 pixel
window around the central pixel. Feature vectors for coordinates that are
within 1 pixel of a ground-truth annotation are classified as positive, and
all other coordinates as negative. The texture features are normalized. For
classification, the linear support vector classifier (LSVC) was found to work
well, using 1e-7 stopping tolerance, balanced class weighting, l2 penalty,
C=1.0, and 1e7 max iterations. Viewing the coefficient weights for the trained
LSVC showed that very few features were significant, so a reduced feature set
was designed; 1548/1572 features were removed. The remaining features are
first order energy and variance, NGTDM coarseness, and GLDM Large Dependence
High Gray Level Emphasis (LDHGLE). NGTDM coarseness was kept over GLSZM Large
Area High Gray Level Emphasis (LAHGLE, which was weighted slightly higher) as
it is simpler to compute. These are only assessed at a distance of 3 pixels,
and GLDM cut-off of 2 pixels. The reduced feature set extraction process runs
about 10 times faster. Each classifier was trained on 115 of the annotated
patches, with the remaining 29 being used for validation.

The prediction outputs for each feature vector are used to produce
segmentation maps, which are binarized with a configurable gray threshold
value. The watershed transform is used to aggregate neighboring values into a
set of coordinates that can be directly compared against the ground truth
coordinates. The minimum pixels in each detection (area threshold) is
configurable. Figure 4 shows the segmentation map and final coordinates for an
example patch.

Figure 4: The output of the feature segmentation method. The leftmost image
shows the input image data, the middle image shows the segmentation map, and
the final image shows the detections (in red) after the watershed transform is
applied. The ground truth points are shown as yellow diamonds.

##  4 Results

The classifiers were evaluated against the same set of 19 patches used to
assess expert inter-rater agreement. No comparison was made with other
existing methods, due to their lack of output granularity and automation (as
discussed in section 2). No patches from the same tissue cores were present
across both the training and validation datasets. Both the classifier using
the full feature set and the classifier using the reduced feature set were
evaluated. $F_{1}$-score (based on matching annotation pairs) was used as the
evaluation metric for each classifier. The best scores for each classifier
were very similar for both classifiers, with the reduced feature set
classifier only performing slightly worse (0.571 $F_{1}$-score) than the
classifier using the full feature set (0.572 $F_{1}$-score). Both classifiers
performed similarly to the expert inter-rater agreement (0.596 $F_{1}$-score).
Additional metrics (precision and recall) are also shown in table 1.

Table 1: Multiple accuracy measurements for both classifiers Classifier |  $F_{1}$-score | Precision | Recall
---|---|---|---
Full feature set classifier | 0.572 | 0.626 | 0.527
Reduced feature set classifier | 0.571 | 0.633 | 0.521
Experts | 0.596 | 0.682 | 0.530

To illustrate the segmentation characteristics and hyperparameter (gray
threshold and area threshold) selection process, a surface plot showing the
$F_{1}$-score at each gray threshold and area threshold is shown in figure 5.
Since both classifiers produced very similar surfaces, only the plot for the
reduced feature set classifier is shown. The plot displays good $F_{1}$-scores
at most thresholds, only dropping significantly at gray thresholds higher than
150. This means that both classifiers are resilient to small changes in the
RNAscope detection size on the segmentation map, and to the intensity of these
detections. This also implies that the feature extraction is stable, producing
coherent results for neighboring pixels.

Figure 5: The $F_{1}$-score of the feature-based segmentation method (reduced
feature set) when using differing gray thresholds and area thresholds for
segmentation post-processing. The max $F_{1}$-score is at 2 pixel area
threshold and gray threshold of 132.

###  4.1 Feature Analysis

Figure 6: A breakdown of the coefficient weight contribution of each colour
channel to each feature type in the full feature set LSVC. Categories with
weight smaller than 0.01 were excluded. Subcategories with weight smaller than
0.05 were not labelled. Categories which were kept for the reduced feature
classifier are labelled with an asterisk.

Of all of the features extracted, few are needed to produce an accurate
segmentation map. This is evidenced by the fact that reducing the feature set
from 1572 to 24 features only decreased the classification $F_{1}$-score from
0.572 to 0.571. Figure 6 shows the relative coefficient weights of each
category of features, and the weighting of each colour channel. Weighting for
a feature is aggregated across all parameters (such as distance, cutoff)
measured for that feature.

The feature with highest weighting is the first order energy, which is simply
the magnitude of pixel values within the area. It accounts for 35.5% of the
total coefficient weight magnitude. For the smaller 7x7 pixel window, lower
gray and RNAscope channel energy values (indicating darker staining) were
positively correlated with RNAscope detection; whereas, lower haematoxylin
values (indicating denser, darker tissue) were negatively correlated. For the
larger 11x11 pixel window, the opposite correlation (albeit weaker) occurred
for each of the three channels. This makes sense as the 11x11 pixel window
would mostly consist of the tissue surrounding the transcript, not just the
transcript itself.

The second highest weighted feature is the GLDM LDHGLE, which accounts for
28.2% of the total weight. The LDHGLE measures the occurrence of high (light)
gray valued regions with homogeneous texture. This is positively correlated
with RNAscope detection on the RNAscope channel at window size 11x11 pixels.
This positive correlation at the larger window size is likely because the
immediate area surrounding an RNAscope transcript generally contains little to
no RNAscope staining. The third highest weighted feature, GLSZM LAHGLE,
indicates large connected areas of high gray value, and carries the same
positive correlation on the RNAscope channel at window size 11x11 pixels.
Given that these two features are so similar, including both would be largely
redundant.

The fourth highest weighted feature is the NGTDM coarseness, which accounts
for 7.2% of the total weight. It assesses the average difference of gray
values from their neighbors within a small (1-3 pixel) radius, or the spatial
rate of change. Lower rates of change on the RNAscope and gray channels
correlate with transcript detection, whereas the opposite association exists
on the haematoxylin channel. This could be because a dot with smooth edges
would produce low coarseness values on the RNAscope channel, and the
haematoxylin channel will naturally be coarse in the areas near nuclei, where
RNA is primarily found.

No GLRLM or GLCM features were highly weighted. Since GLRLM assesses
horizontal runs of same-valued pixels, the noisy and two dimensional nature of
slide image data would reduce its suitability. Given that GLCM assesses co-
occurrence of gray values at small, fixed offsets, it does not seem well
suited for detecting simple dot structures atop heterogeneous tissue.

##  5 Conclusion

In this paper, the viability of gray level texture features for segmentation
of chromogenic dye RNAscope was investigated. A linear support vector
classifier using a variety of these features was developed, and subsequently
pruned to only include the most significant features. It performed similarly
to two experts who annotated data for this study, achieving an $F_{1}$-score
of 0.571 for identifying closely matching RNAscope transcript coordinates. The
baseline expert $F_{1}$-score for the same task was 0.596. Feature analysis
revealed a small set of gray level features well suited to segmentation of
chromogenic dots (RNAscope) from histological tissue specimen slide images.
Automated methods for chromogenic RNAscope segmentation and subsequent
quantification are crucial for the usability of RNAscope in routine pathology
workflows, and could greatly aid in cancer diagnosis with further development.
For future work, more nuanced deep learning methods will be investigated
because they could potentially better handle the tissue heterogeneity that
makes this segmentation task difficult.

## Acknowledgment

This study used data supplied by Associate Professor Logan Walker, Dr. Arthur
Morley-Bunker, and Dr. George Wiggins from the University of Otago,
Christchurch.

The RNAscope transcripts on a total of 144 image patches were annotated by Dr.
Arthur Morley-Bunker (a trained pathology scientist) and Dr. Gavin Harris (an
anatomical pathologist) for use in this study.

We wish to thank Heather Thorne, Eveline Niedermayr, Sharon Guo, all the
kConFab research nurses and staff, the heads and staff of the Family Cancer
Clinics, and the Clinical Follow Up Study (which has received funding from the
NHMRC, the National Breast Cancer Foundation, Cancer Australia, and the
National Institute of Health (USA)) for their contributions to this resource,
and the many families who contribute to kConFab. kConFab is supported by a
grant from the National Breast Cancer Foundation, and previously by the
National Health and Medical Research Council (NHMRC), the Queensland Cancer
Fund, the Cancer Councils of New South Wales, Victoria, Tasmania and South
Australia, and the Cancer Foundation of Western Australia.

## References

  * [1] Amadasun, M., King, R.: Textural features corresponding to textural properties.  IEEE Transactions on Systems, Man, and Cybernetics 19(5), 1264–1274 (1989).  10.1109/21.44046
  * [2] Arnold, M., Morgan, E., Rumgay, H., Mafra, A., Singh, D., Laversanne, M., Vignat, J., Gralow, J.R., Cardoso, F., Siesling, S., Soerjomataram, I.: Current and future burden of breast cancer: Global statistics for 2020 and 2040\.  The Breast 66, 15–23 (2022).  10.1016/j.breast.2022.08.010
  * [3] Chowdhary, C.L., Acharjya, D.: Segmentation and feature extraction in medical imaging: A systematic review.  Procedia Computer Science 167, 26–36 (2020).  https://doi.org/10.1016/j.procs.2020.03.179.  International Conference on Computational Intelligence and Data Science
  * [4] Galloway, M.M.: Texture analysis using gray level run lengths.  Computer Graphics and Image Processing 4(2), 172–179 (1975).  https://doi.org/10.1016/S0146-664X(75)80008-6
  * [5] Ghaznavi, F., Evans, A., Madabhushi, A., Feldman, M.: Digital imaging in pathology: Whole-slide imaging and beyond.  Annual Review of Pathology: Mechanisms of Disease 8(1), 331–359 (2013).  10.1146/annurev-pathol-011811-120902
  * [6] Graschew, G., Roelofs, T.A., Rakowsky, S., Schlag, P.M.: E-health and telemedicine.  International Journal of Computer Assisted Radiology and Surgery 1(1), 119–135 (2006).  10.1007/s11548-006-0012-1
  * [7] van Griethuysen, J.J., Fedorov, A., Parmar, C., Hosny, A., Aucoin, N., Narayan, V., Beets-Tan, R.G., Fillion-Robin, J.C., Pieper, S., Aerts, H.J.: Computational Radiomics System to Decode the Radiographic Phenotype.  Cancer Research 77(21), e104–e107 (2017).  10.1158/0008-5472.CAN-17-0339
  * [8] Haralick, R.M., Shanmugam, K., Dinstein, I.: Textural features for image classification.  IEEE Transactions on Systems, Man, and Cybernetics SMC-3(6), 610–621 (1973).  10.1109/tsmc.1973.4309314
  * [9] Morley-Bunker, A.E., Wiggins, G.A., Currie, M.J., Morrin, H.R., Whitehead, M.R., Eglinton, T., Pearson, J., Walker, L.C.: Rnascope compatibility with image analysis platforms for the quantification of tissue-based colorectal cancer biomarkers in archival formalin-fixed paraffin-embedded tissue.  Acta Histochemica 123(6), 151,765 (2021).  https://doi.org/10.1016/j.acthis.2021.151765
  * [10] Rozario, S.Y., Sarkar, M., Farlie, M.K., Lazarus, M.D.: Responding to the healthcare workforce shortage: A scoping review exploring anatomical pathologists’ professional identities over time.  Anatomical Sciences Education n/a(n/a) (2023).  https://doi.org/10.1002/ase.2260
  * [11] Ruifrok, A.C., Johnston, D.A.: Quantification of histochemical staining by color deconvolution.  Anal Quant Cytol Histol 23(4), 291–299 (2001)
  * [12] Sun, C., Wee, W.G.: Neighboring gray level dependence matrix for texture classification.  Computer Vision, Graphics, and Image Processing 23(3), 341–352 (1983).  https://doi.org/10.1016/0734-189X(83)90032-4
  * [13] Suzuki, S., be, K.: Topological structural analysis of digitized binary images by border following.  Computer Vision, Graphics, and Image Processing 30(1), 32 – 46 (1985).  https://doi.org/10.1016/0734-189X(85)90016-7
  * [14] Thibault, G., FERTIL, B., Navarro, C., Pereira, S., Lévy, N., Sequeira, J., Mari, J.L.: Texture indexes and gray level size zone matrix application to cell nuclei classification.  In: International Conference on Pattern Recognition and Information Processing (PRIP’09), pp. 140–145 (2009)
  * [15] Wang, F., Flanagan, J., Su, N., Wang, L.C., Bui, S., Nielson, A., Wu, X., Vo, H.T., Ma, X.J., Luo, Y.: Rnascope: a novel in situ rna analysis platform for formalin-fixed, paraffin-embedded tissues.  The Journal of molecular diagnostics : JMD 14(1), 22–29 (2012).  10.1016/j.jmoldx.2011.08.002


0 Research algorithm/technique Joao Pinheiro is with Universidad Catolica San
Pablo. E-mail<EMAIL_ADDRESS>Jorge Poco is with Fundação Getulio
Vargas, Brazil. E-mail<EMAIL_ADDRESS>Biv et al.: Global Illumination for
Fun and Profit

# ChartText: Linking Text with Charts in Documents

Joao Pinheiro  and Jorge Poco

###### Abstract

Recent works show that interactive documents connecting text with
visualizations facilitate reading comprehension. However, creating this type
of content requires specialized knowledge. We present ChartText, a method that
links text with visualizations in this work. Our approach supports documents
that include bar charts, line charts, and scatter plots. ChartText receives
the visual encoding of the visualization and its associated text as input. It
then performs the linking in two stages: The matching stage creates individual
links relating simple phrases between the text and the chart. Then, it
combines the individual links according to the visual channels in the grouping
stage, building more meaningful connections. We use two datasets to design and
evaluate our method; the first comes from web documents (24 bar charts and
texts) and the second from academic documents (25 bar charts, 25 line charts,
and 25 scatter plots with their texts). Our experiments show that our method
obtains F1 scores of 0.50 and 0.66 on both datasets. We can also use a semi-
automatic approach correcting individual links; in this case, the scores rise
to 0.68 and 0.84, respectively. To show the usefulness of our technique, we
implement two proofs of concept. We create interactive documents using graphic
overlays in the first one, facilitating the reading experience. We use voice
instead of text to annotate charts in real-time in the second. For example, in
a videoconference, our technique can automatically annotate a chart following
the presenter’s description.

K.6.1Management of Computing and Information SystemsProject and People
ManagementLife Cycle; K.7.mThe Computing ProfessionMiscellaneousEthics

##  1 Introduction

Visualizations are relevant in the reading comprehension process because they
help understand data more naturally. Many content sources, such as scientific
papers and news websites, present the information using charts, diagrams, or
tables that complement the text to improve understanding of concepts and ideas
[10]. However, to read this kind of document, we must split our attention
between elements and mentally integrate the story, requiring more cognitive
work [34]. In response to this problem, some content creators incorporate
interactive charts in their documents that serve as a visual guide for the
reader [32]. In Figure 1, we can see an example where the user hovers a text,
and the chart highlights the associated region. This type of interactivity
facilitates the exploration of the document content.

For instance, imagine a news website writer who wants to embed an existing
graphic that complements a text presented in an article to generate an
interactive experience that guides the reader to a quick understanding of the
text-graphic document. Generating interactive content is a tedious task, even
with programming knowledge. Also, the content creator must manually annotate
the links between the text and the graphic. Moreover, the writer does not
always have access to edit an existing chart or the underlying data to
redesign the chart for this type of content. Having a tool that reduces the
effort in creating interactive content from text-graphics is the primary
motivation for this research. To build a tool that automatically generates
interactive text-chart documents, it is necessary to combine the three main
tasks studied in recent years: (i) it identifies the chart in a document and
maps it with the text that references it [7, 33] (ii) obtain the underlying
data that makes up the chart (_i.e._ , identify the type of chart, get the
texts that appear on the chart, identify the visual channels) [31, 27, 28, 8,
12], and (iii) find the links between the text and parts of the chart [20,
17]. The latter is the least addressed, and the existing techniques are
limited and do not automatically solve this task.

Figure 1: Screenshot of an interactive document taken from the news website
Reuters Graphics [29]. It shows how the links between the text and the chart
are highlighted to provide a visual guide to the reader.

In this work, we propose ChartText, a new method that automatically connects
text with its respective area associated with the chart. Our method is the
starting point for building a tool to help content creators generate
interactive visualizations from static documents. Moreover, it is essential to
clarify that we take for granted tasks (i) and (ii) to achieve this research
result. This approach can also work in semi-automatic mode, allowing the user
to fix intermediate errors. Our method comprises two stages: In the matching
stage, we generate individual links using different comparison operators
between phrases in the text and textual elements in the chart. We combine the
individual links in the grouping stage using each link’s visual channel
information. The grouped links, in general, are more meaningful and allow the
creation of multiple applications. Here, we present two prototypes: In the
first one, we make interactive documents using graphic overlays, facilitating
the reading experience. We use voice instead of text to annotate charts in
real-time in the second. For example, in a videoconference, our technique can
automatically annotate a chart following the presenter’s description. We
evaluate our results by measuring how similar the inferred links are to links
manually annotated by experts in two datasets from academic and web documents.
Also, we compare our results with KongExtraction [20], a non-automated method
based on crowdsourcing, which works only with bar charts. In summary, the
contributions of this work are:

  * •

A method for linking text with visualization for multiple chart types. This
method can work in two modes, fully automated and semi-automatic.

  * •

An application to make interactive documents by visualizing the links
generated by our technique.

  * •

A proof of concept where we use audio instead of text to create video
annotated charts in real-time.

  * •

A new dataset (text and charts) fully annotated for future comparisons. This
dataset includes three chart types (bar chart, line chart, and scatter plots).

##  2 Related Work

Our work relates to two main areas: (i) chart interpretation and (ii)
techniques and applications to facilitate the reading of documents.

###  2.1 Chart Interpretation

In recent years, methods and techniques have been developed to automatically
extract the underlying data and visual encoding that build charts. These
techniques have applications such as the reuse and redesign of charts. One of
the pioneers in this area is ReVision [31]. ReVision extracts the data that
generates a visualization from an image in a bitmap format. ReVision proposed
a pipeline where initially the images are classified using the Support Vector
Machine model [13] to determine the type of chart (_i.e._ , bar chart, line
chart, and scatter plot). Then, they used different image processing
techniques and heuristics. These methods are adapted to the chart type and
then used to filter elements in the image and identify the data. Other works
maintain the initial idea of ReVision but vary some components with more
sophisticated models. For instance, FigureSeer [33] and Scatterract [8] use a
Convolutional Neural Network [24]. The input of these systems consists of
images and features based on the spatial information of the chart elements to
enhance the classification task. They also take advantage of the axis
information and text labels to infer the data on line charts and scatter
plots. Later, ChartSense [16] extended the data extraction support for five
chart types (line chart, bar chart, area chart, foot chart, and radar chart),
diversifying the heuristics to extract the data for each chart type but
keeping the main idea proposed in the previous works.

Other works focused on recovering the visual encoding of the chart. For
instance, Harper et al. [12] developed a system to deconstruct visualizations
created with the D3 library [3]. This technique uses the data and graphic
elements, which are easier to access in SVG images, to compose the visual
encoding. In the same vein, Poco and Heer [27] proposed a pipeline to recover
the visual encoding but for bitmap images. The technique exploits only the
textual elements and retrieves a visual encoding in a declarative language
similar to Vega-Lite [30]. Later, Chen et al. [6] presented a method to
deconstruct bitmap infographics and generate templates that can be reused to
create new infographics. This method uses deep learning models to identify
graphics elements. Although we do not deconstruct charts in our work, we
assume visual encoding as input. This assumption is valid since we can create
visual encoding using the methods described above.

###  2.2 Facilitating Reading of Documents

Recently many works have appeared in this line. Kong and Agrawala [19]
introduced Graphical Overlays to create interactive charts by accessing the
data and visual encoding (using ReVision [31]). We also have a similar
application; however, we generate overlays based on the text accompanying a
chart; in this sense, our goal is to facilitate document reading.

Other works take advantage of additional information that is accessible to
enrich documents and visualizations. For instance, Metoyeret et al. [26]
presented a technique that works on data-rich stories (_i.e._ , texts with
access to additional data such as tables). They proposed using text analysis
to connect narrative sentences with visual elements that describe the rich
data. Bryan et al. [4] proposed a method to explore temporal data and find,
through heuristics, points of interest in the data that allow the creation of
explanatory visualizations. Hullman et al. [15] proposed a system that takes
time series data on a company’s stock to automatically create a line chart
enriched with annotations from text analysis and metadata from news articles
related to the company. These approaches are helpful when we have access to
the data that accompanies the document or visualization. However, we only need
the chart texts available in the visual encoding, and it is easy to extract
with fewer errors in our approach.

Later, Kong et al. [20] presented a crowdsourcing-based method for extracting
links between charts and text in documents. Nevertheless, they use people to
annotate the document using crowdsourcing, unlike our work, which is fully
automatic. On the other hand, some works use tables instead of using
visualizations. Kim et al. [17] presented an automatic technique to find links
between tables and paragraphs. This work uses NLP and heuristic techniques to
connect table cells with text sentences. Similarly, Badam et al. [2] proposed
a web tool that relates the text to the tables and creates visualizations
using these links. To reduce the effort to create interactive documents, Latif
et al. [22] proposed a framework for systematically generating documents that
support text-visualization interaction. This system allows users to annotate
the links in declarative language. Then, Cui et al. [9] proposed a method that
converts text into infographics to facilitate reading. First, they identify
parts of the text that refer to some data (_e.g._ , numbers) and then use this
information in templates that generate the image. Later, Latif et al. [23]
presented an approach for producing interactive documents for graph data, also
using a declarative syntax that requires little to no programming skills.
Recently, Lai et al. [21] proposed a system with similar capabilities.
Nevertheless, they focus only on individual links; in our case, we have a
second stage grouping the individual link to make more meaningful associations
Additionally, they require access to the encoded data, whereas our technique
only requires the textual elements, representing a more straightforward task
and less prone to error.

##  3 Dataset

In this section, we first define the terminology used throughout the document.
Then, we describe the data set supported by our technique and our assumptions.
Finally, we depict how we create our dataset, the gold standard set in our
evaluations.

Figure 2: (a) Example of a visual encoding that describes the x-axis of the
line chart. (b) Example of individual links. The phrases in the paragraph are
related to the texts in the chart. We use the same color and numbers to
represent the individual links. These individual links are later combined to
form grouped links.

###  3.1 Definitions

Before describing our technique, we must define some terms we often use
throughout the document.

Visual Encoding: It describes how we create the visualization, i.e.,
converting data into a visual representation. In a visual encoding, we specify
the graphical marks (_e.g._ , points, bar, line) and map data variables to the
visual channels (_e.g._ , position, length, size). For instance, Figure 2 (a)
is an example of visual encoding that describes the channel associated with
the x-axis of the line chart shown in Figure 2 (b). Note that we have the
scale information in this specification — which maps values from the data
domain to the image range. Our visual encoding also has information about the
labels used in the chart (text and bounding boxes) for this particular case.

Our technique assumes that we receive the visual encoding specification
containing visual channel information (_i.e._ , x-axis, y-axis, and color-
legend), which can be automatically generated by methods proposed in previous
works [27, 12].

Textual Element: It is a region on the chart that contains the text. For
instance, in Figure 2 (b), the x-axis has textual elements such as “100
million”, and the color legend has the textual element “deptype”.

Phrase: A phrase is any text fragment in a paragraph. For example, in Figure 2
(b), the colored texts represent three phrases.

Individual Link: We define as an individual link a connection between a phrase
and a textual element. Figure 2 (b) shows some examples of individual links,
each formed by a textual element and a phrase with the same number and color.

Grouped Link: A grouped link is a set of individual links. ChartText combines
the links shown in Figure 2 (b) to create a grouped link. Using this
meaningful link, we can create a graphical overlay on the chart (more details
in Sec. 4.2).

Case: A case is a pair consisting of a chart and its associated paragraph. A
case could contain many grouped links.

###  3.2 Supported Visualizations

Our method supports bar, line, and scatter plots as a result of the
restrictions we defined to delimit the scope of this investigation: i) the
charts must represent the data using the cartesian space and a single color
channel. We do not support visual channels such as orientation or size (_e.g._
, it does not work for bubble charts); ii) we do not support charts composed
of multiple views. Trellis plot is an example of a chart type violating this
constraint; and iii) we only consider charts that use a single visual mark.
For instance, Pareto plots violate this rule because it uses two graphical
marks: bars and lines.

###  3.3 Gold Standard (GS) Set

To evaluate our results, we compare our outcomes with the grouped links
manually annotated by visualization experts — from now on, we will call it the
Gold Standard (GS) set. Our cases come from two data sources: news websites
and academic documents.

Web Documents (WEB): These are 24 cases taken from news websites and
specialized platforms such as Pew Research (pewresearch.org) and The Economist
(economist.com). This dataset was collected and annotated in Kong et al. [20]
and is available on the paper’s website. The authors’ complete datasets
contain 40 cases, of which we chose 24 that meet our restrictions (all images
are bar charts with their associated texts). Since our method requires visual
encoding, we had to create the visual encoding manually.

Academic Documents (ACA): This dataset has 75 cases extracted from the
SemanticScholar (semanticscholar.org/) database, including figures and texts
from scientific papers from computer science conferences such as ACM CHI and
ACM UIST. The cases in this dataset include the three chart types: 25 bar
charts, 25 line charts, and 25 scatter plots. Since this data had no
annotations, we had to annotate the grouped links and visual encoding
manually. For this task, we follow the guideline proposed in Kong et al. [20].
In each paragraph-chart, we link the phrases with the textual elements. Each
phrase must contain the minimum number of characters to refer to a textual
element. For instance, in Figure 2 (b), it would be possible to connect the
phrase “DepType system” with the textual element “DepType” in the chart since
adding the word “system” does not modify the meaning of the link. However,
according to our guideline, we use “DepType” because it minimizes the selected
text amount. The annotated groups comprise individual links (_i.e._ , phrases
and textual elements) that cover a fact entirely relating to text and chart.
For instance, in Figure 2 (b), the following fact is described in the text:
“In the end with 100 million tokens, the accuracy of the Deptype system rises
to 27%.” and the individual links are highlighted in the paragraph and the
chart (each individual link has the same color and numbering). If we remove
any individual link in the example, the resulting group could not refer to the
text’s fact and would be meaningless. The two authors of this paper (experts
in the visualization area) created the gold standard set. Each worked
separately to generate the individual and grouped links using printed text and
charts. Afterward, they discussed the annotations, and the cases where they
did not match were analyzed jointly (for this dataset, 30% of the cases were
discussed).

Figure 3: Constituency tree for the paragraph related to the line chart on the
bottom.

##  4 Automatic Links Extraction

We divide our method into two stages: matching and grouping. In the _matching_
stage, we make a series of comparisons between the paragraph text and the
textual elements to find individual links. Then, in the _grouping_ stage, we
combine the individual links that describe the same fact. We take advantage of
the information in the individual links’ visual channels in this stage.

###  4.1 Matching

This stage aims to link single phrases to textual elements to obtain
individual links. First, we split the paragraph into syntactically consistent
sentences. (_e.g._ , nominal phrases, verbal phrases, and prepositional
phrases). For that, we decompose the paragraph into sentences using the
sentence segmentation module of the spaCy library [14]. Then, we compute the
syntactic constituency tree for each sentence using the constituency parser
method proposed by Kitaev et al. [18] — the constituency tree subdivides the
sentence into syntactic phrases. Each node in the tree represents a syntactic
phrase; for instance, Figure 3 shows the tree for the sentence “With 100
million tokens, the accuracy of the DepType system rises to 27.0%”. In this
example, the first node with the label _S_ represents the first syntactic
phrase corresponding to the complete sentence. This phrase is then subdivided
into three children nodes: _PP_ , _NP_ and _VP_ that correspond to three
syntactic phrases, _prepositional phrase_ , _noun phrase_ and _verbal phrase_
, respectively. After that, we traverse the tree and check whether the phrase
associated with a node is related to any textual element in the chart. For
instance, in Figure 3, the node _QP_ (green box) corresponds to the phrase
“100 million”, and it is related to the textual element “100 million” in the
x-axis label. The same happens with the other two phrases in Figure 3 (orange,
purple and red boxes). Finally, we use different operators to compare the
phrases with the textual elements: direct, unique word, semantic, and
numerical value comparison.

Direct Comparison. For each previously generated sentence, we check if any
textual element is written precisely the same. If there is, then we create an
individual link. This comparison helps identify proper nouns that are not
defined in a vocabulary _e.g._ , in Figure 3, the phrase “DepType” is the same
as the textual element in the color legend.

Figure 4: (a) Example of matching with the unique words strategy. (b) Example
of matching with the semantic comparison strategy.

Unique Word Comparison. It is common to have repeated terms in the textual
elements with the same text role in the chart (see, for instance, legend
labels in Figure 4 (a). This problem causes our direct comparison to generate
false negatives because the phrases tend to reference the legend’s terms that
differentiate them (we call these terms unique words). To solve this issue, we
eliminate repeated words and keep the unique words on each textual element
with the same text role. In our example, we keep the terms “correct” and
“incorrect” in the textual elements. These words are better candidates to
match the phrases in the paragraph.

Semantic Comparison. Sometimes phrases and textual elements are written using
different words with the same meaning. For instance, in Figure 4 (b), the
phrase “religious wear” is related to the textual element “religious items or
clothing”. We compare the phrases and textual elements to cover these cases
using a word embedding approach that captures the sentence’s semantic. We
tested multiple state-of-the-art techniques to convert phrases into high
dimensional vectors; in the end, we opt for the Universal Sentence Encoder [5]
technique because it got the best results in our datasets. We perform an
analogous process for textual elements. Finally, we compute the similarity
between a pair of vectors; we use the metric proposed by Cer et al. [5], which
calculates the cosine similarity and then use the $\arccos$ to convert the
similarity into an angular distance
($\mathrm{sim}(\mathbf{u},\mathbf{v})=(1-\arccos(\frac{\mathbf{u}\cdot\mathbf{v}}{\|\mathbf{u}\|\|\mathbf{v}\|})/\pi)$).
Experimentally they noticed that this technique performs better than basic
cosine similarity. We form an individual link if the similarity is more
significant than a threshold of $0.78$ — we set up this value by calculating
the F1 score (described in Sec. 5.1) on the web document data (WEB) using
different thresholds. We use the threshold that returns the best score.

Figure 5: Initial errors in numerical comparison: (a) Assignment of the number
(“72”) to an incorrect axis. (b) Number “30%” could be associated to both
numerical axes.

Numerical Value Comparison. In this comparison, we first identify the
numerical values in the paragraph using SpaCy’s Named Entity Recognition [14]
module — we only use the entities of type PERCENT, MONEY, CARDINAL, and DATE.
We then verify the numerical values in the scales’ domain in the chart (x/y
axes). For this, we use channels with numerical variables in the visual
encoding. If the numerical value is within any scale, we create an individual
link.

The above procedure can generate multiple false positives presented as two
types of errors. Figure 5 shows these error types.

Error type I. In Figure 5 (a), the phrase “72%” is within the y-axis domain.
However, the “72%” does not refer to the “accuracy” (y-axis) or any other
axis. To fix the first type error, we proceed as follows. First, we make sure
that the numerical phrase has a NUM tag. Then, using the dependency tree, we
find the NOUN associated with this number. After that, we verify if it
coincides with some of the four cases shown in Figure 6 (a). Finally, once we
have the NOUN phrase, we compare it with the titles of the chart’s axes (we
use the same comparisons of the previous stage: direct, unique, and semantic)
and create the individual link; otherwise, we discard it. The four cases we
show in Figure 6 (a) results from an extensive analysis with different
documents to cover as many scenarios as possible.

Error type II. In Figure 5 (b), we have an “ambiguity” error, where the phrase
“30%” could belong to both the x-axis and the y-axis. However, reading the
paragraph deduces that it refers to the x-axis (“Coverage PC”). To solve these
problems, we generate a data structure that contains two types of information
for each word: i) the syntactic dependency tree and ii) the Part of Speech
(POS) tags. To fix the second type of error, we proceed as follows. Remember
that the numerical phrase is associated with both axes (ambiguity). Then, we
look for the axes’ titles (textual elements) and the numerical phrase in the
tree. This search uses the same comparison as the previous steps. In Figure 6
(b), we can see an example for Figure 5 (b); text in red is the numerical
phrase, and the titles of the x-axis and y-axis are in green and blue,
respectively. After that, we calculate the paths between the numerical phrase
and the axes titles in the dependency tree (in Figure 6 (b), we can see these
paths with blue and green lines). Finally, we select the phrase that has the
shortest route. In our example, we will choose the phrase “Coverage PC”,
discarding the link with the y-axis.

Figure 6: (a) Patterns are used to identify the attribute of a numerical
phrase. (b) Ambiguity problem, using the syntactic dependency tree, we find
the distances between the numerical phrase (“30%”) and the phrases associated
with the axes titles (“coverage” and “C4.5 error”), we keep the phrase with
the shortest distance.

###  4.2 Grouping

While individual links can already help generate interactive documents, as
presented in Lai et al. [21]. Combining individual links can get more
meaningful relationships related to a narrative aspect. For instance, Figure 2
(b) shows the individual links; with that, we can only highlight the areas
where the textual elements and phrases are located in the chart or paragraph,
respectively. In this example, phrases “100 million”, “27.0%”, and “DepType”
correspond to the x-position, y-position, and color channels, respectively.
However, if we combine them, as the phrase “In the end, with 100 million
tokens, the accuracy of the DepType system rises to 27.0%” suggests, we can
find a specific location in the plotting area. To do so, we noticed that in
our Gold Standard set, the grouped links annotated by the experts usually
combine textual elements from different visual channels (x-position,
y-position, and color). Thus, our method for grouping attempts to imitate how
a human would do it.

Figure 7: Constituency tree used in the grouping stage. In the amplified
region, we see a node that forms a grouped link (group 2) because it holds
three different visual channels.

Transferring of the Visual Channel to the Phrases. The first step is to
transfer the visual channel information (_i.e._ , x-position, y-position, or
color) from the textual elements to the associated phrases in the links. For
example, in Figure 3, the phrase “100 million” is linked to the textual
element corresponding to an x-axis label. Therefore, we transfer the channel
x-position to that phrase (“100 million”). We apply the same procedure for the
other sentences (“Deptype” and “27.0”).

Searching Groups in Constituency Tree. We group the individual links whose
phrases are close enough in the constituency tree and are associated with
different visual channels. To achieve this, nodes associated with individual
link phrases inherit the phrase’s visual channel (see Figure 7 as an example).
To form the groups, we traverse the constituency tree, and for each node, we
count the number of visual channels associated with its descendants. Then, we
choose those nodes that maximize the variability of visual channels, minimize
the number of children nodes, and maximize the tree level — this last
condition makes the sentences shorter. In Figure 7, the highlighted region
shows how the node S is selected to form a group (group 2). The node _S_ meets
the conditions to form a group because it is an ancestor of nodes _NP_ ,
_ADJP_ , and _NP_ associated with visual channels x-position, color, and
y-position. Finally, we combines the individual links related to this node and
create a _grouped link_.

Figure 8: Precision and Recall calculation in the different stages of the
metrics. To calculate the Precision, we connect the ChartText results to the
gold standard set (GS) one by one, sum the similarities among the connected
pairs, and divide by the number of connections. Similar reasoning is used to
calculate the recall, but we go from the GS to the ChartText results. (a)
Precision calculation among grouped links. (b) Recall calculation among
grouped links. (c) Precision and Recall calculation among textual elements.
(d) Precision and Recall calculation among phrases. Figure 9: (a) Similarity
for each case WEB datasets, obtained by our method and by the method proposed
by Kong et al.[20]. (b) Similarity for each case of the ACA dataset. Each bar
represents the similarity with respect to the GS set.

##  5 Evaluation

In this section, we first describe the metric used for our evaluations. Then,
we discuss some quantitative and qualitative results to reason about good and
bad scenarios.

###  5.1 Metric

Our work’s metric adapts the metrics proposed in object detection in images
and information retrieval in texts. The objective is to evaluate how similar
our grouped links are to the Gold Standard (GS) set. Figure 8 will serve as a
reference to explain each stage. The F1 score gives us a value of 1 when two
sets are identical and 0 when they are different. To interpret this score, let
us see Figure 11 as an example. On the right side of the figure, we can see
the links on the GS, and on the left side, the links obtained by our method.
If we had found all the correct links, we would have an F1 score of 1, but we
failed to find the textual element for the phrase “84%” — ChartText wrongly
assigned the phrase with a numeric value in the x-axis, when in fact it
belongs to the y-axis — causing the F1 score to decrease to 88%. As the errors
increase, the score decrease, _e.g._ , if we remove the links related to the
phrases “10%” and “active agents” we will obtain a score of 66%. As we can
see, each error in an individual link is, on average, a 12% error, which gives
us a clearer idea of how our method is behaving in all the cases.

To facilitate the description, we concentrate on a single _case_ , _i.e._ , a
set of grouped links. In Figure 8(a), the blue and red circles represent the
grouped links of ChartText and the GS set, respectively. Ideally, there should
be a one-to-one pairing; however, this seldom happens. The similarity value we
use is the $F1=\frac{2\cdot precision\cdot recall}{precision+recall}$ score;
therefore, we need to calculate the precision and recall. To calculate the
precision, we take each grouped link from ChartText and look for the most
similar grouped link in the GS set — later, we will explain what we mean by
the most similar. Once we have the matching, we add their similarities and
divide them by the total number of pairs (for our example in (a), the
precision is 0.65). Intuitively, the precision tells us how good are the
grouped links generated by ChartText. Similar reasoning applies for recall but
goes from the GS set to the grouped links generated by ChartText. (in our
example in (b), the recall is 0.47). Intuitively, this value tells us to
recover all the grouped links from the GS set. Finally, the F1 score for our
case is 0.55.

In the previous paragraph, we mentioned that we look for the most similar
grouped link in the other set given a grouped link. To assign a similarity
value between grouped links, we first calculate the similarity between textual
elements and phrases. The average of these values will be the similarity of
the grouped link. Next, we explain how to compute both values.

Textual Elements Similarity. Here, we also use the F1 score, and the procedure
is similar to before but using the textual elements from the ChartText and GS
set. For precision, we take each textual element from ChartText and look for
its corresponding one in the GS set (this value will be 1 or 0, if it is
correct or not), then we add these values and divide them by the number of
textual elements in ChartText. For our example in (c), the precision is 1.
Again, similar reasoning applies to the recall. For our example in (c), the
recall is 0.5, and the F1 score is 0.66.

Phrases Similarity. Similarly, we use the F1, precision, and recall scores as
in the previous paragraph. The difference is that we use two sets of phrases,
and the similarity between phrases is calculated with the Intersection over
Union (IoU) metric since the sentences generated by ChartText do not precisely
match the GS set. In our example in (d), we have a precision of 0.4, a recall
of 0.5, and an F1 score of 0.5.

###  5.2 Quantitative Results

Figure 10: Scores for each phase in our pipeline; we can see the similarity
(score F1), precision and recall for each phase in both data sets.

Using the metrics described above, we performed a series of experiments to
validate the results generated by ChartText.

Comparison with KongExtraction [20]. We performed this comparison with the WEB
dataset, which is the same used by KongExtraction. Our method obtains an
average similarity of 50%, while KongExtraction obtains 58%. Figure 9 (a)
shows the similarity for each case in this dataset. It is important to note
that although KongExtraction has a higher similarity, ChartText has certain
benefits: i) our approach is fully automatic, while KongExtraction requires
human participation through crowdsourcing; ii) KongExtraction needs to access
the underlying data of the chart, while ChartText only requires access to the
text present in the chart (available in the visual encoding), which is a
simpler and less error-prone task in automation. Additionally, we evaluate
ChartText with the ACA dataset, and we get an average similarity of 66%.
Figure 9 (b) shows the similarity for each case in this dataset. In this
experiment, we could not compare with KongExtraction since its code is
unavailable, and it would be necessary to recreate the crowdsourcing
environment.

Evaluating phases for individual links. This experiment’s objective is to
validate the usefulness of each of the comparisons in the matching stage. For
that, we compute our metrics after each comparison (_i.e._ , before grouping):
Direct, Unique Word, Semantic, and Numerical-value. Given that we do not have
groups at this stage yet, we need to distribute the grouped links of the GS
set into individual links. In Figure 10, we can see the results of this
experiment. As we advance in the pipeline, the F1, precision, and recall
scores increase, validating each comparison’s need. However, the precision
score decreases slightly in the last comparison (Numerical-value). To explain
this, let’s remember that our method finds new individual links in each step,
and some of them can be false positives. Nevertheless, this effect is
compensated by an increase in the recall since we bring individual links with
numerical values necessary for the grouping stage and the applications.

Semi-automatic evaluation. On the other hand, it is possible to use ChartText
in a semi-automatic mode where the user can intervene to solve specific
problems (more details about human intervention in the Applications section).
In the semi-automatic mode, the user can manipulate and fix individual links
so that the grouping stage will get better results. We calculate each case’s
similarities to validate this statement, assuming that we have the perfect
individual links. Our results show similarities increased from 50% to 68% for
the WEB dataset and from 66% to 84% for the ACA dataset.

Figure 11: ChartText poorly-performed results. On the left and right side of
each case are shown the overlays obtained using ChartText and the gold
standard set, respectively. (a) ChartText fails to link the phrase “84” to the
x-axis of the chart. (b) ChartText does not find any link between the text and
the chart.

###  5.3 Qualitative Results

In this section, we present qualitative observations to explain the behavior
in some relevant cases, taking into account the design of our method.

Well-Performed Results. ChartText has good results when it is easy to
associate the phrases with the visual channels (x-position, y-position, and
color). For instance, In Figure 12, the phrase “Oracle” is easily associated
with a color legend label (color channel) in the chart. Similarly, the phrase
“96.23” is directly associated with a value on the y-axis (y-position
channel), without any chance of being wrong because the x-axis only admits
values less than or equal to 10. Finally, the phrase “k = 10” explicitly
indicates that the value 10 belongs to the axis with the title “k” which is
the x-axis (x-position channel). In this example, we notice that the three
phrases that form the grouped link belongs to the short phrase “the Oracle
achieve 96.23% when k = 10”. We designed our method to create grouped links
with multiple channels, and the phrases are close to each other. This behavior
emulates how experts annotated the grouped links following this same
criterion.

Poorly Performed Results. ChartText does not get correct results when there is
ambiguity in assigning the visualization role to a phrase. Consider the case
illustrated in Figure 11 (a); On the left side, we can see the wrong output
generated by ChartText, and on the right side, we can see the correct overlays
generated with the grouped links in the Gold Standard Set. Analyzing this
case, notice that ChartText identified all the phrases correctly; however, the
phrase “84” was incorrectly associated with the x-axis. This error happens in
the matching stages. The phrase “84” is assigned to the x/y-axis because the
value 84 fit in both scale’s domains. Then, using our rule-based approach and
the syntactic distances to the phrases, ChartText determines that the phrase
“84” belongs to the axis associated with the nearby phrase “active agents”
(x-axis). In other words, our approach could not identify the attribute (NOUN)
related to the numerical phrase. For this particular example, we can see in
the text that no phrase indicates that the “84%” refers to the y-axis
(“Cumulative probability”).

Another problematic case is shown in Figure 11 (b). The main problem is that
ChartText requires additional context information to reference a phrase with a
textual element. In the GS set, the experts found two phrases associated with
the chart. For instance, experts decided that the phrase “40-something” means
people between 40 and 49. It makes sense to link this phrase with the textual
element “35-49” since it includes this age range. The second phrase,
“retires”, was associated with people over 65, which appears in the chart as
“65+”. ChartText would need additional information to generate this link, such
as the meaning of the phrase “40-something”. It is a challenging problem and
requires more sophisticated models.

##  6 Applications

To demonstrate the usefulness of ChartText, we developed two applications: (1)
A interactive document web tool and (2) Voice2Overlays, a tool that turns
voice into overlays.

###  6.1 Interactive Documents

Using CharText, we build a proof-of-concept that generates interactive
overlays from a chart in bitmap format based on a text that describes it. This
application exemplifies how our method would allow us to build tools that
generate interactive documents. This idea can be easily extrapolated in other
more valuable applications in the real world. For instance, a document editor
allows a content creator (such as a web news writer) to embed a chart on a
text and generate interactive documents with minimal effort and without
programming knowledge. Even at a more practical level, a plugin for a browser
or a document reader (_e.g._ , Adobe reader [1]) that allows the end-users to
select the graphic and the text on which they wish to inquire and
automatically generate the highlights between the chart and the phrases of the
corresponding text.

Overlays Generation. We built a web tool that supports text visualization
interaction to show the overlays. Figure 12 shows an example of how the tool
produces an overlay when the user selects a phrase in the text. In the
example, the tool highlights a point at the end of the curve that corresponds
to the color legend “Oracle” (intersection of red lines). This overlay
corresponds to three phrases (highlighted in yellow): “Oracle”, “K=10”, and
“96.23%”. These phrases belong to individual links in the same group, and we
use the visual channel associated with each link to build the overlay. For
instance, we use the color information associated with the phrase “Oracle” to
filter out only the pixels that correspond to that color in the chart. Using
this strategy, we highlight the correct curve in the chart. To identify the
point at the end of the curve, we use the spatial information associated with
the phrases “K=10” and “96.23”. Different overlays can be generated, depending
on the type of chart and the visual channel associated with the group’s links.
Now we explain the six overlay types supported by our application:

Figure 12: Web application to visualize the overlays. On the left is shown the
chart, and on the right, the text describing it. When the user selects a part
of the text, the system automatically generates the overlays that are shown on
the chart. Figure 13: Different types of overlays that are possible to
generate with our automatic method. Each type depends on the information
associated with the visual channel of each link that makes up the grouped
link.

a) Line Segment.

This overlay is used to represent a single value of an axis. We use the
numerical value in the grouped link to draw a line segment along the chart’s
width or height. The orientation depends on the visual channel associated with
the textual element (x-position or y-position). Figure 13 (a), shows an
example when the numerical value belongs to the x-axis.

b) Target Point.

This type of overlay is generated when, in the grouped link, there are two
textual elements related to two different chart numerical axes. In this case,
as shown in Figure 13 (b), we draw two line segments from the axes (orthogonal
directions). The intersection represents the target point to which the phrase
refers.

c) Color Filter.

When a phrase in the grouped link refers to a color legend on the chart, we
filter out the chart’s pixels with other colors, highlighting only the
relevant color. To achieve this, we increase the transparency of the pixels
that do not correspond to the target color (Figure 13 (c)).

d) Bounding Box.

We use this type of overlay to highlight a bar or a group of bars, as shown in
Figure 13 (d). The Bounding Box is used when only one individual link refers
to a nominal axis (only for bar charts).

e) Highlighted Bar.

This type of overlay is proper when we want to highlight a bar (Figure 13
(e)). We generate this overlay when there are two phrases in the grouped link:
one related to a color legend and another related to a numerical axis label.
To create this overlay, we increase the pixels’ transparency that does not
correspond to the target color (we could still have multiple bars with the
same color). Then, we increase the pixels’ transparency that does not
correspond to the axis label in the grouped link, reducing it to one
highlighted bar.

f) Target Bar.

This overlay is a specialization of the target point but applied to bar
charts. Figure 13 (f) shows an example of this type of overlay. It is
generated when the grouped link contains textual elements from all three
visual channels (x-position, y-position, and color). We need to highlight the
bar and the numerical value associated with the bar. We first highlight the
bar using the same strategy to generate the Highlighted Bar to create this
overlay. Then, we draw a line segment representing the numerical value of the
axis.

Semi-automatic Tool. Our tool has a semi-automatic mode that allows users to
interact with the interface to fix intermediate results. Specifically, the
user can verify, correct, or remove individual links. Once the user is
satisfied with the results, ChartText continues with the grouping stage and
generates the overlays.

For example, in Figure 14 (a), we correct the problem presented in Figure 11
(a). When we double click on the phrase “84%”, the system displays a panel
where it is possible to select the textual element associated with the phrase.
In this particular case, ChartText erroneously linked the phrase “84%” with an
x-axis; however, we fix this link, choosing the correct value in the y-axis.
By making this change and executing the grouping stage, ChartText manages to
generate the correct grouped link, which is displayed as an overlay in Figure
11 (a-right) Similarly, we can fix the error presented in Figure 11 (b); we
double click the “retires” in the text to pop-up the panel in Figure 14 (b),
then we select the correct textual element (“+65”).

Figure 14: Examples of the use of our semi-automatic component to correct
individual links in our overlay web tool.

###  6.2 Voice2Overlays

We can use ChartText as the primary system component to convert voice to
overlays. This functionality provides interactivity to static charts in
environments such as conferences and video calls. The application receives as
input the audio containing the reference to the chart. Then, we use a speech-
to-text tool to convert audio into text. We take advantage of these tools to
detect pauses in speech and generate sentences; This segmentation is
beneficial since our method works better with short phrases, improving the
analysis during the grouping stage. This feature also helps us determine how
long an overlay should remain in view. For this, we take the time when the API
generates a new sentence as a reference. We add the new overlay and remove the
previous one based on it. Once we have the sentence, we use ChartText to get
the grouped links and generate the graphical overlays.

Figure 15: The application uses our proposed method to convert voice into
visual guides or overlays on the chart. First, the application receives a
video where a presenter describes a chart to the audience. The video’s voice
is extracted and converted into text using a speech-to-text tool. The
application also receives the visual encoding of the chart in the video as
input. Then, the texts extracted from the audio are processed by ChartText,
generating as output the overlays shown in the lower part of the figure.

To demonstrate the usefulness of this application, we decided to conduct a
proof of concept. We selected two videos from Ted Talks conferences, where
presenters explain a chart to the audience, an ideal scenario for our
application. We extracted the audio fragment for each video and reconstructed
the chart’s visual encoding shown in the video. Then, we use the Google
speech-to-text API [11] to convert the audio to text. Finally, we use our
method to extract the links and generate the overlays for each sentence in the
audio.

The first video is entitled “Why you should love statistics”; in the video,
the presenter uses a bar chart to explain the percentage of young people with
low numeracy in 12 countries. Figure 15 shows the results obtained for the
first video. Figure 15-top shows the frames of the original video and the
sentences referring to the chart. Figure 15-bottom shows the overlays on the
chart generated by ChartText. For instance, for the first phrase “Leading the
way the USA nearly 40 percent”, there is an overlay that highlights the first
bar of the chart. We can see the results of this example in the accompanying
video.

The second video is entitled “Germany: Low Crime, Clean Prisons, Lessons for
America”. The presenter explains the U.S. incarceration rate between 1925 and
2012 using a scatter plot in this video. Figure 16 shows the results obtained.
The top part shows the sentences extracted from the original video with a
reference frame as in the first video. The bottom part shows the overlays
generated by our method. In this example, we obtained three overlays: The
first overlay highlights a date range (“1925 to 1975”) on the x-axis and a
value (“100”) on the y-axis. The second and third overlays highlight specific
dates on the x-axis (“1940” and “1970”, respectively).

##  7 Limitations and Future Work

Our work has some limitations and possible improvements presented in this
section.

Dataset. One of the significant limitations of this work is the small amount
of annotated data available to relate the text to visualizations. In this
work, we are making another fully annotated dataset, including three chart
types; however, this is still insufficient to archive this task successfully.
It is impossible to explore other techniques and approaches without an
extensive dataset, such as using machine learning models instead of the rule-
based system currently used. As future work, we propose to design the
processes of annotating an extensive dataset to relate text and
visualizations.

Improvement in the Numerical-value phase. As we saw in our evaluation, the
Numerical-value comparison is necessary for the grouping stage, affecting the
generation of overlays. However, our method is not robust enough, generating
false positives in the individual links. The main problem lies in using
heuristics and templates to identify the numerical phrases that do not
correspond to the chart’s axes. In future work, we intend to experiment with
more robust methods from natural language processing. We will have to generate
a larger dataset with annotated text examples. Using it, we could train a
machine learning model to identify the NOUNs associated with the numerical
phrases more precisely without rules.

Support for complex charts. In this research, we limited the scope of
ChartText to simple charts but were commonly used to display results with
descriptive statistics (bar charts, line charts, and scatterplots). However,
this is a starting point for future works that perform the same task on more
sophisticated graphics (i.e., that overload graphic markup and visual
channels). For instance, there are two graphic marks in a Pareto chart: bars
and lines. Our technique does not support this chart type but supports each
type separately. A possible extension of our approach, having the data
properly annotated, would help solve this problem.

Figure 16: Result of the Voice2Overlays application to convert a presenter’s
voice into overlays in real-time. At the top are shown the frames and the
extracted phrases from the original video used as input. At the bottom are
shown the overlays that our method obtained as output.

End-to-end Generation of Interactive Documents. ChartText assumes that we
have: i) visual encoding and ii) chart and text location within a document.
These are valid assumptions since previous works [7, 25, 28, 12] address these
problems with relatively good results under certain conditions. In this work,
we decided not to enter into those areas because we focused on finding the
links between the text and the chart, which is already difficult. As future
work, we propose implementing and integrating all the components mentioned
above to build a fully automatic system that takes as input a document (in PDF
format) and generates an interactive document. With this framework, it will be
possible to implement plugins for web browsers or document readers tools such
as Adobe Acrobat Reader [1].

##  8 Conclusions

This paper presents ChartText, a method automatically extracting links between
text and visualizations. We evaluated our results by comparing them with
manual annotations made by experts on two datasets (web and academic
documents). We compare our ChartText with KongExtraction [20] (a non-automatic
method based on crowdsourcing strategy) for the web documents. Our approach
obtained a similarity of 50% compared to 58% of KongExtraction. Despite our
lower score, it is essential to highlight that: i) our method provides a fully
automatic approach, ii) we support more chart types, and iii) we do not need
access to the underlying data; we only need the charts’ textual elements. We
also make a second dataset containing 75 cases, including three chart types
from academic documents. In this dataset, Chartext got an average similarity
of 66%. Although our method is automatic, we have a semi-automatic mode that
allows the user to fix intermediate errors (individual links). We experimented
with perfect individual links to evaluate this functionality, and our scores
rose to 68% and 84% for the web and academic documents, respectively. Finally,
we presented two applications. The first is a tool to generate interactive
documents. The second is Voice2Overlays, which converts a speaker’s voice into
visual aids on a chart.

## References

  * [1] Adobe.  Adobe acrobat reader.  https://get.adobe.com/, 2019.  Accessed: 2019-10-04.
  * [2] S. K. Badam, Z. Liu, and N. Elmqvist.  Elastic documents: Coupling text and tables through contextual visualizations for enhanced document reading.  IEEE Transactions on Visualization and Computer Graphics, 25:661–671, 2018.
  * [3] M. Bostock, V. Ogievetsky, and J. Heer.  D3 data-driven documents.  IEEE Transactions on Visualization and Computer Graphics, 17(12):2301–2309, Dec. 2011. doi: 10 . 1109/TVCG . 2011 . 185
  * [4] C. Bryan, K.-L. Ma, and J. Woodring.  Temporal summary images: An approach to narrative visualization via interactive annotation generation and placement.  IEEE Transactions on Visualization and Computer Graphics, 23(1):511–520, Jan. 2017. doi: 10 . 1109/TVCG . 2016 . 2598876
  * [5] D. Cer, Y. Yang, S.-y. Kong, N. Hua, N. Limtiaco, R. St. John, N. Constant, M. Guajardo-Cespedes, S. Yuan, C. Tar, B. Strope, and R. Kurzweil.  Universal sentence encoder for English.  In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 169–174. Association for Computational Linguistics, Brussels, Belgium, Nov. 2018. doi: 10 . 18653/v1/D18-2029
  * [6] Z. Chen, Y. Wang, Q. Wang, Y. Wang, and H. Qu.  Towards automated infographic design: Deep learning-based auto-extraction of extensible timeline.  IEEE transactions on visualization and computer graphics, 2019.
  * [7] C. Clark and S. Divvala.  Pdffigures 2.0: Mining figures from research papers.  In Proceedings of the 16th ACM/IEEE-CS on Joint Conference on Digital Libraries, JCDL ’16, p. 143–152. Association for Computing Machinery, New York, NY, USA, 2016. doi: 10 . 1145/2910896 . 2910904
  * [8] M. Cliche, D. Rosenberg, D. Madeka, and C. Yee.  Scatteract: Automated extraction of data from scatter plots.  In M. Ceci, J. Hollmén, L. Todorovski, C. Vens, and S. Džeroski, eds., Machine Learning and Knowledge Discovery in Databases, pp. 135–150. Springer International Publishing, Cham, 2017.
  * [9] W. Cui, X. Zhang, Y. Wang, H. Huang, B. Chen, L. Fang, H. Zhang, J.-G. Lou, and D. Zhang.  Text-to-viz: Automatic generation of infographics from proportion-related natural language statements.  IEEE transactions on visualization and computer graphics, 2019.
  * [10] M. d. P. P. Echeverría and N. Scheuer.  External representations as learning tools: An introduction.  In Representational systems and practices as learning tools, pp. 1–18, 2009.
  * [11] Google.  Cloud speech-to-text api.  https://cloud.google.com/speech-to-text/docs/apis, 2019.  Accessed: 2019-10-04.
  * [12] J. Harper and M. Agrawala.  Deconstructing and restyling d3 visualizations.  In Proceedings of the 27th Annual ACM Symposium on User Interface Software and Technology, UIST ’14, pp. 253–262. ACM, New York, NY, USA, 2014. doi: 10 . 1145/2642918 . 2647411
  * [13] M. A. Hearst.  Support vector machines.  IEEE Intelligent Systems, 13(4):18–28, July 1998. doi: 10 . 1109/5254 . 708428
  * [14] M. Honnibal and I. Montani.  spacy 2: Natural language understanding with bloom embeddings, convolutional neural networks and incremental parsing.  To appear, 2017.
  * [15] J. Hullman, N. Diakopoulos, and E. Adar.  Contextifier: Automatic generation of annotated stock visualizations.  In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI ’13, p. 2707–2716. Association for Computing Machinery, New York, NY, USA, 2013. doi: 10 . 1145/2470654 . 2481374
  * [16] D. Jung, W. Kim, H. Song, J.-i. Hwang, B. Lee, B. Kim, and J. Seo.  Chartsense: Interactive data extraction from chart images.  In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, CHI ’17, pp. 6706–6717. ACM, New York, NY, USA, 2017. doi: 10 . 1145/3025453 . 3025957
  * [17] D. H. Kim, E. Hoque, J. Kim, and M. Agrawala.  Facilitating document reading by linking text and tables.  In Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology, UIST ’18, pp. 423–434. ACM, New York, NY, USA, 2018. doi: 10 . 1145/3242587 . 3242617
  * [18] N. Kitaev and D. Klein.  Constituency parsing with a self-attentive encoder.  In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, Melbourne, Australia, July 2018.
  * [19] N. Kong and M. Agrawala.  Graphical overlays: Using layered elements to aid chart reading.  IEEE Transactions on Visualization and Computer Graphics, 18(12):2631–2638, Dec 2012. doi: 10 . 1109/TVCG . 2012 . 229
  * [20] N. Kong, M. A. Hearst, and M. Agrawala.  Extracting references between text and charts via crowdsourcing.  In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI ’14, pp. 31–40. ACM, New York, NY, USA, 2014. doi: 10 . 1145/2556288 . 2557241
  * [21] C. Lai, Z. Lin, R. Jiang, Y. Han, C. Liu, and X. Yuan.  Automatic annotation synchronizing with textual description for visualization.  In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, CHI ’20, p. 1–13. Association for Computing Machinery, New York, NY, USA, 2020. doi: 10 . 1145/3313831 . 3376443
  * [22] S. Latif, D. Liu, and F. Beck.  Exploring interactive linking between text and visualization.  06 2018. doi: 10 . 2312/eurovisshort . 20181084
  * [23] S. Latif, K. Su, and F. Beck.  Authoring Combined Textual and Visual Descriptions of Graph Data.  In J. Johansson, F. Sadlo, and G. E. Marai, eds., EuroVis 2019 - Short Papers. The Eurographics Association, 2019. doi: 10 . 2312/evs . 20191180
  * [24] Y. LeCun, Y. Bengio, and G. Hinton.  Deep learning.  Nature, 521:436–44, 05 2015. doi: 10 . 1038/nature14539
  * [25] A. Mayhua, E. Gomez-Nieto, J. Heer, and J. Poco.  Extracting visual encodings from map chart images with color-encoded scalar values.  Conference on Graphics, Patterns and Images, 31. (SIBGRAPI), 2018\.
  * [26] R. Metoyer, Q. Zhi, B. Janczuk, and W. Scheirer.  Coupling story to visualization: Using textual analysis as a bridge between data and interpretation.  In 23rd International Conference on Intelligent User Interfaces, IUI ’18, p. 503–507. Association for Computing Machinery, New York, NY, USA, 2018. doi: 10 . 1145/3172944 . 3173007
  * [27] J. Poco and J. Heer.  Reverse-engineering visualizations: Recovering visual encodings from chart images.  Comput. Graph. Forum, 36(3):353–363, June 2017. doi: 10 . 1111/cgf . 13193
  * [28] J. Poco, A. Mayhua, and J. Heer.  Extracting and retargeting color mappings from bitmap images of visualizations.  IEEE transactions on visualization and computer graphics, 24(1):637–646, 2018.
  * [29] Reuters Graphics.  Reuter graphics.  https://graphics.reuters.com/, 2019.  Accessed: 2019-10-04.
  * [30] A. Satyanarayan, D. Moritz, K. Wongsuphasawat, and J. Heer.  Vega-lite: A grammar of interactive graphics.  IEEE Transactions on Visualization and Computer Graphics, 23(1):341–350, Jan 2017. doi: 10 . 1109/TVCG . 2016 . 2599030
  * [31] M. Savva, N. Kong, A. Chhajta, L. Fei-Fei, M. Agrawala, and J. Heer.  Revision: Automated classification, analysis and redesign of chart images.  In Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology, UIST ’11, pp. 393–402. ACM, New York, NY, USA, 2011. doi: 10 . 1145/2047196 . 2047247
  * [32] E. Segel and J. Heer.  Narrative visualization: Telling stories with data.  IEEE transactions on visualization and computer graphics, 16:1139–48, 01 2011. doi: 10 . 1109/TVCG . 2010 . 179
  * [33] N. Siegel, Z. Horvitz, R. Levin, S. Divvala, and A. Farhadi.  Figureseer: Parsing result-figures in research papers.  In B. Leibe, J. Matas, N. Sebe, and M. Welling, eds., Computer Vision – ECCV 2016, pp. 664–680. Springer International Publishing, Cham, 2016\.
  * [34] J. Sweller, P. Ayres, and S. Kalyuga.  The split-attention effect.  In Cognitive load theory, pp. 111–128. Springer, 2011.