# The Semantic Reader Project: Augmenting Scholarly Documents through AI-
Powered Interactive Reading Interfaces

Kyle Loα Joseph Chee Changα Andrew Headψ Jonathan Braggα Amy X. Zhangω Cassidy
Trierα
Chloe Anastasiadesα Tal Augustα Russell Authurα Danielle Braggρ Erin Bransomα
Isabel Cacholaι Stefan Candraα Yoganand Chandrasekharα Yen-Sung Chenα
Evie Yu-Yen Chengα Yvonne Chouα Doug Downeyα Rob Evansα Raymond Fokω
Fangzhou Huα Regan Huffα Dongyeop Kangυ Tae Soo Kimκ Rodney Kinneyα
Aniket Kitturχ Hyeonsu Kangχ Egor Klevakα Bailey Kuehlα Michael Langanα
Matt Latzkeα Jaron Lochnerα Kelsey MacMillanα Eric Marshα Tyler Murrayα
Aakanksha Naikα Ngoc-Uyen Nguyenα Srishti Palaniσ Soya Parkτ Caroline Paulicα
Napol Rachatasumritχ Smita Raoα Paul Sayreα Zejiang Shenτ Pao Siangliulueα
Luca Soldainiα Huy Tranα Madeleine van Zuylenα Lucy Lu Wangω
Christopher Wilhelmα Caroline Wuα Jiangjiang Yangα Angele Zamarronα
Marti A. Hearstβ Daniel S. Weldα αAllen Institute for AI ωUniversity of
Washington βUniversity of California, Berkeley
ψUniversity of Pennsylvania χCarnegie Mellon University τMassachusetts
Institute of Technology κKAIST
ιJohns Hopkins University υUniversity of Minnesota σUniversity of California,
San Diego ρMicrosoft Research  kylel, josephc, jbragg, cassidyt,
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
(2023)

###### Abstract.

Scholarly publications are key to the transfer of knowledge from scholars to
others. However, research papers are information-dense, and as the volume of
the scientific literature grows, the need for new technology to support the
reading process grows. In contrast to the process of finding papers, which has
been transformed by Internet technology, the experience of reading research
papers has changed little in decades. The PDF format for sharing papers is
widely used due to its portability, but it has significant downsides
including: static content, poor accessibility for low-vision readers, and
difficulty reading on mobile devices. This paper explores the question “Can
recent advances in AI and HCI power intelligent, interactive, and accessible
reading interfaces—even for legacy PDFs?” We describe the Semantic Reader
Project, a collaborative effort across multiple institutions to explore
automatic creation of dynamic reading interfaces for research papers. Through
this project, we’ve developed ten research prototype interfaces and conducted
usability studies with 300+ participants and real-world users showing improved
reading experiences for scholars. We’ve also released a production research
paper reader that will incorporate novel features as they mature. We structure
this paper around challenges scholars and the public face when reading
research papers—discovery, efficiency, comprehension, synthesis, and
accessibility—and present an overview of our progress and remaining open
challenges.

††copyright: acmcopyright††journalyear: 2023††doi: XXXXXXX.XXXXXXX

##  1\. Introduction

The exponential growth of scientific publication (Bornmann et al., 2020;
Brainard, 2020) and increasing interdisciplinary nature of scientific progress
(Van Noorden et al., 2015; Okamura, 2019) makes it increasingly hard for
scholars to keep up with the latest developments. Academic search engines,
such as Google Scholar and Semantic Scholar help scholars discover research
papers. Automated summarization for research papers (Cachola et al., 2020)
helps scholars triage between research papers. But when it comes to actually
reading research papers, the process, based on a static PDF format, has
remained largely unchanged for many decades. This is a problem because
digesting technical research papers is difficult (Bazerman, 1985; Bem, 1995).

In contrast, interactive and personalized documents have seen significant
adoption in domains outside of academic research. For example, news websites
such as the New York Times often present interactive articles with explorable
visualizations that allow readers to understand complex data in a personalized
way. E-readers, such as the Kindle, provide in-situ context to help readers
better comprehend complex documents, showing inline term definitions and
tracking occurrence of characters in a long novel. While prior work has
focused on authoring support tools (Conlen et al., 2021; Latif et al., 2021;
Conlen and Heer, 2022) that can reduce effort in creating interactive
scientific documents (Hohman et al., 2020; Head et al., 2022), they have not
seen widespread adoption due to a lack of incentive structure (Editorial Team,
2021). Furthermore, millions of research papers are locked in the rigid and
static PDF format, whose low-level syntax makes it extremely difficult for
systems to access semantic content, augment interactivity, or even provide
basic reading functionality for assistive tools like screen readers (Bigham et
al., 2016).

Figure 1. The Semantic Reader Project consists of research, product, and open
science resources. The Semantic Reader product1 is a free interactive
interface for research papers. It supports standard reading features (e.g.,
(A) table of contents), integration with Semantic Scholar (e.g., (B) save to
library), useful augmentations atop the existing PDF (e.g., (C) in-situ Paper
Cards when clicking inline citations), and integration with third-party
features (e.g. (D) Hypothes.is1 for user highlights). We continues to
integrate research features into this product as they mature (e.g., (E) Scim
automated highlights §3.1).

Fortunately, recent work on layout-aware document parsing (Xu et al., 2019;
Huang et al., 2022; Shen et al., 2022) and large language models (Beltagy et
al., 2019; Raffel et al., 2019; Brown et al., 2020) show promise for accessing
the content of PDF documents, and building systems that can better understand
their semantics. This raises an exciting challenge: _Can we create
intelligent, interactive, and accessible reading interfaces for research
papers, even atop existing PDFs?_

To explore this question, we present the Semantic Reader Project, a broad
collaborative effort across multiple non-profit, industry, and academic
institutions to create interactive, intelligent reading interfaces for
research papers. This project consists of three pillars: research, product,
and open science resources. On the research front, the Semantic Reader Project
combines AI and HCI research to design novel, AI-powered interactive reading
interfaces that address a variety of user challenges faced by today’s
scholars. We developed research prototypes and conducted usability studies
that clarify their benefits. On the product front, we are developing the
Semantic Reader (Figure 1),111Semantic Reader:
https://www.semanticscholar.org/product/semantic-reader a freely available
reading interface that integrates features from research prototypes as they
mature.222Available for over 369K papers as of February 2023. Finally, we are
developing and releasing open science resources that drive both the research
and the product. These resources together open-source software,333For UI
development: https://github.com/allenai/pdf-component-library444For processing
PDFs: https://github.com/allenai/papermage AI models (Cohan et al., 2020;
Cachola et al., 2020; Shen et al., 2022; Kang et al., 2020), and open datasets
(Kinney et al., 2023; Lo et al., 2020) to support continued work in this area.

In this paper, we focus on summarizing our efforts under the _research pillar_
of the Semantic Reader Project. We structure our discussion around five broad
challenges faced by readers of research papers: 55footnotetext: Hypothes.is:
https://web.hypothes.ishttps://web.hypothes.is

  * •

Discovery: Following paper citations is one of the main strategies that
scholars employ to discover additional relevant papers, but keeping track of
the large numbers of citations can be overwhelming. In §2, we explore ways to
visually augment research papers to help readers prioritize their paper
exploration during literature reviews.

  * •

Efficiency: The exponential growth of publication makes it difficult for
scholars to keep up-to-date with the literature—scholars need to skim and read
many papers while making sure they capture enough details in each. In §3, we
explore how support for non-linear reading can help readers consume research
papers more efficiently.

  * •

Comprehension: Research papers can be dense and contain terms that are
unfamiliar either because the author newly introduces them or assumes readers
have prerequisite domain knowledge. In §4, we explore how providing in-situ
definitions and summaries can benefit readers especially when reading outside
of their domains.

  * •

Synthesis: The sensemaking (Russell et al., 1993) process of synthesizing
knowledge scattered across multiple papers is effortful but important. It
allows scholars to make connections between prior work and identify
opportunities for future research. In §5, we explore how to help readers
collect information from and make sense of many papers to gain better
understanding of broad research topics.

  * •

Accessibility: Static PDFs are an ill-suited format for many reading
interfaces. For example, PDFs are notoriously incompatible with screen
readers, and represent a significant barrier for blind and low vision readers
(Bigham et al., 2016). Furthermore, an increasing number of scholars access
content on mobile devices, on which PDFs of papers are difficult to read. In
§6, we explore methods for converting legacy papers to more accessible
representations.

Specifically, we present ten research prototypes developed in the Semantic
Reader Project—CiteSee (Chang et al., 2023), CiteRead (Rachatasumrit et al.,
2022), Scim (Fok et al., 2023), Ocean (Park et al., 2022), ScholarPhi (Head et
al., 2021), Paper Plain (August et al., 2023), Papeo (team, sion), Threddy
(Kang et al., 2022a), Relatedly (Palani et al., 2023), and SciA11y (Wang et
al., 2021b; team, tion)—and explain how they address these reading challenges.
We conclude by discussing ongoing research opportunities in both AI and HCI
for developing the future of scholarly reading interfaces. We provide pointers
to our production reading interface and associated open resources to invite
the broader research community to join our effort.

##  2\. Unlocking Citations for Discovery

Scholars use many methods to discover relevant research papers to read,
including search engines, word of mouth, and browsing familiar venues.
However, once they find one research paper, it’s especially common for
scholars to use its references and citations to further expand their knowledge
of a research area. This behavior, sometimes referred to as _forward/backward
chaining_ or _footnote chasing_ , is ubiquitous and has been observed across
many scholarly disciplines (Palmer et al., 2009). Supporting this, one popular
feature in the Semantic Reader is in-situ Paper Cards that pop up when readers
click on an inline citation, dramatically reducing the interaction cost caused
by jumping back-and-forth between inline citations and their corresponding
references at the end of a research paper (Figure 1). Despite this affordance,
during literature reviews, readers may still be overwhelmed trying to make
sense of the tens to hundreds of inline citations in each paper (Denney and
Tewksbury, 2013; Peroni et al., 2015; Chang et al., 2023). Conversely, when
reading a given paper, a reader cannot see relevant follow-on research papers
that cited the current paper. Here we discuss how interactive reading
interfaces can help scholars more effectively explore citations to important
relevant work in both directions with two systems called CiteSee (Chang et
al., 2023) and CiteRead (Rachatasumrit et al., 2022).

###  2.1. Augmenting Citations with CiteSee

Figure 2.  CiteSee (Chang et al., 2023) highlights citations to familiar
papers (e.g., recently read or saved in their libraries) as well as unfamiliar
papers to help readers avoid overlooking important citations when conducting
literature reviews. Clicking on _Expand_ surfaces additional context, such as
citing sentences from recently read papers.

While most prior work on supporting research paper discovery has focused on
developing bespoke interfaces of recommender systems or visualizations based
on paper contents (Sugiyama and Kan, 2010; Philip et al., 2014), the citation
graph (Huang et al., 2002; Gori and Pucci, 2006; Xia et al., 2016; Mackinlay
et al., 1995; Chau et al., 2011; He et al., 2019; Ponsard et al., 2016), or a
combination of the two (Wang and Blei, 2011; Cohan et al., 2020), research
paper discovery via inline citations in a reading interface is important but
under-explored. One study estimates that reading and exploring inline
citations accounts for around one in five research paper discoveries during
active research (King et al., 2009). However, while all inline citations are
relevant to the current research paper, it is likely that some are more
relevant to the current reader than others. For example, a reader reading
papers about _aspect extraction of online product reviews_ to learn more about
_natural language processing techniques_ would be less interested in citations
to research papers around _e-commerce and marketing_. In addition, citations
to the same research papers often have different surface forms across papers
(i.e., reference numbers), making it all the more difficult for readers to
keep track of all the inline citations they should explore or have already
explored during literature reviews.

To address this, CiteSee provides a personalized research paper reading
experience by automatically identifying and resolving inline citations in PDFs
to research paper entities in our academic graph (Kinney et al., 2023), and
visually augmenting inline citations based on their connections to the current
reader. First, CiteSee leverages a reader’s reading behavior and history as a
way to capture their short-term and fluid interests during literature reviews.
Using this signal, CiteSee scores and highlights inline citations to help the
reader triage them and discover prior work that are likely relevant to their
literature review topics (Figure 2). Second, CiteSee leverages research papers
saved in the reader’s Semantic Scholar paper library and the reader’s
publication record (Kinney et al., 2023) to understand their longer-term
research interests. Using this signal, CiteSee changes the colors of the
inline citations to familiar papers so that the reader can both better
contextualize the current paper and keep track of citations to papers they
have already explored. In addition, CiteSee also helps readers better make
sense of the cited papers by showing how they connect to a reader’s previous
activities; for example, showing which library folders they were saved under
or the citing sentences from a familiar research paper (Figure 2). Based on
lab and field studies, CiteSee showed promise that providing visual
augmentation and personalized context around inline citations in an
interactive reading environment can allow readers to more effectively discover
relevant prior work and keep track of their exploration during real-world
literature review tasks.

Figure 3.  CiteRead (Rachatasumrit et al., 2022) finds subsequently published
citing research papers, extracts the citation context, and localizes it to
relevant parts of the current research paper as margin notes. This allows
readers to become aware of important follow on work and explore them in-situ.

###  2.2. Exploring Future Work with CiteRead

While augmenting inline citations helps readers to triage them, many relevant
research papers are not cited in a research paper in the first place, for
example, because they were published afterwards. CiteRead is a novel reading
interface that helps readers discover how follow-on work has built on or
engaged with the research paper (Rachatasumrit et al., 2022). Much like social
document annotation systems (Zyto et al., 2012), CiteRead annotates text in
the paper with margin notes containing relevant commentary from citing papers
(Nakov et al., 2004), thereby helping the reader to become aware of the citing
paper and its connection. In order to produce these annotations automatically,
CiteRead first filters citing research papers for ones that are most relevant
to the reader using a trained model atop a number of features representing
citational discourse and textual similarity, i.e. from scientific paper
embeddings (Cohan et al., 2020). CiteRead then localizes citing papers to
particular spans of text in the paper being read, and extracts relevant
information from the citing paper. Figure 3 shows a research paper annotated
with this information from citing papers. Localization is a technical
challenge because while inline citations reference cited papers, they do not
typically reference specific locations in the cited paper; CiteRead determines
location by looking for overlapping spans of text (e.g., a number in common in
the citing paper and the cited paper) or localizes to the relevant section
when this overlap is unavailable. With CiteRead, a reader can directly examine
follow-on work while keeping the citation contexts of both the current paper
and the citing paper. In a lab study, CiteRead helped readers better
understand a research paper and its follow-on work compared to providing
readers with a separate interface for faceted browsing of follow-on work.

##  3\. Navigation and Efficient Reading

An important part of reading a paper is knowing what and where to read.
Scholars often read papers non-linearly; they might return to a previously-
read passage to recall some information, or jump forward to a different
section of the paper (or to another paper) to satisfy an information need
before jumping back. While jumping can help scholars orient their reading to
sections of interest, it can also be a distraction by causing readers to
constantly switch contexts. Non-linear navigation can be especially burdensome
when the reader is interested in a particular _type_ of information (e.g.,
skimming a paper for the main results), but doesn’t know precisely where to
find it within the paper. In this section we discuss two systems, Scim (Fok et
al., 2023) and Ocean (Park et al., 2022), which demonstrate different
approaches to helping readers navigate efficiently through a paper toward
high-value, relevant information.

###  3.1. Guided Reading with Scim

Figure 4. The Scim (Fok et al., 2023) interface guides reader attention using
color highlights corresponding to discourse facets. A sidebar allows users to
toggle facets on/off. Clicking a color-coded snippet scrolls the reader to the
relevant passage.

Scholarly reading can be considered a type of sensemaking represented as a
continuous interplay between two processes: information foraging in which
readers identify relevant paper content, and comprehension in which readers
attempt to integrate the new information into their working model of the paper
and with relevant prior knowledge (Pirolli and Card, 1999; Russell et al.,
1993). Distinguishing between relevant and irrelevant content could help
facilitate efficient reading. Paper abstracts offer one such separation, in
essence an author-crafted determination of relevant content. However, static
paper abstracts can leave readers to desire additional details that then
require them to skim the paper itself.

Scim (Fok et al., 2023) addresses this problem via an augmented reading
interface designed to guide readers’ attention using automatically-created in-
situ faceted highlights (Figure 4). Though prior work has explored
highlighting as a visual cue for guiding reader attention (Wecker et al.,
2014; Chi et al., 2005; Yang et al., 2017), the efficacy for reading of
scholarly text is less well-understood. Scim investigated the following design
goals for intelligent highlights in scholarly reading: highlights should be
(1) evenly-distributed throughout a paper, (2) have just the right density
(too few highlights will present the guise of an inept tool, and too many will
slow a reader down), and (3) highlight several key categories of information
in the paper. Because readers often skim for common types of information, Scim
uses a pretrained language model (Wang et al., 2021a) to classify salient
sentences within papers into one of four information facets: research
objectives, novel aspects of the research, methodology, and results, coupled
with heuristics that ensure an even distribution of highlights. Usability
studies of Scim have shown these highlights can reduce the time it takes
readers to find specific information within a paper. Readers found Scim
particularly useful when skimming text-dense papers, or for papers that fell
outside their area of expertise. Moreover, readers learned to use both Scim’s
inline highlights and a sidebar summary of highlights to augment their
existing reading strategies.

###  3.2. Low-Vision Navigation Support and Reader-Sourced Hyperlinks with
Ocean

The task of navigating between sections and retrieving content can be
particularly challenging for blind and low-vision readers due to limitations
in auditory information access or small viewports under high magnification
(Szpiro et al., 2016). Even when related content is linked, a small viewport
can make navigation difficult and necessitate scrolling (Park et al., 2022).
Most existing tools such as for auditory skimming (Khan et al., 2020) do not
address such challenges associated with low-vision and magnification.

Ocean (Park et al., 2022) minimizes scrolling requirements for low-vision
readers by providing bi-directional, viewport-preserving hyperlinks that
enable navigating to and from associated content without disrupting the
viewport. Based on reported findings from interviews with low-vision readers,
Ocean also allows for easily revisiting portions of the paper with tabbed
reading. Since papers do not always provide hyperlinks and automated link
creation is imperfect, Ocean includes an authoring interface that allows
readers to create and share paper links during reading. An exploratory field
deployment study with mixed-ability groups of low-vision and sighted readers
revealed that readers found value in creating and consuming these links, and
that reader-created links can increase trust.

##  4\. In-Situ Explanations for Better Comprehension

Could an augmented reading application help readers understand a paper by
reducing the cognitive load associated with reading a paper? In this section,
we discuss several ways in which interactive reading aids can help a reader
understand a paper with less work through three systems: ScholarPhi (Head et
al., 2021), PaperPlain (August et al., 2023) and Papeo (team, sion). In
particular, papers can be augmented with definitions of terms and symbols,
provide plain-language summaries of paper passages, and connect readers with
alternative forms of expression (for instance, video clips of research talks)
that offer more approachable explanations of the paper’s content.

###  4.1. Defining Terms and Symbols with ScholarPhi

Figure 5. ScholarPhi (Head et al., 2021) shows definitions of terms and
symbols in pop-up tooltips. When a reader selects a formula, all known
definitions of symbols are shown simultaneously. To let readers select nested
symbols (e.g., “$h$” in “$V^{(j)}_{h}$”), ScholarPhi supports “drill-down”
subsymbol selection.

Understanding a paper requires understanding its vocabulary. However, this is
by no means an easy task—a typical paper may contain dozens of acronyms,
symbols, and invented terms. And often, these terms appear without
accompanying definitions (Murthy et al., 2022). How can we design interactive
aids that present definitions of terms when and where readers most need them?
ScholarPhi (Head et al., 2021) takes as its basis the term gloss—an extension
to a reading interface that shows a reader an explanation of a phrase when
they click it. Glosses appeared in early research interfaces for reading
hypertext (Zellweger et al., 1998) and have since become part of widely-used
reading interfaces including Wikipedia and Kindle.

That said, familiar gloss designs do not work well for scientific papers,
where glosses run the risk of distracting readers, terms have multiple
meanings, and phrases (specifically math symbols) are difficult to
unambiguously select. The ScholarPhi design addresses these challenges. First,
it aims to reduce distraction by showing definitions with high economy:
glosses show multiple definitions and and in-context usages within a compact
tooltip. Second, it provides position-sensitive definitions, revealing
definitions that appears most recently prior to the selected usages of terms.
Terms and definitions are automatically identified using a pretrained language
model (Kang et al., 2020). Finally, it provides easier access to definitions
of mathematical symbols. Readers can access definitions of both a symbol and
the subsymbols it is made of through a multi-click, “drill-down” selection
mechanism. Furthermore, when a reader selects a formula, they can see
definitions for all symbols at once, automatically placed adjacent to the
symbols in the formula’s margins (see Figure 5).

In a usability study, the above interactions reduced the time it took readers
to find answers to questions involving the understanding of terminology. All
readers reported they would use the definition tooltips and formula diagrams
often or always if available in their PDF reader tools.

###  4.2. Simplifying Complex Passages with Paper Plain

Figure 6.  Paper Plain (August et al., 2023) provides in-situ plain language
summaries of passages called “gists” to help readers who are overwhelmed by
complex textual passages. Readers access gists by clicking a flag next to a
section header. These gists are generated by large language models.

Helping a reader understand individual terms and phrases only addresses part
of the problem. Papers often contain passages so dense and complex that
individual definitions are not enough to help someone read the passages,
especially if they are a novice or non-expert in a field (Britt et al., 2014).
Can we make complex texts more approachable by incorporating plain language
summaries in the margins of the text? With Paper Plain (August et al., 2023),
when a reader encounters a section they find difficult to read, they can
access a plain language summary of that section by clicking a button adjacent
to the section header (see Figure 6). These summaries are generated by
prompting a large language model with section text (Brown et al., 2020).

Furthermore, Paper Plain helps guide readers using these summaries as an
“index” into the text. A sidebar containing questions a reader may have about
the text (e.g., _What did the paper find?_ or _What were the limitations?_)
provides links into answering passages identified using a question-answering
system (Yoon et al., 2019) alongside their associated plain language
summaries. These features were designed to help readers understand the “gist”
of passages that contain unfamiliar vocabulary, providing support beyond that
of individual term definitions. Drawing inspiration from prior interactive
reading affordances for term definitions (Jain et al., 2018), in-situ question
answering (Zhao and Lee, 2020; Chaudhri et al., 2013), and guiding reading
(Dzara and Frey-Vogel, 2019), Paper Plain seeks to bring these features
together into a holistic system capable of supporting reading of a paper by a
non-expert readership. In a usability study, readers made more frequent use of
passage summaries than definition tooltips when both were available,
suggesting the potential value of plain language summaries as allowing readers
to bypass definitions of individual terms when acquiring a broad understanding
of a paper.

###  4.3. Fusing Papers and Videos with Papeo

Sometimes, the best explanation of an idea is non-textual. Videos can enhance
understanding (Mayer and Moreno, 1998a) while also requiring less mental load
(Mayer and Moreno, 1998b), and various tools have been designed to facilitate
searching and browsing for explanations in informational videos such as
lectures (Kim et al., 2014a; Liu et al., 2018; Pavel et al., 2014; Krosnick,
2015) and tutorials (Kim et al., 2014b; Truong et al., 2021; Khandwala and
Guo, 2018). Similarly, for research papers, an algorithm might be better
explained through an animation, a user interface might be better showcased
through an screen recording, compared to the proses of a paper (Höffler and
Leutner, 2007). Instead of consuming the two formats independently, could
interactive reading interfaces offer readers access to these alternative, more
powerful descriptive forms as they read? For this, Papeo (team, sion) was
developed as a tool that supplements papers with more engaging, concise,
dynamic presentations of information by linking excerpts of talk videos to
corresponding paper passages. To grant authors more control over how their
work is presented, we developed an AI-supported authoring interface for
linking paper passages and videos efficiently: candidate passages are linked
to excerpts of videos as suggestions using a pretrained language model (Wang
et al., 2020), and an author interactively confirms or refines them.

Unlike text-skimming with Scim (§3.1) and Paper Plain (§4.2), video-skimming
in Papeo combines multiple modalities to explain complex information. For
example, instead of reading a long text description of an interactive system,
readers could see the system’s behavior in a screen recording video with the
author’s commentary, and switch to corresponding passages to see
implementation details or design motivations if desired. Our early-stage
evaluations of Papeo suggest that readers can use these interactions to
fluidly transition between watching video and reading text, using video to
quickly understand, and then selectively descending into the text when they
desire a detailed understanding of the paper.

##  5\. Scaffolding Synthesis with Related Work Sections

Scientific breakthroughs often rely upon scholars synthesizing multiple
published works into broad overviews to identify gaps in the current
literature (Portenoy et al., 2022). For this, scholars periodically compile
survey articles to help other scholars gain a comprehensive overview of
important research topics. For example, some fields have dedicated outlet for
such articles (e.g., the Psychological Bulletin (Bem, 1995)). However, survey
articles require significant time and effort to synthesize, and can quickly
become outdated with the exponential growth of scientific publication
(Bornmann et al., 2020).

Instead, scholars in fast-paced disciplines often rely on the related work
section when they need to better understand the broader background when
reading a paper. While related work sections also summarize multiple prior
works, unlike comprehensive survey articles, they typically provide partial
views of the larger research topic most relevant to a single paper. There is
an opportunity to build better tooling for scholars to consume and synthesize
multiple related work sections across many papers to gain richer and more
comprehensive overviews of fast-paced domains. The Threddy (Kang et al.,
2022a) and Related (Palani et al., 2023) projects explored this opportunity
using two different approaches: clipping and organizing research threads
mentioned across papers (Kang et al., 2022a), and directly exploring and
reading related work sections extracted across many papers (Palani et al.,
2023).

Figure 7. Papeo (team, sion) enables authors to map segments of talk videos to
relevant passages in the paper, allowing readers to fluidly switch between the
two formats. Color-coded bars show the mapping between the two formats, and
allow readers to scrub through video segments for quick previews.

###  5.1. Clipping and Synthesizing across Papers with Threddy

Clipping and note-taking is one common approach to supporting synthesis across
multiple documents. Prior work has pointed to the importance of tightly
integrating clipping and synthesis support in the reading process, and how
incurring significant context-switching costs can be detrimental to
sensemaking (Kittur et al., 2013; Russell et al., 1993; Pirolli and Card,
1999). Therefore, recent work has developed tools aimed at reducing the
cognitive and interaction costs of clipping (Liu et al., 2022b; Chang et al.,
2016a) and structuring (Chang et al., 2020; Kuznetsov et al., 2022; Liu et
al., 2022a, 2019; Subramonyam et al., 2020) to support everyday online
researchers (Chang et al., 2020), programmers (Liu et al., 2019), and students
(Subramonyam et al., 2020). However, designing clipping and synthesis support
tools for research papers is relatively under-explored and introduces exciting
new research opportunities. For example, additional organizational structures
for literature reviews (e.g., threads of prior work instead of tables (Liu et
al., 2019; Chang et al., 2020)), and research paper discovery (e.g., based on
inline citations in clipped text).

For this, Threddy (Kang et al., 2022a) is a thread-focused clipping tool
integrated into scholars’ paper reading process to support literature review
and discovery. Using Threddy, readers can select and save sentences into a
sidebar from the related work sections of a paper. The system maintains rich
context for each clip, including its provenance and inline citations. This
allows readers to navigate back to the clipped paper and cited papers
afterward. In the sidebar, readers can further organize clips collected across
papers into a hierarchy of threads to form their view of the research
landscape. The content of the sidebar is preserved across papers that were
read over time, and provides valuable context for subsequent reading based on
the emerging threads of research the reader have curated. Finally, readers can
further expand their coverage by exploring paper recommendations for each
thread, based on the referenced papers in the corresponding clips. A lab study
showed that Threddy was able to lower the interaction costs of saving clips
while maintaining context, allowed participants to curate research threads
without breaking reading flows, and discover interesting new papers to further
grow their understanding of the research fields.

###  5.2. Reading and Exploring Related Work Sections across Papers with
Relatedly

In contrast to Threddy, which aims to improve readers’ existing literature
review process through enhanced in-situ clipping and synthesis (Kang et al.,
2022a), the Relatedly system introduced a novel workflow that allows readers
to explore many related work sections across papers in an interactive search
and reading interface to quickly gain a comprehensive overview of rich
research topics (Palani et al., 2023). While prior work have explored
providing overview structure of multiple documents based on citations (Ponsard
et al., 2016; Chau et al., 2011), semantic similarity (Hearst, 2006; Shahaf et
al., 2012), or human computation (Hahn et al., 2016; Chang et al., 2016b;
Luther et al., 2015), they could still lead to complex structures that are
hard to interpret (Hearst, 1999) or require significant crowdsourcing efforts.
Relatedly sidesteps these issues by _reusing_ existing related work paragraphs
in published papers which already cite sets of related references with
descriptions connecting them (Palani et al., 2023). As an example, consider a
scholar trying to better understand the space of _online misinformation_. With
_online misinformation_ as the query term, Relatedly shows the reader a list
of paragraphs that describe and cite multiple relevant prior work. Using a
pretrained language model for summarization (Lewis et al., 2020), Relatedly
generates short and descriptive titles for each paragraph, and uses a
diversity-based ranking algorithm so that the reader can quickly see and
explore paragraphs describing different research threads, such as _Fact
Checking Datasets_ , _Social Media and Misinformation_ , and _Fake News
Detection Techniques_.

One challenge here is that paragraphs of the same threads often cite
overlapping prior work, making them hard to explore and read while keeping
track of which papers were new versus already explored. For this, Relatedly
provides reading and cross-referencing support by keeping track of paragraphs
and references explored by the readers. This allows Relatedly to help readers
prioritize their reading for both breadth and depth. Specifically, Relatedly
dynamically re-ranks paragraphs and highlights sentences to spotlight
unexplored and dissimilar references for breadth, but also allow readers to
explore clusters of paragraphs that cited similar references for depth. A
usability study comparing Relatedly to a strong document-centric baseline
showed that Relatedly led to participants writing summaries that were rated
significantly more coherent, insightful, and detailed after 20 minutes of
literature review.

##  6\. Dynamic Documents for Improved Accessibility

A range of disabilities cause people to read scientific documents using a wide
variety of devices and reading tools. For example, blind and low vision
readers may use assistive reading technology such as screen readers, screen
magnification, or text-to-speech to read documents (Szpiro et al., 2016).
Furthermore, people without disabilities face situational impairments, such as
the inability to view a screen while driving or may have a preference for
consuming content on a small, mobile device.

Many of these reading tools, such as screen readers, do not function properly
on document formats designed for print such as PDF unless the document has
been manually post-processed to add information about reading order, content
type, etc., which is rarely performed on scientific documents (Bigham et al.,
2016; Wang et al., 2021c). Further, certain content elements such as figures
require the addition of alternative text in order to be read aloud at all
(figure captions typically assume the reader can see the figure and do not
provide the same semantic content as alt text). High magnification reduces the
viewport (the amount of visible content) and can dramatically increase the
amount of scrolling and panning required, especially for multi-columnar
formats that are commonly used by scientific documents. Visual scanning for
information may be impacted or unavailable in these settings, making it more
difficult to find and navigate between content in the document (Park et al.,
2022).

One way to render legacy PDF content more accessibly is to parse and convert
it into a more flexible format, such as XML or HTML, which can then be
formatted for mobile devices and augmented for reading by screen readers. The
SciA11y system666A demo of a subsequent version is available at
https://papertohtml.org/ demonstrates this approach, automatically converting
12M academic PDFs to HTML (Wang et al., 2021b); a user study with blind and
low vision participants demonstrated strong user appreciation of the output,
though some errors remain (e.g., failing in certain cases to distinguish
footnotes from body text, difficulty parsing math equations) (Wang et al.,
2021c). When available, alt text can be automatically categorized into
semantic content types, enabling new reading experiences that allow skipping
or prioritizing certain types (Chintalapati et al., 2022). Other approaches
provide complementary benefits, such as interfaces tailored for low-vision
readers (§ 3.2), as well as the range of reading support systems outlined
above.

##  7\. Discussion and Future Work

There are additional directions to explore to better support scholarly
activities through the Semantic Reader Project.

#### Towards a full-featured reading experience.

One question is how to integrate the different kinds of functionality across
these projects into one coherent user interface, especially as we migrate
research features into the production interface. Another question is how to
develop support for the oftentimes social and collaborative nature of
scholarly reading. Scholars frequently leverage their social networks and
other social signals for paper discovery (Kang et al., 2022b), work in groups
to conduct literature review triage and synthesis, or engage in reading group
discussions to aid comprehension. Existing augmentations within the Semantic
Reader product could imbue social information, such as providing signals from
one’s co-author network (e.g., in CiteSee §2.1) or aggregate navigation traces
(e.g., in Scim §3.1). The publicly-available Semantic Reader tool could also
scaffold the creation of novel crowd- or community-sourced content, such as
author- or reader-provided explanations, commentary, or verification of paper
content. Finally is the question of how we can allow the scholarly community
to step in where current AI systems fall short, such as by fixing improperly-
extracted content or incorrect generated text which are especially problematic
for interfaces such as SciA11y (§6).

#### Advancing AI for scholarly documents.

The Semantic Reader Project presents an opportunity for further AI research in
scholarly document processing, especially when paired with human-centered
research grounded in user-validated systems and scenarios. The bar for
deploying AI models to support real-world reading is high; we often found
during iterative design and usability studies that even slight errors in these
models can have detrimental effects on the readers. Until recently, interface
design could require months of development of bespoke AI models which creates
a barrier for quickly iterate different system designs. Recent advancements in
scaling large language models (LLMs) has altered this landscape by enabling
researchers to experiment with a wide range of new NLP capabilities at
relatively low cost (Brown et al., 2020). This has the potential of
significantly lowering the cost of human-centered AI design by incorporating
user feedback in earlier stages of system development to create AI systems
that work in symphony with the users beyond pure automation (Shneiderman,
2022). For example, when developing Paper Plain (§4.2), LLMs enabled us to
quickly test different granularities and complexity-levels of plain language
summaries with participants, eschewing the need for expensive changes to data
requirements and model retraining. In the near-term, we will revisit interface
designs relying on bespoke AI models to evaluate whether LLMs can close the
gap between research prototype and ready-for-production (e.g., more accurate
definition identification for ScholarPhi §4.1). Longer-term, we will explore
whether LLMs can power new interactions (e.g., user-provided natural language
queries while reading (Dasigi et al., 2021; Wadden et al., 2020)). While
recent work has shown that these models can occasionally make critical errors
or generate factually incorrect text when processing scientific text
(Otmakhova et al., 2022), we remain cautiously optimistic about developing
ways to address their limitations (Dove et al., 2017; Lee et al., 2010).

#### Ethics of augmented papers

Finally, all the new interfaces for reading that we propose pose a number of
important ethical considerations that will require further research and
discussion. One aspect that arises with any system for elevating certain
papers or certain content over others is bias. For instance, using signals
such as citation counts faces the risk of a “rich get richer” bias, which can
reflect other kinds of documented biases (Beel and Gipp, 2009; Maliniak et
al., 2013; Way et al., 2019). As a result, systems such as CiteSee (§2.1) or
Relatedly (5.2) should carefully consider additional signals of relevance such
as semantic similarity to surface newer and overlooked papers. Another tension
that we have encountered is the potential discrepancy between author desires
and reader desires for how a work is presented and how much control to provide
authors. For instance, in our work on Papeo (§4.3), we found that authors
desired control over placement of their talk video snippets, even as they
found automated mapping support to be helpful. In other cases, authors might
not have the requisite expertise (e.g., they may not have a good sense of
reader needs or what non-experts are confused by) or may have the wrong
incentives. Future work should consider author perspectives on these augmented
experiences. A related issue is around systems for more efficient reading or
synthesis, which may encourage readers to take shortcuts that lead to
incorrect understanding, sloppy research, or even outright plagiarism. Instead
of simply seeking to increase reading throughput uniformly, our systems should
enable triage, so that readers can dedicate time for thoughtful and careful
reading when the content is important. For instance, our systems could design
pathways that, while they may be more efficient, do not obfuscate the full
context (e.g., Scim §3.1), and that encourage good practices such as
verification and provenance tracing. A final consideration is around what is
ethical reuse of a paper’s contents to support reader experiences outside of
that paper and its licensing implications. For instance, CiteRead (§2.2)
extracts paper citances and places them in the cited paper, and Relatedly
(§5.2) extracts related work sections from different papers for users to
explore. Recent trends in _open science and datasets_ (McKiernan et al., 2016;
McKiernan, 2000; Ginsparg, 2011; Lo et al., 2020) point to a promising future
where we could continue to explore different ways to _remix and reuse_
scholarly content across context so that future scientists can take fuller
advantage of prior research.

##  8\. Conclusion

This paper describes the Semantic Reader Project, which currently consists of
ten research prototypes focusing on supporting scientists around Discovery
(Chang et al., 2023; Rachatasumrit et al., 2022), Efficiency (Fok et al.,
2023; Park et al., 2022), Comprehension (Head et al., 2021; August et al.,
2023; team, sion), Synthesis (Palani et al., 2023; Kang et al., 2022a), and
Accessibility (Wang et al., 2021b; team, tion) when reading research papers.
Validating our approach of augmenting existing PDFs of research papers, we
have seen tremendous adoption of the freely-available Semantic Reader product1
which has grown to 10k weekly users.777As of late February, 2023 While we
focused on augmenting PDF documents to support common scholar reading
practices, all of our reading interfaces are built with web
technologies—allowing these novel interactions to extend to future publication
formats which can be rendered in web browsers. We plan to continue
experimenting with novel AI-powered intelligent reading interfaces, as well as
migrating successful interactive features into the product. Finally, we offer
a collection of freely-available resources to the larger research community,
including datasets of open-access research papers (Lo et al., 2020), APIs for
accessing the academic citation graph (Kinney et al., 2023), machine learning
models for processing and understanding research papers (Cohan et al., 2020;
Cachola et al., 2020; Shen et al., 2022; Kang et al., 2020),4 and open-source
software for rendering and augmenting PDF documents for developing reading
interfaces.3 We hope by providing these resources we can enable and encourage
the broader research community to work on exciting novel intelligent reading
interfaces for research papers with us.

###### Acknowledgements.

This project is supported in part by NSF Grant OIA-2033558, NSF Grant
CNS-2213656. NSF RAPID Award 2040196, and ONR Grant N00014-21-1-2707.

## References

  * (1)
  * August et al. (2023) Tal August, Lucy Lu Wang, Jonathan Bragg, Marti A. Hearst, Andrew Head, and Kyle Lo. 2023\.  Paper Plain: Making Medical Research Papers Approachable to Healthcare Consumers with Natural Language Processing.  _ACM Transactions on Computer-Human Interaction_ (2023).  To appear.
  * Bazerman (1985) Charles Bazerman. 1985\.  Physicists reading physics: Schema-Laden Purposes and Purpose-Laden Schema.  _Written Communication_ 2, 1 (Jan. 1985), 3–23.
  * Beel and Gipp (2009) Joeran Beel and Bela Gipp. 2009.  Google Scholar’s ranking algorithm: The impact of citation counts (An empirical study).  _2009 Third International Conference on Research Challenges in Information Science_ (2009), 439–446.
  * Beltagy et al. (2019) Iz Beltagy, Kyle Lo, and Arman Cohan. 2019.  SciBERT: A Pretrained Language Model for Scientific Text. In _Conference on Empirical Methods in Natural Language Processing_.
  * Bem (1995) Daryl J Bem. 1995\.  Writing a review article for Psychological Bulletin.  _Psychological Bulletin_ 118, 2 (1995), 172\.
  * Bigham et al. (2016) Jeffrey P. Bigham, Erin L. Brady, Cole Gleason, Anhong Guo, and David A. Shamma. 2016.  An Uninteresting Tour Through Why Our Research Papers Aren’t Accessible. In _Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems_ (San Jose, California, USA) _(CHI EA ’16)_. Association for Computing Machinery, New York, NY, USA, 621–631.  https://doi.org/10.1145/2851581.2892588
  * Bornmann et al. (2020) Lutz Bornmann, Ruediger Mutz, and Robin Haunschild. 2020\.  Growth rates of modern science: a latent piecewise growth curve approach to model publication numbers from established and new literature databases.  _Humanities and Social Sciences Communications_ 8 (2020), 1–15.
  * Brainard (2020) Jeffrey Brainard. 2020\.  Scientists are drowning in COVID-19 papers. Can new tools keep them afloat.  _Science_ 13, 10 (2020), 1126.
  * Britt et al. (2014) Mary Anne Britt, Tobias Richter, and Jean-François Rouet. 2014\.  Scientific Literacy: The Role of Goal-Directed Reading and Evaluation in Understanding Scientific Information.  _Educational Psychologist_ 49 (2014), 104 – 122.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.  Language Models are Few-Shot Learners. In _Advances in Neural Information Processing Systems_ , H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (Eds.), Vol. 33. Curran Associates, Inc., 1877–1901.  https://proceedings.neurips.cc/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf
  * Cachola et al. (2020) Isabel Cachola, Kyle Lo, Arman Cohan, and Daniel S. Weld. 2020\.  TLDR: Extreme Summarization of Scientific Documents. In _Findings of EMNLP_.
  * Chang et al. (2016a) Joseph Chee Chang, Nathan Hahn, and Aniket Kittur. 2016a.  Supporting Mobile Sensemaking Through Intentionally Uncertain Highlighting. In _Proceedings of the 29th Annual Symposium on User Interface Software and Technology_ (Tokyo, Japan) _(UIST ’16)_. Association for Computing Machinery, New York, NY, USA, 61–68.  https://doi.org/10.1145/2984511.2984538
  * Chang et al. (2020) Joseph Chee Chang, Nathan Hahn, and Aniket Kittur. 2020\.  Mesh: Scaffolding Comparison Tables for Online Decision Making. In _Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology_ (Virtual Event, USA) _(UIST ’20)_. Association for Computing Machinery, New York, NY, USA, 391–405.  https://doi.org/10.1145/3379337.3415865
  * Chang et al. (2016b) Joseph Chee Chang, Aniket Kittur, and Nathan Hahn. 2016b.  Alloy: Clustering with crowds and computation. In _Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems_. 3180–3191.
  * Chang et al. (2023) Joseph Chee Chang, Amy X Zhang, Jonathan Bragg, Andrew Head, Kyle Lo, Doug Downey, and Daniel S Weld. 2023.  CiteSee: Augmenting Citations in Scientific Papers with Persistent and Personalized Historical Context.  _ArXiv_ abs/2022.99999 (2023).
  * Chau et al. (2011) Duen Horng Chau, Aniket Kittur, Jason I Hong, and Christos Faloutsos. 2011.  Apolo: making sense of large network data by combining rich user interaction and machine learning. In _Proceedings of the SIGCHI conference on human factors in computing systems_. 167–176.
  * Chaudhri et al. (2013) Vinay K. Chaudhri, Britte Haugan Cheng, Adam Overholtzer, Jeremy Roschelle, Aaron Spaulding, Peter E. Clark, Mark T. Greaves, and David Gunning. 2013.  Inquire Biology: A Textbook that Answers Questions.  _AI Mag._ 34 (2013), 55–72.
  * Chi et al. (2005) Ed H. Chi, Lichan Hong, Michelle Gumbrecht, and Stuart K. Card. 2005.  ScentHighlights: highlighting conceptually-related sentences during reading. In _Proceedings of the 10th International Conference on Intelligent User Interfaces_. Association for Computing Machinery, San Diego, CA, USA, 272.
  * Chintalapati et al. (2022) Sanjana Shivani Chintalapati, Jonathan Bragg, and Lucy Lu Wang. 2022\.  A Dataset of Alt Texts from HCI Publications: Analyses and Uses Towards Producing More Descriptive Alt Texts of Data Visualizations in Scientific Papers. In _Proceedings of the 24th International ACM SIGACCESS Conference on Computers and Accessibility_ (Athens, Greece) _(ASSETS ’22)_. Association for Computing Machinery, New York, NY, USA, Article 30, 12 pages.  https://doi.org/10.1145/3517428.3544796
  * Cohan et al. (2020) Arman Cohan, Sergey Feldman, Iz Beltagy, Doug Downey, and Daniel S Weld. 2020.  Specter: Document-level representation learning using citation-informed transformers.  _arXiv preprint arXiv:2004.07180_ (2020).
  * Conlen and Heer (2022) Matthew Conlen and Jeffrey Heer. 2022.  Fidyll: A Compiler for Cross-Format Data Stories & Explorable Explanations.  _ArXiv_ abs/2205.09858 (2022).
  * Conlen et al. (2021) Matthew Conlen, Megan Vo, Alan Tan, and Jeffrey Heer. 2021\.  Idyll studio: A structured editor for authoring interactive & data-driven articles. In _The 34th Annual ACM Symposium on User Interface Software and Technology_. 1–12.
  * Dasigi et al. (2021) Pradeep Dasigi, Kyle Lo, Iz Beltagy, Arman Cohan, Noah A. Smith, and Matt Gardner. 2021\.  A Dataset of Information-Seeking Questions and Answers Anchored in Research Papers. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_. Association for Computational Linguistics, Online, 4599–4610.  https://doi.org/10.18653/v1/2021.naacl-main.365
  * Denney and Tewksbury (2013) Andrew S. Denney and Richard Allan Tewksbury. 2013.  How to Write a Literature Review.  _Journal of Criminal Justice Education_ 24 (2013), 218 – 234.
  * Dove et al. (2017) Graham Dove, Kim Halskov, Jodi Forlizzi, and John Zimmerman. 2017\.  UX design innovation: Challenges for working with machine learning as a design material. In _Proceedings of the 2017 chi conference on human factors in computing systems_. 278–288.
  * Dzara and Frey-Vogel (2019) Kristina Dzara and Ariel S Frey-Vogel. 2019.  Medical Education Journal Club for the Millennial Resident: An Interactive, No-Prep Approach.  _Academic pediatrics_ (2019).
  * Editorial Team (2021) Editorial Team. 2021\.  Distill Hiatus.  _Distill_ 6, 7 (July 2021).  https://doi.org/10.23915/distill.00031
  * Fok et al. (2023) Raymond Fok, Hita Kambhamettu, Luca Soldaini, Jonathan Bragg, Kyle Lo, Andrew Head, Marti A. Hearst, and Daniel S. Weld. 2023\.  Scim: Intelligent Skimming Support for Scientific Papers. In _28th Annual Conference on Intelligent User Interfaces (IUI ’23)_.
  * Ginsparg (2011) Paul Ginsparg. 2011\.  ArXiv at 20.  _Nature_ 476, 7359 (2011), 145–147.
  * Gori and Pucci (2006) Marco Gori and Augusto Pucci. 2006.  Research Paper Recommender Systems: A Random-Walk Based Approach.  _2006 IEEE/WIC/ACM International Conference on Web Intelligence (WI 2006 Main Conference Proceedings)(WI’06)_ (2006), 778–781.
  * Hahn et al. (2016) Nathan Hahn, Joseph Chang, Ji Eun Kim, and Aniket Kittur. 2016\.  The Knowledge Accelerator: Big picture thinking in small pieces. In _Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems_. 2258–2270.
  * He et al. (2019) Jiangen He, Q. Ping, Wen Lou, and Chaomei Chen. 2019\.  PaperPoles: Facilitating adaptive visual exploration of scientific publications by citation links.  _Journal of the Association for Information Science and Technology_ 70 (2019).
  * Head et al. (2021) Andrew Head, Kyle Lo, Dongyeop Kang, Raymond Fok, Sam Skjonsberg, Daniel S. Weld, and Marti A. Hearst. 2021.  Augmenting Scientific Papers with Just-in-Time, Position-Sensitive Definitions of Terms and Symbols.  _Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems_ (2021).
  * Head et al. (2022) Andrew Head, Amber Xie, and Marti A Hearst. 2022.  Math Augmentation: How Authors Enhance the Readability of Formulas using Novel Visual Design Practices. In _CHI Conference on Human Factors in Computing Systems_. 1–18.
  * Hearst (1999) Marti A Hearst. 1999\.  The use of categories and clusters for organizing retrieval results.  _Natural language information retrieval_ (1999), 333–374.
  * Hearst (2006) Marti A Hearst. 2006\.  Clustering versus faceted categories for information exploration.  _Commun. ACM_ 49, 4 (2006), 59–61.
  * Höffler and Leutner (2007) Tim Niclas Höffler and Detlev Leutner. 2007.  Instructional animation versus static pictures: A meta-analysis.  _Learning and Instruction_ 17 (2007), 722–738.
  * Hohman et al. (2020) Fred Hohman, Matthew Conlen, Jeffrey Heer, and Duen Horng Chau. 2020.  Communicating with Interactive Articles.  _Distill_.  https://doi.org/10.23915/distill.00028
  * Huang et al. (2022) Yupan Huang, Tengchao Lv, Lei Cui, Yutong Lu, and Furu Wei. 2022.  LayoutLMv3: Pre-training for Document AI with Unified Text and Image Masking.  _Proceedings of the 30th ACM International Conference on Multimedia_ (2022).
  * Huang et al. (2002) Zan Huang, Wingyan Chung, Thian-Huat Ong, and Hsinchun Chen. 2002\.  A graph-based recommender system for digital library. In _JCDL ’02_.
  * Jain et al. (2018) Abhinav Jain, Nitin Gupta, Shashank Mujumdar, Sameep Mehta, and Rishi Madhok. 2018.  Content Driven Enrichment of Formal Text using Concept Definitions and Applications.  _Proceedings of the 29th on Hypertext and Social Media_ (2018).
  * Kang et al. (2020) Dongyeop Kang, Andrew Head, Risham Sidhu, Kyle Lo, Daniel Weld, and Marti A. Hearst. 2020\.  Document-Level Definition Detection in Scholarly Documents: Existing Models, Error Analyses, and Future Directions. In _Proceedings of the First Workshop on Scholarly Document Processing_. Association for Computational Linguistics, Online, 196–206.  https://doi.org/10.18653/v1/2020.sdp-1.22
  * Kang et al. (2022a) Hyeonsu B Kang, Joseph Chee Chang, Yongsung Kim, and Aniket Kittur. 2022a.  Threddy: An Interactive System for Personalized Thread-based Exploration and Organization of Scientific Literature.  _arXiv preprint arXiv:2208.03455_ (2022).
  * Kang et al. (2022b) Hyeonsu B Kang, Rafal Kocielnik, Andrew Head, Jiangjiang Yang, Matt Latzke, Aniket Kittur, Daniel S Weld, Doug Downey, and Jonathan Bragg. 2022b.  From Who You Know to What You Read: Augmenting Scientific Recommendations with Implicit Social Networks. In _Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems_ (New Orleans, LA, USA) _(CHI ’22)_. Association for Computing Machinery, New York, NY, USA, Article 302, 23 pages.  https://doi.org/10.1145/3491102.3517470
  * Khan et al. (2020) Taslim Arefin Khan, Dongwook Yoon, and Joanna McGrenere. 2020\.  Designing an Eyes-Reduced Document Skimming App for Situational Impairments.  _Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems_ (2020).
  * Khandwala and Guo (2018) Kandarp Khandwala and Philip J. Guo. 2018.  Codemotion: expanding the design space of learner interactions with computer programming tutorial videos.  _Proceedings of the Fifth Annual ACM Conference on Learning at Scale_ (2018).
  * Kim et al. (2014a) Juho Kim, Philip J. Guo, Carrie J. Cai, Shang-Wen Li, Krzysztof Z Gajos, and Rob Miller. 2014a.  Data-driven interaction techniques for improving navigation of educational videos.  _Proceedings of the 27th annual ACM symposium on User interface software and technology_ (2014).
  * Kim et al. (2014b) Juho Kim, Phu Tran Nguyen, Sarah A. Weir, Philip J. Guo, Rob Miller, and Krzysztof Z Gajos. 2014b.  Crowdsourcing step-by-step information extraction to enhance existing how-to videos.  _Proceedings of the SIGCHI Conference on Human Factors in Computing Systems_ (2014).
  * King et al. (2009) Donald W. King, Carol Tenopir, Songphan Choemprayong, and Lei Wu. 2009.  Scholarly journal information‐seeking and reading patterns of faculty at five US universities.  _Learned Publishing_ 22 (2009).
  * Kinney et al. (2023) Rodney Kinney, Chloe Anastasiades, Russell Authur, Iz Beltagy, Jonathan Bragg, Alexandra Buraczynski, Isabel Cachola, Stefan Candra, Yoganand Chandrasekhar, Arman Cohan, et al. 2023\.  The Semantic Scholar Open Data Platform.  _arXiv preprint arXiv:2301.10140_ (2023).
  * Kittur et al. (2013) Aniket Kittur, Andrew M. Peters, Abdigani Diriye, Trupti Telang, and Michael R. Bove. 2013.  Costs and Benefits of Structured Information Foraging. In _Proceedings of the SIGCHI Conference on Human Factors in Computing Systems_ (Paris, France) _(CHI ’13)_. Association for Computing Machinery, New York, NY, USA, 2989–2998.  https://doi.org/10.1145/2470654.2481415
  * Krosnick (2015) Rebecca Krosnick. 2015\.  VideoDoc : combining videos and lecture notes for a better learning experience.
  * Kuznetsov et al. (2022) Andrew Kuznetsov, Joseph Chee Chang, Nathan Hahn, Napol Rachatasumrit, Bradley Breneisen, Julina Coupland, and Aniket Kittur. 2022\.  Fuse: In-Situ Sensemaking Support in the Browser. In _Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology_ (Bend, OR, USA) _(UIST ’22)_. Association for Computing Machinery, New York, NY, USA, Article 34, 15 pages.  https://doi.org/10.1145/3526113.3545693
  * Latif et al. (2021) Shahid Latif, Zhengzhong Zhou, Yoon Kim, Fabian Beck, and Nam Wook Kim. 2021.  Kori: Interactive Synthesis of Text and Charts in Data Documents.  _IEEE Transactions on Visualization and Computer Graphics_ PP (2021), 1–1.
  * Lee et al. (2010) Min Kyung Lee, Sara B. Kiesler, Jodi Forlizzi, Siddhartha S. Srinivasa, and Paul E. Rybski. 2010\.  Gracefully mitigating breakdowns in robotic services.  _2010 5th ACM/IEEE International Conference on Human-Robot Interaction (HRI)_ (2010), 203–210.
  * Lewis et al. (2020) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020.  BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_. Association for Computational Linguistics, Online, 7871–7880.  https://doi.org/10.18653/v1/2020.acl-main.703
  * Liu et al. (2018) Ching Liu, Juho Kim, and Hao-Chuan Wang. 2018.  ConceptScape: Collaborative Concept Mapping for Video Learning.  _Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems_ (2018).
  * Liu et al. (2019) Michael Xieyang Liu, Jane Hsieh, Nathan Hahn, Angelina Zhou, Emily Deng, Shaun Burley, Cynthia Taylor, Aniket Kittur, and Brad A Myers. 2019.  Unakite: Scaffolding developers’ decision-making using the web. In _Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology_. 67–80.
  * Liu et al. (2022a) Michael Xieyang Liu, Aniket Kittur, and Brad A. Myers. 2022a.  Crystalline: Lowering the Cost for Developers to Collect and Organize Information for Decision Making. In _Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems_ (New Orleans, LA, USA) _(CHI ’22)_. Association for Computing Machinery, New York, NY, USA, Article 68, 16 pages.  https://doi.org/10.1145/3491102.3501968
  * Liu et al. (2022b) Michael Xieyang Liu, Andrew Kuznetsov, Yongsung Kim, Joseph Chee Chang, Aniket Kittur, and Brad A. Myers. 2022b.  Wigglite: Low-Cost Information Collection and Triage. In _Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology_ (Bend, OR, USA) _(UIST ’22)_. Association for Computing Machinery, New York, NY, USA, Article 32, 16 pages.  https://doi.org/10.1145/3526113.3545661
  * Lo et al. (2020) Kyle Lo, Lucy Lu Wang, Mark Neumann, Rodney Kinney, and Daniel Weld. 2020.  S2ORC: The Semantic Scholar Open Research Corpus. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_. Association for Computational Linguistics, Online, 4969–4983.  https://doi.org/10.18653/v1/2020.acl-main.447
  * Luther et al. (2015) Kurt Luther, Nathan Hahn, Steven P Dow, and Aniket Kittur. 2015\.  Crowdlines: Supporting synthesis of diverse information sources through crowdsourced outlines. In _Third AAAI Conference on Human Computation and Crowdsourcing_.
  * Mackinlay et al. (1995) Jock D. Mackinlay, Ramana Rao, and Stuart K. Card. 1995\.  An organic user interface for searching citation links. In _CHI ’95_.
  * Maliniak et al. (2013) Daniel Maliniak, Ryan Powers, and Barbara F. Walter. 2013\.  The Gender Citation Gap in International Relations.  _International Organization_ 67 (2013), 889 – 922.
  * Mayer and Moreno (1998a) Richard E. Mayer and Roxana Moreno. 1998a.  A Cognitive Theory of Multimedia Learning: Implications for Design Principles. In _CHI 1998_.
  * Mayer and Moreno (1998b) Richard E. Mayer and Roxana Moreno. 1998b.  A Split-Attention Effect in Multimedia Learning: Evidence for Dual Processing Systems in Working Memory.  _Journal of Educational Psychology_ 90 (1998), 312–320.
  * McKiernan et al. (2016) Erin C McKiernan, Philip E Bourne, C Titus Brown, Stuart Buck, Amye Kenall, Jennifer Lin, Damon McDougall, Brian A Nosek, Karthik Ram, Courtney K Soderberg, et al. 2016\.  How open science helps researchers succeed.  _elife_ 5 (2016).
  * McKiernan (2000) Gerry McKiernan. 2000\.  arXiv.org: the Los Alamos National Laboratory e-print server.  _International Journal on Grey Literature_ (2000).
  * Murthy et al. (2022) Sonia K Murthy, Kyle Lo, Daniel King, Chandra Bhagavatula, Bailey Kuehl, Sophie Johnson, Jonathan Borchardt, Daniel S Weld, Tom Hope, and Doug Downey. 2022\.  ACCoRD: A Multi-Document Approach to Generating Diverse Descriptions of Scientific Concepts.  _arXiv preprint arXiv:2205.06982_ (2022).
  * Nakov et al. (2004) Preslav I Nakov, Ariel S Schwartz, Marti Hearst, et al. 2004\.  Citances: Citation sentences for semantic analysis of bioscience text. In _Proceedings of the SIGIR_ , Vol. 4. Citeseer, 81–88.
  * Okamura (2019) Keisuke Okamura. 2019\.  Interdisciplinarity revisited: evidence for research impact and dynamism.  _Palgrave Communications_ 5, 1 (2019), 1–9.
  * Otmakhova et al. (2022) Yulia Otmakhova, Karin Verspoor, Timothy Baldwin, and Jey Han Lau. 2022.  The patient is more dead than alive: exploring the current state of the multi-document summarisation of the biomedical literature. In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_. Association for Computational Linguistics, Dublin, Ireland, 5098–5111.  https://doi.org/10.18653/v1/2022.acl-long.350
  * Palani et al. (2023) Srishti Palani, Aakanksha Naik, Doug Downey, Amy X. Zhang, Jonathan Bragg, and Joseph Chee Chang. 2023\.  Relatedly: Scaffolding Literature Reviews with Existing Related Work Sections.  22 (2023).
  * Palmer et al. (2009) Carole L Palmer, Lauren C Teffeau, and Carrie M Pirmann. 2009\.  Scholarly information practices in the online environment.  _Report commissioned by OCLC Research. Published online at: www. oclc. org/programs/publications/reports/2009-02. pdf_ (2009).
  * Park et al. (2022) Soya Park, Jonathan Bragg, Michael Chang, Kevin Larson, and Danielle Bragg. 2022.  Exploring Team-Sourced Hyperlinks to Address Navigation Challenges for Low-Vision Readers of Scientific Papers. In _Proceedings of the 25th ACM Conference On Computer-Supported Cooperative Work And Social Computing_.
  * Pavel et al. (2014) Amy Pavel, Colorado Reed, Björn Hartmann, and Maneesh Agrawala. 2014.  Video digests: a browsable, skimmable format for informational lecture videos.  _Proceedings of the 27th annual ACM symposium on User interface software and technology_ (2014).
  * Peroni et al. (2015) Silvio Peroni, Alexander Dutton, Tanya Gray, and David M. Shotton. 2015.  Setting our bibliographic references free: towards open citation data.  _J. Documentation_ 71 (2015), 253–277.
  * Philip et al. (2014) Simon Philip, Peter Bamidele Shola, and Abari Ovye John. 2014.  Application of Content-Based Approach in Research Paper Recommendation System for a Digital Library.  _International Journal of Advanced Computer Science and Applications_ 5 (2014).
  * Pirolli and Card (1999) Peter Pirolli and Stuart Card. 1999.  Information Foraging.  _Psychological Review_ 106, 4 (1999), 643–675.  https://doi.org/10.1037/0033-295x.106.4.643
  * Ponsard et al. (2016) Antoine Ponsard, Francisco Escalona, and Tamara Munzner. 2016\.  PaperQuest: A Visualization Tool to Support Literature Review.  _Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems_ (2016).
  * Portenoy et al. (2022) Jason Portenoy, Marissa Radensky, Jevin D West, Eric Horvitz, Daniel S Weld, and Tom Hope. 2022\.  Bursting scientific filter bubbles: Boosting innovation via novel author discovery. In _Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems_. 1–13.
  * Rachatasumrit et al. (2022) Napol Rachatasumrit, Jonathan Bragg, Amy X. Zhang, and Daniel S. Weld. 2022.  CiteRead: Integrating Localized Citation Contexts into Scientific Paper Reading.  _27th International Conference on Intelligent User Interfaces_ (2022).
  * Raffel et al. (2019) Colin Raffel, Noam M. Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2019.  Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer.  _ArXiv_ abs/1910.10683 (2019).
  * Russell et al. (1993) Daniel M. Russell, Mark J. Stefik, Peter Pirolli, and Stuart K. Card. 1993.  The Cost Structure of Sensemaking. In _Proceedings of the INTERACT ’93 and CHI ’93 Conference on Human Factors in Computing Systems_ (Amsterdam, The Netherlands) _(CHI ’93)_. Association for Computing Machinery, New York, NY, USA, 269–276.  https://doi.org/10.1145/169059.169209
  * Shahaf et al. (2012) Dafna Shahaf, Carlos Guestrin, and Eric Horvitz. 2012\.  Metro maps of science. In _Proceedings of the 18th ACM SIGKDD international conference on Knowledge discovery and data mining_. 1122–1130.
  * Shen et al. (2022) Zejiang Shen, Kyle Lo, Lucy Lu Wang, Bailey Kuehl, Daniel S. Weld, and Doug Downey. 2022\.  VILA: Improving Structured Content Extraction from Scientific PDFs Using Visual Layout Groups.  _Transactions of the Association for Computational Linguistics_ 10 (2022), 376–392.  https://doi.org/10.1162/tacl_a_00466
  * Shneiderman (2022) Ben Shneiderman. 2022\.  _Human-centered AI_.  Oxford University Press.
  * Subramonyam et al. (2020) Hariharan Subramonyam, Colleen Seifert, Priti Shah, and Eytan Adar. 2020.  TexSketch: Active Diagramming through Pen-and-Ink Annotations. In _Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems_ (Honolulu, HI, USA) _(CHI ’20)_. Association for Computing Machinery, New York, NY, USA, 1–13.  https://doi.org/10.1145/3313831.3376155
  * Sugiyama and Kan (2010) Kazunari Sugiyama and Min-Yen Kan. 2010.  Scholarly paper recommendation via user’s recent research interests. In _JCDL ’10_.
  * Szpiro et al. (2016) Sarit Felicia Anais Szpiro, Shafeka Hashash, Yuhang Zhao, and Shiri Azenkot. 2016.  How People with Low Vision Access Computing Devices: Understanding Challenges and Opportunities.  _Proceedings of the 18th International ACM SIGACCESS Conference on Computers and Accessibility_ (2016).
  * team (tion) Semantic Reader team. Unpublished demo application.  Paper to HTML.  https://papertohtml.org
  * team (sion) Semantic Reader team. Unpublished demo application; in submission..  Papeo.  https://papeo.app
  * Truong et al. (2021) Anh Tuan Truong, Peggy Chi, D. Salesin, Irfan Essa, and Maneesh Agrawala. 2021.  Automatic Generation of Two-Level Hierarchical Tutorials from Instructional Makeup Videos.  _Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems_ (2021).
  * Van Noorden et al. (2015) Richard Van Noorden et al. 2015\.  Interdisciplinary research by the numbers.  _Nature_ 525, 7569 (2015), 306–307.
  * Wadden et al. (2020) David Wadden, Shanchuan Lin, Kyle Lo, Lucy Lu Wang, Madeleine van Zuylen, Arman Cohan, and Hannaneh Hajishirzi. 2020.  Fact or Fiction: Verifying Scientific Claims. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_. Association for Computational Linguistics, Online, 7534–7550.  https://doi.org/10.18653/v1/2020.emnlp-main.609
  * Wang and Blei (2011) Chong Wang and David M. Blei. 2011.  Collaborative topic modeling for recommending scientific articles. In _KDD_.
  * Wang et al. (2021b) Lucy Lu Wang, Isabel Cachola, Jonathan Bragg, Evie Yu-Yen Cheng, Chelsea Haupt, Matt Latzke, Bailey Kuehl, Madeleine N van Zuylen, Linda Wagner, and Daniel Weld. 2021b.  SciA11y: Converting Scientific Papers to Accessible HTML. In _Proceedings of the 23rd International ACM SIGACCESS Conference on Computers and Accessibility_ (Virtual Event, USA) _(ASSETS ’21)_. Association for Computing Machinery, New York, NY, USA, Article 85, 4 pages.  https://doi.org/10.1145/3441852.3476545
  * Wang et al. (2021c) Lucy Lu Wang, Isabel Cachola, Jonathan Bragg, Evie Yu-Yen Cheng, Chelsea Hess Haupt, Matt Latzke, Bailey Kuehl, Madeleine van Zuylen, Linda Wagner, and Daniel S. Weld. 2021c.  Improving the Accessibility of Scientific Documents: Current State, User Needs, and a System Solution to Enhance Scientific PDF Accessibility for Blind and Low Vision Users.  _arXiv: 2105.00076 [cs.DL]_ (2021).  arXiv:2105.00076 [cs.DL]
  * Wang et al. (2021a) Wenhui Wang, Hangbo Bao, Shaohan Huang, Li Dong, and Furu Wei. 2021a.  MiniLMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained Transformers. In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_. Association for Computational Linguistics, Online, 2140–2151.  https://doi.org/10.18653/v1/2021.findings-acl.188
  * Wang et al. (2020) Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. 2020.  MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers. In _Advances in Neural Information Processing Systems_ , H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (Eds.), Vol. 33. Curran Associates, Inc., 5776–5788.  https://proceedings.neurips.cc/paper_files/paper/2020/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf
  * Way et al. (2019) Samuel F Way, Allison C Morgan, Daniel B Larremore, and Aaron Clauset. 2019.  Productivity, prominence, and the effects of academic environment.  _Proceedings of the National Academy of Sciences_ 116, 22 (2019), 10729–10733.
  * Wecker et al. (2014) Alan J. Wecker, Joel Lanir, Osnat Mokryn, Einat Minkov, and Tsvi Kuflik. 2014.  Semantize: visualizing the sentiment of individual document. In _Proceedings of the 2014 International Working Conference on Advanced Visual Interfaces_. Association for Computing Machinery, Como, Italy, 385–386.
  * Xia et al. (2016) Feng Xia, Haifeng Liu, Ivan Lee, and Longbing Cao. 2016\.  Scientific Article Recommendation: Exploiting Common Author Relations and Historical Preferences.  _IEEE Transactions on Big Data_ 2 (2016), 101–G112.
  * Xu et al. (2019) Yiheng Xu, Minghao Li, Lei Cui, Shaohan Huang, Furu Wei, and Ming Zhou. 2019\.  LayoutLM: Pre-training of Text and Layout for Document Image Understanding.  _Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_ (2019).
  * Yang et al. (2017) Qian Yang, Gerard de Melo, Yong Cheng, and Sen Wang. 2017\.  HiText: Text Reading with Dynamic Salience Marking. In _Proceedings of the 26th International Conference on World Wide Web Companion_. Association for Computing Machinery, Perth, Australia, 311–319.
  * Yoon et al. (2019) Wonjin Yoon, Jinhyuk Lee, Donghyeon Kim, Minbyul Jeong, and Jaewoo Kang. 2019.  Pre-trained Language Model for Biomedical Question Answering. In _PKDD/ECML Workshops_.
  * Zellweger et al. (1998) Polle T. Zellweger, Bay-Wei Chang, and Jock D. Mackinlay. 1998\.  Fluid Links for Informed and Incremental Link Transitions. In _Proceedings of the Conference on Hypertext and Hypermedia_. ACM, 50–57.
  * Zhao and Lee (2020) Tian Zhao and Kyusong Lee. 2020.  Talk to Papers: Bringing Neural Question Answering to Academic Search.  _ArXiv_ abs/2004.02002 (2020).
  * Zyto et al. (2012) Sacha Zyto, David Karger, Mark Ackerman, and Sanjoy Mahajan. 2012\.  Successful classroom deployment of a social document annotation system. In _Proceedings of the sigchi conference on human factors in computing systems_. 1883–1892.