# The Semantic Scholar Open Data Platform

Rodney Kinney, Chloe Anastasiades, Russell Authur , Iz Beltagy, Jonathan
Bragg, Alexandra Buraczynski, Isabel Cachola11footnotemark: 1 , Stefan Candra,
Yoganand Chandrasekhar, Arman Cohan11footnotemark: 1 , Miles
Crawford11footnotemark: 1 , Doug Downey, Jason Dunkelberger, Oren Etzioni, Rob
Evans, Sergey Feldman, Joseph Gorney, David Graham, Fangzhou Hu, Regan Huff,
Daniel King11footnotemark: 1 , Sebastian Kohlmeier11footnotemark: 1 , Bailey
Kuehl, Michael Langan, Daniel Lin, Haokun Liu11footnotemark: 1 , Kyle Lo,
Jaron Lochner, Kelsey MacMillan, Tyler Murray, Chris Newell, Smita Rao,
Shaurya Rohatgi11footnotemark: 1 , Paul Sayre, Zejiang Shen11footnotemark: 1 ,
Amanpreet Singh, Luca Soldaini, Shivashankar Subramanian11footnotemark: 1 ,
Amber Tanaka, Alex D. Wade11footnotemark: 1 , Linda Wagner11footnotemark: 1 ,
Lucy Lu Wang11footnotemark: 1 , Chris Wilhelm, Caroline Wu, Jiangjiang Yang,
Angele Zamarron, Madeleine Van Zuylen11footnotemark: 1 , Daniel S. Weld
Allen Institute for Artificial Intelligence
Seattle, Washington
<EMAIL_ADDRESS>Work done while at the Allen Institute for AI

###### Abstract

The volume of scientific output is creating an urgent need for automated tools
to help scientists keep up with developments in their field. Semantic Scholar
(S2) is an open data platform and website aimed at accelerating science by
helping scholars discover and understand scientific literature. We combine
public and proprietary data sources using state-of-the-art techniques for
scholarly PDF content extraction and automatic knowledge graph construction to
build the Semantic Scholar Academic Graph, the largest open scientific
literature graph to-date, with 200M+ papers, 80M+ authors, 550M+ paper-
authorship edges, and 2.4B+ citation edges. The graph includes advanced
semantic features such as structurally parsed text, natural language
summaries, and vector embeddings. In this paper, we describe the components of
the S2 data processing pipeline and the associated APIs offered by the
platform. We will update this living document to reflect changes as we add new
data offerings and improve existing services.

##  1 Introduction

Semantic Scholar111https://semanticscholar.org (S2), was launched in 2015 by
the Allen Institute for Artificial Intelligence222https://allenai.org (AI2) to
help scholars combat information overload and more efficiently discover and
understand the most relevant research literature. Through a growing number of
partnerships with scientific publishers and preprint services, Semantic
Scholar has built a comprehensive and open corpus of scientific publications
as a public service. While the Semantic Scholar website provides many
features, such as automatically-generated author pages, personalized
libraries, and paper recommendations, most of the site’s data and
functionality is also available via data download, open-source libraries, and
API services.

In this paper, we overview the primary technology used to build the corpus,
and the API services and downloads we provide to access it. We hope that the
resources described in this article can further accelerate a variety of work
that depends critically on high-quality scholarly data. Research into
scientific NLP and science of science benefit directly from such resources.
More generally, we are enabling the development of new applications that help
scientists discover and understand the literature of their field. The need for
timely and comprehensive scholarly data has become more imperative since the
2021 sunsetting of the Microsoft Academic Graph (MAG) Sinha et al. (2015),
which was long a standard source for scholarly data in the community.

The remainder of the paper proceeds as follows. In Section 2, we give an
overview of the data and services that comprise the platform. In Section 3, we
describe the platform’s data processing pipeline. In Section 4, we describe
our public APIs and downloadable datasets. In Section 5, we discuss related
work. In Section 6, we conclude and discuss future work.

Figure 1: Illustration of the Semantic Scholar platform

##  2 Platform Overview

The purpose of the Semantic Scholar Open Data Platform is to build and
distribute the Semantic Scholar Academic Graph, or S2AG (pronounced ”stag”).
S2AG is a disambiguated, high-quality, bibliographic knowledge graph. The
nodes of S2AG represent papers, authors, venues, and academic institutions.
The edges represent papers written by an author, papers cited by another
paper, papers published in a venue, and authors affiliated with an
institution. S2AG is built by ingesting a variety of data sources into our
data processing pipeline, and is distributed via a variety of publicly-
available APIs and datasets, as illustrated in Figure 1.

The work we describe here is an update of a previous version of the Semantic
Scholar data pipeline described in Ammar et al. (2018). The core structure of
the graph (authors, papers and their relationships) is the same, although the
components that build the knowledge graph have been refined. The entity
extraction and linking described in Ammar et al. (2018) have been deprecated
in favor of the semantic features described below.

Table 1 summarizes the overall size of S2AG as of January 2023.333We are
populating the author-affiliation links at the time of writing, so we have
given the expected eventual number. S2AG covers many natural, physical, and
social sciences. Table 2 summarizes the number of paper records in different
academic fields of study. Some papers are assigned to multiple fields, whereas
many are unclassified (“n/a”), due to lack of sufficient information.

##  3 Data Processing Pipeline

At the core of Semantic Scholar lies a sophisticated data processing pipeline
that continually ingests documents and metadata from numerous sources,
extracting full text and metadata from PDFs, normalizing and disambiguating
authors, institutions, and venues, classifying each paper’s field of study,
generating a textual summary of its key results, and more.

The pipeline builds S2AG by ingesting academic paper metadata and PDF content
from a variety of _Data Sources_. A _PDF Content Extraction_ system extracts
structured data from unstructured PDFs. The extracted content, along with
structured metadata from the input sources, is processed by a series of
_Knowledge Graph Construction_ systems that build S2AG. A set of models add
_Semantic Features_ to the graph, such as such as paper summarization and
vector embeddings.

Many components of our pipeline are available as open software or models.
Table 3.2 provides a high-level summary alongside links to publicly available
code, models and datasets, where applicable. Many of these also have an
associated published research article, to which we refer readers seeking
further details.

###  3.1 Data Sources

The pipeline has more than 50 input sources. They include non-profit
organizations such as Crossref, preprint servers such as arXiv, academic
publishers through negotiated agreements, and our own internet web crawler.
Sources may provide metadata in the JATS444https://jats.nlm.nih.gov/ format,
or a variety of proprietary formats. Data may be pushed or pulled via FTP,
fetched via an API, or downloaded in bulk via HTTP. Most sources are updated
daily. An important additional source is human-created data. Users of the web
site can claim the identity of an author in our corpus and manually curate the
papers associated with that author. Our team also makes manual corrections in
response to email requests from users. The first task of the pipeline is to
fetch the latest data from each source and parse it into a normalized format.
Sources typically provide limited information about a paper in structured
form: typically the title, author names, venue, and date, often linked to a
PDF file.

Nodes
---
paper |  205M
author |  80M
publication venue |  550k
academic institution |  ∗100k
Edges
citation |  2.4B
paper-author |  580M
paper-venue | 40M
author-affiliation |  ∗100M

$*$ = anticipated in 2023
---

Table 1: Approximate size of S2AG.

###  3.2 PDF Content Extraction

We augment the structured metadata by parsing the unstructured information in
a paper’s PDF into structured form, resulting in fine-grained information
about the complete text. A critical output of our PDF content extraction is a
structured bibliography from which we can construct the citation graph. Other
important outputs include section headers, paragraphs, figures and tables, and
inline references to figures, tables, and bibliography entries. PDFs are
fundamentally a print-format specification, and extracting structured
information from them is difficult and subject to errors. Nevertheless, they
are the de-facto representation of a paper’s official content, and we have
invested heavily in our extraction technology. There are three steps to PDF
content extraction: _Text Extraction_ , _Visual Region Annotation_ , and _Text
Span Annotation_.

_Text Extraction_ is the process of converting the set of commands that
indicate where characters should appear on the page into plain text, i.e.,
inferring word boundaries and word order. For this, we use existing open-
source toolkits, including pdfalto,555https://github.com/kermitt2/pdfalto
PDFPlumber,666https://github.com/jsvine/pdfplumber and/or
PDFMiner.777https://github.com/pdfminer/pdfminer.six

Field of Study | Count
---|---
n/a | 60.0M
Medicine | 31.8M
Biology | 20.4M
Physics | 11.6M
Engineering | 10.2M
Computer Science | 9.7M
Chemistry | 9.1M
Education | 7.4M
Materials Science | 7.4M
Environmental Science | 7.0M
Economics | 6.2M
Psychology | 6.2M
Agricultural and Food Sciences | 5.9M
Business | 5.6M
Mathematics | 3.7M
History | 3.4M
Political Science | 2.9M
Art | 2.8M
Geology | 2.6M
Sociology | 1.4M
Philosophy | 1.4M
Law | 1.1M
Linguistics | 1.1M
Geography | 350k
Table 2: Paper records in S2AG for different academic fields

It is possible to extract decent-quality content using only the plain text
from the Text Extraction step. For this, we have used
Grobid888https://github.com/kermitt2/grobid as well as our own extractor
called ScienceParse.999https://github.com/allenai/science-
parse,101010https://github.com/allenai/spv2 Our latest pipeline uses visual
recognition, described below, to improve the quality of the extraction.

The _Visual Region Annotation_ step first generates a visual image for each
page, using poppler.111111https://poppler.freedesktop.org/. Within each page,
we run an object detection model121212EfficientDet Tan et al. (2020) model
trained on PubLayNet Zhong et al. (2019) from the LayoutParser Shen et al.
(2021) library which identifies bounding boxes and labels each one with visual
layout categories such as figure, table, paragraph. The code, model, and data
are available at https://github.com/Layout-Parser/layout-parser.

_Text Span Annotation_ is the process of giving semantic labels to tokens from
the Text Extraction step. For this, we use the I-VILA model trained on the
S2-VL dataset, both introduced in Shen et al. (2022), to tag tokens with
categories such as title, author name, section header, body text, figure
caption, bibliography, etc. The code, model, and data are available at
https://github.com/allenai/VILA.

Visual Region and Text Span annotations are converted into structured data
(e.g., section headers are associated with section content, figures are
associated with their captions, and bibliography entries are decomposed into
their constituent elements) using the PaperMage library, which makes its code,
models and data available at https://github.com/allenai/papermage. The final
output of the PDF extraction pipeline is a structured data object suitable for
encoding in JSON format. It includes core metadata such as title, authors, and
abstract, as well as the full body text with detailed structural information
about the text.

Task | Model | Datasets | GitHub
---|---|---|---
| Visual Region
---
Annotation
| EfficientDet Tan et al. (2020) via
---
LayoutParser Shen et al. (2021)
| PubLayNet
---
Zhong et al. (2019)
Layout-Parser/layout-parser
[0.5pt/0.5pt]  | Text Span
---
Annotation
| I-VILA
---
Shen et al. (2022)
| S2-VL
---
Shen et al. (2022)
allenai/VILA
[0.5pt/0.5pt]  | Paper
---
Deduplication
S2APLER | S2APLER | allenai/S2APLER
[0.5pt/0.5pt]  | Author
---
Disambiguation
| S2AND
---
Subramanian et al. (2021)
| S2AND
---
Subramanian et al. (2021)
allenai/S2AND
[0.5pt/0.5pt]  | Affiliation
---
Normalization
S2AFF | S2AFF | allenai/S2AFF
[0.5pt/0.5pt]  | TLDR
---
Summarization
| BART Lewis et al. (2019) with
---
CATTS Cachola et al. (2020)
| SciTLDR
---
Cachola et al. (2020)
allenai/SciTLDR
[0.5pt/0.5pt]  | Citation Intent
---
Classification
Cohan et al. (2019) |  | SciCite
---
Cohan et al. (2019)
allenai/SciCite
[0.5pt/0.5pt]  | Field of Study
---
Classification
S2FOS | S2FOS | allenai/s2_fos
[0.5pt/0.5pt]  | Influential Citation
---
Classification
| Valenzuela et al. (2015)
---
meaningful-citations | -
[0.5pt/0.5pt]  | Paper
---
Embedding
| SPECTER
---
Cohan et al. (2020)
| SciDocs Cohan et al. (2020) &
---
SciRepEval Singh et al. (2022)
| allenai/SPECTER
---
allenai/SciRepEval
Table 3:  Selected models and datasets used by the pipeline.

###  3.3 Knowledge Graph Construction

The output of the PDF processing above, like the structured data in our input
sources, consists of plain-string names for real world entities. Those
entities include authors, the institutions with which those authors are
affiliated, the venues in which the papers were published, and of course, the
papers themselves. To build S2AG, we must assign an ID to each real-world
entity and associate each plain-string name with the appropriate ID.

_Paper Deduplication_ is necessary because we process data from many
independent sources. Paper titles are not unique, and can vary in how they are
expressed. Authors can release updated versions of a paper with a slightly
modified, or almost completely different, title. Our latest paper
deduplication model is named S2APLER. It works by grouping papers into blocks
using title, such that papers with similar but non-identical titles end up in
the same block, then scoring the pairwise similarity of papers in each block
with a trained model. The model uses string-similarity features for title,
abstract, author names, venue name, etc. The synthetic training dataset uses
paper data from authoritative sources that provide either a PDF or a
DOI.131313https://www.doi.org Paper pairs with matching DOI/PDFs from
different sources are used as positive training examples. Pairs with similar
titles but non-matching DOI/PDFs are used as negative examples. We make
S2APLER available at https://github.com/allenai/S2APLER.

_Citation Linking_ is the system for finding references to one paper in
another paper’s bibliography. We also use the Text Span Annotation output to
associate each citation link with the text of the sentence containing the
citation. Citation Linking is a very similar problem to paper deduplication,
except that instead of scoring the similarity between two papers, we score the
similarity between a paper and a bibliography entry produced by the PDF
Extraction system. We are using fuzzy text-matching heuristics on title and
authors for citation linking, but anticipate that the S2APLER model can
eventually be adapted to this problem.

For _Publication Venue Normalization_ , we combine data from
Fatcat141414https://fatcat.wiki/ and MAG to build a comprehensive set of
normalized venues. To match normalized venues, we index all known variant
titles for the venue, including ISO-4
normalization,151515https://en.wikipedia.org/wiki/ISO_4 in a direct lookup
table. We apply regular-expression-based rules to the unnormalized venue
strings we obtain from extracted or input sources and look for exact matches
in the knowledge base.

In _Author Disambiguation_ , we use a system named S2AND, introduced in
Subramanian et al. (2021), to assign an ID to each author mention (a name of
an author appearing in a particular paper). S2AND operates in three stages:
(1) Grouping author mentions into candidate blocks, (2) Scoring similarity
between records within a block using a LightGBM model Ke et al. (2017), and
(3) Clustering mentions within a block. The similarity scoring is a trained on
a large dataset for author disambiguation also introduced in Subramanian et
al. (2021). The code, model and dataset for S2AND is available at
https://github.com/allenai/S2AND.

For _Author Affiliation Normalization_ , we link to ROR,161616https://ror.org
a registry of persistent identifiers for research organizations. Our linking
model is named S2AFF. It first parses unnormalized affiliation strings with a
trained NER model into main institute, child institute, and address
components. It then fetches the top 100 candidates from a Jaccard-overlap
retrieval index and ranks them using a pairwise LightGBM model Ke et al.
(2017) trained using internally-gathered human annotations. We make the code
and data for S2AFF available at https://github.com/allenai/S2AFF. We expect
the API to include normalized affiliation links in 2023.

###  3.4 Semantic Features

We now turn to describing the models that provide semantic features on top of
the knowledge graph.

####  3.4.1 TLDR Summarization

To facilitate faster understanding and decision making when scanning lists of
papers, we distribute short summaries of scientific papers, or TLDRs, as
introduced in Cachola et al. (2020).

Generating TLDRs of scientific papers can be a challenging task that involves
high source compression and requires domain-specific expertise. We use a BART
Lewis et al. (2019) model trained with CATTS Cachola et al. (2020), training
on paper titles as a scaffolding task to overcome the problem of limited
annotated training data. We trained this model on a combined dataset
consisting of examples from SciTLDR Cachola et al. (2020) and a separately
collected set of summaries for biomedical papers from the Semantic Scholar
corpus. The code, model, and data are available at
https://github.com/allenai/SciTLDR.

####  3.4.2 Citation Intent and Influence Classification

Citations play a critical role in scientific papers, and understanding the
intent of a citation is helpful for automated analysis of scholarly
literature. We use the model and dataset, called SciCite, introduced in Cohan
et al. (2019) to classify each citation into one of three categories:
_background information_ , _use of methods_ , or _comparing results_. The
code, model, and data for SciCite are available at
https://github.com/allenai/SciCite.

We also classify whether each citation is _Highly Influential_. Based on the
dataset and findings from Valenzuela et al.
(2015),171717https://allenai.org/data/meaningful-citations we use a feature-
based heuristic: (1) Only citations between papers with no overlapping authors
are considered eligible, (2) A citation is classified as Highly Influential if
it appears at least three times in a sentence in which no other papers are
cited, if the citing sentence contains terms such as ”build upon” ”following”
or ”inspired by”, or if the citing sentence has references to tables or
figures (indicating a direct comparison with the cited work).

####  3.4.3 Fields-of-Study Classification

Prior to the discontinuation of MAG, Semantic Scholar made use of the fields-
of-study classifications that MAG provided, using their level 0 taxonomy.
After MAG’s deprecation, we deployed our own classification model, adding
Education, Law, and Linguistics to the existing MAG list. These additions were
based on user feedback and comparison to other popular academic data sources
such as Dimensions.

We trained our own fields-of-study classifier, named
S2FOS,181818https://blog.allenai.org/9d2f641949e5 using a multilabel linear
SVM using character n-gram TF-IDF representations (the 300k most common
character unigrams to 5-grams). For training data, we manually labeled a
number of publication venues, and then propagated those labels to all papers
published in their respective venues. The code, model, and data are available
at https://github.com/allenai/s2_fos

####  3.4.4 Paper Embeddings

Vector representations (embeddings) of papers can be useful in a variety of
downstream applications. Our pipeline uses them for author disambiguation and
recommendations, and we publish embeddings so that others may use them in
their own applications. We produce embeddings using SPECTER Cohan et al.
(2020), which generates document-level embeddings from SciBERT Beltagy et al.
(2019). SPECTER takes the paper title and abstract as input, and is trained to
minimize a triplet margin loss that encourages paper pairs with a citation
relationship to have more similar embeddings than those without. We evaluated
on the SciDocs benchmark, also introduced in Cohan et al. (2020), as well as a
newer benchmark SciRepEval Singh et al. (2022), which increases the number and
difficulty of tasks. The code, models, and data for SPECTER are available at
https://github.com/allenai/SPECTER and for SciRepEval at
https://github.com/allenai/SciRepEval.

####  3.4.5 Recent Paper Recommendations

We dynamically train recommendation models to surface relevant new papers to
users. Our recommender takes a set of positively- and negatively-annotated
papers, and outputs a ranked list of recommended papers. The model generates
recommendations in three steps: Ranker Training, Candidate Selection, and
Candidate Ranking.

Ranker Training is an extension of Arxiv Sanity.191919https://arxiv-sanity-
lite.com/,202020https://github.com/karpathy/arxiv-sanity-lite From each user’s
positive/negative annotations, it trains two linear Support Vector Machine
models: one from the TF-IDF representations of the annotated papers and one
from the SPECTER embeddings. We augment negative user annotations with
randomly selected negative examples, the latter having less weight. During
Candidate Selection, we use
FAISS212121https://github.com/facebookresearch/faiss to search an approximate
k-nearest neighbor index of the SPECTER embeddings of $\sim$1M papers
published in the last 60 days (refreshed nightly). Finally, for Candidate
Ranking we select $\sim$500 papers nearest the centroid of the positively-
annotated papers and rank them using the average of the two model scores.

##  4 APIs and Datasets

The outputs of our data processing pipeline and semantic models are made
available through a suite of APIs and datasets described below. Because we
develop and refine our models over time, the data served by the APIs may
shift, or it may come from a mixture of models as we migrate from one system
to its successor. Where appropriate, we will update our live
documentation222222https://api.semanticscholar.org/api-docs to reflect any
changes.

For unauthenticated users, we offer a low volume of API requests and samples
of the datasets. For full datasets and high request volumes, we ask that users
obtain an authentication key, at no charge, subject to terms of
use.232323https://www.semanticscholar.org/product/api##Partner-Form To date,
we have issued over 700 authentication keys to various partners, for uses
varying from student research projects to non-profit organizations to
commercial products, serving close to 150 million requests in December 2022.

Field of Study | Count
---|---
Medicine | 2.9M
Biology | 2.2M
Physics | 1.2M
Computer Science | 810k
Mathematics | 580k
Psychology | 540k
Chemistry | 430k
Materials Science | 400k
Environmental Science | 400k
Engineering | 390k
Agricultural And Food Sciences | 380k
Education | 260k
Business | 230k
Economics | 210k
Political Science | 150k
Geology | 110k
Art | 50k
Sociology | 50k
History | 40k
Linguistics | 30k
Philosophy | 30k
Law | 20k
Geography | 20k
Table 4: Full-text availability of papers in S2ORC for different academic
fields

###  4.1 Graph

The Graph API242424https://api.semanticscholar.org/api-docs/graph provides the
most current data from the Semantic Scholar Academic Graph. Papers can be
retrieved by our internal ID, or by using identifiers from arXiv, PubMed, DOI,
and others. Papers can also be retrieved via their bidirectional citation
relationship to other papers, or by author. Keyword search, with some
filtering options, is available for both papers and authors. We place size
restrictions on the number of search results and encourage users to use the
bulk snapshots when retrieving large volumes of data.

###  4.2 Datasets

Download links to monthly snapshots of our knowledge graph can be obtained via
our Datasets API.252525https://api.semanticscholar.org/api-docs/datasets Each
dataset is a collection of gzipped JSON files, where records in one dataset
refer to records in other datasets by ID. The datasets are:

  * •

papers: Core metadata of papers

  * •

abstracts: Abstract text for papers, where allowed by licensing

  * •

authors: Core metadata of authors

  * •

citations: Citation links between papers, with citation context and intent and
influential classifications

  * •

embeddings: SPECTER embeddings of papers

  * •

paper-ids: Mapping between different IDs used to identify a paper. Useful for
tracking deduplication between releases.

  * •

tldrs: TLDRs for papers

  * •

publication-venues: Core metadata for publication venues

  * •

S2ORC: Introduced in Lo et al. (2020), S2ORC is the largest publicly-available
collection of full text for open-access scientific papers. S2ORC’s full text
is annotated with automatically-identified structural and semantic elements of
the paper: section headings, paragraphs, bibliography entries, inline citation
mentions, table/figure references, etc. Table 4 shows the number of open-
access full-text papers broken down by academic field. Since its original
release as a static collection, S2ORC has grown in size and is being kept up-
to-date as part of our PDF processing pipeline. For further details on S2ORC,
we refer the reader to Lo et al. (2020) and documentation at
https://github.com/allenai/s2orc.

###  4.3 Recommendations

The Recommendations API262626https://api.semanticscholar.org/api-
docs/recommendations generates recommendations, selected from papers published
within the past 60 days, based on positive/negative paper annotations. The
caller provides at least one paper ID as a positive example, and any number of
paper IDs as negative examples. The response is a relevance-ordered list of
recently-published papers and their metadata.

Resource | URL | Article Count | Access | Services
---|---|---|---|---
Aminer | aminer.org | 321.5M | open | D∗
[0.5pt/0.5pt] arXiv | arxiv.org | 2M | open | D∗∗,F,S
[0.5pt/0.5pt] BASE | base-search.net | 180.5M | open | S
[0.5pt/0.5pt] CORE | core.ac.uk | 207.3M | open | D∗, S
[0.5pt/0.5pt] Dimensions | app.dimensions.ai | 123.8M | subscription | D, F, M, S
[0.5pt/0.5pt] Google Scholar | scholar.google.com | ? | - | -
[0.5pt/0.5pt] The Lens | lens.org | 240.4M | subscription | D, M, S
[0.5pt/0.5pt] Meta | - | - | terminated 3/31/22 | -
[0.5pt/0.5pt] Microsoft Academic | - | - | terminated 12/31/21 | -
[0.5pt/0.5pt] OpenAlex | openalex.org | 205.2M | open | D, F, M
[0.5pt/0.5pt] PubMed Central | ncbi.nlm.nih.gov/pmc/ | 7.5M | open | D∗∗,F,P,S
[0.5pt/0.5pt] ResearchGate | researchgate.net | 135.0M | - | -
[0.5pt/0.5pt] Scopus | scopus.com | 84.0M | subscription | F, M, S
[0.5pt/0.5pt] Semantic Scholar | semanticscholar.org | 205M | open | D, F, M, P, S, T
[0.5pt/0.5pt] Web of Science Core | webofknowledge.com | 83.2M | subscription | F, M, S
Key: | D=data download; F=field-of-study classification; M=advanced metadata;
---|---
| P=semantically parsed text; S=title and abstract search; T=natural language
summarization
| *=data more than a year stale; **=restricted fields of study
| Article count does not include patents or datasets.
Table 5: Comparison of leading scholarly data providers

###  4.4 Peer Review

The Peer Review API272727https://api.semanticscholar.org/api-docs/peer-review
supports the process of peer reviewer matching and conflict of interest
detection.282828https://blog.allenai.org/24ab9fce2324 Users, such as journal
editors or conference organizers, can upload information about potential
reviewers (their Semantic Scholar author ID) and paper submissions (title,
abstract, and list of authors, including Semantic Scholar author ID). The Peer
Review API returns conflict-of-interest (COI) and matching scores for all
reviewer-submission pairs. The COI score is a binary indicator saying whether
the reviewer has co-authored with any of the submission’s authors in the past.
The reviewer match score is as the average SPECTER distance between submission
and the three most similar papers written by the reviewer.

##  5 Related Work

Table 4.3 summarizes major providers of scholarly data along three key
dimensions: comprehensiveness, access, and services offered. Some providers,
such as Google Scholar, do not offer any programmatic services at all. Others,
such as MAG, have been discontinued. The major open provider of parsed
content, PubMed Central, is not cross-disciplinary. Other providers require a
subscription. Semantic Scholar is unique in providing a comprehensive and open
knowledge base with the widest array of services.

##  6 Conclusion and Future Work

We have described the Semantic Scholar data platform, which offers code bases,
data sets, and APIs covering scientific literature. The Semantic Scholar
Academic Graph (S2AG) consists of hundreds of millions of papers and billions
of citation links, created by a state-of-the-art PDF extraction and knowledge
graph normalization pipeline described in this paper. The platform also offers
semantic features such as summarization, vector embeddings and
recommendations.

In the future, we hope to expose selected semantic features as services, for
users to apply to their own data. We hope to add richer semantic labels to our
full-text annotations. We plan to add more personalized functionality, such as
access to library content and reading history. We will expand our tools for
collecting human data corrections, and possibly collect automated annotations
from external collaborators. Of course, we will continue to improve our
existing knowledge graph construction and semantic feature models. We hope
that providing these resources will enable application development and
research using scholarly data to promote the advancement of science globally.

## Acknowledgements

The Semantic Scholar Open Data Platform, including S2AG and various dataset
and API offerings, is the product of years of work by members of the Semantic
Scholar team. The authors of this paper have all contributed directly to the
creation and continued maintenance of this platform, including software and
model development, data curation, evaluation, design, product management, and
more.

This work was supported in part by NSF Grant CNS-2213656.

## References

  * (1)
  * Ammar et al. (2018) Waleed Ammar, Dirk Groeneveld, Chandra Bhagavatula, Iz Beltagy, Miles Crawford, Doug Downey, Jason Dunkelberger, Ahmed Elgohary, Sergey Feldman, Vu A. Ha, Rodney Michael Kinney, Sebastian Kohlmeier, Kyle Lo, Tyler C. Murray, Hsu-Han Ooi, Matthew E. Peters, Joanna L. Power, Sam Skjonsberg, Lucy Lu Wang, Christopher Wilhelm, Zheng Yuan, Madeleine van Zuylen, and Oren Etzioni. 2018.  Construction of the Literature Graph in Semantic Scholar. In _NAACL_.  https://doi.org/10.18653/v1/N18-3011
  * Beltagy et al. (2019) Iz Beltagy, Kyle Lo, and Arman Cohan. 2019.  SciBERT: A Pretrained Language Model for Scientific Text. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_. Association for Computational Linguistics, Hong Kong, China, 3615–3620.  https://doi.org/10.18653/v1/D19-1371
  * Cachola et al. (2020) Isabel Cachola, Kyle Lo, Arman Cohan, and Daniel S. Weld. 2020\.  TLDR: Extreme Summarization of Scientific Documents. In _FINDINGS_.  https://doi.org/10.18653/v1/2020.findings-emnlp.428
  * Cohan et al. (2019) Arman Cohan, Waleed Ammar, Madeleine van Zuylen, and Field Cady. 2019.  Structural Scaffolds for Citation Intent Classification in Scientific Publications.  _NAACL_ (2019).
  * Cohan et al. (2020) Arman Cohan, Sergey Feldman, Iz Beltagy, Doug Downey, and Daniel S. Weld. 2020.  SPECTER: Document-level Representation Learning using Citation-informed Transformers.  _arXiv_ 2004.07180 (2020).  https://doi.org/10.18653/v1/2020.acl-main.207
  * Ke et al. (2017) Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye, and Tie-Yan Liu. 2017\.  LightGBM: A Highly Efficient Gradient Boosting Decision Tree. In _NIPS_.
  * Lewis et al. (2019) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019\.  Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  _arXiv preprint arXiv:1910.13461_ (2019).
  * Lo et al. (2020) Kyle Lo, Lucy Lu Wang, Mark Neumann, Rodney Michael Kinney, and Daniel S. Weld. 2020.  S2ORC: The Semantic Scholar Open Research Corpus. In _ACL_.  https://doi.org/10.18653/V1/2020.ACL-MAIN.447
  * Shen et al. (2022) Zejiang Shen, Kyle Lo, Lucy Lu Wang, Bailey Kuehl, Daniel S Weld, and Doug Downey. 2022\.  VILA: Improving structured content extraction from scientific PDFs using visual layout groups.  _Transactions of the Association for Computational Linguistics_ 10 (2022), 376–392.
  * Shen et al. (2021) Zejiang Shen, Ruochen Zhang, Melissa Dell, Benjamin Charles Germain Lee, Jacob Carlson, and Weining Li. 2021.  LayoutParser: A unified toolkit for deep learning based document image analysis. In _International Conference on Document Analysis and Recognition_. Springer, 131–146.
  * Singh et al. (2022) Amanpreet Singh, Mike D’Arcy, Arman Cohan, Doug Downey, and Sergey Feldman. 2022.  SciRepEval: A Multi-Format Benchmark for Scientific Document Representations.  https://doi.org/10.48550/ARXIV.2211.13308
  * Sinha et al. (2015) Arnab Sinha, Zhihong Shen, Yang Song, Hao Ma, Darrin Eide, Bo-June Paul Hsu, and Kuansan Wang. 2015.  An Overview of Microsoft Academic Service (MAS) and Applications.  _Proceedings of the 24th International Conference on World Wide Web_ (2015).  https://doi.org/10.1145/2740908.2742839
  * Subramanian et al. (2021) Shivashankar Subramanian, Daniel King, Doug Downey, and Sergey Feldman. 2021.  S2AND: A Benchmark and Evaluation System for Author Name Disambiguation.  _ArXiv_ abs/2103.07534 (2021).
  * Tan et al. (2020) Mingxing Tan, Ruoming Pang, and Quoc V Le. 2020.  Efficientdet: Scalable and efficient object detection. In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_. 10781–10790.
  * Valenzuela et al. (2015) Marco Valenzuela, Vu A. Ha, and Oren Etzioni. 2015.  Identifying Meaningful Citations. In _AAAI Workshop: Scholarly Big Data_.
  * Zhong et al. (2019) Xu Zhong, Jianbin Tang, and Antonio Jimeno Yepes. 2019\.  PubLayNet: largest dataset ever for document layout analysis. In _2019 International Conference on Document Analysis and Recognition (ICDAR)_. IEEE, 1015–1022.  https://doi.org/10.1109/ICDAR.2019.00166


# Demo: Real-Time Semantic Communications with a Vision Transformer

Hanju Yoo,1 Taehun Jung,1 Linglong Dai,2 Songkuk Kim,1 and Chan-Byoung Chae1
1School of Integrated Technology, Yonsei University, Korea
2Department of Electronic Engineering, Tsinghua University, China
{hanju.yoo, taehun.jung, songkuk<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Semantic communications are expected to enable the more effective delivery of
meaning rather than a precise transfer of symbols. In this paper, we propose
an end-to-end deep neural network-based architecture for image transmission
and demonstrate its feasibility in a real-time wireless channel by
implementing a prototype based on a field-programmable gate array (FPGA). We
demonstrate that this system outperforms the traditional 256-quadrature
amplitude modulation system in the low signal-to-noise ratio regime with the
popular CIFAR-10 dataset. To the best of our knowledge, this is the first work
that implements and investigates real-time semantic communications with a
vision transformer.

###### Index Terms:

Semantic communications, image transmission, real-time wireless
communications, deep neural network

##  I Introduction

With the emergence of immersive contents, such as extended reality (XR),
communication systems must simultaneously achieve high data rates and low
latencies. These contradictory requirements have led to ongoing races for
wider bandwidths and higher costs. As a result of the rapid advances in
computing power and more data, researchers have been attempting to improve the
spectral efficiency by introducing deep neural networks (DNNs) to
communications to enable the faster and more accurate transmission of symbols
with limited bandwidths.

What matters, however, is the effective delivery of meaning to achieve goals,
not the bits. Some early research regarding semantic communications has been
conducted in the scope of text [1] or speech transmission [2]. Nevertheless,
none of the research has addressed the image transmission problem, which is
the primary source of human information perception and accounts for most
traffic usage on the internet. Furthermore, their results were validated only
in simulated environments and did not equalize the system data rate for a fair
comparison.

In this paper, we introduce an end-to-end DNN-based communication system for
images and implement a real-time wireless image transmission platform based on
a field-programmable gate array to demonstrate the feasibility of the system
on actual channel conditions. We compare the proposed method to the
traditional 256-quadrature amplitude modulation (QAM) system to demonstrate
its superiority in the low signal-to-noise (SNR) regime.

Figure 1: (a) System setup and (b) proposed deep neural network (DNN) system
architecture. Figure 2: (a) Transmitted images and (b) structural similarity
index measure (SSIM) results of the proposed and baseline systems.

##  II System Setup

###  II-A System Architecture

As presented in Figure 1, the system consists of three blocks: an image
encoder, channel layer, and image reconstruction block (decoder). The encoder
and decoder are trainable neural networks whereas the channel layer is either
a simulated (for training) or physical wireless channel based on a software-
defined radio (for demonstration).

Image encoder. The encoder network $F_{\text{encode}}$ consists of four
consecutive $3\times 3$ convolutional layers that follow the prominent
bottleneck residual block [3] design, known for good early visual processing
and feature extraction performance. Given an image sequence
$s\in\mathbb{R}^{H\times W\times C}$ (where $H,W$, and $C$ are the image
height, width, and channel, respectively), the encoder network
$F_{\text{encode}}$ transforms $s$ into symbols $x\in\mathbb{R}^{H\times
W\times C}$, denoted as follows:

$x=F_{\text{encode}}(s).$ (1)

Channel layer. The channel layer conducts two main tasks: power normalization
and physical channel simulation. In this block, the symbol $x$ is divided by
the power normalizer coefficient $c$ to satisfy the power constraint
$\mathop{\mathbb{E}}\|x\|^{2}=1.$ Then, the normalized symbol $\bar{x}=x/c$ is
transmitted via a communication link, which can be modeled as follows:

$y=h\bar{x}+n,$ (2)

with a channel coefficient $h$ and an additive Gaussian noise
$n\sim\mathcal{N}(0,\,\sigma^{2})$ with noise variance $\sigma^{2}$.

Image reconstruction block. The transmitted symbol $y$ is transformed to
produce the naively decoded symbol ${\hat{x}=c\cdot y/h}$. We assumed that the
perfect channel source information and the power normalization coefficient are
known. Then, the decoder network $F_{\text{decode}}$ reconstructs the image
sequence $\hat{s}$ from $\hat{x}$. The decoder network consists of a vision
transformer (ViT) [4] followed by denoising autoencoder (DAE) [5] layers.
Vision transformer is known for useful properties such as resilience to
typical image occlusions, and DAE effectively denoises images by extracting
critical features from the corrupted image. A shortcut between two layers is
introduced to facilitate the recovery of detailed textures, which may be lost
on denoising. The overall decoder operation can be expressed by the following:

$x_{0}=F_{\text{vit}}(\hat{x}),$ (3) $\hat{s}=Wx_{0}+F_{\text{dae}}(x_{0}),$
(4)

where $\hat{s}$ denotes the reconstructed image sequence, and $W$ is a
trainable matrix. Moreover, $F_{\text{vit}}$ and $F_{\text{dae}}$ are the ViT
and DAE blocks, respectively.

###  II-B Experiments

We trained the system with a CIFAR-10 image dataset containing 50,000 training
and 10,000 test images. The total parameter count of the system was 0.76M. We
trained the network for 150 epochs with a Rayleigh channel at 35 dB SNR and
with the pixel-wise mean squared error loss. Then we fine-tuned the system for
another 150 epochs at 15 dB with the mean absolute error loss. The Adam
optimizer with $\ell$$r=0.002$ was adopted for parameter optimization.

We evaluated the model on two channel models, additive white Gaussian noise
and Rayleigh, with a SNR ranging from 0 to 40 dB. A 256-QAM system with
average symbol power normalization was selected for the baseline, as the
proposed system sends one symbol per image pixel (which can have 256
levels).111The 256-QAM system does not have sophisticated coding techniques.
Future work will consider various source and/or channel coding schemes. We
used the structural similarity index (SSIM) [6] to evaluate the system
performance.

###  II-C Results

Figure 2 illustrates the transmitted images and their SSIM scores for the
proposed and baseline 256-QAM systems. For Figure 2(a), we assumed a Rayleigh
channel and sent random images to produce the results. For Figure 2(b), we
randomly selected 320 testing images and computed the average SSIM scores. The
proposed system outperforms the 256-QAM system in the low SNR regime (e.g., 0
to 20 dB), possibly due to the system utilization of inherent redundancies to
recover noisy pixels. However, as a tradeoff, the image details tend to be
lost even on high SNR regions (e.g., 40 dB), where the traditional 256-QAM
system has advantages over the neural network-based methods. At IEEE ICC, we
plan to take a photo of visitors and transmit the image through the real-time
semantic communication system with the visual transformer.

##  III Conclusion

In this paper, we proposed a DNN-based image semantic communication system and
implemented a real-time image transmission system to demonstrate its
feasibility in actual wireless environments. The proposed method exhibits
promising results compared to the traditional 256-QAM based system, especially
at low SNR levels. We expect these results to provide insight into the future
network design for 6G.

## References

  * [1] K. Lu, R. Li, X. Chen, Z. Zhao, and H. Zhang, “Reinforcement learning-powered semantic communication via semantic similarity,” _arXiv preprint arXiv:2108.12121_ , 2021.
  * [2] Z. Weng and Z. Qin, “Semantic communication systems for speech transmission,” _IEEE Journal on Selected Areas in Communications_ , 2021.
  * [3] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Proc. IEEE Conf. on Comp. Vision and Pattern Recog._ , 2016, pp. 770–778.
  * [4] A. Dosovitskiy _et al._ , “An image is worth 16x16 words: Transformers for image recognition at scale,” _arXiv preprint arXiv:2010.11929_ , 2020.
  * [5] P. Vincent, H. Larochelle, Y. Bengio, and P.-A. Manzagol, “Extracting and composing robust features with denoising autoencoders,” in _Proc. Int. Conf. on Machine Learning_ , 2008, pp. 1096–1103.
  * [6] Z. Wang, A. Bovik, H. Sheikh, and E. Simoncelli, “Image quality assessment: from error visibility to structural similarity,” _IEEE Trans. on Image Proc._ , vol. 13, no. 4, pp. 600–612, 2004.