[datatype=bibtex, overwrite] [fieldset=editor, null] [fieldset=language, null]
[fieldset=note, null]

# Representing the disciplinary structure of physics: a comparative evaluation
of graph and text embedding methods

Isabel Constantino  Sadamori Kojaku  Thomas J. Watson College of Engineering
and Applied Science, Department of Systems Science and Industrial Engineering,
Binghamton University, State University of New York  Santo Fortunato  Yong-
Yeol Ahn<EMAIL_ADDRESS>
###### Abstract

Recent advances in machine learning offer new ways to represent and study
scholarly works and the space of knowledge. Graph and text embeddings provide
a convenient vector representation of scholarly works based on citations and
text. Yet, it is unclear whether their representations are consistent or
provide different views of the structure of science. Here, we compare graph
and text embedding by testing their ability to capture the hierarchical
structure of the Physics and Astronomy Classification Scheme (PACS) of papers
published by the American Physical Society (APS). We also provide a
qualitative comparison of the overall structure of the graph and text
embeddings for reference. We find that neural network-based methods outperform
traditional methods and graph embedding methods such as node2vec are better
than other methods at capturing the PACS structure. Our results call for
further investigations into how different contexts of scientific papers are
captured by different methods, and how we can combine and leverage such
information in an interpretable manner.

##  1 Introduction

New discoveries and innovations build upon existing knowledge, often by
combining previous knowledge together
kauffmanEmergentPropertiesRandom1984weitzmanRecombinantGrowth1998steelDynamicsBirthDeath2020.
Subsequently, new knowledge can be used in many different domains in the
future. In this sense, each publication can be largely characterized by its
_references and citations_ (i.e. the knowledge upon which the publication
builds, and how future publications use its knowledge), and its _content_.
These two sources of information have thus been used to investigate how the
scientific enterprise discovers new knowledge.

Previous studies have analyzed the references of both patents and scientific
publications to understand the relationship between the ideas referenced by
inventors and scientists, and the impact of their work. A study by [15] uses
patent data to compare the effects of combining novel versus familiar ideas,
finding that new combinations and new components result in less useful or less
successful patents (i.e. patents receive fewer citations). However, this also
results in greater variability in future success. The riskiness of exploring
new or otherwise unusual combinations of ideas has also been explored in
scientific research. Papers that reference both atypical and conventional
combinations of ideas are more likely to be highly cited, as well as more
“disruptive”
uzziAtypicalCombinationsScientific2013wuLargeTeamsDevelop2019linNewDirectionsScience2022.
That is, such papers are more likely to be cited in future work than the
papers they referenced or built on. Furthermore, research citing novel
combinations of ideas are more likely to experience delayed recognition, but
are also more likely to be cited by other fields [42]. In these studies,
atypicality and novelty are measured in terms of the number of cocitations
between pairs of referenced journals. Atypicality and novelty can then be
framed as the “distance” between reference pairs: ideas are more distant if
they are less likely to be cited together, and combining distant ideas are
more likely to result in risky but high-impact research.

Previous studies have also used the keywords and titles of publications to
study the interrelatedness and integration of scientific fields over time.
[11] proposed a co-word analysis of keywords in artificial intelligence papers
by creating a keyword-keyword matrix where each entry corresponds to the
number of papers that contain both of these keywords. They then show how
clusters in this keyword co-occurrence matrix correspond to various subdomains
in AI, and how they change over time. In another study by [25], combinations
of publication keywords within the same paper are used to measure subfield
integration in sociological research. They find that papers that span multiple
domains, especially unique combinations of domains, are cited more frequently.

Science mapping integrates various efforts to quantify the “space” of
knowledge, by obtaining a spatial representation of journals, fields, and
papers, where distance and similarity can be quantified. For instance, Boyack,
Klavans, and Börner developed a map of scientific journals and disciplines,
where similar (i.e. in terms of cocitations and/or co-references) journals are
closer to one another
boyackMappingBackboneScience2005bornerDesignUpdateClassification2012boyackCreationHighlyDetailed2014.
On the other hand, the distance between referenced disciplines has been
integrated into measures of the interdisciplinarity of research, such as the
Rao-Stirling index
rafolsDiversityNetworkCoherence2010leydesdorffIndicatorsInterdisciplinarityJournals2011leydesdorffInterdisciplinarityDiversityCitation2019.
These studies in mapping science and evaluating the distance between ideas are
crucial in understanding how scientists collectively create, develop, and
combine ideas in the context of the “space” of knowledge.

The main caveat in using existing disciplinary categorizations of scientific
work, such as the Web of Science subject categories of journals, is that they
tend to be rigid and change slowly over time, even when science rapidly
develops. In an effort to overcome this limitation, previous studies have
applied methods in topic modeling to analyze trends and “hot topics” in
science. Specifically, latent Dirichlet allocation (LDA) is used to learn
representations of publications as probability distributions over scientific
topics, and representations of topics as probability distributions over words
bleiLatentDirichletAllocation2003griffithsFindingScientificTopics2004bleiProbabilisticTopicModels2012.
LDA has also been used to learn representations of authors’ body of work as
probability distributions over topics [38]. However, one challenge with LDA
and other topic modeling methods is the need to specify the number of topics
in the corpus, which is not always known, or may change over time. Moreover,
topic modeling approaches documents as “bags of words”; it ignores the order
and context of words.

By contrast, representative learning (embedding) methods take into account the
context within which words or entities appear, and it represents them into
continuous space without the need to use categorization (whether provided or
learned from the data). These methods have been shown to be capable at
meaningfully representing the space of scientific knowledge. For instance, in
a study by [9], papers published in American Physical Society (APS) journals
are represented as “bags of topics”. The embeddings are then used to map the
various specialties of physics research, and study the production of physics
research by various countries. In another study by [33], journals in the Web
of Science (WOS) are represented using their citation network, which are then
used to learn analogy graphs of journals, and visualize the “spectra” of soft
versus hard science.

Despite the increasing use of embedding methods in the study of scientific
production and innovation, relatively little attention has been paid to the
systematic comparison of two primary embedding approaches—content and graph
embedding. While previous studies
zhangP2VLargescaleAcademic2019kozlowskiSemanticRelationalSpaces2021 have
explored the use of various graph and text embedding methods to represent
scientific publications, the methods were evaluated separately. Here, we
compare popular graph and text embedding methods in terms of how well the
resulting embeddings capture the disciplinary structure of physics. We focus
on the field of physics, particularly papers published in the American
Physical Society (APS) [34], which, until recently, has used a hierarchical
classification system called the Physics and Astronomy Classification Scheme
(PACS). We then conduct a series of experiments to evaluate the embeddings’
ability to capture the hierarchical structure of PACS.

##  2 Data and Methods

###  2.1 Data

The APS provides a citation network of all articles published in its journals,
as well as papers’ metadata on authors, affiliations, titles, publication
dates, etc. We consider the citation network of all peer-reviewed scientific
papers, reviews, and letters in all APS journals published up to the year
2010. We remove articles such as announcements, comments and replies, errata,
and retractions. The resulting citation network consists of $452{,}096$ papers
and $4{,}931{,}143$ citations. We note that this includes papers published
before 1979, which do not have PACS codes. These are included in all
embeddings; however, they are not considered in the evaluation.

For the content embedding, although it would be ideal to use the full text of
papers, this is usually not available to researchers. Therefore, we consider
the titles and abstracts of the papers in the citation network. Because the
data provided by the APS does not contain abstract information, we match
entries in the APS to entries in the Web of Science using exact DOI matching.
Among the $358{,}478$ papers with exact DOI matches to the Web of Science up
to 2010, $159{,}375$ have abstracts. We limit the preprocessing of text data
to removing HTML and MathML tags, as well as removing extraneous whitespace.

###  2.2 Embedding

####  2.2.1 Graph embedding

We embed the citation network of the 452K papers by using graph embedding.
Graph embedding places one paper in a space in relation to other papers. There
are two types of relations, i.e., references (out-going citations) and
citations (in-coming citations), which provides semantically different views
of a paper. The references of a paper imply the knowledge relevant to the
content of the paper, while the citations to a paper imply how the paper is
utilized by the future papers. We incorporate both aspects of relations into
consideration by ignoring the edge directionality and producing the graph
embeddings using the undirected citation network.

A traditional family of graph embedding methods is based on matrix
factorization, with one of the most popular being Laplacian Eigenmap [2].
Laplacian Eigenmap is a spectral embedding method, which obtains an
$m$-dimensional embedding by concatenating the second to the $(m+1)$-th
largest eigenvectors of the symmetric normalized Laplacian
$L=I-D^{-\frac{1}{2}}AD^{\frac{1}{2}}$, where I is the diagonal identity
matrix, D is the diagonal degree matrix $D_{ii}=\sum_{j}A_{ij}$, and $A$ is
the graph adjacency matrix.

The second graph embedding method is node2vec [17]. node2vec is a direct
application of the skip-gram negative sampling method (SGNS) word2vec
mikolovEfficientEstimationWord2013NIPS2013_9aa42b31, a word embedding model
that produces an embedding of words from given sentences. node2vec is adapted
to network embedding by using a random walk, where each sentence is composed
of nodes visited by a random walk. node2vec allows the random walker to
preferentially visit the previous nodes, or nodes that are not adjacent to
previous nodes. This bias is controlled by hyperparameters $p$ and $q$. A
value of $p=1$ and $q=1$ is equivalent to a uniform random walk. For this
study, we use $p=1$ and $q=1$, similar to previous studies evaluating graph
embeddings guPrincipledApproachSelection2021dehghan-
kooshkghaziEvaluatingNodeEmbeddings2022liuSTABLEIdentifyingMitigating2023.

Finally, the third graph embedding method is residual2vec [21]. residual2vec,
also a random walk-based embedding method, uses the bias removal mechanism of
the negative sampling method used in node2vec to remove any prescribed bias
from the embedding. Here, we consider several structural biases present in the
citation network. First, older papers are more likely to have a higher degree
(i.e. more citations) due to the time it has to accumulate citations. Second,
papers tend to cite more recent references. To address these biases, we use
the configuration model as a null model for residual2vec. In other words, we
remove the impact of each paper’s degree from the embedding. For all graph
embeddings, we use dimension $d=128$, given that most real-world networks do
not require a large number of dimensions, and that the embedding tends not to
suffer much from overparametrization of the number of dimensions [18]. Other
parameters we use are: walk length $l=80$, walks per node $r=10$, context
window size $k=10$, and random walk restart probability $s=0.01$.

####  2.2.2 Text embedding

We use three popular text embedding methods: doc2vec [24], SciBERT [3], and
Sentence-BERT [37]. doc2vec learns document representations by training them
to predict words in the document, similar to how word2vec learns word
representations by predicting words within its context. We specifically use
the Paragraph Vector-Distributed Memory (PV-DM) model of doc2vec, where the
paragraph (or document) vector serves as a “memory” that provides a broader
context. For doc2vec, we use dimension $d=128$, window size 5, and minimum
word instance count 5.

The second text embedding method is SciBERT [3], an extension of BERT [13].
BERT is a pretrained deep neural language model that is trained using a
bidirectional transformer network, and can be fine-tuned for tasks such as
classification and question answering. SciBERT is a BERT model finetuned on a
dataset of computer science and biomedical science papers. We use the
allenai/scibert_scivocab_uncased model hosted on the Huggingface Transformers
library [45]. As BERT generates representations for word pieces or punctuation
marks (also known as tokens), some extra processing is required to obtain
representations of “sentences” or documents. The processing is as follows
[37]: (1) For each text input or document, obtain the special CLS token; (2)
for each CLS token, obtain the vectors from the last 4 hidden layers of the
neural network; and (3) sum up the 4 layers.

The last text embedding method is Sentence-BERT [37]. Sentence-BERT extends on
BERT using a Siamese network architecture to obtain sentence embeddings that
can be directly compared using measures such as cosine similarity. The
pretrained model used is sentence-transformers/paraphrase-mpnet-base-v2, also
hosted by Huggingface. As Sentence-BERT already results in an embedding for
each piece of text, no further processing is required. We note that both
SciBERT and Sentence-BERT support a maximum of 512 word tokens; any input
beyond 512 tokens is ignored.

###  2.3 Evaluation

What makes a good embedding? There are a multitude of ways in which the
parameters of each embedding method can be tuned, and many other ways to judge
“similarity” or distance” between two embedded entities. Because we are
interested in how each of the two main families of embedding methods can
encode the hierarchical disciplinary structure of physics, we evaluate the
embeddings based on how well they reflect an existing hierarchical
categorization method, the Physics and Astronomy Classification Scheme (PACS)
[35]. We summarize the embedding and evaluation framework in Figure 1.

Figure 1: Embedding and evaluation framework for APS papers.

####  2.3.1 Classification

We first evaluate the embeddings’ ability to capture the general clusters of
physics research, by running a $k$-nearest neighbor (KNN) classification task
to predict the level 1 (L1) PACS code, that is, its general physics
classification. KNN is a supervised machine learning algorithm where data is
labeled according to the labels of its $k$ nearest neighbors, as calculated by
a distance metric. We first take the citation network consisting of papers
published on or before the year 2010. For each paper, we take all full PACS
codes, and take the first digit of each PACS code to get the L1 PACS code. We
then take 80% of the papers to be used as a training set, and take note of
their L1 PACS codes. The remaining 20% of papers, or the testing set, are
labeled with the L1 PACS code of the majority of its $k$ nearest neighbors, as
measured by the cosine distance between the paper vectors in the embedding
space. For this study, we implement KNN using the Faiss library [19], which
provides a faster calculation of distance in large, high-dimensional datasets.
We repeat for the values $k\in\\{2,4,8,16,32,64,128\\}$. The resulting
classification is then evaluated using the micro-F1 score.

####  2.3.2 Evaluating hierarchical disciplinary structure

We then evaluate whether the embeddings can capture the hierarchical structure
of PACS. We do this in two ways. First, we check whether papers of more
specific subcategories are closer to one another, compared to papers of
general subcategories. Intuitively, sets of papers of a same general topic
“cover” more material than sets of papers of a more specific topic. We
consider each PACS code at each level. At the highest level, there are 10 PACS
codes (00–90) each corresponding to a general physics category such as
particle physics, nuclear physics, etc. Each succeeding level then corresponds
to a more specific subtopic. Below is an example of a six-digit PACS code:
61.30.Jf. The first level, 6, or 60, corresponds to “the structural,
mechanical, and thermal properties of condensed matter”. The second level, 61,
then represents “crystallography” and “solid or liquid structures”. The third
level, 61.30, refers to “liquid crystals”. Finally, the fourth or lowest
level, 61.30.Jf, refers to “defects in liquid crystals”. We want to check
whether each succeeding PACS level has smaller and smaller variability among
its constituent papers. We do this using a measure inspired by the radius of
gyration (ROG), or the root mean square distance of an object’s parts to its
center of mass. In our case, we measure the root mean square cosine distance
of each vector, to the centroid or average of these vectors:

$\text{ROG}_{C}=\sqrt{\frac{1}{N}\sum_{\mathbf{x}\in
C}{\Big{(}1-\cos(\mathbf{x},\mathbf{\bar{x}})\Big{)}^{2}}}$ (1)

where $C$ is the PACS code of interest, $\mathbf{x}$ a vector representation
of a paper in $C$, $\mathbf{\bar{x}}$ the centroid of all vectors in $C$, and
$\cos(\mathbf{a},\mathbf{b})$ the cosine similarity between vectors
$\mathbf{a}$ and $\mathbf{b}$. We calculate the ROG of each PACS code at each
of the 4 levels. We then get the distribution of ROG values at each level. To
confirm whether each finer level has smaller or larger ROG values, we perform
a one-sample Wilcoxon signed-rank text
wilcoxonIndividualComparisonsRanking1945conover1999practical to test the null
hypothesis that the median ROG distribution of a PACS level is greater than or
equal to the median ROG distribution of the next lower PACS level.

Second, we evaluate whether pairs of papers of the same PACS code are more
likely to be closer to one another in the embedding space, compared to papers
of different PACS codes. We sample $15{,}000$ pairs of papers in each of the
following categories: (1) random, (2) different discipline or PACS code, (3)
same discipline (L1 PACS code), and (4) same subdiscipline (L2 PACS code). We
then get the distribution of cosine distance between the pairs of papers in
each subcategory. We compare these distributions in two ways. First, we
compute the Jensen-Shannon divergence between each of the cosine distance
distributions. The Jensen-Shannon (JS) divergence is a similarity measure for
two distributions [14]. Unlike the popular KL divergence [23], it is symmetric
and has a finite range of $0$ to $1$. However, The Jensen-Shannon divergence
does not indicate direction; that is, whether the cosine distance of same-
discipline pairs is the same of that of random pairs. Hence, we also conduct
one-sample t-tests to compare the distributions. In particular, we test the
null hypothesis that the mean distance between random pairs is less than or
equal to than the mean distance between same-discipline pairs.

####  2.3.3 Link prediction

Finally, we evaluate whether the embeddings are useful for predicting links
between papers and references, specifically, whether a paper cites or
references a previous paper adamicFriendsNeighborsWeb2003liben-
nowellLinkpredictionProblemSocial2007shibataLinkPredictionCitation2012. We
first take the citation network of APS papers published up to the year 2010.
We then take the largest connected component of this citation network, then
sample 50% or approximately $149{,}000$ edges from papers published in 2010
(i.e. edges corresponding to citations coming from papers in 2010). This is
the “positive” example set. We then sampled the same number of paper pairs
(such that one of the papers in each pair is from 2010) that have no edge
between them, to serve as the “negative” examples. The “positive” examples or
edges are removed from the citation network, and the resulting network is
embedded using each of the embedding methods as described in section 2.2.
Then, for each embedding, we calculate the cosine similarity of the positive
and negative examples. In this case, positive examples, or pairs of papers
linked by citation, should have a high cosine similarity, while negative
examples should have low cosine similarity. The results are evaluated using
the ROC curve, used in binary classification: the ROC curve plots the false
positive rate against the true positive rate, denoting how the true positive
rate changes as the false positive rate threshold is increased. The area under
the ROC curve (AUC) is then calculated to summarize the results. The AUC
ranges from 0 to 1, where 1 represents a perfect classification, and an AUC of
0.5 represents a random classification.

##  3 Results

Figure 2 shows UMAP projections of each embedding, using a sample of 10,000
papers. Each point is colored according to its level 1 PACS code. Here,
Sentence-BERT—both title and abstract embeddings—node2vec, and residual2vec
provide paper-level embeddings that follow the general clustering structure of
research published in APS. The Laplacian Eigenmap embedding also somewhat
exhibits a general cluster structure, although it also presents many overlaps
among the clusters.

Figure 2: UMAP projections on a sample of the embeddings show that Sentence-
BERT,node2vec, and residual2vec all follow the general clustering structure of
physics research.

Figure 3 shows the results for the $k$-nearest neighbor classification. Among
the text embeddings, Sentence-BERT has the highest micro-F1 score of
approximately $0.75$, across multiple values of $k$. All text embeddings
performed worse than the tested graph embeddings, which all resulted in
micro-F1 scores of approximately $0.76$ to $0.81$. In Laplacian Eigenmap,
higher values of $k$ result in lower micro-F1 scores. Finally, all embeddings
except node2vec performed worse than a baseline citation network-based
classification which predicts a paper’s level 1 PACS code using a majority
vote among its references, although this may indicate a tendency for authors
to self-assign PACS codes similar to those of their references.

Figure 3: Classification of PACS by a k-nearest neighbor algorithm (a) with
$k=2$ to 128; and (b) with micro-F1 score $>0.64$ and $k\in\\{2,4,8\\}$. The
graph embedding methods outperform all text embedding methods. Among the text
embeddings, Sentence-BERT performs best, though not as well as the graph
embeddings. With doc2vec, the abstract embedding results in improved
classification performance compared to the title embedding.

Figure 4 shows the distributions of ROG at each PACS level, and Table 1 shows
the results of the one-sample Wilcoxon test for the distributions. For all
tested graph embedding methods, as well as in Sentence-BERT, the median ROG at
each level is consistently lower than the median ROG of the preceding level.
Hence, for these embeddings, the more specific the subdiscipline of physics,
the closer together its papers are in the embedding space. Meanwhile, for
doc2vec and SciBERT, the median ROG of the highest PACS level is lower than or
equal to the median ROG of the second PACS level. This may be explained by the
“scale” of these text embeddings, where all points are relatively close to one
another, thereby resulting in very small differences in ROG across the PACS
levels. By contrast, in the graph embeddings, the points are more spread out
across the embedding space, which results in more dramatic decreases in ROG as
the PACS level increases. This is especially apparent in node2vec and
residual2vec.

Figure 4: Box plots indicating the distributions of PACS code ROG show that a deeper PACS level (left to right) only results in a lower or more left-skewed ROG distribution in the Sentence-BERT, Laplacian Eigenmap, node2vec, and residual2vec embeddings. |  |  |  # of samples |  Median (current) |  Median (previous) |  Wilcoxon statistic |  p-value
---|---|---|---|---|---|---|---
Property |  Embedding method |  PACS level |  |  |  |  |
Undirected Graph |  Laplacian Eigenmap |  2 |  82 |  0.7928 |  0.8693 |  43 |  0.0000
|  |  3 |  941 |  0.6611 |  0.7928 |  17101 |  0.0000
|  |  5 |  6434 |  0.4983 |  0.6611 |  2050166 |  0.0000
|  node2vec |  2 |  82 |  0.1127 |  0.1502 |  156 |  0.0000
|  |  3 |  941 |  0.0793 |  0.1127 |  53679 |  0.0000
|  |  5 |  6434 |  0.0487 |  0.0793 |  3724942 |  0.0000
|  residual2vec |  2 |  82 |  0.3430 |  0.4419 |  382 |  0.0000
|  |  3 |  941 |  0.2381 |  0.3430 |  60733 |  0.0000
|  |  5 |  6434 |  0.1360 |  0.2381 |  3723006 |  0.0000
Title |  doc2vec |  2 |  82 |  0.1202 |  0.1210 |  1668 |  0.4385
|  |  3 |  946 |  0.1123 |  0.1202 |  167476 |  0.0000
|  |  5 |  6495 |  0.0871 |  0.1123 |  5446270 |  0.0000
|  SciBERT |  2 |  82 |  0.0050 |  0.0053 |  1339 |  0.0469
|  |  3 |  942 |  0.0045 |  0.0050 |  168165 |  0.0000
|  |  5 |  6442 |  0.0033 |  0.0045 |  6409506 |  0.0000
|  Sentence-BERT |  2 |  82 |  0.1871 |  0.2020 |  982 |  0.0004
|  |  3 |  942 |  0.1624 |  0.1871 |  76993 |  0.0000
|  |  5 |  6442 |  0.1312 |  0.1624 |  3161325 |  0.0000
Abstract |  doc2vec |  2 |  76 |  0.1693 |  0.1778 |  1059 |  0.0182
|  |  3 |  844 |  0.1477 |  0.1693 |  91863 |  0.0000
|  |  5 |  5418 |  0.1148 |  0.1477 |  3279790 |  0.0000
|  SciBERT |  2 |  76 |  0.0104 |  0.0106 |  1216 |  0.1005
|  |  3 |  844 |  0.0091 |  0.0104 |  136064 |  0.0000
|  |  5 |  5418 |  0.0045 |  0.0091 |  4877620 |  0.0000
|  Sentence-BERT |  2 |  76 |  0.1364 |  0.1494 |  883 |  0.0013
|  |  3 |  844 |  0.1151 |  0.1364 |  61858 |  0.0000
|  |  5 |  5418 |  0.0929 |  0.1151 |  2434065 |  0.0000
Table 1: One-sample Wilcoxon test results for comparing the median of each
PACS level to the previous level (rows highlighted in yellow indicate null
hypotheses that we failed to reject). This further shows that for the graph
embeddings and Sentence-BERT, the ROG decreases as the PACS level increases or
becomes more specific. However, this more clearly shows that in doc2vec and
SciBERT, the ROG is also decreasing in all but the most general PACS levels.

Figure 5 shows the distributions of sampled paper pairs, Figure 6 shows the
Jensen-Shannon (JS) distance matrices for each embedding, and Table 2 shows
the results of the t-test to compare these embeddings’ distance distributions.
In the representations generated using the graph embeddings and Sentence-BERT,
the mean cosine distance of random paper pairs is consistently greater than
that of pairs with the same L1 or L2 PACS code. In addition, the JS distances
between the distributions in the node2vec and residual2vec embeddings are
increasing. That is, the JS distance between random pairs and pairs with
different PACS codes is less than the JS distance between pairs with the same
L1 and L2 PACS codes. However, on doc2vec embeddings on titles, we fail to
reject the hypothesis that the mean cosine distance of random pairs is not
equal to that of random pairs. Moreover, there are no clear differences in
Jensen-Shannon distance among the cosine distance distributions of random
versus other paper pairs. Meanwhile, on abstracts, although there are also no
visible differences in Jensen-Shannon distance, we do reject the hypothesis
that the mean cosine distance of random pairs is not equal to that of random
pairs.

Figure 5: A comparison of embedding distance distribution between sampled paper pairs also show that Sentence-BERT (title and abstract embeddings), node2vec, and residual2vec embeddings are more likely to embed papers of the same discipline closer to one another than random pairs of papers. Figure 6: Jensen-Shannon (JS) distance matrices between cosine distance distributions of paper pairs. Here, node2vec and residual2vec have the most visible differences in JS distance across the multiple distributions. In particular, the JS distances are smallest along the diagonal, with the distances increasing for paper pairs with the same L1 vs. L2 PACS codes. Property |  Embedding method |  $H_{a}$ |  T |  p-value
---|---|---|---|---
Undirected Graph |  Laplacian Eigenmap |  $\mu(\text{Different PACS})\neq\mu(\text{Random})$ |  3.3851 |  0.0007
|  |  $\mu(\text{Same L1 PACS})<\mu(\text{Different PACS})$ |  -15.3104 |  0.0000
|  |  $\mu(\text{Same L2 PACS})<\mu(\text{Same L1 PACS})$ |  -9.7929 |  0.0000
|  node2vec |  $\mu(\text{Different PACS})\neq\mu(\text{Random})$ |  34.1876 |  0.0000
|  |  $\mu(\text{Same L1 PACS})<\mu(\text{Different PACS})$ |  -173.5711 |  0.0000
|  |  $\mu(\text{Same L2 PACS})<\mu(\text{Same L1 PACS})$ |  -88.5119 |  0.0000
|  residual2vec |  $\mu(\text{Different PACS})\neq\mu(\text{Random})$ |  35.3063 |  0.0000
|  |  $\mu(\text{Same L1 PACS})<\mu(\text{Different PACS})$ |  -143.2327 |  0.0000
|  |  $\mu(\text{Same L2 PACS})<\mu(\text{Same L1 PACS})$ |  -82.2185 |  0.0000
Title |  doc2vec |  $\mu(\text{Different PACS})\neq\mu(\text{Random})$ |  -1.1306 |  0.2582
|  |  $\mu(\text{Same L1 PACS})<\mu(\text{Different PACS})$ |  -12.1402 |  0.0000
|  |  $\mu(\text{Same L2 PACS})<\mu(\text{Same L1 PACS})$ |  -3.2786 |  0.0005
|  SciBERT |  $\mu(\text{Different PACS})\neq\mu(\text{Random})$ |  0.0219 |  0.9825
|  |  $\mu(\text{Same L1 PACS})<\mu(\text{Different PACS})$ |  4.7426 |  1.0000
|  |  $\mu(\text{Same L2 PACS})<\mu(\text{Same L1 PACS})$ |  -10.2161 |  0.0000
|  Sentence-BERT |  $\mu(\text{Different PACS})\neq\mu(\text{Random})$ |  18.0919 |  0.0000
|  |  $\mu(\text{Same L1 PACS})<\mu(\text{Different PACS})$ |  -104.7975 |  0.0000
|  |  $\mu(\text{Same L2 PACS})<\mu(\text{Same L1 PACS})$ |  -32.8416 |  0.0000
Abstract |  doc2vec |  $\mu(\text{Different PACS})\neq\mu(\text{Random})$ |  2.1069 |  0.0351
|  |  $\mu(\text{Same L1 PACS})<\mu(\text{Different PACS})$ |  -9.2880 |  0.0000
|  |  $\mu(\text{Same L2 PACS})<\mu(\text{Same L1 PACS})$ |  -9.9860 |  0.0000
|  SciBERT |  $\mu(\text{Different PACS})\neq\mu(\text{Random})$ |  -0.2382 |  0.8117
|  |  $\mu(\text{Same L1 PACS})<\mu(\text{Different PACS})$ |  -1.9838 |  0.0236
|  |  $\mu(\text{Same L2 PACS})<\mu(\text{Same L1 PACS})$ |  -0.2722 |  0.3927
|  Sentence-BERT |  $\mu(\text{Different PACS})\neq\mu(\text{Random})$ |  9.3059 |  0.0000
|  |  $\mu(\text{Same L1 PACS})<\mu(\text{Different PACS})$ |  -65.0798 |  0.0000
|  |  $\mu(\text{Same L2 PACS})<\mu(\text{Same L1 PACS})$ |  -17.3435 |  0.0000
Table 2: t-test results for comparing the mean cosine distance for each
distribution. Rows highlighted in yellow indicate null hypotheses that we
failed to reject.

Figure 7 shows the ROC curves and AUC for the link prediction experiments. The
graph embedding methods have the highest performance for link prediction, as
measured by the area under the ROC curve (AUC). Sentence-BERT performs best
among the text embedding methods, but it does not perform as well as the
random-walk based embedding methods node2vec and residual2vec.

Figure 7: Link prediction results show that the graph embedding methods
perform very well in link prediction, especially node2vec and residual2vec.
Among the text embedding methods, Sentence-BERT also performs relatively well.

##  4 Discussion

We have found that while graph embedding methods use the rich information
stored in the citation network structure and generally perform better,
Sentence-BERT—whether it is trained on titles or abstracts—has a remarkable
performance. Our results suggest that the disciplinary structure of physics
may be better encoded with graph embedding rather than content embedding. In
other words, titles—which are often the only “content” information available
for many papers—may not fully capture the content of the paper. Abstracts are
able to encode the disciplinary structure better than titles, although
abstracts tend to be more limited in availability. Meanwhile, citations are
more informative and provide more accurate information about the “location” of
a paper. In the future, we may explore further the variety of information
captured by different embedding methods, as well as test embedding methods
that use both text data and citation information. Embedding text and citation
data provides many exciting opportunities to study how scientific knowledge is
referenced, created, and shared.

## Data and Code Availability

The code used in the analysis is available at
https://github.com/sabsconstantino/emb-comp-aps. The American Physical Society
dataset is available upon request at https://journals.aps.org/datasets.
Abstract data from the Web of Science is proprietary and cannot be made
available.

## Author Contributions

Isabel Constantino—Conceptualization, Data curation, Methodology, Software,
Visualization, Writing—original draft, Writing—review and editing. Sadamori
Kojaku—Methodology, Software, Writing—original draft, Writing—review and
editing. Santo Fortunato—Conceptualization, Funding acquisition, Methodology,
Writing—review and editing. Yong-Yeol Ahn—Conceptualization, Funding
acquisition, Methodology, Supervision, Writing—original draft, Writing—review
and editing.

## Funding Information

I.C., S.K., and Y.Y.A. are supported by the Air Force Office of Scientific
Research under award number FA9550-19-1-0391. S.F. is supported by the
National Science Foundation under award number 1927418, the Air Force Office
of Scientific Research under award number FA9550-19-1-0354, and the National
Institute of General Medical Sciences of the National Institutes of Health
under award number R35GM146974.

## References

  * [1] Lada A Adamic and Eytan Adar  “Friends and neighbors on the Web”  In _Social Networks_ 25.3, 2003, pp. 211–230  DOI: 10.1016/S0378-8733(03)00009-1
  * [2] Mikhail Belkin and Partha Niyogi  “Laplacian Eigenmaps for Dimensionality Reduction and Data Representation”  In _Neural Computation_ 15.6, 2003, pp. 1373–1396  DOI: 10.1162/089976603321780317
  * [3] Iz Beltagy, Kyle Lo and Arman Cohan  “SciBERT: A Pretrained Language Model for Scientific Text”  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ Hong Kong, China: Association for Computational Linguistics, 2019, pp. 3613–3618  DOI: 10.18653/v1/D19-1371
  * [4] David M. Blei  “Probabilistic topic models”  In _Communications of the ACM_ 55.4, 2012, pp. 77–84  DOI: 10.1145/2133806.2133826
  * [5] David M. Blei, Andrew Y. Ng and Michael I. Jordan  “Latent dirichlet allocation”  In _The Journal of Machine Learning Research_ , 2003, pp. 993–1022  DOI: 10.5555/944919.944937
  * [6] Katy Börner et al.  “Design and Update of a Classification System: The UCSD Map of Science”  In _PLOS ONE_ 7.7, 2012, pp. e39464  DOI: 10.1371/journal.pone.0039464
  * [7] Kevin W. Boyack and Richard Klavans  “Creation of a highly detailed, dynamic, global model and map of science”  In _Journal of the Association for Information Science and Technology_ 65.4, 2014, pp. 670–685  DOI: 10.1002/asi.22990
  * [8] Kevin W. Boyack, Richard Klavans and Katy Börner  “Mapping the Backbone of Science”  In _Scientometrics_ 64.3, 2005, pp. 351–374  DOI: 10.1007/s11192-005-0255-6
  * [9] Matteo Chinazzi, Bruno Gonçalves, Qian Zhang and Alessandro Vespignani  “Mapping the Physics Research Space: A Machine Learning Approach”  In _EPJ Data Science_ 8.1, 2019, pp. 33  DOI: 10.1140/epjds/s13688-019-0210-z
  * [10] W.J. Conover  “Practical nonparametric statistics”, Wiley series in probability and statistics  Wiley, 1999  URL: https://books.google.com/books?id=dYEpAQAAMAAJ
  * [11] Jean-Pierre Courtial and John Law  “A co-word study of artificial intelligence”  In _Social Studies of Science_ 19.2 Sage Publications, 1989, pp. 301–311  DOI: 10.1177/030631289019002005
  * [12] Arash Dehghan-Kooshkghazi et al.  “Evaluating Node Embeddings of Complex Networks”, 2022  arXiv:2102.08275 [cs.SI]
  * [13] Jacob Devlin, Ming-Wei Chang, Kenton Lee and Kristina Toutanova  “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding”  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ Minneapolis, Minnesota: Association for Computational Linguistics, 2019, pp. 4171–4186  DOI: 10.18653/v1/N19-1423
  * [14] D.M. Endres and J.E. Schindelin  “A new metric for probability distributions”  In _IEEE Transactions on Information Theory_ 49.7, 2003, pp. 1858–1860  DOI: 10.1109/TIT.2003.813506
  * [15] Lee Fleming  “Recombinant Uncertainty in Technological Search”  In _Management Science_ 47.1, 2001, pp. 117–132  DOI: 10.1287/mnsc.47.1.117.10671
  * [16] Thomas L. Griffiths and Mark Steyvers  “Finding scientific topics”  In _Proceedings of the National Academy of Sciences_ 101.suppl 1, 2004, pp. 5228–5235  DOI: 10.1073/pnas.0307752101
  * [17] Aditya Grover and Jure Leskovec  “Node2vec: Scalable Feature Learning for Networks”  In _Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining - KDD ’16_ San Francisco, California, USA: ACM Press, 2016, pp. 855–864  DOI: 10.1145/2939672.2939754
  * [18] Weiwei Gu, Aditya Tandon, Yong-Yeol Ahn and Filippo Radicchi  “Principled approach to the selection of the embedding dimension of networks”  In _Nature Communications_ 12.1, 2021, pp. 3772  DOI: 10.1038/s41467-021-23795-5
  * [19] Jeff Johnson, Matthijs Douze and Hervé Jégou  “Billion-scale similarity search with GPUs”  In _IEEE Transactions on Big Data_ 7.3, 2019, pp. 535–547  DOI: 10.1109/TBDATA.2019.2921572
  * [20] Stuart A. Kauffman  “Emergent properties in random complex automata”  In _Physica D: Nonlinear Phenomena_ 10.1, 1984, pp. 145–156  DOI: 10.1016/0167-2789(84)90257-4
  * [21] Sadamori Kojaku, Jisung Yoon, Isabel Constantino and Yong-Yeol Ahn  “Residual2Vec: Debiasing Graph Embedding with Random Graphs”  In _Thirty-Fifth Conference on Neural Information Processing Systems_ , 2021
  * [22] Diego Kozlowski, Jennifer Dusdal, Jun Pang and Andreas Zilian  “Semantic and Relational Spaces in Science of Science: Deep Learning Models for Article Vectorisation”  In _Scientometrics_ Springer International Publishing, 2021, pp. 1–30  DOI: 10.1007/s11192-021-03984-1
  * [23] S. Kullback and R.. Leibler  “On Information and Sufficiency”  In _The Annals of Mathematical Statistics_ 22.1, 1951, pp. 79–86  DOI: 10.1214/aoms/1177729694
  * [24] Quoc Le and Tomas Mikolov  “Distributed Representations of Sentences and Documents”  In _International Conference on Machine Learning_ PMLR, 2014, pp. 1188–1196
  * [25] Erin Leahey and James Moody  “Sociological Innovation through Subfield Integration:”  In _Social Currents_ , 2014  DOI: 10.1177/2329496514540131
  * [26] Loet Leydesdorff and Ismael Rafols  “Indicators of the Interdisciplinarity of Journals: Diversity, Centrality, and Citations”  In _Journal of Informetrics_ 5.1 Elsevier, 2011, pp. 87–100  DOI: 10.1016/j.joi.2010.09.002
  * [27] Loet Leydesdorff, Caroline S. Wagner and Lutz Bornmann  “Interdisciplinarity as Diversity in Citation Patterns among Journals: Rao-Stirling Diversity, Relative Variety, and the Gini Coefficient”  In _Journal of Informetrics_ 13.1, 2019, pp. 255–269  DOI: 10.1016/j.joi.2018.12.006
  * [28] David Liben-Nowell and Jon Kleinberg  “The link-prediction problem for social networks”  In _Journal of the American Society for Information Science and Technology_ 58.7, 2007, pp. 1019–1031  DOI: 10.1002/asi.20591
  * [29] Yiling Lin, James A. Evans and Lingfei Wu  “New directions in science emerge from disconnection and discord”  In _Journal of Informetrics_ 16.1, 2022, pp. 101234  DOI: 10.1016/j.joi.2021.101234
  * [30] David Liu and Tina Eliassi-Rad  “STABLE: Identifying and Mitigating Instability in Embeddings of the Degenerate Core”  In _SIAM International Conference on Data Mining_ Minneapolis, Minnesota: SIAM, 2023
  * [31] Tomas Mikolov, Kai Chen, Greg Corrado and Jeffrey Dean  “Efficient Estimation of Word Representations in Vector Space”, 2013  arXiv:1301.3781 [cs.CL]
  * [32] Tomas Mikolov et al.  “Distributed Representations of Words and Phrases and Their Compositionality”  In _Advances in Neural Information Processing Systems_ 26 Curran Associates, Inc., 2013
  * [33] Hao Peng et al.  “Neural Embeddings of Scholarly Periodicals Reveal Complex Disciplinary Organizations”  In _Science Advances_ 7.17 American Association for the Advancement of Science, 2021, pp. eabb9004  DOI: 10.1126/sciadv.abb9004
  * [34] “Physical Review Journals”, https://journals.aps.org/about
  * [35] “Physics and Astronomy Classification Scheme (PACS)”  In _Physical Review Journals_ , http://journals.aps.org/PACS, 2008
  * [36] Ismael Rafols and Martin Meyer  “Diversity and Network Coherence as Indicators of Interdisciplinarity: Case Studies in Bionanoscience”  In _Scientometrics_ 82.2, 2010, pp. 263–287  DOI: 10.1007/s11192-009-0041-y
  * [37] Nils Reimers and Iryna Gurevych  “Sentence-BERT: Sentence Embeddings Using Siamese BERT-Networks”  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ Hong Kong, China: Association for Computational Linguistics, 2019, pp. 3982–3992  DOI: 10.18653/v1/D19-1410
  * [38] Michal Rosen-Zvi, Thomas Griffiths, Mark Steyvers and Padhraic Smyth  “The author-topic model for authors and documents”  In _Proceedings of the 20th conference on uncertainty in artificial intelligence_ , UAI ’04  Arlington, Virginia, USA: AUAI Press, 2004, pp. 487–494
  * [39] Naoki Shibata, Yuya Kajikawa and Ichiro Sakata  “Link prediction in citation networks”  In _Journal of the American Society for Information Science and Technology_ 63.1, 2012, pp. 78–85  DOI: 10.1002/asi.21664
  * [40] Mike Steel, Wim Hordijk and Stuart A. Kauffman  “Dynamics of a Birth–Death Process Based on Combinatorial Innovation”  In _Journal of Theoretical Biology_ 491, 2020, pp. 110187  DOI: 10.1016/j.jtbi.2020.110187
  * [41] B. Uzzi, S. Mukherjee, M. Stringer and B. Jones  “Atypical Combinations and Scientific Impact”  In _Science_ 342.6157, 2013, pp. 468–472  DOI: 10.1126/science.1240474
  * [42] Jian Wang, Reinhilde Veugelers and Paula Stephan  “Bias against Novelty in Science: A Cautionary Tale for Users of Bibliometric Indicators”  In _Research Policy_ 46.8, 2017, pp. 1416–1436  DOI: 10.1016/j.respol.2017.06.006
  * [43] Martin L. Weitzman  “Recombinant Growth*”  In _The Quarterly Journal of Economics_ 113.2, 1998, pp. 331–360  DOI: 10.1162/003355398555595
  * [44] Frank Wilcoxon  “Individual Comparisons by Ranking Methods”  In _Biometrics Bulletin_ 1.6, 1945, pp. 80–83  DOI: 10.2307/3001968
  * [45] Thomas Wolf et al.  “Transformers: State-of-the-Art Natural Language Processing”  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ Online: Association for Computational Linguistics, 2020, pp. 38–45  DOI: 10.18653/v1/2020.emnlp-demos.6
  * [46] Lingfei Wu, Dashun Wang and James A. Evans  “Large teams develop and small teams disrupt science and technology”  In _Nature_ 566.7744, 2019, pp. 378–382  DOI: 10.1038/s41586-019-0941-9
  * [47] Yi Zhang, Fen Zhao and Jianguo Lu  “P2V: Large-Scale Academic Paper Embedding”  In _Scientometrics_ 121.1 Springer International Publishing, 2019, pp. 399–432  DOI: 10.1007/s11192-019-03206-9


11institutetext: Davan Harrison and Marilyn Walker 22institutetext: Natural
Language and Dialogue Systems Lab, UCSC, 22email<EMAIL_ADDRESS>33institutetext: Rishi Rajasekaran 44institutetext: Amazon Alexa AI, 44email:
<EMAIL_ADDRESS>
This work was done while Rishi Rajasekaran was a student at UCSC.

# A Transformer-based Response Evaluator for Open-Domain Spoken Conversation

Davan Harrison Rishi Rajasekaran  and Marilyn Walker

###### Abstract

Many open-domain dialogue systems rely on multiple response generators, any of
which can contribute a response to the dialogue in a particular context. Thus
the ability to compare potential responses and then select the best plays an
important role in ensuring a dialogue system is coherent and engaging.
Dialogue coherence goes beyond simply remaining on topic – some trivia may be
on topic and engaging when mentioned out of the blue, but may not be coherent
and grounded in the context of the conversation. We carry out experiments on
response selection in the Athena system, an Alexa Prize SocialBot that has
dedicated content and multiple topic-specific response generators for a large
number of topics. First, we collect a corpus of Athena conversations with live
human traffic, where potential responses from all enabled response generators
are logged and subsequently annotated for response quality. We compare several
off-the-shelf response ranking methods for open-domain dialogue to Athena-
Heuristic, a heuristic response ranker that was field-tested in Athena during
the third Alexa Prize competition. We also compare these to a transformer-
based response ranker we call Athena-RR, that we train on our Athena
conversations. Athena-RR uses both the conversational context and the dialogue
state to rank the potential responses. We find that Athena-RR with a Recall@1
of 70.79% outperforms Athena-Heuristic and all of the off-the-shelf rankers by
a large margin. We then conduct a live A/B study comparing Athena-Heuristic to
Athena-RR in a 6,358 conversations with Alexa users. We show that Athena-RR
leads to significantly longer conversations that receive significantly higher
user ratings than the heuristic rule-based ranker.

##  1 Introduction

Open-domain spoken dialogue systems aim to carry out a natural conversation
with a human user on any topic. While this remains an extremely challenging
problem, a number of advances have been made in recent years, partly through
competitions such as the Alexa Prize SocialBot Grand Challenge Khatri et al.
(2018); Gabriel et al. (2020). However, given the diversity of user interests
and utterances in conversation, even the best of these systems can struggle to
come up with good responses in many contexts Pichl et al. (2020); Paranjape et
al. (2020); Chen et al. (2018); Fang et al. (2017); Curry et al. (2018);
Bowden et al. (2019). As a result, most such systems utilize multiple response
generators that contribute to a pool of potential responses in every dialogue
context. Thus, the ability to rank several potential responses and then select
the most appropriate one plays an important role in ensuring that a dialogue
system remains coherent and engaging Shalyminov et al. (2018); Hedayatnia et
al. (2022); Ghazarian et al. (2022); Kim et al. (2020); Mehri and Eskenazi
(2020b); Ghazarian et al. (2020). The aim of this work is to develop improved
response selection methods for open-domain spoken conversation.

The recent growth and popularity of open-domain systems has been accompanied
by many new reference-free dialogue response ranking methods. Frameworks such
as DialogRPT, DynaEval, USR, RUBER, and FED, inter alia, provide off-the-shelf
tools for estimating the quality of a turn in an ongoing conversation Gao et
al. (2020); Zhang et al. (2021); Yeh et al. (2021); Mehri and Eskenazi
(2020a); Tao et al. (2018); Mehri and Eskenazi (2020b). We test several of
these off-the-shelf rankers on the task of response ranking in Athena, a
socialbot that competed in the Alexa Prize Socialbot Grand Challenge over the
past few years Harrison et al. (2020); Omkar Patil and Walker (2021).

Athena’s dialogue system architecture, shown in Figure 1, follows typical
socialbot design principles, where the response sub-component consists of
multiple response generators (RGs) for each topic. Unlike open-domain chatbots
based on LLMs such as GPT3 or DialoGPT Zhang et al. (2019); Radford et al.
(2019), Athena includes topic-specific response generators that use live
content from WikiData and other databases, as well as topic-specific trivia.
For each system turn, a Dialogue Manager (DM) queries a subset of RGs for
responses given the topic and the dialog context, resulting in a pool of
candidate responses, from which a response is constructed and spoken to the
user. During the third Alexa Prize competition, we fielded Athena-Heuristic, a
heuristic response ranker. In this paper, we report experiments on training a
new transformer-based response ranker we call Athena-RR. We first create a
dataset of annotated (conversation-context, response-pool) examples, from live
customer traffic, by logging all the responses that could be produced by the
RGs at any point in an Athena dialogue.111We cannot release this dataset due
to user privacy issues, however a checkpoint for the trained model is
available at github.com/Vrindiesel/athena-rr . We annotate the goodness of all
the responses, and then use the annotated corpus to train a new response
ranker.

Figure 1: Athena’s architecture showing Multiple Types of Response Generators

We compare the performance of the DynaEval, DialogRPT, and FED off-the-shelf
response-rankers, to Athena-Heuristic and Athena-RR on the annotated test
corpus. We also test Athena-Heuristic and Athena-RR in the production Athena
system in an A/B study where we can measure user ratings and conversation
lengths. We show that Athena-RR ranker achieves an Recall@1 of 70.79% at
selecting highly rated responses, and that deploying Athena-RR to live
customer traffic resulted in an increase in mean conversation ratings from
3.64 to 3.77 and an increase in mean conversation length from 15.02 to 24.77
turns. The best off-the-shelf ranker is the DialogRPT Depth metric with a
Recall@1 of 50.56%. We find that the DynaEval metric does not perform as well
on our data as on other corpora, with a Recall@1 of only 21.34. We also find
that the 8 FED metrics are very highly correlated with one another, and that
their Recall@1 values range between 28.09% and 41.57%.

##  2 Related Work

There has recently been an upsurge of interest in response ranking for open-
domain dialogues Georgila et al. (2020); Deriu et al. (2021); Yeh et al.
(2021). In many cases, these are trained on an existing corpus, that doesn’t
have negative examples, so that that the ranker is trained to select the true
response as opposed to a randomly selected one Lowe et al. (2017); Liu et al.
(2016). More recent work uses quality annotations on dialogues, either from
crowd-sourced workers who created the dialogue turns, or from independent
judges Zhang et al. (2018); Yeh et al. (2021), with the types of dialogues
ranging from short chit-chat conversations such as DailyDialogue Li et al.
(2017), to longer information dense conversations such as Topical Chat
Gopalakrishnan et al. (2019a).

Our experiments test the off-the-shelf rankers DialogRPT, DynaEval, and FED.
DialogRPT is trained on Reddit threads, and uses up/down votes, number of
direct responses, and length of response threads to produce different methods
of ranking responses Gao et al. (2020). DynaEval has multiple versions trained
on Empathetic Dialogues, ConvAI2 and Daily Dialogue. We use the Empathetic
Dialogues variant that was used by the original authors for their out-of-
domain evaluation on FEDZhang et al. (2022). The FED ranker uses the
pretrained transformer DialoGPT to score responses on various dimensions of
conversational quality, using the language model probabilities of the response
eliciting a follow-on user reply such as “That’s interesting” Mehri and
Eskenazi (2020b).

There are also other response rankers specifically used in the context of the
Alexa Prize. Kim et al. uses a large corpus labelled with the output of an
acoustically-based user sentiment detector to train a model that predicts the
users’ sentiment towards a potential response Kim et al. (2020). Other work
has used specific user responses such as “Ask me if I care” to label the
previous turn as good or bad and then trained a ranker See and Manning (2021);
Shalyminov et al. (2018). Another approach relied on conversational data
annotated at the response level for its coherence and relevance in the context
Yi et al. (2019). In a similar approach to what we take here, Hedayatnia et
al. trained a response ranker on an annotated set of Alexa Prize conversations
with multiple responses generated by different neural response generation
models (NRGs), and then compared its performance to off-the-shelf models,
using different methods for negative and positive sampling Hedayatnia et al.
(2022). In this work, there were true negative responses, as in the Athena
annotated corpus described in Section 4. However, unlike Athena, Hedayatnia’s
system had only two topic-specific RGs, one for movies and another for music,
and thus the response pools are quite different than Athena’s. We make Athena-
RR publicly available222github.com/Vrindiesel/athena-rr in the hope that other
Alexa Prize teams will test it on their systems.

##  3 The Athena System

Athena is an Alexa Prize socialbot that has competed in the Alexa Prize for
the last two years Harrison et al. (2020); Juraska et al. (2021). During the
competition, Athena carries out thousands of conversations a week with real
users who access Alexa Prize systems by simply saying “Let’s Chat” to an
Alexa-enabled device. At the end of each conversation, the user has the option
to rate the conversation on a scale of 1 to 5, indicating their interest in
speaking with Athena again in the future. We posited that an improved response
ranker would increase Athena’s average rating and lead to longer
conversations.

Figure 1 in Section 1 illustrates Athena’s architecture. Response ranking in
Athena is especially challenging due to the large pool of RGs that can be
queried in any given turn. The types of response generators (RGs) are shown in
Figure 1, but there are Flow-RGs, KG-RGs and Entity-based Indexing (trivia)
RGs for each topic that Athena supports. The Dialogue Manager tracks the
discourse history and the context, as well as the topic under discussion, and
uses this state information to decide which RGs to run.

Figure 2: Distribution of Utilization of Athena’s Response Generators over a 6
month period

Figure 2 shows the utilization of Athena’s RGs over a six month period. Note
the large number of topics, including rare or niche topics such as the Marvel
Comic Universe (MCU), nutrition, astronomy, pirates and dinosaurs. Our
response ranker needs to work well for all of these topics. Figure 2 also
shows how topics, such as music and movies, have both a KG and a Flow-based RG
that compete with one another to take the next turn. For example, Music-KG,
accesses knowledge triples about musicians and albums from the WikiData
Knowledge Graph Reed et al. (2022), while the Music RG is a flow-based RG that
structures the conversation by asking the user about their music preferences
and opinions. The RG labelled as CENTER in Figure 2 represents the Entity
Indexing RGs that provide trivia, with almost all topics having a
corresponding trivia RG, which could also compete for the turn. The EVI QA
engine was also called in every context to provide alternative possible
responses. Neural Response Generators (NRGs) provided by Amazon were primarily
run in the background in Athena due to latency issues and the difficulty with
controlling their frequent topic changes Rajasekaran (2020); Omkar Patil and
Walker (2021); Chi et al. ; Saha et al. (2021).

While this multiple-RG setup allows for flexibility in handling unplanned user
behavior and results in more engaging and diverse conversations, it makes
response selection given the context of the conversation a very crucial task.
Figure 3 illustrates how Athena-RR, discussed in more detail below, allows the
Athena system to flexibly interleave responses from the three different
response generators for Music.

Figure 3: Dialogue illustrating the use of Multiple Response Generators for a
single topic. The RG column indicates the type of RG, also indicated through
different colors for each RG type.

##  4 Corpus Creation

Speaker |  Dialogue Context | Response Generator
---|---|---
ATHENA |  Yeah. A lot of people have cool hobbies, but what about me? What kind of hobby do you think I would like? | Intro RG
USER |  what’s the name for music |
ATHENA |  I love talking about music! Is there a musician or band that you like? | Music-KG
USER |  no |
Rank |  Response Candidates |  Response Generator
D |  I recently read this fact about Britney Spears. Spears had gotten her start in fame through being on The Mickey Mouse Club with other artists like Christina Aguilera. | Center-Music
A |  no problem! it can be hard to think of one. i like talking about the band Fleetwood Mac. Do you like Fleetwood Mac? | Music-KG
C |  I recently read an interesting fact about the world’s largest piano. The record for the largest piano ever made is held by Adrian Mann, a piano tuner from New Zealand. It took him four years to build the piano, and this grandiose instrument weighs 1.4 tonnes and is 5.7 meters long. | Center-Music
A |  So tell me, why do you like music? | Music-Flow
C |  I just read this fascinating fact about the history of Drums. Did you know that drums are one of the oldest musical instruments in the world? | Center-Music
C |  Do you know the size of the world’s smallest violin is? Well, I heard in Guangzhou, China, a 66-year-old senior violinist Chen Lianzhi made a 1-centimeter-long violin which sets the new world record for the world’s smallest violin. | Center-Music
|  None of the above. |

Table 1: Response pool annotation task example for Music. An “A” indicates an
excellent response, while an“B” indicates a response that could be used, and a
“C” indicates a response that might be okay in another context, while a “D”
indicates a bad response.

We use Athena’s live traffic with real human users to create our annotated
corpus. We take a random sample of 1,600 response pool instances with their
associated conversation histories. The 1,600 instances are comprised of 100
random samples from each of 16 internal topics used by Athena, as illustrated
in Figure 2. The responses are selected such that we get 20 instances from
conversations belonging to each 1-5 rating value, whenever possible.

Speaker |  Dialogue Context | Response Generator
---|---|---
ATHENA |  Nice choice. And for me? It’s hard for me to pick a specific scene, but I certainly liked Eli Wallach in this movie. What did you think of her portrayal of Tuco? | Movies-Flow
USER |  i’m not familiar with that actress |
ATHENA |  In order to capture legal conversations, the Director of Marriage story would meet with actual divorce attorneys. | Center-Movies
USER |  i don’t know anything about that |
Rank |  Response Candidate Text | Response Generator
A |  Seen any good movies recently? | Movies-KG
A |  I was wondering what you think about this. Would you rather star in a commercial blockbuster or an independent film that wins prestigious awards? | Movies-Flow
C |  Show business seems like it would be a really confusing business to navigate. Keanu Reeves was actually tricked into appearing in the film when a friend forged his signature on the contract. | Center-Movies
C |  I already though Charlize Theron was really interesting, and then I heard this. When she was a baby, Charlize Theron had jaundice, and the doctors put her on antibiotics. Her baby teeth rotted because of the medicine, so she did not have a normal set of teeth until she was 11 years old and her adult teeth grew in. | Center-Movies
B |  The director would explain the concept of the divorce at the center of the story then ask what kind of things these lawyers would target in the characters. | Center-Movies
A |  Who is your favorite actor or actress? | Movies-KG
C |  Is there a director that you like? Is there a movie that you want to talk about? | Movies-KG
C |  I was a little disappointed that The Two Popes had some historical inaccuracies. One thing The Two Popes got right though is that Pope Benedict liked Orange Fanta, he drank at least 4 cans of the soda every day! | Center-Movies
|  None of the above. | None-candidate

Table 2: Response pool annotation task example for Movies. An “A” indicates an
excellent response, while an“B” indicates a response that could be used, and a
“C” indicates a response that might be okay in another context, while a “D”
indicates a bad response.

Athena particularly struggles with selecting good responses to questions, so
we also bias the sampling to include 40/100 instances where the user’s most
recent utterance is a question. This is not always possible because question
dialogue acts occur less frequently within some topics compared to others. So
we collect an additional random sample of 494 instances of users asking
questions, without regards to topic. This amounts to a total of 2,094 response
pool instances. When collecting response pools, we limit the sampling to
instances where the response pool contains more than one candidate response.
An example response pool for a Music conversation is shown in Table 1 and one
for Movies is shown in Table 2.

We collect annotations for the corpus via a response selection task where
human judges rate response candidates. In each annotation instance, the judges
are presented with 4 turns of conversation history ending with the user’s most
recent utterance, with a pool of response candidates listed below. The data is
in spreadsheet format. Annotators are asked to select a response that they
would say next by filling in the cell next to the selected response candidate.
The annotations for the two response pool examples are shown in the LHS of
Table 1 and of Table 2. Annotators are allowed to select multiple response
candidates if multiple responses would be appropriate to say next. Notice that
in both Table 1 and Table 2 multiple responses are indicated with an “A”
indicating they are equally good. Notice also that the response pools show how
different RGs for the same topic compete with one another to take the turn.
The Center RGs provide trivia, and they are part of the response pool in
general to guarantee that Athena always has something to say, but even though
they are “on topic” they are often not as good as responses produced by the KG
response generators or the scripted Flow response generators. Annotators are
always given a ”None of the above” option for situations where the response
pool does not contain any good candidates. The annotations are performed by
the authors and recruited lab-mates. Each response pool received one
annotation therefore we did not measure annotator agreement.

Corpus Descriptive Statistics | Un-normalized Corpus | Normalized Corpus
---|---|---
# topics | 16 | # annotated responses | 7,915 | # annotated responses | 6,113
# instances per topic | 100 | mean response pool size | 4.43 | mean response pool size | 3.42
# topical instances | 1,600 | median response pool size | 4.0 | median response pool size | 3.0
# extra question instances | 494 |  |  |  |
Total # of instances | 2,094 |  |  |  |

Table 3: Corpus statistics for annotated Athena response pool dataset.

Overall statistics on the annotated corpus of Athena response pools can be
seen in Table 3 and Table 4. Table 3 summarizes how the response pool covers
all of Athena’s topics, as well as a general pool of questions. The mean and
median response pool sizes are artificially reduced to make it easier for the
annotators. In general, in any dialogue context, there are many more than 3 or
4 possible responses as illustrated by the response pool annotation examples
in Figure 1 and Figure 2. This is because there are many Center-RG potential
responses as well as multiple responses from the knowledge graph RGs.

Data | Topical instances | Question instances
---|---|---
# initial response pools | 1,600 (%100.0) |
# pools with only negative responses | 306 (%19.1) | -
# pools with positive responses | 1,294 (%80.8) | 494
# included in training set | 1,294 | 494
Table 4: Response Pools positive and negative instances

Table 4 breaks the corpus statistics down further, to show how many positive
and negative examples there are in each category of topical instances and
question instances. Despite the many different RGs, 19.1% of the response
pools are judged to have no good response, perhaps illustrating how it can be
difficult to fully predict how users will behave. This observation is also
supported by other work on Alexa Prize, where approximately 33% of the
response pools had no good responses Hedayatnia et al. (2022).

##  5 Ranking Responses: Experiment Overview

Our primary goal is to evaluate how well different response ranking methods,
when given the conversation context, are able to select the best response out
of a pool of responses. We train our own Athena specific response ranker
Athena-RR, and test three recently proposed models for domain independent,
reference free response ranking, namely DialogRPT, DynaEval and FED Zhang et
al. (2022); Mehri and Eskenazi (2020b); Gao et al. (2020). We compare these to
Athena-Heuristic, a response ranker that was fielded in the third Alexa Prize
competition. We evaluate response ranking methods in two ways: (1) accuracy
against our test corpus; and (2) by deployment into the production Athena
system as part of an A/B study on user ratings and conversation lengths.

###  5.1 Response Ranking Models

Athena’s Heuristic Response Ranker. Athena’s heuristic response ranker is
based on a topic-agnostic approach to dialogue management based on theories of
discourse coherence Grosz et al. (1995); Prasad et al. (2010) to dynamically
structure a dialogue. The heuristic ranker applies rules on the current
dialogue state to try to remain on topic until the user requests a topic
change or the topic has been exhausted. Topic-specific RGs can provide
additional signals to the DM along with their response such as an indication
that they must continue, can continue, or have ended their conversation.
Preference is first given to the RG that had control on the last turn, in
order to increase local coherence and promote conversation flow. Then
preference would be given to another RG on the same topic, for example when a
FlowRG for music was exhausted, the KG-RG for music and the Center RG for
music would be equally preferred.

DialogRPT Response Ranker. DialogRPT333https://github.com/golsun/DialogRPTGao
et al. (2020) is a transformer-based dialogue response ranking model trained
by predicting human feedback (i.e., upvotes or replies) of responses on a
large-scale social media platform. The model was trained on 100’s of millions
of Reddit conversation threads. DialogRPT is trained with a contrastive
ranking loss where the model learns to perform pairwise classification,
maximizing the likelihood of the positive class instance over that of the
negative class instance.

Multiple versions of DialogRPT were trained using different types of human
feedback. We use three variants of DialogRPT whose checkpoints were made
publicly available by the original authors. The DialogRPT-updown was trained
using post upvote information, DialogRPT-width and DialogRPT-depth were
trained using conversation thread width and depth information, respectively.
We use the implementation and model available through Huggingface
Tranformers.444https://github.com/huggingface/transformers

DynaEval Ranker. DynaEval Zhang et al. (2022) leverages a Graph Neural Network
to model the dialog-level interactions between a user and a system. This
allows it to distinguish well-formed dialogues from carefully constructed
negative samples that allow it to better capture dialogue level coherence.
Previous work that tested DynaEval on DailyDialog, the USR subset of Topical
Chat, ConvAI2 and Empathetic Dialogues showed that it performs considerably
better than other metrics across these different datasets Yeh et al. (2021).
We evaluate DynaEval for turn-level response ranking and selection by
selecting the candidate response with the highest dialog-level coherence
score.

The FED Ranker. FED is a method proposed to perform reference-free evaluation
of 18 fine-grained dialogue qualities using a large pretrained model Mehri and
Eskenazi (2020b). These dialogue qualities are further divided into 8 turn-
level and 10 conversation-level qualities. The turn-level qualities measure
whether a specific turn of a conversation was Interesting, Engaging, Specific,
Relevant, Correct, Semantically Appropriate, Understandable, and Fluent. We
select FED as one of our baselines for turn-level evaluation in Athena because
the Meena and Mitsuku datasets that it showed good results for are similar to
Athena conversations Adiwardana et al. (2020).

Metric |  Type |  Probe
---|---|---
Interesting |  Positive |  Wow that is really interesting. / That’s really interesting! / Cool! That sounds super interesting.
|  Negative |  That’s not very interesting. / That’s really boring. / That was a really boring response.
Engaging |  Positive |  Wow! That’s really cool! / Tell me more! / I’m really interested in learning more about this.
|  Negative |  Let’s change the topic. / I don’t really care. That’s pretty boring. / I want to talk about something else.
Relevant |  Positive |  —
|  Negative |  That’s not even related to what I said. / Don’t change the topic! / Why are you changing the topic?
Correct |  Positive |  —
|  Negative |  You’re not understanding me! / I am so confused right now! / I don’t understand what you’re saying.
Semantically |  Positive |  That makes sense. / You have a good point. ,
Appropriate |  negative |  That makes no sense!
Fluent |  Positive |  That makes sense! / You have a good point.
|  Negative |  Is that real English? / I’m so confused right now! / That makes no sense!
Table 5:  Sample User response probes for FED turn level metrics

The turn-level metrics are computed by determining the relative likelihood of
a system response in context being followed by a set of predefined positive
and negative probe responses for a given dialoguequality. The probe responses
are shown in Table 5. For each conversation in the normalized corpus, the
turn-level FED metric scores given the previous context is computed for each
candidate response using a DialoGPT-large model. We exclude dialogues that do
not have any context turns, since the initial turns are greeting turns and do
not involve response selection. This yields a final corpus of 5927 candidate
responses in 1743 dialogues.

FED is meant to provide general metrics based on LLMs that are not a
reflection of a particular training set. Previous work suggests that the
correlations between FED metrics and human annotations can be as high as .21
at the turn level and as high as .443 at the dialogue level Mehri and Eskenazi
(2020b). Our hypothesis was that the different turn-level FED measures
strongly overlap in terms of what they are trying to capture.

To explore this, we calculated the the Pearson correlation for the individual
metric scores for all the responses from the conversation corpus. The
correlation heatmap is shown in Figure 4. The heatmap clearly shows (light
cream blocks) that sets of metrics are highly correlated with one another. In
particular, the following groups of metrics exhibit a high degree of
correlation with each other — (i) interesting, engaging, specific (ii)
relevant, correct, (iii) semantically appropriate, understandable, fluent. The
heatmap also shows that the interesting, engaging, and specific metrics have a
strong negative correlation with the semantically appropriate, understandable,
fluent metrics.

Figure 4: FED metric Correlations on Athena corpus (DialoGPT-large)

We posit two causes for this result — one being the conversations that
DialoGPT is trained on are very different from Athena conversations and the
other being many of the chosen reference sentences used to compute the FED
scores are very similar across the correlated metrics. Note that the reference
sentences shown in Table 5 for semantically appropriate, understandable, and
fluent are based on the same positive prompts while using very similar or
overlapping examples of negative prompts. Also relevant, correct both contain
only negative prompts while interesting, engaging, specific use positive
prompts that can be construed as being similar. As a result, the metrics in
each correlated group appear to measure the same attribute, which is in line
with related work that report similar correlations in the human annotation
scores of the conversation-level qualities in the FED datasetZhang et al.
(2022).

We also explored whether there’s a significant difference between the FED
metric scores for the population of human preferred and dispreferred responses
in the normalized Athena corpus. We did a two-sided t-test for the FED metrics
between the samples of human preferred and dispreferred utterances. We found
that Interesting, Engaging, Specific scores for the human preferred responses
were actually significantly lower than those of dispreferred responses; in
other words they go the opposite direction of what they are supposed to
measure. The two samples for Relevant, Correct, Semantically Appropriate,
Fluent showed no statistically significant difference. The only metric that
worked as predicted was Understandable, where the scores for human preferred
responses were significantly higher than those of the dispreferred responses.
Overall, the FED metrics demonstrate weak results on Athena conversations for
response ranking and may not be a suitable measure for dialogue quality
evaluations.

Athena-RR: A Response Ranking transformer.

Much of the recent research using pre-trained transformer models use an end-
to-end dialogue learning approach. Rather than aiming to construct an end-to-
end model, we want to enhance the dialogue system that we already have.
Therefore, we will not be able to “copy” the problem setup used in other work
Gopalakrishnan et al. (2019b); Dinan et al. (2019); Wolf et al. (2019b). We
also want Athena’s response ranker to take advantage of all the information
contained in Athena’s internal dialogue state representation. We develop a
method of incorporating the additional information into the model by extending
the input sequence dialogue tokens using special token representations of
Athena’s internal state.

We train a BERT based model, Athena-RR, using our annotated corpus of Athena’s
response pools. Athena-RR is trained by optimizing a next-utterance
classification loss, similar to that used in the next-sentence prediction task
used in BERTDevlin et al. (2019). The model performs binary classification on
each response candidate to predict whether or not a potential utterance occurs
next for a given conversation context. Our end goal is to have a model capable
of picking responses from a response pool rather than just performing binary
classification on responses.

Our model is pre-trained on the Topical Chats corpus using random sampling for
False next-utterances. We take the model checkpoint with the highest
validation accuracy on the Topical Chats corpus and fine-tune the model on
Athena conversation data. For the sake of evaluation on this intermediate
binary classification task, we use a random training/evaluation partition of
the dataset using a $90:10$ ratio.

To smooth the distribution of positive and negative class instances in the
data, the annotated corpus is down sampled to smooth the distributions of
response candidate selections. Some response candidates were presented to
annotators a disproportionate number of times, resulting in some utterances
having been not selected most of the times they were in a response pool. This
is due to the internal working of Athena, and the way that the response pools
were collected. In cases where an example occurs more than 8 times as a
negative example, we cap its frequency using the cap value $N_{\text{cap}}$,
computed as: $N_{\text{cap}}=8+2\cdot\text{log}(\bar{C})$ where $\bar{C}$ is
the number of times annotators had the opportunity to select the candidate but
didn’t.

Input Representation. Model inputs are the conversation context and a
candidate response. Conversation context includes four turns of history
represented as sequence of tokens with a special [SEP] token between turns.
The context also includes special tokens that represent Athena’s internal
dialogue state, such as the current topic and the previous topic. We also add
special tokens to the front of the response candidate token sequence that
denote the RG it came from.

Figure 5: Speaker aware token embeddings.

Speaker-Aware turn embeddings. Athena-RR is implemented using the HuggingFace
TransformersWolf et al. (2019a) implementation of the BERT transformer in our
experiments. Similar to (Wolf et al., 2019b), we adapt the model by adding
special embeddings that indicate the speaker and the dialogue state
information. Figure 5 shows the speaker aware embedding token and embedding
function.

###  5.2 Evaluation Methods and Setup

Corpus Evaluation. We evaluate all five ranking models, DialogRPT, DynaEval,
FED, Athena-Heuristic and Athena-RR in terms of recall@1 on a held-out test
set of 89 (context, response pool) instances. To measure the performance of
the off-the-shelf models, we test whether the response with the highest model
score is a human preferred response in each conversation.

Evaluation: Live User Study. Because Athena is a real system that interacts
with human users, we were also able to incorporate our response ranking model
into Athena’s dialogue manager and deploy it in an A/B study with the A
version using a pre-existing heuristic response ranker, and the B version
using Athena-RR. We trained a new version of the model using our entire
annotation corpus including our test set. During training, the model learns to
perform binary classification on a given response candidate: say it next or
don’t say it next. We incorporate the model into Athena by scoring each
candidate response separately for a given response pool, and then returning
the response with the highest log-likelihood. Note that Athena-RR does not do
topic-selection, this is driven by a model of user topic preferences and the
dialogue context. Athena-RR is also not used when the response pool contains
only one response candidate.

##  6 Results

Model | Recall@1
---|---
DialogRPT-updown | 29.21%
DialogRPT-width | 46.07%
DialogRPT-depth | 47.19%
DialogRPT-human-vs-rand | 42.70%
DialogRPT-human-vs-machine | 38.20%
DynaEval | 21.34%
FED-Interesting | 28.09%
FED-Engaging | 30.34%
FED-Specific | 26.97%
FED-Relevant | 39.33%
FED-Correct | 41.57%
FED-Semantically Appropriate | 41.57%
FED-Understandable | 40.45%
FED-Fluent | 39.33%
Athena-Heuristic | 56.18%
Athena-RR | 70.79%
Table 6: Performance of the various response rankers on the test set

Table 6 shows that DynaEval and the DialogRPT-updown metric have the poorest
performance with accuracies of only 21.34% and 29.21% respectively. The
DialogRPT width and depth metrics perform much better with the depth metric
achieving an accuracy of 47.19%. DialogRPT is better than the best FED metric.
Previous work suggests that the DialogRPT human-vs-random ranker performs the
best Hedayatnia et al. (2022), though the human-based metrics are not better
than DialogRPT-width and DialogRPT-depth on the Athena corpus. The FED metrics
Interesting, Engaging, Specific all have accuracies at 30% or worse. Relevant,
Correct, Semantically Appropriate, Understandable, and Fluent have the highest
accuracies, with best accuracy at 41.57%, but the differences between them are
not statistically significant (p $=$ .58). The Athena-Heuristic ranker is
better than DialogRPT-depth and better than FED Semantically Appropriate (p
$=$ 0.037), which is the highest accuracy FED metric. The relatively poorer
performance of the FED metrics in comparison to the Athena-Heuristic ranker
that simply tries to stay on topic indicates that FED does not have any notion
of topic relevance. The Athena-RR ranker, trained specifically for this task,
performs the best, as might be expected. The accuracy of 70.79% is much better
than that achieved by the previous heuristic Athena response ranker.

|  | Median | Mean | Mean
---|---|---|---|---
Version | N | # turns | # turns | rating
Heur. | 3,502 | 10 | 15.02 | 3.64
Trans. | 2,856 | 19 | 24.77 | 3.77
|  |  |  p<0.01 |  p<0.01
Table 7: A/B study results: heuristic response ranker versus AthenaRR.

Human Evaluation A/B Study. We then deployed Athena-RR and Athena-Heuristic
simultaneously in a live A/B study. Because the metrics used in the Alexa
Prize are the average user rating and length (number of turns) for
conversations, we evaluate those measures in the A/B study. Athena-RR is not
used until the fourth system turn so we only consider conversations that meet
this length requirement. We also note that there are a number of possible
situations in a conversation where response ranking is not used Whang et al.
(2021). For instance, if the user says ”Let’s talk about something else”, the
flow-of-control returns to the DM to select a new topic. Also, Athena-RR is
not used when the system needs to perform a functional speech act, e.g., user
asks the system to repeat itself, or Athena needs to deflect a user seeking
adult content.

Table 7 shows the A/B study results for conversations where Athena-RR was used
on one or more turn, compared to conversations where Athena-Heuristic was used
on every turn. The results show that Athena-RR outperforms the Athena-
Heuristic model for both conversation length and ratings, with a significant
(p $<$ .01) and large increase in both measures.

##  7 Discussion and Conclusion

This paper describes a response ranker for the Athena SocialBot system, a
competitor in the Alexa Prize. Response ranking is challenging in Athena
because it has the capacity to provide many different on-topic responses in a
particular dialogue context: the response ranker needs to discriminate between
different on-topic responses. We developed a corpus of Athena conversations
with real human users by logging all possible responses that Athena could
produce for many different dialogue contexts, and then sampled these so as to
represent all of Athena’s topics, as well as to have a good distribution of
contexts where users ask questions.

We then explore different models for response ranking and selection. We train
a new conversational response ranking model, Athena-RR, based on the speaker-
aware BERT architecture and compare its performance to the reference-free
dialogue evaluation metrics DialogRPT, DynaEval and FED, as well as to Athena-
Heuristic, the rule-based response ranker fielded in Alexa Prize 3. We find
that DialogRPT-depth provides the highest performance of the off-the-shelf
metrics with a Recall@1 of 47.19%. We show that DynEval performs much worse on
Athena than on other corpora Yeh et al. (2021). We also demonstrate that the
turn-level metrics within FED exhibit strong correlations with each other, as
also found in previous work Zhang et al. (2022), and that FED performs
significantly worse than DialogRPT-depth, with the best turn-level FED metrics
achieving a Recall@1 of 41.57%. Finally, we show that Athena-RR significantly
outperforms Athena-Heuristic in an offline evaluation with a Recall@1 of
70.79% as compared to 56.18%, and that it is also significantly better than
any of the off-the-shelf metrics.

One surprising result was that Athena-Heuristic, which simply tries to stay on
topic, significantly outperformed all the off-the-shelf metrics (p $=$ 0). One
would imagine that DialogRPT, which is trained on thousands of Reddit
conversations, would have internalized a notion of topical coherence. Perhaps
DialogRPT performs more poorly than expected because Reddit conversations are
very different in style than Athena: the conversations are between humans,
topical coherence may not be an issue, and the number of human responses to an
utterance may reflect their individual opinions rather than utterance quality.

Finally we conduct an A/B study of Athena-RR and Athena-Heuristic in our live
deployed system with over 6000 conversations with end-users. Our results show
that Athena-RR led to significantly better conversation ratings (3.77 compared
to 3.64) and longer conversations (24.77 turns as compared to 15.02 turns)
when compared to those chosen by a heuristic-based response ranker.

## References

  * Adiwardana et al. [2020] Daniel Adiwardana, Minh-Thang Luong, David R So, Jamie Hall, Noah Fiedel, Romal Thoppilan, Zi Yang, Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu, et al.  Towards a human-like open-domain chatbot.  _arXiv preprint arXiv:2001.09977_ , 2020.
  * Bowden et al. [2019] Kevin K Bowden, Jiaqi Wu, Wen Cui, Juraj Juraska, Vrindavan Harrison, Brian Schwarzmann, Nicholas Santer, Steve Whittaker, and Marilyn Walker.  Entertaining and opinionated but too controlling: a large-scale user study of an open domain alexa prize system.  In _Proceedings of the 1st International Conference on Conversational User Interfaces_ , pages 1–10, 2019.
  * [3] Ethan A Chi, Caleb Chiam, Trenton Chang, Swee Kiat Lim, Chetanya Rastogi, Alexander Iyabor, Yutong He, Hari Sowrirajan, Avanika Narayan, Jillian Tang, et al.  Neural, neural everywhere: Controlled generation meets scaffolded, structured dialogue.
  * Chen et al. [2018] Chun-Yen Chen, Dian Yu, Weiming Wen, Yi Mang Yang, Jiaping Zhang, Mingyang Zhou, Kevin Jesse, Austin Chau, Antara Bhowmick, Shreenath Iyer, et al.  Gunrock: Building a human-like social bot by leveraging large scale real user data.  _Alexa Prize Proc._ , 2018.
  * Curry et al. [2018] Amanda Curry, Ioannis Papaioannou, Alessandro Suglia, Shubham Agarwal, Igor Shalyminov, Xinnuo Xu, Ondřej Dušek, Arash Eshghi, Ioannis Konstas, Verena Rieser, et al.  Alana v2: Entertaining and informative open-domain social dialogue using ontologies and entity linking.  _Alexa Prize Proc._ , 2018.
  * Deriu et al. [2021] Jan Deriu, Alvaro Rodrigo, Arantxa Otegi, Guillermo Echegoyen, Sophie Rosset, Eneko Agirre, and Mark Cieliebak.  Survey on evaluation methods for dialogue systems.  _Artificial Intelligence Review_ , 54(1):755–810, 2021.
  * Devlin et al. [2019] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _ArXiv_ , abs/1810.04805, 2019.
  * Dinan et al. [2019] Emily Dinan, Stephen Roller, Kurt Shuster, Angela Fan, Michael Auli, and Jason Weston.  Wizard of wikipedia: Knowledge-powered conversational agents.  _arXiv:1811.01241_
  * Fang et al. [2017] Hao Fang, Hao Cheng, Elizabeth Clark, Ariel Holtzman, Maarten Sap, Mari Ostendorf, Yejin Choi, and Noah A Smith.  Sounding board–university of washington’s alexa prize submission.  _Alexa Prize Proc._ , 2017.
  * Gabriel et al. [2020] Raefer Gabriel, Yang Liu, Anna Gottardi, Mihail Eric, Anju Khatri, Anjali Chadha, Qinlang Chen, Behnam Hedayatnia, Pankaj Rajan, Ali Binici, et al.  Further advances in open domain dialog systems in the third alexa prize socialbot grand challenge.  _Alexa Prize Proc._ , 2020.
  * Gao et al. [2020] Xiang Gao, Yizhe Zhang, Michel Galley, Chris Brockett, and William B Dolan.  Dialogue response ranking training with large-scale human feedback data.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 386–395, 2020.
  * Georgila et al. [2020] Kallirroi Georgila, Carla Gordon, Volodymyr Yanov, and David Traum.  Predicting ratings of real dialogue participants from artificial data and ratings of human dialogue observers.  In _Proceedings of the 12th Language Resources and Evaluation Conference_ , pages 726–734, 2020.
  * Ghazarian et al. [2020] Sarik Ghazarian, Ralph Weischedel, Aram Galstyan, and Nanyun Peng.  Predictive engagement: An efficient metric for automatic evaluation of open-domain dialogue systems.  In _Proc. of the AAAI Conference on Artificial Intelligence_ , volume 34, pages 7789–7796, 2020.
  * Ghazarian et al. [2022] Sarik Ghazarian, Behnam Hedayatnia, Alexandros Papangelis, Yang Liu, and Dilek Hakkani-Tur.  What is wrong with you?: Leveraging user sentiment for automatic dialog evaluation.  In _Findings of the Association for Computational Linguistics: ACL 2022_ , pages 4194–4204, 2022.
  * Gopalakrishnan et al. [2019a] Karthik Gopalakrishnan, Behnam Hedayatnia, Qinlang Chen, Anna Gottardi, Sanjeev Kwatra, Anu Venkatesh, Raefer Gabriel, Dilek Hakkani-Tür, and Amazon Alexa AI.  Topical-chat: Towards knowledge-grounded open-domain conversations.  _Proc. Interspeech 2019_ , pages 1891–1895, 2019a.
  * Gopalakrishnan et al. [2019b] Karthik Gopalakrishnan, Behnam Hedayatnia, Qinlang Chen, Anna Gottardi, Sanjeev Kwatra, Anu Venkatesh, Raefer Gabriel, and Dilek Hakkani-Tür.  Topical-Chat: Towards Knowledge-Grounded Open-Domain Conversations.  In _Proc. Interspeech 2019_ , pages 1891–1895, 2019b.  doi: 10.21437/Interspeech.2019-3079.  URL http://dx.doi.org/10.21437/Interspeech.2019-3079.
  * Grosz et al. [1995] Barbara J. Grosz, Scott Weinstein, and Aravind K. Joshi.  Centering: A framework for modeling the local coherence of discourse.  _Computational Linguistics_ , 21(2):203–225, June 1995.  ISSN 0891-2017.
  * Harrison et al. [2020] Vrindavan Harrison, Juraj Juraska, Wen Cui, Lena Reed, Kevin K Bowden, Jiaqi Wu, Brian Schwarzmann, Abteen Ebrahimi, Rishi Rajasekaran, Nikhil Varghese, et al.  Athena: Constructing dialogues dynamically with discourse constraints.  _Proc. of the Alexa Prize 2020_ , 2020.
  * Hedayatnia et al. [2022] Behnam Hedayatnia, Di Jin, Yang Liu, and Dilek Hakkani-Tur.  A systematic evaluation of response selection for open domain dialogue.  In _Proceedings of the 23rd Annual Meeting of the Special Interest Group on Discourse and Dialogue_ , pages 298–311, 2022.
  * Juraska et al. [2021] Juraj Juraska, Vrindavan Harrison, , Lena Reed, Kevin Bowden, Wen Cui, Omkar Patil, Adwait Ratnaparkhi, and Marilyn Walker.  Athena 2.0: Contextualized dialogue management for an alexa prize socialbot.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 124–133, 2021.
  * Khatri et al. [2018] Chandra Khatri, Behnam Hedayatnia, Anu Venkatesh, Jeff Nunn, Yi Pan, Qing Liu, Han Song, Anna Gottardi, Sanjeev Kwatra, Sanju Pancholi, et al.  Advancing the state of the art in open domain dialog systems through the alexa prize.  _arXiv preprint arXiv:1812.10757_ , 2018.
  * Kim et al. [2020] Yelin Kim, Joshua Levy, and Yang Liu.  Speech sentiment and customer satisfaction estimation in socialbot conversations.  In _Interspeech 2020_ , 2020.
  * Li et al. [2017] Yanran Li, Hui Su, Xiaoyu Shen, Wenjie Li, Ziqiang Cao, and Shuzi Niu.  Dailydialog: A manually labelled multi-turn dialogue dataset.  In _Proc. of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 986–995, 2017.
  * Liu et al. [2016] Chia-Wei Liu, Ryan Lowe, Iulian Vlad Serban, Mike Noseworthy, Laurent Charlin, and Joelle Pineau.  How not to evaluate your dialogue system: An empirical study of unsupervised evaluation metrics for dialogue response generation.  In _Proc. of the 2016 Conference on Empirical Methods in Natural Language Processing_ , pages 2122–2132, 2016.
  * Lowe et al. [2017] Ryan Lowe, Nissan Pow, Iulian Vlad Serban, Laurent Charlin, Chia-Wei Liu, and Joelle Pineau.  Training end-to-end dialogue systems with the ubuntu dialogue corpus.  _Dialogue & Discourse_, 8(1):31–65, 2017.
  * Mehri and Eskenazi [2020a] Shikib Mehri and Maxine Eskenazi.  Usr: An unsupervised and reference free evaluation metric for dialog.  In _Annual Meeting of the Association for Computational Linguistics, ACL20_ , 2020a.
  * Mehri and Eskenazi [2020b] Shikib Mehri and Maxine Eskenazi.  Unsupervised evaluation of interactive dialog with dialogpt.  In _Proc. of the 21th Annual Meeting of the Special Interest Group on Discourse and Dialogue_ , pages 225–235, 2020b.
  * Omkar Patil and Walker [2021] Kevin K. Bowden Juraj Juraska Wen Cui Vrindavan Harrison Rishi Rajasekaran Angela Ramirez Cecilia Li Eduardo Zamora Phillip Lee Jeshwanth Bheemanpally Rohan Pandey Adwait Ratnaparkhi Omkar Patil, Lena Reed and Marilyn Walker.  Athena 2.0: Discourse and user modeling in open domain dialogue.  _Proc. of the Alexa Prize_ , 2021.
  * Paranjape et al. [2020] Ashwin Paranjape, Abigail See, Kathleen Kenealy, Haojun Li, Amelia Hardy, Peng Qi, Kaushik Ram Sadagopan, Nguyet Minh Phu, Dilara Soylu, and Christopher D Manning.  Neural generation meets real people: Towards emotionally engaging mixed-initiative conversations.  _arXiv preprint arXiv:2008.12348_ , 2020.
  * Pichl et al. [2020] Jan Pichl, Petr Marek, Jakub Konrád, Petr Lorenc, Van Duy Ta, and Jan Šedivỳ.  Alquist 3.0: Alexa prize bot using conversational knowledge graph.  _arXiv preprint arXiv:2011.03261_ , 2020.
  * Prasad et al. [2010] Rashmi Prasad, Aravind K Joshi, and Bonnie L Webber.  Exploiting scope for shallow discourse parsing.  In _LREC_ , 2010.
  * Radford et al. [2019] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever.  Language models are unsupervised multitask learners.  _OpenAI blog_ , 1(8):9, 2019.
  * Rajasekaran [2020] Rishi Rajasekaran.  Transfer-learning approaches towards controllable response generation for conversational agents.  _MS Project Report, UCSC_ , 2020.
  * Reed et al. [2022] Lena Reed, Cecilia Li, Angela Ramirez, Liren Wu, and Marilyn Walker.  Jurassic is (almost) all you need: Few-shot meaning-to-text generation for open-domain dialogue.  In _Conversational AI for Natural Human-Centric Interaction_ , pages 99–119. Springer, 2022.
  * Saha et al. [2021] Sougata Saha, Souvik Das, Elizabeth Soper, Erin Pacquetet, and Rohini K Srihari.  Proto: A neural cocktail for generating appealing conversations.  _arXiv preprint arXiv:2109.02513_ , 2021.
  * See and Manning [2021] Abigail See and Christopher D Manning.  Understanding and predicting user dissatisfaction in a neural generative chatbot.  In _Proceedings of the 22nd Annual Meeting of the Special Interest Group on Discourse and Dialogue_ , pages 1–12, 2021.
  * Shalyminov et al. [2018] Igor Shalyminov, Ondřej Dušek, and Oliver Lemon.  Neural response ranking for social conversation: A data-efficient approach.  In _Proc. of the 2018 EMNLP Workshop SCAI: The 2nd International Workshop on Search-Oriented Conversational AI_ , pages 1–8, 2018.
  * Tao et al. [2018] Chongyang Tao, Lili Mou, Dongyan Zhao, and Rui Yan.  Ruber: An unsupervised method for automatic evaluation of open-domain dialog systems.  In _Proceedings of the AAAI conference on artificial intelligence_ , volume 32, 2018.
  * Whang et al. [2021] Taesun Whang, Dongyub Lee, Dongsuk Oh, Chanhee Lee, Kijong Han, Dong-hun Lee, and Saebyeok Lee.  Do response selection models really know what’s next? utterance manipulation strategies for multi-turn response selection.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 35, pages 14041–14049, 2021.
  * Wolf et al. [2019a] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, R’emi Louf, Morgan Funtowicz, and Jamie Brew.  Huggingface’s transformers: State-of-the-art natural language processing.  _ArXiv_ , abs/1910.03771, 2019a.
  * Wolf et al. [2019b] Thomas Wolf, Victor Sanh, Julien Chaumond, and Clement Delangue.  Transfertransfo: A transfer learning approach for neural network based conversational agents.  _arXiv:1901.08149 [cs]_ , Jan 2019b.  URL http://arxiv.org/abs/1901.08149.  arXiv: 1901.08149.
  * Yeh et al. [2021] Yi-Ting Yeh, Maxine Eskenazi, and Shikib Mehri.  A comprehensive assessment of dialog evaluation metrics.  In _The First Workshop on Evaluations and Assessments of Neural Conversation Systems_ , pages 15–33, 2021.
  * Yi et al. [2019] Sanghyun Yi, Rahul Goel, Chandra Khatri, Alessandra Cervone, Tagyoung Chung, Behnam Hedayatnia, Anu Venkatesh, Raefer Gabriel, and Dilek Hakkani-Tur.  Towards coherent and engaging spoken dialog response generation using automatic conversation evaluators.  In _Proc. of the 12th International Conference on Natural Language Generation_ , pages 65–75, 2019.
  * Zhang et al. [2021] Chen Zhang, Yiming Chen, Luis Fernando D’Haro, Yan Zhang, Thomas Friedrichs, Grandee Lee, and Haizhou Li.  Dynaeval: Unifying turn and dialogue level evaluation.  In _Proc. of the 59th Annual Meeting of the ACL and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 5676–5689, 2021.
  * Zhang et al. [2022] Chen Zhang, Luis Fernando D’Haro, Qiquan Zhang, Thomas Friedrichs, and Haizhou Li.  Fined-eval: Fine-grained automatic dialogue-level evaluation.  In _Proceedings of EMNLP 2022_ , 2022.
  * Zhang et al. [2018] Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur Szlam, Douwe Kiela, and Jason Weston.  Personalizing dialogue agents: I have a dog, do you have pets too?  In _Proc. of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 2204–2213, 2018.
  * Zhang et al. [2019] Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao, Jianfeng Gao, Jingjing Liu, and Bill Dolan.  Dialogpt: Large-scale generative pre-training for conversational response generation, 2019.