Ray-Singer Torsion, Topological Strings and Black Holes

Cumrun Vafa

Jefferson Physical Laboratory, Harvard University, Cambridge, MA 02138, USA

Genus one amplitude for topological strings on Calabi-Yau 3-folds can be
computed using mirror symmetry: The partition function at genus one gets
mapped to a holomorphic version of Ray-Singer torsion on the mirror Calabi-
Yau. On the other hand it can be shown by a physical argument that this gives
a curvature squared correction term to the gravitational action. This in
paticular leads to an effective quantum gravity cutoff known as the species
scale, which varies over moduli space of Calabi-Yau manifolds. This resolves
some of the puzzles associated to the entropy of small black holes when there
are a large number of light species of particles. Thus Ray-Singer torsion, via
its connection to topological strings at genus one, provides a measure of
light degrees of freedom of four dimensional ${\cal N}=2$ supergravity
theories.

Based on a talk given on May 12th, 2023 at the Singer Memorial Conference,
MIT.

## Isadore Singer

Isadore Singer was not only a great mathematician, but also a science
visionary who appreciated the importance of deep links between modern
mathematics and theoretical physics. Moreover, he played an active role in
fostering the development of a new branch of science which is at the interface
of both fields. On the one hand this equips physicists with powerful
mathematical tools to explore laws of nature at its most fundamental level and
on the other hand it provides physical insights to mathematicians in
uncovering and connecting different areas of mathematics.

I have been fortunate to have also known Is as a friend and to discuss these
topics with him frequently. Moreover I have many fond memories of our joint
efforts which helped establish the Simons Center for Geometry and Physics in
Stony Brook.

The talk I presented in the Singer Memorial conference at MIT, was motivated
by giving a concrete example of showing this deep back and forth connection
between physics and mathematics. The example I presented uses a beautiful work
of Singer (Ray-Singer torsion [1, 2]) and its relation to Gromov-Witten
invariants (topological strings) via mirror symmetry. All these ideas have
found applications in the past year in the context of black hole physics and
positive potentials that can appear in physical theories of quantum gravity
[3, 4], which I review in this paper.

##  1 Introduction

Ray-Singer Torsion [1, 2], a concept intertwining mathematics and physics, has
recently become important in understanding quantum gravity and string theory.
We first briefly review its historical development and then discuss its
significance in modern theoretical physics through a particular example.

##  2 Ray-Singer Torsion in Compact Kahler and Calabi-Yau Manifolds

Among the many important works that Singer has done, here I will describe an
application of an object which arises in physics and closely parallels what
Ray and Singer discovered. They considered [1, 2] an arbitrary Riemannian
manifold M (together with a vector bundle E), and defined

$T(M,E)=\prod_{p}\rm det(\Delta_{p})^{\frac{-(-1)^{p}p}{2}}$

where $\Delta_{p}$ is the Laplacian acting on $p$-forms, and the determinant
is made precise by zeta function regularization. It was conjectured by Ray and
Singer that this invariant is an analytic version and equivalent to the
Reidemeister torsion [5]. This was later proven by Cheeger [6, 7] and Muller
[8].

In this talk I will discuss a closely related object, which is a holomorphic
version of Ray-Singer torsion that arises for compact Kahler manifolds (and
more specifically Calabi-Yau manifolds). This is “invariant”, in the sense
that it does not depend on the Kahler metric, but does depend on the complex
structure.

We will encounter a particular combination of the holomorphic Ray-Singer
torsion:

$\prod_{p,q=0}^{n}({\rm det}\ \Delta_{p,q})^{\frac{(-1)^{p+q}pq}{2}}$

where $\Delta_{p,q}$ denotes Laplacian acting on $(p,q)$ forms. We will
explain how this ends up ‘counting’ the number of light particles by invoking
the physics of black holes.

My main emphasis in this note is to provide an example of how different ideas
in physics and mathematics intermingle to lead to insights for both fields. On
the physics side we explore aspects of black holes and their information
content, and on the joint mathematical and physical side consider Gromov-
Witten invariants (also known as topological string amplitudes) on Calabi-Yau
threefolds. Also mirror symmetry between Calabi-Yau threefolds enters this
story, where the Gromov-Witten invariants on a given Calabi-Yau are computed
by going to the mirror and doing the corresponding computation there, which
turns out to be related to Ray-Singer torsion. Thus the holomorphic version of
Ray-Singer torsion ends up computing Gromov-Witten invariants for genus 1
curves. On the physics side, insights on the structure of light fields in
quantum gravitational theories, including bounds on the nature of positive
potentials $V>0$ that can appear in physical theories, all get connected to
Ray-Singer torsion.

This talk is based on old and new works. In particular the old work with
Bershadsky, Cecotti and Ooguri [9] and the new work with van de Heisteeg,
Wiesner and Wu [3, 4]. Road-map for the rest of this note is as follows. In
section 3 we explain the sense in which the Planck scale is viewed as the
short distance cutoff of quantum gravity theories. In section 4 we briefly
explain the Bekenstein-Hawking entropy of black holes. In section 5 we explain
why having many light species raises a puzzle and a resolution is offered by
replacing the Planck scale cutoff with the Species scale cutoff. In section 6
some aspects of string dualities are reviewed with particular attention to how
the light states appear as we approach boundaries of moduli spaces. In section
7 we review the relevance of one-loop topological strings (genus 1 Gromov-
Witten invariants) to certain $R^{2}$ curvature corrections in the action.
Moreover it is shown how mirror symmetry can be used to relate this to a
holomoprhic version of Ray-Singer torsion and how this leads to its
computation. We use this to find the dependence of the species scale on moduli
which leads to interesting bounds on the allowed potential in physical
theories. We also explain why bounds on the shape of potential is an important
topic for theoretical physics today. In section 8 we end with some concluding
remarks.

##  3 Planck Scale as a UV Quantum Gravity Cutoff

In quantum gravity, the leading term of the action is the Einstein-Hilbert
action. However we also expect corrections to the Einstein-Hilbert action:

$S=\int
d^{d}x\,\sqrt{-g}\left[\frac{M_{pl}^{d-2}}{2}\left(R+\sum_{n}\,\frac{R^{n}}{M_{pl}^{2n-2}}+\dots\right)\right]$

Here we have put the fundamental mass scale, the Planck mass $M_{pl}$ to make
the dimensions right, and we expect the coefficients of each term to be of
$O(1)$, which we have not explicitly written. Note that curvature has
dimension of mass squared. Of course not all higher derivative terms will have
coefficients of $O(1)$, but that should be the generic case. This implies that
when curvature $R\sim M_{pl}^{2}$ the perturbation series breaks down and the
geometric description given by the leading Einstein-Hilbert term, breaks down.

We should thus view $\Lambda_{UV}\sim M_{pl}$ as the high energy ultra-violet
cutoff where geometric description breaks down. This means that since length
has inverse dimension to mass, the Planck length $l_{pl}=M_{pl}^{-1}$ should
be viewed as the shortest geometric length scale in the theory.

##  4 Basics of Black Hole Entropy

Black hole is one of the most enigmatic objects not only in nature, but also
for theoretical physics. One of the most important properties of black holes,
which was discovered half a century ago, was the work of Bekenstein and
Hawking who found that when you include quantum corrections a black hole can
be consistent only if it represents not one but many potential objects; i.e.
that it has entropy. Moreover the entropy of the black hole is proportional to
the area of its event horizon (which is the point beyond which no light can
classically escape to infinity). This was originally motivated by the
classical observation of the fact that the area of the black hole horizons
increase over time and made precise by combining with semi-classical
properties of black holes. More precisely if we denote by $A$ the area of the
event horizon measured in Planck units, the entropy of the black hole is given
by

$S_{BH}=\frac{1}{4}A\sim\bigg{(}\frac{R_{BH}}{l_{pl}}\bigg{)}^{d-2}$

The entropy signifies the number of micro-states of a black hole. If we assume
a black hole is made of $N$ independent constituents then this is the same as
expecting $N$ to grow with the radius of the black hole by

$S_{BH}\sim N\sim R_{BH}^{d-2}.$

We will now see why this leads to a contradiction with the expectation that
the Planck scale is the quantum gravity cutoff, when we have many light
species of particles.

##  5 Light Species Puzzle for Black Holes

Suppose we have $N\gg 1$ of light ($m\ll M_{pl}$) states. Then for any black
hole bigger than Planck scale (so that we can ignore the corrections to the
Einstein-Hilbert action), we would expect

$S>N,$

as the light particles can constitute the degrees of freedom of the black
hole.

On the other hand, if we assume the size of the smallest black hole
describable reliably by the Einstein-Hilbert action is Planck length, as one
would naively expect based on quantum gravity, this would suggest $S\sim 1$
for this black hole which is inconsistent with the above entropy bound. This
is the species problem which we now review. Species scale [10] was offered as
a solution to this problem.

###  5.1  Species Scale

Suppose we have a large number $N$ of light species. Then it better be that
gravitational action breaks down before we get to Planck scale:

$S_{BH}\sim 1\quad{\rm for}\ R_{BH}\sim l_{Pl}$

because this would not accommodate $N$ degrees of freedom. Indeed the smallest
size black hole compatible with such an entropy has a larger radius (using
Bekenstein-Hawking entropy):

$\bigg{(}\frac{R_{BH}^{min}}{l_{pl}}\bigg{)}^{d-2}\sim
N\Rightarrow\frac{R_{BH}^{min}}{l_{pl}}\sim N^{\frac{1}{d-2}}.$

How could this be explained? A natural explanation of this would be that
higher derivative corrections to the action are important and compete at the
mass scale

$\Lambda_{s}=\frac{1}{R_{BH}^{min}}\sim\frac{M_{pl}}{N^{\frac{1}{d-2}}}.$

$\Lambda_{s}$ is called the species scale. We thus have a new UV cutoff for
gravity: $\Lambda_{s}<M_{pl}$.

Figure -659: The minimal size of the black hole which can be reliably treated
using the effective action is $1/\Lambda_{s}$ which is larger than the Planck
length $1/M_{pl}$.

In other words

$\bigg{(}\frac{M_{pl}}{\Lambda_{s}}\bigg{)}^{d-2}=N$

is a measure of the number of light degrees of freedom in the theory. The
smaller the $\Lambda_{s}$ the more the number of light states. We thus expect
$\Lambda_{s}$ to replace $M_{pl}$ in higher derivative correction terms:

$S=\int d^{d}x\,\sqrt{-g}\left[\frac{M_{\rm
pl}^{d-2}}{2}\left(R+\sum_{n}\,\frac{R^{n}}{\Lambda_{s}^{2n-2}}+\dots\right)\right]$

##  6 Dualities and Moduli Space

One of the basic features of string theory is the existence of duality
symmetries in the theory. This means that if you change the parameters in the
theory to extreme values, you get a new description of the theory. The
mechanism for this is that at extreme values of parameters light tower of
particles appear that lead to a new weakly interacting description of the
theory. The parameters of the theory are typically moduli of manifolds where
strings are propagating in. Let us parameterize the moduli space of
compactified space by the field $\phi$ (which can be viewed as a coordinate
system on the moduli space). Moreover there is a natural metric $g_{ij}(\phi)$
on this space arising from the kinetic term in the action:

$S=\frac{1}{2}\int g_{ij}(\phi)\ \nabla\phi^{i}\cdot\nabla\phi^{j}+...$

Figure -658: At extreme corners of the moduli space a light tower of particle
appears whose mass scales down exponentially with distance. The effective
number of light fields $N$ increases as we approach the boundaries. These
light fields lead to the dual description of the theory.

Example: Consider compactification of IIA string on Calabi-Yau 3-fold. The
Kahler and complex moduli of CY correspond to massless fields of the 4
dimensional theory. Let us focus on Kahler moduli $k=e^{\varphi}k_{0}$. As
$\varphi\gg 1\Rightarrow m\sim{\rm
exp}\big{[}-\sqrt{\frac{3}{2}}\varphi\big{]}$ these come from low lying
eigenvalues of Laplacian (KK modes) in large volume. Another limit is of the
following type: Consider K3 fibered CY3-fold, and consider the limit the
volume of K3 fiber goes to zero. In this limit we get tensionless strings
coming from 5-branes wrapped on shrinking K3:

$Vol(K3)=e^{-\varphi},\qquad\varphi\gg 1\Rightarrow m\sim{\rm
exp}\big{[}-\frac{\varphi}{{\sqrt{2}}}\big{]}$

where the tension of the string (in lower dimensional Planck units) is $T\sim
m^{2}$. In general the exponent of the decay of the mass with field space
distance satisfies the bound:

$\sqrt{\frac{D-2}{(D-d)(d-2)}}\geq\alpha\geq\frac{1}{\sqrt{(d-2)}}$

where we consider theories compactified on a $D-d$ dimensional manifold down
to $d$ dimensions. In this bound $D\leq 11$ which is the maximum dimension for
M-theory. The lower range of the bound gets saturated when we get a
tensionless string.

This implies the species scale will depend on the value of massless fields
$\varphi$, i.e., $\Lambda_{s}(\varphi)$ (since as we change $\varphi$ the
masses change and so the species scale could change). We thus expect

$S=\int d^{d}x\,\sqrt{-g}\left[\frac{M_{\rm
pl}^{d-2}}{2}\left(R+\sum_{n}\,\frac{R^{n}}{\Lambda_{s}^{2n-2}(\varphi)}+\dots\right)+\frac{1}{2}|\partial\varphi|^{2}+\dots\,\right]$

We are interested in finding the dependence of $\Lambda_{s}$ on $\varphi$. Let
us focus on $d=4$ and the $R^{2}$ term, which is accompanied by
$\frac{1}{\Lambda_{s}^{2}(\varphi)}\sim N(\varphi)$. How to compute this term
$\int d^{4}x\ N(\varphi)\cdot R^{2}$ ?

One can compute this term in string perturbation theory but then it will not
be robust: As we go to strong coupling regime of string theory, the
corrections would be difficult to compute as we would need to include not only
the contributions coming from all genera, but also non-perturbative
contributions. However, there is a magical $R^{2}$-type correction which can
be computed using string perturbation techniques and that it receives no
quantum corrections and it is exact! To explain this we need to turn to
topological strings.

##  7 Topological Strings on CY 3-folds and dependence of Species scale on
moduli

Topological strings on Calabi-Yau 3-folds calculates the Gromov-Witten
invariants, which roughly speaking ‘counts’ holomorphic maps to the CY (for
more precise definition see e.g. [11]). More specifically, the generating
function of this invariant is the partition function of the topological string
on the corresponding CY.

Figure -657: Gromov-Witten invariants ‘count’ holomorphic maps from curves to
Calabi-Yau manifolds.

The topological string partition function at genus g is given by

$F_{g}(k)=\sum_{\bar{\partial}X=0}{\rm exp}(-\int_{\Sigma^{g}}X^{*}(k))$

Where $k={\rm Kahler\ form}$ of CY and $\Sigma^{g}$ is the ${\rm genus\ g\
curve}$. The topological string partition function compute physical terms for
superstring action. In particular $F_{1}$ computes in the four dimensional
action, the term given by

$\int\ F_{1}(k)\cdot(R^{-})^{2}$

where $R^{-}$ is the anti-self-dual part of the curvature. Moreover this term
is exactly computable as it receives no further corrections! In other words
the genus 1 computation of the topological string yields the exact answer for
this quantity! We can thus interpret this quantity in the four dimensional
theory, since $\frac{1}{\Lambda_{s}^{2}}\sim N$, as implying:

$F_{1}(k)\sim N(k).$

So to complete the circle of ideas we need to know how to compute $F_{1}$, the
genus one topological string amplitude? Here we can use mirror symmetry
between Type IIA strings on CY3 with Type IIB on Mirror CY3’ (See [11] for
more detail on this). In particular note that

$h^{p,q}(CY)=h^{p,3-q}(CY^{\prime}).$

This maps the Kahler parameters of CY $k$ (parameterized by $h^{1,1}$) to
complex moduli $t$ of CY’ (parameterized by $h^{1,2}$). Therefore Topological
String on CY which is parameterized by Kahler deformations gets mapped to
complex deformation of CY’.

$F_{g}(k)\Rightarrow F_{g}(t)$

The latter is computed using the topological B-model, which is the ‘quantum
Kodaira-Spencer theory of gravity’ [9]. This in turn is much simpler than
counting curves and is given in terms of a field theory computation, whose
classical equations lead to Kodaira-Spencer equations for the deformation of
complex structure. The genus one amplitude for topological string thus gets
mapped to one-loop corrections of the field theory, which in turn, as in all
field theories, is given by certain combination of determinants.

$F_{1}\ \ \ \ \ \ \ \ \ \Rightarrow\ {\rm one\ loop\ correction}$

Figure -656: The genus 1 Gromov-Witten invariants get mapped using mirror
symmetry to one loop amplitudes of Kodaira-Spencer theory, which in turn is
given by a holomorphic version of Ray-Singer torsion.

This combination of determinants happens to be a combination of holomorphic
Ray-Singer torsion:

$e^{F_{1}}=\prod_{p,q=0}^{3}({\rm det}\
\Delta_{p,q})^{\frac{(-1)^{p+q}pq}{2}}$

which satisfies [12]

${\partial\bar{\partial}}F_{1}=\text{sTr}\ C\bar{C}-\frac{1}{12}G\cdot\chi\,,$

where $C$ is a certain natural action of the tangent of moduli on the harmonic
forms of the CY. This result can also be seen [13, 14] as an application of
Quillen’s work for curvature of the determinant line bundle). Integrating the
above equation yields

$F_{1}=\frac{1}{2}\left(3+h^{1,1}-\frac{\chi}{12}\right)K\,-\frac{1}{2}\log\det
G+\log|f|^{2}\,,$

Where K is the Kahler function on complex structure moduli space, $\chi$ is
the Euler character of CY, G is the metric on moduli space and $f$ is a
holomorphic function of moduli.

To fix everything we need to find the holomorphic function $f$ which is fixed
by the behavior expected at the boundary (which in turn is determined by using
mirror symmetry).

A specially easy case is the CY 3-fold given by a $Z_{2}$ quotient of the
product of $K3$ and the elliptic curve where the $Z_{2}$ acts as inverstion on
$T^{2}$ and acts freely on $K3$ taking the holomorphic 2-form to its negative.
This yields for $CY^{3}=K3\times T^{2}/Z_{2}$:

$F_{1}=-12\ {\rm
log}\big{[}\sqrt{\tau}_{2}\eta(\tau)\eta(\bar{\tau})\big{]}\sim
N\sim\frac{1}{\Lambda_{s}^{2}}$

$N\sim\tau_{2}\sim{\rm exp}(\sqrt{2}\ \varphi)\ \ {\rm for}\ \tau_{2}\gg 1$

consistent with what one expects for the behaviour of $\Lambda_{s}$ and $N$
near the boundary of moduli space.

From this example and many others we find the general picture that the species
scale decays exponential at the boundaries and reaches a maximum in the
interior of moduli space. Moreover, we find that the exponential fall off of
$\Lambda_{s}$ with distance on moduli gives the expected exponent:

Figure -655: A typical behaviour of $\Lambda_{s}^{2}$ as a function of moduli
fields: exponential decay at infinity and a maximum of order 1 in the
interior.

###  7.1 Application to Bounds on positive potentials

One of the most challenging and phenomenologically interesting questions in
quantum gravity is whether we can say anything about the nature of positive
potentials $V>0$. This is particularly difficult because such situations
necessarily imply that supersymmetry is broken and we thus have no exact
techniques to compute physical amplitudes. However, for at least two reasons
putting bounds on $V>0$ is crucial for cosmology: An important question for
early cosmology in the context of inflation is whether long ranged flat
positive potentials $V>0$ can exist and if there are any restrictions on it.

Figure -654: Long flat regions of potential are needed for generic
inflationary model.

We now argue that for flat regions of the potential

$V(\varphi)<\Lambda^{2}_{s}(\varphi)$

This can be argued as follows: For flat regions of potential, the space is
approximately de Sitter (homogenous space with positive constant curvature)
and the maximal size of the space is given by the horizon size

$R_{max}\sim\frac{1}{\sqrt{V}}$

Figure -653: Using the exponential behaviour of the species scale near the
boundaries we can get a bound on the width of regions with flat potential.

As we already argued the minimum distance for which we can use the effective
action is the species scale distance. This leads to a bound on $V$:

$R_{min}\sim\frac{1}{\Lambda_{s}}<R_{max}\sim\frac{1}{\sqrt{V}}\Rightarrow
V<\Lambda_{s}^{2}$

Using this, and the asymptotic nature of the species scale, we can get bounds
on length of regions of flat regions for V from $V<\Lambda_{s}^{2}$. In
particular we learn that high flat regions of potential lead to small range
and the only way to get larger ranges is to decrease the potential:

$\Delta\varphi<a.{\rm log}\bigg{[}\frac{1}{V}\bigg{]}-b$

where

$a\leq\sqrt{(d-2)(d-1)};\ \ b\sim O(1)$

##  8 Conclusion

We have seen that the species scale is an important concept in quantum
gravitational theories and can depend on moduli of light fields. Topological
strings and mirror symmetry lead to a computation of the species scale: the
one-loop topological amplitude counts the light degrees of freedom which in
turn is given by a holomorphic version of Ray-Singer Torsion. This also leads
to bounds on the range of flat regions of potential which scale
logarithmically with the inverse height of the potential [4].

The intertwined mathematical and physical ideas that were discussed here
provide an example of how the old work of Singer and his vision continue to
impact the development of physics and mathematics today.

#### Acknowledgments

I thank my collaborators, whose work I presented here. This work is supported
by a grant from the Simons Foundation (602883,CV), the DellaPietra Foundation,
and by the NSF grant PHY-2013858.

## References

  * [1] D.B. Ray and I.M. Singer, _R-torsion and the laplacian on riemannian manifolds_ , _Advances in Mathematics_ 7 (1971) 145.
  * [2] D.B. Ray and I.M. Singer, _Analytic torsion for complex manifolds_ , _Annals of Mathematics_ (1973) 154.
  * [3] D. van de Heisteeg, C. Vafa, M. Wiesner and D.H. Wu, _Moduli-dependent Species Scale_ , 2212.06841.
  * [4] D. van de Heisteeg, C. Vafa and M. Wiesner, _Bounds on Species Scale and the Distance Conjecture_ , _Fortsch. Phys._ 71 (2023) 2300143 [2303.13580].
  * [5] K. Reidemeister, _Homotopieringe und linsenräume_ , _Abhandlungen aus dem Mathematischen Seminar der Universität Hamburg_ 11 (1935) 102.
  * [6] J. Cheeger, _Analytic torsion and reidemeister torsion_ , _Proc. Nat. Acad. Sci. USA_ 74 (1977) 2651.
  * [7] J. Cheeger, _Analytic torsion and the heat equation_ , _Annals of Mathematics_ 109 (1979) 259.
  * [8] W. Müller, _Analytic torsion and r-torsion of riemannian manifolds_ , _Advances in Mathematics_ 28 (1978) 233.
  * [9] M. Bershadsky, S. Cecotti, H. Ooguri and C. Vafa, _Kodaira-spencer theory of gravity and exact results for quantum string amplitudes_ , _Communications in Mathematical Physics_ 165 (1994) 311.
  * [10] G. Dvali, _Black Holes and Large N Species Solution to the Hierarchy Problem_ , _Fortsch. Phys._ 58 (2010) 528 [0706.2050].
  * [11] K. Hori, S. Katz, A. Klemm, R. Pandharipande, R. Thomas, C. Vafa et al., _Mirror Symmetry_ , vol. 1 of _Clay Mathematics Monographs_ , American Mathematical Society and Clay Mathematics Institute, Providence, RI (2003).
  * [12] M. Bershadsky, S. Cecotti, H. Ooguri and C. Vafa, _Holomorphic anomalies in topological field theories_ , _Nucl. Phys. B_ 405 (1993) 279 [hep-th/9302103].
  * [13] J.-M. Bismut and D. Freed, _The analysis of elliptic families. i. metrics and connections on determinant bundles_ , _Communications in Mathematical Physics_ 106 (1986) 159.
  * [14] J.-M. Bismut and D. Freed, _The analysis of elliptic families. ii. dirac operators, eta invariants, and the holonomy theorem_ , _Communications in Mathematical Physics_ 107 (1986) 103.


# Context-aware Decoding Reduces Hallucination in Query-focused Summarization

Zhichao Xu
<EMAIL_ADDRESS>
University of Utah

###### Abstract

Query-focused summarization (QFS) aims to provide a summary of a single
document/multi documents that can satisfy the information needs of a given
query. It is useful for various real-world applications, such as abstractive
snippet generation or more recent retrieval augmented generation (RAG). A
prototypical QFS pipeline consists of a retriever (sparse or dense retrieval)
and a generator (usually a large language model). However, applying large
language models (LLM) potentially leads to hallucinations, especially when the
evidence contradicts the prior belief of LLMs. There has been growing interest
in developing new decoding methods to improve generation quality and reduce
hallucination. In this work, we conduct a large-scale reproducibility study on
one recently proposed decoding method—Context-aware Decoding (CAD). In
addition to replicating CAD’s experiments on news summarization datasets, we
include experiments on QFS datasets, and conduct more rigorous analysis on
computational complexity and hyperparameter sensitivity. Experiments with
eight different language models show that performance-wise, CAD improves QFS
quality by (1) reducing factuality errors/hallucinations while (2) mostly
retaining the match of lexical patterns, measured by ROUGE scores, while also
at a cost of increased inference-time FLOPs and reduced decoding speed. The
code implementation based on Huggingface Library is made available
111https://github.com/zhichaoxu-shufe/context-aware-decoding-qfs..

##  1 Introduction

Query-focused summarization (QFS) aims to provide a summary of a single
document/multiple documents that can satisfy the information need of a given
query [XC23, Mar06]. It is useful for various real-world applications, such as
abstractive snippet generation [Che+20] or more recent retrieval augmented
generation (RAG) [Lew+20]. A prototypical QFS pipeline consists of a retriever
to retrieve the documents to use as evidence, and a generator (usually based
on large language models) to read the documents and generate natural language
answers/summaries.

While the idea of combining large language models (LLM) and information
retrieval seems tempting [Met+21], a few problems persist. For example,
deploying generative models requires more inference FLOPs and specialized
hardwares (GPUs or TPUs); and more importantly, the usage of LLMs in QFS/RAG
pipeline also potentially leads to the hallucination problem where the
generated summary contains information contradicting the source
documents/evidence [SB22, Ben+21], which might hurt users’ experience in
information seeking processes and potentially escalate the spread of
misinformation. As such, mainstream search engines (Google, Bing, etc.) still
deploy extractive snippets [BC14].

In parallel to training better language models to reduce hallucinations, there
has been growing interest in developing decoding methods to allow the language
models to better attend to evidence or to penalize degeneration, thus
improving generation quality. For example, [Li+23] propose contrastive
decoding, leveraging the output probability distribution difference between an
expert LM and an amateur LM to improve generation quality. DExperts [Liu+21]
focus on reducing toxicity by contrasting the expert LM with toxicity LM.
[XC23] show that directly injecting knowledge constraints can potentially
produce more faithful summarization. A few works [Shi+23, PCM22] make use of
pointwise mutual information to let the LM focus more on the input context to
reduce hallucination.

In this work, we focus on Context-aware Decoding [Shi+23] (CAD). CAD leverages
the idea of pointwise mutual information and proposes a product-of-experts
enhancement to the vanilla decoding method to make the generation more
conditioned on the input evidence. We conduct a large-scale reproducibility
study with different choices of language models to study CAD’s effectiveness
in query-focused summarization, and include more rigorous analysis on
computational complexity and hyperparameter sensitivity. Extensive experiments
show that CAD improves QFS quality by (1) reducing factual
mistakes/hallucinations while (2) mostly retaining the match of lexical
patterns, measured by ROUGE scores. On the other hand, CAD also introduces
additional inference-time FLOPs and potentially slows down decoding speed.

##  2 Background

#### Context-aware Decoding.

Consider a language model parameterized by $\theta$, a context $\mathbf{c}$,
an input query $\mathbf{x}$, and we want to generate a piece of text
$\mathbf{y}$. In QFS paradigm, $\mathbf{c}$ can be the source document,
$\mathbf{x}$ and $\mathbf{y}$ will be the query and summary, respectively. In
the vanilla decoding scenario, each token of generation $\mathbf{y}$ is
sampled autoregressively from the output probability distribution from
language model $\theta$ conditioned on input query $\mathbf{x}$ and context
$\mathbf{c}$:

$y_{t}\sim
p_{\theta}(y_{t}|\mathbf{c},\mathbf{x},\mathbf{y}_{<t})\propto\text{Softmax}\big{(}\,\text{logit}_{\theta}(y_{t}|\mathbf{c},\mathbf{x},\mathbf{y}_{<t})/\tau\big{)}$
(1)

Here $\tau$ is a temperature hyperparameter to control the sharpness of
probability distribution over the candidate vocabulary $\mathcal{V}$. Examples
of the sampling strategy include greedy and Top-$p$ sampling [Hol+19].

As we are operating on a conditional generation problem, we hope the
generation is more conditioned on input context $\mathbf{c}$. [Shi+23] utilize
the idea of pointwise mutual information (PMI). We consider two events
$\mathbf{c}$ and $y_{t}$:

$\mathbf{PMI}(p_{\theta}(y_{t};\mathbf{c},\mathbf{x},\mathbf{y}_{<t}))=\log\Big{(}\frac{p_{\theta}(y_{t}|\mathbf{c},\mathbf{x},\mathbf{y}_{<t})}{p_{\theta}(y_{t}|\mathbf{x},\mathbf{y}_{<t})}\Big{)}$
(2)

The interpretation of the above equation is we would like to measure the
association of event $y_{t}$ (predicting a specific token) and event
$\mathbf{c}$ (the presence of context $\mathbf{c}$).
$p_{\theta}(y_{t}|\mathbf{x},\mathbf{y}_{<t})$ is the prior probability,
representing the model’s prior belief from its parameters $\theta$ when no
presence of context $\mathbf{c}$, whereas
$p_{\theta}(y_{t}|\mathbf{c},\mathbf{x},\mathbf{y}_{<t})$ is the posterior
probability with $\mathbf{c}$. To leverage PMI, [Shi+23] propose to multiply
PMI with the original probability distribution:

$y_{t}\sim\tilde{p}_{\theta}(y_{t}|\mathbf{c},\mathbf{x},\mathbf{y}_{<t})\propto
p_{\theta}(y_{t}|\mathbf{c},\mathbf{x},\mathbf{y}_{<t})\Big{(}\frac{p_{\theta}(y_{t}|\mathbf{c},\mathbf{x},\mathbf{y}_{<t})}{p_{\theta}(y_{t}|\mathbf{x},\mathbf{y}_{<t})}\Big{)}^{\alpha}$
(3)

Here $\alpha$ is a hyperparameter to control the PMI weighting. We still need
to normalize the above equation to acquire a valid probability distribution,
rearranging the terms yields

$y_{t}\sim\text{Softmax}\big{[}\big{(}(1+\alpha)\,\text{logit}_{\theta}(y_{t}|\mathbf{c},\mathbf{x},\mathbf{y}_{<t})-\alpha\,\text{logit}_{\theta}(y_{t}|\mathbf{x},\mathbf{y}_{<t})\big{)}/\tau\big{]}$
(4)

Here larger $\alpha$ yields more weight on the context $\mathbf{c}$ and
$\alpha=0$ reduces to vanilla decoding. [Shi+23] recommend setting
$\alpha=0.5$ for summarization tasks and we also examine this choice in
Section 3.

#### Analysis of Computational Cost.

We follow the notations used in [Kap+20]. Let us consider the Transformer
architecture with following hyperparameters: number of layers
$n_{\text{layer}}$, dimension of the embedding layer and intermediate
representation $d_{\text{model}}$, dimension of the intermediate feedforward
layer $d_{\text{ff}}$, dimension of the attention output $d_{\text{attn}}$ and
number of attention heads $n_{\text{heads}}$; and denote the number of tokens
in the input sequence as $n_{\text{input}}$. The model size $N$ is the number
of _non-embedding_ parameters of the model, $N\approx|\theta|$:

$N\approx
2\,d_{\text{model}}\,n_{\text{layer}}(2\,d_{\text{attn}}+d_{\text{ff}})=12\,n_{\text{layer}}\,d_{\text{model}}^{2}\,\,\,\,\text{with}\,\,d_{\text{attn}}=d_{\text{ff}}/4=d_{\text{model}}$
(5)

The total FLOPs per token in one forward pass is approximately
$C_{\text{forward}}=2N+2n_{\text{layer}}\,n_{\text{input}}\,d_{\text{attn}}$
(see Section 2.1 in [Kap+20] for details). In LLMs, $d_{\text{model}}\gg
n_{\text{input}}/12$, we can roughly ignore the second term and use
$C_{\text{forward}}\approx 2N$.

With the above analysis, we know that performing one forward pass of CAD
yields in total $(2\,|\mathbf{y}_{<t}|+2\,|\mathbf{x}|+|\mathbf{c}|)\cdot
C_{\text{forward}}$ FLOPs compared to
$(|\mathbf{y}_{<t}|+|\mathbf{x}|+|\mathbf{c}|)\cdot C_{\text{forward}}$ from
vanilla decoding.

##  3 Experiments

We carry out experiments on two QFS datasets and two news summarization
datasets, with different choices of backbone language models, including pre-
trained language models and instruction finetuned language models.

###  3.1 Prompting Templates

For QFS task, we treat the source document as context $\mathbf{c}$, and treat
the original query as query $\mathbf{x}$ and prompt the model to generate
summarization $\mathbf{y}$. For the summarization task, the source document is
used as context $\mathbf{c}$, and we manually append a query $\mathbf{x}$ in
the shape of [summarize the following document:]. The detailed prompting
templates can be found in Appendix 7.1.

###  3.2 Experiment Setup

#### Datasets.

We benchmark CAD’s performance on two widely adopted QFS datasets. Debatepedia
dataset is collected by [Nem+17] from 663 debates of 53 diverse categories in
an encyclopedia of debates and consists of 12K/0.7K/1.0K query-document
summarization triplets $(\emph{query},\emph{document},\emph{summarization})$.
PubMedQA [Jin+19] is a long-form abstractive question-answering dataset from
the biomedical domain with the contexts available. We also use two news
summarization datasets from the original CAD paper: CNN Dailymail [SLM17] and
XSUM [NCL18] for reproducibility purposes.

#### Language Models.

We study language models of different architecture choices (encoder-decoder
models vs decoder-only models) and training paradigms (pre-trained vs
instruction finetuned).

  * •

LLaMA-7B [Tou+23] is a decoder-only language model pre-trained with causal
language modeling objective from FAIR.

  * •

Llama-2-7B [Tou+23] is the second iteration of LLaMA model by better pre-
training corpus and minor architectural improvements.

  * •

MPT-7B [Tea23] is a decoder-only language model pre-trained with causal
language modeling objective from MosaicML

  * •

MPT-7B-instruct is based on MPT-7B and further instruction finetuned with an
instruction finetuning dataset mosaicml/instruct-v1.

  * •

Mistral-7B-v0.1 [Jia+23] is a decoder-only language model pre-trained with
causal language modeling objective from Mistral AI.

  * •

Mistral-7B-instruct-v0.1 is based on Mistral-7B-v0.1 and further instruction
finetuned with an internal dataset.

  * •

Flan-T5-xl [Chu+22] (3B) is an encoder-decoder model from T5 family [Raf+20];
it is pre-trained with span-infilling objective and further adapted with
causal language modeling objective. It is instruction finetuned on the Flan
collection.

  * •

Flan-T5-xxl [Chu+22] (11B) undergoes same training procedure as Flan-T5-xl and
demonstrates strong performance on natural language understanding tasks.

We acquire weights of all listed models from Huggingface
library222https://huggingface.co/models.

#### Evaluation.

Similar to [Shi+23], we evaluate the generation quality with ROUGE F1 [Lin04],
BERTScore-precision [Zha+19] and FactKB [Fen+23]. ROUGE measures the match of
lexical patterns while BERTScore measures the semantic similarity between the
generation and the reference. FactKB is a model-based evaluation metric that
classifies whether the generation is faithful to the document. For BERTScore
we use roberta-large-mnli as it shows a higher correlation with human
judgments. For FactKB we use the original classifier trained by the authors
[Fen+23].

#### Hyperparameters

Our experimental settings follow [Shi+23] to a large extent. For decoding, we
set temperature=1, use a combination of Top-$k$ and Top-$p$ sampling, and vary
$\alpha$ to study the effectiveness of CAD. For the main results reported in
Section 4, we use $\alpha=0.3$ and use 0.15, 0.5 as an ablation study.
Detailed choice of hyperparameters can be found in Appendix 7.2.

##  4 Results and Analysis

Table 1: Vanilla decoding vs CAD with news summarization datasets.

Datasets | Model | Decoding | ROUGE-1 | ROUGE-2 | ROUGE-L | BERTScore-P | FactKB
---|---|---|---|---|---|---|---
CNN Dailymail | LLaMA-7B | Vanilla | 27.6 | 8.7 | 24.5 | 73.0 | 85.8
CAD | 30.4 | 10.8 | 27.1 | 73.5 | 90.9
Llama-2-7B | Vanilla | 26.8 | 8.6 | 23.9 | 72.3 | 80.6
CAD | 29.3 | 10.3 | 26.2 | 73.1 | 84.9
MPT-7B | Vanilla | 28.9 | 9.8 | 25.7 | 73.0 | 86.2
CAD | 30.5 | 11.1 | 27.0 | 73.8 | 90.4
MPT-7B-Instruct | Vanilla | 31.0 | 10.6 | 27.4 | 74.2 | 89.2
CAD | 32.2 | 11.9 | 28.5 | 74.9 | 94.0
Mistral-7B | Vanilla | 28.1 | 9.3 | 24.9 | 72.8 | 78.5
CAD | 29.3 | 10.6 | 25.9 | 73.7 | 84.0
Mistral-7B-Instruct | Vanilla | 34.2 | 11.4 | 30.5 | 75.4 | 93.1
CAD | 34.6 | 13.3 | 30.9 | 75.5 | 95.6
Flan-T5-xl | Vanilla | 29.0 | 10.0 | 25.6 | 75.6 | 86.0
CAD | 30.8 | 11.6 | 27.5 | 75.9 | 85.6
Flan-T5-xxl | Vanilla | 27.0 | 8.4 | 24.0 | 74.5 | 82.3
CAD | 28.8 | 10.0 | 25.8 | 75.1 | 81.8
XSUM | LLaMA-7B | Vanilla | 19.7 | 3.9 | 15.5 | 71.5 | 38.9
CAD | 19.9 | 3.8 | 15.6 | 71.8 | 53.8
Llama-2-7B | Vanilla | 19.5 | 3.9 | 15.3 | 71.5 | 47.2
CAD | 20.0 | 4.0 | 15.6 | 71.7 | 58.1
MPT-7B | Vanilla | 19.5 | 3.7 | 15.2 | 71.6 | 56.2
CAD | 20.0 | 3.8 | 15.5 | 72.3 | 63.6
MPT-7B-Instruct | Vanilla | 22.9 | 5.3 | 17.7 | 73.4 | 51.5
CAD | 22.3 | 5.1 | 17.2 | 76.1 | 62.6
Mistral-7B | Vanilla | 23.2 | 5.9 | 18.0 | 72.6 | 43.0
CAD | 22.5 | 5.9 | 17.6 | 73.3 | 51.8
Mistral-7B-Instruct | Vanilla | 25.8 | 6.7 | 20.0 | 74.2 | 55.7
CAD | 25.2 | 6.5 | 19.5 | 73.9 | 63.4
Flan-T5-xl | Vanilla | 34.8 | 12.8 | 27.5 | 80.4 | 53.3
CAD | 35.8 | 13.8 | 28.5 | 80.8 | 55.6
Flan-T5-xxl | Vanilla | 36.0 | 13.9 | 28.2 | 80.8 | 54.8
CAD | 36.7 | 14.7 | 29.3 | 81.1 | 54.9

Table 2: Vanilla decoding vs CAD with QFS datasets.

Datasets | Model | Decoding | ROUGE-1 | ROUGE-2 | ROUGE-L | BERTScore-P | FactKB
---|---|---|---|---|---|---|---
Dbpedia | LLaMA-7B | Vanilla | 13.3 | 2.9 | 11.8 | 68.8 | 29.6
CAD | 13.4 | 3.1 | 12.0 | 68.9 | 36.1
Llama-2-7B | Vanilla | 12.0 | 2.5 | 10.8 | 68.6 | 27.3
CAD | 12.7 | 2.7 | 11.2 | 68.4 | 34.0
MPT-7B | Vanilla | 13.1 | 2.7 | 11.6 | 70.0 | 33.4
CAD | 13.3 | 3.0 | 12.0 | 70.4 | 37.9
MPT-7B-Instruct | Vanilla | 14.8 | 3.7 | 13.1 | 71.5 | 28.1
CAD | 14.5 | 3.4 | 12.8 | 72.2 | 31.3
Mistral-7B | Vanilla | 13.8 | 3.2 | 12.2 | 69.4 | 28.8
CAD | 13.8 | 3.0 | 12.3 | 69.1 | 36.9
Mistral-7B-Instruct | Vanilla | 18.2 | 5.3 | 15.9 | 71.6 | 42.6
CAD | 17.1 | 4.7 | 15.1 | 71.1 | 41.6
Flan-T5-xl | Vanilla | 18.0 | 5.4 | 16.2 | 74.4 | 37.5
CAD | 16.2 | 4.5 | 14.5 | 73.7 | 46.3
Flan-T5-xxl | Vanilla | 20.6 | 6.8 | 18.6 | 75.0 | 33.9
CAD | 16.2 | 4.4 | 14.5 | 73.5 | 43.0
PubMedQA | LLaMA-7B | Vanilla | 25.5 | 7.6 | 21.3 | 71.2 | 79.5
CAD | 26.0 | 8.2 | 21.6 | 71.3 | 86.4
Llama-2-7B | Vanilla | 27.7 | 9.0 | 23.0 | 72.0 | 87.4
CAD | 28.6 | 9.5 | 23.6 | 72.1 | 90.8
MPT-7B | Vanilla | 27.8 | 8.7 | 23.0 | 72.0 | 92.4
CAD | 27.4 | 8.5 | 22.8 | 71.7 | 96.3
MPT-7B-Instruct | Vanilla | 29.0 | 9.7 | 24.0 | 72.9 | 96.4
CAD | 27.6 | 8.7 | 22.6 | 72.2 | 97.3
Mistral-7B | Vanilla | 25.0 | 8.0 | 20.8 | 71.7 | 67.9
CAD | 24.4 | 7.6 | 20.1 | 71.5 | 75.7
Mistral-7B-Instruct | Vanilla | 31.7 | 11.9 | 26.4 | 74.3 | 97.5
CAD | 31.2 | 11.6 | 26.0 | 73.9 | 97.5
Flan-T5-xl | Vanilla | 32.3 | 11.4 | 26.6 | 76.1 | 95.7
CAD | 30.5 | 10.1 | 24.9 | 75.3 | 95.2
Flan-T5-xxl | Vanilla | 33.1 | 11.6 | 27.0 | 76.3 | 95.6
CAD | 31.3 | 10.6 | 25.6 | 75.6 | 94.3

We try to study the following research questions:

  * •

RQ1: Performance-wise, can CAD (1) improve news summarization performance as
discussed by [Shi+23]? and (2) reduce hallucination/factuality errors in QFS?

  * •

RQ2: What is a good choice of hyperparameter $\alpha$?

  * •

RQ3: What is the computational trade-off?

#### Quality of Generation.

We present the results on QFS datasets and news summarization datasets in
Table 2 and Table 1, respectively. On news summarization datasets, CAD
achieves improved ROUGE scores with all eight models on CNN Dailymail dataset,
and improves with six models on XSUM datasets. In terms of factuality errors,
CAD improves FactKB scores on six out of eight models on CNN Dailymail dataset
and eight out of eight models on XSUM dataset. These results support the
finding in [Shi+23] that CAD can improve news summarization performance.

In terms of results on QFS datasets, we notice that while CAD can reduce
hallucination/factuality errors as measured by FactKB score, the ROUGE score
is not improved on Dbpedia dataset and slightly drops on PubMedQA dataset. We
hypothesize that the different performance patterns might be because the
references in two QFS datasets are highly abstractive, and myopically
conditioning on the input context may not achieve an improved ROUGE score.

As for performance between different language models, a clear observation is
that instruction finetuned models (MPT-7B-Instruct and
Mistral-7B-Instruct-v0.1) outperform their pre-trained counterparts. On the
relatively challenging PubMedQA dataset, Mistral-7B-Instruct-v0.1 achieves
comparable performance to Flan-T5-xxl, despite Flan-T5-xxl being trained on
more academic style NLP tasks.

#### Choice of hyperparameter $\alpha$.

We show a comparison between different choices of hyperparameter $\alpha$
against model performance in Table 3. From QFS results in Table 3 we notice
that there is a trade-off between FactKB score and ROUGE score as $\alpha$
increases, whereas on the two news summarization datasets, the ROUGE score and
FactKB score are positively correlated. We hypothesize that the different
trends are due to the design choices in building the datasets. On the news
summarization datasets, setting $\alpha=0.3$ yields an overall better
performance.

Table 3: Performance comparison between difference choices of hyperparameter
$\alpha$, recall that $\alpha=0$ reduces to vanilla decoding. The reported
numbers are averaged over eight models.

Datasets |  $\bm{\alpha}$ value | ROUGE-1 | ROUGE-2 | ROUGE-L | BERTScore-P | FactKB
---|---|---|---|---|---|---
Dbpedia |  $\alpha$=0.0 | 15.5 | 4.1 | 13.8 | 71.2 | 32.7
$\alpha$=0.15 | 14.9 | 3.8 | 13.3 | 71.1 | 35.1
$\alpha$=0.3 | 14.7 | 3.6 | 13.1 | 70.1 | 38.1
$\alpha$=0.5 | 14.5 | 3.6 | 13.0 | 70.9 | 38.7
PubMedQA |  $\alpha$=0.0 | 29.1 | 9.7 | 24.0 | 73.3 | 89.1
$\alpha$=0.15 | 28.7 | 9.6 | 23.8 | 73.2 | 90.6
$\alpha$=0.3 | 28.4 | 9.4 | 23.4 | 73.0 | 91.7
$\alpha$=0.5 | 27.9 | 9.1 | 23.1 | 72.7 | 93.3
CNN Dailymail |  $\alpha$=0.0 | 29.1 | 9.6 | 25.8 | 73.9 | 85.2
$\alpha$=0.15 | 30.4 | 10.9 | 27.1 | 74.3 | 87.2
$\alpha$=0.3 | 30.7 | 11.2 | 27.4 | 74.4 | 88.4
$\alpha$=0.5 | 30.7 | 11.2 | 27.3 | 74.4 | 87.7
XSUM |  $\alpha$=0.0 | 25.2 | 7.0 | 19.7 | 74.5 | 50.1
$\alpha$=0.15 | 25.4 | 7.2 | 19.8 | 74.8 | 55.7
$\alpha$=0.3 | 25.3 | 7.2 | 19.9 | 74.9 | 58.0
$\alpha$=0.5 | 24.8 | 6.9 | 19.4 | 74.9 | 58.1

#### Reporting Decoding Speed.

Previously we have analyzed the additional complexity in terms of FLOPs of
CAD. Now we take a look at the difference in decoding speed on a real-world
dataset. The experiment is conducted on a server with a single Nvidia RTX 3090
GPU, CUDA version 12.1, and PyTorch version 2.1.0. We implement two forward
passes, first with
$p_{\theta}(y_{t}|\mathbf{c},\,\mathbf{x},\,\mathbf{y}_{<t})$, and second with
$p_{\theta}(y_{t}|\mathbf{x},\,\mathbf{y}_{<t})$; another option is packing
two computation into one batch which will increase CUDA memory usage.

On CNN Dailymail dataset, vanilla decoding with FP16 and batch size=1 leads to
a decoding speed of 0.042 second/token, while the decoding speed of CAD with
FP16 and batch size=1 is 0.077 second/token with two forward passes. This
suggests while CAD can improve generation quality, it also slows down decoding
speed or requires more CUDA memory.

##  5 Related Work

#### Search as an Information-seeking Process.

[Mar06] categorize search into three two different information seeking
processes— _Information Lookup_ and _Exploratory Search_. Information lookup
refers to the user looking for simple facts or asking factoid questions, while
exploratory search/search to learn often involves multiple iterations.
Different approaches have been explored to reduce users’ cognitive loads in
the information-seeking process. For example, Perplexity
AI333https://www.perplexity.ai/ transfers web search queries into QA-style
questions, and cites sources in the answers to improve verifiability. Snippets
[BC14] provide short previews for each webpage to allow users to locate
relevant information without examining the webpage. Counterfactual
explanations [Xu+23] suggest counterfactual queries as explanations and
demonstrate search sessions’ effectiveness can be improved.

#### Hallucination in Natural Language Generation.

[Ji+23] present a survey of hallucination in natural language generation.
There are mainly two types of hallucinations, namely intrinsic hallucination
and extrinsic hallucination. Intrinsic hallucination refers to the generated
output that contradicts the source content, whereas extrinsic hallucination
refers to the generated output that cannot be verified from the source
content. In retrieval augmented generation, we mainly focus on intrinsic
hallucination. Pre-trained language models (not instruction finetuned) are
more prone to hallucination, as the gigantic pre-training corpus may contain
untruthful texts and the teacher forcing training procedure lets the models
pick up this untruthful information. [Shu+21] show that retrieval augmentation
can reduce hallucinations in downstream tasks. [Tia+23] show that language
models can be trained to reduce hallucination via reinforcement learning
techniques. [Var+23] propose to use lm logits to detect hallucination in the
generation process and mitigate it by retrieving evidence and rewriting.

#### Decoding Methods to Improve Generation Quality.

[Li+23] propose Contrastive Decoding (CD) to improve generation quality. The
idea is to contrast the stronger expert model’s prediction with a smaller
amateur model. A recent work [OL23] shows that CD can also improve the
reasoning ability of the language models on multiple reasoning benchmark
datasets. Based on a similar idea, [Liu+21] show that by contrasting to a
toxic model (language model trained with toxicity contents), the toxicity of
the expert language model’s generations can be reduced. A different flavor of
controlled generation aims to let the model generate specific tokens/phrases.
[Lu+21] show that such a goal can be achieved by utilizing specific data
structure and interpolation of lm logits.

##  6 Conclusion and Limitations

In this work we carry out a reproducibility study of a recently proposed
decoding method—Context-aware Decoding (CAD). Experiments on query-focused
summarization datasets and news summarization datasets show that CAD can
improve QFS and news summarization quality by reducing factual
mistakes/hallucinations while mostly retaining the match of lexical patterns,
but also with additional inference-time FLOPs and reduced decoding speed. Due
to the limited bandwidth and resources, we only experiment with language
models no larger than 11B (Flan-T5-xxl), larger language models may exhibit
different performance patterns.

## References

  * [BC14] Hannah Bast and Marjan Celikik  “Efficient index-based snippet generation”  In _ACM Transactions on Information Systems (TOIS)_ 32.2 ACM New York, NY, USA, 2014, pp. 1–24
  * [Ben+21] Emily M Bender, Timnit Gebru, Angelina McMillan-Major and Shmargaret Shmitchell  “On the dangers of stochastic parrots: Can language models be too big?”  In _Proceedings of the 2021 ACM conference on fairness, accountability, and transparency_ , 2021, pp. 610–623
  * [Che+20] Wei-Fan Chen et al.  “Abstractive snippet generation”  In _Proceedings of The Web Conference 2020_ , 2020, pp. 1309–1319
  * [Chu+22] Hyung Won Chung et al.  “Scaling instruction-finetuned language models”  In _arXiv preprint arXiv:2210.11416_ , 2022
  * [Fen+23] Shangbin Feng, Vidhisha Balachandran, Yuyang Bai and Yulia Tsvetkov  “Factkb: Generalizable factuality evaluation using language models enhanced with factual knowledge”  In _arXiv preprint arXiv:2305.08281_ , 2023
  * [Hol+19] Ari Holtzman et al.  “The curious case of neural text degeneration”  In _arXiv preprint arXiv:1904.09751_ , 2019
  * [Ji+23] Ziwei Ji et al.  “Survey of hallucination in natural language generation”  In _ACM Computing Surveys_ 55.12 ACM New York, NY, 2023, pp. 1–38
  * [Jia+23] Albert Q Jiang et al.  “Mistral 7B”  In _arXiv preprint arXiv:2310.06825_ , 2023
  * [Jin+19] Qiao Jin et al.  “PubMedQA: A Dataset for Biomedical Research Question Answering”  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ Hong Kong, China: Association for Computational Linguistics, 2019, pp. 2567–2577  DOI: 10.18653/v1/D19-1259
  * [Kap+20] Jared Kaplan et al.  “Scaling laws for neural language models”  In _arXiv preprint arXiv:2001.08361_ , 2020
  * [Lew+20] Patrick Lewis et al.  “Retrieval-augmented generation for knowledge-intensive nlp tasks”  In _Advances in Neural Information Processing Systems_ 33, 2020, pp. 9459–9474
  * [Li+23] Xiang Lisa Li et al.  “Contrastive Decoding: Open-ended Text Generation as Optimization”  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ Toronto, Canada: Association for Computational Linguistics, 2023, pp. 12286–12312  DOI: 10.18653/v1/2023.acl-long.687
  * [Lin04] Chin-Yew Lin  “ROUGE: A Package for Automatic Evaluation of Summaries”  In _Text Summarization Branches Out_ Barcelona, Spain: Association for Computational Linguistics, 2004, pp. 74–81  URL: https://aclanthology.org/W04-1013
  * [Liu+21] Alisa Liu et al.  “DExperts: Decoding-Time Controlled Text Generation with Experts and Anti-Experts”  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ Online: Association for Computational Linguistics, 2021, pp. 6691–6706  DOI: 10.18653/v1/2021.acl-long.522
  * [Lu+21] Ximing Lu et al.  “NeuroLogic Decoding: (Un)supervised Neural Text Generation with Predicate Logic Constraints”  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ Online: Association for Computational Linguistics, 2021, pp. 4288–4299  DOI: 10.18653/v1/2021.naacl-main.339
  * [Mar06] Gary Marchionini  “Exploratory search: from finding to understanding”  In _Communications of the ACM_ 49.4 ACM New York, NY, USA, 2006, pp. 41–46
  * [Met+21] Donald Metzler, Yi Tay, Dara Bahri and Marc Najork  “Rethinking search: making domain experts out of dilettantes”  In _Acm sigir forum_ 55.1, 2021, pp. 1–27  ACM New York, NY, USA
  * [NCL18] Shashi Narayan, Shay B. Cohen and Mirella Lapata  “Don’t Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarization”  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ Brussels, Belgium: Association for Computational Linguistics, 2018, pp. 1797–1807  DOI: 10.18653/v1/D18-1206
  * [Nem+17] Preksha Nema, Mitesh M. Khapra, Anirban Laha and Balaraman Ravindran  “Diversity driven attention model for query-based abstractive summarization”  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ Vancouver, Canada: Association for Computational Linguistics, 2017, pp. 1063–1072  DOI: 10.18653/v1/P17-1098
  * [OL23] Sean O’Brien and Mike Lewis  “Contrastive decoding improves reasoning in large language models”  In _arXiv preprint arXiv:2309.09117_ , 2023
  * [PCM22] Liam Poel, Ryan Cotterell and Clara Meister  “Mutual information alleviates hallucinations in abstractive summarization”  In _arXiv preprint arXiv:2210.13210_ , 2022
  * [Raf+20] Colin Raffel et al.  “Exploring the limits of transfer learning with a unified text-to-text transformer”  In _The Journal of Machine Learning Research_ 21.1 JMLRORG, 2020, pp. 5485–5551
  * [SB22] Chirag Shah and Emily M Bender  “Situating search”  In _Proceedings of the 2022 Conference on Human Information Interaction and Retrieval_ , 2022, pp. 221–232
  * [Shi+23] Weijia Shi et al.  “Trusting Your Evidence: Hallucinate Less with Context-aware Decoding”  In _arXiv preprint arXiv:2305.14739_ , 2023
  * [Shu+21] Kurt Shuster et al.  “Retrieval Augmentation Reduces Hallucination in Conversation”  In _Findings of the Association for Computational Linguistics: EMNLP 2021_ Punta Cana, Dominican Republic: Association for Computational Linguistics, 2021, pp. 3784–3803  DOI: 10.18653/v1/2021.findings-emnlp.320
  * [SLM17] Abigail See, Peter J. Liu and Christopher D. Manning  “Get To The Point: Summarization with Pointer-Generator Networks”  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ Vancouver, Canada: Association for Computational Linguistics, 2017, pp. 1073–1083  DOI: 10.18653/v1/P17-1099
  * [Tea23] MosaicML NLP Team  “Introducing MPT-7B: A New Standard for Open-Source, Commercially Usable LLMs” Accessed: 2023-05-05, 2023  URL: www.mosaicml.com/blog/mpt-7b
  * [Tia+23] Katherine Tian et al.  “Fine-tuning Language Models for Factuality”  In _arXiv preprint arXiv:2311.08401_ , 2023
  * [Tou+23] Hugo Touvron et al.  “Llama: Open and efficient foundation language models”  In _arXiv preprint arXiv:2302.13971_ , 2023
  * [Var+23] Neeraj Varshney et al.  “A Stitch in Time Saves Nine: Detecting and Mitigating Hallucinations of LLMs by Validating Low-Confidence Generation”, 2023  arXiv:2307.03987 [cs.CL]
  * [XC23] Zhichao Xu and Daniel Cohen  “A Lightweight Constrained Generation Alternative for Query-focused Summarization”  In _arXiv preprint arXiv:2304.11721_ , 2023
  * [Xu+23] Zhichao Xu et al.  “Counterfactual Editing for Search Result Explanation”  In _arXiv preprint arXiv:2301.10389_ , 2023
  * [Zha+19] Tianyi Zhang et al.  “Bertscore: Evaluating text generation with bert”  In _arXiv preprint arXiv:1904.09675_ , 2019

##  7 Appendix

###  7.1 Prompting Templates

We report the prompting templates in Table 4.

Table 4: Prompting Templates

Datasets | Model | Decoding | Templates
---|---|---|---
Dbpedia | Decoder-only | Vanilla |  | Question: {query}. Document: {document}.
---
According to the Document, the one-sentence answer to the Question is:
Decoder-only | CAD |  | Question: {query}. Document: {None}.
---
According to the Document, the one-sentence answer to the Question is:
Encoder-Decoder | Vanilla |  | Question: {query}. Document: {document}.
---
According to the Document, the one-sentence answer to the Question is:
Encoder-Decoder | CAD |  | Question: {query}. Document: {None}.
---
According to the Document, the one-sentence answer to the Question is:
PubMedQA | Decoder-only | Vanilla |  | Question: {query}. Document: {document}.
---
According to the Document, the detailed answer to the Question is:
Decoder-only | CAD |  | Question: {query}. Document: {document}.
---
According to the Document, the detailed answer to the Question is:
Encoder-Decoder | Vanilla |  | Question: {query}. Document: {document}.
---
According to the Document, the detailed answer to the Question is:
Encoder-Decoder | CAD |  | Question: {query}. Document: {None}.
---
According to the Document, the detailed answer to the Question is:
CNN Dailymail | Decoder-only | Vanilla |  | News article: {document}.
---
Summary of the above news article:
Decoder-only | CAD |  | News article: {None}.
---
Summary of the above news article:
Encoder-Decoder | Vanilla |  | Summarize the following article in one or two sentences.
---
{document}:
Encoder-Decoder | CAD |  | Summarize the following article in one or two sentences.
---
{None}:
XSUM | Decoder-only | Vanilla |  | News article: {document}.
---
Summary of the above news article:
Decoder-only | CAD |  | News article: {None}.
---
Summary of the above news article:
Encoder-Decoder | Vanilla |  | Summarize the following article in one or two sentences.
---
{document}:
Encoder-Decoder | CAD |  | Summarize the following article in one or two sentences.
---
{None}:

###  7.2 Hyperparameters

We report the hyperparameters in Table 5.

Table 5: Decoding Hyperparameters

Dataset | Top-k | Top-p | num_beams | repetition_penalty | temperature | min_new_tokens | max_new_tokens
---|---|---|---|---|---|---|---
Dbpedia | 50 | 0.9 | 1 | 1.0 | 1.0 | 5 | 30
PubMedQA | 50 | 0.9 | 1 | 1.0 | 1.0 | 40 | 100
CNN Dailymail | 50 | 0.9 | 1 | 1.0 | 1.0 | 30 | 70
XSUM | 50 | 0.9 | 1 | 1.0 | 1.0 | 20 | 50


# “Ask Me Anything”: How Comcast Uses LLMs to
Assist Agents in Real Time

Scott Rome<EMAIL_ADDRESS>0000-0003-0270-7192 ,  Tianwen Chen
<EMAIL_ADDRESS>0009-0008-0477-7366 Comcast AI
TechnologiesPhiladelphiaPennsylvaniaUSA ,  Raphael Tang
<EMAIL_ADDRESS>0009-0007-2873-892X Comcast AI
TechnologiesPhiladelphiaPennsylvaniaUSA ,  Luwei Zhou<EMAIL_ADDRESS>0009-0009-1862-3058 and  Ferhan Ture<EMAIL_ADDRESS>0000-0002-5585-157X Comcast AI TechnologiesPhiladelphiaPennsylvaniaUSA

(2024)

# “Ask Me Anything”: How Comcast Uses LLMs to
Assist Agents in Real Time

Scott Rome<EMAIL_ADDRESS>0000-0003-0270-7192 ,  Tianwen Chen
<EMAIL_ADDRESS>0009-0008-0477-7366 Comcast AI
TechnologiesPhiladelphiaPennsylvaniaUSA ,  Raphael Tang
<EMAIL_ADDRESS>0009-0007-2873-892X Comcast AI
TechnologiesPhiladelphiaPennsylvaniaUSA ,  Luwei Zhou<EMAIL_ADDRESS>0009-0009-1862-3058 and  Ferhan Ture<EMAIL_ADDRESS>0000-0002-5585-157X Comcast AI TechnologiesPhiladelphiaPennsylvaniaUSA

(2024)

###### Abstract.

Customer service is how companies interface with their customers. It can
contribute heavily towards the overall customer satisfaction. However, high-
quality service can become expensive, creating an incentive to make it as cost
efficient as possible and prompting most companies to utilize AI-powered
assistants, or ”chat bots”. On the other hand, human-to-human interaction is
still desired by customers, especially when it comes to complex scenarios such
as disputes and sensitive topics like bill payment.111https://bit.ly/3yaNO9t

This raises the bar for customer service agents. They need to accurately
understand the customer’s question or concern, identify a solution that is
acceptable yet feasible (and within the company’s policy), all while handling
multiple conversations at once.

In this work, we introduce “Ask Me Anything” (AMA) as an add-on feature to an
agent-facing customer service interface. AMA allows agents to ask questions to
a large language model (LLM) on demand, as they are handling customer
conversations—the LLM provides accurate responses in real-time, reducing the
amount of context switching the agent needs. In our internal experiments, we
find that agents using AMA versus a traditional search experience spend
approximately 10% fewer seconds per conversation containing a search,
translating to millions of dollars of savings annually. Agents that used the
AMA feature provided positive feedback nearly 80% of the time, demonstrating
its usefulness as an AI-assisted feature for customer care.

rag, llm, customer care, assistive AI, vector db, reranking

††journalyear: 2024††copyright: rightsretained††conference: Proceedings of the
47th International ACM SIGIR Conference on Research and Development in
Information Retrieval; July 14–18, 2024; Washington, DC, USA††doi:
10.1145/3626772.3661345††isbn: 979-8-4007-0431-4/24/07††ccs: Information
systems Retrieval models and ranking††ccs: Information systems Language models

##  1\. Introduction

Comcast, like many other companies, provides customer service through various
communication channels. Many self-service solutions are available on the
mobile “Xfinity” app (e.g., reviewing latest bill) which also has an option to
chat with an AI-powered bot named “Xfinity Assistant”. While these digital
automation capabilities have been replacing human customer representatives
(also referred to as ”agents”) for many tasks, there are still many situations
that require human-to-human interactions.

A customer trying to simply look up information about their profile, internet
services, or bill, they should be able to do it without an agent’s assistance.
This also holds true if they are trying to carry out a relatively
straightforward task like rescheduling their appointment or make a change to
their services.

Past studies show a human-human interaction is preferred over a human-computer
one in certain customer service situations(Wowak, [n. d.]). For example,
agents might outperform bots in situations that require creative problem
solving. In other situations, the customer might simply prefer to talk to a
agent to benefit from their empathy and emotional intelligence, or to navigate
through cultural sensitivities.

At Comcast, an internal custom tool suite aims to help agents to effectively
and efficiently handle such conversations. However, it still often requires
manually looking up information in multiple places, relating it to what the
customer is saying, then crafting a relevant response that aligns with the
communication guidelines. In this paper, we introduce a new feature to this
tool suite called “Ask Me Anything” (AMA). It leverages large language models
(LLMs) following a retrieval-augmented generation (RAG) approach to generate
contextually relevant responses by combining internal knowledge sources,
indexing existing knowledge articles efficiently at build time, retrieving
relevant chunks of text for a given question at query time, then feeding them
to a _Reader_ LLM to generate a succinct answer with citations provided as
reference. In the next section, we describe the methodology in more detail.

##  2\. Methodology

Our system follows a typical RAG implementation with modifications to improve
performance on proprietary questions. First, the documents are preprocessed to
text and chunked, the chunks are embedded then stored with metadata (e.g.,
associated URL for citations, an identifier, the title, etc.) in a vector
database. We describe our specific choices for processing and embeddings in
Section 2.1 and Section 2.2 respectively with some experimental justification.
Next, we detail how we train and evaluate a reranking model using synthetic
data to improve search result relevancy in Section 2.3. Finally, we discuss
how we generate answers followed by how we evaluate the system in Section 2.4
and 2.5.

###  2.1. Document Preprocessing

We receive documents from various internal clients in different formats. We
standardize the documents into plain text and chunk each document into
snippets using Deepset.ai’s Haystack library (Pietsch et al., 2019). In order
to uniquely reference each chunk of every document after retrieval, we assign
an origin identifier to each document and a local identifier to each chunk.
Finally, we implement role-based access control on each document, so different
users can only view the documents for which they have permission.

In 1(a), we show various chunking parameters for Haystack’s preprocessor and
their evaluation scores. The metric derivation is explained in Section 2.5
(Answer quality assumed the top 3 items were passed to the LLM). We observed a
large improvement from setting a higher `max_chars_check`, which we used as a
proxy for limiting the size of each snippet given to the LLM.

Table 1. Chunking parameters and evaluation of three different settings. Parameter | A | B | C
---|---|---|---
clean_empty_lines | true |  |
clean_whitespace | true |  |
clean_header_footer | true |  |
split_by | word |  |
split_length | 300 | 100 |
split_overlap | 50 | 25 |
split_respect_sentence_boundary | true |  |
max_chars_check | 1000 |  | 3000
Metric |  |  |
Answer Quality | - | -5.7% | +13.2%
MRR | - | -13.3% | 0.0%
R@3 | - | -7.9% | 0.0%
NDCG | - | -10.0% | 0.0%
(a) For clarity, only changes from setting $A$ are found in the table. Empty
parameter values mean they are same as $A$. The metric values are the relative
difference from $A$, i.e., $100\cdot(\mu_{B}-\mu_{A})/\mu_{A}$ for some metric
$\mu$. Metrics are defined in Section 2.5.

###  2.2. Retrieving Relevant Text Snippets

To inform the choice of our retriever model, we conducted pilot experiments on
a curated evaluation set of fifty question–answer pairs. We searched the in-
production system logs for queries starting with a WH-word (who, what, how,
etc.) or ending with a question mark, roughly following the procedure on Bing
query logs from WikiQA (Yang et al., 2015). For each question, we then located
the relevant passage and answer span in our internal knowledge base used by
agents. Queries without answers were also labeled as such. Crucially, this
process avoids back-formulation (Sakai et al., 2004), where queries are
manually written by annotators based on known passages rather than crawled
from logs, resulting in biased evaluation sets.

We experimented with both dense and sparse retrieval models. For the sparse
model, we used Okapi BM25 (Robertson et al., 2009) with $k_{1}=1.0$ and
$b=0.5$. For the dense ones, we experimented with four: dense passage
retrieval (DPR) (Karpukhin et al., 2020), fine-tuned on Natural Questions
(Kwiatkowski et al., 2019); MPNet-base (v1) (Song et al., 2020), trained on
160GB of text corpora including Wikipedia, BookCorpus (Zhu et al., 2015), and
OpenWebText (Gokaslan and Cohen, 2019); OpenAI’s state-of-the-art `ada-002`
embeddings model; and MPNet-base v2, trained further on one billion sentence
pairs for better embedding quality.222Nils Reimers’s open-source contribution:
https://discuss.huggingface.co/t/train-the-best-sentence-embedding-model-ever-
with-1b-training-pairs/7354 Each was deemed to satisfy our computational and
financial constraints at inference time.

In 2(a), we report the recall@3 (R@3) and the mean reciprocal rank (MRR) of
these models on our evaluation set. The choice of recall@3 (versus recall@5 or
10) is from us feeding the top-three retrieved passages into the LLM. As a
sanity check, we also ran a baseline that randomly drew a passage, which
unsurprisingly yielded low scores. Mirroring prior work (Yang et al., 2019),
we found that BM25 remains a strong baseline, outperforming DPR in R@3 and
MRR, respectively. We conjecture that this results from Natural Questions
being substantially out of domain from our data.

Table 2. Results of various retrievers on our pilot evaluation set Method | Recall@3 | MRR
---|---|---
Random | -71.4% | -83.9%
BM25 | - | -
DPR (`single-nq`) | -42.8% | -42.9%
DPR (`multiset-nq`) | -23.8% | -29.0%
Multi-QA MPNet-base | +33.0% | +39.7%
OpenAI embeddings (`ada-002`) | +33.0% | +53.9%
MPNet-base v2 | +38.1% | +54.9%
(a) Statistics presented as relative difference from BM25, i.e.,
$100\cdot(\mu-\mu_{BM25})/\mu_{BM25}$ . Underline denotes statistical
significance relative to DPR.

We observe MPNet-base (v1), OpenAI’s `ada-002`, and MPNet-base (v2) to perform
similarly. Signed-rank tests for R@3 and $t$-tests for MRR also reveal a
significant difference ($p<0.05$) from DPR. Due to operational convenience and
the high performance of OpenAI’s ADA embeddings, we used ADA for the retriever
component for the final system.

For our production retrieval step, we embedded both the title of the article
and the text of the individual chunk and added them together prior to storage
in the vector database. Anecdotally, we found this to yield a more
comprehensive retrieval for a variety of queries, especially when chunks were
missing some descriptive context of the topic of the article.

###  2.3. Reranking Search Results

We found that reranking results using models finetuned on synthetic data
improved the retrieval step. Our approach was inspired by previous synthetic
data generation approaches (Bonifacio et al., 2022; Dai et al., 2022). First,
we used GPT-4 to generate synthetic questions from each snippet in our
dataset. We then ran each question through our search system using OpenAI’s
`text-embeddings-ada-002` (Greene et al., 2022) embeddings. Any questions
where the original snippet used for question generation did not appear in the
top 20 results were discarded. For each synthetic question, we stored the top
20 items retrieved, their relevance as scored by `BGE-reranker-large` (Xiao et
al., 2023), and an indicator that the snippet was the source of the question.
The final rankings were determined by first placing the source snippet as the
”most relevant” result, followed by the snippets in most relevant order as
scored by the `BGE-reranker-large` model.

For training, we used RankNet (Burges et al., 2005) to distill these rankings
into a finetuned MPNet (Song et al., 2020), in particular `all-mpnet-base-v2`
from `sentence-transformer` (Reimers and Gurevych, 2019), which has fewer
parameters requiring less computational resources to deploy into production
than `BGE-reranker-large`. The final dataset after constructing the necessary
pairs for RankNet consisted of over 10 million examples. We set aside 0.5% of
the examples as validation dataset. Our training parameters were listed in
Table 3. We used `DistributedDataParallel` from PyTorch (Paszke et al., 2017)
for distributed training, so the effective batch size is the number of GPUs
multiplied by the batch size. We found the ”Linear Scaling Rule”, where one
scales the learning rate when the batch size increases, to not apply to our
use case (Goyal et al., 2018), but we suspect it is because the original MPNet
architecture was trained with a much larger batch size than we used for
finetuning.

[b]  Parameter Specification Learning Rate $5\times 10^{-6}$ Batch Size 8
Number of GPUs1 10 Warmup Steps 4000 Weight Decay 0.001 Epochs 1 Total
Training Steps 171391 Learning Rate Scheduler Warmup-constant

Table 3. Training hyperparameters.

  * 1

GPU type: g4dn.xlarge (Nvidia T4)

To further evaluate the performance of our reranker model, we randomly sampled
10,000 real questions asked by customer service agents in our production
system. For every retrieved document, we followed the approach in (Thomas et
al., 2023), which showed that an LLM can accurately predict the relevancy of
search results. Specifically, GPT-4 was used to evaluate the overall quality
of each document to the question, which combined the scores from how the
document matches the intent of the question as well as how trustworthy the
document is. The final integer score ranged between 0 and 2, with higher score
meaning higher overall quality. Table 4 compares multiple metrics between ADA
vs. reranker. Since the overall score is non-binary, we compute MRR using the
rank of first document with a score of 2, and recall@3 examines whether the
top 3 documents contain any documents with a score of 2. The results indicate
an improvement in retrieval performance with the reranker model.

Table 4. ADA vs. Reranker Search Results using Production Questions Metric | ADA | Reranker
---|---|---
Recall@3 | - | +12%
MRR | - | +15%
NDCG | - | +4.8%
(a) For clarity, only changes from setting ADA are found in the table. The
metric values are the relative difference from ADA.

###  2.4. Generating the Answer from Snippets

In generating the answer, we follow the conventional wisdom approach in the
RAG literature. We begin our prompt with a preamble of guidelines for the
model, followed by the task description. Due to the length of our snippets of
text from the knowledge base, we are unable to provide few-shot examples. We
have anecdotally found it better to include more of the text to avoid
necessary information being cut off at random. To avoid the ”lost in the
middle” problem (Liu et al., 2023), we reverse the order of the Top K results
when passed into the LLM, formatted as XML capturing the ID, title and content
of the result. We used OpenAI’s `gpt-3.5-turbo` for our production Reader
component. As a final step in our prompt, we ask the LLM to answer the given
question using the search results.

####  2.4.1. Citations

An important product feature of of the AMA solution is providing references to
agents so they can learn more about the answer given. This can be seen in
various RAG implementations, such as Microsoft Copilot. In addition, the goal
was to build confidence in the system’s output and drive adoption internally.
Inspired by the Fact-Checking Rail (Rebedea et al., 2023), our Citation Rail
was accomplished by prompting the LLM to cite its sources in a specific manner
(c.f., Figure 1) combined with a post processing step where the citations were
removed from the text. If no citations were found, then the system would not
return the answer. Practically, there was another benefit from an
observability perspective: through this approach, we identified most ”no
answer” responses from the LLM, as typically the LLM would response similarly
to ”I’m sorry. I was unable to find the answer in the documents” without a
citation.

⬇

Please include a single source at the end of your answer, i.e., [Document0] if
Document0 is the source. If there is more than one source, use
[Document0][Document1] if Document0 and Document1 are the sources.

Figure 1. An example component of a prompt to encourage citations from the LLM
used in the system prompt section.

###  2.5. Offline Response Evaluation

To evaluate the system’s responses, we follow the LLM-as-a-judge methodology
(Zhu et al., 2023), in addition to metrics around retrieval quality typical of
a search system. In particular, a random sample of questions from customers
were pulled from production traffic. Human annotators then wrote correct
answers to each query using internal knowledge bases that are also available
to the AMA system. We were able to compare system answers to correct responses
given by human annotators using GPT-4 to compute ”Answer Quality”.

For each question, the annotators also provided a citation from which their
answers were based. We used this to calculate ”Citation Match Rate”: the
percentage of cases in which the citation from the AMA system matched the
ground truth. Given that our retrieval step returns a list, we calculated
Recall@K by assuming the annotated citation is the only relevant document.

Table 5 shows key metrics for the same two approaches as in Table 4 (`text-
embedding-ada-002` for dense retrieval of relevant documents and rerankering
the ADA-retrieved documents using our finetuned model). We observe that using
reranked documents, LLM is able to achieve a higher answer quality meaning
that the answer from a different document ranking is more accurate according
to GPT-4. The improvement can also be explained by the increased Citation
Match Rate and Recall@3 from the reranked documents directly influencing the
LLM’s ability to answer accurately.

Table 5. Response Quality Metric | ADA | Reranker
---|---|---
Answer Quality | - | +5.9%
Citation Match Rate | - | +2.5%
Recall@3 | - | +16.5%
(a) For clarity, only changes from setting ADA are found in the table. The
metric values are the relative difference from ADA.

##  3\. Deploying AMA to Customer Service Agents

Due to business sensitivity purposes, this section will obscure some details
related to monetary business metrics. The system was piloted with hundreds of
chat agents in late 2023. Over the course of a month-long trial, chat handling
time improved 10% when agents used AMA versus the traditional search option,
which required the agent to open a new tool and perform a search. We believe
this is a good proxy metric for answer quality because an inaccurate or
incomplete response from AMA would require the agent to start over and revert
to the traditional option, duplicating work and taking more time overall.
Explicit feedback, via a simple thumbs up/thumbs down UI element, was also
collected from agents, with nearly an 80% positive feedback rate (there is no
baseline for this rate as such feedback was not requested before the release
of this feature). Shortly after the trial period, the system was rolled out to
all chat agents (in thousands), with AMA-driven search becoming the preferred
way of searching, accounting for two thirds of all typed queries.

##  4\. Online Reranker Experiment

Shortly after the trial from Section 3 concluded, we began an A/B test of the
reranker module described in Section 2.3. The control variant used only the
ADA embeddings for vector retrieval with no reranking component, and the
treatment utilized the reranker component on the top $20$ results from the
ADA-based vector retrieval step. The test ran for three weeks in early 2024.
We powered our tests at 80% and use significance level $\alpha=.01$ for
metrics that applied to every interaction and $\alpha=.05$ when metrics
considered user feedback, as responses were sparse. Due to the limited pool of
agents, our randomization unit, we utilized an agent-day randomization similar
to the cookie-day randomization found in other large systems (Tang et al.,
2010) to increase statistical power. It has been shown in the literature (Deng
et al., 2017) that violations of the independent and identically distributed
(IID) assumption can lead to underestimation of the variance, but these tests
can still be considered trustworthy in practice by using smaller significance
thresholds and when observing larger effect sizes. The delta method (Deng et
al., 2018) was employed to estimate the variance from question-level metrics.

We observed a statistically significant increase in two of our metrics: namely
the ”No Answer Rate”, which is the number of queries with no answer divided by
the total number of queries, and the ”Positive Feedback Rate”, defined as the
number of thumbs up divided by the count of feedback received. Downstream
business metrics like average handle time and escalation rate showed no
significant difference. However, the improvement in No Answer Rate implies
that the system was able to handle more questions than before by providing the
relevant documents to the LLM while also increasing the rate of positive
feedback.

Table 6. A/B Test Results Metric | Effect | p-value
---|---|---
No Answer Rate | -11.9% | p ¡ .001
Positive Feedback Rate | +8.9% | p ¡ .05
(a) Table contains relative change from control as the effect. Lower is better
for No Answer Rate.

##  5\. Conclusions

In this paper, we introduced AMA, a large-scale solution to a common business
need: efficient high-quality customer care. Through the use of third-party
LLMs and proven RAG methodology, we were able to build AMA pretty quickly and
demonstrate clear value as an assistive feature. We showed improvements to
retrieval and answer quality with specific choices for the document
preprocessing, the retrieval model and its embeddings, as well as a custom
reranker model. As we deploy AMA to thousands of agents with tangible business
benefits, we believe that this provides a good example of how humans and AI
can collaborate to better serve customers.

## References

  * (1)
  * Bonifacio et al. (2022) Luiz Bonifacio, Hugo Abonizio, Marzieh Fadaee, and Rodrigo Nogueira. 2022.  InPars: Data Augmentation for Information Retrieval using Large Language Models.  arXiv:2202.05144 [cs.CL]
  * Burges et al. (2005) Chris Burges, Tal Shaked, Erin Renshaw, Ari Lazier, Matt Deeds, Nicole Hamilton, and Greg Hullender. 2005.  Learning to rank using gradient descent. In _Proceedings of the 22nd International Conference on Machine Learning_ (Bonn, Germany) _(ICML ’05)_. Association for Computing Machinery, New York, NY, USA, 89–96.  https://doi.org/10.1145/1102351.1102363
  * Dai et al. (2022) Zhuyun Dai, Vincent Y. Zhao, Ji Ma, Yi Luan, Jianmo Ni, Jing Lu, Anton Bakalov, Kelvin Guu, Keith B. Hall, and Ming-Wei Chang. 2022.  Promptagator: Few-shot Dense Retrieval From 8 Examples.  arXiv:2209.11755 [cs.CL]
  * Deng et al. (2018) Alex Deng, Ulf Knoblich, and Jiannan Lu. 2018.  Applying the Delta Method in Metric Analytics: A Practical Guide with Novel Ideas. In _Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_ (London, United Kingdom) _(KDD ’18)_. Association for Computing Machinery, New York, NY, USA, 233–242.  https://doi.org/10.1145/3219819.3219919
  * Deng et al. (2017) Alex Deng, Jiannan Lu, and Jonthan Litz. 2017.  Trustworthy Analysis of Online A/B Tests: Pitfalls, challenges and solutions. In _Proceedings of the Tenth ACM International Conference on Web Search and Data Mining_ (Cambridge, United Kingdom) _(WSDM ’17)_. Association for Computing Machinery, New York, NY, USA, 641–649.  https://doi.org/10.1145/3018661.3018677
  * Gokaslan and Cohen (2019) Aaron Gokaslan and Vanya Cohen. 2019.  OpenWebText Corpus.  http://Skylion007.github.io/OpenWebTextCorpus.
  * Goyal et al. (2018) Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. 2018.  Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour.  arXiv:1706.02677 [cs.CV]
  * Greene et al. (2022) Ryan Greene, Ted Sanders, Lilian Weng, and Arvind Neelakantan. 2022.  _New and improved embedding model_.  https://openai.com/blog/new-and-improved-embedding-model
  * Karpukhin et al. (2020) Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020.  Dense Passage Retrieval for Open-Domain Question Answering. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_. 6769–6781.
  * Kwiatkowski et al. (2019) Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. 2019\.  Natural questions: a benchmark for question answering research.  _Transactions of the Association for Computational Linguistics_ 7 (2019), 453–466.
  * Liu et al. (2023) Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2023.  Lost in the Middle: How Language Models Use Long Contexts.  arXiv:2307.03172 [cs.CL]
  * Paszke et al. (2017) Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. 2017.  Automatic differentiation in PyTorch.  (2017).
  * Pietsch et al. (2019) Malte Pietsch, Timo Möller, Bogdan Kostic, Julian Risch, Massimiliano Pippi, Mayank Jobanputra, Sara Zanzottera, Silvano Cerza, Vladimir Blagojevic, Thomas Stadelmann, Tanay Soni, and Sebastian Lee. 2019.  Haystack: the end-to-end NLP framework for pragmatic builders.  https://github.com/deepset-ai/haystack.
  * Rebedea et al. (2023) Traian Rebedea, Razvan Dinu, Makesh Sreedhar, Christopher Parisien, and Jonathan Cohen. 2023.  NeMo Guardrails: A Toolkit for Controllable and Safe LLM Applications with Programmable Rails.  arXiv:2310.10501 [cs.CL]
  * Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019.  Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics.  http://arxiv.org/abs/1908.10084
  * Robertson et al. (2009) Stephen Robertson, Hugo Zaragoza, et al. 2009\.  The probabilistic relevance framework: BM25 and beyond.  _Foundations and Trends in Information Retrieval_ 3, 4 (2009), 333–389.
  * Sakai et al. (2004) Tetsuya Sakai, Yoshimi Saito, Yumi Ichimura, Tomoharu Kokubu, and Makoto Koyama. 2004.  The effect of back-formulating questions in question answering evaluation. In _Proceedings of the 27th annual international ACM SIGIR conference on Research and development in information retrieval_. 474–475.
  * Song et al. (2020) Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu. 2020.  MPNet: Masked and Permuted Pre-training for Language Understanding.  arXiv:2004.09297 [cs.CL]
  * Tang et al. (2010) Diane Tang, Ashish Agarwal, Deirdre O’Brien, and Mike Meyer. 2010.  Overlapping Experiment Infrastructure: More, Better, Faster Experimentation. In _Proceedings 16th Conference on Knowledge Discovery and Data Mining_. Washington, DC, 17–26.
  * Thomas et al. (2023) Paul Thomas, Seth Spielman, Nick Craswell, and Bhaskar Mitra. 2023.  Large language models can accurately predict searcher preferences.  arXiv:2309.10621 [cs.IR]
  * Wowak ([n. d.]) Kaitlin Wowak. [n. d.].  Humans vs. automation: Service center agents can outperform technology, study shows.  https://news.nd.edu/news/humans-vs-automation-service-center-agents-can-outperform-technology-study-shows/
  * Xiao et al. (2023) Shitao Xiao, Zheng Liu, Peitian Zhang, and Niklas Muennighoff. 2023.  C-Pack: Packaged Resources To Advance General Chinese Embedding.  arXiv:2309.07597 [cs.CL]
  * Yang et al. (2019) Wei Yang, Kuang Lu, Peilin Yang, and Jimmy Lin. 2019.  Critically examining the ”neural hype”: weak baselines and the additivity of effectiveness gains from neural ranking models. In _Proceedings of the 42nd international ACM SIGIR conference on research and development in information retrieval_. 1129–1132.
  * Yang et al. (2015) Yi Yang, Wen-tau Yih, and Christopher Meek. 2015.  WikiQA: A challenge dataset for open-domain question answering. In _Proceedings of the 2015 conference on empirical methods in natural language processing_. 2013–2018.
  * Zhu et al. (2023) Lianghui Zhu, Xinggang Wang, and Xinlong Wang. 2023.  JudgeLM: Fine-tuned Large Language Models are Scalable Judges.  arXiv:2310.17631 [cs.CL]
  * Zhu et al. (2015) Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015.  Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In _Proceedings of the IEEE international conference on computer vision_. 19–27.