# Optical Darboux Transformer

Auro M. Perego<EMAIL_ADDRESS>Aston Institute of Photonic
Technologies, Aston University, Birmingham, B4 7ET, United Kingdom

###### Abstract

The Optical Darboux Transformer is introduced as a photonic device which
performs the Darboux transformation directly in the optical domain. This
enables two major advances for signal processing based on the nonlinear
Fourier transform: (i) the multiplexing of different solitonic waveforms
corresponding to arbitrary number of discrete eigenvalues of the Zakharov-
Shabat system in the optical domain, and (ii) the selective filtering of an
arbitrary number of individual solitons too. The Optical Darboux Transformer
can be built using existing commercially available photonic technology
components and constitutes a universal tool for signal processing, optical
communications, optical rogue waves generation, and waveform shaping and
control in the nonlinear Fourier domain.

††preprint: APS/123-QED

The concept of optical soliton has been proposed 50 years ago as a temporally
localised light pulse having hyperbolic secant shape and being invariant upon
spatial propagation. This pulse is the analytical solution to the nonlinear
Schrödinger equation (NLSE) describing the evolution of the electric field
slowly varying envelope along a single mode optical fiber[1]. Most
importantly, Zakharov and Shabat in their seminal paper of 1972 [2], have
shown how to obtain bright soliton solutions to the NLSE using the
mathematical technique called inverse scattering transform (IST)[3]. Given a
particular complex waveform $q(t)$, which depends on time $t$, from the
solutions of the Zakharov-Shabat system – a system of two coupled linear
equations which depend on $q$ – it is possible to define a nonlinear spectrum
of the waveform. This nonlinear spectrum consists in general of a set of
discrete points (denoted by eigenvalues $\lambda_{i}$) in the complex half-
plane – each of them corresponding to a soliton – and of a continuous part
$Q(\lambda)$ (being $\lambda$ a spectral parameter performing a role analogous
to frequency) which is associated to dispersive waves. The latter tends to the
classical Fourier spectrum in the low signal power limit [4]. The nonlinear
spectrum provides a set of orthogonal nonlinear eigenmodes which are invariant
upon propagation and constitute a complete basis for the description of a
given signal [5]. Solving the Zakharov-Shabat system for a given signal $q(t)$
and finding the associated nonlinear spectrum, consists in performing the
Nonlinear Fourier Transform (NFT) of $q(t)$.
In the decades following the pioneering contributions by Hasegawa and Tappert
and by Zakharov and Shabat, substantial research on optical solitons has taken
place both from a mathematical [3, 6] and from a a physics and engineering
perspective too [7, 8]. Optical solitons have been used in commercially
deployed optical communications systems and their concept has been generalised
to encompass robust localised solutions existing in dissipative systems too –
non integrable by the IST – thanks to a balance between nonlinearity,
dispersion gain/injection and losses [9]. In this broader sense dissipative
optical solitons are a cornerstones of modern laser mode-locking [10], and a
workhorse for optical frequency combs generation in nonlinear driven optical
resonators [11]. In the recent years, building on the concept of eigenvalue
communications proposed by Hasegawa and Nyu [12], a novel paradigm for soliton
based optical communication systems in fibres has emerged. This approach aims
at encoding information in the discrete spectrum eigenvalues (solitons and
their phases) and in the continuous spectrum too[13, 14, 4, 15, 16, 17].
Further to that, NFT has been suggested as well as a powerful tool for the
analysis of optical signals generated by dissipative/non integrable optical
devices like mode-locked lasers[18] and optical resonators[19, 20] too. This
is enabled by the fact that the nonlinear spectrum – the union of the
continuous and discrete parts – of a given function provides a complete basis
for the description of signals with vanishing boundary conditions at
$t\rightarrow\pm\infty$ [5, 4]. Hence one can project a signal on the NFT
basis obtaining its decomposition in terms of solitons and continuous spectrum
and then compute new NFTs after the signal has propagated in space, monitoring
in this fashion the nonlinear spectrum evolution with consequent creation and
annihilation of spectral components. This mathematical basis featuring
discrete eigenvalues proves advantageous for following the evolution of
coherent structures embedded inside a complex waveform. Indeed via its
discrete spectral points, NFT enables the description of localised structure
using individual eigenvalues, hence condensing the key information in a
reduced number of degrees of freedom (pulse peak, temporal position, velocity
and phase) compared to a multifrequency classical Fourier spectrum.
Despite these recent advances, NFT applications in photonics remain very much
a technology in its infancy, suffering from limitations ranging from poor
algorithmic performances for computing the nonlinear spectrum to the lack of
optical domain nonlinear signal processing techniques. For instance,
apparently simple problems such as (i) how to optically combine different
waveforms containing multiple solitons so that the novel waveform contains
exactly the solitons present into the two original signals, or (ii) how to
optically eject or filter individual solitons from a signal without
irreversibly destroying the whole waveform, remained unsolved so far. This is
rooted in the qualitatively different nonlinear physics of solitons compared
to the linear principles which inspire the customary approach to signal
processing. For instance, a traditional optical filter is not be able to
discriminate between a soliton and the continuous spectrum; nor coherently
combining a soliton with a signal waveform using for instance an optical
coupler will add a discrete eigenvalue to the nonlinear spectrum of the
signal.
To address this fundamental limitation, we present here a novel device which
is based on the optical implementation of a mathematical operation called the
Darboux transformation (DT), and which enables optically multiplexing of
solitons to an existing waveform – including in presence of continuous
spectrum – and selective removal of optical solitons from a signal too,
without undesired effects on the waveform spectral properties. Mathematically
the DT is transformation which takes as an input a solution to the NLSE and a
new discrete spectrum eigenvalue, providing as an output a new solution to the
NLSE having the same nonlinear spectrum as the input one plus the new discrete
eigenvalue [21, 15, 22]. The DT offers a simple and iterative procedure to
build multisoliton solutions to the NLSE. For this reason it has been widely
used in NFT based optical communication systems to digitally generate
multisoliton signals both in single[22, 15, 23, 24, 16] and dual-polarisation
regimes[17, 25]. However, in the examples reported so far its implementation
does not occur in optical domain. The DT is iteratively calculated digitally
till a waveform $q(t)$ featuring the desired nonlinear spectrum is obtained.
Then the function $q(t)$ is transferred from the electrical to the optical
domain by means of a wave form generator and an optical modulator which
suitably modifies a laser beam.
This poses a fundamental limitation to solitonic waveform signal processing
and namely the necessity of completely passing from the optical to the
electric domain and then back to optical one every time one needs to
manipulate a signal e.g. for multiplexing, demultiplexing, regeneration, and
noise filtering. To the best of our knowledge no optical implementation of the
DT has been ever proposed so far. The goal of this work is to introduce a
method for performing the DT optically, and to practically answer the
question: how can one selectively add and remove discrete eigenvalues from an
optical signal without undesired effects on the spectral properties of the
signal itself?
First, we need to briefly introduce the known DT formalism for the nonlinear
Schrödinger equation (NLSE)[24]. Let us consider a given signal $q(z,t)$ whose
propagation along spatial coordinate $z$ is ruled by the focussing normalised
NLSE:

$\displaystyle\frac{\partial q}{\partial z}=i\frac{\partial^{2}q}{\partial
t^{2}}+2i|q|^{2}q.$ (1)

The normalised amplitude, spatial and temporal coordinates are related to
physical fiber parameters by the following relations: $q=E/\sqrt{P}$,
$t=\tau/T_{0}$, and $z=-Z/\mathcal{L}$; where $E$ is the electric field slowly
varying envelope in units $\sqrt{\text{W}}$, $P=|\beta_{2}|/(\gamma
T_{0}^{2})$, $\mathcal{L}=2T_{0}^{2}/|\beta_{2}|$, $\beta_{2}$ is the group
velocity dispersion in ps2km-1, $\gamma$ is the nonlinearity coefficient in
km-1W-1, $\tau$ is a the temporal coordinate in a reference frame co-moving
with the pulse, $Z$ is the spatial coordinate along the fiber in physical
units, and $T_{0}$ is a free normalization parameter. Starting from the
knowledge of $q(t)$ it is possible to obtain a new solution to the NLSE,
$\tilde{q}$ which features the same discrete spectrum as $q$ plus a new
soliton described by the discrete eigenvalue $\lambda_{0}$. If $q$ had non
vanishing continuous spectrum $Q_{c}(\lambda)$ then $\tilde{q}$ has continuous
spectrum given by
$\tilde{Q}_{c}(\lambda)=\frac{\lambda-\lambda_{0}^{*}}{\lambda-\lambda_{0}}Q_{c}(\lambda)$.
The new solution is defined by

$\displaystyle\tilde{q}(t)=q(t)+2i(\lambda_{0}^{*}-\lambda_{0})\frac{v_{2}^{*}v_{1}}{|v_{1}|^{2}+|v_{2}|^{2}}$
(2)

which is the DT. $v_{1}(t)$ and $v_{2}(t)$ are the two components of the
vector $v=(v_{1},v_{2})^{T}$ which is a solution to the Zakharov-Shabat system
for the signal $q(t)$. The vector

$\displaystyle\tilde{v}=(\lambda_{0}\mathbb{I}_{2}-\mathbb{G}_{0})v$ (3)

is instead the solution to the Zakharov-Shabat system for signal $\tilde{q}$;
being $\mathbb{I}_{2}$ the $2\times 2$ identity matrix and
$\mathbb{G}_{0}=\mathbb{\Theta}\mathbb{M}_{0}\mathbb{\Theta}^{-1}$ with
$\mathbb{M}_{0}=\text{diag}(\lambda_{0},\lambda_{0}^{*})$ and

$\displaystyle\mathbb{\Theta}=\begin{pmatrix}v_{1}&v_{2}^{*}\\\
v_{2}&-v_{1}^{*}\end{pmatrix}.$ (4)

Now we generalise the DT to the case when a signal $\tilde{q}$ is generated
starting from the vacuum solution $q=0$ (or from a solution having no discrete
spectrum but continuum spectrum $Q_{c}(\lambda)$), and features $n$ discrete
eigenvalues $\\{\lambda_{1},..,\lambda_{n}\\}$ which are iteratively added to
the spectrum of $q$. This is naturally achieved by iterating $n$ times the
procedure highlighted in Eq.2. We have

$\displaystyle\tilde{q}(t)=q(t)+\sum_{i=1}^{n}\mathcal{D}_{i}$ (5)

where
$\mathcal{D}_{i}=2i(\lambda_{i}^{*}-\lambda_{i})\frac{v_{2}^{(i)*}v_{1}^{(i)}}{|v_{1}^{(i)}|^{2}+|v_{2}^{(i)}|^{2}}$
with the precaution that for every $\lambda_{i}$ added in the iterative
process, the functions $v_{1,2}^{(k)}$ associated to the remaining – yet to be
added – eigenvalues $\lambda_{k}$ ($k=i+1,...,n$) have to be updated as
follows:

$\displaystyle\begin{pmatrix}\tilde{v}_{1}^{(k)}\\\
\tilde{v}_{2}^{(k)}\end{pmatrix}=(\lambda_{k}\mathbb{I}_{2}-\mathbb{G}_{oi})...(\lambda_{k}\mathbb{I}_{2}-\mathbb{G}_{01})\begin{pmatrix}A^{(k)}e^{-i\lambda_{k}t}\\\
B^{(k)}e^{i\lambda_{k}t}\end{pmatrix}$ (6)

being the matrix $\mathbb{G}_{0i}$ computed as a function of $v_{1,2}^{(i)}$
evaluated after $i-1$ iterations of the DT. It can be furthermore shown that
if one wants to generate a signal featuring $n$ eigenvalues
$\\{\lambda_{1},..\lambda_{n}\\}$ corresponding to solitons having spectral
amplitudes determined by the spectral coefficients $\\{b_{1},...b_{n}\\}$,
then the initialisation of the coefficients should be as follows: $A_{i}=1$
and $B_{i}=b_{i}$[24]. Note also that if $q$ is not the vacuum solution but it
has a continuous spectrum $Q_{c}(\lambda)$, then one should initialise the
iterative DT algorithm with signal $q^{\prime}$ (instead of $q$), with the
spectrum
$Q_{c}^{\prime}(\lambda)=Q_{c}(\lambda)\prod_{i=1}^{n}\frac{\lambda-\lambda_{i}}{\lambda-\lambda_{i}^{*}}$
if one wants to have $\tilde{q}$ with continuous spectrum $Q_{c}$[24].

Figure 1: _Schematic of the Optical Darboux Transformer_. In the multiplexer
mode the signal $q_{-}$ (with spectrum featuring 2 discrete eigenvalues) is
combined at a 50/50 coupler with the auxiliary signal $q_{aux}$, phase-shifted
by $-\pi/2$. In the filter mode the input signal is instead $q_{+}$, featuring
three spectral eigenvalues, and is combined with $q_{aux}$ phase-shifted by
$\pi/2$. The output from port 3 is amplified with 3 dB gain by an optical
amplifier (red triangle) and results in signal $q_{out}$ featuring a new
eigenvalue with respect to $q_{-}$ (in purple) or has one eigenvalue removed
with respect to $q_{+}$ (white circle) depending on whether the operation is
in the multiplexer or in the filter mode.

Based on the known formalism for the iterative addition of eigenvalues we can
hence proceed to define the explicit expression for the DT associated to the
addition of _multiple_ eigenvalues to a signal $q_{-}$ by coherently combining
it with a suitable _single_ auxiliary signal $q_{aux}$. We notice that we can
always write Eq.5 as

$\displaystyle q_{+}=q_{-}+q_{aux}$ (7)

where $q_{-}$ is a signal featuring a certain continuous spectrum
$Q_{-}(\lambda)$ and a discrete spectrum with
$\\{\lambda_{1},...,\lambda_{m}\\}$ eigenvalues and spectral amplitudes
$\\{b_{1},...,b_{m}\\}$; while $q_{aux}$ is an auxiliary signal such that
$q_{+}$ features
$\\{\lambda_{1},...,\lambda_{m},\lambda_{m+1},..,\lambda_{n}\\}$ eigenvalues
with spectral amplitudes $\\{b_{1},...,b_{m},b_{m+1},...,b_{n}\\}$. In
particular

$\displaystyle q_{aux}=\sum_{i=m+1}^{n}\mathcal{D}_{i}$ (8)

provided that the functions $v_{1,2}^{(k)}$ ($k=m+1,...,n$) – which determine
$\mathcal{D}_{i}$ – have been calculated according to Eq.6 keeping into
account that the transformation operated on them is a function _also_ of the
eignevalues of $q_{-}$, $\\{\lambda_{1},...,\lambda_{m}\\}$. It follows that
adding eigenvalues $\\{\lambda_{k+1},..,\lambda_{n}\\}$ to the spectrum of
$q_{-}$ requires simply coherently combining together the waveforms $q_{-}$
and $q_{aux}$. We can hence consider a device, schematically illustrated in
Fig.1, that we call the _Optical Darboux Transformer_ (ODT), which can be
built using a 50/50 optical coupler, a phase shifter and an optical amplifier.
In the eigenvalue multiplexing operational regime, the input at port 1 of the
coupler is $q_{-}$ while the input at port 2 is $q_{aux}$ after being phase-
shifted by $-\pi/2$ radians. In this way the output of port 3 reads
$q_{3}=\frac{1}{\sqrt{2}}q_{-}+i\frac{1}{\sqrt{2}}(-iq_{aux})$ and after
amplification by a factor of $2$ for power ($\sqrt{2}$ for amplitude)
corresponding to $\approx 3$ dB gain, results indeed in
$q_{out}=\sqrt{2}q_{3}=q_{+}$ which has eigenvalues
$\\{\lambda_{1},..,\lambda_{n}\\}$. It is easy to notice that the ODT can
operate as an NFT filter too, which removes eigenvalues
$\\{\lambda_{k+1},...,\lambda_{n}\\}$ from the spectrum
$\\{\lambda_{1},...,\lambda_{n}\\}$ of the signal $q_{+}$. In that case we
call $q_{+}$ the input signal entering from port 1 of the 50/50 coupler while
the phase shifter is set to provide a $+\pi/2$ radians phase shift to
$q_{aux}$. The output of port 3 in this case reads
$q_{3}=\frac{1}{\sqrt{2}}q_{+}+i\frac{1}{\sqrt{2}}(iq_{aux})=q_{-}$ which has
discrete spectrum $\\{\lambda_{1},...,\lambda_{k}\\}$. Interestingly, the
possibility of using the DT to iteratively remove single eigenvalues form the
nonlinear discrete spectrum has been suggested mathematically[26] and explored
for developing efficient algorithms for NFT based optical communication
systems digital data processing[27]. We stress the fact that the operations
described above, both concerning the multiplexing and the filtering of
eigenvalues, are _not_ equivalent to simply coherently adding or subtracting,
to a given signal, an auxiliary signal containing hyperbolic secant waveforms
having the eigenvalues one wishes to add or remove to/from the signal
nonlinear spectrum. As it can be appreciated from examples reported below
$q_{aux}$ exhibits in general a much richer spectrum. Indeed the NFT operation
$\mathcal{N}[\cdot]$ is _nonlinear_ , namely in general
$\mathcal{N}[q_{1}]+\mathcal{N}[q_{2}]\neq\mathcal{N}[q_{1}+q_{2}]$. This
nonlinearity is also a reason why an amplifier is needed at the coupler port 3
output, indeed, unless the signal power is very low $\mathcal{N}[aq_{1}]\neq
a\mathcal{N}[q_{1}]$ with $a$ a complex constant. In Figs. 2a)-c) an example
is shown of the ODT working as a multiplexer and performing de facto the
function of a single soliton creation operator which adds one eigenvalue to
the input signal spectrum. In Figs. 2d)-f) instead an example is shown of the
ODT acting as a filter, by performing the role of a single soliton
annihilation operator, and removing an eigenvalue from the input signal
spectrum. In both cases input, output and auxiliary signals and respective
discrete eigenvalue spectrum are shown. It is interesting to notice that the
auxiliary signal $q_{aux}$ features, besides a solitonic spectrum, a non
vanishing continuous spectrum too, while $q_{\pm}$ don’t. In Fig. 3 it is
furthermore shown how the ODT can simultaneously add or remove multiple
eigenvalues to/from the spectrum. Finally in Fig.4 an example is shown where
the ODT adds or removes an eigenvalue to/from an input signal featuring a
continuous spectrum too[28].

Figure 2: _Single eigenvalue addition and filtering._ In the multiplexer mode
the waveforms, discrete, and continuous spectrum are shown in panels a)-c) for
input, output and auxiliary signal (see legend): the input signal $q_{-}$
features spectrum $\\{\lambda_{1}=1+0.5i,\lambda_{2}=0.8i\\}$ with spectral
coefficients $\\{b_{1}=5,b_{2}=0.5\\}$ while the output $q_{+}$ features the
same eignevalues plus the extra eigenvalue $\lambda_{3}=1+i$ with $b_{3}=-1$.
The filter mode is illustrated in panels d)-f): from the input signal $q_{+}$
with spectrum $\\{\lambda_{1}=-0.3+i,\lambda_{2}=0.5i\\}$ and
$\\{b_{1}=1,b_{2}=0.8\\}$ the eigenvalue $\lambda_{2}$ is removed. Figure 3:
_Multiple eigenvalues addition and filtering._ Multiple eigenvalues addition
is shown in panels a)-c) for the input signal $q_{-}$ having spectrum
$\\{\lambda_{1}=0.3+i,\lambda_{2}=0.3i\\}$ with $\\{b_{1}=5,b_{2}=0.5\\}$
where the ODT adds to it the following additional eigenvalues
$\\{\lambda_{3}=1+2i,\lambda_{4}=0.5i,\lambda_{5}=0.1i\\}$ with
$\\{b_{3}=-1,b_{4}=2,b_{5}=0.1\\}$. Multiple eigenvalues filtering is shown in
panels d)-f). Input signal $q_{+}$ has spectrum
$\\{\lambda_{1}=i,\lambda_{2}=1.5i,\lambda_{3}=0.5i,\lambda_{4}=4i,\lambda_{5}=2+i\\}$
with $\\{b_{1}=1,b_{2}=0.2,b_{3}=-2,b_{4}=3,b_{5}=1\\}$, and eigenvalues
$\\{\lambda_{3}=0.5i,\lambda_{4}=4i,\lambda_{5}=2+i\\}$ have been removed.
Figure 4: _Eigenvalues multiplexing and filtering in presence of continuous
spectrum_. In panels a)-c) the multiplexing case is shown for input signal
$q_{-}$ featuring discrete spectrum $\\{\lambda_{1}=i\\}$ with
$\\{b_{1}=3\\}$, and continuous spectrum $Q(\lambda)=A/(i\lambda)e^{-2i\lambda
t_{d}}\left(1-\frac{\Delta}{i\lambda}\text{cot}(\Delta T)\right)^{-1}$ with
$\sqrt{|A|^{2}+\lambda^{2}}$, $A=0.5$, $t_{d}=6$, $T=1$. The ODT output
provides signal $q_{+}$ featuring the additional eigenvalue
$\\{\lambda_{2}=-1+1.5i\\}$ with $\\{b_{2}=-4\\}$. In d)-f) a scenario is
shown where input signal $q_{+}$, featuring discrete spectrum
$\\{\lambda_{1}=i,\lambda_{2}=2+1.5i\\}$ with $\\{b_{1}=2,b_{2}=-1\\}$, and
continuous spectrum
$Q_{f}(\lambda)=\prod_{i=1}^{2}\frac{\lambda-\lambda_{i}}{\lambda-\lambda_{i}^{*}}Q_{m}(\lambda)$,
is transformed into the new signal $q_{-}$ differing by eigenvalue
$\\{\lambda_{2}=2+1.5i\\}$ with $\\{b_{2}=-1\\}$.

The simple architecture of the proposed device can be realised using existing
commercially available optical fiber technology components like couplers,
phase-shifters and amplifiers such as Erbium-doped fiber ones. The auxiliary
signal $q_{aux}$ can be synthesised using a modulator, while its necessary
temporal synchronisation with the input waveform entering port 1 of the
coupler could be achieved for instance exploiting techniques used for signal
synchronisation in time-division multiplexing for optical communications[29].
The ODT presented in this work can find natural generalisation to the dual
polarisation case (Manakov system) too. This can be implemented for instance
using a first polarising beam splitter directing each polarization to a
different ODT, and a second one for recombining the two polarizations after
solitonic waveform processing (additional amplifiers could be employed where
needed to compensate for power losses).
It is important to stress a vital point. A distinctive conceptual feature of
soliton filtering and soliton multiplexing in nonlinear Fourier domain is
indeed the necessity of embedding a knowledge of the signal to be
filtered/multiplexed in the filter/multiplexer itself. This is at variance
with the operational principle of some traditional linear optical components
such as standard optical filters which do not require any information about
the input signal. Similar paradigm changes, despite certainly appearing
surprising for a way of thinking that is used to conceptualise the world based
on a linear approach, are required when developing a truly nonlinear
scientific _Weltanschauung_.
The ODT may find several applications in optical soliton technology. The
possibility of optically controlling and manipulating solitonic waveforms,
including adding/removing solitons from an optical signal or separating the
continuous from the discrete spectrum, or filtering noise, are only some of
the simplest and most obvious functionalities within reach. The proposed ODT,
due to its potential cascadability and scalability could constitute the ”unit
cell” of more complex nonlinear Fourier domain signal processing and
manipulation devices. To conclude, we have demonstrated the possibility of an
optical domain implementation of the DT for the NLSE by means of a simple
photonic architecture. The ODT proposed in this work offers unprecedented
control over solitonic waveforms by enabling selective discrete eigenvalues
multiplexing and filtering. Further to that, by reducing the need of
intermediate electrical domain processing, the ODT provides a paradigm
changing leap forward towards optical signal processing in nonlinear Fourier
domain.

The author acknowledges support by the Royal Academy of Engineering through
the Research Fellowship Scheme, and is grateful to Dr. Morteza Kamalian and
Dr. Mingming Tan for discussions.

## References

  * Hasegawa and Tappert [1973] A. Hasegawa and F. Tappert, Transmission of stationary nonlinear optical pulses in dispersive dielectric fibers. i. anomalous dispersion, Appl. Phys. Lett. 23, 142–144 (1973).
  * Zakharov and Shabat [1972] V. E. Zakharov and A. B. Shabat, Exact theory of 2-dimensional self-focusing and one-dimensional self-modulation of waves in nonlinear media, Sov. Phys. J. Exp. Theor. Phys. 34, 62 (1972).
  * Novikov  _et al._ [1984] S. Novikov, S. V. Manakov, L. P. Pitaevskii, and V. E. Zakharov,  _Theory of Solitons_ (Plenum, 1984).
  * Yousefi and Kschischang [2014a] M. I. Yousefi and F. R. Kschischang, Information transmission using the nonlinear fourier transform, part i: Mathematical tools, IEEE Transactions on Information Theory 60, 4312 (2014a).
  * Ablowitz  _et al._ [1974] M. J. Ablowitz, D. J. Kaup, A. C. Newell, and H. Segur, The inverse scattering transform-fourier analysis for nonlinear problems, Studies in Applied Mathematics 53, 249 (1974).
  * Ablowitz and Segur [1981] M. Ablowitz and H. Segur,  _Solitons and the Inverse Scattering Transform_ (SIAM (Philadelphia), 1981).
  * Hasegawa and Kodama [1995] A. Hasegawa and Y. Kodama,  _Solitons in Optical Communications_ (Oxford University Press, 1995).
  * Mollenauer and Gordon [2006] L. F. Mollenauer and J. P. Gordon,  _Solitons in Optical Fiber: Fundamentals and Applications_ (Academic Press, 2006).
  * Akhmediev and Ankiewickz [2005] N. Akhmediev and A. Ankiewickz,  _Dissipative Solitons_ (Springer, 2005).
  * Grelu and N. [2012] P. Grelu and M. I. A. N., Dissipative solitons for mode-locked lasers, Nat. Photon. 6, 84 (2012).
  * Kippenberg  _et al._ [2018] T. Kippenberg, A. Gaeta, M. Lipson, and M. Gorodetsky, Dissipative kerr solitons in optical microresonators, Science 361, 567 (2018).
  * Hasegawa and Nyu [1993] A. Hasegawa and T. Nyu, Eigenvalue communication, Journal of Lightwave Technology 11, 395 (1993).
  * Turitsyn  _et al._ [2017] S. K. Turitsyn, J. E. Prilepsky, S. T. Le, S. Wahls, L. L. Frumin, M. Kamalian, and S. A. Derevyanko, Nonlinear fourier transform for optical data processing and transmission: advances and perspectives, Optica 4, 307 (2017).
  * Prilepsky  _et al._ [2014] J. E. Prilepsky, S. A. Derevyanko, K. J. Blow, I. Gabitov, and S. K. Turitsyn, Nonlinear inverse synthesis and eigenvalue division multiplexing in optical fiber channels, Phys. Rev. Lett. 113, 013901 (2014).
  * Aref  _et al._ [2018] V. Aref, S. T. Le, and H. Buelow, Modulation over nonlinear fourier spectrum: Continuous and discrete spectrum, J. Lightwave Technol. 36, 1289 (2018).
  * Le  _et al._ [2017] S. Le, V. Aref, and H. Buelow, onlinear signal multiplexing for communication beyond the kerr nonlinearity limit, Nat. Photon. 11, 570–576 (2017).
  * Gaiarin  _et al._ [2018] S. Gaiarin, A. M. Perego, E. P. da Silva, F. D. Ros, and D. Zibar, Dual-polarization nonlinear fourier transform-based optical communication system, Optica 5, 263 (2018).
  * Sugavanam  _et al._ [2019] S. Sugavanam _et al._ , Analysis of laser radiation using the nonlinear fourier transform, Nat. Commun. 10, 5663 (2019).
  * Turitsyn  _et al._ [2020] S. K. Turitsyn, I. S. Chekhovskoy, and M. P. Fedoruk, Nonlinear fourier transform for characterization of the coherent structures in optical microresonators, Opt. Lett. 45, 3059 (2020).
  * Turitsyn  _et al._ [2021] S. K. Turitsyn, I. S. Chekhovskoy, and M. P. Fedoruk, Nonlinear fourier transform for analysis of optical spectral combs, Phys. Rev. E 103, L020202 (2021).
  * Matveev and Salle [1991] V. B. Matveev and M. A. Salle,  _Darboux transformations and solitons_ (Springer, 1991).
  * Yousefi and Kschischang [2014b] M. I. Yousefi and F. R. Kschischang, Information transmission using the nonlinear fourier transform, part iii: Spectrum modulation, IEEE Transactions on Information Theory 60, 4346 (2014b).
  * Buelow  _et al._ [2016] H. Buelow, V. Aref, and W. Idler, Transmission of waveforms determined by 7 eigenvalues with psk-modulated spectral amplitudes, in  _ECOC 2016; 42nd European Conference on Optical Communication_ (2016) pp. 1–3.
  * Aref [2016] V. Aref, Control and detection of discrete spectral amplitudes in nonlinear fourier spectrum (2016), arXiv:1605.06328 [math.NA] .
  * Ros  _et al._ [2019] F. D. Ros, S. Civelli, S. Gaiarin, E. P. da Silva, N. D. Renzis, M. Secondini, and D. Zibar, Dual-polarization nfdm transmission with continuous and discrete spectral modulation, J. Lightwave Technol. 37, 2335 (2019).
  * Lin [1990] J. Lin, Evolution of the scattering data under the classical darboux transform for su(2) soliton systems, Acta Mathematicae Applicatae Sinica 6, 308–316 (1990).
  * Span  _et al._ [2020] A. Span, V. Aref, H. Bülow, and S. t. Brink, Successive eigenvalue removal for multi-soliton spectral amplitude estimation, Journal of Lightwave Technology 38, 4708 (2020).
  * [28] For calculations of nonlinear spectra Fast-NFT algorithms have been used [30].
  * Weber  _et al._ [2006] H.-G. Weber, R. Ludwig, S. Ferber, C. Schmidt-Langhorst, M. Kroh, V. Marembert, C. Boerner, and C. Schubert, Ultrahigh-speed otdm-transmission technology, Journal of Lightwave Technology 24, 4616 (2006).
  * Wahls  _et al._ [2018] S. Wahls, S. Chimmalgi, and P. Prins, FNFT: A Software Library for Computing Nonlinear Fourier Transforms, The Journal of Open Source Software 3, 597 (2018).


# To Point or Not to Point: Understanding How Abstractive Summarizers
Paraphrase Text

Matt Wilber
Cornell University
<EMAIL_ADDRESS>
&William Timkey
Cornell University
<EMAIL_ADDRESS>
&Marten van Schijndel
Cornell University
<EMAIL_ADDRESS>
###### Abstract

Abstractive neural summarization models have seen great improvements in recent
years, as shown by ROUGE scores of the generated summaries. But despite these
improved metrics, there is limited understanding of the strategies different
models employ, and how those strategies relate their understanding of
language. To understand this better, we run several experiments to
characterize how one popular abstractive model, the pointer-generator model of
See et al. (2017), uses its explicit copy/generation switch to control its
level of abstraction (generation) vs extraction (copying). On an extractive-
biased dataset, the model utilizes syntactic boundaries to truncate sentences
that are otherwise often copied verbatim. When we modify the copy/generation
switch and force the model to generate, only simple paraphrasing abilities are
revealed alongside factual inaccuracies and hallucinations. On an abstractive-
biased dataset, the model copies infrequently but shows similarly limited
abstractive abilities. In line with previous research, these results suggest
that abstractive summarization models lack the semantic understanding
necessary to generate paraphrases that are both abstractive and faithful to
the source document.

##  1 Introduction

Recent years have seen great improvements in “abstractive” summarization
models – models that not only concatenate text from the source document, but
can additionally paraphrase to generate summary text. Once limited to sentence
compression Rush et al. (2015), abstractive models now generate multi-sentence
summaries See et al. (2017), even for relatively long documents Cohan et al.
(2018). However, extractive models and mixed models with significant
extractive components continue to show strong performance, and the extent and
manner in which abstraction is used by summarization models is not well
understood.

Previous work has raised concerns about whether models are able to paraphrase
in ways that lead to better summaries. Abstractive models often generate
summaries that are either ungrammatical or unfaithful to the source document
Maynez et al. (2020); Durmus et al. (2020); Kryscinski et al. (2020) and are
prone to repetition in their outputs See et al. (2019); Holtzman et al.
(2020). These issues raise questions about _how_ neural summarizers generate
novel text. Abstractive summarization is differentiated from extractive
summarization by the model’s ability to paraphrase, but paraphrasing ability
is not directly measured by popular metrics, leading to a lack of
understanding of the generative process. Some previous research has aimed to
alleviate these issues in evaluation: Zhang et al. (2018a) propose evaluating
summaries with human evaluations of informativeness and coherence, and Ganesan
(2018) implements a metric to reward models that paraphrase via simple synonym
substitutions according to WordNet. However, synonym substitution is just one
form of paraphrasing, and truly abstractive models should be capable of more
complex paraphrasing strategies.

To understand how abstraction manifests in neural summarization models, we
study a model that has an explicit abstraction/extraction switch, the pointer-
generator model of See et al. (2017). The training objective of this model
causes it to choose the best summarization strategy (abstractive vs
extractive) in different contexts, permitting us to determine the environments
where abstractive summarization is an effective summarization strategy. First,
we show how the switch varies across a full summary and is influenced by the
decoder’s copy and generation distributions. Next, we present a behavioral
probe of the abstraction/extraction switch, to observe how the switch reacts
to lexical, structural, and distributional information as it decodes a
summary. Finally, we modify the switch value, forcing more frequent paraphrase
generation during decoding, revealing the limits of the model’s paraphrasing
capabilities. Ultimately, we find across both the CNN/DailyMail and XSum
datasets that the model’s abstractive capabilities are limited; the model
understands how to identify and combine constituents from the source text in a
grammatical fashion, but lacks the semantic understanding required to produce
grammatical, faithful and meaningful paraphrases.

##  2 Model

###  2.1 The Pointer-Generator Model

We study the pointer-generator model released by See et al. (2017), which uses
an explicit switch, $p_{\text{gen}}$, that blends abstractive and extractive
summarization strategies. We briefly review the pointer-generator model here;
for more details, see the original paper of See et al. (2017).

The final output distribution for a particular word in the summary $P(w)$ is a
weighted sum of the generation distribution and the copy distribution,
weighted by $p_{\text{gen}}$ and $1-p_{\text{gen}}$, respectively. This is
described by Equation 9 in See et al. (2017), modified for clarity here:

$P(w)=p_{\text{gen}}P_{\text{vocab}}(w)+(1-p_{\text{gen}})P_{\text{copy}}(w)$
(1)

$P_{\text{vocab}}(w)$ is the generation distribution over the model’s
vocabulary, and $P_{\text{copy}}(w)$ is the copy distribution over the tokens
in the source document. The $p_{\text{gen}}$ switch explicitly weights the
influence of the generation and copy mechanisms on $P(w)$. For each time step
$t$, $p_{\text{gen}}$ is a function of the context vector $h_{t}^{*}$, the
decoder state $s_{t}$ and the decoder input $x_{t}$,

$p_{\text{gen}}=\sigma(\delta_{h_{t}^{*}}^{T}h_{t}^{*}+\delta_{s}^{T}s_{t}+\delta_{x}^{T}x_{t}+\beta_{\text{ptr}})$
(2)

where $\sigma$ is the sigmoid function and $\delta_{h_{t}^{*}}$, $\delta_{s}$,
$\delta_{x}$ and $\beta_{\text{ptr}}$ are learned parameters.

See et al. (2017) also use a coverage mechanism aimed at reducing repetition,
defining the coverage vector $c^{t}$ as

$c^{t}=\sum_{t^{\prime}=0}^{t-1}P_{\text{copy}(w_{t})}$ (3)

which is passed as another input to the attention mechanism.

###  2.2 Data

We analyze pointer-generator behavior when trained on an extractive-biased
dataset, CNN/DailyMail, and on an abstractive-biased dataset, XSum. The
CNN/DailyMail dataset is made up of multi-sentence summaries of news articles
from CNN and Daily Mail. XSum (Narayan et al., 2018) is a summarization
dataset that uses the first sentence of a news article as a summary of the
article. The dataset treats the remainder of the article as the source
document. As a result, the summaries are both shorter and more difficult to
copy from the source document, compared to the CNN/DailyMail dataset.

###  2.3 Training

Our experiments on CNN/DailyMail use the trained model released by See et al.
(2017), which includes the coverage mechanism described above. We decode
summaries on the test set of at most 120 tokens using beam search with beam
width 4, as in the original paper. For XSum, we trained our own model on the
XSum training partition, using the code released by See et al. (2017).111Code
and full replication details are available at
https://github.com/mwilbz/pointer-generator-analysis.

Like Narayan et al. (2018), we do not include the coverage mechanism for the
XSum model. When coverage is used for the XSum model, ROUGE scores (Lin, 2004)
slightly decrease, and the produced summaries contain more severe
hallucinations. However, adding coverage does “fix” some degenerate summaries
that produce the same sequence of tokens repeatedly – see Appendix B for an
example.

For both datasets, in addition to the output summaries, we record the value of
the $p_{\text{gen}}$ switch for each emitted token, as well as the generation
distribution and the copy distribution at each time step.

##  3 Experiments

(a) CNN/DailyMail

(b) XSum

Figure 1: (Top) Correlation contributions
$\operatorname{CC}(p_{\text{gen}},H_{\text{gen}})$ (green) and
$\operatorname{CC}(p_{\text{gen}},H_{\text{copy}})$ (purple) for a randomly-
sampled summary. (Bottom) Bar plot of per-token $p_{\text{gen}}$ (orange), and
entropy of the generation distribution (green) and copy distribution (purple)
for the same summary.

In Section 3.1 we qualitatively analyze the evolution of the per-token
$p_{\text{gen}}$ and uncertainty in the extractive/abstractive components over
the course of randomly selected summaries. Section 3.2 provides quantitative
evidence of our observations across the full test sets, by modeling the
lexical, structural, and distributional ($P_{\text{vocab}}$ and
$P_{\text{copy}}$) environments that drive the variability of the
${p_{\text{gen}}}$ switch.

Finally, in Section 3.3 we manipulate $p_{\text{gen}}$ of the CNN/DailyMail
model to generate summaries that are more abstractive than those of the base
model, in order to disentangle any abstractive _behavior_ from abstractive
_capabilities_ , finding that the model’s abstractive capabilities are largely
limited to lexical paraphrases, and that forcing the model to generate more
novel text yields unfaithful summaries.

###  3.1 Token-level Analysis

####  3.1.1 Model

The $p_{\text{gen}}$ switch explicitly tells us how much weight is assigned to
the generation and copy distributions. See et al. (2017) make qualitative
claims about the environments where ${p_{\text{gen}}}$ is highest: “We find
that ${p_{\text{gen}}}$ is highest at times of uncertainty such as the
beginning of sentences, the join between stitched-together fragments, and when
producing periods that truncate a copied sentence.” In this section, we
evaluative these observations on randomly selected summaries generated with
each model.

We quantify the notion of “uncertainty” from See et al. (2017) using
information-theoretic entropy Shannon (1948) of the distribution that predicts
the next word ${w_{i}}$ of a generated summary:

$H_{\theta}(w_{i})=\mathbb{E}_{P_{\theta}}\left[-\log
P_{\theta}(w_{i})\right].$ (4)

where $P_{\theta}$ is the predictive distribution over the model vocabulary
$V_{\theta}$ at a given time step. In our experiments, we use _normalized_
entropy, which divides the equation above by $\log_{2}|V_{\theta}|$, to limit
the domain to $[0,1]$ regardless of the vocabulary size. We calculate model-
internal entropies $H_{\text{gen}}$ and $H_{\text{copy}}$ by setting
$P_{\theta}$ equal to $P_{\text{vocab}}$ and $P_{\text{copy}}$, respectively.

Given the entropy of the copy and generation distributions at each decoder
time step, we investigate the relationship between $p_{\text{gen}}$,
$H_{\text{gen}}$, and $H_{\text{copy}}$ by calculating per-token _correlation
contributions_. Intuitively, correlation contribution measures how much an
individual token contributes to either positive or negative correlation
between $p_{\text{gen}}$ and the model entropies.

The Pearson correlation coefficient between two sequences
$\mathbf{x}=[x_{1},\ldots,x_{n}]$ and $\mathbf{y}=[y_{1},\ldots,y_{n}]$ can be
written as

$r=\frac{\sum_{i=1}^{n}(x_{i}-\bar{x})(y_{i}-\bar{y})}{\sqrt{\sum_{i=1}^{n}(x_{i}-\bar{x})^{2}\sum_{i=1}^{n}(y_{i}-\bar{y})^{2}}}$
(5)

We calculate the correlation contribution of the pair $(x_{i},y_{i})$ at index
$i$ to be

$\operatorname{CC}_{i}=\frac{n(x_{i}-\bar{x})(y_{i}-\bar{y})}{\sqrt{\sum_{i=1}^{n}(x_{i}-\bar{x})^{2}\sum_{i=1}^{n}(y_{i}-\bar{y})^{2}}}$
(6)

Note that the correlation between $\mathbf{x}$ and $\mathbf{y}$ is equal to
the average of
$\operatorname{CC}_{1},\operatorname{CC}_{2},\ldots,\operatorname{CC}_{n}$,
but unlike $r$, the correlation coefficient, each component
$\operatorname{CC}_{i}$ is not bounded by $[-1,1]$.

####  3.1.2 Results

Across the test splits, the Pearson correlation betwen $p_{\text{gen}}$ and
$H_{\text{gen}}$ is $-0.47$ for CNN/DailyMail and $-0.55$ for XSum. The
correlation between $p_{\text{gen}}$ and $H_{\text{copy}}$ is $0.12$ for
CNN/DailyMail and $0.54$ for XSum. This suggests that the higher-certainty
(lower $H$) distribution is weighted more heavily when combining the
generation and copy distributions, since $p_{\text{gen}}$ is high when
$H_{\text{gen}}$ is low, and low when $H_{\text{copy}}$ is low.

Visualizing the correlation contributions across a sentence helps us
understand how individual tokens are decoded as a function of uncertainty in
the abstractive and extractive components of the model. We randomly sample
articles from each dataset’s test split, and visualize the correlation
contributions for the generated summaries in Figure 1. Additional examples may
be found in Appendix A.

CNN/DailyMail: The tokens that correlate high $p_{\text{gen}}$ with low
$H_{\text{gen}}$ (high certainty in the abstractive component) are frequently
punctuation, and periods in particular. This punctuation appears to be used to
truncate sentences at a syntactic boundary, a behavior we quantify in Section
3.2. The correlation of high $p_{\text{gen}}$ and high $H_{\text{copy}}$ (low
certainty in the extractive component) comes from tokens including “has”,
“managed”, “.”, and “sterling”; all tokens that appear multiple times in the
source document. This suggests a possible role played by generation to tie
break when the copy distribution has low certainty about which continuation to
copy next.

XSum: The XSum model uses the copy mechanism very infrequently;
$p_{\text{gen}}$ is frequently large. When $p_{\text{gen}}$ is small, we tend
to observe uncertainty in the generative component and certainty in the copy
component, according to entropy measures. In Figure 1, we see this happens
when the proper noun “smiler”, a rollercoaster name, is generated. It also
happens at the beginning of a quotation, indicating that the model has learned
that quotations should be copied from the source document, rather than
generated.

Overall, we see a strong contrast in $p_{\text{gen}}$ values between the two
models. On the extractive-biased CNN/DailyMail dataset, the model learns to
copy frequently, generating where necessary to truncate sentences. On the
generative-biased XSum dataset, the model acts nearly like a simple seq2seq
model, only infrequently using the copy mechanism for the sake of proper nouns
and quotations.222This can also be seen in the contrasting gaps between the
seq2seq and pointer-generator ROGUE scores reported by See et al. (2017) and
Narayan et al. (2018). The former sees a 9-point gap in ROUGE-1, while the
latter reports a 1-point gap.

###  3.2 Probing $p_{\text{gen}}$

In the previous section, we made qualitative observations about the
relationship between $p_{\text{gen}}$ and model entropies, as well as the
linguistic environments where $p_{\text{gen}}$ is highest. In this section, we
quantify these relationships by predicting $p_{\text{gen}}$ with a linear
model of lexical, syntactic and distributional factors.

####  3.2.1 Model Features

In this section, we describe the four feature sets we use to model
${p_{\text{gen}}}$. These include model-internal entropy measures from the See
et al. (2017) summarizer, model-external entropy measures derived from
pretrained language models, structural features derived from syntactic parses
of summaries, and part-of-speech tags.

Summarization model entropies: We use $H_{\text{gen}}$ and $H_{\text{copy}}$
as features, hypothesizing, like See et al. (2017), that the uncertainty in
the copy and generation distributions will have a significant effect on
$p_{\text{gen}}$.

Language model entropies: We also use entropy from three types of language
models with varying degrees of lexical and structural expressiveness: a
trigram model,333A Kneser-Ney trigram model trained on 5.4m tokens of the
articles from the training partition of the summarization dataset. a top-down
incremental constituency parser Roark (2001); Roark et al. (2009), and a
unidirectional recurrent neural language model van Schijndel et al. (2019).
These models allow us to directly measure how much ${p_{\text{gen}}}$ may be
influenced by lexical, syntactic, and distributional uncertainty in the
generated summary independent of the summarization objective.

Structural Features: The summarization model may also condition its decision
to copy or generate on the current syntactic environment. While pointer-
generator models do not explicitly model syntax, they may exhibit some
implicit syntactic knowledge, such as the ability to identify and copy whole
constituents. As mentioned above, See et al. (2017) claim that
$p_{\text{gen}}$ is high at the “the join between stitched-together
fragments.” Structural features allow us to quantify this, seeing whether the
model has learned to prefer copying or generation in particular syntactic
environments.

We incorporate two structural measures into our model: the root distance of
word $w_{i}$, denoted as ${\operatorname{D_{\text{root}}}(w_{i})}$ and the
edge distance between word $w_{i-1}$ and $w_{i}$, denoted as
${\operatorname{D_{\text{edge}}}(w_{i-1},w_{i})}$. These measures are
calculated on parse trees of generated summaries.444Parses and part of speech
tags are generated by the top-down constituency parser. Root distance is the
distance in the parse tree from the current word to the root node, and
corresponds to the depth of the word in the parse tree. This measure will tell
us if there is an association between depth in the tree and the decision to
copy or generate. Edge distance is the number of intervening edges between the
current and previous word in the summary. Edge distance will be smaller within
a constituent than across two constituents. This measure allows us to test
whether the decision to copy or generate is associated with the size of the
syntactic boundary between words.

Part of Speech: In addition to structure, the summarization model may
condition its decision to copy or generate on the syntactic category of the
most recently generated word. For example, in our preliminary qualitative
observations of the CNN/DailyMail model, we found that ${p_{\text{gen}}}$ was
higher when decoding punctuation, main verbs and conjunctions. To test the
association between part-of-speech and ${p_{\text{gen}}}$ formally, we include
the part-of-speech label of the current word in our model.

####  3.2.2 CNN/DailyMail Results

We predicted $p_{\text{gen}}$ using four single feature-set linear models, and
a single linear model including all features. We conducted ANOVA tests on all
combinations of nested models, and found that each set of features
significantly improves the $p_{\text{gen}}$ model (all $p<0.00001$; see Table
1).

Entropies: The coefficients for the model-internal entropy measures
$H_{\text{gen}}$ and $H_{\text{copy}}$ intuitively indicate that as
uncertainty in the generation distribution increases, the model is less likely
to generate, and as uncertainty in the copy distribution increases, the model
is less likely to copy; these relationships were previously explored in
Section 3.1.

The three language model entropy estimates are significantly associated with
${p_{\text{gen}}}$. However, the coefficients are all very small and this
feature set individually does the poorest job of explaining
${p_{\text{gen}}}$’s variance of all the sets we analyzed. This could be due
to the fact that, with the exception of the $n$-gram model, the language model
entropy estimates come from different training data than the summarization
model. Regardless, while language model entropies significantly improved
$p_{\text{gen}}$ prediction, the other feature sets showed a much stronger
relationship with ${p_{\text{gen}}}$. Therefore we do not focus on language
model entropies in subsequent sections.

Feature Set | Feature | $\beta$
---|---|---
Summ. Model Entropies | ${H_{\text{gen}}}$ | -0.052
($R^{2}$ = 0.274) | ${H_{\text{copy}}}$ | 0.035
LM Entropies | ${H_{\text{LSTM}}}$ | 0.009
($R^{2}$ = 0.140) | ${H_{\text{parser}}}$ | 0.003
| ${H_{\text{ngram}}}$ | 0.009
Structural Features | ${\operatorname{D_{\text{edge}}}(w_{i-1},w_{i})}$ | 0.018
($R^{2}$ = 0.204) | ${\operatorname{D_{\text{root}}}(w_{i})}$ | -0.031
| $ | -0.130
| UH | -0.118
| # | -0.116
Part of Speech | NNP | -0.111
($R^{2}$ = 0.593) | WRB | 0.156
| : | 0.254
| , | 0.269
| . | 0.636
Full Model $R^{2}$: 0.648 |  |
Table 1: Table of slope coefficients ${\beta}$ in the full linear model of
${p_{\text{gen}}}$ in the CNN/DailyMail model. Reported below the name of the
feature set is the adjusted ${R^{2}}$ of a model fit only to that feature set.
The eight part of speech tags with the largest magnitude ${\beta}$ are
reported. All reported ${\beta}$ are significant via t-test (all
${p<0.00001}$).

Structural Features: Both structural features are significantly associated
with ${p_{\text{gen}}}$. A model fit using only
$\operatorname{D_{\text{edge}}}$ and $\operatorname{D_{\text{root}}}$ explains
20% of ${p_{\text{gen}}}$’s variance ($R^{2}=0.204$). Edge-distance is
positively associated with ${p_{\text{gen}}}$, meaning the larger the
syntactic boundary between the previous and current word, the more likely the
summarization model is to generate. This provides evidence that the model has
some knowledge of syntactic boundaries, and uses the generation component as a
means of joining together clauses, in line with the observations of See et al.
(2017). We also find that distance to the root node of the parse is negatively
associated with ${p_{\text{gen}}}$. This means that words which are higher in
the parse tree are more likely to be generated than copied. Conversely, this
means that generated components are unlikely to be associated with complex,
deeply nested phrasing, suggesting the generation component only produces
simple shallow substitutions rather than structurally complex paraphrases or
even simple substitutions that modify structurally complex copied elements.

Part-of-Speech: The part of speech tags with the highest negative association
with ${p_{\text{gen}}}$ (i.e. those most likely to be copied) are $ (currency
symbols), UH (interjection), # (pound symbol), followed by NNP (singular
proper nouns). These results are perhaps unsurprising, as interjections and
proper nouns are difficult to paraphrase and are often out-of-vocabulary in
the generation component of the summarization model. $ and # serve as prefixes
to numerical values which cannot be faithfully paraphrased and therefore
should be copied directly from the source text. The tag for a cardinal number
(CD) also has a relatively strong negative correlation with ${p_{\text{gen}}}$
(${\beta}$ = -0.088).

The part-of-speech tags with the highest positive association with
${p_{\text{gen}}}$ (i.e. those most likely to be generated) are “.” (sentence-
final punctuation), “,” (comma), “:” (colon), and WRB (wh-adverbs, such as
”where” or ”when”). All of these categories can link two clauses or complete
sentences, consistent with the “stitching” hypothesis of See et al. (2017).

Figure 2: Distribution of ${p_{\text{gen}}}$ across all tokens in the test
split of the CNN/DailyMail corpus. Sentence-final punctuation makes up 5% of
tokens in the dataset, which accounts for 22% of ${p_{\text{gen}}}$’s mass

The mean ${p_{\text{gen}}}$ value of all tokens in the test dataset was 0.204,
while the mean ${p_{\text{gen}}}$ value for sentence-final tokens was 0.915.
Further inspection of the ${p_{\text{gen}}}$ distribution reveals a cluster of
outliers at ${p_{\text{gen}}}$ = 1.0. Figure 2 shows the distribution of
${p_{\text{gen}}}$ values. We find that, of all tokens with
${p_{\text{gen}}}>0.95$, 92.1% are sentence-final punctuation. Despite making
up 5% of all tokens, periods account for 22.1% of the total mass of
${p_{\text{gen}}}$ in the dataset. This suggests that sentence final
punctuation is entirely controlled by the generation distribution.
Additionally, we find that of all 5-grams in generated-summaries ending with
sentence-final punctuation, 52% are also present in the article text, compared
to 12% in the reference summaries. Despite the large ${p_{\text{gen}}}$ values
exhibited by sentence-final punctuation, the model only generates punctuation
in novel contexts less than half of the time, suggesting that even when the
model heavily utilizes its generative component, it essentially generates a
copy of the source text.

Our explanatory model of ${p_{\text{gen}}}$ shows that model entropy,
syntactic depth, syntactic boundary size, and part-of-speech are associated
with ${p_{\text{gen}}}$. The strongest predictor of ${p_{\text{gen}}}$ is the
part-of-speech of the current word, with copying most strongly associated with
numbers, number prefixes and proper nouns, and generation most strongly
associated with punctuation. We find that sentence-final punctuation is
handled almost entirely by the generative component of the model, despite the
fact that sentence-final punctuation occurs in novel contexts less than half
of the time.

####  3.2.3 XSum Results

Overall, we find that the variance of $p_{\text{gen}}$ in the XSum model is
well explained by model-internal entropy, and relatively poorly explained by
linguistic features. We believe this is driven by the categorically different
behaviors of each model.555The full table of model coefficients can be found
in Table 5 of Appendix C. While the CNN/DailyMail model only uses the
generative component to join together copied constituents, the generative
component dominates the XSum model’s behavior. The mean $p_{\text{gen}}$ value
across all tokens in the XSum dataset was 0.828, compared to 0.204 in the
CNN/DailyMail dataset. While the structural features
${\operatorname{D_{\text{edge}}}(w_{i-1},w_{i})}$ and
${\operatorname{D_{\text{root}}}(w_{i})}$ explained 20.4% of the variance of
$p_{\text{gen}}$ in the CNN/DailyMail model, these features only explain 4.9%
of the variance in the XSum model. Part of speech also does a poorer job of
explaining the variance in XSum’s $p_{\text{gen}}$. While part of speech
explains 59.3% of the variance of $p_{\text{gen}}$ in the CNN/DailyMail model,
part of speech tags only explain 23.0% in the XSum model.

While the CNN/DailyMail model assigned an abnormally high $p_{\text{gen}}$
value to punctuation, we do not observe this behavior in the XSum model. The
CNN/DailyMail model appeared to make use of the “.”, “:” and “,” tokens to
join together copied sentences, but none of these tokens are a significant
predictor of $p_{\text{gen}}$ in the XSum model. This suggests that the XSum
model does not use the generation distribution to connect copied clauses.

While the XSum model appears not to use the copy and generation distributions
in the same way as the CNN/DailyMail model, we still observe some clear and
intuitive associations between part of speech tags and $p_{\text{gen}}$. In
particular, the XSum model appears to use the copy distribution to handle
words which are likely to be out-of-vocabulary for the generation
distribution. For example, singular and plural proper nouns, interjections and
foreign words (NNP, NNPS, UH, and FW respectively) are associated with low
values of $p_{\text{gen}}$ (copying), while all types of verbs are associated
with large values of $p_{\text{gen}}$ (generation).

We conclude that the CNN/DailyMail model primarily makes use of lexical and
syntactic information such as clause boundaries and punctuation to modulate
between copying and generation. By contrast, the XSum model primarily relies
on the generation distribution, and backs off to the copy distribution at
times of high generation uncertainty or high copy certainty, such as when
copying a quote or a proper name.

###  3.3 Modifying $p_{\text{gen}}$

Article Text: raheem sterling has admitted he is not ready to sign a new
contract at liverpool deal despite being offered a # 100,000-a-week deal to
stay with the merseyside club . the 20-year-old wideman edged closer towards
an anfield exit after revealing in an interview with the bbc on wednesday that
he would have signed for a lot less a year ago . however , despite being one
of liverpool ’s star men , sterling has struggled to repeat the impressive
form he showed for the reds last season . the england international has
managed just six goals this season - one less than stoke frontman jon walters
- while his conversion rate and minutes per goal ratio have worsened as the
graphic below shows . raheem sterling has managed just six goals this season -
one less than stoke forward jon walters -lrb- left -rrb- .
---
Reference: raheem sterling has revealed he is not ready to sign a new
liverpool deal . the reds wideman has struggled to repeat last season’s
impressive form . the 20-year-old liverpool star has managed just six goals
this season . read: sterling insists he is not a ’money-grabbing 20-year-old’
sterling: what he said about contract talks… and what he meant . click here
for the latest liverpool news .
$\mathbf{p_{\text{min}}=0}$: raheem sterling has admitted he is not ready to
sign a new contract . the england international has managed just six goals
this season . sterling has managed just six goals this season - one less than
stoke forward jon walters .
$\mathbf{p_{\text{min}}=0.25}$: raheem sterling has admitted he is not ready
to sign a new contract . the england international has managed just six goals
this season . the england international has managed just six goals this season
.
$\mathbf{p_{\text{min}}=0.50}$: raheem sterling has admitted he is not ready
to sign a new contract . the england international has managed just six goals
this season . the england international has managed just six goals this season
.
$\mathbf{p_{\text{min}}=0.75}$: raheem sterling has admitted he is not ready
to sign a new deal . the 20-year-old has scored just six premier league goals
this season . the 20-year-old has scored just three goals this season .
$\mathbf{p_{\text{min}}=1}$: man utd face manchester city in the premier
league on saturday . the striker has scored just four premier league goals
this season . the 19-year-old has scored just three goals this season . click
here for all the latest premier league news .
Table 2: Summaries generated for the same randomly selected article with
varying values of $p_{\text{min}}$. Differences from the base model summary
are highlighted in blue, while non-faithful text is highlighted in red.

####  3.3.1 Model

Taking advantage of the smooth interpolation between the generation and copy
distribution, we experiment with forcing the CNN/DailyMail model to be more
abstractive. This, we expect, will allow us to differentiate between the
abstractive _behavior_ we observe in the model summaries and the abstractive
_capabilities_ that the model may have but which it only uses infrequently in
practice. We do so by artificially modifying $p_{\text{gen}}$ during decoding.
If $p_{\text{min}}\in[0,1]$ is a parameter that represents the minimum value
of $p_{\text{gen}}$ we allow, we then modify $p_{\text{gen}}$ as follows:

$p_{\text{gen}}^{*}=p_{\text{min}}+(1-p_{\text{min}})p_{\text{gen}}$ (7)

This may be viewed as a linear interpolation from the range $[0,1]$ to
$[p_{\text{min}},1]$. As $p_{\text{min}}$ grows, the model is forced to rely
more heavily on the generation distribution rather than the copy
distribution.666We do not run this experiment on the XSum model because it
already usually has a large $p_{\text{gen}}$.

####  3.3.2 Results

We use the same randomly sampled articles used in Section 3.1.777We see
similar patterns in other randomly-sampled summaries, shared in Appendix B.
Generated summaries for $p_{\text{min}}$ values in $[0,0.25,0.50,0.75,1.0]$
can be found in Table 2.

Consistent with previous studies, we find that the model is effective at
producing grammatical output. At small values of $p_{\text{gen}}$, the model
mostly copies sentences verbatim, but shows the ability to cut a sentence
short in a grammatical manner. For example, “raheem sterling has admitted he
is not ready to sign a new contract at liverpool deal…” is shortened to
“raheem sterling has admitted he is not ready to sign a new contract.”

At greater values of $p_{\text{gen}}$, the model continues sentences in a
consistent fashion despite substituting nouns or verbs at the beginning or
middle of the sentences. For example, “sterling has managed just six goals…”
at $p_{\text{min}}=0$ becomes “the 20-year-old has scored just six premier
league goals” at $p_{\text{min}}=.75$. However, we do not observe significant
paraphrasing beyond these simple substitutions, and at high values of
$p_{\text{min}}$, where the model is forced to rely heavily on the generation
distribution, we begin to observe hallucinations where the model inserts
inaccurate information about the player’s age and the number of goals scored.
When $p_{\text{min}}=1$, the model generates a completely hallucinated
sentence, “man utd face manchester city in the premier league on saturday” and
a non-informative advertisement “click here for all the latest premier league
news.”

##  4 Discussion

Understanding the limitations preventing abstractive summarization models from
paraphrasing effectively is our ultimate aim, but answering that question
requires an understanding of current models’ abstraction capabilities. In this
paper, we analyze the abstractions of which the pointer-generator model See et
al. (2017) is capable.

When trained on CNN/DailyMail, we find that sentence truncation is the most
common form of paraphrasing. Punctuation tokens are associated with high
generation rates and low entropy in the generation distribution. Additionally,
high $p_{\text{gen}}$ often results in generating the token that comes next in
a phrase already being copied verbatim, suggesting that high $p_{\text{gen}}$
merely gives the model the _option_ to generate novel text, but that the model
rarely makes use of it. Artificially increasing $p_{\text{gen}}$ does not
significantly change this behavior, introducing increased rates of synonym
substitution as well as increased rates of non-faithful hallucination.

When trained on XSum, the model makes much less use of the copy mechanism,
largely generating novel text with a few exceptions, including the copying of
proper nouns and parts of quotations. The model generally produces topical
summaries, but ones that aren’t necessarily grammatical or faithful to the
original article. For example, the randomly selected summary used in Figure 1
repeats itself and wanders, “… on the smiler rollercoaster on the smiler
rollercoaster in the south west 200 years ago as ‘phenomenal”’. This comes
after a hallucination, “firefighters are continuing to search for a man” even
though the article describes the rescue from the rollercoaster crash in the
past tense. We hypothesize that the phrase “firefighters are continuing to
search” is a relatively common phrase in news articles that the model learned
from the training data. Such frequency biases likely contribute to the
faithfulness issues in abstractive summarizers reported in previous
literature.

Our results give context to previous observations that summarization model
unfaithfulness increases with abstraction Maynez et al. (2020); Durmus et al.
(2020); Kryscinski et al. (2020) and that abstractive models are prone to
output repetition See et al. (2019); Holtzman et al. (2020). To faithfully
paraphrase, a model must understand both the syntax and the semantics of the
original text. The models we studied were able to recognize syntactic
boundaries, proper nouns, and noun phrases that could be substituted with
synonyms. However, the models didn’t appear to comprehend the meaning of the
text well enough to generate faithful complex paraphrases. This is
unacceptable in high-risk domains such as healthcare; Zhang et al. (2018b)
train a model to summarize radiology findings, but only 67% of their summaries
are judged at least as good as human summaries, in a domain where errors can
have a major impact on human lives.

In our work, the explicit switch between abstractive and extractive modes
enabled us to directly observe the conditions under which abstractive
summarization was chosen as a strategy, and to force an abstractive
summarization strategy to disentangle paraphrasing behavior from capabilities.
We found that the See et al. (2017) model trained on CNN/DailyMail did learn
simple forms of paraphrasing, despite the extractive bias of the dataset. We
conclude that pointer-generator models are _capable_ of simple paraphrasing
regardless of training data, even though they _behave_ in ways that rely on
the frequency biases of the training dataset. However, they also appear
_incapable_ of producing significant paraphrases that are grammatical, non-
repetitive, and faithful to the source document. This suggests that using an
abstractive-biased dataset alone is not enough for a model to learn robust and
faithful paraphrasing strategies. Rather, when trained on XSum, the pointer-
generator model seems to simply learn that it should not copy from the source
text. Future work should investigate how either datasets or models can improve
the training signal that allows the model to understand the underlying
semantics of the source document.

Related to our work, Xu et al. (2020) studied the summarization strategies of
state-of-the-art transformer summarization models. Since their models did not
contain an explicit copy/generation switch, they used $n$-gram overlap between
source documents and summaries as a proxy to measure a summary’s
“extractiveness.” They found a similar result to ours, that high $n$-gram
overlap (“copying”) corresponded to low entropy in the decoder’s output
distribution when the model was trained on CNN/DailyMail.888Their findings are
more difficult to interpret when trained on XSum, partially due to the lack of
an explicit extractive/abstractive summarization switch in their models. Their
findings suggest that our results likely generalize to a much broader class of
summarization models than the pointer-generator models studied here.

Finally, Liu and Liu (2010) found that ROUGE metrics poorly correlate with
human evaluations, leading to recent models being evaluated with human
judgements, but these evaluations often disagree on what they are measuring,
whether it is faithfulness, informativity, or the unqualified “quality” of a
summary Zhang et al. (2018a, 2020); Dou et al. (2020). Developing best
practices on how abstractive summarizers should be evaluated for their
paraphrasing ability is another problem we leave for future work.

##  5 Conclusion

In this paper, we presented three experiments that evaluate the abstraction
capabilities of the pointer-generator neural summarization model. Our results
conclude that on extractive training data, the model uses only simple
paraphrasing strategies that truncate sentences at syntactic boundaries,
allowing the model to stay grammatically accurate as well as faithful to the
source document. We explore two ways to make the model use abstractive
summarization strategies: modifying the model so that it relies more heavily
on its abstractive component, and training a new model on an abstractive-
biased dataset. In both cases, the model shows simple paraphrasing
capabilities but frequently generates unfaithful paraphrases. These results
highlight current limitations of abstractive summarization, where in lieu of
semantic understanding, models must rely on extractive heuristics in order to
stay faithful.

## Acknowledgements

We thank our reviewers for their helpful suggestions. We also thank Esin
Durmus, Ryan Benmalek, and Claire Cardie for helpful discussions about
abstractive summarization. Finally, we thank Shashi Narayan and Shay Cohen for
helping us reproduce their pointer-generator model trained on XSum.

## References

  * Cohan et al. (2018) Arman Cohan, Franck Dernoncourt, Doo Soon Kim, Trung Bui, Seokhwan Kim, Walter Chang, and Nazli Goharian. 2018.  A discourse-aware attention model for abstractive summarization of long documents.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)_ , pages 615–621, New Orleans, Louisiana. Association for Computational Linguistics.
  * Dou et al. (2020) Zi-Yi Dou, Pengfei Liu, Hiroaki Hayashi, Zhengbao Jiang, and Graham Neubig. 2020\.  Gsum: A general framework for guided neural abstractive summarization.  _arXiv preprint arXiv:2010.08014_.
  * Durmus et al. (2020) Esin Durmus, He He, and Mona Diab. 2020.  FEQA: A question answering evaluation framework for faithfulness assessment in abstractive summarization.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 5055–5070, Online. Association for Computational Linguistics.
  * Ganesan (2018) K. Ganesan. 2018.  Rouge 2.0: Updated and improved measures for evaluation of summarization tasks.  _ArXiv_ , abs/1803.01937.
  * Holtzman et al. (2020) Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020.  The curious case of neural text degeneration.
  * Kryscinski et al. (2020) Wojciech Kryscinski, Bryan McCann, Caiming Xiong, and Richard Socher. 2020.  Evaluating the factual consistency of abstractive text summarization.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 9332–9346, Online. Association for Computational Linguistics.
  * Lin (2004) Chin-Yew Lin. 2004.  ROUGE: A package for automatic evaluation of summaries.  In _Text Summarization Branches Out_ , pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
  * Liu and Liu (2010) F. Liu and Y. Liu. 2010.  Exploring correlation between rouge and human evaluation on meeting summaries.  _IEEE Transactions on Audio, Speech, and Language Processing_ , 18(1):187–196.
  * Maynez et al. (2020) Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. 2020.  On faithfulness and factuality in abstractive summarization.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 1906–1919, Online. Association for Computational Linguistics.
  * Narayan et al. (2018) Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018.  Don’t give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 1797–1807, Brussels, Belgium. Association for Computational Linguistics.
  * Roark (2001) Brian Roark. 2001.  Probabilistic top-down parsing and language modeling.  _Computational Linguistics_ , 27(2):249–276.
  * Roark et al. (2009) Brian Roark, Asaf Bachrach, Carlos Cardenas, and Christophe Pallier. 2009.  Deriving lexical and syntactic expectation-based measures for psycholinguistic modeling via incremental top-down parsing.  pages 324–333.
  * Rush et al. (2015) Alexander M. Rush, Sumit Chopra, and Jason Weston. 2015.  A neural attention model for abstractive sentence summarization.  In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_ , pages 379–389, Lisbon, Portugal. Association for Computational Linguistics.
  * van Schijndel et al. (2019) Marten van Schijndel, Aaron Mueller, and Tal Linzen. 2019.  Quantity doesn’t buy quality syntax with neural language models.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 5831–5837, Hong Kong, China. Association for Computational Linguistics.
  * See et al. (2017) Abigail See, Peter J. Liu, and Christopher D. Manning. 2017.  Get to the point: Summarization with pointer-generator networks.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1073–1083, Vancouver, Canada. Association for Computational Linguistics.
  * See et al. (2019) Abigail See, Aneesh Pappu, Rohun Saxena, Akhila Yerukola, and Christopher D. Manning. 2019.  Do massively pretrained language models make better storytellers? In _Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL)_ , pages 843–861, Hong Kong, China. Association for Computational Linguistics.
  * Shannon (1948) C. E. Shannon. 1948.  A mathematical theory of communication.  _Bell System Technical Journal_ , 27(3):379–423.
  * Xu et al. (2020) Jiacheng Xu, Shrey Desai, and Greg Durrett. 2020.  Understanding neural abstractive summarization models via uncertainty.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 6275–6281, Online. Association for Computational Linguistics.
  * Zhang et al. (2018a) Fang-Fang Zhang, Jin-ge Yao, and Rui Yan. 2018a.  On the abstractiveness of neural document summarization.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 785–790.
  * Zhang et al. (2020) Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter Liu. 2020.  Pegasus: Pre-training with extracted gap-sentences for abstractive summarization.  In _International Conference on Machine Learning_ , pages 11328–11339. PMLR.
  * Zhang et al. (2018b) Yuhao Zhang, Daisy Yi Ding, Tianpei Qian, Christopher D. Manning, and Curtis P. Langlotz. 2018b.  Learning to summarize radiology findings.  In _Proceedings of the Ninth International Workshop on Health Text Mining and Information Analysis_ , pages 204–213, Brussels, Belgium. Association for Computational Linguistics.

## Appendix A. Additional Correlation Contribution Examples

This appendix includes additional examples of
$\operatorname{CC}(p_{\text{gen}},H_{\text{gen}})$, the per-token correlation
contributions for randomly selected summaries.

(a) CNN/DailyMail Example 2

(b) CNN/DailyMail Example 3

Figure 3: Bar plot of per-token $p_{\text{gen}}$ and entropy of the generation
distribution (purple) and copy distribution (blue), plotted under correlation
contributions $\operatorname{CC}(p_{\text{gen}},H_{\text{gen}})$ (purple) and
$\operatorname{CC}(p_{\text{gen}},H_{\text{copy}})$ (blue) for a randomly-
sampled CNN/DailyMail test summaries.

(a) XSum Example 2

(b) XSum Example 3

Figure 4: Bar plot of per-token $p_{\text{gen}}$ and entropy of the generation
distribution (purple) and copy distribution (blue), plotted under correlation
contributions $\operatorname{CC}(p_{\text{gen}},H_{\text{gen}})$ (purple) and
$\operatorname{CC}(p_{\text{gen}},H_{\text{copy}})$ (blue) for a randomly-
sampled XSum test summaries.

## Appendix B. Additional Summaries With Modified $p_{\text{min}}$

This appendix contains additional examples of randomly selected summaries
generated with modified values of $p_{\text{min}}$.

Article Text: gigi hadid ’s younger sister bella hadid is quickly catching up
with her star sibling ’s phenomenal fashion success , carving out an
impressive career for herself within the industry . the 18-year-old , who is
just 18 months younger than blonde bombshell gigi , 19 , stars in a powerful
and provocative new shoot featured in the may issue of elle magazine , which
sees her modeling a series of risque and revealing ensembles . in one image ,
the dark-haired beauty poses in a sheer fleur du mal lace bodysuit , with her
arms thrown casually above her head . scroll down for video . it ’s good to be
a model : bella hadid models a tantalizing $ 895 fleur du mal lace bodysuit
and $ 1,250 giuseppe zanotti boots . giuseppe zanotti cowboy boots . on sale
for $ 500 at 6pm ! visit site . she may be just 18 , but her sartorial sense
goes far beyond her years . and in her latest spread for elle magazine , bella
hadid wore several risque pieces that you would n’t see on the average
teenager . but then again , the budding supermodel is n’t a normal teen […]
---
Reference: bella, 18, is the younger sister of guess campaign star gigi hadid,
19 . the rising star poses in a series of provocative outfits for the may
issue of elle . fellow fashion favorite hailey baldwin also features in the
issue, appearing in her own separate shoot and interview .
$\mathbf{p_{\text{min}}=0}$: the 18-year-old , who is just 18 months younger
than blonde bombshell gigi , 19 , stars in a powerful and provocative new
shoot featured in the may issue of elle magazine , which sees her modeling a
series of risque and revealing ensembles . in her latest spread for elle
magazine , bella hadid wore several risque pieces that you would n’t see on
the average teenager . but her sartorial sense goes far beyond her years .
$\mathbf{p_{\text{min}}=0.25}$: the 18-year-old , who is just 18 months
younger than blonde bombshell gigi , 19 , stars in a powerful and provocative
new shoot featured in the may issue of elle magazine , which sees her modeling
a series of risque and revealing ensembles . in her latest spread for elle
magazine , bella hadid wore several risque pieces that you would n’t see on
the average teenager .
$\mathbf{p_{\text{min}}=0.50}$: the 18-year-old is just 18 months younger than
blonde bombshell . she is just 18 months younger than blonde bombshell gigi ,
19 . she is just 18 months younger than blonde bombshell gigi .
$\mathbf{p_{\text{min}}=0.75}$: the 18-year-old is just 18 months younger than
blonde bombshell . she is just 18 months younger than blonde bombshell . she
is just 18 months younger than blonde bombshell . she is just 18 months
younger than blonde bombshell .
$\mathbf{p_{\text{min}}=1}$: the [UNK] , 33 , has been dating for three years
. she is the daughter of [UNK] and [UNK] . she is the daughter of [UNK] ’s
daughter . her new album is priced at $ [UNK] -lrb- # [UNK] -rrb- .

Article Text: the owners of this house better not leave too quickly , after a
speed camera was angled directly on their front door . the bright yellow gatso
had previously enforced the 30mph speed limit for motorists along the
residential road in handsworth , birmingham . however , it has not been
working for two years after every single fixed device was switched off in the
west midlands . big brother is watching : a speed camera has been turned round
and is pointing at this house in birmingham , west midlands . the speed camera
has not been working for more than two years […]
---
Reference: speed camera discovered pointing at house in handsworth, birmingham
. fixed cameras switched off across the west midlands in spring of 2013 . site
is not going to be part of a new trial using digital technology . obsolete
camera may now be taken down after engineers examine device .
$\mathbf{p_{\text{min}}=0}$: the bright yellow gatso had previously enforced
the 30mph speed limit . the bright yellow gatso had previously enforced the
30mph speed limit for motorists along the residential road in handsworth ,
birmingham . however , it has not been working for two years after every
single fixed device was switched off in the west midlands .
$\mathbf{p_{\text{min}}=0.25}$: the bright yellow gatso had previously
enforced the 30mph speed limit for motorists along the residential road in
handsworth , birmingham . it has not been working for more than two years
after every single fixed device was switched off in the west midlands . the
speed camera has not been working for more than two years .
$\mathbf{p_{\text{min}}=0.50}$: the bright yellow gatso had previously
enforced the 30mph speed limit . it has not been working for more than two
years . it has not been working for more than two years after every single
fixed device .
$\mathbf{p_{\text{min}}=0.75}$: the bright yellow gatso had previously
enforced the 30mph speed limit . it has not been working for more than two
years . it has not been working for more than two years . it has not been
working for more than two years .
$\mathbf{p_{\text{min}}=1}$: warning : graphic content . it is believed to
have been in the past of the past two years . it is believed to have been in
the past of the past two years .

Table 3: Summaries generated for additional randomly selected articles from
CNN/DailyMail with varying values of $p_{\text{min}}$. Differences from the
base model summary are highlighted in blue, while non-faithful text is
highlighted in red.

Article Text: meaning ” to watch a large number of television programmes (
especially all the shows from one series ) in succession ” , it reflects a
marked change in viewing habits , due to subscription services like netflix .
lexicographers noticed that its usage was up 200 % on 2014 . other entries
include dadbod , ghosting and clean eating . helen newstead , head of language
content at collins , said : ” the rise in usage of ’ binge - watch ’ is
clearly linked to the biggest sea change in our viewing habits since the
advent of the video recorder nearly 40 years ago . ” it ’s not uncommon for
viewers to binge - watch a whole season of programmes such as house of cards
or breaking bad in just a couple of evenings - something that , in the past ,
would have taken months - then discuss their binge - watching on social media
. ” those partaking in binge - watching run the risk of dadbod , one of ten in
the word of the year list […] the list of collins ’ words of the year offers a
fascinating snapshot of the ever - changing english language , ” said newstead
. those words that remain popular could be included in the next print edition
of the collins english dictionary , due in 2018 .
---
Reference: collins english dictionary has chosen binge-watch as its 2015 word
of the year.
Summary: binge - watch ’ binge - watch ’ binge - watch ’ binge - watch ’ binge
- watch ’ binge - watch ’ binge - watch ’ binge - watch ’ binge - watch ’
binge - watch english language .
Summary with Coverage: the risk of binge - watch ’ binge - watch english
language is ” clearly uncommon ” , according to a list of entries from the
collins english media recorder of the year list .

Article Text: writing in her autobiography , she claimed the director ” threw
himself ” on top of her in the back of his limousine and tried to kiss her .
the actress described the encounter as ” an awful , awful moment ” . hedren
added that she did n’t tell anyone because ” sexual harassment and stalking
were terms that did n’t exist ” in the early 1960s . she continued : ” besides
, he was alfred hitchcock […] the actress , now 86 , made the claims in her
autobiography tippi : a memoir , which is published in november . she has
spoken in the past about the director ’s alleged treatment of her , but has
gone into more detail in the memoir . hedren described a later encounter in
hitchcock ’s office where the director ” suddenly grabbed ” her and ” put his
hands ” on her . she wrote : ” it was sexual , it was perverse , and it was
ugly , and i could n’t have been more shocked and more repulsed . ” […] the
actress said hitchcock then made her life difficult , refusing to submit her
work for the oscar nominations or let her take on other acting roles while he
still had her under contract […]
---
Reference: actress tippi hedren has claimed alfred hitchcock sexually harassed
her while they worked together in the 1960s.
Summary: actress hitchcock hedren has said she was ” ugly ” to kiss her as an
” awful ” experience of sexual harassment and stalking in the early hours of
the year , saying she was ” ugly ” .
Summary with Coverage: actress hitchcock hitchcock , best known by the
director of the oscar - winning director , has died at the age of 86 , the
actress has announced on her return to the memoir .

Table 4: Summaries generated for additional randomly selected articles from
XSum with varying values of $p_{\text{min}}$. Summaries with coverage enabled
also included. Non-faithful text is highlighted in red

## Appendix C. Explanatory $p_{\text{gen}}$ Model for XSum Dataset

Feature Set | Feature | $\beta$
---|---|---
Summ. Model Entropies | ${H_{\text{gen}}}$ | -0.099
($R^{2}$ = 0.476) | ${H_{\text{copy}}}$ | 0.093
LM Entropies | ${H_{\text{LSTM}}}$ | 0.009
($R^{2}$ = 0.123) | ${H_{\text{parser}}}$ | 0.003
| ${H_{\text{ngram}}}$ | -0.013
Structural Features | ${\operatorname{D_{\text{edge}}}(w_{i-1},w_{i})}$ | -0.005
($R^{2}$ = 0.049) | ${\operatorname{D_{\text{root}}}(w_{i})}$ | -0.001
| NNPS | -0.166
| FW | -0.162
| UH | -0.143
Part of Speech | NNP | -0.089
($R^{2}$ = 0.230) | VBD | 0.174
| LS | 0.179
| VBN | 0.178
| WP$ | 0.193
Full Model $R^{2}$: 0.547 |  |
Table 5: Table of slope coefficients ${\beta}$ in the full linear model of
${p_{\text{gen}}}$ in the XSum model. Reported below the name of the feature
set is the adjusted ${R^{2}}$ of a model fit only to that feature set. The
eight part of speech tags with the largest magnitude ${\beta}$ are reported.
All reported ${\beta}$ are significant via t-test (all ${p<0.00001}$).