# Investigation of the ensemble of maximal center gauge

Zeinab Dehghan<EMAIL_ADDRESS>Department of Physics, University of
Tehran, Tehran 14395-547, Iran  School of Physics, Institute for Research in
Fundamental Sciences (IPM), P. O. Box 19395-5531, Tehran, Iran  Rudolf
Golubich<EMAIL_ADDRESS>Atominstitut, Technische Universität Wien
Roman Höllwieser<EMAIL_ADDRESS>Department of Physics,
Bergische Universität Wuppertal  Manfried Faber<EMAIL_ADDRESS>Atominstitut, Technische Universität Wien

###### Abstract

Maximal Center Gauge (MCG) aims to detect center vortices by maximizing a
gauge functional and then projecting onto the center elements of the
respective group. The requirement for unrestricted maximization of the gauge
functional has proven to be untenable because it was shown that it leads to an
underestimation of the string tension. To counter this problem, the ensemble
of local gauge maxima is investigated and it is found that the unrestricted
maximization can be replaced by a maximization restricted to the Gaussian
distributed part of the ensemble. Such restricted maximization weakens the
problem of an underestimated string tension.

††preprint: APS/123-QED

##  I Introduction

Important nonperturbative properties of the QCD vacuum are confinement [1] and
dynamical breaking of chiral symmetry [2]. They show that the QCD vacuum is
highly non-trivial and filled with quantum fluctuations. The amount and type
of these quantum fluctuations depend on the temperature. The most interesting
quantum fluctuations that determine the properties in the hadronic phase are
abelian monopoles [3, 4, 5] and center vortices [6, 7]. Abelian monopoles are
defined by a charge in a U(1) subgroup of SU($N$). They move along paths that
percolate through the four-dimensional space-time. Center vortices are closed
color-magnetic flux tubes, quantised to center elements of the gauge group and
form two-dimensional surfaces percolating the vacuum. Projecting the color of
the flux tubes into a U(1) subgroup results in trajectories for abelian
monopoles [8, 9]. The density of these topological excitations determines the
strength of the force between color charges, the QCD string tension [10, 11].
Successful methods for identifying these topological objects are special
gauges, which are determined by maximizing corresponding gauge functionals.
The maximal abelian gauge for magnetic monopoles and the maximal center gauge
(MCG) for vortices have proven to be particularly successful. We examine
maximal center gauge for the SU(2) gauge theory formulated with the Wilson
action (1).

Local maxima of a gauge functional can only be searched for numerically. In
the maximal Abelian gauge, the highest gauge functionals provide the best
description of the QCD vacuum and justify the maximization prescription. Local
maxima with high values of the gauge functional could also be found for MCG,
which predict the correct string tension. However, a more precise search led
to maxima [12, 13] which provide a string tension that is much too low.
Furthermore, it was shown in [14] that with series of random gauge copies with
broken local Z(2) gauge symmetry due to restricting to links with a positive
trace, gauge configurations with a high value of the gauge functional and a
low string tension can easily be generated. These observations strongly
questioned the maximal center gauge and thus the vortex mechanism as an
explanation of confinement. The aim of this work is to show that the error
lies in the requirement of unrestricted maximization of the gauge functional
and that it is not the vortex mechanism itself that is in question. Refs. [15,
16, 17] also point in this direction, in which non-trivial center regions are
determined using a complex computational procedure in order to guide the gauge
fixing.

In this article we explore ensembles of gauge copies for maximal center gauge
to see whether statistical analysis allows for a less resource intensive
extraction of the correct string tension as in [15, 16, 17].

After describing the formalism of maximal center gauge in the second section,
we explain in the third section the problem of the global maximization
prescription of the gauge functional of maximal center gauge. On the way to
the modification of this prescription we examine in the forth section the
distribution of local maxima of the gauge functional. We observe that in a
wide range of the coupling parameter $\beta$ their densities come very close
to Gaussian distributions. In the upper tip of these distributions we find
agreement of the center projected string tensions with the string tensions
extracted from unprojected fields. At the upper boundary of the
$\beta$-window, we find that strong deviations from Gaussians are first
indications for a failing absolute maximization of the gauge functional. In
the further part of the work we try to extend the $\beta$-window to higher
values. First, we remove these deviations by symmetrisation of the
distribution. Another indication is given by the behavior of the slope of the
quark-antiquark potentials of center projected fields. These potentials are
almost linear, as we find in section five. This is an observation that has
already been made in the determination of string tensions from Creutz ratios
[18]. In general, the slope of the potential increases slightly with the size
of Wilson loops. For increasing asymmetry of the gauge functional
distributions, this behavior may invert to its opposite, especially for gauge
configurations with particularly high values of the gauge functional. We find
then a decrease in string tension with the size of the Wilson loops. This may
indicate that the detected P-vortices break apart and do no longer percolate,
as was already assumed in [19]. By removing these vortex detection failures in
section six and symmetrisation of the distributions we can successfully
increase the region where the right tip of the Gaussian distributions
reproduces the asymptotic string tension. In section seven we draw from these
results the conclusion, how to modify the prescription for the maximization
procedure of MCG.

##  II Direct maximal center gauge

We work on a $24^{4}$ lattice with periodic boundary conditions and
investigate field configurations at the inverse coupling constants
$\beta={4}/{g^{2}}\in\\{2.3,2.4,2.5,2.6,2.7\\}$ of the Wilson action for an
SU(2) gauge theory

$S_{\text{gluons}}=\beta\sum_{x,\mu<\nu}\left(1-\frac{1}{2}\mathfrak{Re}\text{Tr}(U_{\mu\nu}(x))\right).$
(1)

The path ordered product of links $U_{\mu}(x)$ delimiting a plaquette in the
$\mu\nu$ plane defines the plaque variable $U_{\mu\nu}(x)$.

In direct maximal center gauge (DMCG), the link variables are transformed by a
SU(2) gauge transformation $g(x)$ shifting them as close as possible to center
elements by maximizing the gauge functional

$R_{\text{MCG}}=\sum_{x}\sum_{\mu}\mid\text{Tr}[{}^{g}U_{\mu}(x)]\mid^{2},$
(2)

where the transformed link variables are
${}^{g}U_{\mu}(x)=g(x+\hat{\mu})\,U_{\mu}(x)\,g^{\dagger}(x)$. These are then
projected onto the nearest center element

${}^{g}U_{\mu}(x)\rightarrow
Z_{\mu}(x)\equiv\mathrm{signTr}[{}^{g}U_{\mu}(x)].$ (3)

A single negative link variable $Z_{\mu}(x)=-1$ in a field of trivial link
variables $Z_{\mu}=+1$ forces the values of the six attached plaquettes to
$Z_{\mu\nu}(x)=-1$. On the dual lattice these six so-called P-plaquettes form
the surface of a cube. It turns out that in the confinement phase, a large
number of negative links are arranged in such a way that the dual P-plaquettes
form extended closed surfaces: vortices percolating through the whole space-
time lattice. The success of the vortex model of confinement is founded in the
fact that the Abelian Stoke’s law applies to Abelian and therefore also to
center vortices. The sign of Wilson loops is flipped by the piercing of thin
projected and also of thick vortices. A high density of random vortices leads
to a fast decrease of the expectation values of Wilson loops with the size of
the loops and thus to large string tensions.

In the unprojected field configurations, vortices are thick magnetic flux
tubes. MCG assumes that despite the strong limitation of the field degrees of
freedom in the projection procedure from SU(2) to Z(2), MCG is able to detect
the shape of thick vortices and therefore to preserve the important long-range
properties of the vacuum configurations.

##  III The problem with the maximization

The gauge functional (2) has a huge number of local maxima. Determining the
global maximum of the gauge functional (2) is an NP-hard problem, therefore
only local maxima can be approached by the conjugate gradient method and the
global maximum can be approximated step by step only. As Engelhardt and
Reinhardt [20] have concluded from analytical considerations, a trivial
maximization prescription cannot be expected to reproduce the correct physics.
A reduction of the string tension with increased values of the gauge
functional was numerically confirmed by Kovacs and Tomboulis [12] and
Bornyakov, Komarov and Polikarpov [13]: the largest maxima of the functional
underestimate the density of vortices and therefore also the string tension. A
possible reason for this is that in the large volume of thick smooth vortices
the center charge is spread over multiple links causing them to be projected
onto the trivial center. The more parts of the thick vortices the P-vortices
miss, the higher are the values of the gauge functional. With rising beta
values, the field configurations generated by the Monte Carlo method and the
Metropolis algorithm become too smooth and it is increasingly difficult to
describe the shape of thick vortices by sudden changes in the link variables
with values of $\pm 1$.

However, the investigations so far show that the gauge functional (2) of MCG
is in principle able to predict the string tension. Here we would like to
mention, (i) the successful predictions by DMCG [21, 22] where only a few
local maxima were examined, (ii) the pre-selection of gauge copies with the
help of the eigenfunctions of the Laplaceoperator [23, 24] which lead to good
results and (iii) averages over the ensemble of local maxima of the gauge
functional [14] which did not suffer from extreme values of $R_{\text{MCG}}$.

The previous prescription for MCG striving for maximization was based on the
assumption that the global maximum corresponds to the most physical gauge.
This objective has clearly been missed due to the nearly linear decrease of
the string tension [14] with increasing value of the gauge functional.

The above enumerated indications encourage us to examine how the information
about the value of string tension is encoded in the properties of the
ensembles of the local maxima. To accomplish this, we determine the
distribution of the ensemble of local maxima and the relation between the
values of the gauge functional and the string tension. Throughout these
investigations, we hope to find hints for which cases the maximization of the
gauge functional leads to incorrect predictions of string tension. Finally,
this should lead us to a modification of the MCG prescription for the vortex
detection.

a)
b)
c)

Figure 1: The distributions of the ensembles of 20 000 local gauge maxima are
depicted for $\beta=2.5$, $\beta=2.6$ and $\beta=2.7$ in logarithmic plots. A
smooth kernel is shown as blue solid line with the histogram of the data
placed in the background. Deviations from the normal distribution are colored
red. Results of distribution tests are shown below each distribution. One can
clearly see increasing deviations from Gaussians and with $\beta$ increasing
skewness.

##  IV Distributions of local maxima

The distribution of the gauge functional values (2) at the local maxima
resembles a Gaussian, see Fig. 1. We have investigated these distributions for
$\beta\in\\{2.3,2.4,2.5,2.6,2.7\\}$ by Monte-Carlo runs with 20 random starts,
3000 initial sweeps and 10 configurations with a distance of 1000 sweeps. For
each of these 200 configurations we performed 100 unbiased random gauge copies
leading to 20 000 values of the gauge functional for each $\beta$. In Fig. 1
we depict logarithmic plots of these distributions for $\beta=2.5,2.6$ and
$2.7$ and bin widths of one-third of the standard deviation. A continuous
replacement for histograms, a smooth kernel distribution, is shown by the full
blue lines. Their differences to Gaussian fits are filled by red areas. Below
the distributions we display the values of the Cram$\acute{e}$r–von Mises
criteria for judging the goodness of the fit to the normal distributions.
p-values below 0.05 indicate statistically significant deviations from normal
distributions. Until a specific value of $\beta$ the ensemble of gauge copies
is well described by a normal distribution. A measure for the deviations from
symmetric distributions is the skewness, the ratio of the third central moment
to the 3/2-power of the second central moment. With increasing $\beta$ the
skewness of the distribution increases as can be seen in Fig. 2.

Figure 2: The skewness of the distributions of the local gauge maxima is
depicted for different values of $\beta$.

This lets us suspect that a partial trivialization of the lattice may occur
during gauge fixing and projection. This effect increases with $\beta$ and
thus an increasing proportion of gauge copies of the ensemble is shifted to
larger values of $R_{\text{MCG}}$ as is hinted in Fig. 1b and can be nicely
seen in Fig. 1c. As we will show in the next sections, this corresponds to a
decrease in the string tension extracted from the expectation values of Wilson
loops.

Fig. 1 shows that deviations from the normal distribution increase with
increasing value of $\beta$. We know already that for the local maxima with
the highest values of the gauge functional (2), the vortex detection fails due
to the above mentioned smoothness of the lattice. This suggests to remove
these misguided detections by symmetrizing the distributions.

Figure 3: Cumulative distribution functions (CDF) of local maxima of gauge
functional (2) for $\beta=2.7$ before and after symmetrisation of the
distribution.

The distributions for $\beta=2.3,2.4$ and $2.5$ are not affected by
asymmetries and should remain unchanged, while $\beta=2.6$ should only be
slightly modified and $\beta=2.7$ should be strongly changed. The cumulative
distribution function (CDF) of local maxima lends itself well to
symmetrisation. This makes it possible to thin out the high ranges of
$R_{\mathrm{MCG}}$-values and leave the remaining ranges unchanged. Fig. 3
shows the CDFs before and after symmetrisation for $\beta=2.7$.

##  V String tensions extracted from center projected Wilson loops

From expectation values of center projected Wilson loops
$W_{\mathrm{cp}}(R,T)$ we extract the potential

$V_{\mathrm{cp}}(R)=\lim_{T\to\infty}\frac{1}{T}\ln W_{\mathrm{cp}}(R,T)$ (4)

and a slope $\sigma_{\mathrm{cp}}$. A characteristic $R$ dependence of
$V_{\mathrm{cp}}$ is shown in Fig. 4.

Figure 4: Potential $V_{\mathrm{cp}}(R)$ of center projected Wilson loops as
functions of their spatial extent $R$ extracted at $\beta=2.5$ from 200
configurations symmetrically arranged around $R_{\text{MCG}}=0.89053$. The
linear approximations for small and large $R$ demonstrate a slight increase in
slope due to short range fluctuations which do not influence the long range
behavior.

The same nearly linear behavior was also observed for Creutz ratios, see Ref.
[18]. It is explained by center vortices randomly piercing the minimal areas
of Wilson loops. We expect this behavior for vortices percolating the lattice.
Short-range fluctuations of vortex surfaces [25] lead to correlated pairs of
piercings for large loops and do not influence their expectation values, but
they increase the number of single piercings for small loops and decrease
therefore the slope of the potential at short distances. This explains the
slight increase of the slope with R for the characteristic example in Fig. 4
from $\sigma_{\mathrm{cp}}=0.0336\pm 3$ to $\sigma_{\mathrm{cp}}=0.0350\pm 3$.

For Creutz ratios, we found [14] that the $\sigma_{\mathrm{cp}}$ values are
approximately linear functions of the local maxima of $R_{\mathrm{MCG}}$. A
corresponding correlation for the slope $\sigma_{\mathrm{cp}}$ of the
potential $V_{\mathrm{cp}}(R)$ is shown in Fig. 5

Figure 5: Comparison of the slopes $\sigma_{\mathrm{cp}}$ of the potential
$V_{\mathrm{cp}}(R)$ with the literature value [26] of the string tension. The
expectation values of the center projected Wilson loops were averaged over 200
local maxima around the indicated $R_{\mathrm{MCG}}$ values. For comparison
the nearly Gaussian distribution of the functional is shown as a dash-dotted
line. The most interesting region is the intersection of the red data with the
literature value. In the right tip of the distribution, we observe the best
agreement of $\sigma_{\mathrm{cp}}$ with the asymptotic string tension.

at small $R$ values in blue and large $R$ in red color. The string tensions
extracted from unprojected configurations [26] is shown by a horizontal black
line. The slope of the potential is generally greater for large distances than
for short distances as explained in the discussion of Fig. 4. We used 200
configurations around the indicated value of $R_{\mathrm{MCG}}$ for the
determination of the potential. Due to the fluctuations of the vortex shapes a
sample of 200 fields out of the 20 000 is obviously still not enough to get a
smooth curve. The larger variations in the slope of the potential for “large
$R$” than for “small $R$” are due to the statistical errors that increase with
the size of the Wilson loops. On average, an almost linear relationship
between the gauge functional and the string tension is observed. In order to
be able to classify the $R_{\mathrm{MCG}}$ values, we show their normalized
density distribution for comparison. With this diagram, we want to focus on
the important region of the distribution of $R_{\mathrm{MCG}}$ values. We can
clearly see that it is the right tip of the distribution around
$R_{\mathrm{MCG}}=0.8905$ where the slopes of the potentials at large
distances approaches the literature value [26] of the string tension. There
the density of local maxima of the gauge functional has already decreased by a
factor of about 10.

For the lowest $\beta$ values, $V_{\mathrm{cp}}(R)$ can only be studied for
smaller spatial extents of Wilson loops, so only the asymptotic string tension
can be determined. In the $2.4\leq\beta\leq 2.6$ range, we find similar
behavior as for $\beta=2.5$.

Of particular interest is $\beta=2.7$, where we found strong deviations from
the Gaussian distribution of $R_{\mathrm{MCG}}$ values in Fig. 1c. The
corresponding string tensions are shown in Fig. 6.

Figure 6: Same as Fig. 5 for $\beta=2.7$. But here we observe that in the
right tip of the distribution $\sigma_{\mathrm{cp}}$ deviates drastically from
the asymptotic string tension. Further, the slope of $V_{\mathrm{cp}}(R)$
extracted from small Wilson loops exceeds the slope for large loops.

Note that $\sigma_{\mathrm{cp}}$ for large R exceeds $\sigma_{\mathrm{cp}}$
for small R at lower values of $R_{\mathrm{MCG}}$ whilst the opposite is the
case for large values of $R_{\mathrm{MCG}}$. A potential with such an
inverted, wrong behavior of the slope is shown in Fig. 7 at
$R_{\mathrm{MCG}}=0.907$. Around $R_{\mathrm{MCG}}=0.908$, we even observe a
collapse of the linear behavior and a transition to a screening potential,
indicating P-vortices of finite extent.

Figure 7: Characteristic potential extracted from Wilson loops for $\beta=2.7$
in the large $R_{\mathrm{MCG}}$ region at $R_{\mathrm{MCG}}=0.907$. The slope
of the center projected potential $V_{\mathrm{cp}}(R)$ decreases with $R$, as
shown by the linear approximations for small and large $R$. This behavior
indicates a failure of the vortex detection.

The unusual, inverted behavior of Fig. 7 is particularly relevant within the
subset of the ensemble that we suggested to discard in the discussion of Fig.
3. This hints at a possibility to resolve some failures of vortex detection
for $\beta=2.7$ where the vortex detection by center vortices became more
complicated.

##  VI Removal of vortex detection failures

For sufficiently large $\beta$, lattices of size $24^{4}$ are usually large
enough to extract potentials from single field configuration by evaluating
their Wilson loop data. This observation allows for $\beta=2.7$ to identify
local maxima where vortex identification fails. We assume that this failing is
caused by vortices being partly overlooked which in turn resembles a broken
vortex percolation. From the 20 000 configurations we have removed those whose
string tension decrease with $R$, i.e. with a behavior as shown in Fig. 7.
This results in a modified, more symmetric distribution of the remaining 16
000 local maxima of $R_{\mathrm{MCG}}$, see Fig. 8. The string tensions
extracted from averages over 200 center projected Wilson loops are shown in
Fig. 8.

Figure 8: Same as Fig. 5 for $\beta=2.7$ after removing from the ensemble Fig.
1c configurations where vortex detection appeared to have failed. The
discrepancy obviously present in Fig. 6 is weakened: The underestimation of
the string tension is improved.

The intersection of $\sigma_{\mathrm{cp}}$ with the horizontal line has
changed only slightly from $R_{\mathrm{MCG}}=$0.9064 to 0.9066 due to the
modification of the ensemble. The agreement between literature value and
extracted string tension corresponds to the right tip of the distribution,
suggesting that the largest maxima within the symmetrized ensemble of local
maxima predict the physical string tension. Hence, we assume that even for
$\beta=2.7$, the ensemble of local maxima of the gauge functional contains the
information about the density of thick vortices when obvious failures of the
vortex identification are discarded from the ensemble.

##  VII Towards a modified definition of MCG

In the previous sections and in Fig. 1 we have found that the distributions of
local maxima are symmetric for $\beta\in\\{2.3,2.4,2.5\\}$, have a tiny
asymmetry for $\beta=2.6$ and a strong asymmetry for $\beta=2.7$. Further, for
the 20 000 fields at $\beta=2.7$, there are several thousands with a slope of
$V_{\mathrm{cp}}(R)$ decreasing with $R$. We remove these fields with wrong
behavior of the slope. Then we symmetrise the ensembles for $\beta=2.6$ and
$2.7$.

The success of DMCG [21, 22] with the request of global maximization of the
gauge functional, done for a few gauge copies only, leads to the conjecture
that the fields with the highest $R_{\mathrm{MCG}}$ values within the
remaining symmetrized ensembles allows to predict the string tensions of the
full quark-antiquark potentials. We are now going to check this hypothesis for
our ensembles which are symmetrized for $\beta=2.6$ and corrected and
symmetrized for $\beta=2.7$. We determine $\sigma_{\mathrm{cp}}$ from the
asymptotic slopes of the center projected quark-antiquark potential. For the
200 physically different fields we select the $N$ gauge copies with the
highest values of $R_{\mathrm{MCG}}$ within the remaining ensembles. Since the
averages of the ensembles lead generally to an overestimate of the string
tension [14], we can expect that the extracted string tension approaches the
literature value for decreasing $N$ from above. The asymptotic slopes for
various values of $N$ are compared in Fig. 9 with the literature values of the
string tension.

Figure 9: For $\beta\in\\{2.3,2.4,2.5,2.6,2.7\\}$ on a $24^{4}$ lattice,
string tensions are extracted from center projected Wilson loops of 200
different physical fields from the $N$ gauge copies with the highest values of
$R_{\mathrm{MCG}}$. For $\beta=2.6$ a symmetrisation of the distribution
removes some of the highest extreme $R_{\mathrm{MCG}}$ values. At $\beta=2.7$
configurations with P-vortices which did not percolate had to be removed. The
horizontal black lines indicate string tensions extracted from unprojected
configurations in Refs. [27, 28, 26, 29],
$\sigma=0.131(4),0.0708(11),0.0350(12),0.0185(11),0.0112(2)$ for $\beta=2.3$
to $2.7$, respectively. The last figure compares the normalized distribution
of local maxima of $R_{\mathrm{MCG}}$ at $\beta=2.5$ for several $N$. We
observe that the average of these $R_{\mathrm{MCG}}$ distributions moves with
increasing $N$ to lower values and reaches for $N=100$ the average of the
whole ensemble of gauge copies.

We observe that already a few gauge copies, with the highest values of
$R_{\mathrm{MCG}}$, are sufficient to approach the literature values of string
tensions extracted from unprojected field configurations. At $\beta=2.3$ the
strong coupling is largest. This manifests itself in the highest value of the
string tension. The expected values of the Wilson loops therefore disappear
most quickly in statistical noise before the asymptotic value of
$\sigma_{\mathrm{cp}}$ is possibly reached. This could be the reason why the
string tension for beta=2.3 is already overestimated for $N=1$. Also for
$\beta=2.4$ and 2.5 no special action is necessary. For two or three gauge
copies we get already agreement between $\sigma_{\mathrm{cp}}$ for large
Wilson loops and the accepted values of the string tension. At $\beta=2.6$ a
few gauge copies with extreme values of $R_{\mathrm{MCG}}$ had to be removed
by symmetrisation of the ensemble, as discussed in section four. At
$\beta=2.7$ failures of the vortex detection can be identified by missing
percolation of vortices leading to a decrease of the string tension with the
size of center projected Wilson loops. The remaining symmetrized ensemble
allows for the same accuracy in the determination of the asymptotic string
tension as for lower $\beta$ values. The last diagram of Fig. 9 depicts the
distribution of $R_{\mathrm{MCG}}$ values for various choices of the number
$N$ of gauge copies for $\beta=2.5$. These distributions are again nearly
Gaussian. With increasing $N$ their average moves towards lower
$R_{\mathrm{MCG}}$ values. Due to the almost linear relation between
$R_{\mathrm{MCG}}$ and $\sigma_{\mathrm{cp}}$, see Fig. 5, increasing $N$
results in an increased string tension.

##  VIII Conclusion

In this article it was shown that the ensemble of local maxima of the MCG
gauge functional $R_{\mathrm{MCG}}$ contains the information about the long
range behavior of the quark-antiquark potential. To counter an underestimation
of the string tension the request for global maximization of the gauge
functional was substituted by the request of maximization within the symmetric
Gaussian ensemble of local maxima. We have shown that there is a window of
$\beta$ values where the asymptotic string tension of center projected Wilson
loops can be easily determined since the ensemble of $R_{\mathrm{MCG}}$ values
is well Gaussian distributed. By restricting the maximization to this Gaussian
subset of the ensemble extreme values of the gauge functional do not play any
role due to their small weights. The tip of maximal $R_{\mathrm{MCG}}$ values
within these Gaussian distributions determines the gauge copies for which the
center projected string tensions $\sigma_{\mathrm{cp}}$ agree well with the
expected values of the asymptotic string tension for unprojected fields.

The $\beta$-window can be enlarged towards higher values by symmetrization of
the ensembles and diminishing the weight of large $R_{\mathrm{MCG}}$ values. A
further increase of the window is possible by excluding gauge copies
experiencing an inverted behavior of the slope of the potential with respect
to the loop sizes. Due to finite separation and thickness of vortices on the
lattice, one can only expect limited increase of the $\beta$-window. Within
this window the analysis of the ensemble shows that the center vortex model is
capable to reproduce the correct string tension by center projection.

An attempt to systematically address the issue of a heuristically defined
gauge has been presented in the context of Coulomb gauge [30].

## References

  * Greensite [2011] J. Greensite, An introduction to the confinement problem, Lect. Notes Phys. 821, 1 (2011).
  * Faber and Höllwieser [2017] M. Faber and R. Höllwieser, Chiral symmetry breaking on the lattice, Prog. Part. Nucl. Phys. 97, 312 (2017).
  * ’t Hooft [1981] G. ’t Hooft, Topology of the gauge condition and new confinement phases in non-abelian gauge theories, Nuclear Physics B 190, 455 (1981).
  * Kronfeld  _et al._ [1987] A. S. Kronfeld, M. L. Laursen, G. Schierholz, and U. J. Wiese, Monopole condensation and color confinement, Physics Letters B 198, 516 (1987).
  * Bornyakov  _et al._ [2022] V. Bornyakov, V. Goy, I. Kudrov, and R. Rogalyov, Decomposition of the static potential in the maximal abelian gauge, EPJ Web of Conferences 258, 02009 (2022).
  * ’t Hooft [1979] G. ’t Hooft, A Property of Electric and Magnetic Flux in Nonabelian Gauge Theories, Nucl. Phys. B 153, 141 (1979).
  * L. Del Debbio, M. Faber, J. Greensite, and Š. Olejník [1997] L. Del Debbio, M. Faber, J. Greensite, and Š. Olejník, Center dominance and Z(2) vortices in SU(2) lattice gauge theory, Phys. Rev. D 55, 2298 (1997), arXiv:9610005 [hep-lat] .
  * Chernodub  _et al._ [2001] M. N. Chernodub, F. V. Gubarev, M. I. Polikarpov, and V. I. Zakharov, Monopoles and confining strings in QCD, in  _18th CFIF Autumn School: Topology of Strongly Correlated Systems_ (2001) arXiv:hep-lat/0103033 .
  * Suganuma and Ohata [2021] H. Suganuma and H. Ohata, Local Correlation among the Chiral Condensate, Monopoles, and Color Magnetic Fields in Abelian Projected QCD, Universe 7, 318 (2021), arXiv:2108.08499 [hep-lat] .
  * Shibata  _et al._ [2019] A. Shibata, K.-I. Kondo, and S. Kato, Confinement/deconfinement phase transition and dual Meissner effect in SU(3) Yang-Mills theory (2019), arXiv:1911.00898 [hep-lat] .
  * Junior  _et al._ [2021] D. R. Junior, L. E. Oxman, and G. M. Simões, From Center-Vortex Ensembles to the Confining Flux Tube, Universe 7, 253 (2021), arXiv:2106.04535 [hep-th] .
  * Kovacs and Tomboulis [1998] T. G. Kovacs and E. T. Tomboulis, Vortices and confinement at weak coupling, Phys. Rev. D57, 4054 (1998), arXiv:hep-lat/9711009 [hep-lat] .
  * Bornyakov  _et al._ [2001] V. Bornyakov, D. Komarov, and M. Polikarpov, P-vortices and drama of gribov copies, Physics Letters B 497, 151–158 (2001).
  * Dehghan  _et al._ [2023] Z. Dehghan, R. Golubich, R. Höllwieser, and M. Faber, A first analysis of the ensemble of local maxima of maximal center gauge, Universe 9, 389 (2023).
  * Golubich and Faber [2021a] R. Golubich and M. Faber, A possible resolution to troubles of su(2) center vortex detection in smooth lattice configurations, Universe 10.3390/universe7050122 (2021a).
  * Golubich and Faber [2021b] R. Golubich and M. Faber, Center regions as a solution to the gribov problem of the center vortex model, Acta Physica Polonica B Proceedings Supplement 14, 87 (2021b).
  * Golubich [2020] R. Golubich, The road to solving the gribov problem of the center vortex model in quantum chromodynamics, Acta Physica Polonica B Proceedings Supplement 10.5506/APhysPolBSupp.13.59 (2020).
  * R. Höllwieser, and M. Engelhardt [2015] R. Höllwieser, and M. Engelhardt, Approaching $SU(2)$ gauge dynamics with smeared $Z(2)$ vortices, Phys. Rev. D92, 034502 (2015), arXiv:1503.00016 [hep-lat] .
  * Golubich [2022] R. C. F. Golubich,  _Improvement of vortex detection in SU (2)-QCD_ , Ph.D. thesis, Technische Universität Wien (2022).
  * Engelhardt and Reinhardt [2000] M. Engelhardt and H. Reinhardt, Center projection vortices in continuum yang-mills theory, Nucl. Phys. B567, 249 (2000), hep-th/9907139 .
  * Del Debbio  _et al._ [1998] L. Del Debbio, M. Faber, J. Giedt, J. Greensite, and S. Olejnik, Detection of center vortices in the lattice Yang-Mills vacuum, Phys. Rev. D 58, 094501 (1998), arXiv:hep-lat/9801027 .
  * R. Höllwieser, M. Faber, J. Greensite, U. M. Heller, and Š. Olejník [2008] R. Höllwieser, M. Faber, J. Greensite, U. M. Heller, and Š. Olejník, Center Vortices and the Dirac Spectrum, Phys. Rev. D78, 054508 (2008), arXiv:0805.1846 [hep-lat] .
  * Faber  _et al._ [2001] M. Faber, J. Greensite, and S. Olejník, Direct laplacian center gauge, Journal of High Energy Physics 2001, 053 (2001).
  * Faber  _et al._ [2002] M. Faber, J. Greensite, and Š. Olejník, Center dominance recovered: Direct laplacian center gauge, Nuclear Physics B - Proceedings Supplements 106-107, 652 (2002).
  * Bertle  _et al._ [1999] R. Bertle, M. Faber, J. Greensite, and S. Olejník, The structure of projected center vortices in lattice gauge theory, Journal of High Energy Physics 1999, 019 (1999).
  * G. S. Bali, K. Schilling, and C. Schlichter [1995] G. S. Bali, K. Schilling, and C. Schlichter, Observing long color flux tubes in SU(2) lattice gauge theory, Phys. Rev. D 51, 5165 (1995), arXiv:hep-lat/9409005 .
  * Caselle  _et al._ [2015] M. Caselle, A. Nada, and M. Panero, Hagedorn spectrum and thermodynamics of su(2) and su(3) yang-mills theories, Journal of High Energy Physics 2015, 10.1007/jhep07(2015)143 (2015).
  * Fingberg  _et al._ [1993] J. Fingberg, U. M. Heller, and F. Karsch, Scaling and asymptotic scaling in the SU(2) gauge theory, Nucl. Phys. B 392, 493 (1993), arXiv:hep-lat/9208012 .
  * Michael and Perantonis [1991] C. Michael and S. Perantonis, Potentials and glueballs at large beta, Nucl. Phys. B Proc. Suppl. 20, 177 (1991).
  * Heinzl  _et al._ [2008] T. Heinzl, A. Ilderton, K. Langfeld, M. Lavelle, and D. McMullan, Ice limit of coulomb gauge yang-mills theory, Physical Review D 78, 10.1103/physrevd.78.074511 (2008).


# More Robust Schema-Guided Dialogue State Tracking via Tree-Based Paraphrase
Ranking

Alexandru Coca†, Bo-Hsiang Tseng‡, Weizhe Lin†, Bill Byrne†
†Department of Engineering, University of Cambridge, United Kingdom
‡Apple
{ac2123, wl356<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
###### Abstract

The schema-guided paradigm overcomes scalability issues inherent in building
task-oriented dialogue (TOD) agents with static ontologies. Instead of
operating on dialogue context alone, agents have access to hierarchical
schemas containing task-relevant natural language descriptions. Fine-tuned
language models excel at schema-guided dialogue state tracking (DST) but are
sensitive to the writing style of the schemas. We explore methods for
improving the robustness of DST models. We propose a framework111Code will be
released here: https://bit.ly/3WYB7Fl for generating synthetic schemas which
uses tree-based ranking to jointly optimise lexical diversity and semantic
faithfulness. The generalisation of strong baselines is improved when
augmenting their training data with prompts generated by our framework, as
demonstrated by marked improvements in average joint goal accuracy (JGA) and
schema sensitivity (SS) on the SGD-X benchmark.

##  1 Introduction

DST is concerned with tracking user goals in task-oriented conversations. The
goals are represented as key-value pair sequences, with the keys known as
slots (e.g. hotel name). Pre-trained language models (PLMs) (Devlin et al.,
2019; Raffel et al., 2020) have helped shift focus from systems that can only
track slots drawn from a database or domain ontology (Henderson et al., 2014)
to models that do not require re-training to parse goals in new domains. The
Schema-Guided Dialogue (SGD) dataset (Rastogi et al., 2020) facilitates this
shift with a large-scale set of conversations grounded in $45$ service APIs or
schemas that describe the domains, slots and user intents that annotate the
conversations (Appendix A). Test set dialogues are grounded in $6$ schemas
seen during training and $15$ unseen ones.

Neural models perform impressively on the difficult schema-guided DST task
Rastogi et al. (2020), but Lee et al. (2022) show that the uniformity of the
descriptive language of the schemas facilitates this. They create the SGD-X
benchmark to evaluate robust zero-shot generalisation of DST models. This is
achieved by grounding the SGD test set conversations in five schema variants
increasingly dissimilar to the SGD schemata222Variants are ordered according
to their lexical similarity to the SGD schemas. The v1 variant is the most
similar whereas v5 is the most dissimilar. See Appendix A for details and
examples and the schemata here: https://bit.ly/3Ev0KrV.. To perform well, a
DST model should correctly track the state of a dialogue when conditioned, in
turn, on prompts constructed from the five variants.

We show how to improve DST robustness by introducing controlled variability in
the data. We contribute to robust DST research by (1) a flexible framework for
generating and ranking diverse outputs of a paraphrase model based on a tree-
clustering algorithm designed to control lexical diversity and semantic
similarity; (2) combine state-of-the-art paraphrase models and language
generation metrics to generate increasingly diverse schemata paraphrases; (3)
show that augmenting the training dataset with these schemata improves the
robustness and generalisation performance of strong DST baselines.

##  2 Related Work

Input variety, data scarcity and domain shifts affect the robustness of DST
models. Liu et al. (2021) investigate the former. They employ word-level data
augmentation (DA) Wei and Zou (2019), turn paraphrasing and speech disfluency
modeling to approximate their field performance. Turn and dialogue generation
are effective in low-resource settings (Campagna et al., 2020; Hou et al.,
2018) but are very difficult to scale to new domains and are not effective in
the high-resource setting we consider Campagna et al. (2020); Mohapatra et al.
(2021). This also applies to word- and sentence-level methods Quan and Xiong
(2019); Louvan and Magnini (2020). Lee et al. (2022) find word-order changes
and deletions to be ineffective in the high-resource, schema-guided setting we
consider.

Schema-guided DST tackles both data scarcity and novel domains by using API
definitions to prompt PLMs Zhao et al. (2022). Yet Lee et al. (2022)
demonstrate the lack of robustness of schema-guided DST models to prompt
styles and vocabulary, creating a new research direction. They show that
augmenting the training data with synthetic prompts obtained via
backtranslation significantly improves models’ ability to track states under
meaning-preserving prompt transformations. Backtranslation is also applied to
improve DST robustness to linguistic variation inherent in user communication
Ma et al. (2019); Einolghozati et al. (2019), which is orthogonal to the
prompt style and vocabulary robustness setting we consider. Reinforcement
learning has also been applied Yin et al. (2020), but works only in the very
constrained single-domain, ontology-driven setting. Other TOD-relevant DA
approaches apply to policy learning Gritta et al. (2021) and response-
generation Gao et al. (2020); Zhang et al. (2020b).

Addressing the dearth of augmentation methods designed to ensure prompt
robustness of schema-guided DST models, we propose to generate schemas by
ranking large paraphrase candidate lists with learned metrics in a tree
ranking scheme.

Figure 1: Ranking paraphrases of Fare per ticket for journey using a tree. Top
level node split is by Jaccard distance ($J$), middle nodes split by
entailment score ($E$) and leave nodes store string similarities ($S$). Here
$f_{\mbox{dec}}=[\mbox{None},\max,\min]$. By using $J$ we guarantee that
candidates with $J=0$ are syntactic paraphrases if $S$ is constrained. Orange
leaves show top ranked candidates. Numbers on paths show ranking order.

##  3 Tree-Based Paraphrase Ranking

Tree construction A large pool of schema candidates is created by generating
paraphrases given grids of generation parameters (eg temperature, number of
beams). The set is filtered to address generation failures (eg toxic and
hallucinated words). We optionally filter candidates with an entailment model
to increase semantic faithfulness (Narayan et al., 2022) (see Appendix B.1).

The tree constructor (Algorithm 1) takes as input an object (Node) that stores
a metric value, val, and the candidate paraphrases which are split at that
node, sents. A list of metrics to be computed between each candidate and the
input is provided by the user. This enables our framework to build arbitrary-
depth trees with custom user metrics. Each unique list of metric values
describing the distance between the input and a candidate generates a path in
the tree (lines 5-13). The $n$-ary tree constructed in this way has the
property that level-order traversal of the first level can yield diverse
candidates with respect to the metric it encodes. In practice, the metrics
measure lexical and semantic distances between their inputs.

1 def  _build_tree(_root : Node, inp: str, cands: list[str], metrics:
list[Callable]_)_:

Data: root, inp input , cands input paraphrases, metrics objects to eval.
dist. between input & cand.

Result: tree splitting cands according to metrics

2

3 curr $\leftarrow$ root ;

4 for _c in cands_:

5 curr $\leftarrow$ root ;

6 for _m in metrics_:

7 m_val = m(inp, c) ;

8 next $\leftarrow$ get_child (curr.children, m_val) ;

9 if _next is NULL_:

10 next $\leftarrow$ Node (val=m_val, sents=[c]) ;

11 curr.children.add(next)

12 else:

13 next.sents.add(c) ;

14 curr $\leftarrow$ next

15

16

17

18 return root

Algorithm 1 Tree building

Ranking Our ranker input is the tree and a list of decision functions, with
elements corresponding to each level in the tree, f_dec. Without loss of
generality, we assume that the first level encodes a metric with respect to
which the user wishes to maximise diversity (eg lexical distance). As shown in
Figure 1, our algorithm traverses breadth-first the level for which diversity
is to be maximised. Each subtree returned in the traversal is traversed depth-
first, guided by the decision functions. For example, in Figure 1 we show that
the node $B=0.77$ is selected by applying the $\max$ decision function to the
children of $J=66$, and that applying $\min$ to the children of $B=0.77$
selects the leaf $S=77$. See Algorithm 3 (Appendix B) for details.

##  4 Experiments

###  4.1 Schema generation

Our paraphrase model is PegasusParaphrase333Available at
https://bit.ly/3vgY7EZ., a fine-tuned Pegasus model (Zhang et al., 2020a).
Using $10$ settings for the number of beams and temperature, we generate $500$
candidates for each input. For efficiency purposes, these are filtered
heuristically (Appendix B). We construct a depth $d=3$ tree which splits the
candidates by Jaccard distance $J$, entailment $E$, and string similarity $S$.
That is, the input to our tree constructor (Algorithm 2) is
$\mbox{metrics}=[J,E,S]$. Here entailment is computed using BART (Lewis et
al., 2020) as described in Appendix B. We prune nodes with $J>0.75$ to limit
hallucination.

We select $k=5$ lexically-diverse paraphrases that maximise entailment given
the constraint that the returned candidates should be lexically diverse.
First, we select a syntactic paraphrase by traversing the subtree rooted at
$J=0$ and minimising $S$. The remainder of the candidates are selected by
constraining the bread-first traversal of the first level, which encodes
lexical distance, to return the nodes sorted from high to low. This procedure
is depicted schematically in Figure 1. We sort the ranked candidate lists for
each description according to the Jaccard distance between them and the SGD
descriptions. Hence we obtain $k=5$ synthetic schema variants, with $v1$ being
the most similar to the SGD schema and $v5$ the most dissimilar. We refer to
this scheme as Pegasus + BART.

###  4.2 State tracking data augmentation

####  4.2.1 Baseline models

D3ST The Description-Driven Dialogue Modelling (D3ST) model (Zhao et al.,
2022) is a state-of-the-art DST model that performs intent tracking, requested
slots prediction, and state tracking in a single pass. See Appendix C for a
visual representation of inputs and targets. We process the data and train the
model as described in Zhao et al. (2022) and Appendix C, selecting models that
maximise the development set JGA.

T5DST We follow Lee et al. (2022) to implement a simplified T5DST (Lee et al.,
2021a). It predicts the value of each slot iteratively, requiring a number of
decoding passes equal to the number of slots in an API to predict the dialogue
state given a dialogue history. Training and inference with this model is very
expensive and we train the models with a fixed computational budget of
$20,000$ gradient steps444This is the number of steps required for maximising
the development set JGA, for all three runs. for the baseline and $40,000$
steps for all augmented data experiments. See Appendix C for prompt structure
and implementation details.

####  4.2.2 Evaluation

On SGD, the JGA ($\mbox{JGA}_{orig}$) is computed for the $4,201$ test set
dialogues. $77\%$ of these have a turn span where the agent calls an API
unseen in training. Only $6$ out of $21$ schemata are seen in training.

Evaluation on SGD-X proceeds as follows. First, the SGD descriptions in the
prompt are replaced, in turn, with descriptions taken from the five SGD-X
variants. The DST model then predicts the state of a given dialogue $5$ times,
conditioned on prompts that are increasingly dissimilar to the SGD test set.
Hence, the $\mbox{JGA}_{v_{1-5}}$ figures reported are averages over
approximately $21,000$ conversations. For all experiments except oracle (see
Section 4.2.3), none of the test time prompts are seen during training: the
seen superscript in the metric names reported in Section 5 identifies
conversations where the SGD test set prompt is seen during training.
Therefore, it quantifies whether the model can robustly identify slots seen in
training by interpreting the meaning of the descriptions rather than relying
on linguistic patters in the training schema. Meanwhile unseen measures the
ability of the model to generalise to new APIs, which may describe new slots
and domains, notwithstanding the language used by developers to phrase the
descriptions. The JGA coefficient of variation (ie schema sensitivity,
$\mbox{SS}_{JGA}$) as the prompt changes measures the sensitivity of a model
to the prompt Lee et al. (2021b).

Metric | Ranking | v1 | v2 | v3 | v4 | v5
---|---|---|---|---|---|---
Jaccard Dist | Pegasus + BART | 17.1 | 63.0 | 69.5 | 72.0 | 76.6
Backtranslation | 18.2 | 29.9 | 43.9 | - | -
EDA | 3.9 | 4.1 | 6.1 | 16.0 | 32.9
SGD-X | 55.6 | 65.6 | 71.2 | 78.1 | 85.7
Entailment | Pegasus + BART | 99.0 | 96.7 | 94.9 | 94.6 | 94.4
Backtranslation | 97.5 | 96.5 | 95.9 | - | -
EDA | 99.1 | 98.5 | 96.6 | 93.2 | 86.4
SGD-X | 89.7 | 88.0 | 88.4 | 86.8 | 87.5
BLEU | Pegasus + BART | 13.4 | 12.5 | 12.5 | 13.2 | 12.7
Backtranslation | 36.4 | 26.0 | 18.9 | - | -
EDA | 72.0 | 63.3 | 47.2 | 42.3 | 44.2
SGD-X | 20.4 | 15.3 | 10.8 | 8.3 | 5.2
self-BLEU | Pegasus + BART | - | 12.0 | 11.4 | 11.0 | 10.9
Backtranslation | - | 49.3 | 41.7 | - | -
EDA | - | 87.0 | 68.8 | 58.0 | 53.1
SGD-X | - | 13.5 | 11.2 | 9.9 | 8.6

Table 1: Automatic synthetic schema evaluation. $J$ is multiplied by $100$ for
readability.

####  4.2.3 Experimental setup

We show our approach is effective by augmenting the DST training data with
synthetic prompts composed from our generated schemata. To study the effect of
controlling prompt diversity augmented datasets are two (2x) to six times (6x)
the SGD size. For 2x, augmented data contains prompts constructed from the v1
synthetic schema, whereas for 6x we use all five generated schemas555Ordering
is from most (v1) to least (v5) similar to SGD..

Baselines We create three synthetic schema by backtranslation. Our pivot
languages are Korean, Japanese and Chinese (Lee et al., 2022). The augmented
DST training dataset is four times (4x) larger than SGD. Following Huang et
al. (2021), we also consider French and Russian as pivot languages to generate
two more synthetic schemas and obtain an augmented dataset six times (6x)
larger than SGD. We also compare with easy data augmentation (EDA) (Wei and
Zou, 2019), a word-level DA approach based on synonym replacement (SR), random
insertion, deletion and substitution. We perform SR with probability $0.25$
and the other operations with equal probability of $0.05$. Just like for
backtranslation, we generate $3$ or $5$ synthetic schemas with this method via
the public API. Augmentation with the SGD-X human schemata paraphrases is
considered an oracle because these models see the SGD-X schemata at training
time.

Model | # | Generation method - Dataset size | $\mbox{JGA}_{orig}\uparrow$ | $\mbox{JGA}_{v_{1-5}}$ | $\mbox{JGA}_{v_{1-5}}^{seen}$ | $\mbox{JGA}_{v_{1-5}}^{unseen}$ | $\mbox{SS}_{JGA}\downarrow$
---|---|---|---|---|---|---|---
| 1 | None - 1x | 69.8 | 56.5 | 73.6 | 50.8 | 70.1
| 2 | Pegasus + BART - 2x | 72.8 | 61.3 | 80.9 | 54.8 | 56.4
| 3 | Pegasus + BART - 4x | 72.6 | 62.5 | 81.7 | 56.1 | 51.0
| 4 | Pegasus + BART - 6x | 71.2 | 63.9 | 85.9 | 56.6 | 46.5
| 5 | EDA - 4x (Wei and Zou, 2019) | 71.0 | 59.0 | 78.5 | 52.5 | 63.0
| 6 | EDA - 6x (Wei and Zou, 2019) | 71.4 | 62.3 | 83.3 | 55.3 | 53.2
| 7 | Backtranslation - 4x (Lee et al., 2021b) | 72.1 | 62.2 | 84.0 | 54.9 | 53.1
| 8 | Backtranslation - 6x (Huang et al., 2021) | 71.5 | 61.0 | 82.5 | 53.8 | 54.4
D3ST | 9 |  SGD-X \- 6x (Lee et al., 2021b) (Oracle) | 73.8 | 69.7 | 92.5 | 62.1 | 27.9
| 10 | None | 70.0 | 50.4 | 58.5 | 47.7 | 87.0
| 11 | Pegasus B + BART - 4x | 71.3 | 55.1 | 71.2 | 49.7 | 70.1
| 12 | Pegasus + BART - 6x | 68.7 | 52.5 | 71.6 | 46.5 | 77.6
| 13 | EDA - 6x (Wei and Zou, 2019) | 72.2 | 51.1 | 55.6 | 49.6 | 84.1
| 14 | Backtranslation - 4x (Lee et al., 2021b) | 72.8 | 53.9 | 67.0 | 49.6 | 76.4
T5DST | 15 |  SGD-X \- 6x (Lee et al., 2021b) (Oracle) | 74.2 | 67.2 | 91.8 | 59.0 | 36.6
Table 2: SGD and SGD-X dialogue state tracking performance when training with
augmented data. Best performance (excluding the oracle setup) is in bold.
Dataset size is the number of times the augmented dataset is larger than SGD.

##  5 Results and Discussion

###  5.1 Synthetic schema generation

Our ranking method generates increasingly lexically diverse schemata as shown
by the increase in Jaccard distance across schema variants (Table 1). This
aspect is much more difficult to achieve with EDA without significantly
affecting semantics. Furthermore, self-BLEU (Zhu et al., 2018) scores indicate
EDA is the least effective in ensuring candidate diversity compared to other
approaches. The BLEU difference between the SGD-X variants v1 and v5 is $15.2$
but smaller ($0.66$) for our approach. Hence, the PEGASUS + BART copies
$n$-grams from the input and includes additional information. This information
is not always meaning-preserving: City where the event is happening is
paraphrased as The bustling city where the event is taking place ($v5$) but
End date for the reservation or to find the house is paraphrased as End date
for hotel reservation to allow time for a replacement both at the struck and
in the run up to the event ($v5$). The self-BLEU of the SGD-X schemas
decreases faster compared to the automatically generated paraphrases,
suggesting that Jaccard distance increases partly due to hallucination.

Entailment scores show that backtranslation is effective in preserving
semantics. For EDA, the semantic similarity drops significantly as more
candidates are generated since more dissimilar schemas are generated with more
edit operations which are likely to affect meaning. The entailment scores for
the SGD-X paraphrases are also lower since they do not always perfectly
semantically overlap with the input by construction Lee et al. (2022) and
because of entailment model errors.

###  5.2 Dialogue state tracking

D3ST Both the robustness and robust generalisation are improved by
augmentation with our synthetic schemas, as demonstrated by maximum
$\mbox{JGA}^{seen}_{v_{1-5}}$ ($12.35\%$) and $\mbox{JGA}^{unseen}_{v_{1-5}}$
($5.85\%$) increases and $23.6\%$ drop in $\mbox{SS}_{JGA}$ (rows 1&4, Table
2). The most benefit is obtained by training with syntactically diverse
prompts (Pegasus + BART 2x). Adding more diverse data (rows 3&4) improves DST
performance. Part of this improvement may arise because paraphrasing leaves
out domain-dependent information: Average review rating of the doctor is
paraphrased as The rating is average, so it’s not perfect, so the model can
learn to identify ratings more generally666It appears in $4$ unseen services
in the test set.. Moreover, inputs are noisy due to hallucination, so the
models trained with our augmentation are less likely to overfit to the
linguistic patterns of the training schemas. BLEU scores indicate high lexical
overlap between EDA-generated and SGD schemas (Table 1). This limits the
magnitude of EDA improvement (row 5) and we perform better with less data
(rows 3&6, 2&5).

Backtranslation is comparable with our method given the same data quantity
(rows 3&7). When we also backtranslate via French and Russian (Huang et al.,
2021) the data diversity does not significantly increase (Table 6, Appendix
D.1). This negatively impacts the DST performance, while our method improves
it (rows 4&8). We can control the schema generation process to match SGD
backtranslation performance (Appendix E).

T5DST777Lee et al. (2021b) report $72.6\%$ JGA on SGD and $64.0\%$ SGD-X but
we could reproduce only $69.98\%$ and $50.42\%$. We outperform EDA (rows
12&13) but not backtranslation (row 14). This may be due to (1) the larger
computational budget needed to maximise T5DST performance888Each training
example is seen only once. and (2) T5DST’s sensitivity to noisy descriptions
owing to its prompt format (Appendix C). We control hallucination by pruning
candidates with $J>0.5$ and entailment smaller than $0.58$ and maximise $J$
while minimising $S$ to produce an augmented dataset 4x larger than SGD
(Pegasus B + BART 4x). Limiting lexical diversity improves entailment compared
to Pegasus + BART 6x (Table LABEL:tab:ranking-algos-comparison-slots, Appendix
D.1), and the scheme improves DST robustness compared to the backtranslation
baseline (rows 11&14).

The best augmentation schemes fail to improve robustness and generalisation
relative to the human baseline (rows 9&15). This is due to the intrinsic
challenge of generating diverse yet semantically faithful paraphrases but also
due to the fact that humans use common sense and schema information when
paraphrasing, so the SGD-X paraphrases are not strictly semantically
equivalent. However, the proposed automatic process of paraphrase generation
enhances DST, yielding non-trivial improvements in model robustness, while
being less costly and more scalable compared to gathering human-written
schemata paraphrases.

##  6 Conclusion and Future Work

We presented a simple tree-based ranking algorithm for optimising lexical
diversity and semantic faithfulness during schema generation. The synthetic
schemas improve both the DST models’ robustness to schemata writing style and
their generalisation. Our framework will allow researchers working on
paraphrase generation and semantic faithfulness to measure the generalisation
of their models in a way that may be difficult to capture by existing
benchmarks: it can generate schemata paraphrases and train SOTA dialogue state
trackers which were shown to benefit from augmentation with high quality,
crowdsourced paraphrases.

## Limitations

The optimality of our ranking method depends on the ability of the underlying
paraphrase model to generate a search space that contains paraphrases which
are lexically and syntactically diverse and preserve the meaning of the input
description. This is sometimes challenging with schema inputs which tend to be
short (e.g. name of event) and contain little information. Our future work
will focus on addressing this by contextualising these inputs to enable the
paraphrase model to produce a richer space of candidates. Secondly, our method
requires that the semantic faithfulness metrics capture semantic similarity
well even as the vocabulary of the candidates and their syntax are very
diverse. Previous work on abstractive summarisation Narayan et al. (2022);
Maynez et al. (2020); Kryscinski et al. (2019) finds entailment scores to be
best correlated with human judgment of faithfulness. However, the correlations
are not perfect so the output of the ranking algorithm is still expected to
contain noisy candidates. For slot description paraphrases, this is
challenging because different inputs are very closely semantically related and
the entailment model may not identify paraphrase model errors that map a slot
description (e.g. departure time) to one with related semantics (e.g. arrival
time). We intend to address this in future work by developing finetuning
schemes for semantic faithfulness metrics.

## Ethics Statement

Our work is concerned with the use of language generation models to augment
training datasets for schema-guided dialogue datasets. The generation phase is
unconstrained, so the model may generate candidates that exhibit biases
inherited from the C4 Raffel et al. (2020) and HugeNews Zhang et al. (2020a)
pre-training datasets. In our experiments, we did not observe toxic or harmful
outputs, but on one occasion the model did generate the word apartheid as part
of an incoherent sentence. For this reason, our filtering stack rejects any
candidates containing sensitive words. The list of words that parameterize the
sensitive words filter is defined by the user.

## Acknowledgements

Alexandru Coca was supported supported by EPSRC grant EP/R513180/1. He would
like to acknowledge Harrison Lee and Raghav Gupta from Google Research for
support and guidance with D3ST and T5DST implementation. Weizhe Lin was
supported by a Research Studentship funded by Toyota Motor Europe
(RG92562(24020)). We also thank Howard Mei from University of Cambridge for
help with editing the final draft. Authors would like to acknowledge the
improvement suggestions made by anonymous reviwers and the EACL program
committee.

## References

  * Bandel et al. (2022) Elron Bandel, Ranit Aharonov, Michal Shmueli-Scheuer, Ilya Shnayderman, Noam Slonim, and Liat Ein-Dor. 2022.  Quality controlled paraphrase generation.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022_ , pages 596–609. Association for Computational Linguistics.
  * Campagna et al. (2020) Giovanni Campagna, Agata Foryciarz, Mehrad Moradshahi, and Monica S. Lam. 2020.  Zero-shot transfer learning with synthesized data for multi-domain dialogue state tracking.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020_ , pages 122–132. Association for Computational Linguistics.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers)_ , pages 4171–4186. Association for Computational Linguistics.
  * Einolghozati et al. (2019) Arash Einolghozati, Sonal Gupta, Mrinal Mohit, and Rushin Shah. 2019.  Improving robustness of task oriented dialog systems.  _CoRR_ , abs/1911.05153.
  * Gao et al. (2020) Silin Gao, Yichi Zhang, Zhijian Ou, and Zhou Yu. 2020.  Paraphrase augmented task-oriented dialog generation.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020_ , pages 639–649. Association for Computational Linguistics.
  * Gritta et al. (2021) Milan Gritta, Gerasimos Lampouras, and Ignacio Iacobacci. 2021.  Conversation graph: Data augmentation, training and evaluation for non-deterministic dialogue management.  _Trans. Assoc. Comput. Linguistics_ , 9:36–52.
  * Henderson et al. (2014) Matthew Henderson, Blaise Thomson, and Steve J. Young. 2014.  Word-based dialog state tracking with recurrent neural networks.  In _Proceedings of the SIGDIAL 2014 Conference, The 15th Annual Meeting of the Special Interest Group on Discourse and Dialogue, 18-20 June 2014, Philadelphia, PA, USA_ , pages 292–299. The Association for Computer Linguistics.
  * Hou et al. (2018) Yutai Hou, Yijia Liu, Wanxiang Che, and Ting Liu. 2018.  Sequence-to-sequence data augmentation for dialogue language understanding.  In _Proceedings of the 27th International Conference on Computational Linguistics, COLING 2018, Santa Fe, New Mexico, USA, August 20-26, 2018_ , pages 1234–1245. Association for Computational Linguistics.
  * Huang et al. (2021) Shuo Huang, Zhuang Li, Lizhen Qu, and Lei Pan. 2021.  On robustness of neural semantic parsers.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, EACL 2021, Online, April 19 - 23, 2021_ , pages 3333–3342. Association for Computational Linguistics.
  * Kryscinski et al. (2019) Wojciech Kryscinski, Nitish Shirish Keskar, Bryan McCann, Caiming Xiong, and Richard Socher. 2019.  Neural text summarization: A critical evaluation.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 540–551, Hong Kong, China. Association for Computational Linguistics.
  * Lee et al. (2021a) Chia-Hsuan Lee, Hao Cheng, and Mari Ostendorf. 2021a.  Dialogue state tracking with a language model using schema-driven prompting.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021_ , pages 4937–4949. Association for Computational Linguistics.
  * Lee et al. (2021b) Harrison Lee, Raghav Gupta, Abhinav Rastogi, Yuan Cao, Bin Zhang, and Yonghui Wu. 2021b.  SGD-X: A benchmark for robust generalization in schema-guided dialogue systems.  _CoRR_ , abs/2110.06800.
  * Lee et al. (2022) Harrison Lee, Raghav Gupta, Abhinav Rastogi, Yuan Cao, Bin Zhang, and Yonghui Wu. 2022.  SGD-X: A benchmark for robust generalization in schema-guided dialogue systems.  In _Thirty-Sixth AAAI Conference on Artificial Intelligence, AAAI 2022, Thirty-Fourth Conference on Innovative Applications of Artificial Intelligence, IAAI 2022, The Twelveth Symposium on Educational Advances in Artificial Intelligence, EAAI 2022 Virtual Event, February 22 - March 1, 2022_ , pages 10938–10946. AAAI Press.
  * Lewis et al. (2020) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020.  BART: denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020_ , pages 7871–7880. Association for Computational Linguistics.
  * Liu et al. (2021) Jiexi Liu, Ryuichi Takanobu, Jiaxin Wen, Dazhen Wan, Hongguang Li, Weiran Nie, Cheng Li, Wei Peng, and Minlie Huang. 2021.  Robustness testing of language understanding in task-oriented dialog.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/IJCNLP 2021, (Volume 1: Long Papers), Virtual Event, August 1-6, 2021_ , pages 2467–2480. Association for Computational Linguistics.
  * Louvan and Magnini (2020) Samuel Louvan and Bernardo Magnini. 2020.  Simple is better! lightweight data augmentation for low resource slot filling and intent classification.  In _Proceedings of the 34th Pacific Asia Conference on Language, Information and Computation, PACLIC 2020, Hanoi, Vietnam, October 24-26, 2020_ , pages 167–177. Association for Computational Linguistics.
  * Ma et al. (2019) Yue Ma, Zengfeng Zeng, Dawei Zhu, Xuan Li, Yiying Yang, Xiaoyuan Yao, Kaijie Zhou, and Jianping Shen. 2019.  An end-to-end dialogue state tracking system with machine reading comprehension and wide & deep classification.  _CoRR_ , abs/1912.09297.
  * Maynez et al. (2020) Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan T. McDonald. 2020.  On faithfulness and factuality in abstractive summarization.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020_ , pages 1906–1919. Association for Computational Linguistics.
  * Mohapatra et al. (2021) Biswesh Mohapatra, Gaurav Pandey, Danish Contractor, and Sachindra Joshi. 2021.  Simulated chats for building dialog systems: Learning to generate conversations from instructions.  In _Findings of the Association for Computational Linguistics: EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 16-20 November, 2021_ , pages 1190–1203. Association for Computational Linguistics.
  * Narayan et al. (2022) Shashi Narayan, Gonçalo Simões, Yao Zhao, Joshua Maynez, Dipanjan Das, Michael Collins, and Mirella Lapata. 2022.  A well-composed text is half done! composition sampling for diverse conditional generation.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022_ , pages 1319–1339. Association for Computational Linguistics.
  * Quan and Xiong (2019) Jun Quan and Deyi Xiong. 2019.  Effective data augmentation approaches to end-to-end task-oriented dialogue.  In _International Conference on Asian Language Processing, IALP 2019, Shanghai, China, November 15-17, 2019_ , pages 47–52. IEEE.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _J. Mach. Learn. Res._ , 21:140:1–140:67.
  * Rastogi et al. (2020) Abhinav Rastogi, Xiaoxue Zang, Srinivas Sunkara, Raghav Gupta, and Pranav Khaitan. 2020.  Towards scalable multi-domain conversational agents: The schema-guided dialogue dataset.  In _The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020_ , pages 8689–8696. AAAI Press.
  * Sellam et al. (2020) Thibault Sellam, Dipanjan Das, and Ankur P. Parikh. 2020.  BLEURT: learning robust metrics for text generation.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020_ , pages 7881–7892. Association for Computational Linguistics.
  * Wei and Zou (2019) Jason W. Wei and Kai Zou. 2019.  EDA: easy data augmentation techniques for boosting performance on text classification tasks.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019_ , pages 6381–6387. Association for Computational Linguistics.
  * Williams et al. (2018) Adina Williams, Nikita Nangia, and Samuel Bowman. 2018.  A broad-coverage challenge corpus for sentence understanding through inference.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 1112–1122. Association for Computational Linguistics.
  * Wolf et al. (2019) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, and Jamie Brew. 2019.  Huggingface’s transformers: State-of-the-art natural language processing.  _CoRR_ , abs/1910.03771.
  * Yin et al. (2020) Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, and Qun Liu. 2020.  Dialog state tracking with reinforced data augmentation.  In _The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020_ , pages 9474–9481. AAAI Press.
  * Zhang et al. (2020a) Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter J. Liu. 2020a.  PEGASUS: pre-training with extracted gap-sentences for abstractive summarization.  In _Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event_ , volume 119 of _Proceedings of Machine Learning Research_ , pages 11328–11339. PMLR.
  * Zhang et al. (2020b) Yichi Zhang, Zhijian Ou, and Zhou Yu. 2020b.  Task-oriented dialog systems that consider multiple appropriate responses under the same context.  In _The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020_ , pages 9604–9611. AAAI Press.
  * Zhao et al. (2022) Jeffrey Zhao, Raghav Gupta, Yuan Cao, Dian Yu, Mingqiu Wang, Harrison Lee, Abhinav Rastogi, Izhak Shafran, and Yonghui Wu. 2022.  Description-driven task-oriented dialog modeling.  _CoRR_ , abs/2201.08904.
  * Zhu et al. (2018) Yaoming Zhu, Sidi Lu, Lei Zheng, Jiaxian Guo, Weinan Zhang, Jun Wang, and Yong Yu. 2018.  Texygen: A benchmarking platform for text generation models.  In _The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, SIGIR 2018, Ann Arbor, MI, USA, July 08-12, 2018_, pages 1097–1100. ACM.

##  Appendix A The SGD and SGD-X datasets

SGD As mentioned in Section 1, the conversations in the SGD dataset are
grounded in schemata, which describe a set of service APIs. The most important
schema elements are999Examples below are taken from the SGD test set.:

  * •

a service name (e.g. Messaging_1) followed by a service description (e.g.
Connect and share locations with your contacts)

  * •

one or more API functions to be invoked as users solve tasks, referred to as
(user) intents; each intent has a name (e.g. ShareLocation) and an intent
description (e.g. Send your location to a contact)

  * •

optional and required arguments for each API function, or slots; each slot has
a name (e.g. location) and a slot description (e.g. Location to share with the
contact)

SGD-X Lee et al. (2022) observe that $71\%$ of intent names and $65\%$ of slot
names from unseen APIs exactly match the train set. Furthermore, descriptions
are stylistically uniform across the train and test sets. For example, all
boolean slots begin with the phrase Boolean flag … or Whether…. Therefore,
they create the SGD-X dataset as follows:

  * •

crowdsource schema element paraphrasing to more than $400$ authors via Amazon
Mechanical Turk. Each crowdworker either paraphrases all names or all
descriptions for a given schema

  * •

manually vet responses for quality and correctness.

The slot names collected are sorted in increasing order of their Levenshtein
distance to the SGD slot names whereas the descriptions are sorted according
to the Jaccard distance between their lemmatized forms (excluding stop words).
An example of SGD-X description paraphrases in shown in Table 3.

Variant | Description
---|---
SGD | Category to which the attraction belongs
v1 | The category that describes what kind of attraction it is
v2 | Category of place of interest
v3 | Type of tourist attraction
v4 | Choose the kind of tourist landmark
v5 | The kind of tourist hotspot
Table 3: Example of descriptions paraphrases from the SGD-X test schemas. The
more similar v1 description contains overlapping vocabulary with the SGD test
set description, whereas v4 and v5 variants are dissimilar both stylistically
and lexically

While the examples above are paraphrases of the SGD input, in general, the
semantic content of the schema element paraphrases is not perfectly
overlapping with the input as the crowdworkers use information from the wider
service context when creating new elements.

Filter name | Filtered example
---|---
contains advice | An appointment is necessary for your hair.
describes action | They commemorate the number of flights to the airport.
has named entities | Enter the doctor’s Leningrad address.
has low frequency words | The address is ofadvisory.
discard multiple sentences | The address is the dentist’s box. Guidelines for hiring a dentist.
has repeated ngrams |  The dentist is Address of the dentist.
has repeated similar bigrams | The type of event is stated in the title of the event.
has consecutive repeated words | Average review rating for a hotel hotel.
is past tense sentence | It was the dentist’s address.
is passive voice sentence | The address was given by abrasives from the dentist.
is question | Is there a balance of the account?
has alphanumeric words |  400 baths in an apartment.
Table 4: Filters implemented along with sample examples they discard

##  Appendix B Ranking Framework

###  B.1 Candidate generation

Algorithm 2 summarises the candidate generation procedure, which takes any
paraphrase model, a list of model-specific generation parameters and,
optionally, a list of filters as an input (line 1). These parameters are
temperature and number of beams for Pegasus, or a grid of lexical, semantic
and syntactic distances for the Quality Controlled Paraphrase Generation
(QCPG) Bandel et al. (2022) model presented in Appendix E. The model generates
one or more paraphrases, which are filtered before returning (lines 3-8). We
describe the filtering process next.

Heuristic filtering Our main motivation for implementing heuristic filters is
to filter the majority of poor quality candidates, without making use of the
large GPU cards required to run the entailment model. We also address the fact
that the model is free to generate a very large number of candidates and
therefore is expected to hallucinate significantly. These filters are general
purpose and are implemented in few lines of code using the spaCy and nltk
libraries. Table 4 lists active filters along with typical examples filtered.

Entailment filtering We implement our entailment filter using BART Lewis et
al. (2020)101010Avaialble at https://huggingface.co/facebook/bart-large-mnli..
This model is pre-trained on the MNLI dataset Williams et al. (2018). To
measure entailment this model consumes a premise and hypothesis in the format
premise <SEP> hypothesis. In our implementation we replace premise with the
description to be paraphrased. By default, the hypothesis is a template of the
form This example is {}., where {} is a placeholder for the user hypothesis,
in our case the paraphrased description. We find that considering alternative
templates improves the reliability of the model, so we consider {}, This
example has the same meaning as {}., This text is about {}., and This example
implies that {}., averaging the entailment scores across templates to
calculate the entailment score. The same procedure is followed when computing
the entailment of candidates during ranking.

1 def  _generate_candidates(_model : Any, inp: str, params: dict, filters:
Optional[list[Callable]]_)_:

Data: model text generation model, inp input sentence, params model specific
parameters, filters a list of boolean functions

Result: cands list of inp paraphrases

2

3 cands $\leftarrow$ [] ;

4 for _p in params_:

5 c $\leftarrow$ model.forward(inp, **p) ;

6 c $\leftarrow$ [p for p in c if not any(_f(p, inp) for f in filters_)]

7 cands.extend(c)

8 return cands

Algorithm 2 Candidate generation

###  B.2 Ranking

1 def  _tree_rank(_root : Node, n: int, f_dec: list[Callable]_)_:

Data: root, n number of candidates, f_dec decision functions

Result: list of n ranked candidates

2

3 ranked $\leftarrow$ syntax_select (root) ;

4 n $\leftarrow$ n \- len(ranked) ;

5

6 while _len(ranked) $\neq$ n_:

7 for _next in level_order (root)_:

8 for _f in f_dec_:

9

10 cand $\leftarrow$ select (next.sents) ;

11 ranked.add(cand) ;

12 prune (next, cand) ;

13

14

15 return ranked

Algorithm 3 Tree ranking

Ranking Algorithm 3 summarises the tree-ranking procedure. This procedure
takes as an input the tree constructed as described in Algorithm 1, along with
a list of decision functions f_dec. Our algorithm starts by selecting a
paraphrase via depth first traversal of the subtree rooted at $J=0$ (line 2).
The remainder of the candidates are selected by traversing the first level in
a breadth-first manner (line 5) and depth-first traversal of each subtree
returned during the level-order traversal (lines 6-8). Here the semantics of
f(next.children) is that the decision function f takes all the children of
next as input and returns a single node which is next in the traversal. A
candidate is selected from the leaf111111There can be multiple, possibly
repeated candidates in a leaf because the generative model may generate the
same output given different parameter settings. We select the most common one
if there are repeated candidates and randomly otherwise.(line 8) and
subsequently removed from the candidates list (line 10). This is to avoid
selecting the same candidate multiple times in situations where the paraphrase
model generates few distinct candidates.

(a) Visual representation of D3ST inputs and targets. Blue font blue,
preceding the [USR] special token represents the prompt, consisting of slot
descriptions extracted from the schema. Each slot is assigned an index, which
is used to recover the slot value pairs during post-processing. Note that slot
4 is categorical, and so the string a) 1 b) 2 c) 3 d) 4 is appended to the
description to indicate the model that it should output one of the choices.
The dialogue history follows the [USR] token. The model outputs only active
slots, in this case omitting slot $1$ because is has not been mentioned. The
entire dialogue state (as opposed to turn state) is generated at every turn.

(b) Visual representation of T5DST inputs and targets. The string [slot]
separates the dialogue history from the slot description. In the first example
the model outputs the special value none to indicate that the slot was not
mentioned by the user. For categorical slots (second row from the top), the
slot description is concatenated with options containing all possible slot
values and the model predicts the correct option. For non-categorical slots
(third row), the exact value is predicted. The ellipsis indicates that none is
predicted for all other slots in the Restaurant_1 schema that are not
mentioned in the dialogue history

Figure 2: Prompt formats for a) D3ST b) T5DST

##  Appendix C State Tracking Baselines

D3ST We process the data as described by Zhao et al. (2022) with the following
differences:

  * •

The indices are separated by the = symbol in both the inputs and the targets,
to avoid a parsing ambiguity which occurs for time slots if : is used as a
separator for targets

  * •

For categorical slots which take the dontcare special value, our output
contains slot_index: dontcare substring and we do not include the dontcare
value in the prefix together with the other options

  * •

We lowercase the inputs and the targets121212This appears in illustrations but
is not explicitly stated by (Zhao et al., 2022)..

We obtain $175,780$ examples from the original SGD dataset, which are
truncated to the last $1,024$ tokens on the input side. See Figure 2(a) for a
visual representation of the model inputs and outputs. We optimise the model
using the Adafactor optimizer and effective batch size $32$, starting from the
initial weights google/t5-v1_1-base published by huggingface Wolf et al.
(2019). We interpolate the learning rate linearly between $0$ and $10^{-4}$
over the first $1000$ steps and keep it constant thereafter. We select the
model by evaluating the development set JGA every $5000$ gradient updates,
stopping the training if said metric fails to improve after $3$ consecutive
evaluations. All numbers in Table 2 are averages of 3 runs, except the SGD-X
experiment for T5DST which is a single run.

T5DST Given a dialogue in the SGD training set we consider all partial
dialogue histories $\\{u_{1},s_{1},...s_{t-1},u_{t}\\}$ with
$t\in\overline{0,T}$ where $T$ is maximum index of the user turn in a
dialogue. The turns in each dialogue history are lowercased and separated
[usr] and [sys] tokens, not treated as special tokens. For each dialogue
history we create a training example for each slot in the ground truth schema,
which contains the concatenated turns suffixed with the string [slot]
[slot_description] where the placeholder [slot_description] is replaced by the
lowercase descriptions extracted from the SGD schemata. This yields
$1,601,356$ examples for the SGD training dataset. See Figure 2(b) for a
representation of the model inputs and outputs.

We optimise the model using the Adafactor optimizer and effective batch size
$256$, starting from the initial weights google/t5-v1_1-base published by
huggingface. We interpolate the learning rate linearly between $0$ and
$10^{-4}$ over the first $1000$ steps and keep it constant thereafter. We
perform $20,000$ optimisation steps131313For a single run, this is
approximately $6$ hours of computation on $8$ nvidia A100-80GB cards.
Moreover, decoding a single run on SGD and SGD-X takes $6$ hours., limiting
the number of training steps to $40,000$ steps141414This is sufficient so that
the model sees every example once when working with an augmented training set
six times the size of SGD. for all augmented data experiments. All numbers in
Table 2 are averages of 3 runs, except the Oracle experiment on T5DST which is
a single run.

Metric | Ranking | v1 | v2 | v3 | v4 | v5
---|---|---|---|---|---|---
Jaccard Dist | Backtr. 4x | 18.2 | 29.9 | 43.9 | - | -
Backtr. 6x | 12.9 | 22.7 | 27.8 | 35.6 | 46.7
Entailment | Backtr. 4x | 97.5 | 96.5 | 95.9 | - | -
Backtr. 6x | 98.0 | 97.5 | 95.2 | 94.8 | 95.5
BLEU | Backtr. 4x | 36.4 | 26.01 | 18.9 | - | -
Backtr. 6x | 51.3 | 37.2 | 29.5 | 23.4 | 18.2
self-BLEU | Backtr. 4x | - | 49.3 | 41.7 | - | -
Backtr. 6x | - | 55.3 | 49.7 | 44.6 | 39.6

Table 5: Effect of using French and Russian as additional pivot languages on
automatic metrics

##  Appendix D Additional Results

###  D.1 Increasing backtranslation dataset size

Index | Augmentation | $\mbox{JGA}_{orig}$ | $\mbox{JGA}_{v_{1-5}}$ | $\mbox{JGA}_{v_{1-5}}^{seen}$ | $\mbox{JGA}_{v_{1-5}}^{unseen}$ | $\mbox{SS}_{JGA}$
---|---|---|---|---|---|---
1 | Pegasus+BART 6x | 71.2 | 63.9 | 85.9 | 56.6 | 46.5
2 | Pegasus+BLEURT 6x | 72.4 | 64.0 | 86.6 | 56.4 | 46.6
3 | QCPG+BLEURT 6x | 72.7 | 63.2 | 85.2 | 55.9 | 47.3
4 | Backtranslation 4x (Lee et al., 2021b) | 72.1 | 62.2 | 84.0 | 54.9 | 53.1
Table 7: Ranking with a more accurate semantic faithfulness metric (row 2) or
generating candidates with a controllable paraphrase model (row 4) can be used
to boost SGD performance over our Pegasus+BART approach (row 1). Bold font
marks column maximum, underlined second largest number.

We include Table 6 to substantiate our intuition that the training with the
Backtranslation 6x scheme does not yield further improvement compared to the
Backtranslation 4x scheme as the additional data does not significantly
increase the prompt diversity. Most clearly, this is indicated by the fact
that the $v5$ variant has similar BLEU to variant $v3$ in Backtranslation 4x,
indicating that a large proportion of additional data has some overlaps more
with the SGD distribution than the data backtranslated to Chinese, Korean and
Japanese. This is also indicated by how self-BLEU decays as more data is
added, comparatively, between Backtranslation 4x and Backtranslation 6x.

###  D.2 Controlling schema generation diversity

Table LABEL:tab:ranking-algos-comparison-slots shows that the alternative
schema scheme generates schemas with lower average Jaccard distance and higher
entailment with respect to the SGD schemata. We find this effectively controls
the noise in the data, leading to improved performance for T5DST and similar
performance to PEGASUS + BART for D3ST.

##  Appendix E Schema Generation with BLEURT and QCPG

BLEURT (Sellam et al., 2020) is a BERT-based natural metric commonly used in
translation, so it is expected to be highly sensitive to semantic differences.
In Table 7 we show that simply re-ranking the Pegasus output space with BLEURT
improves SGD performance comparably with backtranslation (rows 2&4) and the
robustness and generalisation improvements are maintained.

Bandel et al. (2022) exploit high quality examples in paraphrase corpora by
conditioning the model with a string quality parameters string outlining
target semantic, syntactic and lexical distances of the generated paraphrase
during finetuning. At inference one must specify these parameters to obtain
diverse yet high quality paraphrases. We could not apply the quality parameter
selection method proposed by QCPG authors at inference time as the code had
not been fully released at the time of writing. Instead, we generated a large
number of paraphrases with different quality targets and greedy decoding, and
re-ranked the candidates using our framework. This demonstrates the
versatility of our framework. In Table 7 we show that this model can equally
achieve improved performance on SGD. The improvement on SGD-X is slightly less
than achieved by PEGASUS+BART 6x, as expected since greedy decoding and better
semantic faithfulness optimisation generate schemata closer to the SGD
distribution so less out-of-distribution improvement is achieved.

This experiments in this section and Appendix D.2 demonstrate the versatility
of our framework and its usefulness as a tool for generating synthetic schema
prompts.