[table]capposition=top

# Thermal quenching of classical and semiclassical chaos

Vijay Ganesh Sadhasivam<EMAIL_ADDRESS>Yusuf Hamied Department of Chemistry,
University of Cambridge, Lensfield road, Cambridge, CB2 1EW, United Kingdom
Lars Meuser  Yusuf Hamied Department of Chemistry, University of Cambridge,
Lensfield road, Cambridge, CB2 1EW, United Kingdom  Laboratory of Physical
Chemistry, ETH Zürich, 8093 Zürich, Switzerland  Stuart C. Althorpe
<EMAIL_ADDRESS>Yusuf Hamied Department of Chemistry, University of Cambridge,
Lensfield road, Cambridge, CB2 1EW, United Kingdom

###### Abstract

The growth rate of out-of-time ordered correlators (OTOCs) provide for the
notion of a quantum Lyapunov exponent, which quantifies chaos and information
scrambling in quantum systems. In thermal ensembles, this growth rate is
reduced from its corresponding value in the microcanonical distribution, and a
temperature-dependent bound on has been conjectured to exist for the quantum
Lyapunov exponent. We detail a two-fold mechanism which is responsible for the
reduction of the Lyapunov exponents in thermal quantum systems.

††preprint: APS/123-QED

##  I Introduction

In the study of classical chaos in non-linear dynamical systems, Lyapunov
exponents form an important class of dynamical quantifiers that capture phase-
space instability. They measure the relative spread of phase-space
trajectories that are initially close by and their definition is hence
inherently classical. The question of what constitutes ‘quantum chaos’ in
systems that are classically chaotic has prompted intense interest for
decades. Recently, a quantitative tool called the ‘out-of-time-ordered
correlator’ (OTOC)[1, 2] had been introduced as a diagnostic measure of
‘information scrambling’[3], which, in many systems, represents the
delocalisation of information due to chaotic dynamics. They have the following
structural form:

$C(t)=\langle\left[\hat{W}(t),\hat{V}(0)\right]^{\dagger}\left[\hat{W}(t),\hat{V}(0)\right]\rangle$
(1)

where $\hat{W}$ and $\hat{V}$ are hermitian operators representing local
quantum information.

The definition of OTOC is due to Larkin and Ovchinnikov[4] and was inspired
from a quasiclassical theory. Due to this semiclassical connection,the
exponential growth rate of OTOCs provide for the notion of a ‘quantum’
Lyapunov exponent in systems which exhibit exponentially fast scrambling. The
relation between the growth rate of this OTOC and the classical Lyapunov
exponent of the underlying classical system has been a subject of discussion
[5, 6, 7]. In thermal quantum systems, it has been conjectured that the
quantum Lyapunov exponent satisfies a temperature-dependent bound[1]:

$\displaystyle\lambda_{Q}(T)\leq\frac{2\pi k_{B}T}{\hbar}$ (2)

While there is no such constraint on the classical Lyapunov exponent, it has
been pointed out[5] that, in the semiclassical limit, the growth rate of the
quantum OTOC is related to the ‘classical’ growth rate, which is, in general,
different from the classical Lyapunov exponent.

In almost all contexts discussed in previous works, the relevant OTOCs were
computed as a microcanonical ensemble average. For a certain class systems
described in the microcanonical ensemble, it has also been pointed out that
there is a lower bound condition[6] on the growth rate of the OTOC in the
semiclassical limit. The upper bound condition due to Maldacena et al [1]
pertains specifically to the canonical ensemble. Previous works [8, 9, 10, 11]
have shown that this quantum bound has a (quantum-Boltzmann) statistical
origin. In this letter, we call particular attention to the effects of the
thermal averaging on the growth rate of thermal OTOCs and their semiclassical
approximation.

The OTOCs given by eq. (1) form a specific class of non-linear response
functions [12] which serve as a reliable indicator of information scrambling.
It is to be noted that the double commutator structure of eq. (1) enables it
to probe operator growth, especially in thermal ensembles. In fact, it is
well-known[13, 14] from linear response theory that the thermal average of a
single commutator of the form

$C_{T}(t)=\Tr\left[e^{-\beta
H(\hat{q},\hat{p})}\left[\hat{q}(t),\hat{q}(0)\right]\right]$ (3)

(where $\hat{q}$ is the position operator) yields, in the semiclassical limit,

$\displaystyle\begin{split}C^{cl}_{T}(t)&=-i\hbar\int dq\>dp\>e^{-\beta
H(q,p)}\frac{\partial q_{t}}{\partial p}\\\ &=-\frac{i\hbar\beta}{m}\int
dq\>dp\>e^{-\beta H(q,p)}\>q(t)\>p(0)\end{split}$ (4)

a two-point correlation function (agnostic to time-ordering). By using the
Cauchy-Schwarz inequality of weighted inner products, this correlation
function can be shown to be bounded for all temperatures and not exponentially
growing:

$\langle q(t)p(0)\rangle^{cl}_{T}\leq\langle q(t)^{2}\rangle^{cl}_{T}\;\langle
p(0)^{2}\rangle^{cl}_{T}$ (5)

In fact, it has been shown[14] that even higher-order response functions
computed at a single time can be re-expressed in terms of two-point time-
correlation functions which implies that they cannot grow exponentially.
However, it is well-known from various numerical studies[15, 16] that thermal
OTOCs can indeed be exponential and hence are distinct from other non-linear
response functions. Thus, it is the double commutator structure of the OTOC
definition which yields two stability matrix elements in the semiclassical
limit that results in exponential growth. We discuss the exponential behaviour
of thermal OTOCs as a function of temperature.

##  II Growth rate of thermal OTOCs

We start with a thermal OTOC of the form:

$\displaystyle
C^{qq}_{T}(t)=-\langle\left[\hat{q}(t),\hat{q}(0)\right]^{2}\rangle_{T}$ (6)

which under the classical approximation yields:

$\displaystyle C^{cl}_{T}(t)=\hbar^{2}\int dq\>dp\>e^{-\beta
H(q,p)}\left|\frac{\partial q_{t}}{\partial p}\right|^{2}$ (7)

Here, we consider a first-quantized one-dimensional Hamiltonian
$H(q,p)=T(p)+V(q)$ separable in position and momentum variable. If the
classical phase-space dynamics has a maximal Lyapunov exponent (say
$\lambda_{cl}=\lambda$), a ‘naive’ semiclassical approximation[15] of the
thermal OTOC, which neglects the effects of (classical) thermal averaging will
predict its growth rate at very high temperature as:

$\displaystyle C_{T}^{(cl)}(t)\sim\hbar^{2}e^{2\lambda_{cl}t}$ (8)

However, it has been established [15, 11] that at high temperatures, we indeed
have:

$\displaystyle C_{T}^{cl}(t)\sim e^{\lambda t}$ (9)

An explanation to this halving of the growth rate can be provided along the
same lines of eq. (4). The time evolution of the stability matrix element
$M_{qp}$ in eq. (7) (for simplicity, in one dimension) is given by the
following initial value problem of a second-order ODE:

$\displaystyle\begin{split}\frac{d^{2}}{dt^{2}}M_{qp}=-\frac{1}{m}\frac{\partial^{2}V(q)}{\partial
q^{2}}M_{qp}\\\
M_{qp}(0)=0\;\;\;\;\;\;\;\;\;\dot{M}_{qp}=\frac{1}{m}\end{split}$ (10)

If the Hessian is assumed to be a constant negative value, the solutions to
the above ODE will be exponential and it is assumed to constant and positive,
the solutions are trigonometric. Hence, we can expect the following
factorization to hold for the classical stability matrix elements in general:

$\displaystyle\frac{\partial q_{t}}{\partial
p}=\mathcal{S}(t;p,q)\>\exp({\Lambda(t;p,q)})$ (11)

where $q,p$ are the initial momenta and position respectively,
$\mathcal{S}(t;p,q)$ is a trigonometric function of $t$ and $\Lambda(t;p,q)$
is a positive function of $t$, i.e.
$|\mathcal{S}(t;\textbf{p},\textbf{q})|\leq 1$ and
$\Lambda(t;\textbf{p},\textbf{q})\geq 0$.

In the presence of a global maximal Lyapunov exponent $\lambda$ for the
classical phase-space, we then have:

$\displaystyle C^{(cl)}_{T}(t)$ $\displaystyle=\frac{1}{\mathcal{Z}_{cl}}\int
d{p}\>d{q}\>e^{-\beta H({p},{q})}\>\absolutevalue{\frac{\partial
q_{t}}{\partial p}}^{2}$ (12) $\displaystyle\leq e^{\lambda t}\int
d{p}\>d{q}\>e^{-\beta H({p},{q})}\>\mathcal{S}(t;{p},{q})\>\frac{\partial
q_{t}}{\partial p}$ (13)

Integrating the RHS above by parts, we obtain:

$\displaystyle C^{(cl)}_{T}(t)\leq e^{\lambda t}\int d{p}\>d{q}\>e^{-\beta
H({p},{q})}\>q(t)\>\frac{\partial\mathcal{S}(t;{p},{q})}{\partial p}$ (14)

Since the function $e^{-\beta H}$ is positive, by the Cauchy Schwartz
inequality of weighted inner products, we get:

$\displaystyle C^{(cl)}_{T}(t)\leq e^{\lambda t}\>\langle
q(t)^{2}\rangle_{\beta}^{\nicefrac{{1}}{{2}}}\;\;\left\langle\left(\frac{\partial\mathcal{S}}{\partial
p}\right)^{2}\right\rangle_{\beta}^{\nicefrac{{1}}{{2}}}$ (15)

where $\langle A\rangle_{\beta}=\int\int\>d{p}\>d{q}\>e^{-\beta H(q,p)}A$.
Since the thermal expectation values of operators are bounded,

$\displaystyle C_{T}^{(cl)}(t)\leq O(e^{\lambda t})$ (16)

provided $\nicefrac{{\partial\mathcal{S}(t)}}{{\partial p}}$ has, at most, a
polynomial growth rate. Since $S(t)$ is trigonometric by definition, we can
expect this indeed to be the case. The same argument also holds for other
matrix elements (for instance, the $M_{qq}$ element considered in [11]) - the
choice of $M_{pq}$ for the derivation above is purely for convenience. So if
the classical approximation to the OTOC in eq. (6) is exponential, the
resulting classical growth rate ‘trivially’ satisfies the bound

$\displaystyle\lambda^{cl}(T)\leq\frac{2\pi k_{B}T}{\hbar}$ (17)

as long as the temperature satisfies:

$\displaystyle T\geq\frac{\hbar\lambda}{2\pi k_{B}}$ (18)

the RHS of which was identified to be the instanton ‘crossover’ temperature in
[11]. However, it is to be noted that, mathematically, this derivation does
not guarantee that thermal OTOCs are exponential for the parameter values and
temperatures considered.

##  III Lower bound on classical growth rate

The bound above on the classical growth rate in thermal ensembles holds only
in the presence of a maximal Lyapunov exponent. One important class of quantum
systems that have such a maximal exponent are those whose corresponding
classical counterparts have an isolated saddle point in the phase space.
Examples of such systems include the Dicke model[7], Bose-Hubbard dimer[17]
and many potential energy surfaces (PES) of interest in chemistry. It has been
shown in [6] that the semiclassical approximation to the growth rate of
microcanonical OTOCs in such systems obeys a lower bound condition as follows:

$\lambda_{\text{OTOC}}\geq\lambda_{\text{saddle}}$ (19)

We consider the dynamics around such a generic one-dimensional saddle point
described by an inverted harmonic oscillator, which has been the prototype for
saddle-point dynamics in several previous works [15, 18, 11]:

$\displaystyle\begin{split}V(q)&=-\frac{1}{2}m\lambda^{2}q^{2}+P(q)\\\
H(p,q)&=\frac{p^{2}}{2m}+V(q)\end{split}$ (20)

Here, the parameter $\lambda$ determines $\lambda_{\text{saddle}}$. The term
$P(q)$ in the equation above ‘stabilises’ the system and defines the system
size, and is required for thermal equilibrium in a canonical ensemble. This
term hence needs to be atleast quartic in $q$. Making the change of variable
$a_{+}=p+m\lambda q$ and $a_{-}=p-m\lambda q$ as in [6] to denote the unstable
and stable directions of the saddle, we get:

$\displaystyle H(a_{+},a_{-})=\frac{a_{+}a_{-}}{2m}+P(a_{+},a_{-})$ (21)

The dynamics around this saddle point is given by:

$\displaystyle\frac{da_{\pm}}{dt}=\pm\lambda
a_{\pm}+\widetilde{P}(a_{+},a_{-})$ (22)

Following the variable transformation above, it is straightforward to show
that, for a small perturbation $P(q)$,

$|M_{qp}(t)|^{2}\gtrsim\frac{|M_{++}(t)|^{2}}{4m^{2}\lambda^{2}}$ (23)

where $M_{++}$ corresponds to the stability matrix evaluated along the $a_{+}$
coordinate. This gives:

$\begin{split}C_{T}^{cl}(t)&=\int e^{-\beta H(p,q)}|M_{qp}|^{2}dq\;dp\\\
&\gtrsim\int e^{-\beta
H(a_{+},a_{-})}\;\frac{|M_{++}|^{2}}{2m\lambda}\;da_{+}\;da_{-}\end{split}$
(24)

Considering a narrow strip $S_{t}=\\{|a_{+}|\leq\delta e^{-\lambda
t},|a_{-}|\leq\delta\\}$ around the saddle point at the origin as in [6], we
get:

$C_{T}^{cl}(t)\gtrsim\int_{S_{t}}e^{-\beta
H(a_{+},a_{-})}\;\frac{|M_{++}|^{2}}{2m\lambda}\;da_{+}\;da_{-}$ (25)

As long as the temperature is high enough such that $\beta$ satisfies

$\displaystyle\beta P(a_{+},a_{-})_{\text{typical}}<<1$ (26)

we get:

$C_{T}^{cl}(t)\gtrsim\frac{e^{2\lambda
t}}{2m\lambda}\int_{S_{t}}e^{-\beta\nicefrac{{a_{+}a_{-}}}{{2m}}}\;da_{+}\;da_{-}$
(27)

Here, $P(a_{+},a_{-})_{\text{typical}}$ is the value of the perturbation term
evaluated at typical values of $a_{+}$ and $a_{-}$ in the metastable regions
of classical regions far from the saddle point, at thermal equilibrium. As
mentioned earlier, this value quantifies the size of the system and this
condition ensures that the relative energy separation between the saddle point
and the metastable regions is much smaller in comparison to thermal
fluctuations at this temperature. For instance, for $P(q)=gq^{4}$ as in [15],
this corresponds to the condition that the saddle-point energy $V_{b}$
satisfies $V_{b}<<k_{B}T$.

Eq. (27) can be simplified further to yield:

$\displaystyle\begin{split}C_{T}^{cl}(t)&\gtrsim\frac{e^{2\lambda
t}}{\lambda}\frac{4\;\text{Shi}(\nicefrac{{\beta\delta^{2}e^{-\lambda
t}}}{{2m}})}{\beta}\\\ &\gtrsim\frac{2\delta^{2}e^{\lambda
t}}{m\lambda}+\frac{\beta^{2}\delta^{6}e^{-\lambda t}}{36\;\lambda
m^{3}}=O(e^{\lambda t})\end{split}$ (28)

where $\text{Shi}(x)$ is the hyperbolic sine integral. Combining this with eq.
(14) yields

$\displaystyle C_{T}^{cl}(t)\sim\Theta(e^{\lambda t})$ (29)

Thus, in the classical limit, the quantum thermal OTOC can be expected to grow
exponentially, provided the temperature condition in eq. (26) holds. This is
borne out by the fact that thermal OTOCs have been shown to be exponential for
a range of temperatures in [15, 11]. While thermal averaging ‘quenches’
exponential dependence with time (and hence the ability to probe chaos) in
linear response functions and some non-linear response functions, OTOC
survives this quenching, albeit with a factor-two reduction from the
microcanonical growth rate.

Note that, while the derivations above have been worked out for the one-
dimensional case, the same holds for a multidimensional system as well,
provided the index of the saddle remains one, as was discussed in [6].

##  IV Quantum-Boltzmann statistics and ring-polymer Lyapunov exponents

In [11], we showed that at lower temperatures (far lower than that
characterised by eq. (26)), the quantumness of the Boltzmann density operator
starts to affect the scrambling rate and the quantum Lyapunov exponent
obtained from the thermal quantum OTOC in eq. (6) is smaller than the
classical growth rate $\lambda$. It is erroneous to approximate the growth
rate of the quantum OTOC with the classical Lyapunov exponent (whose maximal
value is $\lambda$) below this temperature.

By considering the classical dynamics of fictitious ring-polymers (which
corresponds to imaginary-time paths that describe the quantum-Boltzmann
statistics) in an extended phase space - which forms the basis of ring-polymer
molecular dynamics (RPMD) [19] procedure, we showed that the quantum bound to
chaos in [1] is brought about by the emergence of delocalised instantonic
structures at low temperatures. These structures necessarily emerge at
temperatures below regimes of ‘classicality’ as described above, as they
describe thermally-assisted tunneling pathways. It was also shown that all
chaotic trajectories are dynamically coupled to these bounce instantons at low
temperatures, which suggest that the maximal Lyapunov exponent in the ring-
polymer phase space is smaller than the corresponding classical value.

Figure 1: The exponential growth of the quantum thermal OTOC in (6) and the
corresponding classical (eq. (7) and RPMD approximations (eq. (30)) at various
temperatures. The maximal Lyapunov exponent of the ring polymer centroid
evaluated in the extended phase space of ring-polymers at various
temperatures. They are shown to be obeying the bound (indicated by the dotted
line) in (2)

This reduced maximal value of the ring-polymer Lyapunov exponents can be found
by augmenting an existing symplectic integration procedure[20] used for
computing classical Lyapunov exponents to the extended phase space of ring
polymers. In figure 1, we provide the results for the ‘ring-polymer’ Lyapunov
exponents computed for each temperature in the chaotic two-dimensional double
well potential considered in [11] at the saddle point of the ring-polymer PES.
We also plot the growth rate of the quantum thermal OTOC and its RPMD and
classical approximation as a function of temperature for reference.

As can be seen, the ring-polymer exponents necessarily obey the quantum bound
and they differ from the classical Lyapunov exponent below the ‘cross-over’
temperature

$T_{c}=\frac{\hbar\lambda}{2\pi k_{B}}$

mentioned in (18) at which instantons start to form. These instantons become
the dominant saddle point in the ring-polymer PES below this temperature.

The growth rate of the RPMD (approximation to) OTOC is given as:

$\displaystyle C^{\text{RP}}_{N}(t)=\frac{\hbar^{2}}{\mathcal{Z}}\int
d\textbf{p}^{N}\\!\int
d\textbf{q}^{N}\;e^{-\beta_{N}H_{N}(\textbf{p},\textbf{q})}\left|\frac{\partial
Q_{0t}}{\partial P_{0}}\right|^{2}$ (30)

in which $\partial Q_{t}/{\partial P_{0}}$ is the stability matrix
(corresponding to $M_{qp}$ in the classical case) of the ring-polymer centroid
coordinate. By replacing the classical maximal Lyapunov exponent $\lambda$
with the (temperature-dependent) ring-polymer maximal exponents in fig. 1 and
repeating the derivations above, it is straightforward to see that:

$\displaystyle C_{T}^{RPMD}(t)\leq
O(\>\exp({\lambda^{RP}_{\text{max}}(T)t})\>)$ (31)

Fig. 1 confirms that the growth rate of the RPMD OTOC below $T_{c}$ is indeed
less than the maximal ring-polymer exponents.

We have hence established that the mechanism responsible for the reduction of
the quantum Lyapunov exponent computed from OTOCs in thermal quantum systems
is two-fold: the classical thermal averaging brings about a factor-of-two
reduction from the microcanonical rate. The quantumness of the Boltzmann
distribution further quenches the quantum Lyapunov exponent below this reduced
value. We believe that finding ring-polymer Lyapunov exponents in the extended
phase space is a viable strategy to quantify the latter.

The majority of the derivation outlined in this letter hinges on the existence
of a global maximal Lyapunov exponent independent of the system’s energy.
While first-quantized Hamiltonians with isolated saddle points form an
important class of such systems, we expect a similar mechanism to bring about
the quantum bound on chaos in general Hamiltonian systems. Since it is
expected that the Sachdev-Ye-Kitaev (SYK) model that saturates this bound is
‘mono-fractal’ [21] and has a single (and hence maximal) Lyapunov exponent for
all phase-space trajectories, we believe that the discussions in this letter
pertains to a large class of systems.

We would like to thank xxxx for stimulating discussions and for commenting on
an earlier version of this manuscript. VGS acknowledges the support of a
Manmohan Singh PhD scholarship from St John’s College, Cambridge.

## References

  * Maldacena  _et al._ [2016] J. Maldacena, S. H. Shenker, and D. Stanford, A bound on chaos, Journal of High Energy Physics 2016, 1 (2016).
  * Sekino and Susskind [2008] Y. Sekino and L. Susskind, Fast scramblers, Journal of High Energy Physics 2008, 065 (2008).
  * Swingle [2018] B. Swingle, Unscrambling the physics of out-of-time-order correlators, Nature Physics 14, 988 (2018).
  * Larkin and Ovchinnikov [1969] A. Larkin and Y. N. Ovchinnikov, Quasiclassical method in the theory of superconductivity, Sov Phys JETP 28, 1200 (1969).
  * Rozenbaum  _et al._ [2017] E. B. Rozenbaum, S. Ganeshan, and V. Galitski, Lyapunov exponent and out-of-time-ordered correlator’s growth rate in a chaotic system, Phys. Rev. Lett. 118, 086801 (2017).
  * Xu  _et al._ [2020] T. Xu, T. Scaffidi, and X. Cao, Does scrambling equal chaos?, Physical review letters 124, 140602 (2020).
  * Chávez-Carlos  _et al._ [2019] J. Chávez-Carlos, B. López-del Carpio, M. A. Bastarrachea-Magnani, P. Stránský, S. Lerma-Hernández, L. F. Santos, and J. G. Hirsch, Quantum and classical lyapunov exponents in atom-field interaction systems, Phys. Rev. Lett. 122, 024101 (2019).
  * Murthy and Srednicki [2019] C. Murthy and M. Srednicki, Bounds on chaos from the eigenstate thermalization hypothesis, Phys. Rev. Lett. 123, 230606 (2019).
  * Tsuji  _et al._ [2018] N. Tsuji, T. Shitara, and M. Ueda, Bound on the exponential growth rate of out-of-time-ordered correlators, Phys. Rev. E 98, 012216 (2018).
  * Pappalardi  _et al._ [2022] S. Pappalardi, L. Foini, and J. Kurchan, Quantum bounds and fluctuation-dissipation relations, SciPost Phys. 12, 130 (2022).
  * Sadhasivam  _et al._ [2022] V. G. Sadhasivam, L. Meuser, D. R. Reichman, and S. C. Althorpe, Instantons and the quantum bound to chaos, arXiv preprint arXiv:2212.10202 (2022).
  * Asban  _et al._ [2021] S. Asban, K. E. Dorfman, and S. Mukamel, Interferometric spectroscopy with quantum light: Revealing out-of-time-ordering correlators, The Journal of chemical physics 154, 210901 (2021).
  * Mukamel  _et al._ [1996] S. Mukamel, V. Khidekel, and V. Chernyak, Classical chaos and fluctuation-dissipation relations for nonlinear response, Phys. Rev. E 53, R1 (1996).
  * Dellago and Mukamel [2003] C. Dellago and S. Mukamel, Simulation strategies and signatures of chaos in classical nonlinear response, Physical Review E 67, 035205 (2003).
  * Hashimoto  _et al._ [2020] K. Hashimoto, K.-B. Huh, K.-Y. Kim, and R. Watanabe, Exponential growth of out-of-time-order correlator without chaos: inverted harmonic oscillator, Journal of High Energy Physics 2020, 1 (2020).
  * Akutagawa  _et al._ [2020] T. Akutagawa, K. Hashimoto, T. Sasaki, and R. Watanabe, Out-of-time-order correlator in coupled harmonic oscillators, Journal of High Energy Physics 2020, 1 (2020).
  * Kidd  _et al._ [2021] R. A. Kidd, A. Safavi-Naini, and J. F. Corney, Saddle-point scrambling without thermalization, Phys. Rev. A 103, 033304 (2021).
  * Bhattacharyya  _et al._ [2021] A. Bhattacharyya, W. Chemissany, S. S. Haque, J. Murugan, and B. Yan, The multi-faceted inverted harmonic oscillator: Chaos and complexity, SciPost Physics Core 4, 002 (2021).
  * Craig and Manolopoulos [2004] I. R. Craig and D. E. Manolopoulos, Quantum statistics and classical mechanics: Real time correlation functions from ring polymer molecular dynamics, The Journal of chemical physics 121, 3368 (2004).
  * Skokos and Gerlach [2010] C. Skokos and E. Gerlach, Numerical integration of variational equations, Physical Review E 82, 036704 (2010).
  * Pappalardi and Kurchan [2023] S. Pappalardi and J. Kurchan, Quantum bounds on the generalized lyapunov exponents, Entropy 25, 246 (2023).


# Speech-Text Dialog Pre-training for Spoken Dialog Understanding
with Explicit Cross-Modal Alignment

Tianshu Yu123∗, Haoyu Gao134, Ting-En Lin3, Min Yang1†,
Yuchuan Wu3, Wentao Ma3, Chao Wang3, Fei Huang3, Yongbin Li3
1Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences
2University of Chinese Academy of Sciences
3Alibaba Group 4University of Science and Technology of China
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>Equal contribution. This
work was conducted when Tianshu Yu and Haoyu Gao were interning at Alibaba.†
Min Yang and Yongbin Li are corresponding authors.

###### Abstract

Recently, speech-text pre-training methods have shown remarkable success in
many speech and natural language processing tasks. However, most previous pre-
trained models are usually tailored for one or two specific tasks, but fail to
conquer a wide range of speech-text tasks. In addition, existing speech-text
pre-training methods fail to explore the contextual information within a
dialogue to enrich utterance representations. In this paper, we propose
Speech-text dialog Pre-training for spoken dialog understanding with ExpliCiT
cRoss-Modal Alignment (SPECTRA), which is the first-ever speech-text dialog
pre-training model. Concretely, to consider the temporality of speech
modality, we design a novel temporal position prediction task to capture the
speech-text alignment. This pre-training task aims to predict the start and
end time of each textual word in the corresponding speech waveform. In
addition, to learn the characteristics of spoken dialogs, we generalize a
response selection task from textual dialog pre-training to speech-text dialog
pre-training scenarios. Experimental results on four different downstream
speech-text tasks demonstrate the superiority of SPECTRA in learning speech-
text alignment and multi-turn dialog context.111For reproducibility, we
release our code and pre-trained model at:
https://github.com/AlibabaResearch/DAMO-ConvAI/tree/main/SPECTRA.

##  1 Introduction

In recent years, speech-text pre-training, which learns universal feature
representations from a large training corpus (Chen et al., 2018; Li et al.,
2021; Bapna et al., 2021), has achieved significant success in both uni-modal
(Schneider et al., 2019; Dosovitskiy et al., 2020) and multi-modal (Lu et al.,
2019; Radford et al., 2021) downstream tasks. Existing speech-text pre-
training works mainly employed multi-modal self-supervised pre-training
objectives, such as cross-modal masked data modeling (Li et al., 2021; Kang et
al., 2022a) and cross-modal contrastive learning (Sachidananda et al., 2022;
Elizalde et al., 2022), which align the speech utterance representation to the
corresponding text sentence representation.

Figure 1: An illustration of SPECTRA, which considers dialogue context and
explicit alignment between text and speech during pre-training, and
generalizes well on various downstream tasks.

Despite the remarkable progress of previous speech-text pre-training models,
there are still several technical challenges to constructing an effective and
unified speech-text pre-training model for spoken dialog understanding, which
are not addressed well in prior works. First, previous models are mainly
tailored for specific speech-text tasks, such as speech-to-text translation
Liu et al. (2020b) and speech-language understanding Chung et al. (2021),
failing to conquer a wide range of speech-text tasks. Although Tang et al.
(2022) proposed a unified speech-text pre-training for speech translation and
recognition, it fails to exploit the temporality of an input speech sequence
and cannot learn the fine-grained speech-text alignment.

Second, limited exploration has been attempted to bridge the gap between plain
speeches/texts and human conversations. In particular, existing speech-text
pre-training methods fail to explore the context information within a dialog.
Nevertheless, spoken dialog understanding needs to effectively process context
information so as to help the system better understand the current utterance,
since humans may omit previously mentioned entities/constraints and introduce
substitutions to what has already been mentioned.

In this paper, we propose Speech-text dialog Pre-training for spoken dialog
understanding with ExpliCiT cRoss-Modal Alignment (SPECTRA), which is the
first-ever speech-text dialog pre-training model. We illustrate the framework
of our method in Figure 1 and details in Figure 2. The backbone of SPECTRA is
composed of a text encoder, a speech encoder, and a fusion module, learning
semantic/acoustic information and the interaction between them, and pre-
trained on a large-scale real-world multi-modal (speech-text) dialog corpus.
We propose two pre-training objectives to learn better context-aware
speech/text representations for spoken dialog understanding Dai et al. (2022);
Zhang et al. (2022b). Specifically, to consider the temporality of speech
modality, we design a novel temporal position prediction task to capture the
speech-text alignment by predicting the start and end time of each textual
word in the corresponding speech waveform. In addition, to learn the
characteristics of spoken dialogs Gao et al. (2023); Qian et al. (2023), we
devise a cross-modal response selection objective to consider the context
information within each dialog.

Our contributions are summarized as follows:

  * •

To the best of our knowledge, we are the first to propose a speech-text dialog
pre-training model for spoken dialog understanding, which fully exploits the
characteristics of multi-modal (speech/text) dialogs.

  * •

We introduce two pre-training objectives (temporal position prediction and
multi-modal response selection) to effectively learn speech-text alignment and
dialog context information.

  * •

We conduct extensive experiments on five benchmark datasets belonging to four
downstream speech-text tasks, including emotion recognition in conversation
(ERC), multi-modal sentiment analysis (MSA), spoken language understanding
(SLU), and dialog state tracking (DST). We believe that the release of the
pre-trained model and source code would push forward the research in this
area.

Figure 2: The overview of SPECTRA. The left part shows the illustration of the
temporal position prediction task and the cross-modal response selection task.
The right part shows the overall structure of the pre-trained model.

##  2 Related Work

##### Uni-modal Pre-training

In recent years, pre-trained language models (PLMs), such as BERT (Kenton and
Toutanova, 2019), RoBERTa (Liu et al., 2019), and GPT (Radford et al., 2019a)
have been proposed and applied to many NLP tasks, yielding impressive
performances. PLMs benefit from the rich linguistic knowledge in large-scale
corporaHe et al. (2022c, a). Inspired by the success of PLMs in NLP tasks,
several speech pre-training models, such as Wav2vec (Schneider et al., 2019),
HuBERT (Hsu et al., 2021), and WavLM (Chen et al., 2022), were proposed to
learn high-quality universal speech representations from massive speech data.

##### Multimodal Pre-training

Compared to multi-modal pre-training for vision-and-language tasks, speech-
text pre-training is relatively less explored. SpeechBERT (Chuang et al.,
2020) jointly trained multimodal representations based on a single BERT for
spoken question-answering. CTAL (Li et al., 2021) extended the original
Transformer to cross-modal by modifying the attention mechanism of the
Transformer decoder. ST-BERT (Kim et al., 2021) combined a pre-trained
acoustic model with BERT and took phoneme posterior and subword-level
tokenized text as input. Kang et al. (2022b) explored multimodal pre-training
model in extremely low-resource data scenarios. CLAM (Sachidananda et al.,
2022) employed contrastive and multirate information inherent in audio and
lexical inputs to align acoustic and lexical information. STPT Tang et al.
(2022) proposed a multi-task learning framework to integrate different
modalities in speech-text pre-training.

##### Multimodal Dialog Systems

The demand for multimodal dialog systems Lin et al. (2022) is increasing due
to the ubiquitous multimodal data. Liao et al. (2018) presented a knowledge-
aware multimodal dialog (KMD) model, which leveraged reinforcement learning to
generate human-like responses given multimodal (text-image) dialog context.
Cui et al. (2019) considered the explicit user requirements in the attribute
level and dynamically encoded the multimodal (text-image) dialog context based
on users’ attention. Sunder et al. (2022) proposed an end-to-end spoken
language understanding model, which trained a semantically rich BERT-based
conversation model along with a speech-based model.

Different from previous works, SPECTRA is the first-ever speech-text dialog
pre-training model, which bridges the gap between plain texts/speeches and
human conversations.

##  3 Method

In this section, we introduce the model architecture and pre-training
objectives of SPECTRA.

###  3.1 The Backbone Architecture

Figure 2 shows the overall structure of our model SPECTRA, which consists of a
text encoder, a speech encoder, and a modality fusion module. During pre-
training, we first convert paired text and speech inputs into uni-modal
embeddings, which are then fed into the text encoder and speech encoder
respectively to obtain uni-modal representations. Finally, we concatenate text
representations and speech representations as input of our modality fusion
module to get fused representations for speech-text pre-training.

####  3.1.1 Data Preparation

Before diving into our model, we first prepare input text and speech sequences
for our model. Let $D=\\{T_{1},T_{2},...,T_{n}\\}$ denotes a conversation with
$n$ dialog turns, where every single dialog turn $T_{i}$ consists of a slice
of raw speech waveform $\mathbf{s}_{i}$ and its corresponding text
$\mathbf{t}_{i}=\\{\mathbf{w}_{i1},\mathbf{w}_{i2},...,\mathbf{w}_{im}\\}$.
Here, $\mathbf{w}_{ij}$ is the $j$-th word of $\mathbf{t}_{i}$, and is
annotated with its corresponding start/end time in the speech, denoted as
$s_{ij}$/$e_{ij}$. $m$ is the sentence length of $\mathbf{t}_{i}$. For each
dialog turn $T_{i}$ where $i>1$, we construct a sample $\mathbf{X}_{i}$ with
current utterance $T_{i}=\\{\mathbf{t}_{i},\mathbf{s}_{i}\\}$, previous $k$
($k\geq 1$) turns of textual dialog history
$\\{\mathbf{t}_{i-k},,...,\mathbf{t}_{i-2},\mathbf{t}_{i-1}\\}$ and the
previous speech dialog history $\mathbf{s}_{i-1}$. In this way, each sample
$\mathbf{X}_{i}$ consists of $k$+1 turns of text and 2 turns of speeches,
where the speeches correspond to the latest 2 turns of text. Note that we only
use 2 turns of speech in pre-training for efficiency, since the length of
speech representation is much longer than its corresponding text
representation.

####  3.1.2 Text Embeddings

For each input element, its vector representation is a summation of the
corresponding token embedding, absolute position embedding and segment
embedding. Specifically, we first concatenate all text sentences of each
sample $\mathbf{X}_{i}$ in temporal order to construct the text input:
$I_{i}$=<s>$\mathbf{t}_{i-k}$</s>$\mathbf{t}_{i-k+1}$</s>…</s>$\mathbf{t}_{i-1}$</s>$\mathbf{t}_{i}$</s>.
Note that we use special token <s> to mark the start of the whole sequence,
and </s> to mark the end of each turn. Then, we encode each token in $I_{i}$
using a pre-trained RoBERTa (Liu et al., 2019) tokenizer. We assign learnable
segment embedding $\mathbf{e}_{t,1}$ to tokens of $\mathbf{t}_{i}$ and the
last </s> token, and $\mathbf{e}_{t,0}$ for the rest of the tokens. The
detailed tokenizing and encoding process is described in Appendix A. We denote
$\mathbf{x}_{i}$ as the input text embeddings of $I_{i}$.

####  3.1.3 Uni-modal Encoders

##### Text Encoder

Inspired by the remarkable success of uni-modal pre-trained models on various
downstream tasks, we employ RoBERTa (Liu et al., 2019) as our text encoder. We
pass $\mathbf{x}_{i}$ into text encoder to obtain the sequence
representations:

$\mathbf{H}_{t,i}=\text{RoBERTa}(\mathbf{x}_{i})$ (1)

where $\mathbf{H}_{t,i}\in\mathbb{R}^{n\times d_{h}}$ denotes the output
hidden states of the last layer of RoBERTa, $n$ is the length of input
$I_{i}$, and $d_{h}$ is the dimension of hidden state.

##### Speech Encoder

We design our speech encoder based on the WavLM structure (Chen et al., 2022)
with three key modules: a feature extractor, a feature projection module and a
Transformer encoder module. The feature extractor consists of 8 temporal
convolutional layers and a layer normalization. We implemented the first seven
convolutional layers to be the same as WavLM, and added another convolutional
layer with 512 channels, 5 strides and 5 kernels size, in order to shorten the
length of the output speech features. As a result, each output token of speech
features represents approximately 200ms of speech with a stride of 100ms.

The feature projection layer is a layer normalization followed by a fully
connected layer converting the size of speech features from 512 to $d_{h}$.
The Transformer encoder module is equipped with a convolution-based relative
position embedding layer and 12 WavLM Transformer layers. For each sample, we
directly input speech waveforms $\mathbf{s}_{i-1}$ and $\mathbf{s}_{i}$ into
our speech encoder, and denote the outputs of the feature projection layer for
$\mathbf{s}_{i-1}$ and $\mathbf{s}_{i}$ as $f_{i-1}$ and $f_{i}$:

$\displaystyle f_{i-1}$
$\displaystyle=\text{Proj}(\text{Conv}(\mathbf{s}_{i-1}))$ (2) $\displaystyle
f_{i}$ $\displaystyle=\text{Proj}(\text{Conv}(\mathbf{s}_{i}))$ (3)

Then, we obtain a speech sequence $a_{i}$ by concatenating $f_{i-1}$ and
$f_{i}$ together with a separation token [SEP] and a starting token [CLS]:

$a_{i}=[\text{CLS}]f_{i-1}[\text{SEP}]f_{i}$ (4)

where $a_{i}\in\mathbb{R}^{(m_{i-1}+m_{i}+2)\times d_{h}}$ denotes the
concatenated sequence. $m_{i-1}$ and $m_{i}$ are the lengths of
$\mathbf{s}_{i-1}$ and $\mathbf{s}_{i}$, respectively. We pass $a_{i}$ as the
input of the Transformer encoder module to get the speech sequence
representations:

$\mathbf{H}_{s,i}=\text{WavLM}(a_{i})$ (5)

where $\mathbf{H}_{s,i}\in\mathbb{R}^{(m_{i-1}+m_{i}+2)\times d_{h}}$ denotes
the hidden states of the last Transformer layer.

####  3.1.4 Modality Fusion Module

To integrate two modalities, we employ a single self-attention Transformer
layer as our modality fusion module. We first concatenate the text sequence
representation $\mathbf{H}_{t,i}$ and the speech sequence representation
$\mathbf{H}_{s,i}$ together. Then, we assign text and speech representations
with learnable modality embeddings $\mathbf{e}_{m,0}$ and $\mathbf{e}_{m,1}$
respectively, and add the modality embeddings to the concatenated
representations as the input of our modality fusion module. Finally, we obtain
output hidden representations of modality fusion module
$\mathbf{H}_{i}\in\mathbb{R}^{(n+m_{i-1}+m_{i}+2)\times d_{h}}$ as the speech-
text joint representations.

###  3.2 Pre-training Tasks

We introduce two novel pre-training objectives for our SPECTRA model,
empowering SPECTRA to capture speech-text alignment and multimodal dialog
context effectively.

####  3.2.1 Temporal Position Prediction

Existing speech-text pre-training works mainly learn from prior visual-text
pre-training models. These works ignore that speeches are temporal sequences,
and thus fail to learn fine-grained speech-text alignment. In this work, we
propose a novel temporal position prediction (TPP) objective, which utilizes
the textual part of the hidden representations $\mathbf{H}_{i}$ to predict the
starting and ending time of each word in the speech waveform.

In particular, for each word $\mathbf{w}_{ij}$ in utterance $\mathbf{t}_{i}$
with its start/end time annotations $s_{ij}$/$e_{ij}$, we denote its
first/last token in $\mathbf{H}_{i}$ as
$\mathbf{h}_{s_{ij}}$/$\mathbf{h}_{e_{ij}}$. The goal of the TPP pre-training
objective is to predict its starting and ending time in $\mathbf{s}_{i}$ with
$\mathbf{h}_{s_{ij}}$ and $\mathbf{h}_{e_{ij}}$, respectively. We use squared
error loss to optimize the TPP task:

$\displaystyle\mathcal{L}_{\rm TPP}(t_{i})$
$\displaystyle=\frac{1}{2}\left(\left(\mathbf{W}_{start}\mathbf{h}_{s_{ij}}-\frac{s_{ij}}{L_{a}}\right)^{2}\right.$
(6)
$\displaystyle+\left.\left(\mathbf{W}_{end}\mathbf{h}_{e_{ij}}-\frac{e_{ij}}{L_{a}}\right)^{2}\right)$

where $\mathbf{W}_{start},\mathbf{W}_{end}\in\mathbb{R}^{d_{h}\times 1}$ are
learnable parameters. $L_{a}$ is the maximum speech length limit. By
normalizing $s_{ij}$ and $e_{ij}$ over $L_{a}$, we guarantee that the starting
and ending time falls into [0,1]. Here, we only calculate the TPP loss for the
words in the last two turns of dialog (i.e., $\mathbf{t}_{i-1}$ and
$\mathbf{t}_{i}$) for each sample $\mathbf{X}_{i}$. We calculate the average
TPP loss over all words within those two turns as the TPP loss of dialog
$\mathbf{X}_{i}$:

$\mathcal{L}_{\rm TPP}=\frac{1}{l_{i-1}+l_{i}}[\sum_{j}\mathcal{L}_{\rm
TPP}(\mathbf{w}_{i-1,j})+\sum_{j}\mathcal{L}_{\rm TPP}(\mathbf{w}_{i,j})]$ (7)

where $l_{i-1}$ and $l_{i}$ denote the total lengths of transcripts
$\mathbf{t}_{i-1}$ and $\mathbf{t}_{i}$ in sample $\mathbf{X}_{i}$.

####  3.2.2 Cross-modal Response Selection

Inspired by the success of response selection tasks in textual dialog systems
(Bao et al., 2019), we design a cross-modal response selection objective. For
each sample $\mathbf{X}_{i}$, we randomly replace the text query
$\mathbf{t}_{i}$ or speech query $\mathbf{s}_{i}$ with the utterances or
speech from other dialogs in the dataset. In this way, for each sample
$\mathbf{X}_{i}$, we can obtain three kinds of corrupted samples as negatives:
(1) only the speech query is randomly substituted; (2) only the text query is
randomly substituted; (3) both text and speech queries are randomly
substituted. Note that both text and speech queries remain unchanged as
positive as illustrated in Figure 2

Since the output of the first <s> token can be viewed as the representation of
the whole speech-text sample, we apply a softmax function following a fully
connected layer on top of the hidden state of token <s> as a four-way
classifier, predicting which case the current example belongs to. We utilize
the cross-entropy loss to optimize the cross-modal response selection task,
denoted as $\mathcal{L}_{\rm CRS}$.

####  3.2.3 Cross-modal Masked Data Modeling

Following previous works Li et al. (2021), we also adopt the cross-modal
representations $\mathbf{H}_{f}$ for cross-modal masked language modeling
(CMLM) and cross-modal masked acoustic modeling (CMAM) objectives. For masked
language modeling, we follow the setup of RoBERTa (Liu et al., 2019) to
dynamically mask out textual input tokens with a probability of 15%. For
masked acoustic modeling, we follow Baevski et al. (2020) and Liu et al.
(2020a) to mask continuous speech frames.

We modify the implementation of the original masked acoustic modeling method
in previous works to increase the average number of masked speech frames in
each sample. We provide the details of masked acoustic modeling in Algorithm 1
in Appendix B. The speech token masking step is performed between the feature
extractor and feature projection. We employ the cross-entropy loss for the
CMLM task ($\mathcal{L}_{\rm CMLM}$) and the mean absolute error loss for the
CMAM task ($\mathcal{L}_{\rm CMAM}$).

####  3.2.4 Joint Pre-training Objective

We combine four pre-training objectives to form a joint pre-training objective
for speech-text pre-training:

$\mathcal{L}=\alpha\mathcal{L}_{\rm TPP}+\mathcal{L}_{\rm
CRS}+\mathcal{L}_{\rm CMLM}+\mathcal{L}_{\rm CMAM}$ (8)

###  3.3 Fine-tuning on Downstream Tasks

We fine-tune SPECTRA on four downstream tasks, including multimodal sentiment
analysis (MSA), emotion recognition in conversation (ERC), spoken language
understanding (SLU), and dialog state tracking (DST).

We use the hidden state of <s> token in $\mathbf{H}_{i}$, denoted as
$\mathbf{h}_{i}$ , and pass it through a prediction head with two fully-
connected layers and a GELU activation (Hendrycks and Gimpel, 2016) between
them to get the prediction:

$\mathbf{y}_{i}=\mathbf{W}^{(2)}\sigma(\mathbf{W}^{(1)}\mathbf{h}_{i}+\mathbf{b}^{(1)})+\mathbf{b}^{(2)}$
(9)

where $\sigma$ denotes the GELU activate function,
$\mathbf{W}^{(1)}\in\mathbb{R}^{d_{h}\times d_{h}}$,
$\mathbf{W}^{(2)}\in\mathbb{R}^{d_{h}\times d_{o}}$,
$\mathbf{b}^{(1)}\in\mathbb{R}^{d_{h}}$,
$\mathbf{b}^{(2)}\in\mathbb{R}^{d_{o}}$ are new learnable parameters in the
fine-tuning stage. The output size $d_{o}$ for MSA task is 1, and for ERC and
SLU it is the corresponding number of classes. We adopt the squared error loss
as the fine-tuning loss function for MSA. The cross-entropy loss is utilized
for the rest of tasks.

Task | Dataset | Metric | Previous SOTA | SPECTRA
---|---|---|---|---
Multimodal Sentiment Analysis (MSA) | MOSI | Acc2 | 84.40 (MIB (Mai et al., 2022)) | 87.50 (+3.10)
MOSEI | Acc2 | 86.20 (BBFN (Han et al., 2021)) | 87.34 (+1.14)
Emotion Recognition in Conversation (ERC) | IEMOCAP | Acc | 66.52 (M2FNET (Chudasama et al., 2022)) | 67.94 (+1.42)
Spoken Language Understanding (SLU) | MIntRec | Acc20 | 72.16 (MAG-BERT (Rahman et al., 2020)) | 73.48 (+1.32)
Dialog State Tracking (DST) | SpokenWoz | JGA | 20.90 (SPACE (He et al., 2022b)) | 21.96 (+1.06)

Table 1: The comparison between the key metrics of our model and the previous
SOTA method on five datasets.

##  4 Experiments

###  4.1 Pre-training Data

In this paper, we adopt Spotify100K (Clifton et al., 2020) to pre-train
SPECTRA, which is a real-world scene speech-text dialog dataset. Spotify100K
contains 105,360 podcast episodes, with nearly 60,000 hours of speeches
covering a variety of genres, subject matter, speaking styles, and structure
formats. The corpus also provides automatically-generated word-level textual
transcripts, marking the starting and ending time in the speech for each word.

For a fair comparison with previous speech-text pre-training studies, we only
use the first 960 hours of speech as well as the corresponding transcripts to
pre-train our SPECTRA model.

###  4.2 Experimental Setup

##### Baselines

In addition to state-of-the-art downstream models tailored for MSA, ERC, SLU
and DST (see Section 4.3-4.6), we also compare SPECTRA with three types of
pre-training models, including the text modality pre-training model RoBERTa
Liu et al. (2019), speech modality pre-training model WavLM Chen et al.
(2022), and speech-text multimodal pre-training model CTAL Li et al. (2021).

##### Experimental Settings during Pre-training

We use the first 960 hours of speech and textual transcripts of Spotify100K
dataset for pre-training. We cut the speech waveform into slices of a maximum
length of 10 seconds and view each slice with the corresponding transcripts as
a single dialog turn, forming 356,380 dialog turns in total. By using these
dialogs and setting $k$ to a maximum of 7, we construct 350,784 samples, where
each sample consists of 2~8 dialog turns of texts and 2 turns of speeches.

Besides, we use pre-trained models RoBERTa-base and WavLM-base+ to initialize
our text and speech encoder, respectively. Since our speech encoder has one
more convolution layer than WavLM-base+, we only initialize the first seven
convolution layers with pre-trained parameters and randomly initialize the
last layer. Both text and speech encoders have 12 Transformer layers with a
hidden size $d_{h}$ of 768. We pre-train our SPECTRA model for 100 epochs on 8
Tesla-A100 GPUs with a batch size of 20 per GPU. We use AdamW (Loshchilov and
Hutter, 2018) to optimize our model with a peak learning rate of $1\times
10^{-4}$ and a linear warmup for the first 1% of updates.

##### Experimental Settings during Fine-tuning

For SpokenWoz dataset, each dialog turn consist of two utterances, one from
the user and the other from the system. For other datasets, each dialog turn
is a single utterance. For all datasets we truncate the speech length of each
dialog turn to a maximum of 10 seconds. We fine-tune our pre-trained
checkpoint on each downstream dataset using an AdamW (Loshchilov and Hutter,
2018) optimizer with a peak learning rate of $2\times 10^{-5}$ and a cosine
annealing warmup.

###  4.3 Fine-tuning on MSA

For MSA task Hu et al. (2022), our model aims to predict the positive or
negative sentiment polarities of the given multi-modal input. We conduct
experiments on two multi-modal datasets MOSI (Zadeh et al., 2016) and MOSEI
(Zadeh et al., 2018) to evaluate the effectiveness of our model for the MSA
task. We adopt the accuracy over positive/negative sentiments classification
(denoted as Acc2) as the evaluation metric for our model and baselines. The
experimental results are reported in Table 1.

From the results, we can observe that our model achieves substantially better
performance than previous state-of-the-art (SOTA) methods on both datasets. In
particular, for the MOSI dataset, the accuracy increases by 3.10% over the
strongest baseline MIB (Mai et al., 2022). In addition, as shown in Table 2,
our SPECTRA also significantly outperforms the speech modality pre-training
model WavLM and speech-text pre-training model CTAL.

Settings |  MLM & MAM | TPP | CRS |  Pre-training Data |  Turns of Textual Dialog History | MSA | ERC | SLU | DST
---|---|---|---|---|---|---|---|---|---
MOSI | MOSEI | IEMOCAP | MIntRec | SpokenWoz
Acc2 | Acc2 | Acc | Acc20 | JGA
RoBERTa | - | - | - | - | - | 85.67 | 85.88 | 64.53 | 71.24 | 20.76
WavLM | - | - | - | - | - | 65.85 | 77.90 | 46.90 | 16.63 | -
CTAL | $\checkmark$ | - | - | 960h | - | 72.56 | 80.77 | 55.12 | 53.26 | 15.79
SPECTRA | $\checkmark$ | $\checkmark$ | $\checkmark$ | 960h | 7 | 87.50 | 87.34 | 67.94 | 73.48 | 21.96
(a) |  |  |  | 0 | 1 | 82.16 | 84.30 | 33.17 | 69.21 | 17.59
(b) | $\checkmark$ | $\checkmark$ | $\checkmark$ | 360h | 7 | 85.98 | 86.02 | 66.01 | 72.36 | 20.34
(c) | $\checkmark$ |  | $\checkmark$ | 960h | 7 | 85.52 | 86.19 | 66.15 | 71.69 | 20.87
(d) | $\checkmark$ | $\checkmark$ |  | 960h | 7 | 87.35 | 86.85 | 65.94 | 72.58 | 20.45
(e) | $\checkmark$ | $\checkmark$ | $\checkmark$ | 960h | 1 | 87.20 | 86.93 | 64.98 | 73.03 | 19.78

Table 2: Ablation test results. Here, setting (a) and (b) mean w/o multi-modal
pre-training and using less pre-training data. Setting (c), (d) and (e)
indicate w/o TTP task, w/o CRS task and w/o full turns of textual history,
respectively.

###  4.4 Fine-tuning on ERC

ERC task requires the model to predict the emotion category of an utterance
given a speech clip with its transcripts and dialog history. Here, we fine-
tune our model with the widely-used IEMOCAP dataset (Busso et al., 2008), and
follow the settings with Chudasama et al. (2022) to perform a 6-way
classification task. For each sample, we construct 11 turns of text and 2
turns of speech with a maximum text length of 512.

In Table 1, we report the accuracy of six-way classification for our model and
previous SOTA method M2FNET (Chudasama et al., 2022). In addition, from Table
2, we can observe that our method outperforms uni-modal pre-training models,
as well as speech-text pre-training baseline CTAL. Compared with the uni-modal
baselines RoBERTa and WavLM, our model benefits from multi-modal pre-training
tasks that capture interactions and alignments between modalities. Compared
with CTAL, our model is equipped with better speech-text alignment and multi-
turn dialog context information with the help of TPP and CRS pre-training
tasks.

###  4.5 Fine-tuning on SLU

We also conduct experiments on the spoken language understanding (SLU) task,
which aims to predict the user intent Lin and Xu (2019) given a spoken
utterance with the textual transcript. We use MIntRec (Zhang et al., 2022a) as
the experimental dataset for SLU and adopt classification accuracy for the
evaluation metric.

From Table 1 and 2, we can observe that SPECTRA obtains significantly better
results than previous methods. In particular, our SPECTRA model improves the
results of RoBERTa and the previous SOTA method MAG-BERT (Rahman et al., 2020)
by 1.55% and 2.47%, respectively. Compared to WavLM and CTAL, our model can
capture semantic information in textual data and the context information
within each dialog.

###  4.6 Fine-tuning on DST

For dialogue state tracking, we use an in-house, large-scale, cross-modal
dataset called SpokenWoz. The dataset was collected by crowdsourcing
recordings through phone calls using the Appen platform222https://appen.com/.
Transcriptions were obtained using a commercial ASR system, and speech-text
pairs were annotated using a schema similar to MultiWoz (Eric et al., 2019).
SpokenWoz consists of 204k turns, 5.7k dialog, and 249 hours of recordings.We
adopt joint goal accuracy (JGA) as the evaluation metric, which compares the
predicted and ground-truth dialogue states at each turn. We follow Trippy
(Heck et al., 2020) and substitute its context model BERT with our SPECTRA
model.

As shown in Table 1, our model outperforms the previous SOTA method,
SPACE+WavLM. In addition, our model also surpasses the three pre-training
baselines by a noticeable margin. This demonstrates better speech-text
alignment is critical to tackling complicated conversations.

(a) Case #1: SPECTRA

(b) Case #1: w/o TPP

(c) Case #2: SPECTRA

(d) Case #2: w/o TPP

Figure 3: Visualization of self-attention weights of the fusion module in our
model and the model pre-trained without TPP (w/o TPP). The upper and lower
tokens stand for text and speech tokens, respectively.

Case | Ground-Truth | By SPECTRA | By “w/o TPP” | Given Text
---|---|---|---|---
#1 | Leave |  ✓ Leave |  ✗ Complain | I am going to sit in traffic for 45 minutes and return this.
#2 | Inform |  ✓ Inform |  ✗ Praise | That’s Tommy. He’s lead organizer, total badass.

Table 3: Intent prediction results on test samples from the MIntRec dataset.

##  5 Analysis

###  5.1 Ablation Study

To better understand the effectiveness of our SPECTRA pre-training method, we
investigate the influence of pre-training components and dialog history on the
overall performance of SPECTRA. We report the ablation test results in Table
2.

##### Impact of Pre-training

To demonstrate the efficiency of multi-modal pre-training, we directly use
uni-modal encoders and randomly initialize the modality fusion module. We
observe a significant performance drop by comparing (a) “w/o multi-modal pre-
training” to other pre-training settings on all five datasets. In particular,
setting (a) directly collapses on the ERC task, which is a complicated and
conversational scenario. This verifies the necessity of cross-modal pre-
training and aligning speech-text modalities. In addition, by comparing
SPECTRA and setting (b) “using less pre-training data”, we can find that using
more pre-training data can further improve the performance of our model.

##### Impact of TPP and CRS

By comparing the setting (c) “w/o TPP” to SPECTRA, the performances on all
five datasets drop to different extents, which verifies the generalization and
effectiveness of our TPP pre-training task. Specifically, the performance
drops significantly on SpokenWoz, which requires the model to have a stronger
ability to align two modalities. This demonstrates that our TPP pre-training
task empowers the model with stronger alignment modeling ability. For setting
(d) “w/o CRS” with SPECTRA, the performance drops significantly on multi-turn
dialog tasks such as ERC and DST. This suggests that the CRS task is essential
to model multi-turn dialog context.

##### Impact of Dialog History

In setting (e) “using 1 turn of textual dialog history”, each instance
consists of 2 turns of paired speech and text.The model performance drops
substantially on ERC and DST downstream tasks by comparing it with SPECTRA.
This demonstrates that increasing dialog history in the pre-training stage is
beneficial to the tasks that require multi-turn dialog context.

###  5.2 Case Study

To have a straightforward understanding of how we learn cross-modal
interaction in our proposed SPECTRA model, we conduct a case study by
providing two cases sampled from the MIntRec dataset. These two cases are
incorrectly predicted by the model pre-trained without TPP but correctly
predicted by our SPECTRA model. In Figure 3, we visualize the self-attention
weights of the fusion layer in our model as well as the model pre-trained
without TPP (denoted as w/o TPP). From Figure 3(a) and 3(c), we observe that
there are rich cross-modal interactions in the fusion layer of the proposed
SPECTRA model. Our model can capture fine-grained information between text and
speech for more accurate classification. In contrast, we also visualize the
self-attention weights of the w/o TPP model in Figure 3(b) and 3(d). Both
cases show that text and speech sequences seldom connect to each other in
self-attention layers.

In Table 3, we also illustrate the intent prediction results obtained by
SPECTRA and w/o TPP. From the results, we can observe that our model can
attend to both text and speech sequences effectively to predict correct intent
results. However, w/o TPP is confused by the wrong labels since it hardly
attends to speech tokens, which indicates that it has the propensity to omit
useful information that exists in speech exclusively.

##  6 Conclusion

In this paper, we proposed our model SPECTRA, the first speech-text dialog
pre-training model. Considering the temporality of speech and text modalities,
we introduced a novel temporal position prediction pre-training task to learn
word-level speech-text alignment. To capture multi-modal dialog context in our
model, we generalized the response selection task into multi-modal scenarios.
Extensive experiments show that our pre-training method can learn better
cross-modal interactions as well as multi-modal contextual information and
significantly outperformed other strong baselines. In the future, we would
like to extend speech-text dialog pre-training to more modalities or
generative tasks.

## Limitations

We analyze the limitations of this work, so as to further improve the
performance of our model in future work. Based on our empirical observation,
we reveal several limitations, which can be divided into two primary
categories. (1) First, our proposed SPECTRA method relies on large-scale
spoken dialog corpora with explicit word-level speech-text alignment
annotation, such as Spotify100K. This limits the generality of our model on
more spoken dialog corpora.

In the future, we would like to develop a semi-supervised pre-training method
to leverage both labelled and unlabeled datasets. (2) Second, our method is
mainly designed for speech-text understanding and has not been fully explored
for generative tasks. We plan to devise dialog generation per-training
objective to empower the model with better generation ability. (3) Third, the
work only involves speech and text modalities. We are interested in handling
more modalities, such as images or videos, to enrich cross-modal information
in joint representations.

## Acknowledgements

Min Yang was partially supported by National Key Research and Development
Program of China (2022YFF0902100), Shenzhen Science and Technology Innovation
Program (KQTD20190929172835662), Shenzhen Basic Research Foundation
(JCYJ20210324115614039 and JCYJ20200109113441941), and NSFC (no. 92270122).
This work was supported by Alibaba Group through Alibaba Innovative Research
Program.

## References

  * Baevski et al. (2020) Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. 2020.  wav2vec 2.0: A framework for self-supervised learning of speech representations.  _Advances in Neural Information Processing Systems_ , 33:12449–12460.
  * Bao et al. (2019) Siqi Bao, Huang He, Fan Wang, Hua Wu, and Haifeng Wang. 2019.  Plato: Pre-trained dialogue generation model with discrete latent variable.  _arXiv preprint arXiv:1910.07931_.
  * Bapna et al. (2021) Ankur Bapna, Yu-an Chung, Nan Wu, Anmol Gulati, Ye Jia, Jonathan H Clark, Melvin Johnson, Jason Riesa, Alexis Conneau, and Yu Zhang. 2021.  Slam: A unified encoder for speech and language modeling via speech-text joint pre-training.  _arXiv preprint arXiv:2110.10329_.
  * Busso et al. (2008) Carlos Busso, Murtaza Bulut, Chi-Chun Lee, Abe Kazemzadeh, Emily Mower, Samuel Kim, Jeannette N Chang, Sungbok Lee, and Shrikanth S Narayanan. 2008.  Iemocap: Interactive emotional dyadic motion capture database.  _Language resources and evaluation_ , 42(4):335–359.
  * Chen et al. (2022) Sanyuan Chen, Chengyi Wang, Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, et al. 2022.  Wavlm: Large-scale self-supervised pre-training for full stack speech processing.  _IEEE Journal of Selected Topics in Signal Processing_ , 16(6):1505–1518.
  * Chen et al. (2018) Yi-Chen Chen, Chia-Hao Shen, Sung-Feng Huang, Hung-yi Lee, and Lin-shan Lee. 2018\.  Almost-unsupervised speech recognition with close-to-zero resource based on phonetic structures learned from very small unpaired speech and text data.  _arXiv preprint arXiv:1810.12566_.
  * Chuang et al. (2020) Yung-Sung Chuang, Chi-Liang Liu, Hung-yi Lee, and Lin-shan Lee. 2020.  Speechbert: An audio-and-text jointly learned language model for end-to-end spoken question answering.
  * Chudasama et al. (2022) Vishal Chudasama, Purbayan Kar, Ashish Gudmalwar, Nirmesh Shah, Pankaj Wasnik, and Naoyuki Onoe. 2022.  M2fnet: Multi-modal fusion network for emotion recognition in conversation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 4652–4661.
  * Chung et al. (2021) Yu-An Chung, Chenguang Zhu, and Michael Zeng. 2021.  Splat: Speech-language joint pre-training for spoken language understanding.  In _NAACL_.
  * Clifton et al. (2020) Ann Clifton, Sravana Reddy, Yongze Yu, Aasish Pappu, Rezvaneh Rezapour, Hamed Bonab, Maria Eskevich, Gareth Jones, Jussi Karlgren, Ben Carterette, and Rosie Jones. 2020.  100,000 podcasts: A spoken English document corpus.  In _Proceedings of the 28th International Conference on Computational Linguistics_ , pages 5903–5917, Barcelona, Spain (Online). International Committee on Computational Linguistics.
  * Cui et al. (2019) Chen Cui, Wenjie Wang, Xuemeng Song, Minlie Huang, Xin-Shun Xu, and Liqiang Nie. 2019.  User attention-guided multimodal dialog systems.  In _Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval_ , pages 445–454.
  * Dai et al. (2022) Yinpei Dai, Wanwei He, Bowen Li, Yuchuan Wu, Zheng Cao, Zhongqi An, Jian Sun, and Yongbin Li. 2022.  Cgodial: A large-scale benchmark for chinese goal-oriented dialog evaluation.  _arXiv preprint arXiv:2211.11617_.
  * Dosovitskiy et al. (2020) Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. 2020.  An image is worth 16x16 words: Transformers for image recognition at scale.  _arXiv preprint arXiv:2010.11929_.
  * Elizalde et al. (2022) Benjamin Elizalde, Soham Deshmukh, Mahmoud Al Ismail, and Huaming Wang. 2022.  Clap: Learning audio concepts from natural language supervision.  _arXiv preprint arXiv:2206.04769_.
  * Eric et al. (2019) Mihail Eric, Rahul Goel, Shachi Paul, Abhishek Sethi, Sanchit Agarwal, Shuyang Gao, and Dilek Hakkani-Tür. 2019.  Multiwoz 2.1: Multi-domain dialogue state corrections and state tracking baselines.
  * Gao et al. (2023) Haoyu Gao, Rui Wang, Ting-En Lin, Yuchuan Wu, Min Yang, Fei Huang, and Yongbin Li. 2023.  Unsupervised dialogue topic segmentation with topic-aware utterance representation.  _arXiv preprint arXiv:2305.02747_.
  * Han et al. (2021) Wei Han, Hui Chen, Alexander Gelbukh, Amir Zadeh, Louis-philippe Morency, and Soujanya Poria. 2021.  Bi-bimodal modality fusion for correlation-controlled multimodal sentiment analysis.  In _Proceedings of the 2021 International Conference on Multimodal Interaction_ , pages 6–15.
  * He et al. (2022a) Wanwei He, Yinpei Dai, Binyuan Hui, Min Yang, Zheng Cao, Jianbo Dong, Fei Huang, Luo Si, and Yongbin Li. 2022a.  Space-2: Tree-structured semi-supervised contrastive pre-training for task-oriented dialog understanding.  _arXiv preprint arXiv:2209.06638_.
  * He et al. (2022b) Wanwei He, Yinpei Dai, Min Yang, Jian Sun, Fei Huang, Luo Si, and Yongbin Li. 2022b.  Unified dialog model pre-training for task-oriented dialog understanding and generation.  In _Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval_ , pages 187–200.
  * He et al. (2022c) Wanwei He, Yinpei Dai, Yinhe Zheng, Yuchuan Wu, Zheng Cao, Dermot Liu, Peng Jiang, Min Yang, Fei Huang, Luo Si, et al. 2022c.  Galaxy: A generative pre-trained model for task-oriented dialog with semi-supervised learning and explicit policy injection.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 36, pages 10749–10757.
  * Heck et al. (2020) Michael Heck, Carel van Niekerk, Nurul Lubis, Christian Geishauser, Hsien-Chin Lin, Marco Moresi, and Milica Gašic. 2020.  Trippy: A triple copy strategy for value independent neural dialog state tracking.  In _21th Annual Meeting of the Special Interest Group on Discourse and Dialogue_ , page 35.
  * Hendrycks and Gimpel (2016) Dan Hendrycks and Kevin Gimpel. 2016.  Gaussian error linear units (gelus).  _arXiv preprint arXiv:1606.08415_.
  * Hsu et al. (2021) Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. 2021.  Hubert: Self-supervised speech representation learning by masked prediction of hidden units.  _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , 29:3451–3460.
  * Hu et al. (2022) Guimin Hu, Ting-En Lin, Yi Zhao, Guangming Lu, Yuchuan Wu, and Yongbin Li. 2022\.  Unimse: Towards unified multimodal sentiment analysis and emotion recognition.  _arXiv preprint arXiv:2211.11256_.
  * Kang et al. (2022a) Yu Kang, Tianqiao Liu, Hang Li, Yang Hao, and Wenbiao Ding. 2022a.  Self-supervised audio-and-text pre-training with extremely low-resource parallel data.  _arXiv preprint arXiv:2204.04645_.
  * Kang et al. (2022b) Yu Kang, Tianqiao Liu, Hang Li, Yang Hao, and Wenbiao Ding. 2022b.  Self-supervised audio-and-text pre-training with extremely low-resource parallel data.  _arXiv preprint arXiv:2204.04645_.
  * Kenton and Toutanova (2019) Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova. 2019.  Bert: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of NAACL-HLT_ , pages 4171–4186.
  * Kim et al. (2021) Minjeong Kim, Gyuwan Kim, Sang-Woo Lee, and Jung-Woo Ha. 2021.  St-bert: Cross-modal language model pre-training for end-to-end spoken language understanding.  In _ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pages 7478–7482. IEEE.
  * Li et al. (2021) Hang Li, Wenbiao Ding, Yu Kang, Tianqiao Liu, Zhongqin Wu, and Zitao Liu. 2021.  Ctal: Pre-training cross-modal transformer for audio-and-language representations.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 3966–3977.
  * Liao et al. (2018) Lizi Liao, Yunshan Ma, Xiangnan He, Richang Hong, and Tat-seng Chua. 2018.  Knowledge-aware multimodal dialogue systems.  In _Proceedings of the 26th ACM international conference on Multimedia_ , pages 801–809.
  * Lin et al. (2022) Ting-En Lin, Yuchuan Wu, Fei Huang, Luo Si, Jian Sun, and Yongbin Li. 2022.  Duplex conversation: Towards human-like interaction in spoken dialogue systems.  In _Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining_ , pages 3299–3308.
  * Lin and Xu (2019) Ting-En Lin and Hua Xu. 2019.  Deep unknown intent detection with margin loss.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 5491–5496.
  * Liu et al. (2020a) Andy T. Liu, Shu wen Yang, Po-Han Chi, Po chun Hsu, and Hung yi Lee. 2020a.  Mockingjay: Unsupervised speech representation learning with deep bidirectional transformer encoders.  In _ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE.
  * Liu et al. (2020b) Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020b.  Multilingual denoising pre-training for neural machine translation.  _Transactions of the Association for Computational Linguistics_ , 8:726–742.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_.
  * Loshchilov and Hutter (2018) Ilya Loshchilov and Frank Hutter. 2018.  Fixing weight decay regularization in adam.
  * Lu et al. (2019) Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. 2019.  Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks.  _Advances in neural information processing systems_ , 32.
  * Mai et al. (2022) Sijie Mai, Ying Zeng, and Haifeng Hu. 2022.  Multimodal information bottleneck: Learning minimal sufficient unimodal and multimodal representations.  _IEEE Transactions on Multimedia_.
  * Qian et al. (2023) Yushan Qian, Bo Wang, Ting-En Lin, Yinhe Zheng, Ying Zhu, Dongming Zhao, Yuexian Hou, Yuchuan Wu, and Yongbin Li. 2023.  Empathetic response generation via emotion cause transition graph.  _arXiv preprint arXiv:2302.11787_.
  * Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021.  Learning transferable visual models from natural language supervision.  In _International Conference on Machine Learning_ , pages 8748–8763. PMLR.
  * Radford et al. (2019a) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019a.  Language models are unsupervised multitask learners.  _OpenAI blog_ , 1(8):9.
  * Radford et al. (2019b) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019b.  Language models are unsupervised multitask learners.  _OpenAI blog_ , 1(8):9.
  * Rahman et al. (2020) Wasifur Rahman, Md Kamrul Hasan, Sangwu Lee, Amir Zadeh, Chengfeng Mao, Louis-Philippe Morency, and Ehsan Hoque. 2020.  Integrating multimodal information in large pretrained transformers.  In _Proceedings of the conference. Association for Computational Linguistics. Meeting_ , volume 2020, page 2359. NIH Public Access.
  * Sachidananda et al. (2022) Vin Sachidananda, Shao-Yen Tseng, Erik Marchi, Sachin Kajarekar, and Panayiotis Georgiou. 2022.  Calm: Contrastive aligned audio-language multirate and multimodal representations.  _arXiv preprint arXiv:2202.03587_.
  * Schneider et al. (2019) Steffen Schneider, Alexei Baevski, Ronan Collobert, and Michael Auli. 2019.  wav2vec: Unsupervised pre-training for speech recognition.  _arXiv preprint arXiv:1904.05862_.
  * Sunder et al. (2022) Vishal Sunder, Samuel Thomas, Hong-Kwang J Kuo, Jatin Ganhotra, Brian Kingsbury, and Eric Fosler-Lussier. 2022.  Towards end-to-end integration of dialog history for improved spoken language understanding.  In _ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pages 7497–7501. IEEE.
  * Tang et al. (2022) Yun Tang, Hongyu Gong, Ning Dong, Changhan Wang, Wei-Ning Hsu, Jiatao Gu, Alexei Baevski, Xian Li, Abdelrahman Mohamed, Michael Auli, et al. 2022.  Unified speech-text pre-training for speech translation and recognition.  _arXiv preprint arXiv:2204.05409_.
  * Zadeh et al. (2016) Amir Zadeh, Rowan Zellers, Eli Pincus, and Louis-Philippe Morency. 2016.  Mosi: multimodal corpus of sentiment intensity and subjectivity analysis in online opinion videos.  _arXiv preprint arXiv:1606.06259_.
  * Zadeh et al. (2018) AmirAli Bagher Zadeh, Paul Pu Liang, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency. 2018.  Multimodal language analysis in the wild: Cmu-mosei dataset and interpretable dynamic fusion graph.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 2236–2246.
  * Zhang et al. (2022a) Hanlei Zhang, Hua Xu, Xin Wang, Qianrui Zhou, Shaojie Zhao, and Jiayan Teng. 2022a.  Mintrec: A new dataset for multimodal intent recognition.  In _Proceedings of the 30th ACM International Conference on Multimedia_ , pages 1688–1697.
  * Zhang et al. (2022b) Sai Zhang, Yuwei Hu, Yuchuan Wu, Jiaman Wu, Yongbin Li, Jian Sun, Caixia Yuan, and Xiaojie Wang. 2022b.  A slot is not built in one utterance: Spoken language dialogs with sub-slots.  In _Findings of the Association for Computational Linguistics: ACL 2022_ , pages 309–321, Dublin, Ireland. Association for Computational Linguistics.

Algorithm 1 Method to mask speech tokens

1:the output of temporal convolution layer
$\mathbf{f}=\text{Conv}(\mathbf{s})$ with the shape of $l\times 512$, where
$\mathbf{s}$ is the input speech waveform.

2:Masked convolutional feature output $\tilde{\mathbf{f}}$ and masked indices
$m_{\mathbf{f}}$.

3:$\tilde{\mathbf{f}}=\mathbf{s}$; $m_{\mathbf{f}}=[0,0,...,0]$ with the
length of $l$.

4:Randomly picks an integer $n$ from [20, 50] as the length of masked
continuous speech frames.

5:for $i=0$; $i<l$; $i++$ do

6: Draw a random number $r$ from $U(0,1)$;

7: if $r<0.15$ then $\triangleright$ Mask the continuous speech frames from
index $i$ to $i+n-1$

8: $m_{\mathbf{f}}[i:i+n]=1$;

9: for $j=0$; $j<n$ and $i+j<l$; $j++$ do

10: Draw a random number $t$ from $U(0,1)$;

11: if $t<0.8$ then

12: $\tilde{\mathbf{f}}[i+j]=\mathbf{0}$;

13: else if $t<0.9$ then

14: Replace $\tilde{\mathbf{f}}[i+j]$ with a random speech frame in
$\mathbf{f}$;

15: end if

16: end for

17: $i=i+n-1$

18: end if

19:end for

##  Appendix A Implementation details of the tokenizer

We describe how we convert each $I_{i}$ into input embeddings
$\mathbf{x}_{i}$. First, we split the sequence $I_{i}$ into list of tokens
using a BBPE algorithm Radford et al. (2019b) and convert each token into its
index according to the dictionary of our tokenizer. Then, we pass the token
indexes to the pre-trained token embedding layer of RoBERTa model to get the
token embedding of each token. Finally, we sum up the token embedding, the
absolute positional embedding and the segment embedding ($\mathbf{e}_{t,0}$ or
$\mathbf{e}_{t,1}$) to get the input text embedding of every token in $I_{i}$.

##  Appendix B Method to Mask Speech Tokens

We report our method to mask speech tokens in Algorithm 1. We note that masked
speech tokens are set to 0 at 80% of the time, a random token 10%, and an un-
altered 10%. In our experiments, the maximum length of speech features $f_{i}$
is 99 since the longest slice of our speech input is 10 seconds. We estimate
the expectation of the number of masked frames of our method and the original
MAM method proposed by Liu et al. (2020a) by simulating both masking steps for
1,000,000 times and calculating the average number of masked tokens.
Simulation results show that our method masks approximately 57% of tokens in
the sequence, while the original MAM method masks around 15%.


# A Nonlinear Sum of Squares Search for CAZAC Sequences

Mark Magsino
Department of Mathematics
U.S. Naval Academy
<EMAIL_ADDRESS>Yixin Xu
Department of Mathematics
Ohio State University
<EMAIL_ADDRESS>
###### Abstract

We report on a search for CAZAC sequences by using nonlinear sum of squares
optimization. Up to equivalence, we found all length 7 CAZAC sequences. We
obtained evidence suggesting there are finitely many length 10 CAZAC sequences
with a total of 3040 sequences. Last, we compute longer sequences and compare
their aperiodic autocorrelation properties to known sequences. The code and
results of this search are publicly available through GitHub.

##  I Introduction

Given $x\in\mathbb{C}^{n}$, we define the periodic ambiguity function by

$A_{p}(x)[k,\ell]:=\frac{1}{n}\sum_{j=0}^{n-1}x_{j+k}\overline{x_{j}}e^{-2\pi
ij\ell/n},$ (1)

where $0\leq k,\ell\leq n-1$, and indices are taken modulo $n$. The periodic
autocorrelation is the case with no frequency shift, i.e. when $\ell=0$. In
this case it is convenient to suppress the second input and write

$A_{p}(x)[k]:=A_{p}(x)[k,0]=\frac{1}{n}\sum_{j=0}^{n-1}x_{j+k}\overline{x_{j}}.$
(2)

A CAZAC sequence of length $n$ as a vector $x\in\mathbb{C}^{n}$ with the
properties

  * (i)

$|x_{j}|=1$ for $1\leq j\leq n$,

  * (ii)

$A_{p}(x)[k]=0$ for $1\leq k\leq n-1$.

The first property is known as the constant amplitude property and the second
is known as the zero autocorrelation property, which gives rise to the acronym
CAZAC.

These sequences have several interpretations and applications which motivate
their study. In communication theory they have been used to reduce the cross-
correlation of signals, for uplink synchronization [9], and OFDM for 5G
communication [7][15]. It is also studied as an idealized waveform with
regards to the narrowband ambiguity function in radar [14]. In general, the
constant amplitude allows one to encode information purely in terms of phase
and the zero autocorrelation ensures no interference with shifted copies of
the signal. CAZAC sequences are also known as perfect polyphase sequences and
are also well studied under that name [11][12][13].

CAZAC sequences have perfect periodic autocorrelation, but many applications
require their aperiodic autocorrelation. The aperiodic ambiguity function of a
sequence $x\in\mathbb{C}^{n}$ is given by

$A_{a}(x)[k,\ell]:=\sum_{j=1}^{n-1}x^{(a)}_{j+k}\overline{x^{(a)}_{j}}e^{-2\pi
ij\ell/n},$ (3)

where $0\leq k,\ell\leq n-1$ and $x^{(a)}_{j}$ is defined by

$x_{j}^{(a)}=\begin{cases}x_{j},&\text{ if }0\leq j\leq n-1,\\\ 0,&\text{
otherwise}.\end{cases}$ (4)

The key difference is instead of taking indices modulo $n$, we set the value
to zero when the index falls outside of the usual range. The aperiodic
autocorrelation corresponds to $\ell=0$ and it is convenient to write

$A_{a}(x)[k]:=A_{a}[k,0]=\sum_{j=1}^{n-1}x^{(a)}_{j+k}\overline{x^{(a)}_{j}}.$
(5)

In particular, we use aperiodic autocorrelation to study two properties of
interest. We define the peak sidelobe level (PSL) of $x\in\mathbb{C}^{n}$ by

$\mathrm{PSL}(x)=\frac{1}{|A_{a}(x)[k]|}\max_{k\neq 0}|A_{a}[k]|,$ (6)

and the integrated sidelobe level (ISL) by

$\mathrm{ISL}(x)=\frac{1}{|A_{a}(x)[k]|^{2}}\sum_{k=1}^{n-1}|A_{a}[k]|^{2}.$
(7)

Although CAZAC sequences are well studied, many things about them are still
unknown. Given two CAZAC sequences, $x$ and $y$, we say they are equivalent if
there exists a complex scalar $c$ with $|c|=1$ so that $y=cx$ and make the
representative of the equivalence class the sequence whose first entry is 1.
With this in mind, it is natural to ask: For each $n$, how many CAZAC
sequences of length $n$ are there? As a partial answer, it is known that if
$n$ is prime, then there are at most $\binom{2n-2}{n-1}$ CAZAC sequences [8].
If $n$ is composite and divisible by a perfect square, then there are
infinitely many sequences [6][10]. If $n$ is composite and not divisible by
any perfect square, then it is unknown how many there are. A brute force
calculation verifies that there are finitely many for $n=6$ [5]. Beyond that,
it is currently unknown.

There are additional transformations under which CAZAC sequences are closed
[3]. They are a finite set of transformations so it does not fundamentally
change the question of whether the set of CAZAC sequences of a given length is
finite. We use these to filter out known length 7 CAZAC sequences.

######  Proposition 1.

Let $x\in\mathbb{C}^{n}$ be a CAZAC sequence and let $\omega=e^{2\pi i/n}$.
Then, the following sequences are also CAZAC sequences:

  * (i)

$(T_{k}x)_{j}=x_{j+k}$, $0\leq k\leq n-1$,

  * (ii)

$(M_{\ell}x)_{j}=\omega^{\ell j}x_{j}$, $0\leq\ell\leq n-1$,

  * (iii)

$(D_{m}x)_{j}=x_{mj},\mathrm{gcd}(m,n)=1,$

  * (iv)

$(\overline{x})_{j}=\overline{x_{j}}$.

##  II CAZAC Sequneces of Length 7

The CAZAC sequences of length 7 can be split into quadratic phase sequences
and non-quadratic phase sequences. Suppose $x\in\mathbb{C}^{n}$ is defined by

$x_{j}=e^{\pi ip(j)/n},$

where $p(j)$ is a quadratic polynomial. In this case, we say that $x$ is a
quadratic phase sequence. The polynomials associated with the known quadratic
phase CAZAC sequences are

Zadoff-Chu: $\displaystyle p(j)=j(j-1),\hskip 5.0pt(n\text{ odd})$ P4:
$\displaystyle p(j)=j(j-n).$ Wiener: $\displaystyle p(j)=2kj^{2},\hskip
5.0pt(\mathrm{gcd}(k,n)=1,n\text{ odd}),$ $\displaystyle p(j)=kj^{2},\hskip
5.0pt(\mathrm{gcd}(k,2n)=1,n\text{ even}).$

When $n$ is prime, there are at least $n(n-1)$ CAZAC sequences comprised of
roots of unity, including the quadratic phase sequences [2]. When $n=7$, this
gives at least 42 roots of unity sequences. Moreover, the transformations
described in Proposition 1 will keep the sequence a root of unity sequence. On
the other hand, in [4] Björck constructed CAZAC sequences comprised of non
roots of unity for each prime $p>5$. The construction is as follows.

Given an odd prime $p$, let $\left(\frac{j}{p}\right)$ denote the Legendre
symbol defined by

$\left(\frac{j}{p}\right)=\begin{cases}0,&\\!\\!\\!\\!\text{ if }j\equiv 0\mod
p,\\\ 1,&\\!\\!\\!\\!\text{ if }j\equiv x^{2}\\!\\!\\!\\!\\!\mod p,\text{ for
some }x\neq 0,\\\ -1,&\\!\\!\\!\\!\text{ if }j\not\equiv x^{2}\mod p,\text{
for any }x\neq 0.\end{cases}$

We define the Björck sequence of length $p$ by

$x_{j}=e^{i\theta(j)},\hskip 5.0pt0\leq j\leq p-1,$ (8)

where if $p\equiv 1\mod 4$, then $\theta(j)$ is given by

$\left(\frac{j}{p}\right)\mathrm{arccos}\left(\frac{1}{1+\sqrt{p}}\right),$
(9)

and if $p\equiv 3\mod 4$, then $\theta(j)$ is given by

$\theta(j)=\begin{cases}\mathrm{arccos}\left(\frac{1-p}{1+p}\right)&\text{ if
}\left(\frac{j}{p}\right)=-1,\\\ 0,&\text{ otherwise}.\end{cases}$ (10)

Since $7\equiv 3\mod 4$, the Björck sequence of length 7 is

$x=(1,1,1,e^{i\theta_{7}},1,e^{i\theta_{7}},e^{i\theta_{7}}),$ (11)

where $\theta_{7}=\mathrm{arccos}(3/4)$. Since CAZAC sequences are closed
under the operations outlined in Proposition 1 the Björck sequence generates
up to 252 CAZAC sequences which begin with 1 when p = 7. Some combinations of
transformations result in the same sequence and thus this is an over count,
which our computations reflect. Regardless, a total of at most 294 of the 532
sequences are accounted for, leaving many new sequences of length 7 to be
found.

##  III CAZACs as an Algebraic Variety

The conditions which define a CAZAC sequence can be viewed as a system of
$2n-1$ equations with $n$ variables. A natural question is to ask about the
dimension of the set of solutions to this system. If the set of solutions is
zero dimensional, then it is a discrete set and may be a finite set.
Conversely, if the set of solutions has positive dimension, then there are
infinitely many of them. This question puts the problem in the realm of
algebraic geometry.

The techniques of algebraic geometry only work with systems of polynomials.
However, the conjugations in the sums prevent those conditions from being
interpreted as polynomials. We can get around this by expressing entries in
real and imaginary parts and converting the equations to polynomials of real
variables. Given a vector $X\in\mathbb{C}^{n}$ and expressing its entries as
$x_{j}=a_{j}+ib_{j}$, CAZAC sequences arise as solutions to the system

$\displaystyle a_{j}^{2}+b_{j}^{2}=1,$ $\displaystyle\hskip 10.00002pt0\leq
j\leq n-1,$ (12) $\displaystyle\sum_{j=0}^{n-1}a_{j+k}a_{j}+b_{j+k}b_{j}=0,$
$\displaystyle\hskip 10.00002pt1\leq k\leq n-1,$ (13)
$\displaystyle\sum_{j=0}^{n-1}a_{j}b_{j+k}-b_{j}a_{j+k}=0,$
$\displaystyle\hskip 10.00002pt1\leq k\leq n-1.$ (14)

Since constraining each amplitude has $n$ conditions, and each of the $n-1$
autocorrelation conditions has been split into real and imaginary parts, there
are $3n-2$ equations in this system. Since each entry of the sequence was
split into a real and imaginary part, the system has $2n$ variables. Hence,
the system is a real algebraic variety with $3n-2$ equations and $2n$
variables.

##  IV Nonlinear Least Squares Search

The system of equations (12) - (14) can be converted into an unconstrained
nonlinear sum of squares optimization problem. Let $a=(a_{1},\cdots,a_{n})$
and $b=(b_{1},\cdots,b_{n})$, and for each $1\leq k\leq n-1$ and
$0\leq\ell\leq n-1$, define the functions

$\displaystyle f_{\ell}(a,b)$ $\displaystyle=a_{\ell}^{2}+b_{\ell}^{2}-1,$
(15) $\displaystyle g_{k}(a,b)$
$\displaystyle=\sum_{j=0}^{n-1}a_{j+k}a_{j}+b_{j+k}b_{j},$ (16) $\displaystyle
h_{k}(a,b)$ $\displaystyle=\sum_{j=0}^{n-1}a_{j}b_{j+k}-b_{j}a_{j+k}.$ (17)

Note that $(a,b)$ are the real and imaginary parts, respectively, of a CAZAC
sequence precisely when all $3n-2$ functions are all zero at $(a,b)$. We can
obtain CAZAC sequences as solutions to the unconstrained optimization problem

$\operatorname*{arg\,min}_{(a,b)}\sum_{\ell=0}^{n-1}f_{\ell}^{2}+\sum_{k=1}^{n-1}g_{k}^{2}+h_{k}^{2},$
(18)

for which the objective function is zero.

Since the equations are nonlinear polynomials, optimization procedures are not
guaranteed to work. Specifically, there may be local minima where the
objective function is greater than zero. Moreover, the objective function is
not convex which makes it difficult to give theoretical guarantees about this
method. Nonetheless, in our experiments we found very few such local minima
and generated as many sequences as needed for each test.

##  V Method

The experimental design was implemented in a Jupyter Notebook. All trials were
run on a 2020 MacBook Pro with the 10-core M1 Pro CPU and 16 GB of physical
memory. The optimization problem in (18) was implemented using the least
squares solver in SciPy’s Optimization package. By default the solver uses the
Trust Region Reflective (TRR) algorithm and a finite difference operation to
estimate the Jacobian at each point. The code is available through GitHub at:
https://github.com/magsino-usna/IEEE-SoS-CAZAC.git.

###  V-A Length 7 CAZACS

We perform the optimization on 10,000 random initial starting points in
$[-1,1]^{14}$. For increased precision, we used stronger tolerances of
$10^{-12}$ instead of the default $10^{-8}$ for changes in the gradient, input
variables, and cost function. If the resulting cost function was lower than
$10^{-10}$, we considered the point found as a CAZAC sequence stored it as a
new row in an array.

After generating the list of sequences, we rounded each one to 8 decimal
places and pass the rounded list through NumPy’s algorithm for determining
unique rows in arrays to determine the number of sequences found. After that,
we used the transformations listed in Proposition 1 using the Björck and the
Wiener sequences as base sequences to filter out all previously known CAZAC
sequences.

###  V-B Length 10 CAZACs

Since 10 is composite and not divisible by a perfect square, it is not known
if there are finitely many of them. Our idea for exploring whether the set of
solutions is finite is based on the following intuition.

Solving the sum of squares minimization with an initial point is essentially
projecting the point onto the set of solutions. If the set of solutions is
greater than zero dimensional, most of the initialization points will project
to different points on the continuous set of solutions. Conversely, if the set
of solutions is zero dimensional and finite, then most of the points should
project to that finite set of points. Thus, if the number of unique sequences
found is far smaller than the number of trials, the set is likely finite. This
idea is illustrated in Fig. 1.

Figure 1: Experiments taking 1000 initial points and using nonlinear sum-of-
squares to find points onto two sets. Left: The points traced out the one
dimensional curve $x^{3}-x-y=0$. Right: The points went to the 8 points of the
intersection of $x^{2}+y^{2}-1=0$ and $8x^{4}-10x^{2}-x+2=0$.

For the experiment, we performed the optimization on 200,000 random initial
starting points in $[-1,1]^{20}$ and use the same tolerances and rounding
method as in the length 7 case.

###  V-C Aperiodic Autocorrelations of Longer CAZACs

Using the nonlinear sum of squares optimization method, we computed 1000 CAZAC
sequences of prime lengths $n=11,13,17,23,29,37,43,47$. We then compared the
PSL and ISL of the CAZAC sequences to the PSL and ISL of the corresponding
Zadoff-Chu and Björck sequences. We compare the results using a box plot to
give a sense of how generic CAZAC sequences behave with respect to PSL and ISL
behavior.

##  VI Results

As previously mentioned, all results and code are stored on GitHub: The full
list is available on GitHub: https://github.com/magsino-usna/IEEE-SoS-
CAZAC.git.

###  VI-A Length 7 Sequences

We were able to enumerate all 532 CAZAC, handling equivalence by dividing and
making the first entry 1. This took 15 min to run all 100,000 attempts on the
aforementioned MacBook Pro. The maximum final value of the objective function
across all trials was on the order of $10^{-17}$ which implies the objective
function has no spurious local minima in the length 7 case. We picked out the
new CAZAC with the best PSL and compared its autocorrelation to the Zadoff-Chu
and Björck sequences in Fig. 2.

Figure 2: Autocorrelation of the best performing length 7 CAZAC sequence
plotted against the autocorrelations of the length 7 Zadoff-Chu and Björck
sequences. The new CAZAC sequence achieves a lower PSL than the Zadoff-Chu
sequence.

###  VI-B Length 10 Sequences

We ran three sets of the trials described in Section V-B. In all three trials,
the same 3040 CAZAC sequences were found. This heavily implies that there are
3040 CAZAC sequences of length 10, although it is still possible that points
corresponding to a highly unstable minimum could be missed. However, the
average runtime for each set of these trials was about 13 hours. The maximum
cost function value across all trials was to the order of $10^{-24}$, which
implies once again that every local minimum of the objective function is the
global minimum of 0. All 3040 sequences found are also available through
GitHub as well.

###  VI-C Aperodic Correlations

We were successfully able to find 1,000 CAZAC sequences of each length $n$ =
11,13,17,23,29,37,43,47. For these larger cases a few spurious local minima
were found in initial tests requiring more minimizations than the number of
desired sequences, slowing down the computation. We compare the PSL and ISL
properties of the numerical CAZAC sequences found with the box plot in Fig. 3.

Figure 3:  Top: The PSLs of each numerically found CAZAC are compared to the
Zadoff-Chu and Björck sequences by box plot. Bottom: The ISLs of each
numerically found CAZAC are compared to the Zadoff-Chu and Björck sequences by
box plot. Notably, it seems possible to get better PSL and ISL than the
Zadoff-Chu sequence in some cases.

To further illustrate the properties of CAZAC sequences generated by these
methods, we juxtapose the (modulus) of the ambiguity function of a length 43
numerically found CAZAC sequence and the typically used used Zadoff-Chu
sequence. This is depicted in Fig. 4.

Figure 4: Top: Ambiguity function plot for a computed length 43 CAZAC
sequence. Bottom: Ambiguity function plot for the length 43 Zadoff-Chu
sequence. Although not as flat as the Zadoff-Chu sequence, the energy is more
well spread in the computed sequence so no values exceed 0.3 times the
ambiguity function at $(k,\ell)=(0,0)$.

##  VII Future Directions

This work provides a framework for computing CAZAC sequences by using
nonlinear sum of squares optimization. In addition to the potential problem of
spurious local minima, scalability is an issue as well. It is possible that
other optimization methods or equation solvers would lead to improved
efficiency. The code has been made available through GitHub to help ensure
replicability of the numerical experiments and to provide a base of code for
future directions of work.

It is also likely that the system of equations defining CAZAC sequences has
some symmetries or redundancies. Exploring these could make it possible to
reduce the number of equations and find new sequences more efficiently.

Another alternative is to convert the aperiodic autocorrelations into a system
of polynomials of real variables. We could then use the resulting function to
create objective functions that act as a proxy for PSL or ISL and run an
optimization search. This could also give us insight on what the optimal PSL
and ISL values are for phase-coded waveforms.

Since this work involves an algebraic variety, it is useful to consider
techniques from numerical algebraic geometry. For example, homotopy
continuation [1] is a framework for analyzing isolated solutions of algebraic
varieties. This could be useful in determining whether there are finitely many
CAZAC sequences in currently unknown cases.

## Acknowledgements

The views expressed in the paper are those of the first author and do not
reflect the official policy or position of the Department of the Navy,
Department of Defense, or the U.S. Government.

## References

  * [1] JC Alexander and James A Yorke.  The homotopy continuation method: numerically implementable topological procedures.  Transactions of the American Mathematical Society, 242:271–284, 1978.
  * [2] John J Benedetto, Katherine Cordwell, and Mark Magsino.  Cazac sequences and haagerup’s characterization of cyclic n-roots.  In New Trends in Applied Harmonic Analysis, Volume 2, pages 1–43. Springer, 2019.
  * [3] John J Benedetto and Jeffrey J Donatelli.  Ambiguity function and frame-theoretic properties of periodic zero-autocorrelation waveforms.  IEEE Journal of Selected Topics in Signal Processing, 1(1):6–20, 2007.
  * [4] Göran Björck.  Functions of modulus 1 on Zn whose fourier transforms have constant modulus, and “cyclic n-roots”.  In Recent Advances in Fourier Analysis and its Applications, pages 131–140. Springer, 1990.
  * [5] Göran Björck and Ralf Fröberg.  A faster way to count the solutions of inhomogeneous systems of algebraic equations, with applications to cyclic n-roots.  Journal of Symbolic Computation, 12(3):329–336, 1991.
  * [6] Göran Björck and Bahman Saffari.  New classes of finite unimodular sequences with unimodular fourier transforms. circulant hadamard matrices with complex entries.  Comptes rendus de l’Académie des sciences. Série 1, Mathématique, 320(3):319–324, 1995.
  * [7] Zhenhua Feng, Ming Tang, Songnian Fu, Lei Deng, Qiong Wu, Rui Lin, Ruoxu Wang, Ping Shum, and Deming Liu.  Performance-enhanced direct detection optical OFDM transmission with CAZAC equalization.  IEEE Photonics Technology Letters, 27(14):1507–1510, 2015.
  * [8] Uffe Haagerup.  Cyclic p-roots of prime lengths p and related complex hadamard matrices.  arXiv preprint arXiv:0803.2629, 2008.
  * [9] Mohammad M Mansour.  Optimized architecture for computing zadoff-chu sequences with application to lte.  In GLOBECOM 2009-2009 IEEE Global Telecommunications Conference, pages 1–6. IEEE, 2009.
  * [10] Andrzej Milewski.  Periodic sequences with optimal properties for channel estimation and fast start-up equalization.  IBM Journal of Research and Development, 27(5):426–431, 1983.
  * [11] Wai Ho Mow.  A unified construction of perfect polyphase sequences.  In Proceedings of 1995 IEEE International Symposium on Information Theory, page 459. IEEE, 1995.
  * [12] Wai Ho Mow and S-YR Li.  Aperiodic autocorrelation properties of perfect polyphase sequences.  In [Proceedings] Singapore ICCS/ISITA92, pages 1232–1234. IEEE, 1992.
  * [13] Ki-Hyeon Park, Hong-Yeop Song, Dae San Kim, and Solomon W Golomb.  Optimal families of perfect polyphase sequences from the array structure of fermat-quotient sequences.  IEEE Transactions on Information Theory, 62(2):1076–1086, 2015\.
  * [14] DE Vakman.  The ambiguity function in the statistical theory of radar.  In Sophisticated Signals and the Uncertainty Principle in Radar, pages 98–125. Springer, 1968.
  * [15] Krzysztof Wesolowski, Adrian Langowski, and Krzysztof Bakowski.  A novel pilot scheme for 5g downlink transmission.  In 2015 International Symposium on Wireless Communication Systems (ISWCS), pages 161–165. IEEE, 2015.