††thanks: These authors have contributed equally to this work.††thanks: These
authors have contributed equally to this work.

# Noise Spectroscopy Without Dynamical Decoupling Pulses

Arian Vezvaee  Department of Chemistry, University of Colorado Boulder,
Colorado 80309, USA  Nanako Shitara  Department of Chemistry, University of
Colorado Boulder, Colorado 80309, USA  Department of Physics, University of
Colorado Boulder, Colorado 80309, USA  Shuo Sun  Department of Physics,
University of Colorado Boulder, Colorado 80309, USA  JILA, University of
Colorado Boulder, Colorado 80309, USA  Andrés Montoya-Castillo
<EMAIL_ADDRESS>Department of Chemistry, University of
Colorado Boulder, Colorado 80309, USA

###### Abstract

Spectral characterization of noise environments that lead to the decoherence
of qubits is critical to developing robust quantum technologies. While
dynamical decoupling offers one of the most successful approaches to
characterize noise spectra, it necessitates applying large sequences of $\pi$
pulses that increase the complexity and cost of the method. Here, we introduce
a noise spectroscopy method that utilizes only the Fourier transform of free
induction decay measurements, thus removing the need for the application any
$\pi$ pulses. We show that our method faithfully recovers the correct noise
spectra and outperforms previous dynamical decoupling schemes while
significantly reducing its experimental overhead. We also discuss the
experimental feasibility of our proposal and demonstrate its robustness in the
presence of statistical measurement noise. Our method is applicable to a wide
range of quantum platforms and provides a simpler path toward a more accurate
spectral characterization of quantum devices, thus offering possibilities for
tailored decoherence mitigation.

Nearly all current quantum technology applications rely on a two-level quantum
system (qubit) that is subject to environmental noise. In the pure dephasing
limit this environmental noise causes fluctuations in the frequency of the
qubit that lead to decoherence. Spectral characterization of such environments
is the most crucial step in successfully controlling and suppressing
decoherence. Indeed, characterizing the noise spectrum allows for a filter-
design approach that suppresses the noise and improves the coherence of the
qubit Cywiński  _et al._ (2008); Biercuk  _et al._ (2011); Uhrig (2007);
Biercuk  _et al._ (2009). Therefore, developing methods that can recover the
noise spectrum of qubit environments has been one of the most active fields of
research over the past two decades Yang  _et al._ (2010); Szańkowski  _et al._
(2017); Degen  _et al._ (2017); Suter and Álvarez (2016). Among these efforts,
dynamical decoupling noise spectroscopy (DDNS) Álvarez and Suter (2011); Yuge
_et al._ (2011); Norris  _et al._ (2016); Krzywda  _et al._ (2019) has been
one of the most successful approaches. In this method, applying a sequence of
$\pi$-pulses turns the qubit into a noise probe (approximated as a frequency
comb) that isolates contributions from particular frequencies of the noise
spectrum. The DD framework has been studied extensively theoretically and
implemented experimentally in various platforms such as superconducting
circuits Bylander  _et al._ (2011); Sung  _et al._ (2019), ultracold atoms
Almog  _et al._ (2011), quantum dots Dial  _et al._ (2013); Farfurnik  _et
al._ (2021); Connors  _et al._ (2022), and nitrogen-vacancy (NV) centers in
diamonds Bar-Gill  _et al._ (2012); Romach  _et al._ (2015). A DDNS protocol
based on the Carr-Purcell-Meiboom-Gill (CPMG) sequence Carr and Purcell
(1954); Meiboom and Gill (1958) was proposed by Álvarez and Suter (AS) Álvarez
and Suter (2011) which would ideally yield a system of equations and unknowns
from the measured values of the qubit coherence
$C(t)=|\left\langle\rho_{01}(t)\right\rangle|/|\left\langle\rho_{01}(0)\right\rangle|$,
and specific frequencies of the spectrum. However, this method offers
reasonable performance only when the number of $\pi$-pulses in each sequence
is large. Beyond a pulse economy standpoint, other difficulties, such as
deviations from the ideal frequency comb approximation Szańkowski and Cywiński
(2018), have recently inspired the development of a deep learning approach
Wise  _et al._ (2021) to reconstruct the noise spectrum from the coherence
function of the qubit. The success of this deep learning method suggests the
existence of a one-to-one mapping between the two quantities.

Here, we present a simple and inexpensive method that uniquely maps the
measured coherence function of a qubit to its noise power spectrum, removing
the need for long sequences of $\pi$-pulses at the heart of DDNS. Our approach
only requires Free Induction Decay (FID) measurements of the qubit and employs
a simple Fourier transform to accurately reconstruct the noise spectrum of the
system. While Fourier spectroscopy has been implemented in Nuclear Magnetic
Resonance and on different types of quantum processors Degen  _et al._ (2017);
Yan  _et al._ (2012); Boss  _et al._ (2016), it has not been utilized in the
context of pure dephasing with the filter function formalism. Here, we combine
the Fourier transform technique with the filter function formalism to
introduce an approach we call Fourier transform noise spectroscopy (FTNS) that
significantly enhances one’s ability to reconstruct the power spectrum while
dramatically reducing the required experimental overhead.

We begin by laying out the theoretical basis for the filter function formalism
in a pure dephasing setup Cywiński  _et al._ (2008); Yuge  _et al._ (2011);
Szańkowski  _et al._ (2017); Gu and Franco (2019). In this setup, the qubit
relaxation process (quantified by $T_{1}$) takes much longer than phase
randomization (quantified by $T_{2}^{\star}$), implying that the decoherence
time $T_{2}^{-1}=(2T_{1})^{-1}+T_{2}^{\star{-1}}\approx T_{2}^{\star{-1}}$
becomes a measure of how fast the phase information is lost due to
environmental fluctuations. Frequency fluctuations of a qubit subject to a
stationary, Gaussian noise, $\beta(t)$, can be described by the Hamiltonian
$\hat{H}=\frac{1}{2}[\Omega+\beta(t)]\hat{\sigma}_{z}$, where $\Omega$ is the
natural frequency of the qubit. Here, the coherence function is
$C(t)=e^{-\chi(t)}$, where the attenuation function $\chi(t)$ is given by the
overlap of the noise spectrum and a filter function that incorporates the
effect of the pulses on the system 111This form of filter-function formalism
is the same for both classical and quantum noise models Yuge  _et al._
(2011).:

$\chi(t)=-\text{ln}[C(t)]=\frac{1}{4\pi}\int_{-\infty}^{\infty}d\omega\leavevmode\nobreak\
S(\omega)F(\omega t).$ (1)

The noise spectrum, $S(\omega)=\int_{-\infty}^{\infty}dt\leavevmode\nobreak\
e^{i\omega t}S(t)$, is the Fourier transform of the equilibrium time
correlation function of the environmental noise,
$S(t)=\langle\\{\beta(t),\beta(0)\\}\rangle/2$, where $\\{A,B\\}=AB+BA$ is the
anticommutator. The filter function, $F(\omega t)$, encodes the sign switching
($\pm 1$) of the environmental fluctuations upon application of each $\pi$
pulse in the sequence Cywiński  _et al._ (2008).

To demonstrate the advantages of our proposed FTNS, we first consider what is
arguably the state-of-the-art approach to noise spectroscopy: the AS protocol.
The main insight of the AS method lies in noting that when the number of
pulses is sufficiently large, the filter function reaches the spectroscopic
limit. In this limit, one can approximate the filter function by a
$\delta$-function (frequency comb) with various harmonics: $\chi(t)\approx
t\sum_{0<k<k_{c}}\left|A_{k\omega_{0}}\right|^{2}S\left(k\omega_{0}\right)$,
where $A_{k\omega_{0}}$ are the Fourier coefficients for a given pulse
sequence, truncated at $k_{c}$. However, while applying many $\pi$-pulses
allows the central peak to better resemble a $\delta$-function, it comes at
the cost of increasing the number of harmonics in the frequency interval of
interest. The extreme case of $k_{c}=1$ approximates the filter function as a
single $\delta$-function, discarding many details of the noise spectrum.
Often, one can still account for a limited number of harmonics (set by the
cut-off $k_{c}$), which attenuates the loss of spectral information Szańkowski
_et al._ (2017); Wise  _et al._ (2021). In the latter case, by appropriately
varying the delay time between pulses and the total time of the sequence, one
can form a linear system of equations consisting of coherence values at
selected times and a matrix of contributing Fourier coefficients. Inverting
this system of equations yields the noise spectrum at the probed frequencies,
which are bounded by $|\omega^{\rm DDNS}_{\text{max}}|\leq\pi/\tau$. Here,
$\tau$ is the minimum delay between consecutive $\pi$-pulses required to
minimize the overlap between subsequent pulses and validate the instantaneous
pulse assumption. Furthermore, since $A_{(k=0)}=0$ for balanced pulse
sequences like CPMG, the zero-frequency part of the spectrum cannot be
accessed directly. Thus, going beyond the $|\omega^{\rm
DDNS}_{\text{max}}|\leq\pi/\tau$ limit and reading $S(\omega=0)$ requires
imbalanced sequences such as Concatenated DD (CDD) Norris  _et al._ (2016).
Hence, the experimental overhead, frequency restrictions, and accuracy
dependence on harmonic inclusions of AS Szańkowski and Cywiński (2018)
motivate the development of a more accessible scheme.

We introduce a radically more straightforward approach by inverting Eq. (1)
directly to obtain the noise power spectrum, without the need for any
$\pi$-pulses. Noting that for FID, $F(\omega t)=(4/\omega^{2})\sin^{2}(\omega
t/2)$ Cywiński  _et al._ (2008), substituting $F(\omega t)$ in Eq. (1), and
differentiating both sides twice with respect to time, we obtain

$\ddot{\chi}(t)=\frac{1}{2\pi}\int_{-\infty}^{\infty}d\omega
S(\omega)\cos(\omega t).$ (2)

Since $S(-\omega)=S(\omega)$, we Fourier transform both sides to find

$S(\omega)=\sqrt{2\pi}\leavevmode\nobreak\
\mathcal{F}\big{[}\ddot{\chi}(t)\big{]}.$ (3)

This straightforward derivation demonstrates that there is a simple and
invertible one-to-one mapping between the noise power spectrum $S(\omega)$ and
the experimentally measured coherence function.

To illustrate this method for an analytically solvable case, we adopt a
Gaussian-shaped noise power spectrum $S(\omega)=Ae^{-(\omega/\sigma)^{2}}$.
The coherence function of this noise profile can be obtained analytically:

$C(t)=\exp\left\\{-\frac{A}{\sigma}\Bigg{[}\frac{t\sigma}{2}\text{Erf}\left(\frac{t\sigma}{2}\right)+\frac{e^{-\frac{t^{2}\sigma^{2}}{4}}-1}{\sqrt{\pi}}\Bigg{]}\right\\},$
(4)

where $\text{Erf}(z)=2\pi^{-1/2}\int_{0}^{z}e^{-x^{2}}\leavevmode\nobreak\ dx$
is the Error function. The second derivative of the attenuation function takes
the expected Gaussian form,
$\ddot{\chi}(t)=\frac{A}{\sqrt{2\pi}}e^{-\frac{t^{2}\sigma^{2}}{4}}$, as does
its Fourier transform,
$\mathcal{F}\Big{[}\frac{A}{\sqrt{2\pi}}e^{-\frac{t^{2}\sigma^{2}}{4}}\Big{]}=\frac{A}{\sqrt{2\pi}}e^{-(\omega/\sigma)^{2}}$,
suggesting $S_{\text{rec}}(\omega)=Ae^{-(\omega/\sigma)^{2}}$. This
illustrates that the FTNS method retrieves the original noise spectrum.

Translating the above insights into a NS procedure is straightforward. First,
one measures the coherence function $C(t)$ from FID by performing Ramsey
measurements at various times, yielding an array of coherence values in
$[0,T_{\text{max}}]$ with a sampling interval, or resolution, $\delta t$. One
then takes a logarithm of the data and numerically performs a double
derivative on the sampled $\chi(t)$ values. A Fourier transform of the
resulting data yields the noise spectrum $S(\omega)$. For this, one can employ
a Discrete Fourier transform (DFT) or numerical quadrature to obtain
equivalent results.

Figure 1: (a) A Lorentzian spectrum and its reconstruction using 2-pulse AS,
the $\delta$-function approximation ($k_{c}=1$), and FTNS. One hundred
frequencies have been reconstructed for the AS method, but only selected
points have been marked for clarity. In the FTNS results, only frequencies
corresponding to the marked AS ones are shown. (b) The absolute error compared
to the true spectrum. While DDNS requires many pulses to achieve comparable
accuracy, FTNS outperforms DDNS through using only FID measurements.

To illustrate the power of the FTNS approach, we assess its ability to
reconstruct single- and double-Lorentzian spectra that are relevant to bulk
Bar-Gill  _et al._ (2012) and near-surface Romach  _et al._ (2015) NV centers,
respectively. Figure 1(a) shows a single Lorentzian peak (grey line) and its
spectrum reconstruction using 2-pulse AS (red circles), the $\delta$-function
approximation of 2-pulse AS (green crosses), and FTNS (blue line and squares),
respectively SM . The absolute error of the reconstructed spectrum compared to
the true spectrum is computed as
$\Delta(\omega)=|S(\omega)-S_{\text{rec}}(\omega)|$. As the figure
demonstrates, FTNS outperforms the 2-pulse AS method while only requiring FID
measurements.

Noting the difficulties of the DDNS approach in accessing the noise spectrum
in both $|\omega^{\rm DDNS}_{\rm max}|\leq\pi/\tau$ and $S(\omega\rightarrow
0)$ limits, it is worth considering if and how similar limitations hinder the
FTNS approach. Since our protocol for FTNS relies on the DFT, two quantities
determine the highest accessible frequency ($\omega^{\rm FTNS}_{\rm max}$) and
its spectral resolution ($\delta\omega^{\rm FTNS}$), which also determines the
lowest accessible frequency ($\omega^{\rm FTNS}_{\rm min}$): the sampling
interval, $\delta t$, of the coherence function measurement, and the total
measurement time, $T_{\text{max}}$. Specifically, $|\omega^{\rm FTNS}_{\rm
max}|=\pi/\delta t$ and $|\omega^{\rm FTNS}_{\rm
min}|=\delta\omega=1/T_{\text{max}}$. While $\delta t$ is determined by
limitations of state-of-the-art measuring devices, $T_{\text{max}}$ depends on
the physical problem. Yet, for many cases of physical interest,
$\ddot{\chi}(t)\rightarrow 0$ at times earlier than $T_{\text{max}}$ SM ,
allowing one to zero-pad $\ddot{\chi}(t\geq T_{\text{max}})$ to a new
effective $\tilde{T}_{\text{max}}\gg T_{\text{max}}$, offering sufficient
spectral resolution to access $S(\omega\rightarrow 0)$.

Given the importance of $\delta t$ in allowing FTNS to access high frequencies
and the analogous role that the minimum delay time, $\tau$, plays in DDNS, we
now consider their connection in greater detail. $\tau$ determines the
earliest time (after $t=0$) where one can measure the coherence function,
i.e., $C(\tau)$. Since the DFT requires measurements of $C(t)$ at regular
intervals, one might be tempted to assume that $\delta t=\tau$. However, this
need not be the case. After all, for $t\geq\tau$, the measurement interval
$\delta t$ is not determined by $\tau$ and can be set such that $\delta
t\ll\tau$. While the resolution of the coherence function measurements at
later times can be made as fine-grained as desired, one still needs to perform
measurements in $[0,\tau]$ to achieve a consistent $\delta t$ through
$[0,T_{\text{max}}]$. To achieve this, we suggest employing the limit $\omega
t\ll 1$, which reveals that the attenuation function behaves as
$\chi(t)\approx\alpha t^{2}+\beta t^{4}+\gamma t^{6}$, to fit $C(t)$ at early
times SM . This guarantees that FTNS can be implemented even when constrained
to the same minimum delay time of a DD pulse sequence.

We are now in a position to illustrate the ability of FTNS to capture spectra
composed of a sum of Gaussians (Fig. 2(a)) and a double-Lorentzian (Fig.
2(b)), each compared to a reconstruction using 2-pulse AS subject to the same
$\tau$ constraint. In both panels, the reconstructed spectrum from 2-pulse AS
(red circles) fails to capture the detailed structure within $|\omega^{\rm
DDNS}_{\rm max}|\leq\pi/\tau$ due to the limited number of DD pulses and
cannot access $|\omega|>\omega^{\rm DDNS}_{\rm max}$. To go beyond these
limits in DDNS, one can employ complex CDD sequences. In contrast, FTNS only
encounters a difficulty in resolving the feature at $\omega\sim\pm 3\
\rm{[1/T}_{2}\rm{]}$ in Fig. 2(a) because we restrict $T_{\rm{max}}$ such that
$C(t<T_{\rm{max}})\geq 0.005$ SM . Thus, FTNS uses a simple FID measurement
that successfully reconstructs the spectrum in the bounded frequency range and
in higher frequency ranges, giving access to information that would be
otherwise lost.

Figure 2: Examples of two structured noise spectra, reconstructed with FTNS
(blue line) and a 2-pulse AS method (red circles): (a) A combination of four
Gaussians and (b) a double Lorentzian. The red vertical lines indicate the
frequency limits up to which the AS method can reconstruct the spectrum for
the given minimum delay time: $|\omega^{\rm DDNS}_{\text{max}}|\leq\pi/\tau$.
Our FID-based FTNS reconstructs the noise spectrum to much higher accuracy
both within, and beyond this frequency region.

Since smaller $\delta t$ gives access to higher $\omega^{\rm FTNS}_{\rm max}$
but raises the cost of the experimental procedure, we turn to the trade-off in
FTNS accuracy and the sampling interval $\delta t$. Figure 3 depicts FTNS
spectrum reconstructions using sets of coherence measurements for a fixed
measurement time $T_{\text{max}}$ with varying $\delta t$. Clearly, increasing
the resolution of coherence measurements (i.e., decreasing $\delta t$)
improves the accuracy of the FTNS reconstruction, especially at higher
frequencies. As expected, even low sampling rates accurately reconstruct the
low-frequency part of the spectrum while the high-frequency part can be
systematically improved with finer $\delta t$. The ability of FTNS to capture
the low-frequency component even at low sampling rates is particularly
advantageous for decoherence mitigation purposes, as the low frequency noise
often dominates decoherence Makhlin and Shnirman (2003); Schriefl  _et al._
(2006) 222Equation 1, subject to FID, suggests that a spectrum with a dominant
low frequency component has a faster decoherence time. This is because the
power spectrum is multiplied by $\omega^{2}F(\omega t)=\sin^{2}(\omega t/2)$,
which is sharply peaked at $\omega=0$.. Thus, even when measurement resolution
is limited, FTNS can be expected to perform well at low frequencies and one
can systematically assess its accuracy by checking convergence with finer
$\delta t$.

Figure 3: FTNS reconstructions of the multi-Gaussian spectrum in Fig. 2(a) for
different values of the sampling interval, $\delta t$, for a fixed
$T_{\text{max}}/T_{2}=10$. The legend shows the values of $\delta t/T_{2}$
used to obtain the FTNS results shown.

Since FTNS requires performing two numerical derivatives, it is sensitive to
measurement noise. There are multiple sources of noise that can compromise the
measured value of the coherence function at a given time. These include
background and shot noises, and imperfect fidelity of the applied pulses
Tyryshkin  _et al._ (2010); Souza  _et al._ (2011). In optical setups, photon
losses can also reduce the number of effective measurements. Nevertheless,
various methods to perform controlled numerical derivatives of noisy data are
available Chartrand (2011); Van Breugel  _et al._ (2020). As an example, here
we utilize a simple denoising method that mitigates effect of noise and
preserves all the advantages of FTNS even on structured noise spectra. Figure
4 shows examples of FTNS spectra reconstructed from artificially noisy data
corresponding to an effective measurement error of $0.1\%$. We perform linear
fitting of $\chi(t)$ at late times and apply low-pass filters to recover the
approximate noise spectrum, which shows good agreement with the actual
spectrum, and reveals the essential features in the spectrum SM . Therefore
our simple FID-based NS approach can semiquantitatively recover the frequency
and the height of the peaks of the noise spectrum, which constitute the
minimum required spectral information to design effective filter functions to
mitigate decoherence from a DD perspective.

While our analysis above accounts for theoretical constraints of the DFT, the
experimental feasibility of FTNS is sensitive to $\delta t$ and the minimum
delay time $\tau$. Controlling these parameters requires flexibility in pulse
design, which varies depending on the platform. For instance, solid-state
spins can be controlled either optically with pulses that range from a few
picoseconds up to 1 ns Press  _et al._ (2010); Vezvaee  _et al._ (2022); Bodey
_et al._ (2019); Takou and Economou (2021) or via a microwave with a pulse as
short as 12 ns Nguyen  _et al._ (2019). Such pulses allow for ns-scale minimum
delay time $\tau$ between pulses. Furthermore, tuning the sampling interval
$\delta t$ to ps-order precision is also achievable 333Two microwave pulses
with a controllable delay can be generated with an arbitrary waveform
generator. The sampling rate of the best commercial arbitrary waveform
generators can exceed 100 GS/s (i.e., a delay resolution of 10 ps is
possible). The delay between two optical pulses can be easily controlled by
varying the length of an arm in the Michelson Interferometer. A delay of 1 ps
corresponds to a change of $\sim$0.3 mm in the arm length, which can be easily
achieved with commercial translational stages. . To illustrate how these
timescales satisfy our FTNS requirements, consider the $\delta t$ required to
reconstruct the double-Lorentzian spectrum shown in Fig. 2(b) in the NV center
parameter regime Bar-Gill  _et al._ (2012); Bauch  _et al._ (2018, 2020);
Taylor  _et al._ (2008) with $T_{2}=1.32\leavevmode\nobreak\ \mu$s. To obtain
the accurate reconstruction shown in Fig. 2(b) one would need to measure 400
points with $\delta t=15$ ns. Thus, sufficiently high resolution measurements
that faithfully reconstruct various noise spectra can be comfortably performed
with experimentally available technology.

Figure 4: Spectrum reconstruction using FTNS assuming $0.1\%$ measurement
noise in coherence measurements for the same spectra shown in Figs. 1(a) and
2(b) in panels (a) and (b), respectively. Even subject to noisy data,
utilizing simple denoising techniques allows FTNS to semiquantitatively
capture the height and frequency of the peaks in the spectrum.

Since the feasibility of FTNS also relies on the ability to sufficiently
reduce statistical noise ($\sim 0.1$%) within a reasonable time, we now
consider what current technology can afford. Each point on the coherence curve
arises from a Ramsey measurement at a given time, repeated multiple times to
construct the ensemble average. Hence, to minimize statistical noise
associated with finite sampling, the repetition rate of such experiments needs
to be sufficiently high. We thus consider the time required on available
experimental setups to bring the statistical error to the 0.1% value assumed
in Fig. 4 in the measured coherence of NV centers with $T_{2}\sim$ few $\mu s$
Bar-Gill  _et al._ (2012); Bauch  _et al._ (2018, 2020), with access to
nanosecond microwave pulses. This 0.1% error requires that each point along
the coherence curve be measured $\sim 1\times 10^{6}$ times. Here, a single
measurement takes $\sim 10\leavevmode\nobreak\ \mu$s (including the
initialization and readout). Thus, requiring $\sim 100$ data points along
$C(t)$ takes $\sim 2.8$ hours (assuming a modest photon collection rate of
$\sim 10\%$). Importantly, these numbers represent a conservative estimate and
can be expected to improve significantly with better photon collection rates
or through non-uniform sampling techniques Fessler and Sutton (2003);
D’Addario and Wernecke (1977); Kazimierczuk  _et al._ (2011); Holland  _et
al._ (2011); Jaravine  _et al._ (2006). Further, by requiring only two $\pi/2$
pulses, FTNS avoids measurement noise arising from imperfect pulses, which
accrues significantly in large DD pulse sequences with many $\pi$-pulses.

FTNS performs most efficiently when FID decays sufficiently slowly, as in
trapped-ion systems Bruzewicz  _et al._ (2019); Ruster  _et al._ (2016). We
have also developed a spin-echo FTNS protocol that unlocks fast decaying
systems, allowing us to reconstruct even spectra dominated by strong
inhomogeneous contributions, as in most NV centers Bar-Gill  _et al._ (2012);
Romach  _et al._ (2015). We introduce an iterative scheme to extract
$S(\omega)$ from our spin-echo FTNS in the supplementary materials SM .

In conclusion, we have introduced a novel noise spectroscopy method that
significantly outperforms the current DDNS methods and is significantly easier
to implement from both experimental and theoretical perspectives. Our work
demonstrates the existence of a direct one-to-one invertible map between the
pure dephasing coherence function within the filter function formalism and the
noise power spectrum. Noting that current technology allows one to minimize
measurement noise, it is clear that FTNS provides a promising route to
accurately and inexpensively measure noise power spectra. Our FTNS proposal
should be applicable to a wide range of quantum platforms and can be utilized
as a powerful tool in deducing information about a qubit’s environment.

###### Acknowledgements.

Acknowledgments: A.M.C. acknowledges the start-up funds from the University of
Colorado, Boulder. S.S. acknowledges funding from National Science Foundation
(Grant No. 1734006 and 2016244).

## References

  * Cywiński  _et al._ (2008) L. Cywiński, R. M. Lutchyn, C. P. Nave, and S. Das Sarma, Phys. Rev. B 77, 174509 (2008).
  * Biercuk  _et al._ (2011) M. J. Biercuk, A. C. Doherty, and H. Uys, Journal of Physics B: Atomic, Molecular and Optical Physics 44, 154002 (2011).
  * Uhrig (2007) G. S. Uhrig, Phys. Rev. Lett. 98, 100504 (2007).
  * Biercuk  _et al._ (2009) M. J. Biercuk, H. Uys, A. P. VanDevender, N. Shiga, W. M. Itano, and J. J. Bollinger, Physical Review A 79 (2009), 10.1103/PhysRevA.79.062324.
  * Yang  _et al._ (2010) W. Yang, Z.-Y. Wang, and R.-B. Liu, Frontiers of Physics 6 (2010), 10.1007/s11467-010-0113-8.
  * Szańkowski  _et al._ (2017) P. Szańkowski, G. Ramon, J. Krzywda, D. Kwiatkowski, and Ł. Cywiński, Journal of Physics: Condensed Matter 29, 333001 (2017).
  * Degen  _et al._ (2017) C. L. Degen, F. Reinhard, and P. Cappellaro, Rev. Mod. Phys. 89, 035002 (2017).
  * Suter and Álvarez (2016) D. Suter and G. A. Álvarez, Rev. Mod. Phys. 88, 041001 (2016).
  * Álvarez and Suter (2011) G. A. Álvarez and D. Suter, Phys. Rev. Lett. 107, 230501 (2011).
  * Yuge  _et al._ (2011) T. Yuge, S. Sasaki, and Y. Hirayama, Phys. Rev. Lett. 107, 170504 (2011).
  * Norris  _et al._ (2016) L. M. Norris, G. A. Paz-Silva, and L. Viola, Phys. Rev. Lett. 116, 150503 (2016).
  * Krzywda  _et al._ (2019) J. Krzywda, P. Szań kowski, and Ł. Cywiński, New Journal of Physics 21, 043034 (2019).
  * Bylander  _et al._ (2011) J. Bylander, S. Gustavsson, F. Yan, F. Yoshihara, K. Harrabi, G. Fitch, D. G. Cory, Y. Nakamura, J.-S. Tsai, and W. D. Oliver, Nature Physics 7, 565 (2011).
  * Sung  _et al._ (2019) Y. Sung, F. Beaudoin, L. Norris, F. Yan, D. Kim, J. Qiu, U. von Lüpke, J. Yoder, T. Orlando, S. Gustavsson, L. Viola, and W. Oliver, Nature Communications 10 (2019), 10.1038/s41467-019-11699-4.
  * Almog  _et al._ (2011) I. Almog, Y. Sagi, G. Gordon, G. Bensky, G. Kurizki, and N. Davidson, Journal of Physics B: Atomic, Molecular and Optical Physics 44, 154006 (2011).
  * Dial  _et al._ (2013) O. E. Dial, M. D. Shulman, S. P. Harvey, H. Bluhm, V. Umansky, and A. Yacoby, Phys. Rev. Lett. 110, 146804 (2013).
  * Farfurnik  _et al._ (2021) D. Farfurnik, H. Singh, Z. Luo, A. S. Bracker, S. G. Carter, R. M. Pettit, and E. Waks, ‘All-optical noise spectroscopy of a solid-state spin,’ (2021).
  * Connors  _et al._ (2022) E. Connors, J. Nelson, L. Edge, and J. Nichol, Nature Communications 13 (2022), 10.1038/s41467-022-28519-x.
  * Bar-Gill  _et al._ (2012) N. Bar-Gill, L. Pham, C. Belthangady, D. Sage, P. Cappellaro, J. Maze, M. Lukin, A. Yacoby, and R. Walsworth, Nature communications 3, 858 (2012).
  * Romach  _et al._ (2015) Y. Romach, C. Müller, T. Unden, L. J. Rogers, T. Isoda, K. M. Itoh, M. Markham, A. Stacey, J. Meijer, S. Pezzagna, B. Naydenov, L. P. McGuinness, N. Bar-Gill, and F. Jelezko, Phys. Rev. Lett. 114, 017601 (2015).
  * Carr and Purcell (1954) H. Y. Carr and E. M. Purcell, Phys. Rev. 94, 630 (1954).
  * Meiboom and Gill (1958) S. Meiboom and D. Gill, Review of Scientific Instruments 29, 688 (1958).
  * Szańkowski and Cywiński (2018) P. Szańkowski and L. Cywiński, Phys. Rev. A 97, 032101 (2018).
  * Wise  _et al._ (2021) D. F. Wise, J. J. Morton, and S. Dhomkar, PRX Quantum 2, 010316 (2021).
  * Yan  _et al._ (2012) F. Yan, J. Bylander, S. Gustavsson, F. Yoshihara, K. Harrabi, D. G. Cory, T. P. Orlando, Y. Nakamura, J.-S. Tsai, and W. D. Oliver, Phys. Rev. B 85, 174521 (2012).
  * Boss  _et al._ (2016) J. M. Boss, K. Chang, J. Armijo, K. Cujia, T. Rosskopf, J. R. Maze, and C. L. Degen, Phys. Rev. Lett. 116, 197601 (2016).
  * Gu and Franco (2019) B. Gu and I. Franco, The Journal of Chemical Physics 151, 014109 (2019).
  * Note (1) This form of filter-function formalism is the same for both classical and quantum noise models Yuge  _et al._ (2011).
  * (29) See the Supplementary Materials for figure parameters, an explicit derivation of the linear behavior at long times, a derivation of the early time behavior and its fitting procedure, the details of our denoising procedure, and an extension of the protocol to spin-echo sequence.
  * Makhlin and Shnirman (2003) Y. Makhlin and A. Shnirman, Journal of Experimental and Theoretical Physics Letters 78, 497 (2003).
  * Schriefl  _et al._ (2006) J. Schriefl, Y. Makhlin, A. Shnirman, and G. Schön, New Journal of Physics 8, 1 (2006).
  * Note (2) Equation 1, subject to FID, suggests that a spectrum with a dominant low frequency component has a faster decoherence time. This is because the power spectrum is multiplied by $\omega^{2}F(\omega t)=\sin^{2}(\omega t/2)$, which is sharply peaked at $\omega=0$.
  * Tyryshkin  _et al._ (2010) A. M. Tyryshkin, Z.-H. Wang, W. Zhang, E. E. Haller, J. W. Ager, V. V. Dobrovitski, and S. A. Lyon, ‘Dynamical decoupling in the presence of realistic pulse errors,’ (2010).
  * Souza  _et al._ (2011) A. M. Souza, G. A. Álvarez, and D. Suter, Phys. Rev. Lett. 106, 240501 (2011).
  * Chartrand (2011) R. Chartrand, ISRN Appl. Math. 2011 (2011), 10.5402/2011/164564.
  * Van Breugel  _et al._ (2020) F. Van Breugel, J. N. Kutz, and B. W. Brunton, IEEE Access 8, 196865 (2020).
  * Press  _et al._ (2010) D. Press, K. De Greve, P. Mcmahon, T. Ladd, B. Friess, C. Schneider, M. Kamp, S. Höfling, A. Forchel, and Y. Yamamoto, Nature Photonics 7, 367 (2010).
  * Vezvaee  _et al._ (2022) A. Vezvaee, E. Takou, P. Hilaire, M. Doty, and S. Economou, ‘Avoiding leakage and errors caused by unwanted transitions in lambda systems,’ (2022).
  * Bodey  _et al._ (2019) J. Bodey, R. Stockill, E. Denning, D. Gangloff, G. Éthier Majcher, D. Jackson, E. Clarke, M. Hugues, C. Gall, and M. Atatüre, npj Quantum Information 5 (2019), 10.1038/s41534-019-0206-3.
  * Takou and Economou (2021) E. Takou and S. E. Economou, Phys. Rev. B 104, 115302 (2021).
  * Nguyen  _et al._ (2019) C. T. Nguyen, D. D. Sukachev, M. K. Bhaskar, B. Machielse, D. S. Levonian, E. N. Knall, P. Stroganov, C. Chia, M. J. Burek, R. Riedinger, H. Park, M. Lončar, and M. D. Lukin, Phys. Rev. B 100, 165428 (2019).
  * Note (3) Two microwave pulses with a controllable delay can be generated with an arbitrary waveform generator. The sampling rate of the best commercial arbitrary waveform generators can exceed 100 GS/s (i.e., a delay resolution of 10 ps is possible). The delay between two optical pulses can be easily controlled by varying the length of an arm in the Michelson Interferometer. A delay of 1 ps corresponds to a change of $\sim$0.3 mm in the arm length, which can be easily achieved with commercial translational stages.
  * Bauch  _et al._ (2018) E. Bauch, C. A. Hart, J. M. Schloss, M. J. Turner, J. F. Barry, P. Kehayias, S. Singh, and R. L. Walsworth, Phys. Rev. X 8, 031025 (2018).
  * Bauch  _et al._ (2020) E. Bauch, S. Singh, J. Lee, C. A. Hart, J. M. Schloss, M. J. Turner, J. F. Barry, L. M. Pham, N. Bar-Gill, S. F. Yelin, and R. L. Walsworth, Phys. Rev. B 102, 134210 (2020).
  * Taylor  _et al._ (2008) J. M. Taylor, P. Cappellaro, L. Childress, L. Jiang, D. Budker, P. R. Hemmer, A. Yacoby, R. Walsworth, and M. D. Lukin, Nature Physics 4, 810 (2008).
  * Fessler and Sutton (2003) J. Fessler and B. Sutton, IEEE Transactions on Signal Processing 51, 560 (2003).
  * D’Addario and Wernecke (1977) L. D’Addario and S. Wernecke, IEEE Transactions on Computers 26, 351 (1977).
  * Kazimierczuk  _et al._ (2011) K. Kazimierczuk, M. Misiak, J. Stanek, A. Zawadzka-Kazimierczuk, and W. Koźmiński, Topics in current chemistry 316, 79 (2011).
  * Holland  _et al._ (2011) D. Holland, M. Bostock, L. Gladden, and D. Nietlispach, Angewandte Chemie (International ed. in English) 50, 6548 (2011).
  * Jaravine  _et al._ (2006) V. Jaravine, I. Ibragimov, and V. Orekhov, Nature methods 3, 605 (2006).
  * Bruzewicz  _et al._ (2019) C. D. Bruzewicz, J. Chiaverini, R. McConnell, and J. M. Sage, Applied Physics Reviews 6, 021314 (2019), https://doi.org/10.1063/1.5088164 .
  * Ruster  _et al._ (2016) T. Ruster, C. T. Schmiegelow, H. Kaufmann, C. Warschburger, F. Schmidt-Kaler, and U. G. Poschinger, Applied Physics B 122 (2016), 10.1007/s00340-016-6527-4.


# The Document Vectors Using Cosine Similarity Revisited

Zhang Bingyu△
&Nikolay Arefyev♢,∇,△

△National Research University Higher School of Economics / Moscow, Russia
♢Samsung Research Center Russia / Moscow, Russia
∇Lomonosov Moscow State University / Moscow, Russia
<EMAIL_ADDRESS><EMAIL_ADDRESS>

###### Abstract

The current state-of-the-art test accuracy (97.42%) on the IMDB movie reviews
dataset was reported by Thongtan and Phienthrakul (2019) and achieved by the
logistic regression classifier trained on the Document Vectors using Cosine
Similarity (DV-ngrams-cosine) proposed in their paper and the Bag-of-N-grams
(BON) vectors scaled by Naive Bayesian weights. While large pre-trained
Transformer-based models have shown SOTA results across many datasets and
tasks, the aforementioned model has not been surpassed by them, despite being
much simpler and pre-trained on the IMDB dataset only.

In this paper, we describe an error in the evaluation procedure of this model,
which was found when we were trying to analyze its excellent performance on
the IMDB dataset. We further show that the previously reported test accuracy
of 97.42% is invalid and should be corrected to 93.68%. We also analyze the
model performance with different amounts of training data (subsets of the IMDB
dataset) and compare it to the Transformer-based RoBERTa model. The results
show that while RoBERTa has a clear advantage for larger training sets, the
DV-ngrams-cosine performs better than RoBERTa when the labelled training set
is very small (10 or 20 documents). Finally, we introduce a sub-sampling
scheme based on Naive Bayesian weights for the training process of the DV-
ngrams-cosine, which leads to faster training and better quality.

##  1 Introduction

The word2vec algorithm originally published by Mikolov et al. (2013) is among
the most famous methods to train vector representations of words. Soon after
the emergence of word2vec, a similar method to build vector representations of
documents was originally proposed by Le and Mikolov (2014) and further studied
by Mesnil et al. (2015). It is known under different names, including
Paragraph Vectors, Sentence Vectors, doc2vec, etc. This method jointly learns
word embeddings and document embeddings such that a binary classifier can
predict if a given word occurs in a particular document given only the
corresponding embeddings. More formally, the following objective is minimized:

$\sum_{d\in D}\sum_{w\in
W_{d}}[-\log\sigma(v_{d}^{T}v_{w})-\sum_{w^{\prime}\sim
V}\log\sigma(-v_{d}^{T}v_{w^{\prime}})]$ (1)

Here $D$ denotes the set of documents, $W_{d}$ is the list of words that make
up the document $d$, $w^{\prime}$ is a word randomly sampled from the full
vocabulary $V$, also known as a negative sample (Goldberg and Levy, 2014).
Finally, $v_{d}$ and $v_{w}$ are the learnt embeddings of $d$ and $w$.
Intuitively, for each document, an embedding is learnt that has high
similarity to the embeddings of those words that occur in this document and
low similarity to the embeddings of some random words.

Later Li et al. (2015) switched from single words to n-grams and observed
significant improvements. Building on that, Thongtan and Phienthrakul (2019)
studied different objective functions. They have found that the cosine
similarity outperforms the dot product, which led to a modified model called
the Document Vectors using Cosine Similarity (we will call it DV-ngrams-cosine
for short). The new objective is:

$\displaystyle\sum_{d\in D}$ $\displaystyle\sum_{u\in
U_{d}}[-\log\sigma(\alpha cos(v_{d},v_{u}))$ (2) $\displaystyle-$
$\displaystyle\sum_{u^{\prime}\sim V}\log\sigma(-\alpha
cos(v_{d},v_{u^{\prime}}))],$

where $U_{d}$ denotes the set of all n-grams in $d$, $v_{u}$ is the embedding
of the n-gram $u$ from $d$, $v_{u^{\prime}}$ is the embedding of a randomly
sampled n-gram, and $\alpha$ is a hyperparameter.

In the same paper, the authors proposed an ensemble consisting of the document
embeddings from DV-ngrams-cosine and the Bag-of-N-grams vectors scaled by
Naive Bayesian weights (NB-weighted BON for short). They concatenated these
two representations and trained the logistic regression classifier on top. The
ensemble was reported to have very high test accuracy (97.42%) on the IMDB
movie reviews dataset (Maas et al. (2011)). To the best of our knowledge, this
accuracy remains the SOTA result on IMDB. Even large Transformer-based models
pre-trained on a huge amount of texts, both in-domain and out-of-domain, have
shown lower accuracy on this dataset Yang et al. (2019); Suchin et al. (2020);
Arefyev et al. (2021).

This extraordinary performance of such a simple model motivated us to
thoroughly study the model and its implementation trying to understand the
reasons behind its success. Unfortunately, during this study, we found a bug
in the implementation of the evaluation procedure of the ensemble, which had
made the estimation of the accuracy incorrect.

In our paper, we re-evaluate the ensemble as well as its individual
components. We show that the originally reported test accuracy of the ensemble
(97.42%) is incorrect and shall be corrected to 93.68%, which is only 0.55%
higher than the accuracy on pure DV-ngrams-cosine embeddings.

Additionally, we analyze how the amount of training data affects the
performance of the ensemble, as well as its individual components, and also
the Transformer-based RoBERTa model Liu et al. (2020), which has recently
shown SOTA or near-SOTA results over a variety of tasks and datasets.
Surprisingly, we have observed that DV-ngrams-cosine outperforms RoBERTa when
the number of labelled training examples is small (10 or 20). We also ensemble
RoBERTa with DV-ngrams-cosine, but only have achieved a marginal improvement.
Finally, we propose a modification for the training process of DV-ngrams-
cosine that results in faster training and better accuracy. The code
reproducing our experiments is publicly available
111https://github.com/Bgzh/dv_cosine_revisited.

##  2 Re-evaluation of the ensemble

In the aforementioned ensemble proposed by Thongtan and Phienthrakul (2019),
the NB-weighted BON and the DV-ngrams-cosine are concatenated and fed into the
logistic regression classifier. However, we have found that in the original
implementation the two vectors concatenated to obtain a single training or
test example usually correspond to two different documents of the same class
(see details in Appendix A). Specifically, the DV-ngrams-cosine vectors and
the BON vectors are built from two different files having different orders of
examples. As a result, after the concatenation, each input to the logistic
regression corresponds to a combination of two examples. Due to the special
structure of the files, those examples are guaranteed to belong to the same
class and the same subset. For instance, a positive example from the test set
is concatenated with another positive example from the test set.

In Appendix B.3 we provide an analysis that shows the reasons of high
performance of this concatenation of two representations. From this analysis
it follows that most examples from IMDB are correctly classified with high
confidence (a large logit) using any of two representations, i.e. they are
easy examples. Less than 10% of examples are classified incorrectly by each
representation (hard examples), but they often obtain low confidence (a logit
near zero). Hard examples are more often combined with easy examples just
because of their dominance. In these cases, the logit from the easy example
often outweigh the logit from the hard one resulting in the correct final
prediction.

Thus, in both the training and the test sets, hard examples are often combined
with simpler examples, making the classification task easier. In this process,
the knowledge of the true labels is implicitly exploited to combine the
examples this way, in both training and testing. This leads to an incorrect
estimation of the classification accuracy for future examples.

After fixing this issue, we have observed that the combination of different
representations of the same document leads to the test accuracy of 93.68%
instead of 97.42% originally reported. Compared to the pure DV-ngrams-cosine
embeddings, the ensemble improves the test accuracy by 0.55%, not 4.29%
reported previously. This improvement also better agrees with the improvements
of less than 1% observed by Li et al. (2015) for similar ensembles with the
predecessor model DV-ngram. As a sanity check, Appendix B additionally reports
the accuracy for different schemes of combining the two representations,
showing that higher accuracy can be achieved only by those schemes that
exploit the knowledge of the test labels.

##  3 Further analysis of performance

In his section we further analyze the performance of the ensemble described
above, comparing it to its individual components as well as to the recently
introduced Transformer-based RoBERTa model Liu et al. (2020). We study the
performance of these models depending on the number of labelled examples in
the training set.

Figure 1: The performance of different models on training sets of different
sizes. The mean values and standard deviations were calculated over 10 random
subsets for RoBERTa and 30 random subsets for other models for each training
set size. BON in the legend implies NB-weighted BON.

For a more fair comparison, the most important hyperparameters of each model
were tuned on the validation set, employing the train/validation/test split of
the IMDB dataset provided by (Suchin et al., 2020). Subsets of different sizes
from 10 to 20000 examples were randomly sampled from the training set. The
logistic regression classifier was trained on these subsets using the DV-
ngram-cosine embeddings, the NB-weighted BON vectors, or their concatenation
as its input representation.

We tuned the L2-regularization strength $C$ of the classifier individually for
each subset of the training set. Additionally, we multiplied the DV-ngram-
cosine embeddings before concatenating them to the BON vectors in order to
balance the magnitudes of the two representations, which may help the
classifier to benefit from both representations. The scaling factor was also
selected on the validation set.

The pre-trained RoBERTa base
model222https://pytorch.org/hub/huggingface_pytorch-transformers/ was fine-
tuned on a part (10 out of 30) of the same subsets of the training set, using
the validation set for early stopping. We used a batch size of 32, with a
maximum learning rate of 1e-5, recommended by
fairseq333https://github.com/pytorch/fairseq/blob/main/examples/roberta/README.custom_classification.md.

As shown in Fig. 1, the fine-tuned RoBERTa model usually achieves higher test
accuracy. But when the number of labelled training examples is very small (10
or 20), the logistic regression on the DV-ngrams-cosine embeddings shows
higher mean test accuracy and lower standard deviation. This result
corroborated the notion that small models can be a better choice when the data
are scarce.

On the other hand, logistic regression on the BON vectors performs
significantly worse than all other models across all training set sizes.
Finally, we don’t observe any significant improvements from the ensembling
when the training set size is less than 20k, as the difference is within one
standard deviation.

It is important to notice that the DV-ngrams-cosine embeddings were pre-
trained on the in-domain examples from the whole IMDB dataset, while RoBERTa
was pre-trained on a huge but general-domain corpus. It is likely that the
domain adaptation techniques Suchin et al. (2020) will help RoBERTa when the
number of labelled examples is small. However, for our study, we decided to
compare the most standard approaches to training the corresponding models.

##  4 NB Sub-Sampling

In this section, we improve the training procedure of DV-ngrams-cosine by
applying a sub-sampling procedure based on the Naive Bayesian weights of
ngrams (NB Sub-Sampling) in order to make the model focus more on sentiment-
related ngrams while building the document embeddings.

Figure 2: Training process with and without NB sub-sampling. The test accuracy of the logistic regression built on top of the document vectors is plotted. The mean values and standard deviations were calculated over 3 runs for each type. Model | Test Accuracy %
---|---
Models trained on the original training set of IMDB (25K) |
NB-weighted BON | 91.29
DV-ngrams-cosine | 93.13
DV-ngrams-cosine + NB-weighted BON Thongtan and Phienthrakul (2019) |  $\\#$97.42
DV-ngrams-cosine + NB-weighted BON (re-evaluated) | 93.68
Models trained using the train/dev split from Suchin et al. (2020) (20K/5K) |
DV-ngrams-cosine with NB sub-sampling | 93.36
RoBERTa | 95.79
DV-ngrams-cosine + RoBERTa | 95.92
DV-ngrams-cosine with NB sub-sampling + RoBERTa | 95.94
Table 1: Test results on the IMDB dataset. $\\#$ indicates incorrect
previously reported results.

Inspired by the previous works (Wang and Manning (2012), Arefyev et al.
(2021)), we trained a multinomial Naive Bayesian Classifier and exploited its
weights to calculate the importance of each ngram $f_{i}$ for the final
classification task:

$h_{i}=|\log p(f_{i}|y=1)-\log p(f_{i}|y=0)|$ (3)

In each epoch we put an ngram into training with the probability

$p(f_{i})=min(\exp(h_{i}/n_{a})/n_{b},1),$ (4)

where $n_{a}$ and $n_{b}$ are the hyperparameters. The choices are purely
empirical. We tried different combinations of $n_{a}$ and $n_{b}$ and found 2
and 3 (respectively) to be the best in them.

The comparison of the training process with and without NB sub-sampling is
shown in Fig. 2 (refer to Appendix C for details of the experiments and the
accuracy on the validation set).

The runs with NB sub-sampling progress faster and show a distinct advantage
after 2500 steps. After 30k steps, the runs with NB sub-sampling stagnated and
kept fluctuating in a small region; the vanilla runs stagnated after 50k
steps, in a lower area. It is also worth noticing that although the labels of
the training set are used during pre-training for sub-sampling, we did not
observe any significant overfitting due to that. Neither the validation score
nor the test score showed a tendency to decay long after reaching the plateau,
indicating that this sub-sampling scheme can be used as an add-on to the
original model, boosting its performance while not creating additional
overfitting trouble.

##  5 Ensemble DV-ngrams-cosine and RoBERTa

The ensemble proposed in (Thongtan and Phienthrakul (2019)) and described in
Section 2 combines two different representations of documents, which are the
DV-ngrams-cosine embeddings and the NB-weighted BON vectors. However, we have
observed in Section 3 that the BON vectors are quite weak on their own, while
RoBERTa outperforms all other models unless the number of examples is very
small. Thus, it is interesting if DV-ngram-cosine can help RoBERTa. In this
section, we combine the DV-ngrams-cosine (with or without NB sub-sampling)
with the output of the last hidden layer of RoBERTa, and test on the IMDB
dataset. Again, the train/validation/test splits by Suchin et al. (2020) were
used. A scaling factor on the DV-ngrams-cosine and the hyperparameter $C$ in
the logistic regression were tuned on the validation set.
The results are shown in Table 1. Although RoBERTa is a much stronger model
than DV-ngram-cosine, combining them has shown a small improvement of
0.13-0.15%.

##  6 Conclusion

The ensemble featuring the DV-ngrams-cosine reported by Thongtan and
Phienthrakul (2019) was re-evaluated. The test accuracy of this ensemble on
the IMDB dataset was corrected from 97.42% to 93.68%. The DV-ngrams-cosine
embeddings with the logistic regression on top were compared with RoBERTa
using different amounts of training data. In this comparison, the DV-ngrams-
cosine has surprisingly outperformed RoBERTa for a small number of training
examples (10 or 20 documents). A sub-sampling scheme based on the Naive
Bayesian weights was introduced to the training process of the DV-ngrams-
cosine, resulting in faster training and better quality.

## Acknowledgements

We are grateful to our anonymous reviewers. This research was partially
supported by the Basic Research Program at the HSE University.

## References

  * Arefyev et al. (2021) Nikolay Arefyev, Dmitry Kharchev, and Artem Shelmanov. 2021.  Nb-mlm - efficient domain adaptation of masked language models for sentiment analysis.  _EMNLP_ , pages 9114–9124.
  * Goldberg and Levy (2014) Yoav Goldberg and Omer Levy. 2014.  word2vec explained: deriving mikolov et al.’s negative-sampling word-embedding method.  _CoRR_.
  * Le and Mikolov (2014) V. Quoc Le and Tomas Mikolov. 2014.  Distributed representations of sentences and documents.  _ICML_ , pages 1188–1196.
  * Li et al. (2015) Bofang Li, Tao Liu, Xiaoyong Du, Deyuan Zhang, and Zhe Zhao. 2015.  Learning document embeddings by predicting n-grams for sentiment classification of long movie reviews.  _CoRR_.
  * Liu et al. (2020) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2020.  Ro{bert}a: A robustly optimized {bert} pretraining approach.
  * Maas et al. (2011) Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. 2011.  Learning word vectors for sentiment analysis.  In _Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies_ , pages 142–150, Portland, Oregon, USA. Association for Computational Linguistics.
  * Mesnil et al. (2015) Grégoire Mesnil, Tomas Mikolov, Marc’Aurelio Ranzato, and Yoshua Bengio. 2015.  Ensemble of generative and discriminative techniques for sentiment analysis of movie reviews.  _international conference on learning representations_.
  * Mikolov et al. (2013) Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013.  Efficient estimation of word representations in vector space.  _CoRR_.
  * Suchin et al. (2020) Gururangan Suchin, Marasović Ana, Swayamdipta Swabha, Lo Kyle, Beltagy Iz, Downey Doug, and Noah Smith A. 2020.  Don’t stop pretraining: Adapt language models to domains and tasks.  _ACL_ , pages 8342–8360.
  * Thongtan and Phienthrakul (2019) Tan Thongtan and Tanasanee Phienthrakul. 2019.  Sentiment classification using document embeddings trained with cosine similarity.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: Student Research Workshop_ , pages 407–414, Florence, Italy. Association for Computational Linguistics.
  * Wang and Manning (2012) Sida Wang and Christopher Manning. 2012.  Baselines and bigrams: Simple, good sentiment and topic classification.  In _Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 90–94, Jeju Island, Korea. Association for Computational Linguistics.
  * Yang et al. (2019) Zhilin Yang, Zihang Dai, Yiming Yang, G. Jaime Carbonell, Ruslan Salakhutdinov, and V. Quoc Le. 2019.  Xlnet: Generalized autoregressive pretraining for language understanding.  _ADVANCES IN NEURAL INFORMATION PROCESSING SYSTEMS 32 (NIPS 2019)_ , pages 5754–5764.

Figure 3: Diagram of the ensemble and its input in the original code

##  Appendix A Detailed Description of the Bug

As depicted in Fig. 3, In the original work, neither of the two sets of
vectors (DV-ngrams-cosine and NB-weighted BON) were obtained directly from the
original IMDB movie review datasets. Instead, they were from two preprocessed
versions of the IMDB movie review dataset stored in two files named "alldata-
id_p1gram.txt" and "alldata-id_p3gram.txt", respectively. (We will refer to
them as "p1gram.txt" and "p3gram.txt" in the remaining of this paper for
short.) The file "p1gram.txt" contains unigrams extracted from the IMDB
dataset, while "p3gram.txt" contains unigrams, bigrams and trigrams.

The problem is that the document orders in these two files are not the same.
For example, the 7th document in "p1gram.txt" corresponds to the 5978th
document in "p3gram.txt".

After a closer look at these two files, as shown in Fig. 4, one can see that
in both files, the preprocessed documents are ordered in such a way that they
can be naturally divided into blocks. Each block contains all the documents
within one particular class and one particular subset (train, test or
unlabelled). The orders of these blocks inside the two files are the same. For
example, in both files, the positive examples from the training set occupy the
first 12500 lines.

However, inside each block, the documents are ordered differently in these two
files. As a result, the document order of "p3gram.txt" is one in-block
permutation (permutation within each block) away from the document order of
"p3gram.txt".

In the original code, the DV-ngrams-cosine embeddings are built from
"p3gram.txt", while the NB-weighted BON vectors are built from "p1gram.txt".
They are directly concatenated without any reordering. This leads to the
incorrect estimation of the test accuracy of the ensemble.

##  Appendix B Re-Evaluation of the Ensemble

After finding the aforementioned bug, the next step is evaluating the ensemble
correctly. Apart from re-evaluating the ensemble with the correct matching (so
that the two vector representations of the same document are concatenated
together), we also tried different ways of combining two documents by
performing shuffling on one of the two files. Concatenating vectors from
different documents is perfectly legit if no test labels are required in this
process. Therefore, we compare different shuffling schemes, both using and not
using test labels, for completeness and sanity check.

###  B.1 Experiments

The ensemble was evaluated by the original code, with both the original
matching (the two representations were concatenated according to the document
orders of the files "p1gram.txt" and "p3gram.txt") and the correct matching
(the representations of the same document are concatenated). Some additional
tests with different shuffling schemes were also carried out, only
"p1gram.txt" was shuffled in all of these tests. The shuffling schemes are
shown in Fig. 5.

In both A and C, the test set was shuffled in-class. This requires using the
labels of the test set in order to group documents with the same labels, which
may significantly simplify the classification task. Both B and D only
underwent cross-class shuffles, in which samples from different classes were
mixed and treated equally. This augments the BON vectors with the DV-ngrams-
cosine vectors of some random examples, thus, we basically add noise. In C
this noise is only added to the test set, while in D it is also added to the
training set, so the model can learn to ignore it.

By comparing A vs. B and C vs. D, we can find out whether the aforementioned
leakage of the test labels is necessary to obtain high test accuracy. In
particular, C was designed to reproduce the incorrect evaluation in the
original paper, except for the randomness in the shuffling process. All tests
involving shuffling were run 30 times.

Shuffling | Acc. | Acc.
---|---|---
Scheme | Mean | Std.
original matching | 97.42 |
correct matching | 93.68 |
test set in-class (A) | 96.58 | 0.07
test set cross-class (B) | 61.80 | 0.25
train/test in-class (C) | 97.43 | 0.08
train/test cross-class (D) | 91.64 | 0.08
Table 2: Test accuracy for different shuffling schemes. Figure 4: Two
preprocessed versions of the IMDB movie review dataset: p1gram.txt and
p3gram.txt Figure 5: The shuffling schemes for re-evaluation of the ensemble.
First of all, both the Document Vectors (DV) and the BON vectors are sorted in
the document order of p1gram.txt. Then, 4 shuffling schemes are imposed on the
Document Vectors, respectively. In A and C, the blocks are shuffled
internally, while in B and D, the corresponding positive block and negative
block are mixed and shuffled as a whole. The train-test split is respected
throughout this experiment. Neither of the BON vectors nor the labels are
shuffled.

(a)

(b)

Figure 6: Distributions of the logits (test set). The classification
boundaries of the ensemble and the sub-models are noted as the red diagonal
and the grey dashed lines, respectively. Here we evenly split the intercept of
the ensemble for the 2 sub-models. (a) when the document vectors are
incorrectly concatenated (as in the original paper); (b) when the document
vectors are correctly concatenated.

(a)

(b)

Figure 7: Distributions of the logits of the positive examples. Green/red dots
represent documents that are correctly/incorrectly classified by the ensemble.
(a) positive documents that are misclassified by the DV_ngrams_cosine part,
(b) positive documents that are correctly classified by the DV_ngrams_cosine
part Figure 8: Marginal distributions of the BON part of the positive
examples. (a) positive documents that are misclassified by the
DV_ngrams_cosine part, (b) positive documents that are correctly classified by
the DV_ngrams_cosine part

###  B.2 Results and discussions

The results of the shuffling tests are shown in Table 2. With the correct
matching, the test accuracy of the ensemble was 93.68%, which is much lower
than the test accuracy of 97.42% achieved by the original matching (this
result agrees with the original paper).

Augmenting the BON vectors with the DV-ngrams-cosine embeddings of different
documents from the same classes (A and C) showed much higher performance than
the correct matching. Between them, C got better accuracy due to the
consistency in the training and testing process. On the other hand, the
training examples for A were correctly concatenated so the model suffered from
data drift during testing.

Augmentation using some random documents for the test set only (B) makes the
logistic regression perform almost like a random classifier, while when it is
done for the training set also (D), it can learn to ignore the added noise.
They serve as a sanity check to verify that shuffling without knowing the
labels doesn’t get better accuracy than the "correct matching".

Only the shuffling scheme C yielded accuracy that is very similar to the
original matching, showing that a random in-class shuffle of both the training
and test set can closely reproduce the test accuracy reported in the original
paper.

###  B.3 A closer look at the high estimation of accuracy caused by the
incorrect concatenation

From the accuracy of each part of the ensemble (table 1), we see that each
part of the ensemble is incorrect for a rather small portion of examples (less
than 7% for DV-ngrams-cosine, less than 9% for NB-weighted BON). As can be
seen from the marginal distributions in figure 6, for these incorrectly
predicted examples the absolute values of logits are usually much smaller
compared to the correctly classified examples.

We can also see from the scatter plot of figure 6 (a) that when the vectors
are concatenated in the original order, the logits of the two parts (DV-
ngrams-cosine and NB-weighted BON) look independent conditioned on the true
label, which corroborates our speculation that "p3gram.txt" were randomly
shuffled inside each class.

Thus, there is a high chance that for an incorrect prediction by the DV-
ngrams-cosine part ("hard" example) a random example of the same class will
have larger NB-weighted BON logit with the correct sign ("easy" example) and
fix this prediction. But as shown in the scatter plot of figure 6 (b), when
the vectors are correctly concatenated, the logits of the two parts have a
positive correlation. So, the correct model doesn’t benefit from the
aforementioned independence.

To see it more clearly, Figure 7 (a) shows only the examples that are
incorrectly classified by the DV-ngrams-cosine part. In the original
combination, the complementary BON logit is mostly positive and quit large,
moving many of those examples to the correct side of the decision boundary.
When concatenated correctly, the BON logit is concentrated near zero and
rarely helps. In Figure 7 (b) only those examples that are correctly
classified by the DV-ngrams-cosine part are shown. We can see that adding the
BON logit can sometimes move those examples to the incorrect side of the
boundary. However, the proportion of such cases is similar for the original
and correctly concatenated representations.

Figure 8 shows the marginal distributions of the NB-weighted BON part in the
same settings. Between the two situations (the DV-ngrams-cosine part is
incorrect/correct), the histogram of the BON logits stays in a very similar
shape when the vectors are incorrectly concatenated (blue), while showing a
shift when the vectors are correctly concatenated (orange). These observations
corroborate the argument that the model on the incorrectly concatenated
vectors benefits from the aforementioned conditional independence.

It is also worth noticing from figure 8 that in both (a) and (b), the
distribution of the BON logits of the original concatenation (blue) is more on
the positive side. This indicates that incorrect concatenation may also help
the training of the weights for NB-weighted-BON. We have tentatively run tests
on this, but haven’t found any overall improvements: the ensemble trained on
incorrectly concatenated vectors and tested on correctly concatenated vectors
only has a test accuracy of 93.0%.

##  Appendix C Training DV-ngrams-cosine with NB Sub-Sampling

In the experiments of this section (also in section 4), the
train/validation/test splits (Suchin et al., 2020) of the IMDB dataset were
also used. During the training, after every epoch, logistic regression was
trained on the training set; the hyperparameter $C$ was tuned on the
validation set; then the model with the "best" $C$ was tested on the test set.
The training of each type (with or without NB sub-sampling) was repeated 3
times.

Figure 9: Training process with and without NB sub-sampling. The validation
accuracy of the logistic regression built on top of the document vectors is
plotted. The mean values and standard deviations were calculated over 3 runs
for each type.

The validation accuracy is plotted in Figure 9 and the test accuracy is
plotted in Figure 2. The runs with NB sub-sampling showed slight advantages in
both plots, almost everywhere.