# Investigation and mitigation of noise contributions in a compact heterodyne
interferometer

Yanqi Zhang  Texas A&M University, Aerospace Engineering & Physics, College
Station, TX 77843  Wyant College of Optical Sciences, The University of
Arizona, 1630 E. University Blvd., Tucson, AZ 85721  Adam Hines  Guillermo
Valdes  Felipe Guzman<EMAIL_ADDRESS>Texas A&M University, Aerospace
Engineering & Physics, College Station, TX 77843

###### Abstract

We present a noise estimation and subtraction algorithm capable of increasing
the sensitivity of heterodyne laser interferometers by one order of magnitude.
The heterodyne interferometer is specially designed for dynamic measurements
of a test mass in the application of sub-Hz inertial sensing. A noise floor of
$3.31\text{\times}{10}^{-11}\text{\,}\mathrm{m}\mathrm{/}\sqrt{\mathrm{Hz}}$
at $100\text{\,}\mathrm{mHz}$ is achieved after applying our noise subtraction
algorithm to a benchtop prototype interferometer that showed a noise level of
$2.76\text{\times}{10}^{-10}\text{\,}\mathrm{m}\mathrm{/}\sqrt{\mathrm{Hz}}$
at $100\text{\,}\mathrm{mHz}$ when tested in vacuum at levels of
$3\text{\times}{10}^{-5}\text{\,}\leavevmode\nobreak\ $Torr. Based on the
previous results, we investigated noise estimation and subtraction techniques
of non-linear optical pathlength noise, laser frequency noise, and temperature
fluctuations in heterodyne laser interferometers. For each noise source, we
identified its contribution and removed it from the measurement by linear
fitting or a spectral analysis algorithm. The noise correction algorithm we
present in this article can be generally applied to heterodyne laser
interferometers.

##  I Introduction

In the past decades, displacement measuring interferometry (DMI) has extended
its application to gravitational wave (GW) detection, including free-falling
test mass measurements in space-based gravitational wave detection as the
Laser Interferometer Space Antenna (LISA) and its technology demonstrator,
LISA Pathfinder [1, 2, 3, 4, 5], intersatellite displacement measurements as
in the mission GRACE Follow-On that utilizes a Laser Ranging Interferometer
[6], and inertial sensor development for seismic activity monitoring in
ground-based Laser Interferometer Gravitational-Wave Observatory (LIGO) [7,
8]. The above applications require high sensitivity or, in other words, a low
noise floor in the frequency regime between tens of micro-Hz to a few Hz. The
test mass motion is usually expected to range from a few microns to several
millimeters, where a DMI with a large dynamic range is crucial to avoid phase
readout ambiguities. Among various DMI techniques, heterodyne laser
interferometry has the advantages of multi-fringe measurement range, inherent
directional sensitivity, and fast detection speed. The development of a
common-mode rejection scheme in heterodyne interferometer [1, 9] provides a
promising solution to displacement metrology applications due to the high
rejection ratio to common-mode noise sources. Further improvements on the
interferometer performance towards picometer level sensitivity require the
knowledge of the residual noise sources and the contribution of each noise
source.

Efforts have been made to identify noise sources in heterodyne interferometry
[10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24]. The inherent
periodic error in heterodyne interferometry is mitigated by spatially
separating the beams to reduce frequency and polarization mixing [10, 11]. The
effect of non-linear optical path difference (OPD) noise caused by electric
sideband interference is investigated in [12, 13], where its origin and
mitigation method by active feedback control are introduced. The laser
frequency, as the “ruler” of interferometric measurements, has been studied in
terms of noise behaviors [14] and various stabilization methods [15, 16, 17,
18]. Photoreceivers are characterized in [19], where the predominant noise
sources are identified, and a photodiode with low current noise is designed.
The effects of temperature fluctuations in a low frequency regime are analyzed
in [20, 21]. Other common noise sources in DMI, such as environmental noises
and phasemeter noises, are also identified in previous work [22, 23, 24].
However, an inclusive post-processing algorithm to characterize and correct
multiple noise sources in the system has not yet been published.

In this article, we present a noise identification and mitigation algorithm
that can be applied to general heterodyne interferometers. We have tested this
in a compact laboratory benchtop interferometer. In Section II, we introduce
the interferometer design, which is built to have a high common-mode rejection
ratio and be periodic-error free. In Section III, we analyze residual noise
sources based on the differential measurement results. The contributions from
distinct noise sources including the non-linear OPD noise, laser frequency
noise, and temperature fluctuations, are identified and subtracted from the
differential measurement. The noise correction results show a noise floor of
$3.31\text{\times}{10}^{-11}\text{\,}\mathrm{m}\mathrm{/}\sqrt{\mathrm{Hz}}$
at $100\text{\,}\mathrm{mHz}$, which is enhanced by an order of magnitude from
the original differential measurement, reaching the detection system noise
limit above $1\text{\,}\mathrm{Hz}$.

##  II Compact Heterodyne Laser Interferometer

###  II.1  Design and Benchtop Prototype

Figure 1a shows the layout of the compact interferometer design. After passing
through two individual acousto-optical frequency shifters (AOFS), two laser
beams are shifted by frequencies $\delta f_{1}$ and $\delta f_{2}$,
respectively. The heterodyne frequency $f_{\text{het}}$ is the difference
between the two laser frequencies, where $f_{\text{het}}=\delta f_{1}-\delta
f_{2}$. The two beams enter a 50/50 lateral beam splitter (LBS), generating
four parallel beams. All four beams propagate through a polarizing beam
splitter (PBS) and a quarter-wave plate (QWP) towards the mirrors on the
measurement end. They are then reflected back into the PBS, exiting along an
orthogonal direction. In the second LBS, two beam pairs interfere and are
detected by photodetectors PD${}_{\text{R}}$ and PD${}_{\text{M}}$ ,
respectively. The electric fields of the four beams at the interferometer
output are given by

$\displaystyle E_{1}$ $\displaystyle=E_{0}\exp\left[i2\pi(f+\delta
f_{1})t+\phi_{\text{n}}(0,\delta_{\text{y}})+\Delta\phi_{\text{M}}\right],$
(1) $\displaystyle E_{2}$ $\displaystyle=E_{0}\exp\left[i2\pi(f+\delta
f_{1})t+\phi_{\text{n}}(\delta_{\text{x}},\delta_{\text{y}})+\Delta\phi_{\text{R}}\right],$
(2) $\displaystyle E_{3}$ $\displaystyle=E_{0}\exp\left[i2\pi(f+\delta
f_{2})t+\phi_{\text{n}}(0,0)\right],$ (3) $\displaystyle E_{4}$
$\displaystyle=E_{0}\exp\left[i2\pi(f+\delta
f_{2})t+\phi_{\text{n}}(\delta_{\text{x}},0)\right],$ (4)

where $E_{0}$ is the nominal amplitude of the electric field. The term
$\phi_{\text{n}}(x,y)$ denotes the structual noises that imprints on the beam
optical pathlengths while propagating through the optical components from the
entry plane to the plane defined by the mirror $M$. This noise term
$\phi_{\text{n}}$ is dependent on the beam path (optical axis) position, which
is represented by a coordinate $(x,y)$ in the plane perpendicular to the beam
entry directions, as shown in Figure 1a. The phase terms
$\Delta\phi_{\text{M}}$ and $\Delta\phi_{\text{R}}$ represent the additional
contributions from the optical pathlength differences between mirrors $M$ and
$M_{\text{M}}$, as well as between $M$ and $M_{\text{R}}$. As depicted in
Figure 1b, two beams with the heterodyne frequency difference ($E_{1}$ and
$E_{3}$, $E_{2}$ and $E_{4}$) are combined into one beam pair.

Figure 1b shows a top view of the interferometer. Optical components can be
bonded or cemented together to reduce the overall footprint, increase the
system stability, and simplify the alignment process. In this design, two
interferometers are constructed: (a) the measurement interferometer (MIFO)
that measures the OPD between the target mirror M${}_{\text{M}}$ and the fixed
mirror M, where
$\phi_{\text{M}}=\phi_{\text{n}}(0,\delta_{\text{y}})-\phi_{\text{n}}(0,0)+\Delta\phi_{\text{M}}=\phi_{\text{n}}(0,\delta_{\text{y}})+\Delta\phi_{\text{M}}$;
(b) the reference interferometer (RIFO) that measures the OPD between the
reference mirror M${}_{\text{R}}$ and the fixed mirror M, where
$\phi_{\text{R}}=\phi_{\text{n}}(\delta_{\text{x}},\delta_{\text{y}})-\phi_{\text{n}}(\delta_{\text{x}},0)+\Delta\phi_{\text{R}}=\phi_{\text{n}}(0,\delta_{\text{y}})+\Delta\phi_{\text{R}}$.
The mirror M is affixed to the same base plate with the interferometer to
provide a common reference for MIFO and RIFO. The mirror M${}_{\text{R}}$ is
mounted as closely as possible to M${}_{\text{M}}$ to provide a local
reference for target displacement. The irradiance signal detected by the
photodetectors PD${}_{\text{R}}$ and PD${}_{\text{M}}$ are described by

$\displaystyle I_{\text{R}}$ $\displaystyle=I_{\text{{R0}}}+A\cos\left(2\pi
f_{\text{het}}t+\phi_{\text{n}}(0,\delta_{\text{y}})+\Delta\phi_{\text{R}}\right),$
(5) $\displaystyle I_{\text{M}}$
$\displaystyle=I_{\text{{M0}}}+B\cos\left(2\pi
f_{\text{het}}t+\phi_{\text{n}}(0,\delta_{\text{y}})+\Delta\phi_{\text{M}}\right).$
(6)

The phases $\phi_{\text{M}}$ and $\phi_{\text{R}}$ can be extracted by various
methods such as phase-locked loops (PLL) [25] or discrete Fourier transforms
[1]. By taking the difference between individual phase readouts, the target
displacement $d$ is calculated by

$\displaystyle d$ $\displaystyle=\frac{\Delta\phi}{2\pi\cdot
2}\lambda=\frac{\phi_{\text{R}}-\phi_{\text{M}}}{2\pi\cdot
2}\lambda=\frac{\Delta\phi_{\text{R}}-\Delta\phi_{\text{M}}}{2\pi\cdot
2}\lambda,$ (7)

where $\Delta\phi$ is the differential phase readout, and $\lambda$ is the
source wavelength. This is referred to as the common-mode rejection scheme,
where the noises in the common-mode optical paths
$\phi_{\text{n}}(0,\delta_{\text{y}})$ are canceled with the presence of a
common reference provided by mirror M.

|
---|---
(a) | (b)
Figure 1: (a) The layout of the compact benchtop heterodyne laser
interferometer design; (b) top view of the layout. Two laser beams with
different frequencies enter the lateral beam splitter (LBS) and split into
four beams, constructing two interferometers measuring interferometer (MIFO)
and reference interferometer (RIFO). The MIFO is to measure the optical
pathlength difference (OPD), $\Delta\phi_{\text{M}}$ and the RIFO is to
measure $\Delta\phi_{\text{R}}$. The actual displacement of test mass
M${}_{\text{M}}$ is calculated from the differential measurement between MIFO
and RIFO.

We built a benchtop prototype with commercial optical components and
mechanical mounting parts, as shown in Figure 2. Optical components are
clamped together mechanically in this assembly to demonstrate the design
concept. The laser source is a tunable diode laser (NP Photonics) operating at
a nominal wavelength of $1064\text{\,}\mathrm{nm}$. Two AOFS (G&H
T-M150-0.4C2G-3-F2P) shift the frequencies upwards by
$150\text{\,}\mathrm{MHz}$ and $145\text{\,}\mathrm{MHz}$, respectively, in
two paths, generating a $5\text{\,}\mathrm{MHz}$ heterodyne frequency. A
commercial phasemeter (Liquid Instruments Moku:Lab) is used to extract the
phase from the heterodyne signal using a PLL algorithm with a sampling
frequency of $30.5\text{\,}\mathrm{Hz}$. All fibers in the system are
polarization-maintaining fibers (PMF) to preserve the interference visibility.

Figure 2: Layout of the benchtop compact heterodyne laser interferometer based
on the design depicted in Figure 1. Only one static mirror M is used to
characterize the system’s noise floor. Commercial optical and mechanical
components are used to construct the benchtop system.

###  II.2 Operation Environments

The benchtop system is tested in the vacuum chamber with a pressure of
$3\text{\times}{10}^{-5}\text{\,}\leavevmode\nobreak\ $Torr to reduce the
effects from refractive index fluctuation, acoustic noises, and air
turbulence. The laser source is coupled into the system by a fiber feedthrough
port, while the optical signal is detected after transmitting through an
optical window port. Figure 3 shows the operation environment for the
preliminary test. A pendulum platform consisting of a breadboard and four
stainless steel wires is set inside the chamber for vibration isolation above
its resonance. The first resonance mode of the pendulum platform has a
frequency of $305\text{\,}\mathrm{mHz}$ measured by a ring-down test. Viton
strips are inserted between the frame of the pendulum platform and the base of
the vacuum chamber to mitigate thermal conduction. In addition, a thermal
insulation box is built outside the chamber with Styrofoam to reduce thermal
coupling from the ambient environment.

|
---|---
(a) | (b)
Figure 3: Operation environments. (a) Pendulum platform inside the chamber;
(b) styrofoam thermal insulator outside the chamber. The compact
interferometer is placed on a pendulum stage consisting of a breadboard and
four steel wires. Viton strips are applied between the pendulum frame and the
vacuum chamber to reduce ambient thermal coupling. The entire pendulum
structure is fixed inside the vacuum chamber.

###  II.3 Preliminary Test

To evaluate the noise floor of the interferometer, we utilized a single static
mirror instead of using individual fixed, reference, and test mirrors. Figure
4 shows the measurement results for MIFO and RIFO, as well as their
difference, which represents the sensitivity level of the overall
interferometer. An eight-hour measurement is taken in the operation
environment described in Section II.2. Figure 4a, the logarithmic average of
the linear spectral density (LSD) plot, shows that the individual
interferometers MIFO and RIFO achieve a sensitivity level of
$1.64\text{\times}{10}^{-7}\text{\,}\mathrm{m}\mathrm{/}\sqrt{\mathrm{Hz}}$ at
$100\text{\,}\mathrm{mHz}$. The traces of MIFO and RIFO overlap highly due to
the common paths shared between the two interferometers. With the common-mode
rejection scheme, the sensitivity is enhanced to the level of
$2.76\text{\times}{10}^{-10}\text{\,}\mathrm{m}\mathrm{/}\sqrt{\mathrm{Hz}}$
at $100\text{\,}\mathrm{mHz}$. Figure 4b shows the time series of the measured
displacement for the individual interferometers and the differential
measurement, which is measured to be
$7.56\text{\times}{10}^{-10}\text{\,}\mathrm{m}$ over the duration of 12 min.

|
---|---
(a) | (b)
Figure 4: Preliminary test results. (a) LSD and its logarithmic average of the
MIFO, RIFO, and differential measurements, respectively; (b) time series of a
12-min section of the MIFO, RIFO, and differential measurements. The traces of
measurement results from MIFO and RIFO highly overlap in (a). Experimental
results show a noise floor of
$1.64\text{\times}{10}^{-7}\text{\,}\mathrm{m}\mathrm{/}\sqrt{\mathrm{Hz}}$ at
$100\text{\,}\mathrm{mHz}$ for individual interferometers, and a noise floor
of
$2.76\text{\times}{10}^{-10}\text{\,}\mathrm{m}\mathrm{/}\sqrt{\mathrm{Hz}}$
at $100\text{\,}\mathrm{mHz}$ for the differential measurement, which is
enhanced by three orders of magnitude from individual interferometers. The
drift for differential measurement is
$7.56\text{\times}{10}^{-10}\text{\,}\mathrm{m}$ over a period of 12 min.

##  III Noise Source Characterization and Suppression

The preliminary test demonstrated that the benchtop compact interferometer
reduces the interferometer noise floor by rejecting common-mode pathlength
noises, thus enhancing the sensitivity level by three orders of magnitude. The
residual noise sources in the differential measurement are investigated in
this section to determine their contribution to the overall noise floor. The
interferometer performance is further improved by combining a linear fit
method [26, 27, 28] with a spectral analysis method.

###  III.1 Non-Linear OPD Noise

The noise investigation of the LISA Pathfinder interferometer demonstrated
that the electromagnetic coupling of the radio-frequency (RF) signals between
two AOFS drivers, generates sidebands that imprint on the optical signals [12,
13]. Figure 5 shows the measured RF driving signals applied on both AOFS. The
sidebands interfere with the RF signals at nominal shifting frequencies, and
lead to a heterodyne signal with the frequency equal to the interferometer’s
main heterodyne frequency $f_{\text{het}}$. Therefore, this ”ghost” signal is
detected by the photodetector (PD) and mixed with the actual displacement
readout, resulting in a non-linear noise that is dependent on the individual
phase readouts $\phi_{\text{M}}$ and $\phi_{\text{R}}$.

Figure 5: Spectra of RF driving signals for both AOFS. The frequency interval
between this sideband and the main peak equals to the heterodyne frequency,
leading to a ghost signal of unstable phase responsible for the non-linear OPD
noise.

Based on the analysis presented in references [12, 13], the non-linear OPD
noise caused by the sideband interference can be expressed as

$\displaystyle\delta\phi_{\text{OPD}}=$ (8)
$\displaystyle\left(C_{1}\cos\left(\frac{\phi_{\text{M}}+\phi_{\text{R}}}{2}\right)+C_{2}\sin\left(\frac{\phi_{\text{M}}+\phi_{\text{R}}}{2}\right)\right)\cdot\sin\left(\frac{\phi_{\text{M}}-\phi_{\text{R}}}{2}\right)$
$\displaystyle+\left(C_{3}\cos\left(\phi_{\text{M}}+\phi_{\text{R}}\right)+C_{4}\sin\left(\phi_{\text{M}}+\phi_{\text{R}}\right)\right)\cdot\sin\left(\phi_{\text{M}}-\phi_{\text{R}}\right).$

Equation (8) can be reformatted as

$\displaystyle\delta\phi_{\text{OPD}}$
$\displaystyle=\pmb{C}\cdot\pmb{N}(\phi_{\text{M}},\phi_{\text{R}})$ (9)
$\displaystyle=[C_{1},C_{2},C_{3},C_{4}]\cdot\begin{bmatrix}\cos\left(\frac{\phi_{\text{M}}+\phi_{\text{R}}}{2}\right)\cdot\sin\left(\frac{\phi_{\text{M}}-\phi_{\text{R}}}{2}\right)\\\
\sin\left(\frac{\phi_{\text{M}}+\phi_{\text{R}}}{2}\right)\cdot\sin\left(\frac{\phi_{\text{M}}-\phi_{\text{R}}}{2}\right)\\\
\cos\left(\phi_{\text{M}}+\phi_{\text{R}}\right)\cdot\sin\left(\phi_{\text{M}}-\phi_{\text{R}}\right)\\\
\sin\left(\phi_{\text{M}}+\phi_{\text{R}}\right)\cdot\sin\left(\phi_{\text{M}}-\phi_{\text{R}}\right)\end{bmatrix},$

where $\pmb{C}$ is the coupling coefficient vector, and $\pmb{N}$ is the non-
linear OPD term vector in the phase measurement. A linear fit is then
performed between the differential phase $\Delta\phi$ and the non-linear OPD
terms to estimate the coupling factor $\pmb{\tilde{C}}$. The contribution of
the non-linear OPD noise is removed, following the equation

$\Delta\phi_{\text{OPDcorr}}=\Delta\phi-\delta\tilde{\phi}_{\text{OPD}}=\Delta\phi-\pmb{\tilde{C}}\cdot\pmb{N}(\phi_{\text{M}},\phi_{\text{R}}).$
(10)

Figure 6 shows the LSD for the original differential measurement results and
the results after subtracting the non-linear OPD noise. Applying this noise
correction leads to a reduction in the noise floor from
$2.76\text{\times}{10}^{-10}\text{\,}\mathrm{m}\mathrm{/}\sqrt{\mathrm{Hz}}$
to
$3.86\text{\times}{10}^{-11}\text{\,}\mathrm{m}\mathrm{/}\sqrt{\mathrm{Hz}}$
at $100\text{\,}\mathrm{mHz}$. Table 1 lists the fitted coefficients $\pmb{C}$
and estimated errors $\pmb{\delta C}$ during the linear fit process.

Figure 6: LSD logarithmic average of original differential measurements and the results after noise correction, and the OPD noise contribution. The noise floor is reduced from $2.76\text{\times}{10}^{-10}\text{\,}\mathrm{m}\mathrm{/}\sqrt{\mathrm{Hz}}$ to $3.86\text{\times}{10}^{-11}\text{\,}\mathrm{m}\mathrm{/}\sqrt{\mathrm{Hz}}$ at $100\text{\,}\mathrm{mHz}$ after applying the noise correction algorithm. Table 1: Linear fitting coefficients and estimated fitting errors. $\pmb{i}$ | 1 | 2 | 3 | 4
---|---|---|---|---
$\pmb{C}_{i}$ | $3.62\times 10^{-3}$ | $1.11\times 10^{-3}$ | $1.89\times 10^{-6}$ | $-2.51\times 10^{-6}$
$\pmb{\delta C}_{i}$ | $1.96\times 10^{-6}$ | $1.96\times 10^{-6}$ | $1.23\times 10^{-6}$ | $1.24\times 10^{-6}$

###  III.2 Laser Frequency Noise

The traceability of interferometric measurements is dependent on the accurate
knowledge of the laser source frequency. Therefore, fluctuations of the laser
frequency in an interferometer with unequal armlengths directly impact the
interferometer phase following the equation

$\delta\phi_{\text{freq}}=\frac{2\pi\Delta L}{c}\delta\nu,$ (11)

where $\Delta L$ is the pathlength difference between two arms of the
interferometer, and $\delta\nu$ is the laser frequency fluctuation. For the
proposed interferometer, the phase readout is the differential measurement of
two individual interferometers MIFO and RIFO. The contribution of laser
frequency noise to the measurement can be expressed by

$\delta\phi_{\text{freq}}=\delta\phi_{\text{{fM}}}-\delta\phi_{\text{{fR}}}=\frac{2\pi\delta\nu}{c}\left[(L_{\text{{M1}}}-L_{\text{{R1}}})+(L_{\text{{M2}}}-L_{\text{{R2}}})\right],$
(12)

where $L_{\text{{Mi}}}$ is the length of the $i\text{\tiny th}$ arm in MIFO,
and $L_{\text{{Ri}}}$ is the length of the $i\text{\tiny th}$ arm in RIFO. For
the nominal design, $L_{\text{{M1}}}$ is equal to $L_{\text{{R1}}}$ due to the
common reference surface provided by the fixed mirror $M$. Therefore, the
magnitude of the laser frequency noise is determined by the axial distance
between the reference mirror $M_{\text{R}}$ and the measurement mirror
$M_{\text{M}}$ that is mounted on the test mass. In the preliminary test, only
one static mirror is used, which means both terms
$(L_{\text{{M1}}}-L_{\text{{R1}}})$ and $(L_{\text{{M2}}}-L_{\text{{R2}}})$
are nominally canceled. However, residual arm length differences remain due to
the manufacturing tolerance as well as the imperfect alignment. To evaluate
the magnitude of the residual laser frequency noise in our benchtop system, a
fiber-based delay-line interferometer (DIFO) is built and integrated with the
prototype interferometer as shown in Figure 7.

Figure 7: System layout with the delay-line interferometer integrated inside
the vacuum chamber. The delay-line interferometer is constructed by inserting
a 2-m fiber in one arm of the fiber Mach–Zehnder interferometer to amplify the
effects of laser frequency noise.

The DIFO is a heterodyne interferometer with an intentionally designed unequal
arm length using a 2-meter fiber in one interferometer arm. The laser
frequency fluctuation then gets amplified when coupled into the DIFO phase
readout $\phi_{\text{d}}$ according to Equation (12).

Figure 8 shows the linear spectra of the interferometers when injecting a
$2\text{\,}\mathrm{Hz}$ modulation to the laser frequency. The spectrum of
DIFO at $2\text{\,}\mathrm{Hz}$ is
$4.82\text{\times}{10}^{-7}\text{\,}\mathrm{m}$, while the spectrum of RIFO is
$5.30\text{\times}{10}^{-10}\text{\,}\mathrm{m}$. It can also be noted from
Figure 8 that, aside from the injected laser frequency modulation peak and its
harmonics, the noise floor of DIFO is almost identical to RIFO due to
predominantly common environmental noises. This makes it difficult to extract
any excess laser frequency noise from the DIFO readout. Therefore, we first
subtract the RIFO phase $\phi_{\text{R}}$ from DIFO phase $\phi_{\text{D}}$,
to mitigate the common path noise effects. The residual phase,
$\phi_{\text{D}}^{\prime}=\phi_{\text{D}}-\phi_{\text{M}}$, is used to
estimate the noise coupling factor. Moreover, the residual phase is band-
passed in the frequency regime between $1\text{\,}\mathrm{mHz}$ and
$1\text{\,}\mathrm{Hz}$ to mitigate temperature fluctuation and mechanical
vibration effects.

Figure 8: Linear spectra of DIFO, RIFO, and the differential measurement to
the injected laser frequency noise by intentionally modulating the laser
frequency at a rate of $2\text{\,}\mathrm{Hz}$. The spectrum response of DIFO
to the laser frequency noise is
$4.82\text{\times}{10}^{-7}\text{\,}\mathrm{m}$ while the RIFO response is
$5.30\text{\times}{10}^{-10}\text{\,}\mathrm{m}$ at $2\text{\,}\mathrm{Hz}$.
After performing the differential operation, the spectrum response is reduced
to $5.07\text{\times}{10}^{-12}\text{\,}\mathrm{m}$ at
$2\text{\,}\mathrm{Hz}$.

We perform a linear fit of the band-passed phase, $\phi_{\text{D}}^{\prime}$,
to the differential phase, to estimate the coupling coefficient $\tilde{K}$.
The noise correction procedure, which is applied to the original data, can be
described as

$\Delta\phi_{\text{freqcorr}}=\Delta\phi-\delta\tilde{\phi}_{\text{freq}}=\Delta\phi-\tilde{K}\cdot\phi_{\text{D}}^{\prime}.$
(13)

Figure 9 shows the results of identifying the laser frequency noise
contribution based on the OPD-noise-corrected differential phase described in
Section III.1. The laser frequency contribution is identified to be
$2.28\text{\times}{10}^{-12}\text{\,}\mathrm{m}\mathrm{/}\sqrt{\mathrm{Hz}}$
at $100\text{\,}\mathrm{mHz}$ in this case, and is not the dominant noise
source in this measurement. We injected a $2\text{\,}\mathrm{Hz}$ modulation
to the laser frequency in order to independently determine its coupling factor
and the effective optical arm length mismatch between the interferometers. The
results are shown in Figure 8, and we obtain an effective arm length mismatch,
$\Delta L$, of $2.31\text{\times}{10}^{-5}\text{\,}\mathrm{m}$. From Equation
(11), it is inferred that the effects of laser frequency fluctuations are
suppressed by a small $\Delta L$, and therefore is not a limiting factor for
the current experimental setup. By substituting $\Delta L$ into Equation (11),
the laser frequency fluctuations are estimated to be
$3.47\text{\times}{10}^{8}\text{\,}\mathrm{Hz}\mathrm{/}\sqrt{\mathrm{Hz}}$ at
$1\text{\,}\mathrm{mHz}$.

Figure 9: Laser frequency noise contribution estimated by a linear fit,
compared to the OPD-noise-corrected differential measurement. The laser
frequency noise is estimated to be
$2.28\text{\times}{10}^{-12}\text{\,}\mathrm{m}\mathrm{/}\sqrt{\mathrm{Hz}}$
at $100\text{\,}\mathrm{mHz}$, and is not the limiting noise source in our
benchtop experiment.

###  III.3 Temperature Fluctuation Noise

Temperature fluctuations lead to OPD variations due to changes in the
refractive index of the air, and thermo-elastic distortions of the optical
elements and mounts. Temperature fluctuations usually limit the sensitivity of
an interferometer in the frequency regime below 1 mHz [20, 21]. Efforts such
as inserting Viton strips and building Styrofoam boxes have been made to
mitigate their effects to the operational environment during the experiment.
To evaluate the residual thermal contributions to the differential
measurements, ambient temperature is monitored by a temperature sensor
attached to the outer wall of the vacuum chamber.

In contrast to the two noise sources introduced above, thermal fluctuation
effects are recorded with a delay in the optical readout. In other words, the
group phase between the temperature measurement and the actual effect on the
differential interferometer must be considered in the analysis. Therefore,
when analyzing temperature fluctuations, a spectral analysis method [20] is
adopted instead of a time-domain linear fit.

Before applying the spectral analysis method, we first evaluate the
correlation between the displacement and the temperature measurement by
calculating the coherence between them. Figure 10 shows the amplitude and
phase of the calculated coherence, and we observe a strong correlation
(amplitude larger than 0.5) at frequencies below 1 mHz, while a non-zero phase
is observed in the same frequency bandwidth. In our case, this shows that the
major limitation presented by temperature fluctuations to our interferometer
performance occurs in the sub-mHz regime and has a delay effect on the
displacement measurement.

Figure 10: The amplitude and phase of the coherence function between the
displacement and the temperature measurements. In the low frequency regime
below 1 mHz, it shows a strong correlation between the displacement and the
temperature measurement as the amplitude is large than 0.5, and a time delay
effect as the phase is nonzero.

In this spectral analysis method, the transfer function between the
temperature and the differential measurement is estimated by the equation

$H(\omega)=\frac{S_{\text{T}\phi}(\omega)}{S_{\text{TT}}(\omega)},$ (14)

where $S_{\text{T}\phi}(\omega)$ is the cross power spectral density of
temperature to differential phase, and $S_{\text{TT}}(\omega)$ is the power
spectral density of the temperature measurement. Once the transfer function is
determined, the contribution of the temperature fluctuation can be removed
from the differential measurement in the spectral domain following

$\Delta\phi_{\text{tempcorr}}(\omega)=\Delta\phi(\omega)-H(\omega)\cdot
T(\omega).$ (15)

Figure 11 shows the amplitude of the calculated transfer function between the
displacement and the temperature measurement. From Figure 10, the correlation
in the high frequency bandwidth above 1 mHz is negligible. Hence, directly
applying the transfer function in Equation (15) will add noises in the
uncorrelated frequency regime. A low-pass spectral filter is applied to the
transfer function in order to remove its contribution, which is also plotted
in Figure 11. The cut-off frequency of this low-pass filter is set to be 1 mHz
based on the coherence results.

Figure 11: Transfer function between the displacement and temperature
measurements before and after applying a 1 mHz low-pass filter. The amplitude
of the transfer function is attenuated by a low-pass filter in the frequency
regime where the correlation is negligible.

Figure 12 shows the logarithmic averaged LSD of the OPD-noise-corrected
differential phase and the temperature correction results based on the
spectral analysis method described in Equations (14) and (15). Temperature
fluctuations clearly dominate in the low frequency regime below 0.5 mHz. After
subtracting this noise contribution, the noise floor at 0.1 mHz is reduced
from
$6.5\text{\times}{10}^{-9}\text{\,}\mathrm{m}\mathrm{/}\sqrt{\mathrm{Hz}}$ to
$9.8\text{\times}{10}^{-11}\text{\,}\mathrm{m}\mathrm{/}\sqrt{\mathrm{Hz}}$.

Figure 12: Contributions of temperature fluctuations estimated by a spectral
analysis method, and the measurement results before and after correction of
this noise source. The noise floor is reduced from
$6.5\text{\times}{10}^{-9}\text{\,}\mathrm{m}\mathrm{/}\sqrt{\mathrm{Hz}}$ to
$9.8\text{\times}{10}^{-11}\text{\,}\mathrm{m}\mathrm{/}\sqrt{\mathrm{Hz}}$ at
0.1 mHz after applying the noise correction algorithm.

###  III.4 Detection System Noise Limit

The photodetector (PD) technical noise determines the minimum noise floor that
an interferometric system can achieve, aside from fundamental limits such as
shot noise. In a PD system, typical noise sources include dark current noise
from the photodiode, shot noise originating from potential barriers, Johnson
noise from thermal fluctuation in the resistor of the sensor circuits, and
capacitive noise from the photodiode and circuit [29]. While other noise
sources can be reduced by optimizing the circuit design [19], shot noise
depends exclusively on the optical power received by the PD and its
responsivity. In a heterodyne interferometer with a PLL phase readout, the
minimum detectable displacement $d_{\text{min}}$ can be derived when the
signal-to-noise ratio (SNR) of the system is equal to 1 [30].

Assuming a small displacement that can be modeled as
$A_{\text{d}}\sin(\omega_{\text{d}}t)$, the electric field of the interference
signal in a general case, is expressed by

$\displaystyle E_{\text{D}}$ $\displaystyle=E_{1}+E_{2}$ (16)
$\displaystyle=A_{1}\exp[i(\omega_{1}t+\phi_{1})]$
$\displaystyle+A_{2}\exp[i(\omega_{2}t+\phi_{2}+A_{\text{d}}\sin(\omega_{\text{d}}t))],$

where $A_{i}$ are the amplitudes of the electric fields, $\omega_{i}$ are the
optical frequencies, and $\phi_{i}$ are the phase terms of the individual
beams. The irradiance on PD then can be represented as

$\displaystyle P_{\text{D}}$ $\displaystyle=|E_{1}+E_{2}|^{2}$ (17)
$\displaystyle=|A_{1}|^{2}+|A_{2}|^{2}+2A_{1}A_{2}\cos{[\Delta\omega
t+\Delta\phi+A_{\text{d}}\sin{(\omega_{\text{d}}t)}]}.$

By applying trignometric identities and Bessel expansion to Equation 17, the
irradiance on PD is rewritten as

$\displaystyle P_{D}$
$\displaystyle=P_{1}+P_{2}+2\sqrt{P_{1}P_{2}}\Big{\\{}J_{0}(A_{\text{d}})\cos(\Delta\omega
t+\Delta\phi)$ (18)
$\displaystyle+\sum_{m=1}^{\infty}\Big{\\{}J_{2m}(A_{\text{d}})[\cos[(2m\omega_{\text{d}}t+\Delta\omega
t+\Delta\phi)$ $\displaystyle+\cos(2m\omega_{\text{d}}t-\Delta\omega
t-\Delta\phi)]\Big{\\}}$
$\displaystyle-\sum_{n=0}^{\infty}\Big{\\{}J_{2n+1}(A_{\text{d}})\big{\\{}\cos[(2n+1)\omega_{\text{d}}t+\Delta\omega
t+\Delta\phi]$ $\displaystyle-\cos[(2n+1)\omega_{\text{d}}t-\Delta\omega
t-\Delta\phi]\big{\\}}\Big{\\}}\Big{\\}},$

where $P_{i}$ is the optical power in each interferometer arm, $J_{\alpha}(x)$
is the Bessel function of the first kind. In this case, only small
displacement is considered where $A_{\text{d}}\ll 1$. Based on the small
argument approximation for Bessel function where

$J_{0}(x)=1,J_{1}(x)=\frac{x}{2},J_{2}(x)=\frac{1}{2}\frac{x^{2}}{2},$ (19)

Equation (19) can be simplified as

$\displaystyle P_{\text{D}}$
$\displaystyle=P_{1}+P_{2}+2\sqrt{P_{2}P_{2}}\\{\cos{(\Delta\omega
t+\Delta\phi)}$ (20)
$\displaystyle-\frac{A_{\text{d}}}{2}[\cos(\omega_{\text{d}}t+\Delta\omega
t+\Delta\phi)-\cos(\omega_{\text{d}}t-\Delta\omega t-\Delta\phi)]$
$\displaystyle+O(A_{\text{d}}^{2})\\}.$

Therefore, the time average of the irradiance and the time-averaged incoherent
square sum of the irradiance on the photodetector are

$\begin{split}&\langle P_{\text{D}}\rangle=P_{1}+P_{2},\\\ &\langle
P_{\text{D}}^{2}\rangle=(P_{1}+P_{2})^{2}+P_{1}P_{2}A_{\text{d}}^{2},\end{split}$
(21)

The actual displacement to be measured, $d$, is converted by the relation
$d=\lambda\cdot A_{\text{d}}/(2\pi\cdot 2)$. The PD system noise in this case
can be decomposed to shot noise based on analytical calculation, and other
noises based on experimental measurements, following the expression

$\langle i_{\text{PD}}^{2}\rangle=\langle i_{\text{shot}}^{2}\rangle+\langle
i_{\text{other}}^{2}\rangle=2e\rho\langle P_{D}\rangle
B+\frac{V_{\text{M}}^{2}}{G^{2}},$ (22)

where $\rho$ is the detector responsivity, $B$ is the detector bandwidth,
$V_{M}$ is the measured output voltage from PD without incident optical
signal, and $G$ is the transimpedance gain and is usually specified by the
manufacturer. Assuming the interferometer has a visibility of $V$, the SNR
when PD noises are present can be expressed as

$\text{SNR}=\sqrt{\langle I_{\text{D}}^{2}\rangle/\langle
i_{\text{PD}}^{2}\rangle}=k_{0}d\sqrt{\frac{4\rho^{2}P_{1}P_{2}}{2e\rho
P_{0}B+\langle V_{\text{M}}^{2}\rangle/G^{2}}},$ (23)

where $k_{0}$ is the wavenumber. When SNR is equal to 1, the minimum
detectable displacement is calculated as

$d_{\text{min}}=\frac{\sqrt{2e\rho P_{0}+\langle
V_{\text{M}}^{2}\rangle/(G^{2}\cdot B)}}{k\rho
P_{0}V}[\text{m}/\sqrt{\text{Hz}}]\cdot\sqrt{B}.$ (24)

For the benchtop system, the RMS of output voltage square without detecting
signal is $\langle
V_{\text{M}}^{2}\rangle=$1.29\text{\times}{10}^{-7}\text{\,}$[V]^{2}$, the
optical power on the detector is
$4.5\text{\times}{10}^{-5}\text{\,}\mathrm{W}$, and the interferometer
visibility is 0.45. By substituting the numerical values into Equation (24),
the minimum detectable displacement is calculated to be $d_{\text{min}}=$
$1.39\text{\times}{10}^{-12}\text{\,}\mathrm{m}\mathrm{/}\sqrt{\mathrm{Hz}}$.

###  III.5 Discussions

Figure 13: Flow chart of the overall noise correction algorithm. The non-
linear OPD noise, laser frequency noise and the temperature fluctuation noise
are characterized, and the contribution from each noise source is removed with
this algorithm.

We investigated three common noise sources for heterodyne interferometry,
including non-linear OPD noise, laser frequency noise, and temperature noise.
The correction method for each noise source is identified, and the overall
post-processing correction algorithm is described in the flow chart in Figure
13.

According to this correction algorithm, Figure 14 shows the differential
measurement results (blue trace), the overall noise correction results (red
trace), and the contribution of each noise source. The residual noise floor
after noise correction shown in Figure 14 is
$3.31\text{\times}{10}^{-11}\text{\,}\mathrm{m}\mathrm{/}\sqrt{\mathrm{Hz}}$
at $100\text{\,}\mathrm{mHz}$. It is worth mentioning that this algorithm
applies to uncorrelated noise sources in general heterodyne interferometry. If
there are correlated noise sources, for example as in the DIFO readout and the
temperature readout, the data needs to be pre-processed, before applying the
correction algorithm to determine the frequency range of their correlation.

Figure 14: Overall noise correction results based on the differential
measurement and contribution of each noise source. The residual noise floor is
$3.31\text{\times}{10}^{-11}\text{\,}\mathrm{m}\mathrm{/}\sqrt{\mathrm{Hz}}$
at $100\text{\,}\mathrm{mHz}$, and
$9.8\text{\times}{10}^{-11}\text{\,}\mathrm{m}\mathrm{/}\sqrt{\mathrm{Hz}}$ at
0.1 mHz after correction. The detection system noise level is
$1.39\text{\times}{10}^{-12}\text{\,}\mathrm{m}\mathrm{/}\sqrt{\mathrm{Hz}}$
and also marked in this plot.

This post-processing noise correction algorithm provides an effective
alternative to improve the sensitivity of heterodyne interferometers without
implementing active stabilizations and can achieve high sensitivity over long-
term measurements. Moreover, the application of the post-processing algorithm
allows us to maintain an overall compact system footprint, and minimal
complexity to the control system. The proposed interferometer and noise
correction algorithm achieve a performance that is comparable to a few other
techniques in GW applications, and has its own relative advantages, such as
low hardware complexity and simple interferometer layout.

In addition to the benchtop system we built in the laboratory, this noise
correction algorithm can be applied to general heterodyne interferometers [7,
9]. Furthermore, this algorithm makes it possible to improve the performance
of existing heterodyne optical readout systems without changing their
configurations.

##  IV Conclusions and Outlook

We have presented a noise identification and noise correction algorithm that
applies to general heterodyne interferometry. A compact heterodyne
interferometer has been developed and tested to determine its performance,
when applying this algorithm to its measurement results. We have developed a
mitigation strategy for typical noise sources that are present in heterodyne
interferometers such as non-linear OPD noise, laser frequency noise, and
temperature fluctuations. The contribution from each noise source is estimated
and subtracted from the main phase measurement. After correction, the noise
floor is reduced by an order of magnitude to a level of approximately
$3.31\text{\times}{10}^{-11}\text{\,}\mathrm{m}\mathrm{/}\sqrt{\mathrm{Hz}}$
at $100\text{\,}\mathrm{mHz}$.

There are a few modifications to be made in the future to further improve the
interferometer performance over the bandwidth above 1 mHz. On the operation
environment side, the natural frequency of the pendulum platform can be
reduced below 100 mHz to provide noise attenuation to a broader frequency
band. To mitigate the non-linear OPD noise, better management of the crosstalk
between RF signals can be implemented, such as building electromagnetic shield
for each individual AOFS RF amplifier to reduce the amplitude of the sideband
interference. On the instrument side, the interferometer design can be
optimized to minimize the common-path noise term, $\phi_{\text{n}}$ in
Equation (5), in the individual interferometers MIFO and RIFO. In addition, a
laser source with a higher optical output power will reduce the shot noise
level and, therefore, the overall interferometer noise floor at higher
frequencies. Lastly, it will be interesting to investigate possible
implementations of this algorithm using machine learning techniques and
predicting filters to achieve a real-time operation and real-time sensitivity
enhancement to laser interferometers without adding complex hardware for
active stabilizations.

### Author contributions

Conceptualization, Y.Z. and F.G.; Funding acquisition, F.G.; Investigation,
Y.Z.; Methodology, Y.Z.; Project administration, F.G.; Supervision, F.G.;
Validation, Y.Z., A.S.H. and G.V.; Writing—original draft, Y.Z.;
Writing—review and editing, A.S.H., G.V. and F.G. All authors have read and
agreed to the published version of the manuscript.

### Funding

The authors gratefully acknowledge support from the National Geospatial-
Intelligence Agency (NGA) through Grant HMA04762010016, the National Science
Foundation (NSF) through Grants PHY-2045579 and ECCS-1945832, and the Jet
Propulsion Laboratory (JPL) subcontract 1659175.

### Acknowledgements

The authors acknowledge Cristina Guzman for editing the contents of this
manuscript.

### Conflict of Interest

The authors declare no conflict of interest. The funders had no role in the
design of the study; in the collection, analyses, or interpretation of data;
in the writing of the manuscript; or in the decision to publish the results.

## References

  * Heinzel  _et al._ [2004] G. Heinzel _et al._ , Class. Quant. Grav. 21, S581 (2004).
  * Wand  _et al._ [2006] V. Wand, F. Guzmán, G. Heinzel, and K. Danzmann, Laser Interferometer Space Antenna: 6th International LISA Symposium American Institute of Physics Conference Series, 873, 689 (2006).
  * Aston and Speake [2006] S. M. Aston and C. C. Speake, Laser Interferometer Space Antenna: 6th International LISA Symposium American Institute of Physics Conference Series, 873, 326 (2006).
  * Schuldt  _et al._ [2006] T. Schuldt, H.-J. Kraus, D. Weise, C. Braxmaier, A. Peters, and U. Johann, Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series, 6293, 62930Z (2006).
  * Armano  _et al._ [2009] M. Armano _et al._ , Class. Quant. Grav. 26, 094001 (2009).
  * Abich  _et al._ [2019] K. Abich _et al._ , Phys. Rev. Lett. 123, 031101 (2019).
  * Hines  _et al._ [2020] A. Hines, L. Richardson, H. Wisniewski, and F. Guzman, Appl. Opt. 59, G167 (2020).
  * Robertson  _et al._ [1982] N. Robertson, R. Drever, I. Kerr, and J. Hough, Journal of Physics E: Scientific Instruments 15, 1101 (1982).
  * Joo  _et al._ [2020] K.-N. Joo, E. Clark, Y. Zhang, J. D. Ellis, and F. Guzmán, Journal of the Optical Society of America A 37, B11 (2020).
  * Wu and Deslattes [1998] C.-M. Wu and R. Deslattes, ”Appl. Opt.” 37, 6696 (1998).
  * Joo  _et al._ [2009] K.-N. Joo, J. D. Ellis, J. W. Spronck, P. J. van Kan, and R. H. Schmidt, Optics Letters 34, 386 (2009).
  * Wand  _et al._ [2006] V. Wand _et al._ , Class. Quant. Grav. 23, S159 (2006).
  * Heinzel  _et al._ [2008] G. Heinzel, V. Wand, A. Garcia, F. Guzman, F. Steier, C. Killow, D. L. Robertson, and H. Ward (2008).
  * Salvadé and Dändliker [2000] Y. Salvadé and R. Dändliker, J. Opt. Soc. Am. A 17, 927 (2000).
  * Drever  _et al._ [1983] R. W. P. Drever, J. L. Hall, F. V. Kowalski, J. Hough, G. M. Ford, A. J. Munley, and H. Ward, Appl. Phys. B 31, 97 (1983).
  * Supplee  _et al._ [1994] J. M. Supplee, E. A. Whittaker, and W. Lenth, ”Appl. Opt.” 33, 6294 (1994).
  * Numata  _et al._ [2019] K. Numata, A. W. Yu, H. Jiao, S. A. Merritt, F. Micalizzi, M. E. Fahey, J. B. Camp, and M. A. Krainak, in  _Solid State Lasers XXVIII: Technology and Devices_ , Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series, Vol. 10896 (2019) p. 108961H.
  * Ellis  _et al._ [2010] J. D. Ellis, K.-N. Joo, E. S. Buice, and J. W. Spronck, Optics Express 18, 1373 (2010).
  * Guzman Cervantes  _et al._ [2011] F. Guzman Cervantes, J. Livas, R. Silverberg, E. Buchanan, and R. Stebbins, Class. Quant. Grav. 28, 094010 (2011).
  * Nofrarias  _et al._ [2013] M. Nofrarias, F. Gibert, N. Karnesis, A. F. Garcia, M. Hewitson, G. Heinzel, and K. Danzmann, Phys. Rev. D 87, 102003 (2013).
  * Gibert  _et al._ [2015] F. Gibert _et al._ , Class. Quant. Grav. 32, 045014 (2015).
  * Badami and de Groot [2013] V. Badami and P. de Groot, in  _Handbook of Optical Dimensional Metrology_ (Taylor & Francis, Boca Raton, USA, 2013) 3rd ed., pp. 178–198.
  * Gerberding  _et al._ [2015] O. Gerberding, C. Diekmann, J. Kullmann, M. Tröbs, I. Bykov, S. Barke, N. C. Brause, J. J. Esteban Delgado, T. S. Schwarze, J. Reiche, K. Danzmann, T. Rasmussen, T. V. Hansen, A. Enggaard, S. M. Pedersen, O. Jennrich, M. Suess, Z. Sodnik, and G. Heinzel, Review of Scientific Instruments 86, 074501 (2015).
  * [24] J. D. Ellis, SPIE .
  * Gardner [2005] F. M. Gardner,  _Phaselock Techniques_ , 3rd ed. (John Wiley & Sons, Inc., Hoboken, USA, 2005).
  * Cervantes  _et al._ [2008] F. G. Cervantes, F. Steier, G. Wanner, G. Heinzel, and K. Danzmann, Appl. Phys. B 90, 395 (2008).
  * Marquardt [1963] D. W. Marquardt, Journal of the Society for Industrial and Applied Mathematics 11, 431 (1963).
  * Guzman [2009] F. Guzman, Ph.D. Thesis, Max Planck Institute for Gravitational Physics & Gottfried Wilhelm Leibniz Universität Hannover, Hannover, Germany (2009).
  * Watchi  _et al._ [2018] J. Watchi, S. Cooper, B. Ding, C. M. Mow-Lowry, and C. Collette, Review of Scientific Instruments 89, 121501 (2018).
  * Dehne  _et al._ [2012] M. Dehne, M. Tröbs, G. Heinzel, and K. Danzmann, Opt. Express 20, 27273 (2012).


# Exploring the Long-Term Generalisation
of Counting Behaviour in RNNs

Nadine El-Naggar
City, University of London / United Kingdom
Affiliation / Address line 2
Affiliation / Address line 3
<EMAIL_ADDRESS>
&Pranava Madhyastha
City, University of London / United Kingdom
Affiliation / Address line 2
Affiliation / Address line 3
<EMAIL_ADDRESS>
&Tillman Weyde
City, University of London / United Kingdom
Affiliation / Address line 2
Affiliation / Address line 3
<EMAIL_ADDRESS>

###### Abstract

In this study, we investigate the generalisation capability of LSTM, GRU and
ReLU models on counting tasks over long sequences. Previous theoretical work
established that RNNs with ReLU activation and LSTMs are capable of counting
with suitable configuration, while GRUs have limitations that prevent correct
counting over longer sequences. Despite this and some positive empirical
results for LSTMs on Dyck-1 languages, our experimental results show that
LSTMs fail to learn correct counting behaviour for sequences that are
significantly longer than in the training data. RNNs behave similarly on
longer sequences and in addition often fail in training. The long sequence
performance is empirically and theoretically related to training and
validation losses, but a loss of zero is not practically achievable through
backpropagation. Because of their design LSTMs, GRUs and ReLUs have different
failure modes, which we illustrate with specifically designed sequences. In
summary, none of the models learns generalisable counting behaviour and we
discuss consequences and potential solutions for further research.

##  1 sequence of paper

  * •

(generalisation to long sequences is not good)

  * •

show that training loss correlates with FPF

  * •

does additional training lead to higehr FPF

  * •

indicators (explanation and why models dont work indefinitely due to lack of
saturation)

  * •

same for RNNs (histograms of indicators)

  * •

explain why no LSTMs reach zero point on time due to lack of saturation, hence
earlier failure

  * •

Failure modes

  * •

future work (discretisation of weihgts)

  * •

why and how recurrent networks fail to count in the long run?

###  1.1 to do

  * •

explain that generalisation may not be reliably possible to infinity if we
keep training (not practical, and maybe the gradient can be rounded down to 0
before we reach saturation)

  * •

correlation and linear regression for trained models (ReLU), best 200 by
validation loss value.

  * •

Find the start and end of the depth plots (starting from 0 or 1 etc).

  * •

WRITE THE TEXT!!!

  * •

check where the lowest loss is in the LSTM models to see if more training is
actually beneficial.

  * •

RNN and GRU models on zigzag, make a graph similar to the failure point one
like yesterday from LSTM. DONE

  * •

create depth plots comparing the activation (counter value) from a network to
the sequence depth plot to further clarify the failure modes

  * •

calculate train val loss for the table 1 (for all models that need it, LSTM,
GRU and remaining ReLU models) DONE

  * •

replace acc in 1000-token sequences with FPF (min, max, avg of per-model
average) DONE

  * •

run the inference on new 1000-token dataset  DONE

  * •

generate 50-100 sequences of 1000 tokens DONE - GENERATED 119 TOTAL, shortened
it to 100

  * •

##  2 Introduction

Recurrent Neural Networks (RNNs) have been the go-to NN model for sequential
tasks since the 1980s and have been popular in recent years for a variety of
applications, (see e.g., karpathy2015unreasonable), including Natural Language
Processing (NLP) before the introduction of Transformers. RNNs are highly
expressive, as they are Turing complete DBLP:journals/jcss/SiegelmannS95 and
capable of universal approximation (funahashi1992neural) from multi-layer
perceptrons . The traditional Elman RNNs have a tanh activation function
elman1990finding. Networks with sigmoid activation functions (such as tanh and
logistic function), suffer from the vanishing gradient problem during training
bengio1994learning. Later, variants using a ReLU activation function
DBLP:journals/corr/LeJH15.

Recent work by suzgun2019lstm and DBLP:conf/acl/WeissGY18 follows in the
footsteps of DBLP:journals/tnn/GersS01 to evaluate the ability of RNN models
to learn formal languages. DBLP:conf/acl/WeissGY18 have shown that LSTM and
ReLU networks have the capatcity to count and model certain formal languages
exactly, where LSTMs need to saturate some functions to achieve this. ReLUs
are reported to be difficult to train, while some success on $a^{n}b^{n}$ and
similar languages. DBLP:conf/acl/WeissGY18 also showed that GRUs and Elman
RNNs don’t have this capability. suzgun2019lstm report that LSTMs can learn
counting behaviour to recognise Dyck-1 languages with 100% accuracy in their
experiments. However, whether the counting by LSTMs is precise enough for long
sequences to what extent ReLUs and GRUs can learn counting on long sequences
remains unclear. Therefore, we investigate the limits of to trained ReLU, LSTM
and GRU models in generalizing Dyck-1 sequences that are longer than those in
the training set and analyse the when, how and why they fail. The specific
contributions of this study are:

  1. 1.

We show that all trained models fail to accept Dyck-1 sequences of length 1000
when trained on lengths up to 50.

  2. 2.

The training and validation loss values are predict generalisation well, but
reducing the error with longer training is not practical for achieving exact
counting.

  3. 3.

We illustrate and analyse the different failure modes of ReLU, LSTM, and GRU
cells.

##  3 Generalising Recognition of Dyck-1 Sequences

We evaluate the ability of our models to generalise to longer sequences,
focusing on Dyck-1 sequences. Dyck-1 sequences have a) one type of brackets,
b) an equal number of opening and closing brackets, and c) no closing brackets
before their corresponding opening brackets. We use the same task as
suzgun2019lstm to evaluate our models. The task is to classify which elements
are valid to follow, which is equivalent to recognising complete vs incomplete
Dyck-1 sequences. This is a limited Dyck-1 recognition task, where a sequence
is classified at every time step, but sequences that violate condition c) do
not occur. A more complete language recognition paradigm is described by
DBLP:journals/corr/abs-2004-06866.

Like suzgun2019lstm, we train our models using a dataset of 10000 sequences of
length 2 to 50 tokens. We use a validation set of 5000 sequences of length 2
to 50 tokens. We use a long test set of 5000 sequences of lengths 52 to 100
tokens. The training, validation and long test sets are generated in the same
way as suzgun2019lstm. There is no overlap between the training and validation
datasets.

We use single layer single cell LSTMs, ReLUs and GRUs in our experiments. We
use the Adam optimiser (kingma2014adam), and learning rates of 0.01 for the
ReLU and LSTM models, and 0.001 for the GRU models.

##  4 Long Sequence Generalisation

We evaluate the generalisation limits of our models that train well on short
sequences and generalise well to some longer sequences up to 100 tokens. We
use 100 sequences of 1000 tokens, which are generated like in suzgun2019lstm.

We show results in Table 1. The LSTM models reach 100% accuracy on training,
validation and the long test set, but do not classify the very long sequences
correctly. GRU models achieve good training and validation accuracy, but fail
on the long and very long sequences. The ReLU models often fail to adapt even
to the training set.

Overall, no single model cassifies all 1000 tokens in any sequences correctly.
We know that it is not possible for GRUs to perform this task. However, ReLU
and LSTM can, but the training does not lead to models that count precisely
enough in the long run.

Table 1:  Dyck-1 classification restults. Accuracy is over sequences, i.e. a sequences counts as correctly treated when all tokens are correctly classified. For the 1000-token sequences, we report the First Point of Failure (FPF). report also median accuracy, if time Model | Train | Validation | Long | 1000-token
---|---|---|---|---
LSTM | 100(100/100) | 100(100/100) | 100(100/100) | 919.92(841.21/946.36)
ReLU (all) | 20.02(0.0/100) | 19.45(0.0/100) | 9.59(0.0/100) | 756.95(167.12/902.41)
ReLU (good) | 60.06(0.05/100) | 58.35(0.0/100) | 28.76(0.0/100) | 768.77(167.12/902.41)
GRU | 97.83(78.58/100) | 97.35(73.96/99.98) | 80.92(13.62/94.3) | 469.87(409.57/677.06)

##  5 Loss and Generalisation

In practices, finite counting abilities may be sufficient and the questions
may be whether we can determine at training or validation time how well a
model may generalise, as testing on very long sequences may be expensive. We
can use the observation that the loss value achieved by the models seems to be
related to the FPF. The negative log loss turns out to be well correlated with
the FPF. We quantify this in a linear regression model and report the $r^{2}$
and p values in Table 2.

Figure LABEL:fig:plots_Correlation_zigzag shows scatter plots with the
regression lines. For the LSTMs we see that the lower loss (higher ngeative
log loss) does lead to higher FPF, but there are still many values that are
20-35% lower than the full length of the sequences.

Table 2:  This table shows the $r^{2}$ value and p value for the linear regression. We report these values for the train, validation and long test sets against the average FPF for each of our models on the 1000-token dataset.  Add 2nd row for ReLU with only good models. Replace train loss with validation loss, when results are in.  | Train | Validation | Long
---|---|---|---
Model / Dataset |  $r^{2}$/p |  $r^{2}$/p |  $r^{2}$/p
LSTM/1000-token | 0.66 / $3.83\times 10^{-28}$ | 0.63 / $9.22\times 10^{-25}$ | 0.73 / $2.12\times 10^{-36}$
ReLU/1000-token | 0.004 / 0.96 | 0.10 / 0.28 | 0.08 / 0.39
GRU/1000-token | 0.26 / 0.0001 | 0.33 / $8.78\times 10^{-7}$ | 0.31/5.91
ReLU (good models)/1000-token |  |  |

##  6 Effect of Additional Training on Loss

We examine the effect of training for additional epochs on the training loss
in LSTM models. We observe that the loss value continues to decrease when the
models are trained for longer. However, the decrease becomes much slower as
the training progresses. Theoretically, training for long periods of time may
allow the LSTM models to reach saturation and produce perfect results.
However, this is not a practical solution, as it will take a very long time
for the models to finally reach the perfect solution. Furthermore, it is
possible for the gradients to become negligible in magnitude, and get rounded
to 0 by the computer. This can result in the model not reaching the perfect
solution despite reaching a 0 gradient. DOUBLE CHECK THESE FINDINGS

  * •

training loss is decreasing in LSTM models the longer training goes on

  * •

We train LSTM models for 100 epochs and observe the effect of the longer
training on the generalisation ability of the models. For the LSTMs we observe
a negligible change in the generalisation ability of the models as a result of
the longer training.

  * •

LSTM trained for longer, but not performing better, negligible difference.

##  7 Failure Modes

We inspect the parameters of our LSTM and ReLU models to determine the causes
of the failure. For the LSTM models, DBLP:conf/acl/WeissGY18 state that
saturation of the activation functions of the LSTM result in counting. We
inspect the parameters of our LSTM models and observe that their weights and
biases do not converge to values that result in the saturation of the sigmoid
and $\tanh$ activation functions. They approach the values for saturation, but
do not achieve it. Consequently a large enough deviation accumulates between
the true depth count and the achieved depth count. This results in the models
reaching a depth value of 0 too early. XXXX ADD MORE HERE ABOUT THE
RELATIONSHIP OF THE MAX DEPTH TO THE FPF.

We investigate the differences in the failure of our models. We use a dataset
that consists of repetitions of $j$ opening brackets followed by $j$ closing
brackets for $j=\\{10,20,25,50,100,125,200,250,500,1000\\}$. All sequences in
this dataset are 2000 tokens in length, and distinct. We refer to this dataset
as the zigzag dataset. We test our models on the zigzag dataset and record the
FPF achieved for each model on each element of the dataset.

We plot the distribution of the FPF values for each sequence. We observe
different failures for each of the models, which are shown in Figure
LABEL:fig:failure_modes. The GRU models have squashed activations, and
consequently have a limited ability to count (DBLP:conf/acl/WeissGY18). This
creates a more drastic squashing of the activation value as the number of
increments increases. This causes the increment and decrement values to vary
throughout the sequence, depending on the position. As a result, an
accumulation of consecutive increments is squashed severely and a smaller
number of decrements quickly returns the count value to 0. LSTM models can
count if their sigmoid and $\tanh$ activations are saturated
(DBLP:conf/acl/WeissGY18). However, our LSTM models clearly do not saturate.
This results in a large deviation between the increment and decrement values,
causing the models to count down to 0 prematurely. This causes the majority of
models to fail before the actual point where the count is 0. ReLUs differ from
the LSTM and GRU models because they do not contain any nonlinearities. The
ReLU models clearly do not reach a counting configuration where increments and
decrements are of the same magnitude. This results in models decrementing too
quickly and failing either before reaching the 0 point like the LSTMs, or
decrementing too slowly and failing at the 0 point.

XXXX INCLUDE THE ZIGZAG PLOTS HERE AND FIX THEM BEFORE INSERTING THEM.

  * •

list different failure modes

  * •

explain the reasons for each failure mode. lack of saturation in LSTMs, GRU
squashing activation etc, and ReLU behaving in a more linear manner than the
other 2 because the flat part of the ReLU is not used.

  * •

ReLU models count reach 0 point too soon or not soon enough.

  * •

LSTM models usually count short and reach the 0 point before the actual 0
point (due to lack of saturation)

  * •

GRUs fail in the middle due to squashing activation

##  8 Conclusions and Future Work

In this study, we investigate the generalisation abilities of our models. We
are interested in determining the extent to which our models can generalise in
the long run, uncovering why they eventually fail to generalise effectively,
and finding any correlation between loss and long term generalisation.

Using the same task as suzgun2019lstm, we train our models in the same way,
and test them on sequences of length 1000, which are much longer than those
seen in training. We observe poor generalisation for the majority of models on
the longer sequences. We then investigate the correlation between loss and the
extent to which models generalise, and we do in fact find that loss is a
predictor for long term generalisation. We then try training our LSTM models
for significantly longer to see if there will be a significant reduction in
the final loss value as a result of additional training. We observe a decrease
in the loss, but it is quite small and slow, rendering this approach
impractical to achieve saturation and perfect long term generalisation. We
then create the zigzag dataset, which has a regular pattern and allows us to
analyse our models. We test our models on the zigzag dataset and we observe
that different models fail in different ways, where GRU models fail just over
halfway through an interval with equal increments and decrements, the LSTMs
fail towards the end of the interval due to the lack of saturation, and ReLUs
fail either before or at the end of the interval due to the unequal size of
the increments and decrements.

This study empirically proves and elaborates on the findings by
DBLP:conf/acl/WeissGY18, and extends the experiments by suzgun2019lstm. In the
future, we would like to further investigate the reasons why models do not
find the correct weights to generalise effectively in the long term. We would
like to determine effective weight configurations for the ReLUs and LSTMs to
be able to count and find ways to train our models in a way that leads them to
these configurations. We would also like to investigate the effect of
discretising the model weights and biases and saturating the activation
functions of the LSTMs if saturation is not reached.

  * •

there is a correlation between loss and generalisation

  * •

different models fail in different ways

  * •

training for longer does not improve generalisation

******************************************************************
IGNORE FROM THIS POINT FORWARD

##  9 Counter Behaviour

We focus on Dyck-1 sequences. Dyck-1 sequences are bracket sequences which
consist of one type of bracket, where each sequence has an equal number of
opening and closing brackets, and closing brackets do not precede their
corresponding opening brackets. suzgun2019lstm and DBLP:conf/acl/WeissGY18
state that LSTMs can behave like counters and evaluate their models using
different language recognition tasks that rely on counting. We use the same
task as suzgun2019lstm to evaluate our models. This is a limited Dyck-1
counting and recognition task where a sequence is classified at every time
step as valid or potentially valid. A more complete language recognition
paradigm is described by DBLP:journals/corr/abs-2004-06866.

The goal is to evaluate the generalisation ability of our models to longer
sequences. We would like to find the extent to which a model generalises, and
extend the work by DBLP:conf/acl/WeissGY18 and suzgun2019lstm.

  * •

if you rtain a model that works on training set, how do they generalise to
long sequences?

  * •

point to weiss and suzgun

  * •

limits of generalisation, different for differnt models etc?

##  10 Experimental Setup

Like suzgun2019lstm, we train our models using a dataset of 10000 sequences of
length 2 to 50 tokens. We use a short test set of 5000 elements of length 2 to
50 tokens. We use a longer test set of 5000 elements of lengths 52 to 100
tokens. The training, short and long test sets are generated in the same way
as suzgun2019lstm. We use the Adam optimiser (kingma2014adam), and learning
rates of 0.01 for the ReLU and LSTM models, and 0.001 for the GRU models. To
evaluate the generalisation abilities of our models, we use 3 different
datasets consisting of 2000-token Dyck-1 sequences.

  1. 1.

1000-token: 1000-token Dyck-1 sequences generated like suzgun2019lstm.

  2. 2.

Nested: The 1000-token sequences that are generated are nested into a random
position in the other 1000-token sequences. There are no duplicate sequences
in this dataset.

  3. 3.

Zigzag: repetitions of $j$ opening brackets followed by $j$ closing brackets
for $j=\\{10,20,25,50,100,125,200,250,500,1000\\}$. There are no duplicate
sequences in this dataset.

The sequences in all datasets are valid and complete Dyck-1 sequences. All
sequences in the nested, concat and zigzag datasets are 2000 tokens long. All
datasets do not contain any duplicate sequences, and there is no overlap
between the training and validation datasets.

##  11 Experimental Results

limits of generalisation in practice. move table of results here. comment on
the lack of generalisation to the longer datasets, then introduce
predictability factors.  WHEN DO THEY FAIL?

##  12 Predictability Factors

  * •

correlation between loss and FPF

  * •

correlation between max depth and FPF (figure out what is needed for this)
EXPECTATIONS

    * –

ReLU if trained, will behave like a linear network where the zero part of the
ReLU is not used. The result will not be affected by the count from the
previous time step

    * –

if LSTM not saturated, due to nonlinearities, there may be some dependence on
the count from the previous timestep.

    * –

GRU models are dependent on previous count

    * –

expected stronger correlation between stack depth and FPF in GRU and LSTMs
than ReLUs.

We evaluate a number of factors which can help us predict the behaviour of our
models. We first evaluate the effect of the loss on the performance of the
models, and whether the loss is somehow correlated with the First Point of
Failure. We define the First Point of Failure (FPF) as the point in the
sequence at which a model first produces an incorrect output.

The second set of predictability factors is a number of indicators that show
us whether or not models have fulfilled a set of conditions that will allow
them to count. In order for a model to behave like a counter, "(" should
result in an increment, i.e. the model increasing the output by a fixed value,
and ")" should result in a decrement, i.e the models reducing the output by a
fixed value which is ideally the same as the increment value. The total number
of increments and decrements by the model should result in a final count value
of 0. We use a number of different datasets to evaluate our models.

###  12.1 Correlation between loss and failure/generalisation

We apply different variants of our models to the concatenated, nested and
zigzag datasets. These models used are from different points in the training
and across different training runs. We plot the results of our tests and
perform linear regression to visualise and quantify the correlation between
the FPF and the loss values. We use the same task as suzgun2019lstm.

As shown in Figure LABEL:fig:plots_Correlation_zigzag, there is a correlation
between the value of the validation loss and the generalisation ability of the
models. A higher log of the inverse loss indicates a lower loss value. This
correlation is observed across the zigzag, nested and concatenated datasets
for LSTMs, ReLUs and GRUs. However, we observe the strongest and most
consistent correlation in the LSTMs. The ReLUs are difficult to train, and
therefore, the majority of the ReLU models end up with higher loss values. The
GRU models are only capable of counting to a very limited extent, and do not
generalise well to longer sequences. For GRU models, counting is very limited,
so we focus on the ReLU and LSTM models. We observe that for the ReLUs and
LSTMs, the FPF is generally lower when the zigzag dataset is used. The overall
highest FPF values are observed when the nested dataset is used.

The accuracy achieved by the models on the datasets are recorded in Table 1.
The LSTM models consistently train to 100% accuracy, and achieve 100% on the
validation and longer test set, where the longest sequence is 100 tokens.
However, when the sequences are significantly longer than the training set,
the LSTM models generalise poorly. GRU models do not train to 100% accuracy,
yet they achieve similar accuracy on the validation and long test sets. GRU
models generalise poorly to the significantly longer sequences in the zigzag,
nested and concat datasets. The ReLU models are difficult to train and do not
learn the Dyck-1 language from data. Consequently, it is not surprising that
they do not generalise to the longer sequences. Despite the higher FPF values
achieved on the nested dataset, the overall performance on the dataset was
poor. Also, the zigzag dataset overall has the lowest FPF values, yet the
models perform best on this dataset when compared to the concatenated and
nested datasets. The models do not generalise to the concatenated and nested
datasets, and generalise poorly to the zigzag dataset. We can therefore say
that there is a correlation between loss and a model’s ability to generalise,
but generalisation is still nowhere near perfect when models are exposed to
much longer sequences than those seen in training. The correlation between the
loss and generalisation cannot be used as a sole factor to predict counting
behaviour in RNN models.

###  12.2 Conditions for Counting in LSTMs

  * •

Weiss defined a configuration which results in counting in LSTM models

  * •

We are not sure of whether these conditions are necessary and sufficient but
it is a known configuration.

  * •

inspect model parameters to check if they actually reach these conditions from
the data (SPOILER, THEY DONT).

  * •

this explains poor generalisation

  * •

insert histograms of the indicators, use only well trained models.

###  12.3 Conditions for Counting in ReLUs

  * •

we develop our own conditions for ReLU counting

  * •

Show that ReLUs are extremely difficult to train

  * •

even the somewhat trained models do not find this configuration from the data.

DBLP:conf/acl/WeissGY18 have shown that GRUs and Elman RNNs cannot behave as
counters, unlike ReLUs and LSTMs. They state that ReLUs can count, and LSTMs
can count if saturation is achieved. We determine a set of identifiers for
LSTMs and ReLUs which can help us predict whether or not the models will
behave as an explicit counter. There are different indicators for LSTMs and
ReLUs. We elaborate on the findings by DBLP:conf/acl/WeissGY18 that state that
saturation is a condition for counting when determining the identifiers for
LSTMs. We focus on Dyck-1 sequences. An open bracket should trigger an
addition of constant $m$ to the current bracket count, which we will refer to
as an increment. A close bracket should trigger a subtraction of constant $m$
from the current bracket count, which we will refer to as a decrement.
Increments and decrements have equal magnitudes in opposite directions. An
input string $x$ is processed by the counter one token $x_{t}$ at a time. Our
input to the models is the one-hot encoding for each bracket.

We use a ReLU to realise our implementation of a Dyck-1 counter from
Definition LABEL:dfn:Dyck-1Counter. We map the components of the Dyck-1
counter to the ReLU and summarise it in Table 3.

Table 3:  This table shows the mapping of Dyck-1 counter components to ReLU components. Dyck-1 Counter | ReLU
---|---
Input token $x_{t}$ | Input $x_{t}$
Count c | Recurrent value $h$
Counter update function $+m$ | Non-recurrent component $Wx_{t}+b$
Zero-check function | ReLU activation
State $Q$ | Recurrent weight $U$

Based on the mapping in Table 3, we define the following weight configuration
for a ReLU which results in it behaving like a counter.

  * •

$Wx_{t}+b=\begin{cases}>0,&\text{if }x_{t}=\text{ `(' }\\\ <0,&\text{if
}x_{t}=\text{ `)' }\end{cases}$

  * •

$(Wx_{t}+b)_{(}:(Wx_{t}+b)_{)}=-1$

  * •

$U=1$

We then inspect the parameters of our ReLU models and show that they almost
never find this counter configuration from the data in training. We visualise
this in Figure LABEL:fig:histogram_ReLU_Indicators.

For LSTMs, we follow the counting conditions defined by
DBLP:conf/acl/WeissGY18.

  * •

Count value is calculated in $c_{t}$

  * •

$i_{t}=f_{t}=1$ due to the saturation of their sigmoid activation functions.

  * •

Counter direction is decided based on input $x_{t}$ and state $h_{t-1}$. This
is dependent on the saturation of the $\tanh$ activation function. This
counter direction is represented in $\tilde{c_{t}}$.

  * •

$o_{t}=1$, and this is also dependent on the saturation of the sigmoid
activation function.

We inspect the parameters of our models and visualise the values of these
indicators in Figure LABEL:fig:Indicators_LSTM. The histograms show that the
LSTM models do approach saturation, but do not necessarily achieve it. This
gives us more clarity about the reasons why generalisation is poor on very
long sequences. A large enough deviation accumulates and causes the models to
fail. This leads us to investigate the differences in the failure of different
models.

##  13 Failure modes

  * •

link this to the correlation between stack depth and FPF

  * •

show that there are different failure modes

  * •

prove the findings by weiss that the GRUs cannot count

To further understand why the models fail, and differentiate between the
failure in different models, we combine our empirical findings from the
correlation between loss and FPF with the counting conditions we defined. We
use the concatenated, nested and zigzag datasets mentioned earlier. We plot
the distribution of the FPFs for our models combined with different datasets.
We use the zigzag plots to distinguish between the different failure modes
shown by our models. We observe 3 different failure modes:

  * •

Failure mode 1: Decrement is larger than the increment, so the models reach
the stack depth of 0 too early.

  * •

Failure mode 2: Increment is larger than decrement, models decrement slowly
and do not reach 0 count when the ground truth is 0.

  * •

Failure mode 3: GRU failure when the squashing has reduced the size of the
count significantly with the increase in the stack depth, and once a decrement
starts, it is very large relative to the accumulated count, and the 0 point is
reached too early.

The images in Figure LABEL:fig:Failure_Modes show that there are different
modes of failure exhibited by the models. The RNN and LSTM models fail because
the increments and decrements did not have the same magnitude, resulting in
failure modes 1 and 2. The GRU models evidently fail at the switching point
where it has finished finished processing the opening brackets and
incrementing the count, and has to start processing the closing bracket and
start decrementing the count. This is due to the squashing activation that is
present in GRUs. This is the third failure mode that we observe.

These findings further corroborate the findings by DBLP:conf/acl/WeissGY18
that GRUs cannot count, ReLUs can count but are very difficult to train XXXX
FIND THE CORRECT REFERENCE FOR RELUS BEING DIFFICULT TO TRAIN, and that LSTMs
can count if their sigmoid and $\tanh$ activation functions are saturated. We
reiterate the findings from the previous section that the models do not find
the configurations for counting from the data.

##  14 Experimental Results

Our results are summarised in Table 1 and Figures
LABEL:fig:plots_Correlation_zigzag, LABEL:fig:histogram_ReLU_Indicators,
LABEL:fig:Indicators_LSTM and LABEL:fig:Failure_Modes. Figure
LABEL:fig:plots_Correlation_zigzag shows that there is a correlation between
the loss value and the generalisation ability of a model, where models with
lower loss failed at a later point in a sequence than the models which had
higher loss values. They show that there is a correlation between the loss and
the FPF, where for all models and test datasets, the p value is very small and
the r value is significant, and even as high as 0.7. We observe that despite
high train accuracies for LSTMs and GRUs, the models fail to generalise to the
longer sequences in the zigzag, nested and concatenated sequences. For the
ReLUs, we can observe from the results that despite the possibility of
realising the counting behaviour, the models do not learn the correct solution
from the data, resulting in poor performance on the test sets.

We have determined indicators that we can use to determine whether or not
saturation has occurred, and as a result predict the counter behaviour of
models. We can also point out the different failure modes for LSTMs, RNNs and
GRUs. We observe that the presence of a switching point in a sequence where
the model switches from a non-zero count to a zero count us a point where
errors are likely to occur. This applies to ReLUs, LSTMs and GRUs. Another
problematic point in a sequence is where the count has reached is maximum and
will start decreasing. Because of the squashing activation of GRUs, this point
in a sequence is a point where a lot of errors happen for GRU models.

##  15 Conclusions

In this paper, we evaluate the ability of RNN models to learn to behave as
counters, and learn Dyck-1 sequences. We start by examining the relationship
between the loss and the FPF and do in fact find a strong correlation. We then
mathematically deduce a set of conditions which can be used to predict the
counter behaviour of our models, and inspect our model parameters to see if
they do fulfill the counting conditions we defined. In practice, we observe
that these conditions are difficult to learn from the data. From our counting
conditions and analysis of the models, we are able to distinguish between the
ways the models fail differently. We have define 3 different failure modes,
and the reasons behind the occurrence of each of them.

##  16 NOTES

###  16.1 contributions

  1. 1.

limitations of generalisation in practice (the empirical results on the
different datasets). Only successfully trained models

  2. 2.

FPF correlates inversely with the loss (use all models here), scatter plots
with linear regression.

  3. 3.

Sufficient conditions defined by Weiss for LSTM counting are not met after
training.

  4. 4.

failure modes

  5. 5.

link between stack depth and FPF

  6. 6.

Conditions for RNNs (not met after training)

  7. 7.

RNNs not training well in the first place

###  16.2 questions

  1. 1.

Do the nested sequences have zero points in the middle of the sequence?  YES
THEY DO, NOT SURE IF THIS APPLIES TO ALL OF THEM.

  2. 2.

###  16.3 to do

  1. 1.

find a way to correlate max stack depth with FPF. Check excel sheets of the
zigzag implementation.

  2. 2.

histogram for additional condition for ReLU for increment > 0 and decrement <
0

  3. 3.

check the 1000 token datasets and FPF to see when they failed (and try to
remember why we even decided to go for 2000 tokens)

  4. 4.

check if the validation accuracy is higher than training accuracy on a smaller
sequence. investigate high validation accuracy (maybe try a smaller set with a
different split)

  5. 5.

write about filtering the somewhat decent trained RNNs

  6. 6.

train LSTMs to 100 epochs to see if the loss gets any smaller and can improve
the FPF value.


RANUM” fails for 113 runs (roughly 2X failed runs compared to RANUM). This
study implies that RANUM’s technique helps to improve effectiveness and
efficiency at both steps of failure-exhibiting system test generation compared
to the pure random baseline. The improvement for the first step is mainly from
the effectiveness perspective, and the improvement for the second step is
mainly from the efficiency perspective.

###  -G Threats to Validity

An external threat to validity is the evaluation subjects, which are the GRIST
benchmarks [54], used to evaluate RANUM and baseline approaches. Although the
subjects contain 63 real-world DL programs and are the largest to our
knowledge, they still may not be representative enough. Another external
threat is the approach randomness. To mitigate this threat, we repeat all
randomized approaches for 10 runs. To mitigate bias in the empirical study,
two authors independently conduct the study and discuss all cases to reach a
consensus. A major internal threat to validity comes from the approach
implementation. We reduce this threat by conducting code review and log
checking extensively. An author independently checks the correctness of code
implementation. We also verify the soundness of our static analysis framework
with over 50 carefully designed unit tests.