# A Ramsey Neutron-Beam Experiment to Search for Ultralight Axion Dark Matter
at the ESS

P. Fierlinger  School of Natural Sciences, Technical University Munich, 85748
Garching, Germany  M. Holl  European Spallation Source ERIC, 224 84 Lund,
Sweden  D. Milstead  Physics Department, Stockholm University, 106 91
Stockholm, Sweden  V. Santoro  Faculty of Science, Department of Physics, Lund
University, 221 00 Lund, Sweden
European Spallation Source ERIC, 224 84 Lund, Sweden  W. M. Snow  Department
of Physics, Indiana University, Bloomington, IN 47405, USA  Y. V. Stadnik
School of Physics, The University of Sydney, Sydney, New South Wales 2006,
Australia

###### Abstract

High-intensity neutron beams, such as those available at the European
Spallation Source (ESS), provide new opportunities for fundamental
discoveries. Here we discuss a novel Ramsey neutron-beam experiment to search
for ultralight axion dark matter through its coupling to neutron spins, which
would cause the neutron spins to rotate about the velocity of the neutrons
relative to the dark matter halo. We estimate that experiments at the HIBEAM
beamline at the ESS can improve the sensitivity to the axion-neutron coupling
compared to the current best laboratory limits by up to $2-3$ orders of
magnitude over the axion mass range $10^{-22}\,\textrm{eV}-10^{-16}$ eV.

##  I Introduction

Astrophysical and cosmological observations indicate that about one-quarter of
the total energy and five-sixths of the total matter content of the Universe
is due to dark matter (DM) [1], the identity and microscopic properties of
which remain a mystery. Traditional detection schemes have largely focused on
searching for possible particle-like signatures of weakly interacting massive
particles (WIMPs) with masses in the $\sim\textrm{GeV}-\textrm{TeV}$ range
[2], whose signatures scale to the fourth power of a small interaction
constant between the DM and ordinary matter. On the other hand, ultralight
bosons with sub-eV masses and a high particle number density may produce
distinctive wavelike signatures. One of the leading candidates for DM is the
axion, which is a light pseudoscalar (spin-0, parity-odd) particle originally
proposed to resolve the strong $\mathcal{CP}$ problem of quantum
chromodynamics (QCD) [3, 4, 5, 6, 7, 8, 9]. Besides the canonical QCD axion,
more generic axion-like particles may also exist in nature [10, 11].111In the
following, we do not distinguish between the canonical QCD axion and axion-
like particles and simply refer to both as “axions”. Searches for axion DM
have mainly focused on the axion’s possible electromagnetic coupling to
photons [13].

Neutron-beam experiments have previously been considered for fundamental
physics tests, such as neutron-antineutron oscillations, electric dipole
moment searches, new-boson-mediated forces and investigations of the structure
of the weak interaction; see, e.g., Refs. [14, 15, 16, 17, 18, 19, 20, 21, 22,
23, 24]. With major new neutron sources, such as the European Spallation
Source (ESS) [25], Spallation Neutron Source (SNS) in the U.S. [26] and China
Spallation Neutron Source (CSNS) [27], providing extremely strong, pulsed
neutron beams, it is imperative to maximally leverage the potential of these
state-of-the-art facilities. Here we explore the sensitivity of a Ramsey
experiment at such a pulsed neutron beam facility to a pseudo-magnetic field
effect stemming from the coupling of an ultralight axion DM field to the
neutron. The derivative coupling of axion DM to neutron spins would cause
neutron spins to precess about the velocity of the neutrons relative to the DM
halo. Depending on the orientation of the pseudo-magnetic field with respect
to the experiment, this would cause neutrons in the beam to accumulate an
additional phase as they travel along their flight path. We estimate that with
the proposed HIBEAM neutron beamline [24] at the ESS, the sensitivity to the
axion-neutron coupling can be improved compared to the current best laboratory
limits by up to $2-3$ orders of magnitude over the axion mass range
$10^{-22}\,\textrm{eV}-10^{-16}$ eV.

##  II Theory

Ultra-low-mass axions with very small kinetic energies can be produced
efficiently via nonthermal production mechanisms, such as vacuum misalignment
[28, 29, 30] shortly after the big bang, and can subsequently form a
coherently oscillating classical field: $a(t)\approx a_{0}\cos(\omega t)$,
with the angular frequency of oscillation given by $\omega\approx
m_{a}c^{2}/\hbar$, where $m_{a}$ is the axion mass, $\hbar$ is the reduced
Planck constant, and $c$ is the speed of light in vacuum. The oscillating
axion field carries the energy density $\rho_{a}\approx m_{a}^{2}a_{0}^{2}/2$
and behaves like a cold, nearly pressureless fluid on length scales greater
than the de Broglie wavelength of the field.222Unless indicated otherwise, we
adopt the natural units $\hbar=c=1$ in this paper. Cosmological and
astrophysical observations rule out the possibility of axions with masses
$m_{a}\lesssim 10^{-21}$ eV comprising the dominant fraction of the DM [32,
33, 34, 35], although such ultralight axions may still comprise a sub-dominant
fraction of the DM depending on their mass. Gravitational interactions between
axion DM and ordinary matter during galactic structure formation subsequently
virialise galactic axions ($v_{\textrm{vir}}\sim 300$ km/s locally), which
gives an oscillating axion field in our local Galactic region the finite
coherence time: $\tau_{\textrm{coh}}\sim 2\pi/(m_{a}v_{\textrm{vir}}^{2})\sim
2\pi\times 10^{6}/m_{a}$; i.e., $\Delta\omega/\omega\sim 10^{-6}$
corresponding to nearly monochromatic oscillations of the field.

An axion field may interact with nucleons via the derivative coupling:

$\mathcal{L}_{\textrm{int}}=-\frac{C_{N}}{2f_{a}}\partial_{\mu}a\,\bar{N}\gamma^{\mu}\gamma^{5}N\,,$
(1)

where $N$ and $\bar{N}$ denote the nucleon field and its Dirac adjoint,
respectively, $f_{a}$ is the axion decay constant, and $C_{N}$ is a model-
dependent dimensionless parameter. The spatial components of the derivative
coupling of an oscillating axion DM field in the laboratory frame of
reference, $a(t,\bm{r})\approx a_{0}\cos(m_{a}t-\bm{p}_{a}\cdot\bm{r})$, with
spin-polarised nucleons in Eq. (1) simplifies as follows in the non-
relativistic limit [36, 37]:

$H_{\textrm{int}}(t)\approx\frac{C_{N}a_{0}}{2f_{a}}\sin(m_{a}t)\,\bm{\sigma}_{N}\cdot\bm{p}_{a}\equiv\bm{\sigma}_{N}\cdot\bm{B}_{\textrm{eff}}(t)\,,$
(2)

which resembles the interaction of a nucleon spin $\bm{\sigma}_{N}$ (which has
unity norm, $|\bm{\sigma}_{N}|=1$) with a time-varying pseudo-magnetic field
$\bm{B}_{\textrm{eff}}(t)$. The interaction in Eq. (2) causes nucleon spins to
precess about the direction of the axion DM momentum, $\bm{p}_{a}$, taken in
the laboratory frame. The correlation $\bm{\sigma}_{N}\cdot\bm{p}_{a}$ in Eq.
(2) is modulated at the daily sidereal frequency due to the rotation of Earth
and can be calculated by transforming to a nonrotating celestial coordinate
system; see, e.g., [38, 39, 40] for more details. The average value of
$\bm{p}_{a}$, sampled over many coherence times, is expected to be directed
opposite to the laboratory’s orbital motion about the Galactic Center.
However, due to the stochastic nature of the axion DM field [41], the
magnitude and direction of $\bm{p}_{a}$ during the course of measurements may
differ from the long-term average value [42, 43].

The “axion wind” spin-precession effect described by Eq. (2) induces temporal
variations in the Larmor precession frequency of a polarised nucleon spin
according to:

$\hbar\omega_{L}(t)=|-\gamma_{N}\,\bm{\sigma}_{N}\cdot\bm{B}+2\bm{\sigma}_{N}\cdot\bm{B}_{\textrm{eff}}(t)|\,,$
(3)

principally from the component of $\bm{B}_{\textrm{eff}}(t)$ directed along
the applied magnetic field $\bm{B}$, with $\gamma_{N}$ being the gyromagnetic
ratio of the nucleon. A (pseudo) magnetic-field measurement using a neutron
spin is based on measurement of the phase
$\phi(T)=\int_{0}^{T}\omega_{L}(t)\,dt$, where $T$ is the spin precession
time, resulting in a measurement sensitivity that depends not only on the
coupling constant $C_{n}/f_{a}$, but also on the axion mass: for the most
sensitive scenario in an experiment when $m_{a}\ll\omega_{L}$, in which case
$\bm{B}_{\textrm{eff}}$ is approximately constant during the free precession
phase and a phase can continuously build up. Note that the axion-DM-induced
change in the Larmor precession frequency scales linearly in the small
interaction constant, in contrast to the quadratic scaling in searches for
virtual-axion-mediated spin-dependent forces [44].

##  III EXPERIMENTAL APPARATUS

To measure a change in $\omega_{L}$, Ramsey’s method of separated oscillating
fields [45] is used. The method is based on the comparison of $\omega_{L}$ to
an external frequency $\omega_{1}$ from a stable clock. Polarized neutrons
enter a spin-flipper coil, a region with a magnetic field $\bm{B}_{1}$,
oscillating at a frequency $\omega_{1}$. This closely approximates a rotating
magnetic field at frequency $\omega_{L}$ determined by the constant background
field $\bm{B}_{0}$. $B_{1}$ is adjusted to ultimately result in a spin
rotation by $\pi/2$ radians, into the plane normal to $\bm{B}_{0}$, resulting
in a precession at the Larmor frequency $\omega_{L}$. The neutrons then
precess in the constant magnetic field $\bm{B}_{0}$ for a time $T$, where a
phase $\phi=\omega_{L}T$ is built up. At the end of the flight path, a second
$\pi/2$-flip is applied by passage through a second coil, driven with the same
current that also drives the first coil. In the case when
$\omega_{L}=\omega_{1}$, both $\pi/2$-flips add up, resulting in a downward
pointing polarization of the neutron. A mismatch of the frequencies, such that
a phase $\pi$ is built up during the flight time relative to $\omega_{1}$,
results in the polarization pointing upwards. Performing many independent
Ramsey experiments with different settings for $\omega_{1}$ results in the
well-known Ramsey fringes. An unknown magnetic-field-like effect, as well as
any other magnetic field applied in addition to $\bm{B}_{0}$, will shift the
whole pattern in frequency. Repeatedly determining the central fringe of the
pattern for subsequently different $\bm{B}_{0}$ orientations thus reveals a
slowly oscillating offset field $\bm{B}_{\textrm{eff}}$ or its absence within
the boundaries of statistical and systematic precision. This can then be
converted into the parameter space of coupling constant and axion mass. A
constant magnetic field $\bm{B}_{0}$ is obtained with magnetic shielding
around the field-generating coil and neutron flight path. However, the axion
DM field is not necessarily affected by the passive magnetic shield based on a
ferromagnetic alloy and a (regular) conductive shield [46]. A flight time of
$\sim 50$ ms for a length of $50$ m with high statistics and a 10 Hz
repetition rate allow for a sensitive bandwidth to an oscillating field with
frequency from around 3 Hz (corresponding to an axion mass of
$10^{-14}\,\textrm{eV}$) to below mHz.

As shown in Fig. 1, the apparatus to determine the spin precession frequency
of the neutrons consists of: (1) a chopper; (2) a polarizer; (3) a collimator;
at (4) the neutrons enter the magnetically controlled region, formed by a
2-layer passive magnetic shield, a vacuum chamber and a field coil to produce
a magnetic field transverse to the neutron direction; (5) and (8) are the
$\pi/2$-spin flippers; (6) a neutron guide; (7) an optional $\pi$-spin
flipper; (9) a spin analyzer; and (10) a detector. The magnetic holding field
is applied over the region denoted with the dotted rectangle.

Figure 1: Schematic setup of the experiment with components (longitudinal
cut), neutrons entering from the left (see the main text for details). The
flight path is 50 m.

The setup is situated starting 15 meters from the moderator. The whole length
of the flight path is magnetically shielded and a constant magnetic field is
applied. Initial conditions for the measurement were established using the
ray-tracing software McStas [47]. Collimators are utilized to reduce the beam
area to $10~{}\textrm{cm}\times 10$ cm at the entrance to the experiment.

##  IV SENSITIVITY

Statistical sensitivity. The frequency resolution of a single Ramsey
experiment scales as

$\Delta f=\frac{1}{2\pi T\alpha\sqrt{N}}\,,$ (4)

with $N$ being the number of neutrons and $\alpha$ the contrast in detection
of the different spin states. The sensitivity then increases with the square-
root of the number of repetitions, while the oscillations of the DM field
remain coherent. The initial neutron flux from the moderator into the exit of
the neutron extraction system is $1\times 10^{12}\,n$/s, which is subsequently
reduced by polarization ($\times 0.5$), polarization analysis ($\times 0.5$)
and calibration runs ($\times 0.8$). Here we assume a quasi-perfectly fit
elliptic neutron guide section with $m\sim 3$ to capture practically the whole
flux. Also, the spin flippers are assumed to be perfectly efficient over the
entire velocity range. Assuming only systematic errors uncorrelated with the
frequency of the DM oscillation signal, the experiment is statistically
limited. A frequency resolution of $10^{-8}$ rad$/$s can be reached in one
year of run time for a 50-meter flight path. As the signal is measured as a
time-dependent modulation of the magnetic field, the effect alternatingly adds
and subtracts from the magnetic field, thus effectively doubling the precision
to $5\times 10^{-9}$ rad$/$s. Using Eq. (2), this translates into the
sensitivity estimates shown in Fig. 3, assuming that axions saturate the
observed density of DM in our local Galactic region, $\rho\approx
0.4\,\textrm{GeV/cm}^{3}$ [1]. The sensitivity is also constrained by
systematic issues, which is estimated by a simulation of the so-called Ramsey
fringes, which is the polarization $P_{z}$ along the magnetic field
$\bm{B}_{0}$ as a function of the detuning from the resonance frequency
$\omega_{L}$. The parameters varied in the simulation are the spread of
measurements with different detuning, neutron statistics, duration of the
neutron pulse, magnetic field maps, spin-flip pulses, neutron positions,
momentum directions and velocities, the precession time $T$, the duration of a
$\pi/2$-flip $\tau$, the frequency $\Omega_{R}$ related to $B_{1}$ (the Rabi
frequency), and the detuning $\delta\omega$ from the Larmor frequency.

Magnetic field.  The phase buildup during free precession depends on the
homogeneity of the magnetic field. For the magnetically controlled region, we
aim for a typical homogeneity of the magnetic field over most of the volume
covered by the neutron beam at the level of 10-3, obtained with a field
strength of $B_{0}\sim 10-100$ $\mu$T. Shielding of environmental disturbances
is obtained by a two-layer octagonal passive magnetic shield with open ends
and its axis approximately aligned with the neutron beam central axis; a coil
made from 8 longitudinal wires produces a transverse magnetic field with
strength $B_{0}\sim 10-100$ $\mu$T. Static residual fields smaller than 100 nT
after degaussing do not affect the measurement and the relative
inhomogeneities of the magnetic field over most of the volume covered by the
neutron beam are about 10-3. The actual complexity of the coil configuration
inside the shield and the cross-sectional area of the shielded region occupied
by the neutron beam will ultimately define the homogeneity of the field
experienced by the neutrons. The inset in Fig. 2 shows a cut through the
envisaged configuration, providing such a field with only a few wires.

Figure 2: Ramsey fringes for a realistic magnetic field. The inset shows a cut
through the octagonal field setup with illustrated field lines.

The vacuum tube, which also acts as a shield for higher frequencies, is
accessed for pumping at the detector region only, as the vacuum requirement is
only 10-5 mbar. The passive shield design is similar to that in Ref. [48].

Polarization. When the neutrons enter the shielded region, they pass through a
cell containing polarised 3He. Through spin-dependent nuclear absorption of
neutrons on 3He, an almost perfectly polarized neutron beam ($>$ 99%) can be
obtained, thus hardly affecting contrast in the Ramsey measurement.
Polarization analysis after the Ramsey experiment can be done in the same way.
We use cells made from GE-180 glass to obtain spin life-times of the order of
a day, with polarization done using the spin-exchange optical pumping
technique with a similar setup as in [49] with a length of several cm, a
diameter of 0.1 m and a pressure of up to 10 bar. Cell designs are
conceptually based on [50]. The product of polarization and analysis power
determines the contrast parameter $\alpha$ in Eq. (4). Spin-flipper coils are
composed of approximately 50 turns of wire wound around a nonmetallic support
inside the vacuum chamber to minimize coupling to the chamber and shields;
both spin-flipper coils are serially connected and fed by a function
generator, stabilized by a GPS-locked frequency standard. While the $\pi/2$
condition is only perfectly valid for a single velocity, the analysis works
for a range of velocities but with reduced contrast. With a $B_{1}$ amplitude
of $\sim 100\,\mu$T, corresponding to 3000 spin rotations per second, and for
a $\pi/2$-flip at 1200 m$/$s, the corresponding coil length is 0.1 m. Placed
along the flight path, a $\pi$-flipper is installed, which can be deployed. In
particular, it can reduce wash-out of the phase measurement due to geometry;
e.g., in the form of parity-symmetric magnetic-field deformations along the
shield for a centered beam. A further interesting feature is that it can
recover the flip angles for velocities that do not match the $\pi/2$-flip
conditions and thus massively broaden the velocity acceptance of the Ramsey
setup. However, for our new physics search, a symmetric placement in a
constant field also strongly suppresses sensitivity to the new-physics signal
in the most interesting frequency range and thus will only be used for
instrument characterization. It should be noted that the magnitude of
$\bm{B}_{0}$ does not change the new physics reach. However, a smaller field
makes the experiment more sensitive to background gradients. The second spin
flip and the polarization analysis are similar to the first spin flip and
polarizer, respectively.

Pulse duration and velocity spread. The pulse duration of a pulse at the ESS
is $\sim 3$ ms at a 10 Hz repetition rate. Although the phase information is
washed out, a fit of the central fringes of the Ramsey experiment can in
principle resolve the phase information. However a chopper can be used to
refine the pulse.

Neutron detection. When $2\times 10^{10}$ neutrons per pulse arrive within
$\mathcal{O}(100)\,\textrm{ms}$ at the detector, the count rate is $\sim
10^{11}$ s-1. For a detector with 1000 pixels, the rate becomes $\sim 100$
MHz. In this project, we will use a 2D-position-sensitive segmented ion
chamber using 3He as the absorber in the gas. Similar detectors have been
successfully used in several sensitive polarized neutron transmission
experiments conducted at both pulsed and continuous neutron sources to search
for parity-odd neutron interactions with nuclei and for possible exotic spin-
dependent neutron interactions [51].

Figure 3:  Projected sensitivity of a 50 m scale Ramsey neutron-beam
experiment using the HIBEAM neutron beamline at the ESS (denoted by dashed red
line) to the coupling strength of axion dark matter with a neutron, defined in
Eq. (1), as a function of the axion mass $m_{a}$, assuming one year of run
time and that axions saturate the observed density of dark matter in our local
Galactic region, $\rho\approx 0.4\,\textrm{GeV/cm}^{3}$ [1]. The cyan, blue,
pink, red and purple regions indicate regions of parameter space already
probed by magnetometry-based searches for time-varying spin-precession effects
induced by axion dark matter [39, 52, 53, 54, 55, 56, 57, 58, 59]. The yellow
region denotes the region of parameter space ruled out by a magnetometry-based
search for spin-dependent forces mediated by the exchange of virtual axions
[44]. The pale green region denotes bounds from astrophysical observations of
supernovae [60], which are subject to model-dependent assumptions and may be
evaded altogether [61].

##  V Discussion and Conclusions

Neutron physics offers the potential to probe new physics in ultralight axion
DM searches. As shown in Fig. 3, our proposed experiment at the HIBEAM neutron
beamline at the ESS has significant discovery potential, offering up to $2-3$
orders of magnitude improvement in sensitivity compared to the current best
laboratory limits. Using free neutrons as a probe of the “axion-wind” spin-
precession effect in Eq. (2) provides a clean probe of the neutron interaction
parameter $C_{n}/f_{a}$, which is complementary to the magnetometry approaches
in Refs. [39, 52, 53, 54, 55, 56, 57, 58, 59] that involve nuclei and hence
probe linear combinations of proton and neutron interaction parameters [62].
Our proposed experiment is expected to be competitive with bounds from
astrophysical observations of supernovae [60], whilst also offering the
advantage of a much cleaner and better controlled environment. On the other
hand, the interpretation of rare astrophysical phenomena, such as supernova
explosions, which take place in conditions drastically different from those in
the laboratory, require additional model-dependent assumptions that may not be
valid [61].

###### Acknowledgements.

The work of Y. V. S. was supported by the Australian Research Council under
the Discovery Early Career Researcher Award No. DE210101593. W. M. S.
acknowledges support from the US National Science Foundation (NSF) grant
PHY-2209481 and the Indiana University Center for Spacetime Symmetries. D. M
and V. S gratefully acknowledge support from the Council for Swedish Research
Infrastructure for the Swedish Research Council for the grant ‘HIBEAM pre-
studies’. V. S gratefully acknowledge support from the Stiftelsen för
Strategisk Forskning for the grant ‘Development of a magnetic control beamline
for fundamental physics and condensed matter science at the European
Spallation Source’.

## References

  * [1] Particle Data Group, RL. Workman, et al., “Review of particle physics,” Progress of theoretical and experimental physics, vol. 2022, no. 8, p. 083C01, 2022.
  * [2] L. Roszkowski et al., “Wimp dark matter candidates and searches—current status and future prospects,” Reports on Progress in Physics, vol. 81, p. 066201, may 2018.
  * [3] R. Peccei and H. Quinn, “CP Conservation in the Presence of Instantons,” Phys. Rev. Lett., vol. 38, p. 1440, 1977.
  * [4] S. Weinberg, “A New Light Boson?,” Phys. Rev. Lett., vol. 40, p. 223, 1978.
  * [5] F. Wilczek, “Problem of Strong $P$ and $T$ Invariance in the Presence of Instantons,” Phys. Rev. Lett., vol. 40, p. 279, 1978.
  * [6] J. E. Kim, “Weak Interaction Singlet and Strong CP Invariance,” Phys. Rev. Lett., vol. 43, p. 103, 1979.
  * [7] M. A. Shifman, A. I. Vainshtein, and V. I. Zakharov, “Can Confinement Ensure Natural CP Invariance of Strong Interactions?,” Nucl. Phys., vol. B166, pp. 493–506, 1980.
  * [8] M. Dine, W. Fischler, and M. Srednicki, “A Simple Solution to the Strong CP Problem with a Harmless Axion,” Phys. Lett., vol. 104B, pp. 199–202, 1981.
  * [9] A. R. Zhitnitsky, “On Possible Suppression of the Axion Hadron Interactions. (In Russian),” Sov. J. Nucl. Phys., vol. 31, p. 260, 1980.  [Yad. Fiz.31,497(1980)].
  * [10] J. Jaeckel and A. Ringwald, “The low-energy frontier of particle physics,” Annual Review of Nuclear and Particle Science, vol. 60, no. Volume 60, 2010, pp. 405–437, 2010.
  * [11] P. Arias, D. Cadamuro, M. Goodsell, J. Jaeckel, J. Redondo, and A. Ringwald, “Wispy cold dark matter,” Journal of Cosmology and Astroparticle Physics, vol. 2012, p. 013, jun 2012.
  * [12] In the following, we do not distinguish between the canonical QCD axion and axion-like particles and simply refer to both as “axions”.
  * [13] CB. Adams, N. Aggarwal, A. Agrawal, R. Balafendiev, C. Bartram, M. Baryakhtar, H. Bekker, P. Belov, KK. Berggren, A. Berlin, et al., “Axion dark matter,” 2022.
  * [14] S. Baeßler, V. V. Nesvizhevsky, K. V. Protasov, and A. Yu. Voronin, “Constraint on the coupling of axionlike particles to matter via an ultracold neutron gravitational experiment,” Phys. Rev. D, vol. 75, p. 075006, Apr. 2007.
  * [15] V. V. Nesvizhevsky, G. Pignol, and K. V. Protasov, “Neutron scattering and extra-short-range interactions,” Phys. Rev. D, vol. 77, p. 034020, Feb. 2008.
  * [16] Yu. N. Pokotilovski, “Neutron experiments to search for new spin-dependent interactions,” JETP Lett., vol. 94, pp. 413–417, Nov. 2011.
  * [17] F. M. Piegsa and G. Pignol, “A proposed search for new light bosons using a table-top neutron ramsey apparatus,” J. Phys.: Conf. Ser., vol. 340, p. 012043, Feb. 2012.
  * [18] C. Haddock et al., “A search for possible long range spin dependent interactions of the neutron from exotic vector boson exchange,” Phys. Lett. B, vol. 783, pp. 227–233, Aug. 2018.
  * [19] W. M. Snow, C. Haddock, and B. Heacock, “Searches for exotic interactions using neutrons,” Symmetry, vol. 14, p. 10, Jan. 2022.
  * [20] E. Chanel et al., “The pulsed neutron beam edm experiment,” EPJ Web Conf., vol. 219, p. 02004, 2019.
  * [21] I. Schulthess, E. Chanel, A. Fratangelo, A. Gottstein, A. Gsponer, Z. Hodge, C. Pistillo, D. Ries, T. Soldner, J. Thorne, and F. M. Piegsa, “New limit on axionlike dark matter using cold neutrons,” Phys. Rev. Lett., vol. 129, p. 191801, Nov 2022.
  * [22] D. Dubbers and B. Maerkisch, “Precise measurements of the decay of free neutrons,” Ann. Rev. Nucl. Part. Sci., vol. 71, p. 139, 2021.
  * [23] M. Baldo-Ceolin et al., “A New experimental limit on neutron - anti-neutron oscillations,” Z. Phys. C, vol. 63, pp. 409–416, 1994.
  * [24] V. Santoro et al., “The hibeam instrument at the european spallation source,” 2023.  arXiv:2311.08326 (physics.ins-det).
  * [25] R. Garoby et al., “The European Spallation Source Design,” Physica Scripta, vol. 93, p. 014001, dec 2017.
  * [26] T. Mason, D. Abernathy, I. Anderson, J. Ankner, T. Egami, G. Ehlers, A. Ekkebus, G. Granroth, M. Hagen, K. Herwig, J. Hodges, C. Hoffmann, C. Horak, L. Horton, F. Klose, J. Larese, A. Mesecar, D. Myles, J. Neuefeind, M. Ohl, C. Tulk, X.-L. Wang, and J. Zhao, “The spallation neutron source in oak ridge: A powerful tool for materials research,” Physica B: Condensed Matter, vol. 385-386, pp. 955–960, 2006.
  * [27] W. Sheng, F. Shou-Xian, F. Shi-Nian, L. Wei-Bin, O. Hua-Fu, Q. Qing, T. Jing-Yu, and W. Jie, “Introduction to the overall physics design of csns accelerators,” Chinese Physics C, vol. 33, p. 1, jun 2009.
  * [28] J. Preskill, M. B. Wise, and F. Wilczek, “Cosmology of the invisible axion,” Physics Letters B, vol. 120, no. 1, pp. 127–132, 1983.
  * [29] L. Abbott and P. Sikivie, “A cosmological bound on the invisible axion,” Physics Letters B, vol. 120, no. 1, pp. 133–136, 1983.
  * [30] M. Dine and W. Fischler, “The not-so-harmless axion,” Physics Letters B, vol. 120, no. 1, pp. 137–141, 1983.
  * [31] Unless indicated otherwise, we adopt the natural units $\hbar=c=1$ in this paper.
  * [32] K. K. Rogers and H. V. Peiris, “Strong bound on canonical ultralight axion dark matter from the lyman-alpha forest,” Phys. Rev. Lett., vol. 126, p. 071302, Feb 2021.
  * [33] E. Armengaud, N. Palanque-Delabrouille, C. Yèche, D. J. E. Marsh, and J. Baur, “Constraining the mass of light bosonic dark matter using SDSS Lyman-alpha forest,” Monthly Notices of the Royal Astronomical Society, vol. 471, pp. 4606–4614, 07 2017.
  * [34] T. Kobayashi et al., “Lyman-$\alpha$ constraints on ultralight scalar dark matter: Implications for the early and late universe,” Phys. Rev. D, vol. 96, p. 123514, Dec 2017.
  * [35] N. Bar, D. Blas, K. Blum, and S. Sibiryakov, “Galactic rotation curves versus ultralight dark matter: Implications of the soliton-host halo relation,” Phys. Rev. D, vol. 98, p. 083027, Oct 2018.
  * [36] V. V. Flambaum, “in proceedings of the 9th patras workshop on axions, wimps and wisps, mainz, germany, 2013,” 2013.
  * [37] Y. V. Stadnik and V. V. Flambaum, “Axion-induced effects in atoms, molecules, and nuclei: Parity nonconservation, anapole moments, electric dipole moments, and spin-gravity and spin-axion momentum couplings,” Phys. Rev. D, vol. 89, p. 043522, Feb. 2014.
  * [38] Y. V. Stadnik, Manifestations of dark matter and variations of the fundamental constants in atoms and astrophysical phenomena.  Springer, 2017.
  * [39] C. Abel et al., “Search for axionlike dark matter through nuclear spin precession in electric and magnetic fields,” Phys. Rev. X, vol. 7, p. 041034, Nov. 2017.
  * [40] C. Smorra, Y. V. Stadnik, P. E. Blessing, M. Bohman, M. J. Borchert, J. A. Devlin, S. Erlewein, J. A. Harrington, T. Higuchi, A. Mooser, G. Schneider, M. Wiesinger, E. Wursten, K. Blaum, Y. Matsuda, C. Ospelkaus, W. Quint, J. Walz, Y. Yamazaki, D. Budker, and S. Ulmer, “Direct limits on the interaction of antiprotons with axion-like dark matter,” Nature, vol. 575, pp. 310–314, Nov. 2019.
  * [41] A. Derevianko, “Detecting dark-matter waves with a network of precision-measurement tools,” Phys. Rev. A, vol. 97, p. 042506, Apr 2018.
  * [42] G. P. Centers, J. W. Blanchard, J. Conrad, N. L. Figueroa, A. Garcon, A. V. Gramolin, D. F. J. Kimball, M. Lawson, B. Pelssers, J. A. Smiga, et al., “Stochastic fluctuations of bosonic dark matter,” Nature communications, vol. 12, no. 1, p. 7321, 2021.
  * [43] M. Lisanti, M. Moschella, and W. Terrano, “Stochastic properties of ultralight scalar field gradients,” Phys. Rev. D, vol. 104, p. 055037, Sep 2021.
  * [44] G. Vasilakis, J. M. Brown, T. W. Kornack, and M. V. Romalis, “Limits on new long range nuclear spin-dependent forces set with a k - he 3 comagnetometer,” Phys. Rev. Lett., vol. 103, p. 261801, Dec. 2009.
  * [45] N. F. Ramsey, “A Molecular Beam Resonance Method with Separated Oscillating Fields,” Physical Review, vol. 78, pp. 695–699, June 1950.
  * [46] D. F. Jackson Kimball, J. Dudley, Y. Li, S. Thulasi, S. Pustelny, D. Budker, and M. Zolotorev, “Magnetic shielding and exotic spin-dependent interactions,” Phys. Rev. D, vol. 94, p. 082005, Oct 2016.
  * [47] P. K. Willendrup and K. Lefmann, “Mcstas (i): Introduction, use, and basic principles for ray-tracing simulations,” Journal of Neutron Research, no. Preprint, pp. 1–16, 2020.
  * [48] E. Wodey et al., “A scalable high-performance magnetic shield for very long baseline atom interferometry,” Review of scientific instruments, vol. 91, Mar. 2020.
  * [49] N. Sachdeva et al., “New limit on the permanent electric dipole moment of ${}^{129}\mathrm{Xe}$ using ${}^{3}\mathrm{He}$ comagnetometry and squid detection,” Phys. Rev. Lett., vol. 123, p. 143003, 2019.
  * [50] K. Coulter et al., “Neutron polarization with polarized 3he,” Nuclear Instruments and Methods in Physics Research Section A: Accelerators, Spectrometers, Detectors and Associated Equipment, vol. 270, no. 1, pp. 90–94, 1988.
  * [51] R. Lehnert, W. Snow, and H. Yan, “A first experimental limit on in-matter torsion from neutron spin rotation in liquid 4he,” Physics Letters B, vol. 730, pp. 353–356, 2014.
  * [52] T. Wu, J. W. Blanchard, G. P. Centers, N. L. Figueroa, A. Garcon, P. W. Graham, D. F. J. Kimball, S. Rajendran, Y. V. Stadnik, A. O. Sushkov, A. Wickenbrock, and D. Budker, “Search for axionlike dark matter with a liquid-state nuclear spin comagnetometer,” Phys. Rev. Lett., vol. 122, p. 191302, May 2019.
  * [53] A. Garcon, J. W. Blanchard, G. P. Centers, N. L. Figueroa, P. W. Graham, D. F. J. Kimball, S. Rajendran, A. O. Sushkov, Y. V. Stadnik, A. Wickenbrock, T. Wu, and Dmitry Budker, “Constraints on bosonic dark matter from ultralow-field nuclear magnetic resonance,” Science Advances, vol. 5, no. 10, p. eaax4539, 2019.
  * [54] I. M. Bloch, Y. Hochberg, E. Kuflik, and T. Volansky, “Axion-like relics: new constraints from old comagnetometer data,” Journal of High Energy Physics, vol. 2020, no. 1, pp. 1–38, 2020.
  * [55] M. Jiang, H. Su, A. Garcon, X. Peng, and D. Budker, “Search for axion-like dark matter with spin-based amplifiers,” Nat. Phys., vol. 17, pp. 1402–1407, Dec. 2021.
  * [56] I. M. Bloch, G. Ronen, R. Shaham, O. Katz, T. Volansky, and Or Katz, “New constraints on axion-like dark matter using a floquet quantum detector,” Science Advances, vol. 8, no. 5, p. eabl8919, 2022.
  * [57] I. M. Bloch, R. Shaham, Y. Hochberg, E. Kuflik, T. Volansky, and O. Katz, “Constraints on axion-like dark matter from a serf comagnetometer,” Nat Commun, vol. 14, p. 5784, Sept. 2023.
  * [58] C. Abel, N. J. Ayres, G. Ban, G. Bison, K. Bodek, V. Bondar, E. Chanel, C. B. Crawford, M. Daum, B. Dechenaux, et al., “Search for ultralight axion dark matter in a side-band analysis of a 199hg free-spin precession signal,” SciPost Physics, vol. 15, p. 058, Aug. 2023.
  * [59] J. Lee, M. Lisanti, W. A. Terrano, and M. Romalis, “Laboratory constraints on the neutron-spin coupling of fev-scale axions,” Phys. Rev. X, vol. 13, p. 011050, Mar 2023.
  * [60] P. Carenza, T. Fischer, M. Giannotti, G. Guo, G. Martínez-Pinedo, and A. Mirizzi, “Improved axion emissivity from a supernova via nucleon-nucleon bremsstrahlung,” J. Cosmol. Astropart. Phys., vol. 2019, p. 016, Oct. 2019.
  * [61] N. Bar, K. Blum, and G. D’Amico, “Is there a supernova bound on axions?,” Phys. Rev. D, vol. 101, p. 123025, Jun 2020.
  * [62] Y. V. Stadnik and V. V. Flambaum, “Nuclear spin-dependent interactions: Searches for wimp, axion and topological defect dark matter, and tests of fundamental symmetries,” Eur. Phys. J. C, vol. 75, p. 110, Mar. 2015.


# SpeechCLIP+: Self-supervised multi-task representation learning for speech
via CLIP and speech-image data

###### Abstract

The recently proposed visually grounded speech model SpeechCLIP is an
innovative framework that bridges speech and text through images via CLIP
without relying on text transcription. On this basis, this paper introduces
two extensions to SpeechCLIP. First, we apply the Continuous Integrate-and-
Fire (CIF) module to replace a fixed number of CLS tokens in the cascaded
architecture. Second, we propose a new hybrid architecture that merges the
cascaded and parallel architectures of SpeechCLIP into a multi-task learning
framework. Our experimental evaluation is performed on the Flickr8k and
SpokenCOCO datasets. The results show that in the speech keyword extraction
task, the CIF-based cascaded SpeechCLIP model outperforms the previous
cascaded SpeechCLIP model using a fixed number of CLS tokens. Furthermore,
through our hybrid architecture, cascaded task learning boosts the performance
of the parallel branch in image-speech retrieval tasks.

Index Terms—  SpeechCLIP, visually grounded, self-supervised learning,
multimodal.

##  1 Introduction

Recent research in speech processing has focused on self-supervised learning
(SSL), which pre-trains upstream models on different pretext tasks [1], such
as generation or reconstruction [2, 3], contrastive learning [4, 5],
prediction [6, 7], and knowledge distillation [8, 9]. These pre-trained models
have been shown to outperform supervised models on certain downstream tasks,
like speech recognition [6]. In addition to unimodal SSL methods, it may be
beneficial to leverage different modalities, e.g., contextual image
information can help speech models recognize similar-sounding words, thereby
boosting the model’s performance on speech recognition [10]. Therefore, some
studies have utilized image-speech or image-text pairs for multi-modal
training.

Speech processing models trained using image-speech paired data are known as
Visually Grounded Speech (VGS) models. Many studies [11, 12] have found that
these VGS models are beneficial for many tasks. The recently proposed
SpeechCLIP [13] is a VGS model that leverages the pre-trained CLIP [14] image-
text model and the speech SSL model HuBERT [6]. HuBERT is trained with a
masked language modeling strategy, which provides good initialization for
general speech processing tasks [15]. CLIP employs contrastive learning to
pre-train robust image and text encoders by aligning semantically related
images and text captions. By aligning speech-image pairs during training,
SpeechCLIP learns to transform speech representations into the same embedding
space as the pre-trained CLIP, thereby aligning speech and text together
without transcriptional supervision. Moreover, M-SpeechCLIP [16], an extension
of SpeechCLIP, demonstrates the capability of multilingual speech-to-image
retrieval.

(a) hybrid SpeechCLIP

(b) hybrid SpeechCLIP+ with cascaded SpeechCLIP+

Fig. 1: Illustration of the proposed models. BN and VQ denote batch
normalization and vector quantization processes, respectively. (a) In hybrid
SpeechCLIP, the training loss combines the contrastive loss between the
leftmost CLS token and the output image representation of the CLIP image
encoder (the parallel branch same as parallel SpeechCLIP [13]) and the
contrastive loss between the output speech representation of the CLIP text
encoder for the remaining $K$ CLS tokens and the output image representation
of the CLIP image encoder (the cascaded branch same as cascaded SpeechCLIP
[13]). (b) In cascaded SpeechCLIP+, instead of extracting keyword information
through a fixed number of learnable CLS tokens, CIF is used to segment frame-
level features into subword-level keyword sequences. In hybrid SpeechCLIP+,
the parallel branch is based on parallel SpeechCLIP, and the cascaded branch
is based on cascaded SpeechCLIP+.

SpeechCLIP employs two architectures: parallel and cascaded. Parallel
SpeechCLIP aligns semantically related images and spoken captions with
utterance-level information, while cascaded SpeechCLIP aligns them with
intermediately extracted subword-level information. Each architecture has its
strengths: cascaded SpeechCLIP is capable of extracting a fixed number of
keywords directly from speech without any text supervision and image tagging
system. On the other hand, parallel SpeechCLIP excels at image-speech
retrieval tasks. In this paper, we aim to further improve two types of
architectures in terms of their strengths. Regarding the keyword extraction
task, we have identified some issues in [13]. First, the $K$ CLS tokens in the
original cascaded architecture tend to attend to similar segments in the input
speech, resulting in duplicate keywords in the output. Additionally, the fixed
number of CLS tokens may not be flexible enough when the duration of the input
speech varies significantly.

To address these issues, we apply the Continuous Integrate-and-Fire (CIF) [17]
module to replace a fixed number of CLS tokens in the original cascaded
architecture. We believe that CIF’s monotonic alignment method will alleviate
the issue of duplicate keywords. Furthermore, CIF enables the ability to
output a dynamic number of keywords. The experimental results also manifest
improvement when we do not consider duplicate keywords in the speech keyword
extraction task. For the image-speech retrieval tasks, we posit that
meaningful subwords’ information could be beneficial in addition to the
summary of the entire utterance. Thus, we propose a hybrid architecture, a
multi-task learning framework that integrates the learning objectives of
SpeechCLIP from both cascaded and parallel architectures. Experimental results
manifest the improvements on Flickr8k [18].

##  2 Method

As with the training of SpeechCLIP [13], the input to the model is a batch of
paired images and audio waveforms. Images are passed to CLIP’s image encoder
to extract image features, and audio waveforms are passed to a pre-trained SSL
speech encoder. CLIP’s text and image encoders are frozen during SpeechCLIP
training. They serve as projectors of the pre-trained image-text semantic
space. We follow SpeechCLIP by using HuBERT as our speech encoder. We also
freeze it during model training but employ a set of learnable weights to
perform a weighted sum of all its hidden states for audio feature extraction.

###  2.1 Continuous Integrate-and-Fire (CIF)

CIF is a soft, monotonic alignment method used to segment input sequential
features with assigned weights. Its effectiveness has been demonstrated in
previous speech segmentation tasks [19, 20]. Given sequential features
$\mathbf{x}=(x_{1},x_{2},\cdots,x_{T})$, CIF uses a convolution layer followed
by a feed-forward layer with a sigmoid function to generate the corresponding
weights $\boldsymbol{\alpha}=(\alpha_{1},\alpha_{2},\cdots,\alpha_{T})$.
Segmentation boundaries are determined by accumulating $\alpha_{t}$. As a
result, CIF generates a sequence $\mathbf{c}=(c_{1},c_{2},\cdots,c_{L})$ based
on its segmentation and aggregation process.

During training, for each input speech, a target segmentation number $L$ is
required, which determines the desired output length of $\mathbf{c}$.
Therefore, in [17], the following quantity loss $\mathcal{L}_{\text{QUA}}$ is
used to encourage CIF to produce the correct cumulative sum of
$\boldsymbol{\alpha}$,

$\mathcal{L}_{\text{QUA}}=\left|\sum_{t=1}^{T}\alpha_{t}-L\right|.$ (1)

Additionally, since the segmentation number may sabotage the training
stability, in [17], a scaling strategy is used to generate
$\boldsymbol{\alpha^{\prime}}=(\alpha^{\prime}_{1},\alpha^{\prime}_{2},\cdots,\alpha^{\prime}_{T})$,
where

$\alpha^{\prime}_{j}=\frac{\alpha_{j}}{\sum_{t=1}^{T}\alpha_{t}}\times L,$ (2)

for performing segmentation and aggregation. The scaling strategy ensures that
the output length of $\mathbf{c}$ is adjusted to the desired $L$.

###  2.2 Cascaded SpeechCLIP+

As shown in Fig. 1(b), in cascaded SpeechCLIP+, instead of extracting keyword
information through a fixed number of learnable CLS tokens, we apply CIF
mechanism to segment frame-level features into a subword-level keyword
sequence. During training, we set the target length of the quantity loss in
Eq. (1) to 5% of the input length, based on the average ratio between the
length of the Byte-Pair Encoding (BPE) token sequence and the output feature
length of HuBERT, as observed during our experimentation. Additionally, we
apply the scaling strategy in Eq. (2) to enhance training stability in the
first 5k steps. After the CIF mechanism, the same as cascaded SpeechCLIP, we
employ batch normalization and vector quantization to generate BPE tokens.
These tokens are then input into the CLIP text encoder to extract
representations, which are used to generate representations for computing
contrastive loss $\mathcal{L}_{\text{cascaded}}$ with image features. The
overall loss $\mathcal{L}$ for training cascaded SpeechCLIP+ is the linear
combination of $\mathcal{L}_{\text{cascaded}}$ and $\mathcal{L}_{\text{QUA}}$,

$\mathcal{L}=\lambda_{c}\times\mathcal{L}_{\text{cascaded}}+\lambda_{q}\times\mathcal{L}_{\text{QUA}},$
(3)

where $\lambda_{c}$ and $\lambda_{q}$ are adjustable weights.

###  2.3 Hybrid SpeechCLIP

Two types of SpeechCLIP are expected to benefit speech encoders from
semantically related images through different alignment methods. Parallel
SpeechCLIP enables speech encoders to benefit from utterance summarization,
while cascaded SpeechCLIP enables speech encoders to benefit from capturing
subword-level information from utterances. The advantages of combining the two
approaches are quite intuitive. As shown in Fig. 1(a), there is a total $K+1$
CLS tokens in hybrid SpeechCLIP, the parallel branch refers to the path with
the leftmost single CLS, while the cascaded branch refers to the path with the
remaining $K$ CLS tokens. The parallel branch is the same as the parallel
SpeechCLIP, the first CLS representation of the transformer encoder’s output
is used to compute the contrastive loss $\mathcal{L}_{\text{parallel}}$ with
image features. The cascaded branch is also the same as the cascaded
SpeechCLIP, the remaining $K$ CLS representations of the transformer encoder’s
output will be batch-normalized to match the mean and variance of CLIP’s BPE
token embeddings and vector-quantized into BPE tokens and then used as input
for the CLIP text encoder. The output of the text encoder is used to compute
the contrastive loss $\mathcal{L}_{\text{cascaded}}$ with image features. The
parallel and cascaded branches are trained jointly, and the overall loss
$\mathcal{L}$ is the linear combination of $\mathcal{L}_{\text{parallel}}$ and
$\mathcal{L}_{\text{cascaded}}$,

$\mathcal{L}=\lambda_{p}\times\mathcal{L}_{\text{parallel}}+\lambda_{c}\times\mathcal{L}_{\text{cascaded}},$
(4)

where $\lambda_{p}$ and $\lambda_{c}$ are adjustable weights.

###  2.4 Hybrid SpeechCLIP+

As shown in Fig. 1(b), in hybrid SpeechCLIP+, the parallel branch is the
parallel SpeechCLIP model, and the cascaded branch is the cascaded SpeechCLIP+
model. In the parallel branch, we apply one CLS token to compute the
contrastive loss $\mathcal{L}_{\text{parallel}}$ with image features. In the
cascaded branch, same as cascaded SpeechCLIP+, we apply CIF to segment frame-
level features into a subword-level keyword sequence. During training, we set
the target length of the quantity loss in Eq. (1) to 5% of the input length
and apply the scaling strategy in Eq. (2) in the first 5k steps. After the CIF
mechanism, we employ batch normalization and vector quantization to generate
BPE tokens. These tokens are then input into the CLIP text encoder to extract
representations, which are subsequently used to generate representations for
computing the contrastive loss $\mathcal{L}_{\text{cascaded}}$ with image
features. The parallel and cascaded branches are trained jointly with the
overall loss $\mathcal{L}$ as follows,

$\mathcal{L}=\lambda_{p}\times\mathcal{L}_{\text{parallel}}+\lambda_{c}\times\mathcal{L}_{\text{cascaded}}+\lambda_{q}\times\mathcal{L}_{\text{QUA}},$
(5)

where $\lambda_{p}$, $\lambda_{c}$, and $\lambda_{q}$ are adjustable weights.

Fig. 2: An example of keywords extracted by Cascaded SpeechCLIP+ from the
Flickr8k test set, showing the image, spoken caption, and extracted keywords
with corresponding segments.

##  3 Experiments and Results

###  3.1 Experimental setups

Datasets. Our models are trained on two image-audio datasets, namely the
Flickr8k Audio Captions Corpus [18] and SpokenCOCO [21]. Each image in both
datasets is paired with five spoken captions collected by humans reciting the
corresponding text captions. The training, development, and test sets in
Flickr8k contain 30k, 5k, and 5k utterances, respectively, while SpokenCOCO
has 565k, 25k, and 25k utterances in its training, development, and test sets.
Following SpeechCLIP [13], we use the Karpathy split for SpokenCOCO [21].

Table 1: BPE extraction performance on the Flickr8k and SpokenCOCO test sets. | w/o stop words | w/ stop words
---|---|---
Model | R | P | F1 | R | P | F1
| Flickr8k
Cascaded [13] | 7.39 | 0.94 | 1.66 | 8.3 | 1.68 | 2.79
Cascaded+ | 27.16 | 3.55 | 6.29 | 18.11 | 3.62 | 6.04
Cascaded(h) | 7.01 | 0.85 | 1.52 | 5.41 | 1.06 | 1.77
Cascaded(h)+ | 16.52 | 2.16 | 3.82 | 11.40 | 2.23 | 3.73
Cascaded Large [13] | 5.27 | 0.75 | 1.31 | 14.84 | 3.10 | 5.13
Cascaded+ Large | 27.27 | 3.73 | 6.56 | 20.74 | 4.2 | 6.99
Cascaded(h) Large | 6.73 | 0.86 | 1.53 | 6.18 | 1.25 | 2.08
Cascaded(h)+ Large | 21.69 | 2.92 | 5.15 | 15.48 | 3.08 | 5.14
| SpokenCOCO
Cascaded Large [13] | 2.08 | 0.39 | 0.65 | 12.96 | 2.89 | 4.72
Cascaded+ Large | 20.30 | 2.74 | 4.83 | 18.42 | 3.39 | 5.73
Cascaded(h) Large | 1.68 | 0.45 | 0.71 | 20.37 | 4.48 | 7.35
Cascaded(h)+ Large | 20.56 | 3.73 | 6.32 | 26.21 | 4.95 | 8.33

Compared models. In the following experiments, the baseline cascaded
SpeechCLIP, the proposed cascaded SpeechCLIP+, cascaded SpeechCLIP in hybrid
SpeechCLIP, and cascaded SpeechCLIP+ in hybrid SpeechCLIP+ are denoted as
Cascaded, Cascaded+, Cascaded(h) and Cascaded(h)+, respectively. Parallel
SpeechCLIP can be trained jointly with cascaded SpeechCLIP in hybrid
SpeechCLIP or cascaded SpeechCLIP+ in hybrid SpeechCLIP+. The model trained in
the former way is represented as Parallel(h) to distinguish it from the
Parallel(h)+ model trained in the latter way. Both models are compared with
the baseline parallel SpeechCLIP (denoted as Parallel). All models are trained
using the base or large model size setting in [13]. When the large model size
setting is used, the model is marked “Large”, e.g., Cascaded+ Large.

Implementation details. We use the same transformer encoder and the CLIP [14]
model as in SpeechCLIP [13]. The convolution layer of CIF consists of a single
one-dimensional convolution with $d_{model}$ channels, a stride of 1, and a
kernel width of 3. Here, $d_{model}$ is set to 768 for the base models and
1024 for the large models. The one-dimensional convolution is followed by a
dropout with a probability of 0.5 and a ReLU activation function. Regarding
the loss weights in Eqs. (3), (4), and (5), we set
$\lambda_{c}=\lambda_{p}=1.0$ and $\lambda_{q}=0.25$. We employ the same
optimization strategy as in SpeechCLIP [13]. All models are trained with a
batch size of 256 using two NVIDIA RTX 3090 GPUs with 24GB of memory. The
hybrid+ Large model trained on SpokenCOCO took 4 days to converge, while model
training under other conditions took less than two days.

###  3.2 Keyword extraction

As discussed in SpeechCLIP [13], we evaluate the keyword extraction ability of
cascaded SpeechCLIP+ through qualitative and quantitative analyses. Although
there are other some previous works [22, 23] on unsupervised Bag of Words
prediction, it is challenging to compare them in this task because we use
different levels of tokens (BPE tokens) for training, whereas they use word
tokens. Moreover, they require an image tagging system, while we do not. So we
only compare with [13] in this section.

For qualitative analysis, from the example from the Flickr8k test set in Fig.
2, we can observe the ability of cascaded SpeechCLIP+ to capture semantically
aligned keywords and their corresponding boundaries in the input speech. For
example, our model successfully extracted the words “MAN” and “ROCK” and found
their fragments in the audio waveform. In addition, the extracted word
“CLIMBING” is semantically correlated to ”CLIMB”.

Table 2: Word and BPE extraction performance of Cascaded+ Large on the Flickr8k test set. “Type” refers to the granularity of the units considered, while “Top $K$” represents the top $K$ BPEs retrieved by each CLS. |  | w/o stop words | w/ stop words
---|---|---|---
Type |  Top $K$ | R | P | F1 | R | P | F1
Word | 1 | 19.81 | 12.06 | 14.99 | 14.12 | 13.66 | 13.89
| 2 | 22.80 | 6.84 | 10.53 | 16.54 | 7.86 | 10.65
| 3 | 25.07 | 4.95 | 8.27 | 18.74 | 5.85 | 8.91
| 4 | 26.76 | 3.92 | 6.83 | 19.97 | 4.61 | 7.49
| 5 | 27.59 | 3.20 | 5.74 | 20.86 | 3.82 | 6.45
BPE | 1 | 19.17 | 12.99 | 15.49 | 13.79 | 13.97 | 13.88
| 2 | 22.29 | 7.53 | 11.26 | 16.29 | 8.25 | 10.96
| 3 | 24.52 | 5.54 | 9.03 | 18.46 | 6.24 | 9.32
| 4 | 26.31 | 4.46 | 7.63 | 19.77 | 5.01 | 7.99
| 5 | 27.27 | 3.73 | 6.56 | 20.74 | 4.2 | 6.99

For the quantitative analysis, the results are shown in Table 1. In this
experiment, the top 5 BPEs closest in cosine similarity to each CLS token (for
Cascaded and Cascaded(h)) or each CIF-segmented token (for Cascaded+ and
Cascaded(h)+) are retrieved. Performance is evaluated in terms of recall,
precision, and F1-score to the BPE sequence of the transcription of the spoken
caption. Considering that stop words in a sentence usually have little impact
on the semantics of the sentence, we provide complete evaluation results (see
w/ stop words) and evaluation results that ignore stop words (see w/o stop
words). Stop words are pronouns, articles, prepositions, and conjunctions. We
use the stop word set from the nltk python package.

It is clearly seen from the table that the proposed Cascaded+ models
significantly outperform their corresponding baseline Cascaded models
(Cascaded+ vs Cascaded, Cascaded(h)+ vs Cascaded(h), Cascaded+ Large vs
Cascaded Large, and Cascaded(h)+ Large vs Cascaded(h) Large). Surprisingly,
joint training of parallel and cascaded branches did not always bring
performance improvements to Cascaded and Cascaded+ models. The reason remains
to be further studied.

Table 2 shows the performance of word extraction. Since the previous
experiment shows that cascade SpeechCLIP+ Large (Cascaded+ Large) performs
best on the Flickr8k test set, we use it in this experiment. We use the
extracted neighboring BPEs to construct words. Top $K$ BPEs with $K$ from 1 to
5 are evaluated. The corresponding BPE extraction performance is also provided
for reference. As can been seen from Table 2, Top 1 BPE gives the best F1
score for word extraction with or without considering stop words. An increase
in $K$ can slightly improve the recall rate (R), but decrease the precision
rate (P). BPE extraction performance has the same trend as word extraction
performance.

Table 3: Image-speech retrieval performance on the Flickr8k and SpokenCOCO test sets. Parallel(h) refers to the parallel branch in hybrid SpeechCLIP (Fig. 1(a)), while Parallel(h)+ refers to the parallel branch in hybrid SpeechCLIP+ (Fig. 1(b)). |  Speech $\rightarrow$ Image |  Image $\rightarrow$ Speech
---|---|---
Model | R@1 | R@5 | R@10 | R@1 | R@5 | R@10
| Flickr8k
Parallel [13] | 26.7 | 57.1 | 70.0 | 41.3 | 73.9 | 84.3
Parallel(h) | 29.8 | 60.8 | 73.5 | 40.9 | 71.6 | 83.9
Parallel(h)+ | 29.3 | 60.1 | 73.7 | 39.7 | 72.8 | 83.4
Parallel Large [13] | 39.1 | 72.0 | 83.0 | 54.5 | 84.5 | 93.2
Parallel(h) Large | 43.1 | 75.6 | 85.2 | 54.3 | 85.1 | 93.5
Parallel(h)+ Large | 41.7 | 73.7 | 84.1 | 54.2 | 86.8 | 94.2
| SpokenCOCO
FaST-VGS${}_{\text{CTF}}$ [24] | 35.9 | 66.3 | 77.9 | 48.8 | 78.2 | 87.0
Parallel Large [13] | 35.8 | 66.5 | 78.0 | 50.6 | 80.9 | 89.1
Parallel(h) Large | 32.5 | 60.9 | 72.9 | 44.2 | 73.9 | 83.8
Parallel(h)+ Large | 36.5 | 66.3 | 77.9 | 51.0 | 80.0 | 88.5

###  3.3 Image-Speech retrieval

Next, we evaluate Parallel(h)+ and Parallel(h) on image-speech retrieval
tasks. The “Speech $\rightarrow$ Image” task is to retrieve the corresponding
image given a spoken caption, and the “Image $\rightarrow$ Speech” task is to
retrieve the corresponding spoken caption given an image.

From Table 3, we can see that on the Frickr8k dataset, the parallel branches
in hybrid SpeechCLIP are mostly better than the corresponding baseline
parallel models (Parallel(h) vs Parallel, Parallel(h)+ vs Parallel,
Parallel(h) Large vs Parallel Large, and Parallel(h)+ Large vs Parallel
Large). The results show that through a hybrid architecture, cascaded task
learning improves the performance of parallel branches in image-speech
retrieval tasks. The performance of Parallel(h) and Parallel(h)+ is
comparable, suggesting that both cascaded SpeechCLIP and cascaded SpeechCLIP+
can effectively enhance the parallel branch in joint training. However, on the
SpokenCOCO dataset, the performance of Parallel(h) Large and Parallel(h)+
Large is disappointing, especially Parallel(h) Large. The results in Table 1
show that Cascaded(h) Large and Cascaded(h)+ Large trained on SpokenCOCO have
a relatively better ability to extract stop word information than models under
other experimental settings. This may make Parallel(h) Large (or Parallel(h)+
Large) dominated by stop words captured by Cascaded(h) Large (or Cascaded(h)+
Large) jointly trained under the hybrid architecture.

##  4 Conclusions

In this paper, we attempt to boost the performance of pre-trained speech
models on downstream tasks by leveraging visual content and other pre-trained
modalities. We propose two extensions to SpeechCLIP. First, we apply the
Continuous Integrate-and-Fire (CIF) module to replace a fixed number of CLS
tokens in the cascaded architecture. Second, we propose a new hybrid
architecture that merges the cascaded and parallel architectures of SpeechCLIP
into a multi-task learning framework. In the keyword extraction task, our
cascaded SpeechCLIP+ model significantly outperforms the baseline cascaded
SpeechCLIP model [13]. Experimental results show that using CIF-segmented
representations are more effective and flexible than adding a fixed number of
CLS tokens when extracting subword and word information in speech. In the
image-speech retrieval task, experimental results show that both cascaded
SpeechCLIP and cascaded SpeechCLIP+ can effectively enhance the parallel
branch in hybrid SpeechCLIP through joint training. In future work, we will
investigate other unsupervised speech segmentation [19] and multi-task
learning methods [25].

## References

  * [1] Abdelrahman Mohamed, Hung-yi Lee, Lasse Borgholt, Jakob D Havtorn, Joakim Edin, Christian Igel, Katrin Kirchhoff, Shang-Wen Li, Karen Livescu, Lars Maaloe, et al.,  “Self-supervised speech representation learning: A review,”  IEEE Journal of Selected Topics in Signal Processing, vol. 16, no. 6, pp. 1179–1210, 2022.

  * [2] Aaron Van Den Oord, Oriol Vinyals, et al.,  “Neural discrete representation learning,”  in NIPS, 2017.
  * [3] Yu-An Chung, Wei-Ning Hsu, Hao Tang, and James Glass,  “An unsupervised autoregressive model for speech representation learning,”  in Interspeech, 2019.
  * [4] Alexei Baevski, Steffen Schneider, and Michael Auli,  “vq-wav2vec: Self-supervised learning of discrete speech representations,”  in ICLR, 2020.
  * [5] Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli,  “wav2vec 2.0: A framework for self-supervised learning of speech representations,”  in NeurIPS, 2020.
  * [6] Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed,  “HuBERT: Self-supervised speech representation learning by masked prediction of hidden units,”  IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2021.
  * [7] Sanyuan Chen, Chengyi Wang, Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, et al.,  “WavLM: Large-scale self-supervised pre-training for full stack speech processing,”  IEEE Journal of Selected Topics in Signal Processing, 2022.
  * [8] Heng-Jui Chang, Shu-wen Yang, and Hung-yi Lee,  “DistilHuBERT: Speech representation learning by layer-wise distillation of hidden-unit BERT,”  in ICASSP, 2022.
  * [9] Alexei Baevski, Wei-Ning Hsu, Qiantong Xu, Arun Babu, Jiatao Gu, and Michael Auli,  “data2vec: A general framework for self-supervised learning in speech, vision and language,”  in ICML, 2022.
  * [10] Felix Sun, David Harwath, and James Glass,  “Look, listen, and decode: Multimodal speech recognition with images,”  in SLT, 2016.
  * [11] Puyuan Peng and David Harwath,  “Word discovery in visually grounded, self-supervised speech models,”  arXiv preprint arXiv:2203.15081, 2022.
  * [12] Puyuan Peng and David F. Harwath,  “Self-supervised representation learning for speech using visual grounding and masked language modeling,”  arXiv:2202.03543, 2022.
  * [13] Yi-Jen Shih, Hsuan-Fu Wang, Heng-Jui Chang, Layne Berry, Hung-yi Lee, and David Harwath,  “SpeechCLIP: Integrating speech with pre-trained vision and language model,”  in SLT, 2023.
  * [14] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.,  “Learning transferable visual models from natural language supervision,”  in ICML, 2021.
  * [15] Shu-wen Yang, Po-Han Chi, Yung-Sung Chuang, Cheng-I Jeff Lai, Kushal Lakhotia, Yist Y Lin, Andy T Liu, Jiatong Shi, Xuankai Chang, Guan-Ting Lin, et al.,  “SUPERB: Speech processing universal performance benchmark,”  in Interspeech, 2021.
  * [16] Layne Berry, Yi-Jen Shih, Hsuan-Fu Wang, Heng-Jui Chang, Hung-yi Lee, and David Harwath,  “M-SpeechCLIP: Leveraging large-scale, pre-trained models for multilingual speech to image retrieval,”  in ICASSP, 2023.
  * [17] Linhao Dong and Bo Xu,  “CIF: Continuous integrate-and-fire for end-to-end speech recognition,”  in ICASSP, 2020.
  * [18] Cyrus Rashtchian, Peter Young, Micah Hodosh, and Julia Hockenmaier,  “Collecting image annotations using Amazon’s Mechanical Turk,”  in NAACL HLT Workshop on Creating Speech and Language Data with Amazon’s Mechanical Turk, 2010.
  * [19] Yen Meng, Hsuan-Jui Chen, Jiatong Shi, Shinji Watanabe, Paola Garcia, Hung-yi Lee, and Hao Tang,  “On compressing sequences for self-supervised speech models,”  in SLT, 2023.
  * [20] Zhiyun Fan, Linhao Dong, Meng Cai, Zejun Ma, and Bo Xu,  “Sequence-level speaker change detection with difference-based continuous integrate-and-fire,”  IEEE Signal Processing Letters, 2022.
  * [21] Andrej Karpathy and Li Fei-Fei,  “Deep visual-semantic alignments for generating image descriptions,”  in CVPR, 2015.
  * [22] Herman Kamper, Shane Settle, Gregory Shakhnarovich, and Karen Livescu,  “Visually grounded learning of keyword prediction from untranscribed speech,”  arXiv preprint arXiv:1703.08136, 2017.
  * [23] Leanne Nortje and Herman Kamper,  “Towards visually prompted keyword localisation for zero-resource spoken languages,”  in 2022 IEEE Spoken Language Technology Workshop (SLT), 2023.
  * [24] Puyuan Peng and David Harwath,  “Fast-slow transformer for visually grounding speech,”  in ICASSP, 2022.
  * [25] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi,  “Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation,”  in International Conference on Machine Learning. PMLR, 2022.


Results and speculations concerning Comer relation algebras and the flexible
atom conjecture
Jeremy F. Alm

Department of Mathematics

Lamar University

Beaumont, TX 77710

USA

<EMAIL_ADDRESS>

David A. Andrews
Department of Mathematics
University of Dallas
Irving, TX 75062
USA
<EMAIL_ADDRESS>

###### Abstract

We study some finite integral symmetric relation algebras whose forbidden
cycles are all 2-cycles. These algebras arise from a finite field construction
due to Comer. We consider conditions that allow other finite algebras to embed
into these Comer algebras, and as an application give the first known finite
representation of relation algebra $34_{65}$, one of whose atoms is flexible.
We conclude with some speculation about how the ideas presented here might
contribute to a proof of the flexible atom conjecture.

##  1 Introduction

In this paper, we explore some material related to the construction of Ramsey
algebras over $\mathbb{F}_{p}$ for primes $p$. The method of construction can
be found in Comer’s paper [8]. A _Ramsey algebra in $m$ colors_ is a partition
of a set $U\times U$ into disjoint binary relations $Id,A_{0},\ldots,A_{m-1}$
such that

  1. (I.)

$A^{-1}_{i}=A_{i}$;

  2. (II.)

$A_{i}\circ A_{i}=A^{c}_{i}$;

  3. (III.)

for $i\neq j$, $A_{i}\circ A_{j}=Id^{c}$.

Here, $Id=\\{(x,x):x\in U\\}$ is the identity over $U$, $\circ$ is relational
composition, -1 is relational inverse, and c is complementation with respect
to $U\times U$.

Ramsey algebras have been constructed for $m$ colors for all $m\leq 2000$,
save for $m=8$ and $m=13$. Condition (II.) implies that the relations $A_{i}$
are triangle free, i.e., $A_{i}\circ A_{i}\cap A_{i}=\emptyset$. All known
existence results are computational, and use the following finite field method
of Comer: Fix $m\in\mathbb{Z}^{+}$, and let $X_{0}=H$ be a multiplicative
subgroup of $\mathbb{F}_{q}$ of order $(q-1)/m$, where $q\equiv 1\ \text{(mod
$2m$)}$. Let $X_{1},\ldots X_{m-1}$ be its cosets; specifically, let
$X_{i}=g^{i}X_{0}=\\{g^{am+i}:a\in\mathbb{Z}^{+}\\}$, where $g$ is a generator
of $\mathbb{F}_{q}^{\times}$. Suppose the following conditions obtain:

  1. (i.)

$-X_{i}=X_{i}$;

  2. (ii.)

$X_{i}+X_{i}=\mathbb{F}_{q}\setminus X_{i}$,

  3. (iii.)

for $i\neq j$, $X_{i}+X_{j}=\mathbb{F}_{q}\setminus\\{0\\}$.

Then define $A_{i}=\\{(x,y)\in\mathbb{F}_{q}\times\mathbb{F}_{q}:x-y\in
X_{i}\\}$. It is easy to check that (i.)-(iii.) imply (I.)-(III.), and we get
a Ramsey algebra. Condition (ii.) implies that all the $X_{i}$’s are sum-free.
A few examples of Ramsey algebras have been constructed over non-prime fields,
but for the most part, attention has been restricted to prime fields.

In the present paper we study algebras that are not Ramsey algebras but that
are similar in the sense that they have exactly one equivalence class of
forbidden cycles. For these algebras, the size of $X_{i}+X_{i}$ is the same as
for Ramsey algebras, but the $X_{i}$’s are not sum-free. The results will lead
to some insights and speculations about further avenues for research.

##  2 Some definitions and lemmas

Comer’s construction yields algebras that have “rotational symmetry”:

######  Lemma 1.

Let $n\in\mathbb{Z}^{+}$ and let $p=nk+1$ be a prime number and $g$ a
primitive root modulo $p$.

For $i\in\left\\{0,1,\ldots,n-1\right\\}$, define

$X_{i}=\left\\{g^{i},g^{n+i},g^{2n+i},\ldots,g^{(k-1)n+i}\right\\}\subseteq\mathbb{F}_{p}.$

Then $(X_{0}+X_{j})\cap X_{k}=\emptyset$ if and only if $(X_{i}+X_{i+j})\cap
X_{i+k}=\emptyset$.

The lemma is trivial to prove: just multiply through by $g^{i}$!

######  Definition 2.

Let $p=nk+1$ be prime with $k$ even, and let the $X_{i}$’s as be as in Lemma
1. Let $C(p,n)$ denote the algebra formed by the $A_{i}$’s. Let
$\operatorname{Aut}(C(p,n))$ denote the set of permutations of indices that
induce an automorphism of $C(p,n)$, that is,

$\operatorname{Aut}(C)=\\{\pi\in S_{n}:A_{i}\circ A_{j}\supseteq
A_{k}\Leftrightarrow A_{\pi(i)}\circ A_{\pi(j)}\supseteq A_{\pi(k)}\\}.$

Of course, the condition on the relations $A_{i}$ is equivalent to the
following condition on the $X_{i}$’s:

$X_{i}+X_{j}\supseteq X_{k}\Leftrightarrow X_{\pi(i)}+X_{\pi(j)}\supseteq
X_{\pi(k)}.$

Lemma 1 implies that for any Comer algebra $C(p,n)$ in $n$ colors, we have
that $\operatorname{Aut}(C)$ contains a subgroup isomorphic to
$\mathbb{Z}/n\mathbb{Z}$.

######  Definition 3.

Let $\mathcal{A}_{n}([i,i+j,i+\ell])$ denote the integral RA with $n$
symmetric diversity atoms $a_{0},\ldots,a_{n-1}$ whose forbidden cycles are
those of the form $a_{i}a_{i+j}a_{i+\ell}$, with indices considered modulo
$n$.

We will be particularly interested in $\mathcal{A}_{n}([i,i,i+1])$.

######  Lemma 4.

$\operatorname{Aut}(\mathcal{A}_{n}([i,i,i+1]))\cong\mathbb{Z}/n\mathbb{Z}$.

###### Proof.

Suppose $\pi\in\operatorname{Aut}(\mathcal{A}_{n}([i,i,i+1]))$ and $\pi(0)=x$.
Since $a_{0}a_{0}a_{1}$ is forbidden, $a_{\pi(0)}a_{\pi(0)}a_{\pi(1)}$ is
forbidden as well, so $a_{x}a_{x}a_{\pi(1)}$ is forbidden. But this forces
$\pi(1)=x+1\pmod{n}$, since if $xxy$ is forbidden, $y$ must be $x+1$.
Similarly, $\pi(2)$ must be $x+2\pmod{n}$, and so on. So $\pi$ must take the
form $\pi(s)=x+s\pmod{n}$. All such permutations are clearly in
$\operatorname{Aut}(\mathcal{A}_{n}([i,i,i+1]))$, and we have shown they are
the only ones. So
$\operatorname{Aut}(\mathcal{A}_{n}([i,i,i+1]))\cong\mathbb{Z}/n\mathbb{Z}$. ∎

######  Lemma 5.

If $\gcd(j,n)=1$, then
$\mathcal{A}_{n}([i,i,i+j]))\cong\mathcal{A}_{n}([i,i,i+1])$.

###### Proof.

Let
$\rho:\mathrm{At}(\mathcal{A}_{n}([i,i,i+1]))\to\mathrm{At}(\mathcal{A}_{n}([i,i,i+j]))$
be given by $a_{i}\mapsto a_{j\cdot i\pmod{n}}$. Since $\gcd(j,n)=1$, $\rho$
is a bijection. It is easy to check that $\rho$ preserves the forbidden
cycles. ∎

The next lemma tells us that for Comer algebras, the isomorphism in Lemma 5
arises in a particularly nice way.

######  Lemma 6.

If $C(p,n)$ has forbidden cycles $[X_{i},X_{i},X_{i+j}]$ and $\gcd(j,n)=1$,
then $X_{j}$ contains a primitive root $g$, and reindexing using $g$ as a
generator will give forbidden cycles $[X_{i},X_{i},X_{i+1}]$.

###### Proof.

Let $g$ be the primitive root that gives the indexing with forbidden cycles
$[X_{i},X_{i},X_{i+j}]$. Now $g^{\ell}$ is also a primitive root modulo $p$ if
$\gcd(\ell,p-1)=1$. We want $g^{\ell}\in X_{j}$, so we want to find an integer
$a$ with $\gcd(an+j,p-1)=1$. Since $\gcd(j,n)=1$, Dirichlet’s theorem on
primes in arithmetic progressions gives some prime $p^{\prime}=an+j$, and
clearly $\gcd(p^{\prime},p-1)=1$. Then $g^{p^{\prime}}$ is a primitive root
and is in $X_{j}$. ∎

######  Lemma 7.

If $\gcd(j,n)>1$, then $\operatorname{Aut}(\mathcal{A}_{n}([i,i,i+j]))$
contains a non-identity permutation $\pi$ that has fixed points. Hence
$\mathcal{A}_{n}([i,i,i+j])\not\cong\mathcal{A}_{n}([i,i,i+1])$.

###### Proof.

Let $x=\gcd(j,n)>1$. Consider the permutation $\pi=(0\ x\ 2x\ 3x\ldots)$,
written in cycle notation. We claim that
$\pi\in\operatorname{Aut}(\mathcal{A}_{n}([i,i,i+j])))$. Consider the
forbidden cycle $a_{0}a_{0}a_{j}$. Write $j=bx$ for some positive integer $b$.
Under $\pi$, this cycle $a_{0}a_{0}a_{bx}$ gets mapped to
$a_{x}a_{x}a_{(b+1)x}$, and since $(b+1)x=x+j$, the cycle
$a_{x}a_{x}a_{(b+1)x}$ is forbidden. In fact, $\pi$ just permutes the
forbidden cycles $a_{\ell x}a_{\ell x}a_{\ell x+j}$ and leaves the other
forbidden cycles fixed. ∎

######  Example 8.

Consider $\mathcal{A}_{6}([i,i,i+2])$. Then $j=x=2$. The permutation $(0\ 2\
4)$ permutes the forbidden cycles $a_{0}a_{0}a_{2}$, $a_{2}a_{2}a_{4}$, and
$a_{4}a_{4}a_{0}$. See Figure 1.

Figure 1: Depiction of the action of the permutation $(0\ 2\ 4)$ on the
forbidden cycles of $\mathcal{A}_{6}([i,i,i+2])$.

The following lemma from Alon and Bourgain gives us just what we need to show
that if $p$ is large relative to $n$, then $C(p,n)$ has only flexible
diversity atoms.

######  Lemma 9 ([6], Proposition 1.4).

Let $q$ be a prime power and let $A$ be a multiplicative subgroup of the
finite field $\mathbb{F}_{q}$ of size $|A|=d\geq q^{1/2}$. Then, for any two
subsets $B$, $C\subset\mathbb{F}_{q}$ satisfying $|B||C|\geq q^{3}/d^{2}$,
there are $x\in B$ and $y\in C$ so that $x+y\in A$.

######  Lemma 10.

If $p>n^{4}+5$, then every diversity atom of $C(p,n)$ is flexible.

###### Proof.

We need to show $(X_{i}+X_{j})\cap X_{0}\neq\varnothing$ for arbitrary $i$ and
$j$. Set $q=p$, $A=X_{0}$, $B=X_{i}$, and $C=X_{j}$ in Lemma 9. Then
$|A|=|B|=|C|=(p-1)/n$. Then we need $|B||C|\geq q^{3}/d^{2}$, which translates
to $(p-1)^{4}\geq n^{4}p^{3}$, which is satisfied when $p>n^{4}+5$. Then all
diversity cycles are mandatory by Lemma 1. ∎

##  3 Computational results

When we started this project, we began with the question, “Why does Comer’s
construction generate Ramsey algebras for (almost) any number of colors? Is
there something special about forbidding just $[i,i,i]$? What if we just
forbid a 2-cycle (or rather a class of 2-cycles) like $[i,i,i+1]$ or
$[i,i,i+2]$?”

Some data are summarized in Table 1 below. While for some small $n$, there is
no construction of a Comer RA representation for $\mathcal{A}_{n}([i,i,i+j])$
for $j=0,1,2$, it would seem for large enough $n$ there is always some modulus
$p$ that works.

n | $\mathcal{A}_{n}([i,i,i]))$ | $\mathcal{A}_{n}([i,i,i+1]))$ | $\mathcal{A}_{n}([i,i,i+2]))$
---|---|---|---
1 | 2 | x | x
2 | 5 | x | x
3 | 13 | x | x
4 | 41 | x | x
5 | 71 | 61 | –
6 | 97 | 109 | x
7 | 491 | 127 | –
8 | x | 257 | x
9 | 523 | 307 | –
10 | 1181 | 641 | 421
11 | 947 | 331 | –
12 | 769 | 673 | x
13 | x | 667 | –
14 | 1709 | 953 | x
15 | 1291 | x | x
16 | 1217 | 2593 | 1697
Table 1: Smallest modulus for a representation over a $C(p,n)$, or x if none
exists. The “–” indicates that an entry is redundant (in light of Lemma 5).

Representations of $\mathcal{A}_{n}([i,i,i+1])$ exist for all $35\leq n\leq
500$. In Figure 2, we compare the smallest modulus $p$ for representations
over $C(p,n)$ for $\mathcal{A}_{n}([i,i,i])$ vs $\mathcal{A}_{n}([i,i,i+1])$.
The growth is a bit slower for the latter.

Figure 2: Smallest modulus $p$ over which $\mathcal{A}_{n}([i,i,i])$ and
$\mathcal{A}_{n}([i,i,i+1])$ are representable as a $C(p,n)$

##  4 A cyclic group representation of relation algebra $34_{65}$

As an application, we give the first known finite representation of $34_{65}$.
Relation algebra $34_{65}$ has four symmetric atoms $1^{\prime}$, $a$, $b$,
and $c$, with forbidden cycles $bbc$ and $ccb$. The atom $a$ is flexible,
hence $34_{65}$ is representable over a countable set.

We noticed that it would be sufficient to find a prime $p=nk+1$, $k$ and $n$
both even, such that $C(p,n)$ has $[i,i,i+n/2]$ as its only forbidden class.
Then we could map $b$ to $X_{0}$, map $c$ to $X_{n/2}$, and map $a$ to the
union of all the other $X_{i}$’s; in other words, $34_{65}$ embeds in
$\mathcal{A}_{n}([i,i,i+n/2]))$ for all even $n>4$. There’s no limit to how
big $p$ can be, since $n$ can also be as large as necessary; we just throw
“everything else” into the image of $a$. A computer search using the fast
algorithm from [5] quickly found a hit: for $p=3697$ and $n=24$, $[i,i,i+12]$
alone is forbidden. (This indexing is for $g=5$.)

##  5 Speculations

The proof of Lemma 9, upon which the proof of Lemma 10 relies, makes use of
the fact that multiplicative subgroups of $\mathbb{F}_{p}^{\times}$ are
quasirandom (in the sense of [7]). This is why we get a polynomial bound in
Lemma 10 rather than an exponential bound, which is what one would expect to
get from Ramsey-theoretic considerations. (For example, to force the 1-cycles
$[i,i,i]$ to be mandatory without appealing to the quasirandomness of the
color classes would require at least $3^{n}$, and the best we can _prove_ is
$e\cdot n!$.)

It appears that a similar quasirandomness phenomenon exists at the level of
the atom structures of Comer RAs. While there is no precise definition of
quasirandomness for atom structures that would be analogous to that for
subsets of $\mathbb{F}_{p}$, for example, we can make some observations.

For example, the first author has used Comer RAs to construct finite cyclic
group representations (over $\mathbb{F}_{p}$ for various $p$) for $33_{37}$,
$35_{37}$, $32_{65}$, $34_{65}$ (see above), $59_{65}$, $77_{83}$, $78_{83}$,
$80_{83}$, $82_{83}$, $1310_{1316}$, $1313_{1316}$, and $1315_{1316}$, most of
which were not previously known to be finitely representable. Each one of
these except $77_{83}$ has a flexible atom. In each case, the idea was the
same: conceive of a forbidden class (or classes) $[i,j,\ell]$ such that if
$C(p,n)$ had $[i,j,\ell]$ as its forbidden class, the algebra in question
would embed into $C(p,n)$. In the example in Section 4, we looked for
$[i,i,i+n/2]$ to be the only forbidden class.

Let’s consider $35_{37}$, which has diversity atoms $a$, $r$, $\breve{r}$. The
forbidden cycle is $rrr$. For $35_{37}$, we looked for $[i,i,i]$ to be the
only forbidden class, with $k$ odd, so all the diversity atoms were
asymmetric. (In this case, $[i,i,i]$ has other equivalent forms, like
$[i,i+n/2,i]$ and $[i+n/2,i,i+n/2]$, since $k$ is odd.) We don’t have to worry
about $n$ getting too big, because we can just map $r$ to $X_{0}$, $\breve{r}$
to $X_{n/2}$, and $a$ to the union of all the other $X_{i}$s. Since $a$ is
flexible, it can’t be “too big”. The smallest example is $C(3221,20)$, which
contains $35_{37}$ (as well as $82_{83}$!) as a subalgebra. See [2].

To take another example (previously published in [4]), consider $59_{65}$,
with forbidden cycles $bbb$ and $cbb$. The Comer RA $C(113,8)$ has forbidden
cycles $[i,i,i]$, $[i,i,i+6]$, $[i,i,i+7]$ (for $g=8$). Mapping $b$ to
$X_{0}$, $c$ to $X_{6}\cup X_{7}$, and $a$ to everything else yields a
representation.

In every single case, once we were able to cook up a “forbidden scheme” that
would allow us to embed our algebra into a Comer RA, we quickly found a
$C(p,n)$ with that forbidden scheme. Sometimes $n$ (and hence $p$) had to be
bigger than seemed necessary, but nonetheless we found an instance. The only
limitation seems to be our ability to imagine forbidden schemes that would
allow embeddings. (For example, $33_{65}$ forbids $ccc$, $bcc$, and $cbb$. We
can’t think of a forbidden scheme such that $33_{65}$ would embed in the
corresponding Comer RA. Can you?)

In sum, it seems that the $C(p,n)$s obey the following (admittedly vague)
rule:

> _Any forbidden scheme not ruled out by obvious considerations will be
> instantiated by some $C(p,n)$ for big enough $n$ and $p$._

A good example is that of Ramsey algebras, which are instantiated by $C(p,n)$s
for all $n\leq 2000$ except $n=8$ and $n=13$. (See [1]. We presume that they
are instantiated for all but finitely many $n$.) Why should this be so, unless
something like the rule above holds?

(One caveat: it isn’t quite so obvious what the “obvious considerations”
should be. A first guess is that $\operatorname{Aut}(C)$ contains a subgroup
isomorphic to $\mathbb{Z}/n\mathbb{Z}$. However, Roger Lyndon’s algebras from
projective lines satisfy this condition, but are not representable over any
cyclic group. See [9].)

These observations might provide a path to finding a proof of the Flexible
Atom Conjecture. The first step would be to formulate a suitable notion of
quasirandomness for (sequences of) relation algebra atom structures. (See also
[3].)

######  Conjecture 11.

Any finite integral RA with a flexible atom embeds in some Comer RA, hence is
representable over a finite cyclic group.

##  6 Open questions

######  Problem 1.

Formulate a suitable notion of quasirandomness for sequences of relation
algebra atom structures.

######  Problem 2.

Find a forbidden scheme that would admit a representation of $33_{65}$.

######  Problem 3.

Algebra $\mathcal{A}_{3}([i,i,i+1])$ is $42_{65}$ and is not representable.
Algebra $\mathcal{A}_{4}([i,i,i+1])$ is not representable via Comer’s method.
Is it representable by some other method?

######  Problem 4.

Algebra $\mathcal{A}_{6}([i,i,i+2])$ is not representable via Comer’s method.
Is it representable by some other method?

######  Problem 5.

Algebra $\mathcal{A}_{5}([i,i,i+1])$ is representable over $\mathbb{F}_{61}$.
Is it representable over an infinite set? Over $\mathbb{Z}$?

## References

  * [1] Jeremy F. Alm.  401 and beyond: improved bounds and algorithms for the Ramsey algebra search.  J. Integer Seq., 20(8):Art. 17.8.4, 10, 2017.
  * [2] Jeremy F. Alm.  Directed Ramsey and anti-Ramsey algebras and the flexible atom conjecture.  2018\.  arXiv:1901.06781.
  * [3] Jeremy F. Alm.  Random relation algebras.  Algebras and Lattices in Hawaii-2018, 2018.  arXiv:1802.06170.
  * [4] Jeremy F. Alm and Roger D. Maddux.  Finite representations for two small relation algebras.  Algebra Universalis, 79(4):Art. 87, 4, 2018.
  * [5] Jeremy F. Alm and A. Ylvisaker.  A fast coset-translation algorithm for computing the cycle structure of Comer relation algebras over $\mathbb{Z}/p\mathbb{Z}$.  To Appear.
  * [6] Noga Alon and Jean Bourgain.  Additive patterns in multiplicative subgroups.  Geom. Funct. Anal., 24(3):721–739, 2014.
  * [7] F. R. K. Chung and R. L. Graham.  Quasi-random subsets of $Z_{n}$.  J. Combin. Theory Ser. A, 61(1):64–86, 1992.
  * [8] S. D. Comer.  Color schemes forbidding monochrome triangles.  In Proceedings of the fourteenth Southeastern conference on combinatorics, graph theory and computing (Boca Raton, Fla., 1983), volume 39, pages 231–236, 1983.
  * [9] R. C. Lyndon.  Relation algebras and projective geometries.  Michigan Math. J., 8:21–28, 1961.