# The Dragonfly Galaxy. III. Jet-brightening of a High-redshift Radio Source
Caught in a Violent Merger of Disk Galaxies

Sophie Lebowitz  Steward Observatory, Department of Astronomy, University of
Arizona, 993 North Cherry Avenue, Tucson, AZ, 85721, USA  Department of
Astronomy, The Ohio State University, Columbus OH 43210 USA  Bjorn Emonts
National Radio Astronomy Observatory, 520 Edgemont Road, Charlottesville, VA
22903, USA  Donald M. Terndrup  Department of Astronomy, The Ohio State
University, Columbus OH 43210 USA  Joseph N. Burchett  Department of
Astronomy, New Mexico State University, Las Cruces, NM 88003, USA  J. Xavier
Prochaska  University of California, 1156 High Street, Santa Cruz, CA 95064,
USA  Kavli Institute for the Physics and Mathematics of the Universe, The
University of Tokyo, 5-1-5 Kashiwanoha, Kashiwa, 277-8583, Japan  Division of
Science, National Astronomical Observatory of Japan, 2-21-1 Osawa, Mitaka,
Tokyo 181-8588, Japan  Guillaume Drouart  International Centre for Radio
Astronomy Research, Curtin University, 1 Turner Avenue, Bentley, Western
Australia 6102, Australia  Montserrat Villar-Martín  Centro de Astrobiología,
CSIC-INTA, Ctra. de Torrejón a Ajalvir, km 4, 28850 Torrejón de Ardoz, Madrid,
Spain  Matthew Lehnert  Université Lyon 1, ENS de Lyon, CNRS UMR5574, Centre
de Recherche Astrophysique de Lyon, F-69230 Saint-Genis-Laval, France  Carlos
De Breuck  European Southern Observatory, Karl Schwarzschild Strasse 2, 85748
Garching bei M$\ddot{u}$nchen Germany  Joël Vernet  European Southern
Observatory, Karl Schwarzschild Strasse 2, 85748 Garching bei M$\ddot{u}$nchen
Germany  Katherine Alatalo  Space Telescope Science Institute, 3700 San Martin
Dr, Baltimore, MD 21218, USA  Johns Hopkins University, Department of Physics
and Astronomy, Baltimore, MD 21218, USA

###### Abstract

The Dragonfly Galaxy (MRC 0152-209), the most infrared-luminous radio galaxy
at redshift $z$ $\sim$ 2, is a merger system containing a powerful radio
source and large displacements of gas. We present kpc-resolution data from
ALMA and the VLA of carbon monoxide (6-5), dust, and synchrotron continuum,
combined with Keck integral-field spectroscopy. We find that the Dragonfly
consists of two galaxies with rotating disks that are in the early phase of
merging. The radio jet originates from the northern galaxy and brightens when
it hits the disk of the southern galaxy. The Dragonfly Galaxy therefore likely
appears as a powerful radio galaxy because its flux is boosted into the regime
of high-z radio galaxies by the jet-disk interaction. We also find a molecular
outflow of $(1100\,\pm\,550)$ M⊙ yr-1 associated with the radio host galaxy,
but not with the radio hot-spot or southern galaxy, which is the galaxy that
hosts the bulk of the star formation. Gravitational effects of the merger
drive a slower and longer lived mass displacement at a rate of
$(170\,\pm\,40)$ M⊙ yr-1, but this tidal debris contain at least as much
molecular gas mass as the much faster outflow, namely $M_{\rm
H2}=(3\,\pm\,1)\times 10^{9}\ (\alpha_{\rm CO}/0.8)\ {\rm M}_{\odot}$. This
suggests that both the AGN-driven outflow and mass transfer due to tidal
effects are important in the evolution of the Dragonfly system. The Keck data
show Ly$\alpha$ emission spread across 100 kpc, and C IV and He II emission
across 35 kpc, confirming the presence of a metal-rich and extended
circumgalactic medium previously detected in CO(1-0).

High-redshift galaxies – Galaxy mergers – Galaxy interactions – Radio galaxies
– Radio jets – Radio loud quasars – Starburst galaxies – Ultraluminous
infrared galaxies – Circumgalactic medium – Radio astronomy – Submillimeter
astronomy

††facilities: ALMA, VLA, Keck††software: CASA (CASA Team et al., 2022), KCWI
data reduction pipeline (Neill et al., 2018)

##  1 Introduction

For decades, high-redshift radio galaxies have served as excellent
laboratories for studying the early formation and evolution of galaxies (Miley
& De Breuck, 2008). High-redshift radio galaxies, or HzRGs in short, are often
defined to have a redshift of $z$ $\gtrsim$ 2 and a rest-frame radio power of
$P_{\rm 500\,MHz\,(rest)}>10^{27.5}\,{\rm W}\,{\rm Hz}^{-1}$ (Miley & De
Breuck, 2008). The bright synchrotron emission from their powerful, steep-
spectrum radio sources has long been used as a beacon for tracing the faint
optical signatures of massive galaxies and proto-clusters (e.g., Roettgering
et al., 1994; Chambers et al., 1996; Carilli et al., 1997; Pentericci et al.,
2000, see also reviews by McCarthy 1993 and Miley & De Breuck 2008). High
redshift radio galaxies occupy the high end of galaxy masses, with $M_{*}$ =
1011-12 M⊙ (Seymour et al. 2007; see also Pentericci et al. 2001, De Breuck et
al. 2010, Rocca-Volmerange et al. 2013). They also possess characteristics
that can give valuable insight into the process of galaxy evolution, such as
high rates of star formation and strong Active Galactic Nucleus (AGN) activity
(e.g., Barthel et al., 2012; Drouart et al., 2016; Wilkes et al., 2013), as
well as jet-driven gas outflows (e.g., Villar-Martín et al., 1999; Nesvadba et
al., 2017). High-$z$ radio galaxies are often found in overdense regions, as
expected from the progenitors of giant elliptical galaxies that occupy the
centers of galaxy clusters (e.g., Pentericci et al., 1997; Venemans et al.,
2007; Hatch et al., 2009; Galametz et al., 2012; Wylezalek et al., 2013;
Dannerbauer et al., 2014). These properties had indicated that high-$z$ radio
galaxies represent the most active episodes in the early evolution of massive
galaxies, but more recent studies reveal that many high-$z$ radio galaxies may
be on the way to quenching (e.g., Man et al., 2019; Falkendal et al., 2019).
Accurate techniques for determining the complex spectral energy distributions
(e.g., Drouart & Falkendal, 2018) are starting to reveal that many of the
radio host galaxies may even fall below the main sequence of star-forming
galaxies (Falkendal et al., 2019).

The Dragonfly Galaxy, MRC 0152-209, is a radio galaxy at $z=1.92$ that shows
extreme characteristics, even for a high-$z$ radio galaxy. Its starburst
infrared luminosity is in the regime of Hyper-Luminous Infra-Red Galaxies
(HyLIRGs; $L_{\rm IR}\geq 10^{13}\,{\rm L}_{\odot}$), which is roughly an
order of magnitude higher than other HzRGs at $z\sim 2$ (Drouart et al., 2014;
Falkendal et al., 2019). This reflects a star formation rate ($SFR$) $\sim
2000\,{\rm M}_{\odot}\,{\rm yr}^{-1}$, after correcting for infrared emission
from dust heated by the AGN (Drouart et al., 2014; Falkendal et al., 2019).
Imaging from the Hubble Space Telescope (HST) Near Infrared Camera and Multi-
Object Spectrometer (NICMOS) showed that the Dragonfly appears to be a merger
system (Pentericci et al., 2001). The radio source in the Dragonfly Galaxy is
$\sim$10 kpc in size, and only slightly resolved in existing radio images at
4.5 and 8.2 GHz (Pentericci et al., 2000).

The Dragonfly Galaxy also contains a large mass of cold molecular gas, $M_{\rm
H_{2}}\approx 5\times 10^{10}\ (\alpha_{\rm CO}/0.8)\ {\rm M}_{\odot}$ (Emonts
et al., 2011), with $\alpha_{\rm CO}$ =$M_{\rm H2}$/$L^{\prime}_{CO}$ = 0.8
the CO-to-H2 conversion factor found for ultra-luminous infrared galaxies
(ULIRGS; Downes & Solomon 1998. This is derived from a CO(1-0) luminosity that
is at the high end of what is found in high-$z$ radio galaxies (Emonts et al.,
2014). The CO(1-0) emission revealed a molecular gas reservoir that is spread
across $\sim$60 kpc, likely reflecting widespread tidal debris of cold gas
(Emonts et al. 2015a, hereafter Paper I). Observations of CO(6-5) with the
Atacama Large Millimeter/submillimeter Array (ALMA) in Cycle 2 showed that the
Dragonfly Galaxy is a system of three merging galaxies with one containing a
powerful radio source (Emonts et al. 2015b, hereafter Paper II). This work
revealed that a high mass of cold molecular gas is being displaced between two
of the merging galaxies at a rate between 1200 and 3000 M${}_{\odot}\ {\rm
yr}^{-1}$. This matches the star-formation rate estimated for the system.
However, these Cycle 2 data could not distinguish whether the gas kinematics
in the Dragonfly Galaxy were caused by gaseous outflows, or by gravitational
interactions between two rotating disk galaxies (Paper II).

In this paper, we present new, kpc-scale resolution data from the Atacama
Large Millimeter/submillimeter Array (ALMA) and the Karl G. Jansky Very Large
Array (VLA) that target line-emission from carbon monoxide (6-5), as well as
continuum emission from dust and synchrotron radiation (see also Lebowitz,
2019). We also present data from the Keck Cosmic Web Imager (KCWI) of the
hydrogen recombination-line Ly$\alpha$, as well as other rest-frame ultra-
violet emission lines that trace the ionized gas. This allows us to map in
detail the radio source and host galaxy environment, in order to study the
unusual properties of this high-$z$ radio galaxy. We will also investigate
what drives the rapid displacement of molecular gas in this system: tidal
forces from merging, or outflows driven by the radio jets and/or star
formation.

Throughout this paper, we assume the following cosmological parameters:
$H_{0}=71$ km s-1 Mpc-1, $\Omega_{\rm M}=0.3$, and $\Omega_{\rm\Lambda}=0.7$,
which at $z=1.9212$ (below) corresponds to an angular scale of 8.3 kpc
arcsec-1 and a luminosity distance $D_{\rm L}=14.6\ {\rm Gpc}$ (Wright, 2006).

##  2 Methods

###  2.1 ALMA

The ALMA Cycle 4 observations (project 2016.1.01417.S) were conducted on 9 and
17 August 2017 for 1.2 hours on-source with 45 antennas with baselines up to
$3.5\ {\rm km}$. We configured the spectral windows to cover two 4 GHz bands,
one including the redshifted CO(6-5) line (235.8 – 239.6 GHz) and the other
including only continuum (251.2 – 255.0 GHz).

The data were calibrated using the ALMA calibration pipeline that is included
in the Common Astronomical Software Applications (CASA) version 4.7.2 (CASA
Team et al., 2022), by means of running the script that was supplied with the
data by the North American ALMA Science Center (NAASC). Subsequent imaging was
done manually using CASA version 5.3.0. For the line data, we subtracted the
continuum in the $(u,v)$-domain by fitting a straight line to the line-free
channels. We then combined our Cycle 4 data with 6 min of ALMA Cycle 2
observations taken at the same frequency but in a more compact configuration
(baselines $\geq$17m; Paper II). Imaging and deconvolution were done using a
robust weighting scheme (Briggs, 1995), resulting in an image resolution of
$0.12\arcsec\times 0.08\arcsec$ (1.0 $\times$ 0.7 kpc) at a position angle
(PA) 80∘. The line data were imaged with a spectral resolution of 15 km s-1
for a single channel, resulting in a root-mean-square (rms) noise of 0.24 mJy
beam-1 chan-1. We also produced lower resolution line-data cubes by tapering
the data to a beam-size of $0.23\arcsec\times 0.19\arcsec$ (1.9 $\times$ 1.6
kpc) at PA $=81^{\circ}$, with channels of 45 kms and an rms noise level of
0.15 mJy beam-1 chan-1.

###  2.2 VLA

The VLA observations were conducted on 29 May 2015 in BnA-configuration, 5 and
13 August 2015 in A-configuration, and 30 December 2017 in B-configuration
(projects VLA/15A-316 and VLA/17B-444). The total on-source time was 3.3
hours. The observations were centred at 43 GHz and used an effective bandwidth
of 7.5 GHz. We used calibrator J2253+1608 for calibrating the bandpass
response, J0204-1701 at 4.4∘ distance from our target for calibrating the
complex gains every 60-70 seconds, and 3C147 for applying the absolute flux
scale.

A standard manual data reduction and analysis was performed using CASA version
5.1.1 for calibrating the A- and BnA-configuration data, and CASA version
5.3.0 for calibrating the B-configuration data and subsequent imaging of the
combined data. We used the multi-frequency synthesis method with a robust
weighting scheme to create a continuum image with a resolution of 0.08′′
$\times$ 0.05′′ (0.7 $\times$ 0.4 kpc) at PA $=-6.6^{\circ}$. The data were
cleaned, but the signal was not strong enough to perform a self-calibration.
This means that low-level artifacts due to phase errors persist in the image.
The rms noise level of the VLA continuum image is 24 $\mu$Jy beam-1.

###  2.3 Keck

The Keck observations (project U074) were conducted on 6 October 2018 using
the Keck Cosmic Web Imager at central wavelength 420 nm (Morrissey et al.,
2018). KCWI is an integral field unit (IFU) that effectively provides a
spectrum at every spatial pixel (spaxel). The KCWI large slicer and BL grating
was used to cover the effective wavelength range of 345 – 525 nm, or 118 – 180
nm in the rest frame, with a pixel size of $1.4\arcsec$ (12 kpc) in the E-W
direction and $0.29\arcsec$ (2.4 kpc) in the N-S direction. The spectral
resolution is R $\sim$ 900\. The total exposure time was 1 hour, divided into
six exposures of 10 minutes each. The data were reduced following standard
procedures using the KCWI data reduction pipeline (Neill et al., 2018).

##  3 Results

###  3.1 Molecular Gas and Dust

Figure 1: Left: HST/NICMOS F160W Image of Dragonfly Galaxy $(z=1.92$)
(Pentericci et al., 2001), Middle: 237 GHz (692 GHz rest frame) ALMA continuum
emission in the inner 13 kpc. Right: Total Intensity map of ALMA CO(6-5)
emission in the inner 7 kpc. The black contours are the 237 GHz ALMA continuum
from the middle plot. Contour levels start at 0.1 mJy beam-1 and increase in
steps of 0.1 mJy beam-1. The white contours are the 43 GHz (115 GHz rest
frame) VLA synchrotron emission of a radio hot spot (see Fig. 3 and Sect. 3.2
for more details on the radio source). Contour levels start at 0.4 mJy beam-1
and increase in steps of 0.4 mJy beam-1. Figure 2: Left: Moment 1 Velocity Map
of CO(6-5), Middle: Moment 2 Velocity Dispersion Map of CO(6-5). The contours
show the ALMA and VLA continuum data from Fig. 1. Right: Position Velocity Map
(smoothed to resolution of 0.3′′) taken along the position angle shown as
yellow rectangle in the inset on the bottom left.

Figure 1 (left) shows an HST image of the Dragonfly Galaxy (Pentericci et al.,
2001). On a scale of several tens of kpc, the HST image shows prominent tidal
features that were previously described in Paper I. Figure 1 (middle) shows
the ALMA 237 GHz continuum image, which reveals three components. Components
NW and SE appear in the center of the Dragonfly system and were previous
detected in Paper II. The continuum flux in component SE is roughly twice that
of component NE (Paper II), confirming that the SE galaxy hosts the bulk of
the dust-obscured star formation detected by Drouart et al. (2014) and
Falkendal et al. (2019). Component C is a likely companion galaxy that was
previously detected in CO(6-5) but not the 237 GHz continuum (Paper II). These
new observations reveal a peak flux density of 70 $\mu$Jy beam-1 for the
observed 237 GHz continuum of component C.

Figure 1 (right) shows a total intensity map of the CO (6-5) emission in the
central $\sim$8 kpc. The ALMA data clearly show that the Dragonfly system
consists of two central galaxies, NW and SE, which are separated by
approximately 4 kpc. The high-$J$ line of CO(6-5) traces molecular disks that
have a projected radius of approximately 2 kpc for NW and 1.5 kpc for SE.
Component C could be a third galaxy (see also Paper II).

Figure 2 (left) shows a moment-1 velocity map of CO(6-5). Both NW and SE
possess distinct regions of redshifted and blueshifted gas, indicative of
rotating disks. The disk of the NW galaxy shows less ordered gas kinematics
and a higher velocity dispersion compared to the disk of the SE galaxy. The
two disks are connected by a tidal bridge of gas. This bridge of CO emission
between the galaxies is also visible in the position-velocity plot with the
pseudo-slit taken along the radio axis (Fig. 2 \- right).

Figure 2 (middle) shows the moment-2 map of the CO velocity dispersion, which
is defined as $\sigma_{\rm CO}$ = $FWHM_{\rm CO}$/2.355, with $FWHM_{\rm CO}$
the full width at half the maximum intensity of the CO line. Figure 2 (middle)
shows that the molecular gas in the bridge has a velocity dispersion that
ranges from $\sigma_{\rm CO}$ = 90 $-$ 170 km s-1, where the maximum
$\sigma_{\rm CO}$ = 170 km s-1 is higher than the velocity dispersion seen
throughout the disk of the SE galaxy. Based on the 8.2 GHz radio continuum
image from Pentericci et al. (2000), in Paper II it was argued that that the
NW galaxy must host the radio-loud AGN.

###  3.2 Radio-loud AGN

Figure 3: VLA 43 GHz continuum image of the radio source, with overlaid
previous lower resolution VLA data of the radio source at 8 GHz (white
contours; Pentericci et al., 2000). The VLA 8 GHz contours start at 1 mJy
beam-1 and increase by a factor of 2. Self-calibration of the 8 GHz data could
have affected its positional accuracy, so we shifted the 8 GHz radio continuum
by 0.09 arcsec to create the best possible overlay with the 43 GHz continuum.
The red contours show the ALMA 237 GHz dust continuum from Fig. 1. The inset
on the bottom-right shows the bright 43 GHz hot-spot in the southern lobe,
detected on the right edge of the disk of the SE galaxy. The ALMA dust
continuum may be contaminated by faint 237 GHz synchrotron emission at the
location of the hot-spot. The inset on the top-right shows a weak and
unresolved 43 GHz counterpart to the northern lobe.

The radio source that classifies the Dragonfly Galaxy as a high-redshift radio
galaxy is shown in Figure 3. The contours of the radio jet detected by the VLA
at 8 GHz (Pentericci et al., 2000) are overlaid on our new high-resolution VLA
image at 43 GHz. The alignment of the central axis of the 8 GHz radio source
with the NW galaxy suggests that NW is the host galaxy of the radio source
(Paper II). The two-sided morphology of the radio source is consistent with
our new VLA observations. At 43 GHz, the high resolving power of the VLA left
too little surface brightness sensitivity to pick up any radio lobes
associated with a jet, but a bright spot was detected on the western edge of
the SE galaxy. This bright spot is aligned along the radio axis and has a flux
density of $S_{43}$ = 1.8 $\pm$0.2 mJy. The northern 8 GHz lobe shows a weak
and apparently unresolved 43 GHz counterpart at a 5.5$\sigma$ level, with a
peak flux density of $S_{43}$ = 0.13 $\pm$0.5 mJy beam-1. The projected
distance between the northern and southern radio component detected at 43 GHz
is $1.15\arcsec$, or 9.5 kpc. A central radio core is not detected in our VLA
data.

Previous low-resolution observations performed with the Australia Telescope
Compact Array at 39.9 GHz revealed a continuum flux density of $S_{39.9}$ =
5.1 $\pm$ 1.5 mJy beam-1 with a spectral index of $\alpha$ = -1.3, for
$S_{\nu}$ $\propto$ $\nu^{\alpha}$ (Emonts et al., 2011). These observations
had a spatial resolution of 9.8′′ $\times$7.1′′ (81 $\times$ 59 kpc), hence
did not resolve any of the structures associated with the Dragonfly system.
Based on these previous observations, we expect a total 43 GHz continuum flux
density of $S_{43}$ $\sim$ 4.6 mJy beam-1. This means that we only recover
about fourty percent of the 43 GHz continuum emission in our high-resolution
VLA data. The remaining sixty percent is likely synchrotron emission from more
diffuse lobes that are resolved out in our high-resolution VLA data.

###  3.3 Keck Spectroscopy

Figure 4: Keck spectroscopy. Top: 2D spectrum taken with a $6.8\arcsec$ wide
pseudo-slit along the North-South direction and covering both galaxies. The
pixel-scale on the vertical axis is $0.29\arcsec$ per pixel. The dotted lines
mark the approximate spatial extent of the Ly$\alpha$, C IV and He II lines.
Middle+bottom: 1D spectrum extracted against $9\times 2$ spaxels in the region
of the Dragonfly. Visible are the lines: Ly$\alpha$ ($\lambda_{\rm
rest}=1215.7$ Å), N V ($\lambda_{\rm rest}=1240$ Å), C II ($\lambda_{\rm
rest}=1336$ Å), Si IV ($\lambda_{\rm rest}=1400$ Å), N IV] ($\lambda_{\rm
rest}=1486$ Å), C IV ($\lambda_{\rm rest}=1548+1551$ Å), He II ($\lambda_{\rm
rest}=1640$ Å), O III] ($\lambda_{\rm rest}=1663$ Å), and N III]
($\lambda_{\rm rest}=1750$ Å). The middle panel also shows a zoom-in of the
Ly$\alpha$ (left) and C IV (right) lines, with arrows indicating potential
absorption features. The zero velocity for the close doublet of C IV is
centered on $\lambda_{\rm rest}=1548.2$ Å. Figure 5: Left: Ly$\alpha$ spectrum
of the Dragonfly Galaxy. A velocity correction was applied to account for the
Earth’s rotational and orbital motion, and the motion of the Galaxy with
respect to the Cosmic Microwave Background (CMB) frame. The 0-velocity (black
dashed line) corresponds to $z$ = 1.9212. The red dashed line highlights $v$ =
-750 km s-1. Annotated on the top are three velocity ranges, from -2000 to
-730 km s-1 (A), -730 to 0 km s-1 (B), and 0 - 1500 km s-1 (C). Right: Three
contour plots of the average surface brightness of the Ly$\alpha$ flux across
the velocity ranges A, B, and C. In each plot, contours start at
10${}^{-17}\,\rm erg\,s^{-1}\,cm^{-2}$Å-1 and increase by a factor of two. The
center at (0,0) is the intensity weighted centroid of the Ly-alpha line in the
Keck data. The PSF of the core of the Ly$\alpha$ line has $FWHM$ = 1.3 arcsec
in the North-South dimension and $FWHM$ = 2.2 arcsec in the East-West
dimension.

Figure 4 shows a 2D spectrum of the Keck data, obtained by putting a ‘pseudo-
slit’ with a width of 5 spaxels (6.75 arcsec) aligned in north-south direction
across the Dragonfly Galaxy. We produced this ‘pseudo-slit’ from the IFU data
through a linear arrangement of spaxels. The large spaxel size of the KCWI
data means that the NE and SW galaxies cannot be accurately distinguished in
the Keck imaging. We detect in total nine emission lines, including
Ly$\alpha$, N V, C II, Si IV, N IV], C IV, He II, O III], and N III]. Based on
the Keck spectrum, we can classify this AGN as Type II. Type II AGN are
characterized by a core that is blocked by the torus, producing only narrow
emission lines and lacking the very broad lines from the broad-line region
that are typically seen in Type I AGN.

Figure 4 shows that the brightest emission lines (Ly$\alpha$, He II, and C IV)
appear spatially extended. The Ly$\alpha$ emission stretches across a scale of
at least 100 kpc. This is analogue to spectroscopic results of giant
Ly$\alpha$ halos observed in other high-$z$ radio galaxies (e.g., van Ojik et
al., 1996; Pentericci et al., 1997; Villar-Martín et al., 2003, 2007; Vernet
et al., 2017) and quasars (e.g., Cantalupo et al., 2014; Borisova et al.,
2016). The Ly$\alpha$ line also shows two putative absorption features, one
around the redshift of CO, and the other blueshifted by roughly 600 km s-1.
These absorption systems are tentatively detected also in C IV. Similar
absorption features were seen in other high-$z$ radio galaxies (e.g., Jarvis
et al., 2003; Humphrey et al., 2008; Swinbank et al., 2015; Kolwa et al.,
2019). Emission of the He II and C IV metal lines is visible across
approximately 35 kpc (Fig. 4, top panel).

Figure 5 shows that the widespread Ly$\alpha$ emission is bright enough that
it can be imaged across $\sim$80 kpc. At the lowest velocities, the extended
Ly$\alpha$ emission is found predominantly North to North-East of the
galaxies, while at the highest velocities it is found stretching towards the
South and South-West. Ly$\alpha$ emission at larger scales or extended
emission seen in the other lines (as shown in the 2D spectrum of Fig. 4) has
too low surface brightness to be visible above the noise in similar contour
plots.

Previously, also CO(1-0) emission from cold molecular gas was observed across
$\sim 60$ kpc, predominantly at positive velocities (Paper I). Our new Keck
results add to the evidence that there is a rich circumgalactic medium present
around the Dragonfly system, although a full analysis of the circumgalactic
gaseous environment of the Dragonfly is beyond the scope of the current paper.

##  4 Discussion

###  4.1 Merging disk galaxies

The ALMA CO(6-5) moment maps show that both the SE and NW components are
likely rotating disk galaxies. The galaxies are connected by a tidal bridge of
molecular gas. The presence of this tidal gas connecting NW and SE, along with
their small separation ($\sim 4$ kpc), indicates that these galaxies are in an
early stage of merging (e.g., Mihos & Hernquist, 1996; Prochaska et al.,
2019). In agreement with this, studies at low redshift showed that radio-AGN
activity can be triggered during the pre-coalescent phase of a merger event
(Ramos Almeida et al., 2011; Tadhunter et al., 2011). The center of the bridge
of molecular gas between the two galaxies shows a high velocity dispersion, as
well as a sudden change in gas kinematics. This could indicate that in the
middle of the bridge the gas is strongly shocked, which could convert the bulk
velocity into a turbulence (e.g., Cornuault et al., 2018). Our results are in
good agreement with simulations by Sparre et al. (2021), which reveal that a
major merger between two disk galaxies can create a gas bridge that is
dominated by supersonic turbulence at the midpoint in between the galaxies.
This bridge can form either before or after the first passage of the galaxies.

The center of the NW galaxy contains the highest velocity dispersion in this
system and has more chaotic rotation than the SE galaxy. The maximum velocity
dispersion of $\sigma_{CO}$ $\sim$ 200 km s-1 in the central region of the NW
galaxy is significantly higher than the velocity dispersion typically observed
in both low- and high-$z$ systems (e.g., Swinbank et al., 2011; Girard et al.,
2021). Following Sect. 3.2, the NW galaxy hosts the radio-loud AGN. We
speculate that the AGN activity may be related to the high velocity dispersion
of the gas in the NW galaxy, either because the radio-loud AGN influences the
gas that is present in the NW galaxy (e.g., Morganti et al., 2021; Ramos
Almeida et al., 2022), or because large amounts of molecular gas are being
accreted onto the NW galaxy, which may fuel the AGN activity (e.g., Ruffa et
al., 2019). Alternatively, we cannot rule out that the disturbed rotational
kinematics of the NW galaxy are caused by another, much closer galaxy-galaxy
interaction associated with only the NW component, rather than a rotating
disk. Hibbard & van Gorkom (1996) show examples of nearby galaxy mergers where
gas kinematics leave blue- and redshifted signatures that could be difficult
to distinguish from rotating disks if these systems were to be located at
larger distances. However, we argue that this is unlikely, because the HST and
ALMA continuum imaging do not reveal indications that the NW galaxy consists
of multiple component.

###  4.2 Gas displacement: tidal debris or outflow?

Previous ALMA observations of the CO(6-5) emission at lower spatial resolution
revealed that molecular gas is rapidly being displaced within the Dragonfly
galaxy, with gas displacement rates between 1200 – 3000 ${\rm M}_{\odot}\,{\rm
yr}^{-1}$ (Paper II). Because this is similar to the star formation rate of
$SFR$ $\sim$ 2000 M⊙ yr-1, the mechanism that displaces the molecular gas is
likely critical for the evolution of this system. Paper II could not clearly
distinguish whether this mechanism was related to gravitational interaction
between two disks, AGN- or starburst-driven outflows, or both. The bridge of
CO(6-5) that we detect in our new ALMA data is a clear indication that
gravitational effects of the merger are important, but the bridge likely
represents only a fraction of the total gas displacement in this system. To
further investigate the gas displacement in the Dragonfly Galaxy, we tapered
our ALMA CO(6-5) data to a lower resolution of $0.23\arcsec\times 0.19\arcsec$
(1.9 $\times$ 1.6 kpc; see Sect. 2.1) in order to increase our sensitivity for
tracing molecular gas at lower surface brightness levels.

####  4.2.1 Tidal debris

In addition to the relatively narrow CO bridge between the two galaxies, our
tapered ALMA data show additional CO(6-5) emission at lower surface brightness
levels at velocities $|v|\lesssim 250\,{\rm km}\,{\rm s}^{-1}$ (Fig. 6). This
emission is also seen in the position-velocity plot of Fig. 2. Spectra of the
CO(6-5) data reveal no kinematic evidence for outflows in the region between
the galaxies, so we conclude that the molecular gas in between the galaxies is
mostly tidal debris from the gravitational interaction between the gaseous
disks of the two galaxies. Simulations by Sparre et al. (2021) show that, in
addition to the formation of a gaseous bridge, a major merger may lead to a
substantial mass accretion from entrainment and cooling of the circumgalactic
medium. This process may contribute to the tidal material the we find between
the galaxies in the Dragonfly system.

To estimate the CO(6-5) luminosity of this tidal debris, we extract a pseudo-
slit in perpendicular direction across the bridge, as shown in Fig. 7. The CO
emission detected across the tidal debris is contaminated by emission from the
bright CO in the NW and SE galaxy, as a result of the point-spread-function
(PSF) of the synthesized beam. To estimate this PSF contamination, we assume
that the molecular gas in the central disks of the two galaxies is distributed
roughly symmetrically around the center of the galaxies. As shown in Fig. 7,
we then take two pseudo-slits at ‘off’ positions on opposite side of the NW
and SE galaxy, at equal distance from the center of the galaxies as the
pseudo-slit of the ‘bridge’. To estimate the true CO(6-5) luminosity of the
tidal debris, we subtract the CO signal in regions ‘off-1’ and ‘off-2’ from
the CO signal of the ‘bridge’ region. The resulting CO(6-5) intensity of the
molecular gas in the tidal debris, corrected for PSF contamination of the
galaxies, is $I_{\rm CO(6-5)}$ = 0.3 $\pm$ 0.1 Jy beam-1 $\times$ km s-1.

####  4.2.2 AGN-driven outflow

The CO emission at the highest velocities (|$v$|$\gtrsim$250 km s-1) appears
to be localized around the NW galaxy. Figure 8 (middle) shows the CO(6-5)
spectra from the tapered data against the central region of NW, as well as a
region 0.15′′ (1.2 kpc) to the North along the axis of the radio source. These
spectra appear to have a broad, blueshifted wing to the CO profile. This blue
wing is indicative of an outflow from the NW galaxy. The maximum velocity of
the outflow reaches from $-500\rm\,km\,s^{-1}$ in the center to approximately
$-800\,{\rm km}\,{\rm s}^{-1}$ North of the center. No outflow is seen against
the SE galaxy, which hosts the bulk of the star formation.111Spilker et al.
(2020a, b) argue from OH 119$\mu$m absorption measurements that molecular
outflows driven by star formation are ubiquitous among high-$z$ dusty star-
forming galaxies, but that molecular emission lines may not always be reliable
tracers for these outflows. This, together with the apparent orientation of
the outflow along the jet axis, suggest that the outflow from the NW galaxy is
most likely driven by the radio source.

In the region North of the center of NW, where the outflow reaches its maximum
velocity, we also detect a faint blob in the 237 GHz radio continuum, with a
peak flux-density of roughly 0.09 mJy beam-1 (Fig. 8). Since we do not detect
anything in the 43 GHz continuum at this location, we can set a 3$\sigma$
lower limit to the spectral index of $\alpha$ > 0.12, for $S$ $\propto$
$\nu^{\alpha}$. This inverted spectral index makes it unlikely that this faint
ALMA continuum detection is radio synchrotron emission from a hot-spot in the
radio jet, but if real, its exact nature is not clear.

The Gaussian fitting to the CO profile in the region North of the center (Fig.
8) reveals that the maximum velocity offset between the peak of the broad and
narrow CO component is $\Delta v$ = 240 $\pm$ 60 km s-1, with $FWHM_{\rm CO}$
= 610 $\pm$ 90 km s-1 for the broad component (see Table 1). The total CO
intensity of the outflow is $I_{\rm CO}=0.37\pm 0.10\,{\rm Jy}\,{\rm
beam}^{-1}\times\,{\rm km}\,{\rm s}^{-1}$.

The optical emission lines in the Keck spectrum also reveal a broad,
blueshifted component (Fig. 9). The He II line is a non-resonant line, and
reveals a broad component that has a $FWHM$ $\sim$ 1810 $\pm$ 320 km s-1, with
a peak that is blueshifted by $\Delta v\,=\,-960\,\pm$ 220 km s-1 with respect
to the narrow He II component (Table 1). As shown in Fig. 9, Ly$\alpha$ and C
IV show similar kinematics. Although the spatial resolution of these Keck data
is not sufficient to determine the location of this broad component, it
nevertheless suggests that the outflow seen in CO has a component also in the
warm ionised gas. Moreover, the outflow velocity is more extreme in the
ionised gas (Fig. 9).

Figure 6: CO(6-5) emission at various velocities after tapering our ALMA data
to a lower resolution of 0.23 $\times$ 0.19′′ (1.9 $\times$ 1.6 kpc) at PA
81${}^{{}^{\circ}}$. White contours mark the level of the integrated emission
after binning the tapered data to 45 km s-1 channels. Contour levels start at
2.5$\sigma$ and increase by 1$\sigma$, with $\sigma$ = 0.15 mJy beam-1
(corresponding negative contours are in grey). The black contours show the CO
total intensity image at full spatial resolution from Fig. 2. Contour levels
start at 0.02 Jy beam-1 $\times$ km s-1 and increase by factor 2. The three
lines in the middle panel visualize the axes along which we obtained the
position-velocity (PV) plots to estimate the CO luminosity of the bridge
material. These PV plots are shown in Fig. 7. Figure 7: Position-Velocity
plots of the CO(6-5) emission in the tapered data taken across the bridge
between the two galaxies (middle), and two ‘off’ positions (left and right),
as indicated by the lines in the middle panel of Fig. 6. The ‘off’ slits are
at the same (but opposite) distance from the center of the SE (Off-1) and NW
(Off-2) galaxy as the slit across the bridge. These ‘off’ slits therefore give
an indication how much of the ‘bridge’ emission is contaminated by emission
from the bright central region of the galaxies due to the point-spread-
function (i.e., synthesized beam) of our observations. Contour levels start at
2.5$\sigma$ and increase with $1\sigma$, with $\sigma=0.15\,{\rm mJy}\,{\rm
beam}^{-1}$ (negative contours are in grey).

####  4.2.3 Mass displacement rates

To derive molecular gas masses for the material in the bridge and the outflow,
we first convert the derived CO(6-5) intensities from Sect. 4.2.1 and 4.2.2 to
the expected intensity of the CO ground transition. For this, we assume the
same CO(6-5)/CO(1-0) line ratios observed in Paper II, namely
$R_{I(6\rightarrow 1)}$ = $I_{\rm CO(6-5)}$/$I_{\rm CO(1-0)}$ $\sim$ 13 for
the molecular gas at |$v$|$\lesssim$250 km s-1 (i.e., the tidal debris), and
17 $\lesssim$ $r_{\rm 6-5}$ $\lesssim$ 36 for the molecular gas at
|$v$|$\gtrsim$250 km s-1 (i.e., the outflow). This allows us to estimate the
luminosity of the CO ground-state transition, $L_{\rm CO(1-0)}^{\prime}$,
following Solomon & Vanden Bout (2005, and references therein):

$L^{\prime}_{\rm CO{\bf(1-0)}}=3.25\times 10^{7}\left(\frac{I_{\rm
CO(6-5)}}{R_{I(6\rightarrow 1)}}\right)D_{\rm L}^{2}~{}\nu_{\rm
rest}^{-2}\left(1+z\right)^{-1},$ (1)

with $I_{\rm CO(6-5)}$ the CO(6-5) intensity in Jy beam-1 $\times$ km s-1,
$D_{\rm L}$ the luminosity distance in Mpc, $\nu_{\rm rest}$ the rest
frequency of the CO(1-0) line in GHz (115.27 GHz), and $L_{\rm
CO(1-0)}^{\prime}$ the CO(1-0) luminosity in K km s-1 pc2. To derive molecular
gas masses, we then use the same conservative CO-to-H2 conversion factor of
$\alpha_{\rm CO}$ = $M_{\rm H2}$/$L^{\prime}_{\rm CO(1-0)}$ = 0.8 M⊙ (K km s-1
pc2)-1 as was used in Paper II, which is the conversion factor found for
ULIRGs at low redshifts (Downes & Solomon, 1998). This results in a total
molecular gas mass of MH2 = (3 $\pm$ 1) $\times$ 109 M⊙ for the gas in the
tidal debris between the two galaxies, and (2 $\pm$ 1) $\times$ 109 M⊙ for the
gas in the outflow from the AGN host-galaxy. We note that these are likely
lower limits to the molecular gas masses, as values of $\alpha_{\rm CO}$
$\sim$ 3.6 M⊙ (K km s-1 pc2)-1 are typically found in high-$z$ galaxies (e.g.,
Daddi et al., 2010; Genzel et al., 2010). These mass estimates for the tidal
debris and outflow are in good agreement with molecular gas masses seen in the
tidal bridge in nearby merging galaxies (Lisenfeld et al., 2002; Braine et
al., 2003; da Silva et al., 2011), and outflows in low-$z$ ULIRGs and quasars
(Cicone et al., 2014). The combined molecular mass of the tidal debris and
outflow in the Dragonfly system represents roughly 20 percent of the total
molecular gas mass detected in the combined disks of NW and SE (Paper II).

To derive the re-distribution rate for the molecular gas in the tidal debris
that bridges the two galaxies, we assume that the molecular gas travels the 4
kpc distance between the galaxies at a typical velocity of $v$ $\sim$ 200 km
s-1. This results in a mass re-distribution rate of $\dot{M}_{\rm tidal}$ =
170 $\pm$ 40 M⊙ yr-1 for the gas in the tidal debris and bridge. For the mass
outflow rate of the NW outflow, we assume that the outflowing gas travels
across the distance spanned by half a tapered beam, or $\sim$1 kpc, with a
bulk outflow velocity of $\Delta v\,+\,\frac{1}{2}\,FWHM_{\rm CO}\,\sim\,545$
km s-1 (see Sect. 4.2.2). This results in a total outflow rate of molecular
gas from the NW galaxy of $\dot{M}_{\rm outfl}$ = 1100 $\pm$ 550 M⊙ yr-1.

####  4.2.4 Evolution of the Dragonfly system: effects of merger, outflow, and
star formation

After having derived mass displacement rates of the molecular gas in both the
tidal debris and the AGN-driven outflow, we can further investigate the
relative importance of the various physical phenomena in the Dragonfly system.
While the displacement of molecular gas in the Dragonfly occurs at a much
faster rate in the outflow compared to the tidal debris, the relative effect
that the outflow and gravitational forces have on the overall mass
displacement critically depends on the lifetime of the respective events. Jet
activity in radio galaxies may last for 107-8 years (Blundell & Rawlings,
2000), but is likely orders of magnitude less for (episodic) activity in young
radio sources (e.g., Nyland et al., 2020). On the other hand, the orbital time
of the system is at least 120 Myr, when considering our observed current
distance of 4 kpc between the galaxies and an orbital velocity similar to the
$\sim$200 km s-1 rotational speed of the disks. While it is beyond the scope
of this paper to determine accurate life-times of the radio source and merger
event, the fact that at least as much mass of molecular gas is locked up in
the slower moving tidal debris compared to the fast outflow, makes both AGN-
driven and gravitational effects important for the evolution of the Dragonfly
system (see also Spilker et al., 2022).

The tidal debris and outflow may not have the same effect on the overall gas
reservoir, at least in the short term. Whereas the tidal debris may
redistribute the molecular gas between the two galaxies, and thus potentially
supply molecular gas to the NW galaxy that hosts the AGN (Sect. 4.1), the
outflow removes molecular gas from the central region of the NW galaxy.
However, in Paper II an estimate was made that the escape velocity at a
distance of 4 kpc (i.e., the distance between the centers of the SE and NW
galaxy) has a lower limit of vesp $\sim$ 1000 km s-1, as derived from only the
stellar mass of the Dragonfly system ($M_{*}$ $\sim$ 5.8 $\times$ 1011 M⊙; De
Breuck et al. 2010). Therefore, it is likely that the molecular gas in both
the outflow and the tidal debris will remain gravitationally bound to the
Dragonfly system and will at some point be re-accreted.

In addition to molecular gas in the outflow and tidal debris, the high star
formation rate of $SFR$ $\sim$ 2000 M⊙ yr-1 across the system means that
molecular gas is being consumed by star formation at a rate that surpasses the
mass outflow rate and tidal gas displacement rate. This means that star
formation itself is as effective as the merger and outflow in removing the
molecular gas (see also Man et al., 2019).

We therefore conclude that the gravitational effects of the merger, the AGN-
driven outflows, and the vigorous star formation are all critical to the
evolution of the Dragonfly system.

###  4.3 Jet-disk interaction

The 43 GHz observations conducted by the VLA detected a radio hot spot on the
western edge of the SE galaxy. This hot spot appears to be aligned along the
central axis of the radio jet (Fig. 3). Based on this alignment, and our
conclusion that NW hosts the radio-loud AGN, we theorize that the interaction
of the southern jet with the gas and dust in the disk of the SE galaxy is
causing the radio emission to brighten up at the location of the 43 GHz
detection. This may happen because confinement of the radio lobe and
compression of the magnetic field results in an increased particle density and
enhanced synchrotron radiation (e.g., Gopal-Krishna & Wiita, 1991; Barthel &
Arnaud, 1996; Morganti et al., 2011)

The gas kinematics in the Dragonfly may further support this scenario. First,
Fig. 2 shows that the velocity dispersion of the gas in the SE galaxy is
highest around the radio hot-spot ($\sigma_{\rm CO}$ $\sim$ 140 km s-1) and
not in the center, as is the case for the NW galaxy. Such an enhanced velocity
dispersion is expected in a region where energy and momentum are being
exchanged between the radio source and the gas (Mandal et al., 2021; Meenakshi
et al., 2022).

Second, the detection of a potential blueshifted outflow (Sect. 4.2.2) may
further support the scenario of jet-brightening through the assumed geometry
of the system. If the outflow is oriented towards us along the northern jet,
then the southern jet is receding. At high redshifts, the brightening of the
receding jet cannot be explained by Doppler boosting, which causes intrinsic
brightening of the jet only for relativistic particles moving towards us.
Moreover, we only detect the radio hot-spot, which represent the working
surface of the radio jet, and which is expected to advance at sub-relativistic
speeds. Some Doppler boosting likely still occurs within the hot-spots, but
this typically creates flux-density ratios of order a few between the hot-
spots of a two-sided jet (Komissarov & Falle, 1996), while the Dragonfly shows
a much higher flux-density ratio between the SE and NW hot-spot of $S_{\rm
SW}$/$S_{\rm NE}$ $\sim$ 14 at 43 GHz. This suggests that the southern jet
likely brightens because of its interaction with the SE galaxy, which enhances
the radio flux of the system. The argument that the SE jet is likely receding
would imply that the SE galaxy must be oriented behind the NW galaxy.

In a forthcoming paper, we will further explore this scenario that the radio
source in the Dragonfly brightens as a result of interaction with cold
molecular gas, based on a comparison with other high-$z$ radio galaxies.
Similar to the Dragonfly, a high flux-density ratio between hot-spots is seen
in several other high-$z$ radio galaxies that show alignments between the
radio source and CO-emitting gas reservoirs in the environment. However,
exploration of this is beyond the scope of this paper and will be presented in
Emonts et al (in prep).

If it is indeed the case that the radio flux is boosted as a result of a jet-
disk interaction, then the Dragonfly Galaxy may not be as intrinsically radio
bright as previously thought, making it perhaps an ‘imposter’ radio galaxy in
the high-z Universe, as we will explain in the next Section.

Figure 8: CO(6-5) ALMA image and spectrum with data tapered to a lower
resolution, as in Fig. 6. Left: Total intensity image of the CO(6-5)
integrated across the velocity range from -750 to +400 km s-1. White contours
show the total intensity map starting at 4$\sigma$ and increasing by a factor
$\sqrt{2}$, with $\sigma$ = 0.05 Jy bm-1 $\times$ km s-1. Black contours are
the ALMA 237 GHz continuum at full (untapered) resolution from Fig. 1,
starting at 5$\sigma$ and increasing by a factor $\sqrt{2}$, with $\sigma$ =
15 $\mu$Jy bm-1. Red contours are the VLA 43 GHz continuum from Fig. 3 at
levels 0.3, 0.8, 1.3 mJy bm-1. The red dashed line visualizes the axis of the
radio source. The synthesized beam of the tapered data is 0.23′′ $\times$
0.19′′ (1.9 $\times$ 1.6 kpc) at PA 81∘, as indicated with the ellipse in the
bottom-left corner of the image. Middle: spectra of the NW galaxy (bottom) and
a region north of the NW galaxy that corresponds to a faint blob of ALMA
continuum emission (top). For clarity, a Hanning smoothing has been applied to
the spectra. The spectrum of the NW galaxy nucleus is complex (see Paper II),
while the spectrum north of the galaxy can be fitted with two Gaussian
functions (red lines). The two spectra are not entirely mutually independent,
but suggest the presence of an outflow component that peaks North of the
nucleus along the jet-axis. Figure 9: Gas outflow in the optical emission
lines. Top: Keck spectrum of He II fitted with two Gaussian components (red
dashed lines). The spectrum is from Fig. 4 and does not spatially resolve the
SE and NW galaxies. Bottom: Superposition of the profiles from He II (black
solid), Ly$\alpha$ (blue dashed), C IV (magenta dash-dotted), and CO(6-5)
(grey thin). The optical profiles show the total line flux integrated across
NW and SE (Fig. 4), while the CO profile shows only the emission against the
Northern region from Fig. 8. Fluxes have been normalized for easy comparison.
Table 1: Outflow kinematics from spectral-line fitting.

|  | CO (North) | He II
---|---|---|---
Narrow | $v$ | 60 $\pm$ 10 | 180 $\pm$ 10
| $FWHM$ | 270 $\pm$ 30 | 990† $\pm$ 30
Broad | $v$ | -180 $\pm$ 60 | -780 $\pm$ 220
| $FWHM$ | 610 $\pm$ 90 | 1810† $\pm$ 320

Note. — All units are in km s-1. Errors reflect uncertainties in the fitting.

† Corrected for instrumental broadening.

###  4.4 Archetypal radio source, or radio imposter?

The major merger between two gas-rich disk galaxies likely drives the star-
formation activity in the Dragonfly, and gives the system its hyper-luminous
IR appearance. The fact that the starburst IR luminosity is an order of
magnitude higher than that of other powerful radio galaxies at $z$ $\sim$ 2
could mean that either the pre-coalenscent merger is a short and active early
phase in the evolution of high-$z$ radio galaxies, or the Dragonfly is not a
typical high-z radio galaxy.

The observed radio source in the Dragonfly is compact, with a total extent of
only 9.5 kpc. This could support the idea that the Dragonfly is in an early
phase of radio-AGN activity. Our finding that the flux of the radio source is
likely enhanced as a result of an interaction between the jet and the disk of
the southern galaxy could also indicate that the Dragonfly is an ‘imposter’
radio galaxy, with an intrinsic radio flux that is much lower than what is
typical for high-z radio galaxies. At low redshifts, examples exist of radio
galaxies for which the jet brightens as it encounters a companion galaxy
(e.g., Lacy et al., 1998; Evans et al., 2008; Hota et al., 2022). The idea
that the brightness of the synchrotron emission is enhanced when a young jet
runs into dense clouds of gas has also been suggested as explanation for the
relatively large fraction of young and compact radio sources seen among
starbursting radio galaxies (Tadhunter et al., 2011). These starburst radio
galaxies at low-$z$ likely have a rich interstellar medium as a result of a
major merger. If a similar jet-cloud interaction is causing the radio source
in the merger system of the Dragonfly to brighten, this could push the
Dragonfly into the flux-limited regime by which powerful high-z radio galaxies
are selected.

This could also be an analogue to the nearby Seyfert galaxy PKS 1814-637,
which has been classified as an ‘imposter’ radio galaxy because the
intrinsically weak radio jet interacts with its gaseous environment (Morganti
et al., 2011). Interestingly, PKS 1814-637 is one of very few low-z radio
galaxies that is classified as a genuine disk galaxy. In this respect, the
fact that the radio source in the Dragonfly is hosted by a galaxy with a
rotating gas disk is intriguing in comparison to this low-$z$ work. As a note
of caution, gaseous disks have also been observed in low-$z$ early-type
galaxies (e.g., Struve et al., 2010; Sansom et al., 2019), so further high-
resolution optical or infra-red observations of the galaxies in the Dragonfly
system are necessary to determine whether the AGN host-galaxy is a genuine
disk galaxy. In this respect, Drouart et al. (2016) showed that other high-$z$
radio galaxies show star-formation histories that resemble those of spiral
galaxies. Therefore, a one-on-one comparison between powerful radio galaxies
in the low- and high-redshift Universe requires further investigation.

It is possible that enhancement of the radio flux, through interactions
between the radio jets and a rich molecular environment, is a common way of
“creating” a high-$z$ radio galaxy with bright synchrotron emission. This
could mean that, in general, we may not observe a high-$z$ radio galaxy unless
its jets manage to interact with a surrounding dense medium. In the case of
the Dragonfly system, the dense medium is the gas in the disk of a separate
galaxy, but it may just as well involve gas in the interstellar or
circumgalactic medium of the AGN host galaxy itself (e.g., Steinbring, 2014,
Emonts et al. in prep.). This scenario would still make the Dragonfly
interesting in terms of its young and active starburst environment, but rather
illustrative in terms of the jet-cloud mechanism that shapes bright radio
sources. High-resolution observations of molecular gas and radio jets in other
high-$z$ radio galaxies are essential to study the prevalence of flux-
brightening of radio sources in molecular-rich environments, and whether this
can tell us something profound about the evolutionary importance of the radio-
loud phase and the general nature of powerful high-z radio galaxies.

##  5 Conclusions

We presented new ALMA and VLA data of CO(6-5), dust and synchrotron emission
in the enigmatic Dragonfly Galaxy with a spatial resolution of $\sim$1 kpc. We
complemented this with new optical IFU spectroscopy from the Keck Cosmic Web
Imager. Our main conclusions are:

  * •

The Dragonfly system is an ongoing major merger between two gas-rich galaxies
with rotating disks, which likely triggered the high star-formation rates.
Tidal debris from the merger is observed even in the high-$J$ line of CO(6-5).

  * •

The high velocity dispersion of CO(6-5) in the central region of NW, combined
with the detection of hot-spots from the jet and counter-jet in our VLA 43 GHz
data, is consistent with the NW galaxy hosting the radio-loud AGN.

  * •

An interaction between the radio jet and the disk of the SE galaxy likely
causes the jet to brighten at the hot-spot.

  * •

Molecular gas is being displaced within the Dragonfly system by both a fast
moving outflow ($\dot{M}_{\rm outfl}$ = 1100 $\pm$ 550 M⊙ yr-1) and slower
moving tidal debris ($\dot{M}_{\rm tidal}$ = 170 $\pm$ 40 M⊙ yr-1). In terms
of total mass, both the outflow and the tidal debris contain similar amounts
of molecular gas.

  * •

The gravitational effects of the merger, the AGN-driven outflow, and vigorous
star formation are all important components in the evolution of the Dragonfly
Galaxy.

  * •

The Keck data, which include rest-frame UV lines, identify the Dragonfly as a
Type II AGN.

  * •

Ly$\alpha$ emission of warm ionized gas is visible across 100 kpc, with the
metal-lines C IV and He II detected across approximately 35 kpc, confirming
the presence of a rich and extended CGM previously detected in CO(1-0) in
Paper I.

Our results suggest that the Dragonfly Galaxy may be an imposter radio galaxy,
which we only observe as a powerful radio galaxy as a result of an interaction
between the radio jet and the disk of the secondary galaxy. This chance
interaction provides a unique snapshot of the evolutionary history of galaxies
in which a major-merger event coincides with radio activity. Future work on
the Dragonfly Galaxy will further investigate the nature of radio activity and
feedback in this system.

More research is needed to identify how common these systems may be in the
high-z Universe. Future high-resolution studies of merger systems like the
Dragonfly can help to reveal the prevalence of flux-boosted radio sources in
molecular-rich environments, which may have significant consequences on our
interpretation of the evolutionary importance of the radio-loud phase and our
understanding of the nature of powerful high-$z$ radio galaxies.

Part of this work was completed as part of the National Astronomy Consortium
(NAC) program at NRAO. The National Radio Astronomy Observatory is a facility
of the National Science Foundation operated under cooperative agreement by
Associated Universities, Inc. This paper makes use of the following ALMA data:
ADS/JAO.ALMA$\\#$2016.1.01417.S, 2013.1.00521.S. ALMA is a partnership of ESO
(representing its member states), NSF (USA) and NINS (Japan), together with
NRC (Canada), MOST and ASIAA (Taiwan), and KASI (Republic of Korea), in
cooperation with the Republic of Chile. The Joint ALMA Observatory is operated
by ESO, AUI/NRAO and NAOJ. Some of the data presented herein were obtained at
the W. M. Keck Observatory, which is operated as a scientific partnership
among the California Institute of Technology, the University of California and
the National Aeronautics and Space Administration. The Observatory was made
possible by the generous financial support of the W. M. Keck Foundation. The
authors wish to recognize and acknowledge the very significant cultural role
and reverence that the summit of Maunakea has always had within the indigenous
Hawaiian community. We are most fortunate to have the opportunity to conduct
observations from this mountain. Based on observations with the NASA/ESA
Hubble Space Telescope obtained from the Data Archive at the Space Telescope
Science Institute, which is operated by the Association of Universities for
Research in Astronomy, Incorporated, under NASA contract NAS5-26555. Support
for program numbers HST-AR-16123.001-A and HST-GO-16891.002-A was provided
through a grant from the STScI under National Aeronautics and Space
Administration (NASA) contract NAS5-26555. DT acknowledges partial support
from the National Science Foundation via award AST-2007023 to The Ohio State
University. MVM acknowledges support from grant PID2021-124665NB-I00 by the
Spanish Ministry of Science and Innovation (MCIN) / State Agency of Research
(AEI) / 10.13039/501100011033 and by the European Regional Development Fund
(ERDF) "A way of making Europe".

## References

  * Barthel et al. (2012) Barthel, P., Haas, M., Leipski, C., & Wilkes, B. 2012, ApJ, 757, L26
  * Barthel & Arnaud (1996) Barthel, P. D., & Arnaud, K. A. 1996, MNRAS, 283, L45
  * Blundell & Rawlings (2000) Blundell, K. M., & Rawlings, S. 2000, AJ, 119, 1111
  * Borisova et al. (2016) Borisova, E., Cantalupo, S., Lilly, S. J., et al. 2016, ApJ, 831, 39
  * Braine et al. (2003) Braine, J., Davoust, E., Zhu, M., et al. 2003, A&A, 408, L13
  * Briggs (1995) Briggs, D. S. 1995, PhD thesis, New Mexico Institute of Mining and Technology
  * Cantalupo et al. (2014) Cantalupo, S., Arrigoni-Battaia, F., Prochaska, J. X., Hennawi, J. F., & Madau, P. 2014, Nature, 506, 63
  * Carilli et al. (1997) Carilli, C. L., Röttgering, H. J. A., van Ojik, R., Miley, G. K., & van Breugel, W. J. M. 1997, ApJS, 109, 1
  * CASA Team et al. (2022) CASA Team, Bean, B., Bhatnagar, S., et al. 2022, PASP, 134, 114501
  * Chambers et al. (1996) Chambers, K. C., Miley, G. K., van Breugel, W. J. M., & Huang, J.-S. 1996, ApJS, 106, 215
  * Cicone et al. (2014) Cicone, C., Maiolino, R., Sturm, E., et al. 2014, A&A, 562, A21
  * Cornuault et al. (2018) Cornuault, N., Lehnert, M. D., Boulanger, F., & Guillard, P. 2018, A&A, 610, A75
  * da Silva et al. (2011) da Silva, R. L., Prochaska, J. X., Rosario, D., Tumlinson, J., & Tripp, T. M. 2011, ApJ, 735, 54
  * Daddi et al. (2010) Daddi, E., Elbaz, D., Walter, F., et al. 2010, ApJ, 714, L118
  * Dannerbauer et al. (2014) Dannerbauer, H., Kurk, J. D., De Breuck, C., et al. 2014, A&A, 570, A55
  * De Breuck et al. (2010) De Breuck, C., Seymour, N., Stern, D., et al. 2010, ApJ, 725, 36
  * Downes & Solomon (1998) Downes, D., & Solomon, P. M. 1998, ApJ, 507, 615
  * Drouart & Falkendal (2018) Drouart, G., & Falkendal, T. 2018, MNRAS, 477, 4981
  * Drouart et al. (2016) Drouart, G., Rocca-Volmerange, B., De Breuck, C., et al. 2016, A&A, 593, A109
  * Drouart et al. (2014) Drouart, G., De Breuck, C., Vernet, J., et al. 2014, A&A, 566, A53
  * Emonts et al. (2011) Emonts, B. H. C., Feain, I., Mao, M. Y., et al. 2011, ApJ, 734, L25
  * Emonts et al. (2014) Emonts, B. H. C., Norris, R. P., Feain, I., et al. 2014, MNRAS, 438, 2898
  * Emonts et al. (2015a) Emonts, B. H. C., Mao, M. Y., Stroe, A., et al. 2015a, MNRAS, 451, 1025 (Paper I)
  * Emonts et al. (2015b) Emonts, B. H. C., De Breuck, C., Lehnert, M. D., et al. 2015b, A&A, 584, A99 (Paper II)
  * Evans et al. (2008) Evans, D. A., Fong, W.-F., Hardcastle, M. J., et al. 2008, ApJ, 675, 1057
  * Falkendal et al. (2019) Falkendal, T., De Breuck, C., Lehnert, M. D., et al. 2019, A&A, 621, A27
  * Galametz et al. (2012) Galametz, A., Stern, D., De Breuck, C., et al. 2012, ApJ, 749, 169
  * Genzel et al. (2010) Genzel, R., Tacconi, L. J., Gracia-Carpio, J., et al. 2010, MNRAS, 407, 2091
  * Girard et al. (2021) Girard, M., Fisher, D. B., Bolatto, A. D., et al. 2021, ApJ, 909, 12
  * Gopal-Krishna & Wiita (1991) Gopal-Krishna, & Wiita, P. J. 1991, ApJ, 373, 325
  * Hatch et al. (2009) Hatch, N. A., Overzier, R. A., Kurk, J. D., et al. 2009, MNRAS, 395, 114
  * Hibbard & van Gorkom (1996) Hibbard, J. E., & van Gorkom, J. H. 1996, AJ, 111, 655
  * Hota et al. (2022) Hota, A., Dabhade, P., Vaddi, S., et al. 2022, MNRAS, 517, L86
  * Humphrey et al. (2008) Humphrey, A., Villar-Martín, M., Sánchez, S. F., et al. 2008, MNRAS, 390, 1505
  * Jarvis et al. (2003) Jarvis, M. J., Wilman, R. J., Röttgering, H. J. A., & Binette, L. 2003, MNRAS, 338, 263
  * Kolwa et al. (2019) Kolwa, S., Vernet, J., De Breuck, C., et al. 2019, A&A, 625, A102
  * Komissarov & Falle (1996) Komissarov, S. S., & Falle, S. A. E. G. 1996, in Astronomical Society of the Pacific Conference Series, Vol. 100, Energy Transport in Radio Galaxies and Quasars, ed. P. E. Hardee, A. H. Bridle, & J. A. Zensus, 327
  * Lacy et al. (1998) Lacy, M., Rawlings, S., Blundell, K. M., & Ridgway, S. E. 1998, MNRAS, 298, 966
  * Lebowitz (2019) Lebowitz, S. 2019, in American Astronomical Society Meeting Abstracts, Vol. 233, American Astronomical Society Meeting Abstracts #233, 145.04
  * Lisenfeld et al. (2002) Lisenfeld, U., Braine, J., Duc, P. A., et al. 2002, A&A, 394, 823
  * Man et al. (2019) Man, A. W. S., Lehnert, M. D., Vernet, J. D. R., De Breuck, C., & Falkendal, T. 2019, A&A, 624, A81
  * Mandal et al. (2021) Mandal, A., Mukherjee, D., Federrath, C., et al. 2021, MNRAS, 508, 4738
  * McCarthy (1993) McCarthy, P. J. 1993, ARA&A, 31, 639
  * Meenakshi et al. (2022) Meenakshi, M., Mukherjee, D., Wagner, A. Y., et al. 2022, MNRAS, 516, 766
  * Mihos & Hernquist (1996) Mihos, J. C., & Hernquist, L. 1996, ApJ, 464, 641
  * Miley & De Breuck (2008) Miley, G., & De Breuck, C. 2008, A$\%$ARv, 15, 67
  * Morganti et al. (2011) Morganti, R., Holt, J., Tadhunter, C., et al. 2011, A&A, 535, A97
  * Morganti et al. (2021) Morganti, R., Oosterloo, T., Tadhunter, C., Bernhard, E. P., & Oonk, J. B. R. 2021, arXiv e-prints, arXiv:2109.13516
  * Morrissey et al. (2018) Morrissey, P., Matuszewski, M., Martin, D. C., et al. 2018, ApJ, 864, 93
  * Neill et al. (2018) Neill, D., Development, K., & mattphys. 2018, Keck-DataReductionPipelines/KcwiDRP: KCWI Data Reduction Pipeline: First minor release, v.v1.1.0, Zenodo, doi:10.5281/zenodo.1287322
  * Nesvadba et al. (2017) Nesvadba, N. P. H., De Breuck, C., Lehnert, M. D., Best, P. N., & Collet, C. 2017, A&A, 599, A123
  * Nyland et al. (2020) Nyland, K., Dong, D. Z., Patil, P., et al. 2020, ApJ, 905, 74
  * Pentericci et al. (2001) Pentericci, L., McCarthy, P. J., Röttgering, H. J. A., et al. 2001, ApJS, 135, 63
  * Pentericci et al. (1997) Pentericci, L., Roettgering, H. J. A., Miley, G. K., Carilli, C. L., & McCarthy, P. 1997, A&A, 326, 580
  * Pentericci et al. (2000) Pentericci, L., Van Reeven, W., Carilli, C. L., Röttgering, H. J. A., & Miley, G. K. 2000, A&AS, 145, 121
  * Prochaska et al. (2019) Prochaska, J. X., Neeleman, M., Kanekar, N., & Rafelski, M. 2019, ApJ, 886, L35
  * Ramos Almeida et al. (2011) Ramos Almeida, C., Tadhunter, C. N., Inskip, K. J., et al. 2011, MNRAS, 410, 1550
  * Ramos Almeida et al. (2022) Ramos Almeida, C., Bischetti, M., García-Burillo, S., et al. 2022, A&A, 658, A155
  * Rocca-Volmerange et al. (2013) Rocca-Volmerange, B., Drouart, G., De Breuck, C., et al. 2013, MNRAS, 429, 2780
  * Roettgering et al. (1994) Roettgering, H. J. A., Lacy, M., Miley, G. K., Chambers, K. C., & Saunders, R. 1994, A&AS, 108, 79
  * Ruffa et al. (2019) Ruffa, I., Davis, T. A., Prandoni, I., et al. 2019, MNRAS, 489, 3739
  * Sansom et al. (2019) Sansom, A. E., Glass, D. H. W., Bendo, G. J., et al. 2019, MNRAS, 482, 4617
  * Seymour et al. (2007) Seymour, N., Stern, D., De Breuck, C., et al. 2007, ApJS, 171, 353
  * Solomon & Vanden Bout (2005) Solomon, P. M., & Vanden Bout, P. A. 2005, ARA&A, 43, 677
  * Sparre et al. (2021) Sparre, M., Whittingham, J., Damle, M., et al. 2021, MNRAS, arXiv:2110.03702
  * Spilker et al. (2020a) Spilker, J. S., Phadke, K. A., Aravena, M., et al. 2020a, ApJ, 905, 85
  * Spilker et al. (2020b) Spilker, J. S., Aravena, M., Phadke, K. A., et al. 2020b, ApJ, 905, 86
  * Spilker et al. (2022) Spilker, J. S., Suess, K. A., Setton, D. J., et al. 2022, ApJ, 936, L11
  * Steinbring (2014) Steinbring, E. 2014, AJ, 148, 10
  * Struve et al. (2010) Struve, C., Oosterloo, T. A., Morganti, R., & Saripalli, L. 2010, A&A, 515, A67
  * Swinbank et al. (2011) Swinbank, A. M., Papadopoulos, P. P., Cox, P., et al. 2011, ApJ, 742, 11
  * Swinbank et al. (2015) Swinbank, A. M., Vernet, J. D. R., Smail, I., et al. 2015, MNRAS, 449, 1298
  * Tadhunter et al. (2011) Tadhunter, C., Holt, J., González Delgado, R., et al. 2011, MNRAS, 412, 960
  * van Ojik et al. (1996) van Ojik, R., Roettgering, H. J. A., Carilli, C. L., et al. 1996, A&A, 313, 25
  * Venemans et al. (2007) Venemans, B. P., Röttgering, H. J. A., Miley, G. K., et al. 2007, A$\&$A, 461, 823
  * Vernet et al. (2017) Vernet, J., Lehnert, M. D., De Breuck, C., et al. 2017, A&A, 602, L6
  * Villar-Martín et al. (1999) Villar-Martín, M., Binette, L., & Fosbury, R. A. E. 1999, A&A, 346, 7
  * Villar-Martín et al. (2007) Villar-Martín, M., Sánchez, S. F., Humphrey, A., et al. 2007, MNRAS, 378, 416
  * Villar-Martín et al. (2003) Villar-Martín, M., Vernet, J., di Serego Alighieri, S., et al. 2003, MNRAS, 346, 273
  * Wilkes et al. (2013) Wilkes, B. J., Kuraszkiewicz, J., Haas, M., et al. 2013, ApJ, 773, 15
  * Wright (2006) Wright, E. L. 2006, PASP, 118, 1711
  * Wylezalek et al. (2013) Wylezalek, D., Galametz, A., Stern, D., et al. 2013, ApJ, 769, 79


# Effectiveness and predictability of in-network storage cache for Scientific
Workflows

Caitlin Sim1, Kesheng Wu2, Alex Sim2, Inder Monga3, Chin Guok3
Frank Würthwein4, Diego Davila4, Harvey Newman5, Justas Balcas5
1 University of California at Berkeley<EMAIL_ADDRESS>
2 Lawrence Berkeley National Laboratory<EMAIL_ADDRESS>
3 Energy Sciences Network<EMAIL_ADDRESS>
4 University of California at San Diego<EMAIL_ADDRESS>
5 California Institute of Technology<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Large scientific collaborations often have multiple scientists accessing the
same set of files while doing different analyses, which create repeated
accesses to the large amounts of shared data located far away. These data
accesses have long latency due to distance and occupy the limited bandwidth
available over the wide-area network. To reduce the wide-area network traffic
and the data access latency, regional data storage caches have been installed
as a new networking service. To study the effectiveness of such a cache system
in scientific applications, we examine the Southern California Petabyte Scale
Cache for a high-energy physics experiment. By examining about 3TB of
operational logs, we show that this cache removed 67.6% of file requests from
the wide-area network and reduced the traffic volume on wide-area network by
12.3TB (or 35.4%) an average day. The reduction in the traffic volume (35.4%)
is less than the reduction in file counts (67.6%) because the larger files are
less likely to be reused. Due to this difference in data access patterns, the
cache system has implemented a policy to avoid evicting smaller files when
processing larger files. We also build a machine learning model to study the
predictability of the cache behavior. Tests show that this model is able to
accurately predict the cache accesses, cache misses, and network throughput,
making the model useful for future studies on resource provisioning and
planning.

###### Index Terms:

in-network caching, data throughput, transfer performance, data access trends

##  I Introduction

Large scientific projects often involve thousands of scientists sharing a
massive data collection [1]. These projects, such as the Large Hadron Collider
(LHC), have collaborators around the world, each with their own analysis
tasks, accessing a different portion of the data collection, transferring data
files over long distances, and causing high demand over the limited wide-area
network bandwidth. While efficient tools for data movement over wide-area
network are available, there is a new networking service, in-network storage
caches, that could reduce a significant portion of the traffic on wide-area
network. These caches take advantage of the geographical sharing of data
accesses as there are overlapping in data accesses among the colleagues in the
same institution who often work on related scientific objectives. In
particular, the High Energy Physics (HEP) community has been exploring such a
caching system under the term of regional “data lakes” [2] as a part of their
federated data storage infrastructure [3]. There are evidences that a regional
data cache could improve data accesses [4, 5, 6]. However, real-world
deployment sometimes bring up unexpected challenges. This work studies the
effectiveness of one such in-network deployment to see how to address the
challenges encountered.

This work studies the operational logs of a large-scale deployment of storage
cache nodes. These logs are from the Southern California Petabyte Scale Cache
(SoCal Repo) [7] developed for High-Energy Physics (HEP) data analysis jobs,
where the wide-area network traffic is primarily carried by the Energy
Sciences Network (ESnet) [8]. There have been some reports about the
performance characteristics including number of file requests, cache misses,
and data volumes [9, 10]. The first objective of this work is to understand
the networking characteristics such as network traffic reduction, data
throughput performance, and so on. We expect this part of the study to confirm
that SoCal Repo significantly reduce the traffic over the wide-area network.
Nevertheless, there are surprises due to a special user access pattern.

The second objective of our work is to understand the predictability of the
network utilization to help plan for additional deployments of in-network
caches in the science network infrastructure. For this purpose, we developed a
machine learning model to predict the network utilization. Despite the high
variability in the cache usage, as shown in Figure 1, it is still possible to
model the cache requests with accuracy. Our model takes the SoCal Repo
performance characteristics as the input time series and learns the
performance patterns through a recurrent neural network architecture known as
the Long Short-Term Memory (LSTM) [11, 12]. The errors of the predictions are
significantly less than the standard deviation of the original values. One way
to use this prediction model is to plan for days with unusually high network
demands and maximize the overall system performance.

##  II Background and XCache Log Files

SoCal Repo is a storage cache supporting computing jobs in Southern California
for US Compact Muon Solenoid (CMS) experiment, a HEP collaboration with
participants around the world [13, 3]. The analysis jobs involve files of
different types, for example, analysis object data (AOD), MiniAOD, or NanoAOD
files, where the information content per proton collision event differs by
more than a factor of 10 each going from AOD to MiniAOD to NanoAOD. NanoAOD is
thus O(10,000) smaller than AOD. More than 90% of analyses work with either
MiniAOD or NanoAOD [14, 15]. The analysis work mostly starts with exploration
of NanoAOD or MiniAOD files. In a number of cases, after scientists have
determined the most useful algorithms and found the most promising collision
events for their analysis work, they apply some algorithms on the larger data
file formats (which has more detailed information about the selected events)
to produce the final results. This data usage pattern effectively creates two
types of accesses, one type touches small file formats frequently and the
other retrieves large infrequently. The analysis jobs requiring large file
formats might take a considerable amount of time because the file transfers,
including potentially retrieval from tape archives, and computation both are
time consuming.

The SoCal Repo has approximately 2.5PB of total storage with 24 federated
caching nodes. There are 11 nodes at Caltech with storage sizes ranging from
96TB to 388TB, 12 nodes at UCSD with 24TB each node, and one node at ESnet
Sunnyvale endpoint with 44TB of storage. Furthest distance to the cache node
from the computing resources is about 500 miles from UCSD to ESnet Sunnyvale
endpoint, with an RTT of about 10ms. The measurement data has been collected
from July 2021 through June 2022, consisting of 8.7 million data accesses
where 67.6% are satisfied with files in cache, see Table I for additional
summary statistics. Among the 12.7 PB requested, 4.5 PB (35.4%) could be
served from the cache, while 8.2 PB needs to be transferred over wide-area
network. The difference between 67.6% and 35.4% is one of the issues we seek
to resolve.

TABLE I: Summary of data access from July 2021 to June 2022. About 67.6% of file requests are satisfied by this cache, while 35.4% requested bytes are in the cache. |  # of accesses |  cache miss size (TB) |  cache hit size (TB) |  number of cache misses |  number of cache hits
---|---|---|---|---|---
Total | 8,713,894 | 8,210.78 | 4,499.44 | 2,822,014 | 5,891,880
Daily | 23,808 | 22.43 | 12.29 | 7,710 | 16,098

Cache misses occur when the client’s requested data file is not in any of the
cache nodes and needs to be transferred from a remote storage over the wide-
area network. When the client’s requested data is in one of the cache nodes,
it is a cache hit, and the data is served from the cache without a wide-area
data transfer. The network traffic reduction comes from these cache hits.

The cache nodes run on XCache software [16, 17, 7]. The information used in
this study is extracted from XCache log files. We’ve processed 8,433 log files
amounting to about 3 TB, and extracted information about the request sizes,
how the request is satisfied, etc. From such information, we derived cache
hits, cache misses, along with network performance information such as remote
transfer throughput that are used in later sections.

##  III Network Traffic Reduction

(a) Fraction of daily file requests: cache misses (in orange) and cache hits
(in blue)

(b) Fraction of daily requested bytes: cache misses (in orange) and cache hits
(in blue)

Figure 1: Cache miss rates versus cache hit rates based on (a) files requested
and (b) bytes requested.

(a) Daily file requests (count): cache misses (in orange) and cache hits (in
blue)

(b) Daily traffic volume: cache misses (in orange) and cache hits (in blue)

Figure 2: Daily network traffic statistics: (a) files requested and (b) bytes
requested. Note that only file requests that miss the cache trigger remote
network file transfers.

In this section, we show how much wide-area network traffic is actually saved
by SoCal Repo. Figure 1 shows the daily cache hit rate and cache miss rate,
where Figure 1a shows these rates based on the file request counts and Figure
1b based on bytes requested.111There are two narrow gaps in Figure 1 due to
brief periods of down time. Cache misses trigger wide-area network traffic. We
show the cache miss rates with bright orange in Figure 1 and Figure 2

Overall, the cache miss rates based on files requested (in Figure 1a) are more
stable than those based on bytes requested in Figure 1b. In particular, there
is a 5-month long period between Oct. 2021 and Feb. 2022, where the majority
of bytes requested are cache misses. An examination of the number of files
requested and number of bytes requested in Figure 2 provides more information.

Figure 2a shows the number of files requested, separated into those could be
satisfied with files in the cache (hits, in blue) and those require wide-area
network transfers (misses, in orange). Across all 24 cache nodes in the SoCal
Repo, an average day sees about 16,000 file requests as hits along with 8,000
misses. In terms of bytes requested, Figure 2b shows that about 12.3 TB per
day are served out of the cache during the whole year. In the early part of
the year, between Jul. 2021 and Sep. 2021, the wide-area network traffic is
reduced by about 13 TB per day, and between March 2022 and May 2022, the wide-
area network traffic is reduced by about 29TB per day.

In the middle of our observation period, for example January 13, 2022, there
are about 60,000 cache misses amounting to about 200TB of wide-area traffic.
On average, each of these files is over 3.3GB, which means they are large
among CMS data files. This observation from the cache statistics conforms to
the usage patterns involving large files described in Section II. We also
received additional confirmation from the site operators that these are indeed
a small number of data analyses involving large files.

This particular usage pattern involving large files has the potential of
evicting the smaller files (that are used more frequently)222This is
colloquially known as cache pollution. and reducing the overall effectiveness
of the cache system. The operators of SoCal Repo recognized this usage pattern
and have separated and limited the accesses to the cache nodes based on file
types, which effectively prevents cache pollution. In cases where one couldn’t
differentiate the cache usages based on simple known characteristics, an
alternative strategy could be to have these requests bypass the cache system
[18].

##  IV Modeling Transfer Throughput

Now that we know the storage cache is effective in reducing the traffic on
wide-area network, and there are strategies to mitigate the impact of special
access patterns that pollute the cache, we’d like to see how to provision
additional storage cache nodes in the future. For this purpose, we start to
model the current cache usage and network performance. More specifically, we
build machine learning models for the hourly and daily average data throughput
performance as well as statistics about cache misses. The data throughput is
defined as the data transfer size over the transfer time. This information is
useful for anticipating the time needed for file transfers.

We have decided to use a neural network architecture called LSTM [11, 12]
because it is effective in capturing time series patterns. Our model includes
the following features: cache miss count, cache miss size, cache hit count,
cache hit size, aggregate throughput for cache misses on all nodes, aggregate
throughput on cache hits for all nodes, average throughput for each cache
miss, and average throughput for each cache hit. In the remaining of this
section, we discuss the information relates to cache misses, which are more
relevant to the wide-area network performance. The training data comes from
the first 80% of the whole monitoring period, and the testing data comes from
the last 20%. Table II shows hyper-parameters chosen after exploring about
1400 different parameter combinations.

TABLE II: hyper-parameters of the LSTM models # of LSTM unit |  activation function |  dropout rate |  # of epochs
---|---|---|---
128 | tanh | 0.04 | 50
TABLE III: RMSE of daily/hourly LSTM model results for network storage cache performance. The relative prediction errors (inside parentheses) are measured against the standard deviations. Note that all six rows are about cache misses. |  Training RMSE |  Testing RMSE |  standard deviation
---|---|---|---
Daily cache misses | 4306.01 | 3637.39 (.32) | 11317.08
Hourly cache misses | 175.11 | 99.31 (.17) | 595.81
Daily volume | 9.75 | 14.54 (.49) | 29.46
Hourly volume | 0.19 | 0.49 (.35) | 1.42
Daily average throughput | 33.20 | 23.49 (.21) | 110.43
Hourly average throughput | 27.08 | 22.79 (.19) | 121.36

As an overall performance measure, Table III shows the root-mean-square error
(RMSE) of both the daily and hourly models for the data volume and average
(wide-area) network transfer performance. The column labeled “standard
deviation” is the standard deviation of the input data values. It provides a
reference for us to determine how large are the errors of predictions. The
ratios of testing RMSE and standard deviation are shown inside parentheses. In
all cases shown, this relative error is less than 0.5, indicating the
predictions are pretty accurate.

In all three sets of measures shown in Table III, the LSTM models are more
accurate with the hourly time series than with the daily time series as both
absolute and relative error are smaller. The most likely reason might be there
are more training data records for the hourly time series. Next, we look into
more details of these prediction models.

(a) Daily number of cache misses

(b) Hourly number of cache misses

Figure 3: Number of cache misses

(a) Daily volume of cache misses

(b) Hourly volume of cache misses

Figure 4: Bytes in cache misses

(a) Daily throughput of wide-area transfers

(b) Hourly throughput of wide-area transfers

Figure 5: Average throughput of wide-area file transfers

Figure 3 and Figure 4 show LSTM model output for the number of cache misses
and their associated data volumes. In these cases, we see the predictions on
the hourly time series are closer to the actual values in the last period of
time than the predictions with daily time series, even though the hourly time
series often shows stronger spikes.

Figure 5 shows the LSTM model performance with daily and hourly average
throughput values. From Figure 5a, we see that during the middle of the
observation period (10/21 – 2/22), the wide-area network traffic throughputs
are quite high because the network transfers are dominated by relatively large
files that are typically better able to utilize the network capacity. In the
other time periods, the average throughput is relatively low due to small
files being transferred.

In the hourly model from Figure 5b, there are significant number of spikes
during late March and early May. Examining Figures 3b and 4b, we see that
these spikes occur during time periods with very few cache misses, i.e., very
few wide-area file transfers. We are interested in exploring these spikes
further in the future. For modeling network performance, it is not necessary
for us to capture such spikes precisely. We could use the moving average
method to smooth out the spike to obtain a performance model for the general
trends.

(a) Network throughput with 24-hour moving average

(b) Network throughput with 168-hour moving average

Figure 6: Modeling hourly average wide-area throughput on smoothed time
series.

Figure 6 shows two different versions of Figure 5b with two different moving-
averaged hourly throughput measured by the cache misses. We clearly see that
LSTM results match the moving-averages much better than the original time
series shown in Figure 5b. The testing error (RMSE) of the LSTM predictions on
the 24-hour moving-averages is 15.05 (i.e., model in Figure 6a) and
corresponding error on the 168-hour moving average is 14.56. Both of these
errors are less than 22.79 on the original hourly throughput time series. Even
though the 24-hour moving averages look like the daily throughput time series
shown in Figure 5a, the LSTM predictions matches the 24-hour moving averages
much better based on visual inspection. The RMSE of 15.05 (Figure 6a) is
noticeably smaller than 23.49 (Figure 5a). For anticipating future network
performance, the LSTM model based on the moving averages is likely to work
better.

##  V Conclusion

In this study, we set out to understand the effectiveness of in-network
storage cache used by a distributed scientific collaboration. The source
information is a set of 3TB of operational logs from the XCache servers on
SoCal Repo. The data analysis operations of the collaboration commonly involve
two types of files, the smaller sized ones are used frequently with more
reuse, while the larger sized ones are invoked infrequently with less reuse.
We observed that SoCal Repo could on average serve about 67.6% of files from
its disk cache, while on average only 35.4% of bytes requested could be served
from the cache, because the large files are less likely to be reused. To avoid
cache pollution from this usage pattern of large files, the system operators
have separated the two different types of files with different storage nodes.
During the period where fewer large files are requested (3/2022 – 5/2022), the
wide-area network traffic is reduced by about 29TB per day. Over the whole
period of observation, there is a five-month period where the large file
requests are noticeably high. The average reduction of wide-area network
traffic by this cache over the whole observation period is still about 12.3TB
per day, which is quite significant.

This work also explored an option to model the network performances with a
neural network architecture known as LSTM. Tests show that the prediction
error (measured as RMSE) are quite small. In a case where the original time
series has large variations, we also show that the LSTM model could work quite
well on moving-averaged versions of the time series data. With this model, we
plan to consider how to provision future deployments of in-network caches. We
are also planning to study other storage caches currently under deployment to
gain better understanding of in-network caches.

## Acknowledgment

This work was supported by the Office of Advanced Scientific Computing
Research, Office of Science, of the US Department of Energy under Contract No.
DE-AC02-05CH11231, and also used resources of the National Energy Research
Scientific Computing Center (NERSC). This work was also supported by the
National Science Foundation through the grants OAC-2030508, OAC-1836650,
MPS-1148698, PHY-2121686 and OAC-1541349. This work was also supported in part
by the US Department of Energy, Office of Science, Office of Workforce
Development for Teachers and Scientists (WDTS) under the Science Undergraduate
Laboratory Internship (SULI) program.

## References

  * [1] B. Brown, E. Dart, G. Rai, L. Rotman, and J. Zurawski, “Nuclear physics network requirements review report,” Energy Sciences Network, University of California, Publication Management System Report LBNL-2001281, 2020. [Online]. Available: http://www.es.net/assets/Uploads/20200505-NP.pdf
  * [2] X. Espinal, S. Jezequel, M. Schulz, A. Sciaba, I. Vukotic, and F. Wuerthwein, “The quest to solve the hl-lhc data access puzzle,” _EPJ Web of Conferences_ , vol. 245, p. 04027, 2020. [Online]. Available: http://doi.org/10.1051/epjconf/202024504027
  * [3] L. Bauerdick, D. Benjamin, K. Bloom, B. Bockelman, D. Bradley, S. Dasu, M. Ernst, R. Gardner, A. Hanushevsky, H. Ito, D. Lesny, P. McGuigan, S. McKee, O. Rind, H. Severini, I. Sfiligoi, M. Tadel, I. Vukotic, S. Williams, F. Wurthwein, A. Yagil, and W. Yang, “Using xrootd to federate regional storage,” _Journal of Physics: Conference Series_ , vol. 396, no. 4, p. 042009, 2012.
  * [4] E. Fajardo, D. Weitzel, M. Rynge, M. Zvada, J. Hicks, M. Selmeci, B. Lin, P. Paschos, B. Bockelman, A. Hanushevsky, F. Würthwein, and I. Sfiligoi, “Creating a content delivery network for general science on the internet backbone using XCaches,” _EPJ Web of Conferences_ , vol. 245, p. 04041, 2020. [Online]. Available: http://doi.org/10.1051/epjconf/202024504041
  * [5] T. Kosar, M. Balman, E. Yildirim, S. Kulasekaran, and B. Ross, “Stork data scheduler: Mitigating the data bottleneck in e-science,” _Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences_ , vol. 369, no. 1949, pp. 3254–3267, 2011.
  * [6] B. L. Tierney, J. Lee, B. Crowley, M. Holding, J. Hylton, and F. L. Drake, “A network-aware distributed storage cache for data intensive environments,” in _Proceedings. The Eighth International Symposium on High Performance Distributed Computing (Cat. No.99TH8469)_ , 1999, pp. 185–193.
  * [7] E. Fajardo, A. Tadel, M. Tadel, B. Steer, T. Martin, and F. Wwrthwein, “A federated xrootd cache,” _Journal of Physics: Conference Series_ , vol. 1085, p. 032025, 2018.
  * [8] “Energy sciences network,” https://www.es.net, accessed: 2022-10-12.
  * [9] E. Copps, H. Zhang, A. Sim, K. Wu, I. Monga, C. Guok, F. Wurthwein, D. Davila, and E. Fajardo, “Analyzing scientific data sharing patterns with in-network data caching,” in _4th ACM International Workshop on System and Network Telemetry and Analysis (SNTA 2021)_ , ACM. ACM, 2021.
  * [10] R. Han, A. Sim, K. Wu, I. Monga, C. Guok, F. Wurthwein, D. Davila, J. Balcas, and H. Newman, “Access trends of in-network cache for scientific data,” in _5th ACM International Workshop on System and Network Telemetry and Analysis (SNTA 2022)_ , ACM. ACM, 2022.
  * [11] K. Greff, R. K. Srivastava, J. Koutník, B. R. Steunebrink, and J. Schmidhuber, “Lstm: A search space odyssey,” _IEEE transactions on neural networks and learning systems_ , vol. 28, no. 10, pp. 2222–2232, 2016.
  * [12] A. Sherstinsky, “Fundamentals of recurrent neural network (rnn) and long short-term memory (lstm) network,” _Physica D: Nonlinear Phenomena_ , vol. 404, p. 132306, 2020.
  * [13] A. Dorigo, P. Elmer, F. Furano, and A. Hanushevsky, “Xrootd - a highly scalable architecture for data access,” _WSEAS Transactions on Computers_ , vol. 4, no. 4, pp. 348–353, 2005.
  * [14] A. Rizzi, G. Petrucciani, and M. Peruzzi, “A further reduction in cms event data for analysis: the nanoaod format,” _EPJ Web Conf._ , vol. 214, p. 06021, 2019. [Online]. Available: http://doi.org/10.1051/epjconf/201921406021
  * [15] CMS Collaboration, “The cms offline workbook,” https://twiki.cern.ch/twiki/bin/view/CMSPublic/WorkBook, accessed: Oct. 1st, 2022.
  * [16] D. Weitzel, M. Zvada, I. Vukotic, R. Gardner, B. Bockelman, M. Rynge, E. Hernandez, B. Lin, and M. Selmeci, “Stashcache: A distributed caching federation for the open science grid,” in _PEARC ’19: Proceedings of the Practice and Experience in Advanced Research Computing on Rise of the Machines (learning)_ , 07 2019, pp. 1–7.
  * [17] L. Bauerdick, K. Bloom, B. Bockelman, D. Bradley, S. Dasu, J. Dost, I. Sfiligoi, A. Tadel, M. Tadel, F. Wuerthwein, A. Yafil, and the CMS collaboration, “Xrootd, disk-based, caching proxy for optimization of data access, data placement and data replication,” _Journal of Physics: Conference Series_ , vol. 513, no. 4, 2014.
  * [18] T. Malik, R. Burns, and A. Chaudhary, “Bypass caching: making scientific databases good network citizens,” in _21st International Conference on Data Engineering (ICDE’05)_ , 2005, pp. 94–105.