Paper Title: Constraints on large-scale polarization in northern hemisphere.


APS/123-QED





CAS Key Laboratory for Research in Galaxies and Cosmology, Department of Astronomy, University of Science and Technology of China, Hefei, Anhui 230026, China
School of Astronomy and Space Science, University of Science and Technology of China, Hefei 230026, China
Kavli IPMU (WPI), UTIAS, The University of Tokyo, Kashiwa, Chiba 277-8583, Japan



CAS Key Laboratory for Research in Galaxies and Cosmology, Department of Astronomy, University of Science and Technology of China, Hefei, Anhui 230026, China
School of Astronomy and Space Science, University of Science and Technology of China, Hefei 230026, China



CAS Key Laboratory for Research in Galaxies and Cosmology, Department of Astronomy, University of Science and Technology of China, Hefei, Anhui 230026, China
School of Astronomy and Space Science, University of Science and Technology of China, Hefei 230026, China


[Corresponding author: ]yifucai@ustc.edu.cn
CAS Key Laboratory for Research in Galaxies and Cosmology, Department of Astronomy, University of Science and Technology of China, Hefei, Anhui 230026, China
School of Astronomy and Space Science, University of Science and Technology of China, Hefei 230026, China


[Corresponding author: ]changfeng@ustc.edu.cn
CAS Key Laboratory for Research in Galaxies and Cosmology, Department of Astronomy, University of Science and Technology of China, Hefei, Anhui 230026, China
School of Astronomy and Space Science, University of Science and Technology of China, Hefei 230026, China






             

§ ABSTRACT 
Present cosmic microwave background (CMB) observations have significantly advanced our understanding of the universe's origin, especially with primordial gravitational waves (PGWs). Currently, ground-based CMB telescopes are mainly located in the southern hemisphere, leaving an untapped potential for observations in the northern hemisphere. 
In this work, we investigate the perspective of a northern hemisphere CMB polarization telescope (NHT) to detect PGWs and present mock data for such a project. 
We forecast the detection sensitivity on the tensor-to-scalar ratio r of NHT and compare it with the existed ground-based experiments, also search for optimal experimental configurations that can achieve the best sensitivity of r.
Our results indicate that, considering realistic experimental conditions, the first year of NHT observations combined with Planck can achieve a precision of σ (r)= 0.015, reaching the level of BICEP2/Keck, with significant potential for improvement with subsequent instrumentation parameter enhancements.



                              
Constraints on large-scale polarization in northern hemisphere
    Chang Feng
    August 5, 2024
==============================================================







§ INTRODUCTION


	
Inflation theory is fundamental to modern cosmology, resolving several problems in Big Bang cosmology, such as the horizon, flatness, and monopole issues <cit.>.
Furthermore, inflation predicts that quantum fluctuations in the inflaton field and spacetime produce scalar and tensor modes, leading to primordial perturbations.
Scalar modes cause temperature anisotropies in the cosmic microwave background (CMB), which lead to the formation of large-scale structures <cit.>.
On the other hand, tensor modes, also known as primordial gravitational waves (PGWs), left their imprint on the B-mode polarization of the CMB <cit.>.

Over the past few decades, progress has been made toward understanding the universe's origins through long-term observations of the CMB.
The Planck mission marked a critical milestone in this field, providing precise observations that ruled out the simplest model of power-law inflation <cit.>. Ground-based CMB experiments <cit.> have also achieved even more precise measurements due to much lowered instrumental noise and sophisticated low-temperature detection techniques.
Currently, the BICEP3/Keck experiment has achieved the best constraints on the tensor-to-scalar ratio r, with an upper bound of r < 0.036 (95%) <cit.>.
However, a definitive detection of PGWs remains elusive, and researchers continue to search for these signals <cit.>. 
The CMB polarization studies become an active area of research with considerable potential for uncovering new physics and broadening our understanding of the universe.

However, ground-based telescopes may suffer from complicated technical issues such as geographic locations and sky-scanning strategies which can result in different sky coverage and levels of atmospheric contamination. To date, most ground-based CMB telescopes are located in the Southern Hemisphere, such as the Atacama desert and the south pole.
The Northern Hemisphere would become another important observing location<cit.>.

In this work, we constructed a generic mock survey for a futuristic Northern Hemisphere CMB Telescope (NHT). 
We investigated the detection sensitivity of PGWs with the NHT and examined the impact of instrumental configurations on the PGW detection sensitivity. These forecasts could become a guidance for the design of future NHT.

This paper is organized as follows:
In section <ref>, we introduce the instrument summary and the forecasting methodology used in our calculations. 
We present our forecast results and analyze the impact of various experimental parameters on the sensitivity of detecting PGWs in section <ref>.
Finally, we conclude in section <ref>. 




§ METHODOLOGY
 


 §.§ Instrument Summary
 
Ground-based CMB telescopes face a significant challenge: atmospheric molecules in the millimeter/sub-millimeter band absorb and emit signals, compromising data quality.
Water vapor is particularly problematic due to its strong absorption properties and quick variability.
Precipitable Water Vapor (PWV), which represents the total water depth in an atmospheric column above ground level, is typically used to quantify atmospheric water vapor.
Given the faint nature of CMB signals, especially polarization signals, CMB observation sites must provide exceptionally clear, dry, and stable atmospheric conditions.
According to MERRA-2 data on global mean PWV distribution from July 2011 to July 2016 <cit.>, only Antarctica, the Atacama Desert, Greenland, and the high Tibetan Plateau exhibit minimal PWV levels. These regions are thus the most suitable for ground-based CMB telescopes.

Currently, all ground-based CMB telescopes are located in the southern hemisphere, primarily in Chile and Antarctica. No significant CMB data has been collected from the northern hemisphere.
This situation is set to change with the advancement of Northern Hemisphere telescope projects like AliCPT and GreenPol. These projects aim to provide high-precision measurements of CMB polarization in the northern sky in the near future.
For forecasting purposes, we designate these Northern Hemisphere Telescopes as NHTs. We generate mock data based on a power spectrum independent of specific pipeline methods. To evaluate the potential performance of NHTs, we employ the Markov Chain Monte Carlo (MCMC) method.

The design of a CMB telescope requires the meticulous selection of several parameters. The critical parameters selected for our hypothetical telescope encompass the survey area f_sky, the number of channels, the Full Width at Half Maximum (FWHM), and the noise level w^- 1/2. We opted for two channels to monitor the 95 GHz and 150 GHz bands, with FWHM values of 19 arcmin and 11 arcmin respectively. The survey area f_sky is set at 0.1. The rationale behind the selected noise level will be discussed in the subsequent section.



 §.§ Foreground

Since the foreground for surveys in the northern sky and the southern sky are different, establishing the NHT provides an excellent window for independent verification of signals that may be discovered in the future by southern telescopes.

Foreground contamination is a challenge that needs to be carefully considered for NHT. The galactic emissions, particularly thermal dust and synchrotron emission, are major contributors to the foreground signal that needs to be separated from the desired B-mode signal by the so-called “component separation" methods. These methods are usually divided into blind <cit.>, parametric <cit.>, and template removal techniques <cit.>.

Several studies have forecasted the performance of blind methods on B-mode mock data for some future experiments <cit.>. 
In the case of the northern hemisphere, previous studies also evaluated the performance of various pipeline methods in cleaning foregrounds, with residual foreground contamination at least an order of magnitude below noise for their designed experiments at the multipoles of interest (ℓ≲ 100) for constraining r <cit.>.
Therefore, in our work, we safely neglect the residual foreground and focus on the impact of the main instrument parameters on the constraints of the tensor-to-scalar ratio r.



 §.§ Noise Model

The modeling of our anticipated noise spectra contains the white noise and the 1/f noise component,

    N_l = N_red(ℓ/ℓ_knee)^α_knee + N_white.
 
The 1/f noise component, mainly from atmospheric and electronic noise, is represented by N_red, ℓ_knee, and α_knee. The white noise component is denoted by N_white.
The white noise component is denoted by N_white. 
Using this noise model, it is estimated that a small-aperture ground telescope dedicated to large-scale CMB physics could have a noise model with ℓ_knee≈ 50 and α_knee between -3.0 and -2.4, describing the uncertainty in the B-mode power spectrum, as discussed in <cit.>.
We adopt the assumption that ℓ_knee=30, α_knee=-2.5/-3.0 for 95/150GHz, and we fix N_red=N_white, as shown in Fig. <ref>.
In the polarization noise spectrum at 95 GHz, the 1/f noise unnaturally rises at high ℓ due to the exponential increase of N_white at high ℓ in the parameterization. 
However, this defect does not affect the calculation, as white noise dominates at high ℓ.

We refrain from modeling the 1/f noise on the TT spectrum, as our hypothetical experiment only involves a small aperture telescope, rendering the Large Aperture Telescope modeling scenario irrelevant. From a data perspective, we argue that this omission will not affect the actual observational results, as the TT spectrum has already been precisely measured by WMAP and Planck. From a physical standpoint, when using CMB to detect PGWs, the primary focus is on the BB polarization power spectrum, and there is no need to be concerned about this bias term in the noise model of the TT power spectrum. Even so, we will show in the calculation analysis later that for the polarization mode, the consideration of 1/f noise does not significantly affect the restrictions on PGWs.

To derive a reliable estimate of N_white, we draw from the findings of previous forecasts for northern hemisphere CMB experiments, which employed specific pipeline methods <cit.>, with σ(r)≈0.02. 
The MCMC method, as explained in Section <ref>, is utilized to estimate the noise level with polarized modes w_P=40 μK· arcmin for a single channel. 
Given that the hypothetical NHT has two channels, the final noise power spectrum should be computed according to Eq. <ref>. The resulting effective noise level aligns closely with <cit.>, despite the variation in calculation methodology. We list the initial operating parameters of NHT (named NHT-1) in Table <ref>.

Assuming statistical independence of the noise in temperature and polarization in the CMB power spectra, the white noise spectra in TT, EE, and BB for scenarios with multiple channels are <cit.>:

    N^PP_white = ( ∑_v w_P, νexp[ - ℓ (ℓ + 1)
      θ_FWHM, ν^2/8 ln 2] )^-1, 
        N^TT_white = 1/2 N^PP_white .

Here, N^PP_white denotes the white noise spectra in EE and BB modes, and N^TT_white denotes the white noise spectrum in TT mode. 
Based on the assumed parameters in Table <ref>, the fiducial N^PP_ℓ power spectrum is shown in Fig. <ref>.




For the lensing reconstruction field, the quadratic estimation method is employed to calculate the error on the power spectrum, as detailed in <cit.>. The computations are implemented using the publicly accessible code, QuickLens[https://github.com/dhanson/quicklens]. Given the absence of real data in NHT, we utilize the mock spectra from Planck, provided in <cit.>, for forecasting purposes.





 §.§ Forecasting Procedure


With NHT's equipment parameters, we simulate the CMB TT, EE and BB power spectra as shown in Fig. <ref>. 
We use the best-fit parameters of Planck 2018 <cit.> which is shown in Table <ref> and assume that the tensor to scalar r is 0.014 where the pivot scale k_∗ is selected as 0.05 Mpc^-1 as our fiducial model based on the ΛCDM+r model.
We also forecast the joint NHT and Planck constraints on r for comparison with BICEP series. 



The likelihood function we used is as follows


    -2lnℒ=∑_X=T, E, B∑_ℓ(Ĉ_ℓ^XX-C_ℓ^XX/Δ C_ℓ^XX)^2,


where Ĉ_ℓ^XX are the mock power spectra,  C_ℓ^XX are the fiducial power spectra, and the error term Δ C_ℓ^XX satisfy

    Δ C_ℓ^XX≃√(2/(2ℓ + 1)f_sky)(C_ℓ^XX + N_ℓ^XX).

We utilize the publicly MCMC code Montepython
<cit.> and the publicly available Boltzmann code CLASS <cit.> to perform the calculations.
We choose the Metropolis Hastings algorithm as our sampling method. For each model, we run MCMC calculation until R-1 for each fundamental parameter reach to the order of 0.001, according to the Gelman-Rubin criterion <cit.>.

We choose ℓ_min = 30 in order to consider the influence of PWV on the ground-based telescope follow the prescription in <cit.>. For ℓ_max, we estimate it to be the reciprocal of the best FWHM in the two channels. In our calculation,
it is assumed that ℓ_max = 1000.

This study presents a general discussion on the constraining power of NHT on PGWs and the improvement in detection capability following instrumental parameter optimization.
Rather than focusing on specific pipeline construction methods, foreground separation techniques, atmospheric noise subtraction, or sky survey strategies, we concentrate on the direct influence of key experimental parameters on the observed CMB power spectra.
This approach ensures our results provide generic guidance applicable to any future technical strategy adopted by NHT.




§ RESULT


After parameter estimation for NHT-1, we combined the NHT-1 data with a mock Planck 2018 dataset, utilizing the ΛCDM+r model to compute the MCMC. This approach allows for a direct comparison with the results reported by BICEP series.
Since our forecast for NHT uses a simulated power spectrum, it cannot be combined with real data in the Montepython program. Therefore, for computational convenience, we used the simulated Planck 2018 power spectrum developed by Brickmann <cit.>. The constraints of the simulated Planck dataset on the cosmological parameters of the ΛCDM model differ by no more than 1% from the real Planck 2018 data. 





For the primordial B-mode, we set the fiducial value to r=0.014, consistent with the non-zero central value reported by BICEP3/Keck <cit.>, to ensure a reliable comparison.
Using the NHT-1 only to constrain r, we obtain its constraint sensitivity as σ (r) = 0.023.

In the data combination process, we replace the CMB TT, EE, and BB power spectra of mock Planck in the range of l= 30-1000 with those from NHT-1, corresponding to a 10% sky coverage. 
More specifically, in the joint computation, we used low-ℓ data ranging from ℓ=2-29 with a Planck precision and f_sky of 0.57, data ranging from ℓ=30-1000 with an NHT-1 precision and f_sky of 0.1, data ranging from ℓ=30-1000 with a Planck precision and f_sky of 0.47, and high-ℓ data ranging from ℓ=1001-3000 with a Planck precision and f_sky of 0.57. These experimental parameters are shown in Table <ref>.

In Fig. <ref> to Fig. <ref>, we plot the power spectrum of each dataset.
The joint calculation shows σ (r) = 0.015.
We present in Fig. <ref> a comparison of the sensitivity of constraints on r from NHT-1 only and NHT-1 combined with Planck.
This result shows that the constraining accuracy of NHT-1 + Planck 2018 on PGWs can be better than the results of BICEP2/Keck + Planck 2015 <cit.>, σ (r)<0.020.













To study the impact of red noise in the noise model, we removed the red noise term in Eq. <ref> and calculated the forecast on σ (r) using only NHT-1. In this case, σ (r) = 0.019.
Figure <ref> shows the forecast for the constraint of r by NHT-1, with and without 1/f noise.

Furthermore, we explore how certain instrument parameters can enhance the ability to constrain PGWs.
In the previous calculation, we assumed that the ℓ range is 30ℓ1000 for NHT as discussed in Section <ref>. Considering that the actual construction of the instrument may be inconsistent with our assumptions during the real observation, we evaluated the impact of changing the boundary conditions of ℓ_min on constraining PGWs. The calculation results are listed in the Fig. <ref>, which shows that changing ℓ_min from 30 to 70 has a notable influence on the σ (r) from 0.021 to 0.037. 
This shows that low-ℓ polarization mock data greatly affects σ (r), indicating that stringent atmospheric noise removal techniques are crucial for improving NHT's ability to constrain PGWs. 
Regarding ℓ_max, increasing the telescope aperture and thereby reducing FWHM can increase ℓ_max. However, since when detecting PGWs, the power spectrum region we concern about is ℓ≲ 100, the construction of a large-aperture telescope is not suitable for the scientific goals of the primordial B-mode detection. Nevertheless, previous research has shown that the construction of large-aperture telescopes has significant implications for other scientific goals <cit.>.







In Table <ref>, we compare the marginal effects on the constraining capacity of n_s and r when the noise level of NHT is reduced. 
We do not show the n_s-r panel to avoid possible misunderstandings as a forecast paper. our calculations show that reducing noise levels markedly improves the ability to search for PGWs. 
The improvement of constraints on σ (r) from 20 μK· arcmin to 10 μK· arcmin is more pronounced than from 40 μK· arcmin to 20 μK· arcmin, while the ability to reduce the constraint on σ (n_s) is relatively small. 
In particular, after being combined with Planck, σ (n_s) has little to do with the noise level of NHT. 
This result is easy to understand because Planck provides a very precise temperature power spectrum, which has the primary effect on the n_s constraint; The σ (r) is directly related to the high-precision polarized B-mode signal, so it is greatly affected by the noise level of NHT. 
The noise level is inversely proportional to the detection time or the number of detectors, and this result suggests that we should strive to keep the NHT noise level below 10 μK· arcmin, which can be achieved by accumulating data over some years of observation seasons or adding detector modules. 
For NHT-1 at a noise level of 40 μK· arcmin, even without increasing the number of detectors, it is theoretically possible to reach the level of 10 μK· arcmin after four years of observation, so our results indicate that the marginal benefit of continuous observation and increasing the number of detectors is considerable.

In Fig. <ref>, we show the impact of increasing f_sky on improving the accuracy of the constraint on r with NHT only. When f_sky increases from 0.1 to 0.7, σ (r) decreases from 0.023 to 0.0071, demonstrating a significant effect on increasing the constraining sensitivity of r.
We also plot the curve of σ(r)∝√(f_sky) starting from f_sky=0.3 in the figure. The calculated results deviate from this relationship between f_sky = 0.1 and f_sky = 0.3 because the 1D constraint on r forecasted by NHT-1 is not a standard Gaussian form.
In addition, since NHT's sky surveys rely on specific clean sky areas in the northern hemisphere, such significant strategic expectations may not be possible for the construction of NHT. Nevertheless, this finding highlights the high scientific significance of a global network of ground-based CMB telescopes to jointly detect PGWs.



§ CONCLUSIONS


In this work, we constructed a generic simulated survey data for Northern Hemisphere CMB telescope, NHT, independent of specific pipeline methods, and forecasted its sensitivity to the tensor-to-scalar ratio of PGWs using the MCMC method.
Combining NHT data with the mock data from Planck 2018, we forecast a sensitivity of σ (r) = 0.015 and r < 0.054 (95% C.L.) for a fiducial model with r = 0.014. This is expected to surpass the BICEP2/Keck report.

Moreover, we explore how certain instrument parameters can enhance the ability to constrain PGWs.
We found that low-ℓ polarization mock data significantly affects σ(r), indicating that strict atmospheric noise removal techniques are crucial for improving NHT's ability to constrain PGWs.

Additionally, our calculations show that reducing noise levels significantly enhances the ability to search for PGWs. The improvement from 20 μK· arcmin to 10 μK· arcmin is more substantial than from 40 μK· arcmin to 20 μK· arcmin, indicating significant marginal effects from continuous observation and increased detector numbers

Furthermore, we study the impact of increasing sky coverage f_sky and show that it significantly affects the constraining sensitivity on r.
Since NHT's sky surveys rely on specific clean sky areas, achieving these strategic expectations may be challenging. However, this highlights the high scientific significance of a global network of ground-based CMB telescopes for joint PGW detection.

In summary, based on realistic and feasible experimental parameters, NHT has the scientific potential to achieve leading international constraints on PGWs and make continuous progress.
Additionally, due to the lack of high-precision ground-based CMB data in the northern sky, the addition of NHT can supplement existing data and independently verify or combine with future experiments like BICEP ARRAY or CMB-S4.
We analyzed and demonstrated through mock data that constructing NHT holds significant scientific importance for the near-future detection of PGWs. 



We are grateful to Hong Li, Siyu Li, Yang Liu, Wentao Luo, Toshiya Namikawa, Larissa Santos and Rui Shi for valuable communications. 
This work is supported in part by the National Key R&D Program of China (2021YFC2203100), CAS Young Interdisciplinary Innovation Team (JCTD-2022-20), NSFC (12261131497, 11653002, 12003029), 111 Project for “Observational and Theoretical Research on Dark Matter and Dark Energy” (B23042), Fundamental Research Funds for Central Universities, CSC Innovation Talent Funds, USTC Fellowship for International Cooperation, USTC Research Funds of the Double First-Class Initiative, CAS project for young scientists in basic research (YSBR-006). Kavli IPMU is supported by World Premier International Research Center Initiative (WPI), MEXT, Japan. We acknowledge the use of computing facilities of astronomy department, as well as the clusters LINDA & JUDY of the particle cosmology group at USTC. 









 Paper Title: Mass mixing between QCD axions.





 
lihaijun@itp.ac.cn
Key Laboratory of Theoretical Physics, Institute of Theoretical Physics, Chinese Academy of Sciences, Beijing 100190, China
 

yfzhou@itp.ac.cn
Key Laboratory of Theoretical Physics, Institute of Theoretical Physics, Chinese Academy of Sciences, Beijing 100190, China
School of Physical Sciences, University of Chinese Academy of Sciences, Beijing 100049, China
School of Fundamental Physics and Mathematical Sciences, Hangzhou Institute for Advanced Study, UCAS, Hangzhou 310024, China
International Centre for Theoretical Physics Asia-Pacific, Beijing/Hangzhou, China

ITP-24-166



§ ABSTRACT 

We introduce a novel level crossing in the mass mixing between two QCD axions, one canonical QCD axion and one Z_𝒩 QCD axion.
The level crossing can take place at the QCD phase transition critical temperature or slightly before it, depending on the ratio of the axion decay constants ∼1.69.
The cosmological evolution of the mass eigenvalues in these two cases is similar, however, the transition of axion energy density is completely different.
Finally, we estimate the relic density of the QCD axion dark matter.
This level crossing may also have some cosmological implications.





Mass mixing between QCD axions
    Yu-Feng Zhou
    August 5, 2024
==============================






Introduction.—
The QCD axions are attractive candidates for the cold dark matter (DM).
The canonical QCD axion was predicted by the Peccei-Quinn (PQ) mechanism <cit.> to solve the strong CP problem in the Standard Model (SM) <cit.>.
It obtains a tiny mass from the QCD non-perturbative effects <cit.>.
As the DM candidate, it can be non-thermally produced in the early Universe through the misalignment mechanism <cit.>.
At high cosmic temperatures, the QCD axion is massless and obtains a non-zero mass during the QCD phase transition, the axion oscillation when its mass is comparable to the Hubble parameter explains the observed DM abundance.
See e.g. Ref. <cit.> for a recent review.
The reduced-mass QCD axion, Z_𝒩 axion <cit.>, can also solve the strong CP problem with 𝒩⩾3 <cit.> and account for the DM through the trapped plus kinetic misalignment mechanism <cit.>.
Here 𝒩 is an odd number, the 𝒩 mirror worlds that are nonlinearly realized by the axion field under a Z_𝒩 symmetry can coexist in Nature.
Due to the suppressed non-perturbative effects on axion potential from the 𝒩 degenerate QCD groups, the Z_𝒩 axion mass is exponentially suppressed at the QCD phase transition.
To avoid confusion, the following “QCD axion" only stands for the canonical QCD axion.

The mass mixing in the multiple axions model <cit.> has attracted extensive attention in recent years.[For instance, the mass mixing between the QCD axion and axionlike particle (ALP) <cit.> or sterile axion <cit.>, and the mixing between the Z_𝒩 axion and ALP <cit.>, etc.]
Considering the non-zero mass mixing between these axions, the cosmological evolution process of the mass eigenvalues called the level crossing can take place and induce the adiabatic transition of the axion energy density, which is similar to the MSW effect <cit.> in the neutrino oscillations.
The level crossing has some interesting cosmological implications, such as the modification of the axion relic density and isocurvature perturbations <cit.>, the domain walls formation <cit.>, the gravitational waves emission and primordial black holes formation <cit.>, and also the dark energy composition <cit.>, etc.

In this letter, we investigate the mass mixing between two QCD axions, one QCD axion and one Z_𝒩 axion.
We find that the level crossing can take place at the QCD phase transition critical temperature or slightly before it, depending on the relation of the axion decay constants.
The conditions for level crossing to occur in these two cases are discussed in detail.
The cosmological evolution of the mass eigenvalues in these two cases is similar, while the transition of energy density between the axions is different.
Finally, we briefly estimate the relic density of the QCD axion and Z_𝒩 axion DM through the misalignment mechanism.
To our knowledge, this is the first time to investigate the effect of level crossing in the mass mixing with a multiple QCD axions model.

The model.—
Here we consider a minimal multiple QCD axions model, one QCD axion ϕ and one Z_𝒩 axion φ, with the low-energy effective Lagrangian

    ℒ   ⊃12∑_ϕ, φ∂^μΦ∂_μΦ-m_a^2(T) f_a^2 [1-cos(ϕf_a)]
       -m_𝒩^2(T) F_a^2𝒩^2[1-cos(ϕf_a+𝒩φF_a)]  ,

where m_a(T) and m_𝒩(T) are the temperature-dependent QCD axion and Z_𝒩 axion masses, respectively, f_a and F_a are the QCD axion and Z_𝒩 axion decay constants.
The QCD axion mass m_a(T) can be described by

    m_π f_πf_a√(z)1+z  ,     T≤ T_ QCD
    m_π f_πf_a√(z)1+z(TT_ QCD)^-b  ,     T> T_ QCD
 
where m_π and f_π are the mass and decay constant of the pion, respectively, z≡ m_u/m_d≃0.48 is the ratio of the up to down quark masses, T_ QCD≃150  MeV is the QCD phase transition critical temperature, and b≃4.08 is an index taken from the dilute instanton gas approximation.
The first term in Eq. (<ref>) corresponds to the zero-temperature QCD axion mass m_a,0.
While the Z_𝒩 axion mass m_𝒩(T) can be described by

    m_π f_π√(π) F_a√(1-z1+z)𝒩^3/4z^𝒩/2  ,    T≤ T_ QCD
    m_π f_πF_a√(z1-z^2)  ,    T_ QCD < T ≤T_ QCDγ
    m_π f_πF_a√(z1-z^2)(γ TT_ QCD)^-b  ,    T>T_ QCDγ

where γ∈(0,1) is a temperature parameter.
The first term in Eq. (<ref>) also corresponds to the zero-temperature Z_𝒩 axion mass m_𝒩,0, which is suddenly exponentially suppressed at T_ QCD due to the Z_𝒩 symmetry.
Then the equations of motion of ϕ and φ are given by

    ϕ̈   +3Hϕ̇+m_a^2(T) f_asin(ϕf_a)
       +m_𝒩^2(T) F_a^2𝒩^2 f_asin(ϕf_a+𝒩φF_a)=0  ,


    φ̈+3Hφ̇+m_𝒩^2(T) F_a𝒩sin(ϕf_a+𝒩φF_a)=0   ,

where H(T) is the Hubble parameter, and the dots represent derivatives with respect to the physical time t.
Diagonalizing the mass mixing matrix

    𝐌^2=
    (
    [ m_a^2(T)+m_𝒩^2(T) F_a^2𝒩^2 f_a^2                 m_𝒩^2(T) F_a𝒩f_a;                 m_𝒩^2(T) F_a𝒩f_a                         m_𝒩^2(T) ])  ,

we derive the heavy (a_h) and light (a_l) mass eigenstates

    (
    [ a_h; a_l ])
    =
    (
    [  cosα  sinα; -sinα  cosα ])
    (
    [ ϕ; φ ])  ,
 
with the corresponding mass eigenvalues m_h,l(T)

    m_h,l^2(T)   =12[m_a^2(T)+m_𝒩^2(T)+ m_𝒩^2(T) F_a^2𝒩^2 f_a^2]
       ±12𝒩^2 f_a^2[-4 𝒩^4 m_a^2(T) m_𝒩^2(T) f_a^4
       +((m_a^2(T)+m_𝒩^2(T))𝒩^2 f_a^2
       +m_𝒩^2(T) F_a^2)^2]^1/2  ,
 
where α is the mass mixing angle

    cos^2α=12(1+m_𝒩^2(T)-m_a^2(T)-m_𝒩^2(T) F_a^2𝒩^2 f_a^2m_h^2(T)-m_l^2(T))  .
 

The so-called axion level crossing is that when considering the interaction between two axion fields, there is a nontrivial mass mixing between their temperature-dependent mass eigenvalues m_h,l(T) that as functions of the cosmic temperature T.
It can take place when the difference of m_h^2(T)-m_l^2(T) gets a minimum value.




Case I: Level crossing at T_ QCD.—
We first consider a case that the zero-temperature QCD axion mass m_a,0 is smaller than the second term in Eq. (<ref>) (we can define it as the mass m_𝒩,π)

    m_π f_πf_a√(z)1+z<m_π f_πF_a√(z1-z^2)  ,

it can be characterized as

    ζ≡F_af_a<√(1+z1-z)≃ 1.69   ,

where we have defined the ratio ζ.
On the other hand, in order for the level crossing to occur in this case, the zero-temperature QCD axion mass should be larger than the zero-temperature Z_𝒩 axion mass m_𝒩,0, we have

    m_π f_πf_a√(z)1+z>m_π f_π√(π) F_a√(1-z1+z)𝒩^3/4z^𝒩/2  ,

it can also be characterized as

    ζ   >1√(π)√((1-z)(1+z)^3z^2)𝒩^3/4z^𝒩/2
       ≃ 1.24×0.48^𝒩/2𝒩^3/4  ,

which depends on the value of 𝒩.
Then we show in Fig. <ref> an illustration of level crossing in this case.
We have set f_a=10^12  GeV, F_a=10^11.5  GeV, 𝒩=17, and γ=0.25.
The red and blue lines correspond to the temperature-dependent mass eigenvalues m_h(T) and m_l(T), respectively.
Note that the level crossing occurs at the QCD phase transition critical temperature

    T_× = T_ QCD  .
 
Let us discuss the cosmological evolution of the mass eigenvalues in this case.
At high temperatures, the heavy mass eigenvalue m_h(T) corresponds to the Z_𝒩 axion, while the light one m_l(T) corresponds to the QCD axion.
They will approach to each other at T_ QCD and then move away from each other.
When T < T_ QCD, the m_h(T) represents the QCD axion, and the m_l(T) represents the Z_𝒩 axion.
For a more intuitive description, they can be expressed as

    m_h(T)⇒
    m_a(T)  ,    T< T_×
    
    m_𝒩(T)  ,    T> T_×


    m_l(T)⇒
    m_𝒩(T)  ,    T< T_×
    
    m_a(T)  .    T> T_×
 
The above discussion is focused on the Case I that level crossing occurs at T_ QCD.

To compare with another case discussed below, we show the condition for level crossing to occur in the Case I by using Eqs. (<ref>) and (<ref>).
See Fig. <ref> with the red shadow region in the {𝒩,  ζ} plane.
The blank regions represent that no level crossing occurs.







Case II: Level crossing slightly before T_ QCD.—
Next we consider another case that the zero-temperature QCD axion mass is larger than m_𝒩,π, we have

    ζ>√(1+z1-z)≃ 1.69   .

We show an illustration of this case in Fig. <ref>.
We have set f_a=10^12  GeV, F_a=10^12.5  GeV, 𝒩=13, and γ=0.25.
Different from the Case I, here the level crossing will take place slightly before the temperature T_ QCD.
By solving d(m_h^2(T)-m_l^2(T))/dT=0 at T_ QCD < T < T_ QCD/γ, we obtain the level crossing temperature

    T_×=T_ QCD(√(1-z1+z)ζ)^1/b(1-ζ^2𝒩^2)^-1/(2b)  ,

which is slightly larger than T_ QCD.
The QCD axion mass at T_× is given by

    m_a(T_×)≃ m_𝒩,π√(1-ζ^2𝒩^2)  .

Note that ζ<𝒩 should be satisfied in Eq. (<ref>).
This level crossing will last for a parametric duration

    Δ T_×=|1cosα(T_×)dcosα(T_×)dT|^-1  ,
 
where α(T) is the mass mixing angle given by Eq. (<ref>). 
This is the Case II.
The temperature-dependent behavior of the mass eigenvalues is similar to the Case I.
However, the transition of energy density between them is completely different, which will be discussed later.
In fact, the Case I can transform into the Case II, depending on the value of the ratio ζ∼1.69. 

Then we show in Fig. <ref> (the blue shadow region) the condition for level crossing to occur in the Case II by using Eq. (<ref>) and ζ<𝒩.
Additionally, we also note that at T_ QCD/γ the QCD axion mass should be smaller than the Z_𝒩 axion mass   

    m_π f_πf_a√(z)1+zγ^b<m_π f_πF_a√(z1-z^2)  ,

it can be characterized as

    ζ<√(1+z1-z)γ^-b≃ 1.69  γ^-b  ,

which depends on the value of γ.
Since for the small γ, the constraint of ζ<𝒩 is more stringent than Eq. (<ref>), we do not show it in Fig. <ref>.

Axion relic density.—
Here we briefly estimate the relic density of the axion DM in the Cases I and II through the misalignment mechanism.
The pre-inflationary scenario is considered, in which the PQ symmetry is spontaneously broken during inflation. 

We first discuss the Case I.
As an example, we focus our attention on the Z_𝒩 axion DM.
We begin with the QCD axion field, which is frozen at high temperatures with an arbitrary initial misalignment angle θ_1,a, and starts to oscillate at T_1,a.
The oscillation temperature T_1 is given by m(T)=3H(T) with the Hubble parameter

    H(T)=√(π^2 g_*(T)90)T^2m_ Pl  ,
 
where m_ Pl is the reduced Planck mass, and g_* is the number of effective degrees of freedom of the energy density.
In the following, the subscript “x" stands for the physical quantity at T_x or corresponding to x.
The initial energy density in the QCD axion field is

    ρ_a,1=12m_a,1^2 f_a^2 θ_1,a^2  .

At T_ QCD<T<T_1,a, the QCD axion energy density is adiabatic invariant with the comoving number N_a ≡ρ_a a^3 /m_a, where a is the scale factor.
Then we have the QCD axion energy density just before T_ QCD as

    ρ_a,  QCD=12 m_a,0 m_a,1 f_a^2 θ_1,a^2 (a_1,aa_ QCD)^3   .

In this case, the QCD axion is trapped around θ_ tr≡θ_a(T_ QCD)∼π until T_ QCD with the initial axion velocity θ̇_ tr≡θ̇_a(T_ QCD).
At T=T_ QCD, the light mass eigenvalue m_l(T) will comprise the Z_𝒩 axion, the axion mass is suddenly suppressed and the true minimum ∼ 0 develops.
Then we have the mean velocity of the Z_𝒩 axion

    √(⟨θ̇_ tr^2⟩)=1√(2)ζ√(m_a,0 m_a,1)θ_1,a(a_1,aa_ QCD)^3/2  .

Note that the Z_𝒩 axion energy density at T_ QCD is non-adiabatic, which just after T_ QCD is given by

    ρ_𝒩,  tr=12 F_a^2 θ̇_ tr^2+2m_𝒩^2 F_a^2𝒩^2  ,

where we consider a case that the axion has a large mean velocity.
Then the Z_𝒩 axion will start to oscillate at T_2, which is defined as the temperature when the kinetic energy is equal to the barrier height.
At T_2<T<T_ QCD, the Z_𝒩 axion energy density is conserved with the comoving PQ charge q_ kin=θ̇_𝒩 a^3, and we have the scale factor at T_2 as a_2=(𝒩θ̇_ tr/(2m_𝒩))^1/3a_ QCD.
Then at T_0<T<T_2, the adiabatic approximation is valid, and we have the Z_𝒩 axion energy density at present T_0 as

    ρ_𝒩,0 = m_𝒩,0 F_a^2 θ̇_ tr𝒩(a_ QCDa_0)^3  .

Substituting the mean velocity, it reads

    ρ_𝒩,0 = C m_𝒩,0√(m_a,0 m_a,1)θ_1,a F_a^2√(2)𝒩ζ(√(a_1,a a_ QCD)a_0)^3  ,

where C≃2 is a constant.
 

 

Next we discuss the Case II, focusing on the QCD axion DM.
We begin with the Z_𝒩 axion field, it starts to oscillate at T_1,𝒩 with the initial misalignment angle θ_1,𝒩, and the initial energy density is given by

    ρ_𝒩,1=12m_𝒩,1^2 F_a^2 θ_1,𝒩^2  .

At T_×<T<T_1,𝒩, the Z_𝒩 axion energy density is adiabatic invariant, and at T_× we have 

    ρ_𝒩,×=12 m_𝒩,π m_𝒩,1 F_a^2 θ_1,𝒩^2 (a_1,𝒩a_×)^3   .

At T=T_×, the level crossing occurs, the heavy mass eigenvalue m_h(T) will comprise the QCD axion and the energy density ρ_𝒩,× is transferred to the QCD axion ρ_a,×.
To ensure this axion energy transition is adiabatic at T_×, we need

    Δ t_×≫max[2πm_l(T_×),  2πm_h(T_×)-m_l(T_×)]  ,

where the time Δ t_× corresponding to the temperature in Eq. (<ref>).
We also check it in Fig. <ref> with the parameter set shown in Fig. <ref>.
Then at T_0<T<T_×, the adiabatic approximation is valid again, and we have the present QCD axion energy density 

    ρ_a,0=12 m_a,0 m_𝒩,1 F_a^2 θ_1,𝒩^2 (a_1,𝒩a_0)^3   .

Note that we only show two examples above to estimate the relic density of the Z_𝒩 axion and QCD axion DM in the Cases I and II, respectively, one can also calculate others in a similar way. 
   
Conclusion.—
In summary, we have investigated the mass mixing between the QCD axion and Z_𝒩 axion, and introduced a novel level crossing.
In the mixing, we find that the level crossing can take place at the QCD phase transition critical temperature T_ QCD (Case I) or slightly before T_ QCD (Case II), depending on the ratio of the axion decay constants ζ∼1.69.
The conditions for level crossing to occur in the Cases I and II are discussed in detail.
In the Case I, the zero-temperature QCD axion mass m_a,0 should be smaller than the mass m_𝒩,π and larger than the zero-temperature Z_𝒩 axion mass m_𝒩,0, which can be characterized as 1.24×0.48^𝒩/2𝒩^3/4<ζ<1.69.
While in the Case II, the zero-temperature QCD axion mass m_a,0 should be larger than m_𝒩,π and the ratio should be ζ<𝒩, which can also be characterized as 1.69<ζ<𝒩.
Note that in this case we also have a relatively weak constraint ζ<1.69  γ^-b, which will be stringent for the large value of γ.

In these two cases, the cosmological evolution of the mass eigenvalues is similar, however, the transition of energy density between the axions is different.
In the Case I, the axion energy transition at the level crossing is non-adiabatic (T_ QCD), while in the Case II it is considered to be adiabatic (T_×).
Finally, we briefly estimate the relic density of the axion DM through the misalignment mechanism, focusing on the Z_𝒩 axion and QCD axion DM in the Cases I and II, respectively.
The level crossing in this scenario may also have some other interesting cosmological implications.

Acknowledgments.—
This work was supported by the CAS Project for Young Scientists in Basic Research YSBR-006, the National Key R&D Program of China (Grant No. 2017YFA0402204), and the National Natural Science Foundation of China (Grants No. 11821505, No. 11825506, and No. 12047503).




 Paper Title: MeerKLASS L-band deep-field intensity maps: entering the HI dominated regime.




firstpage–lastpage
Weak ℤ_2 Supertopology
    Andreas P. Schnyder
    August 5, 2024
=======================




§ ABSTRACT 
We present results from MeerKAT single-dish  intensity maps, the final observations to be performed in L-band in the MeerKAT Large Area Synoptic Survey (MeerKLASS) campaign. The observations represent the deepest single-dish  intensity maps to date, produced from 41 repeated scans over 236 ^2, providing 62 hours of observational data for each of the 64 dishes before flagging. By introducing an iterative self-calibration process, the estimated thermal noise of the reconstructed maps is limited to ∼ 1.21 mK (1.2 × the theoretical noise level). This thermal noise will be sub-dominant relative to the  fluctuations on large scales (k ≲ 0.15 h Mpc^-1), which demands upgrades to power spectrum analysis techniques, particularly for covariance estimation. In this work, we present the improved MeerKLASS analysis pipeline, validating it on both a suite of mock simulations and a small sample of overlapping spectroscopic galaxies from the Galaxy And Mass Assembly (GAMA) survey. Despite only overlapping with ∼ 25% of the MeerKLASS deep field, and a conservative approach to covariance estimation, we still obtain a > 4 σ detection of the cross-power spectrum between the intensity maps and the 2269 galaxies at the narrow redshift range 0.39 < z < 0.46. We briefly discuss the  auto-power spectrum from this data, the detection of which will be the focus of follow-up work. For the first time with MeerKAT single-dish intensity maps, we also present evidence of  emission from stacking the maps onto the positions of the GAMA galaxies. 






cosmology: large scale structure of Universe – cosmology: observations – radio lines: general – methods: data analysis – methods: statistical








§ INTRODUCTION


Probing density fluctuations in large-scale cosmic structure is a cornerstone of precision cosmology. Constructing maps that span wide areas and reach deep redshifts provides a rich resource for testing and constraining the standard
ΛCDM cosmological model. Surveys at optical and near-infrared wavelengths have made impressive progress in this pursuit <cit.>. However, extending experiments to a broader range of wavelengths boosts the quantity of available data. Furthermore, multi-wavelength observations provide robust checks of systematic influence due to the alternative instrumental approach required to survey other wavelengths. There can also be vastly different complex astrophysics for tracers at different wavelengths. Thus alternative tracers can validate the consistency of cosmological conclusions.

Of particular interest for surveying large-scale structure is mapping redshifted neutral hydrogen () gas at 21cm radio wavelengths. In the post-reionisation Universe, at redshifts z ≲ 6, the remaining  is overwhelmingly contained within dark matter halos <cit.>, meaning the 21cm radiation spontaneously emitted from  acts as a tracer of the large-scale structure. However, at the redshifts of interest for large-scale structure cosmology (z ≳ 0.1), the 21cm emission becomes faint and hard to detect with the high signal-to-noise required for a comprehensive mapping survey. This problem can be circumvented by abandoning the requirement to resolve individual galaxies and instead integrate the combined unresolved emission with low-aperture radio telescopes. This process is now commonly known as  intensity mapping <cit.>.

The novel approach of intensity mapping presents some challenges. Since it records all radiation entering the telescope receiver within its frequency bandpass, the contribution to the signal will not come from cosmological  alone; in fact, only a small percentage is. There are multiple sources of foregrounds in the frequency ranges corresponding to redshifted 21cm emission. These mostly originate from our Galaxy and come from diffuse synchrotron and free-free emission, but there can also be strong extragalactic point sources <cit.>. These sources dominate the cosmological  brightness temperature by three to five orders of magnitude. Furthermore, instrumental effects can distort the foreground spectra, making their removal additionally challenging <cit.>. In addition to foregrounds, the telescopes will detect terrestrial, human-made radio frequency interference (RFI) <cit.>. This can dominate even more than the foregrounds, but vary with time and sky position so can therefore be flagged and deleted from the final maps. Residual RFI will also average down if repeated scans are performed over the same patch of sky, as is the case with MeerKLASS observations. 

Despite foreground contamination and RFI challenges, the potential benefits from  intensity mapping are too large to ignore, thus justifying the ongoing endeavour to develop the technique into a tool for precision cosmology. Experiments have already delivered promising results, demonstrating how detections are possible even without purpose-built instruments <cit.>. New experiments are starting to arrive <cit.>, many designed specifically with  intensity mapping in mind, creating a bright future ahead.

The MeerKAT telescope,[https://www.sarao.ac.za/science/meerkat/] an array of 64 dishes in the Upper Karoo region of South Africa, has already showcased its capability to perform  intensity mapping both as an interferometer, claiming the first detection of the  power spectrum <cit.>, and also in single-dish mode, where the field-of-view can reach cosmological scales. Using the single-dish pilot survey <cit.>, a detection of the cross-correlation power spectrum was achieved with overlapping WiggleZ galaxies <cit.>.

The MeerKAT Large Area Synoptic Survey (MeerKLASS) <cit.> will continue adding to these observations until it is merged with the SKA Observatory (SKAO) <cit.> on the same site.[https://www.skao.int/en] MeerKAT can observe in two frequency bands relevant for 21cm-cosmology: L-band (900 MHz < ν < 1420 MHz, reaching redshifts z < 0.58) and UHF-band (580 MHz < ν < 1000 MHz, 1.45 > z > 0.4).

In this work, we present results from MeerKAT single-dish observations in the MeerKLASS L-band deep field, a set of observations performed over a single 236 ^2 patch with 41 repeated scans for each of the 64 dishes. Integrating over the sky in this way will average down time-varying contributions from thermal noise and RFI, thus boosting the signal-to-noise of time-constant sky signals, which includes the foregrounds and the cosmological . Previous analysis <cit.> exploited the fact that the foreground cleaned intensity maps were thermal noise dominated when estimating error bars on the power spectrum measurements. For this MeerKLASS deep-field and future MeerKLASS observations, it can no longer be assumed that the thermal noise will be dominant, therefore demanding more emphasis on covariance estimation. We therefore explore using the scatter in the foreground transfer function <cit.>, primarily used to correct for signal loss from foreground cleaning, as a measure of the covariance in measurements. The transfer function is constructed using mock signal injection into the real data, hence its scatter should encapsulate cosmic variance, thermal noise fluctuations, residual RFI contributions, foreground residuals and signal loss correction uncertainties.

We validate the precision and accuracy of the analysis pipeline to ensure estimator accuracy is within the statistical error associated with these MeerKLASS observations. We formalise the main steps in our pipeline, such as beam reconvolution, foreground cleaning and sampling to a Cartesian grid, presenting validation steps on a suite of simulations which emulate the MeerKLASS L-band deep-field observations. We also use a small sample of overlapping spectroscopic galaxies from the Galaxy And Mass Assembly (GAMA) <cit.> survey to study the cross-power spectrum and its covariance under different analysis choices.

The paper is outlined as follows: sec:ObsData outlines the observations for the MeerKLASS deep field, our new calibration strategy (with an iterative self-calibration process) and overlapping GAMA galaxies. sec:Mocks presents the methods for generating our suite of mock data which emulate the observations. sec:FGcleaning provides details of our foreground cleaning technique. sec:PkEst focuses on the power spectra, formalising our estimation and modelling process, and presents the results. sec:stack shows the results from stacking the MeerKLASS deep-field maps onto the locations of GAMA galaxies and discusses the process and complications that come with this. We then finally conclude in sec:conclusion.



§ OBSERVATIONAL DATA




 §.§ MeerKLASS L-band deep-field


Using the same scan strategy as the MeerKLASS L-band pilot survey in <cit.>, a series of observations were conducted using the MeerKAT telescope between September 1st and December 29th 2021. 
Observations with single dishes typically require a scanning strategy where the dishes are rapidly moved across the sky. The MeerKAT antennas were set to scan in azimuth at constant elevation to minimise fluctuations of ground spill and airmass. To make sure the projected speed on the sky is 5 arcmin s^-1, the telescope scan speed was set to 5/cos(el) arcmin s^-1 (where el is the elevation angle) along the azimuth. This speed also ensures that the telescope pointing does not move significantly compared to the width of the primary beam (∼ 1deg) during a single time dump (2 s).

With a time resolution of 2 s, this gives a scan speed of no more than 10 arcmin per time sample, which is well within the beam size. Noise diodes attached to each receiver were fired for 0.585 s once every 19.5 s during the observation to provide a relative time-ordered data (TOD) calibration reference. The dishes are moved back and forth with a slew of about 10 deg sky projection in each direction, corresponding to an observing time of about 120 s per stripe. The duration of each set of scans with 48 strips is about 100 minutes, and before and after each scan we spent  ∼ 8 minutes tracking a nearby celestial point source to use as a bandpass calibrator and absolute flux calibrator.

At fixed elevation, two scans can be performed per night, corresponding to when the field is rising and setting respectively. The two scans will cross each other to achieve good sky coverage in the region of overlap.
There were 41 observations in total (each one referred as a block), the details of each are listed in the Appendix in <ref>. The observations repeated scans covering a patch of sky approximately 236 ^2 in the southern sky, spanning in R.A. (330^∘,360^∘) and Dec. (-36^∘, -25^∘). The 62 hours obtained for each of the 64 dishes before flagging, provide a cumulative 3968 hours of observations,[This number is reduced for the final useable data since some observation blocks were entirely flagged and some dishes failed in observation. We provide details of this later in this section.] likely to be the deepest single-dish intensity maps, in terms of observational hours per square degree, in the MeerKLASS campaign.



 §.§ Standard MeerKLASS calibration


 

We employ the MeerKAT single-dish calibration pipeline KATcali


to calibrate the deep-field observations and create the combined 3D data cubes. KATcali includes several cycles of RFI flagging, calibration, and map-making. The details of the calibration are thoroughly described in <cit.>. Here, we provide a brief overview of the main points. We take a two-step approach to the calibration process. The first step is to use the flux model of a strong point source during tracking observations to calibrate the bandpass and absolute flux, as well as the injection power of the noise diode. The second step then uses the noise diode injections to calibrate the TOD during the constant-elevation scanning phase.

To calibrate the scanning data, we build the model as 


    T_ model(t, ν)= T_ diffuse (t, ν)+ T_ el (t, ν)
        + T_ diode(t, ν)  + T_ rec(t, ν) ,


where T_ el is the antenna temperature model of elevation-dependent terrestrial emission (atmosphere and ground spill),  T_ diode is the noise diode contribution, T_ rec is the receiver temperature, while 
the diffuse celestial components T_ diffuse includes Galactic emission and the cosmic microwave background (CMB), 


    T_ diffuse(t,ν)= T_ Gal(t, ν) +T_ CMB ,


where the CMB emission is set to be a uniform T_ CMB=2.725K <cit.>, and the Galactic emission T_ Gal (t, ν) is derived from PySM
() that is based on the reprocessed Haslam 408 MHz total intensity map (). 

To compare with the raw data from the correlator, we multiply the signal model by the gain, g(t, ν), so that


    T̂_ model(t, ν)= g(t, ν)  T_ model(t, ν) ,


where the hat indicates a temperature in the (arbitrary) correlator units. 

The signal and gain models are then fitted (with a Bayesian framework; see more details in Section 3.3 of ) to the TOD for each polarisation, frequency channel, dish, and observation scan, all of which are treated independently. Using the gain solution, we can then obtain the calibrated temperature,


    T_ cal(t,ν)≡T̂_ raw(t, ν)/g(t, ν) ,


where T̂_ raw is the raw data collected by MeerKAT.

The residual temperature difference between the signal model and calibrated data is then


    T_ res(t,ν)≡ T_ cal(t,ν) - T_ model(t, ν) .


In standard calibration, we assumed that T_ res is noise-dominated during the fitting, ignoring the faint point sources in the target field.  This was a fair assumption when the noise level was relatively high <cit.>. In this work, we seek a better calibration approach by implementing a self-calibration method. 






 §.§ Self-calibration based on the MeerKLASS sky map




The higher signal-to-noise from the MeerKLASS deep-field data may allow us to deal more carefully with the point sources, which were ignored in the standard calibration, to achieve a better calibration result. Additionally, we aim to use a more accurate sky model to replace the T_ Gal model that is based on the Haslam map. 
The 408 MHz radio continuum all-sky map, which has been widely used to study diffuse synchrotron radiation from our Galaxy and works as a Galactic foreground model for cosmological research, combines data from four different surveys that happened more than 40 years ago (; ). The instrument performance in the 1980s limited the data quality, especially for the Galactic substructures and the zero level of the maps <cit.>. The selection of the spectral index map, used to generate Galactic maps at MeerKAT frequencies, is also a matter of debate <cit.>. 

We therefore extend the calibration process described in sec:StandardCalibration and test the behaviour of the results by introducing an iterative self-calibration process with several loops.
This includes 

(1) replacing T_ diffuse (t, ν) with  T_ sky (t, ν), obtained from the previous Level-6 T_ sky result, when the corresponding Level-6 pixel has a hit-count>40 (which ensures that T_ sky values are obtained from high signal-to-noise data); 

(2) setting the time-order weight w(t_i)=1 for all the scan time samples; instead of setting a higher weight for the diode (main calibrator during the standard calibration) injection samples (see more details in Section 3.3 of ).
Thus <ref> becomes 


    T^ upd_ model(t, ν)= T_ sky (t, ν)+ T_ el (t, ν)
        + T_ diode(t, ν)  + T_ rec(t, ν) .


In this way, we have all components (especially point sources and Galactic substructures) included in the input “model”. The difference between the data and our calibrated sky data is


    T^ upd_ res(t,ν)≡ T_ cal(t,ν) - T^ upd_ model(t, ν) ,


which should be noise-dominated compared to that in <ref>. The self-calibration process is iterative for five loops (see <ref>), in which the T_ sky (t, ν) always uses the Level-6 result of the previous calibration loop. Each iteration refines the calibration, reducing the discrepancies between the observed data and the model. We stopped the self-calibration iteration after five loops because the standard deviation of T_ res converged to a stable value, as shown in <ref>.












 §.§ Characteristics of final MeerKLASS data cubes




  §.§.§ Final data cubes


The combined calibrated temperature cubes (with a pixel size of 0.3 deg and frequency resolution of 0.2 MHz) are derived from all available scans and dishes. However, predominantly due to RFI, an identified repeated source of which we discuss in the following sub-section, only 27 of the 41 observation blocks yielded well-calibrated results. We mark them in the last column of <ref>. In <ref> we present the number count of good (flagged time <50%) channels within the 971–1075 MHz band (channels 550–1050) for each block and each dish. In particular, we found that antenna m059 has a higher chance of surviving calibration even when all other dishes fail.[For consistency with other MeerKAT work, we indicate MeerKAT's 64 antennae with this notation spanning m000-m063.] This may be because m059 is the easternmost dish of MeerKAT, located far from the array core, providing it with a more isolated environment.

We show the final sky temperature map from the combination of the remaining 27 scans after 5 loops of self-calibration at 1023 MHz in the middle panel of <ref>. For comparison, the top panel shows the PySM diffuse Galactic emission model with the same sky coverage. We also show in the bottom panel the total number of time samples (across all dishes and observations) that were combined in each pixel.

To demonstrate the improvement in data quality from the deep-field intensity maps, <ref> plots the amplitude of each pixel along the line-of-sight as a function of frequency. We compare this to the previous MeerKLASS L-band pilot survey (top-panel) relative to the deep field used for this work. The maps should be dominated by the continuum synchrotron spectra, thus their smoothness is a rough proxy for how dominant any systematics are. There are numerous perturbations in the pilot survey spectra that have consequences for the effectiveness of foreground cleaning, required to access the cosmological . However, for the deep field, the spectra appear smoother and well-calibrated. In addition to the improvements shown in <ref>, we also find a marginal performance improvement on the line-of-sight rms when using the self-calibration technique relative to the previous standard calibration in the deep-field. The average variance along the line-of-sight of each pixel decreases to 977.2 mK^2 for the self-calibration, down from 984.3 mK^2 for the standard calibration. 













  §.§.§  RFI contaminations

From the 41 observation blocks, we had to remove 14 blocks due to excessive, low-level RFI contaminations. In affected blocks, these contaminations were sometimes recognisable on the map level of individual dish data as horizontal stripes. After careful investigations, we have now identified the source of the RFI contamination as the downlink of the mobile communication tower (∼ 54 km away) that enters through the sidelobes and causes a non-linear gain effect in the entire band. Our aggressive flagging approach in this study, with the entire removal of 14 blocks, was to ensure high data quality for our analysis. We are currently developing a method to correct for this effect, which should allow us to use the currently flagged data for future projects.




  §.§.§ Noise level estimation


Here we estimate the noise levels from the final maps to check if they are consistent with theoretical expectations. We use the final data cube, T_ sky, to estimate the noise level using the difference between four neighbouring channels (the so-called ABBA technique). By adopting the ABBA pattern, this difference statistic removes both constant and linearly varying signals to cancel signals detected by the receiver, leaving only the noise.  The noise level is estimated by


    Δ T_i(ν^⋆)= 1/2( T^i_ sky(ν) + T^i_ sky(ν+δν)  ) 
                     -1/2 ( T^i_ sky(ν-δν)+T^i_ sky(ν+2δν)),


where the index i goes over all pixels in the map and ^⋆ means the combined result from the four channels [ν-δν, ν, ν+δν, ν+2δν]. 
From the observation data, we take a weighted rms,


    Δ T_ RMS^2(ν^⋆) = 
         N_pix/N_pix - 1 (∑_i w_iΔ T^2_i/∑_i w_i -
         [∑_i w_iΔ T_i/∑_iw_i]^2 ),


where the sum is over the number of pixels in the map, N_pix, and we have suppressed the dependence of w_i and Δ T_i on ν^⋆. For the weight, w_i, we used the theoretical expected variance itself, w_i(ν^⋆)=1/σ^2_ th,i(ν^⋆).
The corresponding variance in each pixel would be,


    σ^2_ th,i(ν^⋆)=1/4(σ^2_ th,i(ν)+σ^2_ th,i(ν+δν)+σ^2_ th,i(ν-δν)
                        + σ^2_ th,i(ν+2δν) ).


Here the noise in the initial polarised TOD is given by the radiometer equation 


    σ^2_ th,pol(t,ν) = T^2_ sys(t,ν)/δν δ t,


where δν=0.209 MHz is the frequency width of the MeerKAT L-band channels and δ t=2 s is the length of each time stamp recording. 
For the system temperature we use the calibrated data itself, T_ sys = T_ cal,HH(t,ν) or T_ cal,VV(t,ν)  for horizontal and vertical polarisation data, respectively. 
Thus the noise in the initial intensity TOD is


    σ^2_ th(t,ν) = T^2_ cal,HH(t,ν)+T^2_ cal,VV(t,ν)/4 δν δ t.


In order to get the final data cube, T^i_ sky(ν), the data goes through several stages of averaging. We propagate the variance taking into account this averaging in order to get to the final σ^2_ th,i(ν).

The theoretical rms which can be compared to Δ T_ RMS (ν^⋆) in <ref>, is then simply


    σ^2_ th (ν^⋆) = N_pix ( ∑_i σ^-2_ th,i(ν^⋆)   )^-1.


More details can be found in Section 5.6 of <cit.>. 


 

In the upper panel of <ref>, we show the noise levels (from data and theory) in the 971–1075 MHz frequency band (channels 550–1050). Note here both the real and theory values are related to the real T_ sys and hit counts in corresponding map pixels. The count reduction due to RFI flagging, and unflagged low-level RFIs that are hiding in the data would cause wiggles. Four groups of spikes are present at higher frequencies, while below 1023 MHz, there is much less scatter compared to the previous MeerKLASS pilot survey result. We therefore chose to restrain our cosmological analysis to the 971–1023 MHz region, for which we show the noise level in the middle panel, finding a median of Δ T_ RMS=1.21mK. In the lower panel of <ref>, we also show the histogram of the ratio 
R_ RMS=Δ T_ RMS (ν^⋆) / σ_ th (ν^⋆) in the 971–1023 MHz band. We find a median value of R_ RMS=1.2, while some channels are noisy and R_ RMS can reach 1.5. 





 §.§ GAMA spectroscopic galaxies


Overlapping with the MeerKLASS deep field, is a catalogue of spectroscopic galaxies recorded in the Galaxy and Mass Assembly (GAMA) survey. GAMA began observing in February 2008 <cit.>. The survey aimed to construct catalogues of galaxies to test the cold dark matter paradigm and probe galaxy evolution <cit.>. For our purposes, we exploit the fact that these galaxies will trace dark matter and should therefore correlate with the overlapping MeerKLASS intensity maps after foreground cleaning.  

We use the fourth and final data release from the Galaxy and Mass Assembly survey, GAMA DR4 <cit.>. This includes galaxies from the GAMA II survey phase which enlarged the GAMA I equatorial survey, by including extra survey regions. One of these extra regions was the 23 hr (G23) field, which completely overlaps in area with the MeerKLASS L-band deep field, shown by the blue scatter points in the bottom panel of <ref>, which represent the GAMA G23 galaxies relative to the observational time footprint map for the L-band deep field. The G23 field covered 339  < R.A. < 351 and -35  < Dec. < -30. The magnitude limit of the field was reduced from the planned 19.8 in r-band to 19.2 in i-band due to limited observational time, but this allowed the GAMA II survey to obtain a high 94.2% redshift completion in the G23 region  <cit.>.

The GAMA DR2 only provides random catalogues that emulate the survey selection function of the real galaxies for the G09, G12, and G15 regions, (not the G23 used in this work). Ideally, these would be used to construct a survey completion window function, which would be used in the cross-power spectrum estimation. In the absence of this, we assume a uniform survey completion across the G23 area and a constant distribution with redshift, given the very flat distribution for the redshift range used in this analysis (green region of <ref>). Given our signal-to-noise levels, survey masks will be a sub-dominant effect in the cross-power, thus this is an adequate approximation to make.

Due to our aggressive flagging strategy on the MeerKLASS L-band data, the overlap region in redshift is restricted to 0.39 < z < 0.46. As shown by <ref>, this catches the GAMA survey galaxy redshift distribution, N_g(z), near the end of its optimal range, but still provides 2269 galaxies to cross-correlate with. At this redshift range, the comoving (volumetric) number density of the GAMA galaxies is n̅_g = 4.8 × 10^-4 h^3 Mpc^-3.









§ SIMULATION OF MOCK DATA


To accompany the calibrated observations outlined in sec:ObsData, we produce a suite of 500 mock simulations, used for validating the analysis pipeline, calculating signal loss from foreground cleaning and estimating covariance in the power spectrum measurements. To create this large number of mocks we generate lognormal density fields <cit.> from which a map of  brightness temperatures can be cut, and galaxies Poisson sampled onto for the spectroscopic galaxy counterpart.

An important stage of the analysis pipeline is regridding the intensity mapping voxels with celestial sky coordinates (R.A., Dec., ν) onto a regular 3D Cartesian grid with lengths l_x, l_y, l_z in h^-1Mpc, on which a Fast Fourier Transform (FFT) can be applied for the power spectrum estimation <cit.>. We therefore require the mocks to begin in celestial coordinates so we can validate the regridding process. To achieve this, we begin by fitting a Cartesian cube around the full MeerKLASS deep-field footprint, creating a grid with dimensions l_x,l_y,l_z = 774.9,335.7,252.2 h^-1Mpc over n_cell = 512^3 cells. We use a large number of cells for this initial grid to approximately emulate a continuous sky, aiming to avoid the simulated maps resolving the finite input grid structure. Onto this grid, we simulate a lognormal density field. This is done by interpolating k-modes from a model power spectrum onto the FFT grid. The  model power spectrum is given by


    P_(k)≡ P_(k,μ) = T_^2 b_^2 (1 + f μ^2/b_)^2 P_m(k)/1 + (k μ σ_v/H_0)^2 .


The anisotropy in this model makes it convenient to parameterise k using μ, defined as the cosine of the angle between the line-of-sight and k. The  parameters are the mean  gas temperature (T_) and the  bias (b_). f is the growth rate of structure and is included to model linear redshift space distortions (RSD) which enhance clustering across all scales. The denominator is included to approximately model the non-linear RSD; the so-called Fingers-of-God (FoG), which suppresses clustering at small scales. The FoG are modulated by σ_v, which is the velocity dispersion of the tracer, in this case, the  gas. All of these quantities have a redshift dependence, which we omit for brevity; they will also have little variation given our narrow redshift range. Lastly, H_0 is the Hubble parameter at z = 0, known as the Hubble constant. An alternative process for generating brightness temperatures is pursued for our stacking analysis discussed later in the paper.
More realistic lognormal line intensity mapping mocks are available. For example, in future work, we intend to adopt the approach in <cit.> where luminosities from the spectral lines are painted given a luminosity function. This has been shown to reproduce better RSD for galaxy distributions and models the whole temperature distribution expected in the maps.

Inverse Fourier transforming the model in <ref> provides the  correlation function ξ_ which is then log-transformed via ξ^'_ = ln(1 + ξ_). Fourier transforming ξ^'_ back gives the modified lognormal power spectrum P^'_. Gaussian random numbers are then sampled into each cell of the Fourier grid δ_G(k) with variance σ^2(k) ∝ P^'_(k). This Gaussian random field is then inverse Fourier transformed to real space giving δ_G(x) from which the lognormal  brightness temperature fluctuation field is given by


    δ T_(x) = exp[δ_G(x) - σ^2_G/2] - 1 ,


where σ_G^2 is the measured variance of the Gaussian random number field δ_G. Further details on this process can be found in the Appendix A of <cit.>. For the galaxy over-density field δ_g(x), we follow the same procedure except the input model is for the galaxy power spectrum, given by


    P_g(k)≡ P_g(k,μ) = b_g^2 (1 + f μ^2/b_g)^2 P_m(k)/1 + (k μ σ_v/H_0)^2 ,


where b_g is the bias for the galaxies. We assume that the velocity dispersion σ_v for the  gas and galaxies is the same for simplicity. This will be sufficient for this work where signal-to-noise is low at the small scales where σ_v has the main effect.

The galaxy over-density δ_g is unit-less whereas the intensity maps δ T_ are brightness temperature fluctuations measured in mK. We use the same seed in the random number generation to ensure that both the δ T_ and δ_g correlate. The lognormal code we adopt utilises some functions from the publically available package  <cit.>.[https://github.com/steven-murray/powerbox]



 §.§ Transforming between Cartesian grids and sky maps


As already mentioned, the galaxies and the intensity mapping voxels from observations are in celestial coordinates at a certain frequency (or redshift) i.e. (θ,ν) ≡ (θ_R.A.,θ_Dec.,ν). However, for the final power spectrum estimation, we require the fields be in a Cartesian space with dimensions (l_x,l_y,l_z) in h^-1Mpc. For this we follow the recent work in <cit.> which we discuss further in sec:PkEst.

Creating lognormal mocks in celestial coordinates that emulate observations requires the inverse of this step i.e. δ T_(x) → δ T_(θ,ν), since the lognormal fields are generated onto Cartesian grids measured in h^-1Mpc. Transforming mock galaxy coordinates (the Poisson sampling of which is discussed in sec:Galmocks) from (l_x,l_y,l_z) [h^-1Mpc] → (θ_R.A.,θ_Dec.,ν) is trivial. We can use [https://docs.astropy.org/en/stable/index.html] <cit.> transformation routines on each galaxy's coordinate. However, transforming the tessellating field of Cartesian cells, each with an associated brightness temperature, into a gap-less map of voxels with identical celestial coordinates and frequency binning as the real MeerKAT maps, requires a resampling of the field. For this we also utilise the techniques and public code[https://github.com/stevecunnington/gridimp] outlined <cit.>. 

This process involves a Monte-Carlo particle sampling technique where each Cartesian cell is filled with N_p particles, randomly distributed within the cell. The particles are assigned the brightness temperature of the cell they fall within. The particles are then transformed to celestial coordinates (θ_R.A.,θ_Dec.,z) and added into the map voxels associated with their celestial coordinate and the frequency channel corresponding to their redshift. Each voxel is normalised by the number of particles falling within it. Providing enough sampling particles per cell are used (in this work we use N_p = 10) and the Cartesian grid is of sufficient resolution (hence our choice of n_cell = 512^3), it will provide an intensity map in celestial sky coordinates, that traces the fluctuations generated by the lognormal field. 

Throughout this work, we do not require the necessary extensions of higher-order particle assignment or field interlacing to suppress the effects of aliasing <cit.>, since these effects are below the current statistical noise from this data. We are however, still required to implement a correction to the power spectrum estimation from the Nearest-Grid-Point assignment which smooths the field, which we formalise in sec:PkEst.



 §.§  intensity map mocks


Using the regridding process described in the previous section we are provided with  intensity mapping mocks δ T_(θ,ν), with identical coordinates and dimensions as the observed MeerKAT intensity maps. From here we can add in observational effects and simulate various systematics. The most dominant effect comes from foreground contamination. Systematics will have a large influence on foreground contamination, and this can be extremely complex. Therefore, this is not something we attempt to directly simulate in this work since we risk under-estimating the problem. Instead, we use the observational data itself which has the relevant systematics contained within, albeit only one realisation of them. However, adding these directly onto different realisations of  mocks, still recreates stochasticity in the  signal's response to systematics. We discuss this further in sec:FGcleaning.

Another important addition to the mocks will be a smoothing to the intensity maps corresponding to the angular pixelisation. This is necessary to emulate the radio telescope's beam, which for single-dish intensity mapping is quite broad giving a low angular resolution to the maps and damping small-scale fluctuations perpendicular to the line-of-sight. In this work, we assume the MeerKAT beam profile to be Gaussian, which is a simplification but still valid given the signal-to-noise at small scales where the beam suppresses signal below the noise. We will investigate implementing a more realistic MeerKAT beam profile in future work <cit.>. The assumed Gaussian beam profile has a frequency (ν) dependent standard deviation in radians given by


    σ_beam(ν) = 1/2√(2ln 2)c/ν D_dish ,


where we use D_dish = 13.5 m for the diameter of the MeerKAT dishes and c is the speed of light. To emulate the beam effects we smooth the mock maps by creating a 2D Gaussian array kernel centred on the median angular pixel θ_0 given by


    B_beam(θ,ν_i) = exp[-1/2(θ - θ_0/σ_beam(ν_i))^2] .


This is normalised such that ∑_θB_beam=1, so that convolution with this beam kernel does not change the total brightness temperatures within the maps. The  intensity map mocks are convolved with <ref> at each frequency ν_i to give the beam-smoothed  map;


    δ T_⊛(θ,ν_i) = δ T_(θ,ν_i)⊛B_beam(θ,ν_i) ,


where the ⊛ symbol represents a convolution.


To realistically emulate additional components such as foregrounds, noise and their response to systematics, we add the real observations to each mock iteration. This is to avoid the requirement of simulating the complex and poorly understood systematics. 

In some cases, mostly for studying the  auto-correlation, it will be useful to simulate just the thermal noise originating from the thermal signature of the receivers.

We do this by generating Gaussian random noise the power of which can then be measured with the same pipeline as the rest of the analysis, providing a robust estimate of the thermal noise floor. We begin with an analytical description of the thermal noise fluctuations per pixel using the radiometer equation (as in sec:ObsData but now at map-level)


    σ_N(θ,ν) = T_sys(ν)/√(2 δν t_p(θ,ν)) ,


where t_p is the summed observation time contained in each voxel, which can be taken directly from the time stamp counts as shown in <ref>, with each count representing δ t =2sec. This emulates the real data by creating a non-uniform noise map, generating higher noise around the edges where scan coverage is lower. For the system temperature, we follow <cit.> and use a model tuned to coincide with the estimated system temperature of the MeerKAT telescope <cit.>[See https://skaafrica.atlassian.net/wiki/x/AYCHE], assuming 


    T_sys(ν)=T_rx(ν)+T_spl+T_CMB+T_gal(ν) ,


where we assume the contribution from our own galaxy is T_gal=25K (408 MHz / ν)^2.75, the CMB temperature is T_CMB=2.725 K, as in sec:ObsData. The spill-over temperature is assumed as T_spl=3 K and the receiver temperature we model as


    T_rx(ν) = 7.5K+10K(ν/GHz-0.75)^2 .


Due to the narrow frequency band we use, this gives a near-constant system temperature with the range 15.97<T_sys<16.02 [K]. Gaussian random numbers can then be generated across the MeerKLASS deep-field map with the voxel dependent rms of σ_N(θ,ν).



 §.§ Galaxy mock catalogues


To accompany the mock intensity maps, we also require corresponding galaxy mocks with the same correlated clustering.  Rather than transforming the Cartesian galaxy over-density field δ_g(x) into sky coordinates, it is more realistic to instead Poisson sample the field to generate a catalogue of galaxies with the same number density as the GAMA G23 survey. This also allows additional validity checks such as testing the stacking pipeline which cannot be done without a catalogue of galaxy coordinates. 

We generate the galaxy catalogue coordinates initially in the Cartesian domain to exploit the high n_cell = 512^3 resolution. We first calculate the average galaxy counts in cells ⟨n_g⟩ for the real GAMA G23 region in the redshift range where there is overlap with the intensity map frequency range. We then take the galaxy survey selection window function, which for the GAMA G23 region, we have to assume is constant between 339 < R.A. < 351 and -35 < Dec. < -30 (see discussion in sec:GAMAobs). We then transform this window function into the high resolution Cartesian space. A galaxy number counts field n_g(x) is then given by


    n_g(x) =  ⟨ n_g⟩(δ_g(x) + 1) W_g(x) ,


where we use the transformed window function W_g(x) and the real GAMA average counts ⟨ n_g⟩ to ensure the mock galaxy counts will share the same survey selection as the real galaxy survey, with the same number density. In this process we normalise the window function W_g such that its maximum value is 1, so for the GAMA G23 survey, this becomes a binary mask of 0 and 1. We then Poisson sample the n_g field to introduce some stochasticity which we expect to be present in the real galaxy catalogues. This provides an integer value in each Cartesian cell for the galaxy counts. The last step is to assign random Cartesian coordinates within the cell boundaries for the n_g(x) galaxies within each cell. At this stage we rely on the Cartesian field having sufficiently high resolution such that the random assignment of a coordinate within the cell does not shift the galaxies away from the peaks of the density fluctuations. The catalogue of galaxy coordinates (x, y, z [h^-1Mpc]) can then be transformed into celestial coordinates (θ_R.A.,θ_Dec.,z). 


In this work, we generate 500 mock intensity maps with corresponding mock GAMA galaxy catalogues. As some early validation of these mocks, we show in <ref> how the power spectra for both the  intensity map and galaxy auto-correlations, along with their cross-correlation, agree with the fiducial model (black dashed line). This demonstrates that our estimator is unbiased and processes such as the regridding are performing to within an acceptable tolerance. The power spectrum estimation process used here on the mocks is identical to that used on the real data and is detailed later in sec:PkEst, along with the modelling.



The mocks used in the test of <ref> do not include any additional systematics beyond the telescope beam in the intensity maps. We will investigate the inclusion of foregrounds and systematics in sec:FGcleaning. The large scatter shown by the shaded blue region, which represents the 1 σ range across the 500 mocks, is therefore driven by cosmic variance due to the small volume covered by the data. We see that the errors for the galaxy power spectrum appear larger than the  intensity maps. This is because we have not introduced any thermal noise (or other observational or systematic effects) into the intensity maps. However, the galaxy catalogues will have shot noise contributing to the uncertainties. This nicely showcases the potential for  intensity mapping if noise and systematics can be controlled, however, this is not the purpose of this exercise so is not a point we will labour. There are also additional caveats when considering high-density tracers such as . As shown by <cit.>, despite the low shot noise, stochastic noise and non-linear bias in the  field can limit the potential of constraints inferred from the power spectrum.

The green dashed lines in the residual panels mark the regions where there is > 90% accuracy relative to the model which we can see is achieved at all scales for all three cases. This is also well within the statistical uncertainty (blue-shaded regions). The largest residual comes in the cross-correlation where discrepancies seem to increase with k nearly reaching the -10% boundary. We speculate that this may be caused by the Poisson sampling of mock galaxies followed by random assignment of coordinates within the cell. This extra stochasticity is not repeated in the  intensity map mocks and could therefore be generating a pseudo-cross-correlation coefficient, which would depart from r = 1 at smaller scales (higher-k). However, this appears to be a minor effect. Furthermore, this could be seen as a feature, since we expect r < 1 in the real observations, which will also have a scale dependence. For these reasons, we deem this to be a suitable result for the purposes of this work, and defer a higher precision investigation and validation to future work.  



§ FOREGROUND CLEANING




Foreground cleaning  intensity maps is a well-studied problem <cit.>, and generally blind cleaning techniques are the currently preferred method for single-dish experiments. As their name suggests, blind methods do not rely on prior knowledge of the foreground structure. This is advantageous for current experiments because the foregrounds at these frequencies lack observational constraints. Furthermore, blind techniques avoid placing overwhelmingly stringent calibration requirements on the data, since the dominant foregrounds require beyond sub-percent precision modelling for a successful subtraction. Blind techniques instead work by exploiting the foreground dominance and their frequency-correlated structure to assume that they can be reduced to a distinct set of correlated modes separated from the oscillating, sub-dominant fluctuations which will contain most of the  information.

We outline the formalism for our blind foreground cleaning approach in sec:PCAcleaning, but first we discuss two pre-processing steps before foreground cleaning. These are designed to improve the foreground clean by reducing the impact from residual RFI or systematics, plus any known frequency-dependent observational effects. 



 §.§ Reconvolution
 

All radio intensity mapping experiments will have a chromatic beam whose size evolves with frequency. This can add to the number of degrees of freedom in the radial foreground structure, inhibiting foreground removal since more modes need to be removed to perform a suitable clean. Smoothing the maps differently in each frequency channel can homogenise the angular resolution, giving them a consistent effective beam size at all frequencies. This reconvolution should also suppress residual contamination manifesting as localised amplitude spikes in the temperature maps, smoothing them out so there are fewer discontinuous fluctuations which will impact the PCA and Fourier transforms for power spectrum estimation.

The reconvolution step involves convolving the intensity maps, one frequency channel ν_i at a time with the kernel



    K(θ,ν_i)=
        
        exp[-1/2(θ - θ_0)^2/γσ_max^2-σ_beam^2(ν_i)] ,


where σ_beam is the beam size as previously defined in <ref>, σ_max is the maximum σ_beam(ν_min) value at the lowest frequency,
and γ is a scaling factor which governs how much the final effective resolution decreases. The reconvolution kernel is normalised such that the sum over all pixels is unity i.e. Σ_θK(θ,ν_i) = 1. 

As in previous work <cit.>, we use a weighted reconvolution to mitigate the impact of edge pixels with low scan coverage. From hereon, the weights for the intensity maps derive directly from the time stamp count in each pixel, as shown in <ref>. The weighted reconvolved intensity maps are given by


    δ T_⊛(θ,ν_i) = [δ T(θ,ν_i)w(θ,ν_i) ] ⊛K(θ,ν_i)/w(θ,ν_i) ⊛K(θ,ν_i)


The weights themselves must also be reconvolved to keep them consistent and optimal for the new δ T_⊛ maps. The reconvolved weights are given by 


    w_⊛(θ,ν_i) = [w(θ,ν_i) ⊛K(θ,ν_i)]^2/w(θ,ν_i) ⊛K^2(θ,ν_i) .


In this work we use a value of γ = 1.4 for the reconvolution kernel in <ref>. We find that neglecting this resmoothing step entirely increases the influence of systematics in the results which is likely due to residual RFI or contaminants not being smoothed and depressing their dominance. We discuss this further in sec:ResmoothingAppendix but also plan a dedicated study into this aspect of the analysis where a more robust model of the MeerKAT beam will also be implemented.

The resmoothed, uncleaned map is shown in <ref> (top panel). Careful comparison with <ref> reveals how some of the foreground structure has been smoothed, albeit subtly, which is the aim. Too much reconvolution will suppress power in high k_⊥ modes in the final power spectrum estimate. The reconvolution is more noticeable in the time stamp count field, which we use as the weights w_⊛, shown by the bottom panel of <ref>.






 §.§ Map trimming


Shown by the time stamp count field of <ref> (and the resmoothed version in <ref>), the MeerKLASS scanning strategy of combining numerous rising and setting scans does not generate a uniform coverage. The edges of the patch result in relatively low recorded time stamps due to a lack of overlapping scans in these regions. 

We found that implementing an edge trim to the field improved the quality of the foreground clean and final power spectrum estimates. This is motivated and explored in greater detail in Carucci et al. (in prep.). The improvement from map trimming is because pixels are removed where residual time-varying contamination has been averaged down less, due to fewer scans. 

In <ref>, the black contour shows the cut to the patch we make, removing everything outside the trimmed lines. This amounts to all pixels at 334^∘ < R.A. < 357^∘ being set to zero in both the weights and intensity map field.



 §.§ Blind foreground cleaning with PCA


We represent the intensity maps with matrices  which are equivalent to the T(θ,ν) temperature fields, discretised into arrays on which matrix algebra can be performed.

Blind foreground cleaning assumes the matrix of the observational data cube _obs, which contains the intensity maps with n_θ pixels at each of the n_ν frequency channels, can be generalised by the linear equation


    _obs = + .


 is referred to as the mixing matrix and is the set of N_fg spectral functions estimated to project out the foregrounds.  are the N_fg source maps obtained by projecting  along the data;


    = ^T_obs .


The dot product  thus has shape [n_ν,n_θ], along with the other terms in <ref>, and represents a subset of the observations estimated to contain the bulk of the foreground contamination. The residual term  will contain the remaining  signal and thermal noise. In reality, the situation is more complex and there is cross-term leakage. Both  signal will leak into the subtracted  term causing signal loss, and foreground will leak into the remaining  causing foreground residuals. The control of both these is crucial and we discuss them further throughout the remaining sections.

Different blind foreground techniques only vary in their approach to estimating the mixing matrix .
In this work, we adopt the straightforward principal component analysis (PCA) approach, since it is well-tested and understood. Implementing more sophisticated blind techniques is planned in future work. PCA foreground cleaning begins by computing the frequency covariance matrix of the weighted observations


    =(∘_obs)^T(∘_obs)/n_θ-1 ,


where we have used the intensity mapping weights to optimise the estimation and ∘_obs represents the element-wise product between the weights and the observational data matrix. Since the weights are the stacked average of the counts along the line-of-sight, they are constant in frequency by design to avoid increasing the rank of _obs i.e. not inject additional structure into the frequency spectra. Weighting in this way means  is not a formally normalised covariance estimation, however, this is not a concern for us since we are only extracting unitless eigenvectors from the covariance and using these to project out modes in the real observed data so sensible amplitudes are recovered in the component separated pieces. We discuss this further, with some examples in sec:FGweights.  is then eigen-decomposed by


    = Λ^T ,


where  is the matrix of eigenvectors with the corresponding eigenvalues contained in Λ. The mixing matrix is then formed by taking the first N_fg column vectors u_i from  i.e. = [u_1,u_2,...,u_N_fg]. The first 9 eigenmodes are shown in the Appendix plot <ref>. It is at this stage a choice is required for N_fg, which determines how many modes are removed and deemed as foregrounds. If the N_fg is too high, the  signal loss increases; on the contrary, if N_fg is too low, residual contamination will dominate more. The estimate of the foregrounds is provided by _fg ≡ which is subtracted to provide the residuals, revealing the final cleaned data used in the analysis i.e. _clean ≡ where


    _clean = _obs -   _fg .


<ref> shows the resulting cleaned maps by removing N_fg=10 PCA modes. We show the maps split into six frequency sub-bands (given in the bottom-left of each panel) to illustrate the changing intensity structures. Each panel is the frequency average of the 42 channels in each sub-band showing a reasonably consistent amplitude of temperature fluctuations across the sub-bands. The amplitudes in the maps are still likely driven by non-cosmological contributions, mostly thermal noise, but they are not far from the 0.1 mK fluctuations we expect from . As we will show statistically in sec:PkEst, over large-scale fluctuations, the predominant contribution comes from the cosmological .







 §.§ Foreground cleaning tests using mocks


In order to validate our foreground removal and estimate the signal loss it induces, we run the same PCA cleaning technique on mock intensity maps. As discussed in sec:HIIMmocks, simulating foreground contamination that emulates the systematics and complex observational effects is beyond current capabilities. We therefore use the real observations themselves and inject mock  intensity maps into these. We then apply the foreground removal pipeline on the mock-injected data and estimate the signal loss from the injected mock whose original signal is known.

We follow the formalism outlined in <cit.>, beginning with a  mock, whose matrices we notify by  to distinguish from the real observation . The mock is injected into the real observations i.e. _obs + _. This is then cleaned by projecting out a subset of modes defined by the mixing matrix, as in the previous sections, such that


    _clean = (_obs + _) - ^T(_obs + _) ,


The data in _clean will now contain signal loss emulating that of the real data clean, along with residual systematics and noise from the real data.

We show the results of mocks contaminated by the real data in <ref>. We show measured power spectra averaged over all 500 mocks under different scenarios (details of power spectrum estimation are provided in the following sec:PkEst). Firstly, the grey dotted line shows the -only mocks, _, with no foreground cleaning effects, representing the true power. The red solid line shows the result of adding the real observations to the mock, then PCA cleaning as in <ref>. The power is higher than the grey line, despite the inevitable signal loss the clean will have caused. This is evidence that the _clean maps contain residual systematics and thermal noise, which are additively biasing the power[There will also be a small contribution from the true  signal in the observed data.]. We can demonstrate this by considering the orange solid line which is the cross-correlation of the cleaned mocks _clean with the -only mocks _, a test only possible with simulations. Since the residual systematics and noise will be uncorrelated with the -only mocks, the distortion in power comes purely from signal loss, hence why it is lower than the -only power at all scales.

To isolate the contribution from unknown systematics, we need to account for thermal noise and signal loss from foreground cleaning. The thermal noise contribution can be approximated using <ref> to generate 500 noise maps and add these to the -only mocks. The black-dashed line shows this and demonstrates the boost in power relative to the grey-dotted line, more so at small scales where, due to the beam, the  signal is low relative to the scale-independent noise[The noise is not completely scale-independent because it has gone through the same regridding and reconvolution processes as the data which causes some subtle damping.]. The last result presented in <ref> is the reconstructed power (i.e. correcting for foreground cleaning signal loss) shown by the blue solid line. This utilises the signal-loss evaluated from the cross-power (orange line) to reconstruct the _clean auto-power. This is discussed more in the following sub-section. The discrepancy between the blue-solid and black-dashed line must be caused by systematics, since signal-loss and thermal noise have now been accounted for. We find a similar excess power with the real observations, which we discuss in the following sec:PkEst. The results in <ref> are to validate the realism of the mock pipeline, and also to motivate the signal loss reconstruction process, which we formalise in the next sub-section.





 §.§ Signal loss correction with transfer functions


We can utilise the method of mock signal injection from the previous sec:FGcleanMocks for estimating the signal loss in the power spectrum of the real data. This is now a well-understood process <cit.> and used in previous single-dish intensity mapping analyses <cit.>.

We begin with the _clean term in <ref>, which as demonstrated, will contain signal loss that should emulate the real data. In some cases, the cleaned real data _clean can be subtracted from _clean. This accelerates the convergence of the transfer function calculation by removing the additional noise caused by the real data but has no bearing on the signal loss calculation. Leaving this piece in the calculation of _clean is however crucial when using the transfer function for estimating errors in the power spectrum. We also find that due to the large number of mocks used in this work, an equivalent accuracy of the transfer function is obtained whether subtracting _clean or not. For simplicity, we therefore only use the case as defined by <ref>, without the _clean subtraction.

The transfer function is defined by the power spectrum, signified by the operator P(), of the cleaned mock _clean with the foreground-free -only mock _, and the ratio of this with the power spectrum of the -only mock


    T_(k)=⟨P(_clean,_)/P(_,_)⟩_N_mock .


The angled brackets indicate an averaging over a large number of mocks such that a converged result is obtained. In this work, we use all mocks in our suite, thus N_mock=500. The operator P() can represent any power spectrum estimator i.e. we use this same recipe for estimating signal loss in the spherically averaged power and also in the cylindrical (k_⊥, k_∥) power spectrum, which we later utilise in some analysis.

In this work, we will mostly focus on the MeerKLASS deep field's cross-correlation with the overlapping GAMA galaxies. The signal loss from PCA cleaning in the cross-power spectrum should be identical to that in the auto-power. This was established empirically in <cit.> which showed how removing a subset of PCA modes entirely but leaving their contribution in the other half of the cross-power product (e.g. in the galaxy maps which have no cleaning, thus no signal loss) generates no more power than the auto-. We present further reasoning for why this is the case in sec:TF_signalloss_autoVcross. To summarise, we believe it is explained by the PCA modes being orthogonal, thus maximally uncorrelated. Therefore, the additional remaining modes in the uncleaned field are linearly uncorrelated with what survives in the cleaned , producing no extra power. 

Despite the approximately equal signal loss between auto and cross-correlations, we still define a separate transfer function for the cross-power spectrum purely for covariance estimation purposes whereby the only difference is we switch in the GAMA galaxy mocks _g instead which will emulate the signal strength and shot-noise of the GAMA galaxies, thus providing a more robust error estimate in the cross-power.


    T_,g(k)=⟨P(_clean,_g)/P(_,_g)⟩_N_mock .


Any measured power spectra should be divided by T(k), thus the transfer function represents a reconstruction of signal at the power spectrum level. 



§ POWER SPECTRUM ESTIMATION


In this section we present power spectra estimations from the foreground cleaned MeerKLASS L-band deep field. We first outline the pipeline for performing these measurements then present the results. By default, we use maps that have been foreground cleaned by removing =10 PCA modes, as outlined in the previous section. However, we also examine the results from varying . 



 §.§ Transforming to a Cartesian grid


The first task to perform in order to measure a power spectrum is to transform the foreground cleaned intensity maps onto a Cartesian grid with comoving h^-1Mpc coordinates. This allows for a Fast Fourier transform to be calculated on the Cartesian array, from which the power at different Fourier bandpasses k, can be averaged and presented. 

We discussed the transformation between sky maps and Cartesian grids already in sec:RegriddingMocks when we performed the inverse routine of sampling Cartesian cells from lognormal mocks into celestial sky map voxels that emulate the MeerKLASS intensity maps. We again utilise the recent work and code in <cit.> and first define a grid of size l_x,l_y,l_z = 632.2,341.5,231.5 h^-1Mpc with n_x,n_y,n_z = 88,48,168 cells along each dimension. The choice of cells is a factor of 1.5 smaller than the number of map voxels in R.A., Dec., and ν. Making the Cartesian grid slightly lower resolution than the map and using 10 sampling particles per map voxel ensures each Cartesian cell is well-sampled, avoiding empty cells.

The foreground cleaned maps δ T_(θ,ν) are transformed to this Cartesian grid, giving δ T_(x). The intensity mapping weights also follow an identical transformation process w_(θ,ν) → w_(x). The catalogue of GAMA galaxies can be directly gridded by transforming their celestial coordinates into Cartesian, then assigning them onto the Cartesian grid n_g(x). For all sampling particles and galaxies we use a straightforward nearest grid point (NGP) assignment. <cit.> presented extensions to this method for avoiding aliased power and obtaining sub-percent accuracy. However, given the current signal-to-noise of our data at small scales, aliasing effects will be negligible, thus higher-order interpolations and interlacing are overkill and serve only to increase compute demand. However, we correct for the dominant effect which is the field's smoothing by the (NGP) particle assignment. This requires deconvolving the field, achieved by dividing through by the NGP mass assignment window function in Fourier space <cit.>


    δ̃ ̃T̃_(k) →δ̃ ̃T̃_(k)/W_ngp(k) ,


where the window function W_ngp accounts for the sampling particles discretisation into cells of size H. These are separable functions along each Cartesian dimension, thus modelled by a Fourier transform of a top-hat function, which becomes a sinc function; sinc(x)=sin(x)/x and therefore


    W̃_ngp(k)=sinc(k_x H_x/2) sinc(k_y H_y/2) sinc(k_z H_z/2) .


Previous work <cit.> employed a tapering along the radial direction of the regridded field with the primary purpose of reducing edge effects in the Fourier transform. Our modelling approach should however account for these effects since the model is also convolved with the window function <cit.>. We still found results were worse when omitting a radial tapering though, which we suspect is instead caused by edge-effects from the foreground clean <cit.> which are suppressed by the tapering. We experimented with a less aggressive taper by using a Tukey window, also known as a cosine window (this is also explored in Carucci et al. in prep). This has a shape parameter α which is simply a top-hat with no apodisation when α=0 and equivalent to a Hann window when α=1. We show these window functions in <ref>. By opting for less aggressive Tukey windows we still effectively suppress the extreme edges where foreground cleaning effects may be present but avoid unnecessary tapering on the central data. We will explore this further with future data in MeerKAT's UHF band where the larger useable frequency ranges may render this less of an issue. In this work we found optimal results when using α=0.8 for the Tukey window which is applied to the intensity map weights w_(x). We do not apply this to the galaxy data as we found this increased the correlations between k-bins in the power spectrum. 





 §.§ Power spectrum formalism


With the fields assigned to a Cartesian space and appropriately tapered, the power spectra can then be measured. The Fourier transformed field of the  temperature fluctuation maps δ T_ is given by


    F̃_(k)=∑_xδ T_(x) w_(x) exp (i k·x) ,


where the weights, now denoted by w_ to distinguish them from the galaxy weights, still derive from the field of time-stamp counts, which are regridded following the same process as outlined in the previous sub-section. The Fourier transformed field of the the galaxy count field n_g is given by


    F̃_g(k)=∑_x n_g(x) w_g(x) exp (i k·x) - N_gW̃_g(k) ,


where N_g = ∑ n_g is the total number of galaxies in the optical maps and W̃_g is the weighted Fourier transform of the selection function


    W̃_g(k)=∑_x W_g(x) w_g(x) exp (i k·x) .


The galaxy window function W_g is normally constructed by stacking mock galaxies that carefully follow the survey selection characteristics of the experiment to account for its incompleteness. As discussed previously, there are no GAMA G23 mocks available for this purpose so we therefore construct a simple window function which is 1 within the survey boundaries and 0 everywhere else. Some inclusion of the redshift evolution of N(z) could have been included, but as <ref> showed, the variation over our narrow redshift range is minimal. The window function is normalised so that ∑_xW_g(x) = 1. Since we lack this proper galaxy window function we do not implement the optimal weighting as per <cit.>, but instead use uniform weighting, so equivalent of assuming w_g(x)≡W_g(x). 

The cross-power spectrum estimator is then given by


    P̂_,g(k) = V_cell/∑_x w_(x)w_g(x)W_g(x)Re{F̃_(k)F̃^*_g(k)}1/N_g ,


where V_cell=l_xl_yl_z/(n_xn_yn_z). The hat notation (P̂) indicates that this is the estimated power from the data, to distinguish from the modelled power, which we discuss shortly. The  auto power spectrum estimator is more straightforwardly given by


    P̂_(k) = V_cell/∑_x w_^2(x)|F̃_(k)|^2 ,



















These power spectra are either spherically averaged into bandpowers |k| ≡ k to provide 1D power spectrum results or alternatively cylindrically averaged into 2D bandpowers defined by modes perpendicular (k_⊥) and parallel (k_∥) to the radial direction z, which we assume is the line-of-sight in a plane-parallel approximation, valid for the current MeerKLASS sky sizes. They are finally divided by their relevant foreground transfer functions (<ref> and <ref>) to reconstruct signal loss for each k-bin.



  §.§.§ Modelling the power spectrum


Here we outline the model of the power spectrum estimations for which we mostly follow the work in <cit.>, with some extensions. We model the  power spectrum using the equation already introduced in <ref> when discussing mock simulations. We repeat the model here for convenience;


    P_(k)≡ P_(k,μ) = T_^2 (b_ + f μ^2)^2P_m(k)/1 + (kμσ_v/H_0)^2 .


Additional modelling of observational and mapping effects is also necessary since these will distort the power.  To model the MeerKAT beam and the additional smoothing from the reconvolution process we assume the field has undergone a constant smoothing by a 2D Gaussian kernel whose standard deviation is given by R_beam=√(γ) R^max_beam in comoving units. γ=1.4 accounts for the reconvolution to a common effective resolution and R^max_beam is the Gaussian approximation of the MeerKAT beam size at the deepest redshift (and lowest frequency ν_min)


    R^max_beam = χ(z_max) σ_beam(ν_min) ,


where σ_beam is given as before from <ref> and χ(z) is the comoving distance to redshift z, which in this case, is the maximum redshift for our frequency range. The perpendicular modes will then be damped according to


    B̃_beam(k) = exp[-k^2(1-μ^2) R_beam ^2/2] .


For this work, we find R_beam=14.1 h^-1Mpc. 

Similar to the beam, there will also be distortions to the power caused by the finite resolution of the intensity maps.  We approximate the pixel size in comoving units by s_pix=χ(z_eff)Δθ where Δθ=0.3 is the pixel size used for the MeerKLASS L-band mapping. The frequency channel depth is approximated by taking the mean across all spacings between redshifts corresponding to channel boundaries i.e. s_chan=Σ_i [χ(z_i) - χ(z_i+1)]/N_chan. These quantities are used in damping functions to model the suppressed resolution from mapping into pixels and frequency channels, the former being the far more dominant effect for the wide-pixel MeerKLASS intensity maps. These functions are given as


    B̃_pix(k) = sinc(k√(1-μ^2)  s_pix/2) ,


and


    B̃_chan (k)=sinc(k μ  s_chan /2) ,


The model and damping functions are discretised onto the same 3D grid of k-modes as the FFT for the real data, then convolved with the survey window function. The model for the observed  auto-power spectrum therefore becomes


    P_^obs(k) = W^2_(k) ⊛[P_(k)  B_beam^2(k) B_pix^2(k) B_chan^2(k)]/∑_x w^2_(x) W^2_(x) ,


where we have normalised by the sum of the weights and window function, W_, to obtain an amplitude that matches the weighted and windowed estimator. For the intensity mapping window function, we assume a simple binary function; 1 where we have scan coverage and 0 everywhere else. There is scope for improving this approximation and further work is needed to examine how intensity mapping surveys are impacted by any selection effects. 

A similar approach is followed for modelling the cross-power with GAMA. The cross-correlation power spectrum is given by


    P_,g(k)= T_[r b_ b_g+b_ f μ^2+b_g f μ^2+f^2 μ^4] P_m(k)/1 + (kμσ_v/H_0)^2 .


where we introduce r, the correlation coefficient between the  fluctuations and the GAMA galaxy over-densities. This accounts for the different astrophysics in each tracer and non-linear clustering, which can mean the two fields are not exact multiples of each other and there is instead some stochasticity which will suppress their cross-power <cit.>. We assume r to be constant, which is sufficient for the current signal-to-noise, but in reality it will exhibit some scale dependence. The correlation coefficient only enters on the single term in <ref> because it only influences the biased tracer fields, not their unbiased velocity fields which are modulated by f. The observational cross power is then similarly convolved with the survey window functions but only requires single factors of the damping functions B; 


    P_,g^obs   (k) = 
       W_(k)W^*_g(k) ⊛[P_,g(k)  B_beam(k) B_pix(k) B_chan(k)]/∑_x w_(x) w_g(x) W_(x) W_g(x)  .


We do not include modelling of uncertainties in the GAMA galaxy positioning, such as redshift estimation. The GAMA radial kernel equivalent of <ref> will likely have a larger impact due to spectroscopic redshift estimation errors. However, this will still be a sub-dominant contribution to the overall modelling relative to the beam and intensity map pixelisation, thus it is reasonable to neglect observational effects from the GAMA galaxies.

We perform a simple least-squares fit for an overall amplitude to our cross-power, equivalent to assuming fixed values for all parameters except for a fitted T_. This is deliberately simplistic to keep the model and fitting in a low-dimensional form. The small overlap sample in the cross-correlation means the statistical noise from such measurements is not yet suited to high-dimensional modelling. We are not therefore extending much from previous work <cit.> in this regard. Furthermore, the auto- still has evidence of additive bias, as we will discuss, and therefore can only be trusted for upper limits in parameter inference. 

For the power amplitude fitting, we assume the parameter values outlined by <ref>. The  bias comes from interpolated results from hydrodynamical simulations <cit.>, which can be approximated by the polynomial


    b_(z)=0.842+0.693 z-0.046 z^2 .


The galaxy bias is obtained by fitting the amplitude of the GAMA auto-power spectrum. The growth rate is obtained from f=Ω_m(z_eff)^0.545 for our Planck 18 cosmology. The correlation coefficient r and velocity dispersion σ_v are poorly constrained nuisance parameters tuned by eye to provide an adequate fit. We discuss model and parameter fitting further in sec:ParamConstraints where we present results from a simultaneous MCMC fit on several parameters, demonstrating the degeneracies between them and discuss plans for breaking such degeneracies, but this is not a central focus of this work. 










 §.§ Error estimation with the transfer function


In previous analysis <cit.>, an analytical approach was primarily used for estimating errors on the power spectrum measurements. However, this approach makes various assumptions, mainly that thermal noise dominates the error budget and that Gaussian statistics are valid. Since we are now reaching regions in k-space where thermal noise no longer dominates with MeerKLASS, we are required to relax these assumptions. Furthermore, the assumption of Gaussianity is arguably not solid, given the additive systematics we know to be still be present within the data.

A more robust approach of estimating the errors is to use the transfer function scatter across the different mock realisations T_i(k). This approach has been used in previous analysis <cit.> and has also been validated in simulations <cit.>. For it to perform effectively, the mocks in the transfer function calculation must emulate the real data effectively which we have strived to do so, as outlined in sec:Mocks.

The transfer function for one single realisation is defined by 


    T_i^,g(k)=P(_clean,_g)/P(_,_g) ,


i.e. equivalent to <ref> but without the averaging over N_mock. To reconstruct signal in a measured power spectrum, we take the ensemble average of <ref> i.e. T(k)=⟨T_i(k)⟩_N_mock, and divide the estimated power by this; P̂_rec(k)=P̂(k)/T(k). Note we are dropping the subscript ,g for brevity and also because the formalism we present here is identically applicable to the -auto. The assumption made when using the transfer function for error estimation is that the scatter in the reconstructed power comes from the transfer function distribution T(k)+δT_i(k) contributed by each ith mock. We can isolate the scatter component from the array of calculated transfer functions by simply subtracting the ensemble mean for each ith mock


    δT_i(k) = T_i(k) - T(k) .


Assuming any errors add in quadrature, error propagation suggests


    δP̂^rec_i(k)/P̂_rec(k) = √((δP̂_i(k)/P̂(k))^2 + (δT_i(k)/T(k))^2) ,


however, we assume all quantifiable error lies in the transfer function distribution, hence δP̂_i=0 and so


    δP̂^rec_i(k) = P̂_rec(k)δT_i(k)/T(k) .



Our setup thus provides N_mock realisations of δP̂_i^rec and the covariance can be computed from this for any k-bin.
We reiterate that this formalism is valid whichever way the k-modes are binned e.g. for a 2D cylindrical averaged or 1D spherically averaged power. In this work, our error bars in any 1D power spectrum estimate represent the limits of the 68th percentile regions of the distribution of δP̂^rec_i(k). If the uncertainties are Gaussian symmetric, this is equivalent to 1 σ error bars, however, as we will later show, this is not perfectly the case for all k-modes. 



 §.§ Spherical averaging k-cuts


Before presenting spherically averaged power spectra from the MeerKLASS data, we first present results from the cross-correlation with GAMA galaxies in cylindrical-(k_⊥,k_∥) space. This is shown in <ref> and motivates a discussion on where in k-space we have the most signal-to-noise so we can make optimal spherically averaged P̂(k) measurements. The dashed gray lines on all panels represent the |k|-bin boundaries we use for the spherically averaged power spectra, restricting this to k<0.3 to avoid highly non-linear cosmology and increasingly poor signal-to-noise. The orange solid lines represent boundaries for potential k-cuts which we will discuss.



The first panel of <ref> shows the cross-power between GAMA G23 galaxies and the MeerKLASS deep field, cleaned by removing =10 PCA modes. The k_⊥ k_∥ scaling is for demonstrative purposes so the high power at low-k does not saturate the plot. At k≳0.15 the large MeerKAT beam begins to have an impact and will damp  fluctuations, which explains why the cross-power signal tends to zero at high-k_⊥. Except for the extreme low-k regions, the rest of the k-space reveals a reasonably consistent positive cross-correlation signal, providing strong evidence of consistent clustering between the MeerKLASS maps and GAMA galaxies. As discussed in the previous sec:TFsignalloss, the signal loss in this power has been corrected with the foreground transfer function. The second panel shows the corresponding transfer function for the =10 clean. Low values of T_,g signify high signal loss. This is evidence that nearly all signal is lost for the lowest-k_∥ modes which is expected since these modes should be highly degenerate with the removed long radial modes of the continuum foregrounds. The high signal loss at low-k_⊥ is interesting and not seen in previous MeerKLASS work <cit.> with a similar method, albeit with less robust mocks. This is likely caused by the small GAMA footprint (see <ref>) which means the large-ranging modes at these low-k are not adequately resolved and deliver little signal which is easily coupled to foreground modes and stripped away in the clean. This can be explored further in follow-up work with wider surveys where we expect this not to be a problem.

Due to the noisy power at low-k and damped power at high-k_⊥ in <ref>, we can choose to cut these regions of k-space from our spherically averaged power to obtain a more robust result. The orange solid lines represent chosen boundaries, where modes beyond these boundaries are excluded in the averaging of each k-bin (shown by the dashed gray lines). Providing the modelling of any power spectra also incorporates the k-cuts, then it will not bias any results. The k-cuts approach is further motivated by considering the signal-to-noise of the cross-power which we show in the third panel of <ref>. Here the errors σ(P̂_,g) are computed using the transfer function by taking the standard deviation from the δP̂_i^rec(k_⊥,k_∥) distribution (see previous sec:TFforErrors). This result provides a strikingly clear window where signal-to-noise is optimal; the orange to red regions in the third panel. The orange k-cut boundary lines have been chosen to target this region. One limitation of this approach is that it excludes the regions of k-space with the most modes, as shown by the fourth and final panel. We will present comparisons between the spherically averaged P(k) with and without these k-cuts in the upcoming sec:GAMAcross. 



 §.§  auto-correlation power spectrum


We next examine results for the auto-correlation  power spectrum from the MeerKLASS L-band deep field. Previous single-dish intensity mapping results have been thermal noise-dominated across all scales. We are now crossing an important threshold with MeerKLASS where the  fluctuations should be dominating the noise at large scales. We demonstrate this in <ref>, where we show the estimated  power spectrum from the MeerKLASS deep field (blue data points) and overlay an estimate of the thermal noise (red solid line). As outlined in sec:HIIMmocks, this noise is modelled by generating Gaussian random fields with the expected rms given by <ref> which utilises the observational hit map. Despite being the  auto-power, we still restrict the measurement to the GAMA-field footprint, for consistency with the cross-correlation results. We tested using the full field but found no improvement in the auto-power. We leave exploration of the full-field to follow-up work which focuses on the -auto power spectrum. The results in <ref> include the scale k-cuts discussed in the previous section.

<ref> also shows the expected  power (grey dotted line) from <ref>. For this auto- model, we do not fit the amplitude to avoid additive bias from residual systematics over inflating the model estimate. Instead we use the scale-independent T_ fit from the upcoming cross-power measurement. This model shows how the cosmological  power surpasses the thermal noise on large scales (k≲0.15). The thermal noise will be present even in the absence of clustering, thus the total expected power can be reasonably approximated as a simple addition of the  power, plus the thermal noise (shown by the black dashed line).



The small discrepancy between the deep-field power estimate and the model represents the lowest upper limit from a  auto-correlation on these large scales. We find the mean residual factor between model and data to be just 2.36 across all k. Whilst the model and data are close in agreement, inferred parameter values for the bias and mean  temperature from fitting the amplitude of the auto power yields inconsistent values with the GAMA cross-correlation (discussed next). This is therefore evidence of low-level residual systematics that are additively biasing the power. In this evaluation, we inherently assume that the transfer function reconstructs any signal loss in the Gaussian thermal noise. If signal loss in the Gaussian noise is underestimated, the residual between the model and data may be higher than we report here. We therefore intend to robustly test this assumption when pursuing a more dedicated study of the  auto-correlation.

In follow-up work, we will focus on improving this upper limit from the  auto power which can be done in two ways: Firstly, by implementing some corrections to remove low-level systematic contributions such as the residual RFI structures discussed in sec:zebra. Secondly, we can split the observations into sub-sets i.e. combinations of different maps from different scan times and/or dishes. This is useful for isolating dish- and time-dependent systematics and removing their biasing contributions to the power. Since each scan and dish is observing the same patch of sky, we would ideally expect all maps to agree, thus any differences will represent thermal noise, or systematics such as residual RFI, varying polarisation leakage etc. We do not pursue these ideas in this paper since they warrant their own detailed investigation.



 §.§ Cross-correlation power with GAMA


We now focus on the cross-correlation with the GAMA galaxies for the remainder of the paper. In <ref> we present the cross-correlation power spectrum between the GAMA galaxies and the MeerKLASS deep field. All points are positive so none are lost to the log scaling, indicating a strong, positive correlation. Generally, a good agreement with the model is obtained. We performed a null test on the result by substituting the GAMA mock galaxies (from sec:Galmocks) in place of the real galaxies. The mocks should not correlate with the intensity maps, and indeed, the resulting cross-power converges to zero once averaged over a sufficient number of mocks. The fitted amplitude of the model is equivalent to an inferred parameter of T_=0.166 mK, with all other values in <ref> assumed. We discuss this further in sec:ParamConstraints, but do not focus on parameter inference in detail due to the large statistical noise owing to the small overlap footprint between MeerKLASS and GAMA.



The error bars for the cross-power in <ref> are obtained using the scatter from the foreground transfer function as discussed in sec:TFforErrors. The error size appears to be over-predicted based on the scatter of the data points, which have a reasonably tight agreement with the model relative to what the error bars reflect. This is supported by examining the reduced χ^2 which we find to be χ^2_dof≡χ^2/N_dof=0.42. A χ^2_dof<1 is conventionally indicative of overinflated errors. Using the reduced χ^2 however is not a reliable test where there are non-linear contributions, as is likely the case for these observations that are still prone to residual systematics. In these cases, the number of degrees of freedom cannot be robustly determined <cit.>. For our case we simply assume N_dof=13, for our 14 k-bins minus a single fitted degree of freedom, representing the amplitude fit. Furthermore, noise within the data causes high uncertainty in χ^2 estimates, which despite the relatively low noise levels we are now reaching, may still be a concern. Alternatives to χ^2 have been proposed, and specifically for 21cm applications 
<cit.>, but these have not been pursued in this work. We only use the χ^2 as an approximate guide, mainly for relative comparisons between different cases.

For the estimation of χ^2, we include the full contribution from off-diagonals in the k-bin covariance , which can be trivially estimated for our error estimation method, simply the covariance over P̂_i^rec(k)=P̂_rec(k)+δP̂^rec_i(k) (see sec:TFforErrors). The χ^2 is then estimated from






    χ^2=(p̂-p_ mod)^T ^-1  (p̂-p_ mod) ,


where p̂ is the data vector and p_mod the corresponding model. We present the normalised covariance matrix (or correlation matrix) in <ref>, showing evidence that the k-bins are slightly correlated, especially at low-k. This could be contributing to the large error bars, hence why we include the full covariance in the χ^2 estimation.



To investigate the errors from the cross-power further, we plot the full distributions for the reconstructed power spectra P̂_i^rec from each ith mock in the foreground transfer function computation. <ref> shows this for each k-bin, including an additional low-k bin which we otherwise exclude due to the large error on these scales. We centre the distributions on the model and normalise by it, to aid comparison with the case of no k-cuts (red results), which includes the areas of k-space with low signal-to-noise (discussed in sec:kcuts). The plot shows how the distributions have some non-Gaussian features, which could also impact error estimation, along with the correlated k-bins shown by the covariance. We find the average kurtosis across all k-bins for the k-cut case to be -0.476, indicating the distribution is flatter than a normal distribution with more contribution in the tails. This could also widen the central percentiles and thus increase the error bar size. 



<ref> shows a mild improvement in performance for the case with k-cuts. The distributions around the model are less broad at low-k, which is expected since this was one of the regions where signal-to-noise was poorest (see third panel of <ref>). In terms of the Gaussianity of the distributions, we find a better kurtosis score for the case with k-cuts, suggesting the profiles are more Gaussian. In contrast, we find the results without k-cuts to be less skewed, although this is marginal and in general, the skewness of the distributions is not concerning. Any improvement coming from the k-cuts is small and we also find little difference in detection significance, as we will explore shortly. As the volume of data grows beyond this small overlap sample, the k-cut results may prove more optimal since we will no longer be cutting modes from an already sample variance-limited data set. This is something that future data sets with MeerKLASS will reveal.



  §.§.§ Varying the foreground clean


So far we have only considered the chosen case of removing =10 PCA modes for the foreground clean. <ref> shows how the χ^2 (which we continue to use as a rough goodness-of-fit metric) varies with . As seen in previous work <cit.>, too low an  does not sufficiently clean the foregrounds, causing a poor fit. Going too high with  also diminishes the fit. Ideally, we would expect high  to cause more signal loss which should be reconstructed by the transfer function, giving unbiased results and thus not diminishing the fit. A higher uncertainty could be expected with greater signal loss, but this should be accounted for by the increasing error bars i.e. the χ^2 should stabilise and not start to rise again at high . On further scrutiny, we found that it is a single data point at k∼0.2 with very little power distorting the fits at higher . This is evidence of low-level residual systematics that become more influential as signal is stripped away in the more aggressive foreground clean. Therefore, with future results where systematics are more controlled, this is expected to be less of a problem. For the case with no k-cuts (thin dashed lines), there is a plateau in χ^2 at higher , as we argued should be the expected situation. This is because the no k-cut power spectra were less impacted by the k∼0.2 anomaly at high  which is potentially suggesting that the k-cuts are overly aggressive for this scale. With larger overlapping fields, the more accessible k-bins will naturally mitigate this discrepency. The no k-cut results in <ref> are therefore encouraging that a model and data fit should reach a plateau for high  as expected with the current analysis pipeline. 



The right hand side (red) y-axis in <ref>, also shows the detection significance defined by √(Δχ^2)≡√(χ^2 - χ_null^2), where χ^2_null is the result's χ^2 with a null model with zero power. The cross-correlation with GAMA is returning a modest > 4σ detection for the most optimal choices of foreground clean, lower than previous MeerKLASS detections <cit.>. This is however still impressive, given that galaxies only overlap with approximately a quarter of the full MeerKLASS deep field. The detection significance is also restricted by the more robust approach to error estimation, which as we have already discussed, provides conservative estimates potentially inflated by residual systematics and correlated k-bins. To demonstrate this point, we artificially construct a scenario where a χ^2_dof∼1 is obtained for the =10 case. This is achieved by assuming a perfectly diagonal covariance and
decreasing the diagonal errors by 40%. This boosts the detection significance to 9.8 σ, verifying that it is the conservative approach to errors lowering the detection significance, rather than the data-model residual. This is of course only a toy demonstration but it validates the argument that if large errors, which are suspected to be driven by systematics and k-bin correlations, can be contained, then detection robustness should improve. This is expected in future work where we will pursue dedicated studies into systematic mitigation. Some mode coupling is always likely, driven at least by non-linear clustering. However, we are confident that improvements can be made in covariance. MeerKLASS observations ongoing now are covering wider areas with more overlap with Stage-IV optical galaxy surveys. These observations are also using MeerKAT's UHF-band, which will allow a deeper usable redshift range, extending the 0.39<z<0.46 used in this work to approximately 0.4<z<1.4. This will naturally improve the large statistical noise due to the small volume overlap in this work.





§ STACKING
 

In this section, we explore the possibility of a stacking detection of the  signal in the MeerKLASS L-band deep-field intensity map. The technique of spectral line stacking <cit.> can be used to detect the  content of galaxies below noise limit. In combination with spectroscopic galaxy surveys, spectral line stacking probes the average emission line profile of selected galaxy samples. Complimentary to the intensity mapping technique, the stacking measurement of the emission line profile probes the  mass function of galaxy samples (e.g. ), the relation of  content with its host environment (e.g. ), and the baryonic Tully-Fisher relation (e.g. ). Interferometric observations of MeerKAT using the MeerKAT International GigaHertz Tiered Extragalactic Exploration survey (MIGHTEE; ) have been used to produce  measurements with regard to the scaling relation <cit.> with galaxy properties and the dependency on the large-scale structure environment <cit.>. See also <cit.> for direct detections of  galaxies in the MIGHTEE survey.

For intensity mapping experiments, stacking analysis has been performed on the  intensity maps using the Parkes telescope <cit.> and the CHIME telescope <cit.>, as well as on CO intensity maps from the CO Mapping Array Project <cit.> and Lyα emission of [O III]-selected galaxies from the Hobby-Eberly Telescope Dark Energy Experiment <cit.>.

MeerKLASS, on the other hand, faces unique challenges. The ∼1deg instrument power beam introduces heavy contamination from nearby sources. For the GAMA galaxy sample used, the number density of the sources is ∼ 35  deg^-2 when integrated along the frequency sub-band. Therefore, when stacking each source, the stacking procedure will inevitably include emissions from more than one GAMA galaxy. The velocity width of the  sources further scatter the 21 cm emission into nearby channels. The double-counting of sources from the beam creates a plateau of excess signal as shown in <ref>, and can be resolved by PCA cleaning, which we discuss in [apdx:stack]Appendix <ref>. However, the residual after PCA cleaning also incorporates systematic effects. The chromatic structure of the systematic effects produces oscillations in the stacked profile, with their amplitudes comparable to the signal. For this work, we are only interested in using stacking as part of the validation test. As discussed later, we obtain a detection of a stacked signal in both angular and spectral space, which can be validated by performing a null test with random galaxy positions. The existence of a stacked signal further demonstrates that we have reached the depth needed for the possible detection of the  power spectrum using the MeerKLASS L-band intensity maps. We leave a more detailed study into refining the stacking technique for probing  science to follow-up work.



 §.§ The stacking procedure of the intensity maps

We use the PCA-cleaned map described in [sec:FGcleaning]Section <ref> and the positions of the GAMA galaxies to perform a 3D stacking in (δα,δϕ,δν) space so that

    I(δα,δϕ,δν) = ∑_i I(α_i+δα, ϕ_i+δϕ, ν_i+δν) w_i/N_ g∑_i w_i,

where (α_i,ϕ_i) are the right ascension and declination of the map pixel in which the i^ th galaxy resides, I is the flux density, N_ g is the number of galaxies, and w_i=w_(α_i+δα,ϕ_i+δϕ,ν_i+δν) is the reconvolved weight of each pixel defined in [subsec:reconvolution]Section <ref>. We choose the angular and the spectral bins to be consistent with the map resolution with Δα=Δϕ=0.3deg. The frequency resolution corresponds to an effective velocity resolution of Δ v=62.83km s^-1 at the centre frequency of the sub-band 997.37MHz. Note that no Doppler correction has been performed on the data. For our analysis, the intensity maps are coherently averaged in frequencies for PCA cleaning and therefore can not be averaged in velocity channels.

In order to calculate the statistical noise for the stacked cube, we perform the same stacking routine on random galaxy positions, similar to the shuffling null test in sec:GAMAcross. The positions are randomly sampled uniformly within the region of the GAMA galaxy sample. The redshifts are randomly sampled following the redshift distribution of the survey. The random process is repeated for 100 realisations to calculate the mean and the standard deviation of the reference cube. The reference cube is used for performing the null test, i.e. checking that the stacked emission signal in the reference cube is consistent with zero.

After obtaining the stacked cubes, we create the angular and spectral stacking by summing over the cubes along the radial and transverse directions. To control the systematics generated by the foreground cleaning, we limit the voxels that go into the summation. For angular stacking, only voxels that are within 150 km s^-1 of the central channel are considered, corresponding to the typical width of the HI emission line profile. For spectral stacking, only the voxels whose centre positions are within the 1deg area of the centre voxel in each channel are included, corresponding to the size of the primary beam. The same summation is performed on the random realisation to calculate the mean and the standard deviation of the reference image and spectrum. We note that there is signal loss due to an insufficient number of voxels included. However, as the random realisations follow the same procedure, it does not affect the validation of the detection which is our focus in this work. 







 §.§ The detection of stacked  signal


In <ref>, we show the results for the angular stacking of the GAMA galaxies on the  intensity map. The stacked map shows a clear detection at the centre of the image, with the centre pixel achieving a detection with a statistical significance of 8.35 σ. The stacked  emission follows the structure of the beam. 

The extended structure of emission outside the beam FWHM is a result of the relatively large thermal noise fluctuation and contamination from nearby sources in the stacking procedure as discussed above. We note that, as shown in <ref>, the lower part of the GAMA galaxy sample region sits near the edge of the observed intensity maps. As a result, the lower half of the stacked image has a higher level of statistical fluctuations as shown in the right panel of <ref>.
Furthermore, the reference image is consistent with a null detection, demonstrating that the stacked signal does not arise from systematic effects in the map.



In <ref>, we show the stacked spectrum along the line-of-sight. 
The peak of the stacked spectrum shows a detection with a significance of 6.30 σ. The detection is validated by the reference spectrum which is consistent with a null detection. However, as seen in the reference spectrum, there is a systematic noise component with fluctuations on large frequency intervals. The systematics can be most visibly seen at the peaks and troughs of the stacked signal at v ∼ 2500, 4500, and 7000km/s. We have identified the source of this fluctuation as the residual systematics in the map after PCA cleaning, which is demonstrated in [apdx:stack]Appendix <ref>.
As stated previously, we are only interested in validating the detection against the null test and leave the suppressing of the systematic effects for future work.

The stacking results show a clear detection of  signal in the positions of the GAMA galaxies. The detection of an emission line signal is statistically significant against the null test.

At the same time, the residual systematics in the data, while being sub-dominant comparing to the  signal, induce structures in the stacked spectrum. We intend to follow up on the findings in this work and develop analysis techniques to extract the underlying emission line signal in the future, as well as using the stacked signal to understand the residual systematics.




§ CONCLUSIONS


In this work we have presented L-band observations from the MeerKLASS intensity mapping campaign. The survey covered a single patch of approximately 200 ^2, repeatedly scanned 41 times (27 scans remained after aggressive RFI flagging), which we have referred to as the MeerKLASS L-band deep field. The observations used in our analysis came from only a small section of the full L-band (971.2<ν<1023.6 at redshift 0.39<z<0.46) to avoid RFI which we find is more present in this band relative to the lower frequency UHF-band (580<ν<1000) which all future MeerKLASS surveys will be conducted in. We detailed the calibration strategy used on these observations which included a new approach of self-calibration (sec:SelfCal), exploiting the data's higher signal-to-noise to devise a less model-dependent calibration. The improvement in data quality relative to previous MeerKLASS pilot surveys is evident (see e.g. <ref>) and encouraging for future MeerKLASS surveys gathering ever-larger data sets.

We demonstrated the enhanced signal-to-noise from the calibrated deep-field intensity maps by presenting the auto-correlation power spectrum (<ref>) which revealed the lowest upper-limit from a  power spectrum at these cosmological scales k<0.3. This remains a factor of ∼ 2 above where we would expect the  power to lie. We will pursue this limit further in follow-up work where we will explore a systematic removal strategy to reduce the quantity of flagged data plus the cross-correlation of divided data sets to mitigate the additive bias from time-dependent residual systematics. 

We also presented results from the cross-correlation power spectrum between the MeerKLASS deep-field and 2269 overlapping GAMA galaxies with spectroscopic redshifts. Despite the small field of overlap (∼25% of the deep field patch), we still obtained a clear > 4 σ detection of the cross-power spectrum (<ref>). Furthermore, we presented a detection of  emission from stacking the MeerKLASS maps onto the positions of the GAMA galaxies. This yielded a 8.35 σ and 6.30 σ detection from angular and spectral stacking respectively, the first stacking detections with MeerKAT in single-dish mode (see sec:stack). 

We implemented several novel strategies into the power spectrum analysis pipeline, including a selective procedure to target the region of k-space with the highest signal-to-noise. The third panel of <ref> demonstrated the clear window of Fourier modes where power dominates over the error. By implementing some minima and maxima cuts on k_⊥ and k_∥ we were able to exclude regions of k-space dominated by error. This provides a more robust result since error-dominated modes will not be included in final spherical averaging. Comparison of the error profiles for each k-bin (<ref>) demonstrated a mild improvement in terms of Gaussianity when using the k-cuts, however, the sacrifice of many modes contained at high-k could be restricting this improvement. We plan to investigate this strategy in future work where larger data sets will reveal the optimal approach.

We formalised our approach to covariance estimation for the power spectra measurements which utlises the foreground transfer function distribution from each mock realisation. This distribution will be determined by all foreseeable sources of error; residual systematics, thermal noise fluctuations, galaxy shot-noise, cosmic variance and any uncertainty in signal-loss correction. This should therefore be a robust method for error estimation. However, we found the errors in the spherically averaged cross-power spectrum appear to be large relative to their scatter about the model. This was supported by a χ^2 analysis. Further inspection revealed that the k-bins are mildly correlated and the error profiles for each k-bin are not perfectly Gaussian, both of which could be inflating the errors. We hypothesise that this should be naturally mitigated with future MeerKLASS analysis where systematics can be more reliably controlled. 

This work also presented validation tests of our improved analysis pipeline. This was achieved by generating a suite of simulations that emulate the real data products. The mock data was then put through the same pipeline as the real data where we could validate its performance. Results in <ref> showed good agreement with the expected model based on the known inputs to generate the mocks, thus validating processes such as Cartesian field regridding and power spectrum estimation. Measured power from the mocks in <ref> which included the foreground cleaning also showed good consistency with real data.  



The MeerKLASS campaign is ongoing and will continue to be so in the pre-SKAO era. Further observations have already begun in the telescope's UHF-band. These lower frequencies should be less prone to RFI, meaning far fewer channels will be flagged. Furthermore, the lower frequency and wider wave-band mean a much larger portion of redshift space will be covered, providing intensity maps of unprecedented volume.



§ ACKNOWLEDGEMENTS


The authors would like to acknowledge the ongoing contributions to the MeerKLASS collaboration from members Suman Chatterjee, Karin Fornazier, Tamirat Gogo, Wenkai Hu, Piyanat Kittiwisit and Sifiso Mahlalela which provided valuable insights for this paper. We would also like to thank Jacob Burba, whose UK SKA Regional Centre project involved a full run-through of our post-calibration analysis pipeline, providing useful validation. 

JW acknowledges support from the National SKA Program of China (No. 2020SKA0110100).

SCu is supported by a UK Research and Innovation Future Leaders Fellowship grant [MR/V026437/1]. 

This result is part of a project that has received funding from the European Research Council (ERC) under the European Union's Horizon 2020 research and innovation programme (Grant agreement No. 948764; PB).

JLB acknowledges funding from the Ramón y Cajal Grant RYC2021-033191-I, financed by MCIN/AEI/10.13039/501100011033 and by
the European Union “NextGenerationEU”/PRTR, as well as the project UC-LIME (PID2022-140670NA-I00), financed by MCIN/AEI/ 10.13039/501100011033/FEDER, UE.
SCa acknowledges support from the Italian Ministry of University and Research, PRIN 2022 `EXSKALIBUR – Euclid-Cross-SKA: Likelihood Inference Building for Universe Research', from the Italian Ministry of Foreign Affairs and International
Cooperation (grant no. ZA23GR03), and from the European Union – Next Generation EU. 

IPC is supported by the European Union within the Next Generation EU programme [PNRR-4-2-1.2 project No. SOE0000136, RadioGaGa].

JF acknowledges support of Fundação para a Ciência e a Tecnologia through the Investigador FCT Contract No. 2020.02633.CEECIND/CP1631/CT0002, the FCT project PTDC/FIS-AST/0054/2021, and the research grants UIDB/04434/2020 and UIDP/04434/2020. 

AP is a UK Research and Innovation Future Leaders Fellow [grant MR/X005399/1]. ZC's research is supported by a UK Research and Innovation Future Leaders Fellowship [grant MR/X005399/1].

MGS acknowledges support from the South African Radio Astronomy Observatory and National Research Foundation (Grant No. 84156).

LW is a UK Research and Innovation Future Leaders Fellow [grant MR/V026437/1]. 

The MeerKAT telescope is operated by the South African Radio Astronomy Observatory, which is a facility of the National Research Foundation, an agency of the Department of Science and Innovation. We acknowledge the use of the Ilifu cloud computing facility, through the Inter-University Institute for Data Intensive Astronomy (IDIA).

GAMA is a joint European-Australasian project based around a spectroscopic campaign using the Anglo-Australian Telescope. The GAMA input catalogue is based on data taken from the Sloan Digital Sky Survey and the UKIRT Infrared Deep Sky Survey. Complementary imaging of the GAMA regions is being obtained by a number of independent survey programmes including GALEX MIS, VST KiDS, VISTA VIKING, WISE, Herschel-ATLAS, GMRT and ASKAP providing UV to radio coverage. GAMA is funded by the STFC (UK), the ARC (Australia), the AAO, and the participating institutions. The GAMA website is https://www.gama-survey.org/https://www.gama-survey.org/.

For the purpose of open access, the author has applied a Creative Commons Attribution (CC BY) licence to any Author Accepted Manuscript version arising from this submission. 




§ DATA AVAILABILITY


The data underlying this article will be shared upon reasonable request to the corresponding author. Access to the raw data used in the analysis is public (for access information please contact archive@ska.ac.za).





mnras
 



















§ OBSERVATION DETAILS


<ref> provides the details for the MeerKLASS L-band deep field observations, for all 41 attempted observation blocks.





§ PCA FOREGROUND CLEANING




 §.§ Impact of reconvolution


As discussed in the main text, the reconvolution of the maps before foreground cleaning (outlined in subsec:reconvolution), slightly improves the foreground clean performance. This is not necessarily just because it homogenises the effective resolution of the beam, as is the primary aim of this technique. The smoothing will also suppress systematics in the data. Residual contributions, for example from RFI, will distort the true amplitude of the sky map. If these contributions vary with frequency, which will be the case for most systematics, then this will create small perturbations to the spectra along each pixel. Despite smoothing the maps in the angular direction only, this will still suppress perturbations along the line of sight, hence providing slightly smoother spectra, which may be linked to the improved foreground clean. 

We demonstrate this in <ref> which plots the amplitudes of each pixel along frequency, as previously shown in <ref> (here we also trim the edges of the map, as outlined in sec:maptrim). The top panel shows the spectra without reconvolution. A clear improvement in the smoothness of the spectra is evident when reconvoltion is used (bottom panel). 

Future work is planned where a more realistic MeerKAT beam will be implemented into our modelling. In this work, we will also investigate this technique of reconvolution and aim to conclude if the smoothing to a common resolution is the source of any improvement or if it is just entirely from the suppression of systematics, reducing the perturbations they cause in the spectra which aids the PCA process.





 §.§ Weighted frequency covariance estimation


Here we explore the small impact that opting to use weights in the covariance estimation has on the PCA cleaning process. The weights used are constant along the frequency direction, so not to add structure to the modes and diminish the PCA performance. Furthermore, after trimming the map edges, the time stamps have a more uniform coverage (see <ref>). This means the weights do not vary considerably across our whole field, so we would expect only minor changes by including them. 

<ref> shows the first 9 eigenmodes from , extracted from the diagonalisation of the covariance matrix computed using <ref>. We show the difference between the weighted case (blue solid) and the unweighted (orange dashed), equivalent to omitting the  factor in the covariance calculation. As expected, the differences are marginal, especially when considering that in some modes the discrepancy is mainly an arbitrary change of sign. However, some small differences are still noticeable, particularly for the higher eigenmodes where noise will become more of a component which is effectively what we are weighting for. 

We found little evidence of improved performance in the final results whether weights were used or not in the foreground clean. The difference between the cleaned maps was at the sub-1μK level, and testing the process on the mocks showed no clear performance gain. However, we included them in our final results for completeness. Furthermore, the weighted formalism may be useful in future work focussing on  auto-correlations, where cross-correlating sub-sets of maps from different dishes or time blocks will be pursued. Here, different map weights can be used in a combined cross-covariance computation; then asymmetric singular vectors can be extracted from this, as in the Singular Value Decomposition (SVD) approach of <cit.>.



To accompany the eigenmode plot of <ref>, we also show the projected map from the first 7 eigenmodes in <ref> for the weighted PCA case. We omit the first eigenmode since it is indistinguishable from the synchrotron sky maps we have shown previously. These show at map level, each component that is being removed in the PCA clean. For example, a N_fg=7 clean would remove all these components. No obvious systematic structure is apparent which could otherwise be targetted for removal by other means, rather than relying on an aggressive PCA clean. However, as we discussed in sec:zebra, when cleans are performed individually on smaller sub-sets of time blocks, constant declination stripes become apparent in some of the components which we will aim to correct for in calibration in future work.












 §.§ Power index of the transfer function for signal resconstruction in auto- and cross-correlations


The transfer function defined in <ref> is applied to the measured power spectrum to reconstruct the loss of information due to foreground cleaning. A priori, intuition would lead to applying the transfer function as defined for the cross-power spectrum of  and galaxies, and the square of the transfer function for the  auto-power spectrum, as signal loss occurs twice in that calculation <cit.>. However, <cit.> found empirically that squaring the transfer function over-corrected the signal loss and that the transfer function should be applied equally to cross- and auto-power spectra for correct modelling. In this appendix, we provide simplified analytic arguments to support such a claim. 

Let us assume the idealised scenario where we can decompose the cosmological fluctuations δ T_(k)≡ r(k)+l(k), where r and l correspond to the residual and loss modes of the cosmological signal, respectively. Under this separation, the transfer function can be understood as


    T_(k) = ⟨ r_ m(k)[r_ m(k')+l_ m(k')]^*⟩/⟨[r_ m(k)+l_ m(k)][r_ m(k')+l_ m(k')]^*⟩ ,


where the subscript `m' refers to quantities obtained from mock injection. 
Due to the component separation process in PCA,
r and l are a combination of independent components, and are therefore uncorrelated. The power spectrum of their sum is


    ⟨[r_ m(k)+l_ m(k)].    . [r_ m(k')+l_ m(k')]^*⟩ = 
     =     1/(2π)^3δ_D(k-k')[P_ m^(r)(k)+P_ m^(l)(k)] 
    =     1/(2π)^3δ_D(k-k')P_ m^(k) .


Note that the equation above applies equally to the actual observed cosmological signal. Therefore, the transfer function can be now understood as


    T_(k) ∼P_ m^(r)(k)/P_ m^(r)(k)+P_ m^(l)(k) .


Let us now consider the application of this conceptual transfer function to the power spectra measured from the clean maps. In this case, we have


    ⟨ r(k) r^*(k')⟩/T_(k)∼ P^(r)(k)[ P_ m^(r)(k) + P_ m^(l)(k)/P_ m^(r)(k)]


for the auto-power spectrum, and


    ⟨ r(k)g^*(k')⟩/T_(k)∼ P^(rg)(k)[P_ m^(r)(k) + P_ m^(l)(k)/P_ m^(r)(k)]


for the cross-power spectrum. Assuming P^(r)∼P_m^(r), it is clear that a single power of the transfer function (i.e. T^1) correctly reconstructs the auto-power to the desired P^(r)+P^(l) level. For the cross-power spectrum, let us simplify the situation and assume linear theory, for which δ_g∝δ T_∼ r+l. Then, P^(rg)∝ P^(r); if we denote the proportionality factor generically as A, we have 


    ⟨ r(k)g^*(k')⟩/T_(k)∼AP^(r)(k)[1+P_ m^(l)(k)/P_ m^(r)(k)] ,


which similarly shows that a single power of the transfer function must also be applied to the cross-power spectrum.




§ PARAMETER INFERENCE FROM THE CROSS-POWER


We have not dedicated a lot of attention to the modelling of the power spectra in this work. This is mostly due to the large errors in the cross-power due to the small sample overlap and the small additive bias still present in the auto- power spectrum. This makes high dimensional models unnecessary and we opted instead for a simple amplitude fit of a fiducial model with some assumptions on parameters (see <ref>). Some of these parameters are poorly constrained however, so here we explore what happens if we extend the model and include an MCMC fit of four parameters; φ={Ω_,b_,b_g,σ_v}. Ω_ is the dimensionless  density parameter and can be directly linked to the mean  temperature through


    T_(z)=180 Ω_(z)   h (1+z)^2/√(Ω_m(1+z)^3+Ω_Λ) mK ,


where Ω_m and Ω_Λ are the density parameters for matter and the cosmological constant respectively.

We assume wide and flat priors for the φ parameters given by 


	
  * 0<10^3Ω_<2
	
  * 0.5<b_<2.0
	
  * 0<σ_v<600 [km s^-1]
	
  * 1.8<b_g<2.0


A tighter prior is assumed for the GAMA galaxy bias since we fit the galaxy auto-correlation power spectrum, finding a best-fit bias value of b_g∼1.9. The remaining priors are sufficiently wide, even for these poorly constrained parameters. We stress that this is still an overly simplistic approach, but our aim here is a demonstration of current modelling capabilities, not rigorous parameter inference.

<ref> shows the inferred parameter constraints from the MCMC fit. We use  <cit.> with 500 iterations of 200 walkers. We checked the chains had converged and discarded a burn-in of 40 iterations. Since it is entirely prior dominated in the cross power, we marginalise over b_g and do not show its parameter contour. The nuisance parameter σ_v is also completely prior dominated and suggests that a more refined model is needed on smaller scales, something explored in <cit.>. There is a constraint on Ω_ but this is only due to the restricted prior on b_. Relaxing the  bias prior would inflate the Ω_ constraint indefinitely. 










<ref> thus demonstrates the degeneracy between Ω_ and b_ which has led previous efforts to merge this parameter fit in different ways <cit.>
. This degeneracy can be broken by utilising RSD which introduces an anisotropic dependence on b_ but not on Ω_. We tested a measurement of the power spectrum quadrupole (a smoking-gun of RSD) <cit.> and found a clear non-zero measurement. However, we found that this was overwhelmingly driven by observational effects and higher signal-to-noise is required to make a distinction between this and a true RSD-induced anisotropy. Further discussion on the degeneracies between astrophysics and cosmology and the capabilities of the LIM power spectrum multipoles to constrain them can be found in <cit.>, where optimal parametrisations are proposed



§ EFFECTS OF PCA IN THE STACKED SIGNAL

In this section, we use mock  galaxy signals to generate signal-only maps for stacking, and illustrate the effects of PCA in creating additional systematic noise structure.



As we are only interested in simulating the effects for stacking in this section, we use generate Poisson realisations of  sources with no clustering information as shown in <ref>. For galaxies inside the survey coverage of GAMA, we use the exact same positions as the GAMA galaxies to replicate the double-counting effects. The rest of the map is then filled with galaxies at random positions assuming the same number density as the GAMA field. Random redshifts are assigned to the galaxies following the redshift kernel shown in <ref>. Each galaxy is then assigned a  mass following the  mass function reported in <cit.>. Furthermore, we assign the velocity full width half maximum of the emission line profile w_50 to each galaxy using the Tully-Fisher relation reported in <cit.>. The emission line profile of each galaxy is then calculated and added to the mock signal cube following the simplified busy function <cit.>

    B_2(v) = a/2×(  erf[ b(w_50^2-v^2)  ] +1 ) × (cv^2+1),

where erf is the error function and a,b,c are parameters to be set. For each galaxy, we randomly choose b logarithmically uniformly in [10^-3,10^-2]  km^-2s^2 and c in [10^-2,10^0]  km^-2s^2. The amplitude parameter a is then set by the  mass of the galaxy. We find that varying the choice of the busy function parameters does not significantly impact the results due to the relatively low velocity resolution. The signal is then convolved with the resmoothed beam described in [subsec:reconvolution]Section <ref>.




As discussed in [sec:stack]Section <ref>, the low angular resolution of the map induces leakage from nearby sources in the stacked profile. To illustrate this, we perform the stacking routine on the signal-only mock cube and compare against the input mock signal in <ref>. The input signal is calculated by averaging the emission line profile of each source according to <ref>. The double-counting introduces a large plateau of contamination across the frequencies. We note that the increasing slope of the plateau is due to the redshift distribution of sources. We then apply the PCA cleaning to the mock signal cube and repeat the stacking procedure. Instead of calculating the mixing matrix from the mock cube, we use the exact same mixing matrix obtained from the map for PCA and show the results in <ref> (orange dashed line). The PCA removes the plateau to recover an emission line profile. However, it introduces several effects. There is signal loss due to the cleaning as well as the fact that we did not sum over an area much larger than the beam. Moreover, the PCA extends the width of the profile as seen in the lower panel of <ref>. 

Apart from the effects on the recovered signal, PCA also creates a systematic noise component as seen in the stacked spectrum. To further demonstrate this, we repeat the reference spectrum calculation described in [sec:stack]Section <ref> on the mock signal and present the results in <ref>. 

Compared to the reference spectrum, the stacked mock signal outside the centre area of the peak shows deviations. The deviation from the reference spectrum appears to be coherent over large velocity intervals Δ v∼5000km/s. The scale of the interval is consistent with the systematic effects seen in the stacked spectrum in <ref>. We emphasize that this is purely the combination of the  signal and the PCA cleaning since the mock map does not contain any other component. We further find that, when increasing the angular area for summation from (1  deg)^2 to (3  deg)^2, the amplitude of the coherent structure increases. 

The structure of the PCA cleaning is caused by the residual systematics in the data. Following the structure of the stacked spectrum in <ref>, we generate random Gaussian realisations of systematic noise components and convolve the systematics to the mock signal. We then repeat the stacking procedure as well as reference spectrum calculation and show the results in <ref>. As one can see, the mock signal affected by the systematics shows similar oscillating structure as seen in the data. The reference spectrum, on the other hand, shows consistentcy with a null detection, and demonstrates the fact that the detection reported in [sec:stack]Section <ref> is a result of the detection  signal with residual systematics, and does not arise purely from contamination in the data.



We intend to simulate the mock data cube with foregrounds, noise and systematics to fully understand the impact of residual systematics on the stacked spectrum in the follow-up work.










 Paper Title: Separating deterministic and stochastic gravitational wave signals in realistic pulsar timing array datasets.

 

    
   




    Dipartimento di Fisica “G. Occhialini", Universitá degli Studi di Milano-Bicocca, Piazza della Scienza 3, I-20126 Milano, Italy

    i.ferranti@campus.unimib.it
    INFN, Sezione di Milano-Bicocca, Piazza della Scienza 3, I-20126 Milano, Italy
    INAF - Osservatorio Astronomico di Cagliari, via della Scienza 5, 09047 Selargius (CA), Italy
    INAF - Osservatorio Astronomico di Brera, via Brera 20, I-20121 Milano, Italy
    ASTRON, Netherlands Institute for Radio Astronomy, Oude Hoogeveensedijk 4, 7991 PD Dwingeloo, The Netherlands
    
Not anymore
Ferranti et al





    











  Recent observations by pulsar timing arrays (PTAs) suggest the presence of gravitational wave (GW) signals, potentially originating from supermassive black hole binaries (SMBHBs). These binaries can generate two kinds of signals: a stochastic gravitational wave background (GWB) or deterministic continuous gravitational waves (CGW). Being able to correctly recognize and separate them is crucial for accurate signal recovery and astrophysical interpretation. This paper aims at investigating the interaction between stochastic GWB and deterministic CGW signals within current analysis pipelines. We focus on understanding potential misinterpretations and biases in the parameter estimation when these signals are analysed separately or together. To this end, we performed several realistic simulations based on the European PTA 24.8yr dataset. We first injected either a GWB or a CGW into five datasets (three GWB realisations and two CGW realisations) with identical noise and analysed each signal type independently, and then analysed data sets containing both a stochastic GWB and a single resolvable CGW. We compared parameter estimation using different search models, including Earth term (ET) only or combined Earth and pulsar term (ET+PT) CGW templates, and correlated or uncorrelated power law GWB templates. We show that, when searched independently, the GWB and CGW signals can be misinterpreted as each other, and only a combined search is able to recover the true signal present. For datasets containing both a GWB and a CGW, failure to account for the latter biases the recovery of the GWB, but when performing a combined search, both GWB and CGW parameters can be recovered without any strong bias. Care must be taken with the method used to perform combined searches on these multi-component datasets, as the CGW PT can be misinterpreted as a common uncorrelated red noise. However, this can be avoided by direct searches for a correlated GWB plus a CGW (ET+PT). Our study underscores the importance of combined searches to ensure unbiased recovery of GWB parameters in the presence of strong CGWs. This is crucial to interpret the signal recently found in PTA data, and is a first step towards a robust framework for disentangling stochastic and deterministic GW components in future, more sensitive datasets.
    
    
 


  

  

  

  

  





   Separating deterministic and stochastic gravitational wave signals in realistic pulsar timing array datasets
    Irene Ferranti1,2, Golam Shaifullah1,2,3, Aurelien Chalumeau1,2,5 & Alberto Sesana1,2,4
    August 5, 2024
============================================================================================================







§ INTRODUCTION


Pulsar Timing Array (PTA) collaborations search for gravitational waves (GWs) with frequencies in the nanoHertz (nHz) band (10^-9-10^-7Hz). The primary target of PTA experiments is a stochastic gravitational wave background (GWB) generated by the incoherent superposition of the GWs emitted by all the super-massive black hole binaries (SMBHBs) that populate our universe <cit.>. The signal manifests itself in the data as a red noise process common to all the pulsars in the array, with a specific angular correlation, first derived by <cit.>. Recently, the PTA community found evidence for a signal with both these characteristics: the collaborations reporting this finding were the European PTA collaboration along with the Indian PTA collaboration <cit.>, the North American Nanohertz Observatory for Gravitational Waves collaboration <cit.>, the Parkes PTA collaboration <cit.> and the Chinese PTA collaboration <cit.>. 

While the Hellings and Downs (HD hereinafter) correlation is distinctive of a GW origin of the signal, separating it from possible correlated noise of different nature <cit.>, its source cannot yet be confidently established. In fact, besides SMBHBs, a nHz cosmic GWB can also be produced by a number of physical processes occurring in the early universe, such as cosmic strings networks <cit.>, primordial curvature perturbations <cit.>, or QCD phase transitions in the early universe <cit.>, all of which can provide a viable explanation of the observed signal <cit.>.
Moreover, the HD correlation is not unique of stochastic GWBs; as shown in <cit.> and in <cit.>, the signal generated by a single SMBHB and observed by an array of pulsars results in an angular correlation with the HD expectation value, albeit with a somewhat different variance. Note that the quality of the current data does not allow a description of the signal variance and  therefore, the possibility that a particularly loud source – a massive and nearby SMBHB – can be mistaken by a GWB is real. By loud source, we mean a source that can be singled out from the noise and, if present, from the GWB by means of a deterministic, template based search. Since the SMBHBs producing such signals are far from mergers and persistent in the data, their signal is generally referred to as continuous gravitational wave <cit.>.

CGWs searches have been performed on the latest data release of EPTA <cit.> and on the NANOGrav 15yr dataset <cit.>. They both found evidence in favour of the presence of a CGW with a 2.5-3σ significance, compared to a model where only pulsar noise is present in the data. This value is comparable to the evidence in favour of the presence of a GWB, which has a 3-to-4σ significance). Some simulations carried out in <cit.> already showed how a CGW can be misinterpreted as a GWB and, vice versa, a GWB can be misinterpreted as a CGW. In both the EPTA and NANOGrav analyses, when a GWB is also searched for, evidence for a CGW significantly decreases, indicating that its presence is at the very least not necessary to explain the observations. However, the degeneracy between the pulsar response to the two signals does not allow to conclusively rule out the presence of a CGW. 

The presence of these resolvable sources in the PTA data on the one hand poses a number of interesting challenges, on the other hand opens the door to invaluable opportunities. In fact, while the analysis pipelines currently used to search for a GWB rely on the assumption that the background is a Gaussian, isotropic, and stationary process characterised by a power-law Fourier spectrum <cit.>, the presence of a CGW can result in an excess of power at specific frequencies, which produces some level of anisotropy and non Gaussianity <cit.>. Thus, the standard analysis may depend on inapplicable assumptions and yield biases on the estimated GWB parameters <cit.>. This is a recognised issue within the PTA community and analysis techniques are being developed to go beyond the power law, isotropic Gaussian assumption <cit.> and to pull out of that data GWBs and CGWs simultaneously <cit.>.

Within this context, it is of paramount importance to study the interplay of CGWs and GWB when both are present in the data and what challenges arise in the interpretation of the analysis results. From an exquisitely scientific perspective, conversely, the prospects of detecting CGWs is particularly appealing. In fact due to their deterministic nature, PTAs are able to estimate the source parameters, in particular the sky location via triangulation <cit.> and possibly the distance to the source <cit.>. The detection of one or more resolvable sources <cit.> would be very precious as it would allow to track the orbital evolution of the binary and thus obtain direct information on the Massive Black Holes (MBHs) coupling to the environment <cit.>. This information cannot be well constrained by a measure of the GWB power spectral density, because it is strongly degenerate with the MBH mass and eccentricity distribution. In addition, CGWs offer the exciting opportunity to search for electromagnetic counterparts of the GW signal, opening the prospects of low frequency GW multimessenger astronomy <cit.>. Although PTAs can realistically localise the CGW source only within several deg^2 <cit.> they must be originated by particularly massive and/or nearby SMBHBs, making the search of their galaxy host viable <cit.>. Identifying the host and detecting accretion activity onto the binary will shed light about the physics governing the formation and (thermo)dynamical evolution of SMBHBs and the accretion discs surrounding them. 

In this paper, we carry out the first detailed investigation of the interplay between stochastic GWBs and deterministic CGWs in realistic PTA datasets, assessing the performance of currently used analysis pipelines when both signals are present in the data. We do so by building state of the art mock datasets capturing the complexity of real data, and by injecting them with as realistic as possible GW signals, mimicking the one found in the recent EPTA analyses <cit.>. Realistic GWBs are injected as the incoherent sum of individual waves consistent with astrophysical observations <cit.>, and CGW are modelled according to the candidate identified in <cit.>. Datasets are then analysed with the standard models implemented in the  <cit.> analysis suite, which models the GWB as a common red process characterized by a single power law spectrum, while we also test CGW models including only the Earth term or the full Earth plus pulsar components. 

The paper is organized as follows. In Section <ref> we describe in detail the simulations: the properties of the adopted datasets, the injected signals and the model used for the recovery. In Section <ref> we present the main results of our analysis. Using the methods described in the previous section, we carried out several investigations. We first assess the importance of including the pulsar term in the CGW model (Section <ref>). We then study the degeneracy of signal recovery on datasets containing only a GWB or a CGW (Section <ref>), showing that datasets injected only with a CGW can produce evidence of a GWB (Section <ref>) and vice versa (Section <ref>), and that a joint search is able to pin down the correct nature of the signal (Section <ref>). The performance of a combined analysis of a GWB and a CGW on datasets containing both signals is then evaluated in Section <ref> and we discuss our main findings in Section <ref>.
 Throughout the paper we assume a Λ cold dark matter cosmology with parameters (H_0, Ω_m, Ω_Λ)=(70 km s^-1Mpc^-1, 0.3, 0.7). 
















  


§ METHODS

In this section, we describe both the simulations structure and the models and methods used in the analysis.



 §.§ Simulations




Our simulations are based on the second data release of EPTA <cit.>. In particular, we have reproduced the DR2full dataset: 25 pulsars with a maximum observation time of 24.8 yr. The main properties of the real time of arrivals (ToAs) of each pulsar are reproduced: the observation time, the number of ToAs, the timing model (TM), the multiband measures – the frequency coverage is identical to that of the EPTA DR2full pulsars – and the noise levels. The uneven sampling of the ToAs is also reproduced, but large gaps, which are common in the real data streams, are not present in the simulations (see Fig. <ref> for a comparison between real and simulated residuals).
All the properties of the arrival times were injected in the dataset using the  package <cit.>. The same package is used to inject the three components of the noise budget: white noise (WN), achromatic – independent of the observation radio frequency –  red noise (RN) and chromatic red noise (due to dispersion measure variations, and customarily referred to as DM). 

To model the white noise, the ToAs are sampled from a Gaussian distribution centred at the value predicted by the TM and with a width given by:

    σ_i = √(EFAC_i^2σ_ ToA^2 + EQUAD_i^2),

where σ_ ToA is the errorbar associated to each ToA according to the template-fitting errors obtained in EPTA DR2, the EFAC takes into account for the ToA measurement errors and the EQUAD accounts for any putative extra source of white noise. The two parameters EFAC and EQUAD are specific for each observing backend. In the simulations, each pulsar's backend, as well as the ToAs' uncertainties, are identical those of DR2full; therefore, each pulsar has ToAs taken from ∼12 different backends, which means 12 different couples of values for EFAC and EQUAD. The values used for EFAC and EQUAD are the maximum likelihood values obtained by the single pulsar noise analysis of EPTA DR2full <cit.>.

In addition to the white noise, achromatic and chromatic red noise contribute to the single-pulsar noise budget. These are time-correlated noise components modelled as a stationary Gaussian process with a power-law power spectral density (PSD) of the form

    P(f; A, γ) = A^2/12π^2(f/ yr^-1)^-γ yr^3.

For achromatic red noise, in each ToA, the delay induced in the residuals is proportional to the red noise PSD, 

    Δ_ RN∝√(P(f; A_ RN, γ_ RN)).


This noise component accounts for the long-term variability of the pulsar spin along with other sources, such as unmodelled objects orbiting the pulsar. For chromatic red noise, on the other hand, the induced delay also depends on the observing radio frequency because it is due to the interaction of the radio signal with the ionised interstellar medium (IISM), the interplanetary medium in the solar system, and the Earth's ionosphere. This effect is taken into account during the observations and inside the timing model which considers its value at a reference epoch together
with its first and second derivatives. However, the inhomogeneous and turbulent nature of the IISM also induces stochastic variations in the DM value, which are modelled as chromatic red noise. The residual DM delay is modelled as 

    Δ_ DM∝√(P(f; A_ DM, γ_ DM))×ν^-2

where ν is the observing radio frequency. The values of A_ RN, γ_ RN, A_ DM and γ_ DM used in the simulations are  reported in Table <ref>, together with the number of Fourier components used in the modelling of these processes. These values – amplitude, slope and number of Fourier components – correspond to maximum likelihood values obtained from the single pulsar noise analysis of EPTA DR2full. An example of simulated data compared to the real EPTADR2 dataset is shown in Fig. <ref>.






  §.§.§ Gravitational wave signal injection


Having specified the properties of the adopted PTA, we now turn to the description of the injected GW signals. The signal produced by a SMBHB is composed by two terms, customarily referred to as Earth term (ET) and pulsar term (PT), which are respectively associated to the GW strain hitting the Earth at the ToA of the considered pulse and the strain hitting the pulsar at the time of emission of the same pulse.

Throughout the paper we will always assume circular SMBHBs, evolving under the effect of GW backreaction only (i.e. we neglect any possible coupling with the local stellar or gaseous environment). Under these assumptions, the delays induced in the ToAs at an epoch t are fully defined by 8 parameters: the chirp mass ℳ[Note that in the quadrupolar approximation employed here for the injected waveform, the signal does not depend on the individual binary masses.], the observed frequency f_gw, the redshift z, the inclination angle , the sky location (θ, ϕ) – respectively polar and azimuthal angle – , the initial phase ϕ_0 and the polarization angle ψ. Following <cit.>, we model them as:


    s(t, Ω̂) = F^+(Ω̂)Δ s_+(t) +F^×(Ω̂)Δ s_×(t),

where 

    Δ s_A(t) = s_A(t_p) - s_A(t_e),

the functions F^A(Ω̂) – with A indexing the polarization and Ω̂ being the unit vector pointing from the GW source to the Solar System Barycenter ˆ
(SSB) – are the antenna pattern functions that depend on the relative position of the pulsar and the source <cit.>. s_A(t_p) and s_A(t_e) are the PT and ET respectively. They both take the form


    s_+(t) = ℳ^5/3/d_Lω(t)^1/3(-sin(2(Φ(t)-Φ_0))(1+cos^2)cos(2ψ) 
       - 2cos(2(Φ(t)-Φ_0))cos sin(2ψ))
       s_×(t) = ℳ^5/3/d_Lω(t)^1/3(-sin(2(Φ(t)-Φ_0))(1+cos^2)sin(2ψ) 
       +2cos(2(Φ(t)-Φ_0))cos cos(2ψ)),

where 

    Φ(t) = Φ_0 + 1/32ℳ^5/3(ω_0^-5/3-ω(t)^-5/3),

and 

    ω(t) = (ω_0^-8/3-256/5ℳ^5/3t)^-3/8.

The only difference between PT and ET is that while the latter is the delay induced at the time of observation on Earth t_e, the former has to be evaluated at the time of emission of the radio pulse at the pulsar t_p = t_e - L(1 + Ω̂·p̂), where L is the pulsar distance and p̂ is the pulsar position in the sky. We inject in our data two different GW components: a stochastic GWB and a resolvable CGW. Since the evolution of the signal over the observation time is small, the phase approximation is used both in the injection and in the recovery: ω_ Earth = ω_0 at each epoch t.

Stochastic GWB. The GWB is injected as a realistic superposition of GWs coming from a cosmic population of SMBHBs. The injection pipeline starts from a catalog of SMBHBs, computes the delays induced by each binary in each pulsar ToA according to Eq. (<ref>) and then sums all the delays in the time domain. The distribution of the sources' ℳ, f_gw, z and  determines the observed power spectrum generated by the SMBHBs population: the binned total characteristic strain, where the frequency bin width is Δ f = 1/T with T the time span of the pulsar array, can be evaluated as <cit.>

    h_c^2(f_i) = ∑_j∈Δ f_ih_j^2(f_r)f_r/Δ f_i,

where f_i is the central frequency of the bin Δ f_i, and the sum runs over all the binaries for which the observed frequency f=f_r/(1+z)∈Δ f_i, where f_r is the GW frequency in the binary rest frame. The strain of each GW signal is given by 

    h(f_r) = 2√(1/2(a()^2 + b()^2))(Gℳ)^5/3(π f_r)^2/3/c^4d(z),
 
where a() = 1+cos^2, b() = -2cos and d(z) is the comoving distance of the binary, which is computed as 

    d(z) = c/H_0∫_0^z1/√(Ω_m (1+z)^3 + Ω_Λ)dz.

For the population of circular GW driven SMBHBs, assumed here, the characteristic strain given by Eq. (<ref>) can be approximated as a power law. It is therefore customary to describe the signal as

    h_c(f) = A_ GW(f/ yr^-1)^α,

where α=(3-γ)/2=-2/3 being γ the PSD induced in the timing residual (cf. Eq. <ref>).

The power spectrum h_c^2(f_i) thus depends on the number of emitting sources per unit redshift, mass and frequency d^3N/(dzdℳdf_r). A list of ∼ 120k binaries is constructed by extracting ℳ, f_gw, z as Monte Carlo samples of a numerical distribution d^3N/(dzdℳdf_r), obtained from the observation-based models described in <cit.> and <cit.>. In those papers, the authors construct hundreds of thousands of SMBHB populations in agreement with current constraints on the galaxy merger rate, the relation between SMBHs and their hosts, the efficiency of SMBH coalescence and the accretion processes following galaxy mergers. From this set of models, we selected 100 populations producing a GWB amplitude A ∼ 2.4×10^-15, consistent with the signal observed by EPTA <cit.>. The other parameters of the binaries are then randomly sampled such that sources are isotropically distributed in the sky – cos θ∈Uniform(-1, 1) and ϕ∈Uniform(0, 2π) –, their orbital angular momenta are uniformly distributed on the unit sphere – cos ι∈Uniform(-1, 1) –, the phase and the polarization angle are random – ϕ_0 ∈Uniform(0, 2π), ψ∈Uniform(0, π). 


Resolvable CGW. In some of the dataset, an individually resolvable CGW is injected. The waveform is again injected in each pulsar in the time domain residual according to Eq. (<ref>). The only difference, compared to the SMBHBs contributing to the unresolved GWB is that the amplitude of the signal is large enough that the source can be identified above the pulsar noise and the unresolved GWB by a signal search employing a deterministic filter to match the signal (see next Section). We stress that we always inject both the ET and the PT for every SMBHB, regardless of whether it provides an unresolvable contribution to the GWB or it is a resolvable CGW.

Datasets. For the purpose of this investigation, seven different datasets were produced, with the exact same noise realization, but different gravitational signals:

    
  * dataset injGWB01, injGWB02, injGWB03. These three datasets contain a GWB-only. They are produced injecting three background selected from the 100 populations mentioned above. The three populations are chosen among those displaying a smooth behaviour in the lowest frequency bins, without particularly loud sources.

    
  * dataset injCGW5, injCGW20. These two datasets only contain a single loud CGW-only. The injection procedure is similar to the GWB datasets, but the population file used contains only one binary. The two injected sources have Earth term GW frequencies of 5nHz (injCGW5) and 20nHz (injCGW20). The parameters of the 5nHz source are chosen to approximately reproduce frequency and amplitude of the candidate CGW found in the EPTA DR2new, <cit.>, which in fact has a frequency of ∼5nHz and an amplitude h∼1.1× 10^-14. The source is then located in the Virgo cluster; this choice determines sky location and distance, and consequently the chirp mass. Inclination, initial phase and polarization angles are randomly extracted as in the GWB case. The source injected in injCGW20 has identical parameters except for the higher frequency (20nHz instead of 5nHz), thus resulting in a higher strain (h∼ 2.7× 10^-14). The delays induced in pulsar PSR J1022+1001 by these two CGWs are shown in Fig. <ref>.
    
    
  * dataset injGWB03+CGW5, injGWB03+CGW20. These two datasets are injected with a GWB and a CGW. The GWB is in both cases the one resulting from population 03, while the CGWs are either at 5nHz or 20nHz.










 §.§ Signal modelling and analysis methods

All the analyses are carried using the  <cit.> package. The starting point of the analysis is the likelihood function of the timing residuals δt

    P(δ𝐭|ξ, ζ) = exp[-1/2(δ𝐭 - 𝐌ϵ - 𝐬)^T𝐂^-1(δ𝐭 - 𝐌ϵ - 𝐬)]/√((2π)^ldet𝐂).

The term Mϵ is the timing model error expressed as a linear function of the offset from the nominal timing model parameters, ϵ. ξ are the timing model parameters and ζ are the stochastic components parameters. The likelihood function sampled to search for the gravitational signal is marginalized over the timing model parameters as described in <cit.>. 𝐂=𝐂(ζ) is the covariance matrix that contains all the contributions from stochastic processes, which are:

    
  * WN, modelled according to Eq. (<ref>). All the white noise parameters are fixed at the injected values in the GW searches;
    
  * RN, modelled as a Gaussian process with amplitude of the Fourier components given by Eq. (<ref>)  (<cit.>;
    
  * DM, modelled as a Gaussian process with amplitude of the Fourier components given by Eq. (<ref>) and amplitude depending on the observed radio frequency (see Eq. <ref>). As for RN, the DM is not present in all the pulsars and, if present, the number of Fourier bins used in the Gaussian process depends on the pulsar and it is equal to the number of components used in the injection. For a summary of the noise model characteristics, see Table <ref>.
    
  * common red noise (CRN), if searched for. A red noise process which is common to all the pulsars can be accounted for in the covariance matrix. If included in the analysis, it is also modelled as a Gaussian process. The amplitude of the components are modelled in two ways: as a power law function of the frequency, and therefore the parameters are the slope γ_ CRN and the amplitude A_ CRN; or as a binned spectrum, and thus the parameters are the amplitudes of the common process in each frequency bin f_i. In both cases, the number of components used in the Gaussian process is 30.
    

Although we inject the GWB as  a superposition of unresolved SMBHBs, we follow the standard PTA analysis practice and model it in our likelihood function as a CRN characterized by an amplitude and a spectral index, as in Eq. (<ref>), which can be directly translated in a characteristic strain of the form given by Eq. (<ref>). Note that, depending on the dataset, we use two different flavours of CRN. In most searches, the GWB is modelled as a common uncorrelated red noise (CURN). This means that inter-pulsar correlations induced by the GWB are neglected and the GWB posterior distribution is then reconstructed a posterirori with the reweighting technique <cit.>, when possible. The advantage of this model is that it is much faster since it keeps the correlation matrix in the likelihood function diagonal and it naturally provides the Bayes factor of the reconstructed model versus the  sampled model, besides the posterior distribution of both models. However, the reweighting technique is not always efficient and, as we will see later, using a CURN to model a GWB is not always appropriate; therefore, when needed, we use the HD-correlated CRN model which we referred to as GWB model. In this case, GWB induced inter-pulsar spatial correlation are also modelled, according to the HD curve <cit.>

    Γ_ab=3/21-cosθ_ab/2ln( 1-cosθ _ab/2)-1/41-cosθ_ab/2 +1/2

where a and b are indexes identifying two pulsars in the array and θ_ ab is their angular separation in the sky. Note that searching for an HD-correlated CRN is much more computationally expensive, since the correlation matrix is not diagonal anymore.

The last term in the likelihood function is s, which is the delay induced by an individually resolvable CGW, and is analytically described by Eqs.(<ref>)–(<ref>) which are implemented in the CGW model of . An approximation that can be adopted is to consider in the CGW model the ET only, neglecting the PT. This is computationally very convenient, since the former only depends on the 8 source parameters. Conversely, the latter also depends on the distance of each pulsar in the array. Since those  distances are generally known with large uncertainties (of the order of 20%), they must be considered as additional free parameters in the search. Therefore, the complete model has at least one more parameter per each pulsar with respect to the ET only model. Moreover, as prescribed in <cit.>, in order to ease the search of the best parameters, one extra parameter is added per each pulsar, which is an initial phase at the pulsar, Φ_p. We will assess the performance of an ET only search on realistic signals including ET and PT in Section <ref>.

We mainly adopt Bayesian techniques in our analysis, although we also exploit the frequentist optimal statistics, when useful. The priors used in the search are listed in table <ref> and are all uninformative, except for the pulsar distances' priors that exploit information obtained from independent electromagnetic observations. The prior distributions have a width which is taken as the measurement error listed in <cit.>; for the pulsars that are not included in this paper, the relative error is fixed at 20%. The mean of the prior is chosen such that the pulsar distance used for the injection is within 1σ of the prior distribution for each pulsar. Parameter estimation is then performed by sampling the posterior distribution with the Markov
Chain Monte Carlo (MCMC) sampler , <cit.>. For the 25 pulsar PTA we employ, the dimensionality of the posterior distribution is 62 (2 parameters – slope and amplitude – for each red noise and dispersion measure process included in the model) plus the GW model dimension (2 for the GWB power-law model, 8 or 58 for an ET only or a full ET+PT CGW model search respectively). Model selection is done by computing the Bayes factor, either with the reweighting technique, when it is efficient, or with the product space method implemented in  <cit.>.


































  


§ RESULTS


Using the methods described in the previous section, we carried out several investigations. In Section <ref>, we assess the importance of modelling a CGW including both the Earth and pulsar terms. In Section <ref>, we investigate the degeneracy between the analysis results obtained with a GWB-only dataset and with a CGW-only dataset. In Section <ref>, it is shown how CGW-only datasets can produce strong evidence in favour of an HD-correlated GWB. Viceversa, in Section <ref>, it is shown how a GWB-only dataset can be misinterpreted as a continuous wave. This degeneracy is solved in Section <ref>, by means of a combined analysis that searches simultaneously for a GWB and a CGW on both GWB-only and CGW-only datasets . Finally, in Section 
<ref>, we evaluate the performance of a combined analysis of a GWB and a CGW on datasets containing both signals. 






 §.§ On the importance of the pulsar term in CGW modelling





















As described in section <ref>, the signal imprinted by a CGW in the ToAs is the sum of two terms: the Earth term and the pulsar term. The ET is common to all pulsars and only depends on the 8 parameters describing the source. Conversely, the PT depends on the distance of each pulsar and the phase of the GW signal at the pulsar's location. 
This means that modelling of the full CGW ET+PT in a search performed on an array of 25 pulsars (like the one we are considering here) requires 50 extra free parameters, that must be added to the 8 source parameters and to the 62 parameters of the individual pulsars noise components. It is therefore interesting to compare searches of CGW modelled as ET only and ET+PT on realistic datasets, to see whether the computational efficiency of the ET only search bears any significant drawback.


The two templates have been tested on both CGW-only datasets (injCGW5 and injCGW20), and the main results are shown in Fig. <ref>. From the figure it is clear that the performance of the ET only template depends on the CGW frequency. In fact, when applied to the injCGW20 dataset (right panel of Fig. <ref>), the ET only model performs well on all parameters that it is sensitive to, i.e. all source parameters except the chirp mass and the luminosity distance. Conversely, the same model applied to the injCGW5 dataset results in some non-negligible biases in the recovery. As it can be seen from the left panel of Fig. <ref>, the injected values for the frequency and sky location are not compatible with the 90% C.I. of the recovered posterior distributions. 

The different performance of the ET only template in the two cases is due to the different degree of separability of the ET and PT terms for the two injected sources. For the injected system chirp mass, ℳ=1.16· 10^9M_⊙, the average frequency difference between ET and PT is Δ f∼ 
3.8×10^-6nHzyr^-1Δ t∼ 1.0× 10^-2 nHz∼ 0.02Δ f_bin for the 5nHz CGW and Δ f∼ 1.4× 10^-3nHzyr^-1Δ t∼ 4.0 nHz∼ 3.2Δ f_bin for the 20nHz CGW. Therefore, while in the 20nHz CGW, ET and PT falls at very distinct frequencies, they blend together in the same frequency bin for the 5nHz CGW. In the former case, the ET only template only picks the common power produced by the ET at its frequency, disregarding the PT term; while in the latter, the ET only template is trying to fit a superposition of ET and PT essentially overlapping at the same frequency. This concept is demonstrated in Fig. <ref>, where the reconstructed waveforms are compared to the injected ones. In the 20nHz case (injCGW20 dataset), the ET only template almost perfectly fits the ET of the full injected waveform, while in the 5nHz case (injCGW5), the ET only template absorbs power from both ET and PT and the reconstructed waveforms sits, for many pulsars, somewhat in between the ET and the full injected signal and is also way more uncertain as shown by the much larger credible region.





Conversely, the full ET+PT template is able to correctly reconstruct the signal and, because it is sensitive to the frequency evolution, it also allows to constrain both the source chirp mass and the luminosity distance. The performance is very different for the 5 nHz CGW and the 20 nHz CGW: for the former the chirp mass recovery is ℳ=0.52^+9.98_-0.48× 10^8M_⊙, while for the latter it is ℳ=1.01^+0.07_-0.06×10^9M_⊙, where we give the median and the 68% credible interval. The superior recovery of the chirp mass in the 20 nHz case is due to the much larger frequency evolution of the source between ET and PT, allowing to place a much tighter constrain on ḟ and thus on M.
Another consequence of the large frequency evolution of the 20 nHz CGW is that the model is sensitive to the pulsar distances. This is shown in Fig.<ref> where the reconstructed waveform (left) and the posterior distribution of the pulsar distance (right) are shown for PSR J1713+0747 for the injCGW20 (top row) and injCGW5 (bottom row) datasets. In the first case, the posterior distribution on the distance is narrower than the prior and centred at the injected value: this is allowed by a correct modelling of the frequency evolution, which is reflected in the excellent reconstruction of the waveform. Conversely, for the 5 nHz case, the posterior distribution of the pulsar distance is almost indistinguishable from the prior, but signal reconstruction still matches the injection, as the frequency evolution is too small for the search to be sensitive to it. We note that the GW signal reconstruction and distance determination is not always as good for the 20 nHz CGW. This is shown in the middle row of Fig.<ref> for pulsar PSR J1918-0642. In this case the posterior distribution resembles the prior, and while the reconstructed signal does not strongly disagree with the injection, the median fails to capture the signal shape accurately. In fact, only about 13 pulsars, whose evolution is accurately reconstructed, significantly contribute to the chirp mass recovery.



In addition to better and unbiased recovery of the CGW parameters, there is another rationale for preferring the complete CGW template over the ET-only template: in a joint CGW + GWB search, the contribution to the CGW signal due to the pulsar term, if not taken into account by the CGW model, is absorbed by the GWB model, since it is a common red noise process. It will therefore add extra spurious power to the GWB, biasing the recovery of its parameters. An example of this behaviour is shown in Fig. <ref>, where the GWB+CGW search is applied to the dataset injCGW5: when the CGW is searched with the full template, the posterior distribution of the GWB parameters is unconstrained, when the search is done with the ET-only template, the posterior distribution of the GWB parameters remains well constrained because it absorbs all the power coming from the unmodelled pulsar term.

We therefore conclude that while the ET-only template can be used to search for possible CGW candidates and provides a reasonable estimate of the source parameters <cit.>, the full ET+PT model must be preferred, especially in a joint search with a stochastic GWB. Despite the much lower computational efficiency, we will therefore always model our CGWs as ET+PT in the following.











 §.§ Degeneracy between GWB and CGW

In this section, searches for GWB, CGW and combined searches of the two signals together are applied to the 3 GWB realizations (injGWB01, injGWB02, injGWB03) and to the 2 CGW realizations (injCGW5, injCGW20). The aim is to study the degeneracy between the two signals and how it can be solved.





  §.§.§ CGW misinterpreted as a GWB


We start by performing a GWB analysis on the three injGWB and the two injCGW datasets. To this end we make use of both Bayesian and frequentist techniques following this procedure:

    
  * a power law-shaped and HD-correlated stochastic process is assumed for the GWB, then the slope γ_ GWB and the amplitude log_10A_ GWB of the powerlaw process are sampled, together with the noise parameters to obtain the posterior distribution;
    
  * the Bayes factor ℬ^ GWB_ CURN is evaluated to compare the evidence for an HD-correlated and an uncorrelated red noise process. The bayes factor is computed either with the reweigthing method or with the product space when reweighting is not efficient;
    
  * the averaged angular correlation between the pulsars is estimated with the optimal statistics method averaging over 10 angular bins, each one containing 30 pulsar pairs. A powerlaw-shaped background with variable γ is assumed;
    
  * the SNR of the HD cross-correlation marginalized over the noise parameters is obtained with the optimal statistics method, assuming a powerlaw-shaped background with variable γ.


1.4 





The results of the powerlaw spectrum parameter estimation are listed in the last two columns of Table <ref> for all five datasets and two examples of the recovered GWB parameters' posterior distributions are shown in Fig. <ref> for datasets injGWB02 and injCGW5. In general, the GWB search returns well constrained posteriors regardless of the actual content of the dataset, which in practice means that a CGW can easily be detected as (and mistaken for) a GWB. However, the signal recovered from the two CGW-only datasets (injCGW5, injCGW20) is higher and flatter than the one obtained from the background datasets. This can be clearly seen both from the A and γ values of Table <ref> and from the 2D contours in Fig. <ref>. This is expected since the extra power due to the two single sources is at frequencies which are higher than the loudest GWB frequency bins that dominate the powerlaw spectrum recovery in the GWB-only datasets.






The Bayes factors reported in Table <ref> show that an HD-correlated common process (i.e. a GWB) is generally strongly favoured regardless of the content of the data. In fact, dataset injCGW5 produces a Bayes factor ℬ^ GWB_ CURN of ∼ 10 and HD S/N of 3.0^+0.5_-0.5 (fixing γ to 3), while injCGW20 
gives ℬ^ GWB_ CURN of ∼ 60 and S/N = 3.2^+0.6_-0.6 (fixing γ to 2). Interestingly, injGWB03 does not favour an HD correlated process, displaying ℬ^ GWB_ CURN of ∼ 0.15. This is consistent with <cit.>, who found that, due to the stochastic nature of the signal, individual realizations of the same SMBHB population can result in Bayes factor that differ by several orders of magnitude.

Although the Bayes factors obtained on the  injCGW5 and injCGW20 are compatible with a GWB detection, the resulting HD S/N is significant only in a small range of values of γ that are consistent with the posterior distributions obtained by the Bayesian search, as shown in Fig. <ref>. By fixing γ to the nominal value of 13/3, the HD S/N drops to 1.3^0.4_-0.4 and to 1.1^1.0_-0.9 for the injCGW5 and injCGW20 datasets respectively. Conversely, the 
injGWB01 and injGWB02 datasets (also shown in Fig. <ref>) result in distributions of the HD S/N that are consistently high for all values of γ≥ 4.


Moreover, at higher values of γ, the CGW-only dataset has an increasing evidence for a dipolar-correlated signal, whose S/N surpasses that of HD. This behaviour is not observed in the GWB-only datasets, for which the dipole S/N always stays lower than the HD S/N. Finally, two examples of HD reconstruction are shown in Fig. <ref> for datasets injGWB02 and injCGW5. While the quadrupolar pattern can be seen in both datasets, the scatter around the expected relation appears to be much larger when a CGW only is present in the data, consistent with <cit.> and <cit.>. Moreover, the HD correlation for the injCGW5 dataset is reconstructed assuming γ=3, while it tends to disappear for higher values of γ.














  §.§.§ GWB misinterpreted as a CGW


We now search for a CGW in the same 5 datasets analysed in the previous section. In the search, we estimate the posterior distributions of all the noise parameters – except for the WN, which is fixed – and the CGW parameters – 8 source parameters plus 50 additional parameters for the pulsars' distances and phases. We then use hypermodeling to evaluate the Bayes factor between a model containing a CGW versus a model featuring intrinsic pulsar noise only. 

We can see from Table <ref> that also in this case, regardless on whether the dataset contains a CGW or a GWB, a CGW is recovered with high confidence, with a Bayes factor ℬ^ CGW_ PSRN>1000 in all cases. Table <ref> also shows that the CGW frequency and amplitude are fairly well constrained. The performance of the CGW search on the injCGW datasets has already been assessed in Section <ref>. As shown in Fig. <ref> when a full ET+PT template is used for the CGW, the model can correctly recover the injected signals and estimate their parameters, although with differences depending on the frequency of the source. Here it is particulary interesting to check the CGW recovery on the three injGWB datasets, which is shown in Fig. <ref>: all source parameters are fairly well constrained, including the sky location, which is different in each dataset. These parameters, however, do not correspond to any specific source present in the datasets, as can be seen in the three panels of Fig. <ref>. The recovered CGW frequency always corresponds to the frequency bin that is the most constrained by the HD-correlated free spectrum search; moreover, the amplitude recovered is always compatible with the GWB amplitude in that frequency bin. Thus, it can be concluded that the CGW parameters are constrained because the CGW model absorbs part of the HD-correlated common red noise produced by the incoherent superposition of all the SMBHBs composing the GWB. 

Finally, we note that the CGW amplitudes recovered from the three injGWB datasets are respectively 1.2^+1.5_-0.7× 10^-14, 1.4^+0.6_-0.5× 10^-14  and 1.8^+0.5_-0.4× 10^-14 (median and 68% credible interval). These values are all well compatible with the amplitude of the DR2new CGW candidate, which is 1.0^+0.5_-2.6× 10^-14 (median and 90% credible interval) <cit.>.














  §.§.§ Breaking the degeneracy: combined search for a GWB and a CGW




We now perform a joint CGW + GWB search on the five datasets considered thus far. We perform two flavours of combined searches: (i) common uncorrelated red noise plus CGW (CURN+CGW), and (ii) HD-correlated common red noise plus CGW (GWB+CGW). The CGW is always modelled as the sum of ET and PT, and together with the signal parameters, all the RN and DM parameters are also searched over.

The results of these two searches are reported in Table <ref> for all datasets, while posterior distributions of selected parameters of the model components are shown in Fig. <ref> for datasets injGWB02 and injCGW5. In figure, orange and light purple distributions are obtained respectively from the GWB+CGW and CURN+CGW searches, and results of GWB-only and CGW-only searches are also shown for comparison. To see how searching for both type of signals simultaneously allows to properly characterize the actual content of the data, let us discuss the injGWB and injCGW separately. 

In the injGWB cases, when a common red noise component is added to the model alongside the CGW, the evidence for the latter strongly decreases. This can be seen both from the drop of the Bayes factors in the first three lines of Table <ref>, and from the posterior distributions of the CGW parameters in the left panel of  Fig. <ref>. In Section <ref>, when searching for a CGW on a dataset containing only a GWB, we found  ℬ^ CGW_ PSRN>1000. This is because the GWB produces a common noise that the model prefers to assign to the CGW signal (common to all pulsars) rather than to the individual pulsars' RN components. 
When a CURN is included in the model, ℬ^ CURN+CGW_ CURN drops significantly but is still slightly larger than unity, suggesting that the presence of CGW component is still preferred. This can also be seen in the posterior distribution of the CGW amplitude shown in purple in the left panel of  Fig. <ref> that, while featuring a long tail that extends to the lower end of the prior range, still shows a significant peak around logA≈-14.3. This behaviour is shared by all three injGWB datasets and is due to the fact that the GWB manifests itself primarily as a common red noise process, with a weaker, emerging 'off diagonal' HD-correlated component. The CURN models it as a common red noise process, but is unable to account for the emerging HD correlation, which the model then assigns to the CGW.
Conversely, when a GWB is added to the model, we get ℬ^ GWB+CGW_ GWB<1 (first three lines of Table <ref>), strongly suggesting that the GWB component is now able to describe the full content of the data. This is also clear by looking at the CGW posterior parameters in   the left panel of  Fig. <ref>. The orange distributions show that both the frequency and amplitude posteriors are almost perfectly flat, with the latter displaying a cut-off, which we find to be at a similar value for all three realizations. From this we can extract  a 95% upper limit of the CGW amplitude which is ≈4.0×10^-15, ≈3.8×10^-15 and ≈4.6×10^-15 for datasets injGWB01, injGWB02 and injGWB03 respectively. Note that these upper limits are lower than the GWB level, since the total GW content of the dataset is correctly accounted for by the GWB component. Finally, the recovered source is not localized in the sky, with the posterior distributions of the sky location almost uniform across the sphere, consistent with the prior. 







Differently from the CGW parameters, the GWB parameters' posterior distributions are not significantly affected by the addition of the CGW component to the model: the GWB posterior distributions obtained from the GWB+CGW search and with the GWB-only search are well consistent with each other (see left panel of Fig. <ref> for the injGWB02 dataset and table <ref> for all the datasets) confirming again that the GWB component of the model is correctly accounting for all the HD-correlated power. Only a small variation can be noticed in the CURN+CGW recovery of the background parameters: the posterior distributions are slightly wider with a longer low amplitude tail. This is because the CURN cannot absorb any HD correlated power, which is (spuriously) assigned to the CGW component.  


 
Conversely, when the same analysis is applied to the injCGW datasets, the posterior distributions of the CGW parameters are always well constrained, both with the CURN+CGW and with the GWB+CGW search, as shown in the right panel of Fig. <ref>. Searching for CGW-only, CURN+CGW or GWB+CGW, in this case, gives consistent results for the CGW parameter, with no significant changes in the posterior distributions' shape and width (see right panel of Fig. <ref>). Moreover, GWB+CGW is able to recover the sky location with a precision which is comparable to the one obtained with the CGW-only search and the model comparison strongly favours the GWB+CGW model over the GWB-only model (see table <ref>). As expected, the GWB posterior distribution is instead completely unconstrained when the model applied in the analysis contains both a common red noise and a CGW: the power that generates the GWB signal recovered with the GWB-only search is now correctly completely absorbed by the CGW component (see Fig. <ref>).





  §.§.§ Noise-only datasets

As a further check, we tested the analysis pipelines on a dataset containing only the noise and no GW signal, neither deterministic nor stochastic. The noise realization is the one contained in all the datasets analysed. In this case, neither a GWB nor a CGW is detected. As for the GWB, we find a value of the Bayes factor ℬ^ HD_ CURN∼1.5 and the posterior distributions of A and γ are poorly constrained, with a 95% upper limit on the GWB amplitude of ∼ 4.47× 10^-15. As for the CGW, the Bayes factor ℬ^ CGW_ PSRN is ∼ 0.5 and the CGW parameters are totally unconstrained, with the amplitude showing a 95% upper limit of ∼4.5×10^-15. Although many more noise realizations should be tested to draw robust statistical conclusions, these tests show on fully realistic datasets that current PTA detection pipelines are unlikely to return a spurious detection in absence of an actual GW signal in the data.





 §.§ Joint GWB+CGW analysis on combined CGW+CGW datasets


In this section we present the results of simultaneous searches for a GWB and a CGW, applied to datasets injGWB03+CGW5 and injGWB03+CGW20. We remind the reader that both datasets contain the same noise and GWB realisation from dataset injGWB03. 

In each dataset a loud CGW is superimposed to the GWB: in one case the CGW is at 5nHz (injGWB03CGW5), in the second one it is at 20nHz (injGWB03CGW20). This is done in order to see how the joint analysis performs in presence of a slowly evolving source (the 5nHz CGW) and on a fast evolving one (the 20nHz CGW), and how the two of them interact with the GWB.









  §.§.§ Bias induced in the GWB recovery by the presence of a resolvable source in the SMBHBs population

As already discussed by <cit.> and  <cit.>, the presence of a particularly loud source among the systems contributing to the GWB, if not accounted for in the analysis, can significantly bias the recovery of the overall GWB signal. As a preliminary step in our investigation, we therefore perform a GWB only search on the injGWB03CGW5 and injGWB03CGW20 datasets. As expected, the analysis produces a biased recovery of the GWB parameters: the results are shown in the right panels of Fig. <ref> and <ref>: in both cases, the recovered background is flatter and higher than the injected signal due to the extra power introduced by the loud source. In the two following sections, we demonstrate how this bias can be solved by performing a joint analysis of the GWB and the CGW components.





  §.§.§ Results for the injGWB03+CGW5 dataset


















The results presented here are obtained by sampling the posterior distributions of the CURN+CGW model and then reweighting them to obtain the posteriors of the GWB+CGW model.
The results of the combined search applied to the dataset injGWB03+CGW5 provides good overall results, with the majority of the CGW parameters recovered within 2σ of the injected values. 

The posterior distributions of the CGW parameters are shown in the left panel of Fig. <ref>, while the recovery of the background parameters is shown in the right panel. The recovery of the CGW parameters is successful: the frequency and the sky localization of the source are well constrained – the width of the posterior distributions is ≲10% of the prior width for all three parameters – and compatible within 68% with the injected values: injected frequency is f = 5.02nHz, recovered frequency is f = 5.04^+0.04_-0.04nHz; injected sky location is (12^ h19^ min50^ s, 12.38^∘), recovered sky location is (12^ h 33^+20  min_-19, 10^+5_-5^∘). However, the 3D source location cannot be properly constrained, since the luminosity distance posterior distribution is  extremely wide due to its strong covariance with the chirp mass. Both posterior distributions are roughly bimodal, with a narrow peak close to the injected value – in particular for the chirp mass – and a long, bumpy tail extending to low values of luminosity distance and chirp mass. Note that the posterior of the luminosity distance spans the interval (0.02, 24)Mpc within the 90% credible region; for such a nearby putative source, ancillary information coming from galaxy catalogues can effectively enable the identification of the source host <cit.>. The inclination angle, initial phase and polarization angle are well constrained and compatible with the injected values.

Focusing now on the GWB parameters, the posterior distribution in the right panel of Fig. <ref> shows how accounting for the resolvable source in the model solves the bias in the GWB parameters' recovery. In fact, the posterior distribution obtained with the combined search is well consistent with the one obtained with the GWB-only search applied to the GWB-only dataset injGWB03: they peak at similar values and have comparable widths, showing that the parameters of the stochastic and of the deterministic components are not significantly correlated and the interplay between the two signals does not affect the search.





  §.§.§ Results for the injGWB03+CGW20 dataset













 





The joint search applied to the injGWB03CGW20 dataset unveils some new interesting behaviour of the recovery pipeline. In fact, in this case, when a joint CURN+CGW (ET+PT) search is performed and the GWB is obtained with the reweighting technique, the estimate of the GWB parameters is completely off, as demonstrated by the filled purple distributions shown in the right panel of 
Fig. <ref>. The result is indeed quite similar to a GWB search only (shown in red), meaning that a significant fraction of the CGW power is absorbed in the common background component and the CURN+CGW search followed by reweighting fails in this case, producing strong biases in the results.



The reason of this failure is the following: as shown in Section <ref>, due to the significant difference in the 20nHz CGW between the Earth and the pulsar term GW frequency, in order to correctly account for both terms with the CGW model, the majority of the 50 PT parameters (i.e. 25 pulsar distances and 25 individual PT GW phases) have to be constrained. This is a difficult task for the sampler, which in the CURN+CGW case converges to an easier, alternative solution: the pulsar terms are all absorbed by the common uncorrelated red noise component, which is possible because they are indeed a common sinusoidal process that is uncorrelated, since the pulsars are all at different sky locations and distances from Earth. This justifies the much higher and flatter recovered background shown in the right panel of Fig. <ref>: the recovered amplitude is 1.83^+0.28_-0.26×10^-14, while the injected value is 2.4×10^-15. 

This interpretation is confirmed by the recovered CGW parameters (not shown). Most of them are biased and in particular the chirp mass, which determines the frequency evolution of the signal, is much lower than the injected value: the obtained value is 6. 9^+18.0_-5.0×10^6 M_⊙, which corresponds to a 95% upper limit on the frequency derivative of 4.2×10^-6nHz·yr^-1, ∼300 times smaller than the ḟ of the injected SMBHB. This is consistent with the fact that the values of the CGW parameters are such that the recovered time domain signal matches the injected Earth term and completely neglects the pulsar term contribution. This is confirmed by the shape of the reconstructed CGW shown in  Fig.<ref>. In the top panel we can see that in the CURN+CGW search the recovered CGW waveform for pulsar PSR J1713+0747 reproduces exactly the injected ET contribution. In fact, being HD-correlated, in presence of a CGW component in the model, the Earth term is not absorbed by the CURN component. Indeed, the recovered background is slightly lower than the one obtained by a GWB-only search on the same dataset (shown by the red distributions in the right panel of Fig. <ref>), because in that case both the Earth and pulsar terms are absorbed by the background component. 

We stress that this bias indeed corresponds to the maximum of the posterior distribution. This can be explained by the fact that the CURN model does not exactly match the injected common red noise, which is HD correlated, and therefore it prefers to absorb the pulsar term, since it is much cheaper to describe it as a two-parameter CURN signal, rather than to determine the 2×25 parameters describing each individual sinusoidal pulsar term.

In this dataset, the injected signal components are  properly recovered only by a direct GWB+CGW (ET+PT) search, which is more computationally expensive due to the off-diagonal HD terms that must be computed when evaluating the correlation matrix in Eq. (<ref>). In fact, in a GWB+CGW search the pulsar term is not absorbed by the HD-correlated common red noise component and both signals are properly reconstructed. In the right panel of Fig.<ref> the GWB recovery obtained by this search is shown in filled purple: it agrees very well with that obtained with the GWB-only template applied to the GWB-only dataset (empty purple), although it is slightly larger, which can be explained by the fact that the GWB+CGW model contains 58 more parameters describing the single source. A tiny correlation with this high number of parameters can easily explain the less precise recovery of the background parameters.

The correct recovery of the GWB parameters is a direct consequence of the correct recovery of the CGW parameters. In the left panel of Fig. <ref>, the posterior distributions of the 8 source parameters are shown and in Fig. <ref> the median and 90% credible region of the recovered CGW signal is shown for pulsar PSR J1713+4707. From the recovery, it can be seen how the model is able to account for the frequency evolution of the signal: the frequency recovered is 19.99^+0.21_0.18nHz (injected is 20nHz) and the chirp mass 1.14^+0.11_0.10×10^9M_⊙ (injected is 1.16× 10^9M_⊙). This allows the luminosity distance to be much better constrained than in the low frequency case, with a median and 68% credible interval of 32.7^+15.1_-9.3Mpc. Since the injected value is 21.6Mpc, the value obtained is a mild overestimate, which is probably due to the anti correlation with the inclination angle and the initial phase, which are both slightly underestimated. The 2D sky location recovery is also improved with respect to the 5nHz case, resulting in a declination angle of 12^ h38^+17 min_-16 and a right ascension of 11.5^+3.0_-3.4^∘.

  


§ DISCUSSION AND CONCLUSIONS


In this paper, we have carried out the first comprehensive study of the interplay between stochastic GWBs and deterministic CGWs in realistic PTA datasets, assessing the performance of currently used analysis pipelines. To this end we constructed mock datasets featuring all the complexity of EPTA DR2full <cit.>; we simulated up to 24.8 years of unevenly sampled, multi frequency ToAs for the 25 pulsars of DR2full and injected WN, RN and DM consistent with the maximum likelihood values found in the DR2full single pulsar analysis <cit.>. The injected GW signals were also as realistic as possible, tailored to mimic the findings of the recent EPTA analyses <cit.>. A GWB with nominal amplitude of A_ GW=2.4× 10^-15 was injected as the incoherent sum of individual waves coming from observationally based cosmic populations of SMBHBs <cit.>, while the deterministic CGW was modelled according to the candidate signal identified in the analysis of EPTA DR2new <cit.>. In all cases, SMBHBs were assumed to be circular and both the ET and PT of the GW induced time delays were injected. These GW signals were searched with the standard models implemented in the  analysis suite. The GWB was therefore modelled as a common red process characterized by a single power law spectrum, while the model CGW matched the one used for injecting the signal (i.e., circular, GW driven SMBHB), either including only ET or ET+PT. With this set-up we carried a number of investigations, whose results are summarized in the following.

We first compared the performance of ET-only and ET+PT templates on two datasets containing only a CGW either at 5nHz (injCGW5) or at 20nHz (injCGW20). In both cases, the ET+PT template was able to fully recover the signal, providing an unbiased estimate of all its parameters, including chirp mass, distance and sky location. Things were not so straightforward when the ET only template was employed. In this case, the 5nHz CGW dataset resulted in a biased recovery of the frequency and the sky location, whereas in the 20nHz case, the recovery was unbiased. In fact, in the latter case Earth and pulsar term can be easily disentangled and the ET-only template correctly matches the ET component of the signal. Conversely, in the 5 nHz case, ET and PT fall in the same frequency bin and the ET template tries to fit a mixture of the two, leading to a bias in the signal reconstruction. In both cases, the ET-only model is not sensitive to the chirp mass and therefore to the luminosity distance, preventing 3D source localization.

We also stress that in both cases, modelling ET only leaves behind some unmodelled residual (e.g. the full PT in the 20 nHz case), which will likely be absorbed by the GWB template in a joint search, leading to a biased recovery of the background parameters. These are the reasons why the complete CGW template (ET+PT) must be preferred to the ET-only, even if this means having 2 times the number of pulsars extra parameters (50 in our case) and therefore much larger computational costs for the sampling.

We therefore adopted the full ET+PT template for the CGW and turned to the investigation of datasets in which only one of the two types of signal was injected (either a GWB or a CGW) and the analyses were performed separately with one model at the time (eiter a GWB or a CGW). We analysed 5 datasets: 3 realizations of a realistic GWB (dataset injGWB01,injGWB02, injGWB03) and 2 realizations of single resolvable sources (datasets injCGW5, injCGW20). All datasets contained the same noise realization. These searches showed that a GWB can easily be misinterpreted as a CGW and vice versa. In fact, all sources returned evidence in favour of either GW signal and well constrained recovery of the model parameters. However, when a combined search is performed – meaning that the model used for the recovery contains both an HD-correlated GWB and a CGW – this degeneracy is broken: when only a GWB is present in the data, the posterior distributions of the CGW parameters obtained with the combined search are unconstrained and the CGW amplitude shows an upper limit; on the contrary, when a CGW is the only signal injected, the combined search gives unconstrained posterior distributions for the GWB parameters. No CGW evidence is absorbed by the HD-correlated background if the single source is present in the data. Therefore, the combined search is able to correctly recover the signal that is actually present in the data. Focusing on GWB-only datasets, we highlight that the CGW evidence is totally absorbed when an HD-correlated GWB is present in the model, while it does not completely disappear if only a CURN component is modelled. In this case, even if poorly constrained, the CGW parameters posterior distributions still peak at specific values. Therefore, modelling the common red noise as HD-correlated is necessary to totally absorb the spurious CGW. These results are similar to the ones obtained with the real EPTA DR2new data <cit.>: CURN + CGW searches peaking at a CGW amplitude of ∼10^-14, and peak disappearing once the background is modelled as HD-correlated and the Bayes factor ℬ^ GWB+CGW_ GWB being of order unity. Our simulations show that this behaviour arises when a CGW is indeed not present in the data, strengthening 
  the conclusion reached in 
the EPTA DR2 analysis.

We then analysed datasets containing both a stochastic GWB and a CGW, either at 5 nHz (injGWB03+CGW5) or at 20 nHz (injGWB03+CGW20). The low frequency CGW is an almost monochromatic source, while the high frequency one is drifting and the signal frequency significantly changes between the  PT and the ET. Performing a combined search on these types of datasets is particularly important because a single resolvable source, if not properly taken into account in the search, will induce a bias in the GWB parameters recovery <cit.>.

The results of the combined searches indicate that the bias in the GWB recovery induced by the presence of the single source is resolved when the CGW is correctly accounted for by the deterministic CGW model, so long as it contains both ET and PT. This result is obtained with both datasets, injGWB03CGW5 and injGWB03CGW20. However, in the injGWB03CGW5 case, the same unbiased result can be obtained with two methods: by performing a CURN+CGW search and then reweighing to obtain the HD+CGW posterior distribution, and by directly sampling the HD+CGW posterior distribution. The first method is computationally advantageous, as the CURN model is considerably faster to compute than the HD one. In the injGWB03CGW20 case, however, the first method (reweighting) is unable to recover the unbiased posterior distribution, as the CGW template solely accounts for the ET contribution, while the PT is more easily absorbed by the CURN. The posterior distribution on the CURN parameters is therefore strongly biased with respect to the GWB injection. Performing a direct sampling of the HD+CGW model, on the contrary, gives an unbiased recovery of the GWB parameters: the posterior distributions of the CURN and HD parameters are consequently disjoint, which makes the reweighing technique extremely inefficient. Therefore, while computationally much more expensive, the only method that is able to converge to unbiased results is the direct HD+CGW search. As for the GWB, the CGW recovery is successful, with most of the parameters constrained and compatible with the injection.

Given the recent PTA results indicating the presence of a gravitational signal in the data, it is crucial to be able to distinguish between the two types of signal that can be produced by SMBHBs. This work represents a first thorough study of this problem. Despite the limited parameter exploration, which is constrained by the high computational cost of the analyses, the investigation decisively points towards the importance of performing combined searches. Indeed, in the cases analysed, combined searches were able to both identify which signal was present in the data, without misinterpretation, and to guarantee an unbiased recovery of the GWB parameters in the presence of a loud individual source in the SMBHB population. Looking ahead, it will be important to extend this analysis to richer and more sensitive datasets, investigating the possibility of disentangling multiple searches, also in the presence of eccentricity or environmental coupling. In fact, the high amplitude of the currently observed signal implies that the future SKA might be able to resolve several tens of individual SMBHBs at S/N>5 <cit.>. Being able to successfully identify and disentangle those signals will have invaluable payouts for the understanding of SMBHB evolution and dynamics in the context of multimessenger astrophysics. Given the complexity and high dimensionality of these sources, it is of paramount importance to design more efficient search techniques <cit.>.


       I.F. thanks Stanislav Babak for his hospitality and mentoring at the APC, where this investigation was initiated.
       We thank the B-Massive group at Milano-Bicocca University for useful discussions and comments. A.C., A.S., G.M.S. acknowledge the financial support provided under the European Union’s H2020 ERC Consolidator Grant “Binary Massive Black Hole Astrophysics” (B Massive, Grant Agreement: 818691). A.C. acknowledges financial support provided under the European Union's Horizon Europe ERC Starting Grant "A Gamma-ray Infrastructure to Advance Gravitational Wave Astrophysics" (GIGA).

  

aa







 Paper Title: An ALMA survey of submillimetre galaxies in the Extended Chandra Deep Field South: an unbiased study of SMG environments measured with narrowband imaging.



firstpage–lastpage
Chemical abundance gradients of organic molecules within a protostellar disk
    [
    
============================================================================




§ ABSTRACT 
Submillimetre galaxies (SMGs) are some of the most extreme star-forming systems in the Universe, whose place in the framework of galaxy evolution is as yet uncertain. It has been hypothesised that SMGs are progenitors of local early-type galaxies, requiring that SMGs generally reside in galaxy cluster progenitors at high redshift. 
We test this hypothesis and explore SMG environments using a narrowband VLT/HAWK-I+GRAAL study of  and  emitters around an unbiased sample of three ALMA-identified and spectroscopically-confirmed SMGs at z ∼ 2.3 and z ∼ 3.3, where these SMGs were selected solely on spectroscopic redshift.
Comparing with blank-field observations at similar epochs, we find that one of the three SMGs lies in an overdensity of emission-line sources on the ∼4 Mpc scale of the HAWK-I field of view, with overdensity parameter δ_g = 2.6^+1.4_-1.2. A second SMG is significantly overdense only on ≲ 1.6 Mpc scales and the final SMG is consistent with residing in a blank field environment. 
The total masses of the two overdensities are estimated to be 
log(M_h/ M_⊙) = 12.1–14.4, 
leading to present-day masses of 
log(M_h,z=0/ M_⊙) = 12.9–15.9. 
These results imply that SMGs occupy a range of environments, from overdense protoclusters or protogroups to the blank field, suggesting that while some SMGs are strong candidates for the progenitors of massive elliptical galaxies in clusters, this may not be their only possible evolutionary pathway.





galaxies: star formation – galaxies: photometry – galaxies: evolution – submillimetre: galaxies









§ INTRODUCTION



Since their discovery more than two decades ago, submillimetre galaxies <cit.> have proven to be important laboratories for exploring galaxy formation and evolution. 
These galaxies are identified in (sub)millimetre surveys and have typical infrared (IR) luminosities of L_ IR∼ 10^12-13 L_⊙ corresponding to star formation rates of ∼10^2-3  yr^-1 <cit.>. 
They are massive, with stellar masses of M_∗∼ 10^10-11   <cit.>, 
dust masses of ≳10^8   <cit.>, 
and cold gas masses of ∼10^11  <cit.>, 
but with gas depletion times of just a few hundred Myr <cit.>. 
The SMG redshift distribution peaks at z ∼  2.5 <cit.>, making these massive dusty galaxies the most intense star-forming systems in the Universe during its peak epoch of star formation <cit.>. 
SMGs contribute up to ∼20 percent of the cosmic star-formation rate density at z ∼ 2 <cit.>.

The extreme properties of SMGs has long made them a good test of galaxy evolution models <cit.>, yet questions about their evolution and role in the evolution of other galaxies remain. 
SMGs have similar properties to those expected of the progenitors of local massive elliptical galaxies, which formed most of their stars in short bursts at z ≳ 2 <cit.>. Indeed, the dust emission from SMGs is typically compact <cit.>, which is consistent with a scenario in which a gas-rich z≳2 galaxy undergoes a compact starburst, leading to a compact quiescent galaxy, which eventually evolves into a local elliptical galaxy <cit.>. 
Since local ellipticals are predominantly found in galaxy clusters <cit.> then if SMGs are indeed a progenitor phase in their formation, then it is expected that SMGs should reside in early galaxy clusters, or `protoclusters', at  z ≳ 2. 

Galaxy protoclusters <cit.> are typically defined as structures that will collapse and virialise to form a galaxy cluster by z = 0. Simulations have shown that in a Λ CDM universe, protoclusters form hierarchically at the highest density regions of the matter distribution in the universe <cit.> at z ∼ 4–6 <cit.>. As such, protoclusters are characterised by overdensities of galaxies relative to the average galaxy density in the coeval blank field. Conversely to their present-day descendants, galaxies in a protocluster are generally not bound to a single halo; they instead occupy large structures extended over megaparsec (Mpc) scales, with the main halo containing as little as 20 percent of the member galaxies <cit.>.

Unfortunately, observationally identifying protoclusters is challenging. Methods of detecting galaxy clusters from their X-ray emission <cit.> or by searching for their imprint on the cosmic microwave background at millimetre wavelengths <cit.> via the Sunyaev-Zel'dovich effect <cit.> are rendered impractical due to the lack of a hot intracluster medium (ICM). Similarly, searches for high concentrations of passive early-type galaxies occupying a tight `red sequence' in colour-magnitude space <cit.> become ineffective since the stellar populations of galaxies in protoclusters typically have not evolved sufficiently for a significant 4000 Å break to be detected. 
Consequently, the majority of protocluster surveys resort to searching for overdensities of galaxies at high redshift. Such searches depend on the existence of accurate redshift information across large cosmological volumes, and several protoclusters have been discovered serendipitously through large spectroscopic surveys <cit.>. In lieu of expensive large-scale spectroscopic observations, an alternative method is to use wide-field narrowband photometric surveys to search for overdensities of galaxies with strong emission lines at a particular observed-frame wavelength <cit.>. 

Whether SMGs commonly inhabit protoclusters or protocluster-like environments is as yet uncertain. Several examples of SMGs residing in protoclusters have been documented <cit.>, but these systems were selected for detailed follow-up because of prior evidence of high galaxy densities. I.e. they comprise a highly biased subset and therefore cannot be used to make inferences about the general SMG population. 

Clustering studies have been used to obtain statistical measurements indicative of the whole SMG population. Results from single-dish clustering measurements suggest that on average SMGs at z ∼ 2.5 reside in dark matter halos of mass ∼10^13  <cit.>. 
This is marginally lower than expected for the progenitors of massive ellipticals, and implies that SMGs are instead more likely to evolve into 2–3L^∗ galaxies in groups and small clusters. 
However, these halo mass measurements have typical uncertainties of ∼0.5 dex due to the difficulties associated with obtaining accurate photometric redshifts for SMGs. Furthermore, these clustering measurements relied on the statistical identification of optical/near-IR counterparts to submillimetre sources detected in low-resolution single-dish surveys, which are incorrect in ∼30 percent of cases and incomplete in a further ∼30 percent <cit.>. 
More recently, <cit.> and <cit.> measured the clustering of SMGs which had been followed up interferometrically with the Atacama Large Millimetre/submillimetre Array (ALMA). 
Using a small sample of 50 ALMA-identified SMGs with spectroscopic redshifts and employing forward modelling to correct for incompleteness, <cit.> estimated halo masses that are a factor of ∼ 3.8 lower than other studies of SMGs.  
From a significantly larger parent sample, <cit.> selected a complete sample of ∼350 ALMA-identified SMGs with photometric redshifts to measure halo masses consistent with the results from the single-dish studies, particularly at z>2 <cit.>. 
Overall, the picture from clustering measurements is complex, and differing results from different studies may be methodological, due to sample selection or cosmic variance. Other ways of measuring the environments of SMGs are required.

Statistical photometric redshifts have identified galaxy overdensities around ∼5–60 percent of SMGs <cit.>, but these are subject to significant selection biases <cit.>, and few overdensities have been spectroscopically confirmed.
Instead, in order to determine the nature of a `typical' SMG environment, and thus confirm whether SMGs really are the progenitors of massive elliptical galaxies in local clusters, we need targeted observations of individual SMGs, but with no prior knowledge of their environments in order to avoid biases. To this end, we have conducted a wide-field narrowband survey of the environments of three SMGs identified as part of the ALESS project <cit.>,
in which follow-up observations of submillimetre sources detected in the LABOCA ECDFS Submillimetre Survey <cit.> were conducted using ALMA. We search for overdensities of  or  emitters around these three SMGs to assess whether they reside in protocluster-like environments. The target SMGs were selected on the basis of redshift only, and with no prior information about their environments. 
Our method is similar to that employed by <cit.>, which combined narrowband photometry with submillimetre observations to identify an overdensity of  emitters around two SMGs in SSA 13. However, the SMGs targeted by <cit.> were already known to be closely grouped with each other and three optically-faint radio-galaxies. Our study is the first to perform such an analysis around a sample of  SMGs that are selected without prior knowledge of their environment.

The structure of this paper is as follows: in 2 we describe the SMG sample selection, our observations and data reduction, and the identification of candidate companion galaxies for each target SMG; 3 includes our main results, analysis and discussion; our conclusions are presented in 4. Throughout this paper we use a ΛCDM cosmology with Ω_M = 0.315, Ω_Λ = 0.685 and H_0 = 67.4 km s^-1 Mpc^-1 <cit.>. 
All magnitudes are presented in the AB system, where a 1 μJy source has a magnitude of 23.9 <cit.>.





§ OBSERVATIONS AND GALAXY IDENTIFICATION



In this study we use the High Acuity Wide-field K-band Imager <cit.> on the Very Large Telescope (VLT) to investigate the environments of three ALMA-identified SMGs from ALESS. 
As part of ALESS, extensive follow-up was undertaken to obtain spectroscopic redshifts of the SMGs <cit.>, 
which enables a search for galaxies that share environments with these submillimetre sources. 
The wide-field imaging capabilities and narrowband photometric filters of HAWK-I allow for an efficient emission-line survey of their environments, which are expected to span physical scales on the order of several Mpc if consistent with being protoclusters <cit.>.




 §.§ Sample selection


The blank-field LABOCA ECDFS Submillimetre Survey (LESS) observed  0.5×0.5 degrees in ECDFS with  APEX/LABOCA  and detected 126  sources at >3.7σ at 870- <cit.>. Each of these sources was followed-up with ALMA to yield the 131 ALESS sources described in <cit.>, divided into a main catalogue of 99 SMGs and a supplementary catalogue of 32 SMGs. The SMGs in the main catalogue all lay within the ALMA primary beam full width at half-maximum (FWHM) of the highest-quality maps, while those in the supplementary catalogue were either extracted from outside the primary beam or from lower-quality maps <cit.>. 

Spectroscopic redshifts were obtained for 52 of the 131 ALESS SMGs by <cit.>,  
and targets for our study are selected from these 52 ALMA-identified and spectroscopically-confirmed SMGs. 
We require SMGs with spectroscopic redshifts that shift the  or λ5007 emission lines into the wavelength coverage of the HAWK-I  filter; this requires that the SMGs are located at z = 2.299±0.023 or z=3.324±0.060. Of the 52 ALMA-identified SMGs with spectroscopic redshifts from <cit.>, five (ALESS 6.1, 75.2, 87.1, 102.1, and 112.1) have spectroscopic redshifts within the desired range for Hα. A sixth SMG (ALESS 5.1) has a CO-derived spectroscopic redshift of  z = 3.303 <cit.>, which places  in the  coverage.

These six SMGs were the proposed targets for observations in four HAWK-I pointings (PID: 0103.A-0668). The six SMGs were selected purely based on their spectroscopic redshifts, with no consideration of their environments. Of the four pointings, only two were  observed during the service-mode observations and the choice of pointings was random. The two observed pointings contain three of the six proposed targets: ALESS 5.1, ALESS 75.2, and ALESS 102.1, whose spectroscopic redshifts are z = 3.303, 2.294, and 2.296, respectively. 
Details of these three targeted SMGs are provided in Table <ref>. A total of 16 other ALESS SMGs lie within the two HAWK-I pointings, but these are not considered in this study as their redshifts are such that no bright emission lines are expected in the narrowband filter. 
Indeed, those that are detected in our HAWK-I observations fail to meet our criteria for being emission-line galaxies (see <ref> and Figure <ref>).








 §.§ HAWK-I data


Each pointing was imaged using the HAWK-I instrument <cit.> on the VLT in the  (central wavelength: λ_c = 2.146 ; FWHM: Δλ = 0.324 ) and  (λ_c = 2.165 ; Δλ = 0.030 ) filters <cit.>. The FWHM of the  filter is equivalent to Δ z = 0.046 (0.060) at z ∼ 2.3 (3.3), corresponding to a velocity width of Δ v = 4200 km s^-1. All three observing blocks (OBs) for the field containing ALESS 5.1 and 75.2 (hereafter Pointing 5+75) were executed on 2019 August 21, while the OBs for the field containing ALESS 102.1 (hereafter Pointing 102)  were split among three separate nights from 2019 August 21 to 2020 January 01. The total exposure times for Pointing 5+75 (Pointing 102) were 7.2 ks (6.6 ks) and 900 s (660 s) in the  and  filters, respectively. 
Individual exposures of 120 s () and 30 s () were 
taken using the “HAWKI img obs AutoJitter” template, with five random 
dither positions within a 20 box for each filter in each OB. 
Each pointing covers a 75 × 75 area, except for a cross-shaped gap of width 15 between the detector's four 2k × 2k Hawaii 2RG arrays. Using HAWK-I's GRAAL system <cit.>, we achieved point spread functions (PSFs) of ∼04 in  and ∼03 in  (see Table <ref>). 

The data were reduced using a custom Python-based pipeline, with each of the four detector chips treated separately. Briefly, the pipeline begins by dark-subtracting the data and subsequently using twilight flats to perform flat-fielding. We then use SExtractor <cit.> to detect sources in each of the flattened frames and produce individual masks. 
A final flat field is produced for each frame by median combining all masked frames from the same OB except the frame being flattened; the frames are then flattened using their unique final flat fields. 
The astrometry of each flattened frame is then calibrated by using scamp <cit.> to match our detected sources with sources detected in a reference K-band image, correcting for any distortions across the field of view by fitting a third-order polynomial. The reference images used for Pointing 5+75 and Pointing 102 were taken from the Taiwan ECDFS NIR Survey <cit.> and the MUltiwavelength Survey by Yale-Chile <cit.>, respectively. Different reference images were required for each pointing because while TENIS is deeper and has higher resolution, roughly a quarter of the Pointing 102 field of view lies outside of the TENIS coverage. Finally, the astrometrically-corrected frames were median combined using SWarp <cit.>. The resultant stacks in both bands were photometrically calibrated using MUSYC  data <cit.> such that they all had a zeropoint magnitude of 30.0 mag and ensure a median ( – ) colour of 0.



Source detection and photometry were conducted using SExtractor <cit.> operating in dual-image mode; the stacked  images were used to identify the positions of sources, and then photometry was extracted at these positions in both the  and  images to ensure that any difference between the measured  and  photometry is a purely intrinsic property of the sources and not caused by positional offsets. After masking noisy regions near the edges of the stacked images, we detected a total of 2175 sources in Pointing 5+75 and 1754 in Pointing 102. Apertures with a diameter of 125 were used, as this is large enough to contain the majority of the flux for all detected sources while minimising the amount of additional background noise captured. The 3σ limiting magnitudes measured in these 125 diameter apertures are provided for each filter and each HAWK-I detector chip in Table <ref>. To account for the variation in size of the detected sources, we then estimated total magnitudes in each filter by selecting all bright (m_K_s < 19.5) sources and (for each pointing separately) calculating the median difference between their fixed-aperture magnitudes and their magnitudes measured by SExtractor in adaptively-scaled <cit.> apertures (mag_auto; ); this difference was then added to the fixed-aperture magnitudes of all sources in the pointing to obtain their total magnitudes. 
The choice to use the same aperture size for all sources and apply a correction (as opposed to simply using the mag_auto values) ensures that estimates of the total magnitudes are self-consistent whilst also closely matching existing photometry in the same band. 







 §.§ Ancillary data


There exists a wealth of photometric data in the ECDFS, which  supplements our HAWK-I photometry. Archival TENIS <cit.>, MUSYC <cit.> and HAWK-I (Zibetti, priv. comm.)  data were collated by <cit.> (hereafter S14) and then used to calibrate our astrometry and photometry (see <ref>). 

In <ref> and <ref> we fit spectral energy distributions to galaxies in our sample in order to first derive properties such as photometric redshift, stellar mass and star formation rate. For the photometric redshifts we use <https://github.com/gbrammer/eazy-py> – an updated version of the photometric redshift code eazy <cit.> written in Python (see <ref>) – while for the other galaxy properties we use  <cit.>.
To this end, we also make use of existing ECDFS images spanning the ultraviolet (UV) to mid-infrared (MIR; see <ref>). 
These images were sourced either from the MUSYC 2010 Public Data Release <cit.> or from TENIS <cit.>. 
The MUSYC dataset consists of UU_38BVRI broadband images from the Wide Field Imager (WFI) on the MPG/ESO 2.2m telescope <cit.>; 5000 Å narrowband and z^' broadband imaging from the Mosaic-II camera on the CTIO Blanco 4m telescope <cit.>, with JK_s broadband imaging from the Infrared Sideport Imager on the same telescope <cit.>; 18 medium-band (IA427, IA445, IA464, IA484, IA505, IA527, IA550, IA574, IA598, IA624, IA651, IA679, IA709, IA738, IA767, IA797, IA827, IA856) images taken with the Subaru telescope's Suprime-Cam <cit.>; Spitzer/IRAC images at 3.6, 4.5, 5.8 and 8.0  <cit.>H-band data are also available but our pointings are not covered.. 


We also make use of spectroscopic redshifts in the ECDFS from studies whose areas overlap with our pointings, obtained from publicly-available composite catalogues<https://www.eso.org/sci/activities/garching/projects/goods/MasterSpectroscopy.html>^,<http://member.ipmu.jp/john.silverman/CDFS_vlt.html> <cit.>. The spectroscopic redshifts used are from the VIMOS VLT Deep Survey <cit.>; the GOODS/VIMOS survey <cit.>; the Extended Chandra Deep Field-South Survey <cit.>; and <cit.>. 
Additionally we utilise the results of the spectroscopic study conducted as part of ALESS by <cit.>.

Finally, we make use of the <cit.> Chandra point source catalogue for the identification of AGN in our final sample (see <ref>).





 §.§ Emission line galaxy selection


Star-forming galaxies at the same redshifts as our target SMGs (z ∼ 2.295 ± 0.023 for ALESS 75.2 and ALESS 102.1; z ∼ 3.324 ± 0.030 for ALESS 5.1) have emission lines that are redshifted into the narrow wavelength coverage of the  filter. Since the  filter is near the centre of the  transmission a galaxy without line emission at these wavelengths will have a (–) colour of zero. However, due to the narrow width of the  filter relative to the  filter, galaxies with redshifts that place an emission line in the narrow  filter will  have a (–) colour that is significantly greater than zero. 
We employ the same methodology as previous narrowband surveys <cit.> to identify line-emitting galaxies. This methodology uses two parameters to select sources with a significant, physically-driven narrowband excess, as opposed to an excess due to random noise. 

The first of these parameters, Σ, quantifies the significance of the narrowband excess compared to the expected random scatter for a source with zero (–) colour <cit.>. Σ is given by:


    Σ = 1 - 10^-0.4(BB - NB)/10^-0.4(ZP - NB)√(rms_NB^2 + rms_BB^2) ,


where NB and BB are the apparent magnitudes in the narrowband () and broadband () filters, respectively; ZP is the zeropoint magnitude of the narrowband images; rms_NB and rms_BB are the rms counts in 125 apertures for the individual narrowband and broadband quadrants, respectively. 
We require candidate line-emitters  have Σ > 3, which is consistent with previous narrowband studies <cit.>; see Figure <ref>. Note that this Σ does not correspond directly to signal-to-noise (SNR) in the  filter, but is a separate quantity based on counts; Σ > 3 implicitly excludes sources with SNR ≲ 8 in  <cit.>. 

In addition to having Σ > 3 line emitters are required to have an observed equivalent width (EW) > 50Å. The EW is calculated for each source using:


    EW = Δλ_Brγf_Brγ - f_K_s/f_K_s - f_Brγ (Δλ_Brγ/Δλ_K_s) ,


where Δλ_Brγ and Δλ_K_s are the widths of the two filters and f_Brγ and f_K_s are the flux densities of the source in each filter. The 50 Å lower limit on EW for a source to be selected as a line emitter was chosen to lie above the 3σ scatter in (–) colours for bright ( > 19.5 mag) sources in both pointings (Figure <ref>).




Before applying the selection criteria, we first account for sources that are detected with ≥3σ significance in the  filter but <3σ in . We classify these sources as non-detections in , and replace their aperture magnitudes with the relevant 3σ limiting magnitude (see Table <ref>). However, several of these non-detections have counterparts in the S14 catalogue (within a 1 matching radius) and thus have  magnitudes from either TENIS <cit.>, archival HAWK-I observations (Zibetti, priv. comm.), or MUSYC <cit.>. For these sources, we replace our HAWK-I  photometry with values from one of these surveys, preferentially using TENIS photometry as it is the deepest of the three (with a limiting 3σ magnitude of m_ lim^3σ = 24.45 mag); if no TENIS photometry is available then we opt for the archival HAWK-I values (m_ lim^3σ = 24.36 mag), using MUSYC (m_ lim^3σ = 22.55 mag) only when no photometry exists for either of the other two. Note that while MUSYC  observations are the shallowest of all the data considered here (including our own), there are 11 sources for which only MUSYC photometry is available. However, all of these sources reside in regions of Pointing 102 that are (a) outside of the coverage of the TENIS and archival HAWK-I observations, and (b) close to the quadrant edges in our HAWK-I observations where the noise is at its greatest.


Using the Σ > 3 and EW > 50 Å selection criteria, 81 and 80 candidate line emitters are identified in Pointing 5+75 and Pointing 102, respectively (Figure <ref>). Of these candidates, 30 are  non-detections with no  photometry in the S14 catalogue, and thus Σ and EW are calculated by assuming that their  magnitudes are equal to the 3σ limiting magnitudes of our data. Since this can only provide a lower limit for the (–) colour and thereby underestimate Σ for these sources, we do include these sources in our sample of candidate line emitters. 

Finally, we visually inspect all 161 candidate line emitters, removing stars/quasi-stellar objects and image artefacts. 
The final sample consists of 79 and 68 candidate line emitters in Pointing 5+75 and Pointing 102, respectively (147 sources in total).






 §.§ Identifying line emitters associated with the SMGs


Narrowband excess alone is not sufficient to identify star-forming galaxies in the same environments as our target SMGs; such an excess could be caused by a number of possible emission lines at different redshifts (see Figure <ref>). We therefore use the available multi-band photometric data covering our pointings to estimate photometric redshifts for the narrowband emitters in our sample. The S14 catalogue contains photometric redshift estimates for sources across the ECDFS, however after cross-matching with our data (using a matching radius of 1), a significant fraction (>30 percent) of the line emitters identified in <ref> do not have broadband counterparts in this catalogue and thus lack any redshift information. 
We therefore perform our own spectral energy distribution (SED) fitting using .

To maximise the number of sources for which we can derive photometric redshifts, we extract fixed-aperture photometry at their HAWK-I  positions in the UV-to-MIR images described in <ref>. 
Each image is astrometrically calibrated using scamp <cit.> and SWarp <cit.> to match the astrometry of our HAWK-I images, and then photometrically recalibrated so that all images have a zeropoint magnitude of 30.0 mag. Photometry is extracted in fixed apertures using the photutils Python package <cit.>; apertures of diameter 20 are used for all images except those from Spitzer/IRAC, for which we use apertures of diameter 38 due to the larger point spread function (PSF). 
Aperture corrections are determined for each filter by measuring the median difference between the magnitudes measured in these apertures and those measured in adaptively scaled apertures with SExtractor for bright point sources. 
Final corrections are applied to each filter to account for Galactic attenuation, using values from <cit.> and <cit.>.




 operates using a χ^2-minimisation procedure in which linear superpositions of template SEDs are tested at different redshifts to find an optimal fit to the observed fluxes <cit.>. In keeping with other recent studies which implement  <cit.>, we use the “tweak_fsps_QSF_12_v3” set of 12 template SEDs, which cover a wide range of galaxy types and utilise a <cit.> initial mass function (IMF) and a <cit.> dust attenuation law while assuming solar metallicity. An advantage of these templates is that they include emission lines, such that a narrowband excess can provide a relatively tight constraint on the redshift. 

As discussed in <ref>, there have been several spectroscopic studies in the ECDFS, from which spectroscopic redshifts have been obtained for a number of galaxies across the field. Using a matching radius of 15, we cross-match our data with catalogues from VVDS <cit.>, the GOODS/VIMOS survey <cit.>, the ECDFS spectroscopic survey <cit.>, and the spectroscopic studies conducted by <cit.> and <cit.>. This gives spectroscopic redshifts for 163 (∼4.1 percent) of the 3929 sources detected in our HAWK-I imaging, including nine for which photometric redshifts could not be estimated due to insufficient photometry. Seven of the 163 sources with spectroscopic redshifts are emission line galaxies selected in <ref>; the spectroscopic redshifts for these sources are used for our analyses. We compare the photometric and spectroscopic redshifts for our HAWK-I sources in Figure <ref>. There is strong agreement between the photometric and spectroscopic redshifts, which is quantified using the normalised median absolute deviation (NMAD) of Δ z:


    σ_NMAD = 1.48 ×median(|Δ z - median(Δ z)/1 + z_spec|) ,


where z_spec is the spectroscopic redshift and Δ z = z_spec - z_phot. We obtain σ_NMAD = 0.062 when considering all 152 HAWK-I detections with photometric and spectroscopic redshifts. 





Only five (∼3.4 percent) of our 147 emission line galaxies have neither spectroscopic nor photometric redshifts, the latter being due to a lack of photometry with sufficient depth. Figure <ref> shows the redshifts of the remaining 142 emission line galaxies, compared with the significance of their narrowband excess (Σ; Equation <ref>). 
Peaks in the redshift distribution are visible at z ∼ 2.3 (both pointings) and z ∼ 3.3 (Pointing 5+75 only), as expected of  and  in the environments of the target SMGs. We select as  () emitters any galaxies for which 2.23 < z < 2.37 (3.23 < z < 3.41), where these redshift ranges correspond to 3× the FWHM of the  filter when  () has redshifted to the centre. We represent these selection criteria with shaded regions in Figure <ref>; the highlighted galaxies are henceforth assumed to be   and  emitters at similar redshifts as the target SMGs. 
We identify 44  emitters and 4  emitters in Pointing 5+75, and in Pointing 102 there are  11  emitters ( emitters in Pointing 102 are not further considered because there is no SMG at z ∼ 3.3 in this pointing). 
Table <ref> summarises the results of each step in the sample selection. We note that all of these  and  candidates have an SNR > 8.5 in the  filter as a natural consequence of our selection process (see also <ref>). We therefore do not expect the sizes of these samples to be significantly affected by Eddington bias <cit.>.




To identify any AGN in the sample we use a 1 matching radius to locate counterparts in the <cit.>  Chandra point source catalogue. None of the  emitters and only one  of the  emitters (2.3%) is an X-ray luminous AGN, which is consistent with the rate of X-ray AGN in blank-field surveys of  emitters at the same redshift <cit.>. Since the AGN fraction is the same as in field surveys then this galaxy is kept in our sample to enable a fair like-for-like comparison between the SMG fields and blank field  emitters. 




Figure <ref> shows the distributions of emission line galaxies across the HAWK-I pointings, with  and  emitters highlighted. For all three target SMGs, the companion galaxies are spread across the entire field of view and therefore span several physical Mpc. This is consistent with expectations from simulations <cit.>, in which protoclusters are seen to extend over several Mpc, such that the entire structure is unlikely to be captured by a single HAWK-I pointing. We also note the presence of a dense clump of seven  emitters (three of which are spectroscopically confirmed at z ∼ 2.3) in the northeast of Pointing 5+75, which coincides with a photometrically-identified  blob at z ∼ 2.3 <cit.>. This system will be discussed further in <ref>. 

Of the three SMGs targeted, only ALESS 102.1  is identified as an  (or ) emitter in our data. <cit.> did not identify  ALESS 102.1 as an  emitter in their spectroscopy, because the wavelength coverage with VLT/FORS2 and VLT/VIMOS does not cover  at z∼2.3. 
The original spectroscopic redshift for ALESS 5.1 is from CO(4-3) <cit.> and no emission lines were observed in Keck/DEIMOS, Keck/MOSFIRE or VLT/XSHOOTER observations <cit.>; this is likely because the redshifted  line clashes with a bright OH^- sky line for this source <cit.>. For ALESS 75.2 the original spectroscopic redshift was measured, in part, thanks to a faint  line detected in Keck/MOSFIRE observations <cit.>, which is below the detection limit of our data.









§ RESULTS, ANALYSIS & DISCUSSION





 §.§ Measuring luminosity functions


In order to quantify whether the SMGs reside in significant overdensities of  or  emitters, a comparison to the blank field needs to be drawn. The High Redshift (z) Emission Line Survey <cit.> is a large narrowband survey of emission line galaxies, including  emitters at z = 2.23 <cit.> and  emitters at z = 3.24 <cit.> in COSMOS. The HiZELS results are therefore representative of  and  emitters in regions of average density at redshifts similar to those of our target SMGs, and we use their luminosity functions as a blank-field sample for comparison with our results.

To construct luminosity functions for our  and  emitters we bin them according to line luminosities, making corrections to the observed number counts in each bin to account for completeness, contamination from other emission lines, dust attenuation, and the shape of the narrowband filter profile. Each of these steps is described in more detail below.



  §.§.§ Survey volumes


Approximating the  filter profile as a top-hat function with width equal to the FWHM of the filter (Δλ = 0.030 ), the  emission line should be detectable from z_min = 2.276 to z_max = 2.322, corresponding to a comoving volume per square degree of 5.86×10^5 cMpc^3 deg^-2. The redshift range within which  can be detected extends from z_min = 3.294 to z_max = 3.354, which gives 1.04×10^6 cMpc^3 deg^-2. Accounting for the high-noise regions that were masked prior to source detection (see <ref>) and the gaps between the HAWK-I detector chips, the surveyed areas in Pointing 5+75 and Pointing 102 are 0.0117 and 0.0118 deg^2, respectively. The volumes probed are therefore: 6859 cMpc^3 for  in Pointing 5+75; 6891 cMpc^3 for  in Pointing 102; 12180 cMpc^3 for  in Pointing 5+75; 12230 cMpc^3 for  in Pointing 102. 
In <ref> we correct the derived luminosity functions to account for the fact that the  filter is not a perfect top-hat function, which leads to the volume probed being slightly different for sources with different luminosities. 




  §.§.§ Completeness correction


It is possible that real galaxies with weak emission lines were missed in our selection process (<ref>) despite actually meeting the selection criteria: the sample is incomplete at low emission line fluxes. We correct for this using the method employed by <cit.>, applying it separately for each quadrant of each pointing due to the variation in depth between detector chips (see Table <ref>). For each emission line ( and ), we select sources that failed to meet the emission line galaxy selection criteria (i.e. sources for which EW < 50 Å and/or Σ < 3) with redshifts within the range used to identify the targeted emission line (see <ref>). Due to the size of these samples, we generate ∼1000 mock galaxies by randomly varying the  and  magnitudes of the selected galaxies according to their uncertainties (assuming a Gaussian probability distribution for each magnitude) and placing them at random positions within their quadrant, removing any sources for which these changes result in them being classed as a line emitter. We then artificially inject line flux to each galaxy in this bolstered sample of non-line emitters, beginning with 10^-22 erg s^-1 cm^-2 and incrementally increasing it by 0.05 dex. Line fluxes are calculated as


    F_line = Δλ_Brγf_Brγ - f_K_s/1 - (Δλ_Brγ / Δλ_K_s) ,


where f_Brγ and f_K_s are the  and  flux densities, respectively, in units of erg s^-1 cm^-2 Å^-1. With each increment of injected line flux, we recalculate the EW and Σ and reapply the line emitter selection criteria to determine the catalogue completeness as a function of line flux. This is used to estimate the completeness corrections for our luminosity functions. The uncertainty in the completeness at a given line flux is estimated by regenerating the mock galaxies 1000 times and measuring the standard deviation in the completeness across all iterations. 





  §.§.§ Removing  contamination


The  emission line lies in between a doublet of  lines at rest-frame wavelengths of 6548 Å and 6583 Å, which will contribute to the measured  flux density and therefore affect the observed EW and emission line flux. Using spectroscopic data taken with Subaru/FMOS and Keck/MOSFIRE, <cit.> observed an anti-correlation between the λ6583-to- line flux ratio and the rest-frame EW (EW_rest = EW / (1 + z)) for the  emitters in HiZELS, deriving the following empirical relation:


    F_[Nii]/F_Hα = -0.296 ×log_10(EW_rest,Hα+Nii) + 0.8 .


We adopt this relation to apply corrections to the line fluxes of all  emitters in our sample, resulting in a median decrease of 11^+9_-5 percent in  emission line flux. 




  §.§.§ Relative contributions from λ5007, λ4959 and 


Thus far only the λ5007 emisson line has been considered in the discussion of  emitters at z ∼ 3.3. However, this line is part of a doublet, with its counterpart residing at a rest-frame wavelength of 4959 Å, and there is a narrow range of redshifts (z = 3.336–3.344) in which both lines can contribute to a galaxy's  flux. Furthermore, while the  emission line is sufficiently separated from the  doublet to avoid contaminating the measured  line flux, it is still close enough such that there is the potential for  emitters to be misidentified as  emitters (see Figure <ref>). Therefore, rather than try and separate our sample into λ5007, λ4959 and  emitters, we present a combined + luminosity function; this also allows for a consistent comparison with the blank-field +  luminosity function from <cit.>. 

We do however take into account the results of <cit.> when estimating the total volumes probed by the  filter in the search for  emitters: using spectroscopy, <cit.> find that for HiZELS ∼50 percent of photometrically-selected + emitters at z ∼ 1.4 are λ5007, ∼27 percent are λ4959, ∼16 percent are , with the remaining ∼7 percent being simultaneous detections of λ5007 and λ4959. Based on these results, <cit.> then add to the total volume probed (i.e. the volume probed if searching for λ5007 emitters) 16 percent of the volume that would be probed had their search been for , and 25 percent of the volume had they been searching for λ4959. We thus apply similar corrections to our total volume probed for λ5007 emitters.





  §.§.§ Corrections for dust attenuation


Dust in star-forming galaxies reprocesses light emitted in the rest-frame UV and optical, and can therefore reduce the amount of  and  flux observed. In order to estimate the intrinsic brightness of the emission lines (i.e. their integrated luminosities), one has to correct for the effect of dust attenuation. For an attenuation of A_line (mag) at the emission line wavelength, the conversion from line flux to intrinsic line luminosity is


    L_line = 4 π D_L^2F_line× 10^0.4A_line ,


where D_L is the luminosity distance. We follow <cit.> and assume an attenuation at the  wavelength of A_Hα = 1 mag, which is based on previous HiZELS studies <cit.>. <cit.> do not correct for dust attenuation when plotting their luminosity functions, so we also leave our + luminosities uncorrected to ensure a consistent comparison. <cit.> later go on to calculate the dust-corrected star formation rates (SFRs) of their galaxies, where they then assume an attenuation of A_+ = 1.35 mag, derived by assuming A_Hα = 1 mag and using a <cit.> dust attenuation curve. We thus adopt the same correction when calculating our own SFRs (see <ref>).




  §.§.§ Filter profile volume corrections


The comoving volumes used for our luminosity functions (<ref>) are calculated by approximating the  filter as a top-hat filter with width equal to the  FWHM. Since the filter profile is not a top-hat in reality, this introduces two main effects which need to be accounted for when estimating the galaxy number densities. Firstly, bright emitters whose line falls near the edges of the  filter will suffer a significant loss of line flux and thus appear to be fainter than they really are. This produces an overall bias towards faint sources in our sample. Secondly, any faint emitters close to the filter edges might be missed from our sample, and are therefore only detectable over a narrower redshift range (and thus a smaller volume) than their bright counterparts.

To correct for these effects, we follow the method used by <cit.> and <cit.>, as first proposed in <cit.>. An initial Schechter fit is performed to the uncorrected`Uncorrected' only in terms of the filter profile correction; the results have already been corrected for line completeness and dust attenuation by this stage. data. We then generate a mock sample of 10^5 fake sources with a luminosity distribution that is weighted by the uncorrected Schechter function. These sources are randomly assigned redshifts with a uniform distribution across the whole possible  coverage. They are then convolved through the  filter profile such that their luminosities decrease according to their assigned redshift (i.e. according to the position of the redshifted emission line in the filter profile) and rebinned using the same luminosity bins as for the uncorrected data. Comparing the resultant distribution to the input distribution reveals that bright sources are underestimated relative to the fainter sources, as expected. The real data are corrected using the ratio of these distributions.




  §.§.§ Fitting Schechter functions


Finally, we perform fits to the corrected data using a Schechter function <cit.>:


    Φ(L)dL = ln(10) Φ^∗(L/L^∗)^α + 1 e^-(L/L^∗) dlog L ,


where Φ(L) is the number density at luminosity L, Φ^∗ is the normalisation of the luminosity function, L^∗ is the characteristic luminosity, and α is the slope at the faint end of the luminosity function, where the power-law component dominates. 

The faintest bins (open symbols in Figure <ref>) are excluded from each fit due to their low completeness. For the  emitters, we take `low completeness' to mean that the low-luminosity edge of the bin lies below the 30 percent completeness limit. 
The line luminosities of all + emitters lie above the 90 percent completeness threshold; we therefore do not exclude any from the fit.


Due to the small number of bins left available for fitting, it is impossible to reliably constrain all three free parameters of the Schechter function simultaneously. For the  luminosity functions, we therefore fix the faint-end slope, α, to the value of -1.59 obtained by <cit.> at z = 2.23. For the + emitters, we only have one available bin and thus fix both α and log(L^∗/erg s^-1) to the values for the z = 3.24 sample of + from <cit.>, which are -1.60 and 42.83, respectively. Thus our + luminosity function is effectively a renormalised version of the <cit.> result, with Φ^∗ being the only free parameter. In addition to the  luminosity functions of the individual pointings, we also construct fits to the combined sample of  emitters from both SMG fields, as this provides a more general view of SMGs at z ∼ 2.3 with improved statistics. 

The best-fit parameters for each luminosity function are summarised in Table <ref>. Uncertainties are estimated for each free parameter by randomly perturbing the bin heights according to their uncertainties and then recalculating the fit, and repeating this process until 10^5 fits have been made. The 1σ confidence interval for each parameter is then estimated using the 16th and 84th percentiles of the best-fit values.






 §.§ Analysing luminosity functions





We next use the luminosity functions to assess whether the targeted SMGs reside in overdensities of emission-line galaxies. Figure <ref> compares the observed luminosity functions from the SMG fields to those from the blank field surveys of <cit.> and <cit.>. 
The environment of ALESS 75.2 shows signs of being overdense relative to the field at z ∼ 2.3, with most bins lying significantly above the blank field luminosity function. Conversely the environments of ALESS 5.1 and 102.1 are broadly consistent with the blank field luminosity functions at their respective epochs. 
An overdensity remains when the  emitters from both fields at z ∼ 2.3 are combined, implying that on average SMGs at this epoch reside in overdense, protocluster-like environments, which qualitatively consistent with clustering results <cit.>. The contrast between the individual  luminosity functions suggests that there is significant variation across SMG environments, although observations of additional SMGs are required to confirm and quantify the field-to-field variation. Furthermore, as explored later in this section and shown in Figure <ref>, the field around ALESS 102.1 is itself overdense on smaller scales. 


To quantitatively compare the SMG field and blank field luminosity functions we consider the parameters of the Schechter function fits ( <ref>). The parameters of Schechter function fitting are often correlated, so in Figure <ref> we show the uncertainties of the luminosity function parameters in the L^∗–Φ^∗ plane (as described in <ref> the faint-end slope, α, is fixed), comparing our SMG fields with the blank fields at similar redshifts. 
For the individual SMG environments at z ∼ 2.3 the fit parameters are offset from those of the blank field, although for the ALESS 102.1 region the offset is only at the ∼1σ level. 
These separations are driven by a higher L^∗, and,  in the case of ALESS 75.2, by a larger Φ^∗, which implies that this environment is preferentially overdense in bright line emitters compared to the blank field. 
Meanwhile, the environment of ALESS 5.1 exhibits an offset of <1σ relative to the blank-field value of Φ^∗ at z ∼ 3.3, implying this SMG does not reside in an overdnensity.

 
We quantify the galaxy overdensity in each sample of  and + emitters in two ways. Firstly, we calculate the ratio of the Φ^∗ from the best fit Schechter function to those from the relevant blank-field luminosity functions, Φ^∗/Φ^∗_field. This ratio tells us how much higher the `knee' of each SMG-field luminosity function is relative to the blank-field. These ratios are presented in Table <ref>. The value for the + luminosity function suggests that the environment of ALESS 5.1 at z∼3.3 is 1.2^+0.6_-0.4 times as dense as the field, i.e. it is consistent with the blank field. For the  emitters at z∼2.3, L^∗ is also a free parameter in the Schechter fits, and the L^∗–Φ^∗ correlation means that we must first refit the data with L^∗ fixed to the z ∼ 2.3 blank-field value from <cit.>, i.e. log(L^∗ / erg s^-1) = 42.83. 
These fits give Φ^∗/Φ^∗_field values of 3.6^+0.6_-0.6 and 1.7^+0.3_-0.3 for the ALESS 75.2 and ALESS 102.1 fields, respectively. The combined sample of  emitters from both SMG environments suggests that the average SMG environment at z ∼ 2.3 is 2.6^+0.4_-0.4 times more dense than the blank field at this epoch. 




To derive a more representative estimate of the galaxy overdensity in each environment, we also estimate the number of  or + emitters that one would expect to find in a blank field with the volumes probed by our observations. To do this, we integrate the field luminosity functions across the luminosity range covered by our data, excluding low completeness bins; i.e. we integrate across the ranges 42.5 ≤log(L^∗_ / erg s^-1) < 43.8 and 42.4≤log(L^∗_+ / erg s^-1) < 42.7 and multiply by the volumes probed in each HAWK-I pointing to estimate the expected number counts in an equivalent blank field, N_field. Since these field galaxies would have contributed to the observed number counts, we quantify the galaxy overdensity in each environment using


    δ_g = N_total - N_field/N_field


where N_total is the sum of the counts in our complete bins. Uncertainties in N_total are determined by adding in quadrature the uncertainties in the bin counts. For N_field the uncertainties are estimated by randomly permuting the blank-field Schechter parameters within their uncertainties prior to integrating, then repeating the process 10^5 times and using the 16th and 84th percentiles of the resultant number counts to define the 1σ confidence interval.

The values of δ_g for each sample of  and + emitters are summarised in Table <ref> along with the significance of this overdensity, σ_δ. Based on these values, the environments of ALESS 5.1, 75.2, and 102.1 are overdense by factors of 0.2^+2.5_-0.7 (0.3σ_δ), 2.6^+1.4_-1.2 (2.3σ_δ), and 0.2^+0.6_-0.5 (0.5σ_δ), respectively. If the samples of  emitters from both pointings are considered as one, then the SMG environments at z ∼ 2.3 are overdense by a factor of 1.5^+1.0_-0.8 (1.9σ_δ) on average. 
The above uncertainties do not account for cosmic variance, which could cause a factor ∼2 difference in number counts, as based on the  emitters in two equal depth HAWK-I pointings in COSMOS and UDS <cit.>. Including cosmic variance in our calculations does not change our overall conclusions that the ALESS 75.2 and the combined z∼2.3 SMG fields are overdense, nor does it affect the conclusions that ALESS 5.1 and ALESS 102.1 reside in environments consistent with the blank field. 

While Pointing 102 as a whole is not overdense, the majority of the  emitters in this pointing are contained within the same quadrant as the SMG, as can be seen in Figure <ref> (see also <ref>). We therefore recalculate δ_g for this SMG environment, this time considering only the volume probed within that quadrant (1722 cMpc^3), finding δ_g = 3.8^+2.4_-1.8 (2.1σ) in this area, which suggests that ALESS 102.1 actually does reside in an overdense environment with a physical scale of ∼1.6 Mpc. This high concentration of galaxies surrounding the SMG could be indicative of structure formation on smaller scales than those of protoclusters and it is possible that ALESS 102.1 resides in a protogroup <cit.>. We discuss the spatial distribution of companion galaxies in each SMG environment in more detail in <ref>.



The question remains as to whether the target SMGs reside in protoclusters, which will evolve into bound clusters by the present day. 
To learn more, we compare the overdensities in the SMG fields with previous studies of protoclusters. However, protoclusters exhibit a wide range of galaxy overdensities; a `typical' value of δ_g is not well-defined, though we highlight here structures  at similar redshifts to our target SMGs. For example, δ_g = 2.5 for the z = 1.99 protocluster in the GOODS-N field <cit.>. The protoclusters 4C 10.48 (z = 2.35) and 4C 23.56 (z = 2.48), which were both identified using narrowband selection of  emitters around luminous radio galaxies, were found to have overdensities of δ_g = 11^+2_-2 and δ_g = 4^+5_-3, respectively <cit.>. 
Similarly, <cit.> used a narrowband search for  emitters at z = 2.23 to identify overdensities of δ_g∼ 3, 2, and 2 around a quasi-stellar object overdensity, a high-redshift radio galaxy, and an overdensity of SMGs and optically faint radio galaxies, respectively. 
The protocluster Cl J0227-0421 at z = 3.29 is overdense by a factor of 10.5 ± 2.8 <cit.>. Two protoclusters in the COSMOS field at z = 2.10 and z = 2.47 were found to have overdensities of δ_g∼ 8 and δ_g∼ 3.3, respectively <cit.>. <cit.> confirm overdensities of  emitters in two protocluster candidates, BOSS1244 and BOSS1542, with overdensity factors of δ_g = 5.6 ± 0.3 and δ_g = 4.9 ± 0.3, respectively. 
Comparing δ_g for these protoclusters with our values, we posit that the environments of ALESS 75.2 and ALESS 102.1 (and thus of SMGs on average at z ∼ 2.3) are consistent with being protoclusters, albeit at the lower-density end. For the remainder of the analyses we assume that members of these overdensities will form larger structures by z = 0, although we caution that the significance of these overdensities is relatively low (1.3–2.3σ_δ) and thus it is uncertain whether they will coalesce by z∼0 <cit.>.





 §.§ Spatial distribution of line emitters



In order to investigate the role of environment in shaping the evolution of SMGs, and to assess whether the target SMGs reside in special regions within any surrounding structures, we next explore the spatial distributions of coeval line emitters across our HAWK-I pointings. Due to the small size of the  emitter sample, we limit this part of the analysis to the  emitters around ALESS 75.2 and 102.1. 
Figure <ref> compares the surface density of  emitters as measured in annuli centred on each target SMG (where the density calculations account for masked and unobserved regions by assuming the density is the same as in the observed regions)  with the surface densities one would expect based on the blank-field luminosity function from <cit.>. The annuli have inner and outer radii increasing in increments of 20, and are represented by dashed circles in Figure <ref>. Note that these large annuli are necessary due to the sample sizes, but make it difficult to probe the protocluster structures in detail. 
We therefore also show in Figure <ref> the result of smoothing the distributions of  emitters in Pointing 5+75 (left) and Pointing 102 (right) using a 2D Gaussian kernel with width corresponding to 0.5 Mpc at z ∼ 2.3. This method of visualisation clarifies where the SMGs lie in relation to any density peaks and can highlight any substructures. For simplicity we assume that there are no  emitters in the unobserved region between the detector chips, or in regions of the image that have been masked, and thus the densities shown in these regions are potentially underestimated. 

For ALESS 102.1, there is a noticeable decrease in the surface density of  emitters as a function of radial distance from the SMG, with the innermost bin in Figure <ref> being significantly overdense relative to the field despite the environment not being overdense as a whole (see also <ref>). This is also clear from Figure <ref>,  which shows that the SMG lies ∼20 from a ∼325 (∼1.6 Mpc) density peak. Furthermore, Figure <ref> demonstrates that the innermost 20 annulus contains more than half of the  emitters detected across the Pointing 102 pointing.  
Figure <ref> includes a panel showing the location of the SMGs and our  emitter density maps in the context of the overdensity of  emitters (LAEs) at z ∼ 2.3 mapped by <cit.>. This shows that the small-scale overdensity around ALESS 102.1 is in a broader underdense region, and it is therefore unlikely to be a condensed infalling knot within a larger structure. 


In the case of ALESS 75.2 we show two results in Figure <ref>: one where we include all  emitters in the pointing (open squares), and one where we exclude the dense clump of  emitters in the north-east (filled squares; see <ref>). In both cases, there is no significant trend in the  surface density as a function of separation from the SMG, although it does show signs of decreasing at the outermost radii if the dense north-easterly clump is excluded. This lack of trend implies that ALESS 75.2 does not reside in a particularly special region of the structure, and/or the structure extends beyond the HAWK-I field of view. 
The latter hypothesis is supported by the comparison of the  emitter overdensity with that of the LAEs from <cit.> (Figure <ref>, right), which shows that the whole structure around ALESS 75.2 is within a larger region of LAE overdensity. This suggests that the  emitter structure likely spans a physical distance ≳ 3.5 Mpc at z ∼ 2.3, which is consistent with the simulations of e.g. <cit.>, in which protoclusters are expected to extend over ≳ 10 Mpc at z ∼ 2. The <cit.> structure in this region includes the  blob CDFS-LAB03, which coincides with seven  emitters (see also <ref>). The overall picture is consistent with previous findings, in which  blobs are found to be associated with massive dark matter halos and filamentary large-scale structures <cit.>. 











 §.§ SMG companions: SFRs and stellar masses


In this section we investigate the dust-corrected star formation rates (SFR) and stellar masses (M_⋆) of the individual galaxies around each target SMG, to determine whether they lie on the main sequence of star formation at their epochs. This correlation between SFR and M_⋆ has been observed out to z ∼ 6 <cit.> and galaxies significantly above the main sequence are usually considered to be short-lived starbursts, whereas those significantly below the main sequence are typically quenched. The position of galaxies relative to the main sequence provides insights into their evolutionary state and can be used to infer the role of any environmentally-driven mechanisms enhancing or inhibiting star formation activity. 
Note that while ALESS 5.1 does not appear to reside in an overdense structure, the properties of the coeval + emitters in its vicinity are still of interest and we thus include them in this part of the analysis.

We obtain stellar masses and SFRs for our galaxies by using the SED fitting code,   <cit.>, to fit SEDs to the same fixed-aperture photometry used to derive photometric redshifts in <ref>. Figure <ref> compares the relationship between SFR and stellar mass for the  and + emitters that are SMG companions with the main-sequence at similar epochs using the prescription from <cit.>. These galaxies generally scatter about the main sequence at their respective epochs, following a similar trend of increasing SFR with increasing stellar mass. 
We thus find no significant evidence of enhanced star formation in any of these SMG environments, despite the range of overdensities that they span; this is contrary to some previous studies in which enhanced SFRs have been found in overdense environments at z ≳ 1 <cit.>, however it is consistent with several other studies in which no environmentally-driven SFR enhancement is observed at high redshift <cit.>.




In addition to the -derived SFRs, we also calculate the dust-corrected star formation rates for the  emitters using the scaling relation from <cit.>, modified for a <cit.> IMF: 


    SFR (M_⊙ yr^-1) = 4.65 × 10^-42 L_ (erg s^-1) ,


where the  line flux has been corrected for contamination by the nearby  doublet (see <ref>) and we assume a dust attenuation of A_ = 1 mag (<ref>). For the + emitters we assume an attenuation of A_+ = 1.35 mag following <cit.> and use the relation between SFR and L_+ from <cit.>, similarly modified for a <cit.> IMF:


    SFR (M_⊙ yr^-1) = 4.32 × 10^-42 L_+ (erg s^-1) .


The bottom panel of Figure <ref> shows how the ratio of the magphys-derived and line-derived SFR estimates varies with stellar mass. Also shown is the  dust attenuation required for the SFR derived from Equation <ref>, SFR_ Hα, to agree with the -derived value, SFR_magphys, as given by:


    A_ Hα = 2.5log_10( SFR_magphys/ SFR_ Hα) + 0.4 .


An analagous equation for A_+ can be obtained by adding 0.14 mag. It is evident that as one moves to higher stellar mass, the assumption that A_ Hα (A_+) = 1.0 (1.35) mag results in underestimated SFRs compared with the results from SED fitting. We therefore caution that while such an assumption may be suitable for galaxies with low-to-average stellar mass, it becomes less reliable for high-mass galaxies.


We also include the SMGs themselves in Figure <ref>, with the SFRs and stellar masses for these calculated by <cit.> and <cit.> using . As expected for sources selected due to their infrared-brightness, the SMGs are among the most active galaxies in the observed fields. ALESS 102.1 is also one of the most massive galaxies in its environment, which suggests that if it is in a protocluster then it may be brightest cluster galaxy (BCG) progenitor, i.e. a proto-BCG. 
Similarly, ALESS 5.1 is massive relative to other galaxies in the surrounding region, but given the low density of this environment we deem it unlikely that this SMG is a proto-BCG. 
Conversely, ALESS 75.2 has a lower mass, which is not exceptional for its environment, and which points towards it being more likely to evolve into a normal cluster member. This is consistent with the spatial analysis of  emitters and LAEs (<ref>), which showed that ALESS 75.2 is offset from the densest regions of this field. 

Since ALESS 102.1 has a counterpart  emitter in our sample, we also compare our -derived SFR and stellar mass with those derived by <cit.>. Our stellar mass of log(M_⋆/ M_⊙) = 11.49^+0.18_-0.05 is in good agreement with their value of log(M_⋆/ M_⊙) = 11.42^+0.06_-0.06. Conversely, our SFR of log( SFR/ M_⊙ yr^-1) = 2.75^+0.22_-0.25 is significantly higher than their estimate of log( SFR/ M_⊙ yr^-1) = 2.12^+0.25_-0.23. This is likely due to the inclusion of FIR and radio photometry in their SED fitting which are absent from our own fit; the dust component (and thus the SFR) is better constrained in the <cit.> SED fit. We therefore opt to use their values of SFR and stellar mass for this galaxy instead of our own.




 §.§ Stellar mass functions


We next construct the stellar mass functions of the galaxies around each SMG and compare these with the blank field. The stellar mass functions are derived following a similar procedure as for the luminosity functions (see <ref>), minus the corrections that are only relevant to luminosity functions (dust attenuation, line flux contamination, filter profile corrections). Completeness corrections were applied to each mass bin according to the completeness values estimated in <ref> based on the emission line fluxes. We then fit Schechter functions to the data:


    Φ(M_⋆)dM_⋆ = ln(10) Φ^∗(M_⋆/M^∗)^α + 1 e^-(M_⋆/M^∗) dlog M_⋆ ,


where Φ(M_⋆) is the number density at stellar mass M_⋆, Φ^∗ is the normalisation of the stellar mass function, M^∗ is the characteristic stellar mass, and α is the slope at the faint end of the stellar mass function. Mass bins that are less than 50 percent complete are excluded from the fitting procedure. 
As with the luminosity functions, we also fix the faint-end slope α to the  values derived for blank-field stellar mass functions by <cit.> and <cit.> (i.e. α = -1.37 and α = -1.3 for the for the  and + emitters, respectively). For + emitters we also fix the characteristic stellar mass, M^∗, to the blank-field value of log(M^∗ / M_⊙) = 10.96 <cit.>. 

The stellar mass functions are presented in Figure <ref>, with the parameters in  Table <ref>. Uncertainties in each parameter are estimated following the same procedure as for those of the luminosity functions (see <ref>) and the correlation between the parameters and their uncertainties are shown in  Figure <ref>, which demonstrates that at the upper limit the characteristic stellar mass, M^∗, is poorly constrained for all of our samples except the  emitters around ALESS 75.2. 
However, the lower limit is sufficient to show that in the z∼2.3 SMG regions the characteristic stellar mass is significantly higher than the z∼2.3 field, which suggests that the stellar mass build-up in SMG companion galaxies is further advanced than the coeval field <cit.>. Due to our selection of  emitters the galaxies have non-negligible star-formation rates (though many are below the main sequence; <ref>). Observations using a local galaxy density estimator suggest that local environment has minimal effect on the stellar mass function of star-forming or quiescent galaxies at z=1.5–2 <cit.>. However, there is evidence of protocluster environments being skewed towards containing galaxies with higher masses than the field <cit.>, consistent with our results. 














 §.§ Dark matter halo masses and evolution




We next estimate the total halo masses of the SMG environments in order to place them within the context of existing protoclusters and trace their likely evolution, focusing primarily on the overdense environments of the two SMGs at z ∼ 2.3. 
Since these overdensities are unvirialised and lack a detectable intracluster medium, the classic methods for weighing galaxy clusters cannot be used. 
Instead, we use two methods that have been used in protocluster studies, though the underlying assumptions required can lead to significant uncertainties, as discussed in the following subsections.
The first method is detailed in <ref> and uses the stellar-to-halo mass relation (SHMR) to estimate the high-redshift mass of the clusters (hereafter SHMR method) and evolve it to the local Universe using the Millennium and Millennium-II simulations <cit.>. 
The second method follows <cit.> and assumes the region of interest is a homogeneous sphere undergoing spherical collapse and uses the overdensity parameter to estimate the z=0 descendant mass, which we trace back to high-redshift using the Millennium and Millenium-II simulations. This is referred to as the spherical collapse model (SCM) method and detailed in <ref>.
In <ref> we discuss the evolution of the SMG environments compared with other systems and previous measurements. 




  §.§.§ The SHMR method for deriving halo masses



The SHMR method for estimating protocluster masses involves identifying the most massive galaxy in the structure and converting its stellar mass to a halo mass, and taking this to be the halo mass of the whole structure. This method has been employed in recent protocluster studies <cit.> and implicitly assumes that all member galaxies occupy a single halo at the observed redshift of the structure, which may not be the case if some of the galaxies are still infalling. Nevertheless, we deem this assumption preferable to the commonly-used alternative of estimating the halo masses of each individual galaxy and summing them together <cit.>, which risks `double-counting' overlapping dark matter halos to produce an overestimate of the structure halo mass. 
Note that we perform this calculation even for ALESS 5.1 and its surrounding  emitters despite our analyses revealing no signs of their environment being significantly overdense. This is because this method does not explicitly depend on the density of the surrounding environment, and the high stellar mass of ALESS 5.1 (see <ref> and <ref>) suggests it may yet reside in a massive halo. We do however caution that the result obtained here likely represents an extreme upper limit on the mass of any possible structure around ALESS 5.1.

We estimate halo masses for each target SMG and their candidate companion galaxies using the SHMR from <cit.>. We use the relation as defined at z = 2 for galaxies in the environments of ALESS 75.2 and 102.1, and at z = 3 for galaxies in the environment of ALESS 5.1. 
For the SMGs themselves we use the stellar masses from the literature <cit.>. 
Some of our galaxies have stellar masses which lie above the range at which the SHMR is defined and for these we use the stellar-to-halo mass ratio for the largest halo mass at which the relation is defined <cit.> to convert the stellar mass to a halo mass. This affects only two  emitters from P5_75, along with one from P102 which we have identified as a counterpart to ALESS 102.1 (see <ref> and Figure <ref>). None of the  emitters have stellar masses above the range for which the SHMR is defined at z ∼ 3, but ALESS 5.1 does lie above this range.  Uncertainties on individual galaxy halo masses are estimated based on the stellar mass uncertainties and the uncertainties in the SHMR derived by <cit.>.


We derive halo masses of log(M_h/ M_⊙) = 11.45–14.46 for individual  and  emitters, with medians of log(M_h/ M_⊙) = 12.16^+0.21_-0.16, 11.94^+0.46_-0.35, 11.75^+0.26_-0.12 for galaxies in the environments of ALESS 5.1, 75.2 and 102.1, respectively. The halo masses of the corresponding SMGs are log(M_h/ M_⊙) = 13.94^+0.38_-0.21, 12.02^+0.08_-0.25, 14.39^+0.03_-0.37, derived using their stellar masses reported in the literature <cit.>. 
ALESS 5.1 and 102.1 are both the most massive galaxies in their respective environments; we therefore adopt their halo masses as the total masses of the potential structures at the observed redshifts. ALESS 75.2 is not the dominant galaxy in its environment, being surpassed in stellar (and hence inferred halo) mass by ∼40 percent of its companion  emitters. The most massive of these is a spectroscopically confirmed member <cit.> located in the  emitter density peak associated with the  blob CDFS-LAB03 (see Figure <ref>), with a halo mass of log(M_h/ M_⊙) = 14.38^+0.01_-0.40. We thus assume this is the total mass of the surrounding structure. Since high-redshift radio galaxies are commonly found in protocluster cores <cit.>, we search for radio counterparts for this galaxy in the second data release from the Very Large Array 1.4 GHz survey of the ECDFS <cit.>, for which the typical sensitivity is 7.4  per 28×16 beam. We find no counterparts within 30 of this  emitter and thus rule it out as being a high-redshift radio galaxy.



The total halo masses at the observed redshift obtained using the SHMR method are thus log(M_h/ M_⊙) = 13.94^+0.38_-0.21, 14.38^+0.01_-0.40, 14.39^+0.03_-0.37 for the environments of ALESS 5.1, 75.2, and 102.1, respectively. 
We note that these masses may be affected by systematic uncertainties on the stellar masses (due to uncertainties on star-formation histories and resulting mass-to-light ratios, which is particularly relevant for young starbursts; e.g. ) and on the SHMR for very high mass galaxies, which are present in the simulations from which the SHMR is derived <cit.>. Indeed, predictions from halo mass functions suggest that the halo masses inferred from this method should be sufficiently rare that finding three such structures in the ∼0.25 deg^2 ECDFS is unlikely <cit.>. 
Therefore, we consider these SHMR-derived halo masses to be upper limits and as such they are represented by the upper bounds on Figure <ref> (the lower bounds are derived in <ref>), which compares the halo masses of the SMG environments with previously-studied galaxy clusters and protoclusters. 

To assess whether these SMG environments are true protoclusters, we evolve the masses derived from the SHMR method to the present day masses and compare with known galaxy clusters in the local Universe. This is done using the redshift-dependent formula for the mean mass accretion rate derived from the results of the Millennium and Millennium-II simulations <cit.>:


    ⟨Ṁ⟩_mean =     46.1  M_⊙yr^-1(M_z/10^12M_⊙)(1 + 1.11z) 
       ×√(Ω_m,0(1+z)^3 + Ω_Λ,0) ,


where M_z is the halo mass of the structure at its observed redshift, and Ω_m,0 and Ω_Λ,0 are the present-day density parameters for matter and the cosmological constant according to our assumed cosmology <cit.>. 

For each overdensity we begin with the total halo masses estimated using the SHMR method and apply Equation <ref> to incrementally add mass in small time steps until the present day is reached. The present-day masses obtained with this method are log(M_h,z=0/ M_⊙) = 15.93^+0.62_-0.33, 15.81^+0.01_-0.55, 15.82^+0.04_-0.50 for the overdensities containing ALESS 5.1, ALESS 75.2, and ALESS 102.1, respectively.
These masses suggest that these structures would all evolve into some of the most massive clusters in the Universe, rivalling that of the Coma cluster <cit.> and other massive clusters at z ≲ 1 such as those in the CLASH <cit.>. However, given the rarity of such massive structures seen in the local Universe, we posit that the identification of progenitor structures around all three of our target SMGs is due to the systematics in the calculations, rather than a real occurrence. 
In addition to the possible systematics in the stellar masses and SHMR, as previously described, we also note that the mean mass accretion rate given by Equation <ref> is poorly constrained for halos with masses of log(M_h/ M_⊙) ≳ 14 beyond z ∼ 0.5. 
Furthermore, Equation <ref> alone does not account for the diversity of evolutionary paths that real dark matter halos undergo, being the mean result for many halos in the Millennium simulation. 
Therefore, as for the high-redshift SHMR-derived halo masses, we also take these SHMR-derived z=0 and intermediate masses to be upper limits. 
This upper limit on the halo mass at the SMG redshift and the evolution to the present day is shown in Figure <ref> as the upper edges of the shaded regions for ALESS 75.2 and 102.1, and as a single solid line for ALESS 5.1. 





  §.§.§ The SCM method for deriving halo masses



An alternative method for estimating the present-day mass of each SMG environment is obtained following <cit.>, which approximates each SMG environment as a homogeneous sphere undergoing spherical collapse. In this case the total present-day mass is given by:


    M_h,z=0 = ρ̅ V (1 + δ_m) ,


where ρ̅ is the mean comoving matter density of the Universe, δ_m is the dark matter mass overdensity, and V is the comoving volume of the structure. We refer to this method as the spherical collapse model (SCM) method. Since the assumption of spherical collapse is unphysical for environments that are not overdense, we only perform this calculation for the two SMG environments at z ∼ 2.3.

To estimate the volume of each overdensity, we assume that the structures are spherical and use the spatial extent of the structure on the sky to infer the angular diameter of the sphere containing the member galaxies. 
As discussed in <ref>, the overdensity around ALESS 75.2 extends beyond the confines of the HAWK-I pointing and therefore the size of the HAWK-I field-of-view can be used as a lower limit on the angular diameter of the structure. 
Therefore, for this environment we calculate the comoving volume for a spatial extent of ∼ 7.5 <cit.>, which corresponds to a comoving volume of V ∼ 1000 cMpc^3. This volume should be considered a lower limit, and thus the derived halo mass is also a lower limit. 
For the environment of ALESS 102.1, we assume that the structure is confined to the quadrant containing the SMG (<ref>), such that the angular diameter of the sphere is then ∼ 220 <cit.>, which gives a comoving volume of V ∼ 110 cMpc^3.

The dark matter mass overdensity is linked to the observed galaxy overdensity, δ_g, via 


    δ_m = δ_g/b ,


where b is the bias parameter. To estimate the bias parameters for  emitters at z ∼ 2.3, we make use of the relationship between b and L_Hα derived by <cit.> at z = 2.23. Following a similar method to that of <cit.>, we derive a linear fit to this relation and estimate b at the mean value of L_Hα for each SMG environment. This gives b_Hα = 2.9^+0.2_-0.1  and 2.8^+0.2_-0.3 for the candidate companions of ALESS 75.2 and 102.1, respectively.

Using the above values of b and V along with the δ_g values calculated in <ref>, we obtain present-day halo masses of 
log(M_h,z=0/ M_⊙) = 13.90^+0.09_-0.10 and 13.05^+0.12_-0.14 
for the environments of ALESS 75.2 and ALESS 102.1, respectively. Contrary to the previous present-day mass estimates, these masses suggest that descendant of the environment of ALESS 102.1 is more akin to a galaxy group than a galaxy cluster <cit.>, while the environment of ALESS 75.2 may evolve into a `Virgo-like' or `Fornax-like' cluster by z = 0 <cit.>.

The significant disparity between the present-day halo mass estimates from the SCM method compared with the SHMR method likely arises from the assumptions and uncertainties in the calculations, including the systematics discussed in <ref> and the estimates of the structure volumes. 
Note that for ALESS 75.2 the derived z=0 SCM halo mass is a lower limit, due to the volume used being a lower limit. 
Since the masses derived from the SCM method are all lower than the those from the SHMR method and its uncertainties, we proceed with the SCM estimates and adopt the lower bound of the 1–σ confidence intervals as lower limits for the present-day halo masses.
Thus, the present-day halo mass estimates from the two methods gives a range of plausible evolutionary pathways for the two z ∼ 2.3 SMG environments as shown on Figure <ref>.

We also use the present-day halo mass estimates from the SCM method in combination with  Equation <ref> to trace the evolution of these SMG environments back in time to their observed redshifts, thereby obtaining a second estimate of the total mass at these redshifts. These masses are 
log(M_h/ M_⊙) = 12.93^+0.07_-0.08 and 12.24^+0.10_-0.11 
for the potential structures around ALESS 75.2 and ALESS 102.1, respectively. 
This calculation provides an evolutionary track that connects the lower halo mass limit at the z ∼ 2.3 to the corresponding value at z = 0, and this is what defines the bottom edge of each shaded region in Figure <ref>. 












  §.§.§ Halo masses and evolution



The halo mass estimates of each SMG environment are summarised in Table <ref>, and Figure <ref> compares the SMG halo masses and their evolution with previously-studied galaxy clusters and protoclusters.  
As described in <ref> and <ref>, the upper and lower bounds shown for the SMG halos are derived from the masses calculated using the SHMR method and the lower limits from the SCM method, respectively (with the exception of ALESS 5.1, for which only the SHMR method is used). Thus, this region encompasses the full range of possible halo masses and evolution for the SMGs. 
In Figure <ref> these are compared with clusters from the Cluster Lensing And Supernova Survey with Hubble <cit.>; clusters from the Gemini Cluster Astrophysics Spectroscopic Survey <cit.>; clusters and protoclusters from the Clusters Around Radio-Loud AGN program <cit.>. We also show the halo masses calculated by <cit.> for overdense structures known to be rich in dusty star-forming galaxies (DSFGs), including: the GOODS-N protocluster at z = 1.99 <cit.>; the COSMOS protoclusters at z = 2.10 <cit.> and z = 2.47 <cit.>; the `Spiderweb' protocluster, MRC 1138-256, at z = 2.16 <cit.>; the SSA 22 protocluster at z = 3.09 <cit.>; the GN20 overdensity at z = 4.05 <cit.>. 
In this sample of DSFG-rich protoclusters we additionally include the halo mass of the DRC at z = 4 <cit.>. 
Figure <ref> shows that the potential structures surrounding ALESS 75.2 and 102.1 are consistent with being protoclusters at z ∼ 2.3. The upper limit for the halo mass derived for ALESS 5.1 is also consistent with this environment being a protocluster, but we emphasise that our previous analyses suggest it is unlikely to reside in such a structure. 

Both the present-day  and  high-redshift masses obtained using the SHMR method (<ref>) are significantly higher than those obtained using the SCM method (<ref>), and higher than expected for three structures all found in a survey of ∼0.25 deg^2 based on predictions from halo mass functions <cit.>. This is consistent with the SHMR masses being affected by systematic effects that make them upper limits, as discussed in <ref>. The SCM-derived masses are typically lower than the masses of protoclusters at similar redshifts (see Figure <ref>), but are consistent with the results of SMG clustering studies <cit.>, which generally agree that SMGs reside in halos with log(M_h/ M_⊙) ≲ 13 at z = 1–3. 
However, we note that the studies by <cit.> and <cit.> were both conducted using single-dish observations and are thus limited by false counterpart identification and source blending. Although the high-resolution interferometric studies by <cit.> and <cit.> are not afflicted by these limitations, they present conflicting results for the halo masses of the SMG population, likely stemming from the methodological differences described in <ref>.

Figure <ref> also includes the approximate boundaries separating different gas regimes, as proposed by <cit.>: in halos for which log(M_h/ M_⊙) ≲ 12, inflowing gas is predominantly cold and enables the growth of galaxies; in halos above this mass threshold, these gas inflows are shock-heated resulting in strangulation of the galaxy within. However, if these massive halos still fall below some other, redshift-dependent mass threshold (as marked by the `cold in hot' boundary in Figure <ref>), then penetrating cold gas may still be sustaining galaxy growth. At z ∼ 2.3 this mass threshold is log(M_h/ M_⊙) ∼ 12.9, while at z = 3.3 it is log(M_h/ M_⊙) ∼ 14.0.

Based on our mass estimates, the halo of ALESS 5.1 is likely in the `cold in hot' category at its observed redshift, particularly when noting that the mass of this halo is possibly overestimated. It is therefore probable that ALESS 5.1, along with any other galaxy that may share its halo, is undergoing growth sustained by penetrating cold gas inflows. Conversely galaxies in the environment of ALESS 75.2 are more likely to be undergoing strangulation due to shock-heating in the halo at the observed redshift; the SHMR-derived halo mass lies significantly above the limit for `cold in hot' gas inflows, and the 1σ confidence interval for the SCM-derived mass only just crosses below the limit. 
We cannot conclude anything about the gas regime in the environment of ALESS 102.1 as the `cold in hot' boundary is straddled by the mass estimates for this structure.







§ CONCLUSIONS

We have conducted a wide-field narrowband survey of star-forming galaxies in the environments of three SMGs at z ∼ 2.3 and z ∼ 3.3 to determine whether these SMGs reside in protocluster-like environments. By studying individual SMGs selecting based only on their spectroscopic redshifts we have measured `typical' SMG environments. 
Our main conclusions are as follows:

  
  * Using HAWK-I  and  photometry, we identified a total of 147 candidate emission line galaxies in the two HAWK-I pointings containing the three target SMGs. After extracting photometry from archival UV-to-NIR broadband images, we performed SED fitting with  to obtain photometric redshifts for these galaxies and identified 44, 11, and 4 companion galaxies to the SMGs ALESS 75.2 (z_spec = 2.294), ALESS 102.1 (z_spec = 2.296), and ALESS 5.1 (z_spec = 3.303), respectively.
 
  
  * By constructing luminosity functions for each SMG environment and comparing with blank-field luminosity functions from the literature at similar redshifts, we measure overdensity parameters of δ_g = 0.2^+2.5_-0.7, 2.6^+1.4_-1.2, and 0.2^+0.6_-0.5 across the whole ∼4 Mpc HAWK-I field of view for ALESS 5.1, 75.2, and 102.1, respectively. 
    Whilst ALESS 102.1 is not overdense on these large scales, it does sit in a ∼1 Mpc region with δ_g=3.8^+2.4_-1.8. Therefore 2/3 of the target SMGs reside in overdense environments.
 
  
  * We considered the spatial distribution of companion  emitters in the environments of the two SMGs at z ∼ 2.3 (ALESS 75.2 and 102.1) by measuring their density in annuli around the SMGs and by constructing overdensity maps. For ALESS 75.2 the companion galaxies are spread out across the entire HAWK-I field of view, spanning a few Mpc. This is consistent with simulations, in which protoclusters are seen to extend over several Mpc at z ∼ 2–3. The SMG resides near a possible density peak of  emitters, although a greater peak is seen a few arcminutes eastward which appears to be associated with a previously discovered  blob <cit.>. 
    The overdensity around ALESS 102.1 is smaller (∼1 Mpc) and could instead evolve into a galaxy group locally. 
    
  
  * Stellar masses and SFRs were obtained for the companion galaxies in each SMG environment by performing SED fitting with . The galaxies are generally scattered about the star-forming main sequence at their respective epochs, with no evidence of enhanced star formation activity in either environment.
 
  
  * Two methods were used to estimate the total halo mass of each of the two overdense SMG environments, which provided upper and lower bounds on the halo masses at the observed redshifts and evolving to the present day. 
  These reveal that ALESS 75.2 likely resides in a protocluster, while ALESS 102.1 resides in either a protocluster or a protogroup. 
  
	
  * We therefore surmise that 2/3 of these SMGs are strong candidates for the progenitors of massive elliptical galaxies in clusters, although the possibility remains for them to end up in galaxy groups. If these targets are indeed representative of `typical' SMGs then this suggests that SMGs in general are likely to evolve into massive elliptical galaxies by the present day, as suggested by certain evolutionary models <cit.>, but with significant variation in the surrounding environments.


With this study we have demonstrated the efficacy of narrowband surveys as a means of searching for galaxy overdensities around SMGs selected without bias towards particular environments. 
Future followup with larger samples of SMGs and/or spectroscopic confirmation of companion galaxies would confirm the nature of the overdensities that we have detected, and resolved analyses (e.g. with ALMA and/or JWST) will further reveal how the member galaxies evolve. 








§ ACKNOWLEDGEMENTS


We are extremely grateful to Ian Smail for numerous insightful discussions, which significantly improved this work. The research is based on observations collected at the European Southern Observatory under ESO programme 0103.A-0668(A).
TC received support from the Science and Technology Facilities Council (STFC; 2287406) and the Faculty of Science and Technology at Lancaster University. 
JW acknowledges an STFC Ernest Rutherford Fellowship (ST/P004784/2).
HD acknowledges support from the Agencia Estatal de Investigación del Ministerio de Ciencia, Innovación y Universidades (MCIU/AEI) under grant (Construcción de cúmulos de galaxias en formación a través de la formación estelar oscurecida por el polvo) and the European Regional Development Fund (ERDF) with reference (PID2022-143243NB-I00/10.13039/501100011033). 
The authors would also like to acknowledge the publicly available software Topcat <cit.>, as well as the Python packages NumPy <cit.>, SciPy <cit.>, Astropy <cit.>, Matplotlib <cit.>, and Photutils <cit.>.





§ DATA AVAILABILITY


The data used in this publication can be accessed via the http://archive.eso.org/eso/eso_archive_main.htmlESO Science Archive under Program ID: 0103-0668(A), PI: Wardlow.






mnras
 
















 Paper Title: Finite distance effects on the Hellings-Downs curve in modified gravity.





    guillem.domenech@itp.uni-hannover.de

    apostolos.tsabodimos@stud.uni-hannover.de



^aInstitute for Theoretical Physics, Leibniz University Hannover, Appelstraße 2, 30167 Hannover, Germany.
^b Max-Planck-Institut für Gravitationsphysik, Albert-Einstein-Institut, 30167 Hannover, Germany


§ ABSTRACT 
There is growing interest in the overlap reduction function in pulsar timing array observations as a probe of modified gravity. However, current approximations to the Hellings-Downs curve for subluminal gravitational wave propagation, say v<1, diverge at small angular pulsar separation. In this paper, we find that the ORF for the v<1 case is sensitive to finite distance effects. First, we show that finite distance effects introduce an effective cut-off in the spherical harmonics decomposition at ℓ∼√(1-v^2)  kL, where ℓ is the multipole number, k the wavenumber of the gravitational wave and L the distance to the pulsars. Then, we find that the overlap reduction function in the small angle limit approaches a value given by π kL v^2 (1-v^2)^2 times a normalization factor, exactly matching the value for the autocorrelation recently derived. Although we focus on the v<1 case, our formulation is valid for any value of v.


Finite distance effects on the Hellings-Downs curve in modified gravity
    Apostolos Tsabodimos^a
    August 5, 2024
=======================================================================






§ INTRODUCTION 


The unique nature of pulsars, characterized by their remarkably stable rotational periods, allows for their utilization in a wide range of applications. These applications extend from the creation of Galactic maps to the exploration of Gravitational Waves (GWs) in the frequency range of 1-100 nHz. Of particular interest is the latter, an exciting prospect pioneered by Sazhin <cit.>, Detweiler <cit.>, Hellings and Downs <cit.>, and its subsequent analysis. Since then, substantial progress has been made, encompassing both phenomenological studies <cit.> and efforts to constrain theoretical models and test the predictions of General Relativity <cit.>.

With the rapid evolution of technology and expertise in recent years <cit.>, the prospect of reliably measuring a stochastic GW background in the aforementioned frequencies has become increasingly feasible. This advancement is driven by techniques in the field of Pulsar Timing Array (PTA) <cit.>. In fact, PTA collaborations recently reported 3-4σ evidence of a GW background <cit.>. A key aspect of PTA measurements is the identification of the Overlap Reduction Function (ORF) <cit.>, commonly denoted as Γ_ab(ξ_ab), where ξ_ab is the angle between pulsars “a” and “b”. In General Relativity (GR), the ORF is commonly known as the Hellings-Downs curve <cit.>.

Interestingly, it has been shown that the ORF can also be used to test the phase velocity of the GWs <cit.>. A phase velocity of the GWs with v≠ 1 is a signature of modifications of gravity. For example, it is possible to have v<1 or v>1 in scalar-tensor theories of gravity <cit.> (see Refs. <cit.> for recent reviews and Refs. <cit.> for an Effective Field Theory (EFT) perspective). In terms of metric transformations, v≠ 1 could also be motivated by disformal couplings to a scalar field <cit.>. In massive gravity one finds that the phase velocity is frequency dependent <cit.>. One can also use general EFTs to study modifications to the propagation speed and polarization of GWs <cit.>. It is important to note that although the LIGO/VIRGO/KAGRA collaboration placed stringent constraints on the propagation speed of GWs <cit.> at frequencies of around 100  Hz, it does not exclude departures from v=1 at lower frequencies, see, e.g., the arguments provided by Ref. <cit.>. Thus, PTAs provide a complementary probe to modified gravity. In general, one finds a different shape and frequency dependence of the ORF <cit.>, namely Γ_ab(kL,ξ_ab), where k is the wavenumber of the GW and L the distance to the pulsars. 

However, current analytical approximations to ORF in the v<1 case diverge at the small pulsar separation <cit.>. Recently, Ref. <cit.> found an exact formula for the value of the autocorrelation, i.e., the correlation of timing residuals of a single pulsar, say Γ_aa, for general values of v. In particular, they provide the autocorrelation value for the v<1 case in the large kL limit. However, there is the open question of how the ORF for v<1 approaches the autocorrelation value and why and when the analytical approximation breaks down.


The goal of this paper is to investigate the finite distance effects on the ORF with a special focus on the v<1 case. To do so, we work in the spherical harmonic decomposition and express the Legendre functions of the harmonic coefficients in terms of spherical Bessel functions. As the spherical Bessel functions extend over the whole real line, they have, in our opinion, a more manageable dependence on the multipole number ℓ. With our alternative formulation, we discover that the approximations derived in Refs. <cit.> are valid up to a cut-off at ℓ∼√(1-v^2) kL. Naively, a cut-off for ℓ≳ kL is to be expected as the angular separation corresponding to the multipole number (roughly ξ∼ 1/ℓ) is smaller than the typical features in the antenna response pattern (which is of the order of 1/kL). Thus, multipoles higher than the cut-off do not contribute to the description of the antenna response. in Fig. <ref>, we show the antenna pattern for v={0.8,1,1.2}.[We follow the procedure described in Ref. <cit.> with the antenna response modified to

    R^A(k,Ω̂)=(1-e^-ikL(v+Ω̂·p̂))v/2e^A_ij(Ω̂)p_ip_j/v+Ω̂·p̂ .

The first term inside the parenthesis is the so-called redshift transfer function and the rest is called geometrical projection factors. The subscript A refers to plus + or cross × polarization.
 We thank B. Allen for the suggestion to draw the antenna pattern.] At the end of the paper, we show that such cut-off reconciles the ORF of Refs. <cit.> with the autocorrelation value of Ref. <cit.>.





This paper is organized as follows. In Sec. <ref>, we present an alternative formulation to the harmonic coefficients in terms of one integral of spherical Bessel functions. We dedicate Sec. <ref> to study the behavior of the integral for ℓ<kL and ℓ> kL. We show an exponential suppression in the latter case for ℓ> √(1-v^2) kL. In Sec. <ref>, we compute the ORF for v<1 and show that it is finite in the small angle limit. We conclude our work in Sec. <ref> with discussions and future directions.






§ REVIEW AND RE-FORMULATION IN TERMS OF BESSEL FUNCTIONS 


Our starting point is the expression of the Hellings-Downs curve in the spherical harmonic expansion, which following the notation of Ref. <cit.> reads

    Γ(kL,ξ)= C∑_ℓ=2^∞ a_ℓ P_ℓ(cosξ) ,

where C is a normalization constant, which we will fix to C=3/64 to compare with Ref. <cit.>, 

    a_ℓ=(2ℓ+1)2(ℓ-2)!/(ℓ+2)!|c_ℓ|^2 ,

and

    c_ℓ=∫_-1^1dxv/v+x(1-e^-ikL(v+x))(1-x^2)^2d^2/dx^2P_ℓ(x) .

We refer the reader to Refs. <cit.> for more details on the derivation of the above formulas. In Eq. (<ref>), we denote with v=ω(k)/k the phase velocity of the GWs (where ω(k) is the dispersion relation), which in principle can have any positive value as well as k dependence.[A k dependence in the phase velocity appears in massive gravity <cit.> where ω^2=k^2+m_g^2 with m_g being the graviton’s mass.] It should be noted that we also assumed pulsars at equal distances for simplicity. The main goal of this section is to recast c_ℓ in a more suitable form for integration and analytical approximations. 

We start noting that the terms containing v+x may be written in an integral form, namely

    v/v+x(1-e^-ikL(v+x))=i∫_0^kLdq  e^-i(v+x)q .

We note that similar replacements are often referred to as “Feynman's trick”.
With the identity (<ref>), Eq. (<ref>) becomes[We also used that

    d/dxP_ℓ(x)=ℓ+1/1-x^2(xP_ℓ(x)-P_ℓ+1(x))   and   (ℓ+2)P_ℓ+1(x)=(2ℓ+3)xP_ℓ+1(x)-(ℓ+1)P_ℓ(x) .

See Sec. 14.10 of Ref. <cit.> for more details.
]

    c_ℓ=iv(ℓ+1)∫_0^kLdq  e^-ivq∫_-1^1dx  e^-ixq{((ℓ+2)x^2-ℓ)P_ℓ(x)-2xP_ℓ+1(x)} .

Now we can perform the x integral by noting that (see Eq.18.17.19 in Ref. <cit.>)

    ∫_-1^1dx e^-ixq P_ℓ(x)=2 (-i)^ℓ j_ℓ(q) ,

where j_ℓ(q) is the spherical Bessel of ℓ-th order. Then, integrating Eq. (<ref>) over x and after some integration by parts,[We used that 
    ∫_-1^1 dx  x^n e^-ixq P_ℓ(x)=2(-1)^ℓ i^ℓ+n j^(n)_ℓ(t) ,

where we indicate the n-th derivative with a superscript parenthesis.
] we obtain our main formula, which is given by

    c_ℓ=2(-1)^ℓ    i^ℓ+1v(ℓ+1){((ℓ+2)v^2-ℓ) I_ℓ(kL,v)-2iv  I_ℓ+1(kL,v)
       
    -e^-ivkL[ℓ+ivkL/kL(ℓ+2)j_ℓ(kL)-(ℓ+4) j_ℓ+1(kL)]} ,

where we used that j_ℓ(0)=0 for ℓ>0 and, for convenience, we introduced

    I_ℓ(kL,v)=∫_0^kLdq  j_ℓ(q)e^-ivq .

With the above procedure, we have reduced the calculation of c_ℓ given by Eq. (<ref>) to a definite integral of a spherical Bessel function. Now, it all boils down to calculating I_ℓ(kL,v), which unfortunately does not have an analytical solution in terms of known functions, as far as we are aware. It is interesting to note that the spherical Bessel functions in Eq. (<ref>) can have an amplitude of O(ℓ^2). Thus, for finite kL, cross terms in |c_ℓ|^2 that include Bessel functions can introduce O(1) modulations on the amplitude of a_ℓ. In the calculations of the ORF though, they mostly average out.




§ ANALYTICAL APPROXIMATIONS TO THE INTEGRAL  


Before going into the detailed analytical approximations, it is instructive to look at Eq. (<ref>) qualitatively. First, we note that in PTAs, we have GW frequencies between f∼ 10^-9-10^-8  Hz and distances to pulsars of about L∼ 1000  lyr. It follows that typical values for kL∼ 2π f L are of the order of 10^3-10^4. We will then always consider that kL≫1 in Eq. (<ref>). However, this is not enough to only consider the kL→∞ limit of the integral due to the behavior of the spherical Bessel function. This is because the spherical Bessel function, j_ℓ(q), first grows as q^ℓ, peaks at around q∼ℓ, and continues to infinity with damped oscillations. This difference between q>ℓ and q<ℓ tells us that there will be a different behavior for ℓ≪ kL and ℓ≫ kL.

To understand the change in behaviour of I_ℓ(kL,v) (<ref>) in terms of kL, ℓ and v, it is instructive to take the large ℓ limit approximation of the spherical Bessel function given by

    j_ℓ(μβ)≈1/μ√(cosβ/tanβ)cos(μ(β-tanβ)+π4) ,

where μ=ℓ+1/2. The above equation is valid for 0<β<π/2 and follows from the approximation to J_μ(x) derived by Rayleigh (see Sec. 8.22 of Ref. <cit.>). Within such approximation we have that Eq. (<ref>), for ℓ<kL, reads

    I_ℓ(ℓ<kL,v)≈     I_ℓ(ℓ,v)
       +∫_0^β_kLdβ/2μ√(tanβ/cosβ)(e^iμ(β-tanβ)+iπ4+e^-iμ(β-tanβ)-iπ4)e^-ivμβ ,

where β_kL=arccos(ℓ/kL). In this form, it is clear that the integrand of Eq. (<ref>) contains two highly oscillating functions with frequency proportional to μ (or ℓ). 

We can gain insight into the behavior of Eq. (<ref>) by looking for saddle points of the imaginary part of the integrand. For v<1, we find that the saddle point of the product of the oscillations lies at sinβ=v. This means that the integrand contributes most around q∼μ/√(1-v^2), which for small values v is, unfortunately, in the regime where Eq. (<ref>) is not very reliable, at least regarding the amplitude. Nevertheless, approximating the integral by the integrand at that point captures the essence of the integral. Namely, it will be an oscillating function of ℓ for a given v<1 (actually, the integrand resembles a Hankle function of zeroth order as it goes as e^-iμ(arcsin v+π/4); something close to what we shall find later). In the opposite regime, namely v>1, the saddle point is complex. If we say that β=a+ib, the saddle point is located at a=π/2 and cosh b=v (at the other extreme of the real β line). This means that the integral is exponentially suppressed with ℓ, roughly as e^-μarccosh v. Both behaviors, for v<1 and v>1 are consistent with the kL→∞ solutions to I_ℓ (<ref>), given by Eq. (<ref>). We thus conclude that the saddle point argument, though not completely rigorous, implies that for v<1, no “constructive interference” occurs when

    ℓ≳ℓ_ cut=√(1-v^2) kL .

This is related to the argument in Ref. <cit.> that c_ℓ for ℓ≫ kL is exponentially suppressed. For v>1 the behavior change might happen at lower ℓ as the saddle point corresponded to larger values of q. However, this will not be relevant for the discussion of the ORF as, in this case, only the lower multipoles contribute. We also note that for q<ℓ, the spherical Bessel function oscillates no more, and both cases should have a similar behavior. We proceed to study the integral in two different limits and derive the precise scaling of the integral (<ref>) with ℓ and kL.




 §.§ Analytical approximation in the  regime 


We start by looking at the ℓ≪ kL limit to Eq. (<ref>), which corresponds to the case when there are many GW oscillations within the angular scale corresponding to ℓ. So, effectively we may send kL→∞. In this limit, there are closed formulas in terms of Legendre polynomials.[These formulas can be derived using the formulas in Secs. 10.22 and 14.3 of Ref. <cit.>, in particular using Eq. 10.22.56 and the hypergeometric function representation of the Legendre polynomials. Alternatively, one can ask a software like Mathematica.] Explicitly, we find that

    I_ℓ(kL→∞,v)={   π/2(-i)^ℓ P_ℓ(v)+(-i)^ℓ+1Q_ℓ(v)   (v<1)
       (-i)^ℓ+1 Q_ℓ(v)    (v>1)
    . ,

where Q_ℓ(v) is the Legendre polynomial of the second kind for v<1, also known as Ferrers function, and Q_ℓ(v) is the Legendre function for v>1. It is interesting to note that, in our formulation, the difference between v>1 and v<1 comes from whether the frequency of the oscillations of the exponential is fast enough to interfere with the oscillations of the Bessel function (see also the discussion on the saddle point approximation at the beginning of the section). In the original formulation, the difference between the v>1 and v<1 could be associated with the seeming pole at x=-v <cit.>. Regarding the Hypergeometric representation of Legendre functions, the values v<1 and v>1 correspond to a different choice of branch cuts. Also note that, for the same reason, the integral Eq. (<ref>) is not well-defined for v=1 in the kL→∞ limit, though c_ℓ in (<ref>) is. Now, with Eq. (<ref>), we find that Eq. (<ref>), in the ℓ≪ kL regime, is approximated by

    c_ℓ   (ℓ≪ kL)≈ 4(-1)^ℓ
       ×{   v(ℓ+1)/2[((ℓ+2)v^2-ℓ)(Q_ℓ(v)+iπ2P_ℓ(v))-2v(Q_ℓ+1(v)+iπ2P_ℓ+1(v))]    (v<1)
       1    (v=1)
       v(ℓ+1)/2[((ℓ+2)v^2-ℓ) Q_ℓ(v)-2v Q_ℓ+1(v)]    (v>1)
    . ,

which coincides with the results of Ref. <cit.>.[After using that P_ℓ(-x)=(-1)^ℓP_ℓ(x) and Q_ℓ(-x)=(-1)^ℓ+1Q_ℓ(x).] For the limit of v→1 we took the total expression in terms of Legendre polynomials and used the asymptotic formulas near the singularity.[For example, we have that P_ℓ(x→ 1^-)→ 1 and Q_ℓ(x→ 1^-)→1/2ln(2/1-x)-γ_E-ψ(ℓ+1), where ψ(ℓ) is the digamma function and γ_E≈ 0.577 Euler’s constant.] All divergences cancel, and we obtain the well-known result for v=1.






 §.§ Analytical approximation in the  regime 


We now turn to the ℓ>kL regime, which corresponds to the case when the angular resolution associated with the multipole number is finer than the number of GW oscillations. This is the regime where we expect finite distance effects to take place. Noting that the ℓ>kL regime roughly corresponds to the regime where the spherical Bessel function does not oscillate, we use the power series expression for j_ℓ(q), namely

    j_ℓ(q)=√(π)/2∑_m=0^∞(-1)^m/m! Γ[ℓ+m+3/2](
    q/2)^ℓ+2m .

Using the expansion Eq. (<ref>), we can formally write Eq. (<ref>) as 

    I_ℓ(kL,v)=√(π)   ∑_m=0^∞(-1)^m(
    2v)^-1-ℓ-2m/m! Γ[ℓ+m+3/2]( Ci_ℓ+2m+1[vkL]-i  Si_ℓ+2m+1[vkL]) ,

where Ci_n[z]=∫_0^zt^n-1cos t dt and Si_n[z]=∫_0^zt^n-1sin t dt respectively are the generalized sine and cosine integrals.[See Sec. 8.21 of Ref. <cit.> for more details.] In the current form, there is no resummation formula for Eq. (<ref>), to the best of our knowledge. However, we can resum Eq. (<ref>) under a series of approximations.

We first note that, both Ci_n[z] and Si_n[z], where in our set up z=vkL, decrease similarly with decreasing z for z≪ n, namely Ci_n[z]∝ z^n/n and Si_n[z]∝ z^n/(1+n).[For z≪ 1 we have that Si_n[z]∝ z^n+1. However, this corresponds to the case where vkL≪1, where a good approximation is to neglect the exponential in Eq. (<ref>) as it barely oscillates. Then Eq. (<ref>) is well approximated by 
    ∫_0^kLdq j_ℓ(q)=√(π)/(l+1) Γ[ℓ+3/2](kL/2)^l+1 
       _1F_2(l+12;l+32,l+32;-(kL2)^2) .
] we also notice that the remaining factor is an oscillatory function slowly decaying with z. For the Ci_n[z] it is initially very similar to cos(z) and for the Si_n[z] to sin(z). We then use the approximation that

    Ci_n[z≪ n]≈z^n/ncos(z)   and   Si_n[z≪ n]≈z^n/1+nsin(z) .

Then, we find that

    I_ℓ(kL,v)   ≈√(π)∑_m=0^∞(-1)^m(
    2kL)^1-ℓ-2m/m! Γ[ℓ+m+3/2](cos(vkL)/ℓ+2m+1-isin(vkL)/ℓ+2m+2)
       
    =√(π)/(l+1) Γ[ℓ+3/2](kL/2)^l+1{ 
       _1F_2(l+12;l+32,l+32;-(kL2)^2)cos(vkL)
                       -isin(vkL) (l+1)/l+2 
       _1F_2(l2+1;l2+2,l+32;-(kL2)^2)} ,

    where _1F_2(a;b,c;-x^2) is the generalized Hypergeometric function. We note that this approximation is close to considering a constant phase in the integral Eq. (<ref>) and integrating the spherical Bessel function only. In fact, since v<1, we expect that the approximation should still be valid for ℓ>vkL, which goes below the cut-off at ℓ∼ kL. Thus, this approximation is particularly good for the case with v≪1.


In the current form, Eq. (<ref>) is not very enlightening. So, let us discuss some interesting limits. First, keeping the leading order term in kL/ℓ in a Taylor series expansion yields

    I_ℓ(kL,v)   ≈√(π)/(ℓ+1) Γ[ℓ+3/2](kL/2)^ℓ+1(cos(vkL)-i (l+1)/l+2sin(vkL))
       ≈e^ℓ/√(2)ℓ(kL/2ℓ)^ℓ+1e^-ivkL .

Note that, despite the exponential dependence on ℓ, Eq. (<ref>) quickly decays for ℓ≫ kL.
In the transitional regime, where ℓ≳ kL, the kL/ℓ dependence can be inferred from the following. First, we note that for ℓ≳ kL, the integrand in Eq. (<ref>) peaks at the upper integration limit, that is, at q=kL. We then approximate the integral by the value of the integrand at the maximum, which gives I_ℓ(kL≲ℓ,v)∼ j_ℓ(kL)e^-ivkL up to O(1) errors. We then use the approximation formula in transitional regions to j_ℓ(kL) given by Nicholson (see Sec. 8.43 of Ref. <cit.>), which reads

    j_ℓ(kL≲ℓ)   ≈√(2/3π)√(ν-kL)/kLK_1/3[2 √(2)
       (ν-kL)^3/2/3 √(kL)]
       ≈1/2^3/4ℓ(1-kL/ℓ)^1/4(ℓ/kL)^3/4e^-ℓ√(ℓ/kL)2 √(2)(1-kL/ℓ)^3/2/3 ,

where ν=ℓ+1/2, K_1/3[x] is the modified Bessel function of the second kind and in the last step we took the large ℓ limit and we approximated K_1/3[x]∼√(π2x)e^-x. Thus, for ℓ≳ kL, the initial decay of Eq. (<ref>) is proportional to e^-ℓ√(ℓ/kL), which is close to the behaviour speculated in Ref. <cit.>. For ℓ≫ kL then it falls off as e^ℓ(kL/ℓ)^ℓ+1. In fact,  for practical purposes, we find that the approximation given by

    I_ℓ(kL≲ℓ,v)   ≈e^-ivkL/2^3/4ℓ(1-kL/ℓ)^1/4(ℓ/kL)^3/4e^-ℓ√(ℓ/kL)2 √(2)(1-kL/ℓ)^3/2/3 ,

already gives a good approximation to the integral in the relevant range of ℓ≳ kL, and it is much less numerically expensive than Eq. (<ref>). Though the approximation is not good for ℓ≫ kL, the integral is already several orders of magnitude smaller and does not contribute to the Hellings-Downs curve (<ref>).  




We show in Fig. <ref> the values of the integral I_ℓ(kL,v) for v=1/2 and v=10^-3 with kL=100. We choose a relatively low value of kL for illustration purposes. Later, we will fix kL=1000 to estimate the ORF numerically. In Fig. <ref> we show the numerical results together with the approximation (<ref>) for ℓ<kL and both Eqs. (<ref>) and (<ref>) for ℓ>kL. See how the qualitative approximation given by the modified Bessel function, Eq. (<ref>), describes well the exponential decay of the integral for ℓ>kL. We also checked numerically that the Taylor series expansion of Eq. (<ref>), namely Eq. (<ref>), describes well the decay but only for ℓ≫ kL. It gives a bad approximation for ℓ near kL. We conclude that the integral I_ℓ(kL,v) is well approximated by the large kL approximation (<ref>) and a cut-off at ℓ∼√(1-v^2)kL (see the discussion around Eq. (<ref>)). We also checked numerically that the value of ℓ_ cut describes well the start of the decay.

We end this section by noting that the calculations in this subsection are valid for any value of v. Nevertheless, for v≥ 1, the summation in the Hellings-Downs curve, Eq. (<ref>), converges quickly enough so that the effects are negligible.




§ FINITE DISTANCE EFFECTS TO THE ORF 


Here we compute the ORF (<ref>) and focus on the coincident pulsar limit, where previous approximations gave a divergent result. We start by computing a_ℓ given by (<ref>), which contains |c_ℓ|^2. For numerical calculations, we use the full c_ℓ given by (<ref>). For the analytical approximations we neglect the Bessel functions in (<ref>) and take

    |c_ℓ|^2≈ 4v^2(ℓ+1)^2 |((ℓ+2)v^2-ℓ) I_ℓ(kL,v)-2iv  I_ℓ+1(kL,v)|^2 .
 
Though we neglect the Bessel functions, it is interesting to note that for ℓ≫ kL, a Taylor series expansion of the Bessel term in (<ref>) yields

    ℓ+ivkL/kL(ℓ+2)j_ℓ(kL)-(ℓ+4) j_ℓ+1(kL)≈√(π) (l+2) (kL)^ℓ-1ℓ/ 2^ℓ+1Γ(l+3/2)≈e^ℓ/2^ℓ+3/2(kL/ℓ)^ℓ-1 ,

where in the last step, we took the large ℓ limit. It is interesting to note that in this limit, the contribution from the spherical Bessel functions, Eq. (<ref>) decays with two powers of (kL/ℓ) less than Eq. (<ref>). We confirm numerically that the Bessel contribution to c_ℓ quickly dominates for ℓ> kL. This is nevertheless unimportant as the suppression is huge, and it does not contribute to the ORF.

Because in Sec. (<ref>) we showed that c_ℓ is exponentially suppressed for ℓ>√(1-v^2) kL, we consider for analytical purposes that the ORF (<ref>) for v<1 is well approximated by

    Γ(kL,ξ)= C∑_ℓ=2^ℓ_ cut a_ℓ P_ℓ(cosξ) ,
 
where we introduced a cut-off to the sum at ℓ_ cut=√(1-v^2) kL.[Strictly speaking the sum is from ℓ=2 until the integer number closer to ℓ_ cut. For large ℓ this does not make a difference.] Thus, we only need to know the value of a_ℓ in the ℓ≪ kL regime. As shown in Ref. <cit.>, the large ℓ limit of c_ℓ in the case of v<1 yields[We used that P_ℓ(cosθ)≈√(θ/sinθ)J_0((ℓ+12)θ) and Q_ℓ(cosθ)≈ -π2√(θ/sinθ)Y_0((ℓ+12)θ). ]

    c_ℓ(ℓ≪ kL;ℓ≫1)≈ (-1)^ℓ iπ   
    v(ℓ+1)√(θ/sinθ)
       ×[((ℓ+2)v^2-ℓ)H_0^(1)((ℓ+12)θ)-2vH_0^(1)((ℓ+32)θ)] ,

where cosθ=v and H_0^(1)(x) is the Hankle function of the first kind. Further using that in the ℓ≫1 limit H_0^(1)(x≫1)≈√(2π x)e^i(x-π/4) and that the prefactor in a_ℓ (<ref>) goes as 4/ℓ^3, one obtains that

    a_ℓ(ℓ≪ kL;ℓ≫1)≈ 8π v^2(1-v^2)^3/2 .

Namely, for large enough ℓ the coefficients a_ℓ are constant. 



We show in Fig. <ref> the a_ℓ for the case of kL=100 and kL=1000, respectively, on the left and right panels. The former is for illustration purposes; the latter is of the order of current observations.  In the left panel, we also fixed v=10^-2 as a small v example, but the approximations work well in other cases, too. The exception is v≳ 1/3 where the cut-off in ℓ∼√(1-v^2)kL is noticeably different than kL. In that case, the ℓ>kL approximation (<ref>) gives the right exponential decay but shifted to ℓ=kL. For smaller values of v, the approximations get better, and Eq. (<ref>) is relatively good for some ℓ inside the ℓ<kL regime. In the right panel, we show the case of kL=1000 for v={9/10,1/2,10^-1,10^-2}. There it is clear that there is a very sharp cut-off roughly at ℓ∼√(1-v^2) kL. Let us remind the reader that value for ℓ_ cut is analytically derived in the beginning of Sec. <ref>. Though one could try to numerically find an effective cut-off, we will see shortly that ℓ_ cut given by Eq. (<ref>) recovers the value of the autocorrelation function.

Now, let us discuss the impact of Eq. (<ref>) on the ORF. In Ref. <cit.>, the constant approximation is taken up to ℓ→∞, which, using the resummation formulas of the Legendre polynomials, yields a factor 1/√(1-cosξ). However, this is divergent in the ξ→ 0 limit, as noted in Refs. <cit.>. Physically speaking, divergence is due to the fact in the strict kL→∞ limit, the pulsars are infinitely far away, and the “surfing” <cit.> on top of the GW takes place infinitely. In our terminology, the cut-off is sent to ℓ_ cut→∞, and the sum diverges proportionally to ℓ_ cut. However, for finite values of kL, we have that the value of ORF (<ref>) in the ξ→0 limit is finite and given by 

    Γ(kL,ξ=0)=3/64∑_ℓ=2^ℓ_ cut a_ℓ ,

where we set the normalization constant to 3/64 for comparison with Ref. <cit.>.
We give an analytical estimate as follows. Using that kL≫1 and so that ℓ_ cut≫1, we note that most of the contribution to the coincident limit comes from the large ℓ sum. We then assume that a_ℓ is constant for all ℓ and given by Eq. (<ref>), which yields that

    Γ(kL,ξ→0)≈3/64a_ℓℓ_ cut=3π/8 kL  v^2(1-v^2)^2 .

We numerically checked for kL=1000 that this limit is accurate within 0.1%. This further supports our analytical estimation.[It is interesting that Γ(kL,ξ→0) is not necessarily large despite the fast growth for small but not too small ξ. In fact, for v<1/√(kL), it achieves values less than in the GR case. This limit is, unfortunately, not so interesting from the observational point of view, as in PTAs the significant for v<0.4 the statistical significants drops below 3σ as show in Ref. <cit.>using the CPTA data.] We also note that our limit for ξ→0 of the ORF matches exactly the value of the auto-correlation obtained in Ref. <cit.>; see their Eq. (28)  with the identification of vkL=2π fL. 



We show in Fig. (<ref>) the ORF (<ref>) for kL=100 (top panels) and kL=1000 (bottom panels) and v={9/10,1/2,10^-1,10^-2}. We also include the GR case (v=1) for comparison. In solid lines, we show the numerical result. The nearby dashed lines show the analytical approximation given by Eq. (<ref>) with c_ℓ from Eq. (<ref>) together with Eq. (<ref>). First, look at the top and bottom panels on the right. Note how the analytical approximation agrees well with the numerical one in the small ξ limit. We also see that the ORF approaches a constant for values of ξ below ξ∼ 2/ℓ_ cut. We estimated such value of ξ by requiring that the next order correction in a Taylor series expansion of the Legendre polynomials for ξ≪1 becomes of O(1). This estimate yields log_10ξ≈ -1.7 and log_10ξ≈ -2.6 respectively for kL=100 and kL=1000, which is a reasonable estimate compared with the right panel of Fig. (<ref>). We also see that the asymptotic value for ξ→0 matches the analytical approximation given by Eq. (<ref>) in the large kL limit.

Now, looking at the left top and bottom panels of Fig. (<ref>) we see that the numerical and analytical approximation are similar, though they do not overlap as much as in the small ξ regime. We see that for kL=100, there are more wiggles than kL=1000 in the numerical and analytical cases due to finite distance effects. Increasing the value of kL smooths the curve. Besides the wiggles, we still see some differences between the numerical and analytical results, even for kL=1000. There it is likely that some component is due to numerical errors as the I_ℓ (<ref>) is the integral of highly oscillatory functions. Nevertheless, it is plausible that finite distance effects remain even for relatively large kL. From Fig. <ref>, it is clear that the analytical approximation given by Eq. (<ref>) does not capture the behavior of a_ℓ near ℓ_ cut. It would be interesting to carry out a more detailed and precise numerical study and, perhaps, refine the analytical approximations to the ORF for v<1.




§ CONCLUSIONS 


The ORF in PTAs provides a complementary way to test gravity by measuring the phase velocity v of the GWs. In general, modified gravity predicts a phase velocity with v≠ 1, such as in scalar-tensor theories (see Ref. <cit.> for a review), and possible with a k dependence, as in massive gravity <cit.>. Thus, departures of v=1 in the ORF would hint at modifications of gravity.

Motivated by recent studies <cit.>, that encountered a divergent ORF in the small angle limit when kL→∞ for the v<1 case, we investigated finite distance effects on the ORF. To do that, we reformulated the calculation of the harmonic coefficients c_ℓ in terms of a definite integral of spherical Bessel functions given by Eq. (<ref>). As we argued at the beginning of Sec. <ref>, the approximation derived by Refs. <cit.> is valid up to ℓ_ cut∼√(1-v^2) kL. For ℓ≳ℓ_ cut the integral (and c_ℓ) decays exponentially, roughly as (ℓ/kL)^3/4× e^-ℓ√(ℓ/kL), see Eq. (<ref>) and the discussion around it. For ℓ≫ kL it then decays as e^ℓ/ℓ×(kL/ℓ)^ℓ+1, see Eq. (<ref>).

The physical interpretation of our results, based on the nice physical explanations of Ref. <cit.>, is as follows. First, the factor kL is the ratio of the “arm-length” (the distance to the pulsar) to the wavelength of the GW. Namely, kL tells us how many GW oscillations undergoes the pulse from the pulsar until Earth. More precisely, the factor kL enters in the redshift transfer transfer function (roughly the term inside parenthesis in Eq. (<ref>)) in the antenna pattern of PTAs, yielding an angular pattern where points with vanishing redshift are separated by an angle v+cosξ=2n/kL where n∈ℤ^+ (see also Fig. 5 and 6 of Ref. <cit.> and Fig. <ref>). Thus, the typical angular separation of these features is ξ∼ 1/kL. Then, in the spherical harmonic decomposition, where the multipole number is roughly related to the inverse angular separation (i.e. ℓ∼π/ξ), it is expected that multipole numbers with ℓ≳ kL contain no more information of the antenna response and their coefficients are, therefore, suppressed. 

Focusing on the small angle limit of the ORF, we found that using the large ℓ approximation for a_ℓ (<ref>), first derived by Refs. <cit.>, up to the cut-off ℓ_ cut (<ref>) exactly recovers the value of the autocorrelation derived in Ref. <cit.>. We also checked our analytical estimates for the ORF numerically for kL=1000 and found a good agreement in the small angle regime. We thus showed that the ORF for v<1 is finite and that it correctly approaches the limiting value for ξ→ 0. Outside the small ξ regime, we find that the numerical and analytical approximations are similar, although they do not match as well as in the small ξ limit. This difference could be partly due to numerical errors, but it is possible that finite distance effects on the ORF will remain for relatively large values of kL. Basically, we find that the a_ℓ≈ constant approximation (<ref>) for ℓ≫ 1 is not good for values close to ℓ∼ℓ_ cut. It is therefore important to carry out more precise numerical studies and perhaps refine the analytical templates of the ORF for v<1.

We also note that while we focused on the case v<1 our approach is also valid for any value of v. However, for v≥ 1 the numerical integration of Eq. (<ref>) might be more challenging in the regime ℓ≪ kL, with kL≫1, as the oscillations from the exponential have a higher frequency and there are non-trivial cancellations. Nevertheless, for v>1 there are already analytical expressions for c_ℓ after dropping the exponential in (<ref>), see, e.g., Refs. <cit.>.  Thus, our approach is most suitable for v<1 as it is sensitive to finite distance effects. For instance, we could numerically compute c_ℓ up to ℓ=1100 for kL=1000, which is challenging to do in the formulation of Legendre polynomials (<ref>). Interestingly, Ref. <cit.> tested the value of v using the ORF of CPTA data. They found that while not statistically significant there are two preferred values of v=0.98 and v=1.04 in the data. For v<0.4 though, the statistical significance dropped below 3σ. It would be interesting to derive more precise templates with our formulation for v≲ 1 and test them against the data.

Lastly, we note that throughout the paper, we assumed equal distance pulsars for simplicity. It would be interesting to redo the calculations when the distance to the pulsars is different from each other, which is the more realistic situation. We leave this issue for future work, as it is out of the scope of this paper. We end by speculating that it may be possible to derive an exact formula for the coefficients a_ℓ without assuming the limit kL≫ 1. Looking at the exact expression for the autocorrelation given by Eq. (23) of Ref. <cit.>, we notice that all terms in that formula can be expressed as an infinite sum of the product of spherical Bessel functions and Legendre polynomials (see, e.g., Secs. 6.10 and 10.6 of Ref. <cit.>). It is thus plausible that a_ℓ may also be written as a series expansion in terms of such functions. In the worst-case scenario, it may only be possible in the exact ξ→ 0 limit. We leave this exercise for future work.





§ ACKNOWLEDGMENTS
 
G.D. would like to thank B. Allen, M. Sasaki, K. Schmitz and J. Tränkle for their helpful discussions. G.D. is supported by the DFG under the Emmy-Noether program grant no. DO 2574/1-1, project number 496592360.



 



 Paper Title: Prospects for Cosmological Research with the FAST Array: 21-cm Intensity Mapping Survey Observation Strategies.


	
	
	
 
	
	
	Key Laboratory of Cosmology and Astrophysics (Liaoning) & College of Sciences, Northeastern University, Shenyang 110819, China
	
	Key Laboratory of Cosmology and Astrophysics (Liaoning) & College of Sciences, Northeastern University, Shenyang 110819, China
	
	Key Laboratory of Cosmology and Astrophysics (Liaoning) & College of Sciences, Northeastern University, Shenyang 110819, China
	
	Key Laboratory of Cosmology and Astrophysics (Liaoning) & College of Sciences, Northeastern University, Shenyang 110819, China
	
	
	
	
	
	Corresponding author.
zhangxin@mail.neu.edu.cn
	
	
	Key Laboratory of Cosmology and Astrophysics (Liaoning) & College of Sciences, Northeastern University, Shenyang 110819, China
	Key Laboratory of Data Analytics and Optimization for Smart Industry (Ministry of Education), Northeastern University, Shenyang 110819, China
	National Frontiers Science Center for Industrial Intelligence and Systems Optimization, Northeastern University, Shenyang 110819, China
	
	
		
	
	
	
	
	
	§ ABSTRACT 
	Precise cosmological measurements are essential for understanding the evolution of the universe and the nature of dark energy. The Five-hundred-meter Aperture Spherical Telescope (FAST), the most sensitive single-dish radio telescope, has the potential to provide the precise cosmological measurements through neutral hydrogen 21-cm intensity mapping sky survey. This paper primarily explores the potential of technological upgrades for FAST in cosmology. The most crucial upgrade begins with equipping FAST with a wide-band receiver (0 < z < 2.5). This upgrade can enable FAST to achieve higher precision in cosmological parameter estimation than the Square Kilometre Array Phase-1 Mid frequency. On this basis, expanding to a FAST array (FASTA) consisting of six identical FAST would offer significant improvements in precision compared to FAST. Additionally, compared with the current results from the data combination of cosmic microwave background, baryon acoustic oscillations (optical galaxy surveys), and type Ia supernovae, FASTA can provide comparable constraints. Specifically, for the dark-energy equation-of-state parameters, FASTA can achieve σ(w_0) = 0.09 and σ(w_a) = 0.33.

		
	
	
	
	Prospects for Cosmological Research with the FAST Array: 21-cm Intensity Mapping Survey Observation Strategies
    Xin Zhang
    
==============================================================================================================



§ INTRODUCTION

 
    Dark energy is a component of the universe that causes the late-time cosmic acceleration. Although it is known to comprise 68% of the total energy density of the universe, the nature of dark energy remains unknown. The Λ cold dark matter (ΛCDM) model, the standard model of cosmology, treats dark energy as a constant energy density filling space homogeneously. However, the latest baryon acoustic oscillation (BAO) observation from the Dark Energy Spectroscopic Instrument (DESI) suggests that dark energy may potentially be a dynamical energy density evolving with time <cit.>. This finding indicates the possibility of dark-energy dynamical evolution, but more observations and advanced instruments are needed to fully confirm this and to understand the evolutionary dynamics of dark energy.

BAO from the large-scale structure (LSS) is a late-time cosmological probe that can be used as a standard ruler to study the expansion history of the universe <cit.>. Currently, BAO studies primarily rely on optical galaxy surveys <cit.>. While effective, these surveys are limited in terms of efficiency and coverage.  To overcome these limitations and achieve more precise cosmological measurements, advancements in radio astronomy have developed a promising method known as 21 cm intensity mapping (21 cm IM). After reionization, neutral hydrogen (H  I) mainly exists within galaxies. This technique measures the intensity of the characteristic 21 cm emission line from H  I to study the LSS <cit.>. This method does not require resolving individual galaxies, thus it has high survey efficiency. The characteristics of the 21 cm signal allow us to probe deeper into the universe and directly obtain redshift information. Recent forecasts and observational studies have confirmed the potential of 21 cm IM in cosmology <cit.>.





	
	To achieve more precise cosmological measurements, advanced radio telescopes are utilized in 21 cm IM. These telescopes include those currently in use, such as the Five-hundred-meter Aperture Spherical Telescope (FAST) <cit.>, the MeerKAT <cit.>, the Tianlai <cit.>, the Canadian Hydrogen Intensity Mapping Experiment (CHIME) <cit.>, and those planned for future use, such as the Square Kilometre Array (SKA) <cit.>, the Baryon Acoustic Oscillations from Integrated Neutral Gas Observations (BINGO) <cit.>, and the Hydrogen Intensity and Real-time Analysis eXperiment (HIRAX) <cit.>.

    
    
    
    Among these, FAST is particularly significant due to its large aperture and sensitivity. The current FAST has achieved substantial success in many areas, particularly in fast radio burst and gravitational wave research <cit.>. However, to harness its potential for 21 cm IM and to extend its capability for cosmological studies, several upgrades are planned for FAST  <cit.>. 
    
    This paper aims to explore how potential future upgrades to FAST could promote cosmological research by using 21 cm IM. For the expansion of FAST itself, equipping it with a wide-band receiver aims to extend its 21 cm signal detection capability, enabling broader redshift coverage <cit.>. This will allow us to observe a larger volume of the universe, aiding in constructing a more precise three-dimensional map of the LSS. Moreover, a phased array feed (PAF) is planned to enhance its survey efficiency <cit.>.
    Additionally, the expansion of the construction around FAST is also considered. The plan involves building 40-meter diameter spherical radio telescopes (40 m dishes) in two phases around FAST: the first phase involves constructing six 40 m dishes, and the second phase will increase this number to twenty. 
	The most significant upgrade is the FAST Array (FASTA), which plans to add five more identical 500-meter diameter telescopes to join the current FAST <cit.>. This network will significantly improve sensitivity and survey efficiency, enhancing the ability to explore LSS and providing higher precision in cosmological parameter estimation.
	
    In this paper, we present a forecast for estimating cosmological parameters using simulated 21 cm IM data and discuss how the various extensions of FAST will improve the precision of these constraints.
    
    In Section <ref>, we introduce the 21 cm IM simulation and the methods used to constrain cosmological parameters. In Section <ref>, we discuss the performance of various configurations for 21 cm IM and the resulting cosmological constraints. In Section <ref>, we conclude with perspectives for future research.
	
	In our analysis, we adopt the Planck best-fit ΛCDM model as fiducial cosmology  <cit.>. The parameters are H_0 = 67.3 km s^-1 Mpc^-1, Ω_m = 0.317, Ω_b = 0.0495, Ω_k = 0, σ_8 = 0.812, and n_s = 0.965.
	
	



§ 21 CM INTENSITY MAPPING

   

 §.§ Signal

   21 cm IM measures the intensity of 21 cm signals and converts it into brightness temperature.  The mean temperature of H  I can be expressed as <cit.>
   
    T_b = 3/32 π𝒽 c^3 A_10/k_B m_pν_21^2(1+z)^2/H(z)Ω_H   I(z) ρ_c, 0 ,

   where 𝒽 is the Planck constant, c is the speed of light, A_10 is the Einstein coefficient for spontaneous emission <cit.>, k_B is the Boltzmann constant, m_p is the proton mass, ν_21 is the H  I line frequency, H(z) is the Hubble parameter, Ω_H  I(z) is the fractional density of H  I, and ρ_c, 0 is the current critical density of the universe.
   
    The signal covariance is expressed in terms of the power spectrum
    <cit.>
		
    C^S(q, y) = T_b^2(z_i)/r^2 r_ν (b_H   I + fμ^2)^2 exp(-k^2μ^2σ_NL^2)  P(k, z),

	where q = k_⊥ r and y = k_ r_ν. r denotes the comoving distance to z, and r_ν corresponds to the comoving distance for the frequency interval. b_H   I represents the bias factor of H  I <cit.>. f is the linear growth rate of the structure. μ = k_ / k is the cosine of the angle between the wavevector k and the line of sight. σ_NL=7 Mpc represents the non-linear dispersion scale <cit.>. P(k, z) is the matter power spectrum at wavenumber k and z, which can be calculated by CAMB <cit.>.
	
		
	

 §.§ Noise

	The noise covariance can be written as <cit.>
	
    C^N(q, y)=T_sys ^2/t_tot Δν N_b N_d U_bin B_⊥^-2 B_^-1,

	where T_sys includes the instrumental temperature T_inst and the sky background radiation T_sky≈ 60  K× (ν / 300 MHz)^-2.5.  t_tot represents the total integration time, N_b is the number of beams, N_d is the number of dishes in the array, and Δν is the total bandwidth. U_bin = S_areaΔν̃ represents the survey volume element at a given redshift, where S_area is the survey area and Δν̃ = Δν/ν_21 is the dimensionless bandwidth for a given redshift bin. B_ is the radial beam response
	
    B_(y)=exp(-(y δν / ν_21)^2/16 ln 2),

	where δν is the bandwidth of an individual frequency channel. B_⊥ is the transverse beam response in single dish mode
	
    B_⊥(q)=exp(-(q λ/D_dish)^2/16 ln 2),

	where D_dish is the diameter of the dish.

	

 §.§ Foreground

	The foreground is an important component of the signal received by the telescope. It primarily originates from the galaxy and extragalactic point sources. At this stage, the effectiveness of methods for subtracting foregrounds from observational data has been very limited. The mainstream methods mainly include Polynomial Fitting, Principal Component Analysis, Independent Component Analysis, and Artificial Intelligence <cit.>. We assume a method that can effectively subtract the foreground, and the residual foreground model can be written as <cit.>
	
    C^F(q, y)=ε_FG^2 ∑_X A_X(l_p/2 π q)^n_X(ν_p/ν_i)^m_X,

	where ε_FG is a factor that describes the efficiency of foreground subtraction. ε_FG = 1 represents no foreground subtraction, while ε_FG = 0 represents perfect foreground subtraction. We assume ε_FG=10^-6, as this level is effective for extracting cosmological signals. X is the different foreground component shown in Table <ref>. 




    






 §.§  Instrument parameters

    

    
    
In this paper, we consider several different configurations of FAST and Square Kilometre Array Phase 1 - Mid frequency (SKA1-Mid) for simulating the 21 cm IM. The specific instrument parameters are listed in Table <ref>.

The current capability of FAST to detect 21 cm signals covers the redshift range 0 < z < 0.35. A new wide-band receiver under test extends this range to 0 < z < 1.84 <cit.>. Previous galaxy surveys, however, have performed precise BAO measurements within z < 2.5, achieving 1%-2% accuracy <cit.>. This paper assumes the upgraded new wide-band receiver on FAST can detect 21 cm signals up to 0 < z < 2.5. 

Additionally, constructing 6×40 m dishes or 20×40 m dishes around FAST is considered. These dishes will use the same receiver and feed system as FAST, so their frequency bands and N_b will be identical. Given the outstanding performance of FAST, we also consider constructing five additional identical FAST telescopes to form FASTA, significantly improving survey efficiency. Furthermore, we consider equipping FAST with a PAF to increase the value of N_b to 100.

Regarding instrument temperature, in Ref. <cit.> the author measured the temperature of FAST near boresight to be 20 K in the L-band. We assume that the frequency band can be broadened while maintaining the instrument temperature at 20 K. For the PAF, we assume the temperature to be approximately 30 K due to the increase in the number of beams. The total observation time is assumed to be 10000 hours. The calculation for the survey area is as follows

	
    S_area = η∫_ϕ∫_θsinθdθdϕ ,

	where η represents the fraction of the initial survey area that remains usable for scientific observation after excluding the Milky Way. For FAST/FASTA, we considered η = 0.7 and the range of declination from -14^∘ to +66^∘, yielding an area of approximately 8100 deg^2.

For SKA1-Mid, we considered band I (0.35<z<3) planned for 21 cm IM, with the survey area of 20000 deg^2 and the total observation time of 10000 hours. The instrument temperature is as follows <cit.>

    T_inst=15  K+30  K(f/GHz-0.75)^2 .

	

 §.§ The Fisher matrix

	
	
	
	
    
	
	
	
	
	
	
	
	
	
	To achieve optimal observational results, different telescopes require different survey strategies, so the methods for calculating C^tot also vary. For a single dish or identical dishes observing the same area of the sky (such as FAST or FASTA), the total covariance matrix is expressed as

    C^tot = C^S + C^N + C^F.


For the combination of different telescopes, we consider two scenarios. The first scenario involves two telescopes in close geographic proximity observing the same area of the sky, such as combining 40 m dishes and FAST. This approach maximizes the reduction of thermal noise impact on observations. Since C^S and C^F are only related to the observed sky area, they remain fixed. Different telescopes have varying C^N due to their differing thermal noise integration effects. Therefore, the total covariance matrix can be expressed as

    C^tot = C^S + ((C_1^N)^-1 + (C_2^N)^-1)^-1 + C^F,

where C_1^N and C_2^N are the noise covariance matrices of the different dishes, with (C^N)^-1 denoting their inverse matrices. The second scenario involves two telescopes with significant differences in latitude, resulting in a very small common observable sky area. Therefore, they must observe different areas of the sky, which also maximizes the survey volume. Since both the observed sky areas and the telescopes are different, the total covariance matrix is given by

    C^tot = ((C_1^tot)^-1 + (C_2^tot)^-1)^-1,

where C_1^tot and C_2^tot are the total covariance matrices of the different dishes.

	After obtaining the total Fisher matrix C^tot, we can transform it into the Fisher matrix of observations {p_i} <cit.>
	
    F_i j = 1/2 U_bin∫d^2 q  d y/(2 π)^3[∂ln C^tot(q, y)/∂ p_i∂ln C^tot(q, y)/∂ p_j].

	In this paper, we mainly calculate the Fisher matrix for several key observables to constain the evolution of LSS, as well as the growth and geometry of the universe. These include the transverse distance indicator D_A(z), radial distance indicator H(z) and the parameter combination fσ_8(z) from redshift space distortions (RSDs), where f(z) is growth rate and σ_8 is the amplitude of matter density fluctuations on a scale of 8 h^-1 Mpc (with h the dimensionless Hubble constant).
    
   
    
    
	

 §.§ Cosmological parameter constraints

    We use the Fisher matrix to propagate errors from observational errors to the BAO parameters D_A(z), H(z), and fσ_8(z). Then, we use the errors of BAO parameters to form the likelihood function for cosmological models. Finally, we employ Markov Chain Monte Carlo (MCMC) analysis to constrain the cosmological parameters <cit.>.
 
 To compare with the mainstream observations, we calculated the constraints from the latest observations. The data used include: CMB – Planck 2018 high-ℓ TT, TE, EE, and low-ℓ TT, EE, and PR4 lensing data <cit.>; BAO – DESI data <cit.>; and supernova (SN) – Pantheon+ data <cit.>. The data combination of CMB, BAO, and SN in this paper is also abbreviated as CBS.

	The equation of state (EoS) of dark energy is given by w = p/ρ, where p is the pressure and ρ is the energy density. We employ different models to describe its dynamics. (i) ΛCDM: w = -1. (ii) wCDM: w(z) = w, where w is a constant EoS for dark energy. (iii) w_0w_aCDM: w varies with redshift according to the parameterization w(z) = w_0 + w_az / (1 + z), where w_0 is the present-day value of the equation of state parameter and w_a describes its evolutionary behavior.
	


§ RESULTS AND DISCUSSION

	In this section, we  first discuss the capabilities of various instruments and their combinations in 21 cm IM using the relative errors of BAO parameters {ε(D_A(z)), ε(H(z)), ε(fσ_8(z))}, where ε(ξ) = σ(ξ)/ξ. We then present the cosmological parameter constraints employing FAST, SKA1-Mid, FASTA, and CBS, followed by comparison and discussion of the results.
    
    
    
    
    
	
	

 §.§ Combination of FAST and SKA-Mid

 
 In this subsection, we discuss the capabilities of SKA1-Mid and FAST with a wide-band receiver, as well as their combination in 21 cm IM. Due to geographical factors, the overlapping observable sky area for FAST and SKA1-Mid is very limited, which necessitates a strategy where each telescope observes different sky regions.

From Figure <ref>, it is evident that FAST and SKA1-Mid have distinct advantages across different redshift ranges. SKA1-Mid achieves tight constraints within the range of 0.35 < z < 0.8 due to its high survey efficiency. In this range, the relative errors in BAO parameters are typically below 3%. On the other hand, FAST attains robust constraints within the range of 0.8 < z < 2.5 because of its high transverse resolution and sensitivity. In this range, the relative errors in BAO parameters are typically around or below 4%.


Their combined observations enable precise measurement of BAO parameters across the redshift range of 0 < z < 2.5, achieving constraints on D_A(z), H(z), and fσ_8(z) to below 2% for most of this range, and to as low as 1% at z ≈ 1.

Upgrading the existing receiver to detect 21 cm signals up to 0 < z < 2.5 will enable FAST to utilize its high-redshift advantage in 21 cm IM. Combined with SKA1-Mid, this complementary strategy will improve measurements of BAO parameters across a wide redshift range.

    

 §.§ Combination of FAST and 40 m dishes

	In this subsection, we discuss the extent to which incorporating 40 m dishes with FAST enhances the constraints on BAO parameters in 21 cm IM. An array of 40 m dishes, located in close geographical proximity to FAST, enhances the survey efficiency by enabling observations of the same sky regions.

Figure <ref> illustrates the fractional constraints achievable on D_A(z), H(z), and fσ_8(z) using 21 cm IM with an array of 40 m dishes, FAST, and their combined configurations. The 40 m dishes perform better at low redshifts because they have higher survey efficiency. FAST is more effective at higher redshifts due to its greater sensitivity and resolution. The combination of FAST and 40 m dishes significantly reduces observational errors, particularly at high redshifts. The relative errors in BAO parameters are typically below 4%, with errors reaching as low as 2% at z ≈ 1.

For the configurations of 6× 40 m dishes and 20× 40 m dishes combined with FAST, Figure <ref> shows that the larger array provides only marginal improvements. This result suggests that the combination of FAST with 6× 40 m dishes offers a more optimal configuration for the 21 cm IM survey.

     
    

 §.§ FASTA

	In this subsection, we present the cosmological parameters constraints by using simulations of 21 cm IM data with different instruments and current observational data from CBS. Figure <ref> shows the  parameter constraints under the 1σ and 2σ confidence intervals for the ΛCDM, wCDM, and w_0w_aCDM models. The specific values are listed in Table <ref>.

Compared to the results in Ref. <cit.> for FAST (0 < z < 0.35), a wide-band receiver proposed in this paper utilizes the resolution advantage of the FAST at high redshifts, significantly enhancing the constrains on cosmological parameters. The main specific parameter constraints are as follows: for the ΛCDM model, FAST achieves σ(Ω_m) = 0.0076 and σ(H_0) = 0.51  km s^-1 Mpc^-1. For the wCDM model, the constraint is σ(w) = 0.036. For the w_0w_aCDM model, the constraints are σ(w_0) = 0.14, and σ(w_a) = 0.57. Compared to SKA1-Mid, FAST also has higher sensitivity and lower receiver noise, which enables it to achieve better parameter constraints. For the dark-energy EoS parameters, there is an improvement of approximately 20% for w, 12.5% for w_0, and 21.9% for w_a.

	Further advances are observed with FASTA. From Figure <ref>, FASTA performs well in observing BAO parameters across the entire band, achieving an observational error as low as 1%. This precise observation of BAO parameters translates into tight constraints on cosmological parameters. 
 
In the ΛCDM model, FASTA provides best constraints, σ(Ω_m) = 3.6 × 10^-3 and σ(H_0) = 0.26  km s^-1 Mpc^-1, which surpasses other instruments and CBS.
In the wCDM model, FASTA also provides best constraints, σ(Ω_m) = 0.0037, σ(H_0) = 0.39  km s^-1 Mpc^-1, and σ(w) = 0.02. For the parameter w, FASTA represents an improvement of approximately 55.56% compared to SKA1-Mid, 44.44% compared to FAST, and 16.67% compared to CBS.
In the w_0w_aCDM model, FASTA achieves highly precise parameter constraints, with σ(Ω_m) = 0.012, σ(H_0) = 0.89  km s^-1 Mpc^-1, σ(w_0) = 0.09, and σ(w_a) = 0.33. For w_0 and w_a, FASTA represents an improvement of approximately 43.75% and 54.79% compared to SKA1-Mid, and 35.71% and 42.11% compared to FAST, while achieving constraints comparable to CBS using 21 cm IM  alone.

In summary, FASTA significantly improves cosmological parameter constraints across various models, surpassing other instruments and achieving results comparable to CBS using 21 cm IM  alone.



 
 
 
	
        
	

 §.§ Phased array feed

	In this subsection, we discuss the enhancements and challenges of upgrading the feed of FAST to PAF for 21 cm IM. PAF can achieve various beamforming modalities by adjusting the phase and amplitude of each element in the array, thereby generating multiple beams simultaneously. This capability covers a larger field of view (FOV) and enables rapidly beam steering and scanning <cit.>.  At the same time, different beamforming strategies significantly impact noise characteristics, and data calibration and processing.  We considered two beamforming strategies in terms of survey efficiency.
 
    In the first scenario, we consider a fixed FOV and maximize the N_b within this field. The size of a beam is given by θ = c/(ν D_dish). If there is no overlap between the beams, the maximum N_b can be written as
    
    N_b = FOV·(ν D/c)^2 .


    In this case, C^N follows the same pattern as described in Ref. <cit.>
    
    C^N(q, y) → C^N(q, y) ×1,    ν>ν_crit 
    (ν_crit  / ν)^2,    ν⩽ν_crit  .

	We assume ν_crit=1420 MHz. As shown in Figure <ref>, the errors increase significantly at low frequencies. This is because at low frequencies, the beam size increases, resulting in a decrease in N_b.
 
	Another scenario is a fixed N_b. We assume the case of 100 beams. As shown in Figure <ref>, the increase in the N_b significantly enhances survey efficiency, achieving best constraint result of 2% around z = 1. However, in practice, the FOV does not increase indefinitely as the beam size increases at lower frequencies. As the FOV certainly deviates from the boresight, the survey efficiency decreases. This requires actual measurements to PAF, but here we make an ideal estimate. The real situation is expected to lie between the fixed FOV and the fixed N_b scenarios.


	

§ CONCLUSION

	In this paper, we explore several potential upgrades to FAST using 21 cm IM to enhance cosmological research. Those upgrades include FAST with a wide-band receiver, the combination of 40 m dishes and FAST, FASTA, and FAST with a PAF. We simulated the full 21 cm IM power spectrum to constrain the BAO and RSD parameters D_A(z), H(z), and fσ_8(z), and used these results to evaluate the capabilities of these different configurations. Finally, we used the simulated BAO and RSD data to constrain the cosmological parameters of the ΛCDM, wCDM, and w_0w_aCDM models.

	Firstly, the wide-band receiver utilizes the capabilities of FAST in the redshift range 0.8 < z < 2.5, achieving better cosmological parameter constraints than SKA1-Mid. It also enables FAST to form a complementary observation with SKA1-Mid, which has strengths in the redshift range of 0 < z < 0.8, providing high-precision observations across a wider redshift range.
 
	Secondly, we discuss the combination of the 40 m dishes and FAST. Incorporating 40 m dishes with FAST significantly enhances survey efficiency, thereby reducing BAO parameter observational errors to typically below 2%. For the combination with FAST, 6×40 m dishes are a more balanced configuration compared to 20×40 m dishes for 21 cm IM.
	
	Next, we discuss the cosmological parameter constraints for FASTA. Through BAO parameter measurements approaching 1% accuracy, it shows excellent performance in constraining cosmological parameters. Specifically, in the w_0w_aCDM model, FASTA achieves highly precise parameter constraints, with improvements of 43.75% for w_0 and 54.79% for w_a over SKA1-Mid, and 35.71% for w_0 and 42.11% for w_a over FAST, achieving results comparable to CBS using 21 cm IM  alone.


	Lastly, we analyze FAST with a PAF. We consider two beamforming strategies, a fixed FOV and a fixed N_b. The real situation is likely to fall between these two cases. This provides a basis for understanding the advantages and limitations of different beamforming strategies for 21 cm IM. A balanced beamforming strategy will be beneficial for effectively utilizing PAF.
	
	In summary, these upgrades to FAST have enormous potential to improve the precision of cosmological measurements. Despite facing some technical challenges, they can greatly enhance our understanding of the evolution of the universe and the nature of dark energy.
	
	
	
	We thank Guang-Chen Sun, Ji-Guo Zhang,  Zi-Qiang Zhao, and Lu Feng for helpful discussions and suggestions. We are grateful for the support from the National SKA Program of China (grant Nos. 2022SKA0110200 and 2022SKA0110203), the National Natural Science Foundation of China (grant Nos. 11975072, 11875102, and 11835009), and the National 111 Project (Grant No. B16009).
		
	
	

	


 Paper Title: Mitigating calibration errors from mutual coupling with time-domain filtering of 21 cm cosmological radio observations.



firstpage–lastpage
Magnetic field diagnostics of prominences with the Mgii k line
 3D Stokes inversions versus traditional methods
    
Jiří Štěpán1
Tanausú del Pino Alemán2,3
Javier Trujillo Bueno2,3,4

    Received XXXX; accepted XXXX
===============================================================================================================




§ ABSTRACT 
The 21 cm transition from neutral Hydrogen promises to be the best observational probe of the Epoch of Reionisation (EoR). This has led to the construction of low-frequency radio interferometric arrays, such as the Hydrogen Epoch of Reionization Array (HERA), aimed at systematically mapping this emission for the first time. Precision calibration, however, is a requirement in 21 cm radio observations. Due to the spatial compactness of HERA, the array is prone to the effects of mutual coupling, which inevitably lead to non-smooth calibration errors that contaminate the data. When unsmooth gains are used in calibration, intrinsically spectrally-smooth foreground emission begins to contaminate the data in a way that can prohibit a clean detection of the cosmological EoR signal. In this paper, we show that the effects of mutual coupling on calibration quality can be reduced by applying custom time-domain filters to the data prior to calibration. We find that more robust calibration solutions are derived when filtering in this way, which reduces the observed foreground power leakage. Specifically, we find a reduction of foreground power leakage by 2 orders of magnitude at k_∥≈ 0.5 h Mpc^-1.  








cosmology: dark ages, reionization, first stars – cosmology: observations – methods: data analysis – instrumentation: interferometers








§ INTRODUCTION

The detection of the redshifted 21 cm emission line from neutral Hydrogen during the Epoch of Reionisation (EoR) is one of the main goals of (current and upcoming) low-frequency radio telescopes such as the Low-Frequency Array <cit.>, the Murchison Widefield Array <cit.>, the Giant Metrewave Radio Telescope EoR experiment <cit.>, the Hydrogen Epoch of Reionisation Array <cit.>, and the Square Kilometre Array <cit.>. The EoR is one of the least constrained areas of cosmology, with several cosmological models predicting different reionisation scenarios. Thus, more sensitive power spectrum limits are still required to discriminate between reionisation models. The achievement of sensitive power spectrum limits will be challenged by the presence of systematic errors, either caused by instrument response <cit.> or by calibration errors due to the use of incomplete sky models during calibration <cit.>.

Measurements of the 21 cm signal are challenged by the presence of foreground emission from the Galaxy and extra-galactic sources, which are orders of magnitude brighter <cit.>. Fortunately, foreground emission is spectrally smooth, unlike the 21 cm emission line, which fluctuates rapidly <cit.>. The delay spectrum approach is a method designed to separate the 21 cm signal from the foreground emission, making use of interferometric delays to isolate the power spectrum of the 21 cm emission <cit.>. Due to the unsmooth spectral nature of the 21 cm signal, its power spectrum appears at all k modes. In contrast, the foreground emission, due to its smooth spectral emission, is limited to a wedge-like region in k-space <cit.>. The region in k space where the 21 cm signal is isolated from the foreground emission is referred to as the EoR window. 

However, the clean separation of the 21 cm signal requires the precision calibration of interferometric data <cit.>. Closely packed radio interferometric arrays, like HERA, are prone to effects of mutual coupling, where the sky signal incident on an antenna is partially reflected and subsequently received by a nearby antenna. Mutual coupling in a multi-element array such as HERA is complex, and it is difficult to fully model and characterise its impact directly <cit.>. If the effects of mutual coupling are not accounted for during calibration, they ultimately lead to unsmooth calibration errors, and 

cause foreground emission to leak into the EoR window <cit.>.  

Custom time-domain filters, called fringe rate filters, have been widely used to partially mitigate these systematics in drift-scan radio observations as a post processing step <cit.>. Recent works have also explored how these filters can be used to mitigate calibration errors from inaccurate sky models <cit.>, and from antenna displacement errors <cit.>.
However, previous works have not explored how these filters could be used to mitigate mutual coupling systematics in the raw data prior to calibration in order to improve calibration quality.

In this work, we investigate the prospect of improving HERA's calibration pipeline by filtering some of the mutual coupling features in the data before running calibration, in order to mitigate their impact on the recovered antenna gain solutions. We aim to do this through the application of fringe rate filters <cit.>. We consider two specific kinds of filters: a simple baseline-independent high-pass filter, which we refer to as the notch filter, and a baseline-dependent filter, referred to as the main-lobe filter <cit.>. We assess the calibration improvement by computing the reduced chi-squared after calibration and by further examining the structure of the recovered gains in Fourier space to assess the amount of residual spectral structure. Lastly, we show how our technique helps mitigate the foreground leakage in the EoR window. Note that the exact form of the adopted main-lobe filter is specific to this work: other works may use similar kinds of "main-lobe filters" but may have slightly different parameterizations depending on their desired outcome.

This paper is divided into three sections, in Section <ref> we give an overview of calibration and the simulation set up, Section <ref>  we discuss the application of fringe rate filters before calibration and the relevant metrics that we use to measure the improvement in calibration, and we conclude in Section <ref>. 
 





§ CALIBRATION OVERVIEW





 §.§ Sky-model based calibration


When observing the sky with a radio interferometer, the electromagnetic waves from celestial sources are measured by two antennas that are correlated, forming interferometric visibilities.
The measured complex-valued visibilities are related back to the intrinsic sky emission and the instrumental response of the antennas via the radio interferometer measurement equation <cit.>.

    V^0_ij(ν)=∬ J(s,ν))  I(s, ν)   J^*(s,ν)   e^-2πν/cb_ij·s  dldm/n(s),

where V^0_ij is the visibility formed between antenna i and j, and ^* denotes the complex conjugate, J_i(s,ν) denotes Jones terms describing the propagation effects along the path from the source to the i-th antenna in the array. I(s,ν) is the brightness distribution of the sky, ν is the frequency of the incoming electromagnetic wave, b_ij is a baseline vector connecting antenna i and j, s = [l, m, n]^T is the position vector on the celestial sphere with an origin centred at the target field (phase centre), and (l, m, n) are the direction cosines, with n = √(1 - l^2 - m^2). Note that in this paper, we will only consider a single polarization. The Jones terms are classified into two main groups: direction-dependent and direction-independent. The direction-dependent Jones varies with observing direction, these effects include the primary beam E(s,ν) response of the antenna, which varies across the sky for a fixed pointing. Whilst the direction-independent Jones terms are independent of observing direction. These effects include (not limited to) the band-pass response of the antenna B(ν) and gain amplification term G(ν).  

Note that in this paper, Equation <ref> assumes a trivial primary beam response in that the signal from the celestial source is completely absorbed by array antennas. However, in real observations, a portion of the signal is reflected due to departure from the conjugate match at the terminals of antenna feeds. If the array is compact, the reflected signal will be absorbed by a neighbouring antenna. This effect, known as mutual coupling, is a known concern for interferometric 21 cm experiments. More analysis of the upper limits of the power spectrum cite mutual coupling as one of the unresolved possibilities for their excess power <cit.>. 

Recent works have developed semi-analytic approaches to modelling the complex nature of coupling effects within an interferometric array.
<cit.> developed a semi-analytic approach for single antenna-pair coupling, which was highly effective in suppressing observed coupling in HERA Phase I data <cit.>.
Future work expanded upon this approach for multi-antenna coupling, which showed new phenomenology that matched the observed systematics in HERA Phase II data <cit.>.

In this work, we will consider only first-order, multi-antenna mutual coupling effects as described in <cit.>, where each antenna in the array absorbs most of the incident signal from the source, and some of the signal is reflected and completely absorbed by other antennae in the array (i.e., the incident signal is only reflected once). If we consider unpolarised radiation from the sky and assume that all beams in the array have identical electromagnetic properties, that is they have the same reflection and absorption coefficients, then the first-order visibilities (visibilities with first-order mutual coupling effects) are given by: 


    V^1_ij =  V^0_ij + (∑_k ≠ i  V_ik^0 X^*_jk  + X_ik V_kj^0 ),
 
where

    X_ik= i η_0/4λΓ/R   |b_ij|  e^2π i ν/c |b_ij|   h_0^2   J(b_ik) J^*(b_ki)


where η_0 is the impedance of free space, Γ is the frequency-dependent reflection coefficient, R is the real part of the impedance term and J(b_ki) denotes direction-dependent effects (primary beam response) along the path travelled by the reflected signal from antenna k to antenna i and h_0 is the antennas so-called effective height <cit.>.


In real observations, antennas in an array also impart direction-independent corruptions to the visibilities, which can be modelled using antenna-based complex gain terms. The corrupted visibilities are related to intrinsic visibilities as

    V^c_ij=g_i  V_ij  g_j^*   + n_ij,

where V^c_ij are the corrupted (or measured) visibilities, g_i and g_j are the complex gain terms of antenna i and j, and n_ij is the complex thermal noise generated by the telescope.

Antenna gain calibration is the process of deriving the direction-independent gain terms (g_i) from the data and then correcting the data to remove their effects.
This is commonly done by constructing a model of the true visibilities and then minimising a χ^2 statistic:

    χ^2 =∑_i,j |V_ij^c-g_i  g_j^*V^m_ij|^2/σ_ij^2,

where V_ij^m are the constructed model visibilities, and σ_ij^2 is the variance of the visibility, and the sum runs over all unique antenna pairs.

The chi-square minimisation depends on the accurate modelling of the intrinsic sky visibilities V_ij. It is crucial, therefore, that the sky model matches the intrinsic sky brightness as closely as possible. Unmodelled mutual coupling effects can cause significant deviation between intrinsic sky visibilities and the model visibilities in calibration, leading to calibration errors that can impart spectral structure on smooth foregrounds <cit.>.




 §.§ Redundant calibration


Redundant calibration is less reliant on a sky model compared to sky-base calibration, to solve for antenna gains, the gains are obtained by exploiting the internal redundancy of the interferometric array. Redundant baselines have the same length and orientation and, therefore, measure the same Fourier mode of the sky brightness distribution.
For example, assume we have a single baseline type, A, uniquely identified via its baseline vector b_A. For all antenna pairs i,j that share this baseline vector, the calibration equation, Equation <ref> from before now becomes

    V^c_ij = g_i   V_A  g_j^* + n_ij,

where V_A is now a parameter of the model, called the “redundant model visibility”.
This is repeated for all unique baseline types, eventually building up an overconstrained system of equations, which can be solved via a χ^2 minimization <cit.>:


    χ^2 =∑_i,j |V_ij^c-g_i  g_j^*V_A|^2/σ_ij^2,

where V_A is the corresponding redundant model visibility for the b_ij baseline vector. Note that in this paper, the χ^2 is both a function of frequency and time. The gain solutions obtained from the system of linear equations set up in the redundant calibration are not unique; there are at least four degenerate parameters that need to be solved for after redundant calibration: the overall gain amplitude, the model visibility amplitude and the phase gradient across the array in the east-west and north-south orientations <cit.>. These parameters are, in principle, a function of polarization, frequency and time and require a sky model to be constrained.  
It is in this step that, redundant calibration itself makes use of a sky model. In this study, we will refer to this step in calibration as absolute calibration to differentiate this from sky calibration or the typical sky-based calibration discussed previously.

In the case of perfect calibration, where the sky model is complete, model visibilities only differ from the data by noise. The expectation value of chi-square ⟨χ^2 ⟩ is thus two times the degrees of freedom (DoF) for complex data <cit.>. The number of DoF, in general, is given by the difference between the number of data points and the number of fitted parameters. If we consider a single polarization and the case of sky-based calibration, the DoF are given by the difference between the number of visibilities and the number of antennas:

    DoF = N(N-1)/2 - N = N_ bl - N,

where N is the number of antennas and N_ bl is the number of baselines given N antennas. If we histogram the χ^2/ DoF quantity, it is expected to follow a theoretical chi-square distribution <cit.>. 
In the case of redundant calibration, the DoF is given by:

    DoF = N_ bl-N_ ubl-N+2,

where N_ ubl is the number of unique baselines that have a single sky model visibility V_A, i.e. the number of redundant baseline groups <cit.>.





 §.§ Simulation set up


In this paper, we simulate sky models composed of diffuse emission from GSM <cit.> and compact source flux density consistent with the GLEAM catalogue <cit.>, but with no spatial clustering of sources (uniformly distributed). We also include bright radio galaxies <cit.>, modelled as compact sources. To simulate mutual coupling effects, we make use of the mutual coupling simulation formalism from <cit.> to produce visibility products with first order mutual coupling effects. Their formalism included modelling the first-order antenna-antenna coupling in radio interferometers.  Specifically, they derive a semi-analytic model of the interferometric visibility Equation <ref>, in which first-order coupling effects are considered. In their model, incident radiation enters all interferometric elements in the array, and then each element absorbs most of the incident radiation, but some of the power is then reflected due to departure from the conjugate match at the terminals of the antenna feeds. Thus, each element in the array not only absorbs the incident array from the sky but re-radiates it across the array, which is subsequently absorbed by other elements. This simulation only considers the first-order formalism, and thus, it only considers the effects of re-radiated radiation being absorbed only once by all the array elements. In this paper, we refer to simulated visibilities containing effects of mutual coupling as first order visibilities. And simulated visibilities without effects of mutual coupling are to as zeroth order visibilities.  

The simulated visibilities span a frequency range of 140-190 MHz, with a frequency resolution of 122.07 kHz. We consider observations in the Local Sidereal Time (LST) range 0^h-4^h, with a time resolution of dt=58 s. The simulation is done for a HERA array consisting of 174 antennas shown in Figure <ref>. 
Figure <ref> shows the visibility spectra from 14 m East-West redundant baselines with (first order visibilities) and without mutual coupling (zeroth order visibilities). The visibility spectra from redundant baselines have a higher amplitude at low frequency, as expected, due to the brightness of diffuse emission at low frequencies <cit.> for both zeroth and first order visibility. Primary beam models containing the effects of mutual coupling differ from the primary beam model that is without mutual coupling <cit.>. As a result, the visibility spectra from the 14 m baselines without mutual coupling (solid red line) differ significantly from the visibility spectra of the same baseline with mutual coupling. Also, notably, the visibility spectra from 14 m redundant baselines with mutual coupling deviate from each other, and these deviations are as high as 6% across the frequency band. 


We make use of gains that are similar to ones simulated by <cit.> to create mock raw data.  The gain g for the jth antenna is given by:

    g_j(ν)=A_j (ν) e^i ϕ_j(ν),

 where the amplitude A follows a frequency power law 

    A_j(ν)=A_j ( ν/150 MHz)^b_j,

and A_j and b_j are drawn for each antenna from a Gaussian distribution 𝒩∼ (x̅_A=0.30, σ_A=0.001) and 𝒩∼ (x̅_b=-2.6,σ_b=0.2), respectively. The phase ϕ_i(ν) is modeled as:


    ϕ_j(ν)=sin(w_a  ν) + cos(w_b  ν),

where w_a and w_b are drawn for each antenna from a Gaussian distribution 𝒩∼ (x̅_w=0.005, σ_w=0.0005). The mean values of the parameters A and b, as well as their standard deviation σ_A and σ_b, are informed by the gain solutions from actual HERA observations <cit.>. The mean phase variation w̅ is based on single antenna phase dependency of actual HERA gains where the cable delay and geometric phase offset have been removed, and the variation of the mean phase between antenna stations, i.e. σ_w, is chosen to be within 10%. We then add noise to the data, such that signal-to-noise ratio (SNR) of the simulated bandwidth ∼ 10^4, mutual coupling effects are known to be prominent at this dynamic range <cit.>. Note that this SNR is far higher than in real observations <cit.>.  We simulate two sets of raw data products; one includes the effects of mutual coupling, and the other does not. The latter raw data product is used to verify our pipeline and serves as a benchmark to quantify the effects of mutual coupling. 

We subdivide the band into two sub-bands, which we name low-band and high-band, with a frequency range of 140-165 MHz and 165-190 MHz, respectively. We make use of a sky model that contains no mutual coupling effects, i.e., our sky model is composed of zero-order visibilities only. First, we make use of redundant calibration to calibrate raw data. The gains from the redundant calibration step are then applied to the raw data, i.e., the raw data is partially calibrated. The calibrated visibilities are then further calibrated using the absolute calibration step. Here, the data is calibrated using the sky model (zeroth order visibilities), and the gains recovered from this step are combined with those recovered from the redundant calibration step to make a full set of gains. We then apply the full set of gains to calibrate the raw data. To quantify the robustness of the calibration,  we compute the reduced chi-square from redundant calibration and absolute calibration steps. 



 §.§ Calibration with mutual coupling effects


Figure <ref> shows the reduced chi-square obtained from the redundant calibration step for an ideal calibration scenario, i.e., where the raw data does not contain the effects of mutual coupling. The histogrammed reduced chi-square from the data set without mutual coupling, as expected, follows the theoretical χ^2 distribution, indicating that model and raw data visibilities only differ by noise, and the recovered gains from calibration are perfect (match the true gains), thus also verifying the calibration pipeline works as expected. Figure <ref> shows the recovered chi-square from the data set with mutual coupling for both low and high sub-bands. The chi-square is notably biased to higher values for both sub-bands, but notably significantly higher at the low sub-band. The mean value of the reduced chi-square χ^2 /DoF is 3.3× 10^3 and 1.1× 10^3 for low and high sub-band, respectively. This bias is due to differences in visibility products from redundant baselines (see Figure <ref>) caused by the effects of mutual coupling, since redundant calibration makes no use of the sky model. Given the mean value of the reduced chi-square, we can also infer that the redundant calibration errors are larger at lower frequencies (low band) than at high frequencies. The reason for this is that the increased brightness at lower frequencies of the diffuse emission, which couples to the primary beam response, thereby effectively enhancing differences in the apparent sky observed by two antennas in the array. Thus, when redundantly calibrating the data, larger differences are found between the visibility spectra of supposedly redundant baselines. Figure <ref> further shows that the reduced chi-square depends upon the looking direction, thus indicating that the chi-square depends on the brightness of the diffuse emission. 

The right panel of Figure <ref> show the chi-square obtained after the absolute calibration step. The mean value of the recovered chi-square χ^2 /DoF from the absolute calibration step is 1.6× 10^6 and 1.1× 10^4 for both low and high sub-band, respectively. At the low sub-band, the mean value is at least 10^2 times greater than that obtained from the redundant calibration step. The increase in chi-square is caused by the fact in our sky model we do not incorporate the effects of mutual coupling, which are found in the data, thus, the model and raw visibility products differ significantly (see Figure <ref>). In addition, calibration errors in the redundant calibration step are carried through to the absolute calibration step, and hence, thus calibration errors accumulate, and the obtained chi-square is significantly larger in the absolute calibration step. We note the recovered chi-square values are large compared to the values (χ^2 /DoF around 1-3) one would find in typical HERA observations <cit.>, this is just a consequence of the low noise (high SNR) in our simulation compared to actual observations. Small differences between model and raw data visibilities are effectively 'amplified' by the small variance σ^2 ∼ 10^-8 in our simulation, whereas in actual observation the noise variance is typically σ^2 > 10^-4.















§ FOURIER REPRESENTATION OF INTERFEROMETRIC DATA


We will now define relevant quantities that will be used to quantify the impact of mutual coupling in calibration. Interferometric visibilities are a function of frequency (ν) and time (t). We refer to the Fourier dual of frequency as the delay domain, formed by taking a Fourier transform of the visibilities across the frequency axis:


    Ṽ(τ) = ∫ V(ν)  ^-2 π i τν dν,


where τ is a delay (in seconds) and Ṽ(τ) is the Fourier pair of the frequency. 

We can also define delay-transformed antenna gains to be the Fourier transform of the antenna gains along the frequency axis:


    g̃(τ) = ∫ g(ν)  ^-2 π i τν dν.

We define the Fourier dual of time as the fringe rate and the fringe-rate visibility as the Fourier transform of the visibility along the time axis:

    Ṽ(f) = ∫ V(t)  ^-2 π i f t dt,

where f is the fringe rate (in units of Hz). For interferometric arrays that operate in drift-scan mode, the fringe rate basis also separates signals based on their relative motion through the fixed interferometric fringes, acting as another form of separation of signals on the sky <cit.>.

 


 §.§ Recovered gains from calibration

Figure <ref> shows the gains recovered from absolute calibration for both low and high sub-band. Let's first consider the gains recovered from the low-band data set. The recovered gains show a pronounced frequency structure in both amplitude and phase, caused by calibration errors resulting from unmodelled mutual coupling effects. The calibration errors are unsmooth, resulting in excess power at delays τ>250 ns (Figure <ref>). The dynamic range, i.e., the ratio of the gain amplitude at delay τ=0 to that of a specific delay τ, of the gains without mutual coupling at τ = 250 ns, is ∼ 10^5. Due to the effects of mutual coupling, this is reduced to ∼ 10^2.  Notably, the mutual coupling effects lead to excess power in the gains at a broader range of delays (τ>250 ns) compared to calibration errors from incomplete sky modelling (200 <τ< 600 ns) <cit.>. Indeed <cit.> also found that mutual coupling effects tend to contaminate a wider area of the EoR window. The recovered gains with mutual coupling at the higher band, however, have comparatively less excess power, and the excess power spans a narrower delay range 250< τ <2000 ns. This is due to the reduced brightness of the diffuse emission, within the high sub-band.









 §.§ Application of Fringe Rate Filters



We propose the use of a fringe rate filter as a way to reduce the impact of mutual coupling in calibration. The idea is to filter out some mutual coupling effects in the raw data, thus bringing the raw data closer to the sky model. We employ a similar approach to that used by <cit.> to improve calibration. We also apply fringe rate filtering to both the raw data and the sky model visibilities in the same manner for each of the different filters that we use (described below). We then apply the derived gains to the unfiltered visibilities, thus mitigating concerns of a possible cosmological signal loss.


Note that, for the two filters described below, we apply them to the data using the DAYENU filtering formalism described in <cit.>, which relies on the Discrete Prolate Spheroidal Sequences <cit.>.



  §.§.§ Notch Filters


We first consider a symmetric baseline-independent notch filter F(f) centred at f = 0 mHz fringe rate,  i.e. a high-pass filter, defined as 


    F(f) = 
    {[       10^-8, |f| ≤ f_ max;           1, |f| > f_ max ]},


with 

f_ max = 0.25 mHz,  we refer to the filter as f_25. See Figure <ref> for a visual representation of the notch filter.




  §.§.§ Main Lobe Filters



We also considered a second type of filter, which we refer to as a “main lobe” filter, because it aims to suppress emission from outside the primary beam field-of-view.
In contrast to the baseline-independent notch filter, which only filters out emission in a region near f∼0 mHz, the baseline-dependent main-lobe filter suppresses the signal everywhere except near the peak emission of the sky model in fringe rate.

Technically, the main-lobe filter is a frequency-dependent filter, but because we operate over a relatively small bandwidth (25 MHz within each band), we approximate it as frequency-independent. The response of the filter within the pre-defined fringe rate bounds is uniform, such that it can be thought of as a top-hat filter.

The bounds of the baseline-dependent main-lobe filter are determined by its centre f_0  and its half-width f_w. These parameters are determined for each baseline by fitting a Gaussian profile G(f) to the sky model visibilities in fringe rate space:


    G(f) = A   e^-(f - f_0)^2/2 σ^2,


where A is the amplitude of the Gaussian, σ is its standard deviation, and f_0 is its mean.
After the fit, we set the main lobe filter centre to be f_0 and its half-width to be f_w = 2σ, such that the full width of the main lobe filter is 4× the fitted Gaussian's standard deviation. We denote the baseline dependent main lobe filter as M_de (Figure <ref>).







 §.§ Using the Notch Filter to Improve Calibration

In Figure <ref>, we have noted that the recovered chi-square is dependent on the looking direction (LST), we have shown that the chi-square is coupled to the diffuse emission, that is, it is higher when the diffuse emission is bright. To show the effectiveness of the fringe rate filters we consider a field where the chi-square is largest (bright diffuse emission), i.e., we are considering the 'worst case-scenario'. We consider a field centred at an LST range 0^h-1^h. We first examine the χ^2 obtained after the absolute sky calibration step. The sky emission can be separated in fringe space, as different parts of the sky occupy different fringe rate values, for an array observing in tracking mode i.e., with the sky emission drifting across the array <cit.>. For a baseline in the HERA array, we note, especially, that emission originating from the celestial South Pole will occupy fringe rate value zero, as sources at the celestial South Pole do not drift across the array <cit.>. Thus, we expect these sources to fall on primary beam side lobes. Due to the unsmooth spectra of the primary beam with mutual coupling <cit.>, when this emission couples with primary beam response, we expect the resulting calibration errors to be unsmooth, as shown in Figure <ref>. Note, however, that this is only true for a baseline with a non-zero East-West projection <cit.>. <cit.> also showed that mutual coupling effects occupy in fringe rate space occupy fringe rate values near zero. Thus, in this paper, we design fringe rate filters to filter out mutual coupling effects at zero fringe rate. We employ a baseline cut and downweight all baselines during the calibration process that have an East-West projection of less than 30 m, to ensure that all baselines have a peak of the main lobe emission (in fringe rate space) at fringe rate values greater than 0.5 mHz.  

Figure <ref> shows the chi-square obtained for the low sub-band when a notch fringe filter is applied to the data set with mutual coupling before calibration. The chi-square has reduced significantly, with a mean value of 1.7× 10^3. This represents a reduction in χ^2 of factor ∼ 10^3. This occurs because the filter suppresses some of the mutual coupling features in the raw data, thus bringing the raw data closer to the model, leading to a smaller chi-square value. A smaller χ^2 value is indicative of robust calibration. Thus, we expect the recovered gains to improve as well. Figure <ref> shows the recovered gains after applying fringe rate filters. The recovered gains have comparatively less frequency structure in both amplitude and phase, and as a result, the delay transform of the gains has comparatively less excess power at delays τ >250 ns for both the main lobe and notch filter. The dynamic range is improved by a factor ∼ 10 at delays τ≈ 250 ns for both high and low sub-band, when the notch filter f_25 is applied. However, the gains recovered after applying the main lobe filter show comparatively less improvement (compared to the notch filter), in terms of dynamic range at delays 250<τ<1000 ns at low sub-band. Similarly, at the high sub-band, the notch filter performs comparatively better than the main lobe filter over the delay range  250<τ<500 ns. Notably, the gains recovered from the notch-filtered data have a higher noise floor compared to unfiltered data, and this is due to lower SNR as some of the sky signal is suppressed by the filter, thus reducing the SNR. However, the gains recovered when the main lobe filter is used have a lower noise floor compared to the notch filter, and this is because the main lobe filter retains most of the sky signal, and in addition, it also significantly attenuates some of the noise. Note that the current HERA calibration pipeline includes smoothing the derived gains, i.e., a prior on the spectral structure of gains is imposed, and this mitigates the some of effects of mutual coupling <cit.>. However, the advantage of using fringe rate filters is that, if there is real spectral structure in the intrinsic antenna gains, say above ∼ 100 ns, then the use of fringe rate filters has a better chance of finding the true gain response.

We can see that the notch filter performs better than the main-lobe filter at lower delays τ<500 ns, given the difference in the dynamic range of the recovered gains. The main key to understanding the improved performance of notch filters is to examine closely the filtering and calibration process. The main key to getting robust gains after calibration is to minimise the differences between the sky model and raw data visibility products. The main lobe filter is designed to retain the main lobe emission in fringe rate, and for a baseline with an East-West baseline projection greater than 30 m, this emission also falls within the main lobe of the primary beam, with sidelobe emission occupying fringe rate near zero. Although the effects of mutual coupling are smaller in the main lobe of the primary beam compared to the sidelobes, there are still significant differences in the antenna primary beam main lobe, between mutual coupling and non-mutual coupling beam models <cit.>, and these primary beam differences would contribute significantly to calibration errors, even more so when bright emission falls on the main lobe of the beam (see Figure <ref>). Unlike the sky emission falling on the sidelobe, the sky emission falling on the main lobe (with some mutual coupling effects) is not significantly attenuated. Thus, it dominates calibration. Mutual coupling effects in the main lobe of the beam have a much larger impact on the calibration compared to mutual coupling from the sidelobe of the primary beam, and are typically relatively smooth in frequency compared to side lobe mutual coupling effects. Thus, we would expect calibration errors due to mutual coupling effects to be relatively smooth in frequency as compared to mutual coupling effects from the sidelobes. The key goal of the filters is to suppress a large portion of emission where mutual coupling effects are pronounced, whilst retaining a good portion of the sky signal that can be used for calibration. On baselines with short East-West projection (main lobe emission is close to  f∼ 0.5 mHz), the baseline independent notch filter, filters a portion of the emission that lies in the main lobe emission,  whereas the main lobe filter retains this emission, which may contain some mutual coupling effects. 

An example of such baseline emission is shown in Figure <ref>. The shoulder emission (emission at fringe rate value f≈ 0.25 mHz) in the main lobe is filtered out by the notch filter (green), minimising the impact of mutual coupling effects in the main lobe emission, thus bringing the raw data closer to the sky model data, reducing errors in calibration. However, the baseline-dependent main lobe filter (black) retains this emission. Figure <ref> shows the visibility spectra of the main lobe and notch filtered visibility of both zeroth order (without mutual coupling, solid line) and first order (with mutual coupling, dotted line) visibilities. The main lobe filtered visibilities (with and without mutual coupling) deviate significantly from each other as the main lobe filter essentially fails to attenuate the relatively smooth frequency varying unmodelled mutual coupling terms, which can be seen as the ripples in spectra (black dotted line). Thus, model and raw data visibilities differ significantly, leading to larger calibration errors. However, the notch-filtered visibility closely match with each other, as ripples in visibility spectra caused by mutual coupling effects are attenuated significantly, leaving an overall smooth visibility spectra for both filtered and unfiltered. The difference between filtered and unfiltered visibility spectra is notably small for the notch filter compared to the main lobe filter,  thus, the notch filter would yield smaller calibration errors, leading to more robust recovered gains that can be used to calibrate the data. In order to achieve the same improvement as the notch filter in the calibration, particularly at the low delays τ <500 ns, we need to implement a E-W baseline cut larger than 50 m. In Figure <ref>, we show that the recovered gains from calibration using the notch filter have the same dynamic range as gains recovered from the main lobe filter, but with an E-W baseline cut of 50 m. 



















 §.§ Foreground Power Spectrum


In this section, we will introduce the power spectrum as a metric to assess the impact of fringe rate filters on calibration. We compute the per-baseline power spectrum of the visibilities, P(τ, b), following the delay approximation <cit.>.



    P(τ, b)=|V_b(τ)|^2  (λ^2/2k_B)^2(D_c^2Δ D_c/B_ eff)(1/Ω B_ eff),



where λ is the centre wavelength of observing bandwidth, k_B is the Boltzmann constant, B_ eff is the effective bandwidth, D_c is the comoving distance at the redshift of our measurement, Δ D_c is comoving distance parallel to the line of sight, Ω is the field of view solid angle, Ṽ_b is the Fourier transformed visibility, and b is the visibility baseline length.
We can map τ to the line-of-sight cosmological Fourier wavevector k_∥ using the relation <cit.>:

    k_|| = 2πν_21 H_0 E(z)/c(1+z)^2 τ,

where ν_21 = 1420 MHz, H_0 is the Hubble constant, E(z)=[Ω_m(1+z)^3+Ω_Λ]^1/2, Ω_m is the normalized matter content and Ω_Λ is the normalized dark energy content.

In Figure <ref>, we show the residual visibilities obtained after calibrating raw data with gains recovered from the data set with and without mutual coupling. The residual visibilities are obtained by taking the difference between calibrated visibilities and the sky model visibilities with mutual coupling effects. Recall that simulated true sky model visibilities contain effects of mutual coupling, thus a perfect calibration needs to yield calibrated visibilities that match closely with the sky model visibilities that contain effects of mutual coupling. As expected, when the raw data visibility products are without mutual coupling, we get a perfect calibration, yielding robust gains, and thus, the calibrated visibilities differ from the sky model visibilities by noise. Hence, the residual visibilities obtained are just noise (left panel). However, when our simulated raw data includes the effects of mutual coupling, our calibration is not perfect, and the recovered gains (Figure <ref>), which are subsequently used to calibrate the raw data are erroneous. Thus calibrated visibilities differ significantly from the true sky model visibilities. Additionally, when the recovered gains are used to calibrate the raw data because of their non-smooth spectral structure (due to calibration errors) they impart spectral structure on the relatively smooth foreground spectra, some of the foreground emission is pushed to higher delays beyond the horizon limit (white line). This is most prominent on the shorter baselines. In the panel second panel from the left, the residual visibilities have foreground emission at short baselines beyond the horizon limit, and this residual foreground emission extends up to delays of τ∼ 500 ns.  When the notch filter f_25 is applied (third panel from left) before calibration, more robust gains are used to calibrate the unfiltered raw data. We find that the calibrated visibilities closely match the sky model visibilities. As a result, the foreground residual emission beyond the horizon limit is significantly reduced by a factor of ∼ 10^2. The excess residual foreground within the horizon line poses no issue, as this region in delay space is expected to be occupied by foreground emission. However, when the main lobe filter M_be is used (right panel), there is comparatively less reduction in the foreground emission in residual emission compared to the notch filter. This is expected as the gains recovered when the main lobe filter is used are not as robust as gains obtained through the use of the notch filter, and can therefore cause foreground leakage when they are used to calibrate the data.  

Figure <ref> further shows the residual power spectrum for 14 m redundant baselines for both low and high sub-bands. The foreground residual power is significantly less in the high sub-band than in the low sub-band as the foreground residual emission is reduced to the noise level at k_∥∼ 0.75 h Mpc^-1 in high sub-band, but at low sub-band, there is still residual power at k_∥ > 1.5 h Mpc^-1 (cyan line). This is again due to the increased mutual coupling effects at the low sub-band.  We also note the main-lobe filter also suppresses foreground residual emission (black line). However, the suppression is not as significant as the notch filter (green line) in both the low and high sub-band. In the low band, the notch filter suppresses residual foreground by a factor ∼ 10 more than the main lobe filter at k_∥ < 0.75 h Mpc^-1. Additionally in the low sub-band, with the notch filter, the residual emission is reduced to levels below the noise floor at k_∥≈ 0.60 h Mpc^-1, whereas for the main lobe filter, this only occurs at k_∥≈ 0.75 h Mpc^-1. In the high sub-band, the reduction of residual power by the use of the notch filter is by a factor of ∼ 10^3 larger than that of the main lobe filter at k_∥≈ 0.5 h Mpc^-1. Thus, the use of the f_25 notch filter produces robust gains that significantly reduce the foreground power leakage.


Lastly, Figure <ref> shows the residual power from 14 m redundant baselines at an LST centred at 2^h, typically used for calibration of HERA data <cit.>. The sky at this LST has strong compact sources and the diffuse component of the sky is colder compared to the LST of 0^h. As a result, the error in the calibration is reduced significantly, as evident by the low recovered chi-square (Figure <ref>). We again note that in this field the use of fringe rate filters improves the calibration as the foreground residual power is reduced by 10^2 at k_∥≈ 0.5 h Mpc^-1. We, however, note that there is no significant difference in the improvement in calibration when either the main lobe or notch filter is used, particularly at the high sub-band. There is slightly better performance of the main lobe filter at low sub-band at k_∥≈ 0.25 h Mpc^-1. In this field, the diffuse emission component falling on the main lobe is cold, compared to bright compact sources, however, the hot diffuse component still resides in the side lobes of the primary beam and thus would introduce calibration errors if the emission is not filtered out (cyan line). When this emission is filtered through with a notch filter (green line), the calibration is improved, and even more so when the main lobe is used, due to more of the emission being filtered. In summary, both filters have shown effectiveness at reducing foreground power leakage due to mutual coupling effects, with the notch filter showing better performance when the diffuse emission dominates the sky. Both filters were also shown to be effective at mitigating calibration due to the use of an incomplete sky model during calibration <cit.>, especially the main lobe filter at LST of 2^h. Thus, the main lobe filter can be used to mitigate both calibration errors due to incomplete sky modelling and the effects of mutual coupling.  











§ CONCLUSION



In this paper, we investigate the prospect of mitigating the effects of mutual coupling in calibration through the application of fringe rate filters prior to calibration. We make use of two types of filters; baseline dependent main lobe filter and baseline independent notch filter. The main lobe filter is designed to attenuate all emission that in not within the main lobe of the primary beam, whilst the notch filter is designed to attenuate emission that is near zero fringe rates. For a baseline in the HERA array with an East-West projection greater than 30 m, this emission will typically resides on the sidelobes of the primary beam. 

We show that mutual coupling effects lead to non-smooth calibration errors, which impart spectral structure in gains derived from calibration in the delay τ >250 ns. When these gains are used to calibrate data, we find significant contamination of the EoR window by foreground emission, which is about 4 orders of magnitude at k_∥≈ 0.5 h Mpc^-1 at z=8.3 for a 14 m redundant baselines. By applying the filters before calibration we find in this paper that the calibration significantly improves, as the recovered gains are much more robust i.e. much closer to the true gains, and when these gains are used to calibrate data we observe a significant decline in foreground contamination of EoR window. The foreground contamination is reduced by at least 2 orders of magnitude at  k_∥≈ 0.5 h Mpc^-1. In addition, we find that the application of the notch filter leads to reduced calibration errors compared to the main lobe filter. We find that the foreground contamination is an order of magnitude below for notch filter compared to the main lobe filter. This is mainly due to mutual coupling effects that are within the main lobe of the primary beam, to which the main lobe filter retains, thus resulting in erroneous gains compared to the notch filter, where a portion of this emission is filtered.

Although the use of fringe rate filters has shown significant improvement in the calibration, we, however, note that more investigation is still needed, because real observations normally include flags, that can potentially affect the filtering of data. When data containing flags is Fourier transformed along the time axis, we expect there to be “spectral leakage” resulting in the mixing of fringe rate modes of the sky due to the sidelobe structure introduced by the null/flags in the data. Thus, flagging on the time axis can reduce the effectiveness of the filter. To reduce the impact of missing data points, the data can be inpainted prior to applying the filters thus preventing the mixing of fringe rate modes of sky prior to filtering the data. A variety of inpainting techniques for radio data could be leveraged in this effort <cit.>.



§ DATA AVAILABILITY

Data used in this work may be made available upon reasonable request to the corresponding author. The code used for the analysis of this work is publicly available at https://github.com/Ntsikelelo-Charles/Fringe_rate_filtershttps://github.com/Ntsikelelo-Charles/Fringe_rate_filters. This work relied on publicly available and open-sourced community Python software, including numpy<cit.>. Visibility simulations were made using matvis <cit.>, and simulation of mutual coupling effects hera_sim (https://github.com/HERA-Team/hera_simhttps://github.com/HERA-Team/hera_sim) and pyuvdata <cit.>. 



§ ACKNOWLEDGEMENTS


This work was supported in part by the Italian Ministry of Foreign Affairs and International Cooperation”, grant number ZA23GR03. NK acknowledges support from NASA through the NASA Hubble Fellowship grant #HST-HF2-51533.001-A awarded by the Space Telescope Science Institute, which is operated by the Association of Universities for Research in Astronomy, Incorporated, under NASA contract NAS5-26555. MGS acknowledges support from the South African Radio Astronomy Observatory and National Research Foundation (Grant No. 84156). OMS's research is supported by the South African Research Chairs Initiative of the Department of Science and Technology and National Research Foundation (grant No. 81737). AL and RP acknowledge support from the Trottier Space Institute, the Canadian Institute for Advanced Research (CIFAR) Azrieli Global Scholars program, a Natural Sciences and Engineering Research Council of Canada (NSERC) Discovery Grant, a NSERC/Fonds de recherche du Québec-Nature et Technologies NOVA grant, the Canada 150 Research Chairs Program, the Sloan Research Fellowship, and the William Dawson Scholarship at McGill.






mnras
 















