less evolved stars, and our validation showed that the masses should be used
with caution for evolved stars. We have therefore added quality information in
flags_flame and flags_flame_spec which takes a value of ’1’ as the first
character to indicate that the star is a giant with a published mass (and
usually an age) and that these corresponding parameters should be used with
caution. Additional validation showed that results for giants with ${M}$ $>2$
$\,{{M}}_{\odot}$ are mis-classified and should not be used.

HR diagrams are shown in Fig. 24 using a random subset of 2 million sources
from Gaia DR3: the top panel shows lum_flame versus teff_gspphot colour-coded
by evolstage_flame while the bottom panel shows lum_flame_spec versus
teff_gspspec colour-coded by age_flame_spec.

Figure 24: HR diagrams colour-coded by FLAME parameters. Note that the colour-
scale is linear and not a density plot. Top: lum_flame versus teff_gspphot
colour-coded by evolstage_flame for stars with relative parallax errors better
than 10%. We have applied the recommended FLAME filter for giants. Lower:
lum_flame_spec versus teff_gspspec colour-coded by age_flame_spec for stars
with flags_gspspec like ’0000000000000%’. In the background the red clump can
be seen in grey. These have no age values associated with them. Even though we
have made selections on quality on certain parameters, there are still some
artefacts that can be seen, such as the high-luminosity, low
$T_{\mathrm{{eff}}}$ giants in the upper panel colour-coded by yellow, or the
high-luminosity low mass main sequence stars in the lower panel. These
artefacts can be removed by filtering on luminosity uncertainty, or requiring
that the $T_{\mathrm{{eff}}}$ from both GSP-Spec and GSP-Phot agree to e.g.
300 K, or filtering on spectra SNR.

####  6.4.2 GSP-Phot

Given GSP-Phot’s use of isochrones in a forward model context, see Sect.
6.2.1, GSP-Phot also provides estimates of absolute magnitude MG and radius
${R}$, mg_gspphot and radius_gspphot, and upper and lower confidence levels.
These parameters are found in the astrophysical_parameters table for 470
million sources, and the results for the individual libraries are found in the
astrophysical_parameters_supp table. From these GSP-Phot results, the user
could also compute GSP-Phot estimates of the (bolometric) luminosity,

$\frac{L}{L_{\odot}}=\left(\frac{R}{R_{\odot}}\right)^{2}\left(\frac{T_{\mathrm{{eff}}}}{5772{\rm
K}}\right)^{4}\,\textrm{.}$ (2)

While GSP-Phot does not directly provide absolute magnitudes in the BP or RP
bands, GSP-Phot does estimate the distance and extinctions $A_{\mathrm{BP}}$
and $A_{\mathrm{RP}}$. Therefore, the user can compute absolute magnitudes
$M_{\textrm{XP}}$ using the observed apparent magnitudes $G_{\textrm{XP}}$ via

$M_{\textrm{XP}}=G_{\textrm{XP}}-A_{\textrm{XP}}-5\log_{10}d+5\,\textrm{,}$
(3)

where XP stands for BP or RP, and $d$ is the distance in parsec. Given those,
the user can then compute bolometric corrections in those bands via

$\textrm{BC}_{\textrm{XP}}=4.74-2.5\log_{10}(L/L_{\odot})-M_{\textrm{XP}}\,\textrm{.}$
(4)

Uncertainties on those additional quantities can be obtained from the GSP-Phot
MCMC samples (see Appendix B) by processing all samples through those
equations and then computing, e.g., their median values and quantiles.

###  6.5 Extragalactic redshifts

The redshifts of extragalactic objects are produced by two modules, QSOC and
UGC, which analyse BP/RP spectra of quasars and galaxies, respectively. The
selection of the processed sources uses the Combmod class probabilities of
DSC, classprob_dsc_combmod_quasar and classprob_dsc_combmod_galaxy. The CU8
extragalactic parameters are found in the qso_candidates and galaxy_candidates
tables. More details on the quality and processing of these parameters are
given in the accompanying Paper III (Delchambre, 2022).

####  6.5.1 QSOC

Figure 25: Galactic sky distribution of the fraction of QSOC predictions that
do not raise warning flags (i.e. flags_qsoc$=0$), even if they are based on
BP/RP spectra of lower quality (i.e. flag Z_BAD_SPEC$=16$ can be set).
Fractions are computed within radii of $1^{\circ}$ over the whole celestial
sphere.

QSOC predicts quasar redshifts redshift_qsoc and associated confidence levels
redshift_qsoc_lower and redshift_qsoc_upper in the range $0.0826<z<6.1295$.
Intentionally, the module chose to be complete and produced results on 6.4
million sources, which is three times the expected number of quasars that Gaia
should theoretically observe. Although this choice may seem questionable, it
gives the final user a much higher chance of finding the redshift of the
sources they are interested in, at the expense of having many contaminating
stars amongst these predictions (see however Bailer-Jones, 2022, their Section
8 for the selection of purer samples).

In order to help discriminating valuable redshift predictions from those where
potential processing issues may arise, we defined two quality measurements:
ccfratio_qsoc and zscore_qsoc. The ccfratio_qsoc field is associated with the
$\chi^{2}$ resulting from the fit of the BP/RP spectra to the templates at the
predicted redshift. Predictions whose redshift is associated with a minimal
$\chi^{2}$ (compared to the $\chi^{2}$ resulting from alternative redshifts)
have ccfratio_qsoc $=1$ and less than one otherwise. The zscore_qsoc field is
associated with the successful modelling of common quasar emission lines. We
have that zscore_qsoc $=1$ if all covered quasar emission lines appear in the
spectrum whereas the miss of a single emission line often leads to very low
values of zscore_qsoc. These quality measurements are summarised in the
flags_qsoc field where boolean flags are set that principally depend on the
values of the ccfratio_qsoc and zscore_qsoc fields.

To illustrate the potential filtering that can be done using flags_qsoc, in
Fig. 25 we show the sky distribution of the fraction of QSOC predictions for
which all flags other than the Z_BAD_SPEC262626The Z_BAD_SPEC flag is raised
for sources of lower quality, and rely on a combination of the source $G$
magnitude and number of BP/RP spectral transits. flag are set to zero. These
correspond to sources where no processing error occurs, even though some
predictions are based on spectra of lower quality (i.e. predictions with
either flags_qsoc $=0$ or flags_qsoc $=16$). We can clearly see that high
stellar density regions (Galactic plane, Magellanic Clouds, globular clusters
and nearby galaxies) usually have a lower fraction of predictions with
flags_qsoc $=\\{0,16\\}$. Imprints of the scanning law are also seen. These
arise from the higher or lower number of spectral transits that leads to a
higher or lower SNR of the spectra and hence more or less confident
predictions by QSOC.

####  6.5.2 UGC

The UGC module provides galaxy redshift parameters redshift_ugc, with $0.0\leq
z\leq 0.6$, and associated uncertainties redshift_ugc_lower and
redshift_ugc_upper, for $1,367,153$ sources in the galaxy_candidates table.
Note that these uncertainties are computed from the standard deviation of the
SVM predictions of sources with known redshift and should accordingly not be
considered as per-source confidence intervals but rather as a measure of the
SVM performance.

As the sources in UGC may have a relatively low probability to be galaxies
(our selection is classprob_dsc_combmod_galaxy $>0.25$), we expect a number of
misclassified quasars to contaminate the redshift_ugc results. Potentially,
about $1\%$ of these are true quasars with redshifts $z>0.6$. It is also
expected that some true high-redshift galaxies are erroneously processed by
UGC, however their number is estimated to be negligibly small. Their predicted
redshifts would therefore be underestimated by UGC. Estimated redshifts below
0.02 or larger than 0.40 are not well constrained, and there is a suspicious
peak of sources in a very narrow bin at $0.0707<z<0.0709$ (for details see
Section 11.3.13 of the online documentation). In Fig. 26 we show
$G_{\mathrm{RP}}$ versus $G_{\mathrm{BP}}$ diagrams illustrating the magnitude
ranges for which we find galaxies with specific redshifts. Higher-redshift
galaxies ($z\sim 0.6$) are only found at the very faint end, as expected.

Figure 26: Magnitude-magnitude diagrams for the 1.4 million galaxies for which
redshifts are provided by UGC. Each panel shows the density distribution of a
different redshift range in a different colour, while the distribution in grey
shows the whole sample.

###  6.6 Outlier analysis

The results produced by OA can be helpful to perform an extensive analysis of
those sources that were assigned a lower classification probability by DSC.
Such sources are usually faint stars or extragalactic objects, which can be
studied through the parameters generated by OA, but they also contain known
objects such as white dwarfs and brown dwarfs. The sources classified as
outliers are given in the astrophysical_parameters table, as explained in
Sect. 6.1.2. Two multi-dimensional tables are also available to further
interpret these data.

The oa_neuron_information table is a self-organising map (SOM) arranged in a
rectangular lattice composed of 900 neurons, where each neuron groups similar
objects that are described by means of different statistical parameters. In
order to assess the quality of the clustering, different indices are available
in this same table, so that high quality or low quality neurons can be
identified and filtered as required by the user to perform their own analysis
or to isolate specific types or groups of objects. To ease such an analysis,
an indication of the astronomical type of the sources is also provided for the
best quality neurons. The SOM is shown in Fig. 12 and the coloured neurons are
the best quality ones with class labels.

Each neuron is associated with a synthetic BP/RP spectrum, the so-called
prototype, which is representative of the spectra of the sources that are
assigned to a certain neuron. In addition, for those neurons where a class
label is provided, the BP/RP spectrum of the template is found in the
oa_neuron_xp_spectra table. Further information is provided in the
accompanying Paper III (Delchambre, 2022).

###  6.7 Auxiliary data products

The auxiliary data products comprise quality metrics, convergence indicators,
flags, the name of the best library for the GSP-Phot results, and the
bolometric correction. Most of these auxiliary data products have been
described in one of the above subsections. These are also listed towards the
bottom of Table 5.3.

##  7 Validation of results

The aim of this paper is to explain the production of and overall content of
the astrophysical parameters from CU8 in Gaia DR3. The accompanying Papers II
and III focus on the validation and quality of the results for stellar-based
APs (Fouesneau, 2022) and the non-stellar content and source classification
(Delchambre, 2022). Additional validation results are also given in the
dedicated online documentation chapter. Validation on GSP-Phot, GSP-Spec, and
ESP-CS-specific products are also found in their dedicated papers (Andrae,
2022; Lanzafame, 2022; Recio-Blanco, 2022b). Here we briefly describe our
validation procedures.

The validation of the CU8 data products included several steps. At a first
level, many Apsis test runs were performed prior to receiving the upstream
data during the DR3 development stage (2018 -- 2020). This repeated validation
was done on a module-by-module basis for a limited number of mostly random
sources (10 million). The teams compiled many validation tests to ensure that
the software performed as intended. Such tests comprise checking the
astrophysical content of the data, e.g. HR diagrams such as Fig. 24, which
helped to point out weaknesses in the codes in certain parameter spaces.
Comparisons with external data allowed the teams to check whether their
results are consistent with what is already known in the literature. It is
important to point out, however, that Apsis does not do any calibration of its
APs to mimic external catalogues. The external catalogues were merely used as
a consistency check. Once the final input data had been received (six months
before operations), further test runs were performed to refine the codes and
to adapt parameter settings to the final data.

A higher level of validation was performed using a validation database hosted
at the ESAC operations center in Madrid. This allowed the CU8 team to perform
many cross-checks between the individual parameters, e.g. Fig. 14, and
provided important feedback for necessary modifications to the code before the
full operations sequence. It also allowed a statistical check on the full
dataset, which then allowed the post-processing codes to be prepared for
filtering results and setting archive flags.

A third level of validation was then performed by the Coordination Unit 9
(CU9) archive team once the operational data had been delivered. In essence
they were the first users of the data, and had an external view of the full
results in the archive just as a user would, see Babusiaux et al. (2022). Once
the CU8 results were final, the validation by CU9 only allowed us to perform
minor updates on parameters through post-processing, e.g. removing results for
some sources or removing fields from the Gaia archive. There are, nonetheless,
some issues that are now known that could not be corrected, and these along
with some caveats are summarised in the next section.

##  8 Caveats and known issues

There are several caveats that the user should be aware of before using the
data. Additionally, a number of issues have been found following extensive
validation. We list both caveats and the main issues known to us at the time
of writing, starting with general comments on variability and crowding,
followed by a discussion on a module-by-module basis. The user should consider
these issues when using the APs in Gaia DR3.

##### Variability:

Apsis processes the mean BP/RP and RVS spectra, astrometry, and mean
magnitudes provided by upstream processing systems. Therefore, we advise users
to consider the variability of their source before deciding whether the APs
from Apsis are adapted to their specific science case. As a concrete example,
RR Lyrae stars have large amplitude variability, and a mean spectrum for these
stars will not necessarily represent the mean state of such a star.
Additionally, as these spectra vary significantly, the concept of a
$T_{\mathrm{{eff}}}$ from one mean spectrum does not make astrophysical sense,
see e.g. Clementini et al. (2022).

##### Crowding:

Crowding is a major limitation in dense regions such as stellar clusters. As
an example, Fig. 27 shows that CU8 results differ significantly between the
dense core and the outer regions of the globular cluster Omega Centauri
(Calamida et al., 2020). For low-resolution BP/RP spectra, the allocated CCD
window is 3.5 arcsec $\times$ 2.1 arcsec (Carrasco et al., 2021), i.e.
theoretically $\sim$1.76 million windows would fit into one square degree. In
practice though, windows on a source over a range of observation epochs will
have quasi-random orientations on the sky and Fig. 27a suggests that CCD
windows already start to overlap at $\sim$600 000 sources per square degree,
thereby producing blended BP/RP spectra (and photometry, see Fig. 27b) that
lead to systematically incorrect CU8 results. For RVS spectra, the window size
is much larger than for BP/RP spectra (74.2 arcsec $\times$ 1.8 arcsec prior
to June 2015, 75.3 arcsec $\times$ 1.8 arcsec after June 2015, see Cropper et
al., 2018) but RVS spectra are deblended (Seabroke & et al., 2022).

Figure 27: Crowding effects in the globular cluster Omega Centauri. Black
contours are identical in all panels and indicate source density dropping by
factors of 2. Panel a: source density. Panel b: excess factor in photometric
flux. Panel c: galaxy class probability from DSC-Combmod. Panel d: extinction
estimate $A_{0}$ from GSP-Phot. Panel e: metallicity estimate from GSP-Phot.
Panel f: age estimate from FLAME.

##### DSC:

Performance on white dwarfs and physical binaries is poor, and in general the
probabilities may not be well calibrated. The purity of quasars and galaxies
on the full sample is low, but this improves (to $\sim$80%) when excluding low
Galactic latitudes ($|b|<10^{\circ}$) and using classlabel_dsc_joint. Note
that these purities account for the expected dominance of contaminating stars
in a random sample selected from Gaia.

##### GSP-Phot:

Distance estimates tend to be underestimated beyond 2--3 kpc due to a harsh
extinction prior. Yet, distances remain reliable for high-quality parallax
measurements ($\frac{\varpi}{\sigma_{\varpi}}>20$) even out to 10kpc.
Metallicities show an offset of about --0.2 dex compared to external
literature sources with [M/H] $>-1$ dex, and additional systematics exist
below --1.0 dex. We recommend correcting these metallicities using the
empirical correction that has been made available to the community, see
Appendix. E. Also, the uncertainties are known to be underestimated. This is
most probably due to ignoring the off-diagonal elements of the covariance
matrix272727Ignoring the covariance matrix could also lead to overestimated
uncertainties, but tests with GSP-Phot have shown that the effects are
primarily to underestimate them.. Another possible explanation could be in the
mismatches between model SEDs and observed BP/RP spectra (see Fig. 1 in Andrae
2022). This is the subject of ongoing investigations. Comparisons with
external data show median absolute differences on the order of 120 K for FGK-
type stars, 340 K for A stars and 1600 K for B stars, see Table 11.19 in the
online documentation.

##### GSP-Spec:

Quality flags with up to 41 characters have been provided for the best use of
the data; users are strongly encouraged to use these for selecting best sample
stars. In the case of Matisse-Gauguin (astrophysical_parameters table), the
parameters $\log\,g$ and [M/H] show some biases with respect to the
literature, and these depend on $\log\,g$. Recio-Blanco (2022b) have proposed
a $\log\,g$ and [M/H]-calibration procedure, to be applied only if the user-
specific science case requires it. ANN results (astrophysical_parameters_supp)
also show biases with respect to the literature, as discussed in Recio-Blanco.

##### MSC

uses a solar-metallicity prior for deriving $T_{\mathrm{{eff}}}$ and $\log\,g$
of binary components, and $\log\,g$ values are in general overestimated with
respect to external catalogues. Additionally, as MSC treats all stars as
binaries, one would need an external catalogue to identify a reliable set of
binary stars. We also note that the value reported as the parameter in the
astrophysical_parameters table is the median value of the last 100 values
available in the mcmc_samples_msc table, while the 16th and 84th percentiles
come from the full MCMC chain from the processing.

##### FLAME

uses a solar-metallicity prior for deriving masses, ages, and evolutionary
stage, therefore for stars with known metallicities $<-0.5$ the age should be
used with caution. The uncertainties in the FLAME masses and ages are also
underestimated, mainly due to the underestimated uncertainties in
$T_{\mathrm{{eff}}}$. For the use of the masses of giants i.e. with the first
digit of flags_flame(_spec) = 1, the published results should only be used
within the range 1--2 $\,{{M}}_{\odot}$ (approximately 14 million of 27
million sources).

##### ESP-HS

uses a solar-metallicity prior to derive spectroscopic parameters of hot
stars. The validity of this assumption should be considered in the context of
the user-specific science case.

##### ESP-UCD

uses empirical training data for the prediction of $T_{\mathrm{{eff}}}$, so
sources deviating significantly from the median metallicities or gravities of
the training set, solar metallicities and $\log\,g\approx 5-5.5$, may have
biased estimates of $T_{\mathrm{{eff}}}$. Also, the list of UCD candidates in
the quality class 2 contains some contaminants due to incorrect astrometry.
This is visible in the distribution of UCD candidates on the celestial sphere
as overdensities in the Galactic disk plane (see Sect. 11.3.10 of the online
documentation). Finally, comparisons with effective temperatures of limited
samples in the literature show a systematic difference in the sense that ESP-
UCD estimates are $\sim 65$ K lower than the literature values for the hot end
of the sample ($T_{\rm eff}>2300$ K; see Creevey, 2022).

##### QSOC

aims for completeness, not purity, and accordingly processed a large fraction
of stars. The prediction of the redshifts of $0.9<z<1.3$ quasars are
complicated by the sole detectable presence of the Mg ii emission lines in the
BP/RP spectra over this redshift range. QSOC is designed to process
Type-I/core-dominated quasars with broad emission lines in the optical and
accordingly yields only poor predictions on galaxies, type-II AGN, and BL
Lacertae/Blazar objects. Use of the flags is also encouraged.

##### UGC:

Some contamination by high redshift galaxies and high redshift quasars is
expected in the sample. There is also a suspicious peak of sources in the
$0.0707<z<0.0709$ range.

##### TGE:

While the TGE extinction maps show excellent agreement with comparable
extinction maps, there is a small bias at very small extinctions ($A_{0}$ <
0.1 mag), and possibly at large extinctions ($A_{0}$ > 4 mag). It is not
advisable to use the TGE maps at very low ($|b|<5^{\circ}$) Galactic
latitudes, see Delchambre (2022) for further details.

##  9 Conclusions

Gaia DR3 contains one of the most extensive catalogues of astrophysical
parameters to be exploited by the community, and it is based on Gaia-only
data. It contains valuable information on stellar and non-stellar sources, and
these parameters appear in ten main archive tables. A minor subset of APs also
appear in gaia_source to simplify querying for a new user of ADQL. There are
up to 1.6 billion classifications of objects (star, galaxy, ...), along with
470 million stellar-based APs and 6 million extragalactic redshifts of quasar
(6M) and galaxy (2M) candidates, along with a self-organising map of outliers,
total Galactic extinction maps and MCMC samples. All of these were produced by
the Apsis analysis system (Sect. 3).

Stellar-based analyses were performed using general methods by the Apsis
modules GSP-Phot and GSP-Spec which derived the spectroscopic parameters
($T_{\mathrm{{eff}}}$, $\log\,g$, ...) using the BP/RP and RVS spectra,
respectively, assuming each source to be a single star. The MSC module also
analysed the BP/RP spectra in a general way but assumed the BP/RP spectrum of
each source to be a composition of two unresolved components of a physical
binary. The FLAME module derived evolutionary parameters (${R}$, ${M}$, ...)
using the spectroscopic parameters from GSP-Phot and GSP-Spec together with
distance measures and photometry. Furthermore, specialised modules produced
results focussing on specific types of stars: ESP-CS, ESP-HS, ESP-UCD, and
ESP-ELS analysed cool active stars, hot stars, ultra cool dwarfs, and
emission-line stars, respectively to provide class probabilities and labels of
emission-line stars, activity index, $T_{\mathrm{{eff}}}$, $v\sin i$,
H-$\alpha$ equivalent widths, and spectral types. All of the parameters are
found in the main astrophysical_parameters table, with supplementary results
from GSP-Phot, GSP-Spec, and FLAME in the astrophysical_parameters_supp table.
Samples of the MCMC chains are also available for GSP-Phot and MSC, see
Appendix B.

Redshifts of galaxies and quasars are available in the qso_candidates and
galaxy_candidates tables. Two-dimensional total Galactic extinction maps are
provided, which are based on the extinction tracers provided by GSP-Phot, at
four different HEALPix levels and an optimal one. The results of an
unsupervised analysis of outliers are found in the oa_neuron_information and
oa_neuron_xp_spectra tables.

As with any extensive catalogue analysing all sources in a homogenous way,
there are both high quality as well as low quality results. Caveats and known
issues with the catalogue are summarised in Sect. 8. Further information on
the quality, validation and use of the data is provided in the accompanying
Papers II and III which target the stellar parameters and the non-stellar
content, respectively (Fouesneau, 2022; Delchambre, 2022). More technical
details are available in the online documentation. Dedicated papers focusing
on specific Apsis modules are also available, for GSP-Phot (Andrae, 2022),
GSP-Spec (Recio-Blanco, 2022b), ESP-CS (Lanzafame, 2022) and QSOC (Delchambre,
2018). In addition to these data, several tools are made available to the
community to aid their exploitation, see Appendix E.

Examples of the use and performance of the astrophysical parameters are shown
in several papers accompanying Gaia DR3. Bailer-Jones (2022) explores the
extra-galactic content of Gaia, by combining results from Apsis with other
Gaia results. Drimmel (2022) exploits the Apsis results along with kinematics
and known variable sources to map the spiral arms of the Milky Way. Creevey
(2022) focusses on several regions of the HR diagram to provide golden samples
of APs, including a focus on Carbon stars and solar-analogues. Recio-Blanco
(2022a) illustrates the chemo-dynamical analysis of disc and halo populations
using the chemical analysis from Apsis, and Schultheis (2022) traces the
spatial structure of the Galactic interstellar medium using the $\lambda$862
nm DIB.

Gaia DR4 (2025) will be based on the analysis of 66 months of Gaia data which
is almost twice as long as the data in the current data release. This fact
along with the application of improved data processing methods will ensure a
significant improvement in the quality of the astrometric, photometric and
spectroscopic data. This, in turn, will impact the quality of the
astrophysical parameters with improved control of sources of systematic
errors, to deliver an even more extensive catalogue of Gaia-based
astrophysical parameters to the community.

###### Acknowledgements.

We thank the referee for their careful reading of the paper and for providing
constructive feedback. This work presents results from the European Space
Agency (ESA) space mission Gaia. Gaia data are processed by the Gaia Data
Processing and Analysis Consortium (DPAC). Funding for the DPAC is provided by
national institutions, in particular the institutions participating in the
Gaia MultiLateral Agreement (MLA). The Gaia mission website is
https://www.cosmos.esa.int/gaia. The Gaia archive website is
https://archives.esac.esa.int/gaia. Acknowledgments from the financial
institutions are in Appendix F. We thank our DPAC colleague Hector Canovas for
providing Python routines to download spectra and MCMC samples. The data
analysis made use of Vaex (Breddels & Veljanoski, 2018), TOPCAT (Taylor,
2005), and R (R Core Team, 2013).

## References

  * Abia et al. (2020) Abia, C., de Laverny, P., Cristallo, S., Kordopatis, G., & Straniero, O. 2020, A&A, 633, A135
  * Aguado et al. (2019) Aguado, D. S., Ahumada, R., Almeida, A., et al. 2019, ApJS, 240, 23
  * Ahumada et al. (2020) Ahumada, R., Prieto, C. A., Almeida, A., et al. 2020, ApJS, 249, 3
  * Allard et al. (2013) Allard, F., Homeier, D., & Freytag, B. 2013, Mem. Soc. Astron. Italiana, 84, 1053
  * Altavilla et al. (2015) Altavilla, G., Marinoni, S., Pancino, E., et al. 2015, Astronomische Nachrichten, 336, 515
  * Álvarez et al. (2021) Álvarez, M. A., Dafonte, C., Manteiga, M., Garabato, D., & Santoveña, R. 2021, Neural Computing and Applications
  * Andrae (2022) Andrae, R. 2022, A&A in prep.
  * Andrae et al. (2018) Andrae, R., Fouesneau, M., Creevey, O., et al. 2018, A&A, 616, A8
  * Arenou (2022) Arenou, F. 2022, A&A in prep.
  * Babusiaux et al. (2022) Babusiaux, C., Fabricius, C., & et al. 2022, A&A in prep.
  * Bailer-Jones (2022) Bailer-Jones, C. 2022, A&A in prep.
  * Bailer-Jones (2011) Bailer-Jones, C. A. L. 2011, MNRAS, 411, 435
  * Bailer-Jones (2021) Bailer-Jones, C. A. L. 2021, GAIA-C8-TN-MPIA-CBJ-094, link
  * Bailer-Jones et al. (2019) Bailer-Jones, C. A. L., Fouesneau, M., & Andrae, R. 2019, MNRAS, 490, 5615
  * Bailer-Jones, C. A. L. et al. (2013) Bailer-Jones, C. A. L., Andrae, R., Arcay, B., et al. 2013, A&A, 559, A74
  * Blanton et al. (2017) Blanton, M. R., Bershady, M. A., Abolfathi, B., et al. 2017, AJ, 154, 28
  * Breddels & Veljanoski (2018) Breddels, M. A. & Veljanoski, J. 2018, A&A, 618, A13
  * Calamida et al. (2020) Calamida, A., Zocchi, A., Bono, G., et al. 2020, ApJ, 891, 167
  * Carrasco et al. (2021) Carrasco, J. M., Weiler, M., Jordi, C., et al. 2021, A&A, 652, A86
  * Chang & Lin (2011) Chang, C.-C. & Lin, C.-J. 2011, ACM Transactions on Intelligent Systems and Technology, 2, 27:1, software available at http://www.csie.ntu.edu.tw/~cjlin/libsvm
  * Chen et al. (2015) Chen, Y., Bressan, A., Girardi, L., et al. 2015, MNRAS, 452, 1068
  * Clementini et al. (2022) Clementini et al. 2022, A&A in prep.
  * Coifman & Lafon (2006) Coifman, R. R. & Lafon, S. 2006, Applied and Computational Harmonic Analysis, 21, 5, special Issue: Diffusion Maps and Wavelets
  * Contursi et al. (2021) Contursi, G., de Laverny, P., Recio-Blanco, A., & Palicio, P. A. 2021, A&A, 654, A130
  * Cortes & Vapnik (1995) Cortes, C. & Vapnik, V. 1995, Machine Learning, 20, 273
  * Creevey (2022) Creevey, O. 2022, A&A in prep.
  * Creevey & Lebreton (2022) Creevey, O. L. & Lebreton, Y. 2022, Gaia Data Processing and Analysis Consortium (DPAC) technical note GAIA-C8-TN-OCA-OLC-035, http://www.cosmos.esa.int/web/gaia/public-dpac-documents
  * Cropper et al. (2018) Cropper, M., Katz, D., Sartoretti, P., et al. 2018, A&A, 616, A5
  * Dafonte et al. (2016) Dafonte, C., Fustes, D., Manteiga, M., et al. 2016, A&A, 594, A68
  * De Angeli et al. (2022) De Angeli et al. 2022, A&A in prep.
  * Delchambre (2018) Delchambre, L. 2018, MNRAS, 473, 1785
  * Delchambre (2022) Delchambre, L. 2022, A&A in prep.
  * Dieterich et al. (2014) Dieterich, S. B., Henry, T. J., Jao, W.-C., et al. 2014, AJ, 147, 94
  * Dorman et al. (1993) Dorman, B., Rood, R. T., & O’Connell, R. W. 1993, ApJ, 419, 596
  * Drimmel (2022) Drimmel, R. 2022, A&A in prep.
  * Ducourant (2022) Ducourant, C. 2022, A&A in prep.
  * El-Badry & Rix (2018) El-Badry, K. & Rix, H.-W. 2018, MNRAS, 480, 4884
  * Fitzpatrick (1999) Fitzpatrick, E. L. 1999, PASP, 111, 63
  * Fouesneau (2022) Fouesneau, M. 2022, A&A in prep.
  * Frémat et al. (2005) Frémat, Y., Zorec, J., Hubert, A. M., & Floquet, M. 2005, A&A, 440, 305
  * Gaia Collaboration et al. (2016) Gaia Collaboration, Brown, A., Vallenari, A., et al. 2016, A&A, 595, A2
  * Gaia Collaboration et al. (2018) Gaia Collaboration, Brown, A. G. A., Vallenari, A., et al. 2018, A&A, 616, A1
  * Gaia Collaboration et al. (2021a) Gaia Collaboration, Brown, A. G. A., Vallenari, A., et al. 2021a, A&A, 650, C3
  * Gaia Collaboration et al. (2021b) Gaia Collaboration, Smart, R. L., Sarro, L. M., et al. 2021b, A&A, 649, A6
  * Georgy et al. (2013) Georgy, C., Ekström, S., Granada, A., et al. 2013, A&A, 553, A24
  * Geurts et al. (2006) Geurts, P., Ernst, D., & Wehenkel, L. 2006, Mach. Learn., 63, 3
  * Grevesse et al. (2007) Grevesse, N., Asplund, M., & Sauval, A. J. 2007, Space Sci. Rev., 130, 105
  * Gustafsson et al. (2008) Gustafsson, B., Edvardsson, B., Eriksson, K., et al. 2008, A&A, 486, 951
  * Hastie & Stuetzle (1989) Hastie, T. & Stuetzle, W. 1989, Journal of the American Statistical Association, 84, 502
  * Hidalgo et al. (2018) Hidalgo, S. L., Pietrinferni, A., Cassisi, S., et al. 2018, ApJ, 856, 125
  * Holtzman et al. (2015) Holtzman, J. A., Shetrone, M., Johnson, J. A., et al. 2015, AJ, 150, 148
  * Jönsson et al. (2020) Jönsson, H., Holtzman, J. A., Allende Prieto, C., et al. 2020, AJ, 160, 120
  * Kohonen (2001) Kohonen, T. 2001, Self-Organizing Maps, 3rd edn., Springer Series in Information Sciences (Berlin Heidelberg: Springer-Verlag)
  * Lanzafame (2022) Lanzafame, A. 2022, A&A in prep.
  * Lindegren et al. (2021) Lindegren, L., Bastian, U., Biermann, M., et al. 2021, A&A, 649, A4
  * Liu et al. (2012) Liu, C., Bailer-Jones, C. A. L., Sordo, R., et al. 2012, MNRAS, 426, 2463
  * Manara et al. (2020) Manara, C. F., Natta, A., Rosotti, G. P., et al. 2020, A&A, 639, A58
  * Manteiga et al. (2010) Manteiga, M., Ordóñez, D., Dafonte, C., & Arcay, B. 2010, PASP, 122, 608
  * Marigo et al. (2017) Marigo, P., Girardi, L., Bressan, A., et al. 2017, ApJ, 835, 77
  * Marinoni et al. (2016) Marinoni, S., Pancino, E., Altavilla, G., et al. 2016, MNRAS, 462, 3616
  * Montegriffo et al. (2022) Montegriffo et al. 2022, A&A in prep.
  * Noyes et al. (1984) Noyes, R. W., Hartmann, L. W., Baliunas, S. L., Duncan, D. K., & Vaughan, A. H. 1984, ApJ, 279, 763
  * Pancino et al. (2012) Pancino, E., Altavilla, G., Marinoni, S., et al. 2012, MNRAS, 426, 1767
  * Pâris et al. (2018) Pâris, I., Petitjean, P., Aubourg, É., et al. 2018, A&A, 613, A51
  * Passegger et al. (2018) Passegger, V. M., Reiners, A., Jeffers, S. V., et al. 2018, A&A, 615, A6
  * Pastorelli et al. (2020) Pastorelli, G., Marigo, P., Girardi, L., et al. 2020, MNRAS, 498, 3283
  * Plez (2012) Plez, B. 2012, Turbospectrum: Code for spectral synthesis, Astrophysics Source Code Library
  * Queiroz et al. (2020) Queiroz, A. B. A., Anders, F., Chiappini, C., et al. 2020, A&A, 638, A76
  * R Core Team (2013) R Core Team. 2013, R: A Language and Environment for Statistical Computing, R Foundation for Statistical Computing, Vienna, Austria
  * Rabus et al. (2019) Rabus, M., Lachaume, R., Jordán, A., et al. 2019, MNRAS, 484, 2674
  * Rasmussen & Williams (2006) Rasmussen, C. & Williams, C. 2006, Gaussian Processes for Machine Learning, Adaptive Computation and Machine Learning (Cambridge, MA, USA: MIT Press), 248
  * Recio-Blanco (2022a) Recio-Blanco, A. 2022a, A&A in prep.
  * Recio-Blanco (2022b) Recio-Blanco, A. 2022b, A&A in prep.
  * Recio-Blanco et al. (2006) Recio-Blanco, A., Bijaoui, A., & de Laverny, P. 2006, MNRAS, 370, 141
  * Recio-Blanco et al. (2016) Recio-Blanco, A., de Laverny, P., Allende Prieto, C., et al. 2016, A&A, 585, A93
  * Reiners et al. (2007) Reiners, A., Homeier, D., Hauschildt, P. H., & Allard, F. 2007, A&A, 473, 245
  * Riello et al. (2021) Riello, M., De Angeli, F., Evans, D. W., et al. 2021, A&A, 649, A3
  * Rybizki et al. (2020) Rybizki, J., Demleitner, M., Bailer-Jones, C., et al. 2020, PASP, 132, 074501
  * Sartoretti et al. (2018) Sartoretti, P., Katz, D., Cropper, M., et al. 2018, A&A, 616, A6
  * Schultheis (2022) Schultheis, M. 2022, A&A in prep.
  * Seabroke & et al. (2022) Seabroke, G. & et al. 2022, A&A in prep.
  * Smart et al. (2017) Smart, R. L., Marocco, F., Caballero, J. A., et al. 2017, MNRAS, 469, 401
  * Smart et al. (2019) Smart, R. L., Marocco, F., Sarro, L. M., et al. 2019, MNRAS, 485, 4423
  * Stephens et al. (2009) Stephens, D. C., Leggett, S. K., Cushing, M. C., et al. 2009, ApJ, 702, 154
  * Tang et al. (2014) Tang, J., Bressan, A., Rosenfield, P., et al. 2014, MNRAS, 445, 4287
  * Tanga (2022) Tanga, P. 2022, A&A in prep.
  * Taylor (2005) Taylor, M. B. 2005, in Astronomical Society of the Pacific Conference Series, Vol. 347, Astronomical Data Analysis Software and Systems XIV, ed. P. Shopbell, M. Britton, & R. Ebert, 29
  * Vallenari (2022) Vallenari, A. 2022, A&A in prep.
  * Wenger et al. (2000) Wenger, M., Ochsenbein, F., Egret, D., et al. 2000, A&AS, 143, 9
  * Zhao et al. (2021) Zhao, H., Schultheis, M., Recio-Blanco, A., et al. 2021, A&A, 645, A14
  * Zwitter et al. (2004) Zwitter, T., Castelli, F., & Munari, U. 2004, A&A, 417, 1055

##  Appendix A Empirical training in Apsis modules

Here we provide details of algorithm training using empirical data for the
modules that take this approach, namely DSC, UGC, ESP-UCD, ESP-ELS, and MSC.

###  A.1 DSC

DSC classifies sources into one of five classes: quasar, galaxy, star, white
dwarf, physical binary star. DSC is trained empirically, meaning it is trained
on a labelled subset of the Gaia data it is later applied to. We select an
external catalogue for each class of objects (e.g. quasars), assumed to have
high purity, and cross-match this to Gaia sources. The BP/RP spectra,
photometry, and astrometry of those matched sources (of order $10^{5}$) are
then used as the training data for that class. Once trained, the machine
learning models -- ExtraTrees and Gaussian Mixture Models -- map the Gaia data
to the class probability for each source.

For the quasar and galaxy classes, sources were selected from
spectroscopically-confirmed objects in SDSS catalogues (Pâris et al. 2018;
Aguado et al. 2019). White dwarfs were taken from the Montreal White Dwarf
Database282828http://www.montrealwhitedwarfdatabase.org/tables-and-
charts.html. Binaries were constructed artificially by combining real BP/RP
spectra of single stars; hence this class (only) is defined by objects that do
not appear explicitly in the Gaia data. The final class in DSC, ‘star’, is
simply a sample of objects drawn at random from Gaia DR3 that are not in the
other training sets. This class is, therefore, strictly an anonymous class of
objects. But as the vast majority of sources in Gaia are stars (well over
99%), and most of these appear single in Gaia data, this class is essentially
equivalent to apparently single stars.

As usual in empirical training, the selection of the training objects defines
the DSC classes. Note, therefore, that the galaxy class does not include all
conceivable galaxies, but only that subset of galaxies that both Gaia and SDSS
observe. Note further that not all types of galaxies have complete data in
Gaia (many lack parallaxes and proper motions, for example, meaning they
cannot be classified by DSC-Allosmod). DSC is a posterior probabilistic
classifier, meaning that its published probabilities depend both on a
likelihood model, and on a prior for each class. This prior is set explicitly
to reflect their expected global occurrence in Gaia DR3, in particular the
rareness of extragalactic objects. Such a prior must be used in any
classification project to avoid misleading performance, as explained in more
detail in Bailer-Jones et al. (2019). More details of the training sets and
the DSC models can be found in the online documentation for Gaia DR3.

###  A.2 UGC

UGC uses support vector machines (SVM) through the LIBSVM (Chang & Lin 2011)
package for the SVM model development, configuration, tuning, training, and
testing. The SVM-models for redshift prediction are trained empirically, using
a set of BP/RP spectra of unresolved galaxies observed by Gaia and sampled by
SMSgen. The selection of sources suitable for this was taken from the SDSS
DR16 archive (Ahumada et al. 2020; Blanton et al. 2017). There are
$2\,714\,637$ sources with SDSS class ‘‘GALAXY’’ that have reliable photometry
as well as spectroscopic redshifts. Of these sources, $1\,189\,812$ have been
observed by Gaia while BP/RP spectra are available for $711\,600$ of them.
These spectra, along with the corresponding SDSS redshifts, were used for
training and testing the SVM models. Because of the very small number of
galaxies with redshifts lower than 0.01 and higher than 0.6, the redshift
range for which the SVMs were trained was limited to $0.01\leq{}z\leq{}0.60$,
leaving $709\,449$ galaxies, of which $6000$ were selected for the SVM models
training set, while the rest were used for performance testing.

The suitability of the galaxies selected for training has been established via
a number of rules and conditions involving various source parameters. The
allowed range of magnitudes was limited to $13.0\leq{}G\leq{}21.0$, and the
galaxy class probability as estimated by DSC-Combmod had to be larger than
$0.25$. Moreover, the galaxy image size, as defined in SDSS by the Petrosian
radius at 50% of the total flux in the r-band, was restricted to
$0.5^{\prime\prime}\leq{}petroRad50\\_r\leq 5.0^{\prime\prime}$ so as to avoid
both suspiciously compact sources and very extended ones. An interstellar
extinction (as defined by the SDSS r-band) upper limit of 0.5 was also
applied, to exclude significantly reddened sources. Finally, it was required
that the mean BP/RP spectrum was constructed from at least six epoch spectra,
and that the mean fluxes lie within limits $0.3\leq$ bp_flux_mean $\leq 100$
e-s-1 and $0.5\leq$ rp_flux_mean $\leq 200$ e-s-1, thus removing spectra with
low signal-to-noise ratios or with suspiciously high signals. Details on the
SVM model preparation can be found in Delchambre (2022).

###  A.3 ESP-UCD

The effective temperatures inferred by the ESP-UCD module come from a Gaussian
Process regression model (Rasmussen & Williams 2006) trained on the set of
Gaia RP spectra described below. Empirical training was adopted for ESP-UCD
because systematic differences were found between the simulated RP spectra
(see Section 4.6 above) obtained for the synthetic library of BT Settl spectra
(Allard et al. 2013) and the observed RP spectra of ultracool dwarfs in the
Gaia UltraCool Dwarf Sample (Smart et al. 2017, 2019, hereafter GUCDS). The
empirical training set is composed of a total of 995 observed Gaia RP spectra.
The first 36 are GUCDS sources defined as spectral standards and their
effective temperatures are derived from the calibration by Stephens et al.
(2009). We then added 282 sources with temperature determinations from high
resolution spectra (Passegger et al. 2018) or derived using interferometric
radii (see e.g. Rabus et al. 2019; Dieterich et al. 2014), most (but not all)
of them are hotter than the UCD regime but included to allow for
extrapolation. These 36+282 examples with effective temperature estimates were
not enough and did not cover homogeneously the range of effective temperatures
of the UCD regime. In order to complete the training set, we added 679 sources
selected from the Gaia Catalogue of Nearby Stars (GCNS Gaia Collaboration et
al. 2021b), lying outside the high source density regions of the sky (Galactic
bulge and disk, and the Magellanic Clouds), with $G<19$ mag and
$G-G_{\mathrm{RP}}<1.8$ mag. Their RP spectra were visually inspected to
confirm their UCD nature. Since these 679 sources did not have effective
temperature estimates, we had to infer them -- as described below -- in order
to use them as part of the training set.

The data set then consists of 995 RP spectra with 120 flux values each. The
features of these RP spectra were assumed to vary smoothly as a function of
temperature so that we could use the labelled spectra (those of the 36+282
sources with effective temperatures in the literature) to calibrate the
relation and assign temperatures to the unlabelled sources. If this assumption
holds, it is justified to add the 679 sources to the training set. Quantifying
the relation between the 120 fluxes of each spectrum and the effective
temperatures is however a high-dimensional problem. In order to simplify the
task we constructed a Diffusion Map (Coifman & Lafon 2006) to reduce the
dimensionality of the data set and found that, as hypothesized, the 995 RP
spectra trace a curve in the first two Diffusion Map coordinates with a very
small scatter around it. The two-dimensional curve shown in Figure 28
represents in practice an ordering of the 995 RP spectra where the position
along the curve (non-linearly) parametrizes the temperature as shown by the
coloured circles.

Figure 28: Projection onto the first two diffusion map coordinates of the set
of 995 spectra defined in the text. The colour code reflects the temperatures
assigned according to their spectral types (filled circles) or literature
values (empty circles). Black dots represent the entire set of sources used to
construct the Diffusion Map. The inset plot is a zoom of the problematic zone
around 2000 K where temperature variations causes only very subtle changes in
the RP spectrum.

We used the position of the sources with effective temperature estimates from
the literature to calibrate the relation between the Diffusion Map coordinates
and temperature. This was done by fitting a Principal Curve (Hastie & Stuetzle
1989) to the first two Diffusion Map coordinates and the
$G+5\log_{10}(\varpi)+5$ (=MG) values of the 995 sources. The third coordinate
was introduced to avoid the non-monotonicity of the curve in the Diffusion Map
coordinates. The Principal Curve represents the minimum scatter maximum-
likelihood fit and implicitly defines a parameter $\lambda$ along it. We then
calibrated the relation between the curve parameter $\lambda$ and the
effective temperature using the labelled examples and a spline regression
model. The resulting calibration is shown in Figure 29. Finally, we used this
regression model to infer effective temperatures for the set of 679 sources
which were then included in the training set.

Figure 29: Calibration in temperature of the position of sources along the
principal curve fit in the 3D space comprised of the two first Diffusion Map
coordinates and the absolute magnitude, MG. The position along the curve is
parametrized with $\lambda$ and artificial scatter is added to help visualize
the number of sources used. Black points represent sources with effective
temperatures derived from high-resolution spectra and GUCDS standards while
orange points represent the rest of GUCDS sources.

###  A.4 ESP-ELS

Three Random Forest classifiers were used during the processing of ESP-ELS:
ELSRFC1, ELSRFC2, and ELSRFC3.

##### ELSRFC1

provides a spectral type tag to each star in order to avoid processing carbon
stars and to allow ESP-HS to preselect O-, B- and A-type stars. It was trained
on the MARCS, OB, A, and BTSettl synthetic spectra, see Sect. 4.1) with
spectral type tags: ‘‘O’’, ‘‘B’’, ‘‘A’’, ‘‘F’’, ‘‘G’’, ‘‘K’’, ‘‘M’’ (see Table
6) and on the observed BP/RP spectra of the Galactic carbon N stars compiled
by Abia et al. (2020) with spectral type tag ‘‘CSTAR’’. The wavelength domain
considered during the training varied from 340 to 600 nm in BP, and from 640
to 850 nm in RP. Both passbands were normalized individually to their
respective integrated flux, while their colour indices
($G_{\mathrm{BP}}$$-$$G$) and ($G$$-$$G_{\mathrm{RP}}$) were added to the flux
arrays.

Table 6: Adopted effective temperature range covered by the spectral type tag assigned to each synthetic spectrum during the training of ELSRFC1. tag |  $T_{\mathrm{{eff}}}$ range [K]
---|---
O |  | $>$ | 30 000
B | 9 790 | - | 30 000
A | 7 300 | - | 9 790
F | 5 940 | - | 7 300
G | 5 150 | - | 5 940
K | 3 840 | - | 5 150
M |  | $\leq$ | 3 840

##### ELSRFC2

identifies those BP/RP spectra belonging to the Wolf-Rayet stars, WC and WN,
and planetary nebula, PN. To these three classes we added the tag ‘‘unknown’’
to be given to all targets not showing features expected in Wolf Rayet stars
or planetary nebulae. The classification is based on the flux measured in
various wavelength domains extracted from the BP/RP spectra and normalized at
their edges, as well as on the astrophysical parameters (i.e.
$T_{\mathrm{{eff}}}$, $\log\,g$, and $A_{0}$). These wavelength ranges were
selected following the line features generally expected to be seen in emission
in various classes of ELS (see Fig. 8). Before the extraction and
normalisation of the features, the spectra are divided by the instrument
response provided by the MIOG simulator. For the training, we adopted the
available MARCS, OB, A, and BTSettl synthetic libraries as representative of
the spectra of non-emission line stars. We further added the observed BP/RP
spectra of Be, Herbig Ae/Be, T Tauri, and dMe stars (i.e. as a representation
of targets that are not Wolf-Rayet nor planetary nebulae), and observed BP/RP
spectra of WC, WN, and PNe. The observed data of targets with known stellar
classification were carefully inspected to only keep those spectra with
striking and unambiguous emission features. The number of targets finally
considered is given in the online documentation.

##### ELSRFC3

only processed targets with previously detected H$\alpha$ emission. It was
trained on the same features as ELSRFC2, but this time only extracted and
normalized for the reference Be, Herbig Ae/Be, T Tauri, and active M dwarf
stars. To the spectroscopic information, we further also added the
astrophysical parameters derived by GSP-Phot during the processing (i.e. no
filters and correction applied).

###  A.5 MSC

MSC uses an ExtraTrees algorithm (Geurts et al. 2006) to initialise its MCMC
chain. The algorithm was trained on stellar parameters from a wide binary
sample (El-Badry & Rix 2018) for which we artificially summed the BP/RP
spectra292929The stellar parameters were inferred from Gaia parallaxes and
photometry using https://github.com/jan-rybizki/isochrone_fitting_example with
PARSEC isochrones (Marigo et al. 2017) under the assumption of equal age,
extinction, metallicity and distance. The extinction was fixed using a 3D
extinction map from Rybizki et al. (2020).. The forward model used in the MCMC
inference is based on an empirical BP/RP spectra grid (next paragraph). The
distance and extinction prior use data from Rybizki et al. (2020) and the flux
ratio and HR-diagram prior are based on the wide binary sample’s parameter
distribution.

MSC uses a model grid of empirical BP/RP spectra instead of simulated spectra.
We thereby circumvent the problem of instrument modelling and the unavoidable
mismatch between simulated and real spectra. The grid is a function of
$T_{\mathrm{{eff}}}$, $\log\,g$, [M/H], and $A_{0}$ in the space of absolute
BP/RP spectra (i.e. flux at 10 pc distance). We used the ExtraTrees machine
learning algorithm (Geurts et al. 2006) on data from a sample of 80 000 APOGEE
(Holtzman et al. 2015) stars for which distance and extinction estimates are
available from StarHorse (Queiroz et al. 2020) and $T_{\mathrm{{eff}}}$,
$\log\,g$, and [M/H] estimates from the ASPCAP pipeline (Jönsson et al. 2020)
crossmatched with Gaia BP/RP spectra.

##  Appendix B Accessing MCMC chains for GSP-Phot and MSC

MCMC chains are provided for GSP-Phot and MSC through the Gaia Archive
DataLink. There are two methods to access them.

First, for any ADQL query results, the user can click on the right-most symbol
showing two interlocked links. This will open a pop-up window where the user
needs to select ‘‘Gaia DR3’’ as data release and ‘‘RAW’’ as data structure,
and the MCMC data available for download will be listed (together with, e.g.,
BP/RP spectra also available for download). However, this is limited to a
maximum of 5000 MCMC chains.

Second, MCMC chains (and also BP/RP or RVS spectra) can be downloaded without
the 5000 limit using Python. A tutorial with example Python scripts for
downloads can be found here: https://www.cosmos.esa.int/web/gaia-
users/archive/datalink-products#datalink_jntb_get_above_lim

##  Appendix C Accessing outlier results

The OA products are essentially contained in three different tables of the
archive:

  * •

oa_neuron_information table contains non-multidimensional parameters related
to the neurons, such as statistical descriptions for different Gaia products
($G$, $G_{\mathrm{BP}}$, $G_{\mathrm{RP}}$, $\varpi$,
$G_{\mathrm{BP}}-G_{\mathrm{RP}}$, etc.), and some quality measurements about
the clustering itself. Moreover, a class label is also provided for the best
quality neurons.

  * •

oa_neuron_xp_spectra table contains multidimensional data related to BP/RP
spectrophotometry of the neurons, so that the preprocessed spectra for both
prototypes (xp_spectrum_prototype_flux) and templates (if available,
xp_spectrum_template_flux) for each neuron can be retrieved.

  * •

astrophysical_parameters table contains information about the correspondence
between the sources and the neurons, among other results produced by many
different modules from Apsis. Regarding OA parameters, these are the
identification of the neuron within which a source lies (if it was processed
by OA, neuron_oa_id), distance between the source BP/RP spectra and the neuron
prototype (neuron_oa_dist, and neuron_oa_dist_percentile_rank). Additionally,
it also provides a processing flag (flags_oa).

The following examples can guide the interested users to access OA results in
the Gaia Archive:

  1. 1.

Retrieve the identifications of all the sources that belong to a certain
neuron. In this example the first neuron (located at $(0,0)$, and identified
by $\href
https://gea.esac.esa.int/archive/documentation/GDR3/Gaia_archive/chap_datamodel/sec_dm_astrophysical_parameter_tables/ssec_dm_oa_neuron_information.html#oa_neuron_information-
neuron_id=202105281205440000$) is used.

SELECT a.source_id FROM gaiadr3.oa_neuron_information n JOIN
gaiadr3.astrophysical_parameters a ON n.neuron_id = a.neuron_oa_id WHERE
n.neuron_id = 202105281205440000;

* 2.

Retrieve the identifications of all the sources that belong to a SOM neuron
that were assigned a specific class label. For this example all galaxy labels
were considered. Please, note that this type of query could potentially lead
to a huge amount of data being accessed.

Query: SELECT a.source_id FROM gaiadr3.oa_neuron_information n JOIN
gaiadr3.astrophysical_parameters a ON n.neuron_id = a.neuron_oa_id WHERE
n.class_label ILIKE ’%GAL%’;

* 3.

Obtain all the neurons that achieve a certain quality category threshold. In
this example, just high quality neurons are retrieved.

Query: SELECT * FROM gaiadr3.oa_neuron_information WHERE quality_category < 4;

##  Appendix D Selection function

The processing limits and filtering performed during post-processing can lead
in some cases to a complicated selection function, see also Fig. 10. We
summarise the filters that were imposed on the results for each module in the
Apsis pipeline in Section 11.4 of the online documentation.

##  Appendix E CU8 tools for the community

CU8 has made available a number of tools and datasets for the community to
accompany Gaia DR3 APs.

The datasets are made available on the Gaia cosmos webpage at
https://www.cosmos.esa.int/web/gaia/dr3-auxiliary-data. The first of these is
a set of parameter simulation
files303030https://www.cosmos.esa.int/web/gaia/dr3-astrophysical-parameter-
inference used in Apsis. These comprise the full set of parameters that
defines the BP/RP spectral simulations of stellar sources, along with
extinction in $G$, $G_{\mathrm{BP}}$, and $G_{\mathrm{RP}}$ passbands, and the
corresponding $V$ and $E(B-V)$, see Sections 4.1 and 4.2 for details. We also
provide the wavelength sampling
scheme313131https://www.cosmos.esa.int/web/gaia/dr3-aps-wavelength-sampling
that was used in Apsis processing.

Information on the tools provided by CU8 can be found on the Gaia DR3 tools
pages https://www.cosmos.esa.int/web/gaia/dr3-software-tools. These are the
following: \-- A visualisation
tool323232https://www.cosmos.esa.int/web/gaia/dr3-oa-self-organising-map-tool
GUASOM flavour DR3333333https://guasom.citic.udc.es has been developed to
analyse the outputs produced by OA (Álvarez et al. 2021), and it allows the
user to interactively explore the SOM maps through several visualisations. \--
The results from TGE have been implemented into the python dustmaps
package343434https://dustmaps.readthedocs.io, and allows one to retrive the
total Galactic extinction based on
coordinates353535https://www.cosmos.esa.int/web/gaia/dr3-extinction-as-
function-of-l-b. \-- A $G$-band bolometric correction
function363636https://www.cosmos.esa.int/web/gaia/dr3-bolometric-correction-
tool has been provided in
python373737https://gitlab.oca.eu/ordenovic/gaiadr3_bcg based on the method
that was used in FLAME. \-- A GSP-Phot metallicity
calibration383838https://www.cosmos.esa.int/web/gaia/dr3-gspphot-metallicity-
calibration tool has been made available in
python393939https://github.com/mpi-astronomy/gdr3apcal based on LAMOST
DR6404040http://dr6.lamost.org/v2/catalogue \-- A set of tools to compute
photometric extinction
coefficients414141https://www.cosmos.esa.int/web/gaia/dr3-extinction-
coefficients-in-various-passbands has been made available through the
dustapprox424242https://mfouesneau.github.io/dustapprox/ python package.

##  Appendix F DPAC Acknowledgements

The Gaia mission and data processing have financially been supported by, in
alphabetical order by country: \-- the Algerian Centre de Recherche en
Astronomie, Astrophysique et Géophysique of Bouzareah Observatory; \-- the
Austrian Fonds zur Förderung der wissenschaftlichen Forschung (FWF) Hertha
Firnberg Programme through grants T359, P20046, and P23737; \-- the BELgian
federal Science Policy Office (BELSPO) through various PROgramme de
Développement d’Expériences scientifiques (PRODEX) grants, the Research
Foundation Flanders (Fonds Wetenschappelijk Onderzoek) through grant
VS.091.16N, the Fonds de la Recherche Scientifique (FNRS), and the Research
Council of Katholieke Universiteit (KU) Leuven through grant C16/18/005
(Pushing AsteRoseismology to the next level with TESS, GaiA, and the Sloan
DIgital Sky SurvEy -- PARADISE); \-- the Brazil-France exchange programmes
Fundação de Amparo à Pesquisa do Estado de São Paulo (FAPESP) and Coordenação
de Aperfeicoamento de Pessoal de Nível Superior (CAPES) - Comité Français
d’Evaluation de la Coopération Universitaire et Scientifique avec le Brésil
(COFECUB); \-- the Chilean Agencia Nacional de Investigación y Desarrollo
(ANID) through Fondo Nacional de Desarrollo Científico y Tecnológico
(FONDECYT) Regular Project 1210992 (L. Chemin); \-- the National Natural
Science Foundation of China (NSFC) through grants 11573054, 11703065, and
12173069, the China Scholarship Council through grant 201806040200, and the
Natural Science Foundation of Shanghai through grant 21ZR1474100; \-- the
Tenure Track Pilot Programme of the Croatian Science Foundation and the École
Polytechnique Fédérale de Lausanne and the project TTP-2018-07-1171 ‘Mining
the Variable Sky’, with the funds of the Croatian-Swiss Research Programme;
\-- the Czech-Republic Ministry of Education, Youth, and Sports through grant
LG 15010 and INTER-EXCELLENCE grant LTAUSA18093, and the Czech Space Office
through ESA PECS contract 98058; \-- the Danish Ministry of Science; \-- the
Estonian Ministry of Education and Research through grant IUT40-1; \-- the
European Commission’s Sixth Framework Programme through the European
Leadership in Space Astrometry (ELSA) Marie Curie Research Training Network
(MRTN-CT-2006-033481), through Marie Curie project PIOF-GA-2009-255267 (Space
AsteroSeismology & RR Lyrae stars, SAS-RRL), and through a Marie Curie
Transfer-of-Knowledge (ToK) fellowship (MTKD-CT-2004-014188); the European
Commission’s Seventh Framework Programme through grant FP7-606740
(FP7-SPACE-2013-1) for the Gaia European Network for Improved data User
Services (GENIUS) and through grant 264895 for the Gaia Research for European
Astronomy Training (GREAT-ITN) network; \-- the European Cooperation in
Science and Technology (COST) through COST Action CA18104 ‘Revealing the Milky
Way with Gaia (MW-Gaia)’; \-- the European Research Council (ERC) through
grants 320360, 647208, and 834148 and through the European Union’s Horizon
2020 research and innovation and excellent science programmes through Marie
Skłodowska-Curie grant 745617 (Our Galaxy at full HD -- Gal-HD) and 895174
(The build-up and fate of self-gravitating systems in the Universe) as well as
grants 687378 (Small Bodies: Near and Far), 682115 (Using the Magellanic
Clouds to Understand the Interaction of Galaxies), 695099 (A sub-percent
distance scale from binaries and Cepheids -- CepBin), 716155 (Structured
ACCREtion Disks -- SACCRED), 951549 (Sub-percent calibration of the
extragalactic distance scale in the era of big surveys -- UniverScale), and
101004214 (Innovative Scientific Data Exploration and Exploitation
Applications for Space Sciences -- EXPLORE); \-- the European Science
Foundation (ESF), in the framework of the Gaia Research for European Astronomy
Training Research Network Programme (GREAT-ESF); \-- the European Space Agency
(ESA) in the framework of the Gaia project, through the Plan for European
Cooperating States (PECS) programme through contracts C98090 and
4000106398/12/NL/KML for Hungary, through contract 4000115263/15/NL/IB for
Germany, and through PROgramme de Développement d’Expériences scientifiques
(PRODEX) grant 4000127986 for Slovenia; \-- the Academy of Finland through
grants 299543, 307157, 325805, 328654, 336546, and 345115 and the Magnus
Ehrnrooth Foundation; \-- the French Centre National d’Études Spatiales
(CNES), the Agence Nationale de la Recherche (ANR) through grant
ANR-10-IDEX-0001-02 for the ‘Investissements d’avenir’ programme, through
grant ANR-15-CE31-0007 for project ‘Modelling the Milky Way in the Gaia era’
(MOD4Gaia), through grant ANR-14-CE33-0014-01 for project ‘The Milky Way disc
formation in the Gaia era’ (ARCHEOGAL), through grant ANR-15-CE31-0012-01 for
project ‘Unlocking the potential of Cepheids as primary distance calibrators’
(UnlockCepheids), through grant ANR-19-CE31-0017 for project ‘Secular
evolution of galxies’ (SEGAL), and through grant ANR-18-CE31-0006 for project
‘Galactic Dark Matter’ (GaDaMa), the Centre National de la Recherche
Scientifique (CNRS) and its SNO Gaia of the Institut des Sciences de l’Univers
(INSU), its Programmes Nationaux: Cosmologie et Galaxies (PNCG), Gravitation
Références Astronomie Métrologie (PNGRAM), Planétologie (PNP), Physique et
Chimie du Milieu Interstellaire (PCMI), and Physique Stellaire (PNPS), the
‘Action Fédératrice Gaia’ of the Observatoire de Paris, the Région de Franche-
Comté, the Institut National Polytechnique (INP) and the Institut National de
Physique nucléaire et de Physique des Particules (IN2P3) co-funded by CNES;
\-- the German Aerospace Agency (Deutsches Zentrum für Luft- und Raumfahrt
e.V., DLR) through grants 50QG0501, 50QG0601, 50QG0602, 50QG0701, 50QG0901,
50QG1001, 50QG1101, 50QG1401, 50QG1402, 50QG1403, 50QG1404, 50QG1904,
50QG2101, 50QG2102, and 50QG2202, and the Centre for Information Services and
High Performance Computing (ZIH) at the Technische Universität Dresden for
generous allocations of computer time; \-- the Hungarian Academy of Sciences
through the Lendület Programme grants LP2014-17 and LP2018-7 and the Hungarian
National Research, Development, and Innovation Office (NKFIH) through grant
KKP-137523 (‘SeismoLab’); \-- the Science Foundation Ireland (SFI) through a
Royal Society - SFI University Research Fellowship (M. Fraser); \-- the Israel
Ministry of Science and Technology through grant 3-18143 and the Tel Aviv
University Center for Artificial Intelligence and Data Science (TAD) through a
grant; \-- the Agenzia Spaziale Italiana (ASI) through contracts I/037/08/0,
I/058/10/0, 2014-025-R.0, 2014-025-R.1.2015, and 2018-24-HH.0 to the Italian
Istituto Nazionale di Astrofisica (INAF), contract 2014-049-R.0/1/2 to INAF
for the Space Science Data Centre (SSDC, formerly known as the ASI Science
Data Center, ASDC), contracts I/008/10/0, 2013/030/I.0, 2013-030-I.0.1-2015,
and 2016-17-I.0 to the Aerospace Logistics Technology Engineering Company
(ALTEC S.p.A.), INAF, and the Italian Ministry of Education, University, and
Research (Ministero dell’Istruzione, dell’Università e della Ricerca) through
the Premiale project ‘MIning The Cosmos Big Data and Innovative Italian
Technology for Frontier Astrophysics and Cosmology’ (MITiC); \-- the
Netherlands Organisation for Scientific Research (NWO) through grant
NWO-M-614.061.414, through a VICI grant (A. Helmi), and through a Spinoza
prize (A. Helmi), and the Netherlands Research School for Astronomy (NOVA);
\-- the Polish National Science Centre through HARMONIA grant
2018/30/M/ST9/00311 and DAINA grant 2017/27/L/ST9/03221 and the Ministry of
Science and Higher Education (MNiSW) through grant DIR/WK/2018/12; \-- the
Portuguese Fundação para a Ciência e a Tecnologia (FCT) through national
funds, grants SFRH/BD/128840/2017 and PTDC/FIS-AST/30389/2017, and work
contract DL 57/2016/CP1364/CT0006, the Fundo Europeu de Desenvolvimento
Regional (FEDER) through grant POCI-01-0145-FEDER-030389 and its Programa
Operacional Competitividade e Internacionalização (COMPETE2020) through grants
UIDB/04434/2020 and UIDP/04434/2020, and the Strategic Programme
UIDB/00099/2020 for the Centro de Astrofísica e Gravitação (CENTRA); \-- the
Slovenian Research Agency through grant P1-0188; \-- the Spanish Ministry of
Economy (MINECO/FEDER, UE), the Spanish Ministry of Science and Innovation
(MICIN), the Spanish Ministry of Education, Culture, and Sports, and the
Spanish Government through grants BES-2016-078499, BES-2017-083126,
BES-C-2017-0085, ESP2016-80079-C2-1-R, ESP2016-80079-C2-2-R, FPU16/03827,
PDC2021-121059-C22, RTI2018-095076-B-C22, and TIN2015-65316-P (‘Computación de
Altas Prestaciones VII’), the Juan de la Cierva Incorporación Programme
(FJCI-2015-2671 and IJC2019-04862-I for F. Anders), the Severo Ochoa Centre of
Excellence Programme (SEV2015-0493), and MICIN/AEI/10.13039/501100011033 (and
the European Union through European Regional Development Fund ‘A way of making
Europe’) through grant RTI2018-095076-B-C21, the Institute of Cosmos Sciences
University of Barcelona (ICCUB, Unidad de Excelencia ‘María de Maeztu’)
through grant CEX2019-000918-M, the University of Barcelona’s official
doctoral programme for the development of an R+D+i project through an Ajuts de
Personal Investigador en Formació (APIF) grant, the Spanish Virtual
Observatory through project AyA2017-84089, the Galician Regional Government,
Xunta de Galicia, through grants ED431B-2021/36, ED481A-2019/155, and
ED481A-2021/296, the Centro de Investigación en Tecnologías de la Información
y las Comunicaciones (CITIC), funded by the Xunta de Galicia and the European
Union (European Regional Development Fund -- Galicia 2014-2020 Programme),
through grant ED431G-2019/01, the Red Española de Supercomputación (RES)
computer resources at MareNostrum, the Barcelona Supercomputing Centre -
Centro Nacional de Supercomputación (BSC-CNS) through activities
AECT-2017-2-0002, AECT-2017-3-0006, AECT-2018-1-0017, AECT-2018-2-0013,
AECT-2018-3-0011, AECT-2019-1-0010, AECT-2019-2-0014, AECT-2019-3-0003,
AECT-2020-1-0004, and DATA-2020-1-0010, the Departament d’Innovació,
Universitats i Empresa de la Generalitat de Catalunya through grant
2014-SGR-1051 for project ‘Models de Programació i Entorns d’Execució
Parallels’ (MPEXPAR), and Ramon y Cajal Fellowship RYC2018-025968-I funded by
MICIN/AEI/10.13039/501100011033 and the European Science Foundation
(‘Investing in your future’); \-- the Swedish National Space Agency
(SNSA/Rymdstyrelsen); \-- the Swiss State Secretariat for Education, Research,
and Innovation through the Swiss Activités Nationales Complémentaires and the
Swiss National Science Foundation through an Eccellenza Professorial
Fellowship (award PCEFP2_194638 for R. Anderson); \-- the United Kingdom
Particle Physics and Astronomy Research Council (PPARC), the United Kingdom
Science and Technology Facilities Council (STFC), and the United Kingdom Space
Agency (UKSA) through the following grants to the University of Bristol, the
University of Cambridge, the University of Edinburgh, the University of
Leicester, the Mullard Space Sciences Laboratory of University College London,
and the United Kingdom Rutherford Appleton Laboratory (RAL): PP/D006511/1,
PP/D006546/1, PP/D006570/1, ST/I000852/1, ST/J005045/1, ST/K00056X/1,
ST/K000209/1, ST/K000756/1, ST/L006561/1, ST/N000595/1, ST/N000641/1,
ST/N000978/1, ST/N001117/1, ST/S000089/1, ST/S000976/1, ST/S000984/1,
ST/S001123/1, ST/S001948/1, ST/S001980/1, ST/S002103/1, ST/V000969/1,
ST/W002469/1, ST/W002493/1, ST/W002671/1, ST/W002809/1, and EP/V520342/1.

The GBOT programme uses observations collected at (i) the European
Organisation for Astronomical Research in the Southern Hemisphere (ESO) with
the VLT Survey Telescope (VST), under ESO programmes 092.B-0165, 093.B-0236,
094.B-0181, 095.B-0046, 096.B-0162, 097.B-0304, 098.B-0030, 099.B-0034,
0100.B-0131, 0101.B-0156, 0102.B-0174, and 0103.B-0165; and (ii) the Liverpool
Telescope, which is operated on the island of La Palma by Liverpool John
Moores University in the Spanish Observatorio del Roque de los Muchachos of
the Instituto de Astrofísica de Canarias with financial support from the
United Kingdom Science and Technology Facilities Council, and (iii) telescopes
of the Las Cumbres Observatory Global Telescope Network.


]Physics and Astronomy, Sejong University, Seoul, South Korea, and INAF-OAS
Bologna, via P. Gobetti, 101, I-40129 Bologna, Italy

]Cerebras, 1237 E. Arques Ave Sunnyvale, CA 94085

]Lawrence Berkeley National Laboratory, 1 Cyclotron Rd Berkeley, CA 94720

]Cerebras, 1237 E. Arques Ave Sunnyvale, CA 94085

# Slide FFT on a homogeneous mesh in wafer-scale computing

Maurice H.P.M. van Putten  [<EMAIL_ADDRESS>Leighton Wilson  [  Adam W.
Lavely  [  Mark Hair  [

###### Abstract

Searches for signals at low signal-to-noise ratios frequently involve the Fast
Fourier Transform (FFT). For high-throughput searches, we here consider FFT on
the homogeneous mesh of Processing Elements (PEs) of a wafer-scale engine
(WSE). To minimize memory overhead in the inherently non-local FFT algorithm,
we introduce a new synchronous slide operation (Slide) exploiting the fast
interconnect between adjacent PEs. Feasibility of compute-limited performance
is demonstrated in linear scaling of Slide execution times with varying array
size in preliminary benchmarks on the CS-2 WSE. The proposed implementation
appears opportune to accelerate and open the full discovery potential of FFT-
based signal processing in multi-messenger astronomy.

FFT, wafer-scale, memory overhead, efficiency

††preprint: APS/123-QED

##  I Introduction

The Fast Fourier Transform (FFT) is central to signal processing and high-
performance computing [9]. High-throughput signal analysis hereby critically
depends on FFT performance. The FFT algorithm optimizes efficiency in compute,
notably in the Cooley-Tukey algorithm [6] with $5n\log_{2}n$ floating point
operations (FLOPs) per transform of length $n$ [23]. This performance bears
out well in CPU-based implementations [e.g. 8, 18, 11, 19, 21, 13]. However,
real-time performance sensitively depends on memory overhead [16] which varies
with the architecture of the computing device.

Optimizing real-time signal processing has been approached on dedicated
hardware including field-programmable gate arrays (FPGAs) [e.g. 32] and
graphics processor units (GPUs) [e.g. 5]. These approaches can be advantageous
in low latency searches for signals at low signal-to-noise ratios (SNR) by
FFT-based matched filtering [22]. Filtering a continuous stream in real-time
requires a filter throughput faster than the input data rate. This is
particularly opportune in searches of un-modeled transient signals at
detector-limited sensitivity from astrophysical gravitational-wave sources in
the Local Universe [26, 27].

The theoretical limit of FFT compute performance is inevitably below
theoretical peak compute-performance due to memory overhead on any
architecture. This is not surprising since the Fourier transform is non-local.
While GPUs offer substantial acceleration over CPUs, performance drops
significantly when FFT-array sizes exceed the limits of Local Memory. In fact,
the resulting bandwidth-limited performance is significantly below theoretical
compute-performance (Fig. 1).

Figure 1: FFT performance on a GPU (Radeon VII with HBM2) by efficiency (left
scale) and throughput (right scale) in matched filtering, expressed by the
product of transform size $N=2^{m}$ and FFT transform rate. Results are
computed with clFFT [5] in complex single-precision and out-of-place as a
function of transform size $N$ in batch mode, shown across memory allocations
$M=64$ MB and $M=1024$MB in Global Memory. Efficiency is limited by memory
bandwidth especially when transform sizes exceed the size of Local Memory,
noticeably across $N=2^{12}$, leaving about 8% of theoretical peak compute
performance in f32. (After Fig. D.1 of [27].)

Wafer-Scale Engines (WSE) offer a radically new architecture for massively
parallel computing on a homogeneous mess of Processing Elements (PEs) with
fast on-chip interconnect [20]. This appears promising provided memory
overhead is kept low relative to compute [12]. The Cerebras WSE CS-2 presents
a novel architecture in a single chip consisting of 850,000 PEs (Fig. 4). CS-2
utilizes the maximum silicon square that can be cut from a 300 mm diameter
wafer: a square array of 84 dies over a surface area of 550 mm2. The die is
interconnected via a custom interconnect by on-chip routers, one per PE. It
presents a homogeneous 2D grid fabric of PEs, each connected to its nearest
neighbors North, South, East, and West. At 850 MHz, the 850,000 PEs of CS-2
offer a theoretical performance of 240 TFLOP/s at about 3 clock cycles per
FLOP. In FFT transform sizes limited to Local Memory, 3D FFT of size $512^{3}$
($\sim 3$ TFLOPs) is realized in $\sim$ 1 ms [15].

Here, we study memory overhead in FFT (§2) on a homogeneous mesh (§3) by
synchronous slide of data to facilitate transforms of arbitrary size (§4). The
potential of compute-limited performance is demonstrated on the CS-2 by linear
scaling of memory overhead with data-size (§5). We summarize our outlook in
(§6).

Figure 2: (Left panel.) Following index permutation of input data, FFT takes a
path in reverse over $m=\log_{2}n$ levels $p=m,m-1,\cdots,1$. At each $p$,
adjacent segments $E$, $O$ are concatenated following a rotation (2), amenable
to embarrassingly parallel computing of $n/2^{p}$ crossings. (Right panel.)
Crossing diagram of rotations (12) of adjacent segments showing (2) (wiggly
line), subsequent to alignment of $E$ and $O$ in shared memory, followed by
sliding output $L,R$ back as input to level $p-1$.

##  II Memory overhead in FFT

To revisit memory overhead in FFT, we consider an array of samples
$D=\left[d_{0},d_{1},\cdots,d_{n-1}\right]$ of size $n=2^{m}$. It starts with
a permutation of indices produced by sequential partitioning in segments of
size $n/2^{p-1}$ over levels $p=1,2,\cdots,m$. Following this pre-processing,
FFT is completed over a reverse path by concatenating adjacent elements
following crossings on levels $p=m,m-1,\cdots,1$ (further below).

To illustrate, index permutation is evaluated over $m=\log_{2}n$ steps
(Appendix A), e.g.,

$\displaystyle I=\begin{bmatrix}0&1&2&3&4&5&6&7\cr 0&2&4&6&1&3&5&7\cr
0&4&2&6&1&5&3&7\end{bmatrix}$ (1)

for an array of $n=8$ elements over $m=3$ levels. These permutations are
readily tabulated, and therefore are not included in evaluation of efficiency.

FFT proceeds by merging adjacent segments $Y=\left(E~{}O\right)^{T}$ of size
$n/2^{p-1}$ in step $p=m,m-1,\cdots,1$ of even ($E$) and odd ($O$) indexed
elements of the array (with indices permuted). Merging is a concatenation of
the transforms $Y^{\prime}=\left(L~{}R\right)$ by a sum and difference,

$\displaystyle L=E+O^{\prime},~{}~{}R=E-O^{\prime},$ (2)

equivalent to a rotation,

$\displaystyle Y^{\prime}=\sqrt{2}\left(\begin{array}[]{rr}0&1\\\
1&0\end{array}\right){\cal R}_{\pi/4}\left(\begin{array}[]{cc}1&0\\\
0&U_{p}\end{array}\right)Y,$ (7)

where

$\displaystyle{\cal
R}_{\pi/4}=\frac{1}{\sqrt{2}}\left(\begin{array}[]{cr}1&1\\\
1&-1\end{array}\right).$ (10)

Here, the trigonometric part in FFT at level $p$ is in the diagonal matrix

$\displaystyle U_{p}={\rm
dia}\left(1,e^{-i\frac{2\pi}{N}},\cdots,e^{-\frac{2\pi}{N}\left(N/2-1\right)}\right)$
(11)

of size $N=2^{m-p+1}$, representing $N/2$ samples on the semicircle of $S^{1}$
in the lower half-plane with determinant $\left|U_{p}\right|=1$ $(p=1)$ and
$\left|U_{p}\right|=i$ $(1<p\leq m)$.

Figure 2 schematically shows

$\displaystyle E\times O\rightarrow RL,$ (12)

aligning $E$ and $O$ to shared memory for compute (7) and concatenation of
output $R,L$. The transform (12) appears one or multiple times in each level
$p$ in mutually independent operations, amenable to embarrassingly parallel
computing at each transform level (Fig. 2, Appendix A). This is ideally suited
for a uniform mesh of PEs provided memory overhead in alignment is negligible
(Fig. 3).

Figure 3:  Memory overhead in aligning $E$ and $O$ in (2) at level $p$. From
top to bottom: coherent shifts to a shared memory location, $E\times
O\rightarrow LR$, followed by coherent shifts back to the original location of
$E,O$. The concatenated segment (white) is $E$ or $O$ at level $p-1$.

##  III Performance in Slide FFT

When data are distributed over a linear array of PEs, Eq. (12) is non-local
and incurs data-transfers between the Local Memory of neighboring PEs. In this
event, (2) triggers memory overhead in aligning $E$ and $O$ and in
concatenation by sliding:

  * •

Forward: aligning $E$ and $O$ for compute in shared memory location;

  * •

Backward: updating the original memory locations of $E$ and $O$ with output
$L$ and $R$, respectively.

It appears natural to slide $E$ and $O$ to a shared memory location half-way
in between (shown in Fig. 3), supporting sufficient allocation for $E,O,L$ and
$R$ following compute in the rotation Eq. (2).

Sliding data over a mesh incurs memory overhead that scales with $n/2^{p-1}$
in sliding data. Sliding time depends on the velocity of propagation
determined by bandwidth between adjacent processors. On the CS-2, this incurs
a minor latency in sending data up or down a PE’s ramp connecting its CE to
its router which communicates with neighboring PEs. Cumulatively over each
transform level $p=m,m-1,\cdots,1$, communication overhead is $n/2+n/4+\cdots
1\simeq n$, and double this amount including the sliding of $L,R$ and back to
the original locations of $E$ and, respectively, $O$.

Given raw compute performance for multiply/add, compute efficiency subject to
memory overhead in data transfer across neighboring PEs depends on the number
of clock cycles $a$ per transfer and the associated compute,

$\displaystyle\alpha=\frac{\\#{\rm clock\,cycles\,per\,transfer}}{\\#{\rm
clock\,cycles\,per\,FLOP}}=\frac{a}{b},$ (13)

where $a$ and $b$ are per datum. Given aforementioned $5n\log_{2}n=5nm$ FLOPs
per FFT, compute efficiency of Slide FFT satisfies

$\displaystyle{\cal\eta}=\frac{5bmn}{an+5bmn}\simeq
1-\frac{\alpha}{5\,m}+{\cal
O}\left(\left(\frac{\alpha}{5\,m}\right)^{2}\right).$ (14)

In Slide FFT extending over $2^{k}$ PEs - a wave of length $k$ on the WSE -
the time to align $E$ and $O$ segments (Fig. 3) scales linearly with the
number of array elements stored at each processor, e.g., $2^{m-k}$. For
processors with one thread compute time handling Eq. (2) scales with the same.
While total processing time scales inversely with $2^{k}$, efficiency Eq. (13)
is invariant under the choice of $k$. By (14), it follows that efficiency of
Slide FFT satisfies

$\displaystyle\frac{\alpha}{5\,m}\ll 1.$ (15)

##  IV The Slide operation

On a WSE, Eq. (2) is realized at Eq. (15) by standard operations of floating
point multiply/add on data local to each PE and one radically new operation
when work space exceeds Local Memory. Facilitating array sizes in excess of
Local Memory, we propose sliding data arrays over a homogeneous mesh of PEs.

Let ${\bf p}$ index the PE positions on a rectangular grid of a WSE. We define
a Slide of data $A$ at ${\bf p}_{0}={\&}A$ to $B$ at ${\bf p}_{1}={\&}B$,

$\displaystyle B={{*}}{\rm{\bf Slide}}_{{\bf p}}({\&}A),$ (16)

by translation over

$\displaystyle{\bf p}={\bf p}_{1}-{\bf p}_{0}.$ (17)

The memory overhead Eq. (16) is expected to satisfy linear scaling with size
of $A$, distributed over Local Memory of the PEs in a choice of wave length
covering the desired FFT transform size. This overhead represents a waiting
time prior and subsequent to compute in Eq. (12), Ideally, this is overhead is
invariant of the choice of wave length $k$, provided it contains sufficient
storage in Local Memory aggregated over $k$ PEs. If so, this leaves Eq. (13)
invariant with respect to $k$.

##  V Slides on the CS-2

Figure 4: Architecture of the wafer-scale engine CS-2 comprising a homogeneous
grid of 850,000 PEs (left). Each PE consists of a Compute Element (CE)
supported by 48 kB of SRAM Local Memory. Local Memory is connected over a bus
to a router enabling bi-directional communication to its nearest neighbors
(right).

On the CS-2, PE performance per clock cycle is as follows:

  * •

Compute: fused multiply-accomulate (FMAC) in eight 16-bit or two 32-bit FMAC
operations by the CE;

  * •

Local Memory: a 128-bit read and a 64-bit write to 48 kB Local Memory (SRAM);

  * •

Router bandwidth: 32-bit packets (bi-directional) between its nearest
neighbors.

Additionally, there is a modest latency in communicating between PEs in data-
transfer between a CE and its router, i.e., about 2-7 cycles per a 32-bit
packet transfer. Taken together, the above represents a few clock cycles for a
floating point operation (FLOP).

Figure 5:  Performance of Slide Eq. (16) in cycles per array element versus
total number of elements in individual benchmarks on the CS-2 (left top)
versus simulated in the Cerebras SDK (right top). Data are distributed evenly
over 8, 16, and 32 PEs, the number of elements per PE varying from 1 to 500.
(Lower panel.) Results for maximal data arrays over $4k$ PEs
($k=2,3,\cdots,8)$ shows performance to be essentially invariant to the number
of PEs used by regular convergence to about 1.3 cycles per data element.

On the CS-2, Eq. (16) is implemented in the Cerebras Software Language (CSL)
by the Cerebras Software Development Kit (SDK 1.0.0), enabling developers to
write custom kernels for Cerebras systems. CSL provides direct access to low-
level hardware features while providing higher-level constructs such as loops
and functions. The SDK provides a CS-2 simulator that allows software to be
debugged and optimized without the physical wafer and provides cycle-accurate
values for performance evaluation.

Fig. 5 shows performance of our Slide operation Eq. (16) in the fabric
simulator and on a real CS-2 system. Realized is the desired linear scaling of
slide-time with data-size in bytes and number of number of PEs involved.

Here, $\alpha=2/3$ given $a=2$ for a 64 bit datum in complex single precision,
while typically $b=3$ including communication with Local Memory. Thus, (15) is
satisfied by an ample margin for typical FFT transform sizes.

##  VI Conclusions

The homogeneous mesh of PEs in WSEs offers a radically new architecture for
low latency high-throughput signal processing. Deep searches for signals at
small signal-to-noise ratios is frequently pursued by matched filtering
evaluated in the Fourier domain by FFT.

Seeking to exploit WSE, we propose compute-limited FFT by a new Slide
operation Eq. 16 facilitating arbitrary transform sizes. Performance Eq. (14)
obtains by exploiting fast on-chip interconnect in Eqs. (14-15).

Sliding data is required when transform sizes exceed the limitations of Local
Memory per compute unit - the PE of a WSE. This approach circumvents the need
for calls to Global Memory otherwise common in GPUs (Fig. 1), illustrative for
the combined challenge of compute and memory overhead in seeking high
throughput FFT-based signal processing on architectures different from CPUs.

Preliminary benchmarks demonstrate the desired linear scaling in Slide times
with array size (Fig. 4), essentially invariant to wave length - the number of
PEs involved in the transform size. The proposed Slide operation hereby seems
to provide a suitable starting point to ameliorate memory overhead in FFT on a
homogeneous mesh architecture such as CS-2.

To exploit the full compute potential of CS-2 (240 TFLOPs), preserving the
presented performance (Fig. 5) to arbitrary transform sizes extended to batch
mode further calls for limiting output back to the host device by post-
callback functions, originally developed for implementation on GPUs [25]. With
such additional steps in place to control memory overhead over the external
bus, WSEs such as CS-2 hold realistic promise to accelerate the performance of
high-throughput signal analysis by over an order of magnitude over
conventional approaches.

This outlook is of particular interest to searches for un-modeled signals by
the FFT-based butterfly matched filtering algorithm [24], nominally over 32 s
segments of LIGO-Virgo-KAGRA data comprising $n=2^{17}$ samples [25]. This
approach recently identified the physical trigger GW170817B - the central
engine - of GRB170817A [26, 27, 28, 1, 29].

Supplementary Data. The CSL code of the Cerebras SDK generating the data of
Fig. 5 is available on GitHub https://github.com/leightonw-cerebras/sliding-
window.

###### Acknowledgements.

The authors gratefully acknowledge stimulating discussions with M.A.
Abchouyeh.

## References

  * Abchouyeh et al. [2023] Abchouyet, M.A., van Putten, M.H.P.M., Amati, Lorenzo, 2023, ApJ, 952, 157
  * Abernathy [2011] Abernathy, M., et al. (ET Science Team), 2011, Tech. Rep. ET–0106C–10
  * Abbott et al. [2021] Abbott, R., et al. (LIGO Scientific, Virgo), 2021, ApJ, 913, L7
  * Akutsu et al. [2019] T. Akutsu et al. (KAGRA), 2019, Nature Astron., 3, 35
  * Bragadeesh [2015] Bragadeesh, N., 2015, https://github.com/ clMathLibraries/clFFT
  * Cooley&Tukey [1965] Cooley, J.W., Tukey, J.W., 1965, Math. Comput. 19, 297
  * Evans et al. [2021] Evans, M., 2021, CE Techn. Rep. CE–P2100003–v7, https://arxiv.org/abs/2109.09882
  * Frigo & Johnson [2005] Frigo, M., & Johnson, S.G., 2005, in Proc. IEEE 93, 2, 216; https://doi.org/10.1109/JPROC.2004.840301
  * Garrido et al. [2009] Garrido, M., Parhi, K.K., Grajal, J., 2009, IEEE Trans. Circ. Syst. I: Regul. Pap. 56, 2634
  * Iglesias et al. [2015] Iglesias, V., Grajal, J., Sanchez, M.A., López-Vallejo, M., 2015, IEEE Trans. Instrum. Meas. 64, 338
  * Lee et al. [2013] Lee, M., Malaya, N., Moser, R.D., 2013, in SC’13: Proc. Int. Conf. HPC, Networking, Storage and Analysis, IEEE, 1
  * Li & Dong [2010] Li, P., & Dong, W., 2010, 3rd Int. Symp. Parallel Architectures, Algorithms and Programming, pp. 369
  * Lu et al. [2021] Lu, T., Chen, Y.-F., Hechtman, B., Wang, T., & Anderson, J., 2021, IEEE Access, 9, 93422
  * Maggiore et al. [2020] Maggiore, M., et al., JCAP, 3, 50
  * Orenes-Vera et al. [2023] Orenes-Vera, M., Sharapov, I., Schreiber, R., et al. 2023, $in$ Proc. 37th Int. Conf. Supercomp. (ICS) 2023, 180 https://doi.org/10.1145/3577193.3593708
  * Jeong et al. [2018] Jeong, H., Low, T.M., and P. Grover, P., 2018, 56th Ann. Allerton Conf. Commununication, Control, and Computing (Allerton), pp. 887
  * Joshi [2015] Joshi, S.M., 2015, Int. J. Comput. Appl. 116, 1
  * Pekurovsky [2012] Pekurovsky, D., 2012, SIAM J. Sc. Comp. 34, 4, C192
  * Plimpton [2018] Plimpton, S., 2018, fftMPI, a distributed-memory parallel FFT library; https://lammps.github.io/fftmpi/
  * Rocki et al. [2020] Rocki, K., et al., 2020, arXiv:2010.03660v1
  * Takahashi et al. [2003] Takahashi, D., Sato, M., & Boku, T., 2003, in Int. Workshop OpenMP Appl. and Tools (Springer-Verlag), 99
  * Turin [1960] Turin, G. L., 1960, An introduction to matched filters, IRE T Inform. Theory., 6, 311
  * van Loan [1992] Van Loan, C., 1992, Computational frameworks for the fast Fourier transform, Frontiers in Appl. Math., SIAM
  * van Putten et al. [2014] van Putten, M.H.P.M., Guidorzi, C., & Frontera, F., 2014, ApJ, 786, 146
  * van Putten [2017] van Putten, M.H.P.M., 2017, PTEP, 093F01
  * van Putten et al. [2019] van Putten, M.H.P.M., & Levinson, A., Frontera, F., Guidorzi, C., Amati, L., & Della Valle, M., 2019, EPJ Plus, 134, 547
  * van Putten & Della Valle [2023] van Putten, M.H.P.M., & Della Valle, M., 2023, A&A, 669, A37
  * van Putten [2023] van Putten, M.H.P.M., 2023, Universe, 9, 279
  * van Putten [2023] van Putten, M.H.P.M., 2023, 32nd Texas Symp. Rel. Astroph., https://indico-tdli.sjtu.edu.cn/event/1582/, https://zenodo.org/records/10393716
  * Wang et al. [2015] Wang, Z., Liu, X., He, B., Yu, F., 2015, IEEE Trans. Very Large Scale Integr. (VLSI) Syst. 23, 973
  * Wong et al. [2017] S. Wong, Beck, K. Bertels, and L. Carro (Eds.), Applied Reconfigurable Computing (LNCS 10216), Proc. 13th Int. Symp., ARC 2017
  * Woods et al. [2017] Woods, R., McAllister, J., Lightbody & Yi, Y., FPGA-based Implementation of Signal Processing Systems (Wiley 2018)

##  Appendix A Emulation of FFT

FFT is initialized by a permutation (Fig. 6) over levels $p=1,2,\cdots,m$
partitioned by segments of size $n/2^{p-1}$ into adjacent $I^{\prime}$ and
$I^{\prime\prime}$ with even $(E)$ and, respectively, odd $(O)$ indices of
$I$:

$\displaystyle{\small\begin{array}[]{l}I(1,:)=0:n-1\\\
I(2,:)=[I^{\prime}(1,:)\,\,I^{\prime\prime}(1,:)]\\\ \mbox{{\bf
while}}\,m>2\\\ ~{}~{}m=m/2\\\ ~{}~{}N=n/m\\\ ~{}~{}\mbox{{\bf for}}\,v=1:N\\\
~{}~{}~{}~{}M=(1+m(v-1)):mv\\\
~{}~{}~{}~{}I(k+1,M)=[I^{\prime}(k,M))\,\,I^{\prime\prime}(k,M)]\\\
~{}~{}\mbox{{\bf end}}\\\ ~{}~{}k=k+1\\\ \mbox{\bf end}\\\ \mbox{clear}\,J\\\
\mbox{\bf for}\,p=1:m\\\ ~{}~{}\mbox{\bf for}\,i=0:2^{m}-1\\\
~{}~{}~{}~{}s=(I(p,:)==i)\\\ ~{}~{}~{}~{}[s,a]=\mbox{\bf
sort}(s,\mbox{\bf'descend'})\\\ ~{}~{}~{}~{}J[(p,i+1)=a(1)-1\\\
~{}~{}\mbox{\bf end}\\\ \mbox{\bf end}\end{array}}$ (37)

Subsequently, FFT is evaluated by crossings (2) over $m$ levels (2-7),
illustrated in Eq. 1 for $m=3$ levels. The potential of embarrassingly
parallel computing follows from the following:

$\displaystyle{\small\begin{array}[]{ll}{\bf Index\,level}\,m\\\
~{}~{}I=J(m,:)\\\ {\bf Data\,indexed\,for\,level}\,m\\\
~{}~{}Y=J(1,I+1)^{T}\\\ N=2\\\ {\bf for}\,p=m:-1:1\\\ ~{}~{}K=(0:N/2-1)^{T}\\\
~{}~{}Up={\bf exp}\left(-(2\pi/N)\sqrt{-1}K\right)\\\
\begin{array}[]{|l}~{}~{}{\bf for}\,v=1:n/N\\\ ~{}~{}~{}~{}(M_{1},M_{2})\,{\bf
index\,adjacent\,}E,O\\\ ~{}~{}~{}~{}m_{1}=1+(v-1)N;m_{2}=m_{1}+N/2-1\\\
~{}~{}~{}~{}M_{1}=m_{1}:m_{2};M_{2}=M_{1}+N/2\\\ ~{}~{}~{}~{}{\bf
Fetch\,adjacent\,}E,O\\\ ~{}~{}~{}~{}E=Y(M_{1})\\\ ~{}~{}~{}~{}O=Y(M_{2})\\\
~{}~{}~{}~{}Op=Up.*O\\\ ~{}~{}~{}~{}{\bf Rotate\,}E,O^{\prime}\\\
~{}~{}~{}~{}L=E+Op\\\ ~{}~{}~{}~{}R=E-Op\\\ ~{}~{}~{}~{}{\bf
Update\,adjacent\,}L,R\\\ ~{}~{}~{}~{}Y(M_{1})=L\\\ ~{}~{}~{}~{}Y(M_{2})=R\\\
~{}~{}{\bf end}\end{array}\\\ ~{}~{}N=2N\\\ {\bf end}\end{array}}$ (64)

In (64), the loop $v=1:n/N$ (left bar) at levels $p$ is amenable to parallel
processing on a homogeneous mesh of PEs.

Figure 6: Array permutation of size $n=2^{m}$ ($m=4$ shown) over a
partitioning of segments of size $n/2^{p-1}$ in even ($E$, blue) and odd ($O$,
red) indexed elements, iterated over levels $p=1,2,\cdots m$ ($m=4$ shown).