# Multi-Class Deep SVDD: Anomaly Detection Approach in Astronomy with Distinct
Inlier Categories

M. Pérez-Carrasco  G. Cabrera-Vives  L. Hernández-García  F. Förster  P.
Sánchez-Sáez  A. Muñoz Arancibia  N. Astorga  F. E. Bauer  A. Bayo  M. Cádiz-
Leyton  M. Catelan  P. A. Estévez

###### Abstract

With the increasing volume of astronomical data generated by modern survey
telescopes, automated pipelines and machine learning techniques have become
crucial for analyzing and extracting knowledge from these datasets. Anomaly
detection, i.e. the task of identifying irregular or unexpected patterns in
the data, is a complex challenge in astronomy. In this paper, we propose
Multi-Class Deep Support Vector Data Description (MCDSVDD), an extension of
the state-of-the-art anomaly detection algorithm One-Class Deep SVDD,
specifically designed to handle different inlier categories with distinct data
distributions. MCDSVDD uses a neural network to map the data into
hyperspheres, where each hypersphere represents a specific inlier category.
The distance of each sample from the centers of these hyperspheres determines
the anomaly score. We evaluate the effectiveness of MCDSVDD by comparing its
performance with several anomaly detection algorithms on a large dataset of
astronomical light-curves obtained from the Zwicky Transient Facility. Our
results demonstrate the efficacy of MCDSVDD in detecting anomalous sources
while leveraging the presence of different inlier categories. The code and the
data needed to reproduce our results are publicly available at
https://github.com/mperezcarrasco/AnomalyALeRCE.

Machine Learning, Anomaly Detection, Astronomy

##  1 Introduction

With modern survey telescopes producing unprecedented volumes of data, it has
become unfeasible to analyze astronomical data massively by human inspection.
Consequently, the necessity for automated pipelines arises, enabling the
extraction of knowledge from these vast datasets in a data-driven manner.
Among the intriguing and complex astronomical challenges lies the task of
anomaly detection (AD), which refers to the identification of irregular or
unexpected patterns that deviate from our existing understanding of the data
(Chandola et al., 2009).

In recent years, the application of machine learning techniques in the field
of astronomy has led to remarkable advancements in detecting anomalous sources
in a systematic manner. For instance, Xiong et al. 2010 employed a
hierarchical probabilistic model, while Baron & Poznanski 2016 utilized an
unsupervised Random Forest approach to identify outliers within galaxy spectra
from the Sloan Digital Sky Survey (SDSS). More recently, Villar et al. 2020
applied an IForest algorithm to a latent space derived from a Variational
Recurrent Autoencoder (VRAE), specifically targeting a simulated dataset of
supernovae (SNe). Similarly, Sánchez-Sáez et al. 2021a employed a VRAE
architecture to analyze the light curves of active galactic nuclei (AGN)
within the Zwicky Transient Facility Data Release 5 (ZTF DR5). Furthermore,
the work of Ishida et al. 2021 employed Gaussian Processes to extract features
and train Isolation Forest algorithm in an active learning manner using the
Open Supernova Catalog dataset and the Photometric LSST Astronomical Time-
series Classification Challenge (PLAsTiCC; Kessler et al. 2019; Hložek et al.
2020). In a similar vein, Muthukrishna et al. 2022 used a bayesian parametric
model to find anomalous sources using simulated light curves from SNANA
(Kessler et al., 2009). Previous works share a common training procedure: they
utilize a sample of normal and well-identified objects categorized as inliers
to train an algorithm that can detect samples deviating from these inliers.
This approach, commonly known as one-class anomaly detection (Schölkopf et
al., 1999), represents the current state-of-the-art for various anomaly
detection problems. However, when different inlier categories with distinct
data distributions are present in the training set, it remains unclear how to
effectively utilize and leverage these methods.

To address this challenge, we present Multi-Class Deep Support Vector Data
Description (MCDSVDD), an extension of the state-of-the-art anomaly detection
algorithm One-Class Deep SVDD (Ruff et al., 2018). Our proposed method employs
a neural network to map the data into hyperspheres, where each hypersphere
encapsulates the data representation of a specific inlier category.
Consequently, the ”weirdness” of each sample is determined by its distance
from the center of the closest hypersphere. To evaluate the effectiveness of
our approach, we compare the performance of several one-class anomaly
detection algorithms on a large dataset of astronomical light-curves obtained
from the Zwicky Transient Facility (ZTF; Bellm et al. 2018). Through our
comprehensive evaluation, we demonstrate the efficacy of MCDSVDD in detecting
anomalous sources while effectively leveraging the presence of different
inlier categories.

##  2 Data

In this work we use data alerts from the from the ZTF data stream (Bellm et
al., 2018). In this data stream, an alert is triggered by an object in the sky
whose current (science) image has a significant difference with respect to a
template (reference) image (Masci et al., 2018). For alerts to be streamed by
ZTF, they need to pass the cut-off criteria defined by the real/bogus
detection system designed by the ZTF Collaboration. These criteria include
signal-to-noise ratios, near-edge image positioning, negative and bad pixels,
and morphological and photometric features (Mahabal et al., 2019; Duev et al.,
2019).

Light-curves are constructed using the same procedure of Section 4.4 of
Förster et al. (2021). We perform crossmatch with the AllWISE111The AllWISE
Data Release can be found at
http://wise2.ipac.caltech.edu/docs/release/allwise/ public source catalog
(Wright et al., 2010; Mainzer et al., 2011), using a matching radius of 2
arcseconds, obtaining W1, W2, and W3 photometry. Then, 152 features are
calculated for each light-curve using the ALeRCE broker feature extractor
(Förster et al., 2021). These features include detection and non-detection
features (see Section 3 of Sánchez-Sáez et al. 2021b for details). We only use
objects with at least six detections in either $g$ or $r$ bands.

Our training set consisted in labeled light-curves that follow a taxonomy with
three main classes (hereafter, the top level classes): transient, stochastic,
and periodic. Each of these categories is then subdivided into the following
subclasses:

  * •

Transient: Type Ia supernova (SNIa), Type Ibc supernova (SNIbc), Type II
supernova (SNII), and super-luminous supernova (SLSN);

  * •

Stochastic: Type 1 Seyfert galaxy (AGN; i.e., host-dominated active galactic
nuclei), Type 1 quasar (QSO; i.e., nucleus-dominated AGN), blazar (blazar;
i.e, beamed jet dominated AGN), young stellar object (YSO; including bursters,
dippers and purely rotation modulated lightcurves of pre-main sequence stars),
and cataclysmic variable/Nova (CV/Nova);

  * •

Periodic: long-period variable (LPV; includes regular, semi-regular, and
irregular variable stars), RR Lyrae (RRL), Cepheid (CEP), eclipsing binary
(E), and $\delta$ Scuti (DSCT).

In Section 4 we explain in detail how the above-mentioned categories are used
to train and evaluate our anomaly detection algorithms by hiding the light
curves from a given subclass and considering all the others as inliers. This
allows for a rigorous comparison of algorithms that helps to select the most
promising method to be applied in a real-world scenario.

##  3 Method

In this work we present Multi-Class Deep SVDD (MCSVDD), an extension of the
One-Class Deep SVDD method (Ruff et al., 2018).

###  3.1 One-Class Deep SVDD

Deep Support Vector Data Description (Deep SVDD; Ruff et al., 2018) is a
neural network-based approach related to OCSVM (Schölkopf et al., 1999). The
general idea is to map the data from an input space
$\mathcal{X}\subseteq\mathbb{R}^{d}$ into a new feature space
$\mathcal{F}\subseteq\mathbb{R}^{p}$ using a neural network
$\phi(\cdot;\Theta):\mathcal{X}\rightarrow\mathcal{F}$ with $L$ hidden layers
and parameters $\Theta=\\{\theta^{1},...,\theta^{L}\\}$. Specifically, we want
the neural network $\phi$ to learn a representation that minimizes the volume
of a hypersphere of radius $R>0$ and center $\mathbf{c}$ that enclose the
normal data in the output space $\mathcal{F}$.

The objective function for this task is defined by:

$\min_{\Theta,R}{R^{2}+\frac{1}{N}\sum_{i=1}^{N}\max\\{0,||\phi(\mathbf{x}_{i};\Theta)-\mathbf{c}||^{2}}-R^{2}\\}+\frac{\lambda}{2}\mathbf{\Theta}^{\top}\mathbf{\Theta},$
(1)

where $\lambda$ is a hyperparameter that controls the weight decay on the
network parameters $\theta$. In practice, the encoding part of an autoencoder
can be used to map the data into the latent space and define the center
$\mathbf{c}$. The parameters $\theta$ and $R$ are alternately and iteratively
optimized via gradient descent and line-search respectively.

In cases when training data contains only normal samples, one can simply
penalize the squared distance of all the data points with respect to a center
$\mathbf{c}$ as follows (Ruff et al., 2018, 2019):

$\min_{\Theta}{\frac{1}{N}\sum_{i=1}^{N}\max\||\phi(\mathbf{x}_{i};\Theta)-\mathbf{c}||^{2}}+\frac{\lambda}{2}\mathbf{\Theta}^{\top}\mathbf{\Theta}.$
(2)

This loss function will be used for comparisons in Section 5.

###  3.2 Multi-Class Deep SVDD

We extend the One-Class Deep SVDD to take into account the different
categories presented in our taxonomy. Instead of modeling a single hypersphere
to enclose the normal data, MCSVDD models multiple hyperspheres, where each
one corresponds to a given class and enclose normal data for that class. The
idea is to learn a neural network that maps objects from the same class close
to each other and far from objects from different classes. As abnormal samples
come from unseen classes, their distances to each hypersphere should be larger
than those of normal datapoints. We named this method Multi-Class Deep SVDD
(MCSVDD) as we now have multiple inlier classes.

Assuming normal data pairs coming from $M$ different classes
$y\in\\{1,...,M\\}$, the objective function for a training set composed by
different inlier classes is given by:

$\min_{\mathbf{\Theta}}{\sum_{j=1}^{M}\frac{1}{N_{j}}\sum_{i=1}^{N}\mathbbm{1}_{(y_{i}=j)}||\phi(\mathbf{x}_{i};\Theta)-\mathbf{c}_{j}||^{2}}+\frac{\lambda}{2}\mathbf{\Theta}^{\top}\mathbf{{\Theta}},$
(3)

where $\lambda$ is a hyperparameter that controls the weight decay regularizer
on the network parameters $\mathbf{\theta}$, $\mathbbm{1}_{(y_{i}=j)}$ is an
indicator function that becomes $1$ if $y_{i}=j$ and $0$ otherwise, and
$N_{j}$ is the number of data points that belong to class $j$.

By following this approach, it is possible to define an anomaly score
$A(\cdot)$ based on the distance of the data points to the centers of the
hyperspheres as follows:

$A(x_{i})=\min_{j}\\{||\phi(\mathbf{x}_{i};\mathbf{\Theta^{*}})-\mathbf{c}_{j}||^{2}\\}.$
(4)

where $\Theta^{*}$ are the parameters of the trained neural network.

##  4 Methodology

For fair comparisons, we follow the same training and evaluation procedure for
all our experiments. From the ZTF light-curves, we extracted 152 features as
described in Sec. 2. All the features were normalized between -1 and 1 using
the quantile normalization method. We randomly split the data into a training
set ($80\%$) and a test set ($20\%$) in a stratified fashion in order to
preserve the proportion of samples per class. The training set is divided into
five stratified subsets in order to perform 5-fold cross-validation for model
selection. Fig. 1 shows a scheme of our training and evaluation methodology.

Figure 1: Methodology for training and evaluation of the anomaly detection
algorithms. We split the data into a training set and test set, composed by
$80\%$ and $20\%$ of the data, respectively. The training set is subdivided
into transient, stochastic, and periodic data. For each of these classes, we
choose each subclass as the outlier class. The outlier class is removed from
the training set and added to the test set (TS2). Then, an anomaly detection
algorithm is trained using the remaining objects of each of the classes, and
is evaluated using TS2.

| Transient | Stochastic | Periodic
---|---|---|---
Method | SLSN | SNII | SNIa | SNIbc | AGN | Blazar | CV/Nova | QSO | YSO | CEP | DSCT | E | RRL | LPV
IForest | $0.640$ | $0.721$ | $0.428$ | $0.490$ | $0.573$ | $0.710$ | $0.975$ | $0.468$ | $0.913$ | $0.359$ | $0.295$ | $0.469$ | $0.549$ | $0.971$
(Liu et al., 2012) | $\pm 0.014$ | $\pm 0.021$ | $\pm 0.032$ | $\pm 0.038$ | $\pm 0.017$ | $\pm 0.009$ | $\pm 0.001$ | $\pm 0.016$ | $\pm 0.003$ | $\pm 0.007$ | $\pm 0.012$ | $\pm 0.021$ | $\pm 0.033$ | $\pm 0.007$
OCSVM | $0.577$ | $0.587$ | $0.434$ | $0.492$ | $0.532$ | $0.443$ | $0.909$ | $\mathbf{0.517}$ | $0.792$ | $0.432$ | $0.557$ | $0.555$ | $0.539$ | $0.943$
(Schölkopf et al., 1999) | $\pm 0.014$ | $\pm 0.014$ | $\pm 0.021$ | $\pm 0.011$ | $\pm 0.008$ | $\pm 0.002$ | $\pm 0.001$ | $\mathbf{\pm 0.005}$ | $\pm 0.005$ | $\pm 0.004$ | $\pm 0.005$ | $\pm 0.003$ | $\pm 0.004$ | $\pm 0.001$
AE | $\mathbf{0.736}$ | $0.807$ | $0.438$ | $0.537$ | $\mathbf{0.701}$ | $\mathbf{0.762}$ | $\mathbf{0.980}$ | $0.443$ | $\mathbf{0.990}$ | $0.564$ | $0.367$ | $0.864$ | $0.907$ | $\mathbf{0.996}$
(Rumelhart & McClelland, 1987) | $\mathbf{\pm 0.022}$ | $\pm 0.021$ | $\pm 0.015$ | $\pm 0.019$ | $\mathbf{\pm 0.010}$ | $\mathbf{\pm 0.006}$ | $\mathbf{\pm 0.016}$ | $\pm 0.004$ | $\mathbf{\pm 0.001}$ | $\pm 0.024$ | $\pm 0.015$ | $\pm 0.009$ | $\pm 0.015$ | $\mathbf{\pm 0.000}$
VAE | $0.669$ | $0.690$ | $0.404$ | $0.522$ | $0.596$ | $0.597$ | $0.849$ | $0.500$ | $0.795$ | $0.442$ | $0.417$ | $0.561$ | $0.451$ | $0.936$
(Kingma & Welling, 2014) | $\pm 0.015$ | $\pm 0.023$ | $\pm 0.018$ | $\pm 0.025$ | $\pm 0.007$ | $\pm 0.010$ | $\pm 0.028$ | $\pm 0.009$ | $\pm 0.009$ | $\pm 0.010$ | $\pm 0.007$ | $\pm 0.007$ | $\pm 0.006$ | $\pm 0.007$
Deep SVDD | $0.644$ | $0.731$ | $0.475$ | $0.507$ | $0.496$ | $0.607$ | $0.932$ | $0.411$ | $0.901$ | $0.707$ | $0.482$ | $0.636$ | $0.774$ | $0.785$
(Ruff et al., 2018) | $\pm 0.043$ | $\pm 0.043$ | $\pm 0.040$ | $\pm 0.040$ | $\pm 0.025$ | $\pm 0.044$ | $\pm 0.015$ | $\pm 0.008$ | $\pm 0.022$ | $\pm 0.027$ | $\pm 0.054$ | $\pm 0.055$ | $\pm 0.068$ | $\pm 0.025$
MCDSVDD | $0.686$ | $\mathbf{0.828}$ | $\mathbf{0.624}$ | $\mathbf{0.584}$ | $0.706$ | $0.512$ | $0.770$ | $0.483$ | $0.854$ | $\mathbf{0.858}$ | $\mathbf{0.819}$ | $\mathbf{0.945}$ | $\mathbf{0.953}$ | $0.953$
(This work) | $\pm 0.051$ | $\mathbf{\pm 0.024}$ | $\mathbf{\pm 0.039}$ | $\mathbf{\pm 0.032}$ | $\pm 0.069$ | $\pm 0.113$ | $\pm 0.127$ | $\pm 0.080$ | $\pm 0.041$ | $\mathbf{\pm 0.025}$ | $\mathbf{\pm 0.015}$ | $\mathbf{\pm 0.006}$ | $\mathbf{\pm 0.003}$ | $\pm 0.008$

Table 1: Evaluation of the performance of each model when applied to each of
the top level taxonomy (transient, stochastic, periodic). Each row represents
a different outlier detection algorithm, and each column represents the
subclass considered as outlier. The performance is evaluated using the AUROC
scores. Best metrics per class are marked in boldface

###  4.1 Training

To facilitate the problem at hand, our training dataset is divided into three
primary classes: transient, stochastic, and periodic. These classes are
further stratified into 14 subclasses, as described in Section 2. For each
main class, we employ a dedicated anomaly detector, resulting in three
separate detectors. Since anomalies are typically unknown in real-world
scenarios, during cross-validation, we designate a subclass from each main
class as the anomalous class and exclude it from the training set.
Consequently, the model is trained on the remaining subclasses, with the
excluded subclass reserved solely for evaluation purposes. This approach
ensures that the model does not incorporate data from the chosen anomalous
subclass during the training phase. The process is iterated for each subclass,
thereby providing a comprehensive evaluation of the anomaly detectors’ overall
performance. The assessment primarily focuses on the detectors’ ability to
successfully identify the removed subclasses, aligning with established
practices in the machine learning literature (Ruff et al., 2018).

###  4.2 Evaluation

Although real outlier events are typically unavailable for evaluation, we aim
to create a realistic scenario to identify the most promising anomaly
detection models. As previously mentioned, 20% of the dataset is designated as
the test set (TS1) and is never utilized for training any model. We create a
second test set (TS2), including all the objects from TS1 that belong to the
inlier subclasses, and outliers from both TS1 and those who were removed from
the training set. To ensure a realistic distribution, TS2 is composed of 10%
outliers and 90% inliers. This setup enables a comprehensive assessment of the
models’ ability to detect the chosen outlier class within TS2. See Figure 1
for an illustrative diagram of training en evaluation methodology.

To evaluate model performance when a subclass is treated as an outlier, we
employ the Area Under the Receiver Operating Characteristic curve (AUROC;
Davis & Goadrich 2006).

##  5 Results

We conducted a comprehensive comparison of our proposed method, Multi-Class
Deep SVDD (MCDSVDD), against five state-of-the-art algorithms for anomaly
detection in the field of astronomy. The baseline algorithms we considered
were Isolation Forest (IForest; Liu et al. 2012), One-class Support Vector
Machine (OCSVM; Schölkopf et al. 1999), Autoencoder (AE; Rumelhart &
McClelland 1987), Variational Autoencoders (VAE; Kingma & Welling 2014), and
Deep Support Vector Data Description (Deep SVDD; Ruff et al. 2018). See
Appendix I for details and hyperparameters of all the algorithms used in this
work. Table 1 presents the 5-fold AUROC values obtained by each anomaly
detection model on the TS2 dataset. The table showcases the performance of
each model when considering each of the 14 subclasses as an outlier in
separate trials. The highest AUROC values for each subclass are highlighted in
bold, indicating the best-performing model for that particular task.

As can be seen, our proposed method MCDSVDD consistently outperforms all other
methods we considered for the transient and periodic objects. Our method is
able to detect SNII, SNIa, and SNIb as outliers with higher AUROC values than
the rest. For the SLSN subclass, AE’s performance is statistically
indistinguishable from MCDSVDD (i.e., not statistically significant; $p$-value
= 0.0789). In terms of the periodic classes, MCDSVDD shows superior
performance in detecting CEP, DSCT, E, and RRL as outliers, while AE
demonstrates better performance in detecting LPVs. Regarding stochastic
sources, AE outperforms the other methods for four out of the five subclasses
(AGN, Blazar, CV/Nova, and YSO), while OCSVM achieves the highest performance
for QSOs. It is worth mentioning that QSO light curves often exhibit slow and
smooth temporal variations, making them more challenging to detect as
anomalies. Therefore, AE emerges as the best outlier detection algorithm for
stochastic sources, whereas MCDSVDD excels in detecting transient and periodic
sources. We hypothesize that the high variability of stochastic light curves
may impact MCDSVDD’s capability to detect different types of sources, unlike
AE, which benefits from its reconstruction loss function.

##  6 Conclusion

In this paper, we addressed the challenge of effectively utilizing different
inlier categories with distinct data distributions in anomaly detection for
astronomical data. We proposed Multi-Class Deep SVDD (MCDSVDD), an extension
of the state-of-the-art One-Class Deep SVDD algorithm. MCDSVDD employs a
neural network to map the data into hyperspheres, where each hypersphere
represents a specific inlier category. By measuring the distance of each
sample from the centers of these hyperspheres, MCDSVDD determines the anomaly
scores of the samples. We evaluated the performance of MCDSVDD along with
several one-class anomaly detection algorithms on a large dataset of
astronomical light-curves from the Zwicky Transient Facility (ZTF). The
results demonstrated that MCDSVDD outperforms other algorithms in detecting
anomalous sources while effectively leveraging the presence of different
inlier categories. Our findings highlight the importance of considering
distinct inlier categories in anomaly detection tasks and showcase the
potential of MCDSVDD for identifying anomalies in astronomical data. Finally,
the code and data to reproduce the results presented in this work are publicly
available at https://github.com/mperezcarrasco/AnomalyALeRCE.

##  7 Acknowledgements

This work gratefully acknowledges funding from ANID through: Millennium
Science Initiative Programs ICN12_009 (MPC, GCV, LHG, FF, AMMA, IR, FEB, MC,
PAE, GP) and NCN19_171 (AB); CATA-Basal AFB-170002 (FEB, MC), ACE210002 (FEB,
MC) and FB210003 (FEB, MC, FF); BASAL project FB210005 (AMMA); FONDECYT
Regular 1231877 (GCV), 1200710 (FF), 1190818 (FEB), 1231637 (MC), and 1200495
(FEB); FONDECYT Postdoctorado 3200250 (PSS); and infrastructure funds
QUIMAL140003 and QUIMAL190012; FONDECYT 1220829 (PAE). We acknowledge support
from REUNA Chile, which hosts and maintains some of our infrastructure. This
work has been possible thanks the use of AWS-U.Chile-NLHPC credits and
computational resources provided by Data Observatory. Powered@NLHPC: This
research was partially supported by the supercomputing infrastructure of the
NLHPC (ECM-02). This research makes use of the community built resource VSX
available at the AAVSO. We additionally thank G. Hajdu for useful discussions.

## References

  * Baron & Poznanski (2016) Baron, D. and Poznanski, D.  The weirdest SDSS galaxies: results from an outlier detection algorithm.  _Monthly Notices of the Royal Astronomical Society_ , 465(4):4530–4555, 11 2016.  ISSN 0035-8711.  doi: 10.1093/mnras/stw3021.  URL https://doi.org/10.1093/mnras/stw3021.
  * Bellm et al. (2018) Bellm, E. C., Kulkarni, S. R., Graham, M. J., Dekany, R., Smith, R. M., Riddle, R., Masci, F. J., Helou, G., Prince, T. A., Adams, S. M., Barbarino, C., Barlow, T., Bauer, J., Beck, R., Belicki, J., Biswas, R., Blagorodnova, N., Bodewits, D., Bolin, B., Brinnel, V., Brooke, T., Bue, B., Bulla, M., Burruss, R., Cenko, S. B., Chang, C.-K., Connolly, A., Coughlin, M., Cromer, J., Cunningham, V., De, K., Delacroix, A., Desai, V., Duev, D. A., Eadie, G., Farnham, T. L., Feeney, M., Feindt, U., Flynn, D., Franckowiak, A., Frederick, S., Fremling, C., Gal-Yam, A., Gezari, S., Giomi, M., Goldstein, D. A., Golkhou, V. Z., Goobar, A., Groom, S., Hacopians, E., Hale, D., Henning, J., Ho, A. Y. Q., Hover, D., Howell, J., Hung, T., Huppenkothen, D., Imel, D., Ip, W.-H., Ivezić, Ž., Jackson, E., Jones, L., Juric, M., Kasliwal, M. M., Kaspi, S., Kaye, S., Kelley, M. S. P., Kowalski, M., Kramer, E., Kupfer, T., Landry, W., Laher, R. R., Lee, C.-D., Lin, H. W., Lin, Z.-Y., Lunnan, R., Giomi, M., Mahabal, A., Mao, P., Miller, A. A., Monkewitz, S., Murphy, P., Ngeow, C.-C., Nordin, J., Nugent, P., Ofek, E., Patterson, M. T., Penprase, B., Porter, M., Rauch, L., Rebbapragada, U., Reiley, D., Rigault, M., Rodriguez, H., van Roestel, J., Rusholme, B., van Santen, J., Schulze, S., Shupe, D. L., Singer, L. P., Soumagnac, M. T., Stein, R., Surace, J., Sollerman, J., Szkody, P., Taddia, F., Terek, S., Sistine, A. V., van Velzen, S., Vestrand, W. T., Walters, R., Ward, C., Ye, Q.-Z., Yu, P.-C., Yan, L., and Zolkower, J.  The zwicky transient facility: System overview, performance, and first results.  _Publications of the Astronomical Society of the Pacific_ , 131(995):018002, dec 2018.  doi: 10.1088/1538-3873/aaecbe.  URL https://doi.org/10.1088%2F1538-3873%2Faaecbe.
  * Chandola et al. (2009) Chandola, V., Banerjee, A., and Kumar, V.  Anomaly detection: A survey.  _ACM Comput. Surv._ , 41(3), July 2009.  ISSN 0360-0300.  doi: 10.1145/1541880.1541882.  URL https://doi.org/10.1145/1541880.1541882.
  * Davis & Goadrich (2006) Davis, J. and Goadrich, M.  The relationship between precision-recall and roc curves.  In _Proceedings of the 23rd International Conference on Machine Learning_ , ICML ’06, pp. 233–240, New York, NY, USA, 2006. Association for Computing Machinery.  ISBN 1595933832.  doi: 10.1145/1143844.1143874.  URL https://doi.org/10.1145/1143844.1143874.
  * Duev et al. (2019) Duev, D. A., Mahabal, A., Masci, F. J., Graham, M. J., Rusholme, B., Walters, R., Karmarkar, I., Frederick, S., Kasliwal, M. M., Rebbapragada, U., and Ward, C.  Real-bogus classification for the Zwicky Transient Facility using deep learning.  _Astrophysical Journal_ , 489(3):3582–3590, November 2019.  doi: 10.1093/mnras/stz2357.
  * Förster et al. (2021) Förster, F., Cabrera-Vives, G., Castillo-Navarrete, E., Estévez, P. A., Sánchez-Sáez, P., Arredondo, J., Bauer, F. E., Carrasco-Davis, R., Catelan, M., Elorrieta, F., Eyheramendy, S., Huijse, P., Pignata, G., Reyes, E., Reyes, I., Rodríguez-Mancini, D., Ruz-Mieres, D., Valenzuela, C., Álvarez-Maldonado, I., Astorga, N., Borissova, J., Clocchiatti, A., De Cicco, D., Donoso-Oliva, C., Hernández-García, L., Graham, M. J., Jordán, A., Kurtev, R., Mahabal, A., Maureira, J. C., Muñoz-Arancibia, A., Molina-Ferreiro, R., Moya, A., Palma, W., Pérez-Carrasco, M., Protopapas, P., Romero, M., Sabatini-Gacitua, L., Sánchez, A., San Martín, J., Sepúlveda-Cobo, C., Vera, E., and Vergara, J. R.  The Automatic Learning for the Rapid Classification of Events (ALeRCE) Alert Broker.  _The Astronomical Journal_ , 161(5):242, May 2021\.  doi: 10.3847/1538-3881/abe9bc.
  * Hložek et al. (2020) Hložek, R., Ponder, K. A., Malz, A. I., Dai, M., Narayan, G., Ishida, E. E. O., Allam, T., Bahmanyar, A., Biswas, R., Galbany, L., Jha, S. W., Jones, D. O., Kessler, R., Lochner, M., Mahabal, A. A., Mandel, K. S., Martínez-Galarza, J. R., McEwen, J. D., Muthukrishna, D., Peiris, H. V., Peters, C. M., and Setzer, C. N.  Results of the photometric lsst astronomical time-series classification challenge (plasticc), 2020.  URL https://arxiv.org/abs/2012.12392.
  * Ishida et al. (2021) Ishida, E. E. O., Kornilov, M. V., Malanchev, K. L., Pruzhinskaya, M. V., Volnova, A. A., Korolev, V. S., Mondon, F., Sreejith, S., Malancheva, A. A., and Das, S.  Active anomaly detection for time-domain discoveries.  _Astronomy and Astrophysics_ , 650:A195, June 2021.  doi: 10.1051/0004-6361/202037709.
  * Kessler et al. (2009) Kessler, R., Bernstein, J. P., Cinabro, D., Dilday, B., Frieman, J. A., Jha, S., Kuhlmann, S., Miknaitis, G., Sako, M., Taylor, M., and Vanderplas, J.  SNANA: A Public Software Package for Supernova Analysis.  _Publications of the Astronomical Society of the Pacific_ , 121(883):1028, September 2009.  doi: 10.1086/605984.
  * Kessler et al. (2019) Kessler, R., Narayan, G., Avelino, A., Bachelet, E., Biswas, R., Brown, P. J., Chernoff, D. F., Connolly, A. J., Dai, M., Daniel, S., Stefano, R. D., Drout, M. R., Galbany, L., González-Gaitán, S., Graham, M. L., Hložek, R., Ishida, E. E. O., Guillochon, J., Jha, S. W., Jones, D. O., Mandel, K. S., Muthukrishna, D., O’Grady, A., Peters, C. M., Pierel, J. R., Ponder, K. A., Prša, A., Rodney, S., Villar, V. A., Collaboration, T. L. D. E. S., the Transient, and Collaboration), V. S. S.  Models and simulations for the photometric lsst astronomical time series classification challenge (plasticc).  _Publications of the Astronomical Society of the Pacific_ , 131(1003):094501, jul 2019.  doi: 10.1088/1538-3873/ab26f1.  URL https://dx.doi.org/10.1088/1538-3873/ab26f1.
  * Kingma & Welling (2014) Kingma, D. P. and Welling, M.  Auto-encoding variational bayes.  In _International Conference on Learning Representations_ , 2014.
  * Liu et al. (2012) Liu, F. T., Ting, K. M., and Zhou, Z.-H.  Isolation-based anomaly detection.  _ACM Trans. Knowl. Discov. Data_ , 6(1), March 2012.  ISSN 1556-4681.  doi: 10.1145/2133360.2133363.  URL https://doi.org/10.1145/2133360.2133363.
  * Mahabal et al. (2019) Mahabal, A., Rebbapragada, U., Walters, R., Masci, F. J., Blagorodnova, N., van Roestel, J., Ye, Q.-Z., Biswas, R., Burdge, K., Chang, C.-K., Duev, D. A., Golkhou, V. Z., Miller, A. A., Nordin, J., Ward, C., Adams, S., Bellm, E. C., Branton, D., Bue, B., Cannella, C., Connolly, A., Dekany, R., Feindt, U., Hung, T., Fortson, L., Frederick, S., Fremling, C., Gezari, S., Graham, M., Groom, S., Kasliwal, M. M., Kulkarni, S., Kupfer, T., Lin, H. W., Lintott, C., Lunnan, R., Parejko, J., Prince, T. A., Riddle, R., Rusholme, B., Saunders, N., Sedaghat, N., Shupe, D. L., Singer, L. P., Soumagnac, M. T., Szkody, P., Tachibana, Y., Tirumala, K., van Velzen, S., and Wright, D.  Machine learning for the zwicky transient facility.  _Publications of the Astronomical Society of the Pacific_ , 131(997):038002, jan 2019.  doi: 10.1088/1538-3873/aaf3fa.  URL https://doi.org/10.1088%2F1538-3873%2Faaf3fa.
  * Mainzer et al. (2011) Mainzer, A., Bauer, J., Grav, T., Masiero, J., Cutri, R. M., Dailey, J., Eisenhardt, P., McMillan, R. S., Wright, E., Walker, R., Jedicke, R., Spahr, T., Tholen, D., Alles, R., Beck, R., Brandenburg, H., Conrow, T., Evans, T., Fowler, J., Jarrett, T., Marsh, K., Masci, F., McCallon, H., Wheelock, S., Wittman, M., Wyatt, P., DeBaun, E., Elliott, G., Elsbury, D., Gautier, T., Gomillion, S., Leisawitz, D., Maleszewski, C., Micheli, M., and Wilkins, A.  PRELIMINARY RESULTS FROM NEOWISE: AN ENHANCEMENT TO THEWIDE-FIELD INFRARED SURVEY EXPLORERFOR SOLAR SYSTEM SCIENCE.  _The Astrophysical Journal_ , 731(1):53, mar 2011\.  doi: 10.1088/0004-637x/731/1/53.  URL https://doi.org/10.1088%2F0004-637x%2F731%2F1%2F53.
  * Masci et al. (2018) Masci, F. J., Laher, R. R., Rusholme, B., Shupe, D. L., Groom, S., Surace, J., Jackson, E., Monkewitz, S., Beck, R., Flynn, D., Terek, S., Landry, W., Hacopians, E., Desai, V., Howell, J., Brooke, T., Imel, D., Wachter, S., Ye, Q.-Z., Lin, H.-W., Cenko, S. B., Cunningham, V., Rebbapragada, U., Bue, B., Miller, A. A., Mahabal, A., Bellm, E. C., Patterson, M. T., Jurić, M., Golkhou, V. Z., Ofek, E. O., Walters, R., Graham, M., Kasliwal, M. M., Dekany, R. G., Kupfer, T., Burdge, K., Cannella, C. B., Barlow, T., Sistine, A. V., Giomi, M., Fremling, C., Blagorodnova, N., Levitan, D., Riddle, R., Smith, R. M., Helou, G., Prince, T. A., and Kulkarni, S. R.  The zwicky transient facility: Data processing, products, and archive.  _Publications of the Astronomical Society of the Pacific_ , 131(995):018003, dec 2018.  doi: 10.1088/1538-3873/aae8ac.  URL https://doi.org/10.1088%2F1538-3873%2Faae8ac.
  * Muthukrishna et al. (2022) Muthukrishna, D., Mandel, K. S., Lochner, M., Webb, S., and Narayan, G.  Real-time detection of anomalies in large-scale transient surveys.  _Astrophysical Journal_ , 517(1):393–419, November 2022.  doi: 10.1093/mnras/stac2582.
  * Ruff et al. (2018) Ruff, L., Vandermeulen, R., Goernitz, N., Deecke, L., Siddiqui, S. A., Binder, A., Müller, E., and Kloft, M.  Deep one-class classification.  In Dy, J. and Krause, A. (eds.), _Proceedings of the 35th International Conference on Machine Learning_ , volume 80 of _Proceedings of Machine Learning Research_ , pp. 4393–4402, Stockholmsmässan, Stockholm Sweden, 10–15 Jul 2018. PMLR.  URL http://proceedings.mlr.press/v80/ruff18a.html.
  * Ruff et al. (2019) Ruff, L., Vandermeulen, R., Görnitz, N., Binder, A., Müller, E., and Kloft, M.  Deep support vector data description for unsupervised and semi-supervised anomaly detection.  06 2019.
  * Rumelhart & McClelland (1987) Rumelhart, D. E. and McClelland, J. L.  _Learning Internal Representations by Error Propagation_ , pp. 318–362.  1987\.
  * Sánchez-Sáez et al. (2021a) Sánchez-Sáez, P., Lira, H., Martí, L., Sánchez-Pi, N., Arredondo, J., Bauer, F. E., Bayo, A., Cabrera-Vives, G., Donoso-Oliva, C., Estévez, P. A., Eyheramendy, S., Förster, F., Hernández-García, L., Arancibia, A. M. M., Pérez-Carrasco, M., Sepúlveda, M., and Vergara, J. R.  Searching for Changing-state AGNs in Massive Data Sets. I. Applying Deep Learning and Anomaly-detection Techniques to Find AGNs with Anomalous Variability Behaviors.  _The Astronomical Journal_ , 162(5):206, November 2021a.  doi: 10.3847/1538-3881/ac1426.
  * Sánchez-Sáez et al. (2021b) Sánchez-Sáez, P., Reyes, I., Valenzuela, C., Förster, F., Eyheramendy, S., Elorrieta, F., Bauer, F. E., Cabrera-Vives, G., Estévez, P. A., Catelan, M., Pignata, G., Huijse, P., De Cicco, D., Arévalo, P., Carrasco-Davis, R., Abril, J., Kurtev, R., Borissova, J., Arredondo, J., Castillo-Navarrete, E., Rodriguez, D., Ruz-Mieres, D., Moya, A., Sabatini-Gacitúa, L., Sepúlveda-Cobo, C., and Camacho-Iñiguez, E.  Alert Classification for the ALeRCE Broker System: The Light Curve Classifier.  _The Astronomical Journal_ , 161(3):141, March 2021b.  doi: 10.3847/1538-3881/abd5c1.
  * Schölkopf et al. (1999) Schölkopf, B., Williamson, R., Smola, A., Shawe-Taylor, J., and Platt, J.  Support vector method for novelty detection.  In _Proceedings of the 12th International Conference on Neural Information Processing Systems_ , NIPS’99, pp. 582–588, Cambridge, MA, USA, 1999. MIT Press.
  * Villar et al. (2020) Villar, V. A., Cranmer, M., Contardo, G., Ho, S., and Lin, J. Y.-Y.  Anomaly Detection for Multivariate Time Series of Exotic Supernovae.  In _34th Conference on Neural Information Processing Systems_ , 10 2020.
  * Wright et al. (2010) Wright, E. L., Eisenhardt, P. R. M., Mainzer, A. K., Ressler, M. E., Cutri, R. M., Jarrett, T., Kirkpatrick, J. D., Padgett, D., McMillan, R. S., Skrutskie, M., Stanford, S. A., Cohen, M., Walker, R. G., Mather, J. C., Leisawitz, D., Gautier, T. N., McLean, I., Benford, D., Lonsdale, C. J., Blain, A., Mendez, B., Irace, W. R., Duval, V., Liu, F., Royer, D., Heinrichsen, I., Howard, J., Shannon, M., Kendall, M., Walsh, A. L., Larsen, M., Cardon, J. G., Schick, S., Schwalm, M., Abid, M., Fabinsky, B., Naes, L., and Tsai, C.-W.  THE WIDE-FIELD INFRARED SURVEY EXPLORER (WISE): MISSION DESCRIPTION AND INITIAL ON-ORBIT PERFORMANCE.  _The Astronomical Journal_ , 140(6):1868–1881, nov 2010.  doi: 10.1088/0004-6256/140/6/1868.  URL https://doi.org/10.1088%2F0004-6256%2F140%2F6%2F1868.
  * Xiong et al. (2010) Xiong, L., Poczos, B., Connolly, A., and Schneider, J.  Anomaly detection for astronomical data, Jun 2010.  URL https://kilthub.cmu.edu/articles/journal_contribution/Anomaly_Detection_for_Astronomical_Data/6475475/1.

##  Appendix A Appendix I

In this work, six anomaly detection algorithms are examined in order to
compare their performances in finding outliers. The algorithms and their
respective hyperparameters are:

Isolation Forest

The Isolation Forest algorithm is based on the concept of isolation trees. It
randomly selects features and divides them into distinct non-overlapping
regions using a randomly selected threshold criterion. The anomaly score is
proportional to the number of splits required to isolate each object in the
sample. Anomalous objects are expected to require fewer splits to be isolated.
To address the issue of overfitting, Isolation Forest employs ensembles of
isolation trees.

We utilized the implementation of Isolation Forest provided by scikit-learn.
The hyperparameters were set as follows: number of trees = 100, number of
samples to train each base estimator = 256, and contamination parameter = 0.1.
These values were recommended in the original work by Liu et al. (2012). Since
outliers were not used for training, we did not perform hyperparameter
selection through cross-validation.

One-Class Support Vector Machine

The One-Class Support Vector Machine (OCSVM) is an anomaly detection method
based on Support Vector Machines (SVM). OCSVM maps the data into a new feature
space, where the inner product between two objects can be represented with a
kernel function, such as the Gaussian kernel. It learns a hyperplane in this
feature space that separates the region where most of the data lie. During
testing, the anomaly score is computed by evaluating the distance of data
points with respect to the learned hyperplane.

We used the One-Class SVM implementation provided by scikit-learn with the
radial basis function (RBF) kernel. The hyperparameters were set to $\nu=0.01$
and the contamination parameter $c=0.1$, following the default values. As the
anomalous samples were assumed to be unknown, we did not perform
hyperparameter selection and used the default settings.

Autoencoder

Autoencoders (AE) are unsupervised neural network algorithms that aim to
reconstruct the input data using a lower-dimensional representation called the
latent space. They consist of an encoder function that maps the input data to
the lower-dimensional representation and a decoder function that reconstructs
the original data from the latent space. AE’s effectiveness in anomaly
detection is attributed to the assumption that anomalies are incompressible
and cannot be effectively reconstructed from low-dimensional projections. The
reconstruction error, computed as the mean squared error, serves as the
anomaly score.

We implemented the Autoencoder using PyTorch 1.0.0. Our model architecture
consisted in an encoder with 4 hidden layers with {512, 256, 128, 64} neurons,
and a decoder with {64, 128, 256, 512} neurons. We use batch normalization and
Leaky ReLU as activation function, except for the last layer that used a Tanh
activation function. We selected the hyperparameters by considering the
reconstruction error over a validation set consisting only of inliers.

Variational Autoencoder

Variational Autoencoders (VAE) extend autoencoders by incorporating a
regularization term. This term enforces the latent space to follow a known
distribution, typically a normal distribution. By generating multiple
reconstructions for each sample and averaging their reconstruction errors,
VAEs compute an anomaly score.

We implemented the Variational Autoencoder using PyTorch 1.0.0. We use the
same model architecture as the autoencoder, but we add two extra layers at the
end of the encoder to learn the parameters $\mu$ and $\sigma$ of the normal
distribution. We selected the hyperparameters using the unsupervised loss
function over a validation set consisting only of inliers.

Deep Support Vector Data Description

To train the Deep Support Vector Data Description (Deep SVDD), an Autoencoder
is trained until convergence. Then, the decoder is removed, and the center of
the hypersphere is estimated as the average of the encoder’s outputs on the
training data. Subsequently, the parameters of the encoder are re-optimized
using the pretrained parameters, with the objective of minimizing the distance
between the encoder’s output and the center of the hypersphere, along with a
weight decay regularizer.

During testing, the anomaly score is defined as the distance between each data
point and the center of the hypersphere.

We implemented Deep SVDD using PyTorch 1.0.0. We use the same model
architecture as the encoder of the autoencoder. The hyperparameter
$\lambda=0.5\times 10^{-6}$ was selected by measuring the unsupervised loss
function over a validation set of inliers.

Multi-Class Deep SVDD

Similar to Deep SVDD, an Autoencoder is trained until convergence, and the
decoder is removed. The centers of the hyperspheres are estimated as the
averages of the encoder’s outputs, considering only the data points of each
class. The parameters of the encoder are re-optimized using a similar
objective as Deep SVDD, but considering multiple hyperspheres.

During testing, the anomaly score is determined by measuring the distance
between each data point and the center of its closest hypersphere.

We implemented MCDSVDD using PyTorch 1.0.0. We use the same model architecture
as the encoder of the autoencoder. We set the hyperparameter
$\lambda=0.5\times 10^{-6}$ through cross-validation of the unsupervised loss
function over a validation set consisting only of inliers.


# Squeeze-and-Excite ResNet-Conformers for Sound Event Localization,
Detection, and Distance Estimation for DCASE2024 Challenge

###### Abstract

This technical report details our systems submitted for Task 3 of the DCASE
2024 Challenge: Audio and Audiovisual Sound Event Localization and Detection
(SELD) with Source Distance Estimation (SDE). We address only the audio-only
SELD with SDE (SELDDE) task in this report. We propose to improve the existing
ResNet-Conformer architectures with Squeeze-and-Excitation blocks in order to
introduce additional forms of channel- and spatial-wise attention. In order to
improve SELD performance, we also utilize the Spatial Cue-Augmented Log-
Spectrogram (SALSA) features over the commonly used log-mel spectra features
for polyphonic SELD. We complement the existing Sony-TAu Realistic Spatial
Soundscapes 2023 (STARSS23) dataset with the audio channel swapping technique
and synthesize additional data using the SpatialScaper generator. We also
perform distance scaling in order to prevent large distance errors from
contributing more towards the loss function. Finally, we evaluate our approach
on the evaluation subset of the STARSS23 dataset.

Index Terms—  Sound event localization and detection, sound distance
estimation, Conformer, feature extraction

##  1 Introduction

The sound event localization and detection (SELD) task for the DCASE 2024
Challenge now introduces the additional task of sound distance estimation
(SDE). Not only are systems required to perform sound event detection (SED)
and its corresponding direction of arrival (DOA) estimation, systems now also
need to estimate the distance between the recording device and the active
sound source. The introduction of the SDE task into the challenge problem
makes the task significantly more complex. Although there is a wealth of
literature relating to SELD and SDE tasks separately, little work has been
done on the joint SELD and SDE (SELDDE) task. In [1], Krause et al. introduced
the first form of investigation into this newly modeled SELDDE problem. The
authors extend the existing multi activity-coupled Cartesian distance of
arrival (multi-ACCDOA) vector proposed by Shimada et al. [2] to include
distance estimation, which they term the multi activity-coupled Cartesian
distance and DOA (multi-ACCDDOA) method. The authors then modify the existing
convolutional recurrent neural network (CRNN) SELDNet, which is used for many
SELD tasks, to concurrently estimate the presence of active sound events and
their corresponding spatial location. This modified CRNN is then used as the
baseline system for DCASE 2024 Challenge Task 3.

CRNNs have also formed the basis of many state-of-the-art (SOTA) work in SELD.
Most of the top-ranking submissions in the DCASE 2023 Challenge Task 3 utilize
some form of CRNN-based architecture [3, 4, 5], with the most common being the
ResNet-Conformer system. We adopt the ResNet-Conformer architecture for our
approach, using the multi-ACCDDOA format as its output. To further enhance
SELD performance, we use the highly effective Spatial Cue-Augmented Log-
Spectrogram (SALSA) features proposed by Tho et al. [6]. As the SALSA features
utilize more frequency bins as it does not use the traditional Mel frequency
bins, we include Squeeze-and-Excitation blocks [7, 8] to introduce forms of
spatial attention to make full use of the rich amount of spatial information
provided by the SALSA features. In addition, we use data augmentation
techniques adapted from [9] and synthesize additional data using the
SpatialScaper generator [10] to compliment the STARSS23 dataset [11]. In
general, we show that our approach achieves significant improvements over the
baseline system without model ensembling.

##  2 Proposed Method

###  2.1 Feature Extraction

The STARSS23 dataset provides audio data in both the first-order Ambisonics
(FOA) and tetrahedral microphone (MIC) formats. We utilize only the FOA array
signals in our proposed approach. We extract the SALSA features in a similar
manner as detailed in [6]. We used a sampling rate of 24kHz, a window length
of 512 samples, hop size of 300 samples, 512-point FFT and the Hann window.
Similarly, we also linearly compress frequency bands above 9kHz to reduce the
feature dimensions to reduce the training time. This results in an input shape
of $(7\times 400\times 200)$ for number of input channels, time bins and
frequency bins respectively.

The SED, DOA, and SDE target labels were extracted and converted to the multi-
ACCDDOA format. In [1], the authors note that larger distances may contribute
more significant errors. To mitigate this issue, we first standardize and then
scale the distance distribution such that all distance values lie within
$[-1,1]$. Moreover, as all elements in the multi-ACCDDOA vector are now lie
within the same range of $[-1,1]$, we can also apply the tanh operation to the
model’s output logits. This helps to introduce a further form of non-linearity
to the system. The distance scaling method consists of two steps –
standardization followed by scaling. The standardization of the distance
distribution is given by

$d_{stand}=\frac{d-\bar{d}}{\sigma_{d}},$ (1)

where d represents the distance distribution in the data set in meters, with
$\bar{d}$ and $\sigma_{d}$ representing the mean and standard deviation of
this distribution, respectively. This operation yields us the standardized
distribution of distance values, $d_{stand}$, with approximately zero mean and
unit standard deviation. The next step is to scale the distance distribution
to $d_{scaled}$, such that $d_{scaled}\in[-1,1]$. The following equation shows
the scaling operation,

$d_{scaled}=\frac{d_{stand}}{\text{max}(d_{stand})}\cdot$ (2)

To retrieve the original distance values in meters, we simply perform (1) and
(2) in reverse order to yield the original $d$. We store $\bar{d},\sigma_{d}$
and $\text{max}(d_{stand})$ as variables to use to restore the distance values
back from scaled units into meters during post-processing. However, we note
that using this scaling method will limit our output distance values in meters
to be within the minimum and maximum of the original d. In this case, our
output distance values are limited to $(0.04\text{m},7.64\text{m})$.

###  2.2 Dataset

The development set of the STARSS23 dataset consists of 168 recording clips
recorded in 16 unique rooms. The training split of this development set only
consists of 98 recordings captured across 12 rooms, totaling up to around 5.5
hours. In order to compliment the relatively small dataset provided, the DCASE
Challenge organizers also provide an external synthesized
dataset111https://zenodo.org/records/10932241 generated using real spatial
room impulse responses (SRIRs) from the TAU-SRIR database [12]. We refer to
this dataset as the “DCASE synthetic dataset”. The DCASE synthetic dataset
consists of 1200 one-minute long spatialized recordings, and we note that the
dataset has its own imbalance in terms of class and distance distributions. As
a result, we opt to generate our own set of audio recordings instead.

Due to the difficulty of manually annotating audio data, the STARSS23 dataset
is limited by its size and distribution of classes. We address this problem by
spatializing additional training data using the SpatialScaper generator [10].
When synthesizing this additional dataset, we make special note of the SRIRs
used to generate the audio data. Under the assumption that the rooms in the
STARSS23 dataset are of similar size, we only select SRIRs that were recorded
at similar distances to the sound source distances recorded in the development
set of the STARSS23 dataset. We also focus on synthesizing sound event classes
that are under-represented in the STARSS23 dataset, such as “knock” or
“doorCupboard”. We follow the synthesizing configurations of the provided
additional dataset and limit the maximum polyphony to 3. We synthesized around
18 hours worth of additional data using SRIRs from 8 different rooms, which we
refer to as the “SpatialScaper dataset”. During training, we opt to use the
SpatialScaper dataset to compliment the STARSS23 training set as opposed to
the DCASE synthetic dataset. We show the distribution of distances in the
DCASE synthetic dataset and the SpatialScaper dataset compared to the STARSS23
training set in Figure 1.

Figure 1: Distribution of distances within each dataset in comparison with
STARSS23 training set

###  2.3 Data Augmentation

To prevent overfitting and improve general system robustness and
generalization ability, we utilize several data augmentation techniques. All
augmentations are applied on-the-fly during training. We can split the data
augmentation methods into augmentation techniques that are performed at either
the waveform and spectrogram level. At the waveform level, we choose to only
utilize the audio channel swapping (ACS) method as proposed by Wang et al. in
[9]. The ACS augmentation method swaps the audio channels around, thus
creating new DOA representations. The ACS method also strongly improves SELD
in real recordings as noted in [13] as they maintain the reverberation and
diffuse effects of real life recordings. As such, we only apply the ACS
augmentation method to data from the STARSS23 dataset.

At the spectrogram level, we use SpecAugment [14], Mixup [15] along both the
time and frequency domain, as well as frequency shifting [16]. For spectrogram
level augmentations, we adopt a similar augmentation policy as the one
detailed by Wang et al. in [17]. In their work, the authors proposed a data
augmentation approach to dynamically configure the magnitude of augmentation
policies with a cosine annealing scheduler. For our training setup, we also
reduce the magnitude of spectrogram level augmentations as the learning rate
decays. However, we still apply the ACS augmentation method at the same
probability throughout training.

###  2.4 Network Architecture

Figure 2: Proposed system architecture with added sSE and SCSE blocks

We use the ResNet-Conformer system, which was used by most top-ranking teams
in the previous year’s DCASE Challenge Task 3. The conformer architecture was
first proposed by Gulati et al. and refers to convolution-augmented
transformers [18]. A conformer block comprises four modules stacked
consecutively: a feed-forward module, a self-attention module, a convolution
module, and a second feed-forward module. Overall, the attention module helps
to recognize global features, while the convolution module helps to identify
local features. Combining the two yields SOTA results in the field of speech
recognition, and has been used in SELD works as well to positive effect, as
seen from the results of DCASE Challenge 2023 Task 3.

To further improve performance, we utilized a deeper, modified ResNet-18
structure for the encoding layer preceding the conformer blocks as detailed by
Niu et al. in [13]. In the proposed system, we apply pooling operations only
along the frequency dimension after each ResNet block. The authors also noted
that the conformer architecture benefits from a higher temporal resolution,
and find that the information of impulse or impact sound events may be
affected by early time pooling operations. As such, we choose to apply the
time pooling operation after the conformer blocks to preserve temporal
information within the network. We also apply the tanh activation function at
the end of our model, which is only possible due to the distance values being
scaled to be within $[-1,1]$. Our system utilizes the multi-ACCDDOA output
format as proposed in [1]. We train the model using the mean squared error
(MSE) loss function with auxiliary duplicating permutation invariant training,
as described by Shimada et al. [2]. The full proposed model architecture is
shown in Figure 2. The pooling operations are done along the time $\times$
frequency dimensions. We utilize AvgMaxPool, which refers to the arithmetic
sum of the outputs of an average pooling and max pooling operation on an input
feature map. We adopt this method from [19] to maximize the benefit of both
forms of pooling operations.

###  2.5 Squeeze-and-Excitation Blocks

Figure 3: Block diagram of the cSE block (left), sSE block (middle) and SCSE
block (right)

One of the key reasons behind the effectiveness of the SALSA suite of features
is the channel-wise alignment of the time frequency (TF) bins of both the
spectral and spatial features. In SALSA, a linear frequency scale is used
instead of the typical log-mel frequency scale, which helps preserve frequency
information without any form of aggregation. After linearly compressing the
frequency bins above 9kHz in SALSA, we end up with 200 frequency bins, as
compared to the 64 or 128 Mel bins used in typical Mel-spectrogram features.
In order to make full use of the expansive frequency information, we implement
additional functional layers to improve the system’s ability to learn inter-
channel spatial information.

Squeeze-and-Excite (SE) blocks were first introduced by Hu et al. as
additional layers to CNNs [7] for image segmentation work. SE blocks work by
first aggregating global spatial information into a channel descriptor. This
is usually done through global average pooling to form a vector with a length
equal to the number of convolutional channels. After which, a self-gating
mechanism is used to “excite” the vector, which is then used to rescale the
output feature maps. This can be seen as a form of “channel attention”, where
the SE block aims to actively scale the channel importance and model the
inter-channel dependencies. In later work, Roy et al. introduced the idea of
channel- and spatial-wise SE blocks [8]. The original SE block proposed by Hu
et al. can be seen as a spatial squeeze and channel excitation block (cSE).
Roy et al. instead proposed its counterpart, which is the channel squeezing
and spatial excitation block (sSE). For the sSE block, the goal is to learn
the important spatial mapping of the input feature maps and can be seen as a
form of “spatial attention”. For the case of SELD, we propose that the sSE
block can be seen as a method to help the system focus on the important TF
bins across all channels. The authors also propose the concurrent spatial and
channel SE block (SCSE) which is a simple element-wise addition of the sSE and
SE excitations. In the output of the SCSE block, a location will receive a
higher activation if it gets a higher importance from both the channel and
spatial recalibration, or excitation. The SCSE block aims to encourage the
network to learn more meaningful feature maps that are relevant both spatially
and channel-wise. We show the workings of the cSE, sSE and SCSE block in
Figure 3.

In our proposed system, we add sSE blocks into the stem blocks of the ResNet
encoding layer, as this is the section of the system where all information
across the time and frequency dimensions are preserved without aggregation.
Our motive is for the system to learn the important inter-channel information
maps at the highest possible frequency resolution. After which, we include
SCSE blocks into the subsequent residual layers of the ResNet to learn
meaningful spatial and channel representations. We follow the work of [20] and
squeeze the number of channels by a factor of 4 in each cSE block in our
systems.

###  2.6 Training

For training, we first segment the input audio signal into 5 second segments
without overlapping before converting them into SALSA features. We train the
model for 200 epochs using the Adam optimizer, with a peak learning rate of
$5\times 10^{-4}$ and a batch size of 32, using the transformer learning rate
scheduler that was first proposed in [21]. We follow the implementation of the
baseline system and evaluate the model on the test split of the development
set of the STARSS23 dataset, saving the model with the best validation
location-dependent F1-Score as our final model. Furthermore, we further fine-
tune the models for another 50 epochs on only real audio recordings after the
initial training.

##  3 RESULTS

We first compare the performance improvements of each of our optimization
methods on the baseline system using the training split of the STARSS23
dataset. However, we did not compare the usage of the SALSA features using the
baseline system, as the baseline system is optimized for the 64 Mel bins of
the Mel-spectrogram input features. We instead only compare the usage of
distance scaling (DS), spectrogram level augmentations (Aug) and the audio
channel swapping augmentation method (ACS). We evaluate each model on the test
split of the development set of the STARSS23 dataset, using the DCASE
Challenge 2024 macro-averaged evaluation metrics that take SDE into
account222https://dcase.community/challenge2024/task-audio-and-audiovisual-
sound-event-localization-and-detection-with-source-distance-
estimation#evaluation. The jackknife estimation method was used when
calculating the metrics during evaluation. The full breakdown of our
experiments is presented in Table 1, with the first row of the table denoting
the results of the baseline system without any modifications. We also
introduce an aggregated SELDDE error function that is similar to the SELD
aggregated error function used in previous years’ DCASE Challenge Task 3 [22].
The SELDDE error, $\mathcal{E}_{\text{\tiny{SELDDE}}}$, is given by,

$\mathcal{E}_{\text{\tiny{SELDDE}}}=\frac{1}{3}((1-{\text{F}}_{20^{\circ}})+\frac{\text{LE}_{\text{CD}}}{180^{\circ}}+\text{RDE}_{\text{CD}}),$
(3)

where $\text{F}_{20^{\circ}}$, $\text{LE}_{\text{CD}}$ and
$\text{RDE}_{\text{CD}}$ represent the location-dependent F1-score, class-
dependent localization error and class-dependent relative distance error
respectively. This SELDDE error function is not used for any official DCASE
evaluation, but can be seen as a measure of the system’s overall SELDDE
performance.

Table 1: SELDDE performance of the baseline model using various optimization methods ACS | Aug | DS |  $\text{F}_{20^{\circ}}\uparrow$ |  $\text{LE}_{\text{CD}}\downarrow$ |  $\text{RDE}_{\text{CD}}$$\downarrow$ | $\mathcal{E}_{\text{\tiny{SELDDE}}}\downarrow$
---|---|---|---|---|---|---
|  |  |  13.1% |  $36.9^{\circ}$ |  33.0% | 0.468
|  | ✓ |  13.6% |  $33.2^{\circ}$ |  31.0% | 0.453
| ✓ |  |  15.0% |  $29.4^{\circ}$ |  27.3% | 0.429
| ✓ | ✓ |  15.9% |  $25.5^{\circ}$ |  31.6% | 0.433
✓ | ✓ | ✓ |  16.8% |  $21.1^{\circ}$ |  31.0% | 0.420

We show that the simple distance scaling trick can help the system to train
with a more stable loss function, which results in slightly improved SED, DOA
and SDE metrics. The usage of various data augmentation techniques also help
to improve the system’s ability to generalize, resulting in noticeable
improvements in all SELDDE metrics. Combining the two methods also improves
SED, DOA, and SDE performance over the baseline by a substantial margin. As
reported in [9, 13], the ACS method does help to generate additional realistic
DOA representations, which helps to improve DOA metrics significantly.

Table 2: SELDDE performance of each of our submitted systems in comparison with the baseline system (BL) using the FOA format System |  $\text{F}_{20^{\circ}}\uparrow$ |  $\text{LE}_{\text{CD}}\downarrow$ |  $\text{RDE}_{\text{CD}}\downarrow$ |  $\mathcal{E}_{\text{\tiny{SELDDE}}}\downarrow$
---|---|---|---|---
BL |  13.1% |  $36.9^{\circ}$ |  33.0% |  0.468
BL$+$ |  19.0% |  $29.1^{\circ}$ |  30.6% |  0.426
A |  33.9% |  $\textbf{20.4}^{\circ}$ |  30.4% |  0.359
B |  33.8% |  $21.4^{\circ}$ |  30.0% |  0.360
C |  32.7% |  $22.9^{\circ}$ |  30.1% |  0.367
D |  32.7% |  $20.6^{\circ}$ |  31.1% |  0.366

Next, we describe the performance of each of our systems submitted in the test
split of the development set of the STARSS23 dataset in Table 2. BL stands for
the performance of the baseline system, while BL$+$ represents the baseline
system trained on the dataset used for submission. All model submissions use
the underlying ResNet-Conformer architecture. System A uses all SE blocks as
shown in Figure 2. System B removes the final SCSE block from the tail of the
system. System C removes all SCSE blocks from the ResNet blocks, leaving only
the sSE and SCSE blocks in the stem and tail of the system, respectively.
Lastly, system D removes all sSE and SCSE blocks from the system entirely,
resulting in the “bare-bones” ResNet-Conformer system.

Overall, the inclusion of SE blocks into the system architecture improves the
SELDDE performance, as demonstrated by the improved metrics of systems A and
B. We conjecture that it could due to the SE blocks introducing slight forms
of spatial attention along the TF bins to better identify classes which
results in the improved SED F1-scores. The slight difference in the system
performances of systems A and B does also seem to suggest that the placement
of each SE block is crucial to the best performance of the system. We leave
the study for the optimal placement and positioning of SE blocks for future
study.

##  4 Conclusion

This technical report presents our proposed system architecture of combining
ResNets, Conformer modules and Squeeze-and-Excitation blocks for SELDDE tasks.
We also reintroduce SALSA features for polyphonic SELD tasks and forms of
channel- and spatial-wise attention modules within the convolutional layers.
Similarly, for future work, we hope to present another suite of hand-crafted
features dedicated to SELDDE performance. In general, our proposed approach
brings about significant improvements over the baseline method. We recognize
that we are unable to provide a full, detailed evaluation of all our proposed
improvements in this report. We leave this further ablation study for future
work.

##  5 ACKNOWLEDGEMENTS

This research is supported by the Ministry of Education, Singapore under its
Academic Research Fund Tier 2 (MOE-T2EP20221-0014).

## References

  * [1] D. A. Krause, A. Politis, and A. Mesaros, “Sound event detection and localization with distance estimation,” _arXiv_ , 2024.
  * [2] K. Shimada, Y. Koyama, S. Takahashi, N. Takahashi, E. Tsunoo, and Y. Mitsufuji, “Multi-accdoa: Localizing and detecting overlapping sounds from the same class with auxiliary duplicating permutation invariant training,” in _ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2022, pp. 316–320.
  * [3] S.-I. Kang, K. Cho, M. Keum, and Y. Park, “The distillation system for sound event localization and detection of dcase2023 challenge,” DCASE2023 Challenge, Tech. Rep., June 2023.
  * [4] Q. Wang, Y. Jiang, S. Cheng, M. Hu, Z. Nian, P. Hu, Z. Liu, Y. Dong, M. Cai, J. Du, and C.-H. Lee, “The nerc-slip system for sound event localization and detection of dcase2023 challenge,” DCASE2023 Challenge, Tech. Rep., June 2023.
  * [5] J. Hu, Y. Cao, M. Wu, F. Yang, W. Wang, M. D. Plumbley, and J. Yang, “Jless submission to dcase2023 task3: Conformer with data augmentation for sound event localization and detection in real space,” DCASE2023 Challenge, Tech. Rep., June 2023.
  * [6] T. N. T. Nguyen, K. N. Watcharasupat, N. K. Nguyen, D. L. Jones, and W. S. Gan, “SALSA: Spatial Cue-Augmented Log-Spectrogram Features for Polyphonic Sound Event Localization and Detection,” _IEEE/ACM Transactions on Audio Speech and Language Processing_ , vol. 30, pp. 1749–1762, 2022.
  * [7] J. Hu, L. Shen, and G. Sun, “Squeeze-and-excitation networks,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 7132–7141.
  * [8] A. G. Roy, N. Navab, and C. Wachinger, “Concurrent spatial and channel ‘squeeze & excitation’in fully convolutional networks,” in _Medical Image Computing and Computer Assisted Intervention–MICCAI 2018: 21st International Conference, Granada, Spain, September 16-20, 2018, Proceedings, Part I_. Springer, 2018, pp. 421–429.
  * [9] Q. Wang, J. Du, H.-X. Wu, J. Pan, F. Ma, and C.-H. Lee, “A four-stage data augmentation approach to resnet-conformer based acoustic modeling for sound event localization and detection,” _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , vol. 31, pp. 1251–1264, 2023.
  * [10] I. R. Roman, C. Ick, S. Ding, A. S. Roman, B. McFee, and J. P. Bello, “Spatial scaper: A library to simulate and augment soundscapes for sound event localization and detection in realistic rooms,” in _IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , Seoul, South Korea, April 2024.
  * [11] A. Politis, K. Shimada, P. Sudarsanam, A. Hakala, S. Takahashi, D. A. Krause, N. Takahashi, S. Adavanne, Y. Koyama, K. Uchida, Y. Mitsufuji, and T. Virtanen. (2023, Mar.) STARSS23: Sony-TAu Realistic Spatial Soundscapes 2023. [Online]. Available: https://doi.org/10.5281/zenodo.7709052
  * [12] A. Politis, S. Adavanne, and T. Virtanen. (2022, Apr.) TAU Spatial Room Impulse Response Database (TAU- SRIR DB). [Online]. Available: https://doi.org/10.5281/zenodo.6408611
  * [13] S. Niu, J. Du, Q. Wang, L. Chai, H. Wu, Z. Nian, L. Sun, Y. Fang, J. Pan, and C.-H. Lee, “An experimental study on sound event localization and detection under realistic testing conditions,” in _ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2023, pp. 1–5.
  * [14] D. S. Park, W. Chan, Y. Zhang, C.-C. Chiu, B. Zoph, E. D. Cubuk, and Q. V. Le, “Specaugment: A simple data augmentation method for automatic speech recognition,” in _Interspeech 2019_. ISCA, Sept. 2019. [Online]. Available: http://dx.doi.org/10.21437/Interspeech.2019-2680
  * [15] H. Zhang, M. Cisse, Y. N. Dauphin, and D. Lopez-Paz, “mixup: Beyond empirical risk minimization,” 2018.
  * [16] T. N. T. Nguyen, D. L. Jones, K. N. Watcharasupat, H. Phan, and W. S. Gan, “Salsa-Lite: a Fast and Effective Feature for Polyphonic Sound Event Localization and Detection With Microphone Arrays,” _ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings_ , vol. 2022-May, pp. 716–720, 2022.
  * [17] Z. Wang, F. Hou, Y. Qiu, Z. Ma, S. Singh, R. Wang, _et al._ , “Cyclicaugment: Speech data random augmentation with cosine annealing scheduler for auotmatic speech recognition.” in _INTERSPEECH_ , 2022, pp. 3859–3863.
  * [18] A. Gulati, J. Qin, C.-C. Chiu, N. Parmar, Y. Zhang, J. Yu, W. Han, S. Wang, Z. Zhang, Y. Wu, and R. Pang, “Conformer: Convolution-augmented transformer for speech recognition,” 2020.
  * [19] Q. Kong, Y. Cao, T. Iqbal, Y. Wang, W. Wang, and M. D. Plumbley, “Panns: Large-scale pretrained audio neural networks for audio pattern recognition,” 2020.
  * [20] J. Naranjo-Alcazar, S. Perez-Castanos, J. Ferrandis, P. Zuccarello, and M. Cobos, “Sound event localization and detection using squeeze-excitation residual cnns,” 2021.
  * [21] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” 2023.
  * [22] A. Politis, A. Mesaros, S. Adavanne, T. Heittola, and T. Virtanen, “Overview and evaluation of sound event localization and detection in dcase 2019,” _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , vol. 29, pp. 684–698, 2020. [Online]. Available: https://ieeexplore.ieee.org/abstract/document/9306885