# Dual input neural networks for positional sound source localization

E.G.Eric Grinstein  V.W.N.Vincent W. Neo  P.A.N.Patrick A. Naylor  Department
of Electrical and Electronic Engineering, Imperial College London, London, UK

###### Abstract

In many signal processing applications, metadata may be advantageously used in
conjunction with a high dimensional signal to produce a desired output. In the
case of classical Sound Source Localization (SSL) algorithms, information from
a high dimensional, multichannel audio signals received by many distributed
microphones is combined with information describing acoustic properties of the
scene, such as the microphones’ coordinates in space, to estimate the position
of a sound source. We introduce Dual Input Neural Networks (DI-NNs) as a
simple and effective way to model these two data types in a neural network. We
train and evaluate our proposed DI-NN on scenarios of varying difficulty and
realism and compare it against an alternative architecture, a classical Least-
Squares (LS) method as well as a classical Convolutional Recurrent Neural
Network (CRNN). Our results show that the DI-NN significantly outperforms the
baselines, achieving a five times lower localization error than the LS method
and two times lower than the CRNN in a test dataset of real recordings.

sound source localization,

multichannel audio processing,

multimodal machine learning,

convolutional recurrent neural networks,

###### keywords:

3GPP

    3rd Generation Partnership Project
aSLAM

    Acoustic Simultaneous Localization and Mapping
aSLAM

    Acoustic Simultaneous Localization and Mapping
A-SNR

    A-weighted SNR
AAC

    Advanced Audio Coding. A lossy codec used for digital audio.
AAD

    Auditory Attention Detection
AAS

    American Auditory Soc.
AASP

    Audio and Acoustic Signal Processing
ABC

    Analytical with or without Bias Compensation
ABR

    Auditory-Brainstem Response
ACAWD

    Archivable Core Actual-Word Database
ACB

    Adaptive Codebook
ACC

    Accuracy
ACE

    Acoustic Characterization of Environments. A noisy reverberant speech corpus and IEEE challenge run by the SAP group at Imperial College
ACELP

    Algebraic Code-Excited Linear Prediction
ACF

    Autocorrelation Function
ACK

    Acknowledgement
ACL

    Access Control List
ACR

    Absolute Category Rating
AD

    Audio Diarization
ADC

    Analogue-to-Digital Converter
ADM

    Adaptive Differential Microphone
ADPCM

    Adaptive Differential Pulse Code Modulation
ADSL

    Asymmetric Digital Subscriber Line
AE

    Almost Everywhere
AES

    Audio Engineering Society
AES

    Advanced Encryption Standard
AGC

    Automatic Gain Control
AH

    Amplitude Histogram
AI

    Articulation Index
AI

    Artificial Intelligence
AI

    Audio Inpainting
AIC

    Akaike Information Criterion
AIFF

    Audio Interchange File Format
AIR

    Acoustic Impulse Response
AIR

    Aachen Impulse Response
AIRD

    Aachen Impulse Response Database
ALC

    Automatic Level Control
ALCons

    Articulation Loss of Consonants
AM

    Amplitude Modulation
AMDF

    Average Magnitude Difference Function. A function with similar properties to the cross- or autocorrelation but that requires no multiplication to evaluate.
AMR

    Adaptive Multi-Rate
AMR-NB

    Adaptive Multi-Rate Narrow Band
AMR-WB

    Adaptive Multi-Rate Wide Band
AMS

    Amplitude Modulation Spectrogram
ANC

    Adaptive Noise Canceller
ANS

    Autocorrelation-Based Noise Subtraction
ANSI

    American National Standards Institute
ANU

    Australian National University
APLAWD

    Archivable Priority List Actual-Word Database
AR

    Autoregressive
ARD

    Arbeitsgemeinschaft der öffentlich-rechtlichen Rundfunkanstalten der Bundesrepublik Deutschland. ‘Consortium (“Working group”) of the public-law broadcasting institutions of the Federal Republic of Germany’
ARMA

    Autoregressive Moving Average
AS

    Audio Segmentation
AS

    Almost Surely
ASA

    Acoustic Scene Analysis
ASIO

    Audio Stream Input/Output. A computer soundcard protocol with low latency developed by Steinberg.
ASK

    Amplitude Shift Keying
ASLP

    Audio, Speech, and Language Processing
ASM

    Acoustic Scene Mapping
ASR

    Automatic Speech Recognition
ASS

    Approximate Spectrum Substitution
AST

    Acoustic Source Tracking
AST

    Asymmetric Sampling in Time
ATF

    Acoustic Transfer Function. The Fourier Transform of the RIR.
ATLM

    Acoustic Tokenization and Language Modelling
ATR

    Advanced Telecommunications Research Institute International, Kyoto, Japan
AUC

    Area under the curve
AURORA

    Aurora Experimental Framework for the Evaluation of the Performance of Speech Recognition Systems under Noisy Conditions
AUV

    Autonomous Underwater Vehicle
AV

    Audio-Visual
AWGN

    Additive White Gaussian Noise
AWS

    Approximate Waveform Substitution
BASIE

    Bayesian Adaptive Speech Intelligibility Estimation
BCE

    Blind Channel Estimation
BCL

    Bekesy Comfortable Loudness
BCR

    Block-Coordinate Relaxation
BEM

    Boundary Element Method
BER

    Bit Error Rate
BIBO

    Bounded-Input Bounded-Output
BIC

    Bayesian Information Criterion
Bk

    Berksons. A unit for measuring intelligibility.
B&K

    Brüel and Kjær
BLSTM

    Bidirectional LSTM
BM

    Blocking Matrix
BO-SCPHD

    Bearing-only SC-PHD
BO-SLAM

    Bearing-only SLAM
BOT

    Bearing-only tracking
BP

    Basis Pursuit
BPCC

    Basis Pursuit with Clipping Constraints
BPDN

    Basis Pursuit Denoising
BPM

    Beats Per Minute
BPSK

    Binary Phase Shift Keying
BR

    Barrodale and Roberts’ (algorithm)
BRI

    Basic Rate Index
BSD

    Bark Spectral Distortion
BSI

    Blind System Identification
BSIM

    Binaural Speech Intelligbility Model
BSS

    Blind Source Separation
BSTOI

    Binaural STOI
BW

    Bandwidth
BZ

    Back-to-Zero
C4DM

    Centre for Digital Music
$C_{\textrm{50}}$

    Clarity Index
C-GFB

    Combination Gas-fired Boiler
CART

    Classification and Regression Tree
CASA

    Computational Auditory Scene Analysis
CBR

    Constant Bit Rate
CCC

    Cross-Correlation Coefficient
CCCC

    DARPA CSR Corpus Coordinating Committee
CCD

    Charge-Coupled Device
CCI

    Call Clarity Index
CCITT

    Consultative Committee for International Telephony and Telegraphy
CCM

    Contralateral Competing Message
CCR

    Comparison Category Rating
CDB

    Constant Directivity Beamformer
CDMA

    Code Division Multiple Access
CELP

    Code-excited Linear Prediction
CHIEF

    Combined Helmholtz Integral Equation Formulation
CIT

    Constrained Initial Taps
CL

    Clipping Level
CLEAR

    Centre for Law Enforcement Audio Research
CLID

    Cluster Identification Test
CLT

    Central Limit Theorem
CMA

    Constant Modulus Algorithm
CMASI

    Coherence Modulated Acoustic Speckle Interferometry
CMB

    Cosmic Microwave Background
CNC

    Consonant-Nucleus-Consonant
CNG

    Comfort Noise Generation
CNN

    Convolutional Neural Network
CODEC

    Coder-Decoder
CPE

    Customer Premises Equipment
CPHD

    Cardinalized PHD
CRACD

    Codec-robust Automatic Clipping Detector
CRC

    Cyclic Redundancy Check
CRNN

    Convolutional Recurrent Neural Network
CS

    Channel Shortening
CS

    Compressive Sensing
CSP

    Communications and Signal Processing
CSR-WSJ

    Continuous Speech Recognition Wall Street Journal Phase 1 database
CST

    Connected Speech Test speech corpus
CT

    Conversation Test
CTTN

    Comparative Tolerance to Noise
CV

    Coefficient of Variation
CVNN

    Complex-Valued Neural Networks
CV

    Constant Velocity
CVC

    Consonant-Vowel-Consonant
CW

    Continuous Wave
CWM

    Centre-Weighted Median
CWMY

    Centre-Weighted Myriad
CWT

    Continuous Wavelet Transform
DAC

    Digital-to-Analogue Converter
DAM

    Diagnostic Acceptability Measure
DAQ

    Data Acquisition
DARPA

    Defense Advanced Research Projects Agency of the United States Dept. of Defense
DARPA-RMD

     DARPA 1000-Word Resource Management Database for Continuous Speech Recognition
DAW

    Digital Audio Workstation
dB

    Decibel
dBFS

     dB Full Scale
DBN

    Deep Belief Network
DBSTOI

    Deterministic BSTOI
DC

    Direct Current
DCME

    Digital Circuit Multiplexing Equipment
DCR

    Degradation Category Rating
DCT

    Discrete Cosine Transform
DDR

    Direct-to-diffuse ratio
DDR3

    Double Data Rate Type Three
DECT

    Digital European Cordless Telecommunication
DeLILAH

    Detection of Clipping using Least Squares Residuals and Iterated Logarithm Amplitude Histogram
DENBE

     DRR Estimation using a Null-Steered Beamformer
DET

    Detection Error Trade-off
Dev

    Development dataset of the ACE Challenge
DFT

    Discrete Fourier Transform
DI

    Directivity Index
DI-NN

    Dual-Input Neural Network
DirectX

    A programming interface developed by Microsoft for handling tasks related to multimedia.
DIRHA

    Distant-speech Interaction for Robust Home Applications multi-microphone multi-language acoustic speech corpus
DMA

    Differential Microphone Array
DMT

    Discrete Multi-Tone
DMV

    Dynamically Managed Voice system
DNN

    Deep Neural Network
DNR

    Dynamic Noise Reduction
DoA

    Direction-of-Arrival
DOA

    Direction-of-Arrival
DP

    Dynamic Programming
DPCM

    Differential Pulse Code Modulation
DPD

    Direct-Path Dominance
DPD-MUSIC

    Direct-Path Dominance Multiple Signal Classification
DR

    Douglas-Rachford
DR2

    Dynamic Range
DRM

    Diagnostic Rhyme Test
DRR

    Direct-to-Reverberant Ratio
DRNN

    Deep Recurrent Neural Net
DRT

    Diagnostic Rhyme Test
DSB

    Delay-and-Sum Beamformer
DSOBM

    Deterministic SOBM
DSP

    Digital Signal Processing
DSPS

    Double Sides Periodic Substitution
DSR

    Distributed Speech Recognition
DSWS

    Double Sides Waveform Substitution
DTMF

    Dual Tone Multi-Frequency
DTX

    Discontinued Transmission
DWT

    Discrete Wavelet Transform
EARS

    Embodied Audition for RobotS
EBF

    Eigen-beamformer
EBU

    European Broadcasting Union
EC

    Echo Canceller
EDC

    Energy Decay Curve
EDR

    Energy Decay Relief
EDF

    Energy Decay Function
EEG

    Electroencephalography
EER

    Equal Error Rate
EFICA

    Efficient Fast Independent Component Analysis
EIR

    Equalized Impulse Response
EKF

    Extended Kalman Filter
EKF-SLAM

     EKF SLAM
EKF-SLAM

    Extended Kalman Filter SLAM
EL

    Echo Loss
ELF

    Extremely Low Frequency
ELRA

    European Languages Research Association
EM

    Estimation-Maximization. An iterative technique to solve certain optimization problems.
EMIB

    Eigenmike Microphone Interface Box
ENF

    Electrical Network Frequency
EPSRC

    Engineering and Physical Sciences Research Council
EQ

    Equalisation
ERB

    Equivalent Rectangular Bandwidth
ERP

    Ear Reference Point (cf. ITU-T Rec. P.64 1999)
ESA

    Early Stage Assessment
ESM

    Equivalent Source Method
ESPRIT

    Estimation of Signal Parameters via Rotational Invariance Techniques
ETAN

    Equivalent Tolerance to Additional Noise
ETSI

    European Telecommunications Standards Institute
EURASIP

    European Association for Signal Processing
EUSIPCO

    European Signal Processing Conference
Eval

    Evaluation dataset of the ACE Challenge
F1

    F1 Score
FAR

    False Alarm Rate
FastSLAM

    FActored Solution To Simultaneous Localization and Mapping
FastSLAM

    FActored Solution To SLAM
FAU

    Friedrich-Alexander-Universität
FB

    Forward-Backward
FB

    Fullband
FBF

    Fixed Beamformer
FBSS

    Forward-Backward Spatial Smoothing
FCC

    Federal Communications Commission
FC-NN

    Fully Connected Neural Network
FDM

    Frequency Division Multiplexing
FDR

    False Discovery Rate
FDR

    Free-Decay Region
FEC

    Forward Error Correction
FEM

    Finite Element Method
FFI

    Norwegian Defence Research Establishment
FFT

    Fast Fourier Transform
FIFO

    First-In First-Out
FIR

    Finite Impulse Response. A filter whose output is a weighted sum of past input values and whose system function contains only zeros and no poles.
FISM

    Fast Image Source Method
FISST

    Finite Set STatistics
FLOM

    Fractional Lower-Order Moments
FLOS

    Fractional Lower-Order Statistics
FM

    Frequency Modulation
FMM

    Fast Multipole Method
FN

    False Negative
FN2

    Nth Formant
FNR

    False Negative Rate
FORTRAN

    The IBM Mathematical Formula Translating System
FoV

    Field of View
FP

    False Positive
FPR

    False Positive Rate
FPS

    Frames Per Second
FRI

    Finite Rate of Innovation
FSB

    Filter-and-Sum Beamformer
FSK

    Frequency Shift Keying
FT

    Flat-Top
FWER

    Familywise Error Rate
FWSSNR

    Frequency-Weighted Segmental SNR
FWSSRR

    Frequency-Weighted SSRR
G.711

     PCM of Voice Frequencies
GARCH

    Generalized Auto-regressive Conditional Heteroscedasticity
GBW

    Gain Bandwidth Product
GCC

    Generalized Cross-Correlation
GCC-PHAT

    Generalized Cross-Correlation with Phase Transform method of estimating TDoA
GCF

    Global Coherence Field
GGD

    Generalized Gaussian Distribution
G$\Gamma$D

    Generalised Gamma Distribution
GM

    Gaussian Mixture
GM-PHD

    Gaussian Mixture PHD
GMCA

    Generalized Morphological Component Analysis
GMM

    Gaussian Mixture Model. An approximation to an arbitrary probability density function that consists of a weighted sum of Gaussian distributions
GNSS

    Global Navigation Satellite System
GPRS

    General Packet Radio Services
GPS

    Global Positioning System
GSC

    Generalized Sidelobe Canceller
GSM

    Global System for Mobile Communications
GSM-EFR

     GSM Enhanced Full Rate Codec
GSM-FR

     GSM Full Rate Codec
GSM-HR

     GSM Half Rate Codec
GUI

    Graphical User Interface
HAAC

    High Amplitude Audio Capture
HATS

    Head and Torso Simulator
HERB

    Harmonicity-based dEReverBeration
HFT

    Hands-Free Terminal
HI

    Hearing-Impaired
HIE

    Helmholtz Integral Equation
HISAS

    High resolution Interferometric Synthetic Aperture  (SAS)
HINT

    Hearing-in-Noise Test
HLT

    Human Language Technology
HMM

    Hidden Markov Model
HOS

    Higher-Order Statistics
HPF

    High-Pass Filter
HR

    Half Rate (GSM Codec)
HRI

    Human-Robot Interaction
HRTF

    Head-related Transfer Function
HSA

    Hearing, Speech, Audio technology group at Fraunhofer IDMT
HSD

    Hybrid Steepest Descent
HT

    Hannan-Thomson
HTK

    Hidden Markov Model Tool Kit
IBM

    Ideal Binary Mask
IC

    Interference Canceller
ICA

    Independent Component Analysis
ICASSP

    Intl. Conf. on Acoustics, Speech and Signal Processing
ID

    Identifier
IDMT

    Institute for Digital Media Technology
iDEN

    Integrated Digital Enhanced Network
IEC

    International Electrotechnical Commission
IEEE

    Institute of Electrical and Electronics Engineers
IET

    Institute of Engineering and Technology
IETF

    Internet Engineering Task Force
IFFT

    Inverse Fast Fourier Transform
IHC

    Inner Hair Cell
IHT

    Iterative Hard Thresholding
IID

    Independent and Identically Distributed
IIR

    Infinite Impulse Response. A filter whose output is a weighted sum of both past input and past output values and whose system function contains both poles and zeros.
IL

    Iterated Logarithm, the logarithm of the logarithm
ILAH

    Iterated Logarithm Amplitude Histogram clipping detection method
ILD

    Interaural Level Difference
IMCRA

    Improved Minima Controlled Recursive Averaging. A technique for blindly estimating the spectrum of additive noise in a signal.
IMD

    Inter-Modulation Distortion
IMSI

    International Mobile Subscriber Identity
IMU

    Inertial Measurement Unit
INMD

    In-service Non-intrusive Measurement Device
INTERSPEECH

    Annual Conference of the ISCA
IO

    Infinitely Often
IP

    Internet Protocol
IPA

    International Phonetic Association
IPD

    Interaural Phase Difference
IRM

    Ideal Ratio Mask
IRS

    Inverse repeated Sequence. A pseudo random sequence used for impulse response measurement.
IRS

    Intermediate Reference System
IS

    Importance Sampling
ISCA

    International Speech Communication Association
ISDN

    Integrated Services Digital Network
ISFT

    Inverse Spherical Fourier Transform
ISO

    Intl. Organization for Standardization
ISP

    Intensity Spectral Profile
IST

    Iterative Soft Thresholding
ISTFT

    Inverse Short Time Fourier Transform
ISVR

    Institute of Sound and Vibration Research, Southampton University, UK
ITD

    Interaural Time Difference
ITF

    Interaural Transfer Function
ITU

    International Telecommunication Union
ITU-R

    International Telecommunication Union Radiocommunication Sector
ITU-T

    International Telecommunication Union Telecommunication Standardisation Sector
IUWT

    Isotropic Undecimated Wavelet (Starlet) Transform
IWAENC

    Intl. Workshop on Acoustic Signal Enhancement
IWAENC

    Intl. Workshop Acoustic Echo and Noise Control
IWASE

    Intl. Workshop on Acoustic Signal Enhancement
JADE

    Joint Approximate Diagonalization of Eigen-Matrices
JPDA

    Joint Probabilistic Data Association
JPEG

    Joint Photographic Experts Group
KDE

    Kernel Density Estimate
KF

    Kalman Filter
KFD

    Kernel Fisher Discriminant Analysis
KL

    Karhunen-Loéve
KLT

    Karhunen-Loéve Transform
KST

    Kolmogorov-Smirnov Test
LAD

    Least Absolute Deviation
LAN

    Local Area Network
LARS

    Least Angle Regression
L${}_{\textrm{AT}}$

    Equivalent Continuous Sound Level. Also called Leq
LBR

    Low Bitrate Redundancy
LC

    Local Criterion
LCMP

    Linearly Constrained Minimum Power
LCMV

    Linearly Constrained Minimum Variance
LCWM

    Linear Combination of Weighted Medians
LDC

    Linguistic Data Consortium
LEM

    Loudspeaker-Enclosure-Microphone System
L${}_{\textrm{eq}}$

    Equivalent Continuous Sound Level. Also called LAT
LF

    Liljencrants-Fant. The developers of a glottal waveform model
LHS

    Left-Hand Side
LID

    Language Identification
LILAH

     LSR-ILAH clipping detection method
LIME

    LInear Predictive Multi-input Equalization algorithm
LiNoPS

    Lightweight Noise Protection System
LLN

    Law of Large Numbers
LLR

    Log-Likelihood Ratio
LLS

    Logarithmic Least Squares
LMA

    Least Mean Absolute
LMS

    Least Mean Squares adaptive filter
LNA

    Low Noise Amplifier
LoS

    Line-of-Sight
LOT

    Listening-Only Test
LP

    Linear Parameter
LP

    Linear Predictive
LP

    Linear Prediction
LPC

    Linear Predictive Coding. An autoregressive model of speech production.
LQO

    Listening Quality Objective
LREC

    Conf. on Language Resources and Evaluation
LS

    Least-Squares
LSA

    Log Spectral Amplitude
LSB

    Lower Side-Band
LSB

    Least Significant Bit
LSD

    Log Spectral Distortion
LSP

    Line Spectrum Pairs
LSR

    Late Stage Review
LSR

    Least Squares Residuals clipping detection method
LSRT

    Least Squares Residuals with Thresholding clipping detection method
LSTM

    Long Short-Term Memory
LTASS

    Long Term Average Speech Spectrum
LTI

    Linear Time Invariant
LTP

    Long Term Prediction
LU

    Loudness Unit
LUFS

    Loudness Units Full-Scale
MA

    Moving Average
MAC

    Multiply Accumulate Operation
MAD

    Median Absolute Deviation
MAE

    Mean Absolute Error
MAP

    Maximum _a posteriori_
MARDY

    Multichannel Acoustic Reverberation Database at York
MARS

    Multivariate Adaptive Regression Splines
MBF

    Matched Filter Beamformer
MC

    Monte Carlo
MCA

    Morphological Component Analysis
MCC

    Matthew’s Correlation Coefficient
MCEQ

    MultiChannel EQualisation
MCS

    Multidimensional Colouration Space
MDCT

    Modified Discrete Cosine Transform
MDL

    Minimum Description Length
MDS

    Multidimensional Scaling
Mel

    A non-uniform frequency scale corresponding to perceived frequency. It is approximately linear at low frequencies and logarithmic at high frequencies.
MELP

    Mixed Excitation Linear Prediction
MFCC

    Mel-frequency Cepstral Coefficients
MFS

    Method of Fundamental Solutions
MFSK

    Multi-Frequency Shift Keying
MHT

    Multi-Hypotheses Tracking
MI

    Mutual Information
MIMO

    Multiple-Input-Multiple-Output
MINT

    Multiple-input/output INverse Theorem
MIRS

    Motorola Integrated Radio System
MIT

    Massachusetts Institute of Technology
MIT-LCS

    Massacchusetts Institute of Technology Laboratory for Computer Science
ML

    Maximum Likelihood
MLD

    Masking Level Difference
MLE

    Maximum Likelihood Estimation
ML-TDoA

    Maximum Likelihood Time Difference of Arrival
MLMF

    Machine Learning with Multiple Features
MLP

    Multi-layer Perceptron
MLS

    Maximum Length Sequence of pseudo random bits.
MMSE

    Minimum Mean Squared Error
MMT

    Multiscale Median Transform
MNRU

    Modulated Noise Reference Unit
MOM

    Mean of Maximum
MOS

    Mean Opinion Score
MOS-LQO

    Mean Opinion Score - Listening Quality Objective
MP

    Matching Pursuit
MP3

     MPEG-2 Audio Layer III
MPEG

    Moving Picture Experts Group
MRF

    Markov Random Field
MRP

    Mouth Reference Point (cf. ITU-T Rec. P.64 1999)
MRT

    Modified Rhyme Test
MS

    Minimum Statistics
MSB

    Most Significant Bit
MSC

    Mean Square Coherence
MSE

    Mean Square Error
MSN

    Multiple Subscriber Number
MSNR

    Maximum SNR
MTF

    Modulation Transfer Function
MTM

    Modified Trimmed Mean
MUSCLE

    MeasUred Single-CLustEr
MUSHRA

    Multi-stimuli Test with Hidden Reference and Anchor
MUSHRAR

    Multi-stimuli Test with Hidden Reference and Anchor for Reverberation
MUSIC

    Multiple Signal Classification
MVDR

    Minimum Variance Distortionless Response beamformer
MWF

    Multi-channel Wiener Filter
NAH

    Nearfield Acoustic Holography
NB

    Narrowband
NCM

    Normalized Coherence Metric
NH

    Normal-Hearing
NIRA

    Non-Intrusive Room Acoustics
NISE

    Non-Intrusive SNR estimation
NISI

    Non-Intrusive Speech Intelligibility Estimation
NISQ

    Non-Intrusive Speech Quality Estimation
NIST

    National Institute of Standards and Technology
NL

    Noise Level
NLA

    Non-Linear Approximation
NLMS

    Normalized Least Mean Squares adaptive filter
NMCFLMS

    Normalized Multichannel Frequency Domain Least Mean Square
NMF

    Non-negative Matrix Factorization
NOISEX-92

    Database to Study the Effect of Additive Noise on Speech Recognition Systems
NOIZEUS

    Noisy Speech Corpus for Evaluation of Speech Enhancement Algorithms
NOSRMR

    Normalized Overall SRMR
NOS

    Number of Sources
NOSRMR

    Normalized Overall SRMR
NPM

    Normalized Projection Misalignment
NPV

    Negative Predictive Value
NR

    Noise Reduction
NS

    Noise Suppression
NSRMR

    Normalised SRMR
NSRR

    Normalized Signal-to-Reverberation Ratio
NSRMR

    Normalised SRMR
NSV

    Negative-Side Variance
NTP

    Network Time Protocol
NV

    Noise-Vocoding
OBL

    Octave Band Level
ODF

    Overdrive Factor
Ofcom

    Office of Communications, the independent regulator and competition authority for the UK communications industries
OFDM

    Orthogonal Frequency Division Multiplexing
OHC

    Outer Hair Cell
OIM

    Objective Intelligibility Measure
OLA

    Overlap-add
OM-LSA

    Optimally Modified Log-Spectral Amplitude Estimator
OMP

    Orthogonal Matching Pursuit
OSI

    Open Systems Interconnection
OSPA

    Optimal Subpattern Assignment
OSRMR

    Overall SRMR
PAB-SRMR

    Per acoustic band SRMR
PALM

    Passive Acoustic Localization and Mapping
PAMS

    Perceptual Analysis Measurement System
PARCOR

    Partial Correlation Coefficients
PB

    Phonetically Balanced
PBF

    Positive Boolean Function
PCA

    Principal Components Analysis
PCM

    Pulse-Code Modulation
PDA

    Personal Digital Assistant
PDA

    Probabilistic Data Association
PDE

    Partial Differential Equation
pdf

    Probability Density Function
PDF

    Probability Density Function
PE

    Parameter Estimation
PEASS

    Perceptual Evaluation for Audio Source Separation
PEFAC

    Pitch Estimation Filter with Amplitude Compression
PESQ

    Perceptual Evaluation of Speech Quality
PF

    Psychometric Function
p.g.fl.

    Probability Generating Functional
PHAT

    Phase Transform
PHD

    Probability Hypothesis Density
PIP

    Peak-Image Pairing
PIV

    Pseudo-Intensity Vector
PL

    Pseudo-likelihood
PLC

    Packet Loss Concealment
PLL

    Phase Locked Loop
PLP

    Perceptual Linear Prediction
PLR

    Perceived Level of Reverberation
PM

    Phase Modulation
PMF

    Probability Mass Function
PMOS

    Predicted Mean Opinion Score
PNN

    Parameterized Neural Network
POLQA

    Perceptual Objective Listening Quality Analysis
POTS

    Plain Old Telephone Service
PPP

    Poisson Point Process
PPS

    Pulse-Per-Second
PPV

    Positive Predictive Value
PReLU

    Parametric Rectified Linear Unit
PRLM

    Phoneme Recognition and Language Modelling
PRP

    Pair-wise Relative Phase-ratio
PSD

    Power Spectral Density
PSK

    Phase Shift Keying
PSNR

    Peak Signal-to-Noise Ratio
PSOLA

    Pitch Synchronous Overlap Add. A method of scaling a signal in time and pitch independently.
PSQM

    Perceptual Speech Quality Measurement
PSSL

    Positional Sound Source Localization
PSTN

    Public Switched Telephone Network
PWD

    Plane-Wave Decomposition
PTA

    Pure-Tone Audiology
QAM

    Quadrature Amplitude Modulation
QILAH

    Quadrisected Iterated Logarithm Amplitude Histogram clipping detection method
QMF

    Quadrature Mirror Filter
QoE

    Quality-of-Experience
QoS

    Quality-of-Service
QPSK

    Quadrature Phase Shift Keying
RADAR

    RAdio Detection And Ranging
RASTA

    Relative Spectral
RASTA-PLP

    Relative Spectral Perceptual Linear Prediction
RASTI

    Room Acoustics Speech Transmission Index (superseded by STIPA)
RBM

    Restricted Boltzmann Machine
RB-PHD

    Rao-Blackwellised PHD
RBPF

    Rao-Blackwellised Particle Filter
RC

    Relative Criterion
$R_{\textrm{DT}}$

    Reverberation Decay Tail
RDTF

    Relative Direct Transfer Function
ReLU

    Rectified Linear Unit
RF

    Radio Frequency
RFI

    Radio Frequency Interference
RFS

    Random Finite Set
RHS

    Right-Hand Side
RIP

    Restricted Isometry Property
RIR

    Room Impulse Response
RLS

    Recursive Least Squares adaptive filter
RLSD

    Relative Log Spectral Distortion
RMCLS

    Relaxed Multichannel Least Squares
RMCLS-CIT

    Relaxed MultiChannel Least-Squares with Constrained Initial Taps
RMS

    Root Mean Square
RMSE

    Root Mean Square Error
RNN

    Recurrent Neural Network
ROC

    Receiver Operating Characteristic
ROHC

    Robust Header Compression
RP

    Received Pronunciation
RPE

    Regular Pulse Excitation
RRTF

    Relative Real-Time Factor
RS

    Reverberation Suppression
RSM

    Reflector Source Method
RSS

    Received Signal Strength
RSV

    Room Spectral Variance
RT

    Reverberation Time
RTAN

    Robustness to Additional Noise
RTF

    Real-Time Factor
RTF

    Room Transfer Function
RTF

    Relative Transfer Function
RV

    Random Variable
RVP

    Recursive Vector Projection
RWTH

    Rheinisch-Westfälische Technische Hochschule
$S_{50}$

    Intelligibility Function Gradient at the Speech Reception Threshold (SRT)
SA

    Spectral Amplitude
SAA

    Synthetic Aperture Audio
SAP

    Speech And Audio Processing
SAR

    Speech-to-Artifact Ratio
SAR

    Speaker Alternation Rate
SAR

    Synthetic Aperture RAdio Detection And Ranging (RADAR)
SAS

    Synthetic Aperture SOund Navigation And Ranging (SONAR)
SB

    Subband
SC-PHD

    Single Cluster PHD
SCAF

    Single Channel Adaptive Filter
SCB

    Stochastic Codebook
SCNR

    Single-Channel Noise Reduction
SCOT

    Smoothed Coherence Transform
SCNR

    Single-Channel Noise Reduction
SC-PHD

    Single Cluster PHD
SCR

    Signal-to-Competition Ratio
SCRIBE

    Spoken Corpus of British English
SCT

    Speech Corruption Toolkit
SCT2

    Short Conversation Test
SD

    Semantic Differential
SDB

    Superdirective Beamformer
SDD

    Spectral Decay Distributions
SDDMSB

     SDD with Mel-spaced frequency bands
SDDSA

     SDD with Mel-spaced frequency bands and selective averaging
SDDSA-G

     SDDSA with Gerkmann noise estimator
SDDSA-H

     SDDSA with Hendriks noise estimator
SDR

    Software Defined Radio
SDR2

    Speech
SDRAM

    Synchronous Dynamic Random Access Memory
SDT

    Speech Description Taxonomy
SEDF

    Subband Energy Decay Function (EDF)
SELD

    Sound Event Localization and Detection
SEMG

    Surface Electromyography
SFDR

    Spurious Free Dynamic Range
SFM

    Single Feature with Mapping
SFT

    Spherical Fourier Transform
SH

    Spherical Harmonic
SHD

    Spectral Harmonic Decomposition
SHD

    Spherical Harmonic Domain
SIE

    System Identification Error
SII

    Speech Intelligibility Index
SIImod

    Speech Intelligibility Index in the modulation domain
SIM

    Subscriber Identity Module
SIMO

    Single-Input-Multiple-Output
SINAD

    Signal-to-Noise and Distortion Ratio
SIP

    Session Initiation Protocol
SIR

    Signal-to-Interference Ratio
SIR

    Sequential Importance Resampling
SIREAC

    Simulation of REal Acoustics Software Tool
SIS

    Sequential Importance Sampling
SL

    Speech Level
SLAM

    Simultaneous Localization and Mapping
SLLN

    Strong Law of Large Numbers
SLM

    Sound Level Meter
SMA

    Spherical Microphone Array
SMARD

    Single- and Multichannel Audio Recordings Database
SMERSH

    Spatiotemporal Averaging Method for Enhancement of Reverberant Speech
SMIR

    Spherical Microphone array Impulse Response
SMPTE

    Society of Motion Picture and Television Engineers
SMS

    Short Message Service
SNR

    Signal-to-Noise Ratio
SNR

    Speech-to-Noise Ratio
SNT

    Subspace Noise Tracking algorithm
SOBM

    STOI-optimal Binary Mask
SONAR

    SOund Navigation And Ranging
SOLA

    Synchronous Overlap Add. A method of scaling a signal in time and pitch independently.
SPC

    Specificity
SPEECON

    Speech Databases for Consumer Devices
SPHERE

    NIST SPeech Header REsources software with embedded Shorten Compression
SPIN

    Speech Perception In Noise
SPL

    Sound Pressure Level
SPP

    Speech Presence Probability
SPQA

    Speech Quality Assurance Package
SQNR

    Signal-to-Quantization Noise Ratio
SR

    Sparse Representation
SR

    Spectral Rotation
SRA

    Statistical Room Acoustics
SRI

    SRI International. Formerly Standford Research Institute
SRMR

    Speech-to-Reverberation Modulation Energy Ratio
SRP

    Steered Response Power
SRP-PHAT

    Steered Response Power with Phase Transform
SRP-TDE

    Steered Response Power with Time Delay Estimation
SRR

    Signal-to-Reverberation Ratio
SRT

    Speech Reception Threshold (also known as Speech Recognition Threshold)
SS

    Spectral Subtraction
SSB

    Single Side-Band
SSI

    Synthetic Sentence Identification
SSL

    Sound Source Localization
SSN

    Simultaneous Switching Noise
SSN

    Speech-Shaped Noise
SSNR

    Segmental SNR
SSOBM

    Stochastic SOBM
SSRR

    Segmental Signal-to-Reverberation Ratio
SSW

    Staggered Spondaic Word
STFT

    Short Time Fourier Transform
STI

    Speech Transmission Index
STIPA

    Speech Transmission Index for Public Address Systems
STITEL

    Speech Transmission Index for Telecommunication Systems
STMI

    Spectro-Temporal Modulation Index
STNR

     NIST’s Speech-to-Noise Ratio Estimation Algorithm
STOI

    Short-Time Objective Intelligibility measure
STQ

    Speech Processing, Transmission and Quality Aspects
STSA

    Short Time Spectral Analysis
STSA

    Short Time Spectral Amplitude
SUS

    Semantically Unpredictable Sentences
SVD

    Singular Value Decomposition
SVM

    Support Vector Machine
SWSOBM

    Stochastic WSOBM
$T_{\textrm{20}}$

    Reverberation Time to decay by $20$ dB
$T_{\textrm{30}}$

    Reverberation Time to decay by $30$ dB
$T_{\textrm{60}}$

    Reverberation Time to decay by $60$ dB
TBM

    Target Binary Mask
TDE

    Time Delay Estimation
TDHS

    Time Domain Harmonic Scaling. A method of scaling a signal in time and pitch independently.
TDOA

    Time-Difference-of-Arrival
TDT

    Tone Decay Test
TF

    Time-Frequency
TFS

    Temporal Fine Structure
TFGM

    Time-Frequency Gain Modification. An approach to signal enhancement in which a signal is multiplied by a gain function in the time-frequency domain.
THD

    Total Harmonic Distortion
TI

    Texas Instruments, Inc.
TIMIT

     TI-MIT speech corpus
TIPHON

    Telecommunication and Internet Protocol Harmonization Over Networks
TLS

    Total Least-Squares
TN

    True Negative
TNR

    True Negative Rate
TOA

    Time-of-Arrival
TOF

    Time-of-Flight
TOSQA

    Telekom Objective Speech Quality Assessmentt
TP

    True Positive
TP

    Trivial Pursuit
TPCC

    Trivial Pursuit with Clipping Constraints
TPR

    True Positive Rate
TSE

    Taylor Series Expansion
TVAR

    Time-varying Autoregression
UAV

    Unmanned Aerial Vehicle
UDP

    User Datagram Protocol
UFRJ

    Federal University of Rio de Janeiro
UHF

    Ultra High Frequency
UKF

    Unscented Kalman Filter
ULA

    Uniform Linear Array
ULF

    Ultra Low Frequency
UMTS

    Universal Mobile Telecommunications Service
US

    United States
UTBM

    Universal Target Binary Mask
VAD

    Voice Activity Detector
VBR

    Variable Bit-Rate
VCV

    Vowel-Consonant-Vowel
VRD

    Variance of Decay-rates
VGC

    Voice Grade Channel
vMF

    von Mises-Fisher
VoIP

    Voice Over Internet Protocol
VRT

    Vlaamse Radio- en Televisieomroeporganisatie. (Flemish Radio and Television Broadcasting Organization)
VSELP

    Vector Sum-excited Linear Prediction
VST

    Virtual Studio Technology. An interface standard developed by Steinberg for adding plugins to an audio editor.
WADA

    Waveform Amplitude Distribution Analysis
WASN

    Wireless Acoustic Sensor Network
WASPAA

    Workshop on Applications of Signal Processing to Audio and Acoustics
WAV

    Waveform Audio File Format
WAVE

    Waveform Audio File Format
WB

    Wideband
WER

    Word Error Rate
WGN

    White Gaussian Noise
WLAN

    Wireless LAN
WLLN

    Weak Law of Large Numbers
WM

    Working Memory
WMA

    Windows Media Audio
WNG

    White Noise Gain
WSS

    Weighted Spectral Slope
WSOBM

    Weighted SOBM
WSTOI

    Weighted STOI
ZOS

    Zero-Order Statistics

Methodology

##  1 Introduction

Most signals, such as audio and images, contain metadata. Metadata can be
signal-based, which describes quantitative properties of the signal, such as
its sampling rate, as well as semantic, which describes, for example,
contextual properties. In speech processing, semantic metadata could consist
of the speaker’s language or gender. Whether signal-based or semantic,
including metadata as a secondary input into neural network models may provide
relevant information which would translate into an economy of training time,
model parameters and flexibility. However, metadata typically has a different
dimensionality than the input signals, making its incorporation into those
models not trivial.

The main focus of this paper is to study the effectiveness of schemes to
process signals and exploit metadata jointly using neural network models. We
focus on the task of Sound Source Localization (SSL) [1] using distributed
microphone arrays to demonstrate the effectiveness of our proposed approach.
In the context of SSL, relevant metadata which is exploited by classical
methods is the microphone positions, which can be acquired by manual
measurement or using self-calibration [2] methods. Other relevant metadata is
the room dimensions and its reverberation time.

SSL refers to the task of estimating the spatial location of a sound source,
such as a human talker or a loudspeaker. In this scenario, metadata refers to
properties of the acoustic scene such as the coordinates of microphones,
dimensions of the room and, the reflection coefficient of the walls. SSL has
many applications, including noise reduction and speech enhancement [3],
camera steering [4] and acoustic Simultaneous Localization and Mapping (SLAM)
[5]. In turn, distributed microphone arrays have become an active research
topic in the signal processing community due to their versatility. Such arrays
may be composed of multiple network-connected devices, including everyday
devices such as cell phones, smart assistants, and laptops, for example. The
array and the constituent devices may be configured as a Wireless Acoustic
Sensor Network (WASN) [6].

SSL approaches may be divided into classical signal processing-based and data-
driven neural network-based methods. By explicitly exploiting metadata
describing microphone positions and room dimensions, classical approaches may
be applied to different rooms and microphone configurations. Conversely,
neural network approaches have recently achieved state of the art results for
source localization [7, 8, 9], at the expense of requiring one network to be
trained for every microphone topology. One reason current neural approaches do
not incorporate the microphones’ positional information is that the
microphones’ signal and positional data are very different from one another in
nature and dimension.

Previous work which discusses the joint processing of signals and metadata is
[10], where a single input neural network is used to process metadata in
conjunction with a low-dimensional physical signal. However, unlike our work,
the method of [10] is restricted to multilayer perceptron architectures and
one-dimensional input and metadata, limiting its application in practical
scenarios.

Another related field is multimodal fusion [11, 12], although this is usually
concerned with learning representations using two types of signals, such as
audio-visual data. Simultaneously processing signals and metadata have also
been explored using non-neural models for sound source separation [13], where
metadata consists of information about the type of sound (speech, music) and
how the sources were mixed. However, none of the existing work discusses
effective schemes for incorporating and evaluating signals and metadata of
different dimensionality.

Figure 1: Overview of the DI-NN approach.

Our main contribution is the DI-NN neural network architecture, which is
capable of processing high-dimensional signals, namely spectrograms, along
with a relevant metadata vector of lower dimensionality. An overview diagram
of our approach is shown in Fig. 1, which will be discussed in subsection 3.2.
We compare our method against three baselines for the task of Positional Sound
Source Localization (PSSL), namely, a metadata-unaware Convolutional Recurrent
Neural Network (CRNN), a metadata-aware classical signal processing approach,
as well as an alternate metadata-aware neural network. Our proposed method is
able to outperform all baselines by a large margin in realistic scenarios. In
contrast to previous approaches [14, 9], our network dispenses with the need
for training a network for each scenario, broadening our method’s
applicability.

This work continues as follows. In section 2, an overview of neural and non-
neural SSL methods will be discussed. The approach for training our proposed
DI-NN for SSL is described together with several baseline methods in section
3. In section 4, the experiments comparing our approach with the baselines
using multiple datasets are described. Finally, results and conclusions are
drawn in section 5.

##  2 Prior art on sound source localization

###  2.1 Neural-based methods

In recent years, deep neural networks have been widely adopted for the task of
sound source localization. The various approaches differ in the input features
used, the network architectures and output strategies. Most studies focus on
the task of Direction-of-Arrival (DOA) estimation, i.e., estimating the angle
between the propagation direction of the acoustic wavefront due to the source
and a reference axis of the array.

Practicioners have experimented with many types of neural input features, such
as the raw audio samples of the microphone signals [9], their frequency-domain
representation through the Short Time Fourier Transform (STFT) [15], their
cross-spectra [16] or cross-correlation [8]. Multiple architectures have been
also tested, including the Multi-layer Perceptron (MLP) [8], Convolutional
Neural Networks [17] and residual networks [18]. In this work, we focus on the
Convolutional Recurrent Neural Network (CRNN) architecture, which has received
widespread adoption in the field [19, 7, 20]. Finally, approaches differ in
terms of the network’s output strategy. While regression-based approaches
directly estimate the source’s coordinates, classification based-approaches
discretize the source locations to a grid of available positions. We refer to
[21] for a discussion on the merits of both approaches. We also refer the
reader to a substantial survey of neural SSL papers [22].

In this paper, we focus on the task of estimating the absolute Cartesian
coordinates of the source, which we shall refer to as Positional Sound Source
Localization (PSSL), and has applications in robot navigation [5] and noise
reduction [23]. The PSSL task has been much less studied using neural methods.
To the best of our knowledge, only [14] and [9] focus on PSSL. However, both
these approaches only work for the same room with fixed relative microphone
positions. We believe this shortage of studies to be at least in part due to
the lack of an architecture capable of incorporating the scene’s metadata,
which is addressed by our proposed DI-NN. We also refer to the recent L3DAS22
challenge [24], where practitioners were invited to develop 3D PSSL algorithms
for a realistic office environment containing a pair of microphone arrays.

###  2.2 Classical signal processing methods

Classical approaches to SSL have been widely studied within the signal
processing community. In PSSL approaches, the source’s coordinates are
estimated using a model involving signal processing, physics and geometry. By
measuring differences in the microphone signals’ amplitudes and phases,
distance metrics between the microphones and source can be estimated. These
estimates can in turn be combined to estimate the source’s coordinates [1].
Besides the microphones’ signals, the positions of microphones are usually
needed for the position of the source to be estimated. Available approaches
for SSL may be classified as delay-based [25, 1], energy-based [26, 27],
subspace-based [28] and beamforming-based [29, 30] approaches. We shall focus
on delay-based approaches and will provide background for our baseline method.

Delay-based SSL methods usually rely on computing the Time-Difference-of-
Arrival (TDOA) between each microphone pair within the system, which
corresponds to the difference in time taken for the source signal to propagate
to different microphones. The locus of candidate source positions with the
same TDOA with respect to a microphone pair is, when considering planar
coordinates, a hyperbola [1, 25]. The source is located at the intersection of
the hyperbolae defined by all microphone pairs. The multiple TDOAs can be
combined using a Least-Squares (LS) framework [31], or using a Maximum
Likelihood (ML) approach if some noise properties of the system are known [1].
In general, TDOAs are estimated using cross-correlation based methods such as
Generalized Cross-Correlation with Phase Transform (GCC-PHAT) [32], which are
shown to be somewhat robust to reflections produced in the room due to, for
example, the walls, ceiling and furniture, i.e. reverberation [33].

##  3 Method

###  3.1 Signal model and scope of this work

Our scope is restricted to the localization of a static source at the planar
coordinates $\boldsymbol{p}_{s}=[p_{s}^{x},p_{s}^{y}]^{T}$. The source emits
an intermittent signal $s(t)$ at time $t$. In our experiments, $s(t)$ may
consist of White Gaussian Noise (WGN) as well as of speech utterances. Also,
$M$ static microphones with known positions are present in the room, each
placed at coordinates $\boldsymbol{m}_{i}=[m_{i}^{x},m_{i}^{y}]^{T}$. Both
source and microphones are enclosed in a room of planar dimensions
$\boldsymbol{d}=[d^{x},d^{y}]^{T}$. The amount of reverberation in the room is
modeled by its reverberation time $r$, a measure of the amount of time it
takes for a sound to decay by 60 dB from its original level. The signal
$y_{i}$ received at microphone $i$ is

$y_{i}(t)=a_{i}s(t-\tau_{i})+\epsilon_{i}(t)\;.$ (1)

In (1), $a_{i}$ is a scaling factor representing the attenuation suffered by
the wave propagating from $\boldsymbol{p}_{s}$ to $\boldsymbol{m}_{i}$. We
assume that the gains between the microphones are approximately calibrated,
although we show in subsection 4.3 that our method is robust to uncalibrated
microphones of the same kind. $\tau_{i}$ is the time taken for a sound wave to
propagate from the source to microphone $i$, and $\epsilon_{i}(t)$ models the
noise. We assume $\tau_{i}$ to be equal to
$\|\boldsymbol{m}_{i}-\boldsymbol{p}_{s}\|_{2}/c$, where
$\|\boldsymbol{m}_{i}-\boldsymbol{p}_{s}\|_{2}$ is the Euclidean distance
between the source and the microphone located at $\boldsymbol{m}_{i}$, $c$ is
the speed of sound and $\|\cdot\|_{2}$ represents the $L_{2}$-norm.

We also define $\boldsymbol{y}(t)=[y_{1}(t),\dotsc,y_{M}(t)]^{T}$ as the
vector containing all microphone signals at discrete time index $t$. The Short
Time Fourier Transform (STFT) of $y_{i}(t)$ is $Y_{i}(\ell,f)$, for frequency
$f$ and time frame $\ell$, and
$\boldsymbol{Y}(\ell,f)=[Y_{1}(\ell,f),\dotsc,Y_{M}(\ell,f)]^{T}$. The STFT
[34] represents the frequency content of a signal over time, and is a widely
used feature for source localization using neural networks [19, 15]. Figure 2
shows the magnitude representation of $\boldsymbol{Y}$ at the input.

Finally, the metadata vector $\boldsymbol{\phi}\in\mathbb{R}^{N_{\phi}}$ is
the concatenation of the coordinates of the microphones, the room dimensions
and reverberation time, as shown in Fig. 2. We chose the three aforementioned
types of metadata as the room dimensions and microphone coordinates are
explicitly exploited in classical localization methods such as the LS.
Furthermore, we included the reverberation time as an additional metadata to
verify whether its knowledge can reduce the detrimental effect of
reverberation in localization methods. However, other metadata could have been
exploited such as the energy ratio between the microphone signals, or the
absoption coefficients of the walls.

###  3.2 Proposed method: Dual input neural network

Our proposed DI-NN architecture is comprised of two neural networks, a feature
extraction network and a metadata fusion network as can be seen in Fig. 1. An
additional third network, called the metadata embedding network is also used
in the alternative DI-NN-Embedding network, which will be presented in
subsection 3.3 .

The input of the network consists of the STFTof the microphone signals as
defined in subsection 3.1. Instead of using the complex representation
generated by the STFT, we split the real and imaginary parts of the STFT
$\boldsymbol{Y}$ use them as separate channels as in [19], giving rise to
$2*M$ input channels. The role of the feature extraction network is to
transform this high dimensional tensor into a one dimensional feature vector
which compactly represents relevant information for the task in hand. In our
experiments, we adopt a CRNN [35] as our feature extraction network, due to
its wide adoption for SSL [7, 20, 36].

This metadata-unaware vector is then concatenated to the available metadata,
thus creating a metadata-aware feature vector. For our application, the
metadata is a one-dimensional vector consisting of the positions of the
microphones, the dimensions of the room, and its reverberation time. This
metadata-aware feature vector is then fed to a metadata fusion network, whose
role is to merge the metadata and feature vector to produce the result. In our
experiments, we adopt a two-layer Fully Connected Neural Network (FC-NN) which
maps the metadata-aware features to a two dimensional vector corresponding to
the estimated coordinates of the source.

Our feature extractor CRNN is divided into two sequential sub-networks: a CNN
block, responsible for extracting local patterns from the input data and a
Recurrent Neural Network (RNN), responsible for combining these pattens into
global, time-independent features. A diagram representing the components of
the DI-NN network is shown in Fig. 2.

Figure 2: Detailed DI-NN architecture for the task of PSSL.

The convolutional block receives a tensor of shape $(M,L,F)$ representing a
multi-channel complex STFT, where $M$ represents the number of audio channels,
$L$ represents the number of time frames generated by the STFT, and $F$ is the
number of frequency bins used. The role of this block is two-fold: firstly, to
combine local information across all microphone channels, and secondly to
reduce the dimensionality of the data to make it more tractable for the RNN
layer.

The convolutional block consists of four sequential layers, where each
performs three sequential operations. Firstly, a set of $K$ convolutional
filters is applied to the input signal, resulting in $K$ output channels.
Secondly, a non-linear activation function is applied to the result. Finally,
an average pooling operation is applied to the width and height of the
activations, generating an output of reduced size. After passing the input
through the four convolutional layers, we perform a global average pooling
operation across all frequencies, generating a two-dimensional output matrix.

After the convolutional block, the resulting matrix serves as input to a
bidirectional, gated recurrent unit neural network (GRU-RNN) [37]. As sound
may not be present throughout the whole duration of the audio signal, such as
during speech pauses, the RNN is important for propagating location
information to silent time-steps. After this network, we reduce the dimensions
of the features once again by performing average pooling on the time
dimension, resulting in a vector of time-independent features.

The output of the feature extraction network are then concatenated to the
available metadata and serve as input to the metadata fusion network. This
network consists of a set of two fully connected layers which map the
metadata-aware features to a two-dimensional vector corresponding to the
estimated cartesian coordinates of the active source. We jointly train both
networks using the same loss function, defined as the $L_{1}$-norm or the sum
of the absolute error between the network’s estimate of the source coordinates
$\boldsymbol{\hat{p}}_{s}$ and the target $\boldsymbol{p}_{s}$, given by

$\mathcal{L}(\boldsymbol{p}_{s},\boldsymbol{\hat{p}}_{s})=|\boldsymbol{p}_{s}-\boldsymbol{\hat{p}}_{s}|\;.$
(2)

We also considered using the more common squared error loss. Although both
losses yielded similar results in our experiments, we chose the absolute error
for its easier interpretability, since it corresponds to the distance in
metres between target and estimated coordinates.

###  3.3 DI-NN-Embedding

To test whether it is advantageous to process the metadata before combining it
with the microphone features, we also propose a variant of the DI-NN model,
where the metadata $\boldsymbol{\phi}$ is processed by a metadata embedding
network to produce an embedding, which is then concatenated to the microphone
features. This network is represented by the metadata embedding network block
in Figure 1.

###  3.4 Baseline: Least-squares based source localization

Figure 3: Error grid produced by the LS algorithm for an anechoic and a
reverberant room of the same dimensions and microphone coordinates.

Our final comparative baseline is the Least-Squares (LS) algorithm [1] which
uses the signal model defined in subsection 3.1. We provide an overview of the
algorithm below. We define the theoretical TDOA between microphones $i$ and
$j$ with respect to the source coordinates $\boldsymbol{p}_{s}$ as

$\tau_{ij}(\boldsymbol{p}_{s})\triangleq\frac{\|\boldsymbol{m}_{i}-\boldsymbol{p}_{s}\|_{2}-\|\boldsymbol{m}_{j}-\boldsymbol{p}_{s}\|_{2}}{c}\;,$
(3)

where $c$ is the speed of sound. Next, the measured TDOA between microphones
$\hat{\tau}_{ij}$ is estimated from the cross-correlation peak between the
received signals according to

$\hat{\tau}_{ij}\triangleq\operatorname*{arg\,max}_{t}(\mathcal{C}(t;y_{i},y_{j}))\;,$
(4)

where $\mathcal{C}$ denotes the cross-correlation operator, usually computed
in the frequency domain using the GCC-PHAT algorithm [32]. We then aggregate
the total error for all microphone pairs using

$E(\boldsymbol{p}_{s})\triangleq\sum_{i=1}^{m}\sum_{j\neq
i}E_{ij}(\boldsymbol{p}_{s})\;,$ (5)

where
$E_{ij}(\boldsymbol{p}_{s})\triangleq|\tau_{ij}(\boldsymbol{p}_{s})-\hat{\tau}_{ij}|^{2}$
is the squared difference between the theoretical and measured TDOA of each
microphone pair in (3) and (4), respectively. To estimate the source’s
location, we compute the values of $E$ for a set of candidate locations
$\boldsymbol{p}_{s}$ within the room. In the absence of noise and
reverberation, the location with the minimum error corresponds to the true
position of the source [1]. Figure 3 shows the heatmaps or error grids
generated using the LS algorithm in an anechoic and a reverberant room. The
position of the source is estimated by selecting the positions that minimize
the total error,

$\hat{\boldsymbol{p}}_{s}=\operatorname*{arg\,min}_{p_{s}}E(\boldsymbol{p}_{s})\;.$
(6)

Figure 3 illustrates the limitations of the LS algorithm when the
reverberation time is large. The two figures show the results of our algorithm
for two simulations, where one source and four microphones are placed in a
room with the same dimensions. When the room is simulated to be anechoic,
i.e., all the reflections are absorbed, the algorithm produces a sharp blue
peak in the heatmap. Conversely, when the simulated room is reverberant, the
peak becomes much more dispersed. An explanation for this is that the model
used by the LS method assumes anechoic propagation between the source and
microphones, i.e., no reflections are assumed. Conversely, we will show that
the DI-NN model is able to localize sources in reverberant environments, as it
is trained using a reverberant dataset. A study conducted in [38] shows that
speech inteligibility is maximized in rooms with a reverberation time between
0.4 and 0.5 ms, therefore limiting the practical application of the LS method
on those environments.

##  4 Experimentation

This section describes our experiments with DI-NNs with three SSL datasets
representing scenarios of varying difficulties. For each dataset, our approach
is compared to two other methods. The first method is a CRNN with the same
architecture but without using the available metadata, i.e., without the
“Concatenate” block in Fig. 2. By comparing this network’s performance to the
DI-NN, we can see the performance gains of our proposed method. The second
comparative method is the classical LS source localization method described in
subsection 3.4. The experiments will be described below.

All of our experiments consisted of randomly placing one source and four
microphones within a room. The height of the microphones, source and room were
fixed for all experiments. For each experiment, the goal of the proposed
method and baselines was to estimate the planar coordinates of the source
within the room using a one-second multichannel audio signal as well as the
positions of the microphones. We emphasize that the training and testing
samples do not overlap, and hence demonstrate our method’s effectiveness for
handling unseen scenes and metadata. We refer the reader to Appendix A for a
discussion on the independence of our datasets.

To simulate sound propagation in a reverberant room, we used the image source
method [39] implemented by the Pyroomacoustics Python library (MIT license)
[40]. We trained our neural networks using PyTorch (BSD license) [41] along
with the PyTorch Lightning (Apache 2.0 license) library [42]. The models were
trained using a single NVIDIA P100 GPU with 16 GB of RAM memory. The
configuration of our experiments is managed using the Hydra (MIT license)
library [43]. We release the code used for generating the data and training
the networks on GitHub 111Code: https://github.com/egrinstein/di˙nn, as well
as a Kaggle notebook 222Demo notebook:
https://kaggle.com/code/egrinstein/di-nn-training-notebook to allow
reproduction of the experiments without the need for any local software
installation. The hyperparameters used for training the proposed method and
baselines are shown in Table 1.

Table 1: Hyperparameters Parameter | Value
---|---
Num. parameters (DI-NN) | 3.5M
Num. conv. kernels | 64, 128, 256, 512
Conv. kernel size | 2x2
Conv. layer pooling size | 2x2
GRU output size | 256
Metadata fusion net. layer out. sizes |  $512+N_{\phi}$, $2$
Metadata embedding layer out. sizes |  $2N_{\phi}$, $N_{\phi}$
Activation func. last layer | None
Activation func. other layers | Rectified Linear Unit (ReLU)
Num. Discrete Fourier Transform (DFT) bins (for STFT) | 1024
DFT hop length (for STFT) | 512
Input duration | 0.5 secs.
Sampling rate | 16kHz
Grid resolution of LS method | 2 cm
Learning rate | 0.0005
Batch size | 32
Num. epochs | 40
Batch normalization [44] | Only after conv. layers
Optimizer | Adam [45]
Figure 4: Experimental setup. (a) For the anechoic and reverberant
simulations, each of the four microphones $m_{i}$ is placed on a random point
along the the coloured arrows, while the source $s$ is randomly placed on a
point within the rectangle defined by them. (b) The sampling procedure for
subsection 4.3, where positions of the microphones and source are randomly
drawn from each differently coloured set of points.

###  4.1 Simulated anechoic rooms

The goal of this experiment is to evaluate the performance of the DI-NN and
baselines in multiple rooms and microphone positions in the absence of
reverberation. Our dataset generation procedure is shown in Fig. 4a. For each
dataset sample, we randomly select two numbers from a uniform distribution in
the interval [3, 6] m representing the room’s width and length. The height of
the rooms is fixed at 3 m. Next, we randomly place one microphone along a line
segment 0.5 m away and parallel to each room’s walls. We chose to place the
microphones close to the wall as a simplified localization scenario, as our
main goal is to test the effectiveness of our metadata fusion procedure.
Nonetheless, this scenario is realistic in the context of smart rooms, where
the microphones are usually placed in or near the room’s walls.

Finally, the source is randomly placed in the room, following a uniform
distribution while respecting a minimum margin of 0.5 m from the walls. In
this experiment, the source signal is WGN, and 30 dB Signal-to-Noise Ratio
(SNR) sensor noise, simulated using WGN, is also added to each microphone. A
dataset of 15,000 samples is generated, from which 10,000 samples are used for
training, 2,500 for validation, and 2,500 for testing.

###  4.2 Simulated reverberant rooms

The data for the simulated reverberant rooms experiment is generated similarly
to the anechoic experiment. However, instead of simulating sound propagation
in an anechoic environment, each dataset sample is randomly assigned a
reverberation time value for its corresponding room from a uniform
distribution within the range of [0.3 – 0.6] s. This value is used to simulate
reverberation using the image source method [39]. For the source signal, we
use speech recordings from the VCTK corpus [46]. The number of training,
testing and validation samples is same as in the above section.

###  4.3 Real recordings

For this experiment, instead of simulations, we use measurements from the
LibriAdhoc40 dataset [47] (GPL3 license). The signals were recorded in a
highly reverberant room containing a grid of forty microphones and a single
loudspeaker, which was placed in one of four available locations. The
microphones recorded speech sentences taken from the Librispeech [48] corpus,
which were played back through the loudspeaker. The reverberation time
measured by the dataset authors was of approximately 900 ms.

To generate each dataset sample, we subselect four of the forty available
microphones. We restrict our microphone selection to the outermost microphones
of the grid, where one microphone per side is selected. A visual explanation
of our microphone selection procedure is provided in Fig. 4b. There are four
available positions for the microphones near each of the west and east walls
and seven positions near each of the north and south walls. Furthermore, there
are four available source positions. There are, therefore, $4\times 4\times
7\times 7\times 4=$ 3,136 source/microphone combinations available for
selection. Finally, we randomly select four speech utterances for each
combination, resulting in a dataset of 12,544 samples. We use 50% of those
combinations for training, 25% for validation and 25% for testing. To create
the training dataset for this experiment, we augment the aforementioned
training split with the training data of the reverberant dataset described in
subsection 4.2, resulting in a dataset consisting of 10,000 $+$ 6,272 $=$
16,272 signals.

###  4.4 Metadata sensitivity study

In practical scenarios, the metadata, e.g., microphone coordinates and room
reverberation time in PSSL, are uncertain because they are typically estimated
or measured. To investigate the robustness of our approach to such
uncertainties, we conducted a sensitivity study using the test dataset in
subsection 4.2. We modify the dataset by introducing different levels of
perturbations to the input metadata, followed by a computation of the mean
localization error for each level using the model trained on subsection 4.2.

Our first three studies consist of perturbing the microphone coordinates of
the testing dataset with increasing levels of random Gaussian noise. The
reported precision of microphone coordinates measured optically is under a
millimeter [49]. Conversely, when these are estimated using self-localization
algorithms, the reported errors are under 7 cm [50, 51]. We therefore choose
the standard deviation levels of the introduced noise to 1, 10 and 50 cm. In
our fourth study, we introduced random Gaussian noise to the reverberation
time with a standard deviation of 200 ms, based on reported errors obtained on
reverberation estimation procedures [52, 53].

###  4.5 Metadata relevance study

To quantify the contribution of each metadata category to the improvement in
localization performance, we conducted a metadata relevance study where we
trained the DI-NN network using six different combinations of the microphone
positions, room dimensions and reverberation time. The results are summarized
in Table 3.

##  5 Results and discussions

###  5.1 Results

Figure 5: (a) Mean localization error for DI-NNs and baselines on different
datasets. (b) Normalized histogram comparison between the DI-NN and the CRNN
baseline on the recorded dataset. (c) Cumulative version of (b).

Figure 5a compares the average error of our proposed DI-NN and DI-NN-Embedding
methods to the CRNN and LS baselines. To obtain statistically significant
results, we train the DI-NN, DI-NN-Embedding and CRNN models four times
independently for each experiment using random initial network parameters. The
results shown in Fig. 5 are averaged across the four times, with error bars
showing the standard deviation across the runs. Conversely, as the LS method
is deterministic, it does not require multiple runs.

A first remark is that although the LS approach is very effective in the
anechoic scenario, its performance is degraded on the other datasets,
indicating its sensitivity to reverberation. The CRNN outperforms the LS
method in reverberant scenarios without knowledge of the microphone’s
coordinates. Interestingly, the CRNN baseline is also obtains good
localization performance on the recorded dataset, indicating that the network
is able to infer the metadata to an extent when trained on a single room.

However, by exploiting the microphone coordinates, the DI-NN is shown to
significantly improve the performance compared to the CRNN. The most
significant difference is observed in the anechoic case, where an improvement
close to three times is obtained. In this case, the microphone coordinates are
more useful as this information cannot be derived from the signals. In a
reverberant room, however, the network might be able to use reflections to its
advantage, as discussed in [54], to infer the microphone coordinates and
making the metadata less useful. Fig. 5a also shows the errors obtained using
the alternative DI-NN-Embedding architecture were similar to the DI-NN in all
scenarios, indicating no advantage in the proposed embedding, although it
still allows the network to exploit the metadata.

In turn, Fig. 5b compares the normalized error histograms between our approach
and the CRNN baseline on the real recordings test dataset. The mode of the DI-
NN’s error is centred on the 0-15 cm bin compared to the 15-30 cm bin for
CRNN’s error. In other words, only the DI-NN is median-unbiased. The
cumulative distribution for the same data is shown in Fig. 5c. While the DI-NN
is shown to locate over 50% and 80% of the dataset samples with less than 15
and 45 cm error, the CRNN achieves the same errors for less than 20% and 60%
of the data, respectively.

Table 2: Metadata sensitivity analysis Coord. std. (m) | Reverb. std. (ms) | Err. increase (%)
---|---|---
0.01 | 0 | 0.05
0.1 | 0 | 1.02
0.5 | 0 | 32.9
0 | 200 | 0.4

The results of the sensitivity study conducted in subsection 4.4 are displayed
in Table 2. The last column refers to the relative error increase between the
perturbed case and the noiseless experiment conducted in subsection 4.2. The
results show that our approach is robust to the uncertainty inherent in
practical measurements of the microphone coordinates and reverberation time
estimates. The case where the microphone coordinates are disturbed by an
extreme error of 0.5 m (more than five times above typical errors) has been
included to demonstrate the impact of including microphone coordinates for
PSSL, reiterating the importance and improved performance of metadata in our
proposed fusion approach.

Table 3: Metadata relevance analysis Mic. coords. | Room dims. | RT60 | % performance
---|---|---|---
✓ | ✓ | ✓ | 100
✓ | ✓ | ✗ | 102
✓ | ✗ | ✓ | 100
✗ | ✓ | ✓ | 61
✓ | ✗ | ✗ | 104
✗ | ✓ | ✗ | 60
✗ | ✗ | ✓ | 47

Finally, the results of the metadata relevance analysis study described in
subsection 4.5 are displayed in Table 3. Each line represents a version of the
DI-NN model trained on the reverberant dataset. The first three columns
describe which metadata types are used in the model, and the last column shows
the model performance relative to the model using all metadata, represented in
the first line. The results show that the microphone coordinates are the most
relevant for the model. In fact, using the microphone coordinates alone
provides the best results. The results also indicate that the room dimensions
are more relevant than the reverberation time in the absence of the microphone
coordinates.

###  5.2 Limitations and extensions

Our approach exploits the metadata, such as the microphone coordinates and
reverberation time and therefore this data must be known a priori or somehow
measured. We have, however, shown that using this additional information is
justified by a significant improvement in performance. While we have also
assumed that the gains of the microphones are calibrated in our experiments,
which may not be verifiable in practical scenarios, we have shown in
subsection 4.3 that our model can perform well even when using uncalibrated
microphones of the same kind. If calibration cannot be ensured, extracting
gain invariant features from the signal pairs such as the cross spectra [16]
may be used as a preprocessing step.

We have also limited our scope to the localization of one static sound source
using static microphones to focus on metadata fusion. However, extensions to
moving sources and microphones could be possible by using smaller processing
frames, for example. Another extension would be to estimate the three
dimensional coordinates of the source. Finally, a possible extension for
multiple source localization is expanding the output of DI-NN to a vector of
size $2N$, where $N$ is the number of maximum sources, and performing
Permutation Invariant Training (PIT) [55].

##  6 Conclusion

In this work, we proposed DI-NN, a simple yet effective way of jointly
processing signals and relevant metadata using neural networks. Our results
for the task of SSL on multiple simulated and recorded scenarios indicate that
the DI-NN is able to exploit successfully the metadata, as its inclusion
reduced the mean localization error by a factor of at least two compared to
the CRNN baseline, as well as significantly improving localization results in
comparison with the classical LS algorithm in reverberant environments.
Additional relevance and sensitivity studies revealed that the microphone
coordinates the most important metadata, and that the DI-NN is robust to
realistic noise in the metadata.

## Appendix

###  6.1 Validation of metadata independence between training and testing
datasets

The datasets used in sections 4.1 and 4.2 are created entirely synthetically
by generating random training, validation and testing samples. The attributes
generated for each sample are the room’s width and length, the coordinates of
the four microphones, and the source coordinates. Additionally, in section
4.2, the room reverberation time is also randomly sampled. These values are
then used to simulate the microphone recordings using the image source method.
The only difference in the procedure for generating the training and testing
sets is the random seed used for sampling values. Although highly unlikely,
generating a test sample with the exact room dimensions, reverberation time,
microphone and source coordinates as a sample in the training set could be
possible and would violate the machine learning principle of having
independent training and testing sets.

To assure the reader that this has not occurred in our experiments, we compute
a distance metric $D$ between each testing sample and the entire training
dataset. We focus on comparing the microphone coordinates between the training
and testing sets and show that our approach has been validated against unseen
metadata. Each sample comprises four microphone coordinates, each placed near
one of the room’s walls. We define the distance $d(i,j)$ between the $i$-th
testing sample and $j$-th training sample as the sum of the distances of the
microphone coordinates between the samples given by

$d(i,j)=\sum_{k=1}^{4}\|\boldsymbol{m}_{i}^{k}-\boldsymbol{m}_{j}^{k}\|_{2}\;,$
(7)

where $\boldsymbol{m}_{i}^{1}$, $\boldsymbol{m}_{i}^{2}$,
$\boldsymbol{m}_{i}^{3}$ and $\boldsymbol{m}_{i}^{4}$ refer to the coordinates
of the microphones located near the north, south, east and western walls of
the room from the $i$-th sample and $\|\cdot\|$ denotes the $L_{2}$-norm.

To measure the distance between the $i$-th testing sample and the entire
training dataset, we compute (7) for every training sample $j$. We define the
smallest distance $D(i)$ between the $i$-th testing sample and the entire
training set as the minimum distance between $i$ and all training samples $j$,
expressed as

$D(i)=\min\limits_{j}\ \\{d(i,j)\\}\;.$ (8)

This measure quantifies the worst-case similarity between the $i$-th testing
sample and the most similar sample in the entire training set. By plotting a
histogram of $D(i)$ for every $i$-th sample in the testing set, we observe in
Fig. 6 that no training microphone configuration appeared in the testing set.
Moreover, the average minimum distance between the testing and training sets
is around 30 cm. Besides having different microphone coordinates, we like to
emphasize that the room dimensions and reverberation time also vary from
sample to sample, increasing the differences between training and testing sets
even further.

Figure 6: Distance between test dataset’s microphone coordinates and training
dataset.

## Correspondence

Eric Grinstein<EMAIL_ADDRESS>
## Funding

This work was funded through the European Union’s Horizon 2020 research and
innovation programme under the Marie Skłodowska-Curie grant agreement no.
956369 and the UK Engineering and Physical Sciences Research Council (EPSRC)
grant no. EP/S035842/1.

## Abbreviations

Dual Input Neural Network (DINN), convolutional recurrent neural network
(CRNN), sound source localization (SSL)

## Ethics approval and consent to participate

Not applicable.

## Authors’ contributions

Algorithmic development: Eric Grinstein, Patrick A. Naylor
Simulations and results: Eric Grinstein, Vincent W. Neo
Manuscript writing: Eric Grinstein, Vincent W. Neo, Patrick A. Naylor

## References

  * [1] So, H.C.: Source Localization: Algorithms and Analysis. John Wiley & Sons, Ltd, Hoboken, United States (2011)
  * [2] Haddad, D.B., Lima, M.V.S., Martins, W.A., Biscainho, L.W.P., Nunes, L.O., Lee, B.: Acoustic Sensor Self-Localization: Models and Recent Results. Wireless Communications and Mobile Computing 2017, 7972146 (2017). doi:10.1155/2017/7972146
  * [3] Brandstein, M., Ward, D.: Microphone Arrays: Signal Processing Techniques and Applications. Springer Science & Business Media, Berlin, Germany (2001)
  * [4] Wang, H., Chu, P.: Voice Source Localization for Automatic Camera Pointing System in Videoconferencing. In: Proc. IEEE Int. Conf. on Acoust., Speech and Signal Process. (ICASSP), pp. 187–190 (1997)
  * [5] Evers, C., Naylor, P.A.: Acoustic SLAM. IEEE Trans. Audio, Speech, Language Process., 1484–1498 (2018)
  * [6] Bertrand, A.: Applications and Trends in Wireless Acoustic Sensor Networks: A Signal Processing Perspective. In: IEEE Symp. on Commun. and Veh. Technol. in the Benelux (SCVT), pp. 1–6 (2011)
  * [7] Adavanne, S., Politis, A., Virtanen, T.: Direction of Arrival Estimation for Multiple Sound Sources Using Convolutional Recurrent Neural Network. In: Proc. Eur. Signal Process. Conf. (EUSIPCO), pp. 1462–1466 (2018)
  * [8] He, W., Motlicek, P., Odobez, J.-M.: Deep Neural Networks for Multiple Speaker Detection and Localization. In: Proc. Int. Conf. Robotics and Automation, pp. 74–79 (2018)
  * [9] Vera-Diaz, J.M., Pizarro, D., Macias-Guarasa, J.: Towards End-to-End Acoustic Localization using Deep Learning: From Audio Signal to Source Position Coordinates. Sensors (2018)
  * [10] Baldi, P., Cranmer, K., Faucett, T., Sadowski, P., Whiteson, D.: Parameterized Neural Networks for High-Energy Physics. The European Physical Journal C (2016)
  * [11] Atrey, P.K., Hossain, M.A., El Saddik, A., Kankanhalli, M.S.: Multimodal fusion for multimedia analysis: A survey. Multimedia Systems, 345–379 (2010)
  * [12] Baltrušaitis, T., Ahuja, C., Morency, L.-P.: Multimodal Machine Learning: A Survey and Taxonomy. IEEE Trans. Pattern Anal. Mach. Intell., 423–443 (2019)
  * [13] Ozerov, A., Vincent, E., Bimbot, F.: A General Flexible Framework for the Handling of Prior Information in Audio Source Separation. IEEE Trans. Audio, Speech, Language Process. 20(4), 1118–1133 (2012)
  * [14] Sundar, H., Wang, W., Sun, M., Wang, C.: Raw waveform based end-to-end deep convolutional network for spatial localization of multiple acoustic sources. In: Proc. IEEE Int. Conf. on Acoust., Speech and Signal Process. (ICASSP), pp. 4642–4646 (2020)
  * [15] Anonymous, A.: Deep Complex-Valued Convolutional-Recurrent Networks for Single Source DOA Estimation. In: Under Review (2022)
  * [16] Xue, W., Tong, Y., Zhang, C., Ding, G., He, X., Zhou, B.: Sound Event Localization and Detection Based on Multiple DOA Beamforming and Multi-Task Learning. In: Proc. Conf. of Int. Speech Commun. Assoc. (INTERSPEECH), pp. 5091–5095 (2020)
  * [17] Chakrabarty, S., Habets, E.A.P.: Broadband DoA Estimation Using Convolutional Neural Networks Trained with Noise Signals. In: Proc. IEEE Workshop on Appl. of Signal Process. to Audio and Acoust. (WASPAA), pp. 136–140 (2017)
  * [18] Yalta, N., Nakadai, K., Ogata, T.: Sound Source Localization Using Deep Learning Models. J. of Robotics and Mechatronics, 37–48 (2017)
  * [19] Krause, D., Politis, A., Kowalczyk, K.: Comparison of Convolution Types in CNN-based Feature Extraction for Sound Source Localization. In: Proc. Eur. Signal Process. Conf. (EUSIPCO), pp. 820–824 (2021)
  * [20] Perotin, L., Serizel, R., Vincent, E., Guérin, A.: CRNN-Based Multiple DoA Estimation Using Acoustic Intensity Features for Ambisonics Recordings. IEEE J. Sel. Topics Signal Process., 22–33 (2019)
  * [21] Perotin, L., Défossez, A., Vincent, E., Serizel, R., Guérin, A.: Regression Versus Classification for Neural Network Based Audio Source Localization. In: Proc. IEEE Workshop on Appl. of Signal Process. to Audio and Acoust. (WASPAA), pp. 343–347 (2019)
  * [22] Grumiaux, P.-A., Kitić, S., Girin, L., Guérin, A.: A Survey of Sound Source Localization with Deep Learning Methods. arXiv (2021)
  * [23] Taseska, M., Habets, E.A.P.: Spotforming using distributed microphone arrays. In: Proc. IEEE Workshop on Appl. of Signal Process. to Audio and Acoust. (WASPAA) (2013)
  * [24] Guizzo, E., Marinoni, C., Pennese, M., Ren, X., Zheng, X., Zhang, C., Masiero, B., Uncini, A., Comminiello, D.: L3DAS22 Challenge: Learning 3D Audio Sources in a Real Office Environment. In: Proc. IEEE Int. Conf. on Acoust., Speech and Signal Process. (ICASSP) (2022). Comment: Accepted to 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP 2022). arXiv admin note: substantial text overlap with arXiv:2104.05499. 2202.10372
  * [25] Gustafsson, F., Gunnarsson, F.: Positioning Using Time-Difference of Arrival Measurements. In: Proc. IEEE Int. Conf. on Acoust., Speech and Signal Process. (ICASSP) (2003)
  * [26] Li, D., Hu, Y.H.: Energy Based Collaborative Source Localization Using Acoustic Micro-Sensor Array. EURASIP J. on Applied Signal Process. (2003)
  * [27] Liu, Z., Zhang, Z., He, L.-W., Chou, P.: Energy-Based Sound Source Localization and Gain Normalization for Ad Hoc Microphone Arrays. In: Proc. IEEE Int. Conf. on Acoust., Speech and Signal Process. (ICASSP) (2007)
  * [28] Schmidt, R.O.: Multiple Emitter Location and Signal Parameter Estimation. IEEE Trans. Antennas Propag., 276–280 (1986)
  * [29] Dmochowski, J.P., Benesty, J., Affes, S.: A Generalized Steered Response Power Method for Computationally Viable Source Localization. IEEE Trans. Audio, Speech, Language Process., 2510–2526 (2007)
  * [30] Lebarbenchon, R., Camberlein, E., di Carlo, D., Gaultier, C., Deleforge, A., Bertin, N.: Evaluation of an open-source implementation of the SRP-PHAT algorithm within the 2018 LOCATA challenge. arXiv (2018)
  * [31] Huang, Y., Benesty, J., Elko, G.W., Mersereati, R.M.: Real-Time Passive Source Localization: A Practical Linear-Correction Least-Squares Approach. IEEE Trans. Audio, Speech, Language Process., 943–956 (2001)
  * [32] Knapp, C., Carter, G.: The Generalized Correlation Method for Estimation of Time Delay. IEEE Trans. Acoust., Speech, Signal Process., 320–327 (1976)
  * [33] Zhang, C., Florencio, D., Zhang, Z.: Why Does PHAT Work Well in Lownoise, Reverberative Environments? In: Proc. IEEE Int. Conf. on Acoust., Speech and Signal Process. (ICASSP), pp. 2565–2568 (2008)
  * [34] Allen, J.B.: Short Term Spectral Analysis, Synthesis, and Modification by Discrete Fourier Transform. IEEE Trans. Acoust., Speech, Signal Process. 25(3), 235–238 (1977)
  * [35] Choi, K., Fazekas, G., Sandler, M., Cho, K.: Convolutional recurrent neural networks for music classification. In: Proc. IEEE Int. Conf. on Acoust., Speech and Signal Process. (ICASSP), pp. 2392–2396 (2017)
  * [36] Cao, Y., Kong, Q., Iqbal, T., An, F., Wang, W., Plumbley, M.D.: Polyphonic Sound Event Detection and Localization using a Two-Stage Strategy. Proc. Detect. and Classific. of Acoust. Scenes and Events (DCASE), 30–34 (2019)
  * [37] Chung, J., Gulcehre, C., Cho, K., Bengio, Y.: Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling. In: Proc. Neural Inform. Process. Conf (2014)
  * [38] Bistafa, S., Bradley, J.: Reverberation time and maximum background-noise level for classrooms from a comparative study of speech intelligibility metrics. The Journal of the Acoustical Society of America 107, 861–75 (2000). doi:10.1121/1.428268
  * [39] Allen, J.B., Berkley, D.A.: Image Method for Efficiently Simulating Small-Room Acoustics. J. Acoust. Soc. Am. 65(4), 943–950 (1979)
  * [40] Scheibler, R., Bezzam, E., Dokmanić, I.: Pyroomacoustics: A Python Package for Audio Room Simulation and Array Processing Algorithms. In: Proc. IEEE Int. Conf. on Acoust., Speech and Signal Process. (ICASSP), pp. 351–355 (2018)
  * [41] Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., et al.: Pytorch: An imperative style, high-performance deep learning library. In: Proc. Neural Inform. Process. Conf (2019)
  * [42] Falcon, W., The PyTorch Lightning Team: PyTorch Lightning. https://www.pytorchlightning.ai (2019)
  * [43] Yadan, O.: Hydra - A Framework for Elegantly Configuring Complex Applications. https://www.hydra.cc (2019)
  * [44] Ioffe, S., Szegedy, C.: Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. In: Proc. Int. Conf. Machine Learning (ICML), pp. 448–456 (2015)
  * [45] Kingma, D.P., Ba, J.: Adam: A Method for Stochastic Optimization. arXiv (2017). 1412.6980
  * [46] Yamagishi, J., Veaux, C., MacDonald, K.: CSTR VCTK Corpus: English Multi-speaker Corpus for CSTR Voice Cloning Toolkit (Version 0.92) (2019)
  * [47] Guan, S., Liu, S., Chen, J., Zhu, W., Li, S., et al.: Libri-Adhoc40: A Dataset Collected from Synchronized Ad-Hoc Microphone Arrays. In: Asia-Pacific Signal and Inform. Process. Assoc. Annual Summit and Conf. (APSIPA) (2021)
  * [48] Panayotov, V., Chen, G., Povey, D., Khudanpur, S.: Librispeech: An ASR Corpus Based on Public Domain Audio Books. In: Proc. IEEE Int. Conf. on Acoust., Speech and Signal Process. (ICASSP), pp. 5206–5210 (2015)
  * [49] Aurand, A.M., Dufour, J.S., Marras, W.S.: Accuracy map of an optical motion capture system with 42 or 21 cameras in a large measurement volume. Journal of Biomechanics 58, 237–240 (2017)
  * [50] Gaubitch, N.D., Kleijn, W.B., Heusdens, R.: Auto-localization in ad-hoc microphone arrays. In: Proc. IEEE Int. Conf. on Acoust., Speech and Signal Process. (ICASSP), pp. 106–110 (2013)
  * [51] Pertilä, P., Mieskolainen, M., Hämäläinen, M.S.: Passive self-localization of microphones using ambient sounds. In: Proc. Eur. Signal Process. Conf. (EUSIPCO), pp. 1314–1318 (2012)
  * [52] Gamper, H., Tashev, I.J.: Blind Reverberation Time Estimation Using a Convolutional Neural Network. In: Proc. Int. Workshop on Acoust. Signal Enhancement (IWAENC), pp. 136–140 (2018)
  * [53] López, P.S., Callens, P., Cernak, M.: A Universal Deep Room Acoustics Estimator. In: Proc. IEEE Workshop on Appl. of Signal Process. to Audio and Acoust. (WASPAA), pp. 356–360 (2021). 2109.14436
  * [54] Ribeiro, F., Ba, D., Zhang, C., Florêncio, D.: Turning enemies into friends: Using reflections to improve sound source localization. In: IEEE International Conference on Multimedia and Expo, pp. 731–736 (2010)
  * [55] Yu, D., Kolbæk, M., Tan, Z.-H., Jensen, J.: Permutation invariant training of deep models for speaker-independent multi-talker speech separation. In: 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 241–245 (2017). doi:10.1109/ICASSP.2017.7952154

  *[SNR]: Signal-to-Noise Ratio
  *[RIR]: Room Impulse Response
  *[LSTM]: Long Short-Term Memory
  *[SC-PHD]: Single Cluster
  *[SLAM]: Simultaneous Localization and Mapping
  *[STOI]: Short-Time Objective Intelligibility measure
  *[PHD]: Probability Hypothesis Density
  *[DARPA]: Defense Advanced Research Projects Agency of the United States Dept. of Defense
  *[dB]: Decibel
  *[BSTOI]: Binaural
  *[DRR]: Direct-to-Reverberant Ratio
  *[ACE]: Acoustic Characterization of Environments. A noisy reverberant speech corpus and IEEE challenge run by the SAP group at Imperial College
  *[SOBM]: STOI-optimal Binary Mask
  *[EKF]: Extended Kalman Filter
  *[SSRR]: Segmental Signal-to-Reverberation Ratio
  *[PCM]: Pulse-Code Modulation
  *[TDoA]:
  *[GSM]: Global System for Mobile Communications
  *[]:
  *[SAS]: Synthetic Aperture   ()
  *[ISCA]: International Speech Communication Association
  *[LSR]: Least Squares Residuals clipping detection method
  *[ILAH]: Iterated Logarithm Amplitude Histogram clipping detection method
  *[MPEG]: Moving Picture Experts Group
  *[SRMR]: Speech-to-Reverberation Modulation Energy Ratio
  *[SRT]: Speech Reception Threshold (also known as Speech Recognition Threshold)
  *[RADAR]: RAdio Detection And Ranging
  *[SONAR]: SOund Navigation And Ranging
  *[SDD]: Spectral Decay Distributions
  *[SDDSA]:  with Mel-spaced frequency bands and selective averaging
  *[EDF]: Energy Decay Function
  *[NIST]: National Institute of Standards and Technology
  *[WSOBM]: Weighted
  *[TI]: Texas Instruments, Inc.
  *[MIT]: Massachusetts Institute of Technology
  *[LAN]: Local Area Network
  *[SSL]: Sound Source Localization
  *[DI-NN]: Dual-Input Neural Network
  *[PSSL]: Positional Sound Source Localization
  *[CRNN]: Convolutional Recurrent Neural Network
  *[DOA]: Direction-of-Arrival
  *[STFT]: Short Time Fourier Transform
  *[MLP]: Multi-layer Perceptron
  *[TDOA]: Time-Difference-of-Arrival
  *[TDOAs]: Time-Difference-of-Arrival
  *[LS]: Least-Squares
  *[ML]: Maximum Likelihood
  *[GCC-PHAT]: Generalized Cross-Correlation with Phase Transform method of estimating
  *[WGN]: White Gaussian Noise
  *[FC-NN]: Fully Connected Neural Network
  *[CNN]: Convolutional Neural Network
  *[RNN]: Recurrent Neural Network
  *[DI-NNs]: Dual-Input Neural Network
  *[ReLU]: Rectified Linear Unit
  *[DFT]: Discrete Fourier Transform