Dual input neural networks for positional sound source localization
In many signal processing applications, metadata may be advantageously used
in conjunction with a high dimensional signal to produce a desired output. In
the case of classical Sound Source Localization (SSL) algorithms, information
from a high dimensional, multichannel audio signals received by many
distributed microphones is combined with information describing acoustic
properties of the scene, such as the microphones' coordinates in space, to
estimate the position of a sound source. We introduce Dual Input Neural
Networks (DI-NNs) as a simple and effective way to model these two data types
in a neural network. We train and evaluate our proposed DI-NN on scenarios of
varying difficulty and realism and compare it against an alternative
architecture, a classical Least-Squares (LS) method as well as a classical
Convolutional Recurrent Neural Network (CRNN). Our results show that the DI-NN
significantly outperforms the baselines, achieving a five times lower
localization error than the LS method and two times lower than the CRNN in a
test dataset of real recordings.
Eric Grinstein, Vincent W. Neo, Patrick A. Naylor
# Dual input neural networks for positional sound source localization

###### Abstract

In many signal processing applications, metadata may be advantageously used in conjunction with a high dimensional signal to produce a desired output. In the case of classical Sound Source Localization (SSL) algorithms, information from a high dimensional, multichannel audio signals received by many distributed microphones is combined with information describing acoustic properties of the scene, such as the microphones' coordinates in space, to estimate the position of a sound source. We introduce Dual Input Neural Networks (DI-NNs) as a simple and effective way to model these two data types in a neural network. We train and evaluate our proposed DI-NN on scenarios of varying difficulty and realism and compare it against an alternative architecture, a classical Least-Squares (LS) method as well as a classical Convolutional Recurrent Neural Network (CRNN). Our results show that the DI-NN significantly outperforms the baselines, achieving a five times lower localization error than the LS method and two times lower than the CRNN in a test dataset of real recordings.

Keywords:sound source localization; multichannel audio processing; multimodal machine learning; convolutional recurrent neural networks +
Footnote †: journal:

## 1 Introduction

Most signals, such as audio and images, contain metadata. Metadata can be signal-based, which describes quantitative properties of the signal, such as its sampling rate, as well as semantic, which describes, for example, contextual properties. In speech processing, semantic metadata could consist of the speaker's language or gender. Whether signal-based or semantic, including metadata as a secondary input into neural network models may provide relevant information which would translate into an economy of training time, model parameters and flexibility. However, metadata typically has a different dimensionality than the input signals, making its incorporation into those models not trivial.

The main focus of this paper is to study the effectiveness of schemes to process signals and exploit metadata jointly using neural network models. We focus on the task of Sound Source Localization (SSL) [1] using distributed microphone arrays to demonstrate the effectiveness of our proposed approach. In the context of SSL, relevant metadata which is exploited by classical methods is the microphone positions, which can be acquired by manual measurement or using self-calibration [2] methods. Other relevant metadata is the room dimensions and its reverberation time.

SSL refers to the task of estimating the spatial location of a sound source, such as a human talker or a loudspeaker. In this scenario, metadata refers to properties of the acoustic scene such as the coordinates of microphones, dimensions of the room and, the reflection coefficient of the walls. SSL has many applications, including noise reduction and speech enhancement [3], camera steering [4] and acoustic Simultaneous Localization and Mapping (SLAM) [5]. In turn, distributed microphone arrays have become an active research topic in the signal processing community due to their versatility. Such arrays may be composed of multiple network-connected devices, including everyday devices such as cell phones, smart assistants, and laptops, for example. The array and the constituent devices may be configured as a Wireless Acoustic Sensor Network (WASN) [6].

SSL approaches may be divided into classical signal processing-based and data-driven neural network-based methods. By explicitly exploiting metadata describing microphone positions and room dimensions, classical approaches may be applied to different rooms and microphone configurations. Conversely, neural network approaches have recently achieved state of the art results for source localization [7; 8; 9], at the expense of requiring one network to be trained for every microphone topology. One reason current neural approaches do not incorporate the microphones' positional information is that the microphones' signal and positional data are very different from one another in nature and dimension.

Previous work which discusses the joint processing of signals and metadata is [10], where a single input neural network is used to process metadata in conjunction with a low-dimensional physical signal. However, unlike our work, the method of [10] is restricted to multilayer perceptron architectures and one-dimensional