# An embedded multichannel sound acquisition system for drone audition

Michael Clayton, Lin Wang, Andrew McPherson, Andrea Cavallaro Manuscript
received: December 25, 2020The authors are with Centre for Intelligent
Sensing, Queen Mary University of London, London, UK (e-mail: {m.p.clayton,
lin.wang, a.mcpherson<EMAIL_ADDRESS>
###### Abstract

Microphone array techniques can improve the acoustic sensing performance on
drones, compared to the use of a single microphone. However, multichannel
sound acquisition systems are not available in current commercial drone
platforms. To encourage the research in drone audition, we present an embedded
sound acquisition and recording system with eight microphones and a
multichannel sound recorder mounted on a quadcopter. In addition to recording
and storing locally the sound from multiple microphones simultaneously, the
embedded system can connect wirelessly to a remote terminal to transfer audio
files for further processing. This will be the first stage towards creating a
fully embedded solution for drone audition. We present experimental results
obtained by state-of-the-art drone audition algorithms applied to the sound
recorded by the embedded system.

###### Index Terms:

Drone audition, microphone array, embedded system

##  I Introduction

The use of drones for remote sensing has substantially increased in the past
decade, with operation in broadcasting, surveillance, inspection, and search
and rescue [1]. Sensing is primarily based on cameras (optical and thermal)
and lasers[2, 3, 4, 5, 6], whereas microphones are rarely used because of the
inherently challenging sound sensing conditions [7]. When visual data is
unreliable due to low light, poor weather conditions or visual obstructions
[8], drone audition would greatly benefit the above-mentioned applications.
One of the main obstacles when capturing audio on a drone is the strong ego-
noise created by the rotating motors, propellers and the airflow during
flight. The ego-noise masks the target sound sources and causes poor recording
quality.

Microphone array techniques can be used to improve the drone audition
performance through sound enhancement [9, 10, 11, 12, 13, 14] and sound source
localization [15, 16, 17, 18, 20]. An important bottleneck for deploying
microphone array algorithms on drones is the requirement of a multichannel
sound acquisition system to enable sampling the sound from multiple
microphones simultaneously and convert it to multichannel digital signals
before further processing. The sound acquisition system needs to fly with the
drone, which imposes additional constraints on the size and weight of the
system. To the best of our knowledge, there is no dedicated multichannel sound
acquisition device available in current commercial drone platforms.
Researchers have to design and implement their own hardware systems for data
collection on drones, and the processing of the data is often done offline
after the flight due to limited computational resources onboard.

To conduct and encourage the research in the field of drone audition, we
designed an embedded multichannel sound acquisition system that is suitable
for drone audition and can be mounted on a drone for acoustic sensing during
flight. The system is designed based on Bela [21], an embedded computing
platform dedicated to audio processing, and can accommodate up to eight
microphones placed in arbitrary shapes. The system can record and store the
sound locally for on-device processing; and can also transfer the recorded
sound file via wireless communication to a remote terminal. In the remainder
of the paper, we disclose the technical details for hardware, software design
and development.

The paper is organized as follows. Sec. II reviews related works. Sec. III and
Sec. IV present the hardware and software design of the embedded system. Sec.
V presents real data collection with the hardware and presents baseline
processing results with state-of-the-art drone audition algorithms.

TABLE I: Existing multichannel sound acquisition systems on drones. Q - Quadcoptor; H - Hexacopters; O - Octocopter Ref | Number of | Shape | Placement | Audio interface | Drone Type | Remark
---|---|---|---|---|---|---
| microphones | of the array | of the array |  |  |
[11] | 6 | T-shape | Side | Zoom H6 | Self-assembled (Q) | Portable recorder
[7] | 8 | Circular | Top | Zoom R24 | 3DR Iris (Q) | Portable recorder
[23] | 6 | Circular (fixed) | Top | ReSpeaker + Raspberry Pi | Self-assembled (O) | Intelligent voice interface
[24] | 7 | Circular (fixed) | Side | UMA-8 Mic array + Raspberry Pi | Self-assembled (Q) | Intelligent voice interface
[25] | 8 | Circular | Below | MiniDSP USBStreamer I2S-to-USB | Matrice 100 (Q) | Sound card
[17] | 8 | Cubic | Below | 8SoundsUSB | MK-Quadro (Q) | Sound card
[28] | 8 | Circular | Top, below, side | 8SoundsUSB | Matrice 100 (Q) | Sound card
[19] | 12 | Spherical | Side | RASP-ZX | Surveyor MS-06LA (H) | Sound card
[26] | 8 | Circular | Side | RASP-24 | Parrot AR Drone (Q) | Sound card
[18] | 16 | Octagon | Side | RASP-ZX | Surveyor MS-06LA (H) | Sound card
Proposed | 8 | Circular | Top | Bela | Matrice 100 (Q) | Sound card

##  II Related work

Three types of audio hardware are employed: multichannel sound card,
intelligent multichannel voice interface, and portable multichannel sound
recorder.

####  II-1 A portable multichannel sound recorder

This is the easiest way to capture sound from drones as there is no
requirement for any configuration of the system, e.g. Zoom H6 [11] and Zoom
R24 [7, 27]. The hardware supports arbitrary array topology. The drawback is
the the hardware can only achieve recording and does not support sound
processing. Another drawback is that the hardware, e.g. Zoom R24, is usually
too heavy a payload for the drone to fly.

####  II-2 Intelligent multichannel voice interface

This type of hardware integrates the microphone array and sound processing
into a compact IC board, e.g. ReSpeaker [23] and UAM-8 [24]. This hardware
usually requires an additional controller, e.g. Rasberry Pi, for sound
acquisition and sound processing. This hardware is also usually easy to use
and configure for audio purposes. One of the main advantages is that the
hardware is also very compact and light-weight, and is suitable to fly with
the drone. The drawback is the topology of the array is fixed, which limits
the performance and flexibility of microphone array algorithms.

####  II-3 Multichannel sound card

This is the most popular approach for sound recording on drones, using e.g.
RASP series [19, 26, 18], 8SoundUSB [17, 28], USB Streamer [25]. This hardware
supports arbitrary array topology along with sound acquisition and sound
processing. The main drawback is the user requires knowledge of the hardware
circuit design. This particular hardware also requires an operating system to
control sound recording and processing, e.g. the RASP series is used in
combination with the HARK system [29]. A good understanding of the back-end
driver is necessary. The lack of related resources is also intimidating for
algorithm designers.

##  III Hardware design

Fig. 1 and Fig. 2 illustrate the architecture and the real objects of the
multichannel sound acquisition system, respectively. The system mainly
consists of three parts: the microphone array, the drone, the hardware tray
containing the Bela sound acquisition system and the cables. Table II lists
the components used by the system. Fig. 3 illustrates the Bela hardware system
assembly and peripheral connections.

Figure 1: Architecture of the multichannel sound acquisition system. Figure 2: Real objects of the multichannel sound acquisition system. (a) Front view; (b) Side view; (c) Top view; (d) Microphone array. TABLE II: Components used in the hardware system. Component |  Type |  Functionality
---|---|---
Drone |  Matrice 100 |  /
Microphones (8) |  Lapel microphones |  /
Array frame |  3D printing |  Holding microphones
Hardware tray |  3D printing |  Holding hardware and cables
Bela |  BeagleBone Black |  1GHz ARM Cortex-A8 processor
CTAG Beast (2) |  / |  Multichannel audio acquisition
CTAG Molex breakout board (2) |  / |  Audio inputs
Molex to 3.5mm adapter cable (4) |  / |  Connects microphones to Bela
Mono to stereo adapter cable (4) |  / |  Split stereo signal to mono signal
USB LiPo battery |  5V, 2Amp |  Provide power to Bela CTAG BEAST
USB storage |  / |  Save and store recorded audio files locally
WiFi Dongle |  / |  Wireless connection to bela IDE

###  III-A Microphone array and drone

We use a circular microphone array consisting of eight Boya BY-M1 lapel
microphones that are each powered by an LR44 (1.5V) battery. A balanced audio
signal is provided from the microphones. The diameter of the array is 16.5 cm.
The microphone array frame is 3D printed and constructed from Acrylonitrile
butadiene styrene (ABS). The array is mounted on top of the drone to avoid the
air flow from the rotating propellers blowing downward [30]. The vertical
distance from the array to the drone body is 18 cm. For the drone, we use DJI
Matrice 100, which has a payload capacity of 1 kg.

Figure 3: Bela multichannel audio hardware system assembly and peripheral. The
core processing part is highlighted in the red box.

###  III-B Bela-based sound acquisition system

The sound acquisition system consists of four units: the core processing unit,
the storage and transmission unit, and the hardware tray. Fig. 3 illustrates
the hardware connections.

####  III-B1 Core processing unit

The core processing unit consists of one Beaglebone device flashed with the
latest Bela software. To access multichannel audio, Bela uses a customized
expansion board called CTAG BEAST, featuring an audio codec with 4 audio input
and 8 audio output channels. One CTAG BEAST consists of 2 x CTAG FACE capes
pre-configured for use as a BEAST [31], two CTAG Molex breakout boards, and
one external LiPo power battery.

Bela111Bela is an embedded audio programming and processing platform invented
by academia from Queen Mary University of London[21]. The compact size, light-
weight, low-latency and multichannel sound acquisition makes it suitable for
sound processing on drones[22]. Bela also comes with a user friendly browser-
based Integrated Development Environment (IDE), which is used for easy access
for editing, building and managing the system. For this reason, we decided to
develop a multichannel sound acquisition system based on the Bela device. This
is the first time the Bela system has been applied to robotic platforms
assisting audition. is a dedicated audio processing platform based on
BeagleBone Black (BBB) single-board computer, which is featured by a 1GHz ARM
Cortex-A8 processor, two PRUs (Programmable Realtime Units), 512 MB RAM, and a
diverse range of on-board peripherals. Bela is used for controlling the sound
acquisition and audio processing. Bela is externally powered by a LiPo USB
battery that operates at 5V and 2Amp for stability and powering the USB
peripherals. The Bela configuration only requires 5V / 300 - 400mA of power
for operation.

The audio codec operates at 48 kHz sampling rate with 16 bits analogue-to-
digital converter (ADC) and digital-to-analogue converter (DAC) conversion. To
accommodate 8 microphone inputs, two CTAG FACE capes are stacked on top of
each other and connected with Bela via the onboard metal contacts.

####  III-B2 Storage and wireless unit

A external USB hub is connected to the USB socket of the Beaglebone device.
The hub accommodates a USB storage stick, which stores the recording locally,
and a USB WiFi Dongle, eliminating the need for a hard-wired connection to the
system IDE and enabling the recorded audio to be transferred to a remote
processing terminal.

####  III-B3 Hardware tray

A hardware tray is designed to accommodate the Bela system and the cables. The
tray contains a Bela enclosure (made from ABS) and shock case (made from
Thermoplastic Polyurethane - TPU) to aid with protecting the hardware from
impacts in the event of a crash. The tray is produced with 3D printing.

##  IV software design

The software design has three objectives: to run the code in a stand-alone
device; to record the sound locally to the USB storage; to transfer the sound
via WiFi to a remote terminal. All the objectives are achieved with the
assistance of the Bela Integrated Development Environment (IDE).

###  IV-A IDE for stand-alone processing

Figure 4: Interacting with Bela from a local computer via wireless network.
Figure 5: Bela Integrated Development Environment.

The Bela IDE (Fig.5) is a browser based integrated development environment
with features that allow for editing, building and managing projects easily
from a ground station (remote terminal) via a self-organized wireless network.

The IDE software is pre-installed at the Bela device, along with an operation
system. Following the steps in the
tutorial222www.eecs.qmul.ac.uk/$\thicksim$linwang/download/bela_documentation.pdf,
we set up a self-organized wireless network through a WiFi dongle mounted on
the Bela device. Upon system boot, Bela starts a NodeJS server that allows
connection to its system from a ground station via the wireless network. The
WiFi is setup as a peer-to-peer connection to ensure that the board acts as a
dynamic host configuration protocol (DHCP) server.

To connect to the Bela device from the ground station, we first need to select
the WiFi network hosted by the Bela system. After connection, the IDE can be
loaded by entering the IP address of the host device from the web browser. The
IDE interface (Fig. 5) will appear automatically at the web browser of the
ground station.

After compiling, building and running a project from the IDE. The project can
be set to run on boot in the IDE Settings tab by selecting the desired project
in the drop down menu. The program will operate on Bela without connecting to
the ground station as long as the external power is provided.

###  IV-B Sound recording

The code that enables the Bela to function as a multichannel recording device
is written in C++. This allows for quick access in the event that the system
requires any modifications, creating a flexible system. The source code for
sound recording is given in the Appendix, with the processing flow shown in
Fig. 6. In brief, after importing the required library, configuring global
variables and file path, the program sets up the recording task to capture the
multichannel audio data, writes the stream to the audio buffer (memory block),
and stores the data in the pre-defined file path. Once the recording is
finished, a clean-up function finalizes the writing process and closes the
file.

The IDE enables the user to start/stop recording, change settings and download
audio files directly from the system among some other features. After building
and running the project, the recording will start by writing the digital audio
to the specified system path. Pressing the stop icon in the IDE will stop the
recording process. The audio data is continuously written to the local storage
during recording. Once the stop button is pressed the .wav file is finalised
and closed.

Figure 6: Processing flow for sound recording with Bela.

###  IV-C WiFi Network Connection

The WiFi connection enables the user to access the Bela system through the IDE
without a hard-wired connection. Instead of recording the audio to the USB
storage, we can alter the target file-path to the default RAM memory of the
Bela device (see Appendix) in order to have the file appear and update during
the recording process within the resources section of the project explorer tab
in the IDE. The network connection is continuous to allow the user to change
different functions in the IDE.

The current WiFi signal is able to achieve an operational range of 20 metres
between ground station and the drone. When the network connection is lost
momentarily, the IDE user interface stops updating the user about the project
running and dependent on filepath selection, the file size of the current
recording. The IDE recovers after coming back into WiFi signal range. When the
wireless network is re-established the existing file has essentially continued
running on the system and the IDE user interface resumes updating the
recording progress of the file. The WiFi connection is not required to conduct
the recording itself but to monitor its progress.

##  V Experiment

###  V-A Setup

To verify the validity of the developed hardware system, we conduct in-flight
testing and recording. We record the ego-noise and the speech, separately.
When recording the ego-noise, the altitude of the drone during flight is
maintained at about 2 meters above the ground via the flight controller (Fig.
7). We record two types of ego-noise: drone hovering and drone moving. In the
former case, the drone is hovering in the air using the GPS stabilised mode
with additional manual input (correcting small drift) to allow the drone to
remain reasonably stable throughout the recording. In the latter case, the
drone is moving in the air at a speed of around 1 meter/second, with random
rotation and tilting during flight. When recording the speech-only data, the
drone is muted on the ground and a loudspeaker plays sound at a distance of 2
meters. The original sampling rate is 48 kHz. The audio is downsampled to 8
kHz before processing. All the analysis is completed offline and not on the
Bela system. The sound recording is available
online333www.eecs.qmul.ac.uk/$\thicksim$linwang/bela.html.

Figure 7: A drone with microphone array hovering in the air during recording.

###  V-B Ego-noise analysis

Fig. 8(a) depicts time-domain waveform of the ego-noise recorded at the
hovering and the moving status. Fig. 8(c) plots the time-frequency domain
spectrogram, which is computed with a moving window of 128 ms and half
overlap. Fig. 8(b) plots the power at each time frame. Fig. 8(d) plots the
frequency-domain spectrum at the 25-th second and 35-th second of the two ego-
noises, respectively.

From Fig. 8(b), the power of the ego-noise does not show big differences at
the hovering status and the moving status. The mean and standard deviation of
the power across time frames (10$s$-40$s$) are -29.9 dB and 0.41 dB,
respectively, at the hovering status. The mean and standard deviation of the
power across time frames (10$s$-60$s$) are -29.6 dB and 0.42 dB, respectively,
at the moving status. For the moving status, we observe a sudden rise of the
power at 45$s$, which is possibly due to the rotation operation of the drone.

From Fig. 8(c), it can be observed that the ego-noise consists of multiple
harmonics. Since the four motors might operate at a slight different rotating
speed, the harmonic ego-noise presents several pitches, which can be verified
from Fig. 8(d). At the hovering status, the pitch of the ego-noise remains
stable. At the moving status, the pitch of the ego-noise varies with time,
depending on flight status of the drone.

The recording is made in an outdoor environment with a light breeze present.
However, from the spectrogram of the recording we do not observe an evident
influence of the wind in the low frequency. This is possibly due to the
windshield worn by each microphone and also the placement of the microphones
on top of the drone.

Figure 8: Visualization of the ego-noise when the drone is hovering and
moving. (a) Time-domain waveform; (b) Power plot; (c) Time-frequency
spectrogram; (d) Frequency-domain plot. Figure 9: Benchmark performance
achieved by two spatial filters at various input SNRs.  Figure 10: Processing
results (beamformer) for input SNR -20 dB. The output SNR is 3.1 dB. (a) Clean
speech; (b) Noisy signal before procesisng; (c) Noisy signal after processing.

###  V-C Processing results

We synthesize a noisy signal at the microphones by adding the ego-noise
(hovering status) and the speech at different input SNRs, which vary from -35
dB to 0 dB, with an interval of 5 dB. The testing signal is 25 seconds long.
We employ a block-wise processing strategy, using a non-overlap sliding block
of 4 seconds. For simplicity, we just verify the performance of two benchmark
spatial filters enhancing the target sound from the ego-noise. The first
spatial filter is a beamformer based on Multichannel Wiener filtering [14]
which computes the correlation matrix of the target sound and the noise
separately assuming the speech-only and noise-only signals are available. The
second spatial filter is based on blind source separation (BSS) [13], assuming
the permutation ambiguities can be perfectly solved by referencing to the
speech-only signals. The speech enhancement performance is evaluated with the
SNR measure, which is defined, given speech $s(n)$ and noise $v(n)$, as [32]

$\text{SNR}=10\log_{10}\frac{\sum_{n}{s^{2}(n)}}{\sum_{n}{v^{2}(n)}}$ (1)

We average the output SNR across all the processing blocks.

Fig. 9 depicts the output SNR achieved by the two spatial filters at different
input SNRs. BSS performs slightly better than beamformer when the input SNR is
lower than -15 dB, while beamformer performs better at higher input SNRs. On
average, the two spatial filters improve the SNR by about 20 dB.

Fig. 10 illustrates exemplar processing results at input SNR -20 dB with the
beamformer. The output SNR is 3.1 dB. Fig. 10(b) shows that the speech signal
is completely buried in the ego-noise in the time-domain waveform and not
distinguishable between each other in the time-frequency spectrogram. Fig. 10
shows the enhanced speech after processing, where the speech is better
observed in the time-frequency spectrogram.

It should be noted that the two spatial filters are estimated with ideal
assumption (i.e. the correlation matrix of the target and the noise are known)
and thus set the benchmark of the performance of spatial filtering. In
practice, the correlation matrices of the target and noise have to be
estimated from the noisy data, which leads to performance drop in low-SNR
scenarios [14]. A comprehensive evaluation will be left for future work.

##  VI Conclusion

We present an embedded multichannel sound acquisition system that can fly with
the drone. The system can accommodate up to 8 microphones placed in an
arbitrary shape, record the sound locally and can transfer the recorded file
to a remote terminal via a self-organized wireless network. Experimental
results with recordings made with this hardware verify its validity. This will
be the first stage towards creating a fully embedded solution for drone
audition.

Future work would be to conduct a comprehensive evaluation of the state-of-
the-art algorithms for ego-noise reduction and to optimize the code for real-
time processing at Bela, which is able to process audio at very low latency
($<$1 millisecond) [21]. The size and weight of the system can be furthered
reduced by designing the microphone array circuit manually.

## Appendix: Sound recording

Figure 11: Source code in render.cpp

Fig. 11 lists the C++ source code file that is used for multichannel sound
recording with the Bela device. There are several crucial configurations for
sound recording: the file path, the number of channels and the sampling rate.
The file path can be configured by setting the global variable const char*
path in the source code, e.g. the command const char* path =
"/mnt/usb/audiofilename" sets USB storage as the file path. The system can
automatically recognize the the amount of active audio inputs and thus does
not need to configure the number of channels. The sampling rate can be
configured by setting the global variable gAudioSampleRate. The ADC and DAC
gain is adjustable within the IDE settings tab. Once the recording is
finished, the file on the USB storage can be downloaded through IDE after
copying them to the project folder, e.g. using the command cp
/mnt/usb/audiofile.wav /root/Bela/projects/projectname/. Alternatively, we can
remove the USB storage from Bela and insert it into a computer for data
transfer.

A breakdown and interpretation of the source code is given below.

⬇

#include <Bela.h>

#include <libraries/Pipe/Pipe.h>

#include <libraries/sndfile/sndfile.h>

#include <libraries/WriteFile/WriteFile.h>

There are four main library imports that are used in the code. Bela.h is the
central control code for hard real-time audio on BeagleBone Black using PRU
and Xenomai Linux extensions. The pipe library enables the use of a bi-
directional pipe that allows for data to be exchanged between realtime and
non-realtime thread. The Writefile library is imported to enable the use of
the generateUniqueFilename function, that returns a unique filename appending
a number at the end of the original filename. This is important in order to
avoid overwriting existing recordings. The sndfile library allows the use of
the libsndfile API (application programming interface) which is designed to
allow the reading and writing of many different sampled sound file formats.

⬇

const char* path = ”/mnt/usb/audioCh.wav”;

SNDFILE * outfile2;

char originalFilename[] = ”/mnt/usb/audioCh.wav”;

char* uniqueFilename = WriteFile::generateUniqueFilename(originalFilename);

The file path is created first, in this case the usb is used as the main
storage. The outfile2 is the name of the reference for the SNDFILE pointer.
The path to the original filename is assigned and then the
generateUniqueFilename function is called on the original filename to return a
unique filename.

Instead of recording the audio to the USB storage, the filepath can be altered
to const char* path = "./audiofilename" in order to have the file appear and
update during the recording process within the resources section of the
project explorer tab in the IDE.

⬇

AuxiliaryTask gFillBufferTask;

unsigned int gAudioFrames;

unsigned int gAudioInChannels;

float gAudioSampleRate;

Pipe gPipe;

The global variables are established for use later in the code. The
gFillBufferTask is a Auxiliary task variable that is used to write to the
audio buffer. The gPipe, gAudioSampleRate, gAudioInChannels and gAudioFrames
are global variables that utilise the established behaviours present in their
corresponding library class files.

⬇

void openFile() {

SF_INFO sfinfo;

sfinfo.channels = gAudioInChannels;

sfinfo.samplerate = gAudioSampleRate;

sfinfo.format = SF_FORMAT_WAV | SF_FORMAT_PCM_16;

outfile2 = sf_open(uniqueFilename, SFM_WRITE, &sfinfo);

}

The openFile function details the SF_INFO structure and the specified file
format, sample rate, amount of channels. The sf_open function opens the sound
file at the specified path and utilises the write only mode SFM_WRITE and the
sfinfo structure for passing data between the calling function and the library
when opening the file for in this case writing.

⬇

void closeFile() {

sf_write_sync(outfile2);

sf_close(outfile2);

printf(”.wav␣file␣written␣and␣closed\n”);

}

The closeFile function closes and writes the file to disk. sf_write_sync
allows for the file if it is opened using SFM_WRITE to call the operating
system’s function to force the writing of all file cache buffers to disk.
sf_close closes the file, deallocates its internal buffers and returns 0 on
success or an error value otherwise.

⬇

void writeBuffer(void*) {

unsigned int numItems = gAudioFrames * gAudioInChannels;

float buf[numItems];

int ret;

while((ret = gPipe.readNonRt(buf, numItems) ) > 0)

{

sf_write_float(outfile2, &buf[0], ret);

}

}

The writeBuffer function essentially writes data to the buffer. This function
calculates the number of items by multiplying the audio frames and audio input
channels. A buffer array holds the audio frame and input channel items. An
integer variable is declared with the purpose of holding the number of items.
The while loop will loop through the block of code as long as the specified
condition is true. The readNonRt reads data from the non-realtime side. The
sf_write_float function writes the data in the array pointing to the pointer
of the file. For items-count functions, the items parameter specifies the size
of the array and must be an integer product of the number of channels or an
error will occur.

⬇

bool setup(BelaContext* context, void* arg)

{

gAudioSampleRate = context->audioSampleRate;

gAudioFrames = context->audioFrames;

gAudioInChannels = context->audioInChannels;

gPipe.setup(”sndfile-write”, 65536, false, false);

openFile();

if((gFillBufferTask = Bela_createAuxiliaryTask(&writeBuffer, 90,
”writeBuffer”))

== 0) {

return false;

}

return true;

}

The setup function is a user-defined initialisation function which runs before
audio rendering begins. This function runs once at the beginning of the
program, after most of the system initialisation has begun but before audio
rendering starts. This is used to prepare any memory or resources that will be
needed in render. The audio sample rate, frames and audio input channels are
all setup using the Bela context structure. The pipe is setup to write data
with a specified size and whether the reads at the realtime and non-realtime
side should be blocking. The openFile function is called to open the file for
writing the audio data. The auxiliary task is then created with write buffer
parameters.

⬇

void render(BelaContext* context, void* arg)

{

gPipe.writeRt(context->audioIn, context->audioFrames *
context->audioInChannels);

Bela_scheduleAuxiliaryTask(gFillBufferTask);

}

The render function is a user-defined callback function to process audio and
sensor data. This function is called regularly by the system every time there
is a new block of audio and/or sensor data to process. The writeRt function
reads data from the non-realtime side. The context->audioIn is the float* that
points to all the input samples, stored as interleaved channels. The audioIn
is an array of 4 frames * 2 channels = 8 audio input samples. The auxiliary
task which has previously been created is scheduled to run.

⬇

void cleanup(BelaContext* context, void* arg)

{

closeFile();

free(uniqueFilename);

}

The cleanup function runs when the program finishes to free up memory. This
function is called by the system once after audio rendering has finished,
before the program quits. It is used to release any memory allocated in setup
and to perform any other required cleanup. If no initialisation is performed
in setup, then this function will usually be empty. The file is closed and the
file that has been created by the generateUniqueFilename has its block of
memory deallocated.

## References

  * [1] D. Floreano, D. and R. J. Wood, “Science, technology and the future of small autonomous drones,” Nature, vol. 521, no. 7553, pp. 460-466, 2015.
  * [2] G. Parascandolo, H. Huttunen, and T. Virtanen, “Recurrent neural networks for polyphonic sound event detection in real life recordings,” in Proc. IEEE Int. Conf. Acoust., Speech Signal Process., Shanghai, China, 2016, pp. 6440-6444.
  * [3] S. Li and D. Yeung, “Visual object tracking for unmanned aerial vehicles: A benchmark and new motion models,” in Proc. Thirty-First AAAI Conf. Artificial Intelligence, San Francisco, USA, 2017, pp. 4140-4146.
  * [4] P. Misra, A. A. Kumar, P. Mohapatra, and P. Balamuralidhar, “Aerial drones with location-sensitive ears,” IEEE Commun. Mag., vol. 56, no. 7, pp. 154-160, Jul. 2018.
  * [5] L. Wang, R. Sanchez-Matilla, and A. Cavallaro, “Tracking a moving sound source from a multi-rotor drone,” in Proc. IEEE/RSJ Int. Conf. Intell. Robots Sys., Madrid, Spain, 2018, pp. 2511-2516.
  * [6] R. Sanchez-Matilla, L. Wang, and A. Cavallaro, “Multi-modal localization and enhancement of multiple sound sources from a micro aerial vehicle,” Proc. ACM Multimedia, Silicon Valley, USA, 2017, pp. 1591-1599.
  * [7] L. Wang and A. Cavallaro, “Acoustic sensing from a multi-rotor drone,” IEEE Sensors J., vol. 18, no. 11, pp. 4570-4582, Nov. 2018.
  * [8] A. Deleforge, D. Di Carlo, M. Strauss, R. Serizel, and L. Marcenaro, “Audio-based search and rescue with a drone: highlights from the IEEE signal processing cup 2019 student competition,” IEEE Signal Process. Mag., vol. 36, no. 5, pp. 138-144, Sep. 2019.
  * [9] A. Schmidt, H. W. Lollmann, and W. Kellermann, “A novel ego-noise suppression algorithm for acoustic signal enhancement in autonomous systems,” in Proc. IEEE Int. Conf. Acoust., Speech Signal Process., Calgary, Canada, 2018, pp. 6583-6587.
  * [10] B. Kang, H. Ahn, and H. Choo, “A software platform for noise reduction in sound sensor equipped drones,” IEEE Sensors J., vol. 19, no. 21 pp. 10121-10130, Nov. 2019.
  * [11] Y. Hioka, M. Kingan, G. Schmid, R. McKay, and K. A. Stol, “Design of an unmanned aerial vehicle mounted system for quiet audio recording,” Appl. Acoust., vol. 155, pp. 423-427, 2019.
  * [12] L. Wang and A. Cavallaro, “Deep learning assisted time-frequency processing for speech enhancement on drones,” IEEE Trans. Emerging Topics Computational Intelligence, 2020, Early Access, DOI: 10.1109/TETCI.2020.3014934.
  * [13] L. Wang and A. Cavallaro, “A blind source separation framework for ego-noise reduction on multi-rotor drones,” IEEE/ACM Trans. Audio Speech Lang. Process., vol. 28, pp. 2523-2537, 2020.
  * [14] L. Wang and A. Cavallaro, “Microphone-array ego-noise reduction algorithms for auditory micro aerial vehicles,” IEEE Sensors J., vol. 17, no. 8, pp. 2447-2455, Aug. 2017.
  * [15] B. Yen and Y. Hioka, “Noise power spectral density scaled SNR response estimation with restricted range search for sound source localisation using unmanned aerial vehicles,” EURASIP J. Audio Speech Music Process., vol. 2020, no. 1, pp. 1-26, 2020.
  * [16] L. Wang and A. Cavallaro, “Time-frequency processing for sound source localization from a micro aerial vehicle,” in Proc. IEEE Int. Conf. Acoust., Speech Signal Process., New Orleans, USA, 2017, pp. 496-500.
  * [17] M. Strauss, P. Mordel, V. Miguet, and A. Deleforge, “DREGON: dataset and methods for UAV-embedded sound source localization,” in Proc. IEEE/RSJ Int. Conf. Intell. Robot. Syst., Madrid, Spain, 2018, pp. 5735-5742.
  * [18] M. Wakabayashi, H. G. Okuno, and M. Kumon, “Drone audition listening from the sky estimates multiple sound source positions by integrating sound source localization and data association,” Advanced Robotics, pp. 1-12, 2020.
  * [19] K. Hoshiba, K. Washizaki, M. Wakabayashi, T. Ishiki, M. Kumon, Y. Bando, D. Gabriel, K. Nakadai, and H.G. Okuno, “Design of UAV-embedded microphone array system for sound source localization in outdoor environments,” Sensors, vol. 17, no. 11, pp. 1-16, Nov. 2017.
  * [20] K. Furukawa, K. Okutani, K. Nagira, T. Otsuka, K. Itoyama, K. Nakadai, and H. G. Okuno, “Noise correlation matrix estimation for improving sound source localization by multirotor UAV,” in Proc. IEEE/RSJ Int. Conf. Intell. Robot. Syst., Tokyo, Japan, 2013, pp. 3943-3948.
  * [21] A. McPherson and V. Zappi, “An environment for submillisecond-latency audio and sensor processing on BeagleBone Black,” in Proc. Audio Engineering Society Convention 138, 2015, pp. 1-7.
  * [22] A. McPherson, H. J. Robert, and G. Moro, “Action-sound latency: Are our tools fast enough?”, in Proc. Int. Conf. New Interfaces Musical Expression, Brisbane, Australia, 2016, pp. 1-6.
  * [23] M. B. Andra, B. Rohman, and T. Usagawa, “Feasibility evaluation for keyword spotting system using mini microphone array On UAV”, in Proc. IEEE International Geoscience Remote Sensing Symp., Yokohama, Japan, 2019, pp. 2264-2267.
  * [24] Z. W. Tan, A. H. Nguyen, and A. W. Khong, “An efficient dilated convolutional neural network for UAV noise reduction at low input SNR,” in Proc. Asia-Pacific Signal Information Process. Association Annual Summit Conf., Lanzhou, China, 2019, pp. 1885-1892.
  * [25] D. Salvati, C. Drioli, G. Ferrin, and G. L. Foresti, “Acoustic source localization from multirotor UAVs”, IEEE Trans. Industrial Electronics, vol. 67, no. 10, pp. 8618-8628, Oct. 2020.
  * [26] K. Okutani, T. Yoshida, K. Nakamura, and K. Nakadai, “Outdoor auditory scene analysis using a moving microphone array embedded in a quadrocopter,” in Proc. IEEE/RSJ Int. Conf. Intell. Robot. Syst., Vilamoura-Algarve, Portugal, 2012, pp. 3288-3293.
  * [27] L. Wang, R. Sanchez-Matilla, and A. Cavallaro, “Audio-visual sensing from a quadcopter: dataset and baselines for source localization and sound enhancement,” in Proc. IEEE/RSJ Int. Conf. Intell. Robots Sys., Macao, China, 2019, pp. 5320-5325.
  * [28] O. Ruiz-Espitia, J. Martinez-Carranza, and C. Rascon. “AIRA-UAS: An evaluation corpus for audio processing in unmanned aerial system,” in Proc. Int. Conf. Unmanned Aircraft Systems, Dallas, USA, 2018, pp. 836-845.
  * [29] K. Nakadai, H. G. Okuno, and T. Mizumoto, “Development, deployment and applications of robot audition open source software HARK,” J. Robotics and Mechatronics, vol. 29, no. 1, pp. 16-25, Jan. 2017.
  * [30] L. Wang and A. Cavallaro, “Ear in the sky: Ego-noise reduction for auditory micro aerial vehicles”, in Proc. Int. Conf. Adv. Video Signal-Based Surveillance, Colorado Springs, USA, 2016, pp. 152-158.
  * [31] H. Langer and R. Manzke, “Embedded multichannel Linux audiosystem for musical applications,” in Proc. Int. Audio Mostly Conf. Augmented Participatory Sound Music Experiences, London, UK, 2017, pp. 1-5.
  * [32] L. Wang, T. Gerkmann, and S. Doclo, “Noise power spectral density estimation using MaxNSR blocking matrix,” IEEE/ACM Trans. Audio, Speech, Lang. Process., vol. 23, no. 9, pp. 1493-1508, Sep. 2015.