# A General Framework for Learning Procedural
Audio Models of Environmental Sounds

###### Abstract

This paper introduces the Procedural (audio) Variational autoEncoder (ProVE)
framework as a general approach to learning Procedural Audio PA models of
environmental sounds with an improvement to the realism of the synthesis while
maintaining provision of control over the generated sound through adjustable
parameters. The framework comprises two stages: (i) Audio Class
Representation, in which a latent representation space is defined by training
an audio autoencoder, and (ii) Control Mapping, in which a joint function of
static/temporal control variables derived from the audio and a random sample
of uniform noise is learned to replace the audio encoder. We demonstrate the
use of ProVE through the example of footstep sound effects on various
surfaces. Our results show that ProVE models outperform both classical PA
models and an adversarial-based approach in terms of sound fidelity, as
measured by Fréchet Audio Distance (FAD), Maximum Mean Discrepancy (MMD), and
subjective evaluations, making them feasible tools for sound design workflows.

Index Terms—  Procedural Audio, Neural Audio Synthesis, Controllable
Generative Models

##  1 Introduction

The job of the sound designer is to record, synthesize, and process audio
stimuli for immersive mediums such as film, video games, and virtual
environments in such a way that complements visual or other stimuli. The
current paradigm typically involves scanning through extensive audio databases
for a set of recorded sounds of a sound class $C$ and preprocessing them to
fit a project’s needs [1]. This process is time-consuming and may result in a
lack of realism if suitable sounds cannot be found or processed. Storage
limitations may also be a problem, arising from the need for a large number of
distinct recordings for a highly variable sound class, especially in
interactive media such as video games and virtual environments where sound
events are influenced by the user’s context.

In mitigating the aforementioned problems, Procedural audio (PA) is a
different approach to traditional sound design where a sound class is modeled
as a mathematical process [2]. This results in models which reduce storage
limitations, have greater control and expressiveness, and have the capacity to
create unique auditory experiences. PA models differ from physical modeling
synthesis [3], which render sounds through simulations in computational
physics. These physics-based models can produce realistic and dynamic sounds,
but are computationally expensive and require a significant amount of domain
knowledge to develop. On the other hand, PA models are simpler and more
computationally efficient, using algorithms to generate sound based on static
and temporal control variables, usually accompanied by random noise to span
variations in synthesis. Despite its advantage in computational efficiency,
current classical PA models still synthesize sounds of lower quality compared
to using real samples or physical modeling synthesis — a primary reason why
they are not yet in standard use in sound design [2, 4].

The state-of-the-art for enhancing sound synthesis quality involves data-
driven neural audio synthesis, the subset of deep learning techniques for
generative audio. Neural audio synthesis has been extensively researched for
speech synthesis [5, 6, 7] and music generation [8, 9], but its application to
specific environmental sounds or Foley synthesis for immersive media has only
recently been investigated [10]. Furthermore, the topic of controllability in
these models is well established in speech [11] and music [8, 9, 12] domains,
but not yet for environmental sounds. Recently a paper used generative
adversarial approaches [13, 14] to model the environmental sound class of
footsteps on various surface materials [15]. Although the model results in a
sample space of high-fidelity footstep sounds, its controllability is limited
to the surface material selection and its expressivity doesn’t account for
temporal context, such as locomotion speed or the extent of force from the
foot to the ground.

Fig. 1: (i) and (ii) depict the two-stage framework of ProVE, where (i) trains
a class-specific autoencoder on the audio dataset $\textbf{s}_{i}\sim
p_{class}$, and (ii) trains a control parameter encoder with static condition
label, temporal control proxy derived from audio
$\boldsymbol{\hat{\gamma}}_{i}=C\\{\textbf{s}_{i}\\}$, and noise sequence.
(iii) is the final ProVE model, synthesizing audio from control parameters
using the trained control encoder and audio decoder.

##  2 ProVE Framework

###  2.1 Overview

PA models generate sound from static and temporal control variables, usually
accompanied by random noise. An example is the classical PA model for
footsteps developed by Andy Farnell in [16]. At the core of Farnell’s model is
a set of texture synthesis modules, one for each surface. A static surface
selection control selects which texture synthesis module is active, and a
ground reaction force (GRF) curve generator temporally controls the active
texture synthesis module to transform uniform noise into footstep sounds.
Farnell developed the GRF curve generator through the analysis of bipedal
locomotion. The generator is a function of 3 polynomial segments that repeats
its values over fixed intervals. The 3 polynomial segments describe ground
reaction force for the phases of a human’s footstep, i.e., heel, roll from
heel to ball, and ball, resulting in an interpretable synthesis control signal
(see Figure 2).

We propose the Procedural (audio) Variational autoEncoder (ProVE), a two-stage
analysis-synthesis approach to learning PA models which aims to improve
fidelity, while retaining the properties of being controllable and expressive.
This approach uses a neural audio synthesis decoder comprised of
differentiable digital signal processing (DDSP) layers [17]. More
specifically, we synthesize an environmental sound class with a differentiable
filtered noise module, governed by a control signal obtained through analysis,
as opposed to the models that predict Fourier coefficients in the spectral
domain [5, 18], autoregressive time-domain approaches [19, 20], as well as
previously mentioned adversarial approaches, as they are typically unstable to
train [21].

Fig. 2: Sample curve from Farnell’s GRF curve generator [16].

Given the highly variable nature of environmental sounds, ProVE aims to learn
a controlled stochastic process. Here, $s[n]=\\{s_{n}[m]\\}_{m=0}^{\delta-1}$
is a small window of discrete audio of length $\delta$, sampled from:

$s[n]\sim q(l,\gamma[n],u[n])$ (1)

where $q$ is a function of sampled stochastic noise
($u[n]\sim\mathcal{U}{[-1,1]}^{n_{\gamma}}$) dependent on static
($l\in\mathbb{R}^{n_{l}}$) and temporal
($\gamma[n]\in\mathbb{R}^{n_{\gamma}}$) deterministic control. Let $p_{class}$
denote the prior of discrete signals of some sound class of length $L$. When
we sample a discrete signal through overlapping and adding a sequence of
sampled small windows $\textbf{s}=\text{OLA}(\\{s[n]\\}_{n=0}^{K-1})$ where
$K\geq\frac{L}{\delta}$, ideally we would want $\textbf{s}\sim p_{class}$.
This is opposed to models that directly output a sample array of fixed length
$L$ from a static condition label, like the adversarial approach, and allows
us to define our own instances of control with $l$ and
$\boldsymbol{\gamma}=\\{\gamma[n]\\}_{n=0}^{K-1}$.

The ProVE framework comprises two steps, the first of which is to learn a
latent audio representation space which is viewed as “machine-interpretable”
control space, and the second is to learn a mapping from “human-interpretable”
control (i.e. GRF Curves with surface selection) to its corresponding
“machine-interpretable” control. Our training dataset, indexed by $i$,
regardless of sound class, is formatted as 4-tuples:
$(\textbf{s}_{i},l_{i},\boldsymbol{\hat{\gamma}}_{i},\textbf{u}_{i})$.
$\textbf{s}_{i}$ is a discrete-time audio signal from a target sound class
($\textbf{s}_{i}\sim p_{class}$) of length $L$, $l_{i}$ is a static condition
label associated with $\textbf{s}_{i}$, and $\boldsymbol{\hat{\gamma}}_{i}$ is
an approximate temporal control variable that is proxied through signal
analysis on audio $\textbf{s}_{i}$. That is,
$\boldsymbol{\hat{\gamma}}_{i}=\\{\hat{\gamma}_{i}[n]\\}_{n=0}^{K-1}$ is a
proxy to the actual $\boldsymbol{\gamma}_{i}$ which governs the temporal
context of audio $\textbf{s}_{i}$, derived from prior knowledge of
$p_{class}$. Finally, $\textbf{u}_{i}$ is a sequence of random noise assumed
to be assigned to $\textbf{s}_{i}$.

###  2.2 Step 1: Audio Class Representation

Step 1 is to train an autoencoder model to learn a latent sequence
representation $\textbf{z}_{i}=\\{z_{i}[n]\\}_{n=0}^{K-1}$ of audio
$\textbf{s}_{i}\sim p_{class}$. Since this is a general framework, both the
encoder $\textbf{z}_{i}=f_{\theta}(\textbf{s}_{i})$ and decoder
$\hat{\textbf{s}}_{i}=g_{\phi}(\textbf{z}_{i})$ models are architecture-
agnostic. However, in our experiments described in Section 3 we use the
encoder and decoder models from the DDSP library by magenta [17] for
experiments due to its success in learning expressive and controllable
generative models of speech [22] and musical instruments [12], and its ability
to synthesize high-quality audio with relatively good fidelity using smaller
datasets. A trained autoencoder learns a latent sequence space that one can
think of as a “machine-interpretable” control space to the synthesis module.
The autoencoder is optimized using multi-scale spectral loss as mentioned in
[17].

###  2.3 Step 2: Control Mapping

Because the latent control sequence space from step 1 is not “human-
interpretable”, a second step is needed to allow for a controllable model. In
this step, a control encoder
$\hat{z}_{i}[n]=h_{\psi}(l_{i},\hat{\gamma}_{i}[n],u_{i}[n])$ is trained to
approximate the “machine-interpretable” control sequence $\textbf{z}_{i}$,
from the tuple $(l_{i},\boldsymbol{\hat{\gamma}}_{i},\textbf{u}_{i})$ by
optimizing $h_{\psi}$ on $L_{2}$-loss between $\textbf{z}_{i}$ and
$\boldsymbol{\hat{z}}_{i}=\\{\hat{z}_{i}[n]\\}_{n=0}^{K-1}$ . After training,
we compose the sythesis model $q(l,\gamma[n],u[n])=(g_{\phi}\circ
h_{\psi})(l,\gamma[n],u[n])$ from the control encoder $h_{\psi}$ and the audio
decoder $g_{\phi}$.

A limitation of current ProVE framework is the requirement for a simple
control variable proxy $\boldsymbol{\hat{\gamma}}_{i}=C\\{\textbf{s}_{i}\\}$
derived from some analysis procedure $C$ on audio $\textbf{s}_{i}$. In future
work, we aim to mitigate this limitation by developing a procedure $C$ from
existing controllable models targeting $p_{class}$, or without prior knowledge
of $p_{class}$ directly from $\textbf{s}_{i}$.

##  3 Experiments

In this section we demonstrate the training of a ProVE model for footstep
sound effects, and we compare objectively and subjectively to baselines of
Farnell’s classical PA model, as well as the GAN approach in [15]. An
interactive web application 111https://dependanz.github.io/projects/prove.html
is supplemented where users can provide control variables to synthesize from a
pretrained ProVE model of footstep sounds.

The ProVE models trained on the footstep sound effect datasets have a static
control for the selection of surface material like in the adversarial
approach. The temporal control variables $\boldsymbol{\gamma_{i}}$ are the GRF
curves, and its approximation $\boldsymbol{\hat{\gamma}}_{i}$ was defined to
be the smoothed envelope of the footstep sounds. That is,
$\boldsymbol{\hat{\gamma}}_{i}=C\\{\textbf{s}_{i}\\}$ where $C$ is the 1D
Laplacian smoothing of the magnitude of the discrete Hilbert transform.

The motivation behind the choice of proxy was that the magnitude of the
exertion of force influences the magnitude of the conversion of energy which
influences the loudness of an impact-based sound. The control proxy is also
shown to have a 3-polynomial structure for each individual footstep that
occurs in $x_{i}$, as can be seen in figure 3.

Fig. 3: Smoothed envelope of footstep sounds.

###  3.1 Datasets

We train and evaluate ProVE models for footstep sounds on two publicly
available datasets. The first is a set of high-heel footstep sounds from the
Zapsplat222https://www.zapsplat.com/sound-effect-packs/footsteps-in-high-heels
website. This is the same dataset used in [15], and we use it to compare ProVE
to the adversarial approach presented in [15]. The dataset contains one-shot
sounds ( 1 sec) and is categorized by ground material, however, the
demonstrations in the evaluation section show a ProVE model which was trained
on long segments of footsteps walking on a specific material ( 10 seconds)
from the same dataset. The reason to train with longer audio segments instead
of one-shots is that a single footstep usually doesn’t occur in isolation but
rather in a sequence of footsteps that has temporal structure. Thus, to
improve the accuracy of PA models, we consider longer audio segments that
capture this temporal structure. Bipedal acceleration sound perceptually
different than bipedal deceleration, and that is due to how the control
variables (GRF Curves), as well as its corresponding sound, behave over time.

The other dataset is from
PremiumBeat.com333https://www.premiumbeat.com/blog/40-free-footstep-foley-
sound-effects/ and is a set of long recordings ($\sim$15-20 secs) classified
by surface material and locomotion speed (Walking, Running, etc). We use this
dataset to test the ability of ProVE in learning higher fidelity synthesis
while also retaining the same expressivity of classical PA.

###  3.2 Training

For training, we resample the input audio to $f_{s}=16$ kHz sample rate for a
fair comparison to the baseline models, but DDSP is easily scalable to higher
sampling rates. We use a $f_{c}=250$ Hz control rate for $\textbf{z}_{i}$ and
$(\boldsymbol{\gamma}_{i},\textbf{u}_{i})$. The main audio encoder
$\textbf{z}_{i}=f_{\theta}(\textbf{s}_{i})$ used for training and evaluation
is a recurrent neural network from the paper that proposed DDSP [17], which
processes normalized mel-frequency cepstral coefficients (MFCCs). The first 13
normalized MFCCs are then passed to a gated recurrent unit (GRU), whose
outputs pass through a final linear layer with a tanh activation to obtain
latent control sequence $\textbf{z}_{i}$, where $z_{i}[n]\in(-1,1)^{n_{z}}$
with a choice of $n_{z}=512$. The audio decoder $g_{\theta}(\textbf{z}_{i})$
is a recurrent neural network which parameterizes the DDSP subtractive noise
synthesizer. The control encoder
$\hat{z}_{i}[n]=h_{\psi}(l_{i},\hat{\gamma}_{i}[n],u_{i}[n])$ is similar to
the audio encoder, in that ($\boldsymbol{\gamma}_{i}$, $\textbf{u}_{i}$) is
normalized, broadcast-multiplied by a learnable embedding obtained with
$l_{i}$, passed through a GRU and a final linear layer with the tanh
activation, giving us $\hat{z}_{i}[n]\in(-1,1)^{n_{z}}$

During inference, a ProVE footstep sound model is given by
$\boldsymbol{\hat{s}}=\text{OLA}(\\{\hat{s}[n]\\}_{n=0}^{K-1})$ where
$\hat{s}[n]\sim q(l,\gamma[n],u[n])=(g_{\phi}\circ
h_{\psi})(l,\gamma[n],u[n])$. The model is controlled by providing: (i)
surface material condition label $l$, (ii) a GRF curve (e.g., a sample from
Farnell’s GRF curve generator) as the temporal control $\boldsymbol{\gamma}$,
(iii) a sample sequence of random noise u.

###  3.3 Objective Evaluation

We use data-driven and perceptually motivated metrics Frechét Audio Distance
(FAD) [23] and Maximum Mean Discrepancy (MMD) [24] to compare the synthesis
from the ProVE model of footstep sounds to classical PA models and the recent
adversarial approach in [15] (denoted GAN). Both of these metrics calculate a
distance between audio distributions. To evaluate the models, we compute FAD
and MMD between the model outputs, the training set of high-heel sounds
(Heels) from the Zapsplat website, and the rest of the footstep sounds from
the same website (Misc). As the quality of synthesized audio (GAN, PA, ProVE)
increases, the distribution distance to real audio datasets (Heels, Misc)
should decrease. In order to compare to the results of GAN, we strictly
followed the same sampling and evaluation processes of [15], where 120k one
second samples from Farnell’s GRF curve were used to sample from the PA and
ProVE models. With the generated footstep sounds of both models, we calculate
FAD and MMD.

Fig. 4: (i) Frechét Audio Distances and (ii) Maximum Mean Discrepancies of
Real and Synthesized Footstep Sound Datasets

The main results in Figure 4 show that the classical PA model is the furthest
away from both Heels and Misc. The results of calculating FAD also show the
ProVE model is closer to the training data (Heels) than the GAN model is, with
an FAD of 2.41. However, the FAD between the two models and Misc shows GAN is
closer with an FAD of 3.13 whereas the ProVE model has an FAD 3.96. The
results on MMD are similar to FAD, with the ProVE model closer to Heels
(42.71) and GAN further away (53.5).

These findings suggest that the ProVE model surpasses the PA model in terms of
fidelity, where FAD and MMD of the ProVE model are significantly closer to
both datasets, enforcing that the PA model lacks realism. This also shows that
the ProVE model has a consistent distance in both datasets, where the model is
closer to the training Heels dataset, and is proportionally further away to
the out-of-distribution dataset Misc. However, the FAD of the GAN and ProVE
models to the Misc dataset are not dissimilar, and it can still be argued that
the ProVE model also has the ability to generate samples beyond the training
set.

###  3.4 Subjective Evaluation

We conducted subjective listening tests to compare the ProVE model, the
classical PA model, and the training dataset, which was the set of long
recordings from the PremiumBeat website (denoted PB in table 2). We input the
smoothed envelope of 8 second snippets of these recordings as the GRF curves
to the ProVE and PA models. This was done in order to align the synthesized
samples with real recordings for a listening test. We used a modification of
pairwise tests in the Reproducible Subjective Evaluation (reseval) framework
[25] such that participants were asked to choose which of two footsteps sounds
more closely matched a descriptive textural prompt. We recruited 50
participants using Amazon Mechanical Turk with reseval, and each participant
completed 10 pairwise comparisons between the synthesized and real
distributions. We compensated participants with a base payment and a bonus for
completing the entire survey. Tables 1 and 2 display the proportion of
participants that preferred ProVE over PA or the real dataset (in Table 2),
where the p-values are from two-sided binomial tests.

p-value = $6\times 10^{-7}$ | Dirt | Grass | Gravel | Wood
---|---|---|---|---
ProVE | 0.77 | 0.705 | 0.371 | 0.81
PA | 0.23 | 0.295 | 0.629 | 0.19

Table 1: Results of Subjective Evaluation (ProVE, PA) p-value = $0.1$ | Dirt | Grass | Gravel | Wood
---|---|---|---|---
ProVE | 0.5 | 0.486 | 0.431 | 0.387
Real (PB) | 0.5 | 0.514 | 0.569 | 0.613
Table 2: Results of Subjective Evaluation (ProVE, PB)

The results indicate that ProVE and classical PA models differ significantly
by surface material, while comparison between ProVE and the real dataset are
more similar. Participants preferred the synthesis of ProVE over classical PA
for dirt, grass, and wood, but they preferred classical PA over ProVE for
gravel. This is possibly due to the fine-grained sounds of the trajectory of
small pebbles and rocks from gravel, which is not accounted for by simply
using a control variable proxy of the smoothed envelope of footstep sounds.

##  4 Conclusion

The Procedural (audio) Variational autoEncoder (ProVE) was proposed as a
general framework for learning high-fidelity procedural audio models of
environmental sounds. ProVE models map from a “human-interpretable” to
“machine-interpretable” control space learned from an audio autoencoder and
surpass classical PA models in terms of fidelity while retaining
controllability as a controlled stochastic process. Possible research
directions from here include shortening inference time for real time audio
synthesis, as well as using ProVE for automatic Foley generation from
video/animation. The implications from ProVE extend beyond audio signals to
procedural signal generation in general, as a step towards controllability in
latent spaces.

## References

  * [1] Mats Liljedahl and Johan Fagerlönn,  “Methods for sound design: A review and implications for research and practice,”  in Proceedings of the 5th Audio Mostly Conference: A Conference on Interaction with Sound, New York, NY, USA, 2010, AM ’10, Association for Computing Machinery.
  * [2] Andy Farnell,  “An introduction to procedural audio and its application in computer games,” 2007.
  * [3] Julius O. Smith,  Physical Audio Signal Processing,  http://ccrma.stanford.edu/$\sim$jos/pasp/, accessed 02/12/23,  online book, 2010 edition.
  * [4] Niels Böttcher,  “Current problems and future possibilities of procedural audio in computer games,”  Journal of Gaming & Virtual Worlds, vol. 5, no. 3, pp. 215–234, Sept. 2013.
  * [5] Yuxuan Wang, RJ Skerry-Ryan, Daisy Stanton, Yonghui Wu, Ron J. Weiss, Navdeep Jaitly, Zongheng Yang, Ying Xiao, Zhifeng Chen, Samy Bengio, Quoc Le, Yannis Agiomyrgiannakis, Rob Clark, and Rif A. Saurous,  “Tacotron: Towards end-to-end speech synthesis,” 2017.
  * [6] Jonathan Shen, Ruoming Pang, Ron J. Weiss, Mike Schuster, Navdeep Jaitly, Zongheng Yang, Zhifeng Chen, Yu Zhang, Yuxuan Wang, RJ Skerry-Ryan, Rif A. Saurous, Yannis Agiomyrgiannakis, and Yonghui Wu,  “Natural tts synthesis by conditioning wavenet on mel spectrogram predictions,” 2017.
  * [7] Yi Ren, Chenxu Hu, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and Tie-Yan Liu,  “Fastspeech 2: Fast and high-quality end-to-end text to speech,” 2020\.
  * [8] Gautam Mittal, Jesse Engel, Curtis Hawthorne, and Ian Simon,  “Symbolic music generation with diffusion models,” 2021.
  * [9] Prafulla Dhariwal, Heewoo Jun, Christine Payne, Jong Wook Kim, Alec Radford, and Ilya Sutskever,  “Jukebox: A generative model for music,” 2020.
  * [10] Keunwoo Choi, Sangshin Oh, Minsung Kang, and Brian McFee,  “A proposal for foley sound synthesis challenge,” 2022.
  * [11] Meiying Chen and Zhiyao Duan,  “Controlvc: Zero-shot voice conversion with time-varying controls on pitch and rhythm,” 2022.
  * [12] Yusong Wu, Ethan Manilow, Yi Deng, Rigel Swavely, Kyle Kastner, Tim Cooijmans, Aaron Courville, Cheng-Zhi Anna Huang, and Jesse Engel,  “Midi-ddsp: Detailed control of musical performance via hierarchical modeling,” 2021.
  * [13] Jungil Kong, Jaehyeon Kim, and Jaekyoung Bae,  “Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis,” 2020.
  * [14] Chris Donahue, Julian McAuley, and Miller Puckette,  “Adversarial audio synthesis,” 2019.
  * [15] Marco Comunità, Huy Phan, and Joshua D. Reiss,  “Neural synthesis of footsteps sound effects with generative adversarial networks,” 2021.
  * [16] Andy Farnell,  “Marching onwards procedural synthetic footsteps for video games and animation,”  2007\.
  * [17] Jesse Engel, Lamtharn Hantrakul, Chenjie Gu, and Adam Roberts,  “Ddsp: Differentiable digital signal processing,” 2020.
  * [18] Jesse Engel, Kumar Krishna Agrawal, Shuo Chen, Ishaan Gulrajani, Chris Donahue, and Adam Roberts,  “Gansynth: Adversarial neural audio synthesis,” 2019.
  * [19] Nal Kalchbrenner, Erich Elsen, Karen Simonyan, Seb Noury, Norman Casagrande, Edward Lockhart, Florian Stimberg, Aaron van den Oord, Sander Dieleman, and Koray Kavukcuoglu,  “Efficient neural audio synthesis,” 2018.
  * [20] Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu,  “Wavenet: A generative model for raw audio,” 2016.
  * [21] Divya Saxena and Jiannong Cao,  “Generative adversarial networks (gans): Challenges, solutions, and future directions,” 2020.
  * [22] Giorgio Fabbro, Vladimir Golkov, Thomas Kemp, and Daniel Cremers,  “Speech synthesis and control using differentiable dsp,” 2020.
  * [23] Kevin Kilgour, Mauricio Zuluaga, Dominik Roblek, and Matthew Sharifi,  “Fréchet audio distance: A metric for evaluating music enhancement algorithms,” 2018.
  * [24] Arthur Gretton, Karsten M. Borgwardt, Malte J. Rasch, Bernhard Schölkopf, and Alexander Smola,  “A kernel two-sample test,”  J. Mach. Learn. Res., vol. 13, no. null, pp. 723–773, mar 2012\.
  * [25] Max Morrison, Brian Tang, Gefei Tan, and Bryan Pardo,  “Reproducible subjective evaluation,”  in ICLR Workshop on ML Evaluation Standards, April 2022.