Note: The figure plots the estimated coefficients for the explanatory variables of interest in  <ref>, i.e. for the share of the age period from 1 to 6 years during which the individual was exposed to the vaccination program interacted with the measles cases per 100 people prior to the vaccination program and for its interaction with the PGI.

§ AN INTRODUCTION TO GENETICS

Humans have 46 chromosomes stored in every cell of the human body other than gametes (sex-cells). Chromosomes come in pairs; one from one's biological mother and one from one's biological father. A single chromosome consists of a double-strand of deoxyribonucleic acid (DNA) containing a large number of `base pairs': pairs of nucleotide molecules (referred to as the `letters' A (adenine) that binds with T (thymine), and G (guanine) that binds with C (cytosine)) that together make up the human genome. In a population there will be variation in the base pairs at some locations. Such variation is known as a single nucleotide polymorphism (SNP, pronounced `snip') – a change in the base pair at one particular locus (location) – and is the most commonly studied genetic variation. When there are two possible base pairs at a given location (i.e., two alleles), the most frequent base pair is called the major allele, while the less frequent is called the minor allele. As humans have two copies of each chromosome, an individual can have either zero, one, or two copies of the minor allele.

To identify specific SNPs that are robustly associated with a particular outcome of interest, so-called Genome-Wide Association Studies (GWAS) relate each SNP to the outcome in a hypothesis-free approach. As genetic datasets contain more SNPs than individuals, the SNP effects cannot be identified in a multivariate regression model. Instead, a GWAS runs a large number of univariate regressions of the outcome on each SNP. These analyses have shown that most outcomes of interest in the social sciences are `polygenic': they are affected by a large number of SNPs, each with a very small effect. To increase the predictive power of the SNPs, it is therefore customary to aggregate the individual SNPs into so-called polygenic indices (also known as polygenic scores), as:

G_i = \sum_{j=1}^J \beta_j X_{ij},

where $X_{ij}$ is a count of the number of minor alleles (i.e., 0, 1 or 2) at SNP $j$ for  individual $i$, and $\beta_j$ is its effect size obtained from an independent GWAS. Hence, the polygenic indices are weighted linear combinations of SNPs, where the weights are estimated in an independent GWAS. This is motivated by an additive genetic model where all SNPs contribute additively to the overall genetic endowment of an individual  <cit.>.

In our main specification, we use polygenic indices of educational attainment and height, constructed from the summary statistics (i.e., the $\beta_j$'s) derived from a GWAS of educational attainment [Lee et al., 2018] and height [Wood et al., 2014]. Note that the GWAS discovery samples for both PGIs exclude the UK Biobank to avoid over-fitting. We also examine the robustness of our results to the use of differentially-constructed polygenic indices from the polygenic repository [Becker et al., 2021]. We account for linkage disequilibrium between SNPs using LDpred [Vilhjálmsson et al., 2015], setting the fraction of causal SNPs to 1. All polygenic indices are standardised to have zero mean and unit standard deviation in the analysis sample.

Our main specification uses all UK Biobank individuals born between September 1949 and August 1969 in England and Wales. In the additional analysis, we re-run our analysis on a sample of siblings. Although the UK Biobank is not a family study and does not have self-reported relatedness, we observe a large number of siblings, whom we identify using the (genetic) kinship matrix.

§ NOT ACCOUNTING FOR THE RAISING OF THE SCHOOL LEAVING AGE

<ref> shows the estimates from <ref> for the analysis of the long-term impact of the introduction of the measles vaccination in 1968 on educational outcomes, exploiting the fact that the benefits of the vaccination campaign were stronger for districts with previously high rates of measles infections. However, we here do not control for the additional dummy variable indicating whether the cohort was affected by the educational reform that raised the school leaving age from 15 to 16 years (see <ref>) and its interaction with $PreRate_d$. Hence, this analysis does not control for any differential effects this reform may have had on the educational outcomes in districts with low and high measles rates.

Education and average measles exposure during early childhood – The introduction of the measles vaccine; not accounting for differential trends in education that are driven by the increase in minimum school leaving age

Note: The dependent variables are years of education, indicators for completion of any qualification, an upper secondary qualification and a degree qualification respectively.  The explanatory variables of interest are the share of the given age periods during which the individual was exposed to the vaccination program, interacted with the measles cases per 100 people prior to the vaccination program. Individuals born in districts that participated in the 1966 trial are excluded from the sample. The analyses do not control for the differential trends in education driven by the raising of the school leaving age. Standard errors clustered at the district of birth level are shown in parentheses. Significance levels are indicated as follows: * p$<$0.1, ** p$<$0.05, *** p$<$0.01

Column (1) of Panel A shows that full exposure to the post-vaccine period at age 1–6 is associated with 0.32 more years of education in districts with one additional annual measles case per 100 population prior to the introduction of the vaccination, which is robust to the inclusion of district fixed effects (column 2), but not to school year of birth fixed effects (columns 3). Indeed, the estimate reduces in magnitude and with the increase in standard errors, it is no longer statistically significantly different from zero. However, the estimate remains large: in districts with 1 case of measles per 100 population prior to the vaccination campaign, individuals have 0.1 additional years of education compared to districts with no reports of measles. Once county-specific trends are included in column (4), the estimated impact on years of education attenuates to zero.

Panels B, C and D of <ref> illustrate how the educational impact of the measles vaccination varied throughout the educational distribution. Panel B shows strong and statistically significant effects of the vaccine introduction on the probability of completing any educational qualification. Even when controlling for school year, month and district of birth fixed effects, full exposure to the post-vaccine period is found to increase the probability of any qualification by 3.1 percentage points in districts with one additional annual measles case per 100 people prior to the introduction of the vaccination. However, controlling for county-specific trends again attenuates the estimate to zero (column 4). For higher levels of educational qualifications, the effects are smaller once school year of birth fixed effects are included in column (3).

As we show in <ref> and <ref>, the estimate in columns (1)-(3) of <ref> for the impact on the probability of obtaining any qualification is likely to be driven by the differential trends in education for those in districts with high and low measles rates prior to vaccination which is caused by the education reform that increased the minimum school leaving age. Once we control for differential trends at the county level (column 4) or explicitly control for the differential impact of the educational reform (<ref> and <ref>), the estimates reduce and are statistically insignificantly different from zero.

§ DIFFERENCE-IN-DIFFERENCE DESIGN WITH CONTINUOUS TREATMENT

Our empirical strategy follows a difference-in-differences (DiD), or two-way fixed effects (TWFE) approach with a continuous treatment. While the recent literature on DiD estimation has mainly focused on binary treatments, there are some studies analysing DiD settings with a continuous treatment [Callaway et al., 2021, de Chaisemartin et al., 2022, D'Haultfoeuille et al., 2021]. Callaway et al., 2021 propose two parameters that may be of interest in a setting with continuous treatment: a level effect (which compares treatment with dose $d$ to not receiving any treatment) and a slope effect (causal response to a marginal change in dose $d$). Under the “standard” parallel trends assumption, a dose-specific average treatment effect on the treated $ATT(d|d)$ can be identified in the presence of a never-treated control group.[Following the notation in Callaway et al., 2021, $ATT(d|d)$ represents the average effect of treatment with dose $d$ among the group actually treated with dose $d$. ] When no such untreated group exists (as is the case in our application), level effects such as the $ATT(d|d)$ cannot be identified. While slope effects can be identified even in the absence of an untreated group, additional stronger assumptions are required to identify slope effects compared to the setting with binary treatment. The authors show that under the “standard” parallel trends assumption, heterogeneous treatment effects that differ across the distribution of dose $d$ can cause a “selection bias” when comparing treatment effects across different dosage levels to infer slope effects. For example, in a two dose scenario with one group receiving a low and the other a high dose, a “selection bias” would be present if the high and low dose groups differ from each other in their potential benefits (or harm) from receiving the low or high dose. The difference between $ATT(high|high)$ and $ATT(low|low)$ would then be a combination of the causal response of a change in dose from low to high among the high group, $ATT(high|high)-ATT(low|high)$, and the selection bias term $ATT(low|high)-ATT(low|low)$.

The authors show that valid comparisons of level effects across different doses require an additional assumption, which they call “strong” parallel trends. This assumption requires that the average expected change in outcomes across all units from being untreated in $t-1$ to treatment with dose $d$ in $t$ is the same as the average expected change for those who actually experience that dose. Hence, “strong” parallel trends requires that on average across all doses there is no selection; a weaker assumption than requiring homogeneous treatment. In comparison to the “standard” parallel trends assumptions, “strong” parallel trends involves treated potential outcomes and can therefore not be tested in most circumstances. Under “strong” parallel trends, an average treatment effect $ATE(d)$ can be identified in the presence of an untreated control group. Comparisons of $ATE(d)$ across different dosage levels are not subject to selection bias, and the average causal response $ACR(d)$ can be identified; our parameter of interest.

A second issue with the use of the TWFE estimator in a DiD setting with continuous treatment relates to the weights assigned to the different underlying comparisons. In a setting without staggered treatment, these weight are positive and add up to one. However, the TWFE weights may not correspond to the distribution of the dose and instead put higher weights on observations with rare dose levels. Specifically, the TWFE weights are highest at the mean dose rather than the most common dose level. If the dose distribution is substantially different from a normal distribution (e.g. in the case of bi-modal, strongly skewed or uniform distributions), TWFE weights will differ substantially from the dose distribution. The authors hence propose using different weights in these circumstances.

In this paper, we use the interaction of the temporal exposure to the post-vaccination period (at ages 1 to 6) with the district-level measles rates prior to vaccination to measure treatment intensity. This interaction therefore represents the treatment dose in our context, and in the following we explore (1) whether we are likely to have “selection bias”, and (2) whether the TWFE weights differ substantially from the dose distribution.

Since the continuous treatment intensity measure in our analysis consists of two components, we consider the “selection bias” issues separately for each of them. First, the use of a continuous measure of temporal exposure to the post-vaccination period may lead to a selection bias if there are differences in $ATT(d|d)$ between birth cohorts that are exposed to a different degree to the post-vaccination period. For example, if those born in September 1962 (and therefore exposed to the post-vaccination for one year) would benefit more (or less) from exposure to the post-vaccination period for five years than those born in September 1966 who are actually exposed for five years, this could create a selection bias.[Note, that differences in the effects of exposure to measles at different ages does not imply a selection issue. The selection issue is caused by differences in potential treated outcomes and cannot be observed directly in the data.] We are not aware of any reason why this type of cohort-level selection is likely to be a major issue. Furthermore, our results are robust to the use of a binary indicator for exposure to the post-vaccination period (instead of the continuous measure used in our main specifications), for which this cohort-level “selection bias” does not play a role, suggesting cohort-level selection does not impact on our analysis.

The second component of the continuous treatment intensity measure is the district-level measles rate prior to vaccination, capturing the potential benefit from the roll-out of the vaccination programme. A difference in the dose therefore corresponds to differences in prior measles exposure, and any selection would be driven by the shape of the relationship between measles exposure and the outcome. We illustrate this graphically in <ref>. The sub-figures on the left (i.e., sub-figures a, c and e) show examples of relationships between measles rates at baseline $m_0$ (horizontal axis) and an outcome $Y(m_0)$ (vertical axis), with an increasing, decreasing and constant marginal impact.[For simplicity, in <ref> we focus on cases with a negative impact of measles on the outcome. Our discussion, however, is in general terms and applies to both negative and positive relationships.] The sub-figures on the right (i.e., sub-figures b, d, and f) show the corresponding $ATT$ curves and illustrate the direction of any “selection bias”, where the change in the outcome $Y(m_1) - Y(m_0)$ is plotted on the vertical axis against the dose $d=-(m_1-m_0)$ which is the reduction in measles rates following the vaccine roll-out. For illustrative purposes, we assume that measles are fully eradicated so the dose (or measles reduction) corresponds to the prior measles rates.

ATT curves under different scenarios for the relationship between measles and outcomes

Increasing marginal impact

ATT curves under increasing marginal impact

Decreasing marginal impact

ATT curves under decreasing marginal impact

Constant marginal impact

ATT curves under constant marginal impact

Note: The sub-figures on the left show examples of relationships between measles rates at baseline $m_0$ and an outcome $Y(m_0)$, with an increasing, decreasing and constant marginal impact of measles. The sub-figures on the right show the corresponding $ATT$ curves and illustrate the direction of any selection bias. The change in the outcome $Y(m_1) - Y(m_0)$ is plotted against the dose $d=-(m_1-m_0)$ which is the reduction in measles rates following the vaccine roll-out. For illustrative purposes, we assume that measles are fully eradicated so the dose (or measles reduction) corresponds to the prior measles rates.

If the marginal impact of measles is increasing with the rate of measles (sub-figures a and b), then a reduction in measles rates by $d'$ from a high baseline level $h$ will have a larger impact than a reduction of the same size from a low baseline level $l$. This means $ATT(d|h)$ will be above $ATT(d|l)$ as illustrated in sub-figure b, and a comparison of these two $ATT$s will capture both the causal response to a change in dose from $l$ to $h$ and a selection term biasing estimates away from zero.

If the marginal impact of measles is decreasing (sub-figures c and d), then a reduction in measles rates by $d'$ from a high baseline level $h$ will have a smaller impact than a reduction of the same size from a low baseline level $l$. This means $ATT(d|h)$ will be below $ATT(d|l)$ as illustrated in sub-figure d, and a comparison of these two $ATT$s will capture both the causal response to a change in dose from $l$ to $h$ and a selection term biasing estimates towards zero.

If the relationship between measles and the outcome is linear and therefore the marginal impact constant (sub-figures e and f), then a reduction in measles rates by $d'$ from high and low baseline levels will have the same impact. This means average treatment effects are the same for any group, and hence $ATE(d)=ATT(d|l)=ATT(d|h)$. Comparisons of treatment effects across doses capture the causal response to a dose change, there is no selection bias.

Hence, the presence and direction of any district-level “selection bias” depends on whether the marginal impact of measles on the outcome differs between different measles rates. We next discuss the likely implications for our analysis. First, years of education: Measles rates may affect years of education in two ways: (1) contracting measles may disrupt the (biological) cognitive development of a child (i.e., a direct impact), and (2) measles cases during the school year may cause disruption to the educational progress of schoolchildren due to complete suspension of lessons or due to slower teaching progress when many children are off sick (i.e., an indirect impact). We expect the marginal direct impact of measles on individuals' cognitive development to be independent of measles rates, since this is mainly driven by a biological mechanism which arguably is similar across children. One argument against this, however, is if severe measles cases (i.e., with larger direct impacts) are disproportionally (and non-linearly) concentrated in areas with high measles rates. If the risk of severe illness increases convexly form with the measles rate, the impact of an additional measles case would be larger in areas with higher measles rates compared to areas with lower rates, causing a “selection bias” in our estimates away from zero. Whether the marginal indirect impact differs between different measles rates is unclear. A higher measles rate implies more children are affected, perhaps leading to school or class closures. However, its impact on children's educational outcomes depends on the `threshold' at which the disruption (e.g., the closure of schools/classes) occurs. Indeed, if schools/classes are suspended when, e.g., 5% of pupils are infected, the disruption is likely to be similar in high and low measles areas, since even in low measles areas, this is an easy-to-reach threshold. In contrast, if schools/classes are only suspended when, e.g., 80% of pupils are infected, this may be more likely to have higher impacts on areas with higher measles rates. Hence, a priori, it is unclear to what extent (if at all) there would be any “selection bias”.

Next, height: The relationship between measles rates and health outcomes such as height can be expected to be largely due to the direct effects of contracting measles, rather than any indirect effects. Similar to the above, therefore, since this is mainly driven by a biological mechanism, we expect the marginal impact of measles on individuals' height to be independent of the measles rates. Hence, a priori, it seems unlikely there would be any substantial selection bias in our estimates of the long-term impact on height.

The second potential issue of using TWFE is the weighting of the underlying comparisons. Specifically, the weights used by TWFE to aggregate the average causal responses on the treated $ACRT(d|d)$ across the dose distribution may not have any theoretical foundation and can differ substantially from the distribution of doses in the sample. If the dose distribution is symmetric and similar to the normal distribution, TWFE weights will be similar to the dose distribution [Callaway et al., 2021]. However, if doses are not (approximately) normally distributed, the weights will differ substantially from the dose distribution which can result in rare doses receiving large weights and common doses receiving small weights.

Distribution of pre-vaccination rates

Note: Shown is the distribution of pre-vaccination measles rates among the sample used in our main analysis.

In our main analysis, the dose distribution derives from the pre-vaccination rates of measles ($PreRate_d$), and the temporal exposure to the post-vaccination period ($Post_{age=a}$). In the robustness analysis, we also estimate the models using a binary indicator for exposure to the post-vaccination period (see <ref>), in which case the dose only derives from the pre-vaccination measles rates.  <ref> presents the distribution of the rates. This shows that it does not differ substantially from a normal distribution. Hence, we can be confident that TWFE weights will be appropriate in these estimations.

<ref> shows the density distribution of the interaction term between the share of ages 1 to 6 during which an individual was exposed to the post-vaccination period and the district-level measles rates prior to the vaccination programme (i.e., $Post_{age=a} \times PreRate_d$). For comparison, <ref> also displays the weights that would be used to aggregate the average causal responses on the treated $ACRT(d|d)$ in a simple two-period TWFE estimation with this distribution of doses, based on Proposition 4 of Callaway et al., 2021.[Weights cannot be easily derived for the more complex estimations used in our analysis, as they involve repeated cross-sections and control for district, school year of birth and month of birth fixed effects as well as other covariates.] In comparison to the dose distribution, TWFE puts slightly higher weights on observations with a lower dose, and slightly lower weights on observations with a higher dose. Overall, however, the differences between the dose distribution and the TWFE weights are not large and hence should not be a major concern in our analysis.

Distribution of doses and TWFE weights

Note: The grey bars show the density distribution of positive doses, i.e. of the interaction term between the temporal exposure to the post-vaccination period at ages 1 to 6 and the the district-level measles rates prior to the vaccination programme. The solid line represents the weights that would be used to aggregate the average causal responses on the treated $ACRT(d|d)$ in a simple two-period TWFE estimation with this distribution of doses, as derived in Proposition 4 of Callaway et al., 2021.


11institutetext: University of Rochester, Rochester, NY, USA 22institutetext:
Codec Avatars Lab, Meta, Pittsburgh, PA, USA
22email:
{chaohuang,chenliang.xu}<EMAIL_ADDRESS>
# Modeling and Driving Human Body Soundfields through Acoustic Primitives

Chao Huang 11 Dejan Marković 22 Chenliang Xu 11 Alexander Richard 22

###### Abstract

While rendering and animation of photorealistic 3D human body models have
matured and reached an impressive quality over the past years, modeling the
spatial audio associated with such full body models has been largely ignored
so far. In this work, we present a framework that allows for high-quality
spatial audio generation, capable of rendering the full 3D soundfield
generated by a human body, including speech, footsteps, hand-body
interactions, and others. Given a basic audio-visual representation of the
body in form of 3D body pose and audio from a head-mounted microphone, we
demonstrate that we can render the full acoustic scene at any point in 3D
space efficiently and accurately. To enable near-field and realtime rendering
of sound, we borrow the idea of volumetric primitives from graphical neural
rendering and transfer them into the acoustic domain. Our acoustic primitives
result in an order of magnitude smaller soundfield representations and
overcome deficiencies in near-field rendering compared to previous approaches.
Our project page: https://wikichao.github.io/Acoustic-Primitives/.

###### Keywords:

Human Body Pose Acoustic Primitives AR/VR

##  1 Introduction

Learning, rendering, and animating 3D human body representations has been a
long standing research area with applications in gaming, movies, and more
recently also AR/VR. MetaHumans [1] and Codec Avatars [3] provide highly
realistic models and advances in neural rendering have pushed the visual
quality to new frontiers [49, 37, 31]. Animating full-body models has seen
significant progress with the availability of generative models, ranging from
pose-based animation [3] to audio- and text-driven animation [29, 40, 46].
Overall, visual representations of 3D humans these days are of excellent
quality and drivable from pose, audio, and text inputs.

However, on the acoustic side of the problem, _i.e_., rendering spatial sound
in 3D for these full-body representations, the research landscape looks dire.
It has been shown that accurate audio-visual modeling is important for an
immersive 3D experience [14] but still almost no research exists that would
allow to render spatial audio of virtual humans. Analogous to visual full-body
models, acoustic full-body models have similar requirements: first, it must be
possible to render spatial sounds produced by a virtual human at any position
in 3D space, and second, the soundfield needs to be drivable. In this work, we
focus on generating and driving full-body soundfields from 3D body pose and
head-mounted microphones.

Figure 1: Single high-order soundfield (a) vs. acoustic primitives (b).
Existing approach [44] predicts a high-order ambisonic soundfield around the
human body, preventing sound from being rendered in the near-field; our
proposed acoustic primitives, represented as small spheres attached to the
body, successfully model a complete and accurate 3D body soundfield.

This problem has recently been addressed in pioneering work by Xu _et al_.
[44], who developed a neural soundfield rendering system for full body
avatars, driven by body pose and headset microphone input. However, [44] has
several major limitations: The approach relies on a single high order
ambisonic (spherical harmonics) representation that models the sound emitted
from the surface of a sphere around the human body, with a diameter of about
2m. Sound can only be modeled outside of this sphere, such that near-field
modeling of signals closer to the body is not possible, see Fig. 1a. Moreover,
accurate sound reproduction in [44] relies on extremely high-order ambisonic
coefficients which are expensive to compute and instable to estimate. To get
around this instability, [44] does not predict the ambisonic coefficients
directly, but instead predicts the raw audio signal on 345 positions
surrounding the body, and then uses traditional signal processing to compute a
17-th order ambisonic representation from these 345 raw waveforms. This
mechanism is computationally inefficient and prevents realtime sound
rendering.

In this work, we propose a novel sound rendering method based on acoustic
primitives which solves the problems of [44]:

Near-field Rendering. We take inspiration from recent methods in visual neural
rendering that rely on volumetric primitives like cuboids [25] or Gaussians
[20]. Instead of modeling the body soundfield by an ambisonic representation
on a single sphere around the full human body as in [44], we attach multiple
acoustic primitives (small spheres each representing low-order ambisonics) to
the 3D human skeleton and model the sound radiating from each of these
acoustic primitives separately, see Fig. 1b. The full soundfield produced by
all primitives together is given by the sum of the individual rendered sound
from each primitive. This way, sound can be modeled arbitrarily close to the
body.

Efficient Soundfield Representation. Instead of a single 17-th order ambisonic
representation, we model body sounds by multiple low-order (typically second
order) ambisonic primitives. This reduces the number of parameters
characterizing the acoustic scene by an order of magnitude and allows for a
more compact and efficient soundfield representation.

Efficient Rendering. Instead of predicting 345 raw audio signals and relying
on traditional, costly high-order ambisonic encoders and decoders, we predict
the low-order ambisonic coefficients of each primitive directly. Efficient
sound rendering can then be achieved using spherical wave functions as
described in Sec. 3.2.

Drivability. Same as [44], our method can be driven from body pose and a head
mounted microphone, _i.e_. 3D soundfields can be generated for novel acoustic
input and body motion. Note that this is in stark contrast to its visual
counterparts [20, 26] which are designed to synthesize novel views of fixed
scenes, but are typically not drivable from user input.

In summary, we propose an efficient and drivable 3D sound rendering system
with

  1. 1.

audio-visual driving: given body pose and an audio signal from head-mounted
microphones, we can accurately render the soundfield produced by the body
(speech, snapping, clapping, footsteps, etc) in 3D;

  2. 2.

real-time rendering: the introduction of acoustic primitives allows for
efficient real-time rendering of 3D sound scenes;

  3. 3.

high quality: although relying only on low-order ambisonic representations, we
achieve comparable quality to [44] but avoid the high computational cost.

##  2 Related Works

Spatial Audio Modeling. Existing works on spatial audio rendering are either
based on traditional signal processing and linear filters [39, 4, 6] or on
more recent neural binaural renderers [35, 33, 12]. While these approaches can
typically produce spatial audio in an efficient way, they come at strong
restrictions, particularly, they need to know the exact location of each sound
source to render as well as the clean sound signal for each sound location.
Such information is available in fully synthetic, artist-created scenes but is
usually unknown in real environments and real acoustic scenes. Our approach,
in contrast, does not rely on such knowledge and implicitly learns to separate
an aggregated audio signal into its distinct sources (the acoustic primitives)
through inverse acoustic rendering. More recently, data-driven methods aim to
produce binaural audio from audio-visual input information. Chen _et al_. [5]
propose a system to render a pre-recorded 3D acoustic scene from novel
viewpoints, however, this method can not handle new acoustic scenes. Liang _et
al_. [23] propose to reconstruct the 3D audio-visual scene from videos, but
the scene is static. Gao _et al_. [10] analyze a 2D image of a visual scene to
generate binaural audio for a given monaural sound source. Note that this
approach can only deal with single sound sources and can’t handle complex 3D
sound scenes. In [28], the authors propose a method to generate spatial audio
from mono acoustic input and 360 degree camera inputs. They demonstrate that
their approach can correctly localize sound sources in the scene and generate
correct spatial audio at a coarse resolution.

Primitives in Volumetric Rendering. Neural rendering has been revolutionized
by volumetric rendering methods like neural volumes [24] and neural radiance
fields [26]. Follow-up works build on volumetric primitives such as cuboids
[25] or Gaussians [20] and render via ray-marching or splatting. These
technologies have unlocked real-time rendering for animatable avatars [7, 25,
32, 49, 37]. We borrow the idea of volumetric primitives and transfer them
from the visual domain into the acoustic domain to build an efficient and low-
parameter characterization of soundfields.

Audio-Visual Learning. Audio-visual learning has been widely applied to find
connections between acoustic and 2D visual signals, _e.g_. in audio-visual
localization [30, 15, 41, 19, 27, 17], for source-separation [8, 48, 9, 11,
16], or to learn associations from 360-degree videos [28, 22, 18]. Application
of audio-visual learning to 3D settings is mostly limited to 3D visual
scenarios, such as audio-visual driving of avatars [46, 29, 36, 43, 34] or
audio-driven gesture synthesis [13, 21, 2, 47]. While these works use audio-
visual input to learn information about a scene, they operate on visual
outputs and do not model acoustic scenes.

Most closely related to our work is [44], who address the same task we address
in this work. However, as outlined above, [44] has some significant drawbacks
such as the inability to render near-field audio, or the lack of real-time
rendering capabilities.

##  3 Pose-Guided Soundfield Generation using Acoustic Primitives

###  3.1 Problem Definition

Let $\mathbf{a}_{1:T_{a}}(a_{1},\dots,a_{T_{a}})$ be the input audio signal
from one or multiple head-mounted microphones, and
$\mathbf{p}_{1:T_{p}}=(p_{1},\dots,p_{T_{p}})$ be the corresponding sequence
of 3D body pose, where each $p_{t}\in\mathbb{R}^{J\times 3}$ is a vector
containing 3D body joint coordinates. We aim to predict a sound signal
$\mathbf{s}$ at an arbitrary 3D position $(r,\theta,\varphi)$. Note that we
use spherical coordinates. In other words, we learn to aim a mapping from 3D
body pose, headset audio signal, and a 3D position in space to the audio
signal at that 3D spatial position,

$\displaystyle\mathbf{a}_{1:T_{a}},\mathbf{p}_{1:T_{p}},(r,\theta,\varphi)\mapsto\mathbf{s}_{1:T_{a}}.$
(1)

The core challenge is how to get training data to learn such a model. It is
impossible to place microphones at all positions in 3D space to get dense
sampling of the space. Instead, we follow the strategy of [44] (and actually
use the same public dataset for our work) and sample soundfield signals
$\mathbf{s}_{1:T_{a}}$ only on a sphere around the human body. This poses the
challenge of rendering the soundfield at positions that are not on the surface
of the sphere on which data has been captured. Note the analogy to graphics
neural rendering: approaches like NeRF [26] also don’t have spatially dense
samples of 2D images taken from a scene, yet through inductive biases such as
the rendering equation, they succeed in synthesizing the 3D scene from any
novel viewpoint. We apply the same strategy for audio, and learn the mapping
in Eq. (1) by differentiation through the wave propagation function, which we
explain in the next section.

###  3.2 Sound Radiation using Spherical Wave Functions

The general solution of the homogeneous, time-dependent wave equation in the
spherical coordinate system is given by [42, 50]:

$\mathbf{w}(t,f,r,\theta,\varphi)=\sum_{n=0}^{\infty}\sum_{m=-n}^{n}\left[b_{nm}(t,f)\cdot
j_{n}(kr)+c_{nm}(t,f)\cdot h_{n}(kr)\right]\cdot Y_{nm}(\theta,\varphi),\\\ $
(2)

where $(r,\theta,\varphi)$ are arbitrary coordinates inside a source-free
region, $t$ and $f$ denote the time and frequency (we will omit them in the
following sections for clarity), and $k=2\pi f/v_{sound}$ is the corresponding
wavenumber; $Y_{nm}(\theta,\varphi)$ represents the spherical harmonic of
order $n$ and degree $m$, and $j_{n}(kr)$ and $h_{n}(kr)$ are, respectively,
$n$th-order spherical Bessel and Hankel functions. Coefficients $b_{nm}(t,f)$
and $c_{nm}(t,f)$ describe, respectively, incoming and outgoing waves. In
particular, considering the scenario depicted in Fig. 1a, only the radiating
field component is present, _i.e_. $b_{nm}=0$, which in literature is known as
exterior domain problem [42, 38].

Given recorded or predicted microphone signals on the surface of the sphere
surrounding the body, SoundingBodies [44] approaches the sound field modeling
task as a traditional exterior domain problem and estimates the sound field
coefficients $c_{nm}(t,f)$. While the general solution requires an infinite
number of harmonic orders, the practical estimates are limited by the
available number of microphones $M$ as $N=\sqrt{M}-1$,

$\hat{\mathbf{w}}(r,\theta,\varphi)=\sum_{n=0}^{N}\sum_{m=-n}^{n}\hat{c}_{nm}\cdot
h_{n}(kr)\cdot Y_{nm}(\theta,\varphi).\\\ $ (3)

This reliance on a generic solution of the exterior domain problem limits the
practicability of [44]. While the network uses pose information to predict
microphone signals, the successive DSP processing of these signals required
for spatial sound rendering (_e.g_. binauralization) does not leverage pose
information at all. This brings two main issues:

  1. 1.

To model sound sources located further away from the center of the
representation, see $R_{0}$ in Fig. 1a, higher harmonic orders are needed,
which in turn requires the network to predict a high number of microphones
before any spatial rendering can be performed; predicting a smaller number of
signals would limit the harmonic order and, consequently, the rendered scene
would collapse towards the center.

  2. 2.

The wave equation solution is valid only outside the boundary surface
encompassing all sources of sound, as shown in Fig. 1a. Inside this region,
the high harmonic orders produce chaotic results, limiting the minimum
distance at which the scene can be rendered and experienced.

To address the above issues we take a different approach. Instead of using
pose-conditioned network to predict microphone signals, we use the network to
predict sound field coefficients directly. Furthermore, instead of trying to
estimate a generic high-order sound field representation, we leverage the
knowledge of possible positions of sound, given by the body pose, and model
the sound radiation as a superposition of several small-order elementary sound
fields originating from different positions of the body as depicted in Fig.
1b. Similarly to Eq. (3), the sound pressure produced by a single elementary
field of order $N$ is given by

$\begin{split}\mathbf{w}(r,\theta,\varphi)&=\sum_{n=0}^{N}\sum_{m=-n}^{n}\left({c}_{nm}\cdot
h_{n}(kr_{ref})\right)\cdot\frac{h_{n}(kr)}{h_{n}(kr_{ref})}\cdot
Y_{nm}(\theta,\varphi)\\\
&=\sum_{n=0}^{N}\sum_{m=-n}^{n}\tilde{c}_{nm}\cdot\frac{h_{n}(kr)}{h_{n}(kr_{ref})}\cdot
Y_{nm}(\theta,\varphi),\end{split}$ (4)

where we use $h_{n}(kr_{ref})$ with $r_{ref}=0.5\mathrm{m}$ for numerical
stability of the learning process, and refer to harmonic coefficients
$\mathcal{S}=[\tilde{c}_{00},...,\tilde{c}_{NN}]$ as an acoustic primitive.

Given Eq. (4), we can translate the task of modeling 3D spatial sound for the
visual body to learning a set of small acoustic primitives
$\\{\mathcal{S}_{i}\\}_{i=1}^{K}$, which we choose $N$ up to the second order
for harmonic coefficients and set the number of acoustic primitives as $K$. In
practice, capturing ground truth sound field coefficient is infeasible, while
the microphone signals received on the surface of the dome are available with
prior efforts by [44]. Since the produced sound pressure
$\mathbf{w}(r,\theta,\varphi)$ indeed represents the audio signal produced at
spherical position $(r,\theta,\varphi)$, we can therefore decompose the entire
learning process into two sub-steps:

  * •

Learning Acoustic Primitives. The main objective of this step is to design a
neural network $\mathcal{F}$ that consumes audio and pose data as input, and
output the sound field representation

$\\{\mathcal{S}_{i}\\}_{i=1}^{K}=\mathcal{F}(\mathbf{a}_{1:T_{a}},\mathbf{p}_{1:T_{p}}).$
(5)

  * •

Rendering Audio with Learned Acoustic Primitives. With the learned acoustic
primitives $\\{\mathcal{S}_{i}\\}_{i=1}^{K}$, we leverage Eq. (4) as a
differentiable rendering function, denoted as $\mathcal{R}$, to generate the
audio waveform received at the target position

$\hat{\mathbf{s}}_{1:T_{a}}(r,\theta,\varphi)=\mathcal{R}(\\{\mathcal{S}_{i}\\}_{i=1}^{K},r,\theta,\varphi).$
(6)

The hyperparameter in $\mathcal{R}$ is fixed once the harmonic order $N$ and
the number of primitives $K$ are initialized, and all the operations in
$\mathcal{R}$ are differentiable, making it feasible to run end-to-end
training. With the training data tuple
$\left(\mathbf{a}_{1:T_{a}},\mathbf{p}_{1:T_{p}},\mathbf{s}_{1:T_{a}}(r,\theta,\varphi)\right)$
that includes recorded microphone signal at position $(r,\theta,\varphi)$, we
can learn a pose-guided acoustic primitive synthesis system by simply
optimizing the loss between $\hat{\mathbf{s}}_{1:T_{a}}$ and
$\mathbf{s}_{1:T_{a}}$.

Figure 2: Our pose-guided acoustic primitive learning framework takes headset
microphone signals and body pose information as inputs. It outputs the
acoustic primitive representations, weights, and offsets in one pass. The
framework consists of two main stages. In the first stage, we employ separate
encoders to process the audio and pose signals into feature spaces. An Audio-
Pose Feature Fusion Module is then utilized to combine these features. In the
second stage, the fused features are fed into an audio decoder network to
generate the acoustic primitive coefficients. Additionally, two separate MLP
heads are used to predict the weights and offsets for each acoustic primitive.

###  3.3 Multimodal Feature Encoding

Pose Encoder. Human body movements offer crucial clues for how sound is
distributed in space. To capture these rich spatial cues, we employ a pose
encoder that processes the input pose sequence $\mathbf{p}_{1:T_{p}}$. The
pose input is first encoded into a latent feature representation. To capture
temporal relationships, we apply two layers of temporal convolutions with a
kernel size of 5. Finally, we concatenate the encoded features for all joints
and use an MLP to create a compact representation, denoted as
$f_{p}\in\mathbb{R}^{C_{p}\times T^{{}^{\prime}}_{p}}$. Here $C_{p}$ is the
number of feature channels and $T^{{}^{\prime}}_{p}$ represents the temporal
dimension after convolution. More details are provided in the supplementary
material.

Audio Encoder. While sound can originate from various points on the body
(_e.g_., hands, feet), it’s captured by the headset microphone located at a
central position near the head. This difference in location creates a slight
time delay between the moment the sound is produced and when it’s actually
recorded. Previous research has shown that compensating for this time delay
can be beneficial [35, 44]. In our approach, we leverage the pose features as
guidance and use an MLP (as shown in Fig. 2) to estimate the delay for each
acoustic primitive attached to a body joint, and time-warp the audio signal
accordingly.

Via STFT, the warped signals are then transformed into complex spectrograms
$X^{c}_{a}\in\mathbb{R}^{C_{h}\times F\times T}$ where $F$ and $T$ represent
the number of frequency and time bins respectively, and $C_{h}$ is the number
of audio channels. The resulting audio features are then encoded with a
network consisting of convolutional- and LSTM layers to capture both local
context and long-range dependencies within the audio data. The encoder
architecture utilizes four layers, where each layer contains two ResNet
blocks, a temporal LSTM block, and a downsampling block with a factor of 2.
Eventually, we can obtain the latent audio features
$f_{a}=\mathrm{E}_{a}(X_{a})\in\mathbb{R}^{C_{a}\times\frac{F}{16}\times\frac{T}{16}}$.

Audio-Pose Feature Fusion Module. While headset audio reveals the content of
sounds (_e.g_., finger snapping), it lacks precise spatial information about
the source. Conversely, body pose offers strong spatial cues about joint
locations, but cannot identify the sound type (_e.g_., speech) solely from
pose data. Therefore, effectively combining audio and pose features is crucial
for learning acoustic primitives and determining their contribution to the
final sound generation. We first interpolate the pose features $f_{p}$ to
match the temporal size of the audio features $f_{a}$. We then employ a
lightweight fusion module with two ResNet blocks and one attention block to
combine the concatenated audio and pose features, resulting in a new
representation denoted as
$f_{ap}\in\mathbb{R}^{C_{a}\times\frac{F}{16}\times\frac{T}{16}}$.

Figure 3: Illustration on the rendering process with the estimated acoustic
primitives. (a) demonstrates how to render a waveform signal given the learned
harmonic coefficients $\mathcal{S}_{k}$, primitive coordinate offset
$\Delta_{k}$, and the weight $W_{k}$. Next, we show that for all the
primitives, we render audio generated by the primitive at the targeted
location and aggregate them to yield the final rendered audio at target
position $(x,y,z)$.

###  3.4 Acoustic Primitive Decoding

As described in Sec. 3.2, an acoustic primitive determines the audio heard at
arbitrary coordinates inside its sound field. It considers factors like the
harmonic coefficients, the center coordinate of the primitive, and the target
location where the sound is perceived. In this work, we focus on generating
sound based on the target location. This translates to learning two key
components: the primitive’s coordinates and the harmonic coefficients.
However, this is non-trivial as the primitive’s location changes dynamically
as the body moves. Additionally, the harmonic coefficients must capture not
only the sound content but also the spatial cues such as sound directivity. In
the next section, we will explain how our approach addresses these challenges.

Sound Field Decoder. Leveraging the fused features $f_{ap}$, our decoder
$\mathrm{D}_{a}$ simultaneously generates the sound field representations for
all the acoustic primitives. Similar to the input spectrograms $X_{a}$, the
harmonic coefficients have the same spatial dimensions but differ in the
number of channels, which encode the richness of the sound’s spatial
information. A higher number of channels allows for more precise control over
the perceived location of the sound. For simplicity, we design the decoder to
resemble the audio encoder and add skip connections between the encoder and
decoder, with the main difference being the number of output channels. The
decoder outputs ${(N+1)}^{2}\times K$ channels. Here, $N$ represents the order
of the harmonic coefficients, which controls the level of detail captured in
the spatial representation. Finally, we separate the decoder’s output into $K$
distinct harmonic coefficients $\\{\mathcal{S}_{i}\\}_{i=1}^{K}$, one for each
acoustic primitive.

Primitive Offsets. We initialize acoustic primitives to be at body joint
locations, _e.g_. at the wrists, face and ankles. While the initial 3D
coordinates of these body joints provide a reasonable starting point, the
actual locations of the sound sources might differ slightly from the body
joint positions. For example, the chosen keypoints represent wrists, but the
sound of finger snapping originates from the fingers themselves. This
discrepancy between the body joint and sound production locations can affect
the learning process and the accuracy of the rendered spatial audio. To
address this limitation, we learn offsets for the initial coordinates to
better represent the actual positions of acoustic primitives. In practice, we
employ a three-layer MLP network that operates on the fused features $f_{ap}$.
First, we apply mean pooling along the frequency axis of $f_{ap}$ but keep its
time dimension, obtaining $\mathop{\bar{f_{ap}}}$. Then, we generate the
offsets by

$\Delta(x,y,z)=\sigma\cdot\tanh\left(\mathrm{MLP}_{\text{offset}}\left(\mathop{\bar{f_{ap}}}\right)\right).$
(7)

To constrain the predicted offsets within a reasonable range, we use a $\tanh$
activation function and apply a scaling factor of $\sigma=0.2$ to restrict the
offsets to a maximum range of 20 centimeters around the initial locations.

Primitive Weights. At different points in time, primitives have different
importance. For instance, when a finger is snapped, the hand primitive emits
high energy sound while other primitives emit at most low energy. The
relationship is a function of the input audio and body pose. We therefore
explicitly model the weight of each primitive as a function of the combined
audio and pose encodings $\mathop{\bar{f_{ap}}}$,

$W=\mathrm{softmax}\left(\mathrm{MLP}_{weight}\left(\mathop{\bar{f_{ap}}}\right)\right).$
(8)

$W$ is the predicted weight for each primitive at each time instance and
indicates the relative influence in the final rendered sound.

###  3.5 Differentiable Acoustic Primitive Renderer

Given the initial primitive locations (_i.e_. the joint locations to which the
primitives are attached), the learned offsets, harmonic coefficients, and
weights, we can now render the sound field for each primitive (as shown in
Fig. 3). We first compute the primitive’s predicted location by adding the
learned offsets to the corresponding body joint location. Now, given a
listener position in 3D space at which we want to render the sound, we
transform each primitive’s predicted location into spherical coordinates
$(r_{k},\theta_{k},\varphi_{k})$ representing the relative position of the
listener with respect to each of the $K$ primitives. We now use the
differentiable audio renderer from Eq. (6) to render the audio signal
$\mathbf{\hat{s}}_{1:T_{a}}^{k}$ produced by the $k$-th primitive at the
listener’s position,

$\displaystyle\mathbf{\hat{s}}_{1:T_{a}}^{k}=\mathcal{R}(\mathcal{S}_{k}\cdot
W_{k},r_{k},\theta_{k},\varphi_{k}),\ $ (9)

and obtain the full sound field by summation over all acoustic primitives,

$\displaystyle\mathbf{\hat{s}}_{1:T}=\sum_{k}^{K}\mathbf{\hat{s}}_{1:T}^{k}.$
(10)

###  3.6 Loss Function

Since our renderer $\mathcal{R}$ is differentiable, it allows us to
efficiently train the model using loss functions on the final predicted
waveforms. In this work, we employ a multiscale STFT loss [45] between the
predicted audio $\mathbf{\hat{s}}_{1:T_{a}}$ and the ground truth audio
$\mathbf{s}_{1:T_{a}}$ on their amplitude spectrograms, denoted as
$\mathcal{L}_{amp}(\mathbf{\hat{s}}_{1:T_{a}},\mathbf{s}_{1:T_{a}})$ and on
the real and imaginary parts of spectrograms, denoted as
$\mathcal{L}_{ri}(\mathbf{\hat{s}}_{1:T_{a}},\mathbf{s}_{1:T_{a}})$. The
window sizes are set as 2048, 1024, 512, 256. As proposed in [44], a
shift-$\ell 1$ loss helps reduce the spatial alignment error. We therefore add
this loss term as $\mathcal{L}_{s\ell
1}(\mathbf{\hat{s}}_{1:T_{a}},\mathbf{s}_{1:T_{a}})$. Additionally,
determining a primitive’s contribution to the final sound (corresponding to
the primitive weights $W$) can be challenging without additional guidance. To
overcome this, we leverage clip-level labels (denoted by
$\mathrm{y}\in\mathbb{R}^{K}$) that specify which body joint contributes to
the received audio. We apply average pooling along the frequency dimension and
find the maximum value of $W$ for each primitive across all time steps,
resulting in $\tilde{W}\in\mathbb{R}^{K}$. This essentially summarizes whether
an acoustic primitive has contributed to the sound in the audio clip. Finally,
a simple cross-entropy loss function $\mathcal{L}_{cts}(\tilde{W},\mathrm{y})$
is employed to aid in the learning process. Our final loss becomes

$\mathcal{L}_{total}=\lambda_{amp}\mathcal{L}_{amp}+\lambda_{ri}\mathcal{L}_{ri}+\lambda_{s\ell
1}\mathcal{L}_{s\ell 1}+\lambda_{cts}\mathcal{L}_{cts}.$ (11)

Please refer to the supplementary materials for ablation on the loss terms.

##  4 Experiments

###  4.1 Experimental Setting

Dataset. To evaluate our approach, we leverage the publicly available dataset
introduced in [44]
111https://github.com/facebookresearch/SoundingBodies222Note: data used in the
paper and data released publicly differ by 1.5 subjects (8 subjects used in
[44] vs 6.5 publicly released). We updated the performance metrics of the
baseline [44] to account for this difference.. The dataset captures
synchronized audio and visual data in an anechoic chamber, offering multimodal
data specifically designed for speech and body sound field modeling research.
It utilizes 5 Kinect sensors for body tracking and a large microphone array
(345 microphones) arranged in a spherical fashion around the recording area.
The data encompasses various participants performing a diverse range of body
sounds and speech in different settings (_e.g_., standing or sitting). The
recordings are segmented into non-overlapping one-second clips. We adopt the
same train/validation/test splits established by [44], resulting in
10,076/1,469/1,431 clips, respectively.

Implementation Details. In our experimental setup, we employ a sampling rate
of 48 kHz for audio signals and a frame rate of 30 fps for body pose data. The
audio waveforms are converted into complex spectrograms using a Hann window of
size 512 and a hop length of 128 and FFT length of 1022. Within the encoders,
both the pose features $f_{p}$ and audio features $f_{a}$ are configured to
have the same channel size $C_{a}=C_{p}=256$. We set the order of harmonic
coefficients to $N=2$. During training, the batch size is set as 1 per GPU and
we randomly select 20 microphones from the available pool of 345 target
microphones for each forward pass. The AdamW optimizer with a learning rate of
0.0002 is used, and the network is trained for 100 epochs. To balance
different loss terms, we set the weights $\lambda_{amp}=7$, $\lambda_{ri}=3$,
$\lambda_{s\ell 1}=0.5$, and $\lambda_{cts}=1$. The experiments are conducted
on 4 NVIDIA Tesla A100 GPUs, with model training for 100 epochs taking
approximately 55 hours to complete.

Evaluation Metrics. We evaluate the performance of our model using three main
metrics: the signal-to-distortion ratio (SDR), the $\ell_{2}$ error on the
amplitude spectrogram, and the angular error of the phase spectrogram. The SDR
measures the overall quality of the reconstructed sound, with higher values
indicating better quality. The amplitude error shows how well the
reconstructed sound matches the original in terms of the distribution of sound
energy, while the angular error evaluates the timing accuracy of the
reconstructed sound waves relative to the original. We report amplitude errors
multiplied by a factor of 1000 to remove leading zeros.

Table 1: Our method achieves comparable results to the baseline SoundingBodies [44] in SDR, amplitude error, and phase error, while significantly outperforming the baseline in inference speed (15x faster).  |  |  | non-speech |  | speech
---|---|---|---|---|---
Methods |  Speed |  |  SDR $\uparrow$ |  amplitude $\downarrow$ |  phase $\downarrow$ |  |  SDR $\uparrow$ |  amplitude $\downarrow$ |  phase $\downarrow$
[44] | 3.56s |  | $3.052$ | $0.832$ | $0.314$ |  | $9.635$ | $0.701$ | $0.464$
Ours | 0.24s |  | $3.597$ | $0.883$ | $0.323$ |  | $8.448$ | $0.943$ | $0.417$

###  4.2 Comparison with Baseline

We compare our method using $12$ acoustic primitives of $2$nd order with the
SoundingBodies [44] baseline. Results are shown in Tab. 1. We can observe that
the sound field modeling performance of the proposed method is comparable to
[44] while having a much faster inference speed. In particular, proposed
method even performs better than the baseline on SDR metric for non-speech
sounds and phase metric for speech. Regarding the inference speed, we show
average time needed to compute 1 second of audio at 48 kHz. Note that for [44]
we only report the time needed for the network to predict the microphone
signals. In a practical scenario [44] needs also DSP processing of these
microphone signals to obtain the high-order sound field representation, which
would further increase the overall processing overhead.

Figure 4: Sound field visualizations for 4 different kinds of sound. Main sound field is in the center and individual primitive contributions are shown around. We can observe that the method assigns acoustic energy to correct acoustic primitives, e.g. speech comes mostly from the head with only a very small contribution from the shoulder primitives. We can also observe the speech directivity pattern matching the head orientation. For each visualization, the left/right 4 primitives are labeled as follows: foot, hip, hand, and shoulder (from bottom to top), and the middle one is the head. Table 2: Ablation study: the number and harmonic order of acoustic primitives. # $K$ | # $N$ |  | non-speech |  | speech
---|---|---|---|---|---
|  SDR $\uparrow$ |  amp. $\downarrow$ |  phase $\downarrow$ |  |  SDR $\uparrow$ |  amp. $\downarrow$ |  phase $\downarrow$
5 |  |  | $0$th |  | 2.009 | 1.013 | 0.334 |  | 4.775 | 1.225 | 0.559
$1$st |  | 3.534 | 0.896 | 0.322 |  | 7.261 | 0983 | 0.480
$2$nd |  | 3.552 | 0.911 | 0.323 |  | 7.981 | 0.977 | 0.442
9 |  |  | $0$th |  | 3.059 | 0.907 | 0.327 |  | 6.619 | 1.068 | 0.496
$1$st |  | 3.600 | 0.895 | 0.323 |  | 7.479 | 0.983 | 0.467
$2$nd |  | 3.569 | 0.915 | 0.323 |  | 8.200 | 0.952 | 0.434
12 |  |  | $0$th |  | 3.020 | 0.895 | 0.327 |  | 6.893 | 1.031 | 0.472
$1$st |  | 3.616 | 0.879 | 0.325 |  | 7.616 | 0.969 | 0.466
$2$nd |  | $3.597$ | $0.883$ | $0.323$ |  | $8.448$ | $0.943$ | $0.417$
12 | no primitive offset |  $2$nd |  | $3.528$ | $0.919$ | $0.321$ |  | $7.730$ | $0.998$ | $0.456$

###  4.3 Ablation Study

In this section we evaluate the impact of the number of acoustic primitives
and their harmonic order. Zero-order harmonics are able to model only
omidirectional fields, while higher orders allow for increasingly complex
radiation patterns. Note that we limit the maximum order to 2 given that
PyTorch implementations of spherical wave functions are available only up to
the second order. Intuitively, a higher number of acoustic primitives allows
for modeling of more complex overall sound fields. We test three
configurations of acoustic primitives: 5 primitives: head, L/R hand, L/R foot;
9 primitives: head, L/R hand, L/R foot, L/R shoulder, L/R hip; and 12
primitives where head and hands have two primitives associated with the same
key-point (given location offsets these primitives are not bound to be in the
same location allowing approximation of a higher order radiation pattern).
Results are shown in Tab. 2. In general, both higher number of primitives and
higher primitive order improve the performance as expected. This is especially
true for speech. For body sounds on the other hand, increasing from $1$st to
$2$nd order does not seem to be beneficial. We also evaluate the model without
the primitive offset adjustment. From Tab. 2 we can observe that removing the
offset has similar impact as decreasing the number of primitives from 12 to 9,
which intuitively makes sense given that repeated primitives collapse to the
same key-point location.

For more experiments, such as ablation on the loss terms and visualizations
with different harmonic orders, please refer to the supplementary materials.

###  4.4 Qualitative Results

Some sound field visualization examples are shown in Fig. 4. We can observe
that the network is able to correctly associate different kinds of sounds to
the appropriate acoustic primitives. We can also observe the speech radiation
pattern matching the head orientation. Furthermore, Fig. 5 shows predicted and
ground truth waveforms at different microphone locations. We can observe good
temporal alignment and amplitude match for most cases. One exception is the
body tapping sound in which the amplitude does not match across different
microphones. This may be due to the primitive struggling to match a highly
variable radiation pattern.

Figure 5: Predicted and ground truth microphone signals at 5 different
locations around the dome. We can observe good temporal alignment and good
amplitude match except for the low-energy body tapping sound. We recommend
zooming in for better visibility.

##  5 Conclusion

We propose a neural rendering system for sound that allows to generate and
render 3D sound fields from sparse user input like body pose and headset
audio. We demonstrate the we maintain similar quality to state-of-the-art
sound rendering, while improving significantly on speed and soundfield
completeness: our approach is an order of magnitude faster than the approach
from [44] and is capable of rendering sound in the near-field, _i.e_. close to
the transmitter’s body, where the previous approach from [44] failed.

Moreover, we want to highlight the design similarities to successful neural
renderers from computer graphics: By leveraging an acoustic rendering equation
and acoustic primitives, similar to leveraging volumetric primitives in
graphical neural rendering, we design a 3D spatial audio system with a
conceptual duality to its visual counterpart. We hope this work will impact
sound rendering in 3D settings like computer games and AR/VR.

Limitations. Albeit the promising results in terms of quality and efficiency,
our approach is still far from broad availability: model training relies on
data collected with a multi-microphone capture stage that is not broadly
available. Future directions need to aim at enabling learning such acoustic
scenes with simpler setups, ideally with commodity hardware like smartphones.
Generalization beyond human bodies is another natural extension that emerges
from the availability of broader data sources for spatial sound.

Potential Society Impact. Ethical and societal risks in this work are low
since no data is manipulated in a generative fashion - pure spatialization has
little potential for harmful actors.

## References

  * [1] Metahuman creator. https://metahuman.unrealengine.com (2021)
  * [2] Ahuja, C., Lee, D.W., Nakano, Y.I., Morency, L.P.: Style transfer for co-speech gesture animation: A multi-speaker conditional-mixture approach. In: Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XVIII 16. pp. 248–265. Springer (2020)
  * [3] Bagautdinov, T., Wu, C., Simon, T., Prada, F., Shiratori, T., Wei, S.E., Xu, W., Sheikh, Y., Saragih, J.: Driving-signal aware full-body avatars. ACM Transactions on Graphics (TOG) 40(4), 1–17 (2021)
  * [4] Chen, C., Jain, U., Schissler, C., Gari, S.V.A., Al-Halah, Z., Ithapu, V.K., Robinson, P., Grauman, K.: Soundspaces: Audio-visual navigation in 3d environments. In: Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part VI 16. pp. 17–36. Springer (2020)
  * [5] Chen, C., Richard, A., Shapovalov, R., Ithapu, V.K., Neverova, N., Grauman, K., Vedaldi, A.: Novel-view acoustic synthesis. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2023)
  * [6] Chen, C., Schissler, C., Garg, S., Kobernik, P., Clegg, A., Calamia, P., Batra, D., Robinson, P., Grauman, K.: Soundspaces 2.0: A simulation platform for visual-acoustic learning. Advances in Neural Information Processing Systems 35, 8896–8911 (2022)
  * [7] Chen, Z., Hong, F., Mei, H., Wang, G., Yang, L., Liu, Z.: Primdiffusion: Volumetric primitives diffusion for 3d human generation. In: Thirty-seventh Conference on Neural Information Processing Systems (2023)
  * [8] Ephrat, A., Mosseri, I., Lang, O., Dekel, T., Wilson, K., Hassidim, A., Freeman, W.T., Rubinstein, M.: Looking to listen at the cocktail party: a speaker-independent audio-visual model for speech separation. ACM Transactions on Graphics (TOG) 37(4), 1–11 (2018)
  * [9] Gao, R., Feris, R., Grauman, K.: Learning to separate object sounds by watching unlabeled video. In: Proceedings of the European Conference on Computer Vision (ECCV). pp. 35–53 (2018)
  * [10] Gao, R., Grauman, K.: 2.5 d visual sound. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 324–333 (2019)
  * [11] Gao, R., Grauman, K.: Visualvoice: Audio-visual speech separation with cross-modal consistency. In: 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 15490–15500. IEEE (2021)
  * [12] Gebru, I.D., Marković, D., Richard, A., Krenn, S., Butler, G.A., De la Torre, F., Sheikh, Y.: Implicit hrtf modeling using temporal convolutional networks. In: ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). pp. 3385–3389. IEEE (2021)
  * [13] Ginosar, S., Bar, A., Kohavi, G., Chan, C., Owens, A., Malik, J.: Learning individual styles of conversational gesture. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 3497–3506 (2019)
  * [14] Hendrix, C., Barfield, W.: The sense of presence within auditory virtual environments. Presence: Teleoperators and Virtual Environments 5(3), 290–301 (1996)
  * [15] Hu, D., Qian, R., Jiang, M., Tan, X., Wen, S., Ding, E., Lin, W., Dou, D.: Discriminative sounding objects localization via self-supervised audiovisual matching. Advances in Neural Information Processing Systems 33, 10077–10087 (2020)
  * [16] Huang, C., Liang, S., Tian, Y., Kumar, A., Xu, C.: Davis: High-quality audio-visual separation with generative diffusion models. arXiv preprint arXiv:2308.00122 (2023)
  * [17] Huang, C., Tian, Y., Kumar, A., Xu, C.: Egocentric audio-visual object localization. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 22910–22921 (June 2023)
  * [18] Huang, H., Solah, M., Li, D., Yu, L.F.: Audible panorama: Automatic spatial audio generation for panorama imagery. In: Proceedings of the 2019 CHI conference on human factors in computing systems. pp. 1–11 (2019)
  * [19] Jiang, H., Murdock, C., Ithapu, V.K.: Egocentric deep multi-channel audio-visual active speaker localization. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 10544–10552 (2022)
  * [20] Kerbl, B., Kopanas, G., Leimkühler, T., Drettakis, G.: 3d gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics 42(4) (July 2023)
  * [21] Lee, H.Y., Yang, X., Liu, M.Y., Wang, T.C., Lu, Y.D., Yang, M.H., Kautz, J.: Dancing to music. Advances in neural information processing systems 32 (2019)
  * [22] Li, D., Langlois, T.R., Zheng, C.: Scene-aware audio for 360 videos. ACM Transactions on Graphics (TOG) 37(4), 1–12 (2018)
  * [23] Liang, S., Huang, C., Tian, Y., Kumar, A., Xu, C.: Av-nerf: Learning neural fields for real-world audio-visual scene synthesis. In: Conference on Neural Information Processing Systems (NeurIPS) (2023)
  * [24] Lombardi, S., Simon, T., Saragih, J., Schwartz, G., Lehrmann, A., Sheikh, Y.: Neural volumes: Learning dynamic renderable volumes from images. ACM Trans. Graph. 38(4), 65:1–65:14 (Jul 2019)
  * [25] Lombardi, S., Simon, T., Schwartz, G., Zollhoefer, M., Sheikh, Y., Saragih, J.: Mixture of volumetric primitives for efficient neural rendering. ACM Transactions on Graphics (ToG) 40(4), 1–13 (2021)
  * [26] Mildenhall, B., Srinivasan, P.P., Tancik, M., Barron, J.T., Ramamoorthi, R., Ng, R.: Nerf: Representing scenes as neural radiance fields for view synthesis. In: Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part I 16. pp. 405–421. Springer (2020)
  * [27] Mo, S., Morgado, P.: Localizing visual sounds the easy way. In: Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXXVII. pp. 218–234. Springer (2022)
  * [28] Morgado, P., Nvasconcelos, N., Langlois, T., Wang, O.: Self-supervised generation of spatial audio for 360 video. Advances in neural information processing systems 31 (2018)
  * [29] Ng, E., Romero, J., Bagautdinov, T., Bai, S., Darrell, T., Kanazawa, A., Richard, A.: From audio to photoreal embodiment: Synthesizing humans in conversations. In: IEEE Conference on Computer Vision and Pattern Recognition (2024)
  * [30] Qian, R., Hu, D., Dinkel, H., Wu, M., Xu, N., Lin, W.: Multiple sound sources localization from coarse to fine. In: Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XX 16. pp. 292–308. Springer (2020)
  * [31] Qian, S., Kirschstein, T., Schoneveld, L., Davoli, D., Giebenhain, S., Nießner, M.: Gaussianavatars: Photorealistic head avatars with rigged 3d gaussians (2023)
  * [32] Remelli, E., Bagautdinov, T., Saito, S., Wu, C., Simon, T., Wei, S.E., Guo, K., Cao, Z., Prada, F., Saragih, J., et al.: Drivable volumetric avatars using texel-aligned features. In: ACM SIGGRAPH 2022 Conference Proceedings. pp. 1–9 (2022)
  * [33] Richard, A., Dodds, P., Ithapu, V.K.: Deep impulse responses: Estimating and parameterizing filters with deep networks. In: IEEE International Conference on Acoustics, Speech and Signal Processing (2022)
  * [34] Richard, A., Lea, C., Ma, S., Gall, J., de la Torre, F., Sheikh, Y.: Audio- and gaze-driven facial animation of codec avatars. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV). pp. 41–50 (2021)
  * [35] Richard, A., Markovic, D., Gebru, I.D., Krenn, S., Butler, G.A., Torre, F., Sheikh, Y.: Neural synthesis of binaural speech from mono audio. In: International Conference on Learning Representations (2021)
  * [36] Richard, A., Zollhoefer, M., Wen, Y., de la Torre, F., Sheikh, Y.: Meshtalk: 3d face animation from speech using cross-modality disentanglement. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) (2021)
  * [37] Saito, S., Schwartz, G., Simon, T., Li, J., Nam, G.: Relightable gaussian codec avatars (2023)
  * [38] Samarasinghe, P.N., Abhayapala, T.D.: 3D spatial soundfield recording over large regions. In: Proc. Int. Workshop on Acoustic Signal Enhancement (IWAENC) (2012)
  * [39] Savioja, L., Huopaniemi, J., Lokki, T., Väänänen, R.: Creating interactive virtual acoustic environments. Journal of the Audio Engineering Society 47(9), 675–705 (1999)
  * [40] Tevet, G., Raab, S., Gordon, B., Shafir, Y., Cohen-or, D., Bermano, A.H.: Human motion diffusion model. In: The Eleventh International Conference on Learning Representations (2023)
  * [41] Tian, Y., Hu, D., Xu, C.: Cyclic co-learning of sounding object visual grounding and sound separation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 2745–2754 (2021)
  * [42] Williams, E.G.: Fourier Acoustics. Academic Press (1999)
  * [43] Xing, J., Xia, M., Zhang, Y., Cun, X., Wang, J., Wong, T.T.: Codetalker: Speech-driven 3d facial animation with discrete motion prior. arXiv preprint arXiv:2301.02379 (2023)
  * [44] Xudong, X., Markovic, D., Sandakly, J., Keebler, T., Krenn, S., Richard, A.: Sounding bodies: Modeling 3d spatial sound of humans using body pose and audio. In: Thirty-seventh Conference on Neural Information Processing Systems (2023)
  * [45] Yamamoto, R., Song, E., Hwang, M.J., Kim, J.M.: Parallel waveform synthesis based on generative adversarial networks with voicing-aware conditional discriminators. In: ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). pp. 6039–6043. IEEE (2021)
  * [46] Yi, H., Liang, H., Liu, Y., Cao, Q., Wen, Y., Bolkart, T., Tao, D., Black, M.J.: Generating holistic 3d human motion from speech. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR). pp. 469–480 (June 2023)
  * [47] Yoon, Y., Cha, B., Lee, J.H., Jang, M., Lee, J., Kim, J., Lee, G.: Speech gesture generation from the trimodal context of text, audio, and speaker identity. ACM Transactions on Graphics (TOG) 39(6), 1–16 (2020)
  * [48] Zhao, H., Gan, C., Rouditchenko, A., Vondrick, C., McDermott, J., Torralba, A.: The sound of pixels. In: Proceedings of the European conference on computer vision (ECCV). pp. 570–586 (2018)
  * [49] Zielonka, W., Bagautdinov, T., Saito, S., Zollhöfer, M., Thies, J., Romero, J.: Drivable 3d gaussian avatars (2023)
  * [50] Zotter, F., Frank, M.: Ambisonics: A Practical 3D Audio Theory for Recording, Studio Production, Sound Reinforcement, and Virtual Reality. Springer Topics in Signal Processing, Springer International Publishing (2019)

Supplementary Materials for “Modeling and Driving Human Body Soundfields
through Acoustic Primitives”

##  F Rendering Audio with Learned Soundfield

As illustrated in Sec 3.2, the sound pressure, _i.e_. the audio signal,
produced by a learned soundfield of order $N$ is given by

$\begin{split}\mathbf{w}(r,\theta,\varphi)&=\sum_{n=0}^{N}\sum_{m=-n}^{n}\left({c}_{nm}\cdot
h_{n}(kr_{ref})\right)\cdot\frac{h_{n}(kr)}{h_{n}(kr_{ref})}\cdot
Y_{nm}(\theta,\varphi)\\\
&=\sum_{n=0}^{N}\sum_{m=-n}^{n}\tilde{c}_{nm}\cdot\frac{h_{n}(kr)}{h_{n}(kr_{ref})}\cdot
Y_{nm}(\theta,\varphi),\end{split}$ (12)

here, $k=2\pi f/v_{sound}$ is the corresponding wavenumber;
$Y_{nm}(\theta,\varphi)$ represents the spherical harmonic of order $n$ and
degree $m$, which is

$Y_{nm}(\theta,\varphi)\equiv\sqrt{\frac{2n+1}{4\pi}\frac{(n-m)!}{(n+m)!}}P_{nm}(\cos\theta)e^{im\varphi},$
(13)

$P_{nm}(z)$ is an associated Legendre polynomial, and $h_{n}(kr)$ is $n$th-
order spherical Hankel functions. All the functions are implemented with
PyTorch and, therefore are fully differentiable. In this paper, we choose
spherical harmonics up to second order and showcase them in Fig. 6.

Figure 6: Spherical harmonics up to second order.

Similarly to $Y_{nm}(\theta,\varphi)$, the learned soundfield, _i.e_. acoustic
primitive, can also be decomposed into a series of spherical harmonics
functions, each representing a different spatial component of the soundfield.
We demonstrate the decomposition process in Fig. 7. Our learned soundfield
representation is enforced to express the same spatial information as
spherical harmonics because each predicted acoustic primitive has $(N+1)^{2}$
channels, which is equivalent to the number of spherical harmonics.

Figure 7: Decomposition of learned soundfield representation and its
connection to second order spherical harmonics. Figure 8: Illsutration on pose
feature encoding, we consider the absolute coordinates and the relative
coordinates/distances (to the headset) of body joints.

##  G Pose Feature Encoding

To capture these rich spatial cues, we employ a pose encoder that processes
the input pose sequence. This sequence, denoted as $\mathbf{p}_{1:T_{p}}$,
contains the 3D coordinates of body joints for each frame
$\mathbf{p}_{t}\in\mathbb{R}^{J\times 3}$. However, since these coordinates
are captured from a third-person perspective, they might not fully capture the
spatial relationship relevant to the audio, where the sound originates from
the body but is received at the headset. To address this, we enhance the pose
input by selecting the head joint $\mathbf{p}^{h}_{t}$ as an anchor and
calculating relative coordinates and Euclidean distances. This extended pose
input
$[\mathbf{p}_{t},\mathbf{p}_{t}-\mathbf{p}^{h}_{t},dist(\mathbf{p}_{t}-\mathbf{p}^{h}_{t})]\in\mathbb{R}^{J\times
7}$, consisting of original coordinates, relative coordinates to the head, and
distance from the head, provides the pose encoder with a more comprehensive
understanding of the body’s spatial relationship with the sound. Details are
shown in Fig. 8.

##  H More Ablations

Visualization of primitive offsets. In Fig. 9, we observe a time delay between
the predicted and GT audios for the model without offsets, likely caused by
inaccurate primitive coordinates. In contrast, our model with learned offsets
mitigates this issue, resulting in a closer match to the ground truth audio.
Also, we visualize the sound fields generated by our framework for different
primitives after applying the learned offsets. We observe that the learned
offsets generally match location where we would expect the source of sound to
be given the particular sound event such as snap, clap, or footstep.

Figure 9: Ablation on learned acoustic primitive offsets.

Visualization of different harmonic order. Fig. 10 illustrates the impact of
sound field order on the accuracy of predicted audio. As shown, the model’s
prediction using a $2$nd-order sound field exhibits a closer match to the GT
audio in terms of amplitude. This is because higher-order harmonics offer
finer spatial rendering capabilities, allowing the model to capture more
precise directional details of the sound. In contrast, the predicted $0$th-
order sound field is omnidirectional, meaning it radiates sound equally in all
directions. This limitation hinders its ability to encode specific spatial
information, resulting in less accurate audio amplitude prediction.

Figure 10: Ablation on different harmonic orders.

Ablation on the choices of loss function. In Tab. 3, we conduct an ablation
study to investigate the effectiveness of each loss term in the total loss
function (Eq. (11) in the main paper). We remove each loss term from the total
loss $\mathcal{L}_{total}$ one at a time. The results show that including
$\mathcal{L}_{cts}$ improves the overall performance on both speech and non-
speech data, and combining all the loss terms yields the best or second-best
performance across different metrics for both speech and non-speech data and
generally the best performance on average over the metrics.

Table 3: Ablation study on loss function. Each loss term is removed from the total loss function one at a time. The best and second-best results are highlighted in green and blue, respectively. |  | non-speech |  | speech
---|---|---|---|---
Loss |  |  SDR $\uparrow$ |  amplitude $\downarrow$ |  phase $\downarrow$ |  |  SDR $\uparrow$ |  amplitude $\downarrow$ |  phase $\downarrow$
$\mathcal{L}_{total}$ |  | 3.597 | 0.883 | 0.323 |  | 8.448 | 0.943 | 0.417
w/o $\mathcal{L}_{amp}$ |  | 2.099 | 1.179 | 0.303 |  | 8.370 | 1.038 | 0.381
w/o $\mathcal{L}_{ri}$ |  | 3.579 | 0.894 | 0.325 |  | 8.492 | 0.933 | 0.418
w/o $\mathcal{L}_{s\ell 1}$ |  | 3.338 | 0.821 | 0.330 |  | 7.722 | 0.915 | 0.491
w/o $\mathcal{L}_{cts}$ |  | 3.523 | 0.903 | 0.324 |  | 8.239 | 0.938 | 0.422

##  I Demo Video

We have prepared a supplementary video to visually demonstrate the
capabilities of our method in spatial audio rendering. The video showcases a
full-body avatar producing correctly spatialized binaural audio corresponding
to various actions and interactions using our trained model. In particular,
the input of the audio system is a single channel mono audio that contains the
mixture of all sounds being made. Our model can render them with the correct
spatial locations using wearer’s body pose. This means that the wearer can
clap left, clap right, applaud, snap around, and the sounds will be
appropriately positioned. Additionally, the system works with objects that the
wearer may be using, such as an egg shaker.

## Acknowledgement

The authors would like to thank Frank Yu for engineering work on the VR demo.