# SonifyAR: Context-Aware Sound Generation in Augmented Reality

Xia Su  University of WashingtonSeattleWashingtonUSA<EMAIL_ADDRESS>,
Jon E. Froehlich  University of WashingtonSeattleWashingtonUSA
<EMAIL_ADDRESS>,  Eunyee Koh  Adobe ResearchSan JoseCaliforniaUSA
<EMAIL_ADDRESS>and  Chang Xiao  Adobe ResearchSan JoseCaliforniaUSA
<EMAIL_ADDRESS>
###### Abstract.

Sound plays a crucial role in enhancing user experience and immersiveness in
Augmented Reality (AR). However, current platforms lack support for AR sound
authoring due to limited interaction types, challenges in collecting and
specifying context information, and difficulty in acquiring matching sound
assets. We present SonifyAR, an LLM-based AR sound authoring system that
generates context-aware sound effects for AR experiences. SonifyAR expands the
current design space of AR sound and implements a Programming by Demonstration
(PbD) pipeline to automatically collect contextual information of AR events,
including virtual content semantics and real world context. This context
information is then processed by a large language model to acquire sound
effects with Recommendation, Retrieval, Generation, and Transfer methods. To
evaluate the usability and performance of our system, we conducted a user
study with eight participants and created five example applications, including
an AR-based science experiment, an improving case for AR headset safety, and
an assisting example for low vision AR users.

Mixed Reality, Sound, Augmented Reality, Authoring Tool

Work done during internship at Adobe Research

††ccs: Human-centered computing Interaction design††ccs: Human-centered
computing Interactive systems and tools

Figure 1. SonifyAR is an AR sound authoring pipeline that generates context-
matching sounds for AR events in situ. With SonifyAR, the user can first
interact with the AR Scene, while SonifyAR identifies potential sound-
producing events and collects context information as text. This context text,
including the real-world environment information like surface material, is
then processed by LLM, which guides the sound acquisition process using
multiple models and methods. The acquired sounds are then used to sonify AR
events.

SonifyAR overview

##  1\. Introduction

Sound is a critical but often overlooked element in Augmented Reality (AR).
AR-based sound can improve immersion and overall user experience
(roginska2017immersive, ) in applications such as depth perception and task
completion (zhou_role_2007, ), search and navigation
(ruminski2015experimental, ), and even assistance for people with low vision
(ribeiro_auditory_2012, ). Despite its importance, current AR authoring
platforms like Reality Composer (realitycomposer, ), Adobe Aero (adobeaero, ),
and Unity Mars (UnityMARS2023, ) provide only rudimentary sound support.
Specifically, we identified three critical gaps with existing AR authoring
tools:

  1. (1)

Limited real-world context. Existing systems typically support action triggers
linked to virtual objects in AR but lack support for real-world contextual
information (e.g., a virtual toy robot traversing diverse indoor surfaces like
wood, carpet, or glass.).

  2. (2)

Limited interaction specification. Existing systems provide only pre-defined
interaction triggers like “Tap” and “Proximity Enter”. This limits a creator’s
ability to specify interactions outside the provided options, especially those
that involve environmental context (e.g., the user “slides” virtual chalk on a
real-world blackboard).

  3. (3)

Limited sound sources. Existing systems are limited by the sound assets
available in their libraries and the scarcity of suitable sound resources
online. Thus, AR authors struggle to find appropriate sounds for distinct AR
events (e.g., reproducing the wing flutter of a virtual dragonfly or
simulating the eating sound of a virtual dinosaur).

To address these challenges, we present SonifyAR: a context-aware AR sound
authoring system using GPT-4 (openai2024gpt4, ) and a text2audio diffusion
model called AudioLDM (liu2023audioldm, ). SonifyAR makes the following key
technical advancements:

First, context collection using PbD. SonifyAR adopts a Programming by
Demonstration (PbD) (10.1145/291080.291104, ) pipeline to simplify the
specification of complex AR interactions. The PbD pipeline enables users to
demonstrate AR sound interactions while the system automatically detects the
action and collects context information. For example, if a creator wants to
sonify the stomping of a walking robot, they can position the virtual robot on
the target (physical) surface. As the robot walks, the collision between
robot’s feet and the surface, as well as the context information like the
robot’s attributes and the surface’s material, is captured by SonifyAR .

Second, context as text. To utilize AR context information stemming from
multiple sources (e.g. user action, virtual object, real-world environment)
and in different formats (e.g. categorical, 3D shape, image), we use text as
the universal medium to encompass all context information. By describing all
context with text and fitting different parts into a template (”This event is
[Event Type], caused by [Source]. This event casts on [Target Object].
[Additional Information on Involved Entities]”), we represent context
information of AR interactions as text snippets.

Finally, LLM-based sound acquisition. We integrate a suite of four sound
acquisition methods: recommend, retrieve, generate, and transfer, all
controlled by the underlying LLM. For each AR sound interaction, the context
information as text is fed to the LLM for processing. The LLM then provides
text prompts that control the suite of four sound acquisition methods, which
automatically provide matching sound assets for the AR interaction.

To author AR sound with SonifyAR, the creator initiates AR interactions and
the system automatically lists sound options based on context. For example,
imagine trying to make a virtual tea cup chime crisply when being slid on a
table (Figure 1). In traditional approaches, the creator would need to
manually specify this sliding action and find a sound asset to match the
chiming ceramic cup. But with SonifyAR, the creator could demonstrate this
sliding action and the chime sounds are generated automatically for the
creator to choose from.

To explore the potential of SonifyAR, we conducted two evaluations: first, a
qualitative user study with eight designers to examine the usability of the
SonifyAR prototype; second, we apply SonifyAR across five key user scenarios:
from AR education to improving AR headset safety. Our user study highlights
usability of the authoring interface and helpfulness of the generated sounds
while identifying space of improvements in terms of generated sound quality.
Our application scenarios demonstrate the breadth and potential of SonifyAR
incorporating automatically acquired sounds seamlessly into various AR
experiences—a practice that would otherwise take significant time and effort.

Our work makes three primary contributions: (1) a design space for AR sound
authoring tools, which highlights existing gaps in the literature; (2) a novel
context-aware AR sound authoring pipeline using generative AI, called
SonifyAR, that incorporates sensed environmental cues like surface material,
virtual object semantics, and user action; (3) findings from a user study with
eight designers and five application examples that demonstrate SonifyAR’s
potential and key advancements in this space. To our knowledge, we are the
first system to offer automatic, in-context sound authoring for AR.

##  2\. Related Work

We cover prior work in AR sound authoring, AI-based sound generation
techniques, and context-awareness in AR.

###  2.1. AR Sound Authoring

Sound in AR provides many benefits, from directing the user’s attention and
enhancing immersion (roginska2017immersive, ; zhou_role_2007, ) to creating
interactive time-varying experiences and increasing accessibility
(serafin_sonic_2018, ; ruminski2015experimental, ). However, as mentioned, the
current landscape of AR authoring tools and platforms reveals a deficiency of
support in AR sound authorship.

The existing AR authoring tools show diversity in their authoring capability
(monteiro2023teachable, ; nebeling_trouble_2018, ). Some are easy to use,
require no coding skills, and can only attach user-provided sound with certain
AR event triggers (haloar, ; arvid, ; adobeaero, ; realitycomposer, ). Some
are heavily code-based and require significant technical skills, but provide
substantial flexibility in designing AR sound interactions, including the
ability to set parameters like sound decay and code-specific conditions for
activating sound assets(unity-website, ; unreal-engine-website, ). One common
design in these tools is a Programming by Specification (PbS) methodology
(monteiro2023teachable, ), where creators define AR sound content and play
conditions during the design phase. This allows creators to precisely define
the conditions for triggers and the consequent sound effects.

As mentioned in the Introduction, we identified critical gaps with existing AR
authoring tools. These gaps are partially due to the limitation of the PbS
methodology, since the triggers specified in the design stage naturally lack
richness in real-world context, which is crucial for AR sound to be realistic.
Also, the trade-offs between supported trigger types and the tool’s usability
make most tools lack coverage for many sound-producing AR events.
Additionally, all sound assets need to be manually selected or created,
tasking creators with sourcing suitable sound effects to match the specific
triggers. The SonifyAR system aims to address these gaps.

###  2.2. Sound Acquisition

Although no work has explored sound generation with AR events as input, sound
generation conditioned from other input modalities like images, 3D models,
text, and videos has already been widely applied. Various cross-modal
generative models like RNN (Zhou_2018_CVPR, ), GAN (engel2019gansynth, ;
kumar2019melgan, ; ghose2022foleygan, ), VAE (dhariwal2020jukebox, ), and most
recently diffusion models (kong2020diffwave, ; liu2023audioldm, ;
yang2023diffsound, ), are utilized to generate sound. Despite different model
architecture, one common training goal is the mapping between latent
representations of input modalities and output sound, thus sounds can be
generated from prompts like images (10096023, ), videos (Zhou_2018_CVPR, ;
ghose2022foleygan, ), and text (liu2023audioldm, ; yang2023diffsound, ). For
example, FoleyGAN (ghose2022foleygan, ) conditions action sequences of input
videos leading to the generation of visually aligned realistic soundtracks.
AudioLDM (liu2023audioldm, ) utilize CLAP (wu2023large, ) based latent space
to embed input text and generate matching sound with diffusion model. Unlike
the cross-modal generative methods, another notable line of work is physics-
based sound synthesis (raghuvanshi_interactive_2006, ; roodaki_sonifeye_2017,
; ren_example-guided_2013, ; jin_deep-modal_2020, ; jin_neuralsound_2022, ;
liu_sound_2021, ; diaz_rigid-body_2022, ). Traditionally using methods like
spring-mass model (raghuvanshi_interactive_2006, ) and recently improved with
deep learning (jin_deep-modal_2020, ), these synthesis methods can provide
realistic sound simulation given 3D model and material properties.

Besides synthesizing, retrieval methods can also acquire matching sound for
input conditions. For example, Oncescu et al. (Oncescu21a, ) trained cross-
modal embedding models to support text-to-audio retrieval. Multi-modality
models like VAST (chen2023vast, ) also supports text-to-audio retrieval that
could potentially be used to provide sound assets for AR experiences. A recent
work called Soundify (lin2023soundify, ) utilizes retrieved sound assets to
sonify videos. They claim that although sound retrieval do not ensure coverage
for all input descriptions, the retrieved sound assets usually surpass
synthesis results in terms of quality. In this case, combining generation and
retrieval in the sound acquisition pipeline seems to be a natural method that
integrates merits of both. We are going to explore this possibility in this
work.

Unlike other existing sound acquisition methods, AR poses unique challenges in
the matching of input condition and output sound. The multi-modal nature of AR
interactions, including the information of the virtual object (e.g. a toy
robot with a walking animation), real environment (e.g. a living room space
with a wooden table in the middle), and user intent (e.g. user taps on a
virtual model), need to be carefully processed into the sound acquisition
pipeline to result in matching sound assets.

###  2.3. Context-awareness in AR

In AR, context-awareness is an important property that allows AR content to be
dynamically adjusted according to context information, such as location, scene
semantics, and human factors. Context-awareness ensures that the application
adapts not only to the user but also to their environment and specific needs,
making the application more user-friendly and immersive
(krings_development_2020, ). Previous research has explored various aspects of
context-awareness in AR, highlighting its importance across different
applications.

For example, the popular AR game Pokemón Go shows game contents based on
user’s physical location. A more common practice in AR research is adapting
virtual contents to scene semantics—essentially understanding the objects and
their locations in the user’s environment. Qian et al. (qian2022scalar, )
introduced an authoring system that helps user create adaptive AR experiences
to surrounding scene semantics. Lang et al. (lang2019virtual, ) analyzes scene
semantics to guide the positioning of virtual agents. Similarly, Liang et al.
(liang2021scene, ) places virtual pets into AR based on scene geometry and
semantics. This realm of similar work (han2020live, ;
tahara_retargetable_2020, ) usually captures and analyzes surrounding
environment, and designs algorithms that determines possible interaction
between virtual content and real-world scene. Lindbauer et al.
(lindlbauer2019context, ) push the AR context-awareness further by including
task and cognitive load as context to adjust level of detail of AR virtual
interface.

Although there has been substantial discussion in the realm of AR context-
awareness, we observed that prior work generally focuses on the visual
modality of AR, neglecting sound. Additionally, existing work primarily
concentrates on adjusting and arranging virtual content rather than generating
new content. We aim to fill this gap by designing a sound generation pipeline
that processes context information to produce matching sound assets.

##  3\. Design of SonifyAR

To design an effective sound authoring system that generates matching sounds
for AR interaction, three main research questions need to be answered.

  * •

What are the target AR interactions that should produce sound effects?

  * •

How can we efficiently acquire sound assets for these AR interactions?

  * •

How can these AR interactions be specified and represented in the authoring
process?

We aim to elaborate on these questions and the respective design rationale in
this section.

###  3.1. The Triad of User, Virtuality, and Reality

We began by examining the design space of AR interaction sounds. Drawing on
Jain et al.’s (jain_taxonomy_2021, ) sound taxonomy in VR, we attempted a
similar analysis for AR sound interaction. We first investigated several
widely used code-free AR authoring tools for their AR sound capability,
including Adobe Aero (adobeaero, ), Apple Reality Composer (realitycomposer,
), and Halo AR (haloar, ). We found that these tools support AR sound related
to either virtual content interaction or AR-based spatial anchors, e.g. play
sound when users get close to a certain position, scan a QR code, or tap on a
virtual model.

The research literature in AR sound (dam2024taxonomy, ;
rakkolainen2021technologies, ; filus2012using, ), however, envisions a much
broader scope. E.g. Rakkolainen et al. (rakkolainen2021technologies, ) have
classified AR audio into five classes: ambient, directional, musical, speech,
and noise. Dam et al.’s (dam2024taxonomy, ) taxonomy of Audio Augmented
Reality (AAR), defines AR-based sound across Environment Connected (sound
maintaining awareness and interaction with physical environment), Goal
Directed (sound assisting users’ primary goal), and Context Adapted (sound
adaptive to users’ immediate reality), as well as their intersection. This
taxonomy emphasizes that AR sound is often rooted in contextual information
from the real world environment and user intention.

Drawing on these prior work, we introduce a novel AR sound design space that
helps emphasize the sonic interplay between three key factors of AR
experiences: reality, denoting the physical environment in the real world that
hosts the AR experience; virtuality, referring to the virtual object(s) placed
in the AR experience; and the user, representing the individual interacting
with the AR experience. Below, we expand on each element of this proposed
design space and indicate current levels of support in the existing code-free
AR sound authoring tools (adobeaero, ; realitycomposer, ; haloar, ).

Figure 2. The sound-producing opportunities in the triad of User, Virtuality
and Reality.

Virtuality (partially supported): Virtuality refers to sounds that accompany
AR events involving only virtual objects. This could be bound to an object’s
change of status (e.g. a notifying sound when a virtual object shows up), from
an object’s animated behavior (e.g. the mechanical noise made by a virtual
dinosaur roaring), or the interaction between multiple virtual objects (e.g.
two virtual balls clacking when collide). Current authoring tools support
status change and animation playing as sound initiating triggers, but do not
support sound for interactions between virtual objects.

User-Virtuality (well supported): User-Virtuality are the corresponding sounds
that react to the users’ actions with virtual objects, e.g. a virtual dog
barks when users tap on or get close to it. In all of our investigated AR
creation tools, these user-initiated triggers like “Tap” and “Proximity Enter”
are well supported.

User-Reality (not well supported): User-Reality refers to sounds that
accompany user interactions with the physical environment via their AR
devices. These sound feedbacks can improve users’ understanding of the
surrounding environment. E.g. a tap sound when users tapping on a real-world
table surface on their phone screen. None of our investigated AR authoring
tools support user-reality actions as triggers.

Virtuality-Reality (not well supported): Virtuality-Reality denotes sound
feedback that plays crucial roles in enhancing realism when virtual objects
interact with the real-world environment, e.g. the crisp stomping sound of a
virtual robot when it walks on a real-world glass surface. Although physics
simulation between virtual models and real-world surfaces have been supported,
their sound feedback remains unexplored in AR authoring tools.

User-Virtuality-Reality (not well supported): User-Virtuality-Reality are the
sounds that accompany user actions involving both virtual and real elements.
For example, the material-aware sliding sounds when users apply a virtual
scraper on different real-world surfaces (e.g., wooden table, painted wall, or
glass window). None of our investigated authoring tools can support the
specification of this complex interaction.

Others Note that we excluded the domains of Reality and User from the
enumeration since our interest lies in events where the AR system (e.g.
smartphone or AR headset) generate the sound. Thus, sounds naturally generated
by user and reality are excluded from our discussion.

Based on the above analysis and as illustrated in Fig. 2, existing AR
authoring tools fail to offer functional AR sound authoring support for many
of the identified dimensions, especially those involving Reality. SonifyAR
aims to cover this gap.

###  3.2. Acquisition of AR Sound

While the above design space helps highlight the dimensions of AR sound and
current authoring support, below we reflect on key challenges related to the
limitations of sound acquisition methods in terms of handling these different
dimensions.

For example, a mechanical noise of a virtual robot (virtuality) can be easily
sourced from local or online sound databases. However, more specific sounds,
like a virtual steel ball hitting a physical concrete wall surface or a
virtual racecar jumping into a backyard pool (virtuality-reality) requires
physical world understanding and material awareness. The space of possible
sound effects here is near infinite. Thus, in such cases, generating sounds
using text-to-sound models is preferable.

Conversely, certain highly recognizable sounds, like a dog barking, pose
difficulties for current generative models, often resulting in outputs that
are unsatisfactory and noisy. The complexity in user needs and trade-offs in
different methods suggests that the combination of both generation and
retrieval could serve as a practical solution.

###  3.3. Event Representation

To combine generation and retrieval methods, one challenge is how to condition
this suite of different sound acquisition methods. While visual formats such
as images (10096023, ) and videos (Zhou_2018_CVPR, ; ghose2022foleygan, ) have
been widely used as inputs for sound generation models, they fall short in
fully capturing AR context information, particularly user actions. To overcome
this, we explore the use of text as a universal representation for sound
acquisition inputs, which has the following benefits. First, text can
sufficiently and precisely convey necessary context information and the
specifics of sound-generating events (e.g.,“User slides a ceramic cup on a
wooden surface”). Second, given that the AR experience operates within a
hardware-software system with multi-modal sensors (e.g., camera, IMU, GPS), we
can leverage this system to monitor and log events within the AR experience
and summarize them into descriptive text. Third, recent advancements, as
demonstrated by works like HuggingGPT (shen2023hugginggpt, ) which uses a LLM
to control AI tasks, showcase the practicality of using LLMs as controllers to
process text for sound generation.

##  4\. SonifyAR Implementation

Figure 3. Overview of the pipeline of SonifyAR. Our system monitors and logs
context information of AR events, which includes the event type, the subjects
and objects (virtual or real-world), and the attributes of the involved
elements like their materials. This information is compiled into a text
template and then processed by our LLM controller to acquire sound assets. The
results are subsequently presented in our selection panel.

We introduce SonifyAR, a novel PbD sound authoring framework that uses context
recognition and generative AI to create personalized, context-sensitive sounds
for AR interactions (Figure 3). SonifyAR consists of three primary components
: (1) Event Textualization: Every user action is recognized as an AR event,
and the context of these events is transformed into textual descriptions. (2)
Sound Acquisition: An LLM-controlled sound acquisition process that utilize
four methods to produces sound effects based on the context. (3) User
Interface: An interface that allows users to experience the AR scene and
provides the capability to view, modify, and test sound effects for AR events.

###  4.1. Event Textualization

In SonifyAR, we utilize a PbD authoring framework to capture potential sound-
producing AR events and compile the context information into text. When users
interact with the AR space, SonifyAR captures AR events that can lead to sound
feedback. Here we define an AR event as an user action and its subsequent
result (e.g. when a user taps a virtual object to trigger its animation). For
each AR event, we describe it with an event type (e.g. tapping an object ),
action source (e.g. user or virtual object), and action target (e.g. virtual
object or real-world plane). Information about the involved parties, such as
virtual objects or real-world planes, will also be included in the context
information. This collected event data is then transformed into text with a
template.

Figure 4. SonifyAR’s event textualization. Left: a user tapping on a virtual
cup through the SonifyAR interface; Right: the textual context information
extracted by SonifyAR’s internal PbD framework.

#### Event Types.

To support a broad range of AR sound interaction, we implemented six event
types:

  1. (1)

Tap Real World Structure: A user taps on a real-world structure through the AR
interface.

  2. (2)

Slide: A user holds the virtual object and slides it on a real-world surface.

  3. (3)

Collide: A virtual object collides with another or with a real-world surface.
We implemented a specialized collision detection mechanism as introduced in
subsection 4.4

  4. (4)

Show Up: A virtual object shows up in the AR experience.

  5. (5)

Tap Virtual Objects: A user taps on a virtual object through the AR interface.

  6. (6)

Play Animation: A virtual object plays an animation.

#### Scene Context Understanding.

SonifyAR utilizes ARKit (apple-arkit-doc, )’s plane detection functionality
(apple-arkit-plane-tracking, ) to assess the surrounding environment. Since
most AR experiences are anchored to planes, the detected plane information
serves as a crucial context. To enrich such information, we employ the Deep
Material Segmentation model (upchurch_dense_2022, ) to segment the scene and
identify the material of planes (Figure 3), which helps produce realistic
sound effects when an AR event involves this plane. Currently, we support six
materials: wood, carpet, concrete, paper, metal, and glass. Surfaces that do
not fall into one of these six categories are labeled as unknown surface.

#### Virtual Object Understanding

The semantics of the virtual object is also crucial for sound generation. To
ensure sound assets align with the virtual object’s material and its state, we
collect a text description for all the virtual objects (e.g. ”This model is a
toy robot made of metal.”) and their animations (e.g. ”A toy robot walks.”).
These descriptions can be provided by the asset creator or, if necessary, we
prompt the user to add relevant details. Any AR event involving the virtual
object or its animation will incorporate these text descriptions.

#### Event-to-text

To aggregate the multi-source context information described above, we employ a
simple text template: ”This event is [Event Type], caused by [Source]. This
event casts on [Target Object]. [Additional Information on Involved
Entities]”.

In our template, Event Type is the type of event described by the type name,
as listed above. Source refers to the subject of the event, which could be the
user when the event is directly triggered by user, or virtual object when it
interacts with real-world environment. Target Object is the object of the
event, e.g. the plane that gets tapped on, or the animation that gets played.
Additional Information on Involved Entities include details that further
elucidate the source object, the triggered event, and the target object.
Examples include material descriptions and animation details. We leave the
corresponding entity field blank when events are missing a target object or
additional information.

During user interactions, the system actively monitors and logs the events
happening in AR space. The context information is fetched and plugged into the
text template, crafting a coherent description that reflects the user’s
interaction within the AR environment.

###  4.2. Sound Acquisition

Figure 5. SonifyAR’s sound acquisition pipeline.

SonifyAR utilizes the LLM to automatically retrieve or generate context-
matching sound assets of an AR event. Inspired by HuggingGPT
(shen2023hugginggpt, ) and Visual ChatGPT (wu2023visual, ), we utilize the LLM
as a controller of multiple sound authoring methods. The LLM takes the text
description of the event as input and replies with commands for multiple sound
acquisition methods (Figure 5). At our current stage, we support four major
sound authoring methods: local recommendation, online retrieval, text2sound
generation, and text-guided sound transfer.

#### Local Recommendation

The LLM can recommend sound assets stored in the local database based on
semantics in the event description. Similar to other AR authoring tools, we
collect a set of sound effects from Adobe Audition sound effects (adobe-
audition, ), each labeled with a descriptive file name, like “Crash Aluminum
Tray Bang” or “Liquid Mud Suction”. The entire list of sound file name is
provided to LLM. When provided with the event context, the LLM recommends the
top five most suitable local sound effects based on their file names. It then
returns the selected sound effect in the format of method1recommend:FILENAME,
where the FILENAME would be replaced with the actual filename. Upon receiving,
SonifyAR parses the filename and adds the top corresponding sound as one of
the sound options for the event. User can also long hold such option to reveal
other results from the top5 recommendation.

#### Online Retrieval

We also expand our retrieval capability with an online sound asset database
called FreeSound (freesound, ). We use the FreeSound API, which returns a list
of best matching sound effects based on a given query. The queries are
condensed versions of full event descriptions generated by the LLM, returned
in the format method2retrieval:PROMPT, with PROMPT replaced by the specific
search query. The returned results are JSON strings containing information on
all search results. We select the top five matches from the entire set of
sound effects returned by the API, which are then downloaded and presented to
the user as sound options.

#### Sound Generation

We adopt a audio diffusion model AudioLDM (liu2023audioldm, ) to generate
sound effects with text prompt. We ask the LLM to compress the event text
description into a shortened generation prompt (returned in the format
method3generation:PROMPT, where the PROMPT would be replaced with the
generation prompt). Upon receiving such a command from the LLM, SonifyAR sends
the prompt to the AudioLDM model, requesting text2sound generation. These
newly generated sounds are then presented to user as sound options.

#### Sound Transfer

Besides text2audio generation, AudioLDM is also capable of performing text-
based sound style transfer. Specifically, for events like tapping, sliding, or
colliding, instead of generating a new sound from scratch, SonifyAR uses a
default sound effect and initiate a style transfer operation with a text
prompt provided by the LLM. This approach allows the output sounds to match
the length and rhythm of the input sounds, so that they can be well-timed with
actions. Furthermore, when users wish to further modify any provided sound
assets, SonifyAR offers text-based style transfer as a fine-tuning and
customizing option.

All sound authoring processes listed above operate concurrently in the
backend. This ensures uninterrupted AR experience.

###  4.3. User Interface

Figure 6. SonifyAR’s authoring interface. Left: SonifyAR’s phone-based AR
interface. Right: SonifyAR’s authoring panel.

As a PbD authoring framework, the SonifyAR application initiate directly with
an interactive AR experience (Figure 6 left). Users can freely interact with
the scene, performing actions like moving virtual objects or interacting with
the real-world surfaces. When an AR event is detected, a text label appears on
the top-left screen to inform the user (Figure 6. A) and the sound acquisition
component is activated to generate candidate sound effects for the detected
events.

By clicking a button (Figure 6. D), users can prompt an editing interface
(Figure 6. F) that overlays on the AR scene. The interface includes all unique
AR events detected (Figure 6. G) and the automatically generated sound assets
(Figure 6. H), enabling users to explore the detected events and their
corresponding sound effects. Users can click on a sound effect to preview it
and double click to select and activate it during the AR session. After
confirming their choices, users can hide the authoring panel to resume the AR
experience and test the selected sounds. Users can always access the editing
interface to modify their choices.

Furthermore, users can long press a sound asset to call out a menu with a
suite of exploratory options. For recommended or retrieved sounds, the menu
includes an option to list all other sound assets in the top five
recommendations. For all sound assets, the menu provides a “style transfer”
and a “generate similar sounds” feature, enabling users to style transfer
audio effects or to generate similar sounds based on the selected sound. When
this feature is selected, users can type a simple text prompt to guide the
sound generation process. This feature enables users to iteratively refine
sound effects and explore new sound variations.

###  4.4. Technical Implementation

We developed SonifyAR as a mobile app with a user interface that enables AR
interaction and sound authoring actions. We also implemented back-end servers
for scene understanding and sound generation. The current SonifyAR demo is
built with Apple’s ARKit with Swift 5.7 and Xcode 14, and runs as an iOS
application on iPhones. Due to the enhanced performance of plane detection on
devices with LiDAR (available in iPhone’s Pro lineup starting from iPhone Pro
12), we developed and tested the app on an iPhone 13 Pro Max running iOS 16.
We implemented the collision detection between virtual objects and planes with
ARKit’s physics simulation (apple-scenekit-physics-simulation, ) function. To
enable collision detection for animated virtual object parts (e.g. the
stomping of a robot foot on the plane), we bind colliders to the joints of the
virtual object’s animated bones.

We use GPT-4 (openai2024gpt4, ) as the LLM in our system. We adopt the DMS
model (upchurch_dense_2022, ) for material segmentation because of its
detailed material labeling, fast processing time, and high accuracy. For sound
generation, we use the AudioLDM model (liu2023audioldm, ) due to its state-of-
the-art performance and its versatility in text2sound generation and text-
guided sound2sound transfer. All models are hosted on a server equipped with
an NVIDIA RTX 4080 GPU and CUDA 12.2.

##  5\. User Study

To examine the usability and authoring performance of the SonifyAR system, we
conducted a usability study across eight participants, who used the SonifyAR
application in a fixed experimental setting and provided ratings for the
system.

###  5.1. Participants

We recruit eight participants (six male and two female, aged between 26 and
33) with varying experience in AR. We inquired about their prior experience in
AR before the study. Out of the eight participants, six had prior AR
experience, and three had experience in AR authoring tools.

###  5.2. Procedure

The usability study was conducted in a small meeting room and comprised four
steps. We first introduce our study goal with the participants and collect
answers to our demographic questions. We then demonstrated our SonifyAR app
with an example AR experience. During this phase, participants had the
opportunity to ask any questions regarding SonifyAR’s functionalities.
Afterwards, participants were asked to independently explore the SonifyAR app.
We provided a walking robot model which can be added to indoor surfaces. This
model starts walking when tapped. We asked the participants to explore this AR
asset freely and use our automatic sound authoring pipeline to create sound
effects with the model until they are satisfied with the results. The entire
usage process was screen-recorded and the user operations were logged. The
final step involved getting participants’ feedback regarding the usability,
helpfulness and technical performance of the SonifyAR system based their
personal experiences. Participants also provided reasoning and insights to
support their answers. The questions and rating results are shown in Figure 7.

###  5.3. Results

All participants were able to complete the AR sound authoring task using
SonifyAR without difficulty. On average, the authoring process took 406
seconds (SD=137s) to finish. Participants tested an average of 56 (SD=10.48)
sound assets, with an average of 6.4 sounds (SD=1.67) assigned to the
authoring results.

Feedback on the SonifyAR app was largely positive. All eight participants
expressed favorable impressions of the tool. They highly agreed that the
SonifyAR tool would be helpful to AR authoring process (Q1,AVG=6 out of 7,
SD=0.93). There was also a general willingness for using SonifyAR in their own
AR creation practice (Q2, AVG=6.25, SD=1.16). Participants agreed that the
sound interaction involving real-world surfaces will improve the immersiveness
of AR experiences (Q3, AVG=6.38, SD=1.06). Additionally, they preferred the
automated sound authoring process over the manual search for sound assets (Q5,
AVG=6.13, SD=0.99). The only area for improvement was the quality of the
generated sound (Q4, AVG=4.75, SD=1.39). However, this could be enhanced with
the introduction of a more advanced sound generative model, which can be
easily integrated into our system. The full list of questions and the
respective Likert-type results can be viewed in Figure 7.

Figure 7. Likert-type questions and results of our usability study

##  6\. Applications

To demonstrate the potential and technical capability of SonifyAR, we explore
five example application scenarios that utilize SonifyAR’s sound authoring
pipeline in AR applications. All examples are also shown in supplementary
videos for better presentation.

###  6.1. Education

AR have been widely explored in STEM education (sirakaya2022augmented, ). As
existing educational applications try to simulate and visualize physical or
chemical phenomenon in AR (radu2019can, ; irwansyah2018augmented, ), we
envision that SonifyAR can help add sound into the simulation and improve the
blending between real environment and virtual contents.

To showcase such possibility, we implement an AR physics experiment with
SonifyAR demo application. We create a 3D model to show the conservation of
momentum, which is a slope and two metal balls located at top and bottom of
the slope (Figure 8). Upon initiating the experiment, the top ball rolls down
the slope and collides with the bottom ball. This collision causes the bottom
ball to move forward and eventually fall, while the top ball comes to a stop.
With SonifyAR, collision events are automatically detected, and material-
specific sound effects, such as the clashing of two metal balls or a metal
ball dropping onto table, are seamlessly generated. The resulting sound
effects can be experienced in the supplementary video.

Figure 8. An AR physics experiment sonified by SonifyAR. (A) The slope and
metal ball model and simulation process. (B), (C) Context information of two
collisions in the simulation. (D) The acquired sound assets for the
collisions.

###  6.2. Accessibility

AAR (Audio Augmented Reality) has been explored as accessibility assistance
for blind or low vision people (ribeiro2012auditory, ; coughlan2017ar4vi, ).
By sonifying real world environments and virtual contents, blind or low vision
(BLV) people can better interpret visual information in both reality and
virtuality. Following this thread of work, we envision SonifyAR being able to
assist on accessibility in AR experiences in three ways. First, by supporting
AR sound authoring, SonifyAR encourages creators to add sound effects in AR
experiences, which can help BLV users consume visual contents. Second, by
providing context-aware AR sound in 3D audio, people with low-vision can
better navigate virtual objects in AR environment. Third, by enabling user to
interact with real-world surfaces via AR (e.g. tapping on a real world
surface), user can explore the surrounding space via the AR interface.

Figure 9. Robot walking on different surface with different sound. (A), (B)
Robot walking on a wooden surface, and the corresponding context information.
(C), (D) Robot walking on carpet surface, and the corresponding context
information. (E), (F) The acquired sound assets.

We use a toy robot 3D model and the SonifyAR demo app to showcase the
accessibility benefits. The toy robot (Figure 9) can be placed on any indoor
surface and has a walking animation. By authoring different bouncing sound
effects for different surfaces (e.g. wood and carpet), user can better
perceive the location of the robot while it is walking.

###  6.3. Sonify Existing Apps

Due to the extensibility of the text template used in SonifyAR’s sound
acquisition pipeline, we foresee its application in existing AR apps if
integrated as an extension. If implemented at the SDK level (e.g. ARKit and
ARCore), SonifyAR can automatically capture textual description of AR events
and provide sonification using the automatic sound acquisition results.

Figure 10. AR dinosaur sonified by SonifyAR. (A) AR Camera app interface with
a virtual dinosaur playing eating animation in the scene. (B) The context
information of the scene. (C) Text prompt and sound output from SonifyAR’s
pipeline.

We explore this application possibility with a Wizard of Oz (WoZ) prototype on
a phone-based AR application called ARVid (arvid, ), which is an AR app that
puts 3D animated objects into real world environment. We select a dinosaur
animation as an example (Figure 10), and conceptualize that SonifyAR pipeline
is adopted into the AR infrastructure. By manually feeding the context
information (e.g. “The dinosaur is eating”) into the SonifyAR pipeline, we
generated sound assets that well-match the 3D animations. The results are
shown in the supplementary video.

###  6.4. Sonify virtuality-reality interaction on AR/MR headset

As recent AR/MR headsets become increasingly popular in entertainment and
productivity tasks, manufacturers have established safety guidelines, such as
setting up safety zone and clear up indoor spaces. We envision SonifyAR
contributing to this topic by creating material-aware sound effects during
headset usage when application windows or virtual object intersect with real
world surfaces or objects.

Figure 11. Sonification of Vision pro’s window operation. (A) User place a
virtual window on sofa. (B) Context information of this action. (C) Text
prompt and sound output from SonifyAR’s pipeline.

We implement a WoZ prototype based on Apple Vision Pro’s user interaction
(Figure 11). When using Vision Pro, user can actively adjust the position of
application windows by hand gestures. SonifyAR can enhance this interaction by
generating AR sounds when virtual windows are placed on or intersect with
indoor surfaces like walls, desks, and floors. By processing these
interactions and applying sound transfers with text prompts such as “a virtual
window bumps into a wall” and “a virtual window collides with a carpet”,
SonifyAR aims to improve the permanence of virtual windows and increase
awareness of surrounding objects as users navigate through virtual windows.
See supplementary video for more direct inspection of the sound effects.

###  6.5. Sound asset generation for traditional AR authoring process

Besides serving as a PbD authoring experience, SonifyAR’s sound authoring
pipeline also have the potential of improving existing PbS AR authoring tools
in sound assets provision. As the mainstream AR authoring tools like Adobe
Aero and Apple Reality Composer already represent their AR interaction as text
like “Tap & Play Sound” and “Proximity & Jiggle”, these specifications can be
fed into SonifyAR’s text-based sound authoring pipeline. In this case,
SonifyAR works as a plugin of these tools and provide matching sound assets
for AR interactions.

Figure 12. WoZ example of SonifyAR being applied in Reality Composer’s
authoring process. (A) A machine model in Reality Composer’s authoring
interface. (B) Reality Composer’s interface for AR sound. (C) Context
information of the event in B. (D), (E) Two sound assets generated and
retrieved with SonifyAR’s pipeline. (F) The acquired sound being used as
recommended options.

We implement a WoZ prototype using Apple’s Reality Composer and a 3D model of
an animated machine (Figure 12). When authoring the “Play Sound” behavior of
this model, context information like the model description, triggering
condition, and animation description, can be fed into SonifyAR’s text-based
authoring pipeline and lead to two sound assets from generation and retrieval
methods. These two sound assets will appear at top of the sound asset
selection as “SonifyAR Recommended”. See supplementary video for more details.

##  7\. Discussion and Future Work

In this paper, we designed and implemented an AR sound authoring pipeline that
automatically generates sound assets based on context information. We
evaluated and discovered the opportunities in the AR sound interaction space,
and implemented an LLM+Generative AI pipeline for sound acquisition. We tested
the pipeline with a usability study on our prototype and further discussed the
application opportunity with five examples.

###  7.1. Limitations and failure cases

While SonifyAR was positively evaluated in our user study and demonstrated
broad application potential, we observed limitations in the system
adaptability and encountered failure cases in multiple aspects.

Firstly, since the sound acquisition pipeline relies on LLM as the context
collector and overall controller, errors of the LLM can lead to failure cases
in sound acquisition. For example, the LLM can extract the wrong keywords in
the context information and prompt sound generation model with inaccruate text
input (e.g. LLM provide prompt of “sliding plastic on carpet” while the
sliding model is actually a metal toy robot) that lead to non-matching
outputs.

Secondly, although AudioLDM (liu2023audioldm, ) is the state-of-the-art
text2audio generator and performs decently in most cases, it can sometimes
produce subpar outputs that are noisy or do not match prompts. Based on our
experiences, such inconsistency in performance can be hard to predict thus
cannot be completely solved by prompt engineering.

Thirdly, some AR interactions can be highly time-sensitive (e.g. objects
colliding, the playing of animations that shows very clear action sequences)
and require sound assets being closely synchronized with visual contents.
Currently, we provide example sound assets for AR interactions of “Colliding”,
“Tapping” and “’Sliding’. By conducting sound style transfer using well-timed
example sounds as inputs, the output sounds can be formatted to be in sync
with the AR interaction. However, for animations, we currently do not have
methods to sync the generated sound with the key frames in animation, so some
generated sounds can be off-sync with animation. We believe this 3D animation-
to-audio synthesis challenge can be a standalone topic for future researchers
to solve.

###  7.2. Expansion of framework

Currently, we implement the automated AR sound acquisition with six example AR
event types, as listed in section 4. This selection is primarily for proof of
concept. Future work, or other systems adopting SonifyAR’s text-based sound
acquisition pipeline, could easily incorporate additional types of AR
interactions, provided they have clear and concise text descriptions. For
example, event types used in Adobe Aero and Reality Composer, such as “Spin”,
“Emphasize”, “Expand”, or “Jump”, can be well-incorporated into the text
template of SonifyAR. The flexibility and extensibility of our text-based
pipeline is also explored in our application examples.

###  7.3. Future work

We plan to address the following improvements for future work.

First, our current SonifyAR app is a simplified tool that exclusively supports
the authoring of AR sound and uses pre-crafted AR scenes as input. For wider
applicability that reflects the real-world needs of AR content creation, we
recognize the need to incorporate the AR sound authoring pipeline into the
entire AR content creation process. Our future goals include integrating the
SonifyAR framework with established tools like Reality Composer and Adobe
Aero.

Secondly, SonifyAR adopts a straightforward process of using a text template
to compile context information collected from multiple sources, including
virtual object semantics and real world semantics. Currently, we use one
material segmentation model to perceive real-world context information. In
future developments, we aspire to incorporate more sophisticated computer
vision models, or multi-modal foundational models, for deeper understanding of
both virtual objects and real world environment, enhancing the context
acquisition process with greater precision and broader adaptability.

Lastly, the current sound generation output of SonifyAR presents opportunities
for further refinement. Recognizing the rapid advancements in the AI research
domain, our system has been designed as a modular framework, ensuring that as
more advanced models emerge, they can be seamlessly integrated. Every model
(e.g., LLM, CV, Sound Generation) within SonifyAR is replaceable, allowing
SonifyAR to easily adapt and stay at the forefront of innovation in this
field.

##  8\. Conclusion

In this paper, we present SonifyAR, a context-aware sound authoring system
designed for sonifying AR events. Our work implements a PbD authoring pipeline
which enables sound asset acquisition using context information. With
SonifyAR, users could demonstrate AR interactions and have AR sound assets
automatically acquired. Our studies validate the usability and overall
performance of our system. Our five application examples further widens our
application potential. We believe SonifyAR can be applied in application
fields like education, BLV assistance, and AR headset safety. We also envision
it incorporated into existing AR authoring tools to strengthen their sound
authoring capabilities. This research advances literature in AR sound
authoring, while also opening up new research avenues for the application of
LLM and generative models in AR systems.

## References

  * (1) ARVid - Augmented Reality.  Accessed on September 24, 2023.
  * (2) Freesound.  Accessed on September 24, 2023.
  * (3) Halo AR.  Accessed on September 24, 2023.
  * (4) Adobe.  Adobe audition sound effects download page, 2023.  Accessed on Date of Access.
  * (5) Adobe.  Adobe aero.  https://www.adobe.com/products/aero.html, Accessed September 11, 2023.
  * (6) Apple.  Apple arkit documentation, 2023.  Accessed on Oct 9th, 2023.
  * (7) Apple.  Arkit - tracking and visualizing planes, 2023.  Accessed on Oct 9th, 2023.
  * (8) Apple.  Scenekit - physics simulation, 2023.  Accessed on Oct 9th, 2023.
  * (9) Apple.  Reality composer.  https://apps.apple.com/us/app/reality-composer/id1462358802, Accessed September 11, 2023.
  * (10) Chen, S., Li, H., Wang, Q., Zhao, Z., Sun, M., Zhu, X., and Liu, J. Vast: A vision-audio-subtitle-text omni-modality foundation model and dataset, 2023.
  * (11) Coughlan, J. M., and Miele, J. Ar4vi: Ar as an accessibility tool for people with visual impairments.  In 2017 IEEE International Symposium on Mixed and Augmented Reality (ISMAR-Adjunct) (2017), IEEE, pp. 288–292.
  * (12) Dam, A., Siddiqui, A., Leclercq, C., and Jeon, M. Taxonomy and definition of audio augmented reality (aar): A grounded theory study.  International Journal of Human-Computer Studies 182 (2024), 103179.
  * (13) Dhariwal, P., Jun, H., Payne, C., Kim, J. W., Radford, A., and Sutskever, I. Jukebox: A generative model for music.  arXiv preprint arXiv:2005.00341 (2020).
  * (14) Diaz, R., Hayes, B., Saitis, C., Fazekas, G., and Sandler, M. Rigid-Body Sound Synthesis with Differentiable Modal Resonators, Oct. 2022.  arXiv:2210.15306 [cs, eess].
  * (15) Engel, J., Agrawal, K. K., Chen, S., Gulrajani, I., Donahue, C., and Roberts, A. Gansynth: Adversarial neural audio synthesis.  arXiv preprint arXiv:1902.08710 (2019).
  * (16) Filus, M. I. A. M., and Rambli, D. R. A. Using non-speech sound as acoustic modality in augmented reality environment.  In 2012 International Symposium on Computer Applications and Industrial Electronics (ISCAIE) (2012), IEEE, pp. 79–82.
  * (17) Games, E. Unreal engine, 2023.  Accessed on Oct 9th, 2023.
  * (18) Ghose, S., and Prevost, J. J. Foleygan: Visually guided generative adversarial network-based synchronous sound generation in silent videos.  IEEE Transactions on Multimedia (2022).
  * (19) Han, L., Zheng, T., Zhu, Y., Xu, L., and Fang, L. Live semantic 3d perception for immersive augmented reality.  IEEE transactions on visualization and computer graphics 26, 5 (2020), 2012–2022.
  * (20) Irwansyah, F. S., Yusuf, Y., Farida, I., and Ramdhani, M. A. Augmented reality (ar) technology on the android operating system in chemistry learning.  In IOP conference series: Materials science and engineering (2018), vol. 288, IOP Publishing, p. 012068.
  * (21) Jain, D., Junuzovic, S., Ofek, E., Sinclair, M., R. Porter, J., Yoon, C., Machanavajhala, S., and Ringel Morris, M. A Taxonomy of Sounds in Virtual Reality.  In Proceedings of the 2021 International Conference on Multimodal Interaction (Montréal QC Canada, Oct. 2021), ACM, pp. 80–91.
  * (22) Jin, X., Li, S., Qu, T., Manocha, D., and Wang, G. Deep-Modal: Real-Time Impact Sound Synthesis for Arbitrary Shapes.  In Proceedings of the 28th ACM International Conference on Multimedia (New York, NY, USA, Oct. 2020), MM ’20, Association for Computing Machinery, pp. 1171–1179.
  * (23) Jin, X., Li, S., Wang, G., and Manocha, D. NeuralSound: learning-based modal sound synthesis with acoustic transfer.  ACM Transactions on Graphics 41, 4 (July 2022), 1–15.
  * (24) Kong, Z., Ping, W., Huang, J., Zhao, K., and Catanzaro, B. Diffwave: A versatile diffusion model for audio synthesis.  arXiv preprint arXiv:2009.09761 (2020).
  * (25) Krings, S., Yigitbas, E., Jovanovikj, I., Sauer, S., and Engels, G. Development framework for context-aware augmented reality applications.  In Companion Proceedings of the 12th ACM SIGCHI Symposium on Engineering Interactive Computing Systems (New York, NY, USA, June 2020), EICS ’20 Companion, Association for Computing Machinery, pp. 1–6.
  * (26) Kumar, K., Kumar, R., De Boissiere, T., Gestin, L., Teoh, W. Z., Sotelo, J., De Brebisson, A., Bengio, Y., and Courville, A. C. Melgan: Generative adversarial networks for conditional waveform synthesis.  Advances in neural information processing systems 32 (2019).
  * (27) Lang, Y., Liang, W., and Yu, L.-F. Virtual agent positioning driven by scene semantics in mixed reality.  In 2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR) (2019), IEEE, pp. 767–775.
  * (28) Lau, T. A., and Weld, D. S. Programming by demonstration: An inductive learning formulation.  In Proceedings of the 4th International Conference on Intelligent User Interfaces (New York, NY, USA, 1998), IUI ’99, Association for Computing Machinery, p. 145–152.
  * (29) Liang, W., Yu, X., Alghofaili, R., Lang, Y., and Yu, L.-F. Scene-aware behavior synthesis for virtual pets in mixed reality.  In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (2021), pp. 1–12.
  * (30) Lin, D. C.-E., Germanidis, A., Valenzuela, C., Shi, Y., and Martelaro, N. Soundify: Matching sound effects to video.  In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology (2023), pp. 1–13.
  * (31) Lindlbauer, D., Feit, A. M., and Hilliges, O. Context-aware online adaptation of mixed reality interfaces.  In Proceedings of the 32nd annual ACM symposium on user interface software and technology (2019), pp. 147–160.
  * (32) Liu, H., Chen, Z., Yuan, Y., Mei, X., Liu, X., Mandic, D., Wang, W., and Plumbley, M. D. Audioldm: Text-to-audio generation with latent diffusion models.  arXiv preprint arXiv:2301.12503 (2023).
  * (33) Liu, S., and Manocha, D. Sound Synthesis, Propagation, and Rendering: A Survey, May 2021.  arXiv:2011.05538 [cs].
  * (34) Monteiro, K., Vatsal, R., Chulpongsatorn, N., Parnami, A., and Suzuki, R. Teachable reality: Prototyping tangible augmented reality with everyday objects by leveraging interactive machine teaching.  In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (2023), pp. 1–15.
  * (35) Nebeling, M., and Speicher, M. The Trouble with Augmented Reality/Virtual Reality Authoring Tools.  In 2018 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct) (Munich, Germany, Oct. 2018), IEEE, pp. 333–337.
  * (36) Oncescu, A.-M., Koepke, A., Henriques, J., and Akata, Z. Albanie, S. Audio retrieval with natural language queries.  In INTERSPEECH (2021).
  * (37) OpenAI.  Gpt-4 technical report, 2024.
  * (38) Qian, X., He, F., Hu, X., Wang, T., Ipsita, A., and Ramani, K. Scalar: Authoring semantically adaptive augmented reality experiences in virtual reality.  In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (2022), pp. 1–18.
  * (39) Radu, I., and Schneider, B. What can we learn from augmented reality (ar)? benefits and drawbacks of ar for inquiry-based learning of physics.  In Proceedings of the 2019 CHI conference on human factors in computing systems (2019), pp. 1–12.
  * (40) Raghuvanshi, N., and Lin, M. C. Interactive sound synthesis for large scale environments.  In Proceedings of the 2006 symposium on Interactive 3D graphics and games (New York, NY, USA, Mar. 2006), I3D ’06, Association for Computing Machinery, pp. 101–108.
  * (41) Rakkolainen, I., Farooq, A., Kangas, J., Hakulinen, J., Rantala, J., Turunen, M., and Raisamo, R. Technologies for multimodal interaction in extended reality—a scoping review.  Multimodal Technologies and Interaction 5, 12 (2021), 81.
  * (42) Ren, Z., Yeh, H., and Lin, M. C. Example-guided physically based modal sound synthesis.  ACM Transactions on Graphics 32, 1 (Feb. 2013), 1:1–1:16.
  * (43) Ribeiro, F., Florencio, D., Chou, P. A., and Zhang, Z. Auditory augmented reality: Object sonification for the visually impaired.  In 2012 IEEE 14th international workshop on multimedia signal processing (MMSP) (2012), IEEE, pp. 319–324.
  * (44) Ribeiro, F., Florêncio, D., Chou, P. A., and Zhang, Z. Auditory augmented reality: Object sonification for the visually impaired.  In 2012 IEEE 14th International Workshop on Multimedia Signal Processing (MMSP) (Sept. 2012), pp. 319–324.
  * (45) Roginska, A., and Geluso, P. Immersive Sound.  Focal Press, 2017.
  * (46) Roodaki, H., Navab, N., Eslami, A., Stapleton, C., and Navab, N. SonifEye: Sonification of Visual Information Using Physical Modeling Sound Synthesis.  IEEE Transactions on Visualization and Computer Graphics 23, 11 (Nov. 2017), 2366–2371.  Conference Name: IEEE Transactions on Visualization and Computer Graphics.
  * (47) Rumiński, D. An experimental study of spatial sound usefulness in searching and navigating through ar environments.  Virtual Reality 19, 3-4 (2015), 223–233.
  * (48) Serafin, S., Geronazzo, M., Erkut, C., Nilsson, N. C., and Nordahl, R. Sonic Interactions in Virtual Reality: State of the Art, Current Challenges, and Future Directions.  IEEE Computer Graphics and Applications 38, 2 (Mar. 2018), 31–43.  Conference Name: IEEE Computer Graphics and Applications.
  * (49) Sheffer, R., and Adi, Y. I hear your true colors: Image guided audio generation.  In ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (2023), pp. 1–5.
  * (50) Shen, Y., Song, K., Tan, X., Li, D., Lu, W., and Zhuang, Y. Hugginggpt: Solving ai tasks with chatgpt and its friends in huggingface.  arXiv preprint arXiv:2303.17580 (2023).
  * (51) Sırakaya, M., and Alsancak Sırakaya, D. Augmented reality in stem education: A systematic review.  Interactive Learning Environments 30, 8 (2022), 1556–1569.
  * (52) Tahara, T., Seno, T., Narita, G., and Ishikawa, T. Retargetable AR: Context-aware Augmented Reality in Indoor Scenes based on 3D Scene Graph.  In 2020 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct) (Nov. 2020), pp. 249–255.
  * (53) Technologies, U. Unity, 2023.  Accessed on Oct 9th, 2023.
  * (54) Unity Technologies.  Getting started with unity mars.  https://unity.com/products/mars/get-started, 2023.  Accessed: 2024-04-02.
  * (55) Upchurch, P., and Niu, R. A Dense Material Segmentation Dataset for Indoor and Outdoor Scene Parsing, July 2022.  arXiv:2207.10614 [cs].
  * (56) Wu, C., Yin, S., Qi, W., Wang, X., Tang, Z., and Duan, N. Visual chatgpt: Talking, drawing and editing with visual foundation models.  arXiv preprint arXiv:2303.04671 (2023).
  * (57) Wu, Y., Chen, K., Zhang, T., Hui, Y., Berg-Kirkpatrick, T., and Dubnov, S. Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation.  In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (2023), IEEE, pp. 1–5.
  * (58) Yang, D., Yu, J., Wang, H., Wang, W., Weng, C., Zou, Y., and Yu, D. Diffsound: Discrete diffusion model for text-to-sound generation.  IEEE/ACM Transactions on Audio, Speech, and Language Processing (2023).
  * (59) Zhou, Y., Wang, Z., Fang, C., Bui, T., and Berg, T. L. Visual to sound: Generating natural sound for videos in the wild.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (June 2018).
  * (60) Zhou, Z., Cheok, A. D., Qiu, Y., and Yang, X. The Role of 3-D Sound in Human Reaction and Performance in Augmented Reality Environments.  IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans 37, 2 (Mar. 2007), 262–272.  Conference Name: IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans.