# A Mapping Strategy for Interacting with Latent Audio Synthesis Using
Artistic Materials

Shuoyang Zheng<EMAIL_ADDRESS>0000-0002-5483-6028 Centre for
Digital Music, Queen Mary University of LondonUK ,  Anna Xambó Sedó
<EMAIL_ADDRESS>0000-0003-2333-6941 Centre for Digital Music, Queen
Mary University of LondonUK and  Nick Bryan-Kinns<EMAIL_ADDRESS>0000-0002-1382-2914 Creative Computing Institute, University of the Arts
LondonUK

(2024)

###### Abstract.

This paper presents a mapping strategy for interacting with the latent spaces
of generative AI models. Our approach involves using unsupervised feature
learning to encode a human control space and mapping it to an audio synthesis
model’s latent space. To demonstrate how this mapping strategy can turn high-
dimensional sensor data into control mechanisms of a deep generative model, we
present a proof-of-concept system that uses visual sketches to control an
audio synthesis model. We draw on emerging discourses in XAIxArts to discuss
how this approach can contribute to XAI in artistic and creative contexts, we
also discuss its current limitations and propose future research directions.

††copyright: acmlicensed††journalyear: 2024††conference: Explainable AI for
the Arts Workshop 2024; June 23, 2024; Chicago, IL, United States

##  1\. Introduction

Research in the ﬁeld of eXplainable AI for the Arts (XAIxArts) examines how AI
models could be made more understandable for creative activities (Bryan-Kinns,
2024). One approach to increasing the explainability of an AI model is to
provide real-time interaction and feedback (Bryan-Kinns et al., 2021). This
approach allows the user to learn how the model works and find ways to
interact with it by exploring how their actions affect the model’s output
(Murray-Browne and Tigas, 2021b). In practice, it involves mapping a human
control space to the parameter space of the AI model and providing real-time
audio feedback. For example, mapping a user’s spatial coordination in an
immersive environment to the latent space of a deep audio synthesis model so
that the user learns the effect of specific latent space axes by ‘walking
through’ the environment (Scurto and Postel, 2023).

The design of this human control space is an open question in the field of XAI
(Bryan-Kinns et al., 2023), especially in a creative and artistic context.
While typical approaches have explored parametric control mechanisms such as
sliders and 2D pads (Vigliensoni and Fiebrink, 2023; Bryan-Kinns et al.,
2021), it is important to consider other interaction mediums that may be more
expressive in creative and artistic contexts (Clemens, 2023). These
interaction mediums can be the artistic materials from the artists’
performance space (Bryan-Kinns, 2024), for example, the movements of dancers
(Murray-Browne and Tigas, 2021b), sketches (Lobbers and Fazekas, 2023), or
electromyography data of performers (Silva et al., 2020). However, materials
in an artistic performance space can be high-dimensional sensor data, which
are difficult to map to the AI model and use as a medium for interaction (Roma
et al., 2019). To tackle this, we propose a mapping strategy that encodes a
human performance space and maps it to the latent space of an AI model to help
users interact with the model’s generation process. We use a sketch-to-sound
controller we developed in a previous work (Zheng et al., pted) to demonstrate
how this mapping strategy can turn high-dimensional sensor data into control
mechanisms of an AI model.

##  2\. Mapping Strategy

Materials in an artistic performance space can be high-dimensional sensor data
such as sketches (Lobbers and Fazekas, 2023) or movements of dancers (Murray-
Browne and Tigas, 2021b). In order to map artistic materials to the latent
space of an AI model, a two-step approach is needed: (i) feature extraction
techniques are required to encode these data into representative features
(Roma et al., 2019), (ii) these representative features need to be
meaningfully mapped to the latent space of the AI model. Therefore, as shown
in Figure 1, we propose using latent mapping (Murray-Browne and Tigas, 2021b)
and Interactive Machine Learning (IML) (Fiebrink et al., 2009) to achieve
these two steps.

Figure 1. Proposed mapping strategy

Step 1 uses latent mapping (Murray-Browne and Tigas, 2021b), an unsupervised
feature learning approach, to learn representative features from a large
corpus of unlabelled data. A trained latent mapping model encodes high-
dimensional inputs into lower-dimensional latent representations, holding
representative features of the artist’s current movements, sketches, or other
artistic practices, which can be mapped to sound controls. Commonly used
unsupervised feature learning methods include Variational AutoEncoders
(Murray-Browne and Tigas, 2021b) and unsupervised dimensionality reduction
(Roma et al., 2019). In practice, training a latent mapping model requires the
artist to create a training dataset for their artistic materials. In other
words, they need to record a full set of possible outputs that they might
produce in the performance space (Murray-Browne and Tigas, 2021b). For
example, training a latent mapping model for a dancer may require recording
body skeletal data from the dancer of all their possible dance movements,
including intermediate transitions between two movements.

Step 2 uses Interactive Machine Learning (IML) to connect the extracted
features and the AI model’s parameters. IML allows users to construct mappings
between two spaces using a few personalised training examples (Fiebrink et
al., 2009). In this way, it creates a mapping between the human performance
space and the AI model’s output, allowing artists to interact with the AI
model using data from the performance space. In practice, the artist manually
defines a few paired data points in the performance space and the AI model’s
parameter space as training examples. Tools such as Wekinator (Fiebrink et
al., 2009) can be used to facilitate this process.

##  3\. Demonstration

To demonstrate how this mapping strategy works in practice, we build on our
previous work on a sketch-to-sound controller that encodes visual sketches
into sound controls. In this work, we use it as the latent mapping model to
control a latent audio synthesis model. Technical details for the sketch-to-
sound controller can be found in (Zheng et al., pted). We use RAVE (Caillon
and Esling, 2021), a variational autoencoder that synthesises audio from a
latent space in real-time. We use a RAVE model pre-trained on organ music
recordings (Intelligent Instruments Lab, 2023) that runs in a
Max4Live111https://maxforlive.com/library/ device. The pre-trained model has
16 latent space dimensions, and the sketch-to-sound model encodes sketches
into a 32-dimensional latent space. Therefore, the IML model takes a
32-dimensional input and maps it to a 16-dimensional output. The
communications between the sketch-to-sound model, the IML model, and the RAVE
model are achieved through open sound control
(OSC)222https://ccrma.stanford.edu/groups/osc/index.html.

Figure 2 shows two screenshots of the user interface. In each screenshot, the
left panel allows the user to interact with the RAVE audio synthesis model by
sketching on the canvas, while the right panel allows users to train the IML
model. When recording training examples for the IML model, users can start by
either manually adjusting the latent parameters or randomising the RAVE
model’s latent parameters using the “randomise” button, then clicking the
“record” button and drawing sketches that depict the current sound produced by
the RAVE model. After training the IML model, users can click the “run” button
and interact with the RAVE model by drawing sketches.

Figure 2. Two screenshots of user interfaces showing the sketch and the audio
synthesis model’s latent spaces. A full demonstration video can be viewed at
https://tinyurl.com/xaixarts24-mapping.

##  4\. Limitations and Future Work

This section draws on emerging discourses in XAIxArts (Bryan-Kinns et al.,
2023) to discuss the proposed mapping strategy in terms of the temporal and
cross-modal aspects. It also discusses the limitations and contributes with
future research directions.

First, the temporal aspect. Real-time interaction and feedback in an XAI model
can help artists discover and reuse unexpected phenomena for their artistic
expression (Bryan-Kinns, 2024). For example, in our demonstration video,
movements such as quick transitions from short lines to full circles trigger
interesting sounds. Users can reuse these sounds by memorising the sketch
trajectories that trigger them, but is there a way to explain these
trajectories to the users in the first place? Typical XAI approaches focus on
explaining a static state of the model parameters (Dhanorkar et al., 2021),
but explaining the trajectories of sketches focuses on explaining the
transition, or the interpolation from one state to another, which involves
taking account of the temporal dimension. This aspect of explanation
contributes to the embodied knowledge about the AI model because it tells
artists how to act with a system, which is an important component in artistic
and creative contexts (Murray-Browne and Tigas, 2021a).

Second, the cross-modal aspect. Investigating interactions that involve cross-
modalities, such as shape-sound associations, is useful to expand the
explanation mediums in creative and artistic contexts (Clemens, 2023). In
latent audio models, for example, future research can explore how human-
interpretable features in non-audio domains can be applied to audio latent
space. Typical XAI approaches aim to establish meaningful features in the
audio domain, and a number of these features already rely on cross-modal
metaphors to express the sonic characteristics, such as “brightness”,
“noisiness”, and so on (Lobbers and Fazekas, 2023). Our mapping strategy can
be used to study other cross-modal mappings that might be useful in XAI. For
example, future work on our sketch-to-sound system can explore programming and
animating sketches using interpretable features such as rotation, zoom, or
other affine transformations, and then use our mapping strategy to connect
these features with the audio latent space to discover meaningful shape-sound
associations that can be used to explain the audio synthesis model.

The motivation for the mapping strategy presented in this paper was to
encourage explorations on explanation mediums that are salient in the creative
and artistic contexts (Clemens, 2023). However, we only use sketch-to-sound to
demonstrate how the mapping strategy works. It would be necessary to explore
other forms of creative practices and go beyond sketch and sound to generalise
the discussion into wider creative and artistic contexts.

##  5\. Conclusion

Real-time interaction and feedback increase the explainability of an AI model
by supporting users in exploring how their actions affect the audio
generation. Following this direction, this paper proposed a mapping strategy
for interacting with latent audio synthesis models using data from an artistic
performance space. We have described how this mapping strategy works in
practice and demonstrated it with a system that controls an AI audio synthesis
model using visual sketches. We discussed that future work using this mapping
strategy should take account of the temporal and cross-modal XAI aspects, and
it is important to experiment with how this mapping strategy can be used in
other forms of creative practices in future work.

###### Acknowledgements.

Shuoyang Zheng is a research student at the UKRI Centre for Doctoral Training
in Artificial Intelligence and Music, supported by UK Research and Innovation
[grant number EP/S022694/1].

## References

  * (1)
  * Bryan-Kinns (2024) Nick Bryan-Kinns. 2024.  Reflections on Explainable AI for the Arts (XAIxArts).  _Interactions_ 31, 1 (Jan. 2024), 43–47.  https://doi.org/10.1145/3636457
  * Bryan-Kinns et al. (2021) Nick Bryan-Kinns, Berker Banar, Corey Ford, Courtney N Reed, Yixiao Zhang, Simon Colton, and Jack Armitage. 2021.  Exploring XAI for the Arts: Explaining latent space in generative music. In _1st Workshop on eXplainable AI approaches for debugging and diagnosis_.
  * Bryan-Kinns et al. (2023) Nick Bryan-Kinns, Corey Ford, Alan Chamberlain, Steven David Benford, Helen Kennedy, Zijin Li, Wu Qiong, Gus G. Xia, and Jeba Rezwana. 2023.  Explainable AI for the Arts: XAIxArts. In _Creativity and Cognition_. ACM, Virtual Event USA, 1–7.  https://doi.org/10.1145/3591196.3593517
  * Caillon and Esling (2021) Antoine Caillon and Philippe Esling. 2021.  RAVE: A variational autoencoder for fast and high-quality neural audio synthesis.  (2021).  https://arxiv.org/abs/2111.05011 arXiv: 2111.05011.
  * Clemens (2023) Michael Clemens. 2023.  Explaining the arts: Toward a framework for matching creative tasks with appropriate explanation mediums. In _Proceedings of The first international workshop on eXplainable AI for the Arts (XAIxArts)_.
  * Dhanorkar et al. (2021) Shipi Dhanorkar, Christine T. Wolf, Kun Qian, Anbang Xu, Lucian Popa, and Yunyao Li. 2021.  Who needs to know what, when?: Broadening the Explainable AI (XAI) Design Space by Looking at Explanations Across the AI Lifecycle. In _Proceedings of the 2021 ACM Designing Interactive Systems Conference_ _(DIS ’21)_. Association for Computing Machinery, New York, NY, USA, 1591–1602.  https://doi.org/10.1145/3461778.3462131
  * Fiebrink et al. (2009) Rebecca Fiebrink, Dan Trueman, and Perry R. Cook. 2009.  A meta-instrument for interactive, on-the-fly machine learning. In _Proceedings of the International Conference on New Interfaces for Musical Expression_. Zenodo, Pittsburgh, PA, United States, 280–285.  https://zenodo.org/record/1177513
  * Intelligent Instruments Lab (2023) Intelligent Instruments Lab. 2023.  rave-models (Revision ad15daf).  https://doi.org/10.57967/hf/1235
  * Lobbers and Fazekas (2023) Sebastian Lobbers and George Fazekas. 2023.  SketchSynth: A browser-based sketching interface for sound control.  _Proceedings of the International Conference on New Interfaces for Musical Expression_ (May 2023), 637–641.  http://nime.org/proceedings/2023/nime2023_95.pdf
  * Murray-Browne and Tigas (2021a) Tim Murray-Browne and Panagiotis Tigas. 2021a.  Emergent interfaces: Vague, complex, bespoke and embodied interaction between humans and computers.  _Applied Sciences_ 11, 18 (2021).  https://doi.org/10.3390/app11188531
  * Murray-Browne and Tigas (2021b) Tim Murray-Browne and Panagiotis Tigas. 2021b.  Latent mappings: Generating open-ended expressive mappings using variational autoencoders. In _International Conference on New Interfaces for Musical Expression_.  https://doi.org/10.21428/92fbeb44.9d4bcd4b
  * Roma et al. (2019) Gerard Roma, Owen Green, and Pierre Alexandre Tremblay. 2019.  Adaptive mapping of sound collections for data-driven musical interfaces. In _Proceedings of the International Conference on New Interfaces for Musical Expression_ , Marcelo Queiroz and Anna Xambó Sedó (Eds.). UFRGS, Porto Alegre, Brazil, 313–318.  https://doi.org/10.5281/zenodo.3672976 ISSN: 2220-4806.
  * Scurto and Postel (2023) Hugo Scurto and Ludmila Postel. 2023.  Soundwalking deep latent spaces. In _Proceedings of the 23rd International Conference on New Interfaces for Musical Expression (NIME’23)_. Mexico, Mexico.  https://hal.science/hal-04108997
  * Silva et al. (2020) Hugo Silva, Michael Zbyszynski, Atau Tanaka, and Federico Visi. 2020.  Interactive machine learning: Strategies for live performance using electromyography. (2020).  https://research.gold.ac.uk/id/eprint/28215/
  * Vigliensoni and Fiebrink (2023) Gabriel Vigliensoni and Rebecca Fiebrink. 2023.  Steering latent audio models through interactive machine learning. In _In Proceedings of the 14th International Conference on Computational Creativity_. Ontario, Canada.  https://computationalcreativity.net/iccc23/papers/ICCC-2023_paper_100.pdf
  * Zheng et al. (pted) Shuoyang Zheng, Bleiz M. Del Sette, Charalampos Saitis, Anna Xambó, and Nick Bryan-Kinns. Accepted.  Building sketch-to-sound mapping with unsupervised feature extraction and interactive machine learning.  _Proceedings of the International Conference on New Interfaces for Musical Expression_ (Accepted), Utrecht, Netherlands.