# MIXING LEVELS — A ROCK MUSIC SPIRIT LEVEL APP

###### Abstract

To date, sonification apps are rare. Music apps on the other hand are widely
used. Smartphone users like to play with music. In this manuscript, we present
_Mixing Levels_ , a spirit level sonification based on music mixing. Tilting
the smartphone adjusts the volumes of $5$ musical instruments in a rock music
loop. Only when perfectly leveled, all instruments in the mix are well-
audible. The app is supposed to be useful and fun. Since the app appears like
a music mixing console, people have fun to interact with Mixing Levels, so
that learning the sonification is a playful experience.

##  1 Link to APK File

The compiled Mixing Levels app can be found on my GitHub account.

##  2 Introduction

As a systematic musicologist [1] I carried out research in the field of sound
field synthesis for music [2], music mixing [3], and music information
retrieval [4], while I worked at the Institute of Systematic Musicology at the
University of Hamburg. Then, my academic career brought me to the Bremen
Spatial Cognition Center at the University of Bremen, where my focus was
sonification for navigation [5] and other, mostly medical purposes [6]. A
focus on sonification was certainly a good fit, as I could bring in my
expertise in sound design, audio programming and digital signal processing,
psychoacoustics and auditory cognition in the development and evaluation of
audio technology, as well as empiric research. At the same time I learned a
lot about cognitive science, medical research and machine learning. But I
developed sonifications for dedicated purposes, in which the focus was to
achieve the highest possible density of perfectly interpretable information.
Musical elements can add a lot concerning acceptance, enjoyment and motivation
of sonification applications, but typically at the cost of information density
or interpretability. So it was off the table for the purposes of my research
projects at the Bremen Spatial Cognition Center. However, being a musicologist
by heart, I’ve always wanted to include music in my sonification research and
development. Mixing Levels is my first step in this direction.

Mixing Levels is a spirit level sonification based on the Sonic Tilt source
code, the open source version of Tiltification [7]. A major criticism about
Tiltification was that some users were not able to interpret the sound that
they had never heard before [8]. And they were not eager to learn it, because
they disliked the sound. The aim of Mixing Levels is to motivate these users
to learn to use a spirit level sonification. When you look at the app market,
you can easily see that there are a number of music making and mixing apps
around, like ROCKRELAY: Synthesizer TB 303 Bassline, soundandfeury:
Saucillator, HOME BAKE INSTRUMENTS: SYNTHESIZER TYPE ONE, Single Cell
Software: Caustic 3, McCormick IT: PdDroidParty and Irreducible Productions:
Csound for Android. This underlines that the market for music making apps is
much larger than the sonification apps market. In contrast to sonification
apps, music making apps have already been recognized as a suitable means for
education over a decade ago [9]. Many studies concluded that a musical
sonification can be informative, motivating, engaging, easy to learn, and
raise situation awareness [10, 11, 12].

In Mixing Levels, while users play around with the music, they learn to
interpret the sound. The sound is much more engaging and fun, music is a
familiar listening experience, and a reduced information density is fine for a
spirit level smartphone app. To be usable without headphones, the sonification
should stick to mono and work without the need for frequencies outside the
tweeter range between $200$ Hz and $3$ kHz.

##  3 Mixing Levels

The app name _Mixing Levels_ is a wordplay: You rotate your smartphone to
adjust the mixing levels (i.e., the volumes) of musical instruments, like a
music mixing engineer. At the same time, the music mix is supposed to help you
level your furniture.

Music recording and mixing has a lot to do with balancing of spectral,
spatial, dynamic and temporal aspects [13, 14, 15, 16, 17]. In Mixing Levels,
you can play music recording/mixing engineer. You need to level your phone in
order to balance a musical mix. The music is a loop of $16$ beats duration
from my song “Stay strong” that I produced in 2008.

The concept of the app is illustrated in Fig. 1. You can consider the
situation from the viewpoint of a recording engineer: Imagine your smartphone
display is a directive microphone, surrounded by musical instruments. On your
left, you’ve got a keyboard, on your right, you’ve got a piano. Both play the
same melody. Between you and the smartphone, you’ve got an electric guitar,
behind the smartphone, you’ve got a drum kit. Above the smartphone, you’ve got
a squeeky synthesizer, represented by flames. Tilting your smartphone in any
direction alters the gain, i.e., the volumes with which all instruments are
captured.

When you tilt your smartphone to the right, the piano is focused and will be
very loud. When you tilt it towards the left, the volume of the piano will
decrease gradually, while the volume of the keyboard increases. To balance the
volumes of the two, the smartphone needs to be leveled; neither tilted to the
left nor tilted to the right.

When you tilt your smartphone away from you, the drums will be focused and
sound very loud. When you gradually tilt it towards you, the volume of the
drums will decrease gradually, while the volume of the electric guitar
increases. To balance the volumes of the two, the smartphone needs to be
leveled; neither tilted towards you nor away from you.

Only when the smartphone is leveled along both dimensions, you can hear all
four musical instruments equally loud, and well as the blazing, squeeky
synthesizer.

You can also consider the task from the viewpoint of a mixing engineer. The
tilt angles control nothing but the gain factors of $5$ audio tracks,
according to the rule illustrated in Fig. 2: The gain factor for the piano is
represented as a green envelope. When the phone is tilted completely to the
left, the piano is muted. When tilting it gradually to the right, the gain
factor increases linearly. Near $0^{\circ}$, it reaches a plateau. When
tilting it even further to the right, the gain factor rises even further, so
that the piano dominates the mix. The gain factor of the keyboard is
represented in pink. When tilted completely to the left, the gain factor is at
max and the keyboard dominates the mix. While tilting it gradually to the
right, the gain factor decreases, until it reaches a plateau near $0^{\circ}$.
When tilted even further to the right, the gain factor decreases further,
until the keyboard track is muted.

The gain factor of the drum kit is represented as a dashed, red envelope. When
tilting the smartphone away from you, the gain factor of the drums is too
high, so the drums dominate the mix. Tilting it towards you reduces the gain
factor. Near $0^{\circ}$, a plateau is reached, and the drums have an adequate
volume. Tilting the display even further towards you, the gain factor reduces
further, until the drums become inaudible. The gain factor of the electric
guitar behaves in the same manner, but in the opposite direction. When the
smartphone is leveled towards you, the gain factor of the electric guitar is
at max and the guitar will overly dominate the whole mix. While tilting it
away from you, the gain factor will decrease gradually, reaching an adequate
plateau near $0^{\circ}$. When wilting the phone even further away from you,
the gain factor will reduce until the electric guitar becomes inaudible.

The volume of the squeeky synthesizer is not controlled continuously by the
tilt angle of the phone. Instead, you can imagine the squeeky synthesizer as
having a gate effect: Only when a certain threshold is surpassed, the
synthesizer is blazing with its full volume. Otherwise, the synthesizer is
switched off. The threshold is a tilt angle of $1^{\circ}$ along both
dimensions, represented as dashed, yellow square in the figure.

Figure 1: In Mixing Levels you can figuratively consider your mobile phone
display a camera. When tilted to the right, you capture the (green) piano,
when tilted to the left, you capture the (pink) keyboard, when tilted away
from you, you capture the (red) drums and when tilted towards you, you capture
the (blue) electric guitar. Only when perfectly leveled, you capture all
instruments equally, and a massive, squeeky (yellow) synthesizer is blazing.

Figure 2: In Mixing Levels, the tilt angles control the gains of musical
instruments. The green envelope represents the gain factor for the piano, the
red one for the keyboard, the blue one for the electric guitar and the pink
one for the drums. Only when (almost) perfectly leveled, an additional,
squeeky synthesizer (yellow) is added.

A screen capture of Mixing Levels can be found on my YouTube channel under
https://youtu.be/bs0xMiljAbs. Be aware

##  4 Discussion

Mixing Levels uses quite exciting, arousing music. This could be positively
shocking for young users, who are used to an overflow of visual and auditory
stimuli, e.g., from video games. Users will get the feeling that they are
being pushed in the deep end and have to learn how to orientate and make the
music work for them instead of being a distractor.

However, many people will want some time to take a breath and explore the
sound slowly, step-by-step. They cannot concentrate with such obtrusive music.
In everyday-life, people turn down the radio when they need to concentrate
[18]. And this makes sense, because in most occasions the sound is not the
source of information, but a potential distractor. In order to reach the
masses, calm music with an ambient character would be preferable. This may
also be the more appropriate choice for the task of leveling furniture. A
future version may start with a calm piece of music and contain a preset
function, in which people can choose their preferred style of music. This
would also be a nice feature in terms of UX-design, because a) people can
customize the app according to their taste and b) people get even more music
to explore and play around with.

## References

  * [1] R. Bader, Ed., _Springer Handbook of Systematic Musicology_ , ser. Springer Handbooks. Berlin, Heidelberg: Springer, 2018. [Online]. Available: https://doi.org/10.1007/978-3-662-55004-5
  * [2] T. Ziemer, _Psychoacoustic Music Sound Field Synthesis_ , ser. Current Research in Systematic Musicology. Cham: Springer, 2020, vol. 7. [Online]. Available: https://doi.org/10.1007/978-3-030-23033-3
  * [3] C. Stirnat and T. Ziemer, “Spaciousness in music: The tonmeister’s intention and the listener’s perception,” in _KLG 2017. klingt gut! 2017 – international Symposium on Sound_ , ser. EPiC Series in Technology, P. Kessling and T. G\"orne, Eds., vol. 1. EasyChair, 2019, pp. 42–51. [Online]. Available: https://easychair.org/publications/paper/q73J
  * [4] T. Ziemer, Y. Yu, and S. Tang, “Using psychoacoustic models for sound analysis in music,” in _Proceedings of the 8th Annual Meeting of the Forum on Information Retrieval Evaluation_ , ser. FIRE ’16. New York, NY, USA: ACM, 2016, pp. 1–7. [Online]. Available: http://doi.acm.org/10.1145/3015157.3015158
  * [5] T. Ziemer, N. Nuchprayoon, and H. Schultheis, “Psychoacoustic sonification as user interface for human-machine interaction,” _International Journal of Informatics Society_ , vol. 12, no. 1, pp. 3–16, 2020. [Online]. Available: http://www.infsoc.org/journal/vol12/12-1
  * [6] S. Schwarz and T. Ziemer, “A psychoacoustic sound design for pulse oximetry,” in _Proceedings of the 25th International Conference on Auditory Display (ICAD 2019)_ , Newcastle, June 2019. [Online]. Available: http://hdl.handle.net/1853/61504
  * [7] M. Asendorf, M. Kienzle, R. Ringe, F. Ahmadi, D. Bhowmik, J. Chen, K. Huynh, S. Kleinert, J. Kruesilp, Y. Lee, X. Wang, W. Luo, N. Jadid, A. Awadin, V. Raval, E. Schade, H. Jaman, K. Sharma, C. Weber, H. Winkler, and T. Ziemer, “Tiltification — an accessible app to popularize sonification,” in _Proc. 26th International Conference on Auditory Display (ICAD2021)_ , Virtual Conference, June 2021, pp. 184–191. [Online]. Available: http://hdl.handle.net/1853/66331
  * [8] T. Ziemer and N. M. Jadid, “Recommendations to develop, distribute and market sonification apps,” in _The 27th International Conference on Auditory Display (ICAD 2022)_ , Virtual Conference, June 2022. [Online]. Available: https://icad2022.icad.org/wp-content/uploads/2022/06/ICAD2022_3.pdf
  * [9] P. Gouzouasis and D. Bakan, “The future of music making and music education in a transformative digital world,” _The University of Melbourne refereed e-journal_ , vol. 2, no. 2, pp. 127–154, 2011.
  * [10] J. Middleton, J. Hakulinen, K. Tiitinen, J. Hella, T. Keskinen, P. Huuskonen, J. Linna, M. Turunen, M. Ziat, and R. Raisamo, “Sonification with musical characteristics: a path guided by user engagement,” in _24th International Conference on Auditory Display (ICAD2018)_ , 2018. [Online]. Available: https://doi.org/10.21785/icad2018.006
  * [11] G. Leplatre and S. A. Brewster, “An investigation of using music to provide navigation cues,” in _5th International Conference on Auditory Display (ICAD1998)_ , Glasgow, UK, Nov. 1998. [Online]. Available: http://hdl.handle.net/1853/50713
  * [12] H. Roodaki, K. Filippatos, A. Eslami, and N. Navab, “Surgical soundtracks: automatic acoustic augmentation of surgical procedures,” _International Journal of Computer Assisted Radiology and Surgery_ , vol. 13, pp. 1345–1355, 2018\.
  * [13] E. Hamidovic, _The Systematic Mixing Guide_. Melbourne: Systematic Productions, 2012.
  * [14] K. Greenebaum and R. Barzel, _Audio Anecdotes III: Tools, Tips, and Techniques for Digital Audio_ , ser. Ak Peters Series. Taylor & Francis, 2007.
  * [15] T. Ziemer, “Source width in music production. methods in stereo, ambisonics, and wave field synthesis,” in _Studies in Musical Acoustics and Psychoacoustics_ , A. Schneider, Ed. Cham: Springer International Publishing, 2017, pp. 299–340. [Online]. Available: https://link.springer.com/chapter/10.1007/978-3-319-47292-8_10
  * [16] C. Kaiser, _1001 Mixing Tipps_. Heidelberg et al.: mitp, 2012.
  * [17] C. Kaiser, _1001 Recording Tipps_. Heidelberg et al.: mitp, 2012.
  * [18] P. Sörqvist, O. Dahlström, T. Karlsson, and J. Rönnberg, “Concentration: The neural underpinnings of how cognitive load shields against distraction,” _Frontiers in Human Neuroscience_ , vol. 10, 2016. [Online]. Available: https://www.frontiersin.org/articles/10.3389/fnhum.2016.00221