 aspects involved in VR applications (for an example questionnaire, see Tab. 1).
Table 1: Example of a short questionnaire for the assessment of an audiovisual scene derived from the studies on spatial audio quality inventory and that of presence and immersive tendency questionnaires. Subjects rate each perceptual quality using for instance a 5-point rating scale.
Describes the distinctness with which a sound source is perceived within or outside the head regardless of the distance (Lindau et al., 2014). / How compelling was your sense of externalization in the virtual environment?
In a permanently moving world (and its virtualization) one constantly has to deal with a combination of different parameters in always new and different weighting. In VR applications, the directions of the virtual sources must be shifted according to the head movement to the respectively desired horizontal and vertical axis. The detection and computational operationalization of the head movement simultaneously minimizes the risk of in-head sound localization and front-back confusion (externalization, Brimijoin et al., 2013). For the perception of moving sound sources in VE, not only temporal aspects of the sound source and its speed must be taken into account, but also other perceptual aspects. These are e.g. the influence of different or changing room acoustic conditions (reverberation, early reflections, distance to the sound source, etc.) or the influence of visual perception on the location of a sound source (cross-/multi-modal perception).
To produce a plausible spatial reproduction via headphones in VR, externalization is an important quality characteristic. Externalization is a measure of how inside or outside the head an auditory event is perceived (Lindau et al., 2014). The addition of reverberation alone does not provide externalization (Boyd et al., 2012), however, visual aids and familiarity with the signal can enhance externalization (Gil-Carvajal et al., 2016). For the evaluation of externalization Hendrickx et al. (2017) use a six-point scale to report externalization (from 0 "The source is at the center of my head" to 5 " The source is externalized and remote").
The psychoacoustic phenomenon front-back confusions occur via headphones when an audio event is perceived by the listener behind, even though the actual sound event was at the front or vice versa (Wightman and Kistler, 1999). In order to reduce front-back confusion, head movements are sufficient by 4° in the horizontal and 16° in the vertical direction (McAnally and Martin, 2014). Furthermore, visual dominance enhances sound localization while motion (Wallach, 1940). For the evaluation of front-back confusions, the rating scale of the front-back position is bipolar: it is either confused or not confused (Lindau et al., 2014).
The integration of visual impressions on auditory perception plays an essential role in investigations in VR. Hicks et al. (2004) investigate the effects of performance within audio-visual and audio-only conditions. Their study indicates that the presence of shared visual information enhances collaborative problem solving. Further, two VR experiments conducted by Po et al. (2005) suggest the presence or absence of certain visual cues, such as tracking cursors and asymmetric frames, can influence voice and gestural interaction for spatial target selection. Especially on designing synthetic expressive music instruments in VR, Mion and d'Incà (2006) showed by focusing on the expression via simple musical visual gestures of violin and flute performances, the enhancement of the audio in virtual reality applications. Liu and Yu (2015) created a method for real-time acoustic rendering which shows that improved computational efficiency and quality of the audio interpolation leads to more immersive virtual reality scenarios.
The audio reproduction becomes more realistic when a matching visual scene is presented, and the subject is in a congruent orientation (vestibular organ) and in a congruent state with the auditory scene (proprioception) (Burr and Alais, 2006). A prominent case of interaction between the various modalities is ventriloquism (Shelton and Searle, 1980; Warren et al., 1981). In the case of ventriloquism, the visual position of a sound source dominates over the auditory position. Whenever there is a discrepancy between auditory and visual localization, the auditory adjusts to visual localization (Knudsen, 1994). For audiovisual localization tests, Bertelson and Radeau (1981) observed that with a 7° difference between an auditory and visual stimulus in the horizontal plane, the auditory stimulus shifts 4° in the direction of the visual stimulus (Hendrickx et al., 2015). However, further studies have shown that it is not the visual that dominates the auditory, but perceptual acuity that determines which is more dominant by the two senses (Alais and Burr, 2004). Moreover, time (e.g. visual object is displayed before the auditory object and vice versa, Getzmann, 2007) and space (e.g. position of the visual object is not equal to the position of the auditory object) affect the perception of audiovisual or multisensory objects. These interactions have to be considered in investigations (Recanzone, 2009).
The study by Munhall et al. (2004) examines visual prosody in combination with speech intelligibility. Four different audiovisual male head animation versions (different kinematics of face and head movement) of 20 Japanese sentences were rated by 12 subjects in a speech-in-noise task. Subjects identified more syllables when there was a natural head movement in the animation than when it was eliminated or distorted. Another exciting effect on the influence of the visual on the auditory is the McGurk effect (McGurk and Macdonald, 1976). This audio-visual effect refers to the change in speech perception due to the movements of the mouth. The subjects saw lip movements to the syllable [ga], the acoustic stimulus, however, was [ba] and finally perceived was [da]. Colors also have a major impact on our auditory sense. Observations by Patsouras et al. (2002), for example, suggest that red trains are considered faster with the same road noise than trains of other colors.
A published study by Berger et al. (2018) investigates localization with general HRTFs in VR. The results show that the combination of a dynamic auditory stimulus with a spatio-temporally aligned visual counterpart improves the localization of sound sources with generic HRTFs. They indicate that generic HRTFs may be enough to enable good auditory source localization in VR. However, their study only took place at the horizontal level (individuality of HRTF less crucial, see section 2.3). It is therefore questionable whether their claim in the title "Generic HRTFs may be good enough in VR" with a study using a generic dataset alone is justified without comparison to individualized HRTFs.
In a study by Poirier-Quinot and Katz (2018), a VR shooter game was constructed in which participants conducted two sessions, using their best and worst HRTFs (selected from seven perceptually quality-weighted HRTFs according to Katz and Parseihian, 2012). Participants were instructed to shoot at the incoming targets, destroying as many as possible, as fast as possible, in the given time limit. The results indicate a significant performance improvement (speed and motion efficiency) for HRTFs with the best possible match.
When it comes to audio VR experience, not only pure numbers are interesting as in localization errors measured in degrees, but also seeing these attributes as perceptual qualities such as localizability (more difficult – easier), front-back position (confused / not confused), externalization (more internalized - more externalized) and tone color (darker – brighter). These parameters for the assessment of the audiovisual scene can be measured on a continuous scale with a questionnaire. Further perceptual qualities such as realism, immersion, presence and authenticity/plausibility are promising for future evaluation of spatial auditory experience in VR applications. Especially the perceptual quality realism correlate with individual vs. non-individual HRTFs, which has already been shown in some studies (e.g. Jenny and Reuter, 2020). The result of this study shows that most realistic simulation of sound sources in VR can be achieved by using individualized HRTFs, which leads to an improvement in terms of the following perceptual qualities: localizability, front-back position, externalization, tone color, and realism. For an overview of studies summarizing the research presented in this paper using HFRT in their research results, see Tab. 2.
Table 2: Overview of studies listed in this review categorized by the type of measurement (localization, VR, etc.) with individual and non-individual HRTFs. real free field with loudspeaker, virtual simulation with headphone. ind individual HRTF, non-ind non-individual HRTF, MIT KEMAR dummy head HRTF, DTF directional transfer function, BRIR binaural room impulse response. # number of subjects in the study, f female, n.s. not specified (no indication of gender).
"Does it sound realistic?" This question addresses important perceptual state related to the auditory space. How do we decide whether the mental representation of some aspects of our auditory environment is realistic or not? Coming from the field of creating VE (visual and/or auditory), terms like fidelity, immersion, presence, authenticity, plausibility, and realism have been coined in order to address these and similar questions. However, it seems difficult to distinguish this terminology not only for the layman, but also for specialized researchers. Do not all of these terms mean the same thing? In the following, these perceptual quality characteristics and their meaning depending on the different discipline are discussed.
The attribute fidelity in audio reproduction relates to sound quality. Already around the 1950s, the term Hi-Fi (for "high fidelity") was brought to the marketplace as a quality