Some research has examined the influence of past experience with interactive games on factors such as presence and CS. Knight and Arns (2006) identified an inverse association between an individual's experience playing video games and the level of CS experienced. Various studies report no relationship between presence and gameplay experience (Schuemie et al., 2005; Alsina-Jurnet and Gutiérrez-Maldonado, 2010; Ling et al., 2013). Others have found minimal evidence for an effect of video game experience on presence ratings or CS. Gamito et al. (2010) experimentally manipulated gameplay experience using a training procedure, and reported that increased gaming experience leads to improved presence (PQ) whereas CS (SSQ) was unaffected by training. In another study that employed a similar task, Gamito et al. (2008) found no relationship between measures of presence (PQ, ITQ) or CS (SSQ) and the previous gameplay experience of participants. At the same time, the authors found an increase in physiological markers of heart rate with increasing experience with video games. Those authors considered heart rate as a measure of anxiety, but we note that others have taken similar markers to indicate presence (e.g., Meehan et al., 2001, 2003; Wiederhold et al., 2001) and also CS (e.g., Kim et al., 2005; Dennison et al., 2016). Accordingly, one should be cautious when interpreting physiological markers purported to measure these factors given the relative paucity and inconsistency of data of this sort reported to date.

# Conclusion: Associated Variables

When taken together, the evidence obtained from the review above begins to clarify the type of relationship that exists between presence and CS:

The relationship between CS and presence can be understood if the associated variables described above are considered with respect to their effect on sensory mismatch. Immersiveness (sensory submersion) likely plays a key role here: Experimental manipulations that increase immersiveness tend to produce both CS and presence, because the compelling nature of stimuli in an immersive virtual space fosters a high perceptual weighting of cues to self-motion and spatial orientation, which enhances the impact of conflicts between expected and obtained sensory signals that are generated by the compelling stimulus (Prothero, 1998; Lombard et al., 2000; Prothero and Parker, 2003). Put differently, immersiveness enhances the magnitude of violated expectations. Thus, increasing field-of-view size, adding stereoscopy, or providing congruent multisensory information can increase both presence and CS. Given that immersiveness (which increases the weight of sensory conflicts, Prothero, 1998) can also lead to increased vection (which is inversely related to sensory conflicts, e.g., Weech and Troje, 2017)

it is unsurprising that research on the vection-presence-CS relationship has concluded that the link is extremely complex (Keshavarz et al., 2015; Palmisano et al., 2017).

On the other hand, within immersive conditions, individuals who experience high presence tend to experience low levels of CS. This relationship may be a result of differences in individual sensitivity to sensory conflicts: Higher sensitivity will result in high CS and low presence, whereas low sensitivity to cue conflicts will lead to low levels of CS and a heightened sense of presence. Individual differences in sensitivity to sensory conflicts have been documented, and there is some limited evidence that these differences relate to CS and presence (Viaud-Delmon et al., 2000, 2006). The advantage in terms of presence and CS observed for "gamers" may be attributed to the process of sensory reweighting that occurs during continuous exposure to cue conflicts (see Reason and Brand, 1975; Weech et al., 2018a). Indeed, a sex effect on presence and CS possibly relates to the superior ability of men to adapt to sensory conflicts compared with women (Viaud-Delmon et al., 1998). Further research on these individual differences will be required in order to test the proposition that variance in sensory conflict sensitivity underlies the experience of both presence and CS.

There is a significant challenge involved in identifying the factors that might mediate the link between presence and CS. Primarily this difficulty arises from the substantial relationships between the associated factors identified above, such as sex and gameplay experience, or vection and field-of-view. In addition to these known relationships, a number of other understudied variables may have significant associations with both presence and CS. For instance, the sense of embodiment is known to form a core aspect of presence (Kilteni et al., 2012), but the relationship between the embodiment of a virtual body and CS is currently unclear. Furthermore, questions remain about how prior experience with VR systems interacts with presence and CS; experiential factors are currently understudied due to the novelty of widely available VR technology. It is also evident that there is a lack of research that combines measurements of presence, CS, and other factors in high-powered studies. The collection of large datasets that encompass multiple individual factors (age, sex, personality type) with several behavioral response measures (objective and subjective measures of presence and CS) permits the use of a modeling approach that would enhance our understanding of the complex relationship between presence, CS, and other mediating factors (a similar approach was adopted for CS alone in Weech et al., 2018b). Through the execution of such studies in the future, further interactions will be uncovered between the associated factors outlined above.

Questions remain with regard to what questionnaires of presence and CS are truly measuring. Studies have identified relationships between CS and spatial presence, but no relationship between CS and immersive tendencies, which correlates highly with spatial presence (e.g., Ling et al., 2013). This raises the problem that the degree to which an individual's tendency to report feeling presence or CS may not determine their experience of either factor. This is an inherent issue in questionnaire studies; according to a meta-analysis, almost 50% of questionnaire studies documented effects of social desirability on their results (Van de Mortel, 2008). It is therefore important that future research takes into account the possibility that response bias modulates measures of factors like presence and CS. Several approaches could be adopted to achieve this, including pre-task questionnaires that assess social desirability (Crowne and Marlowe, 1960; Van de Mortel, 2008) or developing and using questionnaires according to principles of minimizing bias (Choi and Pak, 2005).

# GENERAL CONCLUSION AND FUTURE DIRECTIONS

Literature supports the idea that presence and CS are inversely related, and that the relationship is likely to be mediated by factors including vection, navigation control, and display factors. These factors can be unified in terms of their effect on sensory mismatch, which appears to drive presence and CS in opposite directions. This presents the possibility that interventions targeted at increasing presence could reduce CS, and vice versa. While the results obtained across studies are often discordant, with many sources reporting a positive relationship between presence and CS, these outcomes may be related to the fact that immersive displays are likely to generate both a compelling sense of "being there," as well as symptoms of physiological discomfort. Other noise sources that may contribute to variability in findings include problematic measurement techniques, or differences in the operational definition of the underlying factors among studies.

How can future experimentation best serve the advancement of our understanding of the presence-CS association? The issue of measurement validity must be a major focus of future studies, where the cross-validation of metrics should be undertaken in well-controlled paradigms. Improving the robustness of findings in this area may also require a careful consideration of factors that could play a role in response bias, such as social desirability. Although a limited number of high quality studies have collected large datasets related to human factors in VR, future experiments will need to combine these measures with a modeling approach that can help to interpret the structure of the relationship between these factors. Relatedly, there is a prevalent lack of statistical power in many of the studies reviewed here, and this limits the ability for the field to infer answers about variables that are so inherently noisy. Future studies will benefit from careful a priori considerations of effect sizes, which we have compiled here where available. An additional factor that may reduce the variability in findings across future studies is the natural emergence of standardized VR head-mounted hardware. Note, however, that the findings of studies using lower-immersive systems such as projection screens will still prove valuable, as these will help to identify the impact of immersiveness and vergenceaccommodation mismatch on CS and presence. One particular gap in our understanding revealed by the current review is how presence is affected when sensory mismatch is experimentally manipulated. Given the prospective modulatory role of sensory mismatch in the association between presence and CS, future studies will need to overcome the challenges in manipulating and assessing sensory mismatch in empirical research. Through a careful consideration of the literature critique provided here, we envisage that the next wave of studies on the presence-CS link will help to make major advances toward understanding this complex relationship.

# AUTHOR CONTRIBUTIONS

fpsyg-10-00158 January 31, 2019 Time: 18:41 # 15

SW, SK, and MB-C designed the concept of the article and conducted revisions to the manuscript. SW and SK

# REFERENCES

conducted the literature search. SW wrote the initial draft of the article.

# FUNDING

This research was supported by grants from Oculus Research and the Natural Sciences and Engineering Research Council of Canada (NSERC) [(Grant no. RGPIN-05435-2014)] awarded to MB-C. The funding sources had no input in the preparation of the manuscript.

environment. Hum. Factors Ergon. Manuf. Serv. Ind. 25, 523–533. doi: 10.1002/ hfm.20566

phobic and non-phobic participants in therapeutic virtual environments derived from computer games. Cyberpsychol. Behav. 6, 467–476. doi: 10.1089/<PHONE_NUMBER>69710497

Telepresence Theory, Measurement & Technology, eds M. Lombard, F. Biocca, J. Freeman, W. IJsselsteijn, and R. J. Schaevitz (Cham: Springer), 35–58.

**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2019 Weech, Kenny and Barnett-Cowan. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# How Can On-Road Hazard Perception and Anticipation Be Improved? Evidence From the Body

Mariaelena Tagliabue\*, Michela Sarlo and Evelyn Gianfranchi

Department of General Psychology, University of Padua, Padua, Italy

The present research is aimed at investigating processes associated with learning how to drive safely. We were particularly interested in implicit mechanisms related to the automatic processing system involved in decision making in risky situations (Slovic et al., 2007). The operation of this system is directly linked to experiential and emotional reactions and can be monitored by measuring psychophysiological variables, such as skin conductance responses (SCRs). We focused specifically on the generalization of previously acquired skills to new and never before encountered road scenarios. To that end, we compared the SCRs of two groups of participants engaged, respectively, in two distinctive modes of moped-riding training. The active group proceeded actively, via moped, through several simulated courses, whereas the passive group watched video of the courses performed by the former group and identified hazards. Results indicate that the active group not only demonstrated improved performance in the second session, which involved the same simulated courses, but also showed generalization to new scenes in the third session. Moreover, SCRs to risky scenes, although present in both groups, were detectable in a higher proportion in the active group, paralleling the degree of risk confronted as the training progressed. Finally, the anticipatory ability demonstrated previously (and replicated in the present study), which was evident in the repeated performance of a given scenario, did not seem to generalize to the new scenarios confronted in the last session.

Keywords: hazard perception, moped-riding simulator, learning generalization, implicit learning, skin conductance response

# INTRODUCTION

In the field of road safety research, several studies have gathered indirect and/or direct evidence that supports the idea of the crucial role played by hazard perception in predicting crash likelihood (Horswill, 2016a). As Horswill (2016b) noted, conducting hazard perception research is not feasible in actual on-road situations, as exposing humans to hazards and the consequent potential dangers for the purpose of research raises ethical and other related issues. For this reason, considerable efforts have been devoted to creating adequate tools for measuring this skill in safe contexts.

Two principal methods are currently employed to improve hazard perception among learners (i.e., unlicensed drivers) and novice drivers: one involves watching video clips during which the

#### Edited by:

Massimo Bergamasco, Sant'Anna School of Advanced Studies, Italy

#### Reviewed by:

Andrea Kleinsmith, University of Maryland, Baltimore County, United States Anna Granà, University of Palermo, Italy

\*Correspondence: Mariaelena Tagliabue<EMAIL_ADDRESS>
#### Specialty section:

This article was submitted to Human-Media Interaction, a section of the journal Frontiers in Psychology

Received: 21 August 2018 Accepted: 17 January 2019 Published: 01 February 2019

#### Citation:

Tagliabue M, Sarlo M and Gianfranchi E (2019) How Can On-Road Hazard Perception and Anticipation Be Improved? Evidence From the Body. Front. Psychol. 10:167. doi: 10.3389/fpsyg.2019.00167

learners must identify onscreen hazards and the other relies on engaging learners in virtual driving experiences via simulators that administer a variety of hazardous scenarios. The first method is a component of traditional licensing programs in such countries as England and Australia, and the second method has been a compulsory part of Japan's licensing program for years (Haworth et al., 2005).

Recently, studies aimed at investigating the efficacy of these training methods have demonstrated a positive correlation between a learner's increased engagement in the task, the efficacy of the training, and the likelihood that the improved ability will translate to actual on-road performance; the degree of engagement is itself increased through the requirements conveyed by instructions and the quality of the feedback delivered (Horswill, 2016b; Horswill et al., 2017). Torres et al. (2017) have provided a coherent account of their finding that contingent negative feedback (i.e., losing license points after unsafe decisions in risky scenarios), delivered in response to decisions about whether or not to brake after the presentation of static onroad scenes, yielded to faster and safer decisions. Moreover, Megías et al. (2017) showed that performance in a mopedriding simulator became safer (in terms of the number of accidents, average speed, average time exceeding the speed limit) among participants trained via a feedback learning task that delivered emotional feedback (i.e., pictures of real accidents) with negative valence. Specifically, the learning task consisted of deciding whether it was appropriate to brake, given a set of traffic images. Participants who received negative emotional feedback in 50% of the trials in which they decided not to brake upon being presented with a risky scenario (i.e., risky decisionmaking behavior) demonstrated safer behavior in the subsequent moped-riding simulator test.

In this regard, one important consideration is that when learning involves simulation driving, feedback is embedded within the task; to wit, the driver directly experiences the consequences of risky decisions in the form of the dangers incurred (i.e., crashes or near misses). The use of simulators intrinsically involves a degree of uncertainty with regard to the extent that such results could be generalized to an on-road context (de Winter et al., 2012; Rosenbloom and Eldror, 2014), given that definitive proof would require prolonged monitoring of very large samples and the fact that the likelihood of crash occurrence is relatively low throughout the general population (although regarded as too high to satisfy the safety standard requirements). Nonetheless, some indirect evidence already collected indicates that driving styles recorded via simulator resemble, to some extent, the corresponding on-road driving styles (Goode et al., 2013; Meuleners and Fraser, 2015; Branzi et al., 2017). In particular, the behavioral changes, observed during simulated driving tasks as training progresses, toward safer driving behaviors (i.e., reduction of the probability to incur a crash) are comparable to on-road behaviors, and learning acquired via simulated driving persists over time (Vidotto et al., 2011, 2015).

Moreover, the advantage of using driving simulators to deliver feedback resembling real-life consequences is intrinsically linked to the interactive nature of the virtual environment in the sense that the world changes in response to the driver's behavior (de Winter et al., 2012). Thus, we can assume that quality, degree of contingency, and valence of feedback are crucial factors contributing to the improvement of hazard perception. Consequently, simulated driving should be regarded as more effective at promoting a defensive driving style than passive forms of training because the feedback provided to the trainee is coherent and directly linked to his or her behavior. Further, it can be hypothesized that simulators are potentially more engaging from an emotional perspective, as the first-hand experience of a virtual accident might, in principle, be recognized as more emotionally intense than the simple imposition of a virtual penalty. Moreover, experiencing a car crash for which the driver is personally responsible might produce a more vivid experience than watching images of a crash that is caused by and affects other people.

The aforementioned dynamic characteristic of simulations is thereby related to another advantage yielded by this kind of technology. We refer to the extent to which simulations involve the potential exposure of the driver to hazards (de Winter et al., 2012). First, the simulated nature of this exposure limits the ethical issues that would otherwise plague such research. Consequently, it enables the reproduction of road conditions characterized by the highest statistical probability of accidents occurring, and in which the hazardous element develops "spontaneously" and directly via the specific way in which each driver behaves in each condition. Thus, the use of driving/riding simulators ensures both the standardization of experimental (or training) conditions and the modulation of the hazard degree, pursuant to the specific driving style of each learner.

Another feature that has emerged as a critically important variable is the emotional involvement associated with participation in hazard perception ability training. Currently, a consensus has been reached about the claim that human beings respond to risks in ways that are often not supported by rational rules. According to Slovic et al. (2007), decisions in risky situations are made via an automatic processing system that relies on emotion and experience, which is mostly irrational and more rapid than the controlled processing system.

The mechanism to which Slovic et al. (2007) explicitly referred is the one accounted for by Bechara et al. (1994) and Damasio (1994). By comparing the performances of healthy individuals and ventromedial prefrontal patients in a decision-making task [i.e., the Iowa Gambling Task (IGT)], Bechara et al. (1994) observed that healthy participants develop, over the course of the task, an anticipatory ability that prevents them from choosing from decks that, in previous trials, had yielded losses that outweighed their gains. This occurs via psychophysiological activation [measured through skin conductance responses (SCRs)] that function as signals to alert participants that they are approaching a deck of cards that they have previously experienced as disadvantageous. This ability cannot develop in patients with damage to the ventromedial prefrontal cortex who, consequently, show smaller gains (if not explicitly greater losses) by the conclusion of the task. Interestingly, decision-making ability, as measured by the IGT, seems to interact with other

personality variables that have demonstrably correlated with dangerous driving styles (Gianfranchi et al., 2017a); to wit, it has been shown that IGT performance and sensation-seeking behavior interact in contributing to simulated motorcycle riding style (Gianfranchi et al., 2017b).

Within this framework, a reasonable amount of evidence has been collected on the psychophysiological correlates of hazard perception and reactions in the context of driving. For instance, in one of the first studies, Helander (1978) showed that, in situations that required the use of brakes, participants demonstrated electrodermal responses that were interpreted as deriving from mental activity preceding muscular contraction and, consequently, brake activation.

In a more recent study, Kinnear et al. (2013) demonstrated that, when participants were shown video clips of hazardous road scenarios and asked to identify oncoming dangers, learners and novice drivers exhibited a smaller proportion of anticipatory SCRs than experienced drivers. Moreover, focusing on novice drivers, Tagliabue and Sarlo (2015) showed that employing a driving simulator ensures greater emotional involvement in the task, relative to traditional methods that consist exclusively of exposure to video clips (passive training), as demonstrated by a larger proportion of SCRs. Further, the administration of the same road-simulated scenarios twice, the first a week before the second, has been proven to lead to earlier SCRs (Tagliabue et al., 2017), suggesting improved anticipatory ability, in line with the somatic marker hypothesis (Damasio, 1994): When people encounter the same situation that previously led to an emotional reaction, they experience a reactivation of the same emotions, via their "bodily reactions," which provide them with an advance signal that alerts them to the oncoming risk.

Given the importance and potential implications of such results for driver training programs, it is worth considering two crucial issues arising from the aforementioned evidence. The first concerns the effectiveness of passive training in improving the anticipatory recognition of hazards. The second deals with the need to understand whether or not this anticipation generalizes to new and different scenarios. Both issues have important implications for the design of future programs for learners and novice drivers.

# THE STUDY

The present work is part of a project aimed at investigating the processes involved in hazard perception improvement induced by the Honda Riding Trainer (HRT), a moped simulator. The HRT has proven to be a useful tool for promoting a defensive riding style among novice teenaged riders (Vidotto et al., 2011) by training the attentional skills (Tagliabue et al., 2013) that presumably underlie successful hazard detection; further, the demonstrated improvement in riding style persists for over a year (Vidotto et al., 2015). The HRT has been used to investigate a variety of the processes that underpin driving abilities, such as mental workload during driving, cognitive resources needed to respond to in-vehicle warning systems, and patterns of gaze exploration (Di Stasi et al., 2009, 2010, 2011). Starting from these results, we wanted to investigate the processes underlying hazard perception that could feasibly account for the improvements observed.

As noted above, earlier studies indicated that participants who undertook active training in the form of simulated moped-riding courses demonstrated a greater SCR percentage than participants who engaged in passive training by identifying hazards on these same courses upon request. Further, by comparing the SCRs of active participants who tackled the same courses in the HRT in two sessions (1 week apart) it was shown that SCR onset decreased during the second session, indicating that the first experience negotiating the hazardous scenarios enabled the riders to recognize these now-familiar hazards some 3 m of (virtual) road before (Tagliabue and Sarlo, 2015; Tagliabue et al., 2017).

Three central questions remain open, and these revolve around whether: (a) the anticipatory response does, in fact, generalize to new hazardous conditions; (b) the improvement in anticipatory ability related to hazard detection and reaction differs on the basis of whether the training is active or is passive; and (c) the overall profile of the performance, besides the probability that a crash will occur, is impacted by the active training involving the simulator or the passive training consisting of viewing video clips of hazardous scenarios.

Concerning the third point, in the above-mentioned studies that employed the HRT, improvement in performance was essentially measured by calculating the percentage of accidents. However, another crucial aspect of road safety involves learning to drive in a way that facilitates averting hazard development. To wit, learners may demonstrate a lower percentage of crashes either because they have learned to execute certain last-minute maneuvers designed to avoid impending collisions or because they have learned to drive in a generally safer way. This issue can be addressed by analyzing the level of potential hazards that develop during active training. Indeed, the HRT simulator facilitates such an analysis by providing a score for each potentially hazardous scene based on the degree of hazard development: when the rider behaves in a way that is totally safe, the scene receives a score of A; when the rider relies on mildly unsafe maneuvers (e.g., moderate violations of the speed limit, slightly sudden braking, stopping with insufficient headway), the scene receives a B score; when the rider executes seriously unsafe maneuvers (e.g., strong sudden braking that results in a concrete risk of losing control of the vehicle, stopping dangerously close to the preceding vehicle), the scene is assigned a score of C, generally reflecting conditions that resemble near-misses; finally, a score D corresponds to scenes in which an accident actually occurs.

By analyzing how the scores collected during training reflect a trend of behavioral change toward generally safer behavior, it may be possible to acquire further information on the mechanisms underlying the learning process.

To that end, in the present work, participants in the active group of the study conducted by Tagliabue et al. (2017) were assigned to complete a third training session consisting of the administration of six new courses to investigate issues related to the generalization of learning (aim a). A new group of matched participants (passive group) was recruited, and they engaged in

training by watching the scenes attempted by the active group, and identifying hazards (aim b).

Both behavioral and SCR data were collected to measure how participants learned to detect and anticipate hazards. We formulated three hypotheses, as follows:

# Methods

#### Participants

Thirty-eight undergraduates at the University of Padua were included in the present study. Data from one male participant of the control group were excluded from analyses due to electrode malfunction during skin conductance recording in the third session. Consequently, the data from the matched participant of the experimental group were also eliminated. Thus, the final sample included 36 undergraduates (18 females and 18 males; mean age 19.47 years; range 18–24 years). The experimental group was the same as used in the study conducted by Tagliabue et al. (2017), with the inclusion of three new participants to attain the sample size of 18 participants. The other 18 participants, assigned to the control (passive) group, were all new to this set of studies. All participants were novice drivers/riders; they held a driver's license for no longer than 3 years (range 5–36 months; mean 9.8 months). Nine students had a riding license, but reported road exposures under 5,000 km. The experimental and control groups were balanced for age (mean age = 19.88 and 19.05 years, respectively) and gender (nine males and nine females in each group).

The procedure included three experimental sessions. The task assigned to the experimental group was to ride a motorcycle simulator along some virtual courses. Participants in the control (passive) group were asked to detect hazards while they watched videos of the experimental (active) participants riding the simulators through the virtual courses.

All participants had normal or corrected-to-normal vision. They were paid €39 for their participation. The study was approved by the Ethical Committee for the Psychological Research of the University of Padua.

### Apparatus and Stimuli

The HRT is a two-wheeled-vehicle riding simulator that consists of a Pentium 4 PC with a Windows XP operating system (see **Figure 1** for an image of the HRT and examples of risky scenes).

The PC is placed on a base connected to an LCD monitor (1,024 × 768 resolution) displaying the virtual environment and to a chassis equipped with moped-like controls. For our procedure, a second screen was placed on a table behind participants who were seated on a moped-like seat approximately 80 cm from the HRT monitor. Two speakers, in addition to reproducing the acoustic effects of the engine and traffic noise, provided instructions to the active participants in the experimental group about the path they had to follow. Participants rode along the virtual courses using the moped-like controls, with a transmission set to automatic.

The same five courses, which took place on secondary roads, were employed in the first two sessions, while six courses, which took place on main roads, were administered in the last session. Each course included seven or eight risky scenes (depending on the course), for a total of 39 scenes in each of the first two sessions and 48 in the last one. The scenes represented hazardous road situations based on a European report classifying the most common motorcycle accident scenarios (Motorcycle Accidents In Depth Study [Maids] (2004)).

### Skin Conductance Recording

Skin conductance was recorded with two Ag/AgCl electrodes filled with a K-Y lubricating jelly placed on the left foot, over the abductor hallucis muscle, in a position adjacent to the sole of the foot and midway between the base of the hallucis and a point beneath the ankle. This electrode placement conforms to the international guidelines (Boucsein et al., 2012) indicated in situations where participants need to use their hands for the task itself.

A Grass Model SCA1 skin conductance coupler provided a 0.5-V constant voltage across electrodes. The signal was amplified and filtered with a 10-Hz low-pass filter using a Grass CP122 AC/DC Strain Gage amplifier (Grass Instrument Co., W.

FIGURE 1 | The HRT simulator used in the present study (top-left panel) and three examples of risky scenes.

Warwick, RI, United States). The amplifier—placed adjacent to the second screen, reproducing the output of the HRT monitor returned a display of the ongoing values of the electrodermal activity. A video camera was employed to simultaneously record the electrodermal activity (the values on the amplifier display) and the riding performance (second monitor).

# Procedure

For both groups, the entire procedure lasted for three sessions scheduled 1 week apart. Each session lasted approximately 45–60 min. At the beginning of the first session, after signing informed consent forms, each participant completed a questionnaire involving data related to their age and their driving and riding experience. Then, electrodes for skin conductance recording were attached.

With regard to the experimental group, the participants were instructed on how to use the HRT controls. Their task consisted of riding along the virtual courses, following the audibly vocalized advice, respecting the traffic rules, and avoiding crashes. The instructions also explained the importance of trying to avoid moving their feet during the task. A practice course (3 min in length, with no other road users in the virtual environment) was administered to enable participants to familiarize themselves with both the virtual environment and the HRT controls. All participants in the experimental group attempted the same five courses in the first two sessions, confronting a total of 39 potentially hazardous scenes in each session. In the last session, six new courses were administered, with a total of 48 scenes. Overall, each participant faced 126 hazardous scenes. As in the previous studies (Tagliabue and Sarlo, 2015; Tagliabue et al., 2017), the sequence of the courses was fixed for each session, proceeding from the easiest to the most difficult. The degree of difficulty was derived from the studies conducted by Miceli et al. (2008) and Settanni (2008).

The control group was asked to watch a simulation of some courses (five in the first two sessions and six in the third one) undertaken by an anonymous HRT rider and identify hazards. Whenever a passive participant detected a hazard, he or she pressed a button on the handlebar of the simulator. This detection task had the purpose of ensuring constant attention was paid to the video to avert the possibility of distraction. Given the purpose of the task, there was no need to record the responses. Each participant in the control group, matched to a same-gender participant in the experimental group, watched the replay of the performance of his or her paired participant in the corresponding sessions. At the end of each course, a 3-min rest was assigned to both the experimental and the control group to allow skin conductance to return to the baseline level.

# Data Reduction and Coding

The coding procedure was based on the videos recorded by the camera (in which the electrodermal values displayed by the amplifier and the performance on the HRT were synchronized) and on the .csv files, provided by the simulator, that collected all the variables linked to the riding performance, with a sampling rate of 30 Hz. The electrodermal activity values were coded at this same sampling rate via analysis of the videos obtained for each participant in each session. In this way, each point of the physiological signal was matched with the behavioral variables provided in the .csv files. Among these variables, the HRT also provided an evaluation of participants' riding safety for each scene. Possible scores ranged from A to D, depending on the distance between rider and collision, with A signifying that the participant's behavior was safe enough to prevent any collisions, B indicating a slight increase in the risk of crash, C corresponding to a near-miss, and D representing to an actual crash. Thus, each performed scene received a score that represented its particular level of risk, depending on the rider's behavior.

As in previous works (Tagliabue and Sarlo, 2015; Tagliabue et al., 2017), for each of the hazardous scenes, we identified a clue (i.e., a point on the path, in terms of x/y coordinates, after which hazard development began). As a result, we focused on two temporal windows: a baseline pre-clue window of 5 s and a 10-s post-clue window in which the hazard developed. Note here each participant was provided with the same clues, and this obtained for both the experimental and the control group.

As the participants in the experimental group could influence the development of the hazardous scenes via their riding behavior, the detection of the SCRs' onset was not feasible in terms of timing. For instance, with a lower riding speed, approaching the hazard required more time. Therefore, the SCRs' onset might appear to happen later because of a difference in the time taken to approach the hazard. As such, on the basis of the methodology employed in previous works (Tagliabue and Sarlo, 2015; Tagliabue et al., 2017), the mean level of the electrodermal activity in the baseline window was computed. Then, an SCR was detected as the first increase in amplitude of at least 0.05 µmho (Boucsein et al., 2012; Kinnear et al., 2013; Tagliabue and Sarlo, 2015; Tagliabue et al., 2017) in the post-clue window, with respect to the baseline. The time of SCR onset was then converted into its corresponding position on the path, in terms of the absolute value of x or y coordinates, depending on the dimension along which the participant was moving. Note that each individual change in the coordinates' value corresponds to a change of approximately 1 m in the virtual environment. The onset was therefore computed in terms of spatial distance from the hazard: an anticipation in the SCR onset corresponded to a greater distance from the hazard. The coding procedure was the same for both groups in every session. Indeed, each participant in the control group watched the performance of his or her matched experimental participant, thereby being exposed to the same scenes, with the same x/y coordinates.

# Design

To investigate the effectiveness of the three-session training, we conducted an ANOVA on the dependent variable accident's percentage of participants who actively rode along the virtual courses (active group) with Session as a within-participants factor (three levels).

Moreover, a deeper analysis of the participants' performance was conducted via a MANOVA on the percentage of scores (A, B, C, or D) attributed to each performed scene on the basis of the degree of risk generated by the participants' behavior. Again, Session was the within-participants three-level factor.

To investigate whether psychophysiological responses paralleled improvements in riding performance, and to compare psychophysiological reactivity of the active group to that of the passive group, two ANOVAs were carried out on the dependent variable percentage of SCRs. The first was conducted on the overall SCR percentages (independently from the degree of risk of the scenes) and aimed at comparing the present results with the previous one attained by Tagliabue et al. (2017), with Group (active vs. passive) as a between-participants factor and Session (three levels) as a within-participants factor.

In addition, a further ANOVA with Group as a two-level between-participants factor and Session (three levels) and Risk degree (four levels: A, B, C, and D) as the two within-participants was run to provide more information via the fine-grained analysis of the pattern of psychophysiological changes with reference to the risk degree.

Skin conductance response percentages were calculated for each kind of scene, depending on the degree of risk, as the proportion of SCRs detected over the total scene in each risk degree category. In courses in which one kind of scene (A, B, C, or D) did not occur, depending on the participant performance in the active group, the missing data were replaced by the mean for the same risk category. This happened for one pair of participants for A scenes and five pairs of participants for D scenes.

Last, to test the impact of the three-session training on the anticipatory ability, we analyzed the onset of dependent variable SCR via an ANOVA with Group as a between-participants factor (active vs. passive) and Session (three levels) and Risk degree (four levels: A, B, C, and D) as within-participants factors.

Post hoc analyses using Bonferroni's correction were conducted with α set at 0.05.

# Results

In the ANOVA on the percentage of accidents of the active group, the factor Session attained significance with F(2,34) = 107.46, p < 0.001 (η<sup>p</sup> <sup>2</sup> = 0.86). Participants had 28% of accidents in the first session, 16% in the second session, and 4% in the third session. Post hoc tests showed that all comparisons were significant, thereby confirming that performance improved, both in the second session (which administered the same courses as the first session), and in the third session, when participants had to confront new courses.

In the MANOVA on the percentage of scene's scores, the factor Session attained significance at the multivariate level, Wilks' λ = 0.027, F(6,12) = 71.64, p < 0.001 (η<sup>p</sup> <sup>2</sup> = 0.97). At the univariate level, the factor Session was significant for each score, with F(2,34) = 99.85, p < 0.001 (η<sup>p</sup> <sup>2</sup> = 0.86) for the A score, F(2,34) = 5.01, p < 0.05 (η<sup>p</sup> <sup>2</sup> = 0.23) for the B score, F(2,34) = 37.26, p < 0.001 (η<sup>p</sup> <sup>2</sup> = 0.69) for the C score, and F(2,34) = 103.08, p < 0.001 (η<sup>p</sup> <sup>2</sup> = 0.86) for the D score. The post hoc tests revealed that the percentage of A scores (i.e., totally safe scenes/performance) significantly increased between the first two sessions (15 vs. 26%) and from the second to the third session (47%). The percentage of D scores (i.e., accidents) decreased (28, 16, and 4% for the three sessions, respectively). With regard to B and C scores, no differences were found in the first two sessions, but in the third session, the percentage of both scores significantly differed (see **Figure 2**).

These results testify to the acquisition of learning as the training progressed. In the first session, participants drove in a way that generated a given degree of risk, as evidenced by the lower percentage of A scores relative to B and, even more clearly, to C and D scores that, it is worth noting, were attributable to scenes characterized by the development of a large amount of risk, or even the occurrence of an accident. The improvement in performance in the second session, which required participants to confront the same scenes, is demonstrated by a decrease in crashes. It can reasonably be considered that a certain number of scenes that had previously received a D score received a C or B score for the second session: the gradual modification of performance toward a notably safer level might be the reason why B and C scores appear to remain stable (as D scores become C scores, and C scores become B scores). Finally, this gradual improvement resulted in an unambiguous increase in the proportion of safe performances, as indicated by the enhanced percentage of A (safe performance) and B (low-risk performance) scores for the third session, in which participants faced six new (never before encountered) scenes. The effect of learning was even more apparent, given the significant concomitant reduction of C and D scores.

In the first of the ANOVAs on the SCR percentages, the factor Group and the factor Session attained significance with F(1,34) = 17.60, p < 0.001, (η<sup>p</sup> <sup>2</sup> = 0.34), and F(2,68) = 7.47, p < 0.01, (η<sup>p</sup> <sup>2</sup> = 0.18), respectively. Participants from the active group exhibited a higher percentage of SCRs, relative to the passive group (52 vs. 27%), and the SCR percentages decreased from 47%, for the first session to 39% for the second session and, finally, to 34% for the third session. The post hoc tests indicated that the SCR percentage for the first session was significantly higher than the subsequent percentages for the last two sessions.

In the second ANOVA on the SCR percentages, the factors Group and Risk degree attained significance with F(1,34) = 20.01, p < 0.001 (η<sup>p</sup> <sup>2</sup> = 0.37) and F(3,102) = 53.22, p < 0.001 (η<sup>p</sup> <sup>2</sup> = 0.61), respectively. In addition to the significant difference between the two groups, SCR percentages increased as the scenes became

increasingly risky (32% in A scenes, 35% in B scenes, 46% in C scenes, and 60% in D scenes). The post hoc tests revealed that the SCR percentages in A and B scenes (safe or low-risk level) did not differ, but both were significantly different from SCR percentages in C and D scenes, which were characterized by a greater degree of risk development or the occurrence of an accident. The SCR percentages in C and D scenes were also significantly different from one another.

Moreover, the Group X Risk degree and the Session X Risk degree interactions attained significance with F(3,102) = 6.01, p < 0.01 (η<sup>p</sup> <sup>2</sup> = 0.15) and F(6,204) = 2.85, p < 0.05 (η<sup>p</sup> <sup>2</sup> = 0.08), respectively. Concerning the Group X Risk degree interaction, as is evident in **Figure 3**, the post hoc test confirmed the significant difference between groups in each scene category. Moreover, in the active group, the SCR percentages increased in both C and D scenes, indicating that the psychophysiological reactivity paralleled the analogous rise in the degree of risk. Dissimilarly, in the passive group, a significant increase in SCR percentage was present only in cases of accident occurrence (relative to all other risk degrees).

As far as the Session X Risk degree interaction is concerned (see **Figure 4**), in the first session, A and B scenes elicited similar SCR percentages but were significantly different from C and D scenes, whereas SCR percentages of C and D scenes did not differ. In the second and third session, SCR percentages were higher in C than in B scenes and higher in D than in C scenes, as attested by the post hoc tests. Overall, this interaction suggests a better ability to discriminate the different degree of risk as the training progresses.

In the last ANOVA on the dependent variable SCR onset, the only main source of variance that reached significance was the factor Session, with F(2,68) = 11.94, p < 0.001 (η<sup>p</sup> <sup>2</sup> = 0.26). None of the interactions reached significance. The post hoc tests showed that the onset in the second session (10 m) was different from the SCR onset in both the first (14 m) and third (16 m) sessions. The SCR onset of the first and third sessions (new scenes, never seen before) did not differ significantly. In other words, the anticipatory ability developed in the second

session, when the same scenes had to be faced (as previously demonstrated in Tagliabue et al., 2017), disappeared in the third session when new, potentially risky scenes were performed or viewed by participants.

# Discussion

First, the results of the present paper confirm previous results that show performance improvement as training progresses. The fact that accident percentage decreases in the second session might be due to contextual learning, as the same course was administered in the first two sessions. However, the generalization of the learning acquired, at least at the behavioral level, has been demonstrated by the additional improvement recorded in the third session, in which six new courses were confronted. This improvement is substantiated not only by the reduction in accidents, but also by the overall improvement in performance. Participants who actively rode the moped simulator demonstrated safer behavior in the third session, as scenes in which they rode dangerously and incurred an accident decreased in the third session.

From a psychophysiological perspective, in accordance with previous reasoning, SCRs to potentially approaching hazards should indicate an implicit mechanism responsible for risk detection, in line with the dual processing system articulated by Slovic et al. (2007). Thus, one might wonder why SCR percentages should decrease, given that as the training progresses participants should have learned to react to hazards in a more effective way. Note that a reduction in SCR percentages between the first and second sessions was already observed in Tagliabue et al. (2017). The authors explained this result considering that, as the training develops, participants learn to drive safer, yielding a reduction in the number of near misses and accidents. In this case, there would be less need for the implicit system to react; thus, SCR percentages should decrease. In the present work, a third session was added to further test the hypothesis that the improvement derived from a safer riding style leads to a reduction of the number of hazards to detect and, consequently, of the amount of psychophysiological responses. The present data

favor this explanation in that, in the third session (in which C and D scenes—i.e., near misses and accidents—decreased), the SCR percentage was lower than in the first session.

Moreover, these data replicate the results indicating greater emotional involvement in the active group, which showed higher SCR percentages than the passive group, thereby confirming the results of a study conducted by Tagliabue and Sarlo (2015) using a totally new sample.

The fine-grained analysis of the changes in SCR percentages based on the degree of risk involved in the scenes indicate that differences between the two groups are further evident in the modulation of psychophysiological reactivity pursuant to the risk degree (see **Figure 3**): unlike the active group, the enhancement of the SCR percentages in the passive group obtained only if an accident occurred may indicate a failure by the passive group to discriminate among different levels of hazard. This is a particularly compelling result as, while driving, the correct "categorization" of the risk degree might facilitate the selection of the most appropriate response.

Moreover, the results illustrated in **Figure 4** indicate that accurate discrimination between no- or low-risk scenes, demonstrated by the psychophysiological reactivity, emerged in the second session and was maintained through the third session via transferring acquired learning to the new scenes, thereby providing partial evidence of generalization, at least in the simulated environment.

By contrast, with regard to anticipatory ability, our data did not confirm the generalization of this ability to scenes not yet encountered. Albeit negative, this result, if confirmed, provides important information about the requirements for road safety training, as it highlights the importance of extensive training aimed at exposing novice drivers and riders to as many different hazards as possible under the safer conditions of training (simulator or video clip viewing) to increase the probability that they will recognize such hazards in advance once they are actually on the road.

To summarize, active group participants showed performance improvement, and their learning seemed to generalize to new scenes, as they behaved more safely in both the second and third sessions. Moreover, they learned to discriminate among different degrees of risk via their implicit system (Slovic et al., 2007) and to generalize this achievement to the third session. By contrast, the anticipatory ability, in terms of SCR onset, did not appear to generalize to scenes not previously encountered. Results related to the passive group confirm previous results demonstrating that passive training involves the implicit system of hazard detection and recognition to a lesser extent.

# CONCLUSION

The main findings of the present research consist of demonstrating the following: (a) the experience of adverse consequence in simulated road scenarios yields an improvement in the ability to recognize the risk earlier when it is faced anew; (b) the psychophysiological correlates of this ability indicate that simulation is more effective than passive viewing of risky video clips; and (c) this anticipatory skill develops pursuant to prior experience, as predicted by the dual model of decision making (Slovic et al., 2007) and the somatic marker hypothesis of Damasio (1994).

The present results indicate a need to persist in attempts to develop training modules that permit exposure to the largest possible sample of road hazards before future motorcyclists actually take to the road to render them more capable of safely confronting each potential road risk. In this light, every effort to collect statistics, to map the greatest possible variations of the dynamics associated with the most common circumstances in which crashes occur and the attempt to replicate such dynamics via the simulators, must be firmly supported.

As noted, the hazard scenarios employed in the present research are drawn from the Motorcycle Accidents In Depth Study [Maids] (2004) accident statistics that include 921 situations representing a large amount of recorded motorcycle crashes across Europe. One potentially promising extension of this line of research could be the creation, for each of these prototypical conditions, of a certain number of variants (similar scenes with small differences), to induce learners and novice drivers to generalize, as much as possible, their acquired learning to new and different, but similar, scenes. To wit, the development of learning programs that enhance the probability of transferring the same anticipatory reactions to scenes not yet experienced by promoting generalization to several broader clusters of hazards could represent a further step toward the prevention of road accidents.

Given that experiencing an accident firsthand (albeit virtually) is more emotionally involving than simply viewing someone else's accident, the greater impact of simulation relative to video clip viewing is in line with the evidence provided by studies in the field of affect heuristics that indicate that, the more vivid and affectladen the scenarios, the more effective they are in influencing risk perception (Slovic et al., 2007).

The principal limitation of the present research is related to the generalizability of the observed effects to real on-road conditions. The use of simulators for driving/riding assessment and training is spreading in several countries. However, the benefits and disadvantages of this strategy remain controversial. Rosenbloom and Eldror (2014), for instance, did not find an overall improvement in on-road performance of newly licensed drivers trained with a driving simulator compared to novice drivers who received only the standard driving lessons. In fact, the former group showed a worsening in safe driving intention, probably due to overconfidence, and a slight reduction of headway events not associated with a reduction of their severity. The same group also showed an increase in the amount of brake pressure, which is interpreted by the authors as reflecting a less safe driving style. Goode et al. (2013) argued in favor of a certain amount of effectiveness associated with this technology when it is aimed at refining higher order cognitive skills, such as visual scanning and hazard perception; de Winter et al. (2012) cited the advantages and disadvantages of simulators and stressed the need for deeper investigation. More recently, evidence of correspondence between simulator and on-road

driving parameters was reported by Branzi et al. (2017) by comparing driving speeds in on-road and simulated driving.

Despite the fact that the advantages and disadvantages of employing simulators are matters of ongoing debate (de Winter et al., 2012; Rosenbloom and Eldror, 2014), any reasonable enterprise aimed at improving the abilities of road users to prevent the development of risky situations should not be overlooked, especially as the available data seem to show that, over the last 5-year period, the goal of the EU, in terms of reducing the number of road deaths, appears far from being achieved (Adminaite et al., 2018). The present data suggest that, even though generalization from previously experienced road scenes is evident in the explicit behavior observed during the training of road users with limited experience, implicit learning requires prior exposure to each specific scenario. As such, any attempt to monitor and map conditions in which accidents occur and expose novice road users to the largest possible sample of such risky scenarios, so as to improve training programs, should be strongly encouraged.

# ETHICS STATEMENT

This study was carried out in accordance with the recommendations of guidelines for psychological research of the Associazione Italiana Psicologia (AIP) with written informed consent from all subjects. All subjects gave written informed consent in accordance with the Declaration of Helsinki.

# REFERENCES

The protocol was approved by the Ethical Committee for the Psychological Research of the University of Padua.

# AUTHOR CONTRIBUTIONS

MT supervised data collection and contributed to statistical analyses and manuscript writing. EG conducted data collection, statistical analyses, and manuscript writing. MT, EG, and MS contributed to research planning, results' discussion and revision of the paper.

# FUNDING

This research was supported by a grant FINA n. TAGL\_FINA18\_01 "Meccanismi sottostanti all'apprendimento alla guida sicura e riduzione dell'incidentalità su strada," from the Department of General Psychology to MT.

# ACKNOWLEDGMENTS

The authors thank I. M. Cordis for helping in data coding and S. Bellomo for helping in data collection and coding. The present work was carried out within the scope of the research program "Dipartimenti di Eccellenza, which is supported by a grant from MIUR to the Department of General Psychology, University of Padua."

of vulnerable road users: first-time motorcycle riders. Transp. Res. Part F Traffic Psychol. Behav. 14, 26–35. doi: 10.1016/j.trf.2010.09.003

**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2019 Tagliabue, Sarlo and Gianfranchi. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# Authenticity, Interactivity, and Collaboration in VR Learning Games

Meredith M. Thompson\*, Annie Wang, Dan Roy and Eric Klopfer

Comparative Media Studies and Writing, Massachusetts Institute of Technology, Cambridge, MA, United States

Decreasing cost and increasing technology access in schools places 3D immersive virtual reality (VR) within the reach of K-12 classrooms (Korbey, 2017). Educators have great interest in incorporating VR into classrooms because they are engaging and often novel experiences. However, long-term curriculum development must be positioned on how to best leverage the unique affordances of VR, be informed by theory and research, and integrate VR in meaningful ways that continue to motivate students even after experiences are no longer novel. We propose the theoretical framework of embodied learning and discuss how VR and reflect on current research findings to outline effective applications of VR and provide guidelines in developing educational materials using those tools. We discuss two particular examples: spatial awareness and collaboration. We share our perspectives on the benefits and challenges of applying these principles in a learning game about cellular biology.

#### Edited by:

Maria V. Sanchez-Vives, Institut d'Investigacions Biomèdiques August Pi i Sunyer (IDIBAPS), Spain

#### Reviewed by:

Yiorgos L. Chrysanthou, University of Cyprus, Cyprus Regis Kopper, Duke University, United States

> \*Correspondence: Meredith M. Thompson<EMAIL_ADDRESS>
#### Specialty section:

This article was submitted to Virtual Environments, a section of the journal Frontiers in Robotics and AI

Received: 14 November 2017 Accepted: 30 November 2018 Published: 19 December 2018

#### Citation:

Thompson MM, Wang A, Roy D and Klopfer E (2018) Authenticity, Interactivity, and Collaboration in VR Learning Games. Front. Robot. AI 5:133. doi: 10.3389/frobt.2018.00133 Keywords: immersive virtual reality, stem education, game based learning, embodied learning, K12 education, collaboration

# BACKGROUND

VR has the potential to broaden the reach of the traditional classroom by addressing limitations of K-12 classroom environments. VR simulations that engage learners as explorers shift the focus from content acquisition to active inquiry (Hew and Cheung, 2010; Merchant et al., 2012; Ahn et al., 2017). Now that these technologies are within reach of classrooms and lecture halls, research needs to move beyond simply asking whether VR can help bolster learning, and consider how best to use these tools in educational contexts (Dalgarno et al., 2011). In doing so, we not only imagine the types of problems that immersive 3D can solve for K-12 educators but consider the larger question of how to craft learning experiences for students that effectively move between and utilize two dimensional, three dimensional, and immersive 3D visualizations.

Our labs have developed a number of learning simulations and games, and are currently developing a game to introduce students to cellular biology. Through this process, we have gained perspective on the benefits and challenges of using VR in creating authentic, interactive, and collaborative experiences that help students learn about the complex topic of cellular biology. First, we explore how VR can be helpful in creating authentic representations in biology. Then, we discuss current understandings of the theoretical frameworks of embodied learning and collaborative learning. Finally, we discuss how we have applied these two perspectives through a collaborative, cross-platform educational game named Cellverse.

**77**

# Authenticity: Cell Biology as a Context for Virtual Reality

Cells and the central dogma are two critical topics in biology standards and curricular materials (National Research Council, 2013). Despite the importance of these concepts, visuals of cells are often oversimplified in introductory resources (Shi et al., 2010; Tibell and Rundgren, 2010) and misunderstood by students and educators alike (Çeliker, 2013; Vlaardingerbroek et al., 2014). Incorporating 3D visualization such as immersive VR into biology curricula may be a solution to improving student learning. Previous research has shown a notable positive correlation between the use of visual models and student scores—just a few class sessions of exposure to a tangible model can result in significant score improvement in beginner biology students (Höst et al., 2013), and highquality animations of cellular processes improve scores and higher long-term memory retention among students (McClean et al., 2005). Although these and other studies have noted a positive correlation between visualization and student learning, there are still challenges to be addressed. Skeptics argue that too much visual information can lead to "cognitive overload" and thus endanger learning, although they too acknowledge that there is definite potential in visualization (Tversky et al., 2002). In fact, what can help visual data become truly effective is "interactivity"—the ability for a user to stop, start, replay, and manipulate visuals at his or her own pace. VR is an excellent platform for designing interactive and manipulatable environments.

# Interactivity: Embodied Learning in VR

VR technologies can engage learnings both cognitively and physically through immersive and interactive experiences. The theory of embodied learning posits that connecting learning events and physical actions creates a stronger impact on the individual (Kiefer and Trumpp, 2012). VR technologies can be responsive to the participants' movements in a way that activates the learners' perception of themselves as a tool for developing understanding (Stolz, 2015).

VR simulations are already widely used to develop physical skills with instruments, as a flight simulator does for a pilot's aviation skills or a surgical simulator for a doctor's surgical technique (Slater and Sanchez-Vives, 2016). VR can also help learners practice laboratory skills during virtual laboratories (Chiu et al., 2015; Lindgren et al., 2016; Jang et al., 2017). More recently, VR has been used by scientists for honing their skills in preparing molecular compounds for microscopy (Leinen et al., 2015) and for envisioning how to modify molecules to develop new pharmaceutical drugs (Cheng et al., 2012; Yuan et al., 2017; Liu et al., 2018). Scientists share computer-based visualizations with the scientific community online, drawing upon 3D models of proteins, molecules, and molecular reactions through online resources such as the Protein Databank and PyMOL (Mwalongo et al., 2016; Yuan et al., 2017). VR has enhanced the process of drug discovery by enabling scientists to investigate molecular structure and function and prompted the development of mixed reality software platforms such as Molecular Rift (Yuan et al., 2017) and Reality Convert (Borrel and Fourches, 2017). These applications of VR for science can be useful in K12 contexts by enabling learners to create embodied analogies for abstract concepts through gesture and movement (Weisberg and Newcombe, 2017). For example, a VR simulation offered higher levels of understanding and retention among high school students learning cellular biology in comparison to traditional 2D models (Tan and Waugh, 2014).

Spatial understanding is related to understanding relative size and scale, a topic many students find challenging (Jones et al., 2003). Size and scale are important to understand in science, technology, engineering, and mathematics (STEM) domains (Weisberg and Newcombe, 2017). While individuals have varying degrees of spatial understanding (Coxon et al., 2016), spatial awareness can be improved (Uttal and Cohen, 2012). Activities such as creating 3D representations of geometric shapes (Burte et al., 2017) and through gesturing while solving spatial problems (Chu and Kita, 2011) can enhance individuals' understanding. Spatial awareness is linked to perception of size and scale, which is also important in STEM topics (Jones et al., 2008). Similar to spatial awareness, understanding of size and scale can be enhanced through direct experience with objects and with distances between objects (Jones et al., 2008) and through using a body as a comparison point (Jones et al., 2009). VR has already been useful as a research tool in understanding spatial awareness (Wilson, 2013), and shows promise in developing spatial skills. VR can provide learners with virtual experience with objects and prompt learners to gesture during problem solving; both activities have the potential to improve spatial understanding and users' perception of size and scale.

Problems that require perspective taking and understanding structure are well-suited to use VR. Virtual environments can help users develop "spatial presence," a perception of the overall VR environment and the relationships between the objects within that environment (Wirth et al., 2007). The level of embodiment achievable in VR is directly related to the level of interactivity between the user and the virtual space. 360 videos and virtual field trips are already being used in classrooms to help students learn geography and cultural awareness due to the lower cost of the equipment, however, the user has limited ability to interact with the experience(Brown and Green, 2016; Korbey, 2017; Minocha et al., 2017). Interactive simulations and virtual laboratories have helped students understand electrostatics and forces in physics (Salzman et al., 1999; Pirker et al., 2013), and mathematics (Mizell et al., 2002; Guerrero et al., 2015). Laboratories and simulations require more resources to design than virtual field trips, but the additional interaction supports a deeper level of embodied learning (Potkonjak et al., 2016; Jang et al., 2017).

# Collaborative: Learning in VR

The movement from room scale CAVE Automatic Virtual Environment (CAVE) to head-mounted displays (HMD) has decreased the cost of VR, yet these technologies have focused heavily on the individual's experience (Hew and Cheung, 2010; Slater and Sanchez-Vives, 2016). As technology and connectivity improves, VR will include collaboration between individuals in HMDs, requiring a new understanding of how technology can enable new forms of communication between individuals (Gugenheimer et al., 2017). Designers must balance the user's attention to their own experience and explore how to create a sense of shared presencse, or co-presence, in the virtual world (Campos-Castillo, 2012).

Principles of collaborative learning such as interdependence, thoughtful formation of groups, individual accountability, and attention to social skill development are also useful considerations in VR environments (Cuseo, 1997; Lee, 2009). Activities that require individuals to work together in order to accomplish goals create what Johnson and Johnson (1989) call "positive interdependence" among team members; the structure of the activity necessitates a joint effort. Since virtual environments are still relatively novel, both rules and roles can be useful in structuring collaboration. Jensen and Konradsen (2018) used games as a way to create rules for social interaction and roles for individuals in virtual problem-based activities. Roles also helped visitors engage with a VR museum exhibit experience on an aircraft carrier (Zhou et al., 2016). Middle school students in the EvoRoom VR environment EvoRoom environment benefited from clear roles in gathering and sharing information with their peers (Lui and Slotta, 2014).

In addition to clear roles, a range of expertise helps foster interdependence in virtual teams (Weber and Kim, 2015). One way to establish roles is to structure distributed teamwork through roles and access to different forms of technology and information (text based, 2D, 3D, VR) that must be synthesized to solve a problem. This redistribution can create power dynamics within the group. In comparing virtual to in person problem solving among teams of people using 2D, 3D, and VR interfaces, (Slater et al., 2000; Slater and Sanchez-Vives, 2016) found that the individual in VR was more likely to emerge as the leader, even if that same person did not take on a leadership role in the in-person project. Spante et al. (2006) also studied puzzle solving across VR and 2D systems; they found that team members assumed both had the same view until they traded places. Having different viewpoints enhanced collaboration, creating what Spante et al. (2006) termed "the good inequality". Gugenheimer et al. (2017) created a system where individuals in HMD could interact with individuals outside of VR through a "FaceDisplay," a touch screen interface. Teamwork can be reinforced by structuring environments to providing team members with complementary information and different views of information; furthermore, students also gain appreciation of how different forms of media may be more appropriate for understanding certain concepts.

# AN EXAMPLE IN PROGRESS CELLVERSE—A COLLABORATIVE LEARNING ENVIRONMENT IN VIRTUAL REALITY (CLEVR)

We now apply some of these ideas about embodiment and collaborative learning to a game-based learning project currently under development, Cellverse. In Cellverse students learn about cells and the process of converting DNA to proteins through an interactive problem-based game. Working in small teams of two or three, students examine a living cell from within. The Explorer wears a head mounted display and moves through the cell in VR to observe function and structure, as shown in **Figure 1**. The Navigator uses a tablet-based toolkit of disease descriptions, stains, tags, and measurement devices to gather data and focus the visualizations using a table, as shown in **Figure 2**. The experience is being designed with a distribution of data available for players in a way that students must communicate to solve the puzzle together.

A central question is—why use VR? Virtual reality allows students to experience the cellular environment as an active explorer, rather than a passive observer. It also gives students an appreciation for the density of the cell, the size and scale of organelles relative to each other and to other molecules in the cell, spatial relationships between the organelles, and the cellular structure. Structure and spatial orientation are both important in understanding the central dogma, when DNA is first transcribed to mRNA and translated by tRNA into long amino acid chains that become proteins.

We draw connections between the effective practices we have found in in the literature using the affordances of VR and our intentions for this project.

# Authentic

We situate student learning in the context of biology, both in the game narrative and the game environment. We have built a cellular environment that matches current research on cells, with ongoing input and feedback from cellular biologists and other cell biology experts. Whenever possible, we have incorporated tools and activities that scientists would use as in-game functions. For example, students can highlight specific organelles and structures within the virtual environment using simultaneous label-free autofluorescence-multiharmonic (SLAM) microscopy (You et al., 2018). Cells are densely packed, which is challenging to render and can be overwhelming to users. We continually balance how to represent the cell most authentically while maintaining presence within the experience and minimizing cognitive load.

# Interactive

The game-based format provides a high degree of interaction between students and the concepts included in the VR environment, which has been linked to deeper learning (Lindgren et al., 2016; Jang et al., 2017). The game-based format also provides ongoing feedback to the players, which also assists the learning process (Merchant et al., 2012). Through the game, we aim to transform a topic that is often passive and vocabularybased into an active, embodied experience.

We also incorporate aspects of biology within the game narrative and the game environment. We are building a cellular environment that matches current research on cells, with ongoing input and feedback from cellular biologists. Students will learn what a cell biologist might do by using tools such as (SLAM) microscopy (You et al., 2018).

# Collaborative

We are building interdependence among team members into the design of the project by creating rules, establishing roles, and distributing resources between players. Rules are established before students take on roles in the game; either as explorers or navigators. The explorer will see the 3D VR view of the world and will complete tasks that involve spatial relationships between organelles, identifying protein structures, and tracking processes within the cell. The navigator has access to information on 2D and 3D flat screen models to help guide the explorer and to work with other team members as they identify organelles, proteins, and even DNA and RNA sequences that could provide helpful clues in the game. We plan to build different levels in the game to allow students to rotate through team roles.

Collaborative activities can be enhanced through different modalities. We implemented a number of functions that would allow users to communicate to each other across platforms, including but not limited to "light beacons" that can be placed within the virtual environment and functions such as SLAM microscopy that reflect how real-world scientists mark organelles. These functions do not only allow users to communicate with each other through non-verbal manners, but also enhance their collaborative experience and create embodied learning within the virtual environment.

# CHALLENGES

There have been a number of challenges that we have confronted while building and implementing Cellverse. As Cellverse is a complex environment with many moving parts, users risk becoming nauseated if there is too much activity, or not enough computer processor power to render the activity in real time. A high frame rate, thus, is vital for a smooth VR experience; too much detail or too many objects within the virtual world can reduce the frame rate and cause nausea (Jerald, 2015). We have had to compromise authenticity with playability, and reduced the details of certain structures in order to maintain a comfortable frame rate.

Creating a balanced flow of information between the two players has been challenging. Effective and worthwhile collaboration happens when each player is equally involved and are able to fill in whatever information their partner does not possess. We have explored different ways to foster collaboration through distribution of information resources to the players.

While our goal is to create an authentic environment, scientific understanding is continually advancing. We have also had to make choices about the specificity of our cellular model and the number of processes we can represent in a realistic design timeframe. We have also noted that in an ever-evolving field like microbiology, application authenticity in educational material remains a challenge. There are new discoveries made regarding cells and cell structure every day, and it is in our best interest to make Cellverse as accurate to these discoveries as possible. However, it sometimes means that we do have to change aspects of the game that may not be immediately noticeable to student players. Although they may not be consciously aware of these changes, it is our belief that making the Cellverse environment authentic will allow students to come away with a more wellrounded understanding of cells.

# FEASIBILITY

We are also attuned to how CLEVR could be integrated into curricula and implemented in classrooms. Our partner teachers

have confirmed that the cell and central dogma are important topics in introductory biology. Teachers are helping us imagine how to incorporate VR technology in a feasible way in today's classrooms, and also provide insight into design features of the game that can help the game run smoothly. For example, while students may be excited to try the 3D VR experience, having all students in the class in headsets simultaneously may be a challenge. Conversely, some students may not want to wear headsets, or may be absent from school during the activities, so the activity should be designed so that team members can take on different roles and responsibilities if not all members are there.

Despite these opportunities, the cost of developing quality educational materials remains relatively high. Although the cost of VR has decreased over the years (Korbey, 2017), investing in VR requires significant resources. The labor involved in creating immersive, interactive, and accurate educational VR material is also great. It is then necessary to capitalize on all possible affordances of VR, and to carefully allocate resources so that more individuals can participate perhaps at once—and benefit from the experience in the long term.

# CONCLUSION

Now that VR technology is within reach of educational settings, learning designers and educators can focus on how best to incorporate VR into educational contexts. In this article, we discuss and provide an example of how VR experiences can represent authentic contexts, focus on embodied experiences, and how to structure the environment to foster teamwork and collaboration by having participants view and synthesize different types of data across immersive and non-immersive formats. These parameters can be used to develop effective and engaging learning environments based on our current understanding of VR in education. When moving forward, researchers, developers, and educators should investigate how each of these factors can be fashioned to optimize learning, identify affordances and challenges that may emerge as VR becomes more widespread, and incorporate findings and feedback into future development.

# ETHICS STATEMENT

This study was carried out in accordance with the The Common Rule, 45 CFR pt. 46, with informed written consent from all subjects. All subjects gave written consent in accordance with the Declaration of Helsinki. The protocol was approved by the Committee on the use of Humans as Experimental Subjects (COUHES) at MIT (#10707095354).

# AUTHOR CONTRIBUTIONS

MT did the planning for the paper, read and synthesized literature notes, and wrote the paper. AW read and compiled paper notes, wrote the section on students' understanding

# REFERENCES

of cells, and helped review drafts of the manuscript. DR helped review the paper from a game designer perspective and edited the paper. EK advised MT as she formulated the paper idea, helped imagine the direction for the paper, and reviewed and edited the paper multiple times, providing indepth feedback.

# FUNDING

The project was funded by a grant from Oculus Education.

Geometry. University College London, 1–7. Available online at: http://citeseerx. ist.psu.edu/viewdoc/download?doi=<IP_ADDRESS>.5391&rep=rep1&type=pdf

**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2018 Thompson, Wang, Roy and Klopfer. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# Can Simulator Sickness Be Avoided? A Review on Temporal Aspects of Simulator Sickness

#### Natalia Duzma ˙ nska ´ 1 \*, Paweł Strojny1,2 and Agnieszka Strojny1,2

<sup>1</sup> R&D Unit, Nano Games sp. z o.o., Kraków, Poland, <sup>2</sup> Institute of Applied Psychology, Faculty of Management and Social Communication, Jagiellonian University, Kraków, Poland

Simulator sickness is a syndrome similar to motion sickness, often experienced during simulator or another virtual reality (VR) exposure. Many theories have been developed or adapted from the motion sickness studies, in order to explain the existence of the syndrome. The simulator sickness can be measured using both subjective and objective methods. The most popular self-report method is the Simulator Sickness Questionnaire. Attempts have also been made to discover a physiological indicator of the described syndrome, but no definite conclusion has been reached on this issue. In the present paper, three temporal aspects of the simulator sickness are discussed: the temporal trajectory of the progression of simulator sickness, possibility of adapting VR users in advance and persistence of the symptoms after VR exposure. Evidence found in 39 articles is widely described. As for the first aspect, it is clear that in most cases severity of the simulator sickness symptoms increases with time of exposure, although it is impossible to develop a single, universal pattern for this effect. It has also been proved, that in some cases a threshold level or time point exists, after which the symptoms stop increasing or begin to decrease. The adaptation effect was proved in most of the reviewed studies and observed in different study designs – e.g., with a couple of VR exposures on separate days or on 1 day and with a single, prolonged VR exposure. As for the persistence of the simulator sickness symptoms after leaving the VR, on the whole the study results suggest that such an effect exists, but it varies strongly between individual studies – the symptoms may persist for a short period of time (10 min) or a relatively long one (even 4 h). Considering the conclusions reached in the paper, it is important to bear in mind that the virtual reality technology still evokes unpleasant sensations in its users and that these sensations should be cautiously controlled while developing new VR tools. Certainly, more research on this topic is necessary.

Keywords: simulator sickness, temporal aspects, time, virtual reality, VR

# INTRODUCTION

# Virtual Reality – A Definition and the Most Commonly Used Devices

The simplest definition of virtual reality states that is "the use of computer-generated virtual environments and the associated hardware to provide the user with the illusion of physical presence within that environment" (Jayaram et al., 1997, p. 576). Virtual reality systems are widely used in the fields of scientific research (e.g., Anderson-Hanley et al., 2011), anxiety disorders therapy

#### Edited by:

Albert Rizzo, University of Southern California, United States

#### Reviewed by:

Eugene Nalivaiko, University of Newcastle, Australia Inmaculada Remolar Remolar, Universitat Jaume I, Spain

\*Correspondence: Natalia Duzma ˙ nska ´<EMAIL_ADDRESS>
#### Specialty section:

This article was submitted to Human-Media Interaction, a section of the journal Frontiers in Psychology

Received: 28 June 2018 Accepted: 16 October 2018 Published: 06 November 2018

#### Citation:

Duzma ˙ nska N, Strojny P and ´ Strojny A (2018) Can Simulator Sickness Be Avoided? A Review on Temporal Aspects of Simulator Sickness. Front. Psychol. 9:2132. doi: 10.3389/fpsyg.2018.02132

**84**

(e.g., Gerardi et al., 2010; Łukowska, 2011) or for professional training [e.g., in the army – (Braithwaite and Braithwaite, 1990); fire department – (Bliss et al., 1997); aviation – (Kennedy et al., 2000); medicine – (Bric et al., 2016)].

Many different virtual reality hardware systems and devices have been developed over the years and will be briefly described herein. Nowadays, the most popular are the head-mounted devices (HMDs), such as HTC Vive or Oculus Rift. The VR user wears a headset and holds two controllers which enable them to move and interact in a three-dimensional environment. Such devices are now being sold commercially. According to a recent Business Insider report (Hollander, 2018), there are four main VR headset types: stand-alone (which do not need any additional hardware to function), smartphone-powered, PCpowered and game console-powered. The report predicts that the stand-alone headsets will grow in popularity in the coming years. This could be of advantage for research employing the VR technology, as eliminating the wire which connects the headset to a PC or a console will make conducting experiments with multiple participants at the same time much easier.

Another example of a VR system is a CAVE (cave automatic virtual environment). In such system, the environment is displayed and generated on several projectors, directed to the walls of the room and the user wears 3D glasses.

Different additional devices are used in order to provide the VR user with a realistic, multisensory experience. For example, treadmills are often used to simulate movement in the virtual environment (e.g., Jaeger and Mourant, 2001; Sinitski et al., 2018). For driving and flight simulators, a part of a plane cockpit or a body of a car may be used (e.g., Feenstra et al., 2011; Domeyer et al., 2013; Reinhard et al., 2017).

# Definition of Simulator Sickness

Simulator sickness is a syndrome similar to motion sickness and can be experienced as a side effect during and after exposure to different virtual reality environments. Originally, the term "simulator sickness" was linked to effects induced by simulators consisting of a platform, often mobile, and with the visual stimuli generated by a computer, without head-tracking. The invention of HMDs led to developing another term, "cybersickness," as such devices generate another issues, which may also lead to the unpleasant symptoms, such as the delay between actual head movements and the generated image. However, nowadays both of the terms are being used by researchers to describe the unpleasant symptoms evoked by the virtual reality technology (e.g., Sharples et al., 2008; Bruck and Watters, 2011; Serge and Moss, 2015; Lee et al., 2017).

The symptomatology and severity of the malaise depend on many variables – e.g., age, gender, stress, anxiety, one's individual proneness to such ailment or the characteristics of the simulator itself (Kolasinski, 1995; Cobb et al., 1999; Mourant and Thattacherry, 2000; Jaeger and Mourant, 2001; Lin et al., 2002; Sharples et al., 2008; Brooks et al., 2010; Bruck and Watters, 2011; Classen et al., 2011; Moss and Muth, 2011; Zuzewicz et al., ˙ 2011; Biernacki and Dziuda, 2012; Dziuda et al., 2014; Helland et al., 2016; Lee et al., 2017). Lin et al. (2002) have also suggested that a relationship between one's enjoyment experienced during simulator training may lead to alleviation of the simulator sickness symptoms. A very detailed list of variables, which may have influence on simulator sickness occurrence and severity, may be found in the report by Kolasinski (1995).

The main aims of this paper are to summarize the existing knowledge on simulator sickness with emphasis on its temporal aspects, to provide an overview of research on this topic and to propose further research directions and practical implications for virtual reality developers.

Firstly, the most common theories which could serve as an explanation of the simulator sickness phenomenon will be discussed. Secondly, the methods of simulator sickness measurement, both subjective and objective, will be described in detail. Thirdly, three temporal aspects of simulator sickness will be discussed based on evidence found in empirical studies. And lastly, general conclusions drawn from the reviewed studies and practical implications for further research will be provided.

# Theories Potentially Explaining Simulator Sickness

Several theories have been developed to explain why individuals suffer from motion sickness. According to authors focused on virtual simulators, they may be also applicable in the field of simulator sickness during exposure to virtual reality (Brooks et al., 2010). The Sensory Conflict Theory, proposed by Reason and Brand (1975), explains motion and simulator sickness through a conflict that arises between different sensory systems; namely the signals from visual, vestibular and non-vestibular proprioceptors differ from one another and inevitably differ with expectations based on previous experience. According to the theory, only the conflict between present sensory information and that retained from immediate past elicits sickness. That is claimed on the basis of observation that continuous exposition to a stimulus results in eventual disappearance of symptoms (adaptation) even if the present conflict still exists (Reason, 1978). The vestibular system, which is responsible for perception and detection of direction, is crucial for occurrence of simulator and motion sickness symptoms (Reason and Brand, 1975).

Reason (1978) proposed the Neural Mismatch Model which identifies the source of simulator sickness in discrepancies between expectations derived on a basis of present moves and contents kept in the neural store which, according to Reason (1978), contains information about typical combination of command signals (efference) and the integrated patterns of inputs from the orientation senses generated by them (reafference). That is the theoretical mechanism of adaptation to motion sickness observed for example by Reason and Brand (1975). To conclude, according to this model, sickness occurs when the received sensory information does not match one's experiences based on past situations.

Another theory, widely used to explain simulator and motion sickness, is the Postural Instability Theory. Riccio and Stoffregen (1991) have criticized the Sensory Conflict Theory – they state that sensory conflicts such as those described by Reason and Brand (1975) happen very often and are nothing unusual. Furthermore, the difference (or lack of it) between what one's

senses experience and what an individual expects to feel is immeasurable. They have proposed that the symptoms of motion or simulator sickness may be experienced when one has been exposed to long-lasting postural instability and has not yet learned how to adjust to this situation and maintain proper balance. The most vivid example of such phenomenon is the feeling of instability one experiences when traveling by ship. A similar situation occurs during rollercoaster rides as well (Riccio and Stoffregen, 1991).

The two aforementioned theories are most prevalent in the literature concerning simulator sickness. Other theoretical approaches to this phenomenon have been developed as well. The Eye Movement Theory developed by Ebenholtz (1992, 2001), uses the vagus nerve stimulation as an explanation for motion and simulator sickness. The mechanism is initiated by two specific eye movements (namely the optokinetic nystagmus and vestibular ocular response<sup>1</sup> ) creating tension in the muscles of the eye, which stimulates the vagus nerve and leads to unpleasant symptoms such as difficulty concentrating, eye strain and headaches.

Bruck and Watters (2011) have also attempted to develop a comprehensive theory of cybersickness. They suggest a following chain of causality: an increase in arousal leads to changes in respiration rate, which causes carbon dioxide levels in cerebral blood flow to decrease. These changes lead to the symptoms of simulator sickness: dizziness, fatigue, difficulty concentrating, fullness of head and anxiety. The authors propose dividing the simulator sickness symptoms into four factors:

The Evolutionary Theory, proposed by Treisman (1977), originally explains the motion sickness, but its assumptions can be adapted to simulator conditions as well. Treisman (1977) suggests that people experience motion sickness, because – evolutionally – our species has not managed to adapt to new transportation modes yet. Therefore, the human body reacts to sensory conflicts with nausea – it acts as if poison had been ingested (Brooks et al., 2010). It can be assumed that similar reasons may stand behind the simulator sickness symptoms, as the human species had even less time to adapt to the virtual reality conditions. Although this theory does not propose any physiological mechanisms that may be responsible for experiencing simulator sickness, it can give a valuable insight on reasons why such ailment exists.

# Measurements of Simulator Sickness Self-Report Measures

#### **Simulator Sickness Questionnaire (SSQ)**

Originally published by Kennedy et al. (1993), the Simulator Sickness Questionnaire (SSQ) is a tool widely used for assessing the subjective severity of simulator sickness symptoms. In the pre-experiment part of the questionnaire, information about the current physical condition and participant's experience with simulators is collected. The questionnaire consists of 16 items, derived from the Pensacola Motion Sickness Questionnaire (MSQ). Data collected during previous simulator studies using the MSQ was gathered and the items describing symptoms with less than 1% frequency of appearance or with no change in frequency between pre- and post-exposure were excluded from further analyses (12 of 28 items of MSQ). The severity of each symptom in the SSQ is measured on a four-point scale (0-3).

According to the results of a factor analysis, the items of the SSQ can be grouped into three factors: nausea (e.g., sweating, difficulty concentrating, stomach awareness), oculomotor disturbance (e.g., headache, eyestrain, blurred vision) and disorientation (e.g., fullness head, dizziness with open and closed eyes, vertigo). The factors are not entirely independent – some of the items were included in more than one factor, e.g., the score on difficulty focusing is used to assess the severity of oculomotor disturbance and disorientation. In total, there are five such items. To calculate scores on each factor, all relevant items' scores should be added (each factor consists of 7 items) and multiplying the obtained sum by a specific weight: for nausea by 9.54 (therefore the scores on this scale range from 0 to 200.34), for disorientation by 13.92 (scores ranging from 0 to 292.32) and for oculomotor disturbance by 7.58 (with scores ranging from 0 to 159.18).

The overall score can be measured as well and it can serve as an indicator of total severity of the simulator sickness. It is calculated by adding scores on the 16 items and multiplying the achieved sum by 3.74, therefore the total score can range from 0 to 179.52. In addition to the quantitative data, qualitative information about peculiar sensations during the simulator experience and symptoms other that those listed in the main part of the questionnaire can be gathered (Kennedy et al., 1993; Biernacki et al., 2016).

Simulator Sickness Questionnaire has been used in numerous studies (e.g., Lampton et al., 1994; Mourant and Thattacherry, 2000; Jaeger and Mourant, 2001; Lin et al., 2002; Min et al., 2004; Sharples et al., 2008; Bruck and Watters, 2009a,b, 2011; Moss and Muth, 2011; Biernacki and Dziuda, 2014; Brunnström et al., 2017). The brevity and simplicity of the questionnaire are its assets, as in many study designs it is being used at least twice to assess the changes in occurrence and severity of simulator sickness' symptoms. In most cases SSQ is used as a paper-andpencil test, but it can also be conducted orally – as in Min et al. (2004) study, where the items of the questionnaire were read to the participants by the experimenter (according to the authors of

<sup>1</sup>Optokinetic nystagmus – an eye pursues a target object from one end of a visual field to the other. When the eye can pursue the object no further, it snaps back to the far side of the visual field where it begins to pursue again. Vestibular ocular response – responsible for keeping a target object on the fovea, the center of the retina where one's vision is sharpest, when the head is turning.

the study, conducting the SSQ orally requires only circa 30–40 s) or in the study by Moss and Muth (2011), where a cassette was pre-recorded and then played back to the participants.

#### **Other self-report measures**

fpsyg-09-02132 November 2, 2018 Time: 17:7 # 4

It should be noted that in some studies self-report methods of measurement different from the Simulator Sickness Questionnaire had been used – Brooks et al. (2010) report having used the Motion Sickness Assessment Questionnaire, Malinska ´ et al. (2014) used a self-developed, concise questionnaire and Helland et al. (2016) measured subjective severity of simulator sickness symptoms simply by asking – "To what extent did you experience simulator sickness during the driving test?". Several other authors used other short self-report measures (e.g., McCauley et al., 1990; Helland et al., 2016; Reinhard et al., 2017). As these methods are either a questionnaire originally created for measuring a different ailment or have not been psychometrically tested, they will not be described more widely herein.

### Physiological Measures

Although a conclusion has not yet been reached on which specific physiological parameters are the best indicators of simulator sickness, some researchers (e.g., Min et al., 2004; Bruck and Watters, 2011; Zuzewicz et al., 2011 ˙ ) have tested various physiological variables and some of them appear promising for evaluating simulator sickness without relying on self-report measures or as a supportive method for questionnaires such as SSQ. It has been noted (Min et al., 2004) that during driving (and most of the studies concerning simulator sickness were conducted with various driving simulators) the increase of autonomic nervous system activation may relate to tension, which then causes the heart rate and skin conductance to increase and skin temperature to decrease. Moreover, the physiological measures may be useful, as it has been proved that the subjective evaluation of simulator sickness (e.g., with the SSQ questionnaire) is slightly delayed when compared to the physiological indicators (Min et al., 2004). Therefore, establishing the best physiological indicators of simulator sickness could shed more light on the exact triggering time of the syndrome and therefore allow a more accurate description of the temporal characteristics of simulator sickness.

As no unambiguous physiological indicators of simulator sickness have been discovered, some examples of use of physiological indicators for measuring this syndrome will be described in this paragraph.

#### **Autonomic nervous system**

Respiration (breaths per minute). According to one of the theories of simulator sickness (or "cybersickness," as referred to by the authors; Bruck and Watters, 2011), the changes in respiration rate are crucial to evoking the unpleasant symptoms, especially when the person subjected to a virtual reality environment has no control. Respiration loads two factors in the theory of cybersickness developed by Bruck and Watters (2011): Vision and Arousal. They even propose that hyperventilation may be the cause of arousal experienced by individuals exposed to high levels of movement in a virtual reality. Empirical evidence of changes in respiration rate during VR exposure were achieved by Kim et al. (2005) – in their study a decrease in the respiration rate (when compared to baseline levels) was observed. What is more, a positive correlation was observed between respiration rate and the Simulator Sickness Questionnaire scores (for all of the subscales and the total score, with the r values oscillating between 0.342 for nausea and 0.392 for the total score).

## **Heart rate**

Bruck and Watters (2011) propose that the heart rate may serve as an indicator of simulator sickness, as it had been previously proved that it correlates with such syndrome. In experiments conducted by Cobb et al. (1999) heat rate tended to accelerate during the simulator task and returned to a resting rate in approximately 30 min after completing the task. Furthermore, the heart rate of the participants who reported more severe simulator sickness symptoms was also higher than the heart rate of the individuals who did not experience such unpleasant sensations. Additionally, the heart rate of the participants who showed symptoms of adapting to the VR (virtual reality) conditions during several exposures decreased over the three sessions. Changes in heart rate were observed in a couple of studies. Dahlman et al. (2008, 2009) have noted an increase in heart rate during a VR exposure. In a study by Gavgani et al. (2016) the subjects participated in three roller coaster simulator rides, which took place on separate days. For the first 2 days, an initial tachycardia and tachypnoea that gradually lowered during the ride was observed. No such patterns were discovered on the third day.

Other autonomic variables. In the course of research, some other measures of the autonomic nervous system activity have been tested. This paragraph will provide a brief overview of them. Kim et al. (2005) have observed an interesting pattern of the gastric tachyarrhythmia changes – in increased significantly in the first 4 min of virtual reality exposure and then continued to increase until the final 4 min of a 9.5 min trial. The eyeblink rate did also change in the study by Kim et al. (2005) – it decreased in the first minute of the exposure (when compared to the baseline rate), but then increased and in the middle of the trial it was significantly higher than the baseline level. Another interesting measure is the skin temperature – as observed by Kim et al. (2005), when measured at the fingertip, the skin temperature decreased in the middle of the trial and remained significantly lower than the baseline level even after leaving the VR environment. Such decrease in skin temperature was also observed by Chung et al. (2007) and Brooks et al. (2010). Furthermore, according to the results obtained by Kim et al. (2005), the respiratory sinus arrhythmia (a variation in heart rate occurring during breath cycle) increases during VR exposure.

What is interesting about the above mentioned measures is the fact that for all of them, except for skin temperature, positive correlations with the subjective measurement of the simulator sickness (SSQ) were observed (Kim et al., 2005), with the Pearson r values ranging between 0.265 (eyeblink rate and oculomotor disturbance scale) and 0.359 (gastric tachyarrhythmia and nausea scale).

Furthermore, in a study by Gavgani et al. (2016), a rapid increase in finger skin conductance levels was observed during

the first minute of the VR exposure – the subjects experienced increased sweating in the finger; this trend was present until the end of the experimental trial. However, what is the most interesting, in the cited study phasic SCL activity in the forehead was observed during the experimental trial (compared to none during baseline measurement). This activity – and only this of all of the measured physiological responses – was proven to be associated with the experience of nausea.

The authors (Gavgani et al., 2016) give an interesting interpretation of their findings, which may shed new light on the physiological components of the simulator sickness experience. Some of the physiological symptoms (initial tachycardia, tachypnoea, finger sweating) were present at the initial phase of the VR exposure, in the time during which no self-reported nausea was present. This conclusion is supported by the fact that the above mentioned effects (except for finger sweating) became non-significant on the last, third exposure. The authors conclude that these symptoms may be evoked by emotions and arousal connected with the novelty of the VR experience. The forehead sweating, however, is related to the development of nausea. These results correspond with Treisman's (1977) evolutionary theory of motion sickness – reducing the body temperature by increasing sweating serves as a survival strategy during intoxication.

#### **Central nervous system**

As a measure of the central nervous system activation, EEG has been used in some of the studies (Min et al., 2004; Chung et al., 2007). According to the results obtained by Min et al. (2004), there are significant differences in brainwaves patterns between rest and driving in a driving simulator. Such results have been obtained both for the frontal (Fz) and parietal lobe (Cz), giving similar patterns. After 5 min of simulator exposure, the δ/total increased and α/total, ß/total and θ/total decreased significantly in 5–35 min of simulator exposure. Furthermore, the δ/total at Fz correlates positively, and both θ/total and ß/total at Fz and Cz negatively, with the total SSQ score. The correlation with the SSQ score was the strongest for the θ/total parameter (r = −0.842 at Fz and r = −0.93 at Cz), therefore the authors of the study (Min et al., 2004) propose that it could serve as the most effective physiological indicator of simulator sickness occurrence. This proposal was also supported by Chung et al. (2007).

# Behavioral Measures – Postural Stability Tests

When relying on the Postural Instability Theory (Riccio and Stoffregen, 1991), one could use a postural stability test in order to assess the lack of postural stability as a specific manifestation of simulator sickness. Mourant and Thattacherry (2000) report using such test in their study. It is a simple and brief method – the person is asked to stand on the leg of their choice for 30 s in two separate trials. The time of standing without putting the other leg down is recorded and can be compared to the results of the same test after experimental manipulation or can serve as an independent measure. Although this method does not give a broad insight into simulator sickness symptoms, it can be useful when assessing changes in postural stability dependent on simulator exposure.

Cobb et al. (1999) report using a more complex set of postural stability tests: in their research program, the following methods of measurement were used: measuring the extent to which a static posture could be held, measuring the extent of hip sway over a 30 s period, walking on the floor and navigating over an uneven path with open eyes. Additionally, the authors administered two scales: task difficulty scale and subjective postural stability scale (Postural Stability Questionnaire – PSQ; Hamilton et al., 1989) after completing all the tasks.

# Temporal Aspects of Simulator Sickness

Questions regarding the temporal characteristics of the virtual reality experience which influence simulator sickness seem to recur in many papers (e.g., Kennedy et al., 2000; Moss and Muth, 2011; Domeyer et al., 2013). Although no unambiguous answers have yet been provided, some useful and promising leads can be found in literature and will be discussed herein. Since the main goal of the present work was to review research on simulator sickness from the temporal perspective, we decided to focus on research regarding one (or more) of the three issues described below.

As Kennedy et al. (2000) have observed, there are two main phenomena regarding the temporal aspect of simulator sickness: that the severity of simulator sickness increases with the increase of exposure duration during a single session, and that subjecting a person to several repeated simulator exposures may result in adaptation to the simulator conditions and thus in decrease of simulator sickness symptoms severity. The aforementioned aspects will be discussed in the present paper, as they seem to be crucial as far as virtual reality development is concerned. Furthermore, according to some research (e.g., Moss and Muth, 2011; Biernacki and Dziuda, 2014; Malinska et al., 2014 ´ ), the simulator sickness symptoms appear to persist for some time after the simulator exposure – this aspect will be discussed below as well.

# MATERIALS AND METHODS

# Search Strategy

A search of literature was performed in three electronic databases (Web of Science 'all databases,' PsychArticles, Scopus) with no publication date restriction. Since temporal aspects of simulator sickness rarely are the main focus of studies, we decided to retrieve a wide range of articles using the broadest term "simulator sickness" and assuming intensive article selection in subsequent stages. Thousand two hundred records were obtained. The search was conducted on 19th April 2018.

# Study Selection

Authors conducted a title and abstract screening, in order to exclude obviously irrelevant articles. Following keywords were used: time, temporal, durat<sup>∗</sup> , adapt<sup>∗</sup> , persist<sup>∗</sup> . The articles which titles and abstracts suggested an irrelevant area of research were excluded on this basis (1086 records). In the second stage of the screening process, full texts were retrieved and duplicated records removed (34 records). For 10 records full texts were unavailable and thus these records were excluded from the database as well. 70 articles were retrieved and evaluated in full text using the following criteria:

fpsyg-09-02132 November 2, 2018 Time: 17:7 # 6

After this process, 30 articles were retrieved. The authors decided to add 5 articles on the basis of hand search and previous knowledge. The final database consisted of a total of 35 articles (41 studies). A flow chart describing the search and screening process is presented in **Supplementary Figure S1**.

# RESULTS

# The Temporal Trajectory of the Progression of Simulator Sickness

Studies on simulator sickness have been conducted since 1990s, using a wide array of virtual reality devices. Therefore, it is important to emphasize the fact that direct comparisons between studies using different hardware should be treated with extreme caution. Some trends may be observed, but it should be always borne in mind that for different devices and scenarios the temporal patterns of simulator sickness may vary significantly. Moreover, as some of the cited studies have been conducted almost 20 years ago, caution should be taken while making conclusions. However, the insight provided by the researchers appears to be valuable – while the technological development might have solved some of the problems, the methodology and qualitative conclusions are worth knowing.

In one of the studies conducted by Cobb et al. (1999), four subjects were immersed in a virtual reality environment for 1– 2 h. Simulator sickness severity was measured with the Simulator Sickness Questionnaire. The participants were asked to remain in the virtual reality for up to 2 h. All participants reported the severity of symptoms increasing up to 1 h of exposure. Two of the participants withdrew after an hour when the simulator sickness symptoms experienced by them were too severe (mean scores for nausea: M = 67, oculomotor disturbance: M = 57 and disorientation: M = 82). The remaining two participants completed the 2-h immersion and reported that after 75 min the severity of symptoms decreased greatly. This suggests that although the simulator sickness symptoms severity increases with time, for some individuals it may be possible to adapt to the VR environment during a single exposure. Unfortunately, the sample in the study was too small to provide information on statistical significance of these effects. Nevertheless, these results are interesting and worth being taken into consideration when planning further experiments on extended VR exposure.

Kennedy et al. (2000) examined SSQ data from a military pilots' flight simulator training database and categorized them by exposure duration into four categories (0–1, 1–2, 2–3 h, 3 or more hours). An analysis of variance revealed that the mean SSQ scores increase gradually when exposure duration increases. This trend proved to be statistically significant. No information on statistical significance of differences between each of the categories was given and it also should be noted that the analyzed data concerned many different simulator environments. It was also a between-subject design, therefore no conclusions about individual temporal patterns of simulator sickness severity can be made.

Min et al. (2004) have tested various measures of simulator sickness severity. In their study, both physiological and self-report methods were used – the Simulator Sickness Questionnaire was used for assessing the subjective severity of the syndrome. Only the results of the psychometric measurement will be reported herein. After baseline signal measurement and pre-experiment SSQ administration, the participants of the study drove a car simulator for 60 min, during which physiological measurements were conducted and the SSQ was completed orally after every 5 min of the simulator exposure, as well as after completing the whole trial. The authors of the study report that all of the participants showed symptoms of nausea, disorientation (after 10 min of simulator exposure) and oculomotor disturbance (after 25 min). The first significant difference between the baseline SSQ score and trial score appeared 10 min after beginning of the trial. The obtained results confirm the hypothesis that the severity of simulator sickness increases with time.

Moss and Muth (2011) tested several characteristics of HMDs as possible factors influencing simulator sickness severity, as well as the effect of a prolonged exposure. Only the latter of these effects will be reported herein. The participants' task was to locate several objects in the virtual environment (a virtual laboratory), according to verbally given instructions, using only head movements. Each participant completed two practice sessions and five 2-min trials with 1-min breaks between them. A number of Simulator Sickness Questionnaire results were collected: before the experiment, after a practice session, after each trial, 5 and 10 min after the experiment. It was noted that the severity of simulator sickness symptoms increased with time – a significant effect of duration of the VR exposure was revealed. The most severe symptoms were noted after the last trial.

The type of walking interaction was the main topic explored by Lee et al. (2017), but their results also provide information about the temporal characteristics of simulator sickness. In their experimental design three types of walking control were included:

All of the participants of the study were exposed to three different VR environments (a cartoon town, a realistic nature environment and a low poly<sup>2</sup> landscape in a three-step walking interaction: they either experienced them in the order of: gamepad, hand interface, walking simulator or in the reverse order – each of the participants completed nine VR experiences in total. The following variables were tested in the study: immersion, presence and simulator sickness (measured with the Simulator Sickness Questionnaire). The authors reported that the simulator sickness symptoms became more severe with time, although on the whole they were of moderate severity.

The above-mentioned study results support the hypothesis, that the severity of simulator sickness does increase with time during a single exposure, to various extents, which may differ depending on many variables (e.g., simulator type and its characteristics, length of the whole exposure, individual characteristics of the participants, etc.). Such results are confirmed in many other studies, which will be briefly summarized herein. Lo and So (2001) have confirmed that the nausea severity (measured by one question with answers ranging from 0 – "no symptom" to 6 – "moderate nausea, want to stop") increases linearly with time during a 20-min exposure. Furthermore, the increase was significant in all of the comparisons, except for the one between the 15th and 20th minute of the trial. A similar study was conducted (So et al., 2001), and during a 30-min exposure the nausea ratings (measured in the same way as above) increased as well, but the differences were significant only in the 5th and 10th minute. Jarchow and Young (2007) have also measured the simulator sickness severity by asking just a single question (with a scale from 0 – "normal" to 20 – "about to vomit"). The subjects were tested on two consecutive days, as the main aim of the study was to assess the adaptation effect. It was however, discovered as well that within a single session the severity of symptoms increases, but this effect was observed in only one of the experimental conditions. In the study by Classen and Owens (2010), simulator sickness severity was measured at three time points: before VR exposure, after a 5-min acclimation exposure and after a 20-min trial. The obtained results indicated that the simulator sickness severity increased between the baseline score and both after-acclimation and post-exposure, but no significant differences were discovered between the after-acclimation and post-exposure scores. Therefore, one may presume, that the peak simulator sickness severity in this study was reached very early. However, no data was gathered during the 20-min exposure, so it is possible that some differences might have been discovered if more systematic simulator sickness measurements had been conducted. A similar procedure was conducted by Sinitski et al. (2018) – they measured the simulator sickness severity (with the SSQ) before the exposure, after an acclimation period (which lasted for 15 min) and after a 45-min trial. In this study, however, only a small increase in the disorientation scale was observed after the acclimation period and these symptoms decreased by the end of the session. Again, the period between the second and the third measurement was quite long, and therefore it is impossible to

An experiment conducted by Moss et al. (2008) consisted of a short practice and five 2-min experimental trials. It was confirmed that the simulator sickness (measured with the SSQ) severity increases with time – it was more severe after the last (5th) trial than: before the practice, after the practice, after the 1st, 2nd, and 3rd trials. As no significant differences were discovered between the 4th and 5th trial, it may be hypothesized that after circa 9 min of exposure the simulator sickness has reached its peak severity and would not become more unpleasant if the exposure duration was even longer. In a similar study (Moss et al., 2011), a phenomenon of the simulator sickness severity (measured with the SSQ) increase with the increased VR exposure duration was confirmed. Serge and Moss (2015) measured simulator sickness severity with the Revised Simulator Sickness Questionnaire and proved that it does increase with time when measured before VR exposure and after 8 and 16 min of exposure. Singer et al. (1998) report as well that the simulator sickness severity increases with time during a VR exposure, although the difference between a "Mid-Experiment" and "Post-Experiment" scores was not significant, suggesting an appearance of a threshold simulator sickness level. The authors, however, did not give information on how long the trials were, and therefore any conclusions drawn from this study should be treated with caution. Feenstra et al. (2011) have discovered a slightly different phenomenon than the ones above described – in their study, the differences in simulator sickness severity began to become statistically significant after the participants spent 10 min in the VR and then it increased until the end of the 20-min trial.

A systematic increase of simulator sickness severity (measured with the SSQ) with time was confirmed by Chung et al. (2007), Park et al. (2008), and Choi et al. (2009) during a 60-min trial and Aldaba et al. (2017), who measured simulator sickness severity with the SSQ, and by Reinhard et al. (2017), who used the Fast Motion Sickness Scale (FMS – a single-item scale, the scores on which range from 0 to 20). An increase of simulator sickness symptoms severity was also observed by McCauley et al. (1990), when it was rated on a 7-point scale ("normal, symptom-free" – "severe discomfort, I am unable to continue") – it increased between measurement time points: before the exposure, in the middle of the 10-min task and after the whole 10-min task. There were 4 such trials and an increase in severity of the symptoms was observed for all of them. A brief summary of all reviewed studies is provided in the **Supplementary Table S1**.

Several conclusions can be drawn from the perspective of the temporal trajectory of the progression of simulator sickness on the basis of the studies retrieved. Firstly, there is empirical evidence to expect that severity of simulator sickness grows along time of exposure, as several studies using various approaches confirmed this hypothesis. In light of the reviewed research, this trend seems to be stable regardless of the technological progress in the field of VR presentation – the oldest studies (McCauley et al., 1990) and the most recent one (Sinitski et al., 2018) lead to the same conclusion. Even using between-subject comparisons leads – in most of the cases – to the conclusion that the severity of simulator sickness symptoms is greater when the exposure

thoroughly analyze the pattern of the symptoms during the whole exposure.

<sup>2</sup>Consisting of a small number of polygons.

duration is longer (e.g., Kennedy et al., 2000). However, it is important to note that several moderators, which are not the main focus of this paper, may play a role here – for example, a simulator control method. Secondly, it is difficult to establish a universal rule regarding the maximum time individuals can spend in VR on the basis of the analyzed study results. On the other hand, in most of the studies the simulator sickness symptoms were experienced by all of the participants, not only the ones who reported some kind of tendency to feel sick.

Moreover, in some of the studies it was observed that the simulator sickness severity increases with time, but after reaching a certain level or after a certain amount of time it either begins to decrease (Cobb et al., 1999; Sinitski et al., 2018) or remains on the same level (Singer et al., 1998; Lo and So, 2001; So et al., 2001; Moss et al., 2008; Classen and Owens, 2010). It can lead to a conclusion, that during a single VR exposure it is possible for some people to achieve simulator sickness adaptation (or, for some simulator types, to evoke the adaptation effect). However, it should be further explored whether this effect transfers to subsequent VR sessions.

On the other hand, it has also been proved that in some cases the simulator sickness symptoms begin to show after some time spent in VR and that this time threshold may be different for various simulator sickness symptoms (Min et al., 2004; Feenstra et al., 2011). Although this type of evidence is less prevalent than the one described above, it is also worth being taken into consideration. If the symptoms start being unpleasant after some time, a single VR session should be short enough to prevent these symptoms from occurring.

Keeping in mind several moderators which may vary between software (e.g., way of control, setting, graphics quality), another strategy of testing temporal tolerance may be reasonable, viz. testing of certain VR software using precisely selected methods. In order to make it possible, various methods need to be integrated, and standardized methodology needs to be developed.

# Possibility of Adapting VR Users in Advance