There have also been studies in gynaecology training programs that show laboratory-based training improves technical skills in a clinical setting. A core curriculum of intensive video laparoscopic skills training improved not only technical but operative performance among residents A prospective randomised trial by Coleman & Muller (49) recruited obstetrics and gynaecology residents (skills cohort, 11; control cohort, 7) to laboratory based training for laparoscopic salpingectomy for treatment of ectopic pregnancy, compared to routine surgical training in residency. The aim of the study was to determine the effect and validity of an intensive laboratory-based laparoscopic skills training curriculum on operative proficiency. Study components included a baseline questionnaire, video skills testing, intraoperative skills assessment and resident skills perception. The residents that were assigned to a laboratory based skills curriculum had significantly higher ratings when performing a laparoscopic salpingectomy on patients. This study demonstrated that a shortterm intensive laboratory-based video laparoscopic skills curriculum could translate into better individual operative proficiency. Banks et al (50) randomly assigned residents to a laboratory based surgical curriculum to teach laparoscopic tubal ligation versus routine surgical training. At baseline there were no differences in skills between the two groups. After completion of the curriculum, facility members blinded to the knowledge of which training the resident had received, assessed all residents in the operating room as to their ability to perform laparoscopic tubal ligation. Residents assigned to the simulation training

There is also good evidence to support the positive transfer of surgical skills after training with VR simulation. A recent study by Larsen et al (37) proved that skills in laparoscopic surgery could be increased in clinically relevant manner using proficiency based virtual reality simulator training. These researchers performed a prospective, randomized, observer blinded, controlled trial. A group of junior gynaecology registrars were divided into a control group and an intervention group (trained to proficiency on a VR simulator). The intervention group was given seven hours of training outside the normal service setting and was found to perform their first laparoscopy on a patient up to intermediate level (20-30 cases). The control group performed at a novice level (0-5 cases) and took twice as long to complete the procedure. The results showed that the performance of novices was increased to the level of intermediate experienced laparoscopists and the operation time to complete the task was halved. They were able to show that VR training in laparoscopic salpingectomy, compared with standard clinical education, was associated with a clinically important improvement of operative skills during the actual procedure. The learning curve in the OT was also shorter. By using simulator training it might be possible to bypass the early learning curve, which is known to be associated with a higher number of complications. These results also show that criterion based procedural training on a VR simulator can help compensate for reduced working hours by bringing trainees to a higher level of performance before they start training. They concluded that simulator training should be considered before trainees carry out laparoscopic procedures. This is possibly the first well-designed trial to show benefit from simulation in surgical training, and therefore Practice distribution refers to the schedule of practice that a trainee is given. 'Distributed practice' refers to a practice schedule in which periods of training are interspersed with rest periods; 'massed practice' refers to a continuous block of uninterrupted training (54,55). With regard to the effectiveness of laparoscopic simulator training, it is unclear whether it is preferable to undergo 'distributed' training or 'massed' training. Metaanalytic reviews indicated that distributed training resulted in better retention of motor skills than massed training, although this difference was dependent on the tasks trained (56). MacKay et al (55) examined the effect of practice distribution in the medical setting. Forty-one novice subjects were randomised into one of three groups to train on a VR simulator. Group A trained for 20 min continuously (n=14), group B trained for 20 min in 5 min blocks (n=14) and group C trained for 15 min in 5 min blocks (n=13). Post training, all groups had a rest followed by a retention test. The authors reported that distributed endoscopic training on MIST VR, with short breaks, was superior to continuous training within one single day (p=0.023), as determined by the retention test. A later study randomly assigned students with no endoscopic experience to distributed VR training on three consecutive days (n=10) or distributed training within 1 day (n=10). The training involved 12 repetitions of three different exercises in three differently distributed training schedules. All students performed a post-test on a VR simulator seven days after training; three technical parameters were measured. The group with training over several days performed faster (p=0.013), with the same number of errors and instrument path length used suggesting that 'rest' results in better consolidation of skills (54). It would appear that distributed training is more effective than massed training, and over several days rather than training on one day, potentially having implications for workshop based programmes.

## **12. Training methods**

Harold et al (57) compared two methods of instruction in a randomised fashion for the teaching of laparoscopic intracorporeal knot tying. The intervention group in this study received instruction by lecture, video, and individual proctoring, which was compared with instruction by manual alone. The intervention group performed better than the control group in this study. Participants in the intervention group had the advantage of not only better understanding through the use of video, but also the advantage of practice and proctoring, which allowed their understanding to be translated into performance.

Virtual Reality Simulation: A Valuable Adjunct to Surgical Training 177

deteriorated more than skills needed for easier tasks. Residents were less likely to fail with increasing experience. Stefanidis et al (39) noted that there was a paucity of literature on skill retention and comparison of the durability of skill between VR and VT simulators. Fourteen surgical residents of varying levels were enrolled to train on VR and VT simulators until proficiency levels were achieved. VR scores were generated automatically and VT scores were based on completion time. Skill retention was evaluated by performing one task on both the VR (manipulate diathermy) and VT (bean drop) simulators. Skill acquisition was similar for both systems (Improvement:VR 59% vs. VT 56%). Despite an early performance decrement (VR 45% vs. VT 17%) the acquired skill persisted over a sevenmonth follow-up period. There was no correlation of skill loss with resident level, duration of training or any of the other parameters. The authors concluded that proficiency-based

The variable anatomy and different degrees of difficulty in live patients makes consistent assessment of technical skills in surgical trainees difficult. Traditionally trainee surgeons have been assessed by an 'expert' colleague, a process which is subjective and potentially prone to bias. A 'gold standard' for OR performance does not exist. Although improvement in surgical skill is usually reported, the extent of the improvement is hard to quantify. Therefore it is difficult to establish the effectiveness of simulation. More recently, studies have demonstrated the value of VR simulators for providing an objective assessment tool (64-67). Smith et al (68) developed a skills assessment device (SAD) incorporating VR and VT technology to quantify both speed and accuracy during laparoscopic skill performance. Untrained subjects performed ten repetitions of a standardised laparoscopic task. Task time improved dramatically during the first three repetitions and then stabilised. However, accuracy continued to improve. The authors concluded that although the time to perform a laparoscopic task improved more quickly than the accuracy of task completion, time alone is poor indicator of technical skill as it fails to account for the more protracted learning curve

training on simulators results in durable skills, more so for VT than VR.

for accuracy. In their opinion, time was not a sufficient measure of proficiency.

subjective rating or time on a task, improve with practice (29,71,72).

Faster completion of a task does not presuppose accurate performance of the task (61,69). Both safety and accuracy need to be considered when assessing technical skills. Although a fast surgeon is not necessarily a safe surgeon, the idea that experience is related to greater efficiency of motion has face validity. Twenty-four subjects, with varying level of experience, were divided into three groups (naïve, junior, expert) depending on the volume of surgical experience. The results indicated that the 'time-error' scores are a valid measure of performance and improved significantly from baseline to final iteration in all groups. On all tasks, the 'expert' group performed significantly better than the naïve group (64). Furthermore, shorter times are indicative of familiarity and confidence with the instruments (28). Hyltander et al (70) found that students who performed tasks accurately also needed the least time. A technically skilled surgeon is one who executes a task quickly, is economical in movements as well as being precise (33). Studies have shown that performance, measured by either a

**14. Assessment of Skills**

Other recent randomised trials further reinforce the point that conceptual understanding and technical performance are both important elements of laparoscopic proficiency. Stefanidis et al (39) reported that a combination of video tutorials and limited feedback were the most efficient way to reach proficiency in a laparoscopic suturing curriculum. Korndorffer et al (58) reported that participants who received video and practiced performed better than those who received instruction by video alone. Leung et al (59) tested the efficacy of video as an educational tool in laparoscopic training. This RCT compared text versus video alone for a laparoscopic procedure. The results showed that video is superior to text alone for achieving quicker and better understanding and greater competency at performing laparoscopic tasks.

Snyder et al (60) randomised 36 medical students into independent or proctored training groups (n=18); no significant differences in demographics. Simulator proficiency was reached after a median of eleven hours of training (range 6-21 hrs.). Trainees in the independent group achieved proficiency with significantly fewer hours of training (HR 2.62; 95% CI, 1.01-6.85; p=0.048). The authors concluded that for proficiency-based VR simulator training, an independent approach was just as effective and potentially less time consuming for trainees than a proctored approach.

## **13. Skill retention**

It is widely accepted that laparoscopic skills improve after simulator training, however little is known regarding skill retention. Surgical competency depends on a combination of procedural knowledge and skill retention. A meta-analysis found that performance decay increased with longer retention intervals; a 92% skill loss at one year was documented (61). More recently, Maagaard et al (62) looked at two groups (novices and experts) who performed 10 sessions on the LapSim VR. Assessment of skill was based on time, economy of movement and error. The authors reported that, although novices showed retention of skills after 6 months, after 18 months, laparoscopic skills had returned to pre-training levels. Sinha et al (63) documented the retention of motor skills over time in 33 surgical residents who trained to established criteria (and passed an exam) on seven technical skills on a VR. Six months after training the residents underwent repeat testing. At retest, significantly more residents failed clip applying and cutting tasks (p<0.05). In failed tests, instrument and tissue handling skills deteriorated more than the speed with which a task was completed. Evidence of skill retention was present for some but not all tasks. Fine motor skills deteriorated more than skills needed for easier tasks. Residents were less likely to fail with increasing experience. Stefanidis et al (39) noted that there was a paucity of literature on skill retention and comparison of the durability of skill between VR and VT simulators. Fourteen surgical residents of varying levels were enrolled to train on VR and VT simulators until proficiency levels were achieved. VR scores were generated automatically and VT scores were based on completion time. Skill retention was evaluated by performing one task on both the VR (manipulate diathermy) and VT (bean drop) simulators. Skill acquisition was similar for both systems (Improvement:VR 59% vs. VT 56%). Despite an early performance decrement (VR 45% vs. VT 17%) the acquired skill persisted over a sevenmonth follow-up period. There was no correlation of skill loss with resident level, duration of training or any of the other parameters. The authors concluded that proficiency-based training on simulators results in durable skills, more so for VT than VR.

#### **14. Assessment of Skills**

176 Virtual Reality in Psychological, Medical and Pedagogical Applications

Harold et al (57) compared two methods of instruction in a randomised fashion for the teaching of laparoscopic intracorporeal knot tying. The intervention group in this study received instruction by lecture, video, and individual proctoring, which was compared with instruction by manual alone. The intervention group performed better than the control group in this study. Participants in the intervention group had the advantage of not only better understanding through the use of video, but also the advantage of practice and

Other recent randomised trials further reinforce the point that conceptual understanding and technical performance are both important elements of laparoscopic proficiency. Stefanidis et al (39) reported that a combination of video tutorials and limited feedback were the most efficient way to reach proficiency in a laparoscopic suturing curriculum. Korndorffer et al (58) reported that participants who received video and practiced performed better than those who received instruction by video alone. Leung et al (59) tested the efficacy of video as an educational tool in laparoscopic training. This RCT compared text versus video alone for a laparoscopic procedure. The results showed that video is superior to text alone for achieving quicker and better understanding and greater competency at

Snyder et al (60) randomised 36 medical students into independent or proctored training groups (n=18); no significant differences in demographics. Simulator proficiency was reached after a median of eleven hours of training (range 6-21 hrs.). Trainees in the independent group achieved proficiency with significantly fewer hours of training (HR 2.62; 95% CI, 1.01-6.85; p=0.048). The authors concluded that for proficiency-based VR simulator training, an independent approach was just as effective and potentially less time consuming

It is widely accepted that laparoscopic skills improve after simulator training, however little is known regarding skill retention. Surgical competency depends on a combination of procedural knowledge and skill retention. A meta-analysis found that performance decay increased with longer retention intervals; a 92% skill loss at one year was documented (61). More recently, Maagaard et al (62) looked at two groups (novices and experts) who performed 10 sessions on the LapSim VR. Assessment of skill was based on time, economy of movement and error. The authors reported that, although novices showed retention of skills after 6 months, after 18 months, laparoscopic skills had returned to pre-training levels. Sinha et al (63) documented the retention of motor skills over time in 33 surgical residents who trained to established criteria (and passed an exam) on seven technical skills on a VR. Six months after training the residents underwent repeat testing. At retest, significantly more residents failed clip applying and cutting tasks (p<0.05). In failed tests, instrument and tissue handling skills deteriorated more than the speed with which a task was completed. Evidence of skill retention was present for some but not all tasks. Fine motor skills

proctoring, which allowed their understanding to be translated into performance.

**12. Training methods**

performing laparoscopic tasks.

for trainees than a proctored approach.

**13. Skill retention**

The variable anatomy and different degrees of difficulty in live patients makes consistent assessment of technical skills in surgical trainees difficult. Traditionally trainee surgeons have been assessed by an 'expert' colleague, a process which is subjective and potentially prone to bias. A 'gold standard' for OR performance does not exist. Although improvement in surgical skill is usually reported, the extent of the improvement is hard to quantify. Therefore it is difficult to establish the effectiveness of simulation. More recently, studies have demonstrated the value of VR simulators for providing an objective assessment tool (64-67). Smith et al (68) developed a skills assessment device (SAD) incorporating VR and VT technology to quantify both speed and accuracy during laparoscopic skill performance. Untrained subjects performed ten repetitions of a standardised laparoscopic task. Task time improved dramatically during the first three repetitions and then stabilised. However, accuracy continued to improve. The authors concluded that although the time to perform a laparoscopic task improved more quickly than the accuracy of task completion, time alone is poor indicator of technical skill as it fails to account for the more protracted learning curve for accuracy. In their opinion, time was not a sufficient measure of proficiency.

Faster completion of a task does not presuppose accurate performance of the task (61,69). Both safety and accuracy need to be considered when assessing technical skills. Although a fast surgeon is not necessarily a safe surgeon, the idea that experience is related to greater efficiency of motion has face validity. Twenty-four subjects, with varying level of experience, were divided into three groups (naïve, junior, expert) depending on the volume of surgical experience. The results indicated that the 'time-error' scores are a valid measure of performance and improved significantly from baseline to final iteration in all groups. On all tasks, the 'expert' group performed significantly better than the naïve group (64). Furthermore, shorter times are indicative of familiarity and confidence with the instruments (28). Hyltander et al (70) found that students who performed tasks accurately also needed the least time. A technically skilled surgeon is one who executes a task quickly, is economical in movements as well as being precise (33). Studies have shown that performance, measured by either a subjective rating or time on a task, improve with practice (29,71,72).

## **15. Factors predicting performance**

In laparoscopic surgery it is important to develop the ability to use both hands equally well. Many basic laparoscopic skills demand dexterity in both hands for successful completion of the task. Powers et al (73) assessed whether hand dominance had any effect on performance in a laparoscopic skills curriculum. Twenty-seven surgical residents underwent a four-week laparoscopic skills curriculum with pre- and post testing on six tasks during weeks one and four. During week two and week three, residents attended proctored practice sessions. The authors concluded that participation improved overall performance. The left-handed surgeons demonstrated better initial performance, but post-test comparison showed no difference.

Virtual Reality Simulation: A Valuable Adjunct to Surgical Training 179

constraints. However, the benefits of simulator training seem to be greatest for the most inexperienced surgeons, in acclimatizing to the 2D environment, new instrumentation and the fulcrum effect (76). This would suggest that the ideal time to introduce the concept of surgical skills training to trainees is during their surgical attachments as a medical student. Bearing in mind that simulation is an adjunct to, not a replacement for, traditional methods

Some studies support the role of VR in surgical skills training; others support VT and claim that a greater improvement in skills acquisition occurs. Other studies show no difference between the two methods. Overall it is unclear which method is superior. The bottom line is that research has demonstrated that practice in surgical simulators leads to improved performance. Furthermore, there is evidence to show that simulator training translates into improved operative performance, but in a limited number of procedures. Further good quality studies are needed to strengthen the evidence base in support of the various types of surgical simulation, and to establish to what extent simulation should be part of the surgical

The increasing popularity of MAS makes it imperative that junior doctors have ample opportunity to master basic laparoscopic skills. At present, despite three decades of development, MAS training is still rather primitive. Worldwide, surgical simulators are playing an increasing role in the training of junior doctors. Evidence is increasing on the nature of the acquisition of surgical skill through the use of simulators rather than the traditional approach. The optimal timing and means of acquiring and retaining these skills

to ensure optimal transfer of skill to the operating room is unknown.

*Department of Gynaecology, King Edward Memorial Hospital, Subiaco, Perth, Australia*

*Department of Gynaecology, King Edward Memorial Hospital, Subiaco, Perth, Australia*

*Department of Gynaecology, King Edward Memorial Hospital, Subiaco, Perth, Australia*

*School of Women's and Infants' Health, University of Western Australia, King Edward Memorial*

*School of Women's and Infants' Health, University of Western Australia, King Edward Memorial*

of training. Supervision and feedback are essential (77).

**16. In summary**

training program.

**Author details**

Angela Beard

Roger Hart

Bernadette McElhinney

Krishnan Karthigasu

*Hospital, Subiaco, Perth, Australia*

*Hospital, Subiaco, Perth, Australia*

**17. In conclusions**

Grantcharov et al (74) assessed impact of gender, hand dominance and computer games experience on psychomotor skills demonstrated with a VR simulator. Male surgeons were faster; no significant difference between genders in the number of errors and unnecessary movements was noted. Right-handedness was associated with fewer unnecessary movements. Computer game users made fewer errors than non-users. In a study by Derossis et al (29), forty-two surgeons viewed an introductory video, and then were tested performing seven laparoscopic tasks. Performance was measured using a scoring system rewarding precision and speed. Each candidate repeated all seven tasks and was rescored. Significant predictors of overall performance were level of training (p=0.002), repetition (p=0.0001) and interaction between level of training and practice (0.001). Construct validity was demonstrated by measuring significant improvement in performance with increasing residency training, and with practice.

As well as dealing with the stress of live operating, surgical trainees have to deal with many other stressors including unfavourable working patterns, sleep deprivation, large volumes of work and time pressures, concerns about patient outcomes, surgical emergencies or complications, team challenges, miscommunication and so forth. Andreatta et al (75) demonstrated that simulation provides an opportunity for trainees to manage stress in practice. They observed 27 medical students completing tasks using a laparoscopic simulator under two conditions; direct observation (stressor) and unobserved (no stressor). A simple stimulus of an evaluator observing the completion of a task incurred a stress reaction in terms of elevated heart rate and increasing performance errors. This has implications for training and assessment in the simulated context in that stressors imposed on the learner in a simulated environment may help support the acquisition of stress management skills that are necessary in the applied clinical setting. Exactly how these stressors influence surgical performance is not well understood, but simulation could be used to teach the trainee how to manage stress by developing coping mechanisms early in their training.

A primary aim for trainees is to practice skills in a safe and non-threatening environment. Evidence is accumulating which demonstrates a positive learning curve and improved surgical skills after training on surgical simulators. The availability of surgical simulators means that they can be incorporated into the surgical training curricula, and enable learning curves to be consigned to skills laboratories, away from live patients. The implementation of laparoscopy into residency training is difficult to achieve because of time and financial constraints. However, the benefits of simulator training seem to be greatest for the most inexperienced surgeons, in acclimatizing to the 2D environment, new instrumentation and the fulcrum effect (76). This would suggest that the ideal time to introduce the concept of surgical skills training to trainees is during their surgical attachments as a medical student. Bearing in mind that simulation is an adjunct to, not a replacement for, traditional methods of training. Supervision and feedback are essential (77).

## **16. In summary**

178 Virtual Reality in Psychological, Medical and Pedagogical Applications

In laparoscopic surgery it is important to develop the ability to use both hands equally well. Many basic laparoscopic skills demand dexterity in both hands for successful completion of the task. Powers et al (73) assessed whether hand dominance had any effect on performance in a laparoscopic skills curriculum. Twenty-seven surgical residents underwent a four-week laparoscopic skills curriculum with pre- and post testing on six tasks during weeks one and four. During week two and week three, residents attended proctored practice sessions. The authors concluded that participation improved overall performance. The left-handed surgeons demonstrated better initial performance, but post-test comparison showed no difference.

Grantcharov et al (74) assessed impact of gender, hand dominance and computer games experience on psychomotor skills demonstrated with a VR simulator. Male surgeons were faster; no significant difference between genders in the number of errors and unnecessary movements was noted. Right-handedness was associated with fewer unnecessary movements. Computer game users made fewer errors than non-users. In a study by Derossis et al (29), forty-two surgeons viewed an introductory video, and then were tested performing seven laparoscopic tasks. Performance was measured using a scoring system rewarding precision and speed. Each candidate repeated all seven tasks and was rescored. Significant predictors of overall performance were level of training (p=0.002), repetition (p=0.0001) and interaction between level of training and practice (0.001). Construct validity was demonstrated by measuring significant improvement in performance with increasing

As well as dealing with the stress of live operating, surgical trainees have to deal with many other stressors including unfavourable working patterns, sleep deprivation, large volumes of work and time pressures, concerns about patient outcomes, surgical emergencies or complications, team challenges, miscommunication and so forth. Andreatta et al (75) demonstrated that simulation provides an opportunity for trainees to manage stress in practice. They observed 27 medical students completing tasks using a laparoscopic simulator under two conditions; direct observation (stressor) and unobserved (no stressor). A simple stimulus of an evaluator observing the completion of a task incurred a stress reaction in terms of elevated heart rate and increasing performance errors. This has implications for training and assessment in the simulated context in that stressors imposed on the learner in a simulated environment may help support the acquisition of stress management skills that are necessary in the applied clinical setting. Exactly how these stressors influence surgical performance is not well understood, but simulation could be used to teach the trainee how to manage stress by

A primary aim for trainees is to practice skills in a safe and non-threatening environment. Evidence is accumulating which demonstrates a positive learning curve and improved surgical skills after training on surgical simulators. The availability of surgical simulators means that they can be incorporated into the surgical training curricula, and enable learning curves to be consigned to skills laboratories, away from live patients. The implementation of laparoscopy into residency training is difficult to achieve because of time and financial

**15. Factors predicting performance**

residency training, and with practice.

developing coping mechanisms early in their training.

Some studies support the role of VR in surgical skills training; others support VT and claim that a greater improvement in skills acquisition occurs. Other studies show no difference between the two methods. Overall it is unclear which method is superior. The bottom line is that research has demonstrated that practice in surgical simulators leads to improved performance. Furthermore, there is evidence to show that simulator training translates into improved operative performance, but in a limited number of procedures. Further good quality studies are needed to strengthen the evidence base in support of the various types of surgical simulation, and to establish to what extent simulation should be part of the surgical training program.

## **17. In conclusions**

The increasing popularity of MAS makes it imperative that junior doctors have ample opportunity to master basic laparoscopic skills. At present, despite three decades of development, MAS training is still rather primitive. Worldwide, surgical simulators are playing an increasing role in the training of junior doctors. Evidence is increasing on the nature of the acquisition of surgical skill through the use of simulators rather than the traditional approach. The optimal timing and means of acquiring and retaining these skills to ensure optimal transfer of skill to the operating room is unknown.

## **Author details**

Bernadette McElhinney *Department of Gynaecology, King Edward Memorial Hospital, Subiaco, Perth, Australia*

Angela Beard *Department of Gynaecology, King Edward Memorial Hospital, Subiaco, Perth, Australia*

Krishnan Karthigasu *Department of Gynaecology, King Edward Memorial Hospital, Subiaco, Perth, Australia*

*School of Women's and Infants' Health, University of Western Australia, King Edward Memorial Hospital, Subiaco, Perth, Australia*

Roger Hart *School of Women's and Infants' Health, University of Western Australia, King Edward Memorial Hospital, Subiaco, Perth, Australia*

#### **18. References**

[1] Davies RJ, Hamdorf JM. Surgical Skills Training and the Role of Skills Centers. British Journal of Urology. 2003;91:3-4

Virtual Reality Simulation: A Valuable Adjunct to Surgical Training 181

[17] Feldman LS, Sherman V, Fried GM. Using simulators to assess laparoscopic

[18] Oostema JA, Abdel MP, Gould JC. Time-efficient laparoscopic skills assessment using

[19] Laguna MP, de Reijke TM, de la Rosette J. How Far Will Simulators be involved into

[20] Thompson JR, Leonard AC, Doarn CR, Roesch MJ, Broderick TJ. Limited value of haptics in virtual reality laparoscopic cholecystectomy training. Surgical Endoscopy.

[21] Salikini MW, Doarn CR, Kiehl N, Broderick TJ, Donovan JF, Gaitonde K (2010) The role of haptic feedback in laparoscopic training using the lapMentor 2. Journal of

[22] Panait L, Akkary E, Bell RL, Roberts KE, Dudrick SJ, Duffy AJ (2009) The Role of haptic feedback in laparoscopic simulation training. Journal of Surgical Research 156;312-16 [23] Van der Meijden OAJ & Schijven MP. The value of haptic feedback in conventional and robotic-assissted minimal invasive surgery and virtual reality training: a current

[25] Schlag PM, Moesta KT, Rakovsky S Graschew G. Telemedicine, The new must for

[26] Satava RM. Emerging Technologies for surgery in the 21st Century Arch Surgery, 1999;

[27] Gor M, McCloy R, Stone R, Smith A. Virtual Reality Laparoscopic Simulator for Assessment in Gynaecology. British Journal of Obstetrics and Gynaecology.

[28] Rosser JC, Rosser LE, Savalgi RS. Objective evaluation of a laparoscopic surgical skill program residents and senior surgeons. Archives of Surgery. 1998;133:657-661 [29] Derossis, AM, Fried GM, Abrahamowicz M, Sigman HH, Barkun JS, Meakins JL. Development of a model for training and evaluation of laparoscopic skills. The

[30] Van Dongen KW, van der Wal WA, Borel Rinkes IHM, Schijven MP, Broeders IAMJ. Virtual reality training for endoscopic surgery: voluntary or obligatory? Surgical

[31] Madan AQ, Frantzides CT, Tebbit C, Quiros RM. Participants opinion on laparoscopic training devices after a basic laparoscopic training course. The American Journal of

[32] Hamilton EC, Scott DJ, Fleming JB, Rege RV, Laycock RM, Bergen PC, Tesfay ST, Jones DB. Comparison of video trainer and virtual reality training systems on acquisition of

[33] Moorthy K, Munz Y, Sarker SK, Darzi A. Objective Assessment of Technical Skills in

[34] Kothari SN, Kaplan BJ, DeMaria EJ, Broderick TJ, Merrell MD. (2002) Training in laparoscopic suturing skills using a new computer based virtual reality simulator

competence: Ready for widespread use? Surgery. 2004;135:28-42

Training? Endourology. 2009;10:97-105

review. Surgical Endoscopy 2009;23:1180-1190

American Journal of Surgery. 1998;175:482-487

laparoscopic skills. Surgical Endoscopy. 2002;16:406-411

Surgery. British Medical Journal. 2003;327:1032-1037

Surgery, Arch Surg, 1999; 134: 1216-1221.

[24] Gorman PJ, Meier AH, Krummel TM. Arch Surg, 2009; 134: 1203-1208.

2011;25:1107-1114

134: 1197-1202.

2003;110:181-187

Endoscopy 2008;22:664-667

Surgery 2005;189:758-761

Endourology 24; 99-102

an augmented-reality simulator. Surgical Endoscopy. 2008;22:2621-2624

[17] Feldman LS, Sherman V, Fried GM. Using simulators to assess laparoscopic competence: Ready for widespread use? Surgery. 2004;135:28-42

180 Virtual Reality in Psychological, Medical and Pedagogical Applications

Gynecologic Laparoscopists. 2003;10:147-153

Human Reproduction. 2001;16:1473-1478

Obstetrics and Gynaecology. 1997;104:595-600

Chirurgiao et Gynaecologiae, 2001; 90 (2): 141-146

Journal of Urology. 2003;91:3-4

[1] Davies RJ, Hamdorf JM. Surgical Skills Training and the Role of Skills Centers. British

[2] Halm EA, Lee C, Chassin MR. Is volume related to outcome in healthcare? A systemic review and methodologic critique of the literature. Annals of Internal Medicine.

[3] Rogers RM, Julian TM. Training the Gynecologic Surgeon. Obstetrics and Gynecology.

[4] Reznick RK. Teaching and Testing Surgical Skills. The American Journal of Surgery.

[5] Shen C-C, Wu M-P, Kung F-T, Huang F-J, Hsieh C-H, Lan K-C, Huang E-Y, Hsu T-Y, Chang S-Y. Major complications associated with laparoscopic –assisted vaginal hysterectomy: ten-year experience. The Journal of the American Association of

[6] Chapron C, Cravello L, Chopin N, Kreiker G, Blanc B, Duboisson JB. Complications during set-up procedures for laparoscopy in gynecology: open laparoscopy does not reduce the risk of major complications. Acta Obstetricia et Gynecologica Scandinavica.

[7] Brosens I, Gordon A. Bowel Injuries during Gynaecological Laparoscopy: a

[8] Makinen J, Johansson J, Tomas C, Tomas E, Heinonen PK, Laatikainen T, Kauko M, Heikkinen A-M, Sjoberg J. Morbidity of 10110 hysterectomies by type of approach.

[9] Jansen FW, Kapiteyn K, Trimbos-Kemper T, Hermans J, Trimbos JB. Complications of Laparoscopy: a prospective multicentre observational study. British Journal of

[10] Chapron C, Querleu D, Mage G, MAdelenat P, Dubuisson JB, Audebert A, Erny R, Buhat MA. Complications of Gynaecologic Laparoscopy. Multicentre study of 7604 laparoscopies. Journal of Gynaecology, Obstetrics and Reproductive Biology.

[11] Chapron C, Querleu D, Bruhat MA. Complications of gynaecological laparoscopy. A

[12] Lenihan JP, Kovanda C, Seshadri-Kreaden U. What is the Learning Curve for Robotic Assisted Gynecologic Surgery? The Journal of Minimally Invasive Gynecology.

[13] Goff BA. Changing the Paradigm in Surgical education. Obstetrics and Gynaecology.

[14] Kaufmann Cr. (2001), Computers in surgical education and the operating room. Annals

[16] Krummel TM. Surgical simulation and virtual reality: the coming revolution. Annals of

French collaborative study. Gynaecological Endoscopy 1998;13:867-72

[15] Resnick RK. Teaching and testing clinical skills. Am J Surg, 2003; 165: 358-61.

multinational survey. Gynaecological Endoscopy, 2001;10:1441-1452

**18. References**

2002;137:511-520

2005, 105; 197-200

1993, 165;358-361

2003:82;1125–1129

1992;21:207-213

2008;15:589-594

2008; 112 (1): 328-332.

Surgery. 1998;228:635-637

(MIST-VR) provides results comparable to those with an established pelvic trainer system. Journal of Laparoendoscopic & Advanced Surgical Techniques, 2002; 12: (3): 167-169.

Virtual Reality Simulation: A Valuable Adjunct to Surgical Training 183

[50] Banks EH, Chudnoff S, Karmin I, Wang C, Pardanani S. Does a surgical simulator improve resident operative performance of laparoscopic tubal ligation? Am J Obstst

[51] Sturm LP, Windsor JA, Cosman PH, Cregan P, Hewett PJ, Maddern GJ. A Systematic Review of skills transfer after surgical simulation training. Annals of Surgery.

[52] Gurusamy K, Aggarwal R, Palanivelu L, Davidson BR. Systemic review of randomised controlled trials on the effectiveness of virtual reality training for laparoscopic surgery.

[53] Paisley AM, Baldwin PJ, Paterson-Brown S. Validity of surgical simulation for the

[54] Verdaasdonk EGG, Stassen LPS, van Wijk RPJ, Dankelman J. The Influence of different training schedules on the learning of psychomotor skills for endoscopic surgery.

[55] MacKay S, Morgan P, Datta V, Chang A, Darzi A. Practice distribution in procedural skills training: a randomised controlled trial. Surgical Endoscopy. 2002;16:957-961 [56] Lee TD & Genovese ED. Distribution of practice in motor skill acquisition: learning and

[57] Harold KL, Matthews BD, Backus CL, Pratt BL, Heniford BT. Prospective randomised evaluation of surgical resident proficiency with laparoscopic suturing after course

[58] Korndorffer JR, Dunn JB, Sierra DR, Stefanidis D, Touchard C, Scott D. Simulator training for laparoscopic suturing using performance goals translates to the operating

[59] Yeung P, Justice T, Paya Pasic R. Comparison of text versus video for teaching laparoscopic knot tying in the novice surgeon: A randomised controlled trial. The

[60] Snyder CW, Vandromme MJ, Tyra SL, Hawn MT. Proficiency-based laparoscopic and endoscopic training with virtual reality simulators: a comparison of proctored and

[61] Arthur W, Bennett WJ, Stanush PL. Factors that influence skill decay and retention: a

[62] Maagaard M, Sorensen JL, Oestergaard J, Dalsgaard T, Grantcharov TP, Ottesen BS, Larsen CR. Retention of laparoscopic procedural skills acquired on a virtual-reality

[63] Sinha P, Hogle NJ, Fowler DL. Do the laparoscopic skills of trainees deteriorate over

[64] Sherman V, Feldman LS, Stanbridge D, Kazmi R, Fried GM. Assessing the learning curve for the acquisition of laparoscopic skills on a virtual reality simulator. Surgical

[65] Grantcharov TP, Rosenberg J, Pahle E, Funch-Jensen PM. An objective method for the evaluation of laparoscopic surgical skills. Surgical Endoscopy. 2001;15:242-244

independent approaches. Journal of Surgical Education. 2009;66:201-207

quantitative review and analysis. Human Performance. 1998;11:57-101

assessment of operative skill. British Journal of Surgery. 2001;88:1525-1533

performance effects reconsidered. Research Quarterly. 1988;59:277-287

Gynecol , 2007; 197: 541, e1-5.

British Journal of Surgery. 2008, 95; 1088-1097

instruction. Surgical Endoscopy. 2002;16:1729-1731

room. The American Journal of Surgery. 2005;201:23-29

surgical trainer. Surgical Endoscopy 2011;25:722-727

time? Surgical Endoscopy. 2008;22:2018-2025

Endoscopy. 2005;19:678-682

Journal of Minimally Invasive Gynaecology. 2009;16:411-415

Surgical Endoscopy. 2007;21:214-219

2008;248:166-178

[50] Banks EH, Chudnoff S, Karmin I, Wang C, Pardanani S. Does a surgical simulator improve resident operative performance of laparoscopic tubal ligation? Am J Obstst Gynecol , 2007; 197: 541, e1-5.

182 Virtual Reality in Psychological, Medical and Pedagogical Applications

randomised controlled trial. BMJ. 2009;338:1-6

competence: Ready for widespread use? Surgery. 2003;135:28-72

167-169.

Issue 1. Art. No: CD006575

Reviews. 2007, 3:CD006575

Endoscopy. 2001;15:242-244

Endoscopy. 1999;13:1077-1081

2002;186:836-842

378-385.

(MIST-VR) provides results comparable to those with an established pelvic trainer system. Journal of Laparoendoscopic & Advanced Surgical Techniques, 2002; 12: (3):

[35] Haque S & Srinivasan S. A meta-analysis of the training effectiveness of virtual reality simulators. IEEE Transactions on Information Technology in Biomedicine. 2006;10:51-58 [36] Gurusamy KS, Aggarwal R, Palanivelu L, Davidson BR. Virtual Reality training for surgical trainees in laparoscopic surgery. Cochrane Database of Systemic Reviews. 2009,

[37] Larsen CR, Soerensen JL, Grantcharov TP, Dalsgaard T, Schouenborg L, Ottosen C, Schroeder TV, Ottesen BS. Effect of virtual reality training on laparoscopic surgery:

[38] Feldman LS, Sherman V, Fried GM. Using simulators to assess laparoscopic

[39] Stefanidis D, Korndoffer JR, Sierra R, Touchard C, Dunne B, Scott DJ. Skill retention following proficiency based laparoscopic simulator training. Surgery. 2005;165:165-170 [40] Munz Y, Kumar BD, Moorthy K, Bann, Darzi A. Laparoscopic Virtual Reality and box

[41] Sutherland LM, Middleton PF, Anthony A, Hamdorf J, Cregan P, Scott D, Maddern GJ. Surgical Simulation A systematic Review. Annals of Surgery. 2006;243:291-300 [42] Gurusamy KS, Aggarwal R, Palanivelu L, Davidson BR. Virtual Reality Training for Surgical Trainees in Laparoscopic Surgery. The Cochrane Database of Systemic

[43] Youngblood PL, Srivastava S, Curet M, Heinrichs WL, Dev P, Wren SM. Comparison of training on two laparoscopic simulators and assessment of skills transfer to surgical

[44] Matsumoto ED, Hamstra SJ, Radomski SB, Cusimano MD. The effect of bench model fidelity on endourological skills: a randomized controlled study,. Am J Surg, 2003; 185:

[45] Goff BA, VanBlaricom A, Mandel L, Chinn M, Nielsen P. Comparison of objective structured assessment of technical skills with virtual reality hysteroscopy trainer and

[46] Grantcharov TP, Rosenberg J, Pahle E, Funch-Jensen PM. Virtual reality computer simulation-an objective method for evaluation of laparoscopic surgical skills. Surgical

[47] Fried GM, Derossis AM, Bothwell J, Sigman HH. Comparison of laparoscopic performance in vivo with performance measured in a laparoscopic simulator. Surgical

[48] Seymour NE, Gallagher AG, Roman SA, O'Brien MK, Bansal VK, Andersen DK, Satava RM. Virtual Reality Training Improves Operating Room performance: Results of a

[49] Coleman RL & Muller CY. Effects of laboratory-based skills curriculum on laparoscopic proficiency: A randomised trial. American Journal of Obstetrics and Gynaecology.

Randomized, Double-Blinded Study. Annals of Surgery. 2002;236:458-464

performance. Journal of the American College of Surgeons. 2005;200:546-551

standard latex hysteroscopy model. J Reprod Med, 2007: 52: 407-12.

trainers: is one superior to the other? Surgical Endoscopy. 2004;18:485-494

	- [66] Grantcharov TP, Bardram L, Funch-Jensen P, Rosenberg J. Learning curves and impact of previous operative experience on performance on a virtual reality simulator to test laparoscopic surgical skills. The American Journal of Surgery. 2003;185:146-149

**Chapter 9**

© 2012 McConville, licensee InTech. This is an open access chapter distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/3.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

© 2012 The Author(s). Licensee InTech. This chapter is distributed under the terms of the Creative Commons Attribution License http://creativecommons.org/licenses/by/3.0), which permits unrestricted use, distribution,

**Virtual Rehabilitation and Training for**

Kristiina M. Valter McConville

http://dx.doi.org/10.5772/50133

**1. Introduction**

Additional information is available at the end of the chapter

concept of VR guided balance and fitness training.

**Postural Balance and Neuromuscular Control**

Virtual Reality (VR) has been used in an increasingly wide range of applications, and technological advances have made it accessible to the everyday consumer. Virtual reality games have become the norm in today's consumer marketplace. This makes it possible to consider such devices for more productive use of leisure time, such as in exercise and health maintenance. Gaming systems such as the *Nintendo® wii*-*fit™* have already introduced the

Studies have shown that virtual reality has potential for assisting recovery from illnesses such as stroke and vestibular disorders and improvements in cerebral palsy. Custom designed VR systems in the laboratory have been successful with balance training [1]. Active video game systems, like the *Sony PlayStation® 2* with the *Sony EyeToy®*, a camera accessory, have also been used in studies of rehabilitation. Flynn et al. used this system in a case study of a stroke patient and documented the types of training for each game [2]. The *Nintendo® wii™* system was designed for in-home training as well as gaming. This system was tested on a 13-year-old patient with cerebral palsy who showed improvements in visual perceptual processing, postural control and functional mobility [3]. Therefore, these gaming systems

Other custom-designed games and virtual environments have also been developed to address specific disabilities and user requirements. Why do these systems work? In order to answer this question, the basics of human postural balance control have to be considered. The balance control system is a complex integration of multiple sensory inputs along with feedback from the motor component. The sensory inputs include visual, vestibular, auditory, tactile, and proprioceptive signals. The motor aspect is the "plant", driven by neuromuscular control. Figure 1 demonstrates the complex interaction between the various sensory systems in the central processing of incoming signals. Different inputs are assigned weights based on motor learning and experience, i.e. the reliability of the incoming cues.

and reproduction in any medium, provided the original work is properly cited.

show promise in rehabilitation, and have the added benefit of entertainment.

## **Virtual Rehabilitation and Training for Postural Balance and Neuromuscular Control**

Kristiina M. Valter McConville

Additional information is available at the end of the chapter

http://dx.doi.org/10.5772/50133

## **1. Introduction**

184 Virtual Reality in Psychological, Medical and Pedagogical Applications

skills. The American Journal of Surgery. 2001;181:547-550

laparoscopic skills. Surgical Endoscopy. 1998;12:1111-1116

surgery. Archives of Surgery. 1997;132:200-204

laparoscopy. Surgical Endoscopy. 2003;17:1082-1085

Training. American Journal of Surgery. 2001;182:137-142

Surgery. 2000;180:208-211

2005;19:673-677

Journal of Surgery. 2004;91:1549-1558

[66] Grantcharov TP, Bardram L, Funch-Jensen P, Rosenberg J. Learning curves and impact of previous operative experience on performance on a virtual reality simulator to test

[68] Smith CD, Farrell TM, McNatt SS, Metreveli RE. Assessing laparoscopic manipulative

[69] Aggarwal R, Moorthy K, Darzi A. Laparoscopic Skills Training and Assessment. British

[70] Hyltander A, Liljegren E, Rhodin PH, Lonroth H. The transfer of basic skills learned in a laparoscopic simulator to the operating room. Surgical Endoscopy. 2002;16:1324-1328 [71] Chung JY & Sackier JM. A method of objectively evaluating improvements in

[72] Rosser JC, Rosser LE, Savalgi RS. Skill Acquisition and Assessment for Laparoscopic

[73] Powers TW, Brentem DJ, Nagle AP, Toyama MT, Murphy SA, Murayama KM. Hand dominance and performance in a laparoscopic skills curriculum. Surgical Endoscopy.

[74] Grantcharov TP, Bardram L, Funch-Jensen P, Rosenberg J. Impact of hand dominance, gender, and experience with computer games on performance in virtual reality

[75] Andreatta PB, Woodrum DT, Birkmeyer JD, Yellamanchilli RK, Doherty GM, Gauger PG, Minter RM. Laparoscopic skills are improved with Lap Mentor training: results of a

[76] Scott DJ, William NY, Tesfay ST, Frawley WH, Rege RV, Jones DB. Laparoscopic Skills

[77] Undre S & Darzi A. Laparoscopy Simulators. Journal of Endourology. 2007;21:274-279

randomised, double-blinded study. Annals of Surgery. 2006;243:854-860

laparoscopic surgical skills. The American Journal of Surgery. 2003;185:146-149 [67] Jordan J, Gallagher, AG, McGuigan J, McGlade K, McClure N. A Comparison between Randomly Alternating Imaging, Normal Laparoscopic Imaging, and Virtual Reality Training in Laparoscopic Psychomotor Skill Acquisition. The American Journal of

> Virtual Reality (VR) has been used in an increasingly wide range of applications, and technological advances have made it accessible to the everyday consumer. Virtual reality games have become the norm in today's consumer marketplace. This makes it possible to consider such devices for more productive use of leisure time, such as in exercise and health maintenance. Gaming systems such as the *Nintendo® wii*-*fit™* have already introduced the concept of VR guided balance and fitness training.

> Studies have shown that virtual reality has potential for assisting recovery from illnesses such as stroke and vestibular disorders and improvements in cerebral palsy. Custom designed VR systems in the laboratory have been successful with balance training [1]. Active video game systems, like the *Sony PlayStation® 2* with the *Sony EyeToy®*, a camera accessory, have also been used in studies of rehabilitation. Flynn et al. used this system in a case study of a stroke patient and documented the types of training for each game [2]. The *Nintendo® wii™* system was designed for in-home training as well as gaming. This system was tested on a 13-year-old patient with cerebral palsy who showed improvements in visual perceptual processing, postural control and functional mobility [3]. Therefore, these gaming systems show promise in rehabilitation, and have the added benefit of entertainment.

> Other custom-designed games and virtual environments have also been developed to address specific disabilities and user requirements. Why do these systems work? In order to answer this question, the basics of human postural balance control have to be considered. The balance control system is a complex integration of multiple sensory inputs along with feedback from the motor component. The sensory inputs include visual, vestibular, auditory, tactile, and proprioceptive signals. The motor aspect is the "plant", driven by neuromuscular control. Figure 1 demonstrates the complex interaction between the various sensory systems in the central processing of incoming signals. Different inputs are assigned weights based on motor learning and experience, i.e. the reliability of the incoming cues.

© 2012 McConville, licensee InTech. This is an open access chapter distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/3.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. © 2012 The Author(s). Licensee InTech. This chapter is distributed under the terms of the Creative Commons Attribution License http://creativecommons.org/licenses/by/3.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Virtual Rehabilitation and Training for Postural Balance and Neuromuscular Control 187

Other inputs include auditory and tactile modalities. Auditory signals provide orientation cues based on sound localization. Tactile cues provide pressure feedback and external inputs regarding contact with surfaces. Proprioceptive inputs are also important, providing us cues about our body position in space, through joint angle feedback signals. These signals are directly relevant to the control of ankles, knees, hips and upper body to maintain balance. Furthermore, the arms can be used for stabilization control and fall

In this chapter, we will investigate both commercial and custom-designed virtual reality interfaces for balance training and rehabilitation, considering the various inputs that are important for balance control. We can then begin to evaluate the important elements of virtual reality games, so that they can be developed more effectively for rehabilitation

Arm movements have been shown to be very important in balance maintenance and recovery [7]. Both unilateral and bilateral arm movements generate anticipatory and reactive responses in postural leg muscles [8] and hip and trunk muscles [9]. Furthermore, reaching and grasping actions are important in balance recovery and

In a recent study we showed the importance of free arm movements in a series of classical balance and mobility tests in which three out of four tests showed significant improvement

3. Timed get-up-and-go test (time to get up from a chair, walk to a target, and return to

Only test (4) did not show signficant improvement, possibly because there is less change of support motion of the body relative to the task of walking, compared to launching a single large or elevated stepping motion, and certainly compared to rising from a chair. These

Arm movements are affected by stroke and progressive disease, such as Parkinson's disease (PD). Given the importance of arm movements for balance, such an effect could result in decreased balancing ability or steadiness of stance in these patients. Arm movement training is thus an important part of rehabilitation. We have developed a virtual reality arm movement training system [11], in which different games can be programmed to suit user abilities and interests. This system uses the *Microsoft Kinect™* interface, along with pressure sensors attached to the index finger and thumb as an object selection tool. This system shows the actual image of the user on the screen, and we have developed games, which appear overlaid

1. Maximal step length test (length and timing of a single maximally long step) 2. Step test (number of steps up and down onto an elevated surface in one minute)

results show the importance of arm movements in controlling body balance.

**2. Arm movement importance and potential training system**

prevention.

applications.

prevention of falls [7].

sitting)

with the free use of arm movements [10]:

4. Walk along an elliptical line

**Figure 1.** Proposed comparator model of postural control: Sensory inputs are compared to each other for consistency, and a matrix of gains [V,C,P] is assigned accordingly. The weighted inputs are combined as S(t) which is the total sensed self-orientation x as a function of time t, with noise η(t) added. This results in a compensatory motor output through body dynamics. Feedback is provided through the senses, but also internally to update a memory or database of previously experienced combinations of sensory inputs. In addition to comparing the sensory inputs to each other, this database is also referenced.

Visual inputs are generally dominant among human sensory inputs for balance control as well as a general source of information, and virtual reality systems have evolved with an emphasis on visual displays. Balance improvement has been demonstrated through both static and dynamic visual stimuli in VR systems. [4] This results in a natural fit for visualmodality systems for both cognitive and neuromuscular control training along with entertainment value. However, other channels of information cannot be discounted, including the vestibular system, which is located in the inner ear and senses head acceleration.

The visual and vestibular systems are very closely linked, for example eye movements respond to vestibular inputs. Since the vestibular system senses head movements, it can be hypothesized that head movements together with visual stimuli should be used in training the balance control system. [5] Cohen et al. showed that purposeful head movement exercises improved balance in vestibular patients. [6] Therefore active games that track player head and body movements could demonstrate potential for balance improvement.

Other inputs include auditory and tactile modalities. Auditory signals provide orientation cues based on sound localization. Tactile cues provide pressure feedback and external inputs regarding contact with surfaces. Proprioceptive inputs are also important, providing us cues about our body position in space, through joint angle feedback signals. These signals are directly relevant to the control of ankles, knees, hips and upper body to maintain balance. Furthermore, the arms can be used for stabilization control and fall prevention.

In this chapter, we will investigate both commercial and custom-designed virtual reality interfaces for balance training and rehabilitation, considering the various inputs that are important for balance control. We can then begin to evaluate the important elements of virtual reality games, so that they can be developed more effectively for rehabilitation applications.

## **2. Arm movement importance and potential training system**

Arm movements have been shown to be very important in balance maintenance and recovery [7]. Both unilateral and bilateral arm movements generate anticipatory and reactive responses in postural leg muscles [8] and hip and trunk muscles [9]. Furthermore, reaching and grasping actions are important in balance recovery and prevention of falls [7].

In a recent study we showed the importance of free arm movements in a series of classical balance and mobility tests in which three out of four tests showed significant improvement with the free use of arm movements [10]:

186 Virtual Reality in Psychological, Medical and Pedagogical Applications

is also referenced.

acceleration.

improvement.

**Figure 1.** Proposed comparator model of postural control: Sensory inputs are compared to each other for consistency, and a matrix of gains [V,C,P] is assigned accordingly. The weighted inputs are combined as S(t) which is the total sensed self-orientation x as a function of time t, with noise η(t) added. This results in a compensatory motor output through body dynamics. Feedback is provided through the senses, but also internally to update a memory or database of previously experienced combinations of sensory inputs. In addition to comparing the sensory inputs to each other, this database

Visual inputs are generally dominant among human sensory inputs for balance control as well as a general source of information, and virtual reality systems have evolved with an emphasis on visual displays. Balance improvement has been demonstrated through both static and dynamic visual stimuli in VR systems. [4] This results in a natural fit for visualmodality systems for both cognitive and neuromuscular control training along with entertainment value. However, other channels of information cannot be discounted, including the vestibular system, which is located in the inner ear and senses head

The visual and vestibular systems are very closely linked, for example eye movements respond to vestibular inputs. Since the vestibular system senses head movements, it can be hypothesized that head movements together with visual stimuli should be used in training the balance control system. [5] Cohen et al. showed that purposeful head movement exercises improved balance in vestibular patients. [6] Therefore active games that track player head and body movements could demonstrate potential for balance Only test (4) did not show signficant improvement, possibly because there is less change of support motion of the body relative to the task of walking, compared to launching a single large or elevated stepping motion, and certainly compared to rising from a chair. These results show the importance of arm movements in controlling body balance.

Arm movements are affected by stroke and progressive disease, such as Parkinson's disease (PD). Given the importance of arm movements for balance, such an effect could result in decreased balancing ability or steadiness of stance in these patients. Arm movement training is thus an important part of rehabilitation. We have developed a virtual reality arm movement training system [11], in which different games can be programmed to suit user abilities and interests. This system uses the *Microsoft Kinect™* interface, along with pressure sensors attached to the index finger and thumb as an object selection tool. This system shows the actual image of the user on the screen, and we have developed games, which appear overlaid on the user's image. This allows the user to reach out to the target letters and "select" them by making contact between the thumb and index finger. Initial testing on healthy subjects provided a baseline scores, so that we can now proceed to evaluate patients' performance.

Virtual Rehabilitation and Training for Postural Balance and Neuromuscular Control 189

## **3. Balance training:** *Sony PlayStation® 2* **with the** *Sony EyeToy®* **and the Harmonix** *AntiGrav™* **game [12]**

Based on the importance of head and arm movements along with visual inputs in balance training, the Harmonix *AntiGrav™* game was selected for evaluation. The purpose of this study was to to characterize balance-related motor learning effects through game performance and to evaluate the effectiveness and specificity of the video game for balance training in healthy subjects. [12]

#### **3.1. Game description**

*Sony's EyeToy®* used video capture technology, in which it imaged the user with a small video camera mounted above the display and used face-recognition and motion-tracking technologies (no body markers were required). [13, 14] The system used twodimensional user motion in the frontal plane as inputs. Each user was individually calibrated before each session, to account for height differences and range of head motion. During this procedure, the subject's own image was shown on the screen overlaid with a calibration figure, instructions and controls. The subject had to move to a position such that his/her face was centered on the figure, and then was instructed to wave his/her hand over the "lock" control. In the subsequent screen, the subject was required to bend to the left and right such that the head was aligned with target positions. If this procedure was not properly executed, it became very apparent in the early stages of game-play that the game was not responsive (the avatar movements did not respond to user movements), and the calibration was repeated. A diagram of the system and screenshots is shown in Figure 2.

The user was portrayed in the game as an avatar and the user's image was not shown after the calibration procedure. The avatar raced on a hoverboard along roadways and rails, over jumps, and when airborne, through rings. The user, standing in front of the display, used head movements to guide the avatar, and was required to jump or duck to avoid obstacles. The user also had to reach with either arm or both to strike targets as the avatar passed them. The arm motions were directed diagonally up or down or to the side, requiring additional head/upper body motions to contact the targets. The game was fast-paced and challenging, and included sound effects and "life or death" consequences. It presented unpredictable situations requiring maintenance of balance and decision making at the same time. As the user gained experience in the game, anticipation of upcoming movements became easier. Upon achieving a pre-set score, the user was advanced to playing both difficulty Level 1 and Level 2 in each training session.

**Harmonix** *AntiGrav™* **game [12]**

system and screenshots is shown in Figure 2.

difficulty Level 1 and Level 2 in each training session.

training in healthy subjects. [12]

**3.1. Game description**

on the user's image. This allows the user to reach out to the target letters and "select" them by making contact between the thumb and index finger. Initial testing on healthy subjects provided a baseline scores, so that we can now proceed to evaluate patients' performance.

**3. Balance training:** *Sony PlayStation® 2* **with the** *Sony EyeToy®* **and the**

Based on the importance of head and arm movements along with visual inputs in balance training, the Harmonix *AntiGrav™* game was selected for evaluation. The purpose of this study was to to characterize balance-related motor learning effects through game performance and to evaluate the effectiveness and specificity of the video game for balance

*Sony's EyeToy®* used video capture technology, in which it imaged the user with a small video camera mounted above the display and used face-recognition and motion-tracking technologies (no body markers were required). [13, 14] The system used twodimensional user motion in the frontal plane as inputs. Each user was individually calibrated before each session, to account for height differences and range of head motion. During this procedure, the subject's own image was shown on the screen overlaid with a calibration figure, instructions and controls. The subject had to move to a position such that his/her face was centered on the figure, and then was instructed to wave his/her hand over the "lock" control. In the subsequent screen, the subject was required to bend to the left and right such that the head was aligned with target positions. If this procedure was not properly executed, it became very apparent in the early stages of game-play that the game was not responsive (the avatar movements did not respond to user movements), and the calibration was repeated. A diagram of the

The user was portrayed in the game as an avatar and the user's image was not shown after the calibration procedure. The avatar raced on a hoverboard along roadways and rails, over jumps, and when airborne, through rings. The user, standing in front of the display, used head movements to guide the avatar, and was required to jump or duck to avoid obstacles. The user also had to reach with either arm or both to strike targets as the avatar passed them. The arm motions were directed diagonally up or down or to the side, requiring additional head/upper body motions to contact the targets. The game was fast-paced and challenging, and included sound effects and "life or death" consequences. It presented unpredictable situations requiring maintenance of balance and decision making at the same time. As the user gained experience in the game, anticipation of upcoming movements became easier. Upon achieving a pre-set score, the user was advanced to playing both Virtual Rehabilitation and Training for Postural Balance and Neuromuscular Control 189

Virtual Rehabilitation and Training for Postural Balance and Neuromuscular Control 191

c) Screenshot of the game in progress. The avatar seen in the middle of the screen is riding on a "rail", which constrains the path while presenting targets for arm movements. While elsewhere, head movements steer the avatar, here head movements are required to lean toward targets (if presented on one side) to be struck with rapid arm movements and to duck or jump over obstacles. The lower right corner has an indicator of the user's head and arm positions in real time. These are also reflected in the avatar actions. In the center, the accumulated points are shown along with the target number of points to beat. The time remaining is shown in the top left corner. The bar on the left side indicates the current

d) Screenshot of the game in progress. Here the user has successfully struck a pair of targets with both

The control (no training, n=7) and experimental (video game training, n=7) groups completed the balance tests (as described below) at the beginning and at the end of a threeweek period. Both the control group and the experimental group were free to conduct their daily activities as normal, and thus, the control group controlled for the effect of the specific experimental video game training. The experimental group trained 9 times during the three-

Game performance was measured by the number of tokens or targets struck with arm movements and the speed and accuracy in navigating the avatar through the race course using head movements. Additional points could be achieved by doing tricks, executed by arm movements while airborne. The scores were provided after each game, and points were displayed when awarded, which reinforced the immediate feedback provided during game

Three balance tests (Balance Board (BB), Tandem Romberg (TR), and One-leg Standing (OL)) were selected to evaluate transfer of the video game training to the real world. These tests evaluated transfer of learning to the real world because they were conducted without the virtual environment and thus tested for changes in balance outside of the game. Many balance-related measures were found in the literature ranging from the Dynamic Gait Index to the Sensory Organization Test to various functional measures and clinical tests. Our tests were selected based on common usage and relevance to overall balance [15-17] rather than

The average scores from two trials for each test were used. To minimize a "ceiling effect" (which occurs if most subjects pass a balance test given for a specified time, e.g. 30 seconds), the time was recorded when balance was lost. This gave a finer measure to detect and

As expected, the game scores in both levels of difficulty improved over the 9 training sessions. The performance in the game was based on the game scores in both levels of

hands, and the user has been given a credit of extra time to complete the race. [12]

week period, and the control group received no training. For details, see [12].

factors such as sensory contribution or functional outcomes such as locomotion.

position along the racetrack.

**3.2. Experimental protocol**

play.

quantify improvement.

**3.3. Results**

*3.3.1. Game scores*

**Figure 2.** a) Conceptual schema of the *PlayStation®2*, *EyeToy®* and hoverboard game (not an actual screenshot) with the user.

b) Screenshot of the first step in calibration. The actual image of the user is shown superimposed (off to the side in this picture to highlight the two images) on an outline of a human frame with an opening at the face.

c) Screenshot of the game in progress. The avatar seen in the middle of the screen is riding on a "rail", which constrains the path while presenting targets for arm movements. While elsewhere, head movements steer the avatar, here head movements are required to lean toward targets (if presented on one side) to be struck with rapid arm movements and to duck or jump over obstacles. The lower right corner has an indicator of the user's head and arm positions in real time. These are also reflected in the avatar actions. In the center, the accumulated points are shown along with the target number of points to beat. The time remaining is shown in the top left corner. The bar on the left side indicates the current position along the racetrack.

d) Screenshot of the game in progress. Here the user has successfully struck a pair of targets with both hands, and the user has been given a credit of extra time to complete the race. [12]

#### **3.2. Experimental protocol**

190 Virtual Reality in Psychological, Medical and Pedagogical Applications

**Figure 2.** a) Conceptual schema of the *PlayStation®2*, *EyeToy®* and hoverboard game (not an actual

b) Screenshot of the first step in calibration. The actual image of the user is shown superimposed (off to the side in this picture to highlight the two images) on an outline of a human frame with an opening at

screenshot) with the user.

the face.

The control (no training, n=7) and experimental (video game training, n=7) groups completed the balance tests (as described below) at the beginning and at the end of a threeweek period. Both the control group and the experimental group were free to conduct their daily activities as normal, and thus, the control group controlled for the effect of the specific experimental video game training. The experimental group trained 9 times during the threeweek period, and the control group received no training. For details, see [12].

Game performance was measured by the number of tokens or targets struck with arm movements and the speed and accuracy in navigating the avatar through the race course using head movements. Additional points could be achieved by doing tricks, executed by arm movements while airborne. The scores were provided after each game, and points were displayed when awarded, which reinforced the immediate feedback provided during game play.

Three balance tests (Balance Board (BB), Tandem Romberg (TR), and One-leg Standing (OL)) were selected to evaluate transfer of the video game training to the real world. These tests evaluated transfer of learning to the real world because they were conducted without the virtual environment and thus tested for changes in balance outside of the game. Many balance-related measures were found in the literature ranging from the Dynamic Gait Index to the Sensory Organization Test to various functional measures and clinical tests. Our tests were selected based on common usage and relevance to overall balance [15-17] rather than factors such as sensory contribution or functional outcomes such as locomotion.

The average scores from two trials for each test were used. To minimize a "ceiling effect" (which occurs if most subjects pass a balance test given for a specified time, e.g. 30 seconds), the time was recorded when balance was lost. This gave a finer measure to detect and quantify improvement.

#### **3.3. Results**

#### *3.3.1. Game scores*

As expected, the game scores in both levels of difficulty improved over the 9 training sessions. The performance in the game was based on the game scores in both levels of difficulty. All subjects except one achieved the target score for Level 1 in the first session, and were moved to training at both Levels 1 and 2 by the second session. Overall, there was a gradual increase in the total scores for both Levels over the period of the experiment.

#### *3.3.2. Model of motor learning through game scores*

The improvement in game scores was consistent with our common understanding of improved performance with practice and with studies of motor learning. [18] The game scores were averaged across subjects and fit to an exponential given in Equations 1 and 2 for Level 1 and Level 2 games respectively [19]:

$$y(n) = 57.500 \text{--} 41.000 \text{e}^{-0.26n} \tag{1}$$

Virtual Rehabilitation and Training for Postural Balance and Neuromuscular Control 193

Romberg tests. The BB and TR tests were discontinued after three and two minutes respectively because of the potential for muscle fatigue, which would interfere with the measure of balancing ability. [20] Of the 7 experimental subjects in each group (training and control), three control subjects reached the maximum time in the BB pre-test and one control

Figure 4 shows the balancing times in seconds averaged for training and control subjects for each test. Trained subjects showed significant improvement in the BB test, moderate improvement in the TR test, and no significant difference in the OL test, while control

Virtual Reality games have the potential for simulator sickness, which is related to motion sickness, with many of the same symptoms. Kennedy et al. have established a Simulator Sickness Questionnaire (SSQ) based on work with the US Navy, which is often used in evaluating simulator sickness. [21] It contains 16 items on which the subjects rated their responses (ranging on a 4 point scale; 0 (none), 1 (slight), 2 (moderate), 3

In this study, simulator sickness symptoms were recorded in every training session. Figure 5 shows the incidence of all symptoms across the 7 subjects and 9 training sessions. Most subjects who experienced symptoms reported slight sweating. No severe symptoms were present, and the symptoms subsided over the course of the training sessions. The presence of the symptoms suggested potential for vestibular rehabilitation, and their mildness

The results showed that game scores improved over the sessions as expected. The average game scores were modeled, and showed an almost linearly increasing component followed by a plateau. Motor learning analysis showed that Level 1 was mastered by the participants while Level 2 was still showing learning. We developed a game difficulty parameter, which could be used in rehabilitation game design to provide appropriate increases in difficulty for

Pre- and post-test results showed that subjects who played the game had an improvement in balance board and tandem Romberg tests over the 9 sessions. One-legged standing tests did not show improvement, which could be the result of training specificity, since the training

The simulator sickness questionnaire showed the presence of several symptoms, especially in the earliest sessions. These were mild and decreased over the training period. The symptoms indicated that the game had a physiological effect on the participants, and the

subjects did not show significant difference on any of the tests.

demonstrated the safety of this system for the general public.

subject in the TR pre-test.

*3.3.4. Simulator sickness*

(severe)).

**3.4. Discussion**

continued learning and training.

was not performed on one leg.

decrease in symptoms paralleled the motor learning.

$$y(n) = \text{57}\_{\text{.}}500 \text{--} 43 \text{.} 500 \text{e}^{\cdot 0.087n} \tag{2}$$

where *n* is the session number and *y* is the curve fit value. All three variables were initially used to obtain curve fits. It was found that constraining the DC term to be the same for both curves did not affect the quality of the fit, and it represented the average performance plateau after a large number of sessions. These curves are shown together with the averaged data in Figure 3. The curve fits reflected the increase and plateau of motor learning as a function of the session number for the first level (Level 1), and suggested that the subjects had not yet mastered Level 2 because of the absence of the plateau.

**Figure 3.** Averaged scores over all 7 training subjects for Level 1 and Level 2 training along with corresponding exponential curve fits (see text). [12]

#### *3.3.3. Balance tests*

Performance on balance tests improved over the training period for the training subjects as compared to controls. During pilot trials, it was found that some volunteers were balancing for longer than three minutes on the balance board and two minutes on the Tandem Romberg tests. The BB and TR tests were discontinued after three and two minutes respectively because of the potential for muscle fatigue, which would interfere with the measure of balancing ability. [20] Of the 7 experimental subjects in each group (training and control), three control subjects reached the maximum time in the BB pre-test and one control subject in the TR pre-test.

Figure 4 shows the balancing times in seconds averaged for training and control subjects for each test. Trained subjects showed significant improvement in the BB test, moderate improvement in the TR test, and no significant difference in the OL test, while control subjects did not show significant difference on any of the tests.

#### *3.3.4. Simulator sickness*

192 Virtual Reality in Psychological, Medical and Pedagogical Applications

*3.3.2. Model of motor learning through game scores*

Level 1 and Level 2 games respectively [19]:

corresponding exponential curve fits (see text). [12]

( Balance tests*

difficulty. All subjects except one achieved the target score for Level 1 in the first session, and were moved to training at both Levels 1 and 2 by the second session. Overall, there was a gradual increase in the total scores for both Levels over the period of the experiment.

The improvement in game scores was consistent with our common understanding of improved performance with practice and with studies of motor learning. [18] The game scores were averaged across subjects and fit to an exponential given in Equations 1 and 2 for

 *y(n)* = 57,500-41,000e-0.26*n* (1)

 *y(n)* = 57,500-43,500e-0.087*n* (2)

where *n* is the session number and *y* is the curve fit value. All three variables were initially used to obtain curve fits. It was found that constraining the DC term to be the same for both curves did not affect the quality of the fit, and it represented the average performance plateau after a large number of sessions. These curves are shown together with the averaged data in Figure 3. The curve fits reflected the increase and plateau of motor learning as a function of the session number for the first level (Level 1), and suggested that the subjects

> Average 1 Average 2

Fit 1 Fit 2

**Figure 3.** Averaged scores over all 7 training subjects for Level 1 and Level 2 training along with

123456789

Training Session

Performance on balance tests improved over the training period for the training subjects as compared to controls. During pilot trials, it was found that some volunteers were balancing for longer than three minutes on the balance board and two minutes on the Tandem

had not yet mastered Level 2 because of the absence of the plateau.

Virtual Reality games have the potential for simulator sickness, which is related to motion sickness, with many of the same symptoms. Kennedy et al. have established a Simulator Sickness Questionnaire (SSQ) based on work with the US Navy, which is often used in evaluating simulator sickness. [21] It contains 16 items on which the subjects rated their responses (ranging on a 4 point scale; 0 (none), 1 (slight), 2 (moderate), 3 (severe)).

In this study, simulator sickness symptoms were recorded in every training session. Figure 5 shows the incidence of all symptoms across the 7 subjects and 9 training sessions. Most subjects who experienced symptoms reported slight sweating. No severe symptoms were present, and the symptoms subsided over the course of the training sessions. The presence of the symptoms suggested potential for vestibular rehabilitation, and their mildness demonstrated the safety of this system for the general public.

#### **3.4. Discussion**

The results showed that game scores improved over the sessions as expected. The average game scores were modeled, and showed an almost linearly increasing component followed by a plateau. Motor learning analysis showed that Level 1 was mastered by the participants while Level 2 was still showing learning. We developed a game difficulty parameter, which could be used in rehabilitation game design to provide appropriate increases in difficulty for continued learning and training.

Pre- and post-test results showed that subjects who played the game had an improvement in balance board and tandem Romberg tests over the 9 sessions. One-legged standing tests did not show improvement, which could be the result of training specificity, since the training was not performed on one leg.

The simulator sickness questionnaire showed the presence of several symptoms, especially in the earliest sessions. These were mild and decreased over the training period. The symptoms indicated that the game had a physiological effect on the participants, and the decrease in symptoms paralleled the motor learning.

Virtual Rehabilitation and Training for Postural Balance and Neuromuscular Control 195

Dizzy (eyes open) Dizzy (eyes closed) Fullness of head Fatigue Eye strain Headache General discomfort Sweating

b) Time that the subjects were able to maintain balance in the Tandem Romberg (TR) test Pre- and Posttraining. The performance for the training subjects (n=7), mean=41 and 102 s, is compared to the performance of the control subjects (n=7) who did not receive training during the 3-week period; mean=59 and 64 s. The trained subjects had a significant difference in balancing times (p<0.05). c) Time that the subjects were able to maintain balance in the One-Leg (OL) standing test Pre- and Posttraining. The performance for the training subjects (n=7), mean=15 and 24 s, is compared to the performance of the control subjects (n=7) who did not receive training during the 3-week period; mean=25 and 19 s. The differences in balancing times were not significant for either group. [12]

**Figure 5.** Simulator Sickness symptom occurrences over 7 subjects and 9 training sessions during a three week period. Some subjects had more than one symptom on a given training day, and some

123456789 **Training Session Number**

The auditory system also provides important cues to an individual's self-orientation in space due to sound-localization capability. This ability is based on the difference in arrival time of a sound to each ear. We have used this concept in a prototype balance training aid, in which both visual and auditory feedback were provided to four healthy subjects. The system was tested with the subjects standing on a balance board. The auditory feedback was provided through stereo headphones worn by the participant. Both the Inter-aural Intensity Difference and the Inter-aural Timing Difference were controlled based on accelerometer signals from the tilt of the balance board. The combined audio-visual feedback resulted in significant improvement in the subjects' balancing ability [22]. Future studies will test for

Tactile inputs can be utilized in VR systems with the increasing availability of haptic feedback. Multimodal feedback increases the sense of immersion in the virtual

the relative contribution of each of the two feedback modalities.

subjects reported no symptoms.

**4. Auditory inputs**

0

2

4

6

**Number of Subjects Reporting Symptoms**

8

10

12

**5. Tactile inputs**

**Figure 4.** a) Time that the subjects were able to maintain balance on a Balance Board (BB) Pre- and Posttraining. The performance for the training subjects (n=7), mean=67 and 142 s, is compared to the performance of the control subjects (n=7) who did not receive training during the 3-week period; mean=97 and 90 s. The trained subjects had a significant difference in balancing times (p<0.005).

b) Time that the subjects were able to maintain balance in the Tandem Romberg (TR) test Pre- and Posttraining. The performance for the training subjects (n=7), mean=41 and 102 s, is compared to the performance of the control subjects (n=7) who did not receive training during the 3-week period; mean=59 and 64 s. The trained subjects had a significant difference in balancing times (p<0.05). c) Time that the subjects were able to maintain balance in the One-Leg (OL) standing test Pre- and Posttraining. The performance for the training subjects (n=7), mean=15 and 24 s, is compared to the performance of the control subjects (n=7) who did not receive training during the 3-week period; mean=25 and 19 s. The differences in balancing times were not significant for either group. [12]

**Figure 5.** Simulator Sickness symptom occurrences over 7 subjects and 9 training sessions during a three week period. Some subjects had more than one symptom on a given training day, and some subjects reported no symptoms.

## **4. Auditory inputs**

194 Virtual Reality in Psychological, Medical and Pedagogical Applications

**Average Balancing Time (s)**

**Average Balancing Time (s)**

**Average Balancing Time (s)**

Balance Board Test

Control Training

Pre-Test Post-Test

Control Training

Pre-Test Post-Test

Control Training

Pre-Test Post-Test

(b)

**One-Legged Standing Test**

(a)

Tandem Romberg Test

**Figure 4.** a) Time that the subjects were able to maintain balance on a Balance Board (BB) Pre- and Posttraining. The performance for the training subjects (n=7), mean=67 and 142 s, is compared to the performance of the control subjects (n=7) who did not receive training during the 3-week period; mean=97 and 90 s. The trained subjects had a significant difference in balancing times (p<0.005).

(c)

The auditory system also provides important cues to an individual's self-orientation in space due to sound-localization capability. This ability is based on the difference in arrival time of a sound to each ear. We have used this concept in a prototype balance training aid, in which both visual and auditory feedback were provided to four healthy subjects. The system was tested with the subjects standing on a balance board. The auditory feedback was provided through stereo headphones worn by the participant. Both the Inter-aural Intensity Difference and the Inter-aural Timing Difference were controlled based on accelerometer signals from the tilt of the balance board. The combined audio-visual feedback resulted in significant improvement in the subjects' balancing ability [22]. Future studies will test for the relative contribution of each of the two feedback modalities.

### **5. Tactile inputs**

Tactile inputs can be utilized in VR systems with the increasing availability of haptic feedback. Multimodal feedback increases the sense of immersion in the virtual environment, and different sensory inputs can be harnessed for sensory substitution, additional information channels and most importantly, redundancy. Redundancy is a means of bringing important signals to the attention of the user, and alerting the user in emergencies. For this reason, we considered it important to include tactile feedback in the design of an alerting system to inform subjects about potential loss of balance.

Virtual Rehabilitation and Training for Postural Balance and Neuromuscular Control 197

The arms were selected for the vibration transducers to naturally produce a reaction in the appropriate arm, and by eliminating unnecessary processing steps, the arm movement could be rapid enough to help stabilize posture. An example result from one subject is shown in Figure 7.

**Figure 7.** The subject was perturbed in the antero-posterior direction, and left and right body tilt were recorded with and without the use of the WVRS system. In both directions, the amount of body tilt was

Such a system can be used as a VR rehabilitation tool, utilizing multimodal feedback with the visual display and providing progress tracking with the data recording capability. It is also anticipated that in the next design phase, the recording could also be housed in the main module, making the system portable. Such a wearable device could be worn as an everyday balance maintenance aid. This is an example of taking the virtual reality into the

In addition to head/body and arm movements, balance also depends on the quality of control in the joints of the lower limbs. In a balance training program, these joints can be involved in general training, or targetted specifically. When a joint injury occurs, for example in the knee

We have demonstrated ankle VR rehabilitation systems in which ankle movements in up to two degrees of freedom can be used to control a custom-designed VR game. The program was motivated by a sports injury, which can often take a long time to heal without properly designed rest and training. The games tracked the progress of the training and could be used as an inexpensive home-based training tool that motivates exercise adherence. The

a. the training has a range of difficulty levels to suit different patient abilities, including

b. the system has the ability to constrain movement in both magnitude and direction so

c. the game provides visual feedback through a game, which provides scores of the ankle

that specific muscle groups can be trained without the risk of re-injury, and

or ankle, then a phased and progressive rehabilitation program is essential.

isometric contraction when no joint rotation is desired

control ability related to game performance.

reduced. [23]

real world, or augmented reality.

system had the following objectives [24]:

**6. Ankle training**

We developed a Wireless Vibrotactile Rehabilitation System (WVRS), which provides vibration feedback during body tilt. [23] The system incorporates three vibrating disks on each arm, which are activated when an accelerometer, mounted on the mid-back detects body tilt within certain ranges. Figure 6 shows the design of the system, which includes a wireless link to a computer. The system can record and display the accelerometer signals. The three vibrating disks on each arm are activated as a function of the degree of body tilt and tilt direction.

**Figure 6.** System design of a vibrating tactile balance feedback system. a) The main module includes a tri-axial accelerometer, Arduino microprocessor, and ZigBee wireless system, incorporating a Wireless Personal Area Network ID to distinguish the communication with the arm subsystems and the communication with the host computer. b) A system of three vibrating disks was provided on each arm, activated in each arm as shown, based on the amount of body tilt in each direction respectively. The vibration was applied at 200 Hz. [23]

The arms were selected for the vibration transducers to naturally produce a reaction in the appropriate arm, and by eliminating unnecessary processing steps, the arm movement could be rapid enough to help stabilize posture. An example result from one subject is shown in Figure 7.

**Figure 7.** The subject was perturbed in the antero-posterior direction, and left and right body tilt were recorded with and without the use of the WVRS system. In both directions, the amount of body tilt was reduced. [23]

Such a system can be used as a VR rehabilitation tool, utilizing multimodal feedback with the visual display and providing progress tracking with the data recording capability. It is also anticipated that in the next design phase, the recording could also be housed in the main module, making the system portable. Such a wearable device could be worn as an everyday balance maintenance aid. This is an example of taking the virtual reality into the real world, or augmented reality.

## **6. Ankle training**

196 Virtual Reality in Psychological, Medical and Pedagogical Applications

and tilt direction.

vibration was applied at 200 Hz. [23]

environment, and different sensory inputs can be harnessed for sensory substitution, additional information channels and most importantly, redundancy. Redundancy is a means of bringing important signals to the attention of the user, and alerting the user in emergencies. For this reason, we considered it important to include tactile feedback in the

We developed a Wireless Vibrotactile Rehabilitation System (WVRS), which provides vibration feedback during body tilt. [23] The system incorporates three vibrating disks on each arm, which are activated when an accelerometer, mounted on the mid-back detects body tilt within certain ranges. Figure 6 shows the design of the system, which includes a wireless link to a computer. The system can record and display the accelerometer signals. The three vibrating disks on each arm are activated as a function of the degree of body tilt

**Figure 6.** System design of a vibrating tactile balance feedback system. a) The main module includes a tri-axial accelerometer, Arduino microprocessor, and ZigBee wireless system, incorporating a Wireless Personal Area Network ID to distinguish the communication with the arm subsystems and the

(b)

(a)

communication with the host computer. b) A system of three vibrating disks was provided on each arm, activated in each arm as shown, based on the amount of body tilt in each direction respectively. The

design of an alerting system to inform subjects about potential loss of balance.

In addition to head/body and arm movements, balance also depends on the quality of control in the joints of the lower limbs. In a balance training program, these joints can be involved in general training, or targetted specifically. When a joint injury occurs, for example in the knee or ankle, then a phased and progressive rehabilitation program is essential.

We have demonstrated ankle VR rehabilitation systems in which ankle movements in up to two degrees of freedom can be used to control a custom-designed VR game. The program was motivated by a sports injury, which can often take a long time to heal without properly designed rest and training. The games tracked the progress of the training and could be used as an inexpensive home-based training tool that motivates exercise adherence. The system had the following objectives [24]:

These objectives have been satisfied with the incorporation of either a balance board simulating a 2 degree-of-freedom foot pedal interface [25], or a boot with a frame and locking pins that either prevent motion, or constrain motion to the plantarflexion/dorsiflexion plane [26]. Force sensors and accelerometers measured the muscle activity and ankle motion and provided inputs to the game. Visual interfaces were developed for calibration, as shown in Figure 8. Game scores were tracked and recorded across the training sessions, so that the patient and also the caregiver can monitor progress.

Virtual Rehabilitation and Training for Postural Balance and Neuromuscular Control 199

**Figure 9.** Instrumented harness providing input to the VR training system. Accelerometers attached to the shoulder harness are positioned on the trainee's shoulder blades, and the belt contains a third

The user can view the image of the avatar reflecting actual measured values from his/her upper body during the lifting motion. The discrepancy between the ideal and actual lifting motions is displayed, which guides the trainee to improve his/her lifting technique. Such a VR training aid provides instant visual feedback to allow the user to learn the body

Different virtual reality training environments were evaluated or developed for improving postural balance and neuromuscular control. Balance training was shown to transfer to the real world from commercial video-game based training, and because balance is related to overall fitness, these training interfaces represent an important means of combining leisure and health maintenance. The results of such training could reduce the likelihood of falls and

The benefits of VR training include the ability to isolate specific sensory inputs and motor outputs. Clearly, VR also provides immediate feedback to the user, which enhances learning. Further, VR provides an automated way to guide and track the progress of the training. Adaptive systems can sense the user's performance and adjust parameters to provide the appropriate level of difficulty in each training session. Finally, VR provides a

relatively safe environment for training compared to many real-world environments.

proprioceptive feedback signals associated with the correct postural positioning.

accelerometer, which is positioned on the lumbar region of the low back. [29]

**8. Conclusions**

injury in everyday activities.

**Figure 8.** Windows interface for accelerometer calibration [25]

The important side-benefit of such a visual gaming interface is the motivation to improve scores, and the entertainment value of the game. Sayenko et al. have shown that a customdesigned VR game provides useful feedback and motivation for a spinal cord-injured patient to prevent muscle atrophy through functional electrical stimulation training. [27] The game encouraged adherence to the exercise program. It is important to conduct further studies on patients to develop optimal training protocols.

### **7. Occupational health and safety: Training for correct lifting posture**

Very specific aspects of postural balance can also be isolated and trained through VR. In the field of occupational safety, VR can be a very important training tool and real-time aid in injury prevention. We have developed a lifting technique trainer, in which an avatar shows the correct motion on the screen, based on published positioning guidelines. [28] The trainee lifts a training object wearing a harness instrumented with accelerometers as shown in Figure 9.

Virtual Rehabilitation and Training for Postural Balance and Neuromuscular Control 199

**Figure 9.** Instrumented harness providing input to the VR training system. Accelerometers attached to the shoulder harness are positioned on the trainee's shoulder blades, and the belt contains a third accelerometer, which is positioned on the lumbar region of the low back. [29]

The user can view the image of the avatar reflecting actual measured values from his/her upper body during the lifting motion. The discrepancy between the ideal and actual lifting motions is displayed, which guides the trainee to improve his/her lifting technique. Such a VR training aid provides instant visual feedback to allow the user to learn the body proprioceptive feedback signals associated with the correct postural positioning.

## **8. Conclusions**

198 Virtual Reality in Psychological, Medical and Pedagogical Applications

**Figure 8.** Windows interface for accelerometer calibration [25]

studies on patients to develop optimal training protocols.

These objectives have been satisfied with the incorporation of either a balance board simulating a 2 degree-of-freedom foot pedal interface [25], or a boot with a frame and locking pins that either prevent motion, or constrain motion to the plantarflexion/dorsiflexion plane [26]. Force sensors and accelerometers measured the muscle activity and ankle motion and provided inputs to the game. Visual interfaces were developed for calibration, as shown in Figure 8. Game scores were tracked and recorded across the training sessions, so that the patient and also the caregiver can monitor progress.

The important side-benefit of such a visual gaming interface is the motivation to improve scores, and the entertainment value of the game. Sayenko et al. have shown that a customdesigned VR game provides useful feedback and motivation for a spinal cord-injured patient to prevent muscle atrophy through functional electrical stimulation training. [27] The game encouraged adherence to the exercise program. It is important to conduct further

**7. Occupational health and safety: Training for correct lifting posture**

object wearing a harness instrumented with accelerometers as shown in Figure 9.

Very specific aspects of postural balance can also be isolated and trained through VR. In the field of occupational safety, VR can be a very important training tool and real-time aid in injury prevention. We have developed a lifting technique trainer, in which an avatar shows the correct motion on the screen, based on published positioning guidelines. [28] The trainee lifts a training Different virtual reality training environments were evaluated or developed for improving postural balance and neuromuscular control. Balance training was shown to transfer to the real world from commercial video-game based training, and because balance is related to overall fitness, these training interfaces represent an important means of combining leisure and health maintenance. The results of such training could reduce the likelihood of falls and injury in everyday activities.

The benefits of VR training include the ability to isolate specific sensory inputs and motor outputs. Clearly, VR also provides immediate feedback to the user, which enhances learning. Further, VR provides an automated way to guide and track the progress of the training. Adaptive systems can sense the user's performance and adjust parameters to provide the appropriate level of difficulty in each training session. Finally, VR provides a relatively safe environment for training compared to many real-world environments.

## **Author details**

Kristiina M. Valter McConville *Electrical and Computer Engineering, Ryerson University, Institute of Biomaterials and Biomedical Engineering, University of Toronto, Canada Toronto Rehabilitation Institute, Toronto, Canada*

## **Acknowledgement**

I would like to thank Dr. Ken Norwich, University of Toronto for his insight and wisdom and endless encouragement. I am also grateful to all of my students who have helped me immensely in this work, especially Matija Milosevic and Sumandeep Virk. I also thank the Natural Sciences and Engineering Research Council of Canada for funding the research.

Virtual Rehabilitation and Training for Postural Balance and Neuromuscular Control 201

[12] McConville KMV, Virk S. Evaluation of an Electronic Video Game for Improvement of

[13] Weiss P, Rand D, Katz N, Kizony R. Video capture virtual reality as a flexible and

[14] Wang S, Xiong X, Xu Y, Wang C, Zhang W, Dai X, Zhang D. Face-tracking as an augmented input in video games: enhancing presence, role-playing and control. SIGCHI Conference on Human Factors in Computing Systems. 22-27 April, 2006,

[15] Quesada PM, Durham MP, Topp RV, Swank AM, Biton D. Quantitative assessment of balance performance on a passively unstable surface. Occupat Ergonom 2007;7(1) 3-

[16] Vereeck L, Wuyts F, Truijen S, Van de Heyning P. Clinical Assessment of Balance,

[17] Curb JD, Ceria-Ulep CD, Rodriguez BL, Grove J, Guralnik J, Willcox BJ, Donlon TA, Masaki KH, Chen R. Performance-based measures of physical function for high-

[18] Newell KM, Liu Y-T, Mayer-Kress G. Time scales in motor learning and development.

[19] McConville KMV, Virk S. Motor learning in a virtual environment for vestibular rehabilitation. Proceedings of the 3rd Int IEEE EMBS Conference on Neural

[21] Kennedy RS, Lane NE, Berbaum KS, Lilienthal ML. Simulator sickness questionnaire: an enhanced method for quantifying simulator sickness. Int J Aviat Psychol 1993;3 203-

[22] Milosevic M, McConville KMV. Audio-visual biofeedback system for postural control.

[23] Tino A, Carvalho M, Preto NF, McConville KMV. Wireless Vibrotactile Feedback System for Postural Response Improvement. IEEE Eng Med Biol Soc Conf. 30 Aug – 3

[24] Holme E, Magnuson SP, Beeher K, Bieler T, Aagaard P, Kjarr M. The effect of supervised rehabilitation on strength, postural sway, position sense and re-injury risk

[25] Ching A, Guirguis R. Haptic Interface For Virtual Reality Training Game Using The Ankle. Thesis. Electrical and Computer Engineering Ryerson University Toronto;

[26] Battaglia B, Brukson A, Nassery M. Ankle Rehabilitation. Thesis. Electrical and

[27] Sayenko DG, Masani K, Milosevic M, Robinson MF, Vette AH, McConville KMV, Popovic MR. Video game-based functional electrical stimulation system for calf muscle

after acute ankle ligament sprain. Scand J Med Sci Sports 1999;9 104-9.

Engineering. 2-5 May, 2007, Kohala Coast, Hawaii. IEEE Press; 2007. p600-3. [20] Gandevia SC. Spinal and Supraspinal Factors in Human Muscle Fatigue. Physiol Rev

Normative data and Gender and Age Effects. Int J Audiol 2008;47 67-75.

Balance. Virtual Reality 2012;In Press. DOI: 10.1007/s10055-012-0212-7.

effective rehabilitation tool. J NeuroEng Rehabil 2004;1-12.

function populations. Am Geriatr Soc 2006;54(5) 737-42.

Psychol Rev 2001;108(1) 57-82.

Int J Disabil Hum Dev 2011;10(4) 321-4.

Sept., 2011, Boston, MA. IEEE Press; 2011. p5203 – 6.

Computer Engineering Ryerson University Toronto; 2012.

training: A case study. Med Eng Phys 2011;33(2) 249-55.

2001;81(4) 1725-89.

10.

20.

2009.

Montréal, Québec, Canada. New York: ACM; 2006. p1097-1106.

## **9. References**

[12] McConville KMV, Virk S. Evaluation of an Electronic Video Game for Improvement of Balance. Virtual Reality 2012;In Press. DOI: 10.1007/s10055-012-0212-7.

200 Virtual Reality in Psychological, Medical and Pedagogical Applications

*Electrical and Computer Engineering, Ryerson University,*

rehabilitation. Asst Technol 2004;16(1) 27-35.

cerebral palsy. Phys Ther 2008;88(10) 1196-207.

*Toronto Rehabilitation Institute, Toronto, Canada*

*Institute of Biomaterials and Biomedical Engineering, University of Toronto, Canada*

I would like to thank Dr. Ken Norwich, University of Toronto for his insight and wisdom and endless encouragement. I am also grateful to all of my students who have helped me immensely in this work, especially Matija Milosevic and Sumandeep Virk. I also thank the Natural Sciences and Engineering Research Council of Canada for funding the research.

[1] Keshner AE, Kenyon VR. Using immersive technology for postural research and

[2] Flynn S, Palma P, Bender A. Feasibility of using the Sony Playstation 2 gaming platform for an individual poststroke: a case report. J Neurol Phys Ther 2007;31 180-89. [3] Deutsch JE, Borbely M, Filler J, Huhn K, Guarrera-Bowlby P. Use of a low-cost, commercially available gaming console (Wii) for rehabilitation of an adolescent with

[4] Suárez H, Suárez A, Lavinsky L. Postural adaptation in elderly patients with instability and risk of falling after balance training using a virtual-reality system. Int Tinnitus J

[6] Cohen H, Kane-Wineland M, Miller LV, Hatfield CL. Occupation and visual/vestibular interaction in vestibular rehabilitation. Otolaryngol. - Head & Neck Surgery 1995;112(4)

[7] Maki BE, McIlroy WE. Control of rapid limb movements for balance recovery: agerelated changes and implications for fall prevention. Age Ageing 2006;35(Suppl 2) ii12-

[8] Mochizuki G, Ivanova TD, Garland SJ. Postural muscle activity during bilateral and unilateral arm movements at different speeds. Exp Brain Res 2004;155 352-61. [9] Yamazaki Y, Suzuki M, Ohkuwa T, Itoh H. Maintenance of upright standing posture during trunk rotation elicited by rapid and asymmetrical movements of the arms. Brain

[10] Milosevic M, McConville KMV, Masani M. Arm Movement Improves Performance in

[11] Seevaratnam S, Attalla R, El-Falou A, Valter McConville KM. Arm Biofeedback System for Parkinson's Disease Patients. Inst Biomat Biomed Eng Sci Day. 7 May, 2012.

Clinical Balance and Mobility Tests. Gait Pos 2011;33(3) 507-9.

[5] Foster CA. Vestibular Rehabilitation. Baillieres Clin Neurol 1994;3(3) 577-92.

**Author details**

Kristiina M. Valter McConville

**Acknowledgement**

**9. References**

2006;12(1) 41-4.

Res Bull 2005;67 30-9.

University of Toronto; 2012. p12.

526-32.

ii18.

	- [28] Genaidy, A. M., & Houshyar, A. Biomechanical Tolerance Limits For Manual Lifting Tasks: A Tool To Control Back Injuries. IEEE Eng Med Biol Soc Conf. 9-12 Nov., 1989, Seattle, WA. IEEE Press; 1989. 3:803-5.

**Chapter 10**

© 2012 Chou et al., licensee InTech. This is an open access chapter distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/3.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

and reproduction in any medium, provided the original work is properly cited.

© 2012 The Author(s). Licensee InTech. This chapter is distributed under the terms of the Creative Commons Attribution License http://creativecommons.org/licenses/by/3.0), which permits unrestricted use, distribution,

**Applications of Virtual Reality Technology**

Virtual reality is an evolving technology that allows for the possibility of interactive environments with stereoscopic, three-dimensional (3D) visual displays, auditory input, haptic feedback, and immersive interaction from a first person perspective [1]. Thus virtual reality technology makes it possible to simulate an environment with better ecological validity and control than previously possible for brain research and clinical applications. Virtual reality is compatible with many brain imaging methods and this has allowed researchers to evaluate typical and atypical brain function when users are immersed in a virtual reality environment. Virtual reality is also being developed as a therapeutic tool for a wide range of clinical populations, and because the brain is a primary mediator of effects of virtual reality treatments brain imaging is an important method for assessing some types of treatment effects. In a very different type of therapeutic application, virtual reality is also being developed to augment clinical use of brain imaging results for presurgical planning. This chapter provides an overview of these kinds of studies that employ both virtual reality and brain imaging technologies. In the first part of the chapter we will describe brain correlates of a few examples of tasks that are very difficult, if not impossible, to employ inside a brain imaging scanner without virtual reality technology. These tasks are spatial navigation, car driving, and social interactions. In the second part of this chapter we will survey examples of the application of virtual reality and brain imaging methods to clinical populations. These applications are in early stages of development and will require further studies to assess the value of using a virtual reality versus conventional task and to provide specific evidence of a therapeutic benefit from a virtual reality treatment. Nonetheless these examples will indicate some innovative ways being explored to apply virtual reality and brain imaging in the clinical sphere. We will conclude with a prospective view on using real-time magnetic resonance imaging (MRI) technology combined with virtual reality

Ying-hui Chou, Carol P. Weingarten, David J. Madden,

**in Brain Imaging Studies**

Allen W. Song and Nan-kuei Chen

http://dx.doi.org/10.5772/48445

**1. Introduction**

Additional information is available at the end of the chapter

environments for research or therapeutic applications.

[29] Feizizadeh S, Baretto E, Joackhim R. Virtual Reality Biofeedback Training Interface For Upper Body Movements In Lifting Technique. Thesis. Electrical and Computer Engineering Ryerson University Toronto; 2008.

## **Applications of Virtual Reality Technology in Brain Imaging Studies**

Ying-hui Chou, Carol P. Weingarten, David J. Madden, Allen W. Song and Nan-kuei Chen

Additional information is available at the end of the chapter

http://dx.doi.org/10.5772/48445

#### **1. Introduction**

202 Virtual Reality in Psychological, Medical and Pedagogical Applications

Engineering Ryerson University Toronto; 2008.

Seattle, WA. IEEE Press; 1989. 3:803-5.

[28] Genaidy, A. M., & Houshyar, A. Biomechanical Tolerance Limits For Manual Lifting Tasks: A Tool To Control Back Injuries. IEEE Eng Med Biol Soc Conf. 9-12 Nov., 1989,

[29] Feizizadeh S, Baretto E, Joackhim R. Virtual Reality Biofeedback Training Interface For Upper Body Movements In Lifting Technique. Thesis. Electrical and Computer

> Virtual reality is an evolving technology that allows for the possibility of interactive environments with stereoscopic, three-dimensional (3D) visual displays, auditory input, haptic feedback, and immersive interaction from a first person perspective [1]. Thus virtual reality technology makes it possible to simulate an environment with better ecological validity and control than previously possible for brain research and clinical applications. Virtual reality is compatible with many brain imaging methods and this has allowed researchers to evaluate typical and atypical brain function when users are immersed in a virtual reality environment. Virtual reality is also being developed as a therapeutic tool for a wide range of clinical populations, and because the brain is a primary mediator of effects of virtual reality treatments brain imaging is an important method for assessing some types of treatment effects. In a very different type of therapeutic application, virtual reality is also being developed to augment clinical use of brain imaging results for presurgical planning. This chapter provides an overview of these kinds of studies that employ both virtual reality and brain imaging technologies. In the first part of the chapter we will describe brain correlates of a few examples of tasks that are very difficult, if not impossible, to employ inside a brain imaging scanner without virtual reality technology. These tasks are spatial navigation, car driving, and social interactions. In the second part of this chapter we will survey examples of the application of virtual reality and brain imaging methods to clinical populations. These applications are in early stages of development and will require further studies to assess the value of using a virtual reality versus conventional task and to provide specific evidence of a therapeutic benefit from a virtual reality treatment. Nonetheless these examples will indicate some innovative ways being explored to apply virtual reality and brain imaging in the clinical sphere. We will conclude with a prospective view on using real-time magnetic resonance imaging (MRI) technology combined with virtual reality environments for research or therapeutic applications.

© 2012 Chou et al., licensee InTech. This is an open access chapter distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/3.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. © 2012 The Author(s). Licensee InTech. This chapter is distributed under the terms of the Creative Commons Attribution License http://creativecommons.org/licenses/by/3.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Many imaging modalities have been employed in virtual reality studies reviewed in this chapter. These imaging modalities include structural MRI [2] and diffusion tensor imaging (DTI) [3] that are used to measure brain anatomical structure, and positron emission tomography (PET) [4], functional MRI (fMRI) [5], electroencephalography (EEG) [6], and magnetoencephalography (MEG) [7] that are used to assess brain functional activity.

Applications of Virtual Reality Technology in Brain Imaging Studies 205

In addition, gender- and age-related differences in spatial navigation have been widely reported from behavioral studies [13, 14]. It has been suggested that the gender-related differences may result from disparate cognitive strategies [12, 15-17]. For example, in a selfreport study [12], men are more likely to report using a spatial strategy whereas women are more likely to report using a non-spatial strategy. Men perform best when using instructions indicating the directions (e.g., North or West) and metric distances (e.g., 100m), whereas women performed best when using instructions indicating the salient landmarks

Age-related deficits in spatial navigation have also been studied extensively in animals and humans. For example, older animals show decreased performance in a water maze task [18- 20]. In this task, rats are placed in a large circular pool of opaque water. Younger rats quickly learn to escape by finding and climbing onto a small platform hidden beneath the water surface if this platform remains in a fixed location over a series of trials. However, older rats take longer to find the hidden platform, travel a longer distance in locating the platform, and may require more trials before reaching a designated criterion performance. Moffat and Resnick [21] developed a virtual water maze task for human application and found that, compared to younger adults, older adults traversed a longer linear distance to locate a hidden platform. Older adults also showed evidence of impairment in cognitive mapping as revealed by their poorer map constructions of the previously visited virtual environment and their impaired ability to locate the platform on the experimenter-provided

With the development of 3D virtual reality environments it has now become possible to probe human brain function involved in spatial navigation using brain imaging methods. Virtual environments, rendered in a first-person view, have been created for spatial navigation brain imaging research. Participants could navigate through the virtual environment with the use of keyboard, control pad, joystick, or mouse. The most frequently used virtual environment is the virtual maze, in which participants were instructed to learn and recall the topographical information to locate a target object or to find their way out of the maze [22-27]. Another frequently used virtual environment is a complex virtual town, in which participants were asked to freely navigate the environment first and later they were required to either head directly toward the goal location or follow a trail of arrows [28-31]. In addition, a water maze task used in animal studies to examine the brain correlates of spatial navigation [32] has also been employed in many virtual reality studies [33-35]. In this

task participants are required to search for a hidden target in a large pool of water.

Based on one PET [29] and five fMRI [24-26, 31, 36] virtual reality studies, the hippocampus (especially in the right hemisphere) is one of the most consistently identified substrates associated with spatial navigation in these virtual reality studies. Involvement of the hippocampus has usually been implicated in allocentric representations of space that allow the computation of the direction from any start location to any goal location [29]. The parietal cortex is another brain region that has been frequently identified as being involved in spatial navigation [24, 26, 29, 31, 36] and has been implicated in providing complementary egocentric representations of locations [37]. Various brain correlates were

(e.g., the purple doors) and egocentric (e.g., left or right) turn directions [17].

2D maps of the virtual environment.

## **2. Measuring brain function in virtual reality environments**

Brain function can be measured using imaging methods in a straightforward fashion for many cognitive and sensorimotor performances, such as attention, memory, facial recognition, finger tapping, or language. However, during these measurements of brain function participants usually must be imaged individually and must remain immobile (especially the head). Therefore it has been very difficult to measure brain function associated with tasks that require participants to move through the environment or to have face-to-face social interactions. Virtual reality technology makes it possible to "walk" or "drive" or to perform social interactions in simulated environments during measurements of brain function. Here we will review selected topics in spatial navigation, car driving, and social cognition for which attempts have been made to relate brain function to these behaviors using virtual reality and brain imaging techniques in healthy adults. We will also describe some studies that probed if the brain responds differently to tasks with twodimensional (2D) versus 3D stimuli or real objects, as steps towards a more complex understanding of the neural correlates of virtual reality tasks.

### **2.1. Spatial navigation**

*"Where am I?"*

*"Where are other places with respect to me?"*

#### *"How do I get to other places from here?"*

These are the main questions associated with spatial navigation [8, p.305]. Spatial navigation is the process of determining and maintaining a course or trajectory to a goal location accurately and efficiently [9] and is a requirement of daily life. Behavioral studies of spatial navigation indicate two important aspects for navigation involving allocentric or egocentric representations [10, 11]. Allocentric representations (e.g., South or North) are linked to a reference frame based on the external environment and independent of one's current location in it, while egocentric representations (e.g., left or right) reference spatial locations in the external world with respect to individual body space. Different cognitive strategies have also been observed during spatial navigation [12]. One is called spatial strategy, which involves a more Euclidean representation of space allowing a target location to be reached in a direct path from any given location. The other is a non-spatial strategy, which is related to using environmental information, such as turning left or right at certain points, without knowing the relationships between the start and the target locations.

In addition, gender- and age-related differences in spatial navigation have been widely reported from behavioral studies [13, 14]. It has been suggested that the gender-related differences may result from disparate cognitive strategies [12, 15-17]. For example, in a selfreport study [12], men are more likely to report using a spatial strategy whereas women are more likely to report using a non-spatial strategy. Men perform best when using instructions indicating the directions (e.g., North or West) and metric distances (e.g., 100m), whereas women performed best when using instructions indicating the salient landmarks (e.g., the purple doors) and egocentric (e.g., left or right) turn directions [17].

204 Virtual Reality in Psychological, Medical and Pedagogical Applications

Many imaging modalities have been employed in virtual reality studies reviewed in this chapter. These imaging modalities include structural MRI [2] and diffusion tensor imaging (DTI) [3] that are used to measure brain anatomical structure, and positron emission tomography (PET) [4], functional MRI (fMRI) [5], electroencephalography (EEG) [6], and