Preface

Chapter 8 **How to Create Suitable Augmented Reality Application to Teach Social Skills for Children with ASD 119**

**VI** Contents

I-Jui Lee, Ling-Yi Lin, Chien-Hsu Chen and Chi-Hsuan Chung

Virtual reality (VR) and augmented reality (AR) have long been attributed to science fiction movies and gaming industry. Although VR was popular for decades, AR has been democra‐ tised through famous Pokemon Go. Combination of VR/AR and the physical environment commonly called mixed reality on smartphones has prompted developers towards new heights of creativity and innovation. No need for high-tech wearables to experience VR/AR environments the cloud is quite helpful in processing large multimedia and code processing. One may think that VR/AR is more about multimedia development and visualisation techni‐ ques; nevertheless, there is a huge algorithmic component behind VR/AR systems. Major ICT players have come up with libraries for digital objects that have further accelerated the development life cycle of VR/AR systems.

Regarding challenges, there are technical limitations but also reluctance from users to adopt VR/AR technology. Nonetheless, few mobile manufacturers are not willing to implement such technologies. The main challenge remains dynamic content that would keep the inter‐ est of the users and with exciting user experience so that applications do not fall obsolete within a short time span. Like Multimedia Message Service (MMS), VR/AR has had poor adoption due to lack of content and appropriate business models between service providers and content developers.

On the other hand, VR/AR has proven to be very much useful in healthcare and engineering fields, without mentioning educational technologies where it is already a legacy. Medical scientists, educators and engineers require deeper understanding of how VR/AR can assist them in their daily activities as well as pertinent issues. So far from user perspective, this book showcases research and development to enlighten readers with background in VR/AR.

Moreover, it compiles latest advancement in the fields of VR/AR from a broad perspective. Chapters cover core areas where presently VR/AR technologies are applied, namely, health‐ care, education, building industry and visualisation. The chapters' content provides experi‐ mental validations and high scientific value.

We would like to express our whole-hearted appreciations to all the authors and their re‐ spective institutions who have contributed for the accomplishment of this book. Everything has been possible thanks to the InTech publishing process managers Mr. Julian Virag and Mrs. Danijela Vladika, the technical board, and the commissioning editor for their endeav‐ our in disseminating knowledge globally.

> **Associate Professor (Dr) Nawaz Mohamudally** University of Technology, Mauritius

**Chapter 1**

**Provisional chapter**

**Introductory Chapter: Enhancing Augmented Reality**

**Introductory Chapter: Enhancing Augmented Reality**

One of most amazing aspects of VR and AR is user experience that can be defined as personal or group feelings about VR and AR applications and products. VR/AR user experience is more than usability, perception or customer adoption patterns, it deals with the emotional interaction with technology as well as physical and virtual environments. Although VR/AR is mostly a visualisation technique, other senses like touch and hearing are increasingly present. One example of how user experience can drive a business model is actually the success of social networks on the Internet. Smartphones too have provided rich user experience with multimedia and VoIP applications. AR has brought richer user experience on smartphones that are accessible through the camera or a VR glass. However the resources limitations on handheld devices have prompted novel methods of computation. Extensive work exists on mobile code offloading towards a server or a cloud to minimise battery depletion. Another method is VM cloning, but despite techniques to overcome hardware and software challenges little is done in view of understanding and improving user experience. This chapter gives a brief apercu of design thinking, a technique that can enhance VR/AR applications user experi-

> © 2016 The Author(s). Licensee InTech. This chapter is distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/3.0), which permits unrestricted use, distribution,

© 2018 The Author(s). Licensee IntechOpen. This chapter is distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/3.0), which permits unrestricted use,

distribution, and reproduction in any medium, provided the original work is properly cited.

and reproduction in any medium, provided the original work is properly cited.

*"In 2021, the augmented and virtual reality market is expected to reach a market size of 215 billion U.S. dollars."*1 AR applications are today multidisciplinary. Below are listed some of the

1 https://www.statista.com/statistics/591181/global-augmented-virtual-reality-market-size/

DOI: 10.5772/intechopen.76850

**User Experience (AR-UX) with Design Thinking**

**User Experience (AR-UX) with Design Thinking**

Nawaz Mohamudally

**1. Introduction**

**2. Use cases**

Additional information is available at the end of the chapter

ence. Focus is on AR applications for smartphones.

Nawaz MohamudallyAdditional information is available at the end of the chapter

http://dx.doi.org/10.5772/intechopen.76850

#### **Introductory Chapter: Enhancing Augmented Reality User Experience (AR-UX) with Design Thinking Introductory Chapter: Enhancing Augmented Reality User Experience (AR-UX) with Design Thinking**

DOI: 10.5772/intechopen.76850

Nawaz Mohamudally

Additional information is available at the end of the chapter Nawaz MohamudallyAdditional information is available at the end of the chapter

http://dx.doi.org/10.5772/intechopen.76850

#### **1. Introduction**

 One of most amazing aspects of VR and AR is user experience that can be defined as personal or group feelings about VR and AR applications and products. VR/AR user experience is more than usability, perception or customer adoption patterns, it deals with the emotional interaction with technology as well as physical and virtual environments. Although VR/AR is mostly a visualisation technique, other senses like touch and hearing are increasingly present. One example of how user experience can drive a business model is actually the success of social networks on the Internet. Smartphones too have provided rich user experience with multimedia and VoIP applications. AR has brought richer user experience on smartphones that are accessible through the camera or a VR glass. However the resources limitations on handheld devices have prompted novel methods of computation. Extensive work exists on mobile code offloading towards a server or a cloud to minimise battery depletion. Another method is VM cloning, but despite techniques to overcome hardware and software challenges little is done in view of understanding and improving user experience. This chapter gives a brief apercu of design thinking, a technique that can enhance VR/AR applications user experience. Focus is on AR applications for smartphones.

#### **2. Use cases**

*"In 2021, the augmented and virtual reality market is expected to reach a market size of 215 billion U.S. dollars."*1 AR applications are today multidisciplinary. Below are listed some of the

1 https://www.statista.com/statistics/591181/global-augmented-virtual-reality-market-size/

© 2016 The Author(s). Licensee InTech. This chapter is distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/3.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. © 2018 The Author(s). Licensee IntechOpen. This chapter is distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/3.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

areas and AR use cases with specific user experiences. AR can transform one's home interior into an archaeological site with artefacts 15,000 years back, it is indeed another level of user experience.

**4. Design thinking for AR**

come with experience or by sharing.

information about user experience.

**Figure 1.** Design thinking framework.

design thinking. The process is iterative and non-linear.

box thinking and observing nature are ways to get inspired.

psychological aspect of the users towards the AR app should be captured.

Design a way of thinking was initially meant for creation and problem solving. It is actually a combination of analytical thinking and intuitive thinking that dates back to the 1950s and 1960s. On its way, design thinking to improve UX has gained momentum. Today's smartphone resulted from this approach. **Figure 1** depicts global view of the main components of

Introductory Chapter: Enhancing Augmented Reality User Experience (AR-UX) with Design…

http://dx.doi.org/10.5772/intechopen.76850

3

*Inspiration:* involves an accurate understanding of a problem or an eventual product where AR can add value. It is like envisioning the application even if it appears to be beyond the capacity of current development tools. Inspiration does not come out of the blue, out of the

*Empathy:* is to look at the needs of the users, not only gathering user requirements but what users would like to experience. It could be higher image resolution, larger display, and quicker response and so on. Ease of use and fulfilling needs are parts of user experience. The

*Ideation*: this is perhaps the most important milestone in enhancing user experience as it deals with challenges and ability to find innovative answers to the findings during the empathy phase but also the opportunities and limitations provide during the prototyping phase. Ideas

*Prototyp*ing: involves implementation and testing of the AR app in an agile manner. Testing is crucial in producing an application that is reliable and answers to the user expectations. The element of surprise or extra excitement is the plus in user experience, a differentiator. A survey conducted on testing of AR apps has revealed that most of the testing fields are technology oriented that are camera settings, native or hybrid app, and so on. There is little

Coming back to the AR development for smartphones, marker-less applications are often more challenging but appreciated compared to marker based applications. Users love the element of surprise and astonishment. But the smartphone is not suitable to plug hardware devices that can add extra sensation apart from video, audio, 3D interaction. Nonetheless, the presence of sensors and access to data from cloud puts forward mobile cloud computing as a serious option for future AR applications with new user experience. Sophisticated scripts and AI code can run on the cloud and render user information and experience in real time on the smartphone. Currently effects are on the graphical objects more than the business rules. Thus an AR developer is first and foremost a graphic designer and a multimedia developer. To

*Gaming:* Gaming experience with AR picked up with the famous PokemonGO. Candy Camera is a simple app that allows to add features to selfies. 3D animation and mixed realities combining physical environments and digital objects provide a special user experience in AR games. Players look for entertainment and sensations that are different from video games with shootings.

*Education:* AR applications bring new learning and teaching experiences aiming at developing critical thinking and self-learning. In engineering visualisation of artefacts with interaction is another form of instructions. Likewise the study of bones in medicine with an AR app enhances the student perspective.

*Healthcare*: AR is more and more adopted for patient rehabilitation and for diagnosis. Recent advances advocate that AR will be able to provide surgeons with an environment to prepare themselves prior to a surgery operation. This is an example of kind of user experience in healthcare that is unique to this situation.

*Building industry:* AR is widely used in building information systems, however AR is used also to safeguard patrimonies in digital format or holograms. History discovery and tourism are typical experiences from such applications.

*Navigation:* Popular examples are Flight Simulator and Immersion experience. In such applications there is contrast between the physical environment and virtual environment. Immersion generally is fully virtual, whereas navigating in a building or a site is about mixed realities.

*VuMark:* This is a product code just like the QR code that uses Vuforia technology to display additional information about a product for example when the camera of a smartphone points at the VuMark. It could provide another type of sales experience for consumers. It has been shown that pricing is not always the major determining factor, sales experience makes a difference.

#### **3. AR apps development life cycle**

Developing AR apps requires 3D Graphics and Object Design software (Example 3D MAX, a game engine (Unity3D), Qualcomm Vuforia (an AR SDK to track Image Targets), Head Mounted Display or a Smartphone with a high resolution camera). To enhance user experience, there are existing software engineering techniques like co-construction and user involvement. The time consuming activity is about design and production of the graphics objects that amount to more than 80% of a project. Google Libraries for AR objects and AR Kit have facilitated this phase.

## **4. Design thinking for AR**

areas and AR use cases with specific user experiences. AR can transform one's home interior into an archaeological site with artefacts 15,000 years back, it is indeed another level of user

*Gaming:* Gaming experience with AR picked up with the famous PokemonGO. Candy Camera is a simple app that allows to add features to selfies. 3D animation and mixed realities combining physical environments and digital objects provide a special user experience in AR games. Players look for entertainment and sensations that are different from video games

*Education:* AR applications bring new learning and teaching experiences aiming at developing critical thinking and self-learning. In engineering visualisation of artefacts with interaction is another form of instructions. Likewise the study of bones in medicine with an AR app

*Healthcare*: AR is more and more adopted for patient rehabilitation and for diagnosis. Recent advances advocate that AR will be able to provide surgeons with an environment to prepare themselves prior to a surgery operation. This is an example of kind of user experience in

*Building industry:* AR is widely used in building information systems, however AR is used also to safeguard patrimonies in digital format or holograms. History discovery and tourism

*Navigation:* Popular examples are Flight Simulator and Immersion experience. In such applications there is contrast between the physical environment and virtual environment. Immersion generally is fully virtual, whereas navigating in a building or a site is about mixed realities.

*VuMark:* This is a product code just like the QR code that uses Vuforia technology to display additional information about a product for example when the camera of a smartphone points at the VuMark. It could provide another type of sales experience for consumers. It has been shown that pricing is not always the major determining factor, sales experience makes

Developing AR apps requires 3D Graphics and Object Design software (Example 3D MAX, a game engine (Unity3D), Qualcomm Vuforia (an AR SDK to track Image Targets), Head Mounted Display or a Smartphone with a high resolution camera). To enhance user experience, there are existing software engineering techniques like co-construction and user involvement. The time consuming activity is about design and production of the graphics objects that amount to more than 80% of a project. Google Libraries for AR objects and AR Kit

experience.

with shootings.

a difference.

enhances the student perspective.

healthcare that is unique to this situation.

are typical experiences from such applications.

2 State of the Art Virtual Reality and Augmented Reality Knowhow

**3. AR apps development life cycle**

have facilitated this phase.

Design a way of thinking was initially meant for creation and problem solving. It is actually a combination of analytical thinking and intuitive thinking that dates back to the 1950s and 1960s. On its way, design thinking to improve UX has gained momentum. Today's smartphone resulted from this approach. **Figure 1** depicts global view of the main components of design thinking. The process is iterative and non-linear.

*Inspiration:* involves an accurate understanding of a problem or an eventual product where AR can add value. It is like envisioning the application even if it appears to be beyond the capacity of current development tools. Inspiration does not come out of the blue, out of the box thinking and observing nature are ways to get inspired.

*Empathy:* is to look at the needs of the users, not only gathering user requirements but what users would like to experience. It could be higher image resolution, larger display, and quicker response and so on. Ease of use and fulfilling needs are parts of user experience. The psychological aspect of the users towards the AR app should be captured.

*Ideation*: this is perhaps the most important milestone in enhancing user experience as it deals with challenges and ability to find innovative answers to the findings during the empathy phase but also the opportunities and limitations provide during the prototyping phase. Ideas come with experience or by sharing.

*Prototyp*ing: involves implementation and testing of the AR app in an agile manner. Testing is crucial in producing an application that is reliable and answers to the user expectations. The element of surprise or extra excitement is the plus in user experience, a differentiator. A survey conducted on testing of AR apps has revealed that most of the testing fields are technology oriented that are camera settings, native or hybrid app, and so on. There is little information about user experience.

Coming back to the AR development for smartphones, marker-less applications are often more challenging but appreciated compared to marker based applications. Users love the element of surprise and astonishment. But the smartphone is not suitable to plug hardware devices that can add extra sensation apart from video, audio, 3D interaction. Nonetheless, the presence of sensors and access to data from cloud puts forward mobile cloud computing as a serious option for future AR applications with new user experience. Sophisticated scripts and AI code can run on the cloud and render user information and experience in real time on the smartphone. Currently effects are on the graphical objects more than the business rules. Thus an AR developer is first and foremost a graphic designer and a multimedia developer. To

**Figure 1.** Design thinking framework.

further enhance UX, more data analytics is required. On the network side higher bandwidth coupled with advancement in cellular networks, for instance 5G has not yet disclosed the potential for richer AR apps.

**Chapter 2**

**Provisional chapter**

**Augmented Reality and Virtual Reality: Initial**

**Augmented Reality and Virtual Reality: Initial**

DOI: 10.5772/intechopen.74317

In this chapter, we will review the applications of augmented reality (AR) and virtual reality (VR) in diagnostic radiology. We begin the chapter by discussing state of the art medical imaging techniques to include scanner hardware, spatial resolution, and presentation methods to the radiologist or other medical professionals. We continue by discussing the methodology of a technique called Depth-3-Dimensional (D3D) imaging, which transforms cross-sectional medical imaging datasets into a left and right eye images that that can be viewed with state of the art AR or VR headsets. We include results of the D3D

The purpose of this chapter is to review the applications of augmented reality (AR) and virtual reality (VR) in diagnostic radiology. This introduction section will provide a brief general summary of AR technologies, VR technologies, AR/VR applications in medicine and AR/VR applications diagnostic radiology. The remainder of the paper will discuss state-of-the-art medical imaging systems, current methods of viewing medical images, the imaging processing techniques for generating an AR/VR image, AR/VR imaging results and path forward.

processed AR/VR imaging and conclude with a discussion of the path forward. **Keywords:** augmented reality, virtual reality, diagnostic radiology, D3D, depth

> © 2016 The Author(s). Licensee InTech. This chapter is distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/3.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

© 2018 The Author(s). Licensee IntechOpen. This chapter is distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/3.0), which permits unrestricted use,

distribution, and reproduction in any medium, provided the original work is properly cited.

**Successes in Diagnostic Radiology**

**Successes in Diagnostic Radiology**

David B. Douglas, Demetri Venets, Cliff Wilke, David Gibson, Lance Liotta, Emanuel Petricoin,

David B. Douglas, Demetri Venets, Cliff Wilke, David Gibson, Lance Liotta, Emanuel Petricoin,

Additional information is available at the end of the chapter

Additional information is available at the end of the chapter

Buddy Beck and Robert Douglas

Buddy Beck and Robert Douglas

http://dx.doi.org/10.5772/intechopen.74317

**Abstract**

perception

**1. Introduction**

#### **5. Conclusion**

This chapter highlights user experience component in the development of AR applications. Design thinking is proposed to augment user experience in AR applications for smartphones. It is an iterative method that can add value to the already the hardware and software challenges in AR applications. AR applications may not always be customer or user centric, so it is on a case to case basis that design thinking must be applied. Survey on the prototyping milestone has shown that in the testing phase UX is often negligible although user testing is present compared to technical issues and Empathy is not straightforward. The future of very rich AR-UX lies in mobile cloud computing. The coming chapters in this book deal with more pertinent and complex scientific issues in VR/AR, nevertheless readers will appreciate how far user experience is pervasive.

#### **Author details**

Nawaz Mohamudally

Address all correspondence to<EMAIL_ADDRESS>
University of Technology, Mauritius

#### **Chapter 2 Provisional chapter**

#### **Augmented Reality and Virtual Reality: Initial Successes in Diagnostic Radiology Augmented Reality and Virtual Reality: Initial Successes in Diagnostic Radiology**

DOI: 10.5772/intechopen.74317

David B. Douglas, Demetri Venets, Cliff Wilke, David Gibson, Lance Liotta, Emanuel Petricoin, Buddy Beck and Robert Douglas David B. Douglas, Demetri Venets, Cliff Wilke, David Gibson, Lance Liotta, Emanuel Petricoin, Buddy Beck and Robert Douglas

Additional information is available at the end of the chapter Additional information is available at the end of the chapter

http://dx.doi.org/10.5772/intechopen.74317

#### **Abstract**

further enhance UX, more data analytics is required. On the network side higher bandwidth coupled with advancement in cellular networks, for instance 5G has not yet disclosed the

This chapter highlights user experience component in the development of AR applications. Design thinking is proposed to augment user experience in AR applications for smartphones. It is an iterative method that can add value to the already the hardware and software challenges in AR applications. AR applications may not always be customer or user centric, so it is on a case to case basis that design thinking must be applied. Survey on the prototyping milestone has shown that in the testing phase UX is often negligible although user testing is present compared to technical issues and Empathy is not straightforward. The future of very rich AR-UX lies in mobile cloud computing. The coming chapters in this book deal with more pertinent and complex scientific issues in VR/AR, nevertheless readers will appreciate how

potential for richer AR apps.

4 State of the Art Virtual Reality and Augmented Reality Knowhow

far user experience is pervasive.

University of Technology, Mauritius

Address all correspondence to<EMAIL_ADDRESS>
**Author details**

Nawaz Mohamudally

**5. Conclusion**

In this chapter, we will review the applications of augmented reality (AR) and virtual reality (VR) in diagnostic radiology. We begin the chapter by discussing state of the art medical imaging techniques to include scanner hardware, spatial resolution, and presentation methods to the radiologist or other medical professionals. We continue by discussing the methodology of a technique called Depth-3-Dimensional (D3D) imaging, which transforms cross-sectional medical imaging datasets into a left and right eye images that that can be viewed with state of the art AR or VR headsets. We include results of the D3D processed AR/VR imaging and conclude with a discussion of the path forward.

**Keywords:** augmented reality, virtual reality, diagnostic radiology, D3D, depth perception

#### **1. Introduction**

The purpose of this chapter is to review the applications of augmented reality (AR) and virtual reality (VR) in diagnostic radiology. This introduction section will provide a brief general summary of AR technologies, VR technologies, AR/VR applications in medicine and AR/VR applications diagnostic radiology. The remainder of the paper will discuss state-of-the-art medical imaging systems, current methods of viewing medical images, the imaging processing techniques for generating an AR/VR image, AR/VR imaging results and path forward.

© 2016 The Author(s). Licensee InTech. This chapter is distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/3.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. © 2018 The Author(s). Licensee IntechOpen. This chapter is distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/3.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

#### **1.1. Introduction to AR/VR**

AR technologies can be classified into AR systems or mixed reality (MR) systems. In both the AR and MR systems, the user wears a head mounted display (HMD), which provides the simultaneous display of a virtual image and the scene of the real-world surroundings [1]. The differences are that in AR, the virtual image is transparent like a hologram and in MR the virtual image appears solid. Examples of AR systems include the Meta and DAQRI systems. An example of an MR system is the Microsoft HoloLens. In AR systems, the user can view the virtual image and interact with the real-world scene.

**2. Current state-of-the-art diagnostic medical imaging**

methods.

focus on the latter.

images.

**2.1. Medical imaging equipment**

*2.1.1. Computed tomography (CT)*

Diagnostic radiology plays a major role in medicine as it provides precise anatomic and physiologic information to physicians enabling diagnosis of complex disease and monitoring response to treatment. This section will be organized into three subsections including medical imaging equipment, conventional imaging techniques and the advanced 3D rendering

Augmented Reality and Virtual Reality: Initial Successes in Diagnostic Radiology

http://dx.doi.org/10.5772/intechopen.74317

7

The field of diagnostic radiology includes a wide variety of imaging equipment including systems that generate inherently 2D images (e.g., chest radiograph) as well as systems that generate volumetric medical imaging datasets (e.g., computed tomography (CT), magnetic resonance imaging (MRI), and positron emission tomography (PET)). In this section, we will

In order to perform a CT scan (also known as a CAT scan), the patient is placed in the horizontal position on the CT scanner table. See **Figure 1**. The table is then translated through a donut shaped device containing both an X-ray tube and X-ray detector. Multiple projection images

Image reconstruction algorithms, such as filtered back projection, are performed in order to generate cross-sectional images in the axial plane (x-y plane). Since sequential, contiguous axial images can be obtained, coronal plane (x-z) and sagittal plane (y-z) plane images can be reconstructed. Data is stored in digital imaging and communication in medicine

**Figure 1.** Photograph of a computed tomography (CT) scanner. The patient is being transferred from onto the mobile CT scanner table that will translate through the donut-shaped device during the CT scan and take multiple projection

are acquired as the X-ray tube and detector assembly rotate around the patient.

VR technologies may be classified as fully immersive, semi-immersive or non-immersive [2]. In fully immersive VR systems such as the Oculus Rift and the HTC Vive, the HMD displays a virtual image and the real-world surroundings are completely occluded from the user's field of view [3]. In semi-immersive VR systems such as the Samsung Gear VR, the HMD displays the virtual image, but the real-world scene is only partially occluded from the user's field of view [3]. In VR systems, the user can maneuver through the virtual world by head movements via HMD tracking or walking via external camera tracking systems. Additional ways that a user can interact with the virtual environment include voice gestures or through handheld devices with haptic feedback. The HMDs for the AR and VR systems display a unique image to each eye; therefore, stereoscopic imaging and depth perceptions is achieved. Both AR and VR are rapidly growing fields. Worldwide revenues for AR/VR were \$5 billion in 2016, but are expected to increase to \$162 billion by 2020 [3].

#### **1.2. Applications of AR/VR in medicine**

In 2015, the United States spent 17% of its gross domestic product on healthcare. 32% of the healthcare spending was during hospital stays [4]. Nearly half of hospital costs are for surgical care [5]. Therefore, approximately 3% of the gross domestic product includes costs related to surgery. There is a drive to improve efficiency in the operation room to both improve patient care and drive down costs. AR/VR holds promise in accomplishing these goals through improving pre-operative planning and enhancing intra-operative surgical procedures [6].

#### **1.3. Applications of AR/VR in diagnostic radiology**

In the United States, the total cost of diagnostic imaging has been estimated to be \$100 billion in 2006 alone [7]. Utilization rates of diagnostic imaging are on the rise [8–10]. Currently, there is no United States Food and Drug Administration (FDA) approved AR/VR system used in diagnostic radiology [11]. AR/VR provides enhanced viewing including depth perception and improved human machine interface (HMI) [12, 13]. AR/VR HMDs provide unique images to each eye yielding depth perception. AR/VR systems leverage advanced gaming controllers and joysticks to improve HMI. Because of these features of AR/VR and others discussed later in this chapter, we believe there will be increasing applications of AR/VR in diagnostic radiology in the future.

### **2. Current state-of-the-art diagnostic medical imaging**

Diagnostic radiology plays a major role in medicine as it provides precise anatomic and physiologic information to physicians enabling diagnosis of complex disease and monitoring response to treatment. This section will be organized into three subsections including medical imaging equipment, conventional imaging techniques and the advanced 3D rendering methods.

#### **2.1. Medical imaging equipment**

**1.1. Introduction to AR/VR**

by 2020 [3].

procedures [6].

virtual image and interact with the real-world scene.

6 State of the Art Virtual Reality and Augmented Reality Knowhow

**1.2. Applications of AR/VR in medicine**

**1.3. Applications of AR/VR in diagnostic radiology**

diagnostic radiology in the future.

AR technologies can be classified into AR systems or mixed reality (MR) systems. In both the AR and MR systems, the user wears a head mounted display (HMD), which provides the simultaneous display of a virtual image and the scene of the real-world surroundings [1]. The differences are that in AR, the virtual image is transparent like a hologram and in MR the virtual image appears solid. Examples of AR systems include the Meta and DAQRI systems. An example of an MR system is the Microsoft HoloLens. In AR systems, the user can view the

VR technologies may be classified as fully immersive, semi-immersive or non-immersive [2]. In fully immersive VR systems such as the Oculus Rift and the HTC Vive, the HMD displays a virtual image and the real-world surroundings are completely occluded from the user's field of view [3]. In semi-immersive VR systems such as the Samsung Gear VR, the HMD displays the virtual image, but the real-world scene is only partially occluded from the user's field of view [3]. In VR systems, the user can maneuver through the virtual world by head movements via HMD tracking or walking via external camera tracking systems. Additional ways that a user can interact with the virtual environment include voice gestures or through handheld devices with haptic feedback. The HMDs for the AR and VR systems display a unique image to each eye; therefore, stereoscopic imaging and depth perceptions is achieved. Both AR and VR are rapidly growing fields. Worldwide revenues for AR/VR were \$5 billion in 2016, but are expected to increase to \$162 billion

In 2015, the United States spent 17% of its gross domestic product on healthcare. 32% of the healthcare spending was during hospital stays [4]. Nearly half of hospital costs are for surgical care [5]. Therefore, approximately 3% of the gross domestic product includes costs related to surgery. There is a drive to improve efficiency in the operation room to both improve patient care and drive down costs. AR/VR holds promise in accomplishing these goals through improving pre-operative planning and enhancing intra-operative surgical

In the United States, the total cost of diagnostic imaging has been estimated to be \$100 billion in 2006 alone [7]. Utilization rates of diagnostic imaging are on the rise [8–10]. Currently, there is no United States Food and Drug Administration (FDA) approved AR/VR system used in diagnostic radiology [11]. AR/VR provides enhanced viewing including depth perception and improved human machine interface (HMI) [12, 13]. AR/VR HMDs provide unique images to each eye yielding depth perception. AR/VR systems leverage advanced gaming controllers and joysticks to improve HMI. Because of these features of AR/VR and others discussed later in this chapter, we believe there will be increasing applications of AR/VR in The field of diagnostic radiology includes a wide variety of imaging equipment including systems that generate inherently 2D images (e.g., chest radiograph) as well as systems that generate volumetric medical imaging datasets (e.g., computed tomography (CT), magnetic resonance imaging (MRI), and positron emission tomography (PET)). In this section, we will focus on the latter.

#### *2.1.1. Computed tomography (CT)*

In order to perform a CT scan (also known as a CAT scan), the patient is placed in the horizontal position on the CT scanner table. See **Figure 1**. The table is then translated through a donut shaped device containing both an X-ray tube and X-ray detector. Multiple projection images are acquired as the X-ray tube and detector assembly rotate around the patient.

Image reconstruction algorithms, such as filtered back projection, are performed in order to generate cross-sectional images in the axial plane (x-y plane). Since sequential, contiguous axial images can be obtained, coronal plane (x-z) and sagittal plane (y-z) plane images can be reconstructed. Data is stored in digital imaging and communication in medicine

**Figure 1.** Photograph of a computed tomography (CT) scanner. The patient is being transferred from onto the mobile CT scanner table that will translate through the donut-shaped device during the CT scan and take multiple projection images.

(DICOM) files with a typical matrix of a CT scan is 512 × 512 pixels. A pixel in the axial plane is a 2D object with a discrete length in the x-direction and discrete length in the y-direction. A voxel is a 3D object created with a pixel by adding a third dimension to create a volume.

*2.1.2. Magnetic resonance imaging (MRI)*

In order to perform an MRI scan, the patient is placed in the horizontal position on the MRI scanner table within a large cylindrical shaped device. A large magnetic field is directed through the long axis of the cylinder. Transmit coils direct a radiofrequency (RF) pulse into the patient's body and receive coils process the returning electromagnetic signal from the body to create an image. Similar to CT, contiguous planar images can be stacked and axial, sagittal and coronal reformats can be reconstructed. The imaging data for a MRI scan is similar to that of a CT scan in matrix size and the fact that each pixel has a comparable gray scale. Unlike CT scanners, MRI scanners do not employ ionizing radiation and are therefore ideal for young children or pregnant women who are more vulnerable to radiation. Furthermore, MRI scans have the ability to perform exceptional contrast resolution between tissues of similar density and can diagnose certain types of traumatic brain injury that cannot be seen on CT scans [15]. Modern MRI scans require significantly more time to perform than CT scans,

Augmented Reality and Virtual Reality: Initial Successes in Diagnostic Radiology

http://dx.doi.org/10.5772/intechopen.74317

9

In order to perform a PET scan, a radiopharmaceutical (e.g., fludeoxyglucose F-18) is administered to the patient. Then, the patient is placed in the horizontal position on the PET scanner table within a donut shaped device. As the radiopharmaceutical decays, photons are emitted from within the patient and are received by the PET detector crystals. As with CT and MRI,

A typical matrix of a PET scan is 128 × 128, which is smaller than that of CT or MRI. However, it is similar in that each pixel has an associated numerical value associated with it indicating gray scale. In the case of PET, the gray scale corresponds to the amount of radioactivity emitted from that location. Since the radiopharmaceuticals can target certain structures in the body, it is possible for improved diagnosis of conditions like Alzheimer's disease compared with MRI [16].

In the previous section, we discussed three types of medical imaging scanners that generate volumetric data. In this section, we will review the current methods of viewing the volumetric

The conventional viewing method for reviewing volumetric datasets is a slice-by-slice viewing method for axial, sagittal and coronal imaging planes or on occasion oblique reformats. It is estimated that most radiologists spend more than 95% of their total time on cross-sectional imaging datasets using this conventional slice-by-slice approach [17]. Certain anatomical structures are easier to visualize on particular imaging planes. As an example, some radiologists have a preference for viewing the midline structures in the brain on the sagittal images,

axial images can be stacked and sagittal and coronal images can be reconstructed.

nearly one-hour of image acquisition time for a MRI scan of the brain.

*2.1.3. Positron emission tomography (PET) scanner*

**2.2. Current methods of viewing medical imaging**

*2.2.1. Conventional viewing of the volumetric data*

data.

as shown in **Figure 3**.

Each pixel has an associated gray-scale value called a Hounsfield Unit (HU), which is a function of the density and composition of the tissue. As a reference, water has a HU of 0. Soft tissues (e.g., brain, kidney, muscle, etc.) are slightly denser than water and have a HU of approximately 30–40. Compact bone can have a density of 400. Fat is slightly less dense than water and has a HU of approximately −100. Air is significantly less dense than water and has a HU of −1000.

A process called "windowing and leveling" is performed by the radiologist to set the window "level" and window "width." The window "level" refers to the HU number where mid-gray can represent. The window "width" is the range of shades of gray such that any value more extreme than the range is white (if more dense) or black (if less dense). See **Figure 2**.

Modern CT scans can be perform a total body head-to-toe scan in less than 15 seconds with spatial resolution of less than 1 mm. Thus, CT scans can be invaluable in the setting of trauma since it enables the radiologist to diagnose critical injuries anywhere in the body, such as traumatic brain injury [14]. This provides critical information to the neurosurgeon who can then perform life-saving interventions.

**Figure 2.** Axial CT scan of the head in "bone window" (A), "brain window" (B) and "scalp window" (C) at the same level within the brain. In bone window (A), the window level is set to 400 (to optimize visualization of bone which has a density of approximately 400 HU) and the width is set to 3000. In the brain window (B), the window level is set to 30 (to optimize visualization of brain which as a density of approximately 30 HU) and the width is set to 80. Any pixels that are below −10 (30 (window level) minus 40 (1/2 window width)) will be black. Air with a HU of −1000 is black because it is less than −10.  Any pixels that are higher than 70 (30 (window level) plus 40 (1/2 window width)) will be white. In this image, the bone is white. In the scalp window (C), the window level is set to 30 and the window width to 800. Note that fat (blue star) with a HU of −100 is now gray since it falls within the range of the window level of 30 and width of 800, which would be 30±400 or −370 to 430.

#### *2.1.2. Magnetic resonance imaging (MRI)*

(DICOM) files with a typical matrix of a CT scan is 512 × 512 pixels. A pixel in the axial plane is a 2D object with a discrete length in the x-direction and discrete length in the y-direction. A voxel is a 3D object created with a pixel by adding a third dimension to cre-

Each pixel has an associated gray-scale value called a Hounsfield Unit (HU), which is a function of the density and composition of the tissue. As a reference, water has a HU of 0. Soft tissues (e.g., brain, kidney, muscle, etc.) are slightly denser than water and have a HU of approximately 30–40. Compact bone can have a density of 400. Fat is slightly less dense than water and has a HU of approximately −100. Air is significantly less dense than water and has

A process called "windowing and leveling" is performed by the radiologist to set the window "level" and window "width." The window "level" refers to the HU number where mid-gray can represent. The window "width" is the range of shades of gray such that any value more extreme than the range is white (if more dense) or black (if less dense). See

Modern CT scans can be perform a total body head-to-toe scan in less than 15 seconds with spatial resolution of less than 1 mm. Thus, CT scans can be invaluable in the setting of trauma since it enables the radiologist to diagnose critical injuries anywhere in the body, such as traumatic brain injury [14]. This provides critical information to the neurosurgeon who can then

**Figure 2.** Axial CT scan of the head in "bone window" (A), "brain window" (B) and "scalp window" (C) at the same level within the brain. In bone window (A), the window level is set to 400 (to optimize visualization of bone which has a density of approximately 400 HU) and the width is set to 3000. In the brain window (B), the window level is set to 30 (to optimize visualization of brain which as a density of approximately 30 HU) and the width is set to 80. Any pixels that are below −10 (30 (window level) minus 40 (1/2 window width)) will be black. Air with a HU of −1000 is black because it is less than −10.  Any pixels that are higher than 70 (30 (window level) plus 40 (1/2 window width)) will be white. In this image, the bone is white. In the scalp window (C), the window level is set to 30 and the window width to 800. Note that fat (blue star) with a HU of −100 is now gray since

it falls within the range of the window level of 30 and width of 800, which would be 30±400 or −370 to 430.

ate a volume.

8 State of the Art Virtual Reality and Augmented Reality Knowhow

a HU of −1000.

**Figure 2**.

perform life-saving interventions.

In order to perform an MRI scan, the patient is placed in the horizontal position on the MRI scanner table within a large cylindrical shaped device. A large magnetic field is directed through the long axis of the cylinder. Transmit coils direct a radiofrequency (RF) pulse into the patient's body and receive coils process the returning electromagnetic signal from the body to create an image. Similar to CT, contiguous planar images can be stacked and axial, sagittal and coronal reformats can be reconstructed. The imaging data for a MRI scan is similar to that of a CT scan in matrix size and the fact that each pixel has a comparable gray scale.

Unlike CT scanners, MRI scanners do not employ ionizing radiation and are therefore ideal for young children or pregnant women who are more vulnerable to radiation. Furthermore, MRI scans have the ability to perform exceptional contrast resolution between tissues of similar density and can diagnose certain types of traumatic brain injury that cannot be seen on CT scans [15]. Modern MRI scans require significantly more time to perform than CT scans, nearly one-hour of image acquisition time for a MRI scan of the brain.

#### *2.1.3. Positron emission tomography (PET) scanner*

In order to perform a PET scan, a radiopharmaceutical (e.g., fludeoxyglucose F-18) is administered to the patient. Then, the patient is placed in the horizontal position on the PET scanner table within a donut shaped device. As the radiopharmaceutical decays, photons are emitted from within the patient and are received by the PET detector crystals. As with CT and MRI, axial images can be stacked and sagittal and coronal images can be reconstructed.

A typical matrix of a PET scan is 128 × 128, which is smaller than that of CT or MRI. However, it is similar in that each pixel has an associated numerical value associated with it indicating gray scale. In the case of PET, the gray scale corresponds to the amount of radioactivity emitted from that location. Since the radiopharmaceuticals can target certain structures in the body, it is possible for improved diagnosis of conditions like Alzheimer's disease compared with MRI [16].

#### **2.2. Current methods of viewing medical imaging**

In the previous section, we discussed three types of medical imaging scanners that generate volumetric data. In this section, we will review the current methods of viewing the volumetric data.

#### *2.2.1. Conventional viewing of the volumetric data*

The conventional viewing method for reviewing volumetric datasets is a slice-by-slice viewing method for axial, sagittal and coronal imaging planes or on occasion oblique reformats. It is estimated that most radiologists spend more than 95% of their total time on cross-sectional imaging datasets using this conventional slice-by-slice approach [17]. Certain anatomical structures are easier to visualize on particular imaging planes. As an example, some radiologists have a preference for viewing the midline structures in the brain on the sagittal images, as shown in **Figure 3**.

**Figure 3.** Image of the brain from an MRI scanner. This is of an axial T1-weighted sequence (A) with coronal (B) and sagittal (C) images reconstructed.

Occasionally, the radiologist needs to view an abnormality in another plane, other than the axial, sagittal, coronal imaging planes. In these instances, oblique plane reformats can be used with the images still viewed in a conventional slice-by-slice approach. See **Figure 4**. Curved planar reformats can also be performed and have been shown to be beneficial [18].

*2.2.2. Challenges in conventional viewing of the volumetric data*

deliberate slice-by-slice method takes considerable time.

rendering and volume rendering.

*<IP_ADDRESS>. Surface rendering*

and keyboard.

*2.2.3. Advanced (non-AR/VR) viewing methods of the volumetric data*

First, is the challenge of information overload. The dramatic improvements in spatial resolution (commonly smaller than 1 mm) of CT and MRI coupled with large portions of the body imaged generate immense datasets and the radiologists face the challenge of information overload. As an example, a CT scan of the chest with an axial matrix of 512 pixels (x-direction) by 512 pixels (y-direction) would have 262,144 pixels on a single slice. Thin-cut imaging of the chest provides 500 axial slices, each containing the 262,144 pixels, or roughly 131 million pixels in the data set. Second, is the challenge of detecting small lesions. One example of this is the challenge is the identification a small pulmonary nodule, which is a topic of great concern for radiologists and a top cause of litigation [19, 20]. Identifying a tumor at an small size and corresponding early stage is important in order to improve patient survival and reduce cost of treatment. A very

**Figure 5.**  Note that four flatscreen monitors are present. Methods for human machine interaction (HMI) to manipulate the images include a mouse

Augmented Reality and Virtual Reality: Initial Successes in Diagnostic Radiology

http://dx.doi.org/10.5772/intechopen.74317

11

Third, is the challenge of mentally building a 3D image from reviewing slices [21]. Depending on the clinical scenario and body part imaged, the radiologist can be tasked with following certain twisting and turning structures through the body such as following a blood vessel or loops of intestines.

Most radiologists spend a small fraction (<5%) of their total time in interpreting their imaging scans with advanced viewing methods [17]. Such non-AR/VR techniques include surface

The first 3D rendering technique to display the human body's anatomy was surface rendering (also known as shaded surface display). Through segmentation techniques such as thresholding to display only a prescribed set of pixels, apparent surfaces are displayed within the body. A virtual light source is used to provide surface shading. In surface rendering, only

A standard viewing method includes a flat screen, high-resolution diagnostic imaging monitor with keyboard and mouse, as shown in **Figure 5**. Typically, the radiologist will use the wheel on the back of the mouse to scroll through the stacks of images. Note that the radiologist also uses a microphone to dictate the radiology reports.

**Figure 4.** CT scan of the chest in lung windows showing oblique reformats (A, B) and coronal reformat (C). Note that the image (A) is the oblique plane image corresponding to the yellow line in (C). Note that the image (B) is the oblique plane image corresponding to the green line in (C). Note that image (C) is the coronal plane image corresponding to the red line in (A) and (B).

Augmented Reality and Virtual Reality: Initial Successes in Diagnostic Radiology http://dx.doi.org/10.5772/intechopen.74317 11

**Figure 5.**  Note that four flatscreen monitors are present.  Challenges in conventional viewing of the volumetric data*

First, is the challenge of information overload. The dramatic improvements in spatial resolution (commonly smaller than 1 mm) of CT and MRI coupled with large portions of the body imaged generate immense datasets and the radiologists face the challenge of information overload. As an example, a CT scan of the chest with an axial matrix of 512 pixels (x-direction) by 512 pixels (y-direction) would have 262,144 pixels on a single slice. Thin-cut imaging of the chest provides 500 axial slices, each containing the 262,144 pixels, or roughly 131 million pixels in the data set.

Second, is the challenge of detecting small lesions. One example of this is the challenge is the identification a small pulmonary nodule, which is a topic of great concern for radiologists and a top cause of litigation [19, 20]. Identifying a tumor at an small size and corresponding early stage is important in order to improve patient survival and reduce cost of treatment. A very deliberate slice-by-slice method takes considerable time.

Third, is the challenge of mentally building a 3D image from reviewing slices [21]. Depending on the clinical scenario and body part imaged, the radiologist can be tasked with following certain twisting and turning structures through the body such as following a blood vessel or loops of intestines.

#### *2.2.3. Advanced (non-AR/VR) viewing methods of the volumetric data*

Most radiologists spend a small fraction (<5%) of their total time in interpreting their imaging scans with advanced viewing methods [17]. Such non-AR/VR techniques include surface rendering and volume rendering.

#### *<IP_ADDRESS>. Surface rendering*

**Figure 4.** CT scan of the chest in lung windows showing oblique reformats (A, B) and coronal reformat (C). Note that the image (A) is the oblique plane image corresponding to the yellow line in (C). Note that the image (B) is the oblique plane image corresponding to the green line in (C). Note that image (C) is the coronal plane image corresponding to the

Occasionally, the radiologist needs to view an abnormality in another plane, other than the axial, sagittal, coronal imaging planes. In these instances, oblique plane reformats can be used with the images still viewed in a conventional slice-by-slice approach. See **Figure 4**. Curved

**Figure 3.** Image of the brain from an MRI scanner. This is of an axial T1-weighted sequence (A) with coronal (B) and

A standard viewing method includes a flat screen, high-resolution diagnostic imaging monitor with keyboard and mouse, as shown in **Figure 5**. Typically, the radiologist will use the wheel on the back of the mouse to scroll through the stacks of images. Note that the radiolo-

planar reformats can also be performed and have been shown to be beneficial [18].

gist also uses a microphone to dictate the radiology reports.

sagittal (C) images reconstructed.

10 State of the Art Virtual Reality and Augmented Reality Knowhow

red line in (A) and (B).

The first 3D rendering technique to display the human body's anatomy was surface rendering (also known as shaded surface display). Through segmentation techniques such as thresholding to display only a prescribed set of pixels, apparent surfaces are displayed within the body. A virtual light source is used to provide surface shading. In surface rendering, only

**Figure 6.** Shaded surface display (SSD) of the bones of the head and neck in three different projections (A, B, C). Note that while the density of the skull is intrinsically the same HU, the gray scale displayed changed due to the apparent light source. Note that true depth perception is not achieved because this figure is displayed on a flat computer screen or paper.

*<IP_ADDRESS>. Volume rendering*

See **Figure 8**.

AR/VR in diagnostic radiology.

The technique of volume rendering has been researched for many years by the computer graphics industry and has recently been applied to diagnostic radiology [22]. In volume rendering, a transfer function is applied to assign a color and opacity to each intensity value. As an example, voxels that correspond to the density of blood vessel are colored red and voxels that correspond to the density of bone are colored white. This has significantly helped radi-

**Figure 8.** Volume rendering image of an magnetic resonance angiogram (MRA), which is a type of magnetic resonance

Augmented Reality and Virtual Reality: Initial Successes in Diagnostic Radiology

http://dx.doi.org/10.5772/intechopen.74317

13

One of the key limitations of volume rendering is the overlapping structures [24, 25]. This limitation is significantly worse in settings such as viewing of the vasculature of the brain.

In the first sections of this chapter, we have reviewed the medical imaging equipment, conventional slice-by-slice techniques and the advanced 3D rendering methods including surface rendering and volume rendering. We will now review AR/VR in diagnostic radiology.

The previously discussed limitation of overlapping structures can be minimized through a process called depth-3-dimensional (D3D) imaging by providing unique images to each eye and display on AR/VR HDUs. D3D transforms cross-sectional imaging datasets and displays them on AR/VR head display units (HDUs). In doing so, an overall immersive viewing experience is created. In AR/VR radiology, the radiologist will wear a head display unit, which can

This section of the paper will be organized into three sections. First, we will discuss the process for generating an AR/VR image from a cross-sectional medical imaging dataset. Second, we will include results. Third, we will discuss the path forward and future opportunities of

be either a VR, AR or MR system. The basic concept is outlined in **Figure 9**.

ologists visualize complex 3D structures [23]. See **Figure 7**.

imaging (MRI) of the brain. Note the extensive areas of overlap, which limits evaluation.

**3. Review of AR/VR in diagnostic radiology**

a single surface is used.  However, there are a few limitations.

One limitation of only displaying a single surface is the fact that thresholding is used for one tissue type at a time and it can be difficult to understand the anatomic relationship of multiple different organ systems when only a single organ system is displayed. Another limitation is the fact that many organs are of similar density to their surroundings and it can be difficult to segment these structures out. Finally, since surface rendering images have been displayed on flat screen images true depth perception is not achieved. See **Figure 6**.

**Figure 7.** Volume rendering images of the bones and blood vessels of the head and neck in three different projections (A, B, C). Note that this differs from the shaded surface display since these volume rendering images showed both the bones and the blood vessels and the SSD only displayed the bones. Note that the carotid artery can be seen in (A). Note that in image (C), the shading from the apparent light source has been removed.

Augmented Reality and Virtual Reality: Initial Successes in Diagnostic Radiology http://dx.doi.org/10.5772/intechopen.74317 13

**Figure 8.** Volume rendering image of an magnetic resonance angiogram (MRA), which is a type of magnetic resonance imaging (MRI) of the brain. Note the extensive areas of overlap, which limits evaluation.

#### *<IP_ADDRESS>. Volume rendering*

**Figure 7.** Volume rendering images of the bones and blood vessels of the head and neck in three different projections (A, B, C). Note that this differs from the shaded surface display since these volume rendering images showed both the bones and the blood vessels and the SSD only displayed the bones. Note that the carotid artery can be seen in (A). Note that in

a single surface is used.  Shaded surface display (SSD) of the bones of the head and neck in three different projections (A, B, C). Note that while the density of the skull is intrinsically the same HU, the gray scale displayed changed due to the apparent light source. Note that true depth perception is not achieved because this figure is displayed on a flat computer screen

One limitation of only displaying a single surface is the fact that thresholding is used for one tissue type at a time and it can be difficult to understand the anatomic relationship of multiple different organ systems when only a single organ system is displayed. Another limitation is the fact that many organs are of similar density to their surroundings and it can be difficult to segment these structures out. Finally, since surface rendering images have been displayed on

flat screen images true depth perception is not achieved. See **Figure 6**.

image (C), the shading from the apparent light source has been removed.

body. However, there are a few limitations.

12 State of the Art Virtual Reality and Augmented Reality Knowhow

or paper.

The technique of volume rendering has been researched for many years by the computer graphics industry and has recently been applied to diagnostic radiology [22]. In volume rendering, a transfer function is applied to assign a color and opacity to each intensity value. As an example, voxels that correspond to the density of blood vessel are colored red and voxels that correspond to the density of bone are colored white. This has significantly helped radiologists visualize complex 3D structures [23]. See **Figure 7**.

One of the key limitations of volume rendering is the overlapping structures [24, 25]. This limitation is significantly worse in settings such as viewing of the vasculature of the brain. See **Figure 8**.

### **3. Review of AR/VR in diagnostic radiology**

In the first sections of this chapter, we have reviewed the medical imaging equipment, conventional slice-by-slice techniques and the advanced 3D rendering methods including surface rendering and volume rendering. We will now review AR/VR in diagnostic radiology.

The previously discussed limitation of overlapping structures can be minimized through a process called depth-3-dimensional (D3D) imaging by providing unique images to each eye and display on AR/VR HDUs. D3D transforms cross-sectional imaging datasets and displays them on AR/VR head display units (HDUs). In doing so, an overall immersive viewing experience is created. In AR/VR radiology, the radiologist will wear a head display unit, which can be either a VR, AR or MR system. The basic concept is outlined in **Figure 9**.

This section of the paper will be organized into three sections. First, we will discuss the process for generating an AR/VR image from a cross-sectional medical imaging dataset. Second, we will include results. Third, we will discuss the path forward and future opportunities of AR/VR in diagnostic radiology.

intensity or texture surrounding a voxel, which provides a classifier algorithm with measurements that are used to distinguish one class from another, where classes correspond to the

Augmented Reality and Virtual Reality: Initial Successes in Diagnostic Radiology

http://dx.doi.org/10.5772/intechopen.74317

15

Historical data from numerous similar imaging exams will be collected and "ground truthed" so that a neural network or deep learning classifier can be used to improve classification accuracy. Techniques will be used to first classify the input imagery into categories such as skeletal, fat, normal breast tissue, breast cancer, etc. [26, 27]. The user will need to input the tissue-type selection until an intelligent system will automatically determine the anatomy of the input imagery. User input of the tissue-type selection will be used as historical data to

The goal of the 3D filtering algorithm is to make the most understandable representation of the imagery to the radiologist [26, 27]. In order to accomplish this goal, it utilizes the image tissue-type from the 3D segmentation algorithm to determine which tissue types should be

Current graphic hardware have limitations on how many voxels that can be displayed. Therefore, it is necessary to allocate percentages to the various tissue types based on priority. Those tissue voxels that must be seen from a clinically importance perspective (e.g., tumor) are given the highest percentage. Other tissues that are not as clinically important (e.g., subcutaneous fat) are sparsely sampled. There are two benefits of this process. First of all, this allows the operator to understand the high priority tissue (e.g., tumor) in proper context (e.g., tumor is touching the spine). It can be difficult or impossible to understand exact position of the tumor when all the surrounding tissue has been removed. Second of all, this lower priority tissue needs to be somewhat more transparent so the higher priority tissue can be seen through the lower priority tissue. However, transparency allocated may not necessarily correspond to tissue density. Additionally, thinning the tissue between the viewing perspectives and the area of interest makes it much easier to see the area of interest

User selected filtering is employed to enable the operator to slice away tissue sections to facilitate viewing of areas of interest. Additional implementation of various geometric volumes and surfaces to temporarily remove sections of the voxel cloud may be beneficial. The opera-

The goal of the 3D rendering is to replicate, insofar as possible, what a radiologist would see if looking with his/her own two eyes through the skin and into the body were possible. A slightly different image would be seen from each eye from its particular vantage point. This difference allows one to see depth. The basic concept is illustrated in **Figure 10**. The geometry

tor can also enable or disable display of specific tissue types.

behind the approach is illustrated in **Figures 11** and **12**.

different tissue types.

drive the classifier.

*3.1.2. 3D filtering algorithm*

given highest priorities.

accurately.

*3.1.3. 3D rendering engine*

**Figure 9.** Overview of the system. Cross-sectional imaging data is sent via the digital recording device to the general purpose processor and then sending a unique image to the left eye and a unique image to the right eye a head display unit (HDU) [26].

#### **3.1. Image processing for AR/VR**

To optimize visualization, the D3D software suite must provide the capability for visualizing medical imagery rendered in a true 3 dimensional representation [26, 27]. In order to accomplish this, input imagery is converted from 2 dimensional images into a 3 dimensional voxel space, segmented into distinct tissue types; and then filtered. Finally, rendering is then performed wherein the rendering engine computes a left and right view. This allows the operator to visualize the data using the same stereopsis that our eyes and brains have spent a lifetime interpreting and processing.

#### *3.1.1. 3D segmentation*

During the 3D segmentation process, each image pixel in the input imagery is treated as a voxel, a three dimensional entity with length, width, and height. Each voxel is read into a large 3D array. The 3D array is analyzed for similarity to its neighbors, and for context clues that imply similarity to historical training data.

Mean and variance statistics largely drive the similarity analysis. Every voxel is compared to all the neighboring voxels in the 3x3 (or 5x5 or larger) nearest neighbor region. The most similar voxels are assigned the same class designation. The 3x3 variance is used to determine the width of the local intensity distribution. The width of the local intensity distribution in turn determines the threshold applied for declaring similarity between voxels. Simple local features on the neighborhood are computed around each voxel to describe the local average intensity or texture surrounding a voxel, which provides a classifier algorithm with measurements that are used to distinguish one class from another, where classes correspond to the different tissue types.

Historical data from numerous similar imaging exams will be collected and "ground truthed" so that a neural network or deep learning classifier can be used to improve classification accuracy. Techniques will be used to first classify the input imagery into categories such as skeletal, fat, normal breast tissue, breast cancer, etc. [26, 27]. The user will need to input the tissue-type selection until an intelligent system will automatically determine the anatomy of the input imagery. User input of the tissue-type selection will be used as historical data to drive the classifier.

#### *3.1.2. 3D filtering algorithm*

**3.1. Image processing for AR/VR**

14 State of the Art Virtual Reality and Augmented Reality Knowhow

interpreting and processing.

that imply similarity to historical training data.

*3.1.1. 3D segmentation*

unit (HDU) [26].

To optimize visualization, the D3D software suite must provide the capability for visualizing medical imagery rendered in a true 3 dimensional representation [26, 27]. In order to accomplish this, input imagery is converted from 2 dimensional images into a 3 dimensional voxel space, segmented into distinct tissue types; and then filtered. Finally, rendering is then performed wherein the rendering engine computes a left and right view. This allows the operator to visualize the data using the same stereopsis that our eyes and brains have spent a lifetime

**Figure 9.** Overview of the system. Cross-sectional imaging data is sent via the digital recording device to the general purpose processor and then sending a unique image to the left eye and a unique image to the right eye a head display

During the 3D segmentation process, each image pixel in the input imagery is treated as a voxel, a three dimensional entity with length, width, and height. Each voxel is read into a large 3D array. The 3D array is analyzed for similarity to its neighbors, and for context clues

Mean and variance statistics largely drive the similarity analysis. Every voxel is compared to all the neighboring voxels in the 3x3 (or 5x5 or larger) nearest neighbor region. The most similar voxels are assigned the same class designation. The 3x3 variance is used to determine the width of the local intensity distribution. The width of the local intensity distribution in turn determines the threshold applied for declaring similarity between voxels. Simple local features on the neighborhood are computed around each voxel to describe the local average The goal of the 3D filtering algorithm is to make the most understandable representation of the imagery to the radiologist [26, 27]. In order to accomplish this goal, it utilizes the image tissue-type from the 3D segmentation algorithm to determine which tissue types should be given highest priorities.

Current graphic hardware have limitations on how many voxels that can be displayed. Therefore, it is necessary to allocate percentages to the various tissue types based on priority. Those tissue voxels that must be seen from a clinically importance perspective (e.g., tumor) are given the highest percentage. Other tissues that are not as clinically important (e.g., subcutaneous fat) are sparsely sampled. There are two benefits of this process. First of all, this allows the operator to understand the high priority tissue (e.g., tumor) in proper context (e.g., tumor is touching the spine). It can be difficult or impossible to understand exact position of the tumor when all the surrounding tissue has been removed. Second of all, this lower priority tissue needs to be somewhat more transparent so the higher priority tissue can be seen through the lower priority tissue. However, transparency allocated may not necessarily correspond to tissue density. Additionally, thinning the tissue between the viewing perspectives and the area of interest makes it much easier to see the area of interest accurately.

User selected filtering is employed to enable the operator to slice away tissue sections to facilitate viewing of areas of interest. Additional implementation of various geometric volumes and surfaces to temporarily remove sections of the voxel cloud may be beneficial. The operator can also enable or disable display of specific tissue types.

#### *3.1.3. 3D rendering engine*

The goal of the 3D rendering is to replicate, insofar as possible, what a radiologist would see if looking with his/her own two eyes through the skin and into the body were possible. A slightly different image would be seen from each eye from its particular vantage point. This difference allows one to see depth. The basic concept is illustrated in **Figure 10**. The geometry behind the approach is illustrated in **Figures 11** and **12**.

**Figure 10.** The 3D volume shows a gray voxel and a black voxel. In volume rendering (A), the gray voxel is hidden behind the black voxel and only the black voxel is displayed on the 2D volume rendering image. In depth-3-dimensional (D3D), both the gray and the black voxels are projected onto the imaging plane; furthermore, the user can distinguish that the gray voxel is farther than the black voxel because the D3D process displayed with AR/VR HDUs provides depth perception.

The interocular distance can of course be altered to change stereopsis. The rendering engine allows the operator to comfortably move the viewing position, zoom, and rotate about the pitch, roll or yaw axes. The angular field of view can be changed. The volume of interest (VOI) can be rotated. The viewing perspectives can also be rotated. See **Figure 14**. And to improve

**Figure 13.** Figure illustrates left eye viewing perspective (LEVP) and right eye viewing perspective (REVP). The angles from the LEVP through the volume of interest can converge to a convergence point "c" to enhance visualization of a

small volume of interest [26, 27].

**Figure 12.** This is an overview illustrating the fact that the angle from the left eye viewing perspective (LEVP) to voxel (0, 64, 64) is different from the angle from the right eye viewing perspective (REVP) to voxel (0, 64, 64). This voxel (0, 64, 64) will be displayed on a different spot on the left eye display as compared with the right eye display within the HDU.

Augmented Reality and Virtual Reality: Initial Successes in Diagnostic Radiology

http://dx.doi.org/10.5772/intechopen.74317

17

**Figure 11.** This is an overview of the depth-3-dimensional (D3D) processing system for the left eye viewing perspective (LEVP). Reprinted from the Douglas et al. [12] with permission from Dove Medical Press Ltd.

The Left Eye Viewing Perspective (LEVP) would be at an observer/ radiologist selected point to inspect the volume. The Right Eye Viewing Perspective (REVP) would be the inter-ocular distance to the right of the LEVP and similar math would be applied to generate the slightly different image presented to the right eye display. The rendering engine generates the left and right eye views to provide a true 3D data visualization to be displayed in the AR/VR HDUs [26, 27]. This component relies heavily on the Graphics Processing Units (GPU) built into the graphics card. The rendering engine supports a convergence depth adjustment for fine-tuning the operator's focal point as shown in **Figure 13**.

Augmented Reality and Virtual Reality: Initial Successes in Diagnostic Radiology http://dx.doi.org/10.5772/intechopen.74317 17

**Figure 12.** This is an overview illustrating the fact that the angle from the left eye viewing perspective (LEVP) to voxel (0, 64, 64) is different from the angle from the right eye viewing perspective (REVP) to voxel (0, 64, 64). This voxel (0, 64, 64) will be displayed on a different spot on the left eye display as compared with the right eye display within the HDU.

The interocular distance can of course be altered to change stereopsis. The rendering engine allows the operator to comfortably move the viewing position, zoom, and rotate about the pitch, roll or yaw axes. The angular field of view can be changed. The volume of interest (VOI) can be rotated. The viewing perspectives can also be rotated. See **Figure 14**. And to improve

The Left Eye Viewing Perspective (LEVP) would be at an observer/ radiologist selected point to inspect the volume. The Right Eye Viewing Perspective (REVP) would be the inter-ocular distance to the right of the LEVP and similar math would be applied to generate the slightly different image presented to the right eye display. The rendering engine generates the left and right eye views to provide a true 3D data visualization to be displayed in the AR/VR HDUs [26, 27]. This component relies heavily on the Graphics Processing Units (GPU) built into the graphics card. The rendering engine supports a convergence depth adjustment for fine-tuning

**Figure 11.** This is an overview of the depth-3-dimensional (D3D) processing system for the left eye viewing perspective

(LEVP). Reprinted from the Douglas et al. [12] with permission from Dove Medical Press Ltd.

**Figure 10.** The 3D volume shows a gray voxel and a black voxel. In volume rendering (A), the gray voxel is hidden behind the black voxel and only the black voxel is displayed on the 2D volume rendering image. In depth-3-dimensional (D3D), both the gray and the black voxels are projected onto the imaging plane; furthermore, the user can distinguish that the gray voxel is farther than the black voxel because the D3D process displayed with AR/VR HDUs provides depth

the operator's focal point as shown in **Figure 13**.

perception.

16 State of the Art Virtual Reality and Augmented Reality Knowhow

**Figure 13.** Figure illustrates left eye viewing perspective (LEVP) and right eye viewing perspective (REVP). The angles from the LEVP through the volume of interest can converge to a convergence point "c" to enhance visualization of a small volume of interest [26, 27].

**Figure 14.** Five examples of augmented reality/virtual reality viewing options with the D3D technology. Notes: (A) initial viewing angle into the volume of interest. (B) Illustrates increasing the interocular distance, which provides for increased binocular disparity. (C) Illustrates changing of the angular FOV, so rather than an α FOV of 40°, it changes to an α FOV of 10° and rather than a β FOV of 40°, it changes to a β FOV of 10°. This serves to focus on a particular region within the volume. (D) Illustrates rotation of the VOI, so that the radiologist can have a different viewing perspective. (E) Illustrates rotation of the viewing perspective, which is similar to the radiologist turning the head to see new features of the image, allowing for improved HMI. Note that in (A–E), the center pixels (i.e., α = 0° and β = 0°) for both LEVP and REVP converge at the center of the VOI (i.e., voxel [32, 32, 32]), such that the VOI is optimally presented to the user. Abbreviations: FOV, field of view; HMI, human machine interface; LEVP, left eye viewing perspective; REVP, right eye viewing perspective; VOI, volume of interest. Reprinted from the Douglas et al. [12] with permission from Dove Medical Press Ltd.

visibility, the operator can select any number of color palettes for different tissue classes. The rendering engine also incorporates a graphical overlay for image markup.

#### **3.2. AR/VR control systems used in diagnostic radiology**

The field of AR/VR is evolving with new and changing HDUs and gaming control systems coming on the market. In order to illustrate what we have used in our prior research, we illustrate example control systems shown and how various control buttons correspond with basic maneuvers and advanced maneuvers. See **Figures 15**–**17**.

**3.3. Initial results of AR/VR in radiology**

movements, which are defined as a single controls input at one time.

movements, which are defined as a single controls input at one time.

As previously discussed, there have been numerous applications of AR/VR in surgery to include pre-operative planning, education, and intra-operative assistance [6]. However, AR/ VR is not yet FDA approved in diagnostic radiology, but is being actively researched by DXC

**Figure 17.** This figure illustrates a graphical user interface (GUI) used by D3D technologies programmed with complex

**Figure 16.** This figure illustrates a graphical user interface (GUI) used by D3D technologies programmed with basic

Augmented Reality and Virtual Reality: Initial Successes in Diagnostic Radiology

http://dx.doi.org/10.5772/intechopen.74317

19

Breast Cancer is one of the leading causes of death in women [28, 29]. Breast calcifications are extremely common and are present in up to 86% of mammograms [30]. The calcifications are classified by distribution with a linear and branching pattern suspicious for ductal carcinoma in situ (DCIS) [31, 32]. Standard mammographic views may not reveal the true linear and

Technologies/D3D Technologies with an initial focus in breast cancer imaging [11–13].

**Figure 15.** Control system used by depth-3-dimensional (D3D) technologies. See example buttons and functions.

Augmented Reality and Virtual Reality: Initial Successes in Diagnostic Radiology http://dx.doi.org/10.5772/intechopen.74317 19

**Figure 16.** This figure illustrates a graphical user interface (GUI) used by D3D technologies programmed with basic movements, which are defined as a single controls input at one time.

**Figure 17.** This figure illustrates a graphical user interface (GUI) used by D3D technologies programmed with complex movements, which are defined as a single controls input at one time.

#### **3.3. Initial results of AR/VR in radiology**

**Figure 15.** Control system used by depth-3-dimensional (D3D) technologies. See example buttons and functions.

visibility, the operator can select any number of color palettes for different tissue classes. The

VOI, volume of interest. Reprinted from the Douglas et al. [12] with permission from Dove Medical Press Ltd.

**Figure 14.** Five examples of augmented reality/virtual reality viewing options with the D3D technology. Notes: (A) initial viewing angle into the volume of interest. (B) Illustrates increasing the interocular distance, which provides for increased binocular disparity. (C) Illustrates changing of the angular FOV, so rather than an α FOV of 40°, it changes to an α FOV of 10° and rather than a β FOV of 40°, it changes to a β FOV of 10°. This serves to focus on a particular region within the volume. (D) Illustrates rotation of the VOI, so that the radiologist can have a different viewing perspective. (E) Illustrates rotation of the viewing perspective, which is similar to the radiologist turning the head to see new features of the image, allowing for improved HMI. Note that in (A–E), the center pixels (i.e., α = 0° and β = 0°) for both LEVP and REVP converge at the center of the VOI (i.e., voxel [32, 32, 32]), such that the VOI is optimally presented to the user. Abbreviations: FOV, field of view; HMI, human machine interface; LEVP, left eye viewing perspective; REVP, right eye viewing perspective;

The field of AR/VR is evolving with new and changing HDUs and gaming control systems coming on the market. In order to illustrate what we have used in our prior research, we illustrate example control systems shown and how various control buttons correspond with basic

rendering engine also incorporates a graphical overlay for image markup.

**3.2. AR/VR control systems used in diagnostic radiology**

18 State of the Art Virtual Reality and Augmented Reality Knowhow

maneuvers and advanced maneuvers. See **Figures 15**–**17**.

As previously discussed, there have been numerous applications of AR/VR in surgery to include pre-operative planning, education, and intra-operative assistance [6]. However, AR/ VR is not yet FDA approved in diagnostic radiology, but is being actively researched by DXC Technologies/D3D Technologies with an initial focus in breast cancer imaging [11–13].

Breast Cancer is one of the leading causes of death in women [28, 29]. Breast calcifications are extremely common and are present in up to 86% of mammograms [30]. The calcifications are classified by distribution with a linear and branching pattern suspicious for ductal carcinoma in situ (DCIS) [31, 32]. Standard mammographic views may not reveal the true linear and branching pattern due to suboptimal view point and lack of depth perception. Therefore, the D3D AR imaging system was tested on a simulated set of microcalcifications. The radiologist who rated the AR system found that when the microcalcifications appeared as a cluster when viewed from a single perspective, but with rotation and the AR HDU appeared as a linear and branching pattern. See **Figure 18**.

In addition to microcalcifications, breast cancer can also present as a mass on imaging. Characterization of the shape and margins is important in determining whether the breast mass is malignant or benign. Dedicated breast CT provides high spatial resolution of breast masses [33]. Recent data has shown the importance of characterizing tumor morphology [34]. Therefore, the D3D AR system was also tested in viewing of a known breast cancer [13]. Malignant features of spiculations were noted to be more conspicuous on the D3D system than the native CT. See **Figure 19**.

**3.4. Future work of AR/VR in radiology**

haptic feedback.

**4. Conclusions**

Both the fields of AR/VR and diagnostic radiology are large and rapidly growing. One of the most common reasons for performing diagnostic imaging is for cancer. Early determination of whether a particular therapy regimen is working would be extremely helpful to improve survival and would save costs. We outline a flow chart below as a recommended process for

Augmented Reality and Virtual Reality: Initial Successes in Diagnostic Radiology

http://dx.doi.org/10.5772/intechopen.74317

21

**Figure 20.** Overview of how D3D AR/VR system could be used to evaluate how a tumor at multiple time points.

It is foreseeable that AR/VR will one day play a major role in diagnostic radiology. Computer aided diagnosis (CAD) will help to identify the abnormalities. The role of the radiologist will include assessing an abnormality in great detail to appreciate subtle changes to diagnose accurately and assess treatment response. Features of AR/VR including depth perception, head tracking, improved GUIs create an overall immersive environment allowing for new opportunities in diagnostic radiology. Radiologists will interact with medical images in ways never before including voice commands, gestures or through handheld devices with

Both diagnostic radiology and AR/VR are large and rapidly growing fields. In this chapter, we reviewed diagnostic medical imaging equipment, data storage, conventional slice-by-slice analysis and advanced 3D rendering techniques including surface rendering and volume rendering. We then introduced D3D processing of images, so volumetric medical imaging can be displayed in AR/VR HDUs. We showed a variety of GUIs including controllers and joysticks

introducing AR/VR evaluating a cancer at multiple time points. See **Figure 20**.

**Figure 18.**  Notes: The microcalcifications are shown in (A), (B), and (C), but in different projections. (A) Initial single projection viewed by the radiologist (EW) was thought to represent a cluster as indicated by the circle. Note that some of the dots are bigger, which is due to the fact that they are closer to the viewing perspective and the calcifications subtend a larger angular resolution from the viewing perspective. (B) The cluster is now rotated and viewed with both an LEVP and an REVP. The top portion (arrows) was thought to represent a linear pattern by the radiologist (EW) with the linear portion indicated by the arrows. (C) Same cluster as (A) and (B), but now rotated once again. The bottom portion (arrows) was thought to represent a branching pattern by the radiologist (EW), where two branches originate at the bottom of the image as indicated by the arrows. In retrospect, the viewing perspective (A) was closest to the top of the microcalcifications as if one were looking down on linear, branching pattern seen from the side view on (B) and (C). Abbreviations: D3D, depth 3-dimensional; LEVP, left eye viewing perspective; REVP, right eye viewing perspective. Reprinted from the, Douglas et al. [12] with permission.

**Figure 19.** (A) Contrast-enhanced breast CT demonstrates the mass with small spiculations extending from the margins. (B and C) same mass from breast CT exam as seen in (A), but viewed with D3D where (B) represents the left eye viewing perspective (LEVP) and (C) represents the right eye viewing perspective (REVP). The red box illustrates the 3D cursor used. (D&E) represent the same mass from the breast CT, but zoomed in and viewed from a different perspective with (D) representing the LEVP and (E) representing the REVP. Red arrows show spiculations extending from the margins of the mass. The red circle represents a spiculation sticking out toward the user, which was well seen when rotating with the D3D system. Reprinted from the Douglas et al., [13] with permission.

Augmented Reality and Virtual Reality: Initial Successes in Diagnostic Radiology http://dx.doi.org/10.5772/intechopen.74317 21

**Figure 20.** Overview of how D3D AR/VR system could be used to evaluate how a tumor at multiple time points.

#### **3.4. Future work of AR/VR in radiology**

branching pattern due to suboptimal view point and lack of depth perception. Therefore, the D3D AR imaging system was tested on a simulated set of microcalcifications. The radiologist who rated the AR system found that when the microcalcifications appeared as a cluster when viewed from a single perspective, but with rotation and the AR HDU appeared as a linear and

In addition to microcalcifications, breast cancer can also present as a mass on imaging. Characterization of the shape and margins is important in determining whether the breast mass is malignant or benign. Dedicated breast CT provides high spatial resolution of breast masses [33]. Recent data has shown the importance of characterizing tumor morphology [34]. Therefore, the D3D AR system was also tested in viewing of a known breast cancer [13]. Malignant features of spiculations were noted

**Figure 18.**  Notes: The microcalcifications are shown in (A), (B), and (C), but in different projections. (A) Initial single projection viewed by the radiologist (EW) was thought to represent a cluster as indicated by the circle. Note that some of the dots are bigger, which is due to the fact that they are closer to the viewing perspective and the calcifications subtend a larger angular resolution from the viewing perspective. (B) The cluster is now rotated and viewed with both an LEVP and an REVP. The top portion (arrows) was thought to represent a linear pattern by the radiologist (EW) with the linear portion indicated by the arrows. (C) Same cluster as (A) and (B), but now rotated once again. The bottom portion (arrows) was thought to represent a branching pattern by the radiologist (EW), where two branches originate at the bottom of the image as indicated by the arrows. In retrospect, the viewing perspective (A) was closest to the top of the microcalcifications as if one were looking down on linear, branching pattern seen from the side view on (B) and (C). Abbreviations: D3D, depth 3-dimensional; LEVP, left eye viewing perspective; REVP, right eye viewing perspective.

**Figure 19.** (A) Contrast-enhanced breast CT demonstrates the mass with small spiculations extending from the margins. (B and C) same mass from breast CT exam as seen in (A), but viewed with D3D where (B) represents the left eye viewing perspective (LEVP) and (C) represents the right eye viewing perspective (REVP). The red box illustrates the 3D cursor used. (D&E) represent the same mass from the breast CT, but zoomed in and viewed from a different perspective with (D) representing the LEVP and (E) representing the REVP. Red arrows show spiculations extending from the margins of the mass. The red circle represents a spiculation sticking out toward the user, which was well seen when rotating with

to be more conspicuous on the D3D system than the native CT. See **Figure 19**.

branching pattern. See **Figure 18**.

20 State of the Art Virtual Reality and Augmented Reality Knowhow

Reprinted from the, Douglas et al. [12] with permission.

the D3D system. Reprinted from the Douglas et al., [13] with permission.

Both the fields of AR/VR and diagnostic radiology are large and rapidly growing. One of the most common reasons for performing diagnostic imaging is for cancer. Early determination of whether a particular therapy regimen is working would be extremely helpful to improve survival and would save costs. We outline a flow chart below as a recommended process for introducing AR/VR evaluating a cancer at multiple time points. See **Figure 20**.

It is foreseeable that AR/VR will one day play a major role in diagnostic radiology. Computer aided diagnosis (CAD) will help to identify the abnormalities. The role of the radiologist will include assessing an abnormality in great detail to appreciate subtle changes to diagnose accurately and assess treatment response. Features of AR/VR including depth perception, head tracking, improved GUIs create an overall immersive environment allowing for new opportunities in diagnostic radiology. Radiologists will interact with medical images in ways never before including voice commands, gestures or through handheld devices with haptic feedback.

#### **4. Conclusions**

Both diagnostic radiology and AR/VR are large and rapidly growing fields. In this chapter, we reviewed diagnostic medical imaging equipment, data storage, conventional slice-by-slice analysis and advanced 3D rendering techniques including surface rendering and volume rendering. We then introduced D3D processing of images, so volumetric medical imaging can be displayed in AR/VR HDUs. We showed a variety of GUIs including controllers and joysticks with a variety of functions achieved by various functions. Imaging cases were illustrated with a specific focus on breast cancer. We concluded with a discussion on future techniques in comparing how a tumor changes appearance over multiple time points.

[8] Mettler FA Jr, Wiest PW, Locken JA, Kelsey CA. CT scanning: Patterns of use and dose.

Augmented Reality and Virtual Reality: Initial Successes in Diagnostic Radiology

http://dx.doi.org/10.5772/intechopen.74317

23

[9] Mitchell DG, Parker L, Sunshine JH, Levin DC. Body MR imaging and CT volume: Variations and trends based on an analysis of medicare and fee-for-service health insur-

[10] Boone JM, Brunberg JA. Computed tomography use in a tertiary care university hospi-

[11] Douglas DB, Wilke CA, Gibson JD, Boone JM, Wintermark M. Augmented reality: Advances in diagnostic imaging. Multimodal Technologies and Interaction. 2017;**1**(4):29

[12] Douglas DB, Petricoin EF, Liotta L, Wilson E. D3D augmented reality imaging system: Proof of concept in mammography. Medical Devices (Auckland). 2016;**9**:277-283 [13] Douglas DB, Boone JM, Petricoin E, Liotta L, Wilson E. Augmented reality imaging system: 3D viewing of a breast cancer. Journal of Natural Sciences. 2016;**2**(9):215

[14] Douglas DB, Chaudhari R, Zhao JM, Gullo J, Kirkland J, Douglas PK, et al. Perfusion imaging in acute traumatic brain injury. Neuroimaging Clinics of North America;

[15] Douglas DB, Iv M, Douglas PK, Anderson A, Vos SB, Bammer R, et al. Diffusion tensor imaging of TBI: Potentials and challenges. Topics in Magnetic Resonance Imaging:

[16] Douglas D, Goubran M, Wilson E, Xu G, Tripathi P, Holley D, et al. Correlation between arterial spin labeling MRI and dynamic FDG on PET-MR in Alzheimer's disease and

[17] David Douglas MDDbcnnmppiC, USA. Dual Board Certified Neuroradiologist Nuclear

[18] Ochi T, Shimizu K, Yasuhara Y, Shigesawa T, Mochizuki T, Ikezoe J. Curved planar reformatted CT angiography: Usefulness for the evaluation of aneurysms at the carotid

[19] White CS, Salis AI, Meyer CA. Missed lung cancer on chest radiography and computed tomography: Imaging and medicolegal issues. Journal of Thoracic Imaging.

[20] Gurney JW. Missed lung cancer at CT: Imaging findings in nine patients. Radiology.

[21] Ferroli P, Tringali G, Acerbi F, Schiariti M, Broggi M, Aquino D, et al. Advanced 3-dimensional planning in neurosurgery. Neurosurgery. 2013;**72**(suppl\_1):A54-A62

[22] Hohne K-H, Bomans M, Tiede U, Riemer M, editors. Display of Multiple 3D–Objects Using the Generalized Voxel-Model. Medical Imaging II. Newport Beach: SPIE; 1988 [23] Calhoun PS, Kuszyk BS, Heath DG, Carley JC, Fishman EK. Three-dimensional volume rendering of spiral CT data: Theory and method. Radiographics: A Review Publication

non-Alzhiemer's disease patients. EJNMMI Physics. 2015;**2**(Suppl 1):83

siphon. American Journal of Neuroradiology. 1999;**20**(6):1025-1030

of the Radiological Society of North America, Inc. 1999;**19**(3):745-764

Medicine Physician Practicing in California USA; 2017

ance databases. American Journal of Roentgenology. 2002;**179**(1):27-31

tal. Journal of the American College of Radiology. 2008;**5**(2):132-138

Journal of Radiological Protection. 2000;**20**(4):353-359

2017;**28**:55-65

1999;**14**(1):63-68

1996;**199**(1):117-122

TMRI. 2015;**24**(5):241-251

#### **Conflict of interest**

All authors either have direct financial interest or are employees of D3D Technologies or DXC Technologies.

#### **Author details**

David B. Douglas1 \*, Demetri Venets2 , Cliff Wilke3 , David Gibson3 , Lance Liotta2 , Emanuel Petricoin2 , Buddy Beck4 and Robert Douglas4

#### **References**

[8] Mettler FA Jr, Wiest PW, Locken JA, Kelsey CA. CT scanning: Patterns of use and dose. Journal of Radiological Protection. 2000;**20**(4):353-359

with a variety of functions achieved by various functions. Imaging cases were illustrated with a specific focus on breast cancer. We concluded with a discussion on future techniques in

All authors either have direct financial interest or are employees of D3D Technologies or DXC

, David Gibson3

, Lance Liotta2

,

, Cliff Wilke3

and Robert Douglas4

[1] Lovo EE, Quintana JC, Puebla MC, Torrealba G, Santos JL, Lira IH, et al. A novel, inexpensive method of image coregistration for applications in image-guided surgery using

[2] Baus O, Bouchard S. Moving from virtual reality exposure-based therapy to augmented reality exposure-based therapy: A review. Frontiers in Human Neuroscience. 2014;**8**:2 [3] Merel T. The 7 Drivers of the \$150 billion AR/VR Industry. Aol Tech; 2015. https://tech-

[4] Statistics NCfH. Health, United States, 2016: With chartbook on long-term trends in

[5] Insider B. Available from: http://www.businessinsider.com/20-most-expensive-surgeries-in-

[6] Douglas DB, Wilke CA, Gibson D, Petricoin EF, Liotta L. Virtual reality and augmented

[7] Mitchell JM, LaGalia RR. Controlling the escalating use of advanced imaging: The role of radiology benefit management programs. Medical Care Research and Review.

augmented reality. Neurosurgery. 2007;**60**(4 Suppl 2):366-371; discussion 71-2

crunch.com/2015/07/08/the-7-drivers-of-the-150-billion-arvr-industry/

reality: Advances in surgery. O Biologico. 2017;**2**(5):1-8

comparing how a tumor changes appearance over multiple time points.

\*, Demetri Venets2

\*Address all correspondence to<EMAIL_ADDRESS>
, Buddy Beck4

22 State of the Art Virtual Reality and Augmented Reality Knowhow

3 Hewlett-Packard/DXC Technologies, VA, USA

**Conflict of interest**

Technologies.

**Author details**

David B. Douglas1

Emanuel Petricoin2

**References**

health. 2017

the-us-2014-2

2009;**66**(3):339-351

1 Stanford University, CA, USA

4 D3D Technologies, VA, USA

2 George Mason University, VA, USA

[24] Fishman EK, Ney DR, Heath DG, Corl FM, Horton KM, Johnson PT. Volume rendering versus maximum intensity projection in CT angiography: What works best, when, and why. Radiographics: A Review Publication of the Radiological Society of North America, Inc. 2006;**26**(3):905-922

**Chapter 3**

**Provisional chapter**

**Connect Smart Cities and Heritage Through**

**Connect Smart Cities and Heritage Through**

DOI: 10.5772/intechopen.74363

This chapter aims to connect the digital resources of knowledge with the historical and cultural heritage in the context of smart cities. Specifically, combining the joint intervention in the Real Sitio of El Pardo and Aranjuez, as well as the Foundation Square of the University of Alcalá, both in Spain. Through traditional historical research and the innovative use of new technologies like augmented reality (AR), a historical, biological and cultural heritage is conserved, consisting of forests, gardens, agricultural spaces, urban centers and palatial residences. Cultural and artistic heritage is a resource of the first magnitude for the sustainable development of smart cities. It evolves with time and society; it is this that determines what goods are to be conserved and protected for posterity, according to the values attributed to them. Hence, the importance of achieving an awareness in society plays an active part in the conservation, enjoyment and dissemination of heritage. In this context, the augmented reality is presented as a powerful tool for contextualizing and disseminating the heritage, as well as to make the resources created more accessible, making an innovative use of the new technologies applied to the transfer of knowledge

and the enhancement of a country's cultural and historical heritage.

by forests, gardens, agricultural spaces, urban centers and palatial residences.

© 2016 The Author(s). Licensee InTech. This chapter is distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/3.0), which permits unrestricted use, distribution,

© 2018 The Author(s). Licensee IntechOpen. This chapter is distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/3.0), which permits unrestricted use,

distribution, and reproduction in any medium, provided the original work is properly cited.

and reproduction in any medium, provided the original work is properly cited.

This project, through the use of new technologies in the dissemination and transfer of knowledge to society, seeks to preserve and improve knowledge of these royal sites and the city of Alcala de Henares, of enormous environmental, cultural and educational value, constituted

**Keywords:** augmented reality, smart cities, heritage, royal sites, historical architecture

Félix Labrador Arroyo, Julián de la Fuente Prieto and

Félix Labrador Arroyo, Julián de la Fuente Prieto

Additional information is available at the end of the chapter

Additional information is available at the end of the chapter

http://dx.doi.org/10.5772/intechopen.74363

**Augmented Reality**

**Augmented Reality**

Enrique Castaño Perea

**Abstract**

**1. Introduction**

and Enrique Castaño Perea

#### **Connect Smart Cities and Heritage Through Augmented Reality Connect Smart Cities and Heritage Through Augmented Reality**

DOI: 10.5772/intechopen.74363

Félix Labrador Arroyo, Julián de la Fuente Prieto and Enrique Castaño Perea Félix Labrador Arroyo, Julián de la Fuente Prieto and Enrique Castaño Perea

Additional information is available at the end of the chapter Additional information is available at the end of the chapter

http://dx.doi.org/10.5772/intechopen.74363

#### **Abstract**

[24] Fishman EK, Ney DR, Heath DG, Corl FM, Horton KM, Johnson PT. Volume rendering versus maximum intensity projection in CT angiography: What works best, when, and why. Radiographics: A Review Publication of the Radiological Society of North

[25] Johnson PT, Heath DG, Kuszyk BS, Fishman EK. CT angiography with volume rendering: Advantages and applications in splanchnic vascular imaging. Radiology.

[26] Douglas D, inventor; US Patent Office, assignee. US 8,384,771 Method and Apparatus for

[27] Douglas D, inventor; US Patent Office, assignee. US 9,349,183 Method and Apparatus for

[28] DeSantis C, Ma J, Bryan L, Jemal A. Breast cancer statistics, 2013. CA: a Cancer Journal

[29] Ferlay J, Shin HR, Bray F, Forman D, Mathers C, Parkin DM. Estimates of worldwide burden of cancer in 2008: GLOBOCAN 2008. International Journal of Cancer Journal

[30] Kopans D. Analyzing the Mammogram-Calcifications. Philadelphia, PA: Lippincott-

[31] Lee KS, Han BH, Chun YK, Kim HS, Kim EE. Correlation between mammographic manifestations and averaged histopathologic nuclear grade using prognosis-predict scoring system for the prognosis of ductal carcinoma in situ. Clinical Imaging. 1999;**23**(6):339-346

[32] Burnside ES, Ochsner JE, Fowler KJ, Fine JP, Salkowski LR, Rubin DL, et al. Use of microcalcification descriptors in BI-RADS 4th edition to stratify risk of malignancy.

[33] Boone JM, Lindfors KK. Breast CT: Potential for breast cancer screening and diagnosis.

[34] Mukhtar RA, Yau C, Rosen M, Tandon VJ, I-Spy T, Investigators A, et al. Clinically meaningful tumor reduction rates vary by prechemotherapy MRI phenotype and tumor subtype in the I-SPY 1 TRIAL (CALGB 150007/150012; ACRIN 6657). Annals of Surgical

America, Inc. 2006;**26**(3):905-922

24 State of the Art Virtual Reality and Augmented Reality Knowhow

for Clinicians. 2014;**64**(1):52-62

Radiology. 2007;**242**(2):388-395

Future Oncology. 2006;**2**(3):351-356

Oncology. 2013;**20**(12):3823-3830

Three Dimensional Viewing of Images. USA; 2013

Three Dimensional Viewing of Images. USA; 2016

International du Cancer. 2010;**127**(12):2893-2917

1996;**200**(2):564-568

Raven; 1998

This chapter aims to connect the digital resources of knowledge with the historical and cultural heritage in the context of smart cities. Specifically, combining the joint intervention in the Real Sitio of El Pardo and Aranjuez, as well as the Foundation Square of the University of Alcalá, both in Spain. Through traditional historical research and the innovative use of new technologies like augmented reality (AR), a historical, biological and cultural heritage is conserved, consisting of forests, gardens, agricultural spaces, urban centers and palatial residences. Cultural and artistic heritage is a resource of the first magnitude for the sustainable development of smart cities. It evolves with time and society; it is this that determines what goods are to be conserved and protected for posterity, according to the values attributed to them. Hence, the importance of achieving an awareness in society plays an active part in the conservation, enjoyment and dissemination of heritage. In this context, the augmented reality is presented as a powerful tool for contextualizing and disseminating the heritage, as well as to make the resources created more accessible, making an innovative use of the new technologies applied to the transfer of knowledge and the enhancement of a country's cultural and historical heritage.

**Keywords:** augmented reality, smart cities, heritage, royal sites, historical architecture

#### **1. Introduction**

This project, through the use of new technologies in the dissemination and transfer of knowledge to society, seeks to preserve and improve knowledge of these royal sites and the city of Alcala de Henares, of enormous environmental, cultural and educational value, constituted by forests, gardens, agricultural spaces, urban centers and palatial residences.

© 2016 The Author(s). Licensee InTech. This chapter is distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/3.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. © 2018 The Author(s). Licensee IntechOpen. This chapter is distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/3.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Thus Aranjuez or El Pardo is today the lung of Madrid, which, being a hunting ground of a royal site, escaped the relentless destruction of the natural environment that occurred as a result of the growth of the capital. It is a testimony to the landscape of yesteryear, a place where the original fauna and flora are preserved, where there is also an urban center, El Pardo, integrated into the natural space and a royal residence, El Pardo Palace, where an important architectural and artistic ensemble is preserved. Therefore, the study of the Royal Sites, in the specific case of this project of El Pardo, cannot do without considering the urban or natural environment in which these are inserted or to which they are opposed and this has to be reflected in the investigations and in the knowledge that society has of them.

Traditionally, those who have paid attention to these aspects, with great quantitative and qualitative richness, have been the art historians or architects, and the utility of them from tourism; what does not correspond, until the last years, is an equal interest on the part of other social sciences. The plurality of ways, elements and lexicons through which these enclaves, in all their components, or the spaces of power, economic development and landscape that develop in them are transformed into a multiform, polysemic sign system and modulated by different levels of perception.

Indeed, one of the great challenges we face is the impact of new technologies and their rapid evolution. According to data from the European Commission, "the global demand for ICTs is worth 2 trillion euros, but only a quarter of this amount is covered by European companies"1 , which is why it is a priority to stimulate an innovative digital society that contributes economic, social and educational benefits. No one can deny the multitude of applications they have for development in the health, architectural, social or educational field, including, for some years, the cultural. But besides being set as a factor of development, these resources have emerged as a particularly suitable means for the interpretation of cultural heritage and to transfer this knowledge to citizens and get them to identify with their past (**Figure 1**).

The architectural constructions, the urban interventions diffused or circumscribed in the territory, the more or less organic projects and the ideal city (more or less translated into practice) are an integral part of the exercise of power; they are codices constituting the authority and not ephemeral representations of it. The act of building and political action is always intimately united.

Indicators such as the ecological footprint, water footprint or ecological debt show that climate change, loss of biodiversity, lack of social justice, the persistence of disadvantaged social sectors, deforestation, the increase of inequalities in the distribution of wealth and so on are currently the environmental problems, that is, sociological and ecological, more importantly. We are not faced with a set of diverse problems; on the contrary, we are facing a systemic problem that obliges not only to try to alleviate its negative consequences but also to analyze

Connect Smart Cities and Heritage Through Augmented Reality

http://dx.doi.org/10.5772/intechopen.74363

27

Education alone is not enough to achieve a more sustainable future. However, without education and learning of sustainable development, we will not be able to achieve this goal. Education for sustainable development allows each human being to acquire the knowledge, skills, attitudes and values necessary to forge a sustainable future. Likewise, this sustainable education requires participatory methods of teaching and learning that motivate students and give them autonomy in order to change their behavior and facilitate the adoption of measures

and act directly on its causes.

**Figure 1.** Past heritage in the Royal Site of Aranjuez using A.R.

**Figure 2.** Present heritage in the Royal Site of Aranjuez.

in favor of sustainable development.

The study, enhancement and recovery of these places, shown in **Figures 2** and **3**, allow the recovery of a common past, the economic improvement of these places and their surroundings, their recovery and so on; also, they are an unbeatable space to advance the principles of an education for sustainable development, through which people acquire knowledge and skills in the field of sustainable development, which will increase their capacities and their self-confidence and increase their opportunities to opt for a healthy and productive life in harmony with nature, respectful of social values, equality between the sexes and cultural diversity, which will result in intercultural and diverse societies.

1 European Commission, "Europe 2020. A strategy for smart, sustainable and inclusive growth". Brussels: Publications Office of the European Union (2010). [Accessed on March 15, 2016]. Available at http://eur-lex.europa.eu/LexUriServ/ LexUriServ.do?uri=COM:2010:2020:FIN:ES:PDF, p. 12 and 15.

**Figure 1.** Past heritage in the Royal Site of Aranjuez using A.R.

Thus Aranjuez or El Pardo is today the lung of Madrid, which, being a hunting ground of a royal site, escaped the relentless destruction of the natural environment that occurred as a result of the growth of the capital. It is a testimony to the landscape of yesteryear, a place where the original fauna and flora are preserved, where there is also an urban center, El Pardo, integrated into the natural space and a royal residence, El Pardo Palace, where an important architectural and artistic ensemble is preserved. Therefore, the study of the Royal Sites, in the specific case of this project of El Pardo, cannot do without considering the urban or natural environment in which these are inserted or to which they are opposed and this has

Traditionally, those who have paid attention to these aspects, with great quantitative and qualitative richness, have been the art historians or architects, and the utility of them from tourism; what does not correspond, until the last years, is an equal interest on the part of other social sciences. The plurality of ways, elements and lexicons through which these enclaves, in all their components, or the spaces of power, economic development and landscape that develop in them are transformed into a multiform, polysemic sign system and modulated by

Indeed, one of the great challenges we face is the impact of new technologies and their rapid evolution. According to data from the European Commission, "the global demand for ICTs is worth 2 trillion euros, but only a quarter of this amount is covered by European companies"1

which is why it is a priority to stimulate an innovative digital society that contributes economic, social and educational benefits. No one can deny the multitude of applications they have for development in the health, architectural, social or educational field, including, for some years, the cultural. But besides being set as a factor of development, these resources have emerged as a particularly suitable means for the interpretation of cultural heritage and to transfer this knowledge to citizens and get them to identify with their past (**Figure 1**).

The architectural constructions, the urban interventions diffused or circumscribed in the territory, the more or less organic projects and the ideal city (more or less translated into practice) are an integral part of the exercise of power; they are codices constituting the authority and not ephemeral representations of it. The act of building and political action is always inti-

The study, enhancement and recovery of these places, shown in **Figures 2** and **3**, allow the recovery of a common past, the economic improvement of these places and their surroundings, their recovery and so on; also, they are an unbeatable space to advance the principles of an education for sustainable development, through which people acquire knowledge and skills in the field of sustainable development, which will increase their capacities and their self-confidence and increase their opportunities to opt for a healthy and productive life in harmony with nature, respectful of social values, equality between the sexes and cultural

European Commission, "Europe 2020. A strategy for smart, sustainable and inclusive growth". Brussels: Publications Office of the European Union (2010). [Accessed on March 15, 2016]. Available at http://eur-lex.europa.eu/LexUriServ/

diversity, which will result in intercultural and diverse societies.

LexUriServ.do?uri=COM:2010:2020:FIN:ES:PDF, p. 12 and 15.

,

to be reflected in the investigations and in the knowledge that society has of them.

different levels of perception.

26 State of the Art Virtual Reality and Augmented Reality Knowhow

mately united.

1

**Figure 2.** Present heritage in the Royal Site of Aranjuez.

Indicators such as the ecological footprint, water footprint or ecological debt show that climate change, loss of biodiversity, lack of social justice, the persistence of disadvantaged social sectors, deforestation, the increase of inequalities in the distribution of wealth and so on are currently the environmental problems, that is, sociological and ecological, more importantly. We are not faced with a set of diverse problems; on the contrary, we are facing a systemic problem that obliges not only to try to alleviate its negative consequences but also to analyze and act directly on its causes.

Education alone is not enough to achieve a more sustainable future. However, without education and learning of sustainable development, we will not be able to achieve this goal. Education for sustainable development allows each human being to acquire the knowledge, skills, attitudes and values necessary to forge a sustainable future. Likewise, this sustainable education requires participatory methods of teaching and learning that motivate students and give them autonomy in order to change their behavior and facilitate the adoption of measures in favor of sustainable development.

**d.** Development of accessible didactic and educational materials with new technologies for

Connect Smart Cities and Heritage Through Augmented Reality

http://dx.doi.org/10.5772/intechopen.74363

29

The theoretical framework justifies the innovative nature of the proposal. Starting from the concept of "Smart City," the technology of augmented reality is considered as the ideal instru-

The concept of "Smart Cities" [1–3] is associated with the principle of sustainable urban development that includes different areas such as infrastructures, technologies or democracy. The efficiency of smart cities has an important component of social development, and there is no better way to connect a city with its population than through the heritage that unites them. The new means of communication have generated a new field of dissemination and protection of cultural heritage [4–6] Since the declaration of UNESCO in 1972 on the preservation of material heritage and in the year 2003 on the intangible heritage, a series of processes, tools and especially networks have been generated that increase the accessibility and participation

use in Moocs or in the new realities of teaching and learning processes.

**2. Theoretical framework**

ment to develop a virtual heritage proposal.

of citizens in the management of cultural assets (**Figure 4**).

**2.1. The "smart cities" and their heritage**

**Figure 4.** Identifying interested places in a smart city.

**Figure 3.** Virtual heritage in the University of Alcalá.

Finally, indicate that in the elaboration of this project, we have taken into account the recommendations and suggestions emanating from the document Getting cultural heritage to work for Europe. Report of the Horizon 2020 Expert Group in Cultural Heritage (2015) of the Conclusions of the Council of Europe (Education, Youth, Culture and Sports) adopted on May 20, 2014.

#### **1.1. Goals**

The main objective of this chapter is to connect digital knowledge resources with historical and cultural heritage in the context of Smart Cities. Specifically, combining the joint intervention in the Real Sitio del Pardo and Aranjuez, as well as the Foundation Apple of the University of Alcalá.

This main aim is developed through five specific objectives:

**d.** Development of accessible didactic and educational materials with new technologies for use in Moocs or in the new realities of teaching and learning processes.

### **2. Theoretical framework**

Finally, indicate that in the elaboration of this project, we have taken into account the recommendations and suggestions emanating from the document Getting cultural heritage to work for Europe. Report of the Horizon 2020 Expert Group in Cultural Heritage (2015) of the Conclusions of the Council of Europe (Education, Youth, Culture and Sports) adopted on

The main objective of this chapter is to connect digital knowledge resources with historical and cultural heritage in the context of Smart Cities. Specifically, combining the joint intervention in the Real Sitio del Pardo and Aranjuez, as well as the Foundation Apple of the

**a.** Creation of an environment of augmented reality of free access, showing the historical evolution of these places and the different changes that have taken place in it, highlighting the historical and cultural heritage of it, as well as the relationship with the environment

**b.** 2D and 3D interpretation of the cartography and planimetry that analyzes the process of configuration, development and consolidation of these places to the present day, which

**c.** Create tourist and/or educational itineraries at various levels, accessible for the transmission of artistic, cultural and landscape heritage and value this heritage using new technologies.

This main aim is developed through five specific objectives:

allows their better transmission and contextualization.

May 20, 2014.

**Figure 3.** Virtual heritage in the University of Alcalá.

28 State of the Art Virtual Reality and Augmented Reality Knowhow

University of Alcalá.

and the landscape.

**1.1. Goals**

The theoretical framework justifies the innovative nature of the proposal. Starting from the concept of "Smart City," the technology of augmented reality is considered as the ideal instrument to develop a virtual heritage proposal.

#### **2.1. The "smart cities" and their heritage**

The concept of "Smart Cities" [1–3] is associated with the principle of sustainable urban development that includes different areas such as infrastructures, technologies or democracy. The efficiency of smart cities has an important component of social development, and there is no better way to connect a city with its population than through the heritage that unites them.

The new means of communication have generated a new field of dissemination and protection of cultural heritage [4–6] Since the declaration of UNESCO in 1972 on the preservation of material heritage and in the year 2003 on the intangible heritage, a series of processes, tools and especially networks have been generated that increase the accessibility and participation of citizens in the management of cultural assets (**Figure 4**).

**Figure 4.** Identifying interested places in a smart city.

The development of a "smart city" inevitably requires efficient management of heritage, which involves its integration into new information and communication technologies [7–9]. In this sense, we now have an exceptional resource when it comes to integrating heritage in the field of these new technologies: augmented reality.

#### **2.2. From virtual reality to augmented reality**

The scope of new technologies is truly extensive and, although all of them work together for the same purpose, the truth is that here we want to make special mention of the technology of augmented reality, which has been shaping up since the last quarter of the twentieth century as a tool with great potential to transmit our past, value and raise public awareness of our heritage ([10] 5).

**2.3. The applications of virtual heritage**

• Reconstruction of buildings in ruins or significantly altered.

**Table 1.** "Reality-virtuality continuum" based on Milgran and Kishino [12].

• Simulation of social or natural environments on historic sites.

material objects according to Gutiérrez and Hernández [17]:

• Speculate with different hypotheses about lost objects.

• Analyze an archeological object in its original environment.

• Test restoration techniques on synthetic models.

of our own reality ([17] 14).

systems shown in **Figures 5** and **6**.

• Recreation of lost or damaged archeological pieces.

The application of augmented reality to the field of historical heritage has generated a new concept that is virtual heritage. Following the discourse of several authors [13–16], we can synthesize three approaches when using augmented reality to disseminate historical heritage:

Connect Smart Cities and Heritage Through Augmented Reality

http://dx.doi.org/10.5772/intechopen.74363

31

To these informative functions, we must add other approaches that also consider useful in scientific contexts the generation of 3D images to simulate and investigate certain intangible

Therefore, it is necessary to propose new forms of content associated with this technology that go beyond the simple virtual reconstruction of historical buildings. Through augmented reality, it should be possible to enjoy new experiences that cannot occur in the real world, even in the one that has already disappeared, and that help to better understand what heritage means beyond its spatial analysis. In this sense, Gutiérrez and Hernández also defend that the incorporation of multimedia, multi-user exploration, telepresence and the possibility of showing worlds in ways that are not subject to the physical limitations of the world we live in will lead this technology to become no longer an emulation of what exists, but an expansion

This coincides perfectly with the objective of the augmented reality that according to Fernández Álvarez [14] is none other than overcoming the comprehension difficulties due to different levels of conceptual abstraction presented by the different traditional representation