**6.2.1 Acyclovir**

Acyclovir is one of the most commonly used antiviral drugs (Kavakl et al, 1998). Treatment of neonatal herpes simplex infections varicella zoster infections with CNS and pulmonary involvement, and herpes simplex encephalitis (Kavakl et al, 1998; Young & Mangum, 2010).

It can be used for the treatment of herpes simplex virus infections of newborn, varicella infections of children taking immunosuppressive and of immunodeficient children (Kavakl et al, 1998).

*Dose:* 20 mg/kg per dose Q8 hours IV by syringe pump over 1 hour. Prolong the dosing interval in prematures infant <34 weeks PMA, or in patients with significant renal impairment or hepatic failure. Treat localized herpes simplex infections for 14 days,

The Administration and Dose of Most Frequently Used Drugs in Pediatrics 315

*Uses:* Treatment of systemic fungal infections resistant to conventional amphotericin B therapy or in patients with renal or hepatic dysfunction (Young & Mangum, 2010).Dose: 5 to 7 mg/kg dose Q24 hours IV infusion by syringe pump over 2 hours (Young & Mangum,

It is administered by IV route every 2-6 hours in a 5% dextrose solution. It is started with 0.25 mg/kg/day and increased by 0.25 mg/kg/day in 1-2 days and can be advanced up to

*Side Effects:* Anemia, thrombocytopenia, hypokalemia, nausea/vomiting, and fever/chills

It is active against oropharynx, esophagus, urinary and systemic candidiasis. It is orally well- absorbed. It interacts with Warfarin, Phenytoin, Cyclosporine and Rifampin (Eroğlu,

It is used in skin and mucosal candidiasis infections (Apak, 1996; Kanmaz, 2010). It resembles to Amphotericin B as structure. It cannot be absorbed via gastrointestinal channels, skin and mucosae. It may have fungicidal or fungistatic effect (Kanmaz, 2010).

*Dose:* Topical: cream or ointment can be used on the affected area every 6 hours. Therapy

PO: from 100.000 U/ml solution 2 ml for term infants, 1 ml for preterm ones every 6 hours for each side of mouth. Therapy should be maintained even after symptoms disappear

*Administration Properties:* Oral suspension must be shaken well before use. This suspension

It is a drug whose analgesic and antipyretic effect is almost equal to aspirin. However it does not resemble to aspirin in terms of gastric mucosa destruction and bleeding. It has no antirheumatic effect being only active against mild and moderate fever. It is used in cases of nuisance, muscle and joint pain, neuralgia and fever. It is also advised in situations where

should be maintained even after symptoms disappear (Kanmaz, 2010).

*Side effects*: Cream or ointment related rash (Kanmaz, 2010).

includes <1% alcohol, saccharin, 50% sucrose (Kanmaz, 2010).

aspirin is contraindicated or not tolerated (Kavakl et al, 1998). *The dose, route of administration, duration and discharge of drug:*

Oral: loading dose 24 mg/kg, maintenance dose 12 mg/kg/dose Rectal: loading dose 30 mg/kg, maintenance dose 20 mg/kg

1-1.5 mg/kg/day (Apak, 1996; Dökmeci, 2001; Eroğlu, 2002).

2010).

2002).

(Young & Mangum, 2010).

**6.3.2 Fluconazole**

**6.3.3 Nystatin**

(Kanmaz, 2010).

**6.4 Other drugs**

**6.4.1 Acetaminophen**

Maintenance dose intervals Preterm ≤32 weeks: 12 hours

disseminated or CNS infections for 21 days (Young & Mangum, 2010).As this drug's intestinal absorption is not that good it should be taken by oral route at high doses. The drug reaches its peak after 1.5-2 hours following being administered by oral route. Almost 30-90% of it is discharged by urinary system (Kavakl et al, 1998).

*Adverse Effects:* The most common side effects are nausea, vomiting, diarrhea and headache. Other reported side effects include agitation, confusion, rash, anemia, and muscle pain. Hypersensitivity reactions, seizures, agitation, confusion, anemia, hepatitis, and muscle pain have also been reported (Apak, 1996; Kavakl et al, 1998; Young & Mangum, 2010).

*The things to be considered by nurse applying the drug:* In application by IV route a vial of 500 mg is diluted with 10 ml sterile water. The vial is shaken until the drug is fully dissolved. The desired dose is taken into syringe. In order to decrease concentration a single dose should be administered every 1 hour with infusion solution and if possible with infusion pump. Fast application may result in phlebitis and kidney destruction. The infusion area should be changed in order to avoid thrombophlebitis. The prepared solution should be used within 12 hours.

As the drug gets discharged from urinary system the child should be provided with adequate hydration pre-application and during application. The nephrotoxicity risk decreases after 2 hours following infusion application.

Nurse should attentively observe the observable side effects of the drug and should notify doctor if any is observed. If necessary, drug use should be ceased at once (Kavakl et al, 1998).


The implementation of semi-(markerless) mixed reality consists of four main components: *initialization*, *tracking and recognition*, *pose estimation*, and *MR scene* [8]. The architecture of the semi-markerless mixed reality system is shown in **Figure 12**. The implementation of this system required two additional platformdependent software packages. The first one was *NyARToolkit* (https://nyatla.jp/nya rtoolkit/wp/) with the core of the mixed reality construction and also an implementation of mathematical calculations used for determination of the pattern/ object position. The second one was the *Emgu.CV* software library (http://www. emgu.com), which provides the already mentioned SURF method implementation

The component *initialization* sets some parameters of the camera, pattern/

The component *tracking and recognition* recognizes the pattern/object from the image captured by the camera. This step can use the SURF method, e.g., from the software library Emgu.CV. This method describes the image by using descriptors. The description with the descriptors generated by this method is invariant to rotation and camera distance from the object being described. Interest points obtained by this method are shown in **Figure 14**. 3D scanning technology and followed recognition can be used also in this component. However, a detailed description of

The component *pose estimation* calculates the transformation matrix, for the establishment of the three-dimension coordinates on the pattern/object. For the calculation (based on [9]) itself, it is necessary to know the projection matrix, which is obtained by camera calibration. The most important part of the calculation is to obtain a transformation matrix that determines the location of the 3D virtual graphic object into 3D space. Placing the virtual model into the real world is needed to determine the parameters of the transformation matrix. In case we have a pattern (square/rectangle) as shown in **Figure 13**, determination of the transformation

Steps 2 and 3 are essential and the most demanding ones. The most commonly used methods for image recognition are based on *SIFT* and *SURF*

deformations, noise, and illumination changes.

for the detection of patterns/objects in the image.

this method goes beyond the scope of this chapter.

matrix parameters is as follows (1) and (2):

at the position.

*Mixed Reality: A Known Unknown*

algorithms.

creation.

object, and 3D object.

**155**

with the virtual object added.

*DOI: http://dx.doi.org/10.5772/intechopen.92827*

**Figure 12.** *The architecture of the semi-markerless system.*

in the image. In this case, however, the computational demands of the application significantly increase, especially when detecting recognized shapes. How an MR system of this type works is shown in **Figure 12**. In this type of system, a learning phase is required. The learning phase involves scanning the environment for examples of objects we need to recognize and acquiring templates of these objects, e.g., in the form of their photographs.

The third type is used primarily in smartphones (see previous subchapter "*Mobile MR implementation*"). It uses the phone camera, which scans the place where the user is looking. Using GPS, the system will detect where the user is and which points he has in his surroundings. The digital compass of the smartphone is used to determine the direction in which he is looking. The use of these features of the smartphone (camera, digital compass, GPS) allows creating MR applications.

The principle of creating an MR without exact markers is similar to creating an MR with exact markers (**Figures 8** and **9**). However, there is a significant difference in the method of recognizing the original and positioning it in the real scene image.

How markerless (semi-markerless) MR works can be, on the basis of **Figure 12**, described by the following steps:

Steps 2 and 3 are essential and the most demanding ones. The most commonly used methods for image recognition are based on *SIFT* and *SURF* algorithms.

The implementation of semi-(markerless) mixed reality consists of four main components: *initialization*, *tracking and recognition*, *pose estimation*, and *MR scene* [8]. The architecture of the semi-markerless mixed reality system is shown in **Figure 12**. The implementation of this system required two additional platformdependent software packages. The first one was *NyARToolkit* (https://nyatla.jp/nya rtoolkit/wp/) with the core of the mixed reality construction and also an implementation of mathematical calculations used for determination of the pattern/ object position. The second one was the *Emgu.CV* software library (http://www. emgu.com), which provides the already mentioned SURF method implementation for the detection of patterns/objects in the image.

The component *initialization* sets some parameters of the camera, pattern/ object, and 3D object.

The component *tracking and recognition* recognizes the pattern/object from the image captured by the camera. This step can use the SURF method, e.g., from the software library Emgu.CV. This method describes the image by using descriptors. The description with the descriptors generated by this method is invariant to rotation and camera distance from the object being described. Interest points obtained by this method are shown in **Figure 14**. 3D scanning technology and followed recognition can be used also in this component. However, a detailed description of this method goes beyond the scope of this chapter.

The component *pose estimation* calculates the transformation matrix, for the establishment of the three-dimension coordinates on the pattern/object. For the calculation (based on [9]) itself, it is necessary to know the projection matrix, which is obtained by camera calibration. The most important part of the calculation is to obtain a transformation matrix that determines the location of the 3D virtual graphic object into 3D space. Placing the virtual model into the real world is needed to determine the parameters of the transformation matrix. In case we have a pattern (square/rectangle) as shown in **Figure 13**, determination of the transformation matrix parameters is as follows (1) and (2):

in the image. In this case, however, the computational demands of the application significantly increase, especially when detecting recognized shapes. How an MR system of this type works is shown in **Figure 12**. In this type of system, a learning phase is required. The learning phase involves scanning the environment for examples of objects we need to recognize and acquiring templates of these objects, e.g., in

The third type is used primarily in smartphones (see previous subchapter "*Mobile MR implementation*"). It uses the phone camera, which scans the place where the user is looking. Using GPS, the system will detect where the user is and which points he has in his surroundings. The digital compass of the smartphone is used to determine the direction in which he is looking. The use of these features of the smartphone (camera, digital compass, GPS) allows creating MR applications. The principle of creating an MR without exact markers is similar to creating an MR with exact markers (**Figures 8** and **9**). However, there is a significant difference in the method of recognizing the original and positioning it in the real scene image. How markerless (semi-markerless) MR works can be, on the basis of **Figure 12**,

1.After initialization, the camera constantly captures the real scene and sends the

2.The software processes the captured image by frame and searches for the pattern(s)/object(s) in the image using the selected detection method.

3.The position and orientation of the object/s (pattern) are computed after it is

the form of their photographs.

*The architecture of the semi-markerless system.*

*Mixed Reality and Three-Dimensional Computer Graphics*

**Figure 12.**

**154**

described by the following steps:

video to the computing system for processing.

recognized (computer vision area).

*Mixed Reality and Three-Dimensional Computer Graphics*

$$
\begin{bmatrix} X\_c \\ Y\_c \\ Z\_c \\ \mathbf{1} \end{bmatrix} = \begin{bmatrix} V\_{11} & V\_{12} & V\_{13} & W\_x \\ V\_{21} & V\_{22} & V\_{23} & W\_y \\ V\_{31} & V\_{32} & V\_{33} & W\_x \\ \mathbf{0} & \mathbf{0} & \mathbf{0} & \mathbf{1} \end{bmatrix} \cdot \begin{bmatrix} X\_m \\ Y\_m \\ Z\_m \\ \mathbf{1} \end{bmatrix} \tag{1}
$$

$$
\begin{bmatrix} X\_c \\ Y\_c \\ Z\_c \\ \mathbf{1} \end{bmatrix} = \begin{bmatrix} V\_{3 \times 3} & W\_{3 \times 1} \\ \mathbf{0} \mathbf{0} & \mathbf{1} \\ \mathbf{1} \end{bmatrix} \cdot \begin{bmatrix} X\_m \\ Y\_m \\ Z\_m \\ \mathbf{1} \end{bmatrix} = T\_{\text{cm}} \cdot \begin{bmatrix} X\_m \\ Y\_m \\ Z\_m \\ \mathbf{1} \end{bmatrix} \tag{2}
$$

*T*cm (transformation from pattern coordinates to camera coordinates) is obtained by analyzing the input image. This transformation matrix consists of the rotation matrix (*V*3�3) and the translation matrix (*W*3�3). Two parallel patterns edges (margins) are reflected in the image. Coordinates of these edges correspond to the equations of lines (3):

$$\begin{aligned} l\_1: a\_1\mathbf{x} + b\_1\mathbf{y} + c\_1 &= \mathbf{0} \\ l\_2: a\_2\mathbf{x} + b\_2\mathbf{y} + c\_2 &= \mathbf{0} \end{aligned} \tag{3}$$

Determination of line parameters *l*1:

*Rectangle ABCD and interest points obtained by SURF method.*

*Mixed Reality: A Known Unknown*

*DOI: http://dx.doi.org/10.5772/intechopen.92827*

*u*

*Semi-markerless augmented system. The virtual model is displayed in the real world.*

! ¼ j j AB , A½ � *a*1, *a*2 , B½ � *b*1, *b*2 *:*

1.Finding of direction vector line

2.*u*1 ¼ *b*1 � *a*1; *u*2 ¼ *b*2 � *a*2.

3. *u*

**157**

**Figure 15.**

**Figure 14.**

! ¼ ð Þ *u*1, *u*2 .

The determination of the line parameters can be calculated in several ways. One of them is a calculation of parameters, if we know at least two points that lie on this line. Because pattern/object has a square or rectangle shape, we can obtain coordinates of its four vertices in the screen coordinate system. These coordinates are obtained using the SURF method after pattern/object recognition in the video image. Denote the pattern as a rectangle *ABCD* (**Figure 14**). Edges *AB* and *CD* are parallel. Corresponding equations for these edges are equations of lines *l*1 and *l*2 (3). Also, the edges *BC* and *DA* are parallel and their equations are *l*3 and *l*4.

**Figure 13.** *The relationship between pattern coordinates and the camera coordinates.*

*Xc Yc Zc* 1

*Mixed Reality and Three-Dimensional Computer Graphics*

5 ¼ *V*3�3 *W*3�1 000 1 � �

*V*11 *V*12 *V*13 *Wx V*21 *V*22 *V*23 *Wy V*31 *V*32 *V*33 *Wz* 0001

�

*T*cm (transformation from pattern coordinates to camera coordinates) is obtained by analyzing the input image. This transformation matrix consists of the rotation matrix (*V*3�3) and the translation matrix (*W*3�3). Two parallel patterns edges (margins) are reflected in the image. Coordinates of these edges correspond

> *l*1 : *a*1*x* þ *b*1*y* þ *c*1 ¼ 0 *l*2 : *a*2*x* þ *b*2*y* þ *c*2 ¼ 0

The determination of the line parameters can be calculated in several ways. One of them is a calculation of parameters, if we know at least two points that lie on this line. Because pattern/object has a square or rectangle shape, we can obtain coordinates of its four vertices in the screen coordinate system. These coordinates are obtained using the SURF method after pattern/object recognition in the video image. Denote the pattern as a rectangle *ABCD* (**Figure 14**). Edges *AB* and *CD* are parallel. Corresponding equations for these edges are equations of lines *l*1 and *l*2 (3).

Also, the edges *BC* and *DA* are parallel and their equations are *l*3 and *l*4.

*The relationship between pattern coordinates and the camera coordinates.*

*Xm Ym Zm* 1

*Xm Ym Zm* 1

*Xm Ym Zm* 1

(1)

(2)

(3)

*Xc Yc Zc* 1

to the equations of lines (3):

**Figure 13.**

**156**

**Figure 14.** *Rectangle ABCD and interest points obtained by SURF method.*

**Figure 15.** *Semi-markerless augmented system. The virtual model is displayed in the real world.*

Determination of line parameters *l*1:

1.Finding of direction vector line

$$
\overrightarrow{\boldsymbol{\mu}} = |\mathbf{A}\mathbf{B}|, \mathbf{A}[\boldsymbol{a}\_1, \mathbf{a}\_2], \mathbf{B}[\boldsymbol{b}\_1, \mathbf{b}\_2].
$$

$$2.u\_1 = b\_1 - a\_1; \\ u\_2 = b\_2 - a\_2.$$

$$\mathbf{3}. \overrightarrow{u} = (u\_1, u\_2).$$

4.Determination of the vector that is perpendicular to it: *n* ! ¼ ð Þ *u*2, �*u*1 .

5.Substitution of the values into the general equation of the line *ax* þ *by* þ c ¼ 0:

$$
u\_2 \ge -\nu\_1 \mathcal{y} + c = 0.$$

6. Substitution of the values *x* and *y* for the point that lies on a line such as coordinates of point *B* and computation of the parameter *c.*

In a similar way, the general equations of lines *l*2, *l*3, and *l*4 are obtained. The next procedure is to calculate the rotation and translation part of the transformation matrix.

The last component *MR scene* displays the virtual model in the real world. To view mixed reality, an appropriate rendering core can be used. The example result is shown in **Figure 15**.

#### **6. Mixed reality as user interface and gesture recognition**

Gestural interfaces offer various features to provide hand tracking for nonverbal interaction [10]. In the mixed reality, hands are the most effective tools that can be used for natural hand-object manipulation. Unlike touch interfaces, there is an opportunity to work with a variety of gestures and transform their semantics to specific commands. Gesture-based interfaces give users the freedom to interact without any limitation than using contact VR controllers.

Considering human-computer interaction (HCI), gesture recognition is performed by a digital system that senses users' handshapes and responds to them [11]. Handshapes are equal to visual patterns, which are recognizable in real time. Nowadays, there are several technologies that can provide full hand tracking.

The *Microsoft HoloLens* (MS HoloLens) introduces an all-in-one head-mounted display, which supports the complete head and hand tracking. In contrast to other MR systems, the MS HoloLens can provide two-handed gestures to ensure more intuitive interaction [12]. The gesture recognition utilizes an infrared depth camera which senses the reflection of the user's hands [13].

The similar technology as MS HoloLens is Microsoft Kinect (MS Kinect), which provides motion sensing of the human's rigid body and hands [14]. The gesture recognition and body tracking utilize the same principles based on the depth sensor including an infrared laser projector. In contrast to MS HoloLens, the MS Kinect can sense multiple persons concurrently, who can interact together [15].

**6.1 Static gestures**

**Figure 17.**

**159**

**Figure 16.**

*Clicking on hologram, static gesture utilization.*

*Mixed Reality: A Known Unknown*

*DOI: http://dx.doi.org/10.5772/intechopen.92827*

The recognition of static hand gestures (**Figure 18**) in mixed reality uses the identification of hand poses in a stream of image frames [18]. The static gesture

Gestural interfaces based on static gesture recognition include several stages to process gesture inputs. The first stage concerns hand tracking technology able to sense human hand in real time. This is usually supported by depth sensors or infrared cameras. In the second stage, the image sequence is performed. The hand detection obtains a hand posture from the image sequence. Using a variety of detection techniques [20] can filter different hand poses. In the third stage, the image segmentation preprocessing is provided. Then all of the detected hand regions are filled by contrasting colors and sharpened on boundaries. The final hand boundary representation is necessary for gesture recognition [21]. In the fifth stage,

the obtained gesture is compared with records from gesture datasets. If the

represents an event executed in the shortest time intervals [19].

*Continuous hologram manipulation by a hand, dynamic gesture utilization.*

In general, mixed reality focuses on gesture recognition to intent powerful and natural HCI. The utilization of IR sensors proves excellent results in development and research [16]. One of the specific systems is *VirtualTouch*. The system supports human-object interaction [17], where virtual objects are merged into physical ones. The user operates with a physically based object which is wrapped by its virtual entity.

In mixed reality, gestures can be utilized to perform a single event or continual activity. The majority of gesture recognition considers two categories that consider gestures duration:

4.Determination of the vector that is perpendicular to it: *n*

*Mixed Reality and Three-Dimensional Computer Graphics*

5.Substitution of the values into the general equation of the line *ax* þ *by* þ c ¼ 0:

*u*2*x* � *u*1*y* þ *c* ¼ 0*:*

In a similar way, the general equations of lines *l*2, *l*3, and *l*4 are obtained. The next procedure is to calculate the rotation and translation part of the transformation

The last component *MR scene* displays the virtual model in the real world. To view mixed reality, an appropriate rendering core can be used. The example result

Gestural interfaces offer various features to provide hand tracking for nonverbal interaction [10]. In the mixed reality, hands are the most effective tools that can be used for natural hand-object manipulation. Unlike touch interfaces, there is an opportunity to work with a variety of gestures and transform their semantics to specific commands. Gesture-based interfaces give users the freedom to interact

Considering human-computer interaction (HCI), gesture recognition is performed by a digital system that senses users' handshapes and responds to them [11]. Handshapes are equal to visual patterns, which are recognizable in real time. Nowadays, there are several technologies that can provide full hand tracking.

The *Microsoft HoloLens* (MS HoloLens) introduces an all-in-one head-mounted display, which supports the complete head and hand tracking. In contrast to other MR systems, the MS HoloLens can provide two-handed gestures to ensure more intuitive interaction [12]. The gesture recognition utilizes an infrared depth camera

The similar technology as MS HoloLens is Microsoft Kinect (MS Kinect), which provides motion sensing of the human's rigid body and hands [14]. The gesture recognition and body tracking utilize the same principles based on the depth sensor including an infrared laser projector. In contrast to MS HoloLens, the MS Kinect can

In general, mixed reality focuses on gesture recognition to intent powerful and natural HCI. The utilization of IR sensors proves excellent results in development and research [16]. One of the specific systems is *VirtualTouch*. The system supports human-object interaction [17], where virtual objects are merged into physical ones. The user operates with a physically based object which is wrapped by its virtual entity. In mixed reality, gestures can be utilized to perform a single event or continual activity. The majority of gesture recognition considers two categories that consider

• Static gestures (considered as events executed in the shortest time intervals,

• Dynamic gestures (considered as an activity with longer time duration, **Figure 17**)

sense multiple persons concurrently, who can interact together [15].

6. Substitution of the values *x* and *y* for the point that lies on a line such as

coordinates of point *B* and computation of the parameter *c.*

**6. Mixed reality as user interface and gesture recognition**

without any limitation than using contact VR controllers.

which senses the reflection of the user's hands [13].

matrix.

is shown in **Figure 15**.

gestures duration:

**Figure 16**)

**158**

! ¼ ð Þ *u*2, �*u*1 .

**Figure 16.** *Clicking on hologram, static gesture utilization.*

**Figure 17.** *Continuous hologram manipulation by a hand, dynamic gesture utilization.*

#### **6.1 Static gestures**

The recognition of static hand gestures (**Figure 18**) in mixed reality uses the identification of hand poses in a stream of image frames [18]. The static gesture represents an event executed in the shortest time intervals [19].

Gestural interfaces based on static gesture recognition include several stages to process gesture inputs. The first stage concerns hand tracking technology able to sense human hand in real time. This is usually supported by depth sensors or infrared cameras. In the second stage, the image sequence is performed. The hand detection obtains a hand posture from the image sequence. Using a variety of detection techniques [20] can filter different hand poses. In the third stage, the image segmentation preprocessing is provided. Then all of the detected hand regions are filled by contrasting colors and sharpened on boundaries. The final hand boundary representation is necessary for gesture recognition [21]. In the fifth stage, the obtained gesture is compared with records from gesture datasets. If the

**Figure 18.** *Detection of static hand gesture interaction in real time.*

classification of detected gesture is similar to its dataset record, then recognition is successful. In the final stage, the gesture is executed into the output command.

activity starts, the trigger gesture is obtained. The whole activity (dynamic gesture) can last over a long time, while the user interacts with virtual content. After the

The human speech represents the most common form of everyday communication [24]. In terms of human communication, extending mixed reality with speech recognition has an effective approach to provide multimodal interfaces. Through voice commands, the user can naturally communicate with the system [25]. This kind of interface frees the user from the touch or haptics interaction. Speech commands can be helpful in situations when users perform activities that engage their hands. The uniformity of speech recognition interfaces results in excellent usage on different platforms. Nowadays, mixed reality applications are utilizing speech interfaces in fields of education, research, medicine, and industry (**Figure 21**).

The whole process of speech recognition includes four stages which concern the

activity fulfills, the ending gesture terminates the action.

**7.

• Analysis of speech inputs

• Feature extraction

The advantage of static gesture recognition concerns the storage of gestural dataset records in simple readable structures such as images and text files. On the other hand, the preparation of new gestures requires the preparation of large dataset records.

#### **6.2 Dynamic gestures**

Continuous dynamic gestures (**Figure 19**) represent the activity sensed over a long time during which the movement of the human hand or limb is carried on [22]. The reason for utilizing continuous gestures in mixed reality refers to the interaction based on continuous manipulation of a virtual object. In contrast to static gestures, the preparation of dynamic gestures utilizes diverse principles in tracking [23]. While static gestures contain detection of hand posture, dynamic gestures equip motion tracking. The motion tracking performs real-time detection of the user's hands and limbs concurrently.

Most mixed reality systems support dynamic gestures to provide natural interaction. During the continual activity, the user can pick up virtual objects and manipulates them. This activity is triggered by static gestures that manage the beginning and terminating of dynamic gestures. As shown in **Figure 20**, before the

**Figure 19.**

*Performing continuous dynamic gesture recognition.*

**Figure 20.**

classification of detected gesture is similar to its dataset record, then recognition is successful. In the final stage, the gesture is executed into the output command.

**6.2 Dynamic gestures**

**Figure 18.**

**Figure 19.**

**160**

user's hands and limbs concurrently.

*Detection of static hand gesture interaction in real time.*

*Mixed Reality and Three-Dimensional Computer Graphics*

*Performing continuous dynamic gesture recognition.*

The advantage of static gesture recognition concerns the storage of gestural dataset records in simple readable structures such as images and text files. On the other hand, the preparation of new gestures requires the preparation of large dataset records.

Continuous dynamic gestures (**Figure 19**) represent the activity sensed over a long time during which the movement of the human hand or limb is carried on [22]. The reason for utilizing continuous gestures in mixed reality refers to the interaction based on continuous manipulation of a virtual object. In contrast to static gestures, the preparation of dynamic gestures utilizes diverse principles in tracking [23]. While static gestures contain detection of hand posture, dynamic gestures equip motion tracking. The motion tracking performs real-time detection of the

Most mixed reality systems support dynamic gestures to provide natural inter-

action. During the continual activity, the user can pick up virtual objects and manipulates them. This activity is triggered by static gestures that manage the beginning and terminating of dynamic gestures. As shown in **Figure 20**, before the *Performing continuous dynamic gesture recognition.*

activity starts, the trigger gesture is obtained. The whole activity (dynamic gesture) can last over a long time, while the user interacts with virtual content. After the activity fulfills, the ending gesture terminates the action.

### **7. Mixed reality speech recognition**

The human speech represents the most common form of everyday communication [24]. In terms of human communication, extending mixed reality with speech recognition has an effective approach to provide multimodal interfaces. Through voice commands, the user can naturally communicate with the system [25]. This kind of interface frees the user from the touch or haptics interaction. Speech commands can be helpful in situations when users perform activities that engage their hands. The uniformity of speech recognition interfaces results in excellent usage on different platforms. Nowadays, mixed reality applications are utilizing speech interfaces in fields of education, research, medicine, and industry (**Figure 21**).

The whole process of speech recognition includes four stages which concern the following [26]:

#### **7.1 Analysis of speech inputs**

In the first stage, the system obtains speech inputs. The speech input can include one or even several words. After the speech input is recorded, it is important to convert its representation into the analog signal.

#### **7.2 Feature extraction**

The speech input can contain surrounding noise that affects the purity of speaking voice. This step focuses on extracting two waveforms from the input, the whole speech, and environmental sounds. The speech input is purified using various techniques based on spoken context, pitch and variation, duration, and frequency of speaking. Most of the mixed reality systems utilize the artificial intelligence components that provide automated feature extraction in short time intervals.

spatial mapping and head tracking movement. The eye gaze is supported by the Pupil Labs system, which tracks eye movement to ensure eye to object interaction. Gesture input is supported by hand tracking, for which MS HoloLens (in MR usage)

The next of CMR systems called Vishnu [29] is concerning the mediation of virtual and real environments for remote guiding on a global scale. The system prepares separate visual outputs for MR and VR platforms. The whole collaboration focuses only on the objects that are captured by the MR side. The MR creates a realtime 3D scan and shares it with the VR side. The VR participant is able to manipulate a 3D scan and also can work together with the MR participant. The technological scope of the Vishnu includes hand tracking (OptiTrack and Kinect) and video-

Another system [30] related to remote guiding through collaborative mixed reality utilizes 3D point cloud data. Two collaborators, the local worker, and remote helper can operate in a commonly shared environment. Both are using the same head-mounted technology (Oculus Rift DK2). The local worker captures his

workspace through Oculus stereo cameras and distributes real-time visual output to the remote helper. The hands of the remote helper are captured by a depth sensor continuously. Their 3D point cloud overlays the visual output of the local worker

The next point cloud collaboration [31] focuses on remote Telepresence where MR and VR are used to engage physically presented (on-site users) and remotely shared users (remote users) in one shared space. The on-site users are physically available in the same physical environment, while the remote users are connected over the network and presented by their 3D point clouds. The system affords interaction between all participants through high-res point clouds that include realistic bodies. All point clouds are captured by depth-sensing through Kinect V1 and V2. The interaction is performed by a gestural interface equipped with free-

The LIRKIS G-CVE [32] introduces global collaborative virtual environments that are fully compatible with mixed reality usage (**Figure 22**). Unlike other collaborative mixed reality software and systems, the LIRKIS G-CVE is accessible through web browsers that ensure cross-platform support for a variety of VR, MR, and AR devices. All collaborative environments are distributed over the network. The system includes several interfaces, which enhance user interaction. There are gesture recognition, haptic interaction, and voice commands. The haptic interaction utilizes VR controllers equipped with three and six degrees of freedom. These immerse participants to interact more naturally and improve object manipulation. Gesture interface offers an intuitive object manipulation through MS HoloLens as grabbing, pulling, throwing, and stretching 3D object. These are currently limited to using only one hand than both. LIRIS project used MR and MS HoloLens for rehabilitation

and LeapMotion (in VR usage) are responsible.

*An example of virtual collaborative environment with multiple avatars.*

**Figure 22.**

*Mixed Reality: A Known Unknown*

*DOI: http://dx.doi.org/10.5772/intechopen.92827*

**163**

even if it necessary to guide him.

see mode through Oculus Rift stereo cameras for MR usage.

hand tracking through MS HoloLens and Leap Motion.

#### **7.3 Speech recognition**

This stage concerns the modeling techniques by using the acoustic and language model [27] to identify words in the speech input. The acoustic model works with audio records and process statistics of every spoken word to recognize syntax. The language model recognizes the semantics resulted from the speech input and detects the language in which the word is spoken. After performing speech identification, the final words are formed.

#### **7.4 Decoding output commands**

After finishing word recognition, the output command is performed. Each of the commands can perform various functions according to final use. Their functionality is fully unlimited. The speech recognition in mixed reality commonly prefers shorter speech inputs that are more effective than sentences. One-word commands are more specific and user-friendly.

#### **8. Collaborative mixed reality**

Mixed reality increases users' experiences utilizing gestural and speech recognition. This feature becomes useful for providing collaborative environments with multiuser interaction. Unlike other collaboration systems, collaborative mixed reality (CMR) offers a virtual and physical environment, where members can interact together. In fact, there are many systems designed for CMR purposes.

The CoVAR [28] introduces a remote collaborative system supporting VR and MR technologies. Participants can collaborate within the same local real-world environment or remotely. In the locally based collaboration, the MR user captures the surrounding physical space and shares its 3D model with other VR users. The remote collaboration utilizes the same principles but also a network to share a collaborative environment over long distances. The whole system primarily utilizes MS HoloLens for MR and HTC Vive for VR usage. In the case of interaction, the system inputs are formed to support head gaze, eye gaze, and hand gestures. The head gaze equips technologies included in VR and MR devices concerning the

• Speech recognition

• Decoding output command

convert its representation into the analog signal.

*Mixed Reality and Three-Dimensional Computer Graphics*

In the first stage, the system obtains speech inputs. The speech input can include one or even several words. After the speech input is recorded, it is important to

The speech input can contain surrounding noise that affects the purity of speaking voice. This step focuses on extracting two waveforms from the input, the whole speech, and environmental sounds. The speech input is purified using various techniques based on spoken context, pitch and variation, duration, and frequency of speaking. Most of the mixed reality systems utilize the artificial intelligence components that provide automated feature extraction in short time intervals.

This stage concerns the modeling techniques by using the acoustic and language model [27] to identify words in the speech input. The acoustic model works with audio records and process statistics of every spoken word to recognize syntax. The language model recognizes the semantics resulted from the speech input and detects the language in which the word is spoken. After performing speech identification,

After finishing word recognition, the output command is performed. Each of the commands can perform various functions according to final use. Their functionality is fully unlimited. The speech recognition in mixed reality commonly prefers shorter speech inputs that are more effective than sentences. One-word commands

Mixed reality increases users' experiences utilizing gestural and speech recognition. This feature becomes useful for providing collaborative environments with multiuser interaction. Unlike other collaboration systems, collaborative mixed reality (CMR) offers a virtual and physical environment, where members can interact

The CoVAR [28] introduces a remote collaborative system supporting VR and MR technologies. Participants can collaborate within the same local real-world environment or remotely. In the locally based collaboration, the MR user captures the surrounding physical space and shares its 3D model with other VR users. The remote collaboration utilizes the same principles but also a network to share a collaborative environment over long distances. The whole system primarily utilizes MS HoloLens for MR and HTC Vive for VR usage. In the case of interaction, the system inputs are formed to support head gaze, eye gaze, and hand gestures. The head gaze equips technologies included in VR and MR devices concerning the

together. In fact, there are many systems designed for CMR purposes.

**7.1 Analysis of speech inputs**

**7.2 Feature extraction**

**7.3 Speech recognition**

the final words are formed.

**7.4 Decoding output commands**

are more specific and user-friendly.

**8. Collaborative mixed reality**

**162**

**Figure 22.** *An example of virtual collaborative environment with multiple avatars.*

spatial mapping and head tracking movement. The eye gaze is supported by the Pupil Labs system, which tracks eye movement to ensure eye to object interaction. Gesture input is supported by hand tracking, for which MS HoloLens (in MR usage) and LeapMotion (in VR usage) are responsible.

The next of CMR systems called Vishnu [29] is concerning the mediation of virtual and real environments for remote guiding on a global scale. The system prepares separate visual outputs for MR and VR platforms. The whole collaboration focuses only on the objects that are captured by the MR side. The MR creates a realtime 3D scan and shares it with the VR side. The VR participant is able to manipulate a 3D scan and also can work together with the MR participant. The technological scope of the Vishnu includes hand tracking (OptiTrack and Kinect) and videosee mode through Oculus Rift stereo cameras for MR usage.

Another system [30] related to remote guiding through collaborative mixed reality utilizes 3D point cloud data. Two collaborators, the local worker, and remote helper can operate in a commonly shared environment. Both are using the same head-mounted technology (Oculus Rift DK2). The local worker captures his workspace through Oculus stereo cameras and distributes real-time visual output to the remote helper. The hands of the remote helper are captured by a depth sensor continuously. Their 3D point cloud overlays the visual output of the local worker even if it necessary to guide him.

The next point cloud collaboration [31] focuses on remote Telepresence where MR and VR are used to engage physically presented (on-site users) and remotely shared users (remote users) in one shared space. The on-site users are physically available in the same physical environment, while the remote users are connected over the network and presented by their 3D point clouds. The system affords interaction between all participants through high-res point clouds that include realistic bodies. All point clouds are captured by depth-sensing through Kinect V1 and V2. The interaction is performed by a gestural interface equipped with freehand tracking through MS HoloLens and Leap Motion.

The LIRKIS G-CVE [32] introduces global collaborative virtual environments that are fully compatible with mixed reality usage (**Figure 22**). Unlike other collaborative mixed reality software and systems, the LIRKIS G-CVE is accessible through web browsers that ensure cross-platform support for a variety of VR, MR, and AR devices. All collaborative environments are distributed over the network. The system includes several interfaces, which enhance user interaction. There are gesture recognition, haptic interaction, and voice commands. The haptic interaction utilizes VR controllers equipped with three and six degrees of freedom. These immerse participants to interact more naturally and improve object manipulation. Gesture interface offers an intuitive object manipulation through MS HoloLens as grabbing, pulling, throwing, and stretching 3D object. These are currently limited to using only one hand than both. LIRIS project used MR and MS HoloLens for rehabilitation

The user interface consists of a scene containing the model itself with appliances and other functional and nonfunctional object models. This simulation model and its smart appliances can be visualized as part of the mixed reality, using Microsoft HoloLens or other data helmets that can run a web browser. In this mode, the user can freely move and control appliances, such as turning on/off the television, lights, sunblind, etc. Users can also choose or modify one of the existing presets. Choosing presets, all appliances will set appropriate states based on the selected profile. For example, choosing "away from home" will turn off lights and TV and lock the doors. In such simulated environment, more users can collaborate because all requests and responses are done on the backend server and all users have actual data about simulated appliances states. This interface is also suitable for controlling

Mixed reality research is progressing quite well, although it requires significant financial resources. On the other hand, this technology offers a very immersive experience for its users. Mixed reality allows to bring gaming, education, training, and presentation of various kinds of designs up to an entirely new level. It represents a new form of visualization of real objects, extended with virtual information. Models can be created using 3D modeling tools, including CAD software, and inserted to a real scene. A mixed reality scene can be then created using one of the available augmented reality systems. The correct placement of virtual models inside a scene is ensured either by markers or by a combination of recognizable objects from the real environment and additional information from other sources, such as positioning systems. Together they create a solution that brings a new form of

This work has been supported by the APVV grant no. APVV-16-0202 "Enhancing cognition and motor rehabilitation using mixed reality" and by the KEGA grant No. 035TUKE-4/2019: "Virtual-reality technologies and handicapped people

© 2020 The Author(s). Licensee IntechOpen. This chapter is distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/ by/3.0), which permits unrestricted use, distribution, and reproduction in any medium,

Branislav Sobota\*, Štefan Korečko, Marián Hudák and Martin Sivý

\*Address all correspondence to<EMAIL_ADDRESS>
households with handicapped people.

*Mixed Reality: A Known Unknown*

*DOI: http://dx.doi.org/10.5772/intechopen.92827*

computing resource utilization.

Technical University of Košice, Slovakia

provided the original work is properly cited.

**Acknowledgements**

education."

**Author details**

**165**

**10. Conclusion**

**Figure 23.** *An example of a patient's view in rehabilitation process using MR.*

of patients after stroke, and training of movement of their hand is also very important. A patient uses MS HoloLens, and he can see real hand and also phantom virtual hand with appropriate movement. Then he can try to perform the suggested movements. An example of a patient's view is illustrated in the **Figure 23**.

The voice commands perform multimodal user inputs when utilizing other interaction techniques. Interacting through voice is limited to simple commands that are responsible for simple operations (enable and disable functions, hiding and showing 3D objects).

#### **9. Mixed reality and SMART environment simulation**

Building a SMART household without testing and implementing it into real operation is complicated and can be very costly. Therefore, simulators are created. The study [33] identified areas in which smart intelligence simulation research is being conducted. The study [33] shows an overview of some simulation tools analyzed for the SMART household. **Figure 24** shows the view from a created simulator of a SMART environment using freeware technologies such as Blender, Python, and JavaScript. The program serves to visualize smart home simulation with few basic appliances, which are used to present the way the simulator works. These appliances can be controlled using the control panel or with a direct approach using clicks and context menu. The control panel sets the profiles for appliances'statuses. It is possible to move freely in the household and interact with the appliances.

**Figure 24.** *Simulation model of SMART household (left) and real SMART household user interface control (right).*

#### *Mixed Reality: A Known Unknown DOI: http://dx.doi.org/10.5772/intechopen.92827*

The user interface consists of a scene containing the model itself with appliances and other functional and nonfunctional object models. This simulation model and its smart appliances can be visualized as part of the mixed reality, using Microsoft HoloLens or other data helmets that can run a web browser. In this mode, the user can freely move and control appliances, such as turning on/off the television, lights, sunblind, etc. Users can also choose or modify one of the existing presets. Choosing presets, all appliances will set appropriate states based on the selected profile. For example, choosing "away from home" will turn off lights and TV and lock the doors. In such simulated environment, more users can collaborate because all requests and responses are done on the backend server and all users have actual data about simulated appliances states. This interface is also suitable for controlling households with handicapped people.

#### **10. Conclusion**

of patients after stroke, and training of movement of their hand is also very important. A patient uses MS HoloLens, and he can see real hand and also phantom virtual hand with appropriate movement. Then he can try to perform the suggested move-

The voice commands perform multimodal user inputs when utilizing other interaction techniques. Interacting through voice is limited to simple commands that are responsible for simple operations (enable and disable functions, hiding and

Building a SMART household without testing and implementing it into real operation is complicated and can be very costly. Therefore, simulators are created. The study [33] identified areas in which smart intelligence simulation research is being conducted. The study [33] shows an overview of some simulation tools analyzed for the SMART household. **Figure 24** shows the view from a created simulator of a SMART environment using freeware technologies such as Blender, Python, and JavaScript. The program serves to visualize smart home simulation with few basic appliances, which are used to present the way the simulator works. These appliances can be controlled using the control panel or with a direct approach using clicks and context menu. The control panel sets the profiles for appliances'statuses. It is possible

ments. An example of a patient's view is illustrated in the **Figure 23**.

*An example of a patient's view in rehabilitation process using MR.*

*Mixed Reality and Three-Dimensional Computer Graphics*

**9. Mixed reality and SMART environment simulation**

to move freely in the household and interact with the appliances.

*Simulation model of SMART household (left) and real SMART household user interface control (right).*

showing 3D objects).

**Figure 23.**

**Figure 24.**

**164**

Mixed reality research is progressing quite well, although it requires significant financial resources. On the other hand, this technology offers a very immersive experience for its users. Mixed reality allows to bring gaming, education, training, and presentation of various kinds of designs up to an entirely new level. It represents a new form of visualization of real objects, extended with virtual information. Models can be created using 3D modeling tools, including CAD software, and inserted to a real scene. A mixed reality scene can be then created using one of the available augmented reality systems. The correct placement of virtual models inside a scene is ensured either by markers or by a combination of recognizable objects from the real environment and additional information from other sources, such as positioning systems. Together they create a solution that brings a new form of computing resource utilization.

#### **Acknowledgements**

This work has been supported by the APVV grant no. APVV-16-0202 "Enhancing cognition and motor rehabilitation using mixed reality" and by the KEGA grant No. 035TUKE-4/2019: "Virtual-reality technologies and handicapped people education."

#### **Author details**

Branislav Sobota\*, Štefan Korečko, Marián Hudák and Martin Sivý Technical University of Košice, Slovakia

\*Address all correspondence to<EMAIL_ADDRESS>
© 2020 The Author(s). Licensee IntechOpen. This chapter is distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/ by/3.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

#### **References**

[1] Azuma R. A survey of augmented reality. Presence Teleoperators and Virtual Environments. 1997;**6**(4): 355-385

[2] Rosenberg LB. The Use of Virtual Fixtures As Perceptual Overlays to Enhance Operator Performance in Remote Environments; Technical Report AL-TR-0089. OH: USAF Armstrong Laboratory, Wright-Patterson AFB; 1992

[3] Milgram P, Kishino AF. Taxonomy of mixed reality visual displays. IEICE Transactions on Information and Systems. 2013:1321-1329

[4] Mann S. Campus Canada, ISSN 0823-4531; Feb–Mar 1985, p. 55; Apr– May 1986, pp. 58–59; Sep–Oct 1986, p. 72

[5] Sobota B, Korečko Š, Hrozek F. Mobile Mixed Reality; ICETA 2013: 11th IEEE International Conference on Emerging eLearning Technologies and Applications: Proceedings; 24–25 October 2013; Stary Smokovec, Slovakia. Danvers: IEEE; 2013. pp. 355-358. ISBN 978-1-4799-2161-4

[6] Lowe DG. Distinctive Image Features from Scale-Invariant Keypoints. Computer Science Department, University of British Columbia; 2004

[7] Tuytelaars BT, Van Gool L. Speed up robust features. European Conference on Computer Vision. 2006;**1**:404-417

[8] Varga M. Markerless augmented reality using SURF method; SCYR 2012. In: Proceedings from Conference: 12th Scientific Conference of Young Researchers; 15 May 2012, Herľany, Slovakia. Košice: TU; 2012. pp. 173-176. ISBN 978-80-553-0943-9

[9] Kato H, Bilinghurst M. Marker Tracking and HMD Calibration for a Video based Augmented Reality Conferencing System, Iwar. IEEE Computer Society; 1999. p. 85

[10] Chang YS, Nuernberger B, Luan B, Höllerer T. Evaluating gesture-based augmented reality annotation. In: 2017 IEEE Symposium on 3D User Interfaces (3DUI). IEEE; 2017. pp. 182-185

SIGGRAPH Symposium on Interactive 3D Graphics and Games. 2013. p. 184

*DOI: http://dx.doi.org/10.5772/intechopen.92827*

*Mixed Reality: A Known Unknown*

[25] Oberhauser R, Lecon C. Towards virtual reality immersion in software structures: Exploring augmented virtuality and speech recognition interfaces. 2018;**11**(1–2):34-44

[26] Boruah S, Basishtha S. A study on hmm based speech recognition system. In: 2013 IEEE International Conference on Computational Intelligence and Computing Research. IEEE; 2013.

[27] Plouffe G, Cretu AM. Static and dynamic hand gesture recognition in depth data using dynamic time warping. IEEE Transactions on Instrumentation and Measurement. 2015;**65**(2):305-316

[28] Piumsomboon T, Dey A, Ens B, Lee G, Billinghurst M. The effects of sharing awareness cues in collaborative mixed reality. Frontiers in Robotics and

[29] Le Chénéchal M, Duval T, Gouranton V, Royan J, Arnaldi BV. Virtual immersive support for HelpiNg users an interaction paradigm for collaborative remote guiding in mixed reality. In: 2016 IEEE Third VR International Workshop on

Collaborative Virtual Environments (3DCVE). IEEE; 2016. pp. 9-12

[31] Kolkmeier J, Harmsen E,

Giesselink S, Reidsma D, Theune M, Heylen D. With a little help from a holographic friend: The OpenIMPRESS mixed reality telepresence toolkit for remote collaboration systems. In: Proceedings of the 24th ACM

Symposium on Virtual Reality Software

and Technology. 2018. pp. 1-11

[32] Hudák M, Sivý M. Web-based collaborative virtual environments to

[30] Gao L, Bai H, Lee G, Billinghurst M. An oriented point-cloud view for MR remote collaboration. In: SIGGRAPH ASIA 2016 Mobile Graphics and Interactive Applications. 2016. pp. 1-4

pp. 1-5

AI. 2019;**6**(5):02

[17] Xiao Y, Zhang Z, Beck A, Yuan J, Thalmann D. Human–robot interaction by understanding upper body gestures. Presence Teleoperators and Virtual Environments. 2014;**23**(2):133-154

[18] Sharma RP, Verma GK. Human computer interaction using hand gesture. Procedia Computer Science.

[19] Cheng K, Ye N, Malekian R,

Wang R. In-air gesture interaction: Real time hand posture recognition using passive RFID tags. IEEE Access. 2019;**7**:

[20] Ibraheem NA, Khan RZ, Hasan MM. Comparative study of skin color based segmentation techniques. International Journal of Applied Information Systems.

[21] Song J, Sörös G, Pece F, Fanello SR, Izadi S, Keskin C, et al. In-air gestures around unmodified mobile devices. In: Proceedings of the 27th Annual ACM Symposium on User Interface Software and Technology. 2014. pp. 319-329

[22] Santos CCD, Samatelo JLA, Vassallo RF. Dynamic gesture recognition by using CNNs and star RGB: A temporal information condensation, 2019. arXiv preprint

[23] Wang X, Xia M, Cai H, Gao Y, Cattani C. Hidden-markov-modelsbased dynamic hand gesture

recognition. Mathematical Problems in Engineering. 2012;**2012**:11. Article ID 986134. DOI: 10.1155/2012/986134

[24] Ranjan R, Dubey RK. Isolated word recognition using HMM for Maithili dialect. In: 2016 International Conference on Signal Processing and Communication (ICSC). IEEE; 2016.

2015;**54**:721-727

94460-94472

2013;**5**(10):24-38

arXiv:1904.08505

pp. 323-327

**167**

[11] Song Y, Zhou N, Sun Q, Gai W, Liu J, Bian Y, et al. Mixed reality storytelling environments based on tangible user interface: Take origami as an example. In: 2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR). IEEE; 2019. pp. 1167-1168

[12] Chaconas N, Höllerer T. An evaluation of bimanual gestures on the Microsoft HoloLens. In: 2018 IEEE Conference on Virtual Reality and 3D User Interfaces (VR). IEEE; 2018. pp. 1-8

[13] Xiao R, Schwarz J, Throm N, Wilson AD, Benko H. MRTouch: Adding touch input to head-mounted mixed reality. IEEE Transactions on Visualization and Computer Graphics. 2018;**24**(4):1653-1660

[14] Kulshreshth A, Zorn C, LaViola JJ. Poster: Real-time markerless Kinect based finger tracking and hand gesture recognition for HCI. In: 2013 IEEE Symposium on 3D User Interfaces (3DUI). IEEE; 2013. pp. 187-188

[15] Gritti AP, Tarabini O, Guzzi J, Di Caro GA, Caglioti V, Gambardella LM, et al. Kinect-based people detection and tracking from small-footprint ground robots. In: 2014 IEEE/RSJ International Conference on Intelligent Robots and Systems. IEEE; 2014. pp. 4096-4103

[16] Melax S, Keselman L, Orsten S. Dynamics based 3D skeletal hand tracking. In: Proceedings of the ACM *Mixed Reality: A Known Unknown DOI: http://dx.doi.org/10.5772/intechopen.92827*

SIGGRAPH Symposium on Interactive 3D Graphics and Games. 2013. p. 184

**References**

355-385

p. 72

Patterson AFB; 1992

Systems. 2013:1321-1329

[1] Azuma R. A survey of augmented reality. Presence Teleoperators and Virtual Environments. 1997;**6**(4):

*Mixed Reality and Three-Dimensional Computer Graphics*

Video based Augmented Reality Conferencing System, Iwar. IEEE Computer Society; 1999. p. 85

(3DUI). IEEE; 2017. pp. 182-185

[12] Chaconas N, Höllerer T. An evaluation of bimanual gestures on the Microsoft HoloLens. In: 2018 IEEE Conference on Virtual Reality and 3D User Interfaces (VR). IEEE; 2018.

[13] Xiao R, Schwarz J, Throm N, Wilson AD, Benko H. MRTouch: Adding touch input to head-mounted mixed reality. IEEE Transactions on Visualization and Computer Graphics.

[14] Kulshreshth A, Zorn C, LaViola JJ. Poster: Real-time markerless Kinect based finger tracking and hand gesture recognition for HCI. In: 2013 IEEE Symposium on 3D User Interfaces (3DUI). IEEE; 2013. pp. 187-188

[15] Gritti AP, Tarabini O, Guzzi J, Di Caro GA, Caglioti V, Gambardella LM, et al. Kinect-based people detection and tracking from small-footprint ground robots. In: 2014 IEEE/RSJ International Conference on Intelligent Robots and Systems. IEEE; 2014. pp. 4096-4103

[16] Melax S, Keselman L, Orsten S. Dynamics based 3D skeletal hand tracking. In: Proceedings of the ACM

2018;**24**(4):1653-1660

pp. 1-8

[11] Song Y, Zhou N, Sun Q, Gai W, Liu J, Bian Y, et al. Mixed reality storytelling environments based on tangible user interface: Take origami as an example. In: 2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR). IEEE; 2019. pp. 1167-1168

[10] Chang YS, Nuernberger B, Luan B, Höllerer T. Evaluating gesture-based augmented reality annotation. In: 2017 IEEE Symposium on 3D User Interfaces

[2] Rosenberg LB. The Use of Virtual Fixtures As Perceptual Overlays to Enhance Operator Performance in Remote Environments; Technical Report AL-TR-0089. OH: USAF Armstrong Laboratory, Wright-

[3] Milgram P, Kishino AF. Taxonomy of mixed reality visual displays. IEICE Transactions on Information and

[4] Mann S. Campus Canada, ISSN 0823-4531; Feb–Mar 1985, p. 55; Apr– May 1986, pp. 58–59; Sep–Oct 1986,

[5] Sobota B, Korečko Š, Hrozek F. Mobile Mixed Reality; ICETA 2013: 11th IEEE International Conference on Emerging eLearning Technologies and Applications: Proceedings; 24–25 October 2013; Stary Smokovec, Slovakia. Danvers: IEEE; 2013. pp. 355-358. ISBN 978-1-4799-2161-4

[6] Lowe DG. Distinctive Image Features

[7] Tuytelaars BT, Van Gool L. Speed up robust features. European Conference on Computer Vision. 2006;**1**:404-417

[8] Varga M. Markerless augmented reality using SURF method; SCYR 2012. In: Proceedings from Conference: 12th Scientific Conference of Young Researchers; 15 May 2012, Herľany, Slovakia. Košice: TU; 2012. pp. 173-176.

[9] Kato H, Bilinghurst M. Marker Tracking and HMD Calibration for a

ISBN 978-80-553-0943-9

**166**

from Scale-Invariant Keypoints. Computer Science Department, University of British Columbia; 2004 [17] Xiao Y, Zhang Z, Beck A, Yuan J, Thalmann D. Human–robot interaction by understanding upper body gestures. Presence Teleoperators and Virtual Environments. 2014;**23**(2):133-154

[18] Sharma RP, Verma GK. Human computer interaction using hand gesture. Procedia Computer Science. 2015;**54**:721-727

[19] Cheng K, Ye N, Malekian R, Wang R. In-air gesture interaction: Real time hand posture recognition using passive RFID tags. IEEE Access. 2019;**7**: 94460-94472

[20] Ibraheem NA, Khan RZ, Hasan MM. Comparative study of skin color based segmentation techniques. International Journal of Applied Information Systems. 2013;**5**(10):24-38

[21] Song J, Sörös G, Pece F, Fanello SR, Izadi S, Keskin C, et al. In-air gestures around unmodified mobile devices. In: Proceedings of the 27th Annual ACM Symposium on User Interface Software and Technology. 2014. pp. 319-329

[22] Santos CCD, Samatelo JLA, Vassallo RF. Dynamic gesture recognition by using CNNs and star RGB: A temporal information condensation, 2019. arXiv preprint arXiv:1904.08505

[23] Wang X, Xia M, Cai H, Gao Y, Cattani C. Hidden-markov-modelsbased dynamic hand gesture recognition. Mathematical Problems in Engineering. 2012;**2012**:11. Article ID 986134. DOI: 10.1155/2012/986134

[24] Ranjan R, Dubey RK. Isolated word recognition using HMM for Maithili dialect. In: 2016 International Conference on Signal Processing and Communication (ICSC). IEEE; 2016. pp. 323-327

[25] Oberhauser R, Lecon C. Towards virtual reality immersion in software structures: Exploring augmented virtuality and speech recognition interfaces. 2018;**11**(1–2):34-44

[26] Boruah S, Basishtha S. A study on hmm based speech recognition system. In: 2013 IEEE International Conference on Computational Intelligence and Computing Research. IEEE; 2013. pp. 1-5

[27] Plouffe G, Cretu AM. Static and dynamic hand gesture recognition in depth data using dynamic time warping. IEEE Transactions on Instrumentation and Measurement. 2015;**65**(2):305-316

[28] Piumsomboon T, Dey A, Ens B, Lee G, Billinghurst M. The effects of sharing awareness cues in collaborative mixed reality. Frontiers in Robotics and AI. 2019;**6**(5):02

[29] Le Chénéchal M, Duval T, Gouranton V, Royan J, Arnaldi BV. Virtual immersive support for HelpiNg users an interaction paradigm for collaborative remote guiding in mixed reality. In: 2016 IEEE Third VR International Workshop on Collaborative Virtual Environments (3DCVE). IEEE; 2016. pp. 9-12

[30] Gao L, Bai H, Lee G, Billinghurst M. An oriented point-cloud view for MR remote collaboration. In: SIGGRAPH ASIA 2016 Mobile Graphics and Interactive Applications. 2016. pp. 1-4

[31] Kolkmeier J, Harmsen E, Giesselink S, Reidsma D, Theune M, Heylen D. With a little help from a holographic friend: The OpenIMPRESS mixed reality telepresence toolkit for remote collaboration systems. In: Proceedings of the 24th ACM Symposium on Virtual Reality Software and Technology. 2018. pp. 1-11

[32] Hudák M, Sivý M. Web-based collaborative virtual environments to support cross-platform access. In: Poster 2019 International Student Scientific Conference, Prague. 2019. pp. 178-182

[33] Synnott J, Nugent C, Jeffers P. Simulation of smart home activity datasets. Sensors. 2015;**15**:14162-14179

support cross-platform access. In: Poster 2019 International Student Scientific Conference, Prague. 2019. pp. 178-182

*Mixed Reality and Three-Dimensional Computer Graphics*

[33] Synnott J, Nugent C, Jeffers P. Simulation of smart home activity datasets. Sensors. 2015;**15**:14162-14179

**168**

## *Edited by Branislav Sobota and Dragan Cvetković*

Mixed reality is an area of computer research that deals with the combination of realworld and computer-generated data, where computer-generated objects are visually mixed into the real environment and vice versa in real time. It is the newest virtual reality technology. It usually uses 3D computer graphics technologies for visual presentation of the virtual world. The mixed reality can be created using the following technologies: augmented reality and augmented virtuality. Mixed and virtual reality, their applications, 3D computer graphics and related technologies in their actual stage are the content of this book. 3D-modeling in virtual reality, a stereoscopy, and 3D solids reconstruction are presented in the first part. The second part contains examples of the applications of these technologies, in industrial, medical, and educational areas.

Published in London, UK © 2020 IntechOpen © Andrey\_A / iStock

Mixed Reality and Three-Dimensional Computer Graphics

Mixed Reality

and Three-Dimensional

Computer Graphics

*Edited by Branislav Sobota*

*and Dragan Cvetković*


**3.3 Production of arbuscular mycorrhizal fungi bio-fertilizers in arid lands**

Both conventional method and in vitro methods are practiced in arid areas to produce AMF bio-fertilizers. Several researches was focused on the increasing of plant production in arid land using AMF inoculum (**Table 2**), conventional method with AMF mixture was the most important technic of production adopted. Nevertheless, in vitro technic was also practiced such as by the energy and resource institute of India (TERI) [34]. They based on the faculty of Glomus genus to provide the possibility of using colonized roots as inoculum material with up to 80% of root colonization attained at 4 and 12 weeks [34, 39]. Despite, arid lands are often localized in underdeveloped country with low economical budget who cannot afford to allot enormous amounts in order to produce bio-fertilizers, so the conventional method remains the most appropriate technique under these conditions. In addition ecological conditions of arid lands give them specific characteristics that are not accommodating with all AMF strain. For that production of native AMF bio-fertilizers adapted to local conditions and to specific abiotic stress is essential [13]. Labidi et al. [39] developed a native AMF bio-fertilizer adapted to calcareous arid Tunisians soils. Abdelsalam et al. [38] produced AMF inoculum of desert saoudian areas using *Sorghum halepense* as trapping plant. Bencherif et al. [13] developed a specific AMF bio-fertilizer for arid saline soils. It is noted that in

**Table 2.**

*AMF bio-fertilizers produced in arid lands.*

all case, efficiency of AMF bio-fertilizers is related to the better combination AMF genotype/host plant genotype/adaptation to specific abiotic stress [13, 27].

### *3.3.1 On-farm method*

To have AMF bio-fertilizer produced at a large scale with low cost, studies has been developed to test the multiplication of AMF spores under field conditions, called the "on-farm method". These studies explore AMF colonization with strains isolates that are environmentally adapted to native environmental conditions, which potentially representing a low-cost alternative for farmers [26, 37]. This technic is based on sowing plants seeds or seedlings in intact soil cores or mixed soil samples for sufficient time to allow development of AMF symbiosis and then roots were sampled, processed and assessed to measure mycorrhiza formation [26]. Indeed, [41] showed that multiplication on-farm with *Rhizophagus clarus* and *Claroideoglomus etunicatum* grown in agro-industrial residues, such as sugarcane bagasse, is a good strategy for the multiplication of AMF, leading to excellent inoculum potential and large numbers of spores. As well as, the on-farm technic allows farmers and nursery workers to access inoculums with the most effective AMF strains for their culture and their soil and climate conditions; furthermore, they can produce seedlings already inoculated with adapted AMF strains, which enhancing their establishment of s in the field conditions. In addition Moreira et al. [26] produced AMF inoculum with *Rhizophagus clarus*, *Claroideoglomus etunicatum* species, and native AMF from pineapple and coffee plantations, using spores multiplied by the on-farm method to enhance the growth of pineapple and coffee plantlets. These authors concluded that AMF inoculum favorite growth of the commercial tested crops with a high viability of AMF spores. This method is recommended for arid land due to their specific AMF strain and low coast, it could be applied and generalized in order to developing agricultural practices in these areas. Furthermore, because fungi carry different amounts of nutrients for plants, they may affect the growth of plants differently [37], it is preferable to use mixed AMF strains adapted to native conditions which could maximize the absorption of limiting nutrients [26]. This phenomenon could provide more benefits compared to colonization with exotic AMF strains or with single AMF species. Moreover, the mixed inoculation of AMF might have the characteristic of complementarity, exploiting the best of each species that colonizes the plant [26–37, 41].

### **3.4 Formulation of AMF bio-fertilizers**

Formulation technologies largely take care of possible adverse environmental effects and factors that may render the inoculum ineffective [34]. In fact, biofertilizers are generally prepared as liquid suspension or more generally solid support containing different types of propagules: spores, fungal mycelium, mycorrhizal root fragments [7]. Generally AMF bio-fertilizers are presented with multi AMF species, which are closer to natural conditions; because in natural ecosystems it is rare to encounter only one species of mycorrhizal fungus. The presence of several fungal species allows the inoculum to respond to a greater diversity of culture conditions [13, 39]. Bio-fertilizer support can be composed of peat, vermiculite, lignite powder, clay, talc, rice bran, granulated rock phosphate, charcoal, soil, straw compost of rice or wheat otherwise a mixture of these materials. In current practice, for better framework life of the bio-fertilizer formulation, the support is selected on the basis of the viability of the microorganisms mixed with them. Likewise, the pre-sterilization of the support and its nutrient enrichment is the other strategy to improve framework, allowing AMF to be maintained

**85**

*Native Arbuscular Mycorrhizal Fungi and Agro-Industries in Arid Lands…*

in a non-competitive microenvironment. Sucrose, maltose, trehalose, molasses, glucose, and glycerol are additional nutrients and cell-protecting agents commonly used along with a support to ensure maximum cell viability and extended shelf life [4, 39]. After production, AMF bio-fertilizer may be in the form of granules,

• The micro granules, between 1 and 4 mm, are easily mixed with support of mycorrhizal plants or brought into the planting surface as close as possible to

• Very fine powders (particles <250 μm) make it possible to prepare a suspension which can be sprayed on growing media or injected into the soil at the base of plants already installed. This type of inoculum can also be used as a seed

• The tablets allow easy dosing of the inoculum to be placed in the plantation area. The inoculum provided is localized in one place and not distributed

• Associated with a praline, the inoculum is particularly suitable for plants with bare roots. In a single operation, the plant is inoculated and its roots protected.

• Liquid suspensions are suitable for coating seeds. These inocula can also be sprayed

The formulated AMF bio-fertilizers should be positioned near the roots, with avoiding the spreading products and favor the injection or burying method. Finally,

• Inoculated planting support, ready to use and particularly suitable for soil-less

○ Pre-inoculated plants (vine, chestnut, etc.) ready for planting and whose

○ Seeds coated with AMF propagules (mainly spores) which make it possible

○ Organic fertilizers and amendments containing AMF propagules [7].

The aim of our present study is to set up a production unit for AMF biofertilizers, which has the advantage of covering the biological deficit of arid soils, a problem which continues to degenerate more and more. Indeed, the installation of a bio-fertilizer unit must follow some criteria such as: appropriate production, location, construction space, equipment, machinery, other laboratory equipment and working capital. The bio-fertilizer production unit should be founded in a homogeneous area based on the interaction of soil characteristics, geomorphology and climate. This place should be qualified as a buffer to reduce the risk of contamination during the process of production and quality control. The overall architecture of the unit is the key element for the success of any economic project. In our case,

mycorrhization has been checked before marketing,

to spread and inoculate a plot in a single pass,

**4. Design of AMF bio-fertilizer production unit**

on growing media or injected into the soil at the base of plants [4–7, 39].

*DOI: http://dx.doi.org/10.5772/intechopen.94084*

powder, tablets, pralins, or liquid suspension:

the roots;

coating.

culture,

evenly over the roots.

it is also possible to produce "2 in 1" products:

#### *Native Arbuscular Mycorrhizal Fungi and Agro-Industries in Arid Lands… DOI: http://dx.doi.org/10.5772/intechopen.94084*

*Mycorrhizal Fungi - Utilization in Agriculture and Forestry*

species that colonizes the plant [26–37, 41].

**3.4 Formulation of AMF bio-fertilizers**

*3.3.1 On-farm method*

all case, efficiency of AMF bio-fertilizers is related to the better combination AMF

To have AMF bio-fertilizer produced at a large scale with low cost, studies has been developed to test the multiplication of AMF spores under field conditions, called the "on-farm method". These studies explore AMF colonization with strains isolates that are environmentally adapted to native environmental conditions, which potentially representing a low-cost alternative for farmers [26, 37]. This technic is based on sowing plants seeds or seedlings in intact soil cores or mixed soil samples for sufficient time to allow development of AMF symbiosis and then roots were sampled, processed and assessed to measure mycorrhiza formation [26]. Indeed, [41] showed that multiplication on-farm with *Rhizophagus clarus* and *Claroideoglomus etunicatum* grown in agro-industrial residues, such as sugarcane bagasse, is a good strategy for the multiplication of AMF, leading to excellent inoculum potential and large numbers of spores. As well as, the on-farm technic allows farmers and nursery workers to access inoculums with the most effective AMF strains for their culture and their soil and climate conditions; furthermore, they can produce seedlings already inoculated with adapted AMF strains, which enhancing their establishment of s in the field conditions. In addition Moreira et al. [26] produced AMF inoculum with *Rhizophagus clarus*, *Claroideoglomus etunicatum* species, and native AMF from pineapple and coffee plantations, using spores multiplied by the on-farm method to enhance the growth of pineapple and coffee plantlets. These authors concluded that AMF inoculum favorite growth of the commercial tested crops with a high viability of AMF spores. This method is recommended for arid land due to their specific AMF strain and low coast, it could be applied and generalized in order to developing agricultural practices in these areas. Furthermore, because fungi carry different amounts of nutrients for plants, they may affect the growth of plants differently [37], it is preferable to use mixed AMF strains adapted to native conditions which could maximize the absorption of limiting nutrients [26]. This phenomenon could provide more benefits compared to colonization with exotic AMF strains or with single AMF species. Moreover, the mixed inoculation of AMF might have the characteristic of complementarity, exploiting the best of each

Formulation technologies largely take care of possible adverse environmental effects and factors that may render the inoculum ineffective [34]. In fact, biofertilizers are generally prepared as liquid suspension or more generally solid support containing different types of propagules: spores, fungal mycelium, mycorrhizal root fragments [7]. Generally AMF bio-fertilizers are presented with multi AMF species, which are closer to natural conditions; because in natural ecosystems it is rare to encounter only one species of mycorrhizal fungus. The presence of several fungal species allows the inoculum to respond to a greater diversity of culture conditions [13, 39]. Bio-fertilizer support can be composed of peat, vermiculite, lignite powder, clay, talc, rice bran, granulated rock phosphate, charcoal, soil, straw compost of rice or wheat otherwise a mixture of these materials. In current practice, for better framework life of the bio-fertilizer formulation, the support is selected on the basis of the viability of the microorganisms mixed with them. Likewise, the pre-sterilization of the support and its nutrient enrichment is the other strategy to improve framework, allowing AMF to be maintained

genotype/host plant genotype/adaptation to specific abiotic stress [13, 27].

**84**

in a non-competitive microenvironment. Sucrose, maltose, trehalose, molasses, glucose, and glycerol are additional nutrients and cell-protecting agents commonly used along with a support to ensure maximum cell viability and extended shelf life [4, 39]. After production, AMF bio-fertilizer may be in the form of granules, powder, tablets, pralins, or liquid suspension:

The formulated AMF bio-fertilizers should be positioned near the roots, with avoiding the spreading products and favor the injection or burying method. Finally, it is also possible to produce "2 in 1" products:

	- Pre-inoculated plants (vine, chestnut, etc.) ready for planting and whose mycorrhization has been checked before marketing,
	- Seeds coated with AMF propagules (mainly spores) which make it possible to spread and inoculate a plot in a single pass,
	- Organic fertilizers and amendments containing AMF propagules [7].