XX XX 8 May/June

# MAGES 4.0: Accelerating the world’s transition to medical VR training

Paul Zikas  ORamaVR SA, University of Crete, ICS-FORTH  Antonis Protopsaltis
ORamaVR SA, University of Western Macedonia  Manos Kamarianakis  Mike Kentros
Nick Lydatakis  ORamaVR SA, University of Crete, ICS-FORTH  Dimitris Angelis
Michalis Tamiolakis  Michael Dodis  George Kokiadis  ORamaVR SA, University of
Crete  Ilias Chrysovergis  Flora Fyka  ORamaVR SA  Maria Pateraki  ORamaVR SA,
National Technical University of Athens, ICS-FORTH  George Papagiannakis
ORamaVR SA, University of Crete, ICS-FORTH

(2023)

###### Abstract

In this work, we propose MAGES 4.0, a novel Software Development Kit (SDK) to
accelerate the creation of collaborative medical training scenarios in VR/AR.
Our solution offers a versatile authoring platform for developers to create
medical simulations in a future-proof, low-code environment. MAGES breaks the
boundaries between realities since students can collaborate using virtual and
augmented reality devices at the same medical scene. With MAGES we provide a
solution to the 150-year-old training model which is unable to meet the level
of healthcare professionals needed. Our platform incorporates, among others,
the following novel advancements: a) 5G edge-cloud remote rendering and
physics dissection, b) realistic real-time simulation of organic tissues as
soft-bodies, c) a highly realistic cutting and tearing algorithm, d) neural
network assessment for user profiling and, e) a VR recorder to record and
replay or resume the training simulation from any perspective.

††journal: Computer Graphics and Applications

The metaverse is still being defined, both literally and figuratively. Along
with it, many questions remain, including “who will build the metaverse and
how?” Such questions often rise when we are referring to the
metaverse111https://bit.ly/3QEmUdJ as a massively scaled and interoperable
network of real-time rendered 3D virtual worlds, effectively experienced
synchronously and persistently by an unlimited number of users. Building such
an ecosystem from scratch would require significant effort involving tasks of
tremendous complexity, especially if a single tool would be used to create it.

The extended pandemic crisis highlighted the need for effective medical
training, along with the inadequacy of the 150-year-old training model.
Computational medical science [1] aims to accelerate world’s transition to VR
medical training in the metaverse and empower medical professionals to enhance
their proficiency and ultimately improve patient outcomes.

To serve the causes above, we present MAGES 4.0, the medical VR industry’s
first Software Development Kit that allows rapid prototyping of any shared,
collaborative networked medical training in VR, in a fraction of time and
cost. It draws its robustness from the 20 years of academic Research &
Development, incorporating the latest advancements into a novel medical VR/AR
software development kit.

The computational results achieved with MAGES SDK exceed those typically
reached by large teams of domain expert developers of similar engines. Being
layered on top of existing game engines, such as Unity3D and Unreal Engine
(see Fig. 1-Left and 2), it brings a low-code virtual world authoring platform
to developers with even a moderate knowledge of these engines. It is thus
possible, even for a single developer, to create a full VR medical training
simulation.

Figure 1: (Left) The visual scripting editor. (Right) A surgeon with two
students performing a collaborative Total Knee Arthroplasty as generated using
MAGES, in Unity3D.

## PREVIOUS WORK

In the past years, a variety of virtual world simulations [2] have been
introduced, tailored to particular research problems, most of them presented
as proof of concept. A great number of applications have been designed for
entertainment/multimedia systems [3], education/training [4, 5, 6],
collaboration, Cultural heritage, and medical training [4, 7]. Each simulation
has various strengths, but because they were often designed with specific use
cases in mind, each is also limited in different ways. Most VR systems provide
high-fidelity photorealistic scenes [3], user immersion and presence that
allow human-centric interaction in VR [8]. Existing simulation environments
are differentiated from the other as they combine VR training functions in
various domains, while others, only very few of them [4], provide realistic
physics simulations on surfaces with rigid body, soft body, cloth, and fluids.
Innovative multi-user virtual environments (VE) [3, 4] provide a great asset
in all fields especially realized in the COVID era.

Some VE authoring tools [5, 4] allow reuse/import of content/assets that
impacts the tool’s usability and effectiveness, as it enables designers to
significantly reduce their effort, and generate custom user-experience that
otherwise would be impossible. In that respect, some novel, innovative, and
interoperable authoring platforms allow users with non or limited programming
knowledge to design a VE fast and easy [4], setting early the basic and
essential requirements for the building of the metaverse [2]. In that respect,
VR software design patterns method [9] is a research concept with already
concrete results that will be utilized in the building process of the next-
generation VR training applications for the metaverse.

## MAGES 4.0 INNOVATIONS

The MAGES SDK is named after its most unique features:

Multiplayer With GA Interpolation (M): A custom Geometric Algebra (GA)
interpolation engine allows up to 300 simultaneous users in the same session
of the VE. To the best of our knowledge, this is the highest number of active
concurrent users in the same scene, whereas state-of-the-art applications
offer only a few tens of concurrent users with only one active and rest being
spectators. This remarkable achievement is accomplished by the transmission
over the network layer of GA-based representation forms of the VR scene
transformation data, which support higher data compression rates. The use of
such alternative forms by MAGES reduce the required data that must be sent
over the network by up to 33%, whereas the reduced amount of data sent in
these forms yield visually better interpolated results, with the overall QoE
being sufficient even when the network quality, in terms of bandwidth,
seriously deteriorates [10].

Analytics (A): A powerful advanced analytic engine system allows tracking and
visualization of the student’s progress. As this feature enables scoring per
user action in a simple manner, it provides an easy way for the tutor or the
students to quickly inspect and identify the parts of the training where they
had great or poor performance.

Geometric Algebra Deformable Animation, Cutting and Tearing (G): The under-
the-hood GA engine used to solve the animation equation is responsible not
only for model deformation but for a series of features that exploit it.
Specifically, the ability to perform cuts, progressive tears and drills on
skinned soft-body meshes is now feasible in real-time, with increased realism
suitable for VR immersive applications.

Editor with Action Prototypes (E): An incorporated visual scripting editor
along with the prototypes of basic building blocks, called _Actions_ , are
used to rapidly accelerate content creation for VR/AR simulations. Based on
novel software design patterns [9], this tool enables creation of medical
training operations, at a fraction of the time and cost previously needed.

Semantically Annotated Deformable, Soft and Rigid Bodies (S): Towards a highly
realistic recreation of a virtual surgery operation, we have designed and
developed a particle system, suitable for real-time elasticity simulations of
human tissues and organs.

## HOW MAGES WORKS IN 5 STEPS

In this section we present the five main steps, a developer should follow, to
create a VR training simulation:

  1. 1.

Design training storyboard: Utilizing our visual scripting editor, developers
create the steps of each scenario.

  2. 2.

Create virtual assets: Gather all the medical 3D content (tools, human anatomy
etc.).

  3. 3.

Author training Actions: Generate the Action scripts using our action
prototype design patterns. Developers create programmable actions along with
analytics and network behavior using the embedded authoring tools.

  4. 4.

Build the medical VR training simulation: Build and deploy the executable
application to a wide range of supported headsets/platforms, operating systems
and computing devices.

  5. 5.

License the simulation: Optionally connect a MAGES SDK provided cloud
management license and analytics server to the training simulation.

## ACTION PROTOTYPES

Code reusability and prototyping are two major principles in software
architecture. The structure of software systems and the communication between
its modules is described through software design patterns. Software design
patterns are reusable solutions for common design problems that often occur
during software development [11].

Especially in training simulations, developers need to implement highly
interactive behaviors for the students to follow. For this reason, we
introduce Action prototypes, as novel software design patterns for low-code
behavioral tasks in training scenarios. We classified the majority of the
physical tasks into programmable and easily extendable code patterns. We
implemented basic behaviors like the insertion, removal, usage or even
cognitive behaviors like questions into separate programmable entities.
Developers can inherit those entities and develop their own Actions with a few
lines of code.

To minimize the need of coding, we developed an authoring tool to
automatically generate Action scripts. Developers can create Action scripts by
selecting the necessary physical objects for each task (e.g tools, specific
parts of the body) and subsequently the system will automatically generate an
Action script. This script contains the basic-default behavior of the
particular object, allowing also to easily to include additional behavior by
mounting extension-scripts.

## THE TRAINING SCENEGRAPH

A training scenario contains a number of carefully defined steps, in a
sequential or multi-path manner. MAGES lie upon training Scenegraph, a highly
dynamic, acyclic graph representing the training scenarios. Each node defines
an _Action_ , a specific task to be completed by the trainee.

Scenegraph is not just a static tree, it’s a dynamic graph. An educational
scenario can lead to multiple paths according to user’s actions and decisions.
To accommodate this need, the training Scenegraph can generate new paths and
deviate from its original form while the user explores the training scenario.

To rapidly accelerate the content creation we built a visual scripting
authoring tool. This is a low-code system that enables easy authoring of
Actions while reveling the scenario from a higher level perspective. The
visual scripting editor, consisting of interactive nodes able to be modified
by developers, is able to generate a training simulation in a future proof and
easy to use graphical environment (see Fig. 1(Left)).

Figure 2: Developing a Total Knee Arthroplasty with MAGES in Unreal engine.

This novel system can create medical simulations eight times faster than
without the visual scripting editor.

## ANALYTICS EDITOR

User assessment is crucial in medical simulations as students can identify
their skills, while teachers can note any difficulties or points of
improvement. For this reason we integrated into MAGES an analytics system to
assess and track the progress of each individual. In our training scenarios,
we are asked from the medical advisors to track specific parameters through
the operation. Those can vary from wrong angle during incisions or drilling,
to time constrained decisions or even contamination issues.

MAGES features a novel analytics system to configure and present the
assessment data. We build an easy to use interface on top of our visual
scripting engine that, with minor modifications, enables analytic
functionality to our Actions. For this reason, we introduced the scoring
factor, a component that tracks data from objects or from the student’s
actions. Some of the scoring factors we implemented are velocity scoring
factors, to track if a user is moving a fragile object too fast, error
collider scoring factors, to check for possible contamination, as well as
angle scoring factors to check the proper placement of implants. Custom
scoring factors are also supported allowing developers to extend the existing
classes and create their own factors. Developers can assign multiple scoring
factors in each Action. The final score for each Action is calculated as a
weighted average of all assigned scoring factors.

After each simulation, users can visualize their analytics from within the
VR/AR environment. Analytic reports are also uploaded automatically to our
online portal allowing supervisors and teachers to view their students’
progress. To implement the upload to the portal, developers need to modify
just a single API call in the MAGES configuration file.

## AUTOMATIC HAND POSTURES

Interactive virtual characters are nowadays common place in VR/AR
applications. Designing a virtual human-like hand which is able to touch and
grasp objects with realistic hand and finger adjustments, often trying to
imitate the human brain way of reasoning, it’s a matter of necessity. Grasping
hand animation itself includes several problematic aspects that make finding a
unique solution extremely hard. Especially concerning human-like hands the
main challenge is trying to imitate the human-brain way of thinking and
replicate the unconscious movements a person makes to grab an object.

Reducing the number of possible grasp movements is fundamental. The applied
methodology should also respect the physiology of the human hand [12]. This
applies to anatomical joint limits, angle limits due to tendon links between
fingers (dynamic constraints) as well as angle limits that force a natural
posture.

Figure 3: A student has just dissected the left part of the liver, during a
hemihepatectomy operation. MAGES provides a realistic simulation of hand
postures when grasping various objects.

In MAGES we designed an algorithm for intuitive object grasping with easy
configuration an high accuracy mimicking the human way of reasoning. This
algorithm is flexible enough to support hand structures having an arbitrary
number of fingers and an arbitrary number of phalanges for each finger. Given
the hand bones structure, the position of the character and an object, the
algorithm finds the most suitable grasping position for the character’s hand
on that object. Grasp poses are generated by a combination of a generic grasp
movement and a target object the fingers collide with. Movements are stored
using only an initial pose and a final pose, so that the algorithm can
generate an arbitrary number of interpolations between them. The algorithm
runs in real-time without performance issues even with complex objects (see
Fig. 3).

To simulate grasp poses, we implemented a procedure that based on a start hand
position and on a grasp movement, generates the grasping pose, taking into
account the object, by computing the final positions and rotations for all the
bones. When the user grabs an object, starting from the root joint of the
hand, we interpolate the joints to reach the final pose. If a bone gets in
contact with the object it is excluded from the following calculations. This
algorithm runs recursively for each joint to reach a firm grip.

## BLENDING REALITIES

Virtual worlds should not get limited from the used medium. They should blend
with the real world in a discrete way. For this reason, MAGES can populate
applications for both virtual and augmented reality headsets without complex
configurations and additional work from the developer. We designed the SDK in
such a way to enable the collaboration between VR and AR headsets in the same
simulation (see Fig. 4). In this regard, if a developer needs to build an
application for an AR headset (e.g HoloLens, Magic Leap etc), the only
additional work required is to import the respective external SDK. Our
universal XR camera works with any headset regardless the targeted reality.

We implemented this methodology with our custom device controller. MAGES
encapsulates a central device controller to describe the needed methods for a
device that runs one of our simulations. Those can vary from the camera
movement to the controller button behaviors, forming a set of rules among the
devices. In this way, the simulation remains clean of additional coding,
regarding the Head-mounted displays (HMDs), while internally the device
controller manages the HMD inputs. If MAGES detects an augmented reality
headset, it automatically culls the unnecessary VE objects, like the walls of
the operating room as they should not be rendered.

It is a breakthrough for virtual application as we break the boundaries
between different technologies that up until recently were used separately.

Figure 4: Collaborative Total Knee Arthroplasty simulation between a Magic
Leap (point of view) and an oculus quest 2 headset (the medical assistant).

## EDGE-CLOUD REMOTE RENDERING

Although the VR hardware landscape evolves rapidly, with SoC solutions that
aim to reduce the performance gap to high-end desktop GPUs , still standalone/
untethered VR solutions are less capable in supporting high quality strong
interactive VR services due to their reduced GPU capabilities and battery
life. This favors the exploitation of software solutions that offload
computationally intensive tasks from end devices to cloud/edge resources to
support processing and storage.

In our approach, a signaling server is integrated to complete the handshake
and the ICE candidate negotiation between the HMD and the offloaded
application, based on RFC 8445. The signaling server establishes a
communication channel, either by using a STUN server to determine connectivity
between the two components, or alternatively a TURN server, and decides which
encoding format is the most appropriate to use for the video stream.

One major road-blocker for VR offloading using Unity3D Render Streaming, is
that Unity3D does not support a capability for stereo render-to-texture. This
further required a re-engineering of the render streaming by using two
complete Unity3D cameras in the scene with customized settings for the left
and right eye to render to separate textures for each eye, which are then
blitted onto the final target. Furthermore, a custom data stream is used to
send transformation and controller data from the HMD to the offloaded
application. Due to the simulation of two Unity3D cameras the frame-rate
between the offloaded component and the HMD decreases to approximately 200%
the frame time or 44.5 fps on average. The offloaded application runs inside
of a Windows virtual machine on a linux host. By using KVM with GPU-
passthrough, performance can come close to bare-metal speeds. Additionally, a
custom TURN server is hosted inside of the VM, which allows WebRTC to
consistently establish communication with the client, even if they happen to
be behind a NAT.

## DISSECTED PHYSICS ENGINE

A typical monolithic game engine pipeline involves the execution of physics
related calculations, performed on CPU, alongside with scene rendering
calculations, performed on CPU/GPU. Both these heavy computations are either
performed on high-end untethered HMDs, or on VR-ready PCs with a tethered HMD.

The designed methods and techniques for the dissection of the Unity3D game
engine pipeline, creates two autonomous, deployed separately, bidirectionally
communicating components: the Host, and the Physics Server. The Host is
responsible of maintaining the game logic and of processing the graphics
rendering, while the Physics Server is responsible for performing physics
computations (see Fig. 5). Main goal of the dissected Unity3D pipeline is to
allow any game object on the Host’s scene to be fully simulated by the
separated Physics Server (see Fig. 6).

The dissection approach is based on the splitting of the Host’s game objects
into graphics objects, residing in the Host, and physics objects, residing in
the Physics Server. This is accomplished by eliminating the physics attributes
(e.g., mass, collider dimensions etc.) from all Host’s game objects, which are
subsequently created in the Physics server as physics objects with the same
parameters, in an easily transmittable and compressible format. The Physics
Server acts on a completely passive nature since it retains no knowledge
regarding the Host’s scene, the game loop or the behaviors. The imposed inter-
calls are streams of transformations between the two services.

Figure 5: A high-level architecture diagram of the physics dissection. The
physics are calculated only on the physics server while the render service
receives the rendered image.

The dissected Physics Server runs inside of a Windows virtual machine on a
linux host. Further containerization of this service is under consideration.
The results of the dissected pipeline, show a minor uptake of 0.03ms on the
latency from the new physics service, producing identical total frame-rate of
44.5 fps. The produced results have confirmed our plans of hosting extremely
high intensity physics computations in a separate edge service orthogonally to
the rendering service, and allow the Physics server to serve multiple
Rendering services, collaborating in the same multi-user session, in a N-1
setup, instead of the current 1-1 setup.

Figure 6: The physics are calculated only to the server while the client
receives the rendered image.

## SOFT-BODIES SIMULATION

In the real world, there exist certain deformable objects, e.g., soft or hard
tissues, which are “naturally” deformed when external forces are applied on
them. MAGES incorporates a framework that aims to achieve this, via the so-
called _soft-body mesh deformation_. The idea behind this methodology is to
create a layer of control points, called _particles_ , on top of the mesh
model that, when translated, would affect the vertices of a model in a
weighted manner. Ultimately, the visual effect we aim is similar to what would
happen in reality if we pinched and pulled the material at that point, towards
the displacement direction of the particle.

The particles are spawned and bound via springs to some initial positions on
the mesh’s surface, uniformly distributed via a Poisson Sampling mechanism.
Particles lying sufficiently close are “connected”, forming the desired layer
of control points. Furthermore, we assign vertices to particles that have a
distance from them below a specific model-dependent threshold.

After this initial setup, every time a particle is displaced away from its
initial position it will trigger several events. Firstly, the particle’s
displacement will amount to a weighted displacement of all particles it is
assigned too, with each vertex’s displacement being inversely proportional to
its distance from the particle. Furthermore, the particle’s movement will
affect connected particles by a fraction also inversely proportional to their
in-between distance. As neighbouring particles are displaced, they will, in
turn, affect vertices in a diminishing extent. Lastly, particles that are
moved away from their initial position tend to return with a velocity that is
proportional to the displacement, similarly to a spring. With these
fundamental rules applied to the model, we are able to simulate elasticity on
the model’s surface.

Table 1: Running times required to tear and cat various softbody models: a liver and a heart. Results were obtained by running simulations in a PC with an i7-11375H CPU, 16 GB of RAM and an RTX 3060 GPU.  Characteristics | Liver | Heart
---|---|---
Number of vertices | 515 | 2527
Number of triangles | 768 | 4968
Number of particles | 191 | 179
Tear Operation | Performance per tear segment
Perform Tear | 0.4 ms | 3 ms
Update particles | 2.3 ms | 7.3 ms
Total Time | 3.25 ms | 11.19 ms
Output | 140 fps | 90 fps
Cut Operation | Performance
Intersection points | 128 | 356
Total Time | 10.9 ms | 17.2 ms
Output | 90 fps | 55 fps

## HIGHLY REALISTIC PROGRESSIVE TEARING AND CUTTING

By exploiting the GA-based interpolation engine, we are able to simulate
realistic unconstrained consecutive tears or cuts, on a soft-body model,
similar to the ones performed in real life by a surgeon in the operating room.
Based on pure geometric operations on the surface mesh, we are able to perform
such actions and obtain real-time results in XR, even in low-spec devices such
as mobile VR HMDs. The specific calculations are performed in real-time within
a 10-30 ms to preserve user immersion.

Although bibliography [13] describes diverse ways on how to cut a 3D model,
most of these methods are not suitable for VR, since the specific calculations
must be performed in a real-time manner within a few ms to preserve user
immersion. Latest developments [14] allow for basic operations, such as
cutting, tearing or drilling on a rigged mesh model, to be run in near real-
time. The significance of this framework lies on the fact that it overcomes
current state-of-the-art limitations, where similar tears on a rigged 3D model
in VR are predefined via linear-blend skinning animations, in order to allow
them to playback in real-time.

Paired with the soft-bodies framework, our tearing and cutting algorithms
allows the simulation of realistic, surgical-grade continuous tears,
especially valuable in the context of medical VR training. Furthermore, our
algorithms are based on simple geometric predicates on the rigged mesh, and
therefore do not require specific model pre-processing. Ultimately, our
framework allows the user to freely cut or tear in a consecutive way any 3D
mesh model, under collaborative networked virtual environment, reaching up to
140 fps

depending on scene complexity (see Table 1). Tears can now be simulated to
“open”, replicating the tissue behaviour in real-life tears, providing
immersive visual results for soft-body materials. And of course, rigged models
can be further re-animated and torn or cut again, enabling a number of complex
surgical operations to be implemented in VR.

Figure 7: Interactive heart model featuring our deformable and tearing
algorithms. The student performed an incision to observe the internal
structures.

## CONVOLUTIONAL NEURAL NETWORK ASSESSMENT

Despite the effort, only limited systems involve procedures for assessing user
progress inside the immersive environment, that either evaluate only trivial
tasks or require a huge amount of time by the reviewers [15]. On the other
hand, the need for real-time automated evaluation of user’s actions is
constantly increasing. State-of-the-art methods for similar tasks either
require the development of complicated task specific computer vision
algorithms or support very simple tasks [16].

MAGES proposes a deep learning based system, that is able to assess, in real-
time, user actions within a VR training scenario. The method enables the rapid
development of trained assessment functions, since it utilizes data
augmentation to minimize the amount of labelled data (e.g.,
poor/mediocre/excellent performance) that need to be collected. Using this
system, we are able to assess e.g. the incisions, i.e., tears and cuts,
performed in VR medical operations by the trainees. The usefulness of this
feature is especially highlighted in cases where we have a predefined
operation, or exemplary task from a teacher, and students must be trained to
replicate the teacher’s movements precisely. Furthermore, by using transfer
learning, these assessment functions can be reconfigured to support similar
tasks, thus reducing even more the amount of training data.

Various underlying techniques were used to make this achievable. Data
collection involved capturing transformation data (translation & rotation) of
the active virtual tool (e.g. a scalpel) on distinct frames. The data set was
made uniform to amend for the fact that execution of an action is user
dependent. Data augmentation techniques were applied to increase the training
data towards enhancing the training process of the $15$-layer CNN . Since low
training and inference times are preferred, the lightweight model used is able
to provide high-accuracy scoring results. Incorporating the trained CNN in the
MAGES allows assessing the user’s actions with minimal performance overhead.

## VR RECORDER FOR ASSESSMENT

Recording a training session in VR can serve as an additional and powerful
educational tool. Especially in virtual and augmented reality, a recorded
session is not only a three dimensional video but the student can relive the
experience from a different perspective. The bibliography shows great interest
on this matter with the majority of projects focusing on video recording and
motion capturing [16].

MAGES incorporates a different, highly accurate and lightweight solution to
record and replay any collaborative training VR/AR session [17]. The
underlying algorithm allows for the first time to record the VR/AR session by
capturing the transformation of objects (position, rotation) and user-driven
events (interaction, decisions). This results in a highly accurate recording
of the scenario that requires minimal storage space (approximately 1MB per
minute per user) and minimal overhead (see Table 2). Additionally, we
efficiently handle and record the user audio and synchronize it using a
timestamp algorithm.

After recording the session, users can select the recordings and replay them
while observing as a spectators. They can join in the same environment as they
performed the scenario and re-live their experience, paying attention to the
details that were not possibly visible when they initially “played” the
scenario.

Table 2: Measuring the FPS burden of a VR application due to the Recorder feature. Results were obtained by running simulations in a PC with an i7-11375H CPU, 16 GB of RAM and an RTX 3060 GPU. Performance | Session without | Session with
---|---|---
VR Recording | VR Recording
Average FPS | 89.56 | 85.13
Minimum FPS | 76.56 | 68.78
Maximum FPS | 93.29 | 92.57

## CASE STUDIES

In this section we present honorable medical simulations built with MAGES SDK.

Figure 8: (Top)A collaborative total knee arthroplasty simulation with 25
participants in the same operating room. (Bottom) The swab insertion process
from the covid test simulation.

Total knee and hip arthroplasty: In collaboration with University of Southern
California and New York Univeristy, we created both orthopaedic operations as
a part of their curricula. We also conducted a clinical trial that proves
skill transfer from virtual to the real world [18].

Emergency trauma scenarios: This is a collection of emergency on-the-field
simulations (extrication from car, first aid) and an intubation process as
part of an online course for the University of Athens.

STARS: Patient education and empowerment through knowledge. We developed this
informative and stress relief applciation for patients in collaboration with
ICS-FORTH and the Human Computer Interaction Lab.

VRADA: In collaboration with the Aristotle University of Thessaloniki, the
University of Thessaly and Biomechanic solutions we created a VR bike
simulation with cognitive questions that allows older people with mild
cognitive impairment symptoms to simultaneously practice physical and
cognitive skills on a dual task. We published a clinical trial with our
results [6].

Behavioral Health, Social Worker: Two cognitive training simulations for soft
skills in collaboration with the Western Governors University and Fayetteville
State University respectively.

Covid-19 PPE & swab testing: During the covid-19 pandemic, we developed this
application to educate medical personnel the proper use of personal protective
equipment and how to take covid samples for test (see Fig. 8). It was
performed in collaboration with Inselspital University hospital of Bern and
the New York University. We conducted two clinical trials [19, 20], to explore
the effectiveness of VR simulations versus traditional learning methods.

Anatomical viewer: Non Nocere, developed an anatomical viewer for laminectomy
and discectomy using MAGES SDK.

REBOA: In collaboration with the Inselspital University hospital of Bern we
developed a endovascular balloon occlusion of the Aorta simulation.

## CONCLUSIONS AND FUTURE WORK

In this work we presented MAGES 4.0, a novel development kit that allows rapid
creation of any collaborative medical VR/AR simulation. The above novelties,
rapidly accelerate content creation while maintaining a stable, multi-plaftorm
authoring tool for medical simulations.

In the future, we aim to improve the developer experience in MAGES. Our goal
is to reach a point where a novice developer can learn and create with all
existing actions, cloud and analytics in a single day. In addition,
experienced developers should also extend the codebase (Actions, prototypes,
mechanics) with ease. Furthermore, we consider to support the Universal Scene
Description (USD), an open ecosystem that empowers the collaboration in 3D
virtual worlds, invented by Pixar Animation Studios. There is a lot of
research in this field which can be beneficial for medical training as we move
towards metaverse.

## ACKNOWLEDGEMENT

The project was partially funded by the European Union’s Horizon 2020 research
and innovation programme under grant agreements No 871793 (ACCORDION), No
101016509 (CHARITY) and No 101016521 (5G-EPICENTRE).

## References

  * [1] Papagiannakis, G., 2021. A computational medical XR discipline. arXiv preprint arXiv:2108.04136.
  * [2] Coelho, H., Monteiro, P., Gonçalves, G., Melo, M. and Bessa, M., 2022. Authoring tools for virtual reality experiences: a systematic review. Multimedia Tools and Applications, pp.1-24.
  * [3] Nebeling, M., Lewis, K., Chang, Y.C., Zhu, L., Chung, M., Wang, P. and Nebeling, J., 2020, April. Xrdirector: A role-based collaborative immersive authoring system. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (pp. 1-12).
  * [4] Papagiannakis, G., Zikas, P., Lydatakis, N., Kateros, S., Kentros, M., Geronikolakis, E., Kamarianakis, M., Kartsonaki, I. and Evangelou, G., 2020. MAGES 3.0: Tying the knot of medical VR. In ACM SIGGRAPH 2020 Immersive Pavilion (pp. 1-2).
  * [5] Cassola, F., Pinto, M., Mendes, D., Morgado, L., Coelho, A. and Paredes, H., 2021, March. A novel tool for immersive authoring of experiential learning in virtual reality. In 2021 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW) (pp. 44-49). IEEE.
  * [6] Hassandra, M., Galanis, E., Hatzigeorgiadis, A., Goudas, M., Mouzakidis C., Karathanasi, EM., Petridou N., Tsolaki, M., Zikas, P., Evangelou, G., Papagiannakis G, Bellis G., Kokkotis, C., Panagiotopoulos, SR., Giakas, G., Theodorakis, Y., ”A Virtual Reality App for Physical and Cognitive Training of Older People With Mild Cognitive Impairment: Mixed Methods Feasibility Study”, JMIR Serious Games 2021;9(1): e24170, doi: 10.2196/24170, 2021
  * [7] Zikas, P., Kamarianakis, M., Kartsonaki, I., Lydatakis, N., Kateros, S., Kentros, M., Geronikolakis, E., Evangelou, G., Catilo, P.A., Papagiannakis, G., Covid-19 VR Strikes Back: Innovative medical VR training, In ACM SIGGRAPH 2021 Immersive Pavilion (SIGGRAPH ’21), Association for Computing Machinery, New York, NY, USA, 202
  * [8] Zhang, L. and Oney, S., 2020, October. Flowmatic: An immersive authoring tool for creating interactive scenes in virtual reality. In Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology (pp. 342-353).
  * [9] Zikas, P., Papagiannakis, G., Lydatakis, N., Kateros, S., Ntoa, S., Adami, I., Stephanidis, C. Immersive visual scripting based on VR software design patterns for experiential training. Vis Comput 36, 1965–1977 (2020). https://doi.org/10.1007/s00371-020-01919-0
  * [10] Kamarianakis, M., Chrysovergis, I., Lydatakis, N., Kentros, M., Papagiannakis, G. (2022). Less Is More: Efficient Networked VR Transformation Handling Using Geometric Algebra. arXiv preprint arXiv:2203.10988.
  * [11] Nystrom, R.: Game Programming Patterns, 3rd edn. Genever Benning (2014)
  * [12] John Lin, Ying Wu and T. S. Huang, ”Modeling the constraints of human hand motion,” Proceedings Workshop on Human Motion, 2000, pp. 121-126, doi: 10.1109/HUMO.2000.897381.
  * [13] Jun Wu, Christian Dick, and Rüdiger Westermann. 2013. Efficient collision detection for composite finite element simulation of cuts in deformable bodies. The Visual Computer 29, 6 (June 2013), 739–749.
  * [14] Manos Kamarianakis and George Papagiannakis. 2021. An All-in-One Geometric Algorithm for Cutting, Tearing, and Drilling Deformable Models. Advances in Applied Clifford Algebras 31 (Jul 2021), 58.
  * [15] Erica Southgate. 2020. Using Screen Capture Video to Understand Learning in Virtual Reality. In 2020 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW). 418–421. https://doi.org/10.1109/VRW50115.2020.00089
  * [16] Aneeq Zia, Yachna Sharma, Vinay Bettadapura, Eric L Sarin, Thomas Ploetz, Mark A Clements, and Irfan Essa. 2016. Automated video-based assessment of surgical skills for training and evaluation in medical schools. Int. J. Comput. Assist. Radiol. Surg. 11, 9 (Sept. 2016), 1623–1636.
  * [17] Kamarianakis, M., Chrysovergis, I., Kentros, M., Papagiannakis, G., Recording and replaying psychomotor user actions in VR, arXiv, 2022, doi: 10.48550/ARXIV.2205.00923
  * [18] Hooper, J., Tsiridis, E., Feng, J., Schwarzkopf, R., Waren, D., Long, W., Poultsides, L., Macaulay, W., Papagiannakis, G., Kenanidis, E., Rodriguez, E., Slover, J., Egol, K., Phillips, D., Friedlander, S., Collins, M., “Virtual Reality Simulation Facilitates Resident Training in Total Hip Arthroplasty: A Randomized Controlled Trial”, The Journal of Arthroplasty, 2019, ISSN 0883-5403, https://doi.org/10.1016/j.arth.2019.04.002
  * [19] Birrenbach T, Zbinden J, Papagiannakis G, Exadaktylos A, Müller M, Hautz W, Sauter T Effectiveness and Utility of Virtual Reality Simulation as an Educational Tool for Safe Performance of COVID-19 Diagnostics: Prospective, Randomized Pilot Trial JMIR Serious Games 2021;9(4):e29586 URL: https://games.jmir.org/2021/4/e29586 DOI: 10.2196/29586
  * [20] Zikas, P., Kateros, S., Lydatakis, N., Kentros, M., Geronikolakis, E., Kamarianakis, M., Evangelou, G., Kartsonaki, I., Apostolou, A., Birrenbach, T., Exadaktylos, A. K., Sauter, T. C., Papapagiannakis G., ”Virtual Reality Medical Training for COVID-19 Swab Testing and Proper Handling of Personal Protective Equipment: Development and Usability”, Frontiers in Virtual Reality, Vol. 2, ISSN=2673-4192, doi: 10.3389/frvir.2021.740197 , 2022

Paul Zikas, is head of product at ORamaVR. He earned his BSc, and MSc in
Computer Science from the University of Crete. Currently he is developing his
PhD research topics at the University of Crete. Contact him at
<EMAIL_ADDRESS>
Antonis Protopsaltis, is an R&D executive at ORamaVR. He holds a Ph.D. in
Computer Science. He is also a Special Teaching Fellow with the University of
Western Macedonia. He specializes in Computer Graphics, and CAD methods.
Contact him at<EMAIL_ADDRESS>
Mike Kentros, is lead SDK developer & ML at ORamaVR. He earned his BSc, and
MSc in Computer Science from the University of Crete. Currently, he is PhD
candidate in the same university. Contact him at<EMAIL_ADDRESS>
Manos Kamarianakis, is an SDK R&D developer & ML at ORamaVR, holding a Ph.D.
in Applied Mathematics. He specializes in Computational Geometry, Geometric
Algebra and Computer Graphics applications. Contact him at
<EMAIL_ADDRESS>
Nick Lydatakis, is head of platform at ORamaVR.He graduated from Computer
Science department, University of Crete where he obtained both this BSc and
MSc on Computer Graphics. Contact him at<EMAIL_ADDRESS>
Dimitris Angelis, is a machine learning developer at ORamaVR. He earned his
BSc in Computer Science from the University of Crete specialized in
classification algorithms. Contact him at<EMAIL_ADDRESS>
Michalis Tamiolakis, is content and soft body developer at ORamaVR. He earned
his BSc in Computer Science from the University of Crete specialized soft body
simulations. Contact him at<EMAIL_ADDRESS>
Ilias Chrysovergis, is a machine learning developer at ORamaVR. He holds an
MSc in Communications Signal Processing from Imperial College London and a
MEng in Electrical and Computer Engineering from Aristotle University of
Thessaloniki. Contact him at<EMAIL_ADDRESS>
George Kokiadis, is a MR developer at ORamaVR. He completed his BSc in 2021
with a Thesis focused on Mobile XR and continues his academic path with an MSc
in Multimedia Technology. Contact him at<EMAIL_ADDRESS>
Michael Dodis, is a networking and Unreal engine developer at ORamaVR. He
first came into contact with programming through Unity, then went on doing
lower-level projects with computer graphics. Contact him at
<EMAIL_ADDRESS>
Flora Fyka, is an interaction and VR embodiement developer at ORamaVR. She is
a computer scientist from the Computer Science Department, University of
Crete, Greece. Contact her at<EMAIL_ADDRESS>
Maria Pateraki, is R&D executive at ORamaVR. She is also an Assistant
Professor in Photogrammetry at the National Technical University of Athens and
an affiliated Researcher of the CVRL at ICS-FORTH. Contact her at
<EMAIL_ADDRESS>
George Papagiannakis, is co-founder and CEO/CTO of ORamaVR. His academic
credentials include serving as Professor of Computer Graphics at the Computer
Science department of the University of Crete, Greece, as Affiliated Research
Fellow at the HCI lab at ICS-FORTH, where he leads the CG Group and as
visiting Prof of CS at the University of Geneva. Contact him at
<EMAIL_ADDRESS>