# Roadmap towards Meta-being

Tianyi Huang1,2
&Stan Z. Li1 &Xin Yuan1,3 &Shenghui Cheng1,2,3 1 Westlake University,
Hangzhou, China
2 Westlake Institute for Advanced Study, Hangzhou, Chin
3 Research Center for the Industries of the Future, Hangzhou, China

###### Abstract

Metaverse is a perpetual and persistent multi-user environment that merges
physical reality with digital virtuality. It is widely considered to be the
next revolution of the Internet. Digital humans are a critical part of
Metaverse. They are driven by artificial intelligence (AI) and deployed in
many applications. However, it is a complex process to construct digital
humans which can be well combined with the Metaverse elements, such as
immersion creation, connection construction, and economic operation. In this
paper, we present the roadmap of Meta-being to construct the digital human in
Metaverse. In this roadmap, we first need to model and render a digital human
model for immersive display in Metaverse. Then we add a dialogue system with
audio, facial expressions, and movements for this digital human. Finally, we
can apply our digital human in the fields of the economy in Metaverse with the
consideration of security. We also construct a digital human in Metaverse to
implement our roadmap. Numerous advanced technologies and devices, such as AI,
Natural Language Processing (NLP), and motion capture, are used in our
implementation. This digital human can be applied to many applications, such
as education and exhibition.

††footnotetext: Correspondence to: Shenghui Cheng
<EMAIL_ADDRESS>
##  1 Introduction

Since the invention of the camera in 1839, photos have been used to present
the appearances of great characters in history for nearly two hundred years.
However, photos cannot satisfy our desire to interact with someone. With the
development of computers, Metaverse which is widely considered to be the next
revolution of the Internet makes this desire come to the truth Cheng et al.
(2022).

Metaverse is a perpetual and persistent multi-user environment with the
mergence of physical reality with digital virtuality Loveys et al. (2022); Van
der Merwe (2021); Mystakidis (2022). It is based on the convergence of
technologies such as Virtual Reality (VR), Augmented Reality (AR), and Mixed
Reality (MR) Azuma (1997); Costanza et al. (2009); Schuemie et al. (2001), and
thus enables multi-sensory interactions with virtual environments, digital
objects, and people. Our desire to interact with someone can be implemented on
the digital humans in Metaverse Fang et al. (2021); 9757550. They are
autonomously-animated virtual people driven by AI and deployed in many
applications, such as healthcare, customer service, and education. However, it
is complex to construct a digital human which can be well combined with the
elements in Metaverse, such as immersion creation, connection construction,
and economic operation.

In this paper, we present the roadmap of Meta-being to construct the digital
human in Metaverse. In this roadmap, we first need to model and render a
digital human model for immersive display in Metaverse. Then we add a dialogue
system with audio, facial expressions, and movements for this digital human.
Finally, we can apply our digital human in the fields of the economy in
Metaverse with the consideration of security. In this way, we can appreciate
the motions and speakings of digital humans and interact with them in
Metaverse as in physical reality. Meta-being is a complex process. As we can
see from Fig. 1, many advanced technologies and devices are used to perform
our Meta-being.

Figure 1: The technologies and devices in Meta-being.

Our contributions are summarized as follows:

  * •

We present the roadmap for the Meta-being of digital humans.

  * •

We construct a digital human to implement Meta-being.

The rest of this paper is organized as follows. Section 2 reviews the
Metaverse and digital human beings. In Section 3, we give the roadmap of Meta-
being. In Section 4, we construct a digital human to implement Meta-being. In
the end, we conclude the paper in Section 5.

##  2 Digital Human Beings in Metaverse

The term Metaverse was invented and first appeared in Neal Stevenson’s science
fiction novel Snow Crash published in 1992 Stephenson (2003). It represented a
parallel virtual reality universe created from computer graphics to make users
worldwide access and connect through goggles and earphones. Digital humans as
an important part of Metaverse are computer-generated and computer-controlled
characters with a human-like appearance and can simulate human face-to-face
interactions Magnenat-Thalmann and Thalmann (2005). At the same time, with the
development of artificial intelligence, intelligent digital human has become
one of the new development directions. Digital humans are often used in
education, movies, games, and other fields. Each application domain requires
different attributes at different levels, such as autonomous behaviour,
natural language communication, recognition of real people, etc. Different
intelligent decision-making techniques, such as artificial neural networks,
need to be used to build a virtual human framework. Noma et al. Noma et al.
(2000) created a digital human presenter based on the JackTM animated agent
system in 2000. The input to this system is in the form of spoken text with
embedded commands (mostly related to the digital presenter’s body language).
The system then has the digital humans as a presenter with presentation skills
in real-time 3D animation synchronized with the voice output. However, due to
technical limitations, their digital human cannot synchronize lip movement
with sound, and the movements are not smooth and natural enough. In 2012,
Rizzo et al. Rizzo et al. (2011) developed a digital human, SimCoach, which
could provide healthcare information and support, especially for military
members, veterans, and their important ones. Specifically, SimCoach breaks
down barriers to online care by providing confidential assistance for military
members and others in exploring and accessing healthcare content and
facilitating on-site care. However, SimCoach’s anthropomorphic degree is not
high, and the interaction with users is insufficient. And he can only provide
support for specific groups of people and specific medical problems. Due to
technical limitations, these digital humans are not so much like virtual
digital humans but like computer systems with human images. In recent years, a
group of virtual people such as Miquela Sousa has appeared Dalayli (2020).
They exist in the network (e.g., social media), and the robot-focused
representations are highly anthropomorphic and interactive. Some researchers
think although they live in the virtual world as humans live in the real
world, with “jobs", “emotions", and “thoughts", they are neither real humans
nor real phenomena, but over time digital humans will be considered human in
social media, which is seen as a simulated environment Hubble (2018); Marwick
(2018). Though the ethical issues that come with it will be a considerable
challenge to tackle.

##  3 Meta-being

The digital cyberspace of Metaverse includes immersion creation, hardware
support, text interpretation, audio processing, connection construction,
economy operation, and security protection Cheng et al. (2022). To create a
digital human which can be well combined with the above elements, we propose
the roadmap towards Meta-being as follows.

  * 1)

The body and the face of our digital human should be modeled.

  * 2)

The resulting model should be rendered in detail.

  * 3)

We should give an immersive display of our digital human in Metaverse by some
display technologies, like VR and holographic projection.

  * 4)

We should generate a dialogue system with audio, facial expressions, and
movements for this digital human.

  * 5)

We can apply our digital human in the fields of the economy in Metaverse with
the consideration of security.

As we can see, Meta-being is a complex process. Many advanced technologies and
devices, such as 3D modeling, AI, and VR, are used to perform our Meta-being.
This roadmap is demonstrated in Fig. 2.

Figure 2: The roadmap towards Meta-being.

###  3.1 Modeling and Rendering

For a digital human in Meta-being, we should construct and then render a
digital human model Demirel and Duffy (2007); Zhang et al. (2022); Cheng and
Mueller (2015). To this end, we first need a human body model as a reference.
This model can be an adult or a child with an athletic body with relevant
muscle mass and no significant body fat Azevedo and Araújo (2020). Some 3D
scanning technologies, such as TEN24, can be used to capture the reference
model Berdic et al. (2017); Ratner (2012). Second, we can use 3D Face Modeling
technologies, like triangulation Morris and Kanade (2000),3D Morphable Model
Abrevaya et al. (2018), to construct the face of our digital human by the
reference photos in detail. Finally, in the rendering step, we add the hair
and clothes to our modeling digital human and fine-tune the appearance
structures, such as the color and texture of the skin. In this modeling, the
triangulation set, $T=\\{t_{1},t_{2},...,t_{n}\\}$ is along with a texture map
set, $A=\\{a_{1},a_{2},...,a_{n}\\}$, on all of the face triangles. The photo
set $V^{\prime}=\\{v^{\prime}_{1},v^{\prime}_{2},...,v^{\prime}_{k}\\}$ taken
with a calibrated camera in Metaverse can be computed as

$\displaystyle V^{\prime}=f(T,A)$ (1)

$a_{i}$ for a given triangulation $t_{i}$ can be estimated directly from the
actual photo set, $V=\\{v_{1},v_{2},...,v_{k}\\}$, where $v_{j}$ is along with
the same camera angle of $v^{\prime}_{j}$, as follows.

$\displaystyle A=f^{-1}(T,V)$ (2)

Thus $V^{\prime}$ from equation equation 1 depends only on $T$ and $V$. With
the assumption that the photos along with the triangulation constraints are
sufficient to define a unique face, our goal is to get the maximum Similarity
$U$ for the actual photo set and the photo set from Metaverse as

$\displaystyle\arg\max_{t_{i}}\sum_{j=1}^{k}\sum_{i=1}^{n}U(v_{j},v^{\prime}_{j}|t_{i})$
(3)

Finally, in the rendering step, we add the hair and clothes to our modeling
digital human of Turing and fine-tune the appearance structures, such as the
color and texture of the skin. Fig. 3 shows the modeling process and the
resulting model with different poses.

Figure 3: The modeling process and the resulting model.

###  3.2 Immersive Display

For the immersion creation in Metaverse, immersive VR mainly presents virtual
environments using cave automatic virtual environments or head-mounted
displays, which can provide users with a strong and immersive feeling of
presence in a 3D virtual world. The immersive display of digital humans is a
critical part of immersion creation in Metaverse for us to interact with
digital humans. Existing immersive display methods of digital humans are
usually based on predefined elements, including body motions and facial
expressions. This could preclude the immersive feeling of human interaction
with a digital human in the Metaverse since these elements may be different
from the real ones with limited predefined elements. To address this issue, we
should capture many real scenes and use those images or videos to generate new
ones and update the scenes in the metaverse within a short period of time,
possibly in real-time eventually. Computational imaging may provide a
promising solution to capture scenes efficiently in a low-cost, low-bandwidth
manner with the help of deep learning Barbastathis et al. (2019); Mait et al.
(2018).

###  3.3 Adding Operations

For a Meta-being digital human, we also need to create the dialogue system,
body movements, and facial expressions for our model. The dialogue of our
digital human is mainly based on our QA system generated by NLP algorithms
Harabagiu et al. (2000). This system first classifies the questions from users
by automatic speech recognition and then extracts the relevant answers Yu and
Deng (2016); Sarkar et al. (2015). The intent of the question could be
understood by identifying the starting phrase or words of this question. To
answer the question, we can identify areas of interest from the voluminous
content for this question and find the related paragraphs, and then score each
sentence in the related paragraphs by the number of proper nouns, the
similarity between this sentence and the question, and so on. A sentence can
be considered a part of the answer if its score is above a particular
threshold.

Human motion capture, like Xsens MVN motion capture, can be used to get the
body motions and for digital human Schepers et al. (2018). It tracks the
motion of the human body defined by a biomechanical model consisting of some
segments, like the neck, head, shoulders, feet, and so on. For each body
segment $B$, all kinematic quantities are expressed in a standard, local
coordinate frame $L$, which is a right-handed Cartesian coordinate system.
Given the sensor $S$, we can obtain the position of each segment ${}^{L}p_{B}$
and corresponding orientation ${}^{LB}q$ by

${}^{LB}q=^{LS}q\otimes^{BS}q^{*}$ (4)

and

${}^{L}p_{B}=^{L}p_{S}+^{LB}q\otimes^{B}r_{BS}\otimes^{LB}q^{*}$ (5)

where ${}^{BS}q$ denotes the relative orientation of the sensor w.r.t. the
body, ${}^{B}r_{BS}$ denotes the position of the sensor w.r.t. the segment
origin expressed in the segment frame, $\otimes$ denotes the quaternion
multiplication, and $*$ denotes the complex conjugate of the quaternion.

###  3.4 Applied in Metaverse

The economic operation in Metaverse is for exchanging virtual goods through a
blockchain digital system. Digital humans are one type of the most important
virtual goods in Metaverse. The applications of Meta-being digital humans are
driven by interaction and social connection. For example, interaction with
digital recreations of historical figures has existed for almost a decade,
e.g., an interactive life-size video of Abraham Lincoln that is used at the
National Civil War Museum. In this case, education has become an important
application of Meta-being digital humans. By Meta-being digital humans with an
artistic style, the children’s attention can be well attracted during the
learning process. Also, proposing a Meta-being digital human curriculum with a
ready-to-use software package would also improve/update the engineering
quality and provide a unique educational experience for students Demirel and
Duffy (2007).

Widespread adoption of the Metaverse comes with many unique threats to user
privacy and security, especially in the interaction with digital humans.
Biometric data reveals a host of personally identifiable information which can
in turn be used to potentially manipulate users on a psychological level
through the creation of avatars that are adaptable to user preferences Buck
and McDonnell (2022). AI may have the capability of privacy protection through
algorithms that automatically and dynamically detect user privacy references
from diverse contexts in the Metaverse Zang et al. (2022). An AI-powered
security system will emerge in the Metaverse.

##  4 The Implementation of Meta-being

As an implementation of Meta-being, we have made a digital human of Alan
Mathison Turing by modeling the digital human model of Turing and then
creating dialogue and movement functions, including body motions and facial
expressions. Then in Metaverse, we can appreciate the body motions, facial
expressions, and speakings of digital Truing. We also can talk to our digital
Turing in Metaverse. Our digital Truing can be applied to many applications,
such as education and exhibition. By mixing VR and AR, our digital Truing can
explain computer knowledge to students and introduce new products to consumers
in an immersive environment. We display three interesting action sequences of
our Turing model in Fig. 4-6.

Figure 4: The actions of our digital Turing

Figure 5: The action sequence of giving a class.

Figure 6: The action sequence of saying goodbye. Figure 7: The holographic
projection of our digital Turing.

Holographic projection technology breaks the traditional presentation of
images by combining 3D technology and holographic technology Elmorshidy
(2010); Huebschman et al. (2003). It has become a hot application technology
in the Metaverse for immersion display Aburbeian et al. (2022). By holographic
projection, we have presented our digital Turing at some expositions. Fig. 7
shows an example of our presentation. As we can see, our vivid digital Turing
is well integrated with the stage’s background.

##  5 Conclusion

In this paper, we have presented the roadmap for the Meta-beings to make
digital humans can be better combined with the elements in Metaverse. In this
roadmap, we model the body and face of a digital human in a 3D space. Then, in
the rendering step, we add clothes and appearance details to our model. We
also need to create a dialogue system, body movements, and facial expressions
for this model. In this manner, we can appreciate its motions and speaking and
interact with him in Metaverse as in physical reality. It is a complex process
to implement our Meta-being on a digital human. Many advanced technologies and
devices, such as AI, NLP, and motion capture, are used to perform our Meta-
being. As an example, we implement our Meta-being on Turing, a great scientist
of computer science.

## References

  * Abrevaya et al. [2018] Victoria Fernández Abrevaya, Stefanie Wuhrer, and Edmond Boyer.  Multilinear autoencoder for 3d face model learning.  In _IEEE Winter Conference on Applications of Computer Vision (WACV)_ , 2018.
  * Aburbeian et al. [2022] AlsharifHasan Mohamad Aburbeian, Amani Yousef Owda, and Majdi Owda.  A technology acceptance model survey of the metaverse prospects.  _AI_ , 3(2):285–302, 2022.
  * Azevedo and Araújo [2020] Hugo de Azevedo and António Araújo.  Towards a canon for digital human anatomy 3d modeling.  In _ARTeFACTo2020. International Conference on Digital Creation in Arts and Communication_ , pages 121–126. Edições Centro de Investigação em Artes e Comunicação, 2020.
  * Azuma [1997] Ronald T Azuma.  A survey of augmented reality.  _Presence: teleoperators & virtual environments_, 6(4):355–385, 1997.
  * Barbastathis et al. [2019] George Barbastathis, Aydogan Ozcan, and Guohai Situ.  On the use of deep learning for computational imaging.  _Optica_ , 6(8):921–943, 2019.
  * Berdic et al. [2017] Nikola Berdic, Dinu Dragan, Srdan Mihic, and Zoran Anisic.  Creation and usage of 3d full body avatars.  _Annals of the Faculty of Engineering Hunedoara_ , 15(1):29, 2017.
  * Buck and McDonnell [2022] Lauren Buck and Rachel McDonnell.  Security and privacy in the metaverse: The threat of the digital human.  2022\.
  * Cheng and Mueller [2015] Shenghui Cheng and Klaus Mueller.  The data context map: Fusing data and attributes into a unified display.  _IEEE Ttransactions on Visualization and Computer Graphics_ , 22(1):121–130, 2015.
  * Cheng et al. [2022] Shenghui Cheng, Yue Zhang, Xiaofei Li, Lin Yang, Xin Yuan, and Stan ZQ Li.  Roadmap towards metaverse: An ai perspective.  _The Innovation_ , page 100293, 2022.
  * Costanza et al. [2009] Enrico Costanza, Andreas Kunz, and Morten Fjeld.  Mixed reality: A survey.  In _Human machine interaction_ , pages 47–68. Springer, 2009.
  * Dalayli [2020] Feyza Ünlü Dalayli.  Representation of robots in the social media with the simulation universe: social media influencers and influencer robot miquela sousa.  _International Journal of Social Science_ , 3(2):87–102, 2020.
  * Demirel and Duffy [2007] H Onan Demirel and Vincent G Duffy.  Applications of digital human modeling in industry.  In _International Conference on Digital Human Modeling_ , pages 824–832. Springer, 2007.
  * Elmorshidy [2010] Ahmed Elmorshidy.  Holographic projection technology: the world is changing.  _arXiv preprint arXiv:1006.0846_ , 2010.
  * Fang et al. [2021] Zhixin Fang, Libai Cai, and Gang Wang.  Metahuman creator the starting point of the metaverse.  In _2021 International Symposium on Computer Technology and Information Science (ISCTIS)_ , pages 154–157. IEEE, 2021.
  * Harabagiu et al. [2000] Sanda M Harabagiu, Dan I Moldovan, Marius Pasca, Rada Mihalcea, Mihai Surdeanu, Razvan C Bunescu, Roxana Girju, Vasile Rus, and Paul Morarescu.  Falcon: Boosting knowledge for answer engines.  In _TREC_ , volume 9, pages 479–488, 2000.
  * Hubble [2018] Claire Hubble.  Miquela sousa and the rise of fake influencers.  _Eureka Street_ , 28(13):20–22, 2018.
  * Huebschman et al. [2003] Michael L Huebschman, Bala Munjuluri, and Harold R Garner.  Dynamic holographic 3-d image projection.  _Optics express_ , 11(5):437–445, 2003.
  * Loveys et al. [2022] Kate Loveys, Mark Sagar, Mark Billinghurst, Nastaran Saffaryazdi, and Elizabeth Broadbent.  Exploring empathy with digital humans.  In _2022 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW)_ , pages 233–237. IEEE, 2022.
  * Magnenat-Thalmann and Thalmann [2005] Nadia Magnenat-Thalmann and Daniel Thalmann.  Virtual humans:thirty years of research, what next?  _The Visual Computer_ , 21(12):997–1015, 2005\.
  * Mait et al. [2018] Joseph N Mait, Gary W Euliss, and Ravindra A Athale.  Computational imaging.  _Advances in Optics and Photonics_ , 10(2):409–483, 2018.
  * Marwick [2018] Alice E Marwick.  The algorithmic celebrity: The future of internet fame and microcelebrity studies.  _Microcelebrity around the globe_ , pages 161–169, 2018.
  * Morris and Kanade [2000] Daniel D Morris and Takeo Kanade.  Image-consistent surface triangulation.  In _Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000 (Cat. No. PR00662)_ , volume 1, pages 332–338. IEEE, 2000\.
  * Mystakidis [2022] Stylianos Mystakidis.  Metaverse.  _Encyclopedia_ , 2(1):486–497, 2022.
  * Noma et al. [2000] Tsukasa Noma, Liwei Zhao, and Norman I Badler.  Design of a virtual human presenter.  _IEEE Computer Graphics and Applications_ , 20(4):79–85, 2000.
  * Ratner [2012] Peter Ratner.  _3-D human modeling and animation_.  John Wiley & Sons, 2012.
  * Rizzo et al. [2011] Albert Rizzo, Belinda Lange, John G Buckwalter, Eric Forbell, Julia Kim, Kenji Sagae, Josh Williams, JoAnn Difede, Barbara O Rothbaum, Greg Reger, et al.  Simcoach: an intelligent virtual human system for providing healthcare information and support.  2011\.
  * Sarkar et al. [2015] Sanglap Sarkar, Venkateshwar Rao, SM Baala Mithra, and Subrahmanya VRK Rao.  Nlp algorithm based question and answering system.  In _Seventh International Conference on Computational Intelligence, Modeling and Simulation_ , 2015.
  * Schepers et al. [2018] Martin Schepers, Matteo Giuberti, Giovanni Bellusci, et al.  Xsens mvn: Consistent tracking of human motion using inertial sensing.  _Xsens Technol_ , 1(8), 2018.
  * Schuemie et al. [2001] Martijn J Schuemie, Peter Van Der Straaten, Merel Krijn, and Charles APG Van Der Mast.  Research on presence in virtual reality: A survey.  _CyberPsychology & Behavior_, 4(2):183–201, 2001.
  * Stephenson [2003] Neal Stephenson.  _Snow crash: A novel_.  Spectra, 2003.
  * Van der Merwe [2021] David Van der Merwe.  The metaverse as virtual heterotopia.  In _3rd world conference on research in social sciences_ , 2021.
  * Yu and Deng [2016] Dong Yu and Li Deng.  _Automatic speech recognition_ , volume 1.  Springer, 2016.
  * Zang et al. [2022] Zelin Zang, Shenghui Cheng, Linyan Lu, Hanchen Xia, Liangyu Li, Yaoting Sun, Yongjie Xu, Lei Shang, Baigui Sun, and Stan Z Li.  Evnet: An explainable deep network for dimension reduction.  _IEEE Transactions on Visualization and Computer Graphics_ , 2022\.
  * Zhang et al. [2022] Xinyu Zhang, Shenghui Cheng, and Klaus Mueller.  Graphical enhancements for effective exemplar identification in contextual data visualizations.  _IEEE Transactions on Visualization and Computer Graphics_ , 2022\.