11institutetext:  Oak Ridge National Laboratory
11email: {archbaldrk, dazevedoef, eisenbackm<EMAIL_ADDRESS>22institutetext:
Georgia Institute of Technology
22email<EMAIL_ADDRESS>33institutetext: University of Tennessee,
Knoxville
33email: {dongarra, flopez<EMAIL_ADDRESS>
33email: {rfebbo<EMAIL_ADDRESS>
33email<EMAIL_ADDRESS>
TOC Author

# Integrating Deep Learning in Domain Sciences at Exascale 111 This manuscript
has been authored by UT-Battelle, LLC, under contract DE-AC05-00OR22725 with
the US Department of Energy (DOE). The US government retains and the
publisher, by accepting the article for publication, acknowledges that the US
government retains a nonexclusive, paid-up, irrevocable, worldwide license to
publish or reproduce the published form of this manuscript, or allow others to
do so, for US government purposes. DOE will provide public access to these
results of federally sponsored research in accordance with the DOE Public
Access Plan (http://energy.gov/downloads/doe-public-access-plan).

Rick Archibald 11 Edmond Chow 22 Eduardo D’Azevedo 11 Jack Dongarra 3311
Markus Eisenbach 11 Rocco Febbo 33 Florent Lopez 33 Daniel Nichols 33
Stanimire Tomov 33 Kwai Wong 33 Junqi Yin 11

###### Abstract

This paper presents some of the current challenges in designing deep learning
artificial intelligence (AI) and integrating it with traditional high-
performance computing (HPC) simulations. We evaluate existing packages for
their ability to run deep learning models and applications on large-scale HPC
systems efficiently, identify challenges, and propose new asynchronous
parallelization and optimization techniques for current large-scale
heterogeneous systems and upcoming exascale systems. These developments, along
with existing HPC AI software capabilities, have been integrated into
MagmaDNN, an open-source HPC deep learning framework. Many deep learning
frameworks are targeted at data scientists and fall short in providing quality
integration into existing HPC workflows. This paper discusses the necessities
of an HPC deep learning framework and how those needs can be provided (e.g.,
as in MagmaDNN) through a deep integration with existing HPC libraries, such
as MAGMA and its modular memory management, MPI, CuBLAS, CuDNN, MKL, and HIP.
Advancements are also illustrated through the use of algorithmic enhancements
in reduced- and mixed-precision, as well as asynchronous optimization methods.
Finally, we present illustrations and potential solutions for enhancing
traditional compute- and data-intensive applications at ORNL and UTK with AI.
The approaches and future challenges are illustrated in materials science,
imaging, and climate applications.

##  1 Background

Deep learning (DL) has become one of the most prevalent technologies today.
Applications extend from image recognition to natural language processing.
While its commercial applications have garnered the efforts of major
technology companies, DL has proven vital to works in numerous scientific
domains.

Some of these domain problems are vital to solving many of today’s global
issues—for instance, in solving modern, data-dense problems in climate
science. This field has seen several advancements with the recent integration
of deep neural networks (DNNs). Ise et. al. [1] propose a modern approach
towards utilizing neural networks to predict global temperature data. They
further conclude that the accuracy of their model increases with the number of
input images used. Thus, the model’s accuracy and utility grow with the amount
of compute power available. Wang et. al. [2] provide further uses for DNNs in
weather forecasting, which compete with the current state-of-the-art,
physically based simulations. The neural networks can also reduce the
computations necessary, as they possess the ability to extend one trained
network to similar environments, or train once, and predict frequently.

Additionally, DL has proven useful in materials science, even leading the
field of materials informatics [3]. Feng et. al. [4] were able to utilize deep
networks to predict material defects with an accuracy above 98%, even for
small data sets. However, these great accuracies require deep and/or pre-
trained networks. As with climate modelling, deep learning can be used to
replace current physically based simulations, which are expensive and/or do
not scale well [5].

Providing high performance hardware and software stacks to facilitate deep
learning has become a focus in the high-performance computing (HPC) field. An
overview of the parallel algorithms and their challenges for distributed, deep
neural networks can be found in Ben-Nun and Toefler [6]. Large-scale modern
systems, such as the US DOE’s Summit, are fully loaded with high-performance
GPUs to support the compute-intensive nature of deep learning. Additionally,
some HPC distributed file systems under-perform in providing I/O for data-
hungry DL training [7]. Even in terms of sheer performance, many state-of-the-
art deep learning models and libraries fail to utilize a significant portion
of the maximum floating-point operations (FLOPs) rate of current top systems
[8]. There is significant research in the area of accelerating DL on cloud and
other commercial systems, but these methods may not integrate well with
varying HPC technologies and modern scientific workflows.

One major area in which HPC can innovate deep learning is model parallelism.
In many domain applications, networks can become large, leading to growth
beyond the memory of a single GPU. By utilizing existing HPC communication
standards, a robust model-parallel training standard needs to be developed in
addition to efficient implementations. Currently, several libraries attempt to
integrate model parallelism [9], but this requires extensive knowledge of the
code base, model, and hardware. There is an active research push into further
developing model parallel approaches [10, 11, 12, 13].

R. Stevens presented a vision of the landscape of machine learning as DOE
moves towards the exascale era [14], stressing the importance of machine
learning as an integral part of DOE exascale applications. The new paradigm of
”HPC + AI” on exascale computing leads towards an ”Integrated Sim, Data, Learn
Stack.” There has been a long history of application code development for
compute-intensive applications in the areas of climate, materials, medical,
high energy, and urban sciences applications. In light of these needs, we will
highlight developments and discuss the necessity of a set of tools and methods
particularly relevant to deploying AI on HPC systems.

Realizing the paradigm of “HPC + AI” requires three major components: (1) the
HPC system and investments in scaling application-driven capabilities; (2) the
AI software achieving the potential performance; and (3) the “+”, innovative
algorithms and implementations/tools that combine the components of compute-
and data-driven applications seamlessly on exascale machines.

##  2 Deep Learning Software on Modern HPC Systems

There are countless deep learning frameworks in existence today: TensorFlow
[15], PyTorch [16], MxNet [17], and many others. Each framework has its own
advantages and each falls short in some areas. One advantage of each of these
frameworks is their strong corporate backings. Companies pour significant
amounts of money into making production-ready libraries, which support various
hardware. However, when using these frameworks in an HPC setting, this can
cause issues. Much of the research and advancements by the library designers
are production focused and target cloud environments. Some of these
advancements translate well onto HPC systems, such as the use of hardware
accelerators and mixed precision. However, many communication paradigms and
implementations differ dramatically between the two. For example, the
parameter server method in TensorFlow is more suited for heterogeneous cloud
environments, although it is more bandwidth efficient than allreduce; and the
socket-based GLOO communication backend built into PyTorch cannot efficiently
utilize high-performance interconnects on HPC systems, although it is more
flexible.

###  2.1 Towards a Deep Learning Framework for HPC

We evaluated a number of existing packages for their readiness to efficiently
run deep learning models and applications on large-scale HPC systems. As
pointed out previously, we found various limitations. Our vision of what is
important and needed in deep leaning AI packages for HPC is incorporated into
the design of the MagmaDNN open source framework [18, 19] (see Figure 1) with
current release MagmaDNN v1.2 [20], which we discuss briefly in what follows.

Firstly, MagmaDNN targets HPC systems, rather than cloud systems. It is
written in C++ and is open source (available at bitbucket.org/icl/magmadnn).

Many of the core components of MagmaDNN are the same as other frameworks:
operations are represented in a compute graph, data is passed around via
tensors, gradients are computed using backpropagation, etc. However, each of
these was designed with an HPC environment in mind.

Figure 1: MagmaDNN software stack.

MagmaDNN is centered around existing high-performance software packages,
enabling easy integration into existing environments and Exascale Computing
Project (ECP) software products. First and foremost, MagmaDNN uses the Matrix
Algebra on GPU and Multicore Architectures (MAGMA) [21] package for
heterogeneous linear algebra, since deep learning is heavily dependent on
linear algebra routines. For strictly CPU computations, the package allows the
use of any Basic Linear Algebra Subprograms (BLAS)/LAPACK packages such as
Intel MKL, OpenBLAS, ATLAS, etc.

To better operate with hardware accelerators, MagmaDNN uses CUDA packages to
harness GPU acceleration. Custom CUDA kernels, along with CUDNN’s routines for
GPU-accelerated deep learning, allows the package to get maximal performance
on GPU-enabled systems.

For distributed computing, MagmaDNN relies on existing HPC tools such as the
Message Passing Interface (MPI) for its communication. Most systems have an
MPI distribution tuned to that hardware, while some even have custom
distributions. Relying on MPI for inter-node communication allows the
framework to utilize existing optimized implementations while providing
simpler integration with existing scientific codes.

This integration of existing HPC frameworks allows MagmaDNN to compile and
link with the fastest libraries for particular hardware. Utilizing existing
optimized packages is a consistent design pattern within HPC libraries, and
MagmaDNN uses interfaces that are commonly installed on HPC clusters. In terms
of integrating with existing systems and scientific codes, this approach
offers the package a significant advantage over other frameworks and
workflows.

Not only does MagmaDNN integrate with current HPC frameworks, it provides
modularity in its design and linking so that users have flexibility in use.
Conforming to standard APIs allows linking with any choice of framework; any
BLAS/LAPACK/MPI choice implementations can be used. However, it is not just
libraries with which MagmaDNN is modular. By utilizing computation graphs and
extendable C++ classes, the package allows users to fully customize their
neural networks. For instance, to utilize a unique network interface for
distributed training, one can define a custom node on the computation graph
which handles the transfer of data on that network component. This can be
further utilized to define any desired data movement or computational
elements. By providing modular components, MagmaDNN provides a structure onto
which domain scientists can easily implement desired custom functionality.

Much of this customizability is fully supported through MagmaDNN’s interfaces.
For instance, working with GPU and distributed memory is abstracted away in
the `MemoryManager` class. Thus, library users do not need to worry about
where memory is stored and how it is transferred. The package identifies the
memory type, and whenever a memory operation is used (e.g., copy, put, get,
etc.), and the MagmaDNN implementation handles the low-level specifics.
Likewise, communication operations between devices or nodes are abstracted
into the library as well. However, users can define their own operations,
meaning they can redefine those in existing code. This makes it easy, for
example, to try out different convolution algorithms and determine which one
performs optimally. Another common example is to determine which distribution
strategy is optimal for a specific machine. The same training code will run on
any machine, and only the communication code might need to be altered.

Looking towards next-generation hardware and exascale systems, MagmaDNN aims
to incorporate more technology stacks such as AMD’s HIP (e.g., hipMAGMA is
already available) and Intel’s OneAPI. Support for more technologies will
allow researchers at various labs and institutions to focus on domain
research—and not on how to integrate their code with HPC stacks. We also hope
to provide streamlined and optimal support for model parallelism. As the size
of models grows, so does the necessary memory requirements on devices. Soon
many cutting-edge networks, which can provide state-of-the-art accuracy in
classification, will require model parallelism. Supporting fast, scalable
model parallelism on HPC systems is crucial to facilitating next-generation
research, and MagmaDNN is moving towards this goal.

###  2.2 Workflow Software for Modern HPC Systems

Another major component in the integration of AI and HPC is the availability
of workflow software: which is critical, for example, in launching, collecting
results from, and analyzing various simulations in hyperparameter tuning, DL
networks discoveries, etc. Exascale applications largely have their own set of
workflow procedures and tools to launch their simulations. Integrating
traditional applications seamlessly with existing AI software—either
TensorFlow, PyTorch, or MagmaDNN—takes on a new level of challenges. Most
workflow frameworks available today focus on cloud deployment, yet a workflow
framework tailored to scale on HPC systems is critical to the success to
”Automate and Accelerate” the task of ”Sim, Data, and Learn Stack”. SWIFT-T
[22] is a tool that offers the choice to run on large-scale HPC systems. At
The University of Tennessee, Knoxville (UTK), we have developed a parallel
workflow framework called openDIEL that aims to give researchers and users of
HPC machines an efficient way to coordinate, organize, and interconnect many
disparate modules of computation in order to effectively utilize and allocate
HPC resources [23, 24]. It provides users an adaptive avenue to run compute-
intensive and data science jobs concurrently, allowing specification of DNN
architectures, data processing, and hyperparameter tuning. Existing ML tools
can be readily used in openDIEL, allowing for easy experimentation with
various models and approaches. Most importantly, openDIEL provides a platform
to run existing ECP applications in their own workflow procedures. When
conducting multi-discipline simulations, it is often required to use a large
variety of software, and serial or parallel codes, to answer a research
question. Utilizing disparate modules of computation requires the careful
coordination of dependencies, communication, and synchronization between them,
and there is not always a clear path. This problem is solved by openDIEL: it
enables researchers to define complex dependencies between modules and
schedule communication between them. OpenDIEL consists of three primary
components: the workflow control file, the communication library (COMMLIB),
and the Executive, shown in Figure 3. Workflow is defined by users via the
control file, which consists of two parts, the definition of functional
modules and the interactions and sequence of execution of modules. OpenDIEL
provides two communication libraries, one for direct point-to-point data
transfer between two modules, and a store-and-move tuple space library for
collective and asynchronous data transfers. The Executive is a lightweight
manager designed to run and scale on heterogeneous HPC platforms.

Figure 2: OpenDIEL Structure.

Figure 3: Workflow in Materials Sciences.

OpenDIEL specifically focuses on unifying modules into a single executable
that uses MPI for communication. Climate and materials sciences simulations,
with typical workflow shown in Figure 3, are two of the applications that are
incorporating deep learning into their simulations.

##  3 Algorithmic Improvements for DNN AI in HPC

In this section, we select and highlight the development of two technologies
that we believe are critical for the scalability and efficient use of HPC
systems. The first (Section 3.1) addresses scalability of DL training,
stressing the benefits and need for asynchronous methods on today’s highly
parallel and heterogeneous systems, featuring powerful nodes and relatively
slow interconnects. The second (Section 3.2) addresses hardware changes
motivated by DL—in particular powerful hardware acceleration for low-precision
arithmetic—and techniques for harnessing this in scientific HPC applications.

###  3.1 Asynchronous Methods

In a multi-core or many-core shared-memory environment, the speed at which the
parameters can be updated by different threads may be a limiting factor for
stochastic gradient descent (SGD) performance. This is because threads must
lock the parameter vector while they are updating it, preventing other threads
from reading or updating the parameters, so that they must wait. Asynchronous
versions of SGD may relax the data dependencies and order of operations
compared to the classical SGD algorithm, allowing the algorithm to update
parameters more rapidly—but also making the asynchronous algorithm
mathematically different and non-deterministic in its execution [25].

In a distributed-memory environment, a common implementation option for SGD is
for each compute node to store a copy of the model parameters [26]. When each
compute node has computed a gradient for its mini batch of data, all the
gradients must be summed and then shared among the compute nodes so that they
can update their copies of the model parameters. This leads to a potential
bottleneck in the global all-reduce computation at every step of SGD. In
addition, nodes that compute their gradients faster than others must wait for
the other nodes (sometimes called “stragglers”). This can be a very high
synchronization overhead that may be difficult to reduce by load balancing on
heterogeneous architectures. An asynchronous approach can address these high
overheads, as we demonstrate below.

Another common implementation option for distributed-memory computers is the
parameter server approach [27]. Here, one or more compute nodes are dedicated
to storing and updating the model parameters. The worker nodes compute
gradients for their partition of the data and send these gradients to the
parameter server. The parameter server then sends back the updated values of
the model parameters. The parameter server model can be used to run SGD
synchronously—with the same issues of stragglers slowing down the iteration
for all nodes—but it has the advantage that it can also be run asynchronously
(i.e., the worker nodes do not have to send gradients at the same time).
Workers, however, may be working with different sets of values for the
parameters at any instant. This makes the computation asynchronous. In
distributed memory implementations, the gradients are typically updated by the
parameter server using atomic memory operations, which differs from the shared
memory case.

Asynchronous parameter server implementations were proposed around 2012 or
earlier, but it was Hogwild! [25], with inconsistent reads and writes of the
parameter vector in shared memory, that popularized the asynchronous approach.
In particular, Hogwild! showed that, at least in cases of convex problems and
gradient sparsity, asynchrony with inconsistent reads and writes does not have
to harm convergence asymptotically. Hogwild! was extended in many directions,
such as HogBatch [28], designed for efficiently running on multi-core CPU
architectures, and Buckwild [29] which exploits low-precision arithmetic to
accelerate the training. Improvements on the analysis of Hogwild! showed that
convergence can be maintained for the non-convex and non-sparse case [30]. In
MagmaDNN, we designed a parallel asynchronous variant of SGD that is similar
to a parameter server approach, but on multi-core CPU and GPU architectures
[31].

In practice, the accuracy of machine learning models can be dramatically
improved by increasing the number of model parameters as well as the sizes of
the training sets. As a result, the memory footprint needed for model training
often exceeds the memory storage available on a single node. In such cases,
models must be trained on large-scale machines where the dataset, and possibly
the model as well, are distributed among the compute nodes. From a performance
perspective, it is crucial to efficiently parallelize the training in this
setup—and this means overcoming the communication costs. It has been shown
that when targeting large-scale platforms, asynchronous algorithms can
outperform the synchronous ones as in the shared-memory case. In Google’s
DistBelief framework [27], for example, the Downpour SGD algorithm uses a
centralized parameter server for storing the model, and distributes the
dataset to other participating nodes. Asynchrony enables communications to
overlap with computations, thus improving resource usage. In addition, the
algorithm is particularly well adapted to heterogeneous computing environments
where resources may have different processing and communication speeds.

In MagmaDNN, we take a different approach, exploiting remote memory access
(RMA) using MPI one-sided communication capabilities for implementing
asynchronous SGD in order to maximize the scalability of parallel training.
This approach has already been proven efficient in implementing several
asynchronous numerical algorithms such as the Jacobi [32] or optimized Schwarz
[33] methods for solving sparse systems of linear equations.

###  3.2 Reduced and Mixed Precision

Deep neural networks can be efficiently trained using low-precision floating-
point arithmetic, such as single (fp32) or lower precision, improving the
training times on modern hardware [34]. Popular floating-point formats include
the 16-bit IEEE half-precision format (fp16) and the 16-bit bfloat16 [35]
format. With fewer bits in the mantissa compared to fp16, the bfloat16 format
offers less precision but has the advantage of having the same range as
fp32—thereby removing the risk of overflow and underflow when converting from
fp32 data.

Motivated by the success of low-precision arithmetic in machine learning
applications, many specialized hardware accelerators have been designed with
reduced- or mixed-precision arithmetic capabilities. For example, the NVIDIA
Tensor Cores introduced in the Volta GPU can issue $4\times 4$ matrix
multiplication $D=C+A*B$ in one clock cycle, where the input matrices $A$ and
$B$ are fp16, whereas $D$ and $C$ can be either fp16 or fp32. The theoretical
performance peak on Volta GPU is 125 teraFLOP/s, which is 8 times the
performance peak for fp32; and, although the accuracy of tensor core
operations is lower than fp32, the ability to accumulate the result of matrix
multiplication in fp32 rather than fp16 yields a considerable gain with
respect to accuracy over fp16 [36]. In [37], authors use Tensor Cores for
computing a reduced-precision LU factorization of a dense linear system,
initially generated in double precision (fp64), and use these factors as a
preconditioner for a Krylov solver in order to retrieve an fp64 solution. By
doing this, they manage to reduce the time to solution up to $4\times$ fp64
arithmetic for the factorization.

Tensor Cores can also be used in performing FFTs [38]. Note that a matrix in
fp32 can be well approximated as the scaled sum of two fp16 matrices

$A_{32}\approx a1_{32}*A1_{16}+a2_{32}*A2_{16}$ (1)

where $a1_{32}$ and $a2_{32}$ are in fp32. Scaling by $a1_{32}(a2_{32})$
ensures $A1_{16}(A2_{16})$ is within the limited dynamic range of fp16.
Conceptually, $A1_{16}$ roughly captures the 3 most significant decimal
digits, and $A2_{16}$ the next 3 lower significant decimal digits. If $A_{32}$
is already well scaled, then one can expect $a1_{32}=1$ and
$a2_{32}=2^{-11}*a1_{32}$. Matrix multiplication of two fp32 matrices can be
approximated by 3 matrix multiplications of fp16 matrices on Tensor Cores,
which may give a theoretical speedup of 8/3$\times$ on Volta GPU. For another
perspective, equation (1) suggests the form of operations in DNN are
theoretically able to well approximate operations of fp32 using mixed-
precision operations in fp16.

##  4 Applications

###  4.1 Materials Science and Microscopy

There are multiple opportunities to exploit machine learning techniques in
materials science. While many applications have concentrated on materials
discovery—as exemplified by the use of ML in conjunction with databases such
as The Materials Project [39], AFLOW [40], or OQMD [41]—a tightly coupled
integration of traditional simulation techniques has great promise for
bridging the accuracy and computational cost divide between first principles
calculations and effective models. One promising combination for a tight
coupling is for first principles statistical mechanics of materials to
calculate the temperature dependence of materials. This requires the
calculations of many possible atomic configurations within these materials
using a Monte Carlo approach, where the probability of the individual states
would be evaluated using an expensive density functional theory calculation
[42]. Thus, we will utilize ML to construct surrogate models as an
intermediate step to link the first principles calculations to the Monte Carlo
simulation. Here, the calculation of ordering phase transitions in alloys
might serve as an example [43]. In a solid solution alloy, different chemical
species can occupy the sites of an underlying lattice structure (note that the
total number of states grows exponentially with the number of atoms in the
system). Each of these possible configurations has a different energy that
determines the probability of the system being found in this state at a given
temperature. To build a model for these interactions, density functional
calculations for representative configurations O(1,000–10,000) will be
performed to train a surrogate model that can replace the expensive (in the
order of multiple node hours per data point) first principles calculation
within the Monte Carlo sampling of possible configurations. While this
approach can be conducted in a linear workflow,
$\textrm{DFT}\rightarrow\textrm{ML}\rightarrow\textrm{MC}$, we envision a
tighter coupled workflow, which augments the original training set with new
points from important regions of the phase space discovered during the MC
simulation, and retrains the model to improve the quantitative predictive
power of this approach.

A long-standing inverse problem in atomic imaging is the loss of phase
information during measurement (a.k.a., the phase problem). Given the sparse
data collection of scanning transmission electron microscopic (STEM) images on
different types of materials, a comprehensive database is needed for the
community to study the subject. State-of-the-art electron microscopes produce
focused electron beams with atomic dimensions and capture of convergent beam
electron diffraction (CBED) patterns. In this dataset [44], we use newly
developed electron scattering simulation codes to generate CBED patterns from
over 60,000 materials (solid-state materials) from a material project
database, representing nearly every known crystal structure. A data sample
from this data set is given by a 3D array formed by stacking 3 CBED patterns
simulated from the same material at 3 distinct material projections (i.e.,
crystallographic orientations). Associated with each data sample in the data
set is a host of material attributes or properties which are, in principle,
retrievable via analysis of this CBED stack. These consists of the crystal
space group to which the material belongs, atomic lattice constants and
angles, and chemical composition, to name but a few. Of note is the crystal
space group attribute (or label).

This dataset could be augmented with experimental data in the future. The
generated dataset, based on simulations, will emphasize the scalability aspect
of the model with the use of very large images ($\sim$ 10GB per image).

###  4.2 Super-Resolution for HPC Simulations

Image compression is a very active field of study, with new methods being
constantly generated [45]. The need for improvements in image compression
quality is growing in the field of HPC simulations because of the exponential
trend in data generation. There exists an untapped potential in this situation
due to the nature of simulated data that is not currently exploited.
Simulation data from numerical systems of partial differential equations exist
on a solution manifold [46]. Thus, the manifold hypothesis in machine
learning—which states that real-world, high-dimensional data lie on low-
dimensional manifolds embedded within the high-dimensional space—is concrete
for simulation data. We can therefore expect that identifying this map to the
low-dimensional manifold will provide ideal compression for HPC Simulations.
In the next paragraph we describe the basic setup that allows researchers to
test in situ machine learning for climate simulations.

The shallow water equations on a sphere is a well-known model used in climate
science to simulate basic dynamics of the Earth’s atmosphere. There exist many
test case scenarios for this model [47, 48, 49, 50], and in this paper we use
the challenging test known as the barotropic instability test [50], which
involves perturbing the atmospheric flow with a localized bump to the balanced
height field. We simulate the shallow water equations on the sphere by using a
fourth-order Runge-Kutta method [51] for time-stepping and a discontinuous
Galerkin (DG) spatial discretization with a Legendre basis on a cubed-sphere
mesh [52, 53].

The data structure for this simulation consists of a time series of images for
each of the six faces of the cube, which for analysis can be converted to a
single series lat-lon gridded images. Arguably, the most commonly used form of
lossy compression is the discrete cosine transform (DCT) method [54]. ECP has
two projects on the development of FFTs and related transformations, one of
which is heFFTe [55], already integrated in the ECP software stack, delivering
high-performance (close to 90% of the theoretical roofline peak), and very
good weak and strong scaling on systems like Summit at ORNL. This common
compression method fills out the setup of the in situ machine learning super
resolution method. At each time step of the barotropic instability simulation,
in situ machine learning methods are exposed to both the compressed image data
and the original image. As the simulation progresses, only the compressed
image data is stored to disk, and the machine learning method adaptively
learns correct super-resolution transformation of the compressed data. Final
stored data contain all compressed image data with the trained machine
learning method. It has been demonstrated in [56] that, using this setup, it
is possible to train in situ networks to reduce the error in lossy
compression—obtaining multiple orders of improvement. Going forward, it will
be necessary to further improve in situ compression and analysis in order to
maximize discovery with HPC simulations.

##  5 Meeting Exascale

Exascale systems will appear in 2021. These machines incorporate accelerators
from NVIDIA, AMD, and Intel. Efforts in preparing the software stack for
exascale systems have been a major focus in the DOE ECP program [57]. We
believe a native DNN framework such as MagmaDNN will pave a unique path to
meet the challenges of exascale platforms. Assisted by the openDIEL parallel
workflow engine, which admits a diverse spectrum of applications as functional
modules, we will be able to exploit the full capabilities of exascale
machines. In-situ data augmentation will be incorporated with compute-
intensive simulations, leading to discovery in multi- and inter-disciplinary,
systems-wide, real-time recommendation systems. From instrumentation
calibration, experimental and observable results, theoretical simulations, to
validation and analysis—exascale computing will be brought to bear on health,
transportation, environment, and social sciences. For example, climate and
weather recommendation systems will be able to integrate models in storms
prediction, rainfall, vegetation, farm growth, pollution, power usage, traffic
flow, etc. [58, 59, 60, 61, 62]. As inputs and outputs from many sensor
devices become ubiquitous, the importance of a scalable AI framework and an
extensible workflow tool will grow.

Challenges will rise from algorithmic approaches as HPC systems continue to
expand and evolve. Multiple-precision implementation will be unavoidable.
Setting the basis and preparing for product diversity from different vendors
will be an important consideration of an AI framework, as well. One challenge
in performance is to reduce the impact of communication while maintaining good
convergence of the algorithm, such as SGD, in the DNN framework. In the
Horovod framework [26], for example, the solution is to distribute the
training of neural network on large machines using an efficient gradient
reduction strategy referred to as ring-allreduce. Notably, in the 2018 Gordon-
Bell award winning paper [63], Horovod was used in the context of TensorFlow
to train two DNN models on large systems, including the Summit machine, to
detect extreme weather patterns from climate data. Similarly, Horovod was used
in [64] to train a large DNN model for tackling inverse problems in imaging.
Algorithmic improvements towards individual or combined synchronous,
asynchronous, and pipeline approaches are essential to improve resource usage
as well as convergence.

There is a trend in increasingly larg model sizes (especially in NLP—the
latest model [65] has 17 Billion parameters), and a need in scientific
applications (e.g., geospatial imaging) to process larger-dimension inputs.
Although there exist exascale deep learning applications [63, 66] on pre-
exascale system such as Summit thanks to the Tensor Core technology, those use
cases push the limit on large-batch training for data parallelism and are not
generally applicable to exascale learning challenges. Early efforts [9, 67] on
model parallelism have made progress, but are not yet mature or generic. An AI
framework that can efficiently exploit various level of parallelisms (data
parallel, model parallel, hyperparameter search, etc) will be in demand.

##  6 Conclusion

Exascale systems are an important investment in the US. With exascale, we
envision a stronger economy and improved quality of life. It will also lead to
important scientific discovery and resolution of complex issues related to
national security. Development in AI software and tools that scale well on
these systems is important, and even more critical for AI frameworks that also
work well across the existing spectrum of exascale applications. Although many
challenges exist, a primary roadblock is the lack of direct collaborative
effort, and a software platform that values performance as the foremost
priority. In this paper, we present a unique set of AI tools and algorithms,
as well as efforts between collegiate and ORNL researchers, demonstrating that
there is a pathway to integrate and deploy machine learning to those ends—with
emphasis on two major applications on exascale systems.

##  7 Acknowledgments

This material is based upon work supported in part by the Laboratory Directed
Research and Development program at the Oak Ridge National Laboratory, which
is operated by UT-Battelle, LLC., for the U.S. Department of Energy under
Contract DE-AC05-00OR22725. The work was partly supported by the Scientific
Discovery through Advanced Computing (SciDAC) program funded by U.S.
Department of Energy, Office of Science, Advanced Scientific Computing
Research, with specific thanks to the FASTMath Institutes.

This work was conducted at the Joint Institute for Computational Sciences
(JICS) and the Innovative Computing Laboratory (ICL), sponsored by the
National Science Foundation (NSF), through NSF REU Award #1659502 and NSF
Award #1709069. This work used hardware donations from NVIDIA as well as the
Extreme Science and Engineering Discovery Environment (XSEDE), which is
supported by NSF grant number ACI-1548562. Computational Resources are
available through a XSEDE education allocation award TG-ASC170031.

## References

  * [1] T. Ise and Y. Oba, “Forecasting Climatic Trends Using Neural Networks: An Experimental Study Using Global Historical Data,” Frontiers in Robotics and AI, vol. 6, p. 32, 2019.
  * [2] J. Wang, P. Balaprakash, and R. Kotamarthi, “Fast Domain-Aware Neural Network Emulation of a Planetary Boundary Layer Parameterization in a Numerical Weather Forecast Model,” Geoscientific Model Development, vol. 12, no. 10, pp. 4261–4274, 2019.
  * [3] A. Agrawal and A. Choudhary, “Deep materials Informatics: Applications of Deep Learning in Materials Science,” MRS Communications, vol. 9, no. 3, p. 779–792, 2019.
  * [4] S. Feng, H. Zhou, and H. Dong, “Using Deep Neural Network with Small Dataset to Predict Material Defects,” Materials & Design, vol. 162, p. 300–310, 2019.  Citation Key: FENG2019300.
  * [5] W. Ye, C. Chen, Z. Wang, I.-H. Chu, and S. P. Ong, “Deep Neural Networks for Accurate Predictions of Crystal Stability,” Nature Communications, vol. 9, p. 3800, Sep 2018.
  * [6] T. Ben-Nun and T. Hoefler, “Demystifying Parallel and Distributed Deep Learning: an Indepth Concurrency Analysis,” ACM Computing Survey, vol. 1, pp. 1–37, January 2019.
  * [7] J. Han, L. Xu, M. M. Rafique, A. R. Butt, and S. Lim, “A quantitative study of deep learning training on heterogeneous supercomputers,” in 2019 IEEE International Conference on Cluster Computing (CLUSTER), pp. 1–12, 2019.
  * [8] Y. You, Z. Zhang, C.-J. Hsieh, J. Demmel, and K. Keutzer, “ImageNet Training in Minutes,” in Proceedings of the 47th International Conference on Parallel Processing, ICPP 2018, (New York, NY, USA), Association for Computing Machinery, 2018.
  * [9] N. Shazeer, Y. Cheng, N. Parmar, D. Tran, A. Vaswani, P. Koanantakool, P. Hawkins, H. Lee, M. Hong, C. Young, R. Sepassi, and B. Hechtman, “Mesh-TensorFlow: Deep Learning for Supercomputers,” in Advances in Neural Information Processing Systems 31 (S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, eds.), pp. 10414–10423, Curran Associates, Inc., 2018.
  * [10] J. Geng, D. Li, and S. Wang, “ElasticPipe: An Efficient and Dynamic Model-Parallel Solution to DNN Training,” in Proceedings of the 10th Workshop on Scientific Cloud Computing, ScienceCloud ’19, (New York, NY, USA), p. 5–9, Association for Computing Machinery, 2019.
  * [11] Y. e. a. Huang, “GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism,” in Advances in neural information processing systems 32, p. 103–112, Curran Associates, Inc., 2019.  Citation Key: NIPS2019_8305.
  * [12] A. Gholami, A. Azad, P. Jin, K. Keutzer, and A. Buluc, “Integrated Model, Batch, and Domain Parallelism in Training Neural Networks,” in Proceedings of the 30th on Symposium on Parallelism in Algorithms and Architectures, SPAA ’18, (New York, NY, USA), p. 77–86, Association for Computing Machinery, 2018.
  * [13] C.-C. Chen, C.-L. Yang, and H.-Y. Cheng, “Efficient and Robust Parallel DNN Training through Model Parallelism on Multi-GPU Platform,” ArXiv, vol. abs/1809.02839, 2018.
  * [14] R. Stevens, “Exascale Computing : The Coming Integration of Simulation, Data and Machine Learning,” 2018.  European Technology Platform for High-Performance Computing (ETP4HPC) ISC’18 Workshop.
  * [15] M. A. et. al., “Tensorflow: Large-Scale Machine Learning on Heterogeneous Distributed Systems,” CoRR, vol. abs/1603.04467, 2016\.
  * [16] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, and et al., “PyTorch: An Imperative Style, High-Performance Deep Learning Library,” in Advances in neural information processing systems 32, p. 8026–8037, Curran Associates, Inc., 2019\.  Citation Key: NIPS2019_9015.
  * [17] T. Chen, M. Li, Y. Li, M. Lin, N. Wang, M. Wang, T. Xiao, B. Xu, C. Zhang, and Z. Zhang, “MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems,” CoRR, vol. abs/1512.01274, 2015.
  * [18] D. Nichols, K. Wong, S. Tomov, L. Ng, S. Chen, and A. Gessinger, “MagmaDNN: Accelerated Deep Learning Using MAGMA,” in Proceedings of the Practice and Experience in Advanced Research Computing on Rise of the Machines (Learning), PEARC ’19, (New York, NY, USA), Association for Computing Machinery, 2019.
  * [19] D. Nichols, N.-S. Tomov, F. Betancourt, S. Tomov, K. Wong, and J. Dongarra, “MagmaDNN: Towards High-Performance Data Analytics and Machine Learning for Data-Driven Scientific Computing,” in ISC High Performance, (Frankfurt, Germany), Springer International Publishing, Springer International Publishing, 2019-06 2019.
  * [20] D. Nichols, R. Febbo, F. Lopez, K. Wong, S. Tomov, and J. Dongarra, “MagmaDNN (Version 1.2),” July 2020.  10.5281/zenodo.3972406.
  * [21] S. Tomov, J. Dongarra, and M. Baboulin, “Towards Dense Linear Algebra for Hybrid GPU Accelerated Manycore Systems,” Parallel Computing, vol. 36, pp. 232–240, June 2010.
  * [22] J. Ozik, N. Collier, J. Wozniak, and C. Spagnuolo, “From Desktop to Large-Scale Model Exploration with Swift/T,” in Proceedings of the 2016 Winter Simulation Conference, 2016.
  * [23] K. Wong and Z. Trzil, “Tuple Space Implementation in A Parallel Workflow Engine,OpenDIEL,” in Student Paper,PEARC ’18, 2018.
  * [24] F. Betancourt, K. Wong, E. Asemota, Q. Marshall, D. Nichols, and S. Tomov, “openDIEL: A Parallel Workflow Engine and Data Analytics Framework,” in Proceedings of the Practice and Experience in Advanced Research Computing on Rise of the Machines Learning, Student Paper,PEARC ’19, (New York, NY, USA), Association for Computing Machinery, 2019.
  * [25] F. Niu, B. Recht, C. Re, and S. J. Wright, “HOGWILD! A Lock-Free Approach to Parallelizing Stochastic Gradient Descent,” in Proceedings of the 24th International Conference on Neural Information Processing Systems, NIPS’11, (Red Hook, NY, USA), p. 693–701, Curran Associates Inc., 2011.
  * [26] A. Sergeev and M. D. Balso, “Horovod: Fast and Easy Distributed Deep Learning in TensorFlow,” arXiv preprint arXiv:1802.05799, 2018.
  * [27] J. Dean, G. Corrado, R. Monga, K. Chen, M. Devin, M. Mao, M. aurelio Ranzato, A. Senior, P. Tucker, K. Yang, Q. V. Le, and A. Y. Ng, “Large Scale Distributed Deep Networks,” in Advances in Neural Information Processing Systems 25 (F. Pereira, C. J. C. Burges, L. Bottou, and K. Q. Weinberger, eds.), pp. 1223–1231, Curran Associates, Inc., 2012.
  * [28] S. Sallinen, N. Satish, M. Smelyanskiy, S. S. Sury, and C. Ré, “High performance parallel stochastic gradient descent in shared memory.,” in IPDPS, pp. 873–882, IEEE Computer Society, 2016.
  * [29] C. D. Sa, C. Zhang, K. Olukotun, and C. Ré, “Taming the Wild: A Unified Analysis of HOG WILD! -Style Algorithms,” in Proceedings of the 28th International Conference on Neural Information Processing Systems - Volume 2, NIPS’15, (Cambridge, MA, USA), p. 2674–2682, MIT Press, 2015.
  * [30] X. Lian, Y. Huang, Y. Li, and J. Liu, “Asynchronous Parallel Stochastic Gradient for Nonconvex Optimization,” in Advances in Neural Information Processing Systems 28 (C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett, eds.), pp. 2737–2745, Curran Associates, Inc., 2015\.
  * [31] F. Lopez, E. Chow, S. Tomov, and J. Dongarra, “Asynchronous SGD for DNN Training on Shared-Memory Parallel Architectures,” Tech. Rep. ICL-UT-20-04 (To appear in IPDPSW’20 proceedings), Innovative Computing Laboratory, University of Tennessee, 2020-03 2020.
  * [32] J. Wolfson-Pou and E. Chow, “Modeling the Asynchronous Jacobi Method without Communication Delays,” Journal of Parallel and Distributed Computing, vol. 128, 6 2019.
  * [33] I. Yamazaki, E. Chow, A. Bouteiller, and J. Dongarra, “Performance of Asynchronous Optimized Schwarz with One-Sided Communication,” Parallel Computing, vol. 86, pp. 66 – 81, 2019.
  * [34] M. Courbariaux, Y. Bengio, and J.-P. David, “Training Deep neural Networks with Low Precision Multiplications,” 2014.
  * [35] Intel Corporation, “BFLOAT16—Hardware Numerics Definition,” Nov. 2018\.  White paper. Document number 338302-001US.
  * [36] P. Blanchard, N. J. Higham, F. Lopez, T. Mary, and S. Pranesh, “Mixed Precision Block Fused Multiply-Add: Error Analysis and Application to GPU Tensor Cores,” SIAM Journal on Scientific Computing, vol. 42, no. 3, pp. C124–C141, 2020.
  * [37] A. Haidar, S. Tomov, J. Dongarra, and N. J. Higham, “Harnessing GPU tensor Cores for Fast FP16 Arithmetic to Speed up Mixed-Precision Iterative Refinement Solvers,” in Proc. of the Intl. Conf. for High Performance Computing, Networking, Storage, and Analysis, pp. 1–11, 11 2018.
  * [38] A. Sorna, X. Cheng, E. D’Azevedo, K. Wong, and S. Tomov, “Optimizing the Fast Fourier Transform Using Mixed Precision on Tensor Core Hardware,” in 2018 IEEE 25th International Conference on High Performance Computing Workshops (HiPCW), pp. 3–7, 2018.
  * [39] A. Jain, S. P. Ong, G. Hautier, W. Chen, W. D. Richards, S. Dacek, S. Cholia, D. Gunter, D. Skinner, G. Ceder, and K. A. Persson, “Commentary: The materials project: A Materials Genome Approach to Accelerating Materials Innovation,” APL Materials, vol. 1, no. 1, p. 011002, 2013\.
  * [40] E. Gossett, C. Toher, C. Oses, O. Isayev, F. Legrain, F. Rose, E. Zurek, J. Carrete, N. Mingo, A. Tropsha, and S. Curtarolo, “Aflow-ml: A restful api for Machine-Learning Predictions of Materials Properties,” Computational Materials Science, vol. 152, pp. 134 – 145, 2018.
  * [41] S. Kirklin, J. E. Saal, B. Meredig, A. Thompson, J. W. Doak, M. Aykol, S. Rühl, and C. Wolverton, “The Open Quantum Materials Database (OQMD): assessing the accuracy of DFT formation energies,” npj Comput Mater, vol. 1, p. 15010, 2015.
  * [42] M. Eisenbach, C.-G. Zhou, D. M. Nicholson, G. Brown, J. Larkin, and T. C. Schulthess, “A Scalable Method for Ab Initio Computation of Free Energies in Nanoscale Systems,” in Proceedings of the Conference on High Performance Computing Networking, Storage and Analysis, SC ’09, (New York, NY, USA), pp. 64:1–64:8, ACM, 2009.
  * [43] M. Eisenbach, Z. Pei, and X. Liu, “First-Principles Study of Order-Disorder Transitions in Multicomponent Solid-Solution Alloys,” Journal of Physics: Condensed Matter, vol. 31, p. 273002, apr 2019.
  * [44] N. Laanait, A. Borisevich, and J. Yin, “A Database of Convergent Beam Electron Diffraction Patterns for Machine Learning of the Structural Properties of Materials,” 10.13139/OLCF/1510313, 5 2019.
  * [45] K. Sayood, Introduction to Data Compression.  The Morgan Kaufmann Series in Multimedia Information and Systems, Elsevier Science, 2017.
  * [46] W. C. Rheinboldt, “On the Computation of Multi-Dimensional Solution Manifolds of Parametrized Equations,” Numerische Mathematik, vol. 53, no. 1, pp. 165–181, 1988.
  * [47] D. Williamson, J. Drake, J. Hack, R. Jakob, and P. Swarztrauber, “A Standard Test Set for Numerical Approximations to the Shallow Water Equations in Spherical Geometry,” Journal of Computational Physics, vol. 102, pp. 211–224, September 1992.
  * [48] R. D. Nair and C. Jablonowski, “Moving Vortices on the Sphere: A Test Case for Horizontal Advection Problems,” Monthly Weather Review, vol. 136, no. 2, pp. 699–711, 2008.
  * [49] McDonald and Bates, “Semi-lagrangian Integration of a Shallow Water Model on the Sphere,” Monthly Weather Review, vol. 117, p. 130, 1989\.
  * [50] J. Galewsky, R. Scott, and L. Polvani, “An Initial-Value Problem for Testing Numerical Models of the Global Shallow-Water Equations,” Tellus, vol. 56A, pp. 429–440, 2004.
  * [51] M. Abramowitz and I. Stegun, eds., Handbook of Mathematical Functions, ch. 9.  Dover Publications, 1972.
  * [52] R. Sadourny, “Conservative Finite-Difference Approximations of the Primitive Equations on Quasi-Uniform spherical grids,” Monthly Weather Review, vol. 100, no. 2, pp. 136–144, 1972.
  * [53] R. Nair, S. Thomas, and R. Loft, “A Discontinuous Galerkin Global Shallow Water Model,” Monthly Weather Review, vol. 133, no. 4, pp. 876–888, 2005.
  * [54] N. Ahmed, T. Natarajan, and K. R. Rao, “Discrete Cosine Transform,” IEEE Transactions on Computers, vol. C-23, no. 1, pp. 90–93, 1974.
  * [55] S. Tomov, A. Ayala, A. Haidar, and J. Dongarra, “FFT-ECP API and High-Performance Library Prototype for 2-D and 3-D FFTs on Large-Scale Heterogeneous Systems with GPUs,” ECP WBS 2.3.3.13 Milestone Report FFT-ECP STML13-27, Innovative Computing Laboratory, University of Tennessee, 2020-01 2020\.  revision 01-2020.
  * [56] S. Lee, Q. Kang, S. Madireddy, P. Balaprakash, A. Agrawal, A. Choudhary, R. Archibald, and W. Liao, “Improving Scalability of Parallel CNN Training by Adjusting Mini-Batch Size at Run-Time,” in 2019 IEEE International Conference on Big Data (Big Data), pp. 830–839, 2019\.
  * [57] “Exascale Computing Project.”  https://www.exascaleproject.org.
  * [58] K. Zhou, Y. Zheng, B. W. D. Li, and X. Zhang, “Forecasting Different Types of Convective Weather: A Deep Learning Approach,” Journal of Meteorological Research, vol. 33, 2019.
  * [59] S. Samsi, C. Mattioli, and V. Mark, “Distributed Deep Learning for Precipitation Nowcasting,” in 2019 IEEE High Performance Extreme Computing Conference (HPEC), 2019.
  * [60] M. Keaney and T. Neal, “Comparing Deep Neural Network and Econometric Approaches to Predicting the Impact of Climate Change on Agricultural Yield,” UNSW Economics Working Paper, 2020-02.
  * [61] X. Xiuwen Yi, J. Zhang, Z. Wang, T. Li, and Y. Zheng, “Deep Distributed Fusion Network for Air Quality Prediction,” in Proceedings of KDD’18, London, United Kingdom, 2018.
  * [62] K. Chen, K. Chen, Q. Wang, Z. Ziyu He, Hu, and J. He, “Short-term Load Forecasting with Deep Residual Networks,” ArXiv, vol. abs/1805.11956v1, 2018.
  * [63] T. Kurth, S. Treichler, J. Romero, M. Mudigonda, N. Luehr, E. Phillips, A. Mahesh, M. Matheson, J. Deslippe, M. Fatica, Prabhat, and M. Houston, “Exascale Deep Learning for Climate Analytics,” in Proceedings of the International Conference for High Performance Computing, Networking, Storage, and Analysis, SC ’18, IEEE Press, 2018.
  * [64] N. Laanait, J. Romero, J. Yin, M. T. Young, S. Treichler, V. Starchenko, A. Borisevich, A. Sergeev, and M. Matheson, “Exascale Deep Learning for Scientific Inverse Problems,” 2019.
  * [65] S. Rajbhandari, J. Rasley, O. Ruwase, and Y. He, “ZeRO: Memory Optimization Towards Training A Trillion Parameter Models,” ArXiv, vol. abs/1910.02054, 2019.
  * [66] N. Laanait, J. Romero, J. Yin, M. T. Young, S. Treichler, V. Starchenko, A. Y. Borisevich, A. Sergeev, and M. Matheson, “Exascale Deep Learning for Scientific Inverse Problems,” ArXiv, vol. abs/1909.11150, 2019.
  * [67] Y. Huang, Y. Cheng, D. Chen, H. Lee, J. Ngiam, Q. V. Le, and Z. Chen, “GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism,” ArXiv, vol. abs/1811.06965, 2018.


# Semantic Information Market For The Metaverse: An Auction Based Approach

Lotfi Ismail1, Dusit Niyato1, Sumei Sun2, Dong In Kim3, Melike Erol-Kantarci4,
and Chunyan Miao1  1School of Computer Science and Engineering, Nanyang
Technological University, Singapore  2Institute for Infocomm Research, A*STAR,
Singapore  3Department of Electrical and Computer Engineering, Sungkyunkwan
University, Suwon, South Korea  4School of Electrical Engineering and Computer
Science, University of Ottawa, Ontario, Canada

###### Abstract

In this paper, we address the networking and communications problems of
creating a digital copy in the Metaverse digital twin. Specifically, a virtual
service provider (VSP) which is responsible for creating and rendering the
Metaverse, is required to use the data collected by IoT devices to create the
virtual copy of the physical world. However, due to the huge volume of the
collected data by IoT devices (e.g., images and videos) and the limited
bandwidth, the VSP might become unable to retrieve all the required data from
the physical world. Furthermore, the Metaverse needs fast replication (e.g.,
rendering) of the digital copy adding more restrictions on the data
transmission delay. To solve the aforementioned challenges, we propose to
equip the IoT devices with semantic information extraction algorithms to
minimize the size of the transmitted data over the wireless channels. Since
many IoT devices will be interested to sell their semantic information to the
VSP, we propose a truthful reverse auction mechanism that helps the VSP select
only IoT devices that can improve the quality of its virtual copy of objects
through the semantic information. We conduct extensive simulations on a
dataset that contains synchronized camera and radar images, and show that our
novel design enables a fast replication of the digital copy with high
accuracy.

###### Index Terms:

Metaverse, sensing-as-a-service, semantic communication, reverse auction.

##  I Introduction

###  I-A Background and Motivations

Driven by the Covid-19 pandemic, the Metaverse has gained huge interest
recently from different industry and public sectors [1, 2, 3]. Considered as
the next generation of the Internet, the Metaverse enables users and objects
to experience near real-life interaction with each other in the virtual
environment through their avatars. The Metaverse is made up from different
emerging technologies such as virtual reality (VR), augmented reality (AR) and
haptic sensors. Furthermore, other emerging technologies such as beyond 5G and
6G are driving the Metaverse from imagination and fiction towards real world
implementation as they enable users to access the Metaverse from anywhere,
anytime instantly [4].

Digital twin modeling of the physical world in the Metaverse have a number of
benefits for different application scenarios. For example, creating a virtual
copy of the workplace enables employers to bridge the gap between working from
office and working from home. Another example appears in the smart vehicles
area, where several companies face significant challenges to train their
autonomous vehicles (AVs) in the real world. A digital twin of the road,
vehicles and other objects can be created, and then the AVs are trained in the
Metaverse. Instead of training the AVs on the real road network which is a
high risk task, training in the Metaverse is much safer as road accidents in
the Metaverse are not reflected to the real world, but the experiences are
identical. More importantly, the experiences faced in the Metaverse are
learnable by the learning model of the AVs [5]. An agriculture company can
enable smart farming by creating a digital twin of the farm. IoT devices
deployed in the field can collect several types of data about the plants and
the soil, and then send it back to the edge server of the company through an
unmanned aerial vehicle base station (UAV-BS) relay [6]. This enables farmers
to simulate and observe effects of interventions with real-time data before
physical interventions can take place.

As the digital twins are required to replicate the physical real-world system
to the finest details [3], generating an accurate 3D model of the physical
system is the first step towards this goal. However, the creation of an
accurate 3D digital copy is challenging for several reasons. First, the
collection of data about the environment requires different types of sensors,
e.g., cameras, radars and LIDARs, which are required to be located at the
region of interest of the VSP. It will be costly for the VSP to deploy IoT
devices equipped with the aforementioned sensors for each task which are
dynamic in both time and location. Second, the collected data by the IoT
devices is huge in size and the available bandwidth for data transmission will
quickly exceed the system limitation. To enable a real-time construction of
the digital twin in the Metaverse, the communication system needs to be
carefully designed as to maximize successful data transmission while
minimizing the latency of packet delivery.

###  I-B Contribution

In this paper, we study the paradigm of digital twin construction over
wireless channels from the communication and computation perspective. As the
study of the Metaverse is still in its infancy, only few works addressed the
aspect of wireless resource allocation for the Metaverse. In [7], an IoT-
assisted Metaverse sync problem is studied in which an evolutionary game is
formulated to enable the IoT devices to select VSPs to work for. However, the
volume of the data and the limited bandwidth problem was not addressed. In
contrast, our work mainly focuses on addressing the issue faced during data
transmission from IoT devices to the VSP. Specifically, we propose the
transmission of only semantic information to the VSP instead of raw data.
Since the semantic extraction from raw data depends heavily on the data type,
we focus in this work on the case of camera and radar sensors for AV systems
[8, 9]. This is motivated by the fact that camera and radar sensors produce
data with high volume and consume a large amount of bandwidth for
transmission. However, our framework is straightforwardly to apply to other
sensor types (e.g., voice recording) and other use cases of the Metaverse with
changes only to the semantic valuation function. We then leverage existing IoT
devices such as existing AVs and smart phones to transmit their data to the
VSP (Figure 1). The VSP sends a UAV-BS to its region of interest to collect
data. Beside the cost efficiency, this solution is scalable as it enables the
VSP to hire IoT devices to satisfy the targeted QoS. However, IoT devices
might take advantage of the importance of their semantic information to the
VSP and increase their revenues by selling their data much higher than their
true value. To overcome this manipulation of the strategic interaction, we
develop a reverse auction mechanism that incentivizes IoT devices to make
their bids to sell semantic information to the VSP truthfully. Our main
contributions are as follows:

  * •

We propose a novel design for digital twin construction in the Metaverse by
leveraging semantic information extraction algorithms. The obtained semantic
information are sent to the VSP instead of the raw data, reducing the size of
the transmitted data and consequently the latency.

  * •

To collect data from IoT devices, we propose that the VSP sends a UAV-BS to
its region of interest. A reverse auction mechanism is developed then to
select the set of IoT devices that can access the wireless bandwidth and sell
their semantic information to the VSP.

  * •

We conduct extensive simulations to validate our proposed model. The major
finding is that the use of semantic information transmission enables more IoT
devices to transmit their data to the VPS compared to traditional raw data
transmission. This enables the VSP to have a variety of data sources and fast
rendering of the digital twin in the Metaverse.

In Section II, we describe our proposed semantic information marketing model.
In Section III we formulate the social welfare maximization problem.
Simulation results are presented in Section IV and Section V concludes the
paper.

##  II System Model: semantic information Market

###  II-A Preliminaries About semantic information

As illustrated in Figure 1, we consider a Metaverse market which consists of a
virtual service provider (VSP) and a set of $\mathcal{N}=\\{1,\dots,N\\}$ IoT
devices, i.e., data owners. The VSP is responsible for the construction of the
digital twin in the Metaverse based on the collected data from the selected
set of IoT devices. Each IoT device has a set of sensors to collect geo-
spatial data from the surrounding environment and has a machine learning (ML)
model to extract semantic information from the collected raw data. Different
from traditional crowd-sensing platforms that collect all the raw data from
the IoT devices, the VSP obtains only semantic information (e.g., semantic
mask for each object with its corresponding class), which is motivated by the
following reasons:

  * •

The number of channels available to the VSP are limited. Hence, if the VSP
allows transmission of raw data by the IoT devices, only few devices will be
able to transmit their data which might be of less importance than the data of
other non-selected IoT devices.

  * •

Raw data is large in size in general (e.g. video and images), which can
increase the transmission delay making the rendering of the digital twin very
slow and obsolete.

  * •

The quality of the constructed digital twin will be higher as more semantic
information about the physical world will be available to the VSP.

Figure 1: Exemplary use cases of semantic information for the Metaverse.

###  II-B Service Cost in the Metaverse Market

The IoT device needs to calculate the cost of collecting data, extracting the
semantics from raw data and the communication costs required to transmit the
semantic information to the edge server.

####  A. Computation Cost

The computation cost for data collection for IoT device $i$ is defined as
follows [10]:

$c_{i}^{R}=\sum\limits_{k=1}^{m_{i}}d_{ik}^{R}\alpha_{ik}^{R}$ (1)

where $d_{ik}^{R}$ is the size of the collected raw data and $\alpha_{ik}^{R}$
is the per unit computation cost for sensor $k$ by the IoT device $i$. $m_{i}$
is the number of sensors supported by the IoT device $i$. The cost of the
semantic extraction is primarily based on the amount of energy required to
perform the semantic extraction which can be formulated as

$c_{i}^{S}=\Delta t\gamma_{i}$ (2)

where $\Delta t$ is the time required for the semantic extraction given the
input raw data and $\gamma_{i}$ is the per time step computation cost. The
total computation cost is then

$c_{i}^{p}=c_{i}^{R}+c_{i}^{S}=\sum\limits_{k=1}^{m_{i}}d_{ik}^{R}\alpha_{ik}^{R}+\Delta
t\gamma_{i}.$ (3)

####  B. Communication Cost

When transmitting semantic information to the VSP on $C_{i}$ channels with
capacity $r$ for each channel111For simplicity, we consider a frequency-
division multiple-access (FDMA) communication scheme, with other schemes
straightforwardly applicable with appropriate adjustment to the cost
function., the resulting communication cost to the IoT device $i$ is defined
as [10]

$c_{i}^{m}=rC_{i}\beta_{i}$ (4)

where $\beta_{i}$ is the per unit energy cost for transmission. By summing (3)
and (4), the total service cost is

$c_{i}=c_{i}^{p}+c_{i}^{m}=\sum\limits_{k=1}^{m_{i}}d_{ik}^{R}\alpha_{ik}^{R}+\Delta
t\gamma_{i}+rC_{i}\beta_{i}.$ (5)

####  C. Semantic Information Value

Not all the extracted semantic information have the same value. For instance,
if the IoT device has predicted a car collision, the device should value this
information more and hence increase its price when selling the semantic
information to the VSP. Therefore, the IoT device should include the valuation
of the derived semantic information based on the sensitivity of its content in
the final service cost valuation. The semantic information value/reward is
then defined as follows:

$R_{i}^{S}=\left[\sum\limits_{j=1}^{O_{t}}\left(\delta^{j}_{1}+\delta^{j}_{2}+\delta^{j}_{3}+\delta^{j}_{4}\right)q(j)\right]\frac{1}{w_{t}}+\frac{1}{d^{R}_{i}}$
(6)

where $O_{t}$ is the number of objects detected at time $t$, $\delta^{j}_{1}$,
$\delta^{j}_{2}$, $\delta^{j}_{3}$ and $\delta^{j}_{4}$ are binary variable
that reflects the existence of semantic information about the relative speed,
size, relative position and moving direction for the object $j$, respectively.
$q(j)$ is the model quality for the detected object $j$. For example, the
quality function can be the intersection over unit (IoU) for image
segmentation or the accuracy for object detection. Finally, $w_{t}$ is the
weather condition at time $t$ scaled between zero and one. If the weather
condition is bad, i.e., close to zero, the derived semantic information
valuation $R_{i}^{S}$ will increase, reflecting its high value to the VSP. The
last term $\frac{1}{d^{R}_{i}}$ indicates the importance of data size to the
VSP. For example, if two IoT devices have the same semantic value for certain
data, the IoT device that provides this data with a smaller size is preferred
by the VSP.

###  II-C Auction Based Metaverse Service Market

In our considered system model, as there are several sellers (the IoT devices)
and one buyer (the VSP), a reverse auction, where the traditional roles of
buyer and sellers is reversed, is appropriate for our system design [11].
Therefore, to select the set of IoT devices to collect data from, the VSP
conducts a reverse auction. Each IoT device bids for the price it is willing
to sell its semantic information to the VSP. This is modeled as a competition
on the channels provided by the VSP through its UAV-BS. Therefore, the IoT
devices tries to increase their bids $\mathbf{b}=(b_{1},b_{2},\dots,b_{N})$ to
gain access to the minimum number of channels required to transmit their data.
Since all sensors do not have the same data quality due to their location and
hardware specifications, the VSP needs to choose the winners based on their
semantic information quality in addition to data size and bid value. The IoT
device $i$ reports its type $t_{i}=\\{b_{i},R_{i}^{S}\\}$ and the number of
channels $C_{i}$ required to transmit its data. The IoT device is single
minded, i.e., it either take all the request set of channels or any [11].
Additionally, The VSP requires information freshness, i.e., the transmitted
data should not exceed a certain time threshold $t_{max}$ which is broadcasted
before the beginning of the auction. The IoT devices use this threshold to
determine the required number of channels to transmit their local data within
the specified threshold.

##  III Social Welfare Maximization Reverse Auction

The IoT device’s utility is the difference between its payment and service
cost (computation cost and communication cost), which is expressed as

$u_{i}=p_{i}-R_{i}^{S}-b_{i}$ (7)

where $b_{i}=c_{i}$. The utility of the VSP is the sum of all IoT devices’
semantic information rewards minus the cost $\hat{c}$ of allocating the
channels from the wireless service provider and the sum of payments to the
winners, which is written as

$\hat{u}=\sum\limits_{i=1}^{\mathcal{|N|}}\xi_{i}R_{i}^{S}-\sum\limits_{i=1}^{\mathcal{|N|}}\xi_{i}p_{i}-\hat{c}$
(8)

where $\xi_{i}$ is a binary decision variable that indicates if IoT device $i$
is chosen among the winners ($\xi_{i}=1$) or not ($\xi_{i}=0$). The social
welfare of the system is defined as the sum of the utilities of all the system
entities (the VSP and the IoT devices), which is written as

$S(\xi)=\sum\limits_{i=1}^{\mathcal{|N|}}\xi_{i}u_{i}+\hat{u}=-\sum\limits_{i=1}^{\mathcal{|N|}}\xi_{i}b_{i}-\hat{c}.$
(9)

The social welfare is regarded as the system efficiency [12] and hence,
maximizing the social welfare implies an efficient Metaverse market system.
The social welfare maximization is formally written as an integer linear
programming (ILP) problem as follows:

$\displaystyle\begin{split}\max_{\xi}S(\xi)=-\sum\limits_{i\in\mathcal{N}}\xi_{i}b_{i}-\hat{c},\end{split}$
(10a) $\displaystyle\begin{split}\hskip
28.45274pts.t.\sum_{i\in\mathcal{N}}\xi_{i}C_{i}\leq B\end{split}$ (10b)
$\displaystyle\begin{split}\hskip 28.45274pt\xi_{i}\in\\{0,1\\},\forall
i\in\mathcal{N}\end{split}$ (10c)

where $B$ is the number of channels provided by the VSP. The ILP presented in
(10) can be solved using a deterministic off-the-shelf solver or using a
heuristic algorithm [13]. In Section IV, we implement and compare both
solutions. To avoid manipulation of the market by malicious IoT devices, e.g.,
gaining a higher utility than deserved or getting a negative utility, the
mechanism should guarantee the properties of incentive compatibility (IC) and
individual rationality (IR). Therefore, in the following, we present the
payment rule for winning IoT devices and prove the properties of IC and IR.

####  A. Payment Rule

The payment rule for winning IoT devices is based on VCG mechanism payment
[12] and is represented as follows:

$p_{k}=S(\xi^{*})-S_{\mathcal{N}\backslash\\{k\\}}(\varphi^{*})+(R_{i}^{S}+b_{k})\xi_{k},$
(11)

where $\xi^{*}$ is the optimal allocation for IoT devices given the bidding
and demand vectors, and $S(\xi^{*})$ is the corresponding maximal social
welfare obtained from (10). $S_{\mathcal{N}\backslash\\{k\\}}(\varphi^{*})$ is
the maximal social welfare when IoT device $k$ is not among the participant in
the auction where $\varphi^{*}$ represents the corresponding optimal
allocation strategy.

####  B. Incentive Compatibility and Individual Rationality

###### Theorem 1

The proposed VCG-based reverse auction mechanism is incentive compatible.

###### Proof:

We consider two cases for the IoT device $k$ in the set of bidders:
Case 1: The submitted bid $b_{k}$ by the IoT device $k$ is equal to its true
valuation $c_{k}$. In this case, by substituting (11) in (7), the utility is
written as

$u_{k}=S(\xi^{*})-S_{\mathcal{N}\backslash\\{k\\}}(\varphi^{*})$ (12)

Case 2: The submitted bid $b_{k}^{{}^{\prime}}$ by the IoT device $k$ is
different from its true valuation, i.e., $b_{k}^{{}^{\prime}}\neq c_{k}$ and
the utility is written as

$u_{k}^{{}^{\prime}}=S(\xi^{*}{{}^{\prime}})-S_{\mathcal{N}\backslash\\{k\\}}(\varphi^{*}{{}^{\prime}})+\xi_{k}^{*}{{}^{\prime}}b_{k}^{{}^{\prime}}-\xi_{k}^{*}{{}^{\prime}}c_{k}$
(13)

We note that the optimal allocation strategy in the absence of the IoT device
$k$ in the auction in both cases is the same, i.e.,
$S_{\mathcal{N}\backslash\\{k\\}}(\varphi^{*})=S_{\mathcal{N}\backslash\\{k\\}}(\varphi^{*}{{}^{\prime}})$.
The difference between the utilities of the above two cases is then calculated
by:

$u_{k}^{{}^{\prime}}-u_{k}=S(\xi^{*}{{}^{\prime}})-S(\xi^{*})+\xi_{k}^{*}{{}^{\prime}}b_{k}^{{}^{\prime}}-\xi_{k}^{*}{{}^{\prime}}c_{k}$
$=\left[-\sum\limits_{i\neq
k}^{\mathcal{|N|}}\xi_{i}^{*}{{}^{\prime}}b_{i}-\xi_{k}^{*}{{}^{\prime}}b_{k}^{{}^{\prime}}\right]-\left[-\sum\limits_{i=1}^{\mathcal{|N|}}\xi_{i}^{*}b_{i}\right]+\xi_{k}^{*}{{}^{\prime}}b_{k}^{{}^{\prime}}-\xi_{k}^{*}{{}^{\prime}}c_{k}$
$=\left[-\sum\limits_{i=1}^{\mathcal{|N|}}\xi_{i}^{*}{{}^{\prime}}b_{i}\right]-\left[-\sum\limits_{i=1}^{\mathcal{|N|}}\xi_{i}^{*}b_{i}\right]$
(14)

If $u_{k}^{{}^{\prime}}-u_{k}>0$, then the IoT device has an incentive to
deviate from $\xi^{*}$. However, since $\xi^{*}$ is the optimal solution for
the ILP in (10), and hence better than any other solution (e.g.,
$\xi^{*}{{}^{\prime}}$), we have $u_{k}^{{}^{\prime}}\leq u_{k}$. Therefore,
the IoT devices cannot obtain any benefit by misreporting their true valuation
of their data. Note that the IoT device might submit a higher semantic value
than its true value to gain access to the channels and sell its data with
higher price. However, both the ILP problem in (10) and the utility of the IoT
device in (12) are independent from the value of the submitted semantic value
$R_{i}^{S}$, giving no incentive to the IoT device to submit untruthful value
of $R_{i}^{S}$, concluding the proof.

∎

###### Theorem 2

The proposed VCG-based reverse auction mechanism has the property of
individual rationality.

###### Proof:

Given that the mechanism is truthful, we consider two cases:

Case 1: $S(\xi^{*})\geq S_{\mathcal{N}\backslash\\{k\\}}(\varphi^{*})$, then
based on (12), $u_{k}\geq 0$. Hence, the IoT device $k$ will get non-negative
utility after joining the auction.

Case 2: $S(\xi^{*})<S_{\mathcal{N}\backslash\\{k\\}}(\varphi^{*})$. In this
case, since $\mathcal{N}\backslash\\{k\\}\subset\mathcal{N}$, the VSP will
have much more options to choose from with $\mathcal{N}$ rather than with
$\mathcal{N}\backslash\\{k\\}$ implying that $S(\xi^{*})\geq
S_{\mathcal{N}\backslash\\{k\\}}(\varphi^{*})$ and $u_{k}\geq 0$. Therefore,
Case 2 cannot be realized and the proposed VCG-based reverse auction mechanism
has the property of individual rationality. ∎

##  IV Experiments

In this section, we conduct experiments on real world data and present
numerical results to evaluate the performance of our proposed reverse auction
mechanism. Unless otherwise stated, we consider that $N=20$, $t_{max}=1s$, and
that the channel’s capacity for each IoT device is fixed to $r=10kbps$.

(a) Camera view.

(b) Range-Doppler view.

Figure 2: A scene from CARRADA dataset from different views with their
respective semantic segmentation, with a cyclist and a pedestrian. The blue
mask in (b) represents the pedestrian while the green mask represents the
cyclist.

###  IV-A Dataset and Experimentation’s Settings

We conduct our experiments on CARRADA dataset [14] which is a recent, open
dataset that contains 30 scenes of synchronized sequences of camera and radar
images. The radar images consists of the range-angle (RA) and the range-
Doppler (RD) views. Figure 2 illustrates sample images from the CARRADA
dataset where a cyclist and a pedestrian are moving into the opposite
directions. To extract the semantic information from the raw data, different
algorithms are used based on the supported sensors on the IoT device. We
consider 4 types of IoT devices that use different semantic extraction
algorithms. For devices with camera sensors only, Mask R-CNN algorithm for
semantic segmentation and bounding boxes is used [15]. For devices with radar
sensors only, RAMP-CNN (for RA view only), FCN-8s (for either RA or RD views)
and, TMVA-NET (for RA and RD views) are used [9, 14, 16]. We adopt simulation
settings similar to [9]. Each IoT device is considered to use one of the
aforementioned algorithms based on a uniform distribution. For devices with
both camera and radar sensors, a combination of these algorithms is used
instead. As an output for the aforementioned semantic segmentation algorithms,
a white background image which contains the masked objects (see Figure 2) is
produced in addition to a meta-data text file (in JSON format) that contains a
mapping between each object in the image with its class and other derived
semantic information (e.g., range and shape). The meta-data text file has a
maximum size of $d_{meta}=1kb$.

###  IV-B Results

####  A. Impact of The Solver on The Social Welfare

First, we evaluate the efficiency of our proposed ILP reverse auction
algorithm. The ILP is solved deterministically using _Gurobi optimizer_ and
the results are compared to the heuristic auction algorithm proposed in [13]
with the payment rule adjusted to follow (11). We observe from Figure 3 that
both solvers reach an equivalent optimal solution when the number of channels
is higher than $30$. However, when the number of channels is lower, i.e., from
$10$ to $25$, the heuristic algorithm is not able to reach the optimality
derived by the deterministic optimizer. This is explained by the fact that the
heuristic algorithm first sorts the IoT devices (bidders) based on their bids
and stops the search for potential winners once the social welfare stops
increasing. This makes the algorithm deviate from the optimal solution as
other potential combination of winners is not explored. In what follows, we
only consider the use of the deterministic optimizer to solve the social
welfare maximization problem.

Figure 3: Impact of the solver on the social welfare.

####  B. Impact of semantic information Transmission on The Metaverse

We consider two types of transmission: the first one consists of transmitting
raw data, i.e., no semantic information extraction, and the second one
consists of transmitting only the semantic information extracted from the raw
data. We observe from Figure 4 that as the number of channels $B$ provided by
the VSP increases, the number of winners for the case of semantic information
transmission also increases (up to 12 winners). However, in the case where
only raw data is transmitted, the number of winners is very low ($1$ to $2$
winners) and almost remain unchanged, making the VSP relies heavily on the
inputs of a small set of IoT devices. This result is explained by the fact
that raw data has significantly a larger size compared to the size of semantic
information and therefore one winner can use up to half of the available
channels. Furthermore, even if the channels are fully allocated in both cases,
the received data by the VSP might be insignificant in case of raw data
transmission (e.g., in case of camera image, the captured image can be taken
from an angle where no object is present). Specifically, if the VSP tries to
create a copy of the physical world using the received raw data, the
constructed virtual world would have a very poor quality and cannot capture
the real dynamics of the system (e.g., congestion of the road). However, in
case of semantic information transmission, the VSP has a variety of data
sources/views which are rich of semantic information, e.g., instances and
their shape or relative speeds, showing the importance of semantic information
and heterogeneity of data sources. This result also indicates that our
formulation of the utility functions is able to select IoT devices with more
valuable semantic information as winners and hence, transmit their data to the
VSP. An important point to highlight is that as more IoT devices send their
data to the VSP, the chances of having redundant information increase, which
helps creating a more reliable virtual environment.

Figure 4: Impact of the number of channels on the number of winners and social
welfare.

Interestingly, we observe from Figure 4 that when the number of channels
increases from $25$ to $30$, the number of winners drops by $1$. This is due
to the fact that some IoT devices that have more valuable information but
require a larger number of channels are not selected as winners when fewer
channels are available, i.e., $B=25$. However, when the number of channels
increases to $30$, the winner list changes to contain fewer IoT devices but
they can provide a higher social welfare, which is clearly observable in
Figure 4 (red dotted curve).

####  C. Impact of the semantic information Extraction Algorithms and The
Input Data on The Winner List

To observe how the winner list changes based on the semantic information
extracted from data, we consider two identical groups of IoT devices where
each group has 5 IoT devices and use different semantic information extraction
algorithms as described in Table I. We provide the first group with scenes
that contains 3 objects while the second group is provided with scenes which
have only one object. The total number of participating IoT devices is then
$N=10$ and the number of channels is varied from 3 to 21. We observe from
Figure 5 that the first IoT devices to join the winner list after the number
of available channels increases are from group 1. This is justified by the
fact that input scenes for group 1 contains more semantic information than
group 2. Interestingly, when the number of channels increases to 15, IoT
device 5 from group 2 join the winner list leaving one element from group 1
not in the winner list. This is due to fact that IoT device 5 has camera and
radar sensors and is using both Mask R-CNN and TMVA-NET algorithms for
semantic extraction which have high quality of extracting significant semantic
information. The remaining element from group 1 is IoT device 2 uses RAMP-CNN
for radar semantic extraction, which has the lowest performance compared to
other algorithms [9].

TABLE I: Semantic extraction algorithms for different IoT devices. IoT device |  Sensor(s) |  Semantic algorithm(s)
---|---|---
1 |  camera |  Mask R-CNN
2 |  radar |  RAMP-CNN (RA)
3 |  radar |  FCN-8s (RA)
4 |  radar |  TMVA-NET (RA and RD)
5 |  camera & radar |  Mask R-CNN + TMVA-NET
Figure 5: Impact of the extracted semantic information on the winner list.
Green boxes represent IoT devices from group 1 and red boxes represent IoT
devices from group 2.

##  V Conclusion

In this paper, we presented a novel design to address the problem of data
transmission over the wireless channels in the Metaverse. Specifically, we
considered the scenario where a VSP is collecting data from the physical world
using IoT devices in the field to create a digital twin of the road for AV
training. We then proposed the use of semantic information extraction
algorithms by the IoT devices to transmit only semantic information to the VSP
instead of raw data. This significantly reduces the data transmission volume
over the wireless network. It also enables more IoT devices to access the
channels and the VSP to access a heterogeneity of data sources. As many IoT
devices are interested in selling their semantic information to the VSP, we
designed a reverse auction mechanism that has the properties of IC and IR to
enable the VSP to choose the winners. Simulation results show that our
proposed novel design is able to intelligently select IoT devices that have
more semantic information and enable the VSP to create a high quality digital
copy of the physical world. To further improve our design, we consider as a
future work the use of a prediction model by the VSP to be trained on masked
images to create scenes which have been lost during communication due to link
failures.

## References

  * [1] H. Jeong, Y. Yi, and D. Kim, “An innovative e-commerce platform incorporating metaverse to live commerce,” _International Journal of Innovative Computing, Information and Control_ , vol. 18, pp. 221 – 229, 2022.
  * [2] G. Papyshev and M. Yarime, “Exploring city digital twins as policy tools: A task-based approach to generating synthetic data on urban mobility,” _Data & Policy_, vol. 3, 2021.
  * [3] R. Minerva, G. M. Lee, and N. Crespi, “Digital twin in the IoT context: A survey on technical features, scenarios, and architectural models,” _Proceedings of the IEEE_ , vol. 108, no. 10, pp. 1785–1824, 2020.
  * [4] M. Chen, W. Saad, and C. Yin, “Virtual reality over wireless networks: Quality-of-service model and learning-based resource management,” _IEEE Transactions on Communications_ , vol. 66, no. 11, pp. 5621–5635, 2018.
  * [5] A. Amini, I. Gilitschenski, J. Phillips, J. Moseyko, R. Banerjee, S. Karaman, and D. Rus, “Learning robust control policies for end-to-end autonomous driving from data-driven simulation,” _IEEE Robotics and Automation Letters_ , vol. 5, no. 2, pp. 1143–1150, 2020.
  * [6] C. Verdouw, B. Tekinerdogan, A. Beulens, and S. Wolfert, “Digital twins in smart farming,” _Agricultural Systems_ , vol. 189, p. 103046, 2021.
  * [7] Y. Han, D. T. Niyato, C. Leung, C. Miao, and D. I. Kim, “A dynamic resource allocation framework for synchronizing metaverse with IoT service and data,” _ArXiv_ , vol. abs/2111.00431, 2021.
  * [8] W. Yang, Z. Q. Liew, W. Y. B. Lim, Z. Xiong, D. Niyato, X. Chi, X. Cao, and K. B. Letaief, “Semantic communication meets edge intelligence,” 2022. [Online]. Available: https://arxiv.org/abs/2202.06471
  * [9] A. Ouaknine, A. Newson, P. Pérez, F. Tupin, and J. Rebut, “Multi-view radar semantic segmentation,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , October 2021, pp. 15 671–15 680.
  * [10] Y. Jiao, P. Wang, D. Niyato, B. Lin, and D. I. Kim, “Toward an automated auction framework for wireless federated learning services market,” _IEEE Transactions on Mobile Computing_ , vol. 20, no. 10, pp. 3034–3048, 2021.
  * [11] N. Nisan, T. Roughgarden, E. Tardos, and V. V. Vazirani, _Algorithmic Game Theory_. USA: Cambridge University Press, 2007.
  * [12] X. Zhang, Z. Huang, C. Wu, Z. Li, and F. C. M. Lau, “Online auctions in IaaS clouds: Welfare and profit maximization with server costs,” _IEEE/ACM Transactions on Networking_ , vol. 25, no. 2, pp. 1034–1047, 2017.
  * [13] L. Ismail, D. Niyato, S. Sun, and D. I. Kim, “Social welfare maximization auction in joint radar communication systems for autonomous vehicles,” in _2021 IEEE Global Communications Conference (GLOBECOM)_ , 2021, pp. 1–6.
  * [14] A. Ouaknine, A. Newson, J. Rebut, F. Tupin, and P. Pérez, “CARRADA Dataset: Camera and automotive radar with range- angle- doppler annotations,” in _2020 25th International Conference on Pattern Recognition (ICPR)_ , 2021, pp. 5068–5075.
  * [15] K. He, G. Gkioxari, P. Dollar, and R. Girshick, “Mask R-CNN,” in _Proceedings of the IEEE International Conference on Computer Vision (ICCV)_ , Oct 2017.
  * [16] X. Gao, G. Xing, S. Roy, and H. Liu, “RAMP-CNN: A novel neural network for enhanced automotive radar object recognition,” _IEEE Sensors Journal_ , vol. 21, no. 4, pp. 5119–5132, 2021.