1480 10.1109/TVCG.2023.3247054 Research algorithm/technique James Mullen
E-mail<EMAIL_ADDRESS>Dinesh Manocha E-mail<EMAIL_ADDRESS>Mullen et al.:
Virtual Agent Interaction

# PACE: Data-Driven Virtual Agent Interaction in Dense and Cluttered
Environments

James F. Mullen Jr0000-0002-4117-1741 and Dinesh Manocha0000-0001-7047-9801
University of Maryland College Park

###### Abstract

We present PACE, a novel method for modifying motion-captured virtual agents
to interact with and move throughout dense, cluttered 3D scenes. Our approach
changes a given motion sequence of a virtual agent as needed to adjust to the
obstacles and objects in the environment. We first take the individual frames
of the motion sequence most important for modeling interactions with the scene
and pair them with the relevant scene geometry, obstacles, and semantics such
that interactions in the agents motion match the affordances of the scene
(e.g., standing on a floor or sitting in a chair). We then optimize the motion
of the human by directly altering the high-DOF pose at each frame in the
motion to better account for the unique geometric constraints of the scene.
Our formulation uses novel loss functions that maintain a realistic flow and
natural-looking motion. We compare our method with prior motion generating
techniques and highlight the benefits of our method with a perceptual study
and physical plausibility metrics. Human raters preferred our method over the
prior approaches. Specifically, they preferred our method 57.1% of the time
versus the state-of-the-art method using existing motions, and 81.0% of the
time versus a state-of-the-art motion synthesis method. Additionally, our
method performs significantly higher on established physical plausibility and
interaction metrics. Specifically, we outperform competing methods by over
1.2% in terms of the non-collision metric and by over 18% in terms of the
contact metric. We have integrated our interactive system with Microsoft
HoloLens and demonstrate its benefits in real-world indoor scenes. Our project
website is available at https://gamma.umd.edu/pace/

###### keywords:

Embodied agents, Virtual humans, Human Factors

We present an approach to create moving virtual agents that interact with a
dense or cluttered 3D scene, either virtual or real, based in motion-captured
human data. We have integrated our system with Microsoft HoloLens and
highlight its benefit in terms of moving virtual agents in real-world scenes.
As key functionality of our method, we adjust or tailor the motion-captured
agents motion to the obstacles in the scene like the chairs and tables in the
scene above. Our optimization-based approach improves the performance in terms
of well-known interaction metrics corresponding to non-collision and contact.
In this figure, which we captured using our Microsoft HoloLens, we observe a
virtual agent navigating around the obstacles in the environment.

Introduction Humans interact with their environment by making contact with
objects and avoiding collisions with obstacles. For example, a human sitting
at a desk is sitting in a chair. Their arms are likely resting on the surface
of the desk, potentially even grasping the mouse and keyboard. When this human
stands up from their desk, they have to navigate around the room, avoiding any
objects within it, before reaching out to grasp the door knob and open the
door. These types of interactions, defined by contact, dictate how humans move
through their environment. In this paper, our goal is to take motion-captured
humans and translate them into virtual humans that interact with virtual or
real scenes in a natural looking way.

To motivate our problem, imagine a virtual office. To make it feel plausible,
virtual human agents or avatars have to be moving around the space and
interacting with objects such as the chairs, desks, or keyboards with no
penetrations or awkward motions. Current methods for this type of dense,
cluttered space are based on animators using modeling or animation tools to
generate such motion sequences that fit the environment. However, this can be
very time consuming and it requires animators with considerable experience
using these tools. Additionally, cost prohibits a large number of specialized
animations or motions from being produced, resulting in the same animations
being repeated in the virtual environment, reducing the sense of realism of
the environment and the users engagement within it. In contrast, our goal is
to leverage the vast quantities of publicly available human motion-captured
data and place virtual agents personifying the motion-capture into the
environment.

Populating an environment with virtual humans that interact with the
environment naturally is an important problem in virtual reality (VR),
augmented reality (AR) [18], game design [37], and human-robot interaction
[33]. Specifically, a key problem in VR is generating plausible motion of
human-like virtual agents. Prior methods are limited to humans moving in open
or large spaces with few obstacles. However, real world environments,
especially indoor ones, are filled with obstacles and can be dense. Most prior
work in VR in terms of dense or cluttered environments is mainly limited to
object selection or manipulation [40, 52, 3]. Recent work in computer vision
has enabled the creation of several real world datasets of indoor scenes
containing dense objects and furniture[6, 9, 64, 8, 10, 17], such as Habitat-
Matterport or Scan-Net, with hundreds of 3D scans of rooms with many objects.
Such 3D models are increasingly used for VR and AR applications. However, we
need to generate appropriate methods for virtual humans to interact with such
environments by generating collision-free and plausible motion.

Prior work in human-scene interaction, or scene affordances, attempts to place
3D human models into 3D scene scans such that the placement matches real human
behavior. Most methods work with static, single-pose humans, [25, 19, 70, 17].
Recent methods enable the generalization of human model placement into any
possible scene [18], and extend this ability to existing short virtual agent
motions [39]. Motion synthesis literature similarly tries to generate 3D human
motion sequences with recent methods accounting for the scene geometry and
affordances [56, 34, 61].

###  0.1 Main Contributions

We leverage widely available indoor scene datasets and populate the complex,
dense, indoor scenes with natural-looking virtual agents that move and
interact with their environment. For the most natural-looking movements
possible, we start with motion-captured human data and tailor it to the unique
nature of each dense indoor scene. This is different from existing synthesis
approaches that instead attempt to create motion sequences to fit the
environment after learning from many valid sequences. When using motion-
captured data, properly fitting a given environment as to make plausible
interactions requires making changes to each pose in the motion sequence, each
with several degrees of freedom, while maintaining the natural look and flow
of the underlying motion sequence. We do this through a novel optimization
method that takes the virtual agent motion sequence and 3D scene as inputs.
From the scene and motion sequence, the optimization method outputs the
location and orientation of the most natural placement of the virtual agent in
the scene based on interaction metrics, and the modified pose and translation
of the individual meshes in the virtual agent motion. While we did our
evaluations with motion-captured data, we can utilize any virtual agent motion
that can be represented as a time-series of 3D human meshes. We make no
assumptions about the scene itself, but results are better if the objects in
it are labeled. In practice, our approach shows the largest improvement over
prior art with virtual agent motions longer than three seconds. Our main
contributions are as follows:

  1. 1.

We present PACE, a method to generate placements for motion-captured virtual
agents into 3D dense scenes with motion optimization. Using existing methods,
PACE first utilizes a deep model to estimate a virtual agent motion’s
potential scene interactions, before using this information alongside the
motion geometry with another deep model to select important frames in the
motion. Using the output of the deep models, we weight meshes in the motion
such that the highest weight is attributed to meshes with maximum diversity
and important scene interactions. This weighting allows us to find an optimal
initial placement of the virtual agent before our novel algorithms optimize
the agent’s motion itself to better fit the unique constraints of a dense,
cluttered scene. Our optimizer utilizes novel pose and motion losses alongside
contact and semantic losses to alter the motion sequence to best fit the scene
while maintaining the essence and natural look or the original motion.

  2. 2.

We qualitatively show natural and physically plausible virtual agent
placement. Through a perceptual user study we show that human raters prefer
PACE 67.9% of the time over an extension of POSA to the time dimension, 81.2%
of the time over a state-of-the-art motion synthesis method, and 59.3% of the
time over PAAK.

  3. 3.

We quantitatively show a significant improvement over POSA[18] and PAAK[39] in
the physical plausibility and interaction metrics[70, 18]. These metrics
include the non-collision and contact metrics which denote a lack of
collisions and interactions with the scene respectively. Specifically, PACE
improves upon PAAK by over 1.2% in the non-collision metric and by over 18% in
the contact metric.

  4. 4.

We have integrated PACE with a Microsoft HoloLens and evaluate the performance
in real world scenes. This allows users to populate their local environment
with moving virtual humans that interact with the environment in natural-
looking and physically plausible ways. The populated virtual agents move at 30
frames per second in a plausible manner.

  5. 5.

We release a dataset with tens of thousands of virtual agents placed into
dense and cluttered indoor scenes. We also release software to render videos
from a camera perspective of the users choosing or interact with the entire 3D
scene-agent pair.

##  1 Related Work

Generating Plausible Virtual Agent Motion. Closely related to our work is the
problem of generating plausible virtual agent motion, especially in cluttered,
dense virtual environments. This is a well studied problem [50, 49, 31, 54,
2], with many trying to direct or maneuver a users motion through a dense,
cluttered environments. More recently, many researchers focused on plotting
virtual agent motion through a virtual environment [69, 29], or generating
virtual agents to fit an environment [71, 32]. We aim to combine many of these
threads of research by creating a method capable of generating agent-scene
pairings such that interactions with the environment are natural-looking and
convincing for the user.

Human Models. Most prior work with virtual human agents utilizes body
skeletons to model 3D humans [21, 53]. However, the surface of the body is
important for rending the virtual agent or modeling interactions with the
scene or any objects. For example, it is essential to know how far the surface
of the back is from the skeleton when placing an agent laying on a surface.
Learned parametric 3D body models have addressed this need [1, 23, 35, 45,
44]. In our work, we utilize SMPL-X [45], a body model that captures face and
hand articulation in addition to the structure of the body itself.

Motion Synthesis. Motion synthesis is a well-studied problem in computer
graphics, VR, and computer vision [27, 46, 59, 16, 56, 34, 66, 20, 48, 61,
60]. Early work in motion synthesis worked to synthesize intermediate states
between two given points in the motion sequence [59, 16, 65]. These methods
were known to not handle large translational position changes effectively.
More recent work began utilizing data-driven deep models for motion synthesis
[66, 20]. These methods show better generalization than the geometric methods
from earlier work. However, most of these methods look at the animation or
generated motion in isolation from the scene or environment.

Some more recent motion synthesis methods account for both the environment and
the virtual agent’s movement [56, 34, 7, 63]. Many of these methods use
greatly simplified scenarios with predefined objects or primitive motion. Our
work is closest to [61] which utilizes arbitrary 3D environments when
synthesizing its motion. The motions produced by [61] and others frame our use
case by falling short of the realism of motion-captured movements. However,
our system, PACE does not synthesize its own motion, but instead leverages
motion capture data and tailors it to the needs of the scene, resulting in the
most natural-looking and physically plausible agent-scene pairing possible.

Figure 1: An overview of PACE. Using exiting methods, we first estimate human-
scene interactions for a given scene and use those interactions to determine
the frame weighting in the virtual agent motion. Through our novel
contributions, we then utilize the scene geometry to optimize the motion of
the virtual agent such that it interacts with the environment, matching both
the interaction in the motion and the geometry of the scene based on our two
interaction metrics: non-collision and contact.

Video Synthesis. Our work is also related to the synthesis of videos
containing human actions. Generative adversarial networks (GANs) [12] and
neural radiance fields (NeRFs) [38] have contributed to a ever growing body of
work attempting to generate videos of humans completing actions in arbitrary
scenes. [41, 47, 43, 68, 57, 67, 14, 72, 28, 5, 42] worked towards extending
NeRF to scenes with multiple objects and arbitrary backgrounds, or to
articulated objects like humans. Closest to our work is [42] which similarly
takes a human action and attempts to place it on a background. Our approach
addresses many of the shortcomings of [42], enabling complex natural-looking
background and natural-looking interactions between the animation and
background.

Human-Scene Interaction. Also closely related to our work is human-scene
interaction (HSI), or scene affordance. Early efforts in HSI were purely
geometric with Gleicher [11] and Kim et al. [25] using contact constraints and
automating 3D skeleton generation respectively. These and subsequent works
exploited the importance of contact with the environment, with some accounting
for the forces present in the environment [24, 30, 13]. As a major step
forward, Gupta et al. [15] estimated the human poses "afforded" by the scene
by predicting a 3D scene occupancy grid and computing the support of 3D
skeletons inside of it.

Data-Driven Methods. Many data-driven approaches have been proposed [58, 22,
51, 71, 18, 32]. [22] and [26] learn to estimate human poses and object
affordances in an RGB-D 3D scene while [62] learned to utilize the affordances
of the scene to optimize pose estimation. Closest related to PACE, PSI [71],
PLACE [70], POSA [18], and PAAK [39] populate scenes with SMPL-X [45] human
bodies. Of these methods, POSA and PAAK are unique in that they are human-
centric, finding estimated interactions in the human meshes and then pairing
those interactions with any relevant affordances in the environment. In PACE,
similar to PAAK, we leverage POSA’s model and utilize the estimated contact
and semantic information it provides about a given virtual agent motion
sequence. Of all of these methods, PAAK is unique in its ability to place
virtual agent motion instead of single, static human meshes. Our key
difference from PAAK is our ability to tailor the virtual agent motions
directly to the unique needs of a scene. PAAK is handcuffed by the geometry of
the motion, resulting in many unnatural-looking or physically implausible
placements. For example, a PAAK placement of a sitting motion is likely to
result in either the buttocks or feet levitating above the surface of the
scene due to a mismatch between the motion captured seat and the one in the
scene. PACE, in contrast, alters the animation to fit the unique geometry of
each scene exactly.

##  2 Putting Moving Virtual Agents into Dense 3D Scenes

In this work, we approach the following problem statement: Given a sequence of
virtual agent motion and a 3D scene, create the most natural-looking and
physically plausible agent-scene interaction. Specifically, our goal is to
take a given virtual agent and tailor it to a scene such that it properly fits
the scene and that any interactions with the scene found in the agents motion
(i.e. touching objects, sitting on a couch) match the affordances of the
scene. When using cluttered or dense indoor environments, it becomes
increasingly difficult to find an appropriate place to put the virtual agent.
We use a data-driven method and make appropriate alterations to the virtual
agents motion that result in significant improvements in the physical
plausibility of agent-scene interactions and result in more natural-looking
placements. At an intuitive level, you can imagine a motion that involves
reaching for an object, but in the scene provided the object is further away
than that modeled in the motion-captured environment. For the most natural-
looking results, it is important that the motion is altered to reach towards
the object present in the scene. Agent-scene pairings that do not alter the
motion in this way will result in unnatural looking interactions that will be
obvious to the user in an XR setting.

###  2.1 Overview

We present an overview of PACE in Figure 2. A list of symbols frequently used
in this work are shown in Table 1. Rarely used symbols are defined where they
are used. In terms of notation, we define a time-series using an uppercase
letter, while a lowercase letter denotes an individual frame in the time-
series.

The inputs to PACE are a set of virtual agent motions and a set of 3D scene
meshes. PACE outputs a location and orientation for the virtual agent in the
scene, as well as the altered pose and translation of each frame in the agents
motion sequence. Each virtual agent motion sequence input, $V_{b}$, consists
of a time series of human meshes, $v_{b}$, which are defined by the SMPL-X
[45] parametric human body model. SMPL-X uses a human skeleton of 24 joints,
all of which we can alter to tailor the motion to the scene. Each 3D scene
input is represented as a static triangle mesh with vertex class labels
denoting what type of object a given surface is. These class labels are
helpful but not required. For the altered pose and translation outputs, PACE
provides the 24 joint pose and xyz translation of each frame in the motion
sequence. This can be redefined by SMPL-X to create a new motion sequence
similar to the original input but properly fitting the environment.

We first take these meshes and estimate likely human scene interactions given
the geometry of the mesh’s surface. We then extract geometrically and
semantically important frames and utilize active learning techniques to
extract a diversity score for each frame. The combination of this information
and the mesh geometry are combined to create the frame weights, $k$. Using the
frame weights, we can find promising initial placements for the virtual agent
in the scene before we optimize both the placement in the scene and the
geometry of the agents motion. To optimize the motion of the virtual agent to
best fit the scene, (as defined by the physical plausibility metrics we define
below) we use the geometry and semantics of the scene and try to match it to
the virtual agents motion, altering the specific poses in the agents motion as
needed to create a match. This results in an agent-scene pairing that is as
natural-looking as possible, with the motion closely matching the geometry of
the scene and the interactions it contains accurately modeled.

Symbols |  Definitions
---|---
$V_{b}$ |  A virtual agent motion, consisting of a time series of 3D human meshes
$v_{b},v_{p},v_{\tau}$ |  A single 3D human mesh, an individual frame from $V_{b}$, the pose of that mesh, and the translation of that mesh relative to the first mesh
$f_{c}$, $f_{s}$ |  Contact labels and Semantic labels, attributed to each vertex in a mesh, $v_{b}$
$k$ |  frame weights
$E_{p}$, $E_{alt}$ |  The objective functions utilized when optimizing the motions placement and geometry
Table 1: List of symbols used and their definitions.

We choose to use an optimization method for altering the agents motion with an
end-to-end deep learning model because it will always find the ideal placement
and motion geometry for the scene. In contrast to this, a deep learning model
would have to learn to generalize over all possible motions and scenes, not
likely finding ideal placements or motion geometry for any of them.
Additionally, data does not currently exist of humans moving and interacting
with dense or cluttered 3D scenes. Our optimization method also allows for
flexibility to take into account other interaction metrics. For example, if
not penetrating the scene is a hard constraint, users can easily alter the
weight of the penetration loss and the optimization can adapt accordingly.

###  2.2 Human-Scene Interaction Estimation

To place a virtual agent motion into a scene such that any interactions
present in the motion are preserved, we must first determine those
interactions. Following the lead of PAAK [39], we directly implement the POSA
pretrained [18] conditional variational autoencoder (cVAE) [55] and feed each
frame of the virtual agent motion into it individually. The POSA cVAE, $f$,
generates an egocentric feature map for each vertex in each mesh, $v_{b}$, in
the virtual agent motion, $V_{b}$. This feature map consists of a contact
label, $f_{c}$, and a set of semantic labels, $f_{s}$, which intuitively
denotes a) whether that vertex should be in contact with the scene, and b),
what it should ideally be in contact with, respectively. For example, if the
mesh is of a person grasping an object, the vertices on the hand should have a
high contact probability and highlight the object semantic label. In contrast,
vertices on the other hand or on the agents back should have a very low
contact probability. We represent POSA as the function $f$ in Equation 1
below.

$f:(v_{b})\rightarrow[f_{c},f_{s}]$ (1)

PACE enables the use of any length of virtual agent motion in contrast to PAAK
which only tested with motions 60 frames (or 2 seconds) in length. We
specifically tested conducted our quantitative tests with motions from 4-15
seconds in length with 60-450 total frames. Using longer motions is only
limited by the scene size, with animations longer than 15 seconds rarely
having a valid fit location in the single-room, small scenes we utilized when
testing PACE. We feed all of the meshes of a given virtual agent motion
individually into $f$ to extract the full set of contact labels, $F_{c}$, and
semantic labels, $F_{s}$, for use as inputs to our optimization methods and
frame weight extractions.

Figure 2: An overview of our novel optimization process. We iteratively modify
the motion of the virtual agent while trying to find the optimal location for
it in the scene. This results in a motion that better fits the scene it is
being placed in, for more natural-looking and physically plausible virtual
agent motion. In this example, note the green circle highlighting changes to
the arm geometry so it does not collide with the table in the final placement.
In the initial motion, the arm penetrated the table when the person sat down.

###  2.3 Frame Weighting

We implement PAAK to extract the a frame weights for each frame in the virtual
agent motion. Intuitively, finding these frame weights allows our optimization
methods to focus on the most important pieces of the motion when placing it
into the scene. For example, you can imagine a virtual agent which stands
still for a long period before sitting down in a chair. The sitting motion
would only be a few frames of the overall motion and it could be easy for the
optimization process to miss these frames and instead focus on optimizing the
long standing still period. However, to create the most natural-looking
placement, it is extremely important that when the sitting motion occurs, it
ends in a chair, while the standing period of the motion can happen almost
anywhere. PAAK strives to find the frames that are most important for a
natural-looking placement in the scene, $K$. PAAK consists mainly of a deep
model, $g$, trained to estimate a geometric frame weighting equation. PAAK
then utilizes active learning techniques to give each frame in the motion a
diversity score. The combination of the model output approximating geometric
and semantic cues, and the diversity score, are used to create a final frame
weighting for the virtual agent motion with each frame in the motion given a
weight. Equation 2 and Equation 3 below show the PAAK active frame weighting
framework.

$K=\lambda_{g}*\hat{K_{g}}+\lambda_{b}*W_{d}$ (2)
$g:(V_{b},F_{c},F_{s})\rightarrow\hat{K_{g}}$ (3)

Specifically, the PAAK deep model, $g$, maps from the virtual agent motion
geometry, contact labels, and semantic labels to an estimation of a geometric
frame weight definition that emphasizes motion and unique semantics. To get
the final frame weights, the output of the model alongside the diversity score
obtained from the models gradients are summed.

For datasets with frame rates higher than 30 fps, such as the AMASS [36] CMU
MoCAP [4] subset with a 120 fps frame rate, we modified the PAAK algorithm to
sample down to an average frame rate of 30 fps. Specifically, we utilize their
deep model and extract the top 25% of the frames by frame weight and discarded
the remaining frames. To make sure we did not create large gaps in motion, we
created a frame rate floor of 10 fps and retained additional frames to make
sure we did not drop below this amount. Sampling down to 30 fps on average
allowed our optimizer to work quicker as the number of meshes it has to
calculate and the number of parameters it has to optimize are greatly
decreased.

###  2.4 Scene Placement and Motion Optimization

As our main novel algorithmic contribution, we place the agent into the scene
such that it makes sense in context and complete any alterations to the motion
needed to better fit the scene. For the placement itself, we mainly key in on
the available semantic, contact, and geometric information of the virtual
agent motion as weighted by the frame weights. However, we make any
alterations to the motion concurrently to finding the placement as the
alterations to the motion may make a given placement significantly better.
Specifically, we optimize two objective functions, $E_{p}$ and $E_{alt}$ where
$E_{p}$ finds the optimal location, $\tau$, and rotation, $\theta$, in the
scene for the virtual agent and where $E_{alt}$ finds the optimum alterations
to the agents motion to maximize the viability of the placement without
generating too much movement as to hurt the perceived realism of the motion.
As in [39] and [18], $E_{p}$ minimizes the sum of an affordance loss,
$\mathcal{L}_{afford}$ and a penetration loss, $\mathcal{L}_{pen}$, calculated
for each frame weighted by the frame weights, $k_{a}$.

$E_{p}(\tau,\theta)=\sum^{|k|}_{i=0}{k_{i}*[\mathcal{L}_{afford,i}}+\mathcal{L}_{pen,i}]$
(4)

Intuitively, $\mathcal{L}_{afford}$ is minimized when the distance to the
scene is small for vertices with a high probability of contact using $f_{c}$
and when the semantic label $f_{s}$ matches the semantics of the object
vertices are in contact with. $\mathcal{L}_{pen}$ heavily penalizes placements
that result in the agents motion penetrating the scene as that is a key factor
in physical plausibility. The frame weights, $k$, allows the optimizer to
accurately find a placement that focuses on the key interactions in the
virtual agent motion.

For the alteration objective, $E_{alt}$, two more loss functions are added
with the intention of maintaining consistency through the virtual agent
motion. $E_{alt}$ minimizes the sum of a pose loss, $\mathcal{L}_{pose}$, and
a motion loss $\mathcal{L}_{mot}$. $\mathcal{L}_{pose}$ is minimized when the
pose of each individual mesh in the motion is close to the original pose, thus
minimizing unrealistic poses. $\mathcal{L}_{mot}$ is minimized when the motion
between two frames is the same as from the original virtual agent motion, thus
minimizing the probability of overly large single motions between frames. The
weighting for these two loss terms are set such that an individually large
difference in pose with a number of small differences in motion is less costly
than large changes in motion. This creates an effectively connected nature
where the optimizer favors a smooth motion towards the goal pose over a sudden
change in pose. The optimizer for $E_{alt}$ operates across the translation of
each mesh in the virtual agent motion and the pose of each skeleton in each
frame of the virtual agent motion. Equation 6 shows the calculation pose loss,
taking effectively the squared error of the pose for each frame. Similarly,
Equation 7 shows the calculation of the motion loss, which is the squared
error of the difference between each sequential pose and translation in the
motion sequence. We added a discount factor, $\lambda_{\tau}$ for the
translation component of the loss as we found we achieved better results when
emphasizing it less, but worse results if we removed it entirely. This process
can be seen visually in Figure 3.

$E_{alt}(V_{\tau},V_{p})=\sum^{|k|}_{i=0}{k_{i}*[\mathcal{L}_{pose,i}}+\lambda_{mot}*\mathcal{L}_{mot,i}]$
(5) $\mathcal{L}_{pose}=\sum^{|k|}_{i=0}{(v_{p,i}-\hat{v_{p,i}})^{2}}$ (6)
$\mathcal{L}_{mot}=\sum^{|k|}_{i=0}{(diff(v_{p})_{i}-diff(\hat{v_{p}})_{i})^{2}}+\\\
\lambda_{\tau}[\sum^{|k|}_{i=0}{(diff(v_{\tau})_{i}-diff(\hat{v_{\tau}})_{i})^{2}}]$
(7)

Figure 3: Comparisons on placing the same virtual agent into the same scene
across PACE, POSA-T, and PAAK. Note that two angles of each placement are
provided. For the first placement, PACE is the only one that maximizes the
interaction with the environment by tailoring the agents motion to maneuver it
around the chair to the right of the table, placing the hand on it as a guide
as a real person might. For POSA-T, the placement not only puts the agent
perpendicular to the chair, but then has it wonder off the scene where its
interactions make less sense. PAAK is in a middle state where it maintains
contact with the environment but penetrates the chair and table and floats
above the ground surface. For placement 2, PACE shows the most probable
interaction given the virtual agent motion provided. It interacts with the
chair and the bed, bracing its movement to the chair with the hand. It does
however result in an awkward yet still valid seating position. For POSA-T, the
second seating position is completely ignored, which makes sense due to its
lack of frame weighting. PAAK finds a valid placement for the virtual agent
but the hand placements are awkward and do not contact the scene in the way a
real human might.

$E_{alt}$ when used in combination with $E_{p}$ is especially powerful as the
optimizer can fit the virtual agent motion to the scene while retaining the
essence and natural look of the real motion sequence. For example, in a
scenario where the virtual agent is sitting in a chair that is smaller than
that which was recorded in the original motion capture, the feet of the agent
would normally go through the floor. However, when using $E_{alt}$ and $E_{p}$
together, the actual poses of the agent can change to make sure the feet do
not penetrate the floor, while maintaining a smooth and natural-looking
sitting motion. Without $E_{alt}$ there are many scenarios where the virtual
agent motions do not closely fit the needs of the scene resulting in
penetrations, floating feet, or inaccurate affordances. These errors result in
physically implausible and unnatural-looking placements for the virtual agents
that quickly break users immersion and satisfaction.

We specifically use a Pytorch implementation of L-BFGS with Strong Wolfe line
search as our optimizer following [18] and [39]. A learning rate of 1 is used
and the optimization executes for 10 steps.

###  2.5 Physical Plausibility Metrics

Quantitatively measuring the physical plausibilty of interactions generated by
PACE is important for validating its results. For this, we use the non-
collision and contact metrics originally defined by Zhang et al. [71] and used
by Hassan et al.[18], and Mullen et al. [39]. Each metric is defined on the
human mesh level. Given the body mesh, $v_{b}$, the scene mesh, $m_{s}$, and a
scene signed distance field (SDF) that stores distances for each voxel, the
non-collision score is computed as the ratio of body mesh vertices with
positive SDF values divided by the total number of vertices. In contrast, the
contact score is calculated as 1 if at least one vertex of $v_{b}$ has a non-
positive distance value to the scene.

##  3 Experiments

###  3.1 Datasets and Baseline

For our PROX based evaluations, we randomly sample 10,000 motions from the 8
training scenes of PROX to use as our virtual agents. We then place these
virtual agents into the 4 testing scenes. For our perceptive user study, these
placement results are then shown to human raters in comparison with a baseline
or ablative method. The rates are shown two placements in the same scene and
asked to pick the more realistic of the two videos. For our quantitative
evaluations of the physical plausibility metric all of the methods are used to
generate results to be compared directly.

For our AMASS based evaluations, we randomly sample relevant virtual agent
motion sequences from the CMU MoCap dataset. This means that we filter out
motions that involve dancing, climbing, or those that would otherwise be
unnatural in a cluttered indoor environment. We then use a HoloLens augmented
reality headset to sample a local scene and populate that scene with a virtual
agent using PACE. This was done as a proof of concept for how this could work
in use. Additionally, we will be releasing a dataset we created using the
AMASS/CMU dataset and scenes from PROX and Matterport3D [6].

Figure 4: Comparisons on placing the same virtual agent into the same scene
across PACE, and two ablation studies. Specifically, we attempt to remove the
$\mathcal{L}_{pose}$ and $\mathcal{L}_{mot}$ terms and evaluate the results.
Note that two view angles of each interactions are provided. For the first
interaction, notice how PACE places the agent such that it has direct, non-
penetrative contact with the table and the ground that matches the motion of
the agent. Conversely, removing $\mathcal{L}_{pose}$resulted in the agent
seeming to float out of the scene, and contort in awkward ways, i.e. a non-
plausible interaction. Removing $\mathcal{L}_{mot}$ resulted in no visible
placement as each individual mesh was separated and thrown far from the scene.
For placement 2, PACE successfully navigates the virtual agent around the
chair, and even results in contact between the hand and the chair for support.
In contrast, the virtual agents motion without $\mathcal{L}_{pose}$ resulted
in some awkwardly leaned poses and a lack of contact for some of the meshes.
Moreover, the motion sequence without $\mathcal{L}_{mot}$ became separated
into pieces, not resembling a valid motion at all. Overall,
$\mathcal{L}_{pose}$ and $\mathcal{L}_{mot}$ improve the plausibility of the
interactions.

We compare PACE with POSA, PAAK, a motion synthesis method, and ablations of
PACE. We describe the baselines in greater detail below:

POSA-T. Hassan et al. [18] propose a method that places a single human mesh
into a scene given its affordances. As this approach does not consider motion,
we use PAAK’s [39] POSA-T baseline which is modified to sum the placement loss
over all the meshes in the motion sequence. It is labeled POSA-T for the
addition of the time dimension.

PAAK. For PAAK, Mullen et al. [39] selects “keyframes" from a virtual agent to
focus on when optimizing the placement of the agent into a scene.

Motion Synthesis. Wang et al. [61] propose a motion synthesis method that
accounts for the affordances of the scene in its motion creation. We used
their approach as designed. We call this baseline Motion Synthesis.

###  3.2 Evaluation

A qualitative evaluation of PACE alongside PAAK and POSA-T can be found in
Figure 4. Generally, we found that PACE generated improved interactions and
fittings over the baselines, especially with regard to the details of each
interaction. POSA-T was especially prone to invalid placements or
interactions, like having the agent sitting in midair. PAAK placements tended
to decrease in quality as the length of the virtual agents motion increases
while its interactions were clumsy and inexact. This is likely due to its
inability to alter the motion itself. This prevented it from tailoring the
motion to the small, cluttered environments in the PROX dataset. These
phenomenon can be seen in Figure 4. Anecdotally, as scene complexity
increased, valid fittings tended to decrease. As a simple example, if you
imagine a scene with a chair placed such that its right side is touching a
wall, a motion that goes from sitting to standing, before immediately turning
right and walking, will not have a valid fit location. These types of issues
become more common as scene complexity increase, and are especially relevant
for PAAK which cannot make small changes to the motion to accommodate the
scene.

| Non-Collision $\uparrow$ | Contact $\uparrow$
---|---|---
POSA-T | 0.983 | 0.733
PAAK | 0.986 | 0.784
PACE | 0.996 | 0.929
PACE w/o $\mathcal{L}_{pose}$ | 0.995 | 0.610
PACE w/o $\mathcal{L}_{mot}$ | 1.00 | 0.340
Table 2: Evaluation of the physical plausibility metrics. A higher score is
better for both the non-collision score and the contact score. PACE
outperforms both of our baselines, especially in the contact metric. Moreover,
PAAK results in a higher number of collisions between the virtual agent and
the obstacles, while also having less meaningful interactions with the scene.
Additionally, while the ablative methods without some of our loss terms
perform well in the non-collision metric, they resulted in far lower contact
scores than PACE.

Physical Plausibility. Following the procedures utilized by [18, 70, 71, 39],
we take 100 virtual agents and place them into the 4 test scenes of PROX. With
the placements providing a body mesh relative to the scene, we use that and a
scene signed distance field (SDF) to compute a non-collision score and contact
score as defined originally in [71] and explained in section 3. The results
are shown in Table 2. A high non-collision score denotes that the meshes in
each motion sequence do not penetrate the scene. PACE is an improvement over
the baselines in the non-collision score.

The contact score is calculated as 1 if at least one vertex of the mesh is in
direct contact with the scene. PACE is a significant improvement over the
baselines, showcasing a unique ability to bend the virtual agents motion
towards the geometry of the scene. The decrease in performance with PAAK and
POSA-T are likely due to mismatches between the motion geometry in the scene
geometry in things like seat size or exact object distances. For example, when
standing up after sitting, if the seat size was not correct, the feet could be
positioned above the floor instead of in contact with it. In contrast, PACE is
able to ensure contact at each frame in the motion sequence, maximizing the
contact metric. These results show that PACE creates more physically plausible
placements than the baselines.

Figure 5: PACE evaluated in a real-world scene using HoloLens. Notice how the
virtual agent smoothly navigates around the obstacles in the real-world
environment.

Ablation Results. We ablated PACE against the use of the $\mathcal{L}_{pose}$
and $\mathcal{L}_{mot}$ loss functions and compared the results qualitatively
and with the physical plausibility metrics. For the physical plausibility
metrics, both of these ablations resulted in similar or less collisions than
PACE. However, the contact metric rapidly decreases. This makes sense when
visualizing the results qualitatively in Figure 5. When ablating against
$\mathcal{L}_{pose}$, we notice that the poses of the human itself rapidly
decomposes to look unnatural and disfigured. The virtual agent itself also
moves rapidly away from the scene as if taking flight. This allowed the
optimizer to negate any penetration with the environment, at the cost of
contacting it in any way and retaining realism in the motion. The results are
also poor when ablating against $\mathcal{L}_{mot}$. As the translational
location and motion of each mesh relative to the previous frame are no longer
enforced, the virtual agent would often be moved entirely out of the frame by
the optimizer. Some individual meshes would sometimes remain when there was
important semantic contact.

HoloLens Study. We developed an augmented reality system for the Microsoft
HoloLens where the application samples the geometry of its current
environment, and places a virtual agent into it using PACE. Figure 6 shows an
example of this implementation. We found a number of unique challenges, mostly
related to the inaccuracies in the HoloLens room scans. It would box some
things off incorrectly, resulting in conflicts. The HoloLens also does not
classify the objects in the room semantically, making it more difficult to
place virtual agents that want to specifically key in on this information. For
example, a virtual agent with a laying interaction may lay on the floor
despite there being a bed nearby. Anecdotal feedback suggests that users were
impressed by the natural-looking motion of the virtual agents, but were not
completely convinced they could be real people. Specifically, users noted that
virtual agents seemed to often be walking around aimlessly which a real human
would not typically do. After precomputing the scene and motion sequences, the
final virtual agents can be populated into the HoloLens at 30 frames per
second.

Figure 6: Example question shown to participants in our user study. Two videos
are presented, one corresponding to PACE, the other corresponding to a
baseline. Users would play each video as many times as they wished and would
see the virtual agent move through the scene. Users would then have to choose
which video was more realistic or plausible.

###  3.3 User Study

We conduct a web-based user study to determine the quality of the virtual
agent-scene pairings PACE creates compared to those created by the baselines.

Procedure. We follow the protocols of Hassan et al. [18], Zhang et al. [70],
and Mullen et al. [39]. The study consisted of 60 questions taking
participants approximately 30 minutes to complete. For each question,
participants were shown two clips of a virtual agent placed into a cluttered
3D scene. They would play these two clips and decide which one they perceived
to look more natural or realistic. An example user study question can be seen
in Figure 7.

To generate our clips, we used the 4 testing scenes from the PROX dataset. We
then took 100 virtual agent samples from the PROX training set for placement
into the scenes. All of the methods, with the exclusion of the synthesis
method, begin with a grid of potential placement locations and orientations in
the scene with each location testing rotations every 30 degrees. We then find
the optimal placement and alteration by minimizing $E_{p}$ and $E_{alt}$. We
then render each agent-scene pair into a video from four different angles so
the human raters can get a sense of the relationships between the agent, its
motion, and the scene.

We had a total of 400 agent-scene pairings for each method, 100 for each of
the 4 utilized scenes. Users were shown a selection of these as to not make
the study prohibitively long. Specifically, 5 clips from each of the four
scenes, for each method, were selected. As we compared against 3 baseline
methods, this resulted in 60 total questions per participant. Participants
were not told which clip was from which method. Moreover, we made sure that
PACE was not always placed above or below the baseline to prevent any
selection bias. As the selection of clips was random, some clips were viewed
and rated more frequently than others by participants.

| Baseline $\downarrow$ | PACE $\uparrow$
---|---|---
POSA-T | 32.1% | 67.9%
Motion Synthesis | 18.8% | 81.2%
PAAK | 40.7% | 59.3%
Table 3: PACE compared to POSA-T, Motion Synthesis [61], and PAAK. Subjects
are shown pairs of motions placed into a 3D scene by PACE and by a baseline
and must choose the most realistic one. A higher percentage indicates the
scene-agent pairing subjects deemed more plausible and exhibits better
interaction. Users preferred PACE over all of the baseline methods.

Evaluation. 42 participants took part in our study. They were recruited by web
advertisements and word of mouth. We did not record demographic information as
to avoid any identifying information being collected. Additionally, unlike
similar studies looking at emotion or gestures, demographic background should
not impact whether a virtual agent’s motion looks physically plausible. As
each participant responded to 60 questions, we had a total of 2520 responses.
The results are shown in Table 2.

PACE is an improvement over PAAK, with participants preferring it 59.3% of the
time. Participants sometimes found it difficult to fully notice the difference
in some placements due to the low resolution of the video. Where users did see
the difference is in the even more cluttered or smaller rooms where PAAK would
often generate interactions that would penetrate the scene (i.e. collisions),
or be entirely at an elevation that does not make realistic sense, like
walking on a bed. This makes sense as PAAK was unable to modify the motion of
the agent to properly respond to the unique characteristics of the
environment, effectively shoehorning in motion that does not accurately fit
into the scene. Participants perceived PACE as more natural-looking than
POSA-T 67.9% of the time. This is likely because POSA-T created much worse
interactions with common occurrences including the virtual agent levitating or
sitting in mid-air. The most plausible explanation for this being POSA-T’s
lack of frame weights and inability to alter the motion of the agent to
properly fit the scene. Users strongly disliked the virtual agents created by
the motion synthesis method, preferring PACE 81.2% of the time. The most
likely explanation for this is the motion synthesis method not consistently
generating smooth, plausible motion. Individual frames of the motion generated
by the synthesis method looked great but when it played as a video, like that
shown to participants, the motion often looked unrealistic. Specifically, legs
frequently did not move in a natural way when walking. This result emphasizes
the importance of using motion-captured virtual agents to create the most
natural-looking agent-scene pairings.

##  4 New Virtual Agent Movement Dataset

Using the AMASS [36] CMU MoCap [4] dataset, we used PACE to create a new
dataset of virtual agent placements in cluttered indoor 3D scenes. We are
releasing this dataset of tens of thousands of agent-scene pairs. We use the
fitting loss, $E$, as a quality marker on the fitting and save pairings with a
loss below a threshold value. Additionally, we filter the placements by the
physical plausibility metric, ensuring that pairings with poor plausibility
are not included. This filtering process allows us to filter out virtual
agents that are poor fits for a given scene and would need an unnatural amount
of alteration to fit properly. Completing this process with the full AMASS/CMU
MoCap dataset alongside the PROX and Matterport3D-R [70] room scans results in
tens of thousands of samples. These samples are labeled by the original CMU
MoCap data labels so users can filter by motion type. The motions included in
our dataset include walking, jumping, stretching, sitting, navigating, among
others. This dataset would be released at the time of publication.

##  5 Conclusions, Limitations, and Future Work

In this paper, we propose PACE, a novel method for generating placements for
virtual agents into a dense cluttered 3D scene with accurately modeled human-
scene interactions and tailored motion. Our key insight is that tailoring the
motion of the virtual agents is essential to creating agent-scene pairings
that are natural-looking, especially in dense or cluttered environments. Human
raters preferred PACE agent placements over existing methods including PAAK
and POSA. Additionally, PACE agent-scene pairs are quantitatively rated as
more physically plausible than existing methods.

Limitations and Future Work. Note that PACE does not always create natural
placements. There are still cases where a given motion is simply too long or
large to fit in a given environment. Additionally, if there is no semantic
match the placement can look unnatural. For example, a sitting motion will not
look right in a scene without any chairs. PACE is currently limited to static
3D scenes, although future work can explore the use of dynamic scenes. Our
optimization method can sometimes miss good placements as it relies on a grid
of initial placements before optimizing the best ones. For example, initial
placements with heavily penalized penetrations could become the best available
with further optimization to the motion sequence. We limited this due to
processing time but more compute could enable improved agent-scene pairings
from PACE. This leads us to the largest limitation of PACE, the amount of
processing time it requires. Future work can look into placement proposal to
limit the number of initial placements PACE must attempt to optimize. This
would work similar to the way a region proposal network works for object
detection models. An additional interesting future direction would extend PACE
to model human-human interactions when placing multiple animations into a
scene. More testing with real-world AR and VR headsets is needed to determine
how users respond to the virtual agents added by PACE.

Acknowledgements. This material is based upon work supported by the National
Science Foundation Graduate Research Fellowship Program under Grant No. DGE
1840340. It is also supported by ARO Grant W911NF2110026 and U.S. Army
Cooperative Agreement W911NF2120076. Any opinions, findings, and conclusions
or recommendations expressed in this material are those of the author(s) and
do not necessarily reflect the views of these funding agencies.

## References

  * [1] D. Anguelov, P. Srinivasan, D. Koller, S. Thrun, J. Rodgers, and J. Davis.  SCAPE: Shape completion and animation of people.  ACM Transactions on Graphics, 24(3):408–416, July 2005. doi: 10 . 1145/1073204 . 1073207
  * [2] F. Argelaguet and C. Andujar.  Efficient 3d pointing selection in cluttered virtual environments.  IEEE Computer Graphics and Applications, 29(6):34–43, 2009.
  * [3] J. N. Bailenson, K. Swinth, C. Hoyt, S. Persky, A. Dimov, and J. Blascovich.  The independent and interactive effects of embodied-agent appearance and behavior on self-report, cognitive, and behavioral markers of copresence in immersive virtual environments.  Presence, 14(4):379–393, 2005.
  * [4] Carnegie Mellon University.  CMU MoCap Dataset.
  * [5] E. R. Chan, C. Z. Lin, M. A. Chan, K. Nagano, B. Pan, S. D. Mello, O. Gallo, L. Guibas, J. Tremblay, S. Khamis, T. Karras, and G. Wetzstein.  Efficient geometry-aware 3D generative adversarial networks.  In arXiv, 2021.
  * [6] A. Chang, A. Dai, T. Funkhouser, M. Halber, M. Niessner, M. Savva, S. Song, A. Zeng, and Y. Zhang.  Matterport3d: Learning from rgb-d data in indoor environments.  International Conference on 3D Vision (3DV), 2017.
  * [7] A. Clegg, W. Yu, J. Tan, C. K. Liu, and G. Turk.  Learning to dress: Synthesizing human dressing motion via deep reinforcement learning.  ACM Transactions on Graphics, 37(6):179:1–179:10, Dec. 2018. doi: 10 . 1145/3272127 . 3275048
  * [8] S. Cruz, W. Hutchcroft, Y. Li, N. Khosravan, I. Boyadzhiev, and S. B. Kang.  Zillow indoor dataset: Annotated floor plans with 360º panoramas and 3d room layouts.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2133–2143, June 2021.
  * [9] A. Dai, A. X. Chang, M. Savva, M. Halber, T. Funkhouser, and M. Nießner.  Scannet: Richly-annotated 3d reconstructions of indoor scenes.  In Proc. Computer Vision and Pattern Recognition (CVPR), IEEE, 2017\.
  * [10] H. Fu, B. Cai, L. Gao, L.-X. Zhang, J. Wang, C. Li, Q. Zeng, C. Sun, R. Jia, B. Zhao, et al.  3d-front: 3d furnished rooms with layouts and semantics.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10933–10942, 2021.
  * [11] M. Gleicher.  Retargetting motion to new characters.  In Proceedings of the 25th Annual Conference on Computer Graphics and Interactive Techniques - SIGGRAPH ’98, pp. 33–42. ACM Press, Not Known, 1998. doi: 10 . 1145/280814 . 280820
  * [12] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio.  Generative Adversarial Nets.  In Advances in Neural Information Processing Systems, vol. 27. Curran Associates, Inc., 2014.
  * [13] H. Grabner, J. Gall, and L. Van Gool.  What makes a chair a chair?  In CVPR 2011, pp. 1529–1536, June 2011. doi: 10 . 1109/CVPR . 2011 . 5995327
  * [14] J. Gu, L. Liu, P. Wang, and C. Theobalt.  STYLENERF: A STYLE-BASED 3D-AWARE GENERA- TOR FOR HIGH-RESOLUTION IMAGE SYNTHESIS.  ICLR, p. 25, 2022.
  * [15] A. Gupta, S. Satkin, A. A. Efros, and M. Hebert.  From 3D scene geometry to human workspace.  In CVPR 2011, pp. 1961–1968. IEEE, Colorado Springs, CO, USA, June 2011. doi: 10 . 1109/CVPR . 2011 . 5995448
  * [16] F. G. Harvey, M. Yurick, D. Nowrouzezahrai, and C. Pal.  Robust motion in-betweening.  ACM Transactions on Graphics, 39(4):60:60:1–60:60:12, July 2020\. doi: 10 . 1145/3386569 . 3392480
  * [17] M. Hassan, V. Choutas, D. Tzionas, and M. Black.  Resolving 3D Human Pose Ambiguities With 3D Scene Constraints.  In 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 2282–2292. IEEE, Seoul, Korea (South), Oct. 2019\. doi: 10 . 1109/ICCV . 2019 . 00237
  * [18] M. Hassan, P. Ghosh, J. Tesch, D. Tzionas, and M. J. Black.  Populating 3D Scenes by Learning Human-Scene Interaction.  In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 14703–14713. IEEE, Nashville, TN, USA, June 2021. doi: 10 . 1109/CVPR46437 . 2021 . 01447
  * [19] E. S. L. Ho, T. Komura, and C.-L. Tai.  Spatial relationship preserving character motion adaptation.  ACM Transactions on Graphics, 29(4):33:1–33:8, July 2010. doi: 10 . 1145/1778765 . 1778770
  * [20] D. Holden, J. Saito, and T. Komura.  A deep learning framework for character motion synthesis and editing.  ACM Transactions on Graphics, 35(4):138:1–138:11, July 2016. doi: 10 . 1145/2897824 . 2925975
  * [21] C. Ionescu, D. Papava, V. Olaru, and C. Sminchisescu.  Human3.6M: Large Scale Datasets and Predictive Methods for 3D Human Sensing in Natural Environments.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 36(7):1325–1339, July 2014. doi: 10 . 1109/TPAMI . 2013 . 248
  * [22] Y. Jiang, H. Koppula, and A. Saxena.  Hallucinated Humans as the Hidden Context for Labeling 3D Scenes.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2993–3000, 2013.
  * [23] H. Joo, T. Simon, and Y. Sheikh.  Total Capture: A 3D Deformation Model for Tracking Faces, Hands, and Bodies.  In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8320–8329. IEEE, Salt Lake City, UT, USA, June 2018. doi: 10 . 1109/CVPR . 2018 . 00868
  * [24] C. Kang and S.-H. Lee.  Environment-Adaptive Contact Poses for Virtual Characters.  Computer Graphics Forum, 33(7):1–10, 2014. doi: 10 . 1111/cgf . 12468
  * [25] V. G. Kim, S. Chaudhuri, L. Guibas, and T. Funkhouser.  Shape2Pose: Human-centric shape analysis.  ACM Transactions on Graphics, 33(4):120:1–120:12, July 2014. doi: 10 . 1145/2601097 . 2601117
  * [26] H. S. Koppula, R. Gupta, and A. Saxena.  Learning human activities and object affordances from RGB-D videos.  International Journal of Robotics Research, 32(8):951–970, July 2013. doi: 10 . 1177/0278364913478446
  * [27] L. Kovar and M. Gleicher.  Flexible automatic motion blending with registration curves.  In Proceedings of the 2003 ACM SIGGRAPH/Eurographics Symposium on Computer Animation, SCA ’03, pp. 214–224. Eurographics Association, Goslar, DEU, July 2003.
  * [28] Y. Kwon, D. Kim, D. Ceylan, and H. Fuchs.  Neural Human Performer: Learning Generalizable Radiance Fields for Human Performance Rendering.  In Advances in Neural Information Processing Systems, vol. 34, pp. 24741–24752. Curran Associates, Inc., 2021.
  * [29] Y. Lang, W. Liang, and L.-F. Yu.  Virtual Agent Positioning Driven by Scene Semantics in Mixed Reality.  In 2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR), pp. 767–775, Mar. 2019. doi: 10 . 1109/VR . 2019 . 8798018
  * [30] K. Leimer, A. Winkler, S. Ohrhallinger, and P. Musialski.  Pose to Seat: Automated design of body-supporting surfaces.  Computer Aided Geometric Design, 79:101855, May 2020. doi: 10 . 1016/j . cagd . 2020 . 101855
  * [31] S. Lessels and R. A. Ruddle.  Movement around real and virtual cluttered environments.  Presence: Teleoperators & Virtual Environments, 14(5):580–596, 2005.
  * [32] X. Li, S. Liu, K. Kim, X. Wang, M.-H. Yang, and J. Kautz.  Putting Humans in a Scene: Learning Affordance in 3D Indoor Environments.  In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 12360–12368. IEEE, Long Beach, CA, USA, June 2019. doi: 10 . 1109/CVPR . 2019 . 01265
  * [33] J. Lin, X. Guo, J. Shao, C. Jiang, Y. Zhu, and S.-C. Zhu.  A virtual reality platform for dynamic human-scene interaction.  In SIGGRAPH ASIA 2016 Virtual Reality Meets Physical Reality: Modelling and Simulating Virtual Humans and Environments, SA ’16, pp. 1–4. Association for Computing Machinery, New York, NY, USA, Nov. 2016. doi: 10 . 1145/2992138 . 2992144
  * [34] H. Y. Ling, F. Zinno, G. Cheng, and M. Van De Panne.  Character controllers using motion VAEs.  ACM Transactions on Graphics, 39(4):40:40:1–40:40:12, July 2020\. doi: 10 . 1145/3386569 . 3392422
  * [35] M. Loper, N. Mahmood, J. Romero, G. Pons-Moll, and M. J. Black.  SMPL: A skinned multi-person linear model.  ACM Transactions on Graphics, 34(6):1–16, Nov. 2015. doi: 10 . 1145/2816795 . 2818013
  * [36] N. Mahmood, N. Ghorbani, N. F. Troje, G. Pons-Moll, and M. J. Black.  AMASS: Archive of motion capture as surface shapes.  In International Conference on Computer Vision, pp. 5442–5451, Oct. 2019.
  * [37] J. Martin.  How to Make Immersive Game Design | University of Silicon Valley.  https://usv.edu/blog/how-to-make-immersive-game-design/.
  * [38] B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng.  NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis.  ECCV, p. 17, 2020.
  * [39] J. F. Mullen Jr, D. Kothandaraman, A. Bera, and D. Manocha.  Placing Human Animations into 3D Scenes by Learning Interaction- and Geometry-Driven Keyframes, Sept. 2022. doi: 10 . 48550/arXiv . 2209 . 06314
  * [40] S. Narang, A. Best, and D. Manocha.  Simulating movement interactions between avatars & agents in virtual worlds using human motion constraints.  In 2018 IEEE Conference on Virtual Reality and 3D User Interfaces (VR), pp. 9–16. IEEE, 2018.
  * [41] M. Niemeyer and A. Geiger.  GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fields.  In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 11448–11459. IEEE, Nashville, TN, USA, June 2021. doi: 10 . 1109/CVPR46437 . 2021 . 01129
  * [42] A. Noguchi, X. Sun, S. Lin, and T. Harada.  Unsupervised Learning of Efficient Geometry-Aware Neural Articulated Representations, Apr. 2022.
  * [43] A. Noguchi, S. Xiao, S. Lin, and T. Harada.  Neural Articulated Radiance Field.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 5762–5772, Oct. 2021.
  * [44] A. A. A. Osman, T. Bolkart, and M. J. Black.  STAR: Sparse Trained Articulated Human Body Regressor.  In A. Vedaldi, H. Bischof, T. Brox, and J.-M. Frahm, eds., Computer Vision – ECCV 2020, vol. 12351, pp. 598–613. Springer International Publishing, Cham, 2020. doi: 10 . 1007/978-3-030-58539-6_36
  * [45] G. Pavlakos, V. Choutas, N. Ghorbani, T. Bolkart, A. A. Osman, D. Tzionas, and M. J. Black.  Expressive Body Capture: 3D Hands, Face, and Body From a Single Image.  In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10967–10977. IEEE, Long Beach, CA, USA, June 2019. doi: 10 . 1109/CVPR . 2019 . 01123
  * [46] V. Pavlovic, J. M. Rehg, and J. MacCormick.  Learning Switching Linear Models of Human Motion.  In Advances in Neural Information Processing Systems, vol. 13. MIT Press, 2000.
  * [47] S. Peng, J. Dong, Q. Wang, S. Zhang, Q. Shuai, X. Zhou, and H. Bao.  Animatable Neural Radiance Fields for Modeling Dynamic Human Bodies.  In 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 14294–14303. IEEE, Montreal, QC, Canada, Oct. 2021\. doi: 10 . 1109/ICCV48922 . 2021 . 01405
  * [48] D. Rempe, T. Birdal, A. Hertzmann, J. Yang, S. Sridhar, and L. J. Guibas.  HuMoR: 3D Human Motion Model for Robust Pose Estimation.  In 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 11468–11479. IEEE, Montreal, QC, Canada, Oct. 2021\. doi: 10 . 1109/ICCV48922 . 2021 . 01129
  * [49] R. A. Ruddle.  The effect of environment characteristics and user interaction on levels of virtual environment sickness.  In IEEE Virtual Reality 2004, pp. 141–285. IEEE, 2004.
  * [50] R. A. Ruddle and D. M. Jones.  Movement in cluttered virtual environments.  Presence, 10(5):511–524, 2001.
  * [51] M. Savva, A. X. Chang, P. Hanrahan, M. Fisher, and M. Nießner.  SceneGrok: Inferring action maps in 3D environments.  ACM Transactions on Graphics, 33(6):212:1–212:10, Nov. 2014. doi: 10 . 1145/2661229 . 2661230
  * [52] J. Shen, J. Dudley, and P. O. Kristensson.  Simulating realistic human motion trajectories of mid-air gesture typing.  In 2021 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), pp. 393–402. IEEE, 2021.
  * [53] L. Sigal, A. O. Balan, and M. J. Black.  HumanEva: Synchronized Video and Motion Capture Dataset and Baseline Algorithm for Evaluation of Articulated Human Motion.  International Journal of Computer Vision, 87(1-2):4–27, Mar. 2010\. doi: 10 . 1007/s11263-009-0273-6
  * [54] A. L. Simeone, I. Mavridou, and W. Powell.  Altering user movement behaviour in virtual environments.  IEEE transactions on visualization and computer graphics, 23(4):1312–1321, 2017.
  * [55] K. Sohn, H. Lee, and X. Yan.  Learning Structured Output Representation using Deep Conditional Generative Models.  In Advances in Neural Information Processing Systems, vol. 28. Curran Associates, Inc., 2015.
  * [56] S. Starke, H. Zhang, T. Komura, and J. Saito.  Neural state machine for character-scene interactions.  ACM Transactions on Graphics, 38(6):209:1–209:14, Nov. 2019. doi: 10 . 1145/3355089 . 3356505
  * [57] S.-Y. Su, F. Yu, M. Zollhöfer, and H. Rhodin.  A-NeRF: Articulated neural radiance fields for learning human shape, appearance, and pose.  In Advances in Neural Information Processing Systems, 2021.
  * [58] F. Tan, C. Bernier, B. Cohen, V. Ordonez, and C. Barnes.  Where and who? Automatic semantic-aware person composition.  In IEEE Winter Conf. on Applications of Computer Vision (WACV), 2018.
  * [59] R. Urtasun, D. J. Fleet, A. Geiger, J. Popović, T. J. Darrell, and N. D. Lawrence.  Topologically-constrained latent variable models.  In Proceedings of the 25th International Conference on Machine Learning - ICML ’08, pp. 1080–1087. ACM Press, Helsinki, Finland, 2008. doi: 10 . 1145/1390156 . 1390292
  * [60] J. Wang, Y. Rong, J. Liu, S. Yan, D. Lin, and B. Dai.  Towards Diverse and Natural Scene-Aware 3D Human Motion Synthesis.  CVPR, p. 10, 2022.
  * [61] J. Wang, H. Xu, J. Xu, S. Liu, and X. Wang.  Synthesizing Long-Term 3D Human Motion and Interaction in 3D Scenes.  In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9396–9406. IEEE, Nashville, TN, USA, June 2021. doi: 10 . 1109/CVPR46437 . 2021 . 00928
  * [62] Z. Wang, L. Chen, S. Rathore, D. Shin, and C. Fowlkes.  Geometric Pose Affordance: 3D Human Pose with Scene Constraints, Dec. 2021.
  * [63] Z. Wang, A. Novikov, K. Żołna, J. T. Springenberg, S. Reed, B. Shahriari, N. Siegel, J. Merel, C. Gulcehre, N. Heess, and N. de Freitas.  Critic regularized regression.  In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS’20, pp. 7768–7778. Curran Associates Inc., Red Hook, NY, USA, Dec. 2020.
  * [64] F. Xia, A. R. Zamir, Z.-Y. He, A. Sax, J. Malik, and S. Savarese.  Gibson env: real-world perception for embodied agents.  In Computer Vision and Pattern Recognition (CVPR), 2018 IEEE Conference on. IEEE, 2018.
  * [65] G. Xia, H. Sun, Q. Liu, and R. Hang.  Learning-Based Sphere Nonlinear Interpolation for Motion Synthesis.  IEEE Transactions on Industrial Informatics, 2019. doi: 10 . 1109/TII . 2019 . 2894113
  * [66] J. Xu, H. Xu, B. Ni, X. Yang, X. Wang, and T. Darrell.  Hierarchical Style-based Networks for Motion Synthesis.  ECCV, p. 16, 2020.
  * [67] G. Yang, D. Sun, V. Jampani, D. Vlasic, F. Cole, C. Liu, and D. Ramanan.  ViSER: Video-Specific Surface Embeddings for Articulated 3D Shape Reconstruction.  In Advances in Neural Information Processing Systems, vol. 34, pp. 19326–19338. Curran Associates, Inc., 2021.
  * [68] G. Yang, M. Vo, N. Neverova, D. Ramanan, A. Vedaldi, and H. Joo.  BANMo: Building Animatable 3D Neural Models From Many Casual Videos.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2863–2873, 2022.
  * [69] Z.-M. Ye, J.-L. Chen, M. Wang, and Y.-L. Yang.  PAVAL: Position-Aware Virtual Agent Locomotion for Assisted Virtual Reality Navigation.  In 2021 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), pp. 239–247, Oct. 2021. doi: 10 . 1109/ISMAR52148 . 2021 . 00039
  * [70] S. Zhang, Y. Zhang, Q. Ma, M. J. Black, and S. Tang.  PLACE: Proximity Learning of Articulation and Contact in 3D Environments.  In 2020 International Conference on 3D Vision (3DV), pp. 642–651. IEEE, Fukuoka, Japan, Nov. 2020. doi: 10 . 1109/3DV50981 . 2020 . 00074
  * [71] Y. Zhang, M. Hassan, H. Neumann, M. J. Black, and S. Tang.  Generating 3D People in Scenes Without People.  In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6193–6203. IEEE, Seattle, WA, USA, June 2020. doi: 10 . 1109/CVPR42600 . 2020 . 00623
  * [72] F. Zhao, W. Yang, J. Zhang, P. Lin, Y. Zhang, J. Yu, and L. Xu.  HumanNeRF: Efficiently Generated Human Radiance Field From Sparse Inputs.  CVPR, p. 11, 2022.