# Zero-Shot Learning for the Primitives of 3D Affordance in General Objects

Hyeonwoo Kim1 Sookwan Han111footnotemark: 1 Patrick Kwon2 Hanbyul Joo1 and
1Seoul National University 2Naver Webtoon AI
https://sshowbiz.github.io/ZSP3A/ Equal contribution

###### Abstract

One of the major challenges in AI is teaching machines to precisely respond
and utilize environmental functionalities, thereby achieving the affordance
awareness that humans possess. Despite its importance, the field has been
lagging in terms of learning, especially in 3D, as annotating affordance
accompanies a laborious process due to the numerous variations of human-object
interaction. The low availability of affordance data limits the learning in
terms of generalization for object categories, and also simplifies the
representation of affordance, capturing only a fraction of the entire
affordance. To overcome these challenges, we propose a novel, self-supervised
method to “generate” the 3D affordance examples given only a 3D object input,
without any manual annotating procedures. The method starts by capturing the
3D object into images and creating 2D affordance examples by inserting humans
into the image via inpainting diffusion models, where we present the Adaptive
Mask algorithm to enable human insertion without altering the original details
of the object. The method consequently lifts inserted humans back to 3D to
create 3D human-object pairs, where the depth ambiguity is resolved within a
depth optimization framework that utilizes pre-generated human postures from
multiple viewpoints. We also provide a novel affordance representation defined
on relative orientations and proximity between dense human and object points,
that can be easily aggregated from any 3D HOI datasets. The proposed
representation serves as a primitive that can be manifested to conventional
affordance representations via simple transformations, ranging from physically
exerted affordances (e.g., contact) to nonphysical ones (e.g., orientation
tendency, spatial relations). We demonstrate the efficacy of our method and
representation by generating the 3D affordance samples and deriving high-
quality affordance examples from the representation, including contact,
orientation, and spatial occupancies.

Figure 1: Given any 3D object, our method can generate 3D affordance examples
involving human-object interactions. By aggregating this dataset into our
novel representation, we obtain a unified knowledge of affordance that can be
transformed into diverse affordance cues, such as contact, orientation
tendency, and spatial relations.

##  1 Introduction

The unique ability of humans is to perceive the functionalities provided by
the environment and to efficiently utilize them; a concept known as affordance
[21]. For example, we know what functions a cup provides, set goals
responsively during interactions, and precisely control our hands to pick up
and use the cup. Such ability allows us to use complex tools in various ways,
serving as a major driving factor for our evolution of intelligence. As it is
a crucial component of human intelligence, there have been many tries to
implement such features in AI and robotics.

Previous studies [94, 89, 92, 96, 25, 24, 11, 107, 30, 16, 35] focus on
learning human and object interaction from 2D or 3D dataset in a supervised
manner. While these methods provide opportunities to effectively mimic the
interactions between humans and environments, such supervised approaches often
suffer from a lack of ability to support unseen categories, consequently
failing to be generalized to the real world’s diverse interactions.
Additionally, the need for laborious annotations or data collection procedures
makes it hard to scale, limiting the representation of affordance to simple
aspects such as expressing affordance categories [6, 32, 47] (e.g., sitting,
grasping) or contact heatmap [1, 49] which is just a part of affordance. Thus,
the current representation does not completely capture the implicit effects of
affordance on the distribution of the agent’s response, such as the
orientation tendency of body parts (e.g., face), proximity relations, etc.,
induced to require various criteria (e.g., physical stability).

Henceforth, we propose a novel method to extract affordance in a zero-shot
manner without requiring any manual annotations. Specifically, we address the
task of inferring affordance distributions from any 3D objects given as mesh;
with its first step generating the samples of interacting 3D human and object
pairs. We enable this by presenting a novel method, leveraging inpainting
diffusion models [75] to insert affordance-aware humans into the renderings of
the 3D asset (Sec. 3.2). By creating affordance examples in the relatively
“unchallenging” domain of 2D, we can generate plausible samples of affordance
without heavy 3D supervision; even with no additional training in 2D as we use
pretrained models. However, the challenge exists as directly leveraging the
inpainting diffusion model will result in altering the original object, which
means the generated affordance is simply a hallucination from a different
object. To mitigate this, we propose Adaptive Mask Inpainting to preserve the
details of the original object and insert humans interacting with the exact
target object, rather than hallucinating different ones. We then uplift the
predicted humans in 2D back to the 3D space (Sec. 3.3) to generate the 3D
affordance samples from 2D images with human pose and shape regresser, where
we aim to use for learning 3D affordance. On its path, we also offer a
framework that leverages generated 2D affordance samples as depth cues to
resolve the inherent depth ambiguity.

Importantly, we also propose a novel affordance representation (Sec. 4)
designed to model the prevalent and plausible 3D spatial relationship between
a human and an object during interactions. Different from the prior approaches
that primarily focus on the 3D contact regions where direct interactions occur
[28, 89, 96], our affordance representation encompasses the entire spectrum of
relationships between all human parts and object regions, not only the
physical contact. Our representation is based on the observation that each
part of the human body has a specific set of possible actions in interacting
with each part of objects. For example, when we use a laptop, such actions
entail not only finger-to-keyboard touch but also involve typical 3D postures,
and relative distances of various body parts, including the face and arms.
Specifically, our representation encapsulates the relative proximity and
orientations between each possible pair of object surface points and human
points from generated 3D affordance samples as a probability distribution.

To summarize, our method contributes novel strategies to generate the 3D
affordance samples from the given input 3D object, including: (1) adaptive
mask inpainting for inserting humans into scenes without harming the original
context of the object, and (2) a framework for resolving depth ambiguity of 3D
human by leveraging depth cue from synthesized human postures of various
viewpoints. Also, we present a (3) novel primitive representation for 3D
affordance that captures the full spectrum of relationships among all pairs of
human and object points, where we demonstrate efficacy by deriving contact,
orientation tendency, and spatial relations.

##  2 Related Work

Data Synthesis for Learning. There have been approaches to supplement data in
fields lacking efficient annotation progress by leveraging generative models,
irrespective of the research area. Many methods employ GANs [22, 42] or
diffusion models [75, 34, 83] to augment data [88, 31] for various vision
tasks, including perception and representation learning [91], classification
[9, 56, 87], segmentation [58, 90, 104], dense visual alignment [70], and
further extending to 3D tasks, such as neural rendering [27, 102], shape
reconstruction [68], and so forth. The major challenge in leveraging
synthesized data (or generative models for cues) is controllability, as the
generator, even when conditioned, typically produces “free” samples. Ali et
al. [38] presents a method to control the viewpoint of the synthesized image
by modeling the viewpoint-free latent space. CHORUS [26] applies cascaded
filtering to the generated dataset to learn 3D human-object spatial relations,
similar to our approach.

Learning Visual Affordance. Affordance was introduced by Gibson [21] as a set
of functionalities which environment furnishes to an agent (e.g., human,
animal). The concept extends to the vision and robotics area, where the focus
is on teaching embodied agents to interact with scenes [1, 64, 19, 48, 93],
mimicking human-object [17, 2, 36, 73, 101, 13, 62, 8, 50, 84, 41, 77, 100,
71, 29], or hand-object [3, 4, 5, 86, 105] interactions. Earlier methods
focused on learning action category labels [6, 32, 47] with bounding boxes but
lack a full description of affordance. Some studies represent affordance with
contact regions and heatmaps on objects [1, 49, 96] or humans [89, 103, 29].
Other than the challenges in representation, annotating affordance, especially
in 3D, is challenging due to numerous variations. Previous approaches provide
3D annotations via multi-camera system [2, 37], motion capture [28], and
manual process [39, 89, 96, 14]; however, they are limited in terms of
categories, thus the model lacks generalization after training. Other
approaches train generative models to create affordance samples in 2D [97, 46]
or 3D human action [94].

Diffusion Model for Synthesizing Affordance Images. While diffusion models
[82, 34, 83] excel at generating realistic images [75, 15], few address the
generation of affordance images (i.e., inserting humans into images).
Inpainting models [75, 54], while a common choice for human insertion, often
compromise scene details, resulting in inconsistent affordance images. Image
editing techniques [59, 33, 61] may offer an alternative but tend to
prioritize style and struggle with creating new geometry. Affordance diffusion
[97] requires an additional layout network to specify the inpainting region
before inserting a hand into the image, and affordance insertion [46] relies
on a diffusion model using human images and masks as input, requiring
additional training and limiting applicability.

##  3 3D Affordance Sample Generation

Figure 2:  3D Affordance Sample Generation Method Overview. The 3D affordane
sample generation phase begins by rendering the given 3D object from multiple
views. The method leverages inpainting diffusion models to insert humans into
rendered object images, creating 2D samples depicting affordance. The method
applies Adaptive Mask Inpainting to enable human insertion without harming
original details of the object. The method then lifts the 2D humans into 3D
through single view 3D human prediction and depth optimization using human
cues extracted from generated 2D affordance samples which originally rendered
in various views.

###  3.1 Overview

The goal of our method is to infer the affordance distributions
$\mathcal{P}=\\{\mathcal{P}_{ij}\\}$ (defined in Sec. 4) for 3D object meshes
from general sources (e.g., Internet) or text-to-3d models [72, 51, 12],
without additional annotations. To this end, we “generate” the 3D affordance
samples, and consequently use them for affordance distribution learning.

Given an input object mesh $\mathbf{M}^{o}_{\text{input}}$, our 3D affordance
sample generation phase aims to create diverse pairs of 3D humans and objects
depicting the type of affordance:

$\mathcal{D}=\\{\mathbf{M}^{o}_{d},\mathbf{M}^{h}_{d}\\}_{d=1}^{D}.$ (1)

A $d$-th object sample $\mathbf{M}^{o}_{d}$ is assumed rigid and parametrized
as $\\{\mathbf{M}^{o}_{\text{input}},\mathbf{R}^{o}_{d},\mathbf{t}^{o}_{d}\\}$
where $\mathbf{t}^{o}_{d}\in\mathbb{R}^{3}$ is the translation and
$\mathbf{R}^{o}_{d}\in\text{SO}(3)$ is the rotation applied to the input mesh
$\mathbf{M}^{o}_{\text{input}}$. The paired $d$-th human is parametrized as
$\mathbf{M}^{h}_{d}=\\{\theta_{d},\beta_{d},\mathbf{R}^{h}_{d},\mathbf{t}^{h}_{d},\mathbf{s}^{h}_{d}\\}$,
where $\theta_{d}\in\mathbb{R}^{54\times 3}$ and $\beta_{d}\in\mathbb{R}^{10}$
are the pose and shape parameters of SMPL-X [69],
$\mathbf{t}^{h}_{d}\in\mathbb{R}^{3}$ is translation,
$\mathbf{R}^{h}_{d}\in\text{SO}(3)$ is the global rotation, and
$\mathbf{s}^{h}_{d}\in\mathbb{R}$ is scale. Note that we use explicit mesh
representation to preserve the surface point orderings
$\\{\mathbf{v}_{i}^{o}\\}_{i=1}^{N_{o}}$,
$\\{\mathbf{v}_{j}^{h}\\}_{j=1}^{N_{h}}$ (used in Sec. 4), where our
affordance distribution is learned.

Our 3D affordance sample generation process can be divided into two steps: (1)
2D affordance image generation, and (2) 2D-to-3D uplifting. The core idea for
the first step is to leverage vision-language models [67] and inpainting
diffusion models [75] on 2D renderings of input mesh
$\mathbf{M}^{o}_{\text{input}}$ to create diverse examples of 2D affordance
from multiple viewpoints. To effectively insert humans into object renderings
without altering the original details of the appearance, we present Adaptive
Mask Inpainting, a method that can be applied to any inpainting diffusion
models. In a nutshell, the method specifies the inpainting region over
diffusion timesteps, enforcing the model to focus on generating within the
adapted mask region and preserving the details outside.

The second step, 2D-to-3D uplifting, focuses on generating 3D samples from the
2D images of humans and objects. Specifically, we first predict the 3D humans
from single view images using off-the-shelf [63] 3D human prediction models.
To resolve the remaining depth ambiguity of 3D human, we present a novel
framework that enables figuring out depth using human cues obtained from
generated 2D affordance samples. Refer to Fig. 2 for the overview, and Supp.
Mat. for additional details.

Algorithm 1 Adaptive Mask Inpainting

Latent Diffusion Model: $\epsilon_{\theta}$

Latent VAE Decoder: $\mathbf{D}$

Segmentation Model: $\mathbf{S}$

DDIMSchedule: $\\{\alpha_{t}\\}_{t=1}^{T}$

Inputs: Prompt ($c$), Initial Mask ($m_{\text{default}}$), Image
($I_{\text{orig}}$)

Initialize Noise Latent: $x_{T}\sim\mathcal{N}(\mathbf{0},\mathbf{I})$

Initialize Adaptive Mask: $m_{T}\xleftarrow[]{}m_{\text{default}}$

for $t$=$T,...,1$ do

$\hat{x}_{0}={1\over\sqrt{\bar{\alpha}_{t}}}(x_{t}-\sqrt{1-\bar{\alpha}_{t}}\epsilon_{\theta}(x_{t};c,m_{t},I_{\text{orig}},t))$

if $t\in\text{ProvokeSchedule}$ then

$s=\mathbf{S}(\mathbf{D}(\hat{x}_{0}))$

if $s\neq\varnothing$ then

$m_{t-1}=\text{Dilate}(s)$

else

$m_{t-1}=m_{\text{default}}$

end if

else

$m_{t-1}=m_{t}$

end if

$x_{t-1}=\text{DDIMStep}(x_{t},\hat{x}_{0},t)$

end for

return $\mathbf{D}(x_{0})$

###  3.2 Affordance Image Generation

Rendering Object from Multi-Viewpoints. Given a 3D object
$\mathbf{M}_{\text{input}}^{o}$, we first render it from multiple viewpoints.
We place the object and camera differently for two object types: static and
dynamic. For objects that are presumed to be fixed on the ground during
interaction (e.g., chair), we fix the object on the ground plane, whereas
objects with dynamic pose during interaction (e.g., cup) are perturbed with
sampled rotation and translation. We empirically find that differentiating the
rendering process per object type improves inpainting quality when using
diffusion models as they inherently possess spatial bias for a given object.
Note that we automate this process by rendering the object at any direction
first and inferring the object type via vision-language model [67]. For
rendering, we use weak perspective cameras $\\{\Pi_{c}\\}_{c=1}^{N}$ installed
around the object with equal interval azimuth and same elevation, where $N$ is
set as 8 for static, 40 for dynamic objects.

Inpainting Mask Selection. After rendering the images, we select the masks on
which we inpaint a human interacting with the object. We move a sliding window
and select the windows if the intersection over the object segmentation mask
satisfies a certain threshold.

Figure 3: Adaptive Mask Inpainting. Without adaptive mask inpainting, the
original object is damaged when inserting humans.

Prompt Generation. Inspired by CHORUS [26], we automatically generate the
prompts that describe the possible interactions with humans and the given
object, which are taken as guidance for inpainting. Specifically, we utilize
the vision-language model [67] to infer the HOI prompt using the rendered
object image and predefined template. We generate 3 prompts per given object.
See Supp. Mat. for additional details.

Adaptive Mask Inpainting for Human Insertion. We synthesize 2D images of
human-object interaction by inpainting the human into rendered object images
using publicly available text-conditional inpainting diffusion models [75].
However, the challenge arises as the object geometry and details within the
mask region are not preserved during inpainting, raising the odds of creating
hallucinations of affordance. To mitigate this problem, we present Adaptive
Mask Inpainting to progressively specify the inpainting region over diffusion
timesteps. Let the sequence of denoised image latent be
$\\{x_{t}\\}_{t=T}^{0}$, where $x_{T}$ is the fully random noise and $x_{0}$
is the denoised latent. Inspired by the observation that the quality of the
predicted denoised image $\hat{x}_{0}$ improves over progress of timestep, and
creates a low-level structure of the target prompt (in this case, human) even
at the early steps as shown in Fig. 3, we aim to adapt the inpainting region
$m_{t}$ over timesteps by spatially discovering the mask from the low-level
structure. Specifically, we apply PointRend [44] to decoded latent
$\mathbf{D}(\hat{x}_{0})$ at specific timesteps to predict the human mask,
consequently use as the mask for the next denoising step. We find our approach
better preserves the target objects as demonstrated in our experiments. Note
that we dilate the predicted human mask to tolerate the imperfectness of the
generated structure during early steps. Refer to Algorithm. 1 for the full
pipeline.

###  3.3 Lifting 2D Affordance to 3D

From the previous section, we obtain a set of affordance images
$\\{I_{d}\\}_{d=1}^{D}$. Even though we have multiple views with geometrically
consistent object renderings, the inserted humans have diversity without
multi-view consistency, and thus it is non-trivial to directly lift 3D humans
from these 2D images. To address this, we first utilize a single view 3D human
prediction model to obtain a corresponding 3D human pose and shape of the
image, and then estimate the depth of the human with weak auxiliary cue
provided by the set of affordance images, generating a 3D affordance sample
corresponds to the image.

Single View 3D Human Prediction. To uplift the 2D affordance images into 3D,
we first predict the pose and shape of 3D humans from the images. We apply
off-the-shelf 3D human prediction model [63] (denoted as
$\mathbf{F}_{\text{human}}$) to predict SMPL-X humans [69] from generated
images:

$\\{\theta_{d},\beta_{d},\mathbf{j}_{d},\mathbf{R}_{d}^{h},\mathbf{s}_{d}^{h},x_{d},y_{d}\\}=\mathbf{F}_{\text{human}}(I_{d})$
(2)

, where $\theta_{d}\in\mathbb{R}^{54\times
3},\beta_{d}\in\mathbb{R}^{10},\mathbf{j}_{d}\in\mathbb{R}^{67\times 3}$ are
the predicted SMPL-X pose, shape, and joints (see Supp. Mat. for the increased
number of joints), respectively. Note that $\mathbf{j}_{d}$ is defined in
world coordinate shared with weak perspective cameras, obtained by
transforming joints with camera rotation and translation.
$\mathbf{R}_{d}^{h}\in\text{SO}(3)$ is global rotation, and the
$\mathbf{s}_{d}^{h},x_{d},y_{d}$ are scale, and $x,y$ direction offsets
representing the parameters of weak-perspective camera $\Pi_{d}$ assigned to
image $I_{d}$. As 3D human lies within the weak perspective camera framework,
there remains depth ambiguity to solve.

Figure 4: Leveraging Inpainted Images for Specifying Depth. From inconsistent
2D affordance images, we apply RANSAC [18] to figure out largest inlier set
which is semi-consistent with reference view image. The inlier set are used to
resolve depth ambiguity.

Depth Optimization using Weak Auxiliary Cue To address the depth ambiguity of
3D human obtained from reference image, we leverage 2D affordance images
generated in advance. Since the human pose in images are geometrically
inconsistent, we select the largest subset inliers $\mathcal{I}$ among them
which are “semi-consistent” with the reference image by applying RANSAC [18]
in terms of joint re-projection error. The joint triangulation is performed on
two images, including the reference image. Note that we generate sufficiently
large amount of 2D affordance images to ensure the existence of the inlier
set. The inliers $\mathcal{I}$ is used for specifying depth $z_{d}$, as shown
in Fig. 4 via depth optimization by minimizing the joint re-projection loss
defined below:

$\mathcal{L}_{\text{re-
projection}}={1\over|\mathcal{I}|}\sum_{I_{d^{\prime}}\in\mathcal{I}}\|{\Pi_{d^{\prime}}}(\mathbf{j}_{d}+z_{d}\bm{f}_{d})-{\Pi_{d^{\prime}}}(\mathbf{j}_{d^{\prime}})\|^{2}$
(3)

where ${\Pi_{d^{\prime}}},\mathbf{j}_{d^{\prime}}$ denotes the weak
perspective projection and predicted 3D joints in world coordinate w.r.t. the
inlier image $I_{d^{\prime}}$, and $\bm{f}_{d}\in\mathbb{R}^{3}$ is the
normalized direction of the camera $\Pi_{d}$. For initial depth before
optimization, we leverage occlusion cues which human segmentation provides as
a depth ordering between human and object, inspired by Jiang et al. [40].
Around the object location, we initialize equispaced 7 SMPL-X along the camera
direction and select the one with the best IoU between the rendered mask
(regarding occlusion) and segmentation from the image. The initialization is
helpful in the case when optimal position of object is between two human body
parts along the camera direction (e.g., body of motorcycle is placed between
human legs while riding), which collision loss (which will be explained next)
often prevents human from crossing the object while optimization. See Supp.
Mat. for additional details.

We also leverage a collision cue to estimate fine-grained depth, considering
physical plausibility. We adopt collision loss from COAP [60] where the human
body is represented as SDF [65] field. We set object vertices as query points
to reduce the collision between human and object, namely
$\mathcal{L}_{\text{collision}}$. We optimize depth $z_{d}$ to minimize the
total loss:

$\mathcal{L}=\mathcal{L}_{\text{re-
projection}}+\lambda_{\text{collision}}\mathcal{L}_{\text{collision}}$ (4)

Filtering. We filter out potentially low-quality 3D human samples in the cases
of: (1) human rendering (regarding occlusion) and predicted human segmentation
do not overlap much; (2) number of inliers during RANSAC is small; (3) human
and object collides significantly in 3D.

##  4 Primitive Representation of 3D Affordance

Figure 5: Visualization of Primitive Representation. For each human and object
point, the primitive is represented as probabilistic distribution of relative
normal and proximity after canonicalization.

Given an ordered 3D surface points of an object, we aim to model the pointwise
affordance distributions that encapsulates proximity and orientations of a
human surface point from human body (or vice versa, which we omit for
brevity). In practice, we use our 3D affordance samples
$\mathcal{D}=\\{\mathbf{M}^{o}_{d},\mathbf{M}^{h}_{d}\\}_{d=1}^{D}$ consists
of object mesh $\mathbf{M}^{o}_{d}$ and generated 3D human mesh
$\mathbf{M}^{h}_{d}$. For each sample in $\mathcal{D}$, we apply Poisson-disk
sampling [57] to uniformly select the points
$\\{\mathbf{v}^{o}_{i}\\}_{i=1}^{N_{o}}$ and
$\\{\mathbf{v}^{h}_{j}\\}_{j=1}^{N_{h}}$ on each surface. The density function
on the $i$-th object vertex $\mathbf{v}^{o}_{i}$ and the $j$-th human vertex
$\mathbf{v}^{h}_{j}$, which we denote as
$\mathcal{P}_{ij}(\mathbf{p},\mathbf{n})$, represents the probability (of the
occurrence) of the relative position $\mathbf{p}\in\mathbb{R}^{3}$ and
relative normal orientation $\mathbf{n}\in\mathbb{S}^{2}$ of the human point
$\mathbf{v}^{h}_{j}$ with respect to object point $\mathbf{v}^{o}_{i}$ within
the 3D affordance samples. When we compute $\mathbf{p}$ and $\mathbf{n}$, we
canonicalize the orientation of the human surface normal $\mathbf{n}^{h}_{j}$
and relative position
$\mathbf{p}^{o\xrightarrow[]{}h}=\mathbf{v}^{h}_{j}-\mathbf{v}^{o}_{i}$,
assuming the object surface normal $\mathbf{n}^{o}_{i}$ is rotated to face
$\mathbf{\hat{n}}=[0,0,1]^{T}$ as shown in Fig. 5 (details in Supp. Mat.). The
$\mathcal{P}_{ij}$ sums up to 1 for all possible $\mathbf{p}$ and
$\mathbf{n}$:

$\int_{\mathbb{R}^{3}}\int_{\mathbb{S}^{2}}\mathcal{P}_{ij}(\mathbf{p},\mathbf{n})d\mathbf{n}d\mathbf{p}=1$
(5)

In practice, we set the domain of $\mathbf{p}$ as voxelized grid and domain of
$\mathbf{n}$ as equispaced grid on $\mathbb{S}^{2}$, obtained via Fibonacci
Spirals [20], and compute the discrete probabilities by aggregating Gaussian
Kernels calculated for $\mathbf{n}$ and $\mathbf{p}$ (we use geodesic metrics
for $\mathbf{n}$). Under the probability distribution
$\mathcal{P}_{ij}(\mathbf{p},\mathbf{n})$, we define three different types of
functions $f(\mathbf{p},\mathbf{n})$ to capture different aspects of
affordances: (1) Contactual affordance, (2) Orientational affordance, (3)
Spatial affordance. Given each function $f(\mathbf{p},\mathbf{n})$ described
below, the pointwise affordance of object vertex $\mathbf{v}^{o}_{i}$ w.r.t.
human vertex $\mathbf{v}^{h}_{j}$ is determined by computing its expectation:

$\mathbb{E}_{\mathbf{p},\mathbf{n}\sim\mathcal{P}_{ij}}\,[f(\mathbf{p},\mathbf{n})]$
(6)

Contactual Affordance. From the learned primitives, we can easily infer the
contactness between arbitrary object surface point $v_{j}^{o}$ and human
surface point $v_{i}^{h}$ regarding proximity and normal alignment as:

$f_{\text{contact}}(\mathbf{p},\mathbf{n})={\left(1-\mathbf{n}\cdot\mathbf{\hat{n}}\over
2\right)}e^{-\|\mathbf{p}\|}$ (7)

where the $e^{-\|\mathbf{p}\|}$ term encourages high score when distance
$\|\mathbf{p}\|$ is near. The ${1-\mathbf{n}\cdot\mathbf{\hat{n}}\over 2}$
term, motivated by the concept of the normal force that physical interactions
are conducted via exerted normal force between contact points, encourages high
score when the normals of $v_{j}^{o}$ and $v_{i}^{h}$ align in antiparallel
direction. As shown in Fig. 6, the normal alignment term improves the
precision of the contact map, especially when small body parts (e.g.,
fingertips) interact with an object and allows more robust affordance learning
even when using noisy HOI datasets, compared to prior work that only
introduces proximity for contact terms [2, 29, 98, 103, 28] or normal for
computing only penetration [23, 95], without considering the alignment of
surface normals.

Figure 6: Effects of Normal Alignment when deriving contact map. The normal
alignment term improves precision when computing contact scores, compared to
when only proximity is considered.

Orientational Affordance. We aim to capture the pattern and tendency of the
orientation of body parts with respect to the object based on the concept of
Shannon entropy [79]. Intuitively, low entropy means there exists less
uncertainty or more typical patterns of human body orientations during the
interactions. To quantify the orientational tendency for affordance
representation, we use negated normalized Shannon entropy to enforce high
value when the human surface normal shows consistent orientational tendency
(low variance) with respect to the object surface normal:

$f_{\text{orientation}}(\mathbf{n})=1+{\log\mathcal{P}_{ij}(\mathbf{n})\over\log
n_{b}}$ (8)

where $n_{b}$ denotes the number of discretized bins of probability measure.
Intuitively, the Orientational Affordance defined by
$\mathbb{E}_{\mathbf{n}\sim\mathcal{P}_{ij}}\,[f_{\text{orientation}}(\mathbf{n})]$
has a low value when marginalized distribution $\mathcal{P}_{ij}(\mathbf{n})$
is close to the uniform distribution, meaning that there is no dominant
orientational pattern. Note that Eq. 8 is agnostic to the proximity term
$\mathbf{p}$, which allows us to model the nonphysical orientational effects
for even far-distance points as shown in Fig. 9.

Spatial Affordance. Following CHORUS [26], we also consider the 3D spatial
occupancy distribution of a human surface point with respect to the object
points by simply counting the occurrence:

$f_{\text{spatial}}(\mathbf{p})=\mathbf{1}(\mathbf{x}-\mathbf{p})$ (9)

where $\mathbf{1}(\cdot):\mathbb{R}^{3}\xrightarrow[]{}\\{0,1\\}$ is a binary
function that returns 1 if the given argument is zero vector else 0, thus the
the Spatial Affordance
$\mathbb{E}_{\mathbf{p}\sim\mathcal{P}_{ij}}\,[f_{\text{spatial}}(\mathbf{p})]$
is a function that outputs scalar occupancy for the given spatial vector
$\mathbf{x}\in\mathbb{R}^{3}$. In practice we implement the function as voxel
array, counting 1 to the voxel that contains $\mathbf{p}$. Note that
$f_{\text{spatial}}$ is agnostic to the normal direction $\mathbf{n}$ via
marginalization. Learning the occupancy informs us about macro positioning of
the human with respect to the given object, exemplified in Fig. 10.

##  5 Experiments

In Sec. 5.2, we perform quantitative evaluation to determine the strength of
our 3D affordance sample generation pipeline. We validate the effect of depth
optimization via weak auxiliary cues by comparing the generated samples with
existing 3D HOI dataset [2]. We also compare our Adaptive Mask algorithm with
original inpainting baseline to prove the scene-preservability during
inpainting. In Sec. 5.3, we qualitatively demonstrate the efficacy of our
generated dataset and our new representation by showcasing multiple examples
of different affordance types. We illustrate the generalizability of our
method by generating high-quality contact maps for various 3D objects from
various sources. Additionally, we analyze contact representations on both
human and object, aggregated on specific subsets of points, exploring the
potential for modeling dense, region-wise contact. Furthermore, we examine the
exemplified results of orientational affordance and spatial affordance.

###  5.1 Datasets

Our method generates the 3D affordance samples in a self-supervised manner. We
prepare the input 3D objects from various sources listed below.

3D Object Datasets. We sample the object meshes from various sources,
including conventional 3D object datasets and Internet source. For
conventional 3D object datasets, we test our method on 3 objects from ShapeNet
[10], 5 objects from InterCap [37] and 20 objects from BEHAVE [2]. We generate
the texture for InterCap using TEXTure [74] as the dataset does not provide
the texture. To test the generalizability of our method, we apply our method
on the objects obtained from the Internet source. Specifically, we use 4
objects from SketchFab [81].

BEHAVE Dataset. For quantitative evaluation, we utilize BEHAVE [2] dataset as
ground-truth to verify the quality of generated 3D affordance samples. BEHAVE
[2] contains video sequences of HOI scenarios for 20 object categories. Each
video frame is annotated with a single 3D human and object, where each 3D
human is annotated as SMPL-H [76, 53] parameters and 3D objects as rotation
and translation applied to the canonical mesh. We preprocess the dataset by
transferring the SMPL-H annotations into SMPL-X format following Choutas et
al. [69], and downsampling human and object mesh into 1000 surface points
described in Sec. 4.

Figure 7: Contact Results on Various 3D Objects. Using only the generated 3D
affordance samples, our representation can effectively infer precise contact
information. Table 1:  Effects of Adaptive Mask Inpainting. Adaptive mask
inpainting effectively inserts hunan into scene images without harming the
original scene.

Inpainting Methods |  RMSE${}_{\text{Background}}\downarrow$ |  mIoU$\uparrow$ |  mIoU${}_{\text{Occlusion Aware}}\uparrow$
---|---|---|---
Vanilla | 36.20 | 0.64 | 0.74
Adaptive Mask | 17.45 | 0.73 | 0.83

Table 2: Effects of Depth Optimization via Multiview Cues. Resolving depth
ambiguity via selecting semi-consistent multiview cues improves the quality of
affordance data in 3D.

Methods |  SIM${}_{\text{Human}}$$\uparrow$ |  SIM${}_{\text{Object}}$$\uparrow$
---|---|---
Ours${}_{\text{w/o Depth Optimization via Weak Auxiliary Cues}}$ | 25.77 | 34.20
Ours${}_{\text{full}}$ | 33.21 | 46.75

###  5.2 Quantitative Evaluation

Effects of Adaptive Mask Inpainting. We verify the effectiveness of our
adaptive mask algorithm, by comparing the amount of background preservation
for inpainted images with the original inpainting diffusion model. We report 3
metrics; (1) $\text{RMSE}_{\text{Background}}$: pixel error between images
excluding the predicted human region; (2) mIoU: average IoU between object
predictions in each inpainted image and original object image; (3)
$\text{mIoU}_{\text{Occlusion Aware}}$: same metric as (2) but excluding the
predicted human region during computation. To ensure fairness, we predict
high-quality human/object segmentation from images using open vocabulary
segmentation model [52, 45], different from PointRend [44] used in adaptive
mask algorithm. As shown in Tab. 1, we demonstrate our adaptive mask algorithm
better preserves the background, not only pixelwise
(RMSE${}_{\text{Background}}$), but also semantic-region-wise (mIOU,
mIOU${}_{\text{Occlusion Aware}}$) than original mask inpainting baseline.
This means that the generated images from adaptive mask inpainting are more
likely to preserve the original object, and not create hallucinated, or
geometric implausible affordance.

Effects of Depth Optimization via Weak Auxiliary Cues. We justify our
optimization pipeline by comparing the quality of the generated 3D affordance
samples produced with and without the component. Specifically, we ablate this
component by directly using the depth initialization results obtained from
human segmentation cues (refer to Sec. 3.3), without further depth
optimization. We extract the primitive representations from the generated
dataset ($\mathcal{P}_{ij}$) and ground-truth primitive representations from
BEHAVE [2] dataset ($\mathcal{P}^{\text{GT}}_{ij}$), where we consequently
compute similarity [85] between aggregated human contact (i.e.,
$\text{SIM}_{\text{Human}}$) and aggregated object contact (i.e.,
$\text{SIM}_{\text{Object}}$), derived from each $\mathcal{P}_{ij}$ and
$\mathcal{P}^{\text{GT}}_{ij}$. Note that we simply select maximum score among
$i$ or $j$ for aggregation. The motivation using contact as metric is because
it is a widely used affordance cue, and a conventional way to represent 3D
affordance knowledge. As shown in Tab. 2, we observe a big performance drop
without our depth optimization framework. This is expected, as solely relying
on segmentation cues for depth prediction will not provide precise depth, and
does not regard the plausibility of human joints observed from different
views, causing 3D artifacts such as floating humans.

Figure 8: Contact for Specified Region. Our representation provides contact
scores for all pairs of human point and object point, which can be used to
infer regionwise contact map.

###  5.3 Qualitative Evaluation

Contact Results on Various 3D Objects. We demonstrate the quality of the
generated dataset, derived human and object contact map from our
representation for various objects in Fig. 7. Note that we show the aggregated
contact map, obtained via maximizing score. The results demonstrate the
quality of the generated dataset, and preciseness of the contact map derived
from our representation.

Contact for Specified Regions. Our representation outputs contact values for
each pair of human point and object point; which means we can infer
corresponding contact region w.r.t. specified human or object region by simply
selecting the points and aggregating the contact values. See the first row of
Fig. 8. For the selected points on the handle of the motorcycle, we can
observe the human contact possessing high values around hand, while selecting
points on the seat of the motorcycle gives high values of human contact around
buttocks. This is plausible, as handle provides utilities to steer motorcycle
by hand, and seat provides a physical support via buttocks. The second row
shows the “inverse” results, where selecting the human surface points around
buttocks gives high contact values for object around the seat, while selecting
the points around back gives high values around backrest. Enabling learning
for such dense contact correspondences for any object opens up a range of
possible future work.

Orientational & Spatial Affordance. Example of orientational affordance is
shown in Fig. 9. As seen in the figure, the orientation tendency shows overall
higher score in the case of chair, compared to the stool. This results comes
from the geometry of the chair, as the chair with backrest and handles will
induce human front torso to head front, as heading back or side will likely
cause collision with the object, making the pose highly likely to be
physically implausible. In contrast, we observe low scores in the case of
stool, as no structure exists above the seating, hence the human becoming
freely rotatable w.r.t. azimuth, and deriving low orientation tendency (high
entropy). Note that we observe high values on thighs and top of the head as
those body parts are implicitly headed antiparallel to gravity while sitting.
In Fig. 10, we also exemplify the use of spatial affordance, where we show
relative human occupancy given the object umbrella. Similar to contact, since
we save the vertex-wise occupancy scores, we can specify the human points and
check the occupancy of the selected points only (e.g., hands or faces). As
shown, we can observe that the hand occupies volume around the handle, and
face in the region inside and near the umbrella, which is plausible assuming
that human grabs the umbrella, and uses it to block face from the rain by
hiding under it.

Figure 9: Orientational Affordance. The chair with backrest provides higher
orientation tendency (given as negated entropy) for the torso than the stool
with arbitrary sitting directions.  Figure 10: Spatial Affordance. By
selecting the points from specific body parts of human, we can derive the
spatial occupancy distribution for any body parts w.r.t. the object.

##  6 Discussion

We introduce a novel method to learn 3D affordance for any 3D object input, by
generating the 3D affordance samples for the specific object, which is a pair
of human and object mesh depicting plausible interaction. The method leverages
zero-shot capabilities of inpainting diffusion models for generalization, with
novel strategies including adaptive mask inpainting to enable scene-preserving
human insertion, and depth optimization framework that leverages generated
human cues from multiple viewpoints to uplift 2D affordance image into 3D.
Furthermore, we propose new primitive affordance representation, where we
demonstrate the potential by deriving various types of affordance, including
contact, orientation tendency, and spatial occupancy. With our method being
generalizable, and our representation being easily extractable, our approach
offers myriads of possible applications. Despite the potentials, the method is
new and possesses limitations such as challenges in learning dexterous grasps.
See Supp. Mat. for further discussion on limitations and potential future
work.

Acknowledgements. This work was supported by Naver Webtoon. The work of SNU
members was also supported by NRF grant funded by the Korean government (MSIT)
(No. 2022R1A2C2092724), and IITP grant funded by the Korean government (MSIT)
(No.2021-0-01343). H. Joo is the corresponding author.

## References

  * Bahl et al. [2023] S. Bahl, R. Mendonca, L. Chen, U. Jain, and D. Pathak.  Affordances from human videos as a versatile representation for robotics.  In _CVPR_ , 2023.
  * Bhatnagar et al. [2022] B. L. Bhatnagar, X. Xie, I. Petrov, C. Sminchisescu, C. Theobalt, and G. Pons-Moll.  Behave: Dataset and method for tracking human object interactions.  In _CVPR_ , 2022.
  * Brahmbhatt et al. [2019a] S. Brahmbhatt, C. Ham, C. C. Kemp, and J. Hays.  Contactdb: Analyzing and predicting grasp contact via thermal imaging.  In _CVPR_ , 2019a.
  * Brahmbhatt et al. [2019b] S. Brahmbhatt, A. Handa, J. Hays, and D. Fox.  Contactgrasp: Functional multi-finger grasp synthesis from contact.  In _IROS_ , 2019b.
  * Brahmbhatt et al. [2020] S. Brahmbhatt, C. Tang, C. D. Twigg, C. C. Kemp, and J. Hays.  Contactpose: A dataset of grasps with object contact and hand pose.  In _ECCV_ , 2020.
  * Cai et al. [2016] M. Cai, K. M. Kitani, and Y. Sato.  Understanding hand-object manipulation with grasp types and object attributes.  In _RSS_ , 2016.
  * Cao et al. [2019] Z. Cao, G. Hidalgo Martinez, T. Simon, S. Wei, and Y. A. Sheikh.  Openpose: Realtime multi-person 2d pose estimation using part affinity fields.  _IEEE TPAMI_ , 2019.
  * Cao et al. [2020] Z. Cao, H. Gao, K. Mangalam, Q.-Z. Cai, M. Vo, and J. Malik.  Long-term human motion prediction with scene context.  In _ECCV_ , 2020.
  * Chai et al. [2021] L. Chai, J.-Y. Zhu, E. Shechtman, P. Isola, and R. Zhang.  Ensembling with deep generative views.  In _CVPR_ , 2021.
  * Chang et al. [2015] A. X. Chang, T. Funkhouser, L. Guibas, P. Hanrahan, Q. Huang, Z. Li, S. Savarese, M. Savva, S. Song, H. Su, J. Xiao, L. Yi, and F. Yu.  ShapeNet: An Information-Rich 3D Model Repository.  Technical Report arXiv:1512.03012 [cs.GR], 2015.
  * Chao et al. [2021] Y.-W. Chao, W. Yang, Y. Xiang, P. Molchanov, A. Handa, J. Tremblay, Y. S. Narang, K. Van Wyk, U. Iqbal, S. Birchfield, et al.  Dexycb: A benchmark for capturing hand grasping of objects.  _CVPR_ , 2021.
  * Chen et al. [2023] R. Chen, Y. Chen, N. Jiao, and K. Jia.  Fantasia3d: Disentangling geometry and appearance for high-quality text-to-3d content creation.  2023\.
  * Chen et al. [2019] Y. Chen, S. Huang, T. Yuan, S. Qi, Y. Zhu, and S.-C. Zhu.  Holistic++ scene understanding: Single-view 3d holistic scene parsing and human pose estimation with human-object interaction and physical commonsense.  In _ICCV_ , 2019.
  * Deng et al. [2021] S. Deng, X. Xu, C. Wu, K. Chen, and K. Jia.  3d affordancenet: A benchmark for visual object affordance understanding.  In _CVPR_ , 2021.
  * et al [2022] A. R. et al.  Hierarchical text-conditional image generation with clip latents.  _arXiv preprint arXiv:2204.06125_ , 2022.
  * Fan et al. [2023] Z. Fan, O. Taheri, D. Tzionas, M. Kocabas, M. Kaufmann, M. J. Black, and O. Hilliges.  Articulated objects in free-form hand interaction.  _CVPR_ , 2023.
  * Fieraru et al. [2020] M. Fieraru, M. Zanfir, E. Oneata, A.-I. Popa, V. Olaru, and C. Sminchisescu.  Three-dimensional reconstruction of human interactions.  In _CVPR_ , 2020.
  * Fischler and Bolles [1981] M. A. Fischler and R. C. Bolles.  Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography.  _Communications of the ACM_ , 1981.
  * Gao and Tedrake [2021] W. Gao and R. Tedrake.  kpam-sc: Generalizable manipulation planning using keypoint affordance and shape completion.  In _ICRA_ , 2021.
  * Garg et al. [2014] M. Garg, P. Garg, and R. Vohra.  Advanced fibonacci sequence with golden ratio.  _IJSER_ , 2014.
  * Gibson [1979] J. J. Gibson.  _The Ecological Approach to Visual Perception_.  Houghton Mifflin, 1979.
  * Goodfellow et al. [2014] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio.  Generative adversarial nets.  In _NeurIPS_ , 2014.
  * Grady et al. [2021] P. Grady, C. Tang, C. D. Twigg, M. Vo, S. Brahmbhatt, and C. C. Kemp.  ContactOpt: Optimizing contact to improve grasps.  In _CVPR_ , 2021.
  * Hampali et al. [2020] S. Hampali, M. Rad, M. Oberweger, and V. Lepetit.  Honnotate: A method for 3d annotation of hand and object poses.  _CVPR_ , 2020.
  * Hampali et al. [2022] S. Hampali, S. D. Sarkar, M. Rad, and V. Lepetit.  Keypoint transformer: Solving joint identification in challenging hands and object interactions for accurate 3d pose estimation.  _CVPR_ , 2022.
  * Han and Joo [2023] S. Han and H. Joo.  Learning canonicalized 3d human-object spatial relations from unbounded synthesized images.  In _ICCV_ , 2023.
  * Hao et al. [2021] Z. Hao, A. Mallya, S. Belongie, and M.-Y. Liu.  Gancraft: Unsupervised 3d neural rendering of minecraft worlds.  In _ICCV_ , 2021.
  * Hassan et al. [2019] M. Hassan, V. Choutas, D. Tzionas, and M. J. Black.  Resolving 3d human pose ambiguities with 3d scene constraints.  In _ICCV_ , 2019.
  * Hassan et al. [2021] M. Hassan, P. Ghosh, J. Tesch, D. Tzionas, and M. J. Black.  Populating 3d scenes by learning human-scene interaction.  In _CVPR_ , 2021.
  * Hasson et al. [2019] Y. Hasson, G. Varol, D. Tzionas, I. Kalevatykh, M. J. Black, I. Laptev, and C. Schmid.  Learning joint reconstruction of hands and manipulated objects.  In _CVPR_ , 2019.
  * He et al. [2022] R. He, S. Sun, X. Yu, C. Xue, W. Zhang, P. Torr, S. Bai, and X. Qi.  Is synthetic data from generative models ready for image recognition?  _arXiv preprint arXiv:2210.07574_ , 2022.
  * Hermans et al. [2011] T. Hermans, J. M. Rehg, and A. Bobick.  Affordance prediction via learned object attributes.  In _Proc. Intl. Conf. on Robotics and Automation_ , 2011.
  * Hertz et al. [2022] A. Hertz, R. Mokady, J. Tenenbaum, K. Aberman, Y. Pritch, and D. Cohen-Or.  Prompt-to-prompt image editing with cross attention control.  _arXiv preprint arXiv:2208.01626_ , 2022.
  * Ho et al. [2020] J. Ho, A. Jain, and P. Abbeel.  Denoising diffusion probabilistic models.  _arXiv preprint arxiv:2006.11239_ , 2020.
  * Huang et al. [2022a] C.-H. P. Huang, H. Yi, M. Höschle, M. Safroshkin, T. Alexiadis, S. Polikovsky, D. Scharstein, and M. J. Black.  Capturing and inferring dense full-body human-scene contact.  In _CVPR_ , 2022a.
  * Huang et al. [2022b] C.-H. P. Huang, H. Yi, M. Höschle, M. Safroshkin, T. Alexiadis, S. Polikovsky, D. Scharstein, and M. J. Black.  Capturing and inferring dense full-body human-scene contact.  In _CVPR_ , 2022b.
  * Huang et al. [2022c] Y. Huang, O. Taheri, M. J. Black, and D. Tzionas.  InterCap: Joint markerless 3D tracking of humans and objects in interaction.  In _GCPR_ , 2022c.
  * Jahanian et al. [2021] A. Jahanian, X. Puig, Y. Tian, and P. Isola.  Generative models as a data source for multiview representation learning.  _arXiv preprint arXiv:2106.05258_ , 2021.
  * Jian et al. [2023] J. Jian, X. Liu, M. Li, R. Hu, and J. Liu.  Affordpose: A large-scale dataset of hand-object interactions with affordance-driven hand pose.  In _ICCV_ , 2023.
  * Jiang et al. [2020] W. Jiang, N. Kolotouros, G. Pavlakos, X. Zhou, and K. Daniilidis.  Coherent reconstruction of multiple humans from a single image.  In _CVPR_ , 2020.
  * Jiang et al. [2022] Y. Jiang, S. Jiang, G. Sun, Z. Su, K. Guo, M. Wu, J. Yu, and L. Xu.  Neuralhofusion: Neural volumetric rendering under human-object interactions.  In _CVPR_ , 2022.
  * Karras et al. [2019] T. Karras, S. Laine, and T. Aila.  A style-based generator architecture for generative adversarial networks.  In _CVPR_ , 2019.
  * Kingma and Ba [2014] D. P. Kingma and J. Ba.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_ , 2014.
  * Kirillov et al. [2020] A. Kirillov, Y. Wu, K. He, and R. Girshick.  Pointrend: Image segmentation as rendering.  In _CVPR_ , 2020.
  * Kirillov et al. [2023] A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y. Lo, P. Dollar, and R. Girshick.  Segment anything.  In _ICCV_ , 2023.
  * Kulal et al. [2023] S. Kulal, T. Brooks, A. Aiken, J. Wu, J. Yang, J. Lu, A. A. Efros, and K. K. Singh.  Putting people in their place: Affordance-aware human insertion into scenes.  In _CVPR_ , 2023.
  * Lee and Grauman [2015] Y. J. Lee and K. Grauman.  Predicting important objects for egocentric video summarization.  _IJCV_ , 2015.  URL https://api.semanticscholar.org/CorpusID:5617021.
  * Levine and Shah [2022] S. Levine and D. Shah.  Learning robotic navigation from experience: principles, methods and recent results.  _Philosophical Transactions of the Royal Society B: Biological Sciences_ , 2022.
  * Li et al. [2023] G. Li, V. Jampani, D. Sun, and L. Sevilla-Lara.  Locate: Localize and transfer object parts for weakly supervised affordance grounding.  In _CVPR_ , 2023.
  * Li et al. [2019] Z. Li, J. Sedlar, J. Carpentier, I. Laptev, N. Mansard, and J. Sivic.  Estimating 3d motion and forces of person-object interactions from monocular video.  In _CVPR_ , 2019.
  * Lin et al. [2023] C.-H. Lin, J. Gao, L. Tang, T. Takikawa, X. Zeng, X. Huang, K. Kreis, S. Fidler, M.-Y. Liu, and T.-Y. Lin.  Magic3d: High-resolution text-to-3d content creation.  In _CVPR_ , 2023.
  * Liu et al. [2023] S. Liu, Z. Zeng, T. Ren, F. Li, H. Zhang, J. Yang, C. Li, J. Yang, H. Su, J. Zhu, et al.  Grounding dino: Marrying dino with grounded pre-training for open-set object detection.  _arXiv preprint arXiv:2303.05499_ , 2023.
  * Loper et al. [2015] M. Loper, N. Mahmood, J. Romero, G. Pons-Moll, and M. J. Black.  SMPL: A skinned multi-person linear model.  _Proc. ACM SIGGRAPH Asia_ , 2015.
  * Lugmayr et al. [2022] A. Lugmayr, M. Danelljan, A. Romero, F. Yu, R. Timofte, and L. Van Gool.  Repaint: Inpainting using denoising diffusion probabilistic models.  In _CVPR_ , 2022.
  * Maninis et al. [2023] K.-K. Maninis, S. Popov, M. Nießner, and V. Ferrari.  Cad-estate: Large-scale cad model annotation in rgb videos.  In _ICCV_ , 2023.
  * Mao et al. [2021] C. Mao, A. Cha, A. Gupta, H. Wang, J. Yang, and C. Vondrick.  Generative interventions for causal learning.  In _CVPR_ , 2021.
  * McCool and Fiume [1992] M. McCool and E. Fiume.  Hierarchical poisson disk sampling distributions.  In _Graphics interface_ , 1992.
  * Melas-Kyriazi et al. [2021] L. Melas-Kyriazi, C. Rupprecht, I. Laina, and A. Vedaldi.  Finding an unsupervised image segmenter in each of your deep generative models.  _arXiv preprint arXiv:2105.08127_ , 2021.
  * Meng et al. [2022] C. Meng, Y. He, Y. Song, J. Song, J. Wu, J.-Y. Zhu, and S. Ermon.  SDEdit: Guided image synthesis and editing with stochastic differential equations.  In _ICLR_ , 2022.
  * Mihajlovic et al. [2022] M. Mihajlovic, S. Saito, A. Bansal, M. Zollhoefer, and S. Tang.  COAP: Compositional articulated occupancy of people.  In _CVPR_ , 2022.
  * Mokady et al. [2022] R. Mokady, A. Hertz, K. Aberman, Y. Pritch, and D. Cohen-Or.  Null-text inversion for editing real images using guided diffusion models.  _arXiv preprint arXiv:2211.09794_ , 2022.
  * Monszpart et al. [2019] A. Monszpart, P. Guerrero, D. Ceylan, E. Yumer, and N. J. Mitra.  imapper: interaction-guided scene mapping from monocular videos.  _ACM TOG_ , 2019.
  * Moon et al. [2022] G. Moon, H. Choi, and K. M. Lee.  Accurate 3d hand pose estimation for whole-body 3d human mesh estimation.  In _CVPRW_ , 2022.
  * Nagarajan and Grauman [2020] T. Nagarajan and K. Grauman.  Learning affordance landscapes for interaction exploration in 3d environments.  In _NeurIPS_ , 2020.
  * Oleynikova et al. [2016] H. Oleynikova, A. Millane, Z. Taylor, E. Galceran, J. Nieto, and R. Siegwart.  Signed distance fields: A natural representation for both mapping and planning.  In _RSS 2016 workshop_ , 2016.
  * OpenAI [2023a] OpenAI.  Chatgpt: Optimizing language models for dialogue.  https://openai.com/blog/chatgpt/, 2023a.
  * OpenAI [2023b] OpenAI.  Gpt-4v(ision) system card.  https://openai.com/research/gpt-4v-system-card, 2023b.
  * Pan et al. [2020] X. Pan, B. Dai, Z. Liu, C. C. Loy, and P. Luo.  Do 2d gans know 3d shape? unsupervised 3d shape reconstruction from 2d image gans.  _arXiv preprint arXiv:2011.00844_ , 2020.
  * Pavlakos et al. [2019] G. Pavlakos, V. Choutas, N. Ghorbani, T. Bolkart, A. A. A. Osman, D. Tzionas, and M. J. Black.  Expressive body capture: 3d hands, face, and body from a single image.  In _CVPR_ , 2019.
  * Peebles et al. [2022] W. Peebles, J.-Y. Zhu, R. Zhang, A. Torralba, A. A. Efros, and E. Shechtman.  Gan-supervised dense visual alignment.  In _CVPR_ , 2022.
  * Petrov et al. [2023] I. A. Petrov, R. Marin, J. Chibane, and G. Pons-Moll.  Object pop-up: Can we infer 3d objects and their poses from human interactions alone?  In _CVPR_ , 2023.
  * Poole et al. [2023] B. Poole, A. Jain, J. T. Barron, and B. Mildenhall.  Dreamfusion: Text-to-3d using 2d diffusion.  In _ICLR_ , 2023.
  * Rempe et al. [2020] D. Rempe, L. J. Guibas, A. Hertzmann, B. Russell, R. Villegas, and J. Yang.  Contact and human dynamics from monocular video.  In _ECCV_ , 2020.
  * Richardson et al. [2023] E. Richardson, G. Metzer, Y. Alaluf, R. Giryes, and D. Cohen-Or.  Texture: Text-guided texturing of 3d shapes.  In _Proc. ACM SIGGRAPH_ , 2023.
  * Rombach et al. [2022] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer.  High-resolution image synthesis with latent diffusion models.  In _CVPR_ , 2022.
  * Romero et al. [2017] J. Romero, D. Tzionas, and M. J. Black.  Embodied hands: Modeling and capturing hands and bodies together.  _Proc. ACM SIGGRAPH Asia_ , 2017.
  * Savva et al. [2016] M. Savva, A. X. Chang, P. Hanrahan, M. Fisher, and M. Nießner.  Pigraphs: learning interaction snapshots from observations.  _ACM TOG_ , 2016.
  * SG_161222 [2023] SG_161222.  Realistic vision v5.1.  https://civitai.com/models/4201?modelVersionId=130090, 2023.
  * Shannon [2001] C. E. Shannon.  A mathematical theory of communication.  _ACM SIGMOBILE_ , 2001.
  * Shen et al. [2021] T. Shen, J. Gao, K. Yin, M.-Y. Liu, and S. Fidler.  Deep marching tetrahedra: a hybrid representation for high-resolution 3d shape synthesis.  In _NeurIPS_ , 2021.
  * Sketchfab [2023] Sketchfab.  Sketchfab.  https://sketchfab.com/, 2023.
  * Sohl-Dickstein et al. [2015] J. Sohl-Dickstein, E. Weiss, N. Maheswaranathan, and S. Ganguli.  Deep unsupervised learning using nonequilibrium thermodynamics.  In _ICML_ , 2015.
  * Song et al. [2020] J. Song, C. Meng, and S. Ermon.  Denoising diffusion implicit models.  _arXiv:2010.02502_ , 2020.  URL https://arxiv.org/abs/2010.02502.
  * Sun et al. [2021] G. Sun, X. Chen, Y. Chen, A. Pang, P. Lin, Y. Jiang, L. Xu, J. Yu, and J. Wang.  Neural free-viewpoint performance rendering under complex human-object interactions.  In _ACM MM_ , 2021.
  * Swain and Ballard [1991] M. J. Swain and D. H. Ballard.  Color indexing.  _IJCV_ , 1991.
  * Taheri et al. [2020] O. Taheri, N. Ghorbani, M. J. Black, and D. Tzionas.  Grab: A dataset of whole-body human grasping of objects.  In _ECCV_ , 2020.
  * Tanaka and Aranha [2019] F. H. K. d. S. Tanaka and C. Aranha.  Data augmentation using gans.  _arXiv preprint arXiv:1904.09135_ , 2019.
  * Trabucco et al. [2023] B. Trabucco, K. Doherty, M. Gurinas, and R. Salakhutdinov.  Effective data augmentation with diffusion models.  _arXiv preprint arXiv:2302.07944_ , 2023.
  * Tripathi et al. [2023] S. Tripathi, A. Chatterjee, J.-C. Passy, H. Yi, D. Tzionas, and M. J. Black.  DECO: Dense estimation of 3D human-scene contact in the wild.  In _ICCV_ , 2023.
  * Tritrong et al. [2021] N. Tritrong, P. Rewatbowornwong, and S. Suwajanakorn.  Repurposing gans for one-shot semantic part segmentation.  In _CVPR_ , 2021.
  * Wu et al. [2023] W. Wu, Y. Zhao, H. Chen, Y. Gu, R. Zhao, Y. He, H. Zhou, M. Z. Shou, and C. Shen.  Datasetdm: Synthesizing data with perception annotations using diffusion models.  _arXiv preprint arXiv:2308.06160_ , 2023.
  * Xie et al. [2022] X. Xie, B. L. Bhatnagar, and G. Pons-Moll.  Chore: Contact, human and object reconstruction from a single rgb image.  In _ECCV_ , 2022.
  * Xu et al. [2021] J. Xu, S. Zhu, H. Guo, and S. Wu.  Automated labeling for robotic autonomous navigation through multi-sensory semi-supervised learning on big data.  _IEEE TBD_ , 2021.
  * Xu et al. [2023] S. Xu, Z. Li, Y.-X. Wang, and L.-Y. Gui.  InterDiff: Generating 3d human-object interactions with physics-informed diffusion.  In _ICCV_ , 2023.
  * Yang et al. [2021] L. Yang, X. Zhan, K. Li, W. Xu, J. Li, and C. Lu.  CPF: Learning a contact potential field to model the hand-object interaction.  In _ICCV_ , 2021.
  * Yang et al. [2023] Y. Yang, W. Zhai, H. Luo, Y. Cao, J. Luo, and Z.-J. Zha.  Grounding 3d object affordance from 2d interactions in images.  In _ICCV_ , 2023.
  * Ye et al. [2023] Y. Ye, X. Li, A. Gupta, S. D. Mello, S. Birchfield, J. Song, S. Tulsiani, and S. Liu.  Affordance diffusion: Synthesizing hand-object interactions.  In _CVPR_ , 2023.
  * Yi et al. [2022] H. Yi, C.-H. P. Huang, D. Tzionas, M. Kocabas, M. Hassan, S. Tang, J. Thies, and M. J. Black.  Human-aware object placement for visual environment reconstruction.  In _CVPR_ , 2022.
  * Zhang et al. [2020a] J. Y. Zhang, S. Pepose, H. Joo, D. Ramanan, J. Malik, and A. Kanazawa.  Perceiving 3d human-object spatial arrangements from a single image in the wild.  In _ECCV_ , 2020a.
  * Zhang et al. [2020b] S. Zhang, Y. Zhang, Q. Ma, M. J. Black, and S. Tang.  Place: Proximity learning of articulation and contact in 3d environments.  In _3DV_ , 2020b.
  * Zhang et al. [2022] X. Zhang, B. L. Bhatnagar, S. Starke, V. Guzov, and G. Pons-Moll.  Couch: Towards controllable human-chair interactions.  In _ECCV_ , 2022.
  * Zhang et al. [2020c] Y. Zhang, W. Chen, H. Ling, J. Gao, Y. Zhang, A. Torralba, and S. Fidler.  Image gans meet differentiable rendering for inverse graphics and interpretable 3d neural rendering.  _arXiv preprint arXiv:2010.09125_ , 2020c.
  * Zhang et al. [2020d] Y. Zhang, M. Hassan, H. Neumann, M. J. Black, and S. Tang.  Generating 3d people in scenes without people.  In _CVPR_ , 2020d.
  * Zhang et al. [2021] Y. Zhang, H. Ling, J. Gao, K. Yin, J.-F. Lafleche, A. Barriuso, A. Torralba, and S. Fidler.  Datasetgan: Efficient labeled data factory with minimal human effort.  In _CVPR_ , 2021.
  * Zhou et al. [2022] K. Zhou, B. L. Bhatnagar, J. E. Lenssen, and G. Pons-Moll.  Toch: Spatio-temporal object correspondence to hand for motion refinement.  In _ECCV_ , 2022.
  * Zhou et al. [2018] T. Zhou, R. Tucker, J. Flynn, G. Fyffe, and N. Snavely.  Stereo magnification: Learning view synthesis using multiplane images.  In _Proc. ACM SIGGRAPH_ , 2018.
  * Zimmermann et al. [2019] C. Zimmermann, D. Ceylan, J. Yang, B. Russell, M. Argus, and T. Brox.  Freihand: A dataset for markerless capture of hand pose and shape from single rgb images.  _ICCV_ , 2019.

##  Appendix A Implementation Details

###  A.1 Affordance Image Generation

####  A.1.1 Rendering Object from Multi-Viewpoints

For static type objects, we install 8 weak perspective cameras with
$45\degree$ azimuth intervals. For dynamic type objects, we install 4 weak
perspective cameras with $90\degree$ azimuth intervals. The elevation is set
constant within [$0\degree$, $30\degree$] range, where the values differ by
category. For dynamic objects, we perturb the object with random rotations and
translations. Specifically, we uniformly sample the rotation in the form of
Euler angle, where yaw, pitch, and roll are uniformly sampled from a
predefined range. Random translations are sampled in a similar way, where each
component of 3D displacement is sampled from a predefined range. Note that we
set weak perspective camera scale and distance from the object as
hyperparameters. We repeat the rendering procedures 10 times with different
perturbations applied, resulting in 40 distinct views.

####  A.1.2 Inpainting Mask Selection

We also render the inpainting masks while rendering the object, using the same
camera parameters. Specifically, for each camera, we place an upright window
perpendicular to the $xy$ plane, also perpendicular to the $xy$ projection of
the camera’s front vector. For each mask, the center of the intersection with
$z=0$ plane lies within the $xy$ projection of the 3D object. Note that the
strides of the upright windows w.r.t. $x,y$ direction are given as
hyperparameters, along with the height and width of the window. We render
these upright windows using assigned cameras to obtain 2D rectangular masks,
consequently used as inpainting masks that occlude the original object. To
reduce the number of unnecessary masks (e.g., masks that do not cover the
object but mostly the background), we only retain the masks if the IoU
(Intersection over Union) between the mask and the original object lies within
the predefined range, also given as hyperparameters.

####  A.1.3 Prompt Generation

We generate human-object interaction prompts via GPT4v [67], where we input
the rendered object image and the following query template:

> Generate at most 3 simple subject-verb-object prompt where subject’s word is
> exactly ‘1 person’ and object’s image is given. You should use diverse and
> general word but no pronoun for subject. Generated prompt must align with
> common sense. Verb must be simple as possible, and should depict physical
> interaction between subject and object. Also, only the interaction with
> given object is allowed, and no other objects should be introduced in the
> prompt.

Table 3: Examples of Generated HOI Prompts. Our method utilizes GPT4v [67] to
generate plausible set of HOI prompts, given the object image and a query
template. We provide example prompts for BEHAVE [2] objects below.

Object Category | Generated HOI Prompts
---|---
Backpack | 1 person wears the backpack
1 person zips the backpack
1 person carries the backpack
Chairblack | 1 person sits on the chair
1 person adjusts the chair
1 person lifts the chair
Keyboard | 1 person types on the keyboard
1 person cleans the keyboard
1 person holds the keyboard
Suitcase | 1 person packs the suitcase
1 person drags the suitcase
1 person opens the suitcase

We do not augment the prompt except “full body”, where we empirically find
this augmentation useful when generating the whole human body instead of a
zoom-in shot of body parts (e.g., hand). We provide examples of generated HOI
prompts in Tab. 3.

####  A.1.4 Adaptive Mask Inpainting

While we use a publicly available inpainting diffusion model (RealisticVision
[75, 78]), we note that our adaptive mask algorithm is applicable to any
inpainting diffusion model. We sample images with 45$\sim$50 denoising steps
using DDIM [83] scheduler, with a classifier-free guidance scale of $11.0$.
During Adaptive Mask Inpainting, we dilate the adaptive mask using $3\times 3$
kernel $k$ with $n_{t}$ times repeat:

$k=\begin{bmatrix}1&1&1\\\ 1&1&1\\\ 1&1&1\end{bmatrix},\quad
n_{t}=\begin{cases}20&\text{50 $>t$ $\geq 45$}\\\ 10&\text{45 $>t$ $\geq
40$}\\\ 5&\text{40 $>t$ $\geq 35$}\\\ 4&\text{35 $>t$ $\geq 30$}\\\ 3&\text{30
$>t$ $\geq 25$}\\\ 2&\text{25 $>t$ $\geq 20$}\\\ 1&\text{20 $>t$ $\geq 15$}\\\
0&\text{15 $>t$ $\geq 0$}\\\ \end{cases}$ (10)

We also employ “Provoke Schedule” (refer to Algorithm. 1) for faster
generation speed. The provoke scheduler determines whether to skip the mask
adaptation step ($t\in\text{ProvokeSchedule}$) or not
($t\not\in\text{ProvokeSchedule}$) during timestep $t$. Specifically, we use
the following schedule:

$\text{ProvokeSchedule}=\\{t~{}|~{}\text{(40 $\geq t$ $\geq$ 2 and $t$ is
even) or $t$ = 45}\\}$

(11)

###  A.2 Lifting 2D Affordance to 3D

####  A.2.1 Number of Joints Used during Triangulation

We use Hand4Whole [63] to predict 3D humans from images
($\mathbf{F}_{\text{human}}$ in Sec. 3.3). The model returns 1 global rotation
(for pelvis joint) and 54 human joint rotations following the SMPL-X [69]
format; which consists of $21$ body joints, ${15+15}$ hand joints, $1$ jaw
joint, and $1+1$ eye joints. During Virtual Triangulation (Sec. 3.3), we add
21 OpenPose [7] joints (“nose”, “right eye”, “left eye”, “right ear”, “left
ear”, “left big toe”, “left small toe”, “left heel”, “right big toe”, “right
small toe”, “right heel”, “left thumb”, “left index”, “left middle”, “left
ring”, “left pinky”, “right thumb”, “right index”, “right middle”, “right
ring”, “right pinky”) and exclude 11 original joints (“spine1”, “spine2”,
“spine3”, “left foot”, “right foot”, “left collar”, “right collar”, “head”,
“jaw”, “left eye smplhf”, “right eye smplhf”), resulting in $1+54+21-11=67$
joints.

####  A.2.2 Depth Initialization

We evenly space 7 human candidates along the orthographic rays, where we place
the pelvis joint of $4^{\text{th}}$ candidate (center candidate) to the
position that minimizes the average distance w.r.t. the all object vertices.
The distance between human candidates is proportional to the width of the
human along the orthographic camera ray, where we set the multiplier as $0.3$.
We initialize the depth using the human candidate with maximum IoU between the
rendered human mask and the predicted human segmentation mask.

####  A.2.3 Selecting Inliers for Multiview Human Cues

Given the human image (i.e., reference image) we aim to specify the depth, we
first select the 2D joints from different view images (generated from the same
prompt) to use for triangulation. Specifically, we first triangulate the 2D
joints w.r.t. the corresponding joints in the reference image human, and
filter the joints if the reprojection error is over $100$. We iterate through
the remaining joints again, where we triangulate the joint with other
remaining joints and consequently apply RANSAC [18] to retain the inlier
joints (we consider the joint as inlier if the reprojection error is less than
$200$). We finally select the maximum inlier set and use them as multiview
cues for depth optimization.

####  A.2.4 Collision Loss

We implement collision loss using COAP [60], which represents the human body
as SDF [65] field with a zero-level set of $f_{\Theta}(q|\mathcal{G})=0$,
where $\mathcal{G}$ is a human bone transformation and $q$ is a query point.
By treating query point $q$ as an object vertex, we reduce collision between
human and object by penalizing object vertices inside the human volume:

$\mathcal{L}_{\text{collision}}=\frac{1}{N_{o}}\sum_{\mathbf{v}_{i}^{o}\in\mathbf{M}^{o}}\sigma(f_{\Theta}(\mathbf{v}_{i}^{o}|\mathcal{G}))\mathbf{1}({f_{\Theta}(\mathbf{v}_{i}^{o}|\mathcal{G})>0})$
(12)

where $\sigma(\cdot)$ is the sigmoid function to convert SDF values to opacity
and $\mathbf{1}(\cdot)$ returns 1 if the argument inside is true else 0. We
set $\lambda_{\text{collision}}=400$, and use Adam [43] optimizer with
learning rate $1\times 10^{-2}$ for $200$ iteration to optimize $\mathcal{L}$.

####  A.2.5 Filtering

We filter out the 3D human samples if (1) the IoU between the human rendering
and predicted human segmentation is below $0.3$ or over $0.8$, (2) number of
inliers after RANSAC [18] is below $\tau_{\text{inlier}}$ (which varies from
$1\sim 50$, based on the given 3D object), or (3) the intersection volume over
human volume is higher than $0.01$.

###  A.3 Canonicalization for Primitive Extraction

We address 2 types of 3D object (including 3D human): (1) the object is
assumed rigid, meaning that current object can be obtained via applying rigid
transformation
$\mathbf{T}^{\text{original}\xrightarrow[]{}\text{current}}\in\text{SE}(3)$ on
the original object; or (2) the object is non-rigid (e.g., 3D human), meaning
that no original object exist, and also the rigid transformation. We provide
the canonicalization procedure that addresses both cases.

Given the human surface normal $\mathbf{n}_{j}^{h}$ and relative position
$\mathbf{p}^{o\xrightarrow[]{}h}$, we rotate them the same amount when
rotating the object surface normal $\mathbf{n}_{i}^{o}$ to face
$\hat{\mathbf{n}}=[0,0,1]^{T}$ (or conversely, rotate object surface normal
$\mathbf{n}_{i}^{o}$ and relative position $\mathbf{p}^{h\xrightarrow[]{}o}$
the same amount when rotating human surface normal $\mathbf{n}_{j}^{h}$ to
face $\hat{\mathbf{n}}=[0,0,1]^{T}$, which we omit for brevity). Specifically,
we canonicalize the human normal $\mathbf{n}_{j}^{h}$ to $\mathbf{n}$
following:

$\mathbf{n}=(\mathbf{n}_{i}^{o}\cdot\hat{\mathbf{n}})\mathbf{n}^{h}_{j}+(\mathbf{n}_{j}^{h}\cdot\mathbf{n}_{i}^{o})\hat{\mathbf{n}}-(\mathbf{n}_{j}^{h}\cdot\hat{\mathbf{n}})\mathbf{n}_{i}^{o}+[{\mathbf{n}_{j}^{h}\cdot(\mathbf{n}_{i}^{o}\times\hat{\mathbf{n}})\over
1+\mathbf{n}_{i}^{o}\cdot\hat{\mathbf{n}}}](\mathbf{n}_{i}^{o}\times\hat{\mathbf{n}})$

(13)

and similarly, we canonicalize the relative position
$\mathbf{p}^{o\xrightarrow[]{}h}$ to $\mathbf{p}$ following:

$\mathbf{p}=(\mathbf{n}_{i}^{o}\cdot\hat{\mathbf{n}})\mathbf{p}^{o\xrightarrow[]{}h}+(\mathbf{p}^{o\xrightarrow[]{}h}\cdot\mathbf{n}_{i}^{o})\hat{\mathbf{n}}-(\mathbf{p}^{o\xrightarrow[]{}h}\cdot\hat{\mathbf{n}})\mathbf{n}_{i}^{o}+[{\mathbf{p}^{o\xrightarrow[]{}h}\cdot(\mathbf{n}_{i}^{o}\times\hat{\mathbf{n}})\over
1+\mathbf{n}_{i}^{o}\cdot\hat{\mathbf{n}}}](\mathbf{n}_{i}^{o}\times\hat{\mathbf{n}})$

(14)

The canonicaliation procedure described in Eq. 13 and Eq. 14 preserves the
length of the vector ($\|\mathbf{n}\|=\|\mathbf{n}^{h}_{j}\|$ &
$\|\mathbf{p}^{o\xrightarrow[]{}h}\|=\|\mathbf{p}\|$) and preserves the
orientation w.r.t. the object normal
($\mathbf{n}_{i}^{o}\cdot\mathbf{n}_{j}^{h}=\hat{\mathbf{n}}\cdot\mathbf{n}$ &
$\mathbf{n}_{i}^{o}\cdot\mathbf{p}^{o\xrightarrow[]{}h}=\hat{\mathbf{n}}\cdot\mathbf{p}$).
Also, the procedure above describes the movement of human normal
$\mathbf{n}_{j}^{h}$ and relative position $\mathbf{p}^{o\xrightarrow[]{}h}$
following the object normal, when the object normal is taking the “shortest
path” to $\hat{\mathbf{n}}$ along the sphere surface $\mathbb{S}^{2}$. Note
that for the case of rigid object, we transform the current object (and
corresponding human) back to the original state using
$(\mathbf{T}^{\text{original}\xrightarrow[]{}\text{current}})^{-1}$ before
applying Eq. 13 and Eq. 14. For non-rigid objects (e.g., human mesh), we
directly apply Eq. 13 and Eq. 14.

###  A.4 Texture Generation for InterCap [37]

Intercap [37] does not provide texture for the 3D objects, hence we generate
the texture for the objects using an off-the-shelf mesh stylization method
[74]. The text prompts used for stylization are generated using ChatGPT [66],
where we insert the following query:

> Give a simple appearance description of an object of given categories as a
> form of “a $\\{\text{{category}}\\}$, $\\{\text{{appearance
> description}}\\}$”.

Figure 11: Application of Contact & Orientation Tendency. Leveraging the
contact and orientation tendency information derived from our method, we
present a fully automatic CAD model fitting pipeline for the task of 3D Scene
Reconstruction from Single Image. The proposed pipeline is zero-shot (i.e.,
applicable for any CAD models in 3D scenes), and does not require any
additional training.

##  Appendix B Applications

Our method can generate 3D affordance samples (given as 3D human and object
pairs), where we can learn the affordance primitives for versatile use. We
provide example applications of the learned primitives below.

###  B.1 Application of Contact & Orientation tendency

We can learn the 3D affordance primitives for any CAD model using our method,
and infer contact and orientation tendency information of the object; which
opens up the possibility of leveraging human cues for CAD-related 3D tasks. We
demonstrate this by addressing the task of 3D Scene Reconstruction from Single
Image. The idea is to generate human cues from the image, and utilize the
human cues along with our contact and orientation tendency information to fit
the 3D CAD models.

Given the image from CAD Estate [55, 10, 106], the pipeline starts by applying
Adaptive Mask Inpainting (Sec. 3.2) to generate multiple images of humans
interacting with the scene. We then predict 3D humans (and weak perspective
camera parameters) from the generated images using off-the-shelf human
prediction model [63]. Using the perspective camera parameters provided by CAD
Estate [55, 10, 106], we convert the 3D humans from the weak perspective
camera into perspective camera framework by optimizing human joint
reprojection error, similar to CHORUS [26].

After placing the humans in 3D space, we utilize the learned contact and
orientation tendency information to fit CAD model into 3D scenes.
Specifically, we declare two losses: (1) Chamfer Distance between contact
regions of human and object to resolve the proximity ambiguity; (2)
Orientation Alignment Loss, where we choose the human vertex with high
orientation tendency ($j^{*}$), and set the normal orientation with highest
marginal probability value $\mathcal{P}_{ij^{*}}(\mathbf{n})$ as target human
normal: which is used to align the object normals $\mathbf{n}_{i}^{o}$ in a
way such that the generated human normal of $j^{*}$th vertex match the target
human normal (after canonicalization via $\mathbf{n}_{i}^{o}$). By optimizing
the rotation and translation of the CAD model w.r.t. the weighted sum of two
losses, we can fit the CAD models into the 3D scene.

Figure 12: Application of Regionwise Contact. We can leverage the learned
regionwise contact information derived from our method, to generalize and
fully automate the existing pipeline [99] for the task of 3D Human-Object
Reconstruction from Single Image. The proposed pipeline is zero-shot (i.e.,
applicable for any 3D Object), and does not require any additional training.

We demonstrate the results in Fig. 11. Our method is fully automated and zero-
shot, meaning it is applicable to any CAD model. Also, since we leverage
generated 3D human cues, our scene reconstruction results are human-aware.

###  B.2 Application of Regionwise Contact

From our learned primitives, we can also infer regionwise contact (as
described in Sec. 5.3) between humans and objects. We can directly apply this
information to improve existing pipelines for the task of 3D Human-Object
Reconstruction from Single Image. Specifically, we can improve PHOSA [99],
which requires manual annotation of regionwise contact as input. By simply
replacing the manual provision of regionwise contact information with our
automatically learned contact (easily derived from the primitives), we can
fully automate the pipeline, only requiring a single image as input.
Furthermore, we can make PHOSA zero-shot, as our method can potentially return
regionwise contact information for any 3D objects, meaning that PHOSA can also
be applied to any 3D objects. We provide results in Fig. 12.

Figure 13: Additional Results for Contactual Affordance. We provide additional
contactual affordance results for boxlarge, boxmedium, basketball and
chairwood objects from BEHAVE [2].

##  Appendix C Additional Qualitative Results

We provide additional contactual affordance examples in Fig. 13, orientational
affordance examples in Fig. 14, and spatial affordance examples in Fig. 15.
Notably, the monitor example in Fig. 14 shows the strength of our method when
modelling affordance even for objects without any physical interaction with
the human. This is possible since the orientational affordance does not depend
on proximity between object and human, as $f_{\text{orientation}}$ only
depends on normal $\mathbf{n}$, thus marginalizing the effect of relative
position $\mathbf{p}$ when computing expectations as described in Sec. 4.

##  Appendix D Limitations & Future Works

Spatial Bias in Inpainting Diffusion Models. Our method utilizes inpainting
diffusion models [75, 78] to insert humans into object images; however, the
diffusion model may possess spatial biases, which may alter the inpainting
results as the properties of inpainting mask (e.g., center location, aspect
ratio, resolution) differs. For example, diffusion model may not be able to
generate humans if the objects that usually interact with hands (e.g., sports
ball) are rendered on the bottom side of the image. Future research can
further improve the spatial bias in diffusion models, or the mask selection
procedure to reduce the number of unnecessary generations.

Incorrect HOI Prompt Generation. During the prompt generation step, there is a
chance for the vision-language model [67] to misidentify the object in the
image, resulting in incorrect HOI prompts that describe implausible situations
for the given object.

Figure 14: Additional Results for Orientational Affordance. We showcase the
orientational affordance results for monitor and tennis racket.  Figure 15:
Additional Results for Spatial Affordance. We showcase the spatial affordance
results per body-part for motorcycle.

Limits and Potentials of Adaptive Mask Inpainting. We propose an adaptive mask
algorithm to preserve the original object during inpainting. However, the
method depends on segmentation models to adapt the inpainting mask, and the
errors during segmentation may affect the inpainting results. For example, if
the segmentation model predicts part of the object as human due to various
reasons (e.g., the texture of the object is similar to the texture of the
generated human), the adaptive mask algorithm may not work well as the
following inpainting mask occludes the object also. One potential approach for
improvement is to use better segmentation models, such as Grounded SAM [52,
45]. Also, while we use adaptive mask inpainting only for the human insertion
task, the algorithm can be applied to any categories the segmentation model
allows, opening possibilities such as open-vocabulary object insertion into
scene image.

Potentials of Virtual Triangulation. During virtual triangulation, we select
the inlier joints from different views to optimize the depth of the 3D human.
Since the inlier joint set exists for every 3D human, one promising future
direction includes utilizing these inlier sets to cluster the multimodal
distribution of affordance and find modes (e.g., a person may sit on the
chair, but also stand on the chair).

Bias due to Filtering. Employing heavy filtering at the end of the pipeline
may result in bias. For example, filtering out humans with high collision may
cause the remaining samples to “slide out” the object, especially if the
object is complex and is highly likely to collide given the plausible posture
(e.g., motorcycle). One alternative is applying soft filtering (i.e., applying
confidence weights instead of removing images with hard thresholds).

Low Granularity. Our primitive representation returns pointwise normal
distribution, which consumes large memory when the resolution of human and
object mesh is high; enforcing the method to downsample the human and object
mesh (refer to Sec. 4). Such limits in resolution may cause a lack of details,
especially when modeling interactions with dexterous objects. It is worth
exploring the use of implicit 3D representation for human and object surfaces
(e.g., SDF [65], DMTet [80]), as such representations model the continuous
surface as function.

Modelling Dexterous Interactions. Besides the limits due to low granularity,
the limits in the diffusion model and 3D human prediction model also hinder
modeling dexterous interactions (e.g., cup). For example, the diffusion model
may struggle to generate high-quality hands when generating humans. For future
research, employing diffusion models specifically trained on hand images to
generate hand images, and using close-view (toward hand) cameras could be
beneficial.

Possible Improvements in Primitives and its Derivatives. We present a novel
primitive representation of 3D affordance, and provide a way to derive contact
information. While the method to derive contact is justified, there is room
for improvement from the current definition of $f_{\text{contact}}$: for
example, one may consider modeling pressure, to further generalize our
representation to deformable objects. The orientational affordance term also
has room for improvement. While the current definition of orientational
affordance quantifies the tendency of orientation effectively (as a negated
entropy term), it does not regard the main source of the tendency. For
example, in the example of chair, the feet also pose a high tendency as feet
usually head toward the ground; however, it is unfounded to say that the
object causes such tendency; it is more likely to say it is due to
gravitational pull. A promising future direction is to distinguish and
quantify the source of orientational tendency.

Expanding to Non-Watertight Mesh. Our primitive representation can be easily
extracted from non-rigid mesh (e.g., human mesh), as long as the mesh provides
a closed surface and surface normal can be defined. One possible future
direction is to improve the primitive representation to be easily extractable
from any 3D surfaces (mesh or other surface representations), including non-
watertight mesh, and sharp meshes where defining surface normals is non-
trivial.

Evaluation Metrics. As the first in this direction, we evaluate and justify
our method with simple metrics proposed in Sec. 5.2. It is worth exploring
additional metrics to effectively quantify the validity of our method,
especially for comparing the volumetric quality of 3D humans and objects for
justification of 2D-to-3D lifting methods.

Potential Applications. Our new method and representation provide multitudes
of possible applications, as demonstrated in Sec. B. We list potential
downstream applications: (1) Large-scale 3D affordance dataset generation; (2)
Object recognition from 3D human posture (similar to Object Pop-up [71]); (3)
Action recognition from human-object interaction sequence; (4) Application for
robotics, especially for humanoids.