#  Multimodal Sense-Informed Forecasting of 3D Human Motions

Zhenyu Lou1 Qiongjie Cui2
Haofan Wang3 Xu Tang3 Hong Zhou1
1Zhejiang University, 2Nanjing University of Science and Technology,
3Xiaohongshu Inc
<EMAIL_ADDRESS><EMAIL_ADDRESS>Corresponding author

###### Abstract

Predicting future human pose is a fundamental application for machine
intelligence, which drives robots to plan their behavior and paths ahead of
time to seamlessly accomplish human-robot collaboration in real-world 3D
scenarios. Despite encouraging results, existing approaches rarely consider
the effects of the external scene on the motion sequence, leading to
pronounced artifacts and physical implausibilities in the predictions. To
address this limitation, this work introduces a novel multi-modal sense-
informed motion prediction approach, which conditions high-fidelity generation
on two modal information: external 3D scene, and internal human gaze, and is
able to recognize their salience for future human activity. Furthermore, the
gaze information is regarded as the human intention, and combined with both
motion and scene features, we construct a ternary intention-aware attention to
supervise the generation to match where the human wants to reach. Meanwhile,
we introduce semantic coherence-aware attention to explicitly distinguish the
salient point clouds and the underlying ones, to ensure a reasonable
interaction of the generated sequence with the 3D scene. On two real-world
benchmarks, the proposed method achieves state-of-the-art performance both in
3D human pose and trajectory prediction. More detailed results are available
on the page: https://sites.google.com/view/cvpr2024sif3d.

##  1 Introduction

Forecasting future human poses, from the observed ones, stands as a
fundamental application in the domain of machine intelligence, autonomous
vehicles, and human-robot collaboration [18, 42, 65, 39, 11, 48, 73, 40, 33,
12, 70].

Human motion is fundamentally intertwined with and constrained by its
environment [59, 36, 55], a reality often overlooked in previous motion
prediction methods [42, 2, 41, 33]. This oversight could lead to noticeable
anomalies and unrealistic outcomes in the predictions. We notice this issue
and aim to solve it.

In robotics, scene information is commonly represented as a point cloud [6,
35, 57, 71], where a large number of points are constructed as a collection to
describe a 3D environment. Existing methods perform motion
prediction/generation by encoding the entire scene information into a unified
global embedding [5, 22, 73]. Despite their good performance, we note that not
all information within the point cloud is equally relevant to the motion
prediction task; instead, only a small subset is salient. Moreover, the global
embedding fails to capture the intricate local details of the scene;
therefore, the generation may deviate from the intended scene semantics. In
addition, the human gaze is a valuable manifestation, providing an intention
that the human is likely to reach or a direction that the human is likely to
move toward [16, 63, 72]. By jointly considering the 3D scene and human gaze,
it is possible to capture the human’s behavior of wanting to move toward a
specific location, and thus generate more reasonable motion sequences.

Towards this end, we propose a novel multimodal sense-informed predictor for
real-world 3D scenarios, named SIF3D. It integrates past motion sequences with
critical data from external 3D scenes and the observer’s internal gaze to
enhance scene-aware human activity representations [8, 5, 22, 73]. Following
the recent progress, both motion encoder and scene encoder are introduced to
extract the motion embedding and scene embedding [53], respectively. To
further refine point cloud data utilization and focus on key points affecting
actions, we propose two novel scene-aware cross-modal attention mechanisms:
ternary intention-aware attention (TIA) integrates human gaze, motion, and
scene data through a comprehensive analysis to concentrate on globally salient
points; semantic coherence-aware attention (SCA) focuses on identifying the
local salient scene points by analyzing their semantic connections with human
poses. SIF3D adeptly identifies salient points and the underlying ones in the
3D scene (as in Figure LABEL:f1), where the salient points are more likely to
interact with the human, and thus the generated motion sequence is more
realistic. Additionally, inspired by [67], we apply a geometry discriminator
to boost the overall authenticity of the predicted motions.

Our contributions are as follows: (1) We introduce SIF3D which considers both
the external scene and the internal human gaze, and is capable of generating
the accurate future motion, as well as the trajectory, within given 3D
scenarios. (2) Both semantic coherence-aware attention and ternary intention-
aware attention are proposed, to explicitly distinguish salient point clouds
to the local human pose representation, and the one to the global trajectory
planning. (3) We show that, on two benchmarks GIMO and GTA-1M, the proposed
SIF3D achieves state-of-the-art performance both in human pose and trajectory
prediction.

##  2 Related Work

3D human motion prediction has witnessed substantial advancements,
particularly with the rise of deep end-to-end approaches [27, 4, 30, 56, 20].
Prior works typically leverage RNNs [43, 44, 15, 18, 19, 61] to capture the
motion temporal dependence, facing the limitations of discontinuity and error
accumulation. To address it, recent alternatives, CNN-based [32] and
transformer-based [1] methods have been proposed. To effectively exploit both
spatial and temporal dependencies, [1] designs a spatial-temporal attention
schema, while [70] proposes a dependencies modeling AuxFormer. Notably, GCNs-
based predictors are the dominant techniques and still evolve nowadays [42,
10, 9, 11, 31, 34, 33].

We also notice that current methods often fail to explicitly consider real 3D
scenes, leading to physically implausible and artifactual predicted
trajectories and poses. To solve these limitations, our SIF3D is proposed.

Scene-aware motion generation. Considering the connection between human
activities and scene context, scene-aware motion generation has become a focal
point of research [8, 5, 22, 73]. Early attempts [8, 5, 67] incorporate object
bounding boxes, 2D scene images, or depth maps to contextualize scenes. [22]
further introduces the object’s geometry to estimate the interaction and
generate interactive motions, while adaptations of the A$*$ algorithm have
been proposed for collision-averse trajectory planning [22, 66].

The adoption of directly accessible 3D point clouds for scene representation
marks a new trend in scene-aware motion generation [73, 68, 25, 74]. [68]
proposes to condition the motion synthesis through text descriptions, while
[25] contributes a conditional generative diffusion model for scene
understanding, and [74] advances a structure-aware motion framework for active
object tracking. Notably, BiFu [73] introduces a bidirectional fusion strategy
for motion prediction. However, its neglect of local scene intricacies limits
salient point detection, leading to ambiguity in cross-modal motion encoding
across the scene. In contrast, our SIF3D adeptly isolates critical scene
points, enhancing the integration of pivotal scene data into motion
forecasting.

Figure 1:  The architecture of SIF3D. SIF3D incorporates three modalities of
input, the past motion sequence, the 3D scene point cloud, and the human gaze.
First, MotionEncoder encodes past motion sequence into a motion embedding
$\boldsymbol{f}_{m}$, and the 3D scene $\boldsymbol{S}$ is encoded into
$\\{\boldsymbol{\hat{S}},\boldsymbol{\hat{S}}_{global}\\}$ through PointNet++
[53]. Then, our TIA mechanism compresses motion embedding in the temporal
dimension and searches for global salient points in the scene for trajectory
planning. In addition, human gaze point $\boldsymbol{G}$ is introduced to
index the scene point cloud for gaze point scene feature extraction. The SCA
mechanism, on the other hand, is designed to capture local salient points in
the scene for each independent pose. A TrajectoryPlanner and a PosePredictor
are applied to predict trajectory and poses, respectively. And finally, the
predicted motion sequence is generated through a MotionDecoder, which is
supervised by the geometric discriminator.

3D scene understanding. 3D Point cloud, a vital geometric data structure, has
been traditionally transformed into regular 3D voxel grids or collections of
images, because of its irregular representation [45, 52, 69, 52, 58]. However,
these methods are either computationally expensive or task-specific, making
them hard to transfer into downstream tasks. To address it, PointNet and
PointNet++ [51, 53] introduces a unified architecture directly consuming point
clouds, proving highly efficient and effective. Since then, various methods
have been proposed for different tasks, _e.g_., 3D object detection [21, 49,
50], semantic segmentation [7, 17, 62], and instance segmentation [14, 23, 24,
28].

Although 3D scenes are crucial for information, effectively utilizing them in
human motion prediction presents a challenge. Previous works [51, 53] have
shown that many point cloud points are often superfluous or irrelevant. SIF3D
advances this understanding by selectively focusing on salient points within
the 3D scene, leading to enhanced prediction quality for human poses and
global trajectories.

##  3 Proposed Method

Figure 1 illustrates the overall architecture of our SIF3D. It begins with the
proposed MotionEncoder and the vanilla PointNet++ [53], to encode the observed
sequence, and 3D point clouds. Then, ternary intention-aware attention (TIA)
and semantic coherence-aware attention (SCA) are used to extract crossmodal
features from salient scene points, followed by a TrajectoryPlanner and a
PosePredictor to plan future trajectories and predict poses, respectively.
Ultimately, the predicted motion sequence is generated by a MotionDecoder,
supervised by the geometric discriminator.

###  3.1 Problem Setup

Given $T$ historical motion
$\boldsymbol{X}_{1:T}=\\{\boldsymbol{x}_{1},\boldsymbol{x}_{2},\cdots,\boldsymbol{x}_{T}\\}$
in the scene cloud $\boldsymbol{S}\in\mathbb{R}^{n\times 3}$, corresponding
with the human gaze point sequence
$\boldsymbol{G}_{1:T}=\\{\boldsymbol{g}_{1},\boldsymbol{g}_{2},\cdots,\boldsymbol{g}_{T}\\}$,
we aim to predict the future motion $\boldsymbol{Y}_{1:\Delta T}$. Here, $n$
denotes the size of the scene point clouds,
$\boldsymbol{g}_{k}\in\mathbb{R}^{3}$ is the position of human gaze (denoted
as the intersection of the human eye ray and 3D scene). Each pose is described
as
$\boldsymbol{x}_{k}=(\boldsymbol{t}_{k},\boldsymbol{o}_{k},\boldsymbol{p}_{k})$
[73], where $\boldsymbol{t}_{k}\in\mathbb{R}^{3}$ denotes the global
translation, $\boldsymbol{o}_{k}\in SO(3)$ denotes the orientation, and
$\boldsymbol{p}_{k}\in\mathbb{R}^{32}$ refers to the body pose embedding. The
SMPL-X body mesh and the joints position
$\boldsymbol{j}_{k}\in\mathbb{R}^{23\times 3}$ can be then obtained via VPoser
[47]. Following [73], we set $T\\!=\\!3\text{-}sec$ and $\Delta
T\\!=\\!5\text{-}sec$ to achieve a more challenging long-term prediction
compared with the previous tasks ($\Delta T=1\text{-}sec$) [42, 33].

In summary, with the model parameter $\theta$, our goal is:

$\arg\max P(\boldsymbol{Y}_{1:\Delta
T}|\boldsymbol{X}_{1:T},\boldsymbol{S},\boldsymbol{G}_{1:T};\theta).\vspace{-0.4em}$
(1)

Note that the motion trajectory involves both global translation
$\boldsymbol{T}_{1:T+\Delta T}$ and orientation $\boldsymbol{O}_{1:T+\Delta
T}$. Our primary focus in trajectory evaluation centers on
$\boldsymbol{T}_{1:T+\Delta T}$ due to its significant impact on the overall
motion. The orientation is implicitly reflected in the joints positions
$\boldsymbol{J}_{1:T+\Delta T}$.

In contrast to numerous prior literature [60, 26], our task incorporates 3D
scene and human gaze information, with the objective of producing semantically
reasonable, and physically stable motion predictions within a given 3D scene.
Our effort aims to bridge the divide between motion prediction and real-world
scenarios, fostering diverse applications grounded in realistic 3D scenes.

###  3.2 Multimodal Encoding

In addition to the observed motion sequence $\boldsymbol{X}_{1:T}$, SIF3D
integrates two additional input modalities: the external 3D scene
$\boldsymbol{S}$, and the internal human’s gaze $\boldsymbol{G}_{1:T}$. The
motion and the scene are first encoded into a unified space, while the gaze
point is used for scene point indexing to obtain the gaze point feature (see
Sec-3.3).

A transformer-based MotionEncoder [64] is proposed to transfer the passing
motion sequence $\boldsymbol{X}_{1:T}$ into a motion embedding
$\boldsymbol{f}_{m}$. Note that, instead of using the original observation
$\boldsymbol{X}_{1:T}$ with $T$ frames, we pad it with the last observed pose
$\boldsymbol{x}_{T}$ repeated $\Delta T$ times to construct the virtual
sequence
$\\{\boldsymbol{X},\boldsymbol{x}_{T},...,\boldsymbol{x}_{T}\\}\in\mathbb{R}^{(T+\Delta
T)\times c_{m}}$, which is taken as the input of the MotionEncoder:

$\boldsymbol{f}_{m}=\text{MotionEncoder}(\\{\boldsymbol{X},\boldsymbol{x}_{T},...,\boldsymbol{x}_{T}\\}),\vspace{-0.4em}$
(2)

where the padded frames need to be predicted, and will be replaced after the
inference to achieve the final prediction. Here,
$\boldsymbol{f}_{m}\in\mathbb{R}^{(T+\Delta T)\times c_{m}}$, and $c_{m}$ is
the dimension.

Meanwhile, 3D point cloud $\boldsymbol{S}$ is encoded using PointNet++ [53] to
obtain per-point scene features $\boldsymbol{\hat{S}}$ and the global scene
embedding $\boldsymbol{\hat{S}}_{global}$:

$\displaystyle\boldsymbol{\hat{S}},\boldsymbol{\hat{S}}_{global}=\text{PointNet++}(\boldsymbol{S}),$
(3)

where $\boldsymbol{\hat{S}}\in\mathbb{R}^{n\times c_{s}}$,
$\boldsymbol{\hat{S}}_{global}\in\mathbb{R}^{c_{s}}$,

###  3.3 Ternary Intention-Aware Attention

Scene information serves as a crucial constrain in predicting the long-term
trajectory [8, 5, 67]. Moreover, as people often direct their gaze toward the
objectives of their movements, human gaze becomes an essential cue, reflecting
the potential destination of the motion trajectory [73].

To enhance trajectory planning by effectively utilizing scene and gaze
information, we propose ternary intention-aware attention (TIA). TIA is
designed to capture global salient scene features and analyze human intention
through 3 distinct modalities: motion embedding $\boldsymbol{f}_{m}$, 3D scene
$\\{\boldsymbol{\hat{S}},\boldsymbol{\hat{S}}_{global}\\}$, and human gaze
$\boldsymbol{G}_{1:T}$.

Since long-term trajectory planning hinges on the semantics of the entire
motion sequence, TIA first extracts a comprehensive global motion
representation. It encodes the input motion embedding to
$\boldsymbol{f}^{\text{TIA}}_{m}$, and then condenses it into a global motion
representation $\boldsymbol{f}^{\text{TIA}}_{gm}$ through a temporal
aggregator. Various methods can be employed to design this aggregator, while
we simply adopt the last motion embedding as the global representation in our
case, aligning with a widely-used technique in NLP [54]:

$\displaystyle\boldsymbol{f}^{\text{TIA}}_{m}$
$\displaystyle=\text{TrajectoryEncoder}(\boldsymbol{f}_{m}),$ (4)
$\displaystyle\boldsymbol{f}^{\text{TIA}}_{gm}$
$\displaystyle=\text{Aggregate}(\boldsymbol{f}^{\text{TIA}}_{m}).$ (5)

The global salient scene features are grounded in the global scene embedding
$\boldsymbol{\hat{S}}_{global}$ as a foundation. To achieve the extraction of
effective global salient scene features, TIA assesses the global salience
$s_{g}\in\mathbb{R}^{n}$ for each scene point to
$\boldsymbol{f}^{\text{TIA}}_{gm}$, focusing only on the salient points. To
exclude underlying points, a simple method arises by multiplying the salience
with the corresponding scene point features. This process yields the global
salience $s_{g}\in\mathbb{R}^{n}$ and the crossmodal global scene-motion
features $\boldsymbol{f}^{\text{TIA}}_{sm}$ through a crossmodal attention
mechanism as follows:

$\displaystyle{\boldsymbol{Q}^{\text{TIA}};\,(\boldsymbol{K}^{\text{TIA}},\boldsymbol{V}^{\text{TIA}})=\text{Linear}(\boldsymbol{f}^{\text{TIA}}_{gm});\,\text{Linear}(\boldsymbol{\hat{S}})},$
(6) $\displaystyle
s_{g}=\text{softmax}\bigg{(}\dfrac{\boldsymbol{Q}^{\text{TIA}}*(\boldsymbol{K}^{\text{TIA}})^{T}}{\sqrt{c}}\bigg{)},$
(7)
$\displaystyle\boldsymbol{f}^{\text{TIA}}_{sm}=\text{broadcast}(\boldsymbol{\hat{S}}_{global}+s_{g}\cdot\boldsymbol{V}^{\text{TIA}}),$
(8)

where $c$ is the attention dimension. We broadcast the global salient scene
features to the same shape as $\boldsymbol{f}_{m}$ to disperse it across the
entire motion sequence.

Moreover, recognizing the informative nature of the gaze regarding the
subject’s intent, we also introduce the human gaze into cross-modal attention.
The gaze point $\boldsymbol{G}_{1:T}$, defined as the intersection between the
human eye’s line of sight and the 3D scene, correlates with a specific scene
point. By indexing the scene point cloud features $\boldsymbol{\hat{S}}$ with
the gaze point, we derive the gaze-related scene feature, representing the
potential human intention. This $T$-frame gaze feature (padded with last frame
for $\Delta T$ times to fit the length) is then encoded to align with the
length of motion sequence using a GazeEncoder:

$\displaystyle\boldsymbol{f}_{gaze}=\text{GazeEncoder}(\\{\boldsymbol{\hat{S}}[\boldsymbol{G}],\boldsymbol{\hat{S}}[\boldsymbol{g}_{T}],...,\boldsymbol{\hat{S}}[\boldsymbol{g}_{T}]\\}),$
(9)

where $[\cdot]$ represents the indexing operation, $\boldsymbol{f}_{gaze}$
represents the encoded gaze features. Finally, TIA integrates the features
from the three modalities: $\boldsymbol{f}_{m}$ for motion,
$\boldsymbol{f}^{\text{TIA}}_{sm}$ for scene, and $\boldsymbol{f}_{gaze}$ for
gaze. The output of TIA is then computed through a standard two-layer MLP
[64]:

$\displaystyle\boldsymbol{f}^{\text{TIA}}_{o}=\text{MLP}(\text{concat}(\boldsymbol{f}_{m},\boldsymbol{f}^{\text{TIA}}_{sm},\boldsymbol{f}_{gaze})).$
(10)

###  3.4 Semantic Coherence-Aware Attention

We introduce semantic coherence-aware attention (SCA) to integrate local
salient scene details into per-frame pose prediction. In contrast to TIA, SCA
computes scene point salience for each frame independently. It initiates with
a PoseEncoder to encode the input motion embedding $\boldsymbol{f}_{m}$ into
pose embedding $\boldsymbol{f}^{\text{SCA}}_{m}$. Then, the local scene point
salience $s_{l}\in\mathbb{R}^{(T+\Delta T)\times n}$ is evaluated between the
scene features $\boldsymbol{\hat{S}}$ and each of the $T+\Delta T$ pose
embedding:

$\displaystyle\boldsymbol{f}^{\text{SCA}}_{m}=\text{PoseEncoder}(\boldsymbol{f}_{m}),$
(11)
$\displaystyle{\boldsymbol{Q}^{\text{SCA}};\,\boldsymbol{K}^{\text{SCA}},\boldsymbol{V}^{\text{SCA}}=\text{Linear}(\boldsymbol{f}^{\text{SCA}}_{m}),\,\text{Linear}(\boldsymbol{\hat{S}})},$
(12) $\displaystyle
s_{l}=\text{softmax}\bigg{(}\dfrac{\boldsymbol{Q}^{\text{SCA}}*(\boldsymbol{K}^{\text{SCA}})^{T}}{\sqrt{c}}\bigg{)},$
(13)

where $c~{}$ represents the attention dimension.

SCA places a greater emphasis on fine-grained local scene information,
exhibiting heightened sensitivity to spatial details. In practice, points
situated closer to the subject and aligned with the human body are empirically
considered more salient. To adopt this spatial attention mechanism, we
incorporate a spatial salience bias to $s_{l}$, denoted as
$s_{spatial}\in\mathbb{R}^{(T+\Delta T)\times n}$, inspired by [38, 37, 13].
The absolute scene points position $\boldsymbol{S}$ is normalized to the
relative positions $\boldsymbol{S}_{rel}\in\mathbb{R}^{(T+\Delta T)\times
n\times 3}$ according to the predicted human translation
$\boldsymbol{\hat{T}}$ and orientation $\boldsymbol{\hat{O}}$ from
TrajectoryPlanner. Then, the spatial salience bias $s_{spatial}~{}$ is
determined based on $\boldsymbol{S}_{rel}$ through a standard two-layer MLP:

$\displaystyle\boldsymbol{S}_{rel}=\text{Normalize}(\boldsymbol{S};\boldsymbol{\hat{T}},\boldsymbol{\hat{O}}),$
(14) $\displaystyle s_{spatial}=\text{MLP}(\boldsymbol{S}_{rel}).$ (15)

The local salient scene features are then extracted based on locally salient
scene points, while the low-salience underlying points are ignored through
salience multiplication:

$\displaystyle\boldsymbol{f}^{\text{SCA}}_{sm}=(s_{l}+s_{spatial})\cdot\boldsymbol{V}^{\text{SCA}}.$
(16)

Finally, SCA produces output by combining the motion features
$\boldsymbol{f}_{m}$ with the local salient scene features
$\boldsymbol{f}^{\text{SCA}}_{sm}$:

$\displaystyle\boldsymbol{f}^{\text{SCA}}_{o}=\text{MLP}(\text{concat}(\boldsymbol{f}^{\text{SCA}}_{m},\boldsymbol{f}^{\text{SCA}}_{sm})).$
(17)

###  3.5 Motion Sequence Generation

Predicting future motion involves considering both the trajectory and poses.
In our way of multimodal feature extraction, TIA adds global scene features
and human intention, while SCA brings in local scene details. Since trajectory
planning looks further ahead, and human poses are more related to the local
scene semantics, it makes sense in practice that TIA could enhance trajectory
prediction, while SCA helps predict poses. Thus we use the features from TIA
to predict the global translation $\boldsymbol{\hat{T}}$ and orientation
$\boldsymbol{\hat{O}}$, while relying on SCA for prediction poses
$\boldsymbol{\hat{P}}$:

$\displaystyle\boldsymbol{\hat{T}},\boldsymbol{\hat{O}}=\text{TrajectoryPlanner}(\boldsymbol{f}^{\text{TIA}}_{o}),$
(18)
$\displaystyle\boldsymbol{\hat{P}}=\text{PosePredictor}(\boldsymbol{f}^{\text{SCA}}_{o}),$
(19)

where TrajectoryPlanner/PosePredictor are linear layers.

The predicted trajectory and pose are then combined to generate the final
motion sequence $\boldsymbol{\hat{Y}}$ through a motion decoder. To achieve
precise adjustment of joint positions, the motion decoder applies a GCN
architecture [29], and we calculate the SMPL-X human skeleton based on the
predicted trajectory and pose embedding through the frozen pre-trained SMPL-X
model and VPoser:

Dataset |  GIMO [73] |  GTA-1M [5]
---|---|---
Traj-path | Traj-dest | MPJPE-path | MPJPE-dest | Traj-path | Traj-dest | MPJPE-path | MPJPE-dest
LTD [42] | 681 | 890 | 158.7 | 204.6 | 691 | 1051 | 175.0 | 251.8
SPGSN [33] | 739 | 910 | 159.5 | 203.1 | 805 | 1115 | 167.3 | 241.4
AuxFormer [70] | 688 | 893 | 173.5 | 205.4 | 688 | 1078 | 187.8 | 262.0
SIF3D | 737 | 888 | 169.6 | 205.6 | 772 | 1072 | 182.0 | 250.2
LTD [42]+Scene | 673 | 890 | 156.4 | 203.0 | 690 | 1029 | 175.8 | 245.6
SPGSN [33]+Scene | 700 | 893 | 157.0 | 202.2 | 767 | 1054 | 167.8 | 236.9
AuxFormer [70]+Scene | 732 | 902 | 165.1 | 202.4 | 745 | 1028 | 167.0 | 246.0
SIF3D w/ Scene | 700 | 823 | 164.2 | 199.6 | 746 | 967 | 175.2 | 231.4
LTD [42]+Gaze | 655 | 807 | 157.6 | 203.5 | 687 | 996 | 169.1 | 246.0
SPGSN [33]+Gaze | 681 | 857 | 158.2 | 201.7 | 748 | 1031 | 164.4 | 234.8
AuxFormer [70]+Gaze | 627 | 755 | 165.8 | 202.8 | 672 | 932 | 176.0 | 256.6
SIF3D w/ Gaze | 603 | 714 | 163.1 | 199.1 | 680 | 917 | 171.5 | 233.0
LTD [42]+Scene+Gaze | 655 | 801 | 155.6 | 202.0 | 702 | 982 | 167.8 | 241.9
SPGSN [33]+Scene+Gaze | 702 | 982 | 156.9 | 202.0 | 737 | 1018 | 162.7 | 238.1
AuxFormer [70]+Scene+Gaze | 622 | 746 | 161.5 | 201.1 | 672 | 916 | 165.3 | 236.8
BiFu [73] w/ Scene+Gaze | 661 | 727 | 167.8 | 205.0 | 683 | 903 | 164.6 | 234.2
SIF3D w/ Scene+Gaze | 590 | 666 | 156.6 | 195.7 | 626 | 836 | 164.9 | 227.7
Table 1:  Comparison of trajectory deviation and MPJPE (in $mm$) over the
sequences of the GIMO [73] and GTA-1M [5] datasets. The best result is
highlighted in bold. From the results, we observe that the 3D scene and gaze
information can boost all the methods, and the proposed SIF3D obtains both
lower trajectory deviation and smaller MPJPE in almost all scenarios compared
to the previous methods.

$\displaystyle\boldsymbol{\hat{J}}=\text{SMPL-X}(\boldsymbol{\hat{T}},\boldsymbol{\hat{O}},\boldsymbol{\hat{P}},\text{VPoser}(\boldsymbol{\hat{P}})),$
(20)
$\displaystyle\boldsymbol{\hat{Y}}=\text{MotionDecoder}(\boldsymbol{\hat{J}}),$
(21)

where $\boldsymbol{\hat{J}}$ represents the reconstructed joints positions,
and $\boldsymbol{\hat{Y}}$ denotes the final predicted motion sequence.

To further improve the physical stability of the predicted motion sequence, a
geometric discriminator is introduced following [67] to classify the fake and
true motion sequences to the ground truth within the 3D scenarios.

###  3.6 Implement Details

Architecture Details: In our experiments, we employ a 6-layer transformer
encoder for the MotionEncoder, a single-layer self-attention mechanism for
both the TrajectoryEncoder and the PoseEncoder, and a 6-layer graph
convolutional network for the MotionDecoder. We configured the stack size of
both TIA and SCA to be 2, indicating that SIF3D incorporates 2 TIA-blocks and
2 SCA-blocks. Furthermore, we set all embedding dimensions, including
attention dimension $c$, motion embedding dimension $c_{m}$, and scene
embedding dimension $c_{s}$, to 256. The MLPs in TIA and SCA are both 2-layer
feed-forward with a hidden dimension of 1024. Finally, the geometry
discriminator is designed with a 3-layer transformer decoder.

Training Setup: We train our model using an AdamW optimizer with an initial
learning rate of 0.0004. An exponential learning rate scheduler is applied
with a decay rate of 0.98. The models are all trained for 100 epochs with a
batch size of 8 on a single NVIDIA RTX3090 GPU.

##  4 Experiments

###  4.1 Experimental Setup

Dataset-1: GIMO [73] records three modalities, including (1) full-body SMPL-X
poses with $\approx$ 129$K$ frames. (2) 3D geometry scene scanned from LiDAR
sensors. (3) human gaze is recorded as a 3D coordinate. We down-sample the
motion sequences to 2fps for long-term prediction and maintain the original
train-test split [73].

Dataset-2: GTA-1M [5] comprises over 1000$K$ frames, which collects the human
pose from real game engine rendering. The dataset has clean 3D human pose and
camera annotations, and large diversity in human appearances, camera views: 10
large houses, 13 weathers, 50 human models, 22 walking styles, and various
actions.

Baselines: Our SIF3D is compared with 4 recent methods, _i.e_., LTD [42],
SPGSN [33], BiFu [73] and AuxFormer [70]. BiFu [73] and AuxFormer[70] are
transformer-based, while LTD [42] and SPGSN[33] are variants of GCNs. For a
fair comparison, for the methods that do not inherently consider 3D scenes or
human gaze, we concatenate the global scene embedding
$\boldsymbol{\hat{S}}_{global}$ to the motion sequence and include the gaze
embedding $\boldsymbol{f}_{gaze}$. Other aspects align with their open-source
codes and original settings.

Metrics: We evaluate SIF3D from 2 main aspects: trajectory and human pose. The
trajectory evaluation [73, 3, 46] quantifies the difference between the
predicted trajectory and the ground truth one. The MPJPE [42, 33, 2, 73]
evaluates the mean error in the position across all human joints.
Specifically, the result is analyzed from the intermediate poses and the final
destination: (1) Traj-path: the average trajectory deviation over all
predicted poses; (2) Traj-dest: the trajectory deviation of the end pose; (3)
MPJPE-path: the average MPJPE over the predicted sequence; (4) MPJPE-dest: the
MPJPE of the destination.

| Method | Gaze& | bedroom | classroom | garden | kitchen | lab | livingroom | seminarroom
---|---|---|---|---|---|---|---|---|---
| Scene |  0.5$s$ |  2.0$s$ |  5.0$s$ |  0.5$s$ |  2.0$s$ |  5.0$s$ |  0.5$s$ |  2.0$s$ |  5.0$s$ |  0.5$s$ |  2.0$s$ |  5.0$s$ |  0.5$s$ |  2.0$s$ |  5.0$s$ |  0.5$s$ |  2.0$s$ |  5.0$s$ |  0.5$s$ |  2.0$s$ |  5.0$s$
Traj | LTD [42] | $\times$ | 109 | 459 | 1147 | 195 | 415 | 613 | 149 | 578 | 1076 | 100 | 457 | 481 | 184 | 644 | 1165 | 127 | 624 | 1212 | 190 | 529 | 851
✓ | 108 | 411 | 966 | 192 | 422 | 524 | 147 | 496 | 560 | 88 | 528 | 589 | 149 | 516 | 878 | 133 | 551 | 1297 | 170 | 527 | 750
SPGSN [33] | $\times$ | 144 | 492 | 922 | 189 | 396 | 564 | 129 | 556 | 813 | 117 | 639 | 660 | 181 | 640 | 1014 | 153 | 703 | 1454 | 182 | 589 | 866
✓ | 112 | 475 | 965 | 180 | 407 | 497 | 172 | 540 | 661 | 156 | 640 | 677 | 120 | 515 | 919 | 135 | 649 | 1258 | 174 | 514 | 753
AuxFormer [70] | $\times$ | 185 | 541 | 1162 | 319 | 442 | 529 | 232 | 630 | 972 | 172 | 509 | 625 | 248 | 745 | 1113 | 151 | 621 | 1030 | 273 | 606 | 881
✓ | 212 | 472 | 813 | 228 | 427 | 532 | 231 | 536 | 577 | 225 | 544 | 555 | 246 | 511 | 621 | 273 | 647 | 1082 | 222 | 546 | 735
BiFu [73] | ✓ | 232 | 540 | 838 | 332 | 442 | 689 | 218 | 532 | 659 | 275 | 467 | 582 | 273 | 551 | 681 | 340 | 607 | 936 | 417 | 659 | 629
SIF3D | ✓ | 183 | 441 | 554 | 155 | 424 | 431 | 200 | 472 | 533 | 176 | 471 | 444 | 245 | 540 | 730 | 257 | 602 | 794 | 219 | 440 | 611
MPJPE | LTD [42] | $\times$ | 92.9 | 142.8 | 263.5 | 98.0 | 160.4 | 169.7 | 73.6 | 231.1 | 209.1 | 86.6 | 102.3 | 141.4 | 70.3 | 99.1 | 222.6 | 97.6 | 128.6 | 252.5 | 109.5 | 150.8 | 204.2
✓ | 85.5 | 134.9 | 236.6 | 110.1 | 192.8 | 159.8 | 67.8 | 227.0 | 204.6 | 91.0 | 97.7 | 128.3 | 74.3 | 104.1 | 237.1 | 111.3 | 140.7 | 271.7 | 117.2 | 156.0 | 199.9
SPGSN [33] | $\times$ | 89.0 | 131.4 | 232.8 | 102.9 | 170.5 | 160.5 | 62.3 | 218.8 | 205.5 | 87.5 | 97.1 | 131.6 | 66.8 | 98.5 | 240.1 | 103.6 | 124.1 | 285.2 | 110.5 | 149.2 | 191.5
✓ | 89.4 | 137.6 | 249.5 | 101.7 | 176.1 | 147.8 | 67.1 | 233.2 | 202.6 | 85.5 | 92.1 | 120.9 | 71.8 | 98.7 | 238.2 | 109.1 | 124.9 | 281.3 | 112.8 | 145.4 | 193.1
AuxFormer [70] | $\times$ | 114.9 | 173.2 | 259.7 | 137.1 | 178.7 | 162.4 | 105.3 | 230.5 | 229.9 | 128.5 | 108.0 | 125.1 | 113.1 | 144.0 | 231.9 | 156.3 | 132.7 | 259.5 | 130.9 | 160.7 | 204.9
✓ | 90.5 | 142.3 | 240.7 | 110.2 | 183.8 | 170.5 | 91.5 | 232.7 | 203.9 | 103.8 | 106.4 | 115.5 | 76.3 | 112.9 | 218.2 | 113.5 | 127.1 | 271.5 | 105.7 | 146.0 | 191.6
BiFu [73] | ✓ | 67.0 | 115.0 | 258.0 | 86.6 | 144.2 | 171.9 | 89.4 | 205.0 | 251.7 | 98.0 | 104.0 | 133.0 | 53.9 | 110.8 | 229.8 | 95.7 | 126.1 | 236.1 | 107.2 | 128.7 | 209.9
SIF3D | ✓ | 82.0 | 105.7 | 185.9 | 100.3 | 147.1 | 160.1 | 74.0 | 190.4 | 199.0 | 100.1 | 101.5 | 114.3 | 65.4 | 94.3 | 218.1 | 108.9 | 121.1 | 188.4 | 101.8 | 137.3 | 179.4
Table 2:  Performance details on the GIMO test set [73] using trajectory
deviation and MPJPE (in $mm$). Our SIF3D demonstrates strong performance
across all scenarios, particularly excelling in long-term predictions, and
achieves the best result on average.

###  4.2 Ablation of Multiple Modalities

The results in Table 1 highlight the vital role of gaze and 3D scene
information in scene-conditioned motion prediction. Note that, as GTA-1M lacks
human gaze information, we approximate it using the intersection point of the
ray facing the human face and the 3D scene. We observe that the introduction
of gaze and scene information improves the performance of all methods,
including the estimated gaze information in GTA-1M, with SIF3D benefiting the
most.

SIF3D excels by adeptly utilizing 3D scene information, surpassing baseline
methods that show only marginal gains. This success is attributed to SIF3D’s
proficient integration of scene features from both global and local
perspectives through TIA and SCA. On the other hand, the human gaze
significantly enhances trajectory planning by providing key insights into the
subject’s intentions and closely aligning with the motion sequence’s
destination, resulting in a notable decrease in trajectory deviation.

While the introduction of both scene and gaze enhances the performance
individually, their simultaneous incorporation yields optimal results across
the board. It evidences the effectiveness of collaborating external scene and
internal gaze features, establishing a robust foundation for improved motion
prediction.

###  4.3 Detailed Results

Figure 2:  Visualizations of our SIF3D compared with the SoTA BiFu, under the
scenarios of (a) living room and (b) bedroom. The top is the results of BiFu
[73], which equally treats all scene points; in contrast, the middle row is
our SIF3D, where the salient points are highlighted in red, and the underlying
points are in blue. For the sake of clarity, the predicted sequence is
presented from the vertical view, whole-seq view, and end-pose view. We note
that the red human meshes are the ground truth, while the blue ones indicate
the predictions. At the bottom, we present the local scene salience heatmap
across time for SIF3D, with a time interval of 2 seconds.

Next, we analyze the performance of each method under different timestamps, as
illustrated in Table 2 and Table 3.

From the result, we observe that AuxFormer [70] demonstrates substantial
improvement, especially in long-term trajectory predictions. BiFu [73]
outperforms baseline methods lacking scene and gaze information and remains
competitive with methods that include such information. We attribute this
success to the better ability of cross-modal feature extraction of
transformers. Similarly, our SIF3D also applies an attention mechanism (TIA
and SCA), ensuring the effective extraction of multi-modal features.

GCN-based methods (LTD [42], SPGSN [33]) show proficiency in short-term
predictions. However, challenges arise in longer-term scenarios, particularly
in 5-second predictions, where Traj-dest averages over 400mm for both LTD and
SPGSN. This difficulty arises due to the challenges GCN-based methods
encounter in obtaining comprehensive whole-person pose features, making it
challenging to estimate global human intention and establish effective cross-
modal interaction with scene features.

| Method |  0.5$s$ |  1.0$s$ |  2.0$s$ |  3.0$s$ |  5.0$s$
---|---|---|---|---|---|---
Traj |  AuxFormer [70] | 212 | 322 | 500 | 771 | 1078
AuxFormer [70]+Scene+Gaze | 198 | 274 | 411 | 747 | 916
BiFu [73] | 192 | 251 | 417 | 647 | 903
SIF3D | 179 | 265 | 392 | 579 | 836
MPJPE |  AuxFormer [70] | 105.3 | 135.7 | 163.7 | 211.5 | 262.0
AuxFormer [70]+Scene+Gaze | 95.5 | 127.3 | 160.7 | 203.4 | 236.8
BiFu [73] | 100.3 | 131.0 | 158.8 | 196.6 | 234.2
SIF3D | 97.9 | 125.7 | 155.9 | 192.4 | 227.7
Table 3:  Performance details on the GTA-1M test set [5] using trajectory
deviation and MPJPE (in $mm$). We observe that our SIF3D consistently
outperforms the baseline methods in motion prediction within game engine-
rendered 3D scenarios.

In contrast, our SIF3D achieves superior performance across all scenarios,
particularly achieving the best metrics in almost all long-term (5-sec)
evaluations. This success is evident in both global trajectory and local pose,
highlighting the effectiveness of TIA’s intention understanding and SCA’s
locally scene-aware pose prediction. It also evidences the importance of
scene-awareness and line of gaze-guidance for motion prediction in 3D scenes.

###  4.4 Visualizations

To delve deeper into SIF3D, we provide the prediction visualizations in two
different scenarios, where BiFu [73] is used as a baseline, due to its built-
in integration of scene and gaze, resulting in superior numerical performance.

In the first scene (Figure 2(a)), the person navigates through the aisle
between the chair and table, proceeding toward the sofa. While BiFu estimates
the overall movement trend, the predicted motion has the person passing
through the coffee table. Moreover, in the final pose, the person appears
suspended before the chair, deviating from physical constraints. In contrast,
SIF3D demonstrates exhibits precise predictions of the individual’s motion.
The global scene point salience heatmap derived highlights SIF3D’s ability to
comprehend the 3D scene, concentrating on key interactive objects (chairs,
sofa, and coffee table) aligned with the observed motion sequence and
corresponding human gaze. This focus enables SIF3D to predict a clear human
intention and devise a plausible trajectory plan.

In the second scene (Figure 2(b)), the individual is poised to traverse the
hallway leading to the main entrance, reaching the upper left corner intending
to open the window. BiFu [73] predicts an accurate destination, but the motion
sequence presents a significant physical inconsistency within the scene,
penetrated by the wall (frames 2-5) and the screen (frames 7-8). The global
salience heatmap of SIF3D reveals the understanding of the semantics of the
entire 3D scene, even if the upper left corner of the room is not visible at
the end of the observation, providing a better action prediction. In addition,
the local salience heatmap also shows SIF3D’s awareness of obstacles like the
walls.

The visualizations demonstrate that SIF3D effectively predicts motions that
align closely with the ground truth, accurately capturing both global
trajectories and local poses.

###  4.5 Ablation Studies

Ablation of each component. We first investigate the impact of key components
within our SIF3D, including (1) TIA, (2) SCA, (3) MotionDecoder, (4)
Discriminator, and (5) PointNet++, as shown in Table 4.

| Traj-path | Traj-dest | MPJPE-path | MPJPE-dest
---|---|---|---|---
w/o TIA | 701 | 850 | 167.9 | 203.8
w/o SCA | 640 | 703 | 164.8 | 200.4
w/o MotionDecoder | 669 | 699 | 159.2 | 198.9
w/o Discriminator | 604 | 677 | 158.2 | 196.8
w/o PointNet++ | 766 | 918 | 170.4 | 218.5
SIF3D | 590 | 666 | 156.6 | 195.7
Table 4:  SIF3D performance with ablations of components, evaluated using
trajectory deviation and MPJPE (in $mm$).

The removal of TIA and SCA results in a significant performance decline.
Specifically, w/o TIA, SIF3D experiences a significant increase of 111mm in
Traj-dest, underscoring a reduced capability to fully understand the 3D scene
and human intentions. Note that SCA’s spatial salience bias $s_{spatial}$
depends on the predicted trajectory, removing TIA also negatively impacts SCA
due to the inaccurate predicted trajectory, which is evident in increases of
11.3mm and 8.1mm in MPJPE-path and MPJPE-dest. In addition, w/o SCA, SIF3D
cannot capture local scene details, resulting in a 37mm increase in Traj-dest
and a 4.7mm rise in MPJPE-dest.

TIA and SCA make independent predictions for global trajectory and local pose.
Eliminating the motion decoder makes the predicted trajectory and pose
identical, which introduces a 20.0mm increase in Traj-dest and a 3.2mm rise in
MPJPE-dest. The discriminator also contributes to a 7.8mm decrease in Traj-
dest and a 1.1mm decrease in MPJPE-dest.

Moreover, the scene encoder PointNet++, plays a crucial role as a vital link
connecting our motion predictor. Notably, when PointNet++ is removed (replaced
with an MLP), there is a remarkable increase in Traj-dest and MPJPE-dest. In
summary, these results highlight the effectiveness of each component in our
SIF3D.

Point cloud size | 512 | 1024 | 2048 | 4096 | 8192 | 16384 | 32768
---|---|---|---|---|---|---|---
Traj-path (mm) | 626 | 605 | 594 | 590 | 588 | 587 | 587
MPJPE-path (mm) | 165.4 | 160.2 | 157.7 | 156.6 | 155.6 | 155.8 | 155.4
VRAM (MB) | 3560 | 3700 | 4358 | 5246 | 7286 | 11916 | 20876
Speed (sample/s) | 99.8 | 96.1 | 88.5 | 82.4 | 64.2 | 42.2 | 24.1
Table 5:  SIF3D performance with various scene point cloud sizes.

(6) Scene point cloud size. The original size of scene point clouds from LiDAR
sensors typically surpasses 500K [73]. To optimize the use of scene point
clouds, we initially down-sample them. However, this approach risks omitting
critical scene details. To strike a balance between computational overhead and
performance, we conduct experiments by varying the scene point cloud sizes to
assess SIF3D’s performance. Table 5 reveals that the performance improves with
the increase in scene point cloud size until the size exceeds 4096. While the
performance continues to advance with the larger point cloud size, a critical
issue emerges in terms of computational overhead and memory consumption (The
VRAM and speed are measured on a NVIDIA RTX3090 GPU during training).
Therefore, in SIF3D, we opt for a trade-off point cloud size of 4096.

Aggregate Method | Last | Mean | Max | Conv | Transformer
---|---|---|---|---|---
Traj-path (mm) | 594 | 595 | 597 | 602 | 605
MPJPE-path (mm) | 157.1 | 157.7 | 159.2 | 161.3 | 160.8
Table 6:  Performance of SIF3D with various temporal aggregators.

(7) Motion aggregator in TIA. In TIA, we consolidate the motion sequence into
a unified motion embedding using a temporal motion aggregator. To investigate
the influence of various action sequence aggregators on performance, we
conduct experiments involving 5 aggregation strategies: (1) Last, capturing
the last motion embedding in the sequence. (2) Mean, averaging across the
entire motion sequence. (3) Max, employing max-pooling throughout the entire
motion sequence. (4) Conv, applying a three-layer convolutional network to
downsample the sequence. (5) Transformer, introducing a single-layer
transformer decoder into the aggregation process.

The results in Table 6 reveal that the performance of SIF3D is not
significantly affected by the choice of motion aggregator, with the first
aggregator (Last) demonstrating the best performance. However, the two
weighted methods (Conv and Transformer) surprisingly yield inferior results.
We attribute this counterintuitive finding to the limited size of the dataset
and the insufficient supervision for the aggregated global motion embedding,
making it challenging for the weighted methods to learn effectively.

##  5 Conclusion

This work presents a pioneering multi-modal sense-informed framework, SIF3D,
designed for human motion prediction within real-world 3D scenes. By
incorporating external 3D scene points and internal human gaze, SIF3D acquires
the ability to perceive the scene and comprehend human intention using SCA and
TIA. In evaluations conducted on two datasets of GIMO and GTA-1M, SIF3D
achieves an average destination trajectory deviation of 666mm and 836mm on
each dataset, and a 195.7mm and 227.7mm average destination MPJPE,
establishing a new state-of-the-art performance both in global trajectory and
local pose prediction. Consequently, our findings underscore the significance
of scene and gaze information in scene-based motion prediction. Furthermore,
we posit that delving into high-fidelity diverse human motion generation in
real-world 3D scenarios holds promise as a compelling avenue for future
exploration.

## Acknowledgements

This work was supported in part by the National Key Research and Development
Program of China (2022YFC3602601), in part by the Natural Science Foundation
of Jiangsu Province (BK20220939), and in part by the National Natural Science
Foundation of China (62306141).

## References

  * [1] Emre Aksan, Manuel Kaufmann, Peng Cao, and Otmar Hilliges.  A Spatio-temporal Transformer for 3D Human Motion Prediction.  In 3DV, pages 565–574. IEEE, 2021.
  * [2] Emre Aksan, Manuel Kaufmann, Peng Cao, and Otmar Hilliges.  A spatio-temporal transformer for 3d human motion prediction.  In 3DV, pages 565–574. IEEE, 2021.
  * [3] Florent Altché and Arnaud de La Fortelle.  An lstm network for highway trajectory prediction.  In 2017 IEEE 20th international conference on intelligent transportation systems (ITSC), pages 353–359. IEEE, 2017.
  * [4] Emad Barsoum, John Kender, and Zicheng Liu.  HP-GAN: Probabilistic 3D Human Motion Prediction via GAN.  In CVPR, pages 1418–1427, 2018.
  * [5] Zhe Cao, Hang Gao, Karttikeya Mangalam, Qi-Zhi Cai, Minh Vo, and Jitendra Malik.  Long-term human motion prediction with scene context.  In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part I 16, pages 387–404. Springer, 2020.
  * [6] Jingdao Chen, Zsolt Kira, and Yong K Cho.  Deep learning approach to point cloud scene understanding for automated scan to 3d reconstruction.  Journal of Computing in Civil Engineering, 33(4):04019027, 2019.
  * [7] Christopher Choy, JunYoung Gwak, and Silvio Savarese.  4d spatio-temporal convnets: Minkowski convolutional neural networks.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3075–3084, 2019.
  * [8] Enric Corona, Albert Pumarola, Guillem Alenya, and Francesc Moreno-Noguer.  Context-aware Human Motion Prediction.  In CVPR, pages 6992–7001, 2020.
  * [9] Qiongjie Cui and Huaijiang Sun.  Towards Accurate 3D Human Motion Prediction From Incomplete Observations.  In CVPR, pages 4801–4810, June 2021.
  * [10] Qiongjie Cui, Huaijiang Sun, and Fei Yang.  Learning Dynamic Relationships for 3D Human Motion Prediction.  In CVPR, pages 6519–6527, 2020.
  * [11] Lingwei Dang, Yongwei Nie, Chengjiang Long, Qing Zhang, and Guiqing Li.  MSR-GCN: Multi-Scale Residual Graph Convolution Networks for Human Motion Prediction.  In ICCV, pages 11467–11476, 2021.
  * [12] Christian Diller, Thomas Funkhouser, and Angela Dai.  Forecasting characteristic 3d poses of human actions.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15914–15923, 2022.
  * [13] Xiaoyi Dong, Jianmin Bao, Dongdong Chen, Weiming Zhang, Nenghai Yu, Lu Yuan, Dong Chen, and Baining Guo.  Cswin transformer: A general vision transformer backbone with cross-shaped windows.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12124–12134, 2022.
  * [14] Francis Engelmann, Martin Bokeloh, Alireza Fathi, Bastian Leibe, and Matthias Nießner.  3d-mpa: Multi-proposal aggregation for 3d semantic instance segmentation.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9031–9040, 2020.
  * [15] Katerina Fragkiadaki, Sergey Levine, Panna Felsen, and Jitendra Malik.  Recurrent Network Models for Human Dynamics.  In ICCV, pages 4346–4354, 2015.
  * [16] Anindita Ghosh, Rishabh Dabral, Vladislav Golyanik, Christian Theobalt, and Philipp Slusallek.  Imos: Intent-driven full-body motion synthesis for human-object interactions.  In Computer Graphics Forum, volume 42, pages 1–12. Wiley Online Library, 2023.
  * [17] Benjamin Graham, Martin Engelcke, and Laurens Van Der Maaten.  3d semantic segmentation with submanifold sparse convolutional networks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 9224–9232, 2018.
  * [18] Liang-Yan Gui, Yu-Xiong Wang, Xiaodan Liang, and José M. F. Moura.  Adversarial Geometry-Aware Human Motion Prediction.  In ECCV, pages 786–803, 2018.
  * [19] Liang-Yan Gui, Kevin Zhang, Yu-Xiong Wang, Xiaodan Liang, José M. F. Moura, and Manuela Veloso.  Teaching Robots to Predict Human Motion.  IROS, pages 562–567, 2018.
  * [20] Xiao Guo and Jongmoo Choi.  Human Motion Prediction via Learning Local Structure Representations and Temporal Dependencies.  In AAAI, pages 2580–2587, 2019.
  * [21] JunYoung Gwak, Christopher Choy, and Silvio Savarese.  Generative sparse detection networks for 3d single-shot object detection.  In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part IV 16, pages 297–313. Springer, 2020.
  * [22] Mohamed Hassan, Duygu Ceylan, Ruben Villegas, Jun Saito, Jimei Yang, Yi Zhou, and Michael J Black.  Stochastic scene-aware motion prediction.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11374–11384, 2021.
  * [23] Ji Hou, Angela Dai, and Matthias Nießner.  3d-sis: 3d semantic instance segmentation of rgb-d scans.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4421–4430, 2019.
  * [24] Ji Hou, Angela Dai, and Matthias Nießner.  Revealnet: Seeing behind objects in rgb-d scans.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2098–2107, 2020.
  * [25] Siyuan Huang, Zan Wang, Puhao Li, Baoxiong Jia, Tengyu Liu, Yixin Zhu, Wei Liang, and Song-Chun Zhu.  Diffusion-based generation, optimization, and planning in 3d scenes.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16750–16761, 2023.
  * [26] Catalin Ionescu, Dragos Papava, Vlad Olaru, and Cristian Sminchisescu.  Human3. 6m: Large scale datasets and predictive methods for 3d human sensing in natural environments.  IEEE transactions on pattern analysis and machine intelligence, 36(7):1325–1339, 2013.
  * [27] Ashesh Jain, Amir Roshan Zamir, Silvio Savarese, and Ashutosh Saxena.  Structural-RNN: Deep Learning on Spatio-Temporal Graphs.  In CVPR, pages 5308–5317, 2016.
  * [28] Haiyong Jiang, Feilong Yan, Jianfei Cai, Jianmin Zheng, and Jun Xiao.  End-to-end 3d point cloud instance segmentation without detection.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12796–12805, 2020.
  * [29] Thomas N Kipf and Max Welling.  Semi-supervised Classification with Graph Convolutional Networks.  arXiv preprint arXiv:1609.02907, 2016.
  * [30] Jogendra Nath Kundu, Maharshi Gor, and R. Venkatesh Babu.  BiHMP-GAN: Bidirectional 3D Human Motion Prediction GAN.  In AAAI, volume 33, pages 8553–8560, 2019.
  * [31] Bin Li, Jian Tian, Zhongfei Zhang, Hailin Feng, and Xi Li.  Multitask Non-Autoregressive Model for Human Motion Prediction.  IEEE Transactions on Image Processing, 2020.
  * [32] Chen Li, Zhen Zhang, Wee Sun Lee, and Gim Hee Lee.  Convolutional Sequence to Sequence Model for Human Dynamics.  In CVPR, pages 5226–5234, 2018.
  * [33] Maosen Li, Siheng Chen, Zijing Zhang, Lingxi Xie, Qi Tian, and Ya Zhang.  Skeleton-Parted Graph Scattering Networks for 3D Human Motion Prediction.  In ECCV, pages 18–36. Springer, 2022.
  * [34] Maosen Li, Siheng Chen, Yangheng Zhao, Ya Zhang, Yanfeng Wang, and Qi Tian.  Dynamic Multiscale Graph Neural Networks for 3D Skeleton Based Human Motion Prediction.  In CVPR, pages 214–223, 2020.
  * [35] Xingyu Liu, Charles R Qi, and Leonidas J Guibas.  Flownet3d: Learning scene flow in 3d point clouds.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 529–537, 2019.
  * [36] Yicheng Liu, Jinghuai Zhang, Liangji Fang, Qinhong Jiang, and Bolei Zhou.  Multimodal motion prediction with stacked transformers.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7577–7586, 2021.
  * [37] Ze Liu, Han Hu, Yutong Lin, Zhuliang Yao, Zhenda Xie, Yixuan Wei, Jia Ning, Yue Cao, Zheng Zhang, Li Dong, et al.  Swin transformer v2: Scaling up capacity and resolution.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12009–12019, 2022.
  * [38] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo.  Swin transformer: Hierarchical vision transformer using shifted windows.  In Proceedings of the IEEE/CVF international conference on computer vision, pages 10012–10022, 2021.
  * [39] Zhenguang Liu, Kedi Lyu, Shuang Wu, Haipeng Chen, Yanbin Hao, and Shouling Ji.  Aggregated Multi-GANs for Controlled 3D Human Motion Prediction.  In AAAI, volume 35, pages 2225–2232, 2021.
  * [40] Tiezheng Ma, Yongwei Nie, Chengjiang Long, Qing Zhang, and Guiqing Li.  Progressively generating better initial guesses towards next stages for high-quality human motion prediction.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6437–6446, 2022.
  * [41] Tiezheng Ma, Yongwei Nie, Chengjiang Long, Qing Zhang, and Guiqing Li.  Progressively Generating Better Initial Guesses Towards Next Stages for High-Quality Human Motion Prediction.  In CVPR, pages 6437–6446, 2022.
  * [42] Wei Mao, Miaomiao Liu, Mathieu Salzmann, and Hongdong Li.  Learning Trajectory Dependencies for Human Motion Prediction.  In ICCV, pages 9489–9497, 2019.
  * [43] Julieta Martinez, Michael J Black, and Javier Romero.  On Human Motion Prediction using Recurrent Neural Networks.  In CVPR, pages 2891–2900, 2017.
  * [44] Angel Martínez-González, Michael Villamizar, and Jean-Marc Odobez.  Pose transformers (potr): Human motion prediction with non-autoregressive transformers.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2276–2284, 2021.
  * [45] Daniel Maturana and Sebastian Scherer.  Voxnet: A 3d convolutional neural network for real-time object recognition.  In 2015 IEEE/RSJ international conference on intelligent robots and systems (IROS), pages 922–928. IEEE, 2015.
  * [46] Nishant Nikhil and Brendan Tran Morris.  Convolutional neural network for trajectory prediction.  In Proceedings of the European Conference on Computer Vision (ECCV) Workshops, pages 0–0, 2018.
  * [47] Georgios Pavlakos, Vasileios Choutas, Nima Ghorbani, Timo Bolkart, Ahmed AA Osman, Dimitrios Tzionas, and Michael J Black.  Expressive body capture: 3d hands, face, and body from a single image.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10975–10985, 2019.
  * [48] AJ Piergiovanni, Anelia Angelova, Alexander Toshev, and Michael S Ryoo.  Adversarial Generative Grammars for Human Activity Prediction.  In ECCV, pages 507–523. Springer, 2020.
  * [49] Charles R Qi, Xinlei Chen, Or Litany, and Leonidas J Guibas.  Imvotenet: Boosting 3d object detection in point clouds with image votes.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4404–4413, 2020.
  * [50] Charles R Qi, Or Litany, Kaiming He, and Leonidas J Guibas.  Deep hough voting for 3d object detection in point clouds.  In proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9277–9286, 2019.
  * [51] Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas.  Pointnet: Deep learning on point sets for 3d classification and segmentation.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 652–660, 2017.
  * [52] Charles R Qi, Hao Su, Matthias Nießner, Angela Dai, Mengyuan Yan, and Leonidas J Guibas.  Volumetric and multi-view cnns for object classification on 3d data.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5648–5656, 2016.
  * [53] Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas.  Pointnet++: Deep hierarchical feature learning on point sets in a metric space.  Advances in neural information processing systems, 30, 2017.
  * [54] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In International conference on machine learning, pages 8748–8763. PMLR, 2021.
  * [55] Noha Radwan, Wolfram Burgard, and Abhinav Valada.  Multimodal interaction-aware motion prediction for autonomous street crossing.  The International Journal of Robotics Research, 39(13):1567–1598, 2020.
  * [56] Alejandro Hernandez Ruiz, Juergen Gall, and Francesc Moreno-Noguer.  Human Motion Prediction via Spatio-Temporal Inpainting.  In CVPR, pages 7134–7143, 2018.
  * [57] Radu Bogdan Rusu, Nico Blodow, Zoltan Csaba Marton, and Michael Beetz.  Close-range scene segmentation and reconstruction of 3d point cloud maps for mobile manipulation in domestic environments.  In 2009 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 1–6. IEEE, 2009.
  * [58] Hang Su, Subhransu Maji, Evangelos Kalogerakis, and Erik Learned-Miller.  Multi-view convolutional neural networks for 3d shape recognition.  In Proceedings of the IEEE international conference on computer vision, pages 945–953, 2015.
  * [59] Zhaoxin Su, Gang Huang, Sanyuan Zhang, and Wei Hua.  Crossmodal transformer based generative framework for pedestrian trajectory prediction.  In 2022 International Conference on Robotics and Automation (ICRA), pages 2337–2343. IEEE, 2022.
  * [60] Omid Taheri, Nima Ghorbani, Michael J Black, and Dimitrios Tzionas.  Grab: A dataset of whole-body human grasping of objects.  In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part IV 16, pages 581–600. Springer, 2020.
  * [61] Yongyi Tang, Lin Ma, Wei Liu, and Wei-Shi Zheng.  Long-Term Human Motion Prediction by Modeling Motion Context and Enhancing Motion Dynamics.  In IJCAI, 2018.
  * [62] Hugues Thomas, Charles R Qi, Jean-Emmanuel Deschaud, Beatriz Marcotegui, François Goulette, and Leonidas J Guibas.  Kpconv: Flexible and deformable convolution for point clouds.  In Proceedings of the IEEE/CVF international conference on computer vision, pages 6411–6420, 2019.
  * [63] Susanne Trick, Dorothea Koert, Jan Peters, and Constantin A Rothkopf.  Multimodal uncertainty reduction for intention recognition in human-robot interaction.  In 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 7009–7016. IEEE, 2019.
  * [64] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is All You Need.  In NeurIPS, pages 5998–6008, 2017.
  * [65] Hongsong Wang, Jian Dong, Bin Cheng, and Jiashi Feng.  PVRED: A Position-Velocity Recurrent Encoder-Decoder for Human Motion Prediction.  IEEE Transactions on Image Processing, 30:6096–6106, 2021.
  * [66] Jingbo Wang, Yu Rong, Jingyuan Liu, Sijie Yan, Dahua Lin, and Bo Dai.  Towards diverse and natural scene-aware 3d human motion synthesis.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20460–20469, 2022.
  * [67] Jingbo Wang, Sijie Yan, Bo Dai, and Dahua Lin.  Scene-aware generative network for human motion synthesis.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12206–12215, 2021.
  * [68] Zan Wang, Yixin Chen, Tengyu Liu, Yixin Zhu, Wei Liang, and Siyuan Huang.  Humanise: Language-conditioned human motion generation in 3d scenes.  Advances in Neural Information Processing Systems, 35:14959–14971, 2022.
  * [69] Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao.  3d shapenets: A deep representation for volumetric shapes.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1912–1920, 2015.
  * [70] Chenxin Xu, Robby T Tan, Yuhong Tan, Siheng Chen, Xinchao Wang, and Yanfeng Wang.  Auxiliary tasks benefit 3d skeleton-based human motion prediction.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9509–9520, 2023.
  * [71] Bisheng Yang, Zhen Dong, Fuxun Liang, and Yuan Liu.  Automatic registration of large-scale urban scene point clouds based on semantic feature points.  ISPRS Journal of Photogrammetry and Remote Sensing, 113:43–58, 2016.
  * [72] Zhibin Yu, Sangwook Kim, Rammohan Mallipeddi, and Minho Lee.  Human intention understanding based on object affordance and action classification.  In 2015 International Joint Conference on Neural Networks (IJCNN), pages 1–6. IEEE, 2015.
  * [73] Yang Zheng, Yanchao Yang, Kaichun Mo, Jiaman Li, Tao Yu, Yebin Liu, C Karen Liu, and Leonidas J Guibas.  Gimo: Gaze-informed human motion prediction in context.  In European Conference on Computer Vision, pages 676–694. Springer, 2022.
  * [74] Fangwei Zhong, Xiao Bi, Yudi Zhang, Wei Zhang, and Yizhou Wang.  Rspt: reconstruct surroundings and predict trajectory for generalizable active object tracking.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 3705–3714, 2023.


# Are Labels Needed for Incremental Instance Learning?

Mert Kilickaya
Eindhoven University of Technology
<EMAIL_ADDRESS>Joaquin Vanschoren
Eindhoven University of Technology
<EMAIL_ADDRESS>
###### Abstract

We introduce a new, self-incremental approach to visually classify object
instances in this paper. Our model, Visual Self-Incremental Instance Learning,
VINIL for short, incrementally observes and classifies a single instance at a
time before discarding it. The challenge in incremental instance learning is
the risk of forgetting over longer learning sessions and the difficulties
associated with instance labeling. Our approach addresses these challenges
through three key innovations: i). The development of VINIL, a self-
incremental learner that can learn object instances in sequence, ii). The use
of self-supervision to eliminate the need for instance labeling, and iii). A
thorough evaluation of VINIL against label-supervised counterparts on two
large-scale benchmarks [33, 6], showcasing its effectiveness in achieving
higher accuracy while reducing forgetfulness.

##  1 Introduction

This paper strives for incrementally learning to recognize visual object
instances. Visual instance recognition aims to retrieve different views of an
input object instance image. It can be seen as fine-grained object
recognition, where the goal is to distinguish different instantiations of the
same object, such as cup 1 from cup 2. Instance recognition finds applications
in many domains, such as in visual search [40], tracking [5, 48, 49] and
localization [60].

Distinguishing between different object instances is a challenging task as
they often differ only by small nuances. Metric learning [52] is a commonly
used approach to learn visual object instances by comparing two views of the
same object using a deep convolutional network, such as ResNet [22]. The
network is trained to bring representations of the same object close together
and separate representations of different objects in a large batch of images.

However, this approach requires iterating over potentially million-scale
datasets multiple times to refine the metric space, which can be impractical
for privacy reasons (some data may have to be deleted) or scale (when dealing
with billions of images). Additionally, using the trained deep net to query a
large database of images by comparing the feature representation of the input
image to the database representations is time-consuming and computationally
expensive.

This paper builds upon incremental learning to mitigate privacy and scale
issues. In incremental learning, the learner observes images from a certain
class for a number of iterations. Then, the data of the previous class is
discarded, and the learner receives examples from a novel category. Such
approach is called class-incremental learning, and receives an increasing
amount of attention recently [36, 27, 37, 57].

Existing class-incremental learners are ill-suited for instance-incremental
learning for two reasons. First, class-incremental learners rely on full label
supervision. Collecting such annotation at the instance level is very
expensive. Second, despite years of efforts, class-incremental learners are
forgetful, since they lose performance on previously observed categories.

Figure 1: Top: Label-incremental learning demands instance-level annotations
and trains a new weight per instance. This approach is not suitable for
handling large numbers of visual instances, and it is also prone to forgetting
previously learned instances. Bottom: In this paper, we introduce VINIL, a
self-incremental instance learning method. VINIL focuses solely on learning a
discriminative embedding and uses Self-Supervised Learning (SSL) to extract
supervision from different views of the same instance. As a result, VINIL is
label-free, more scalable, and significantly less prone to forgetting compared
to label-incremental learning.

This paper proposes Visual Self-Incremental Instance Learning, VINIL, to
perform instance-incremental learning, consider Figure 1. VINIL observes
multiple views of a single instance at a time, which is then discarded from
the dataset. Such examples can be easily captured via turntable cameras [38,
18, 29, 6] or via hand-interactions [34, 15, 50]. Then, VINIL extracts its own
supervision via self-supervision [56], therefore self-incremental. Self-
incremental learning not only is label-efficient, it also consistently
outperforms competitive label-supervised variants, as we will show. In
summary, this paper makes three contributions:

  1. I.

We propose VINIL, a realistic, scalable incremental instance learner,

  2. II.

VINIL performs self-incremental learning, by-passing the need for heavy
instance supervision,

  3. III.

VINIL is trained without labels, and is consistently more accurate and less
forgetful across benchmarks of Core-50 and iLab20M [33, 6].

##  2 Related Work

Visual Instance Recognition. Visual instance recognition has been extensively
researched in recent years and has been applied to various computer vision
problems, including product retrieval [40, 23, 32, 52], object tracking [5,
48, 49], and geo-localization [31, 51, 59, 46, 53]. The most common approach
for these tasks is to induce a discriminative embedding space, often using
metric learning techniques [23, 14]. These methods require access to the
entire dataset and fine-grained similarity labels. In contrast, this paper
presents a novel method for incremental and label-free visual instance
recognition.

Class-Incremental Learning. Class-incremental learning involves expanding a
deep classifier with novel objects, with the goal of maintaining performance
on previous categories and avoiding forgetting [36]. Popular techniques to
prevent forgetting include regularization, which limits abrupt changes in
network weights [28, 30, 43], and memory replay of previous data [4, 47, 45,
24]. Our approach differs from conventional class-incremental learning in two
ways. First, while class-incremental learning focuses on object categories,
our approach operates at the instance level, presenting new challenges.
Second, class-incremental learning requires fully labeled datasets, which is
often not possible in instance learning. To overcome these limitations, we use
self-supervision and adapt relevant evaluation techniques. Specifically, we
use Elastic Weight Consolidation (EwC) as a regularization method [28] and
Replay as a memory technique [45] due to their adaptability for label-free
learning.

Self-Supervised Learning. Self-supervision involves creating pretext tasks to
learn deep representations without using labels. Early methods predicted
rotations [19] or patches [39], but contrastive learning has become dominant
in recent years [12, 8, 11, 21]. In our work, we utilize self-supervision to
extract learning signals in place of instance labels. We experimented with
both BarlowTwins [56] and SimSiam [13] due to their high performance and
adaptation in incremental learning tasks [16, 35]. We found that BarlowTwins
[56] performs better than SimSiam for our incremental learning setup. We
believe this is due to its ability to reduce redundancy across different views
of the input. Reducing visual redundancy is especially important for different
instances of the same object, as visual object instances may only differ in
small details.

Incremental Self-Supervised Learning. Recently, there has been a surge of
interest in use of self-supervision to replace label supervision for
incremental learning. We identify three main directions. i) Pre-training:
Researchers use self-supervised learning either for pre-training prior to
incremental learning stage [17, 7, 26] or as an auxiliary loss function to
improve feature discrimination [58]. However, these papers still require
labels during the incremental learning stage. ii) Replay:  Second line of
techniques propose replay-based methods [42, 10, 35] to supplement self-
supervised learners with stored data within the memory. However, they require
large amounts of exemplars to be stored within the memory to work effectively.
iii) Regularization:  Third line of work proposes to regularize self-learned
representations [16, 20, 35]. In this work, we focus on regularization-based
self-incremental learning. More specifically, we closely follow UCL [35] and
ask ourselves: What is the contribution of self-supervision for instance
incremental learning? Our main observation is that self-supervision
consistently yields less forgetful, more accurate and transferable
representations, as will be shown via large-scale experiments.

Method | Supervision | Input | Memory | Loss
---|---|---|---|---
Fine-Tuning | Label-supervised | $(x,y)$ | ✗ | $CE(y,y^{\prime})$
Fine-Tuning | Self-supervised | $(x)$ | ✗ | $BT(x,x^{\prime})$
EwC | Label-supervised | $(x,y)$ | ✗ | $CE(y,y^{\prime})+Reg(\Theta,y^{\prime})$
EwC | Self-supervised | $(x)$ | ✗ | $BT(x,x^{\prime})+Reg(\Theta)$
Replay | Label-supervised | $(x,y)$ | $(x^{m},y^{m})$ | $CE(y,y^{\prime})+CE(y^{m},y^{m^{\prime}})$
Replay | Self-supervised | $(x)$ | $(x^{m})$ | $BT(x,x^{\prime})+BT(x^{m},x^{m^{\prime}})$
Table 1: VINIL performs incremental instance learning via self-supervision,
and is compared with label-supervision. We use memory replay [45] and weight
regularization [28] as well as simple fine-tuning. Fine-Tuning [44] relies on
Cross-Entropy (CE) or BarlowTwins (BT) [56] to perform incremental learning.
EwC [28] penalizes abrupt changes in network weights via regularization
($Reg(\cdot)$). Replay [45] replays a part of previous data in the form of
input-labels (label-supervised) or input-only (self-supervised).

##  3 VINIL

We present an overview of VINIL in Table 1. The goal of VINIL is to train an
embedding network $f(\cdot)_{\theta_{t}}$ parameterized by $\theta_{t}$. The
network maps an input image $x$ to a $D$-dimensional discriminative embedding,
$h=f{\theta_{t}}(x)$ which will then be used to query the database to retrieve
different views of the input query for instance recognition. Here, $t$ denotes
the incremental learning step, where the tasks are arriving sequentially:
$\mathbf{T}=(\mathbf{T}_{1},\mathbf{T}_{2},...,\mathbf{T}_{t})$. We train
VINIL via minimizing the following objective:

$\displaystyle\mathcal{L}=w_{c}\cdot L_{inst}+(1-w_{c})\cdot L_{incr}$ (1)

where $w_{c}$ controls the contribution of instance classification loss
$L_{inst}$ and incremental learning loss $L_{incr}$. Incremental learning loss
either corresponds to memory replay [45] or weight regularization [28] whereas
instance classification loss $L_{inst}$ is either cross-entropy with labels or
a self-supervision objective.

###  3.1 Incremental Learning

Fine-Tuning. A vanilla way to perform incremental instance learning is to
apply simple fine-tuning via SGD [44]. In fine-tuning, no incremental learning
loss is applied (_i.e_. $w_{c}=1.0$) and the sole objective is classification.

In case of label-supervision, a task is defined by a dataset
$D_{t}^{label}=\\{(x_{i,t},y_{i,t})_{i=1}^{k_{t}}\\}$ where $k_{t}$ is the
data size at time $t$. Then, fine-tuning corresponds to instance
discrimination via cross-entropy $L_{inst}=CE(y_{i,t},y_{i,t}^{\prime})$.
Here, instance category prediction for the instance $i$ at time step $t$ is
obtained with a simple MLP classifier. Notice that this classifier will expand
in size linearly with the number of instance categories.

In case of VINIL, a task is defined by a dataset
$D_{t}^{self}=\\{(x_{i,t})_{i=1}^{n_{t}}\\}$ (_i.e_. no labels). Then, fine-
tuning corresponds to minimizing the self-supervision objective
$L_{inst}=BT(x_{i,t},x_{i,t}^{\prime})$ where $BT(\cdot)$ is the BarlowTwins
[56].

EwC [28]. EwC penalizes big changes in network weights via comparing the
weights in the current and the previous incremental learning step. Originally,
EwC re-weights the contribution of each weight to the loss function as a
function of instance classification logits (_i.e_. label-supervision). In
VINIL, in the absence of labels, we omit this re-weighting and simply use
identity matrix.

Replay [45]. Replay replays a portion of the past data from previous
incremental steps to mitigate forgetting. In case of label-supervision, this
corresponds to replaying both the input data and their labels via cross-
entropy: $CE(y_{i,t}^{m},y_{i,t}^{m^{\prime}})$ where $y_{i,t}^{m^{\prime}}$
is the instance categories for the memory instance $i$ at time $t$. For VINIL,
we simply replay the input memory data and its augmented view via self-
supervsion of BarlowTwins as $BT(x_{i,t}^{m},x_{i,t}^{m^{\prime}})$.

###  3.2 Self-Supervised Learning

In BarlowTwins, the features are extracted from the original and the augmented
view of the input image with a siamese deep network, at time step $t$ as:
$(z_{i,t},z_{i,t}^{\prime})=(f_{\theta^{t}}(x_{i,t}),f_{\theta^{t}}(x_{i,t}^{\prime}))$
where $x_{i,t}^{\prime}=aug(x{i,t})$ is the augmented view of the input.
BarlowTwins minimizes the redundancy across views while maximizing the
representational information. This is achieved via operating on the cross-
covariance matrix via:

$\displaystyle BT=\sum_{i}(1-C_{ii})^{2}+w_{b}\cdot\sum_{i}\sum_{j\neq
i}(C_{ij})^{2}$ (2)

where:

$\displaystyle
C_{ij}=\frac{\sum_{\beta}z_{\beta,i}z^{\prime}_{\beta,j}}{\sum_{\beta}\sqrt{z_{\beta,i}^{2}}\cdot\sum_{\beta}\sqrt{(z_{\beta,j}^{\prime})^{2}}}$
(3)

is the cross-correlation matrix. Here, $w_{b}$ controls invariance-redundancy
reduction trade-off, $i$ and $j$ corresponds to network’s output dimensions.

##  4 Experimental Setup

Implementation. All the networks are implemented in PyTorch [41]. We use
ResNet-$18$ [22] as the backbone $f(\cdot)$, and a single-layer MLP for the
instance classifier. We train for $200$ epochs for each incremental steps with
a learning rate $0.001$ decayed via cosine annealing. We use SGD optimizer
with momentum $0.9$ and batch-size $256$. We use random cropping and scaling
for augmentation.

We follow the original implementation of BarlowTwins [1]. $10\%$ of the data
is stored within the memory for replay [45]. We set scalars as:
$w_{c}=0.7,w_{b}=0.03$

Datasets. We evaluate VINIL on iLab-$20$M [6] and Core-$50$ [33], since they
are large-scale, sufficiently different, and widely adopted in incremental
learning.

iLab-$20$M is a turntable dataset of vehicles. It consists of $10$ objects
(_i.e_. bus, car, plane) with varying ($[25,160]$) number of instances per
category. Objects are captured by varying the background and the camera angle,
leading to $14$ examples per-instance. We use the public splits provided in
[3] with $125$k training and $31$k gallery images.

Core-$50$ is a hand-held object dataset used in benchmarking incremental
learning algorithms. The dataset includes $10$ objects (_i.e_. phones,
adaptors, scissors) with $50$ instances per-category. Each instance is
captured for $300$ frames, across $11$ different backgrounds. We use $120$k
training and $45$k gallery images [2].

Protocol. We first divide each dataset into $5$ tasks, with $2$ object
categories per-task. Then, each task is subdivided into $N$ object instance
tasks depending on the dataset. We discard the classifier of label-supervised
variants after training, and evaluate all models with instance retrieval
performance via k-NN with $k=100$ neighbors on the gallery set, as is the
standard in SSL [12, 8, 11, 21, 13].

We use the mean-pooled activations of layer4 of ResNet to represent images.
All exemplars in the gallery set are used as query.

Metrics. We rely on two well established metrics to evaluate the performance
of the models, namely accuracy and forgetting.

i). Accuracy (Acc) measures whether if we can retrieve different views of the
same instance from the gallery set given a query. We measure accuracy for each
incremental learning steps, which is then averaged across all sessions.

ii). Forgetting (For) measures the discrepancy of accuracy across different
sessions. Concretely, it compares the maximum accuracy across all sessions
with the accuracy in the last step.

##  5 Experiments

Our experiments address the following research questions: Q1: Can VINIL
improve performance and reduce forgetting in comparison to label-supervision?
Q2: Does VINIL learn incrementally generalizable representations across
datasets? Q3: What makes VINIL effective against label-supervision?

###  5.1 How Does VINIL Compare to Label-Supervision?

First, we compare VINIL’s performance to label-supervision. The results are
presented in Table 2.

| Core-$50$ | iLab-$20$M
---|---|---
Method |  $Acc$ ($\uparrow$) |  $For$ ($\downarrow$) |  $Acc$ ($\uparrow$) |  $For$ ($\downarrow$)
FT (Label) | 71.450 | 22.436 | 89.340 | 6.500
FT (VINIL) | 74.914 | 4.802 | 90.398 | 0.000
Replay (Label) | 88.180 | 6.741 | 84.464 | 5.696
Replay (VINIL) | 67.677 | 10.095 | 90.543 | 0.000
EwC (Label) | 75.117 | 18.268 | 87.690 | 4.535
EwC (VINIL) | 73.011 | 2.167 | 90.655 | 0.000
Table 2: Visual Incremental Instance Learning on Core-$50$ [33] and iLab-$20$M
[6]. VINIL outperforms label-supervised variants for $4$ out of $6$ settings,
while significantly reducing forgetfulness on both datasets. This indicates
self-incremental learning is a strong, label-free alternative to label-
supervision.

VINIL Yields Competitive Accuracy. We first compare the accuracies obtained by
VINIL vs. label-supervision. We observe that VINIL yields competitive accuracy
against label-supervision: In $4$ out of $6$ setting, VINIL outperforms label-
supervised variants.

VINIL Mitigates Forgetting. Secondly, we compare the forget rates of VINIL vs.
label-supervision (lower is better). We observe that VINIL consistently leads
to much lower forget rates in comparison to label-supervision. On iLab-$20$M
dataset, VINIL results in no forgetting. On the more challenging dataset of
Core-$50$, the difference across forget rates are even more pronounced: Label-
supervision suffers from $22\%$ forget rate whereas VINIL only by $4\%$, a
relative drop of $80\%$ with fine-tuning.

Label-supervision Leverages Memory. Our last observation is that memory
improves the accuracy and reduces forgetfulness of label-supervision. In
contrast, the use of memory disrupts self-supervised representations. This
indicates that replaying both inputs and labels ($(x_{i},y_{i})$) as opposed
to input-only ($(x_{i})$, as in self-supervision) may lead to imbalanced
training due to limited memory size [9, 25, 54].

In summary, we conclude that VINIL is an efficient, label-free alternative to
label-supervised incremental instance learning. VINIL improves accuracy while
reducing forget rate. We also observe that label-supervision closes the gap
when an additional memory of past data is present. This motivates further
research for improving self-incremental instance learners with memory.

###  5.2 Can VINIL Generalize Across Datasets?

After confirming the efficacy of VINIL within the same dataset, we now move on
to a more complicated setting: Cross-dataset generalization. In cross-dataset
generalization, we first perform incremental training on Core-$50$, and then
evaluate on iLab-$20$M. Then, we perform incremental training on iLab-$20$M
and then evaluate on Core-$50$.

Cross-dataset generalization between Core-$50$ and iLab-$20$M is challenging
due to the following reasons: i). Camera:  Core-$50$ is captured with a hand-
held camera whereas iLab-$20$M is captured on a platform with a turntable
camera, ii). Object Categories: Object categories are disjoint, as no common
objects are present in each dataset, iii). Object Types: iLab-$20$M exhibits
toy objects of vehicles whereas Core-$50$ exhibits hand-interacted daily-life
objects.

The results are presented in Table 3. We present iLab-$20$M to Core-$50$, and
Core-$50$ to iLab-$20$M results, along with the relative drop w.r.t training
and testing on the same dataset (see Table 2).

Train on $\Longrightarrow$ | iLab-$20$M | Core-$50$
---|---|---
Test on $\Longrightarrow$ | Core-$50$ | iLab-$20$M
Method | $Acc_{(\%\Delta(\downarrow))}$ | $Acc_{(\%\Delta(\downarrow))}$
FT (Label) | $59.850_{\%16}$ | $67.249_{\%24}$
FT (VINIL) | $\textbf{66.704}_{\textbf{\%10}}$ | $\textbf{76.302}_{\textbf{\%15}}$
Replay (Label) | $55.692_{\%36}$ | $69.412_{\%17}$
Replay (VINIL) | $\textbf{61.857}_{\textbf{\%8}}$ | $\textbf{76.125}_{\textbf{\%15}}$
EwC (Label) | $59.030_{\%21}$ | $70.087_{\%20}$
EwC (VINIL) | $\textbf{70.648}_{\textbf{\%3}}$ | $\textbf{75.793}_{\textbf{\%16}}$
Table 3: Cross-Dataset Generalization on Core-$50$ and iLab-$20$M. We present:
i) Train on iLab-$20$M and test on Core-$50$, ii) Train on Core-$50$ and test
on iLab-$20$M. In adddition to accuracy, we also present the relative drop
w.r.t training and testing on the same dataset (see Table 2). We observe that
VINIL is consistently more robust in cross-dataset generalization when
compared with label-supervision. The results indicate that self-supervision is
able to extract more domain-agnostic representations, which improves the
generality of visual representations, for instance-incremental setup.

VINIL Yields Generalizable Representations. We first observe that VINIL
consistently yields higher accuracy and lower drop rate across all $6$
settings in both datasets. This indicates that self-supervision extracts more
generalizable visual representations from the dataset.

Label-supervision Overfits with Memory. Secondly, we observe that label-
supervised variants with memory generalizes via overfitting on the training
dataset. Replay with label-supervision leads to the biggest drop rate of
$36\%$ on Core-$50$, when trained with iLab-$20$M. This implies the use of the
memory drastically reduces generality of visual representations. A potential
explanation is that, since replay utilizes the same set of examples within the
limited memory repeatedly throughout learning, this forces the network to
over-fit to those examples.

We conclude that VINIL extracts generalizable visual representations from the
training source to perform instance incremental training. We also conclude
that the astounding performance of label-supervision equipped with memory
comes with the cost of overfit, leading to drastic drop in case of visual
discrepancies across datasets.

###  5.3 What Factors Affect VINIL’s Performance?

VINIL Mitigates Bias Towards Recent Task. We present the heatmaps of the
performance for all $5$ main tasks, when each task is introduced sequentially,
for label-supervision in Figure 3 and for VINIL in Figure 3 on iLab-$20$M [6].
Each row presents the accuracy for each task, as the tasks are introduced
sequentially. For example, the entry $(0,2)$ denotes the performance on
Task-$0$ when the Task-$2$ is introduced.

Figure 2: Task-level performance of Label-supervision (Fine-tuning). Label-
supervision is biased towards recent task.

Figure 3: Task-level performance of VINIL (Fine-tuning). VINIL improves its
performance with incoming data, and is less biased towards recent task.

Considering Figure 3 for label-supervision, observe how the tasks achieve
their peak performance when they are being introduced to the model, hence the
higher numbers within the diagonal. Then, the performance degrades drastically
as more and more tasks are being introduced. This indicates label-supervision
fails to leverage more data. We call such phenomenon ”recency bias”, as the
model is biased towards the most recently introduced task.

In contrast, in Figure 3 for VINIL, the performance on each task improves
sequentially with the incoming stream of new tasks. This indicates self-
supervised representations are less biased towards the recent task, and can
leverage data to improve performance. This renders them as a viable option
when incremental learning for longer learning steps, such as in incremental
instance learning.

VINIL Focuses on the Object Instance. We present the activations of the last
layer of ResNet, at different incremental time steps, in Figure 4.

Figure 4: Activations of the last layer of ResNet [22], throughout the
incremental learning steps. We compare label-supervision with VINIL (Fine-
tuning). Notice how the attention of the label-supervised variant is disrupted
after a few learning tasks. Instead, VINIL learns to segment out the target
object, successfully suppressing the background context, such as the hand or
the background.

Observe how VINIL learns to segment out the target object from the background.
This allows the model to accurately distinguish across different instances of
the same object sharing identical backgrounds. In contrast, label-supervised
variant progressively confuses the object with the background. We call such a
phenomenon ”attentional deficiency” of label-supervised representations.

VINIL Stores Instance-level Information. We present nearest neighbors for
three queries in Figure 5. We use the average-pooled activations of the last
ResNet layer on Core-$50$ trained with fine-tuning.

Figure 5: Five nearest neighbors for three object instance queries on
Core-$50$ [33] with fine-tuning. Green is a success, red is a failure. Observe
how VINIL retrieves object instances in different views. The last column
showcases a failure case, where both models fail to represent an object with
holes (scissor).

Observe how VINIL retrieves the same instance in different viewpoints, such as
for the light bulb and can. In contrast, label-supervision is distracted by
the background context, as it retrieves irrelevant objects with identical
background. This indicates self-supervision generalizes via storing instance-
level information. We present a failure case in the last row, as both models
fail to represent an object with holes and un-familiar rotation.

We conclude that VINIL can improve its performance with incoming stream of
data, and generalizes via focusing on the target object and storing instance-
level details to perform instance-incremental learning.

##  6 Discussion

This paper presented VINIL, a self-incremental visual instance learner. VINIL
sequentially learns visual object instances, with no label supervision, via
only self-supervision of BarlowTwins [56]. Below, we summarize our main
discussion points:

Self vs. Label-supervision? We demonstrate that self-supervision not only
omits the need for labels, but it is also more accurate and less forgetful.

W/ or W/o Memory? Our results show that the use of memory boosts label-
supervised instance incremental learning, however the improvement comes with
the cost of over-fitting on the training source.

Fine-tuning [44] vs. Replay [45] vs. EwC [28]? We demonstrate that with the
use of self-supervision, VINIL closes the gap between simple fine-tuning via
SGD and more complicated, compute-intensive techniques like memory replay or
regularization via EwC.

What Makes VINIL Effective? VINIL retains representations across tasks, and is
able to store and focus on instance-level information, which are crucial for
instance-incremental learning.

Limitation. VINIL is executed with regularization [28] and memory [45]. One
can also consider dynamic networks [55] whose architectures are updated with
incoming task data. VINIL is a scalable alternative to dynamic incremental
network training due to abundant unlabeled data.

##  7 Acknowledgements

Mert Kilickaya’s research is fully funded by ASM Pacific Technology (ASMPT).

## References

  * [1] https://research.facebook.com/publications/barlow-twins-self-supervised-learning-via-redundancy-reduction/.
  * [2] https://vlomonaco.github.io/core50/index.html.dataset.
  * [3] https://github.com/gyhandy/Group-Supervised-Learning.
  * [4] Yogesh Balaji, Mehrdad Farajtabar, Dong Yin, Alex Mott, and Ang Li.  The effectiveness of memory replay in large scale continual learning.  arXiv preprint, 2020.
  * [5] Luca Bertinetto, Jack Valmadre, Joao F Henriques, Andrea Vedaldi, and Philip HS Torr.  Fully-convolutional siamese networks for object tracking.  In ECCV, 2016.
  * [6] Ali Borji, Saeed Izadi, and Laurent Itti.  ilab-20m: A large-scale controlled object dataset to investigate deep learning.  In CVPR, 2016.
  * [7] Lucas Caccia and Joelle Pineau.  Special: Self-supervised pretraining for continual learning.  arXiv preprint, 2021.
  * [8] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin.  Unsupervised learning of visual features by contrasting cluster assignments.  NeurIPS, 2020.
  * [9] Francisco Manuel Castro, Manuel J. Marín-Jiménez, Nicolás Guil Mata, Cordelia Schmid, and Alahari Karteek.  End-to-end incremental learning.  ArXiv, 2018.
  * [10] Hyuntak Cha, Jaeho Lee, and Jinwoo Shin.  Co2l: Contrastive continual learning.  In ICCV, 2021.
  * [11] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton.  A simple framework for contrastive learning of visual representations.  In ICML. PMLR, 2020.
  * [12] Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He.  Improved baselines with momentum contrastive learning.  arXiv preprint, 2020.
  * [13] Xinlei Chen and Kaiming He.  Exploring simple siamese representation learning.  In CVPR, 2021.
  * [14] Sumit Chopra, Raia Hadsell, and Yann LeCun.  Learning a similarity metric discriminatively, with application to face verification.  In CVPR, 2005.
  * [15] Hehe Fan, Tao Zhuo, Xin Yu, Yi Yang, and Mohan Kankanhalli.  Understanding atomic hand-object interaction with human intention.  IEEE TCSVT, 2021.
  * [16] Enrico Fini, Victor G Turrisi da Costa, Xavier Alameda-Pineda, Elisa Ricci, Karteek Alahari, and Julien Mairal.  Self-supervised models are continual learners.  In CVPR, 2022.
  * [17] Jhair Gallardo, Tyler L Hayes, and Christopher Kanan.  Self-supervised training enhances online continual learning.  arXiv preprint, 2021.
  * [18] Jan-Mark Geusebroek, Gertjan J Burghouts, and Arnold WM Smeulders.  The amsterdam library of object images.  IJCV, 2005.
  * [19] Spyros Gidaris, Praveer Singh, and Nikos Komodakis.  Unsupervised representation learning by predicting image rotations.  arXiv preprint, 2018.
  * [20] Alex Gomez-Villa, Bartlomiej Twardowski, Lu Yu, Andrew D Bagdanov, and Joost van de Weijer.  Continually learning self-supervised representations with projected functional regularization.  In CVPR, 2022.
  * [21] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick.  Momentum contrast for unsupervised visual representation learning.  In CVPR, 2020.
  * [22] Kaiming He, X. Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  CVPR, 2016.
  * [23] Alexander Hermans, Lucas Beyer, and Bastian Leibe.  In defense of the triplet loss for person re-identification.  arXiv preprint, 2017.
  * [24] Stella Ho, Ming Liu, Lan Du, Longxiang Gao, and Yong Xiang.  Prototypes-guided memory replay for continual learning.  arXiv preprint, 2021.
  * [25] Saihui Hou, Xinyu Pan, Chen Change Loy, Zilei Wang, and Dahua Lin.  Learning a unified classifier incrementally via rebalancing.  CVPR, 2019.
  * [26] Dapeng Hu, Shipeng Yan, Qizhengqiu Lu, HONG Lanqing, Hailin Hu, Yifan Zhang, Zhenguo Li, Xinchao Wang, and Jiashi Feng.  How well does self-supervised pre-training perform with streaming data?  In ICLR, 2021.
  * [27] Minsoo Kang, Jaeyoo Park, and Bohyung Han.  Class-incremental learning by knowledge distillation with adaptive feature consolidation.  In CVPR, 2022.
  * [28] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al.  Overcoming catastrophic forgetting in neural networks.  Proceedings of the national academy of sciences, 2017.
  * [29] Yann LeCun, Fu Jie Huang, and Leon Bottou.  Learning methods for generic object recognition with invariance to pose and lighting.  In CVPR, 2004.
  * [30] Zhizhong Li and Derek Hoiem.  Learning without forgetting.  TPAMI, 2017.
  * [31] Tsung-Yi Lin, Serge Belongie, and James Hays.  Cross-view image geolocalization.  In CVPR, 2013.
  * [32] Weiyang Liu, Yandong Wen, Zhiding Yu, Ming Li, Bhiksha Raj, and Le Song.  Sphereface: Deep hypersphere embedding for face recognition.  In CVPR, 2017.
  * [33] Vincenzo Lomonaco and Davide Maltoni.  Core50: a new dataset and benchmark for continuous object recognition.  In CoRL. PMLR, 2017.
  * [34] Jian Ma and Dima Damen.  Hand-object interaction reasoning.  arXiv preprint, 2022.
  * [35] Divyam Madaan, Jaehong Yoon, Yuanchun Li, Yunxin Liu, and Sung Ju Hwang.  Representational continuity for unsupervised continual learning.  In ICLR, 2021.
  * [36] Marc Masana, Xialei Liu, Bartlomiej Twardowski, Mikel Menta, Andrew D Bagdanov, and Joost van de Weijer.  Class-incremental learning: survey and performance evaluation on image classification.  arXiv preprint, 2020.
  * [37] Sudhanshu Mittal, Silvio Galesso, and Thomas Brox.  Essentials for class incremental learning.  In CVPR, 2021.
  * [38] Sameer A Nene, Shree K Nayar, Hiroshi Murase, et al.  Columbia object image library (coil-100).  1996\.
  * [39] Mehdi Noroozi and Paolo Favaro.  Unsupervised learning of visual representations by solving jigsaw puzzles.  In ECCV, 2016.
  * [40] Hyun Oh Song, Yu Xiang, Stefanie Jegelka, and Silvio Savarese.  Deep metric learning via lifted structured feature embedding.  In CVPR, 2016.
  * [41] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer.  Automatic differentiation in pytorch.  2017\.
  * [42] Senthil Purushwalkam, Pedro Morgado, and Abhinav Gupta.  The challenges of continuous self-supervised learning.  arXiv preprint, 2022.
  * [43] Matthew Riemer, Ignacio Cases, Robert Ajemian, Miao Liu, Irina Rish, Yuhai Tu, and Gerald Tesauro.  Learning to learn without forgetting by maximizing transfer and minimizing interference.  arXiv preprint, 2018.
  * [44] Herbert E. Robbins.  A stochastic approximation method.  Annals of Mathematical Statistics, 2007.
  * [45] David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy Lillicrap, and Gregory Wayne.  Experience replay for continual learning.  NeurIPS, 2019.
  * [46] Yujiao Shi and Hongdong Li.  Beyond cross-view image retrieval: Highly accurate vehicle localization using satellite image.  In CVPR, pages 17010–17020, 2022.
  * [47] Hanul Shin, Jung Kwon Lee, Jaehong Kim, and Jiwon Kim.  Continual learning with deep generative replay.  NeurIPS, 2017.
  * [48] Bing Shuai, Andrew Berneshawi, Xinyu Li, Davide Modolo, and Joseph Tighe.  Siammot: Siamese multi-object tracking.  In CVPR, 2021.
  * [49] Ran Tao, Efstratios Gavves, and Arnold WM Smeulders.  Siamese instance search for tracking.  In CVPR, 2016.
  * [50] Bugra Tekin, Federica Bogo, and Marc Pollefeys.  H+ o: Unified egocentric recognition of 3d hand-object poses and interactions.  In CVPR, 2019.
  * [51] Shruti Vyas, Chen Chen, and Mubarak Shah.  Gama: Cross-view video geo-localization.  arXiv preprint, 2022.
  * [52] Jian Wang, Feng Zhou, Shilei Wen, Xiao Liu, and Yuanqing Lin.  Deep metric learning with angular loss.  In ICCV, 2017.
  * [53] Daniel Wilson, Xiaohan Zhang, Waqas Sultani, and Safwan Wshah.  Visual and object geo-localization: A comprehensive survey.  arXiv preprint, 2021.
  * [54] Yue Wu, Yinpeng Chen, Lijuan Wang, Yuancheng Ye, Zicheng Liu, Yandong Guo, and Yun Raymond Fu.  Large scale incremental learning.  CVPR, 2019.
  * [55] Jaehong Yoon, Eunho Yang, Jeongtae Lee, and Sung Ju Hwang.  Lifelong learning with dynamically expandable networks.  ArXiv, 2018.
  * [56] Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny.  Barlow twins: Self-supervised learning via redundancy reduction.  In ICML, 2021.
  * [57] Fei Zhu, Zhen Cheng, Xu-Yao Zhang, and Cheng-lin Liu.  Class-incremental learning via dual augmentation.  NeurIPS, 2021.
  * [58] Fei Zhu, Xu-Yao Zhang, Chuang Wang, Fei Yin, and Cheng-Lin Liu.  Prototype augmentation and self-supervision for incremental learning.  In CVPR, 2021.
  * [59] Sijie Zhu, Mubarak Shah, and Chen Chen.  Transgeo: Transformer is all you need for cross-view image geo-localization.  In CVPR, 2022.
  * [60] Sijie Zhu, Taojiannan Yang, and Chen Chen.  Vigor: Cross-view image geo-localization beyond one-to-one retrieval.  In CVPR, 2021.