# FILP-3D: Enhancing 3D Few-shot Class-incremental Learning with Pre-trained
Vision-Language Models

Wan Xu1 Tianyu Huang1 Tianyu Qu1 Guanglei Yang1 Yiwen Guo2 Wangmeng Zuo1
1Harbin Institute of Technology 2Independent Researcher
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Few-shot class-incremental learning (FSCIL) aims to mitigate the catastrophic
forgetting issue when a model is incrementally trained on limited data. While
the Contrastive Vision-Language Pre-Training (CLIP) model has been effective
in addressing 2D few/zero-shot learning tasks, its direct application to 3D
FSCIL faces limitations. These limitations arise from feature space
misalignment and significant noise in real-world scanned 3D data. To address
these challenges, we introduce two novel components: the Redundant Feature
Eliminator (RFE) and the Spatial Noise Compensator (SNC). RFE aligns the
feature spaces of input point clouds and their embeddings by performing a
unique dimensionality reduction on the feature space of pre-trained models
(PTMs), effectively eliminating redundant information without compromising
semantic integrity. On the other hand, SNC is a graph-based 3D model designed
to capture robust geometric information within point clouds, thereby
augmenting the knowledge lost due to projection, particularly when processing
real-world scanned data. Considering the imbalance in existing 3D datasets, we
also propose new evaluation metrics that offer a more nuanced assessment of a
3D FSCIL model. Traditional accuracy metrics are proved to be biased; thus,
our metrics focus on the model’s proficiency in learning new classes while
maintaining the balance between old and new classes. Experimental results on
both established 3D FSCIL benchmarks and our dataset demonstrate that our
approach significantly outperforms existing state-of-the-art methods. Code is
available at Code is available at https://github.com/HIT-leaderone/FLIP-3D

##  1 Introduction

The proliferation of 3D content in recent years has been marked by both
synthetic creations [2, 5] and real-world reconstruction [29, 24, 35]. This
expansion in data scale naturally introduces new classes into the 3D domain.
However, these emerging categories often comprise a limited number of
instances, posing a challenge for existing 3D recognition models.
Consequently, 3D few-shot class-incremental learning (FSCIL) has become
increasingly important in practical applications.

Previous work in 2D few-shot and zero-shot tasks [20, 47] has shown that Pre-
Training Models (PTMs) excel in incremental learning scenarios, often
outperforming non-PTM-based approaches. This superior performance is largely
attributed to the prior knowledge that PTMs acquire [6, 10, 21], which
enhances generalization in downstream tasks. Motivated by these successes, we
aim to leverage PTMs to imbue our model with shape-related prior knowledge.
However, the limited scale of available 3D data hampers the effectiveness of
existing 3D PTMs in downstream tasks. To address this, recent studies [43, 11,
40] have successfully aligned 3D representations with Vision-Language (V-L)
PTM knowledge. Building on this, we map point cloud data into a cross-modal
space, utilizing vision and language pre-training to improve performance in 3D
FSCIL tasks. Specifically, we generate multi-view depth maps from point clouds
and employ template text prompts for classification, such as “an image of a
{class name}”.

Figure 1: CLIP learns a large amount of prior knowledge from massive image-
text pairs. Thus pre-aligned image and text features contain sufficient shape-
related prior knowledge. Along with the elimination of redundant information
(RFE) and the compensator of 3D fine-grained information (SNC), the
performance in 3D FSCIL can be significantly improved.

While the integration of Vision-Language (V-L) Pre-Training Models (PTMs) into
FSCIL tasks has yielded improvements over previous methods, two key challenges
remain to limit performance. First, V-L PTMs are designed to capture detailed
visual features such as color and texture. However, point cloud data, being a
collection of discrete points, lacks the information to represent these
intricate visual details. This discrepancy can lead to inaccurate feature
extraction, adversely affecting performance. Second, the use of V-L PTMs
amplifies the issue of noise sensitivity, particularly in real-world scanned
data. Consequently, classification performance suffers significantly if noise
disrupts the rendering of depth maps and object contours.

To tackle the aforementioned challenges, we introduce Few-shot class
Incremental Learning tasks with Pre-training on 3D (FILP-3D), a framework that
employs CLIP as its backbone and incorporates two innovative components:
Redundant Feature Eliminator (RFE) and Spatial Noise Compensator (SNC), as is
shown in Figure 1. RFE serves as a specialized dimensionality reduction
technique, designed to remove redundant features while preserving semantic
content. By precisely compressing dimensions within the V-L PTMs, RFE
facilitates better alignment between the feature spaces of point clouds and
their corresponding embeddings. On the other hand, SNC is a graph-based 3D
model tailored to extract robust geometric information from point clouds. This
additional layer of information enhances the geometric relationships between
object components, mitigating the loss of knowledge that occurs during
projection, especially in noisy, real-world scanned data. As a result, our
FILP-3D framework gains improved resilience against noise interference.

Furthermore, we observe that the numbers of samples for different classes in
test datasets can vary by up to two orders of magnitude, leading to a
significant imbalance. This imbalance skews traditional accuracy metrics,
making them less reliable for comprehensive evaluation. To address this, we
introduce new evaluation metrics, namely $\rm NCAcc$ and $\rm F_{FSCIL}$.
These metrics are designed to assess the model’s ability to learn new classes
effectively while maintaining a balance between old and new classes, thereby
providing a more nuanced evaluation of the model performance.

In a nutshell, our contributions are three-fold:

  * •

We pioneer the application of Vision-Language Pre-Training Models (V-L PTMs)
to 3D Few-Shot Class-Incremental Learning (FSCIL) tasks, achieving performance
gains over existing models. The general embedding provided by V-L PTM, along
with its embedded prior knowledge, can complement the missing information in
few-shot tasks and alleviate catastrophic forgetting during the incremental
learning process.

  * •

We introduce FILP-3D, a framework that incorporates two innovative modules:
the Redundant Feature Eliminator (RFE) and the Spatial Noise Compensator
(SNC). RFE specifically addresses feature space misalignment, while SNC is
designed to mitigate the adverse effects of noise on the model. FILP-3D yields
notable performance improvements, especially in metrics associated with novel
classes.

  * •

We develop new metrics, namely $\rm NCAcc$ and $\rm F_{FSCIL}$ to provide a
more nuanced evaluation. These metrics assess a model’s ability to adeptly
learn new classes without compromising the performance on existing classes,
offering a more comprehensive evaluation framework for 3D FSCIL tasks.

##  2 Related Work

###  2.1 3D Point Cloud understanding

In recent years, many works have been proposed to classify 3D point cloud
objects. PointNet [18] and PointNet++ [19] design the architecture to maintain
natural invariances of the data. DGCNN [34] connects the point set into a
graph and designs a local neighbor aggregation strategy.

Some works employ masked point modeling [7, 12] as a 3D self-supervised
learning strategy to achieve great success. For example, Point BERT [39] uses
a pre-trained tokenizer to predict discrete point labels, while Point MAE [17]
and Point-M2AE [42] apply masked autoencoders to directly reconstruct the
masked 3D coordinates.

Recently, inspired by the breakthroughs in V-L PTMs [21], a number of
approaches are suggested to transfer 2D PTM to point cloud tasks and show
excellent performance. CLIP2Point [11] transfers CLIP to point cloud
classification with image-depth pre-training. CLIP2 [40] takes a step toward
open-world 3D vision understanding. I2P-MAE [44] leverages knowledge of 2D
PTMs to guide 3D MAE and ULIP [37] improves 3D understanding by aligning
features from images, texts, and point clouds. These models are supported by
prior knowledge from V-L PTMs, which can improve few-shot performance. In this
work, we aim to introduce V-L PTMs to 3D FSCIL.

###  2.2 Few-Shot Class-Incremental Learning

The FSCIL problem was proposed by Tao et al. [28]. Concretely, FSCIL aims at
learning from severely insufficient samples incrementally while preserving
already learned knowledge. TOPIC [28] uses a neural gas network to learn and
preserve the topology of features. Subsequently, CEC [41] utilizes a graph
model to propagate context information between classifiers for adaptation.
Also, some models like FACT [46] try to use virtual prototypes to reserve for
new ones, thus ensuring incremental learning ability. The SOTA method BiDist
[45] utilizes a novel distillation structure to alleviate the effects of
forgetting.

Recently, with the breakthroughs in 2D PTMs, a number of works attempt to
leverage the vast knowledge acquired by 2D PTMs, which is effective in
learning new concepts and alleviating the problem of forgetting [22, 47, 33].
With almost perfect performance, these models have attracted a lot of interest
and attention. The preceding methods are all 2D methods, except for Chowdhury
et al. [3]’s work which explores the FSCIL task in 3D point cloud data.
However, Chowdhury et al. [3]’s work neither supplements knowledge for few-
shot data nor addresses the high-noise nature of real-world scanned data,
resulting in limited performance in 3D FSCIL tasks. In contrast, our FILP-3D
successfully addresses the shortcomings mentioned above by introducing V-L
PTMs and two newly proposed modules.

##  3 Proposed Method

###  3.1 Problem Formulation

Assuming a sequence of $B$ tasks
$\mathcal{D}=\\{\mathcal{D}^{1},\mathcal{D}^{2},\dots,\mathcal{D}^{B}\\}$,
FSCIL methods incrementally recognize novel classes with a small amount of
training data. For the $b$-th task
$\mathcal{D}^{b}=\\{(\mathbf{x}_{i}^{b},y_{i}^{b})\\}_{i=1}^{n_{b}}$, we have
$n_{b}$ training samples, in which an instance $\mathbf{x}_{i}^{b}$ has a
class label $y_{i}^{b}$ and $y_{i}^{b}$ belongs to the label set $Y_{b}$. We
stipulate that $Y_{b}\cap Y_{b^{\prime}}=\emptyset$ when $b\not=b^{\prime}$.
After the $b$-th training task, trained models are required to classify test
sets of all the previous tasks $\\{\mathcal{D}^{1},\dots,\mathcal{D}^{b}\\}$.
Note that, in the 3D FSCIL setting, $\mathbf{x}_{i}^{b}\in\mathbb{R}^{P\times
3}$ denotes a point cloud object, where $P$ donates the number of points in a
point cloud object. $\mathcal{D}^{1}$ is the base task with a large-scale 3D
training dataset, while much fewer samples are included in the incremental
task $\mathcal{D}^{b}$, i.e., $n_{b}\ll n_{1}$ for $b>1$.

###  3.2 Model Overview

Figure 2: Overview of FILP-3D. FILP-3D mainly consists of three components,
i.e., 3D branch (SNC), 2D branch, and textual prototypes. Then, the 3D feature
and the 2D feature will be fused as a global feature, and be used to calculate
probability through the REF module alongside textual prototypes.

The framework of FILP-3D is shown in Figure 2. For each training sample
$(\mathbf{x}_{i}^{b},y_{i}^{b})$, we mainly have three branches: 1) point
cloud $\mathbf{x}_{i}^{b}$ is fed into 3D encoder to obtain an original point
feature $\mathbf{f}^{3D}$ and then aligned to $\mathbf{f}^{p}$, 2)
$\mathbf{x}_{i}^{b}$ is then rendered as multi-view depth maps
${\rm\mathbf{D}}^{b}_{i}$, embedded as depth features ${\rm\mathbf{F}}^{2D}$,
and finally merged into a global depth feature $\mathbf{f}^{d}$, 3) visible
class names are templated into text prompts and encoded as text features
${\rm\mathbf{F}}^{t}$. The point features $\mathbf{f}^{p}$ and the global
depth feature $\mathbf{f}^{d}$ are then fused as a global feature
$\mathbf{f}^{g}$. Afterward, we use pre-processed principal components
$\mathbf{V}$ (refer to Sec. 5 in Suppl. for more details of pre-processing) to
eliminate redundant dimensions of $\mathbf{f}^{g}$ and $\mathbf{F}^{t}$ and
generate $\widetilde{\mathbf{f}}^{g}$ and $\widetilde{\mathbf{F}}^{t}$. The
final predicted probability is calculated by our proposed renormalized cosine
similarity.

###  3.3 PTM is a Good 3D FSCIL Learner

Recent works [20, 47] have shown that PTMs substantially enhance the
performance in incremental learning tasks. However, Chowdhury et al. [3]’s
method, in the absence of shape-related prior knowledge, struggles with the
challenge of catastrophic forgetting in 3D continual learning. Further
complicating matters, the point features extracted by PointNet [18] are not in
alignment with the prototypes generated through word2vec [15]. Thus, Chowdhury
et al. [3]’s work finds it challenging to simultaneously retain features
associated with base classes (representing old knowledge) and integrate
features of novel classes (representing new knowledge).

In response to the aforementioned challenges, our initial approach was to
embed shape-related prior knowledge via a 3D PTM. However, the substantial
disparity in quality and volume between 3D data and its text/2D counterparts
renders current 3D PTMs less effective in generalizing to downstream tasks.
Motivated by this limitation, we turn to an alternative method. Drawing
inspiration from recent studies [43, 11], we leverage CLIP [21] to indirectly
infuse shape-related prior knowledge through projection. We christen this new
framework as SimpleCIL-3D.

Specifically, we project the point cloud data into multi-view depth maps
$\mathbf{D}^{b}_{1:N}$. A pre-trained ViT [8] in CLIP is then deployed to
extract depth features $\mathbf{F}^{2D}\in\mathbb{R}^{N\times C}$. Here, $N$
is the number of views, and $C$ is the embedding dimension of ViT. To allow
incremental tasks, a learnable merger is attached, formulating global depth
features $\mathbf{f}^{d}\in\mathbb{R}^{C}$ as follows,

$\mathbf{f}^{d}=f^{1}_{d}({\rm ReLU}(f^{2}_{d}({\rm
concat}(\mathbf{F}_{1:N}^{2D}))))$ (1)

where $f^{1}_{d}$, $f^{2}_{d}$ are two learnable MLPs.

The efficacy of setting classifier weights to average embeddings (referred to
as prototypes) for CIL tasks has been well-established by [25]. However, while
Chowdhury et al. [3]’s method employs word2vec to construct prototypes for new
classes, it falls short in encapsulating the average semantics of 3D point
clouds. In our approach, the depth features, denoted as $\mathbf{f}^{d}$, are
pre-aligned with text embeddings of CLIP. This enables SimpleCIL-3D to
progressively produce prototypes using a CLIP text encoder. For each class
labeled as $k$ with its respective name $t_{k}$, we introduce a template text
prompt: “an image or projection or sketch of a $t_{k}$”. This is mapped to the
CLIP prototype symbolized by $\mathbf{F}^{t}_{k}\in\mathbb{R}^{C}$. Owing to
the inherent association of our method between image and textual
representations, there is no compulsion to realign the two modalities during
incremental phases, unlike strategies such as in Chowdhury et al. [3]’s work.
As a result, we can directly utilize the cosine similarity between
$\mathbf{f}^{d}$ and $\mathbf{F}^{t}_{k}$ to represent the logit for class
$k$. The ultimate probability prediction, represented by $\mathbf{p}$, is
formulated as follows:

$l_{k}=\cos(\mathbf{f}^{d},\mathbf{F}^{t}_{k}),\quad\mathbf{p}={\rm
softmax}([l_{1},\dots,l_{K}]).$ (2)

where $\cos(\cdot,\cdot)$ denotes the cosine similarity, and $K$ is the number
of visual classes.

###  3.4 Ensure Performance of PTM in 3D FSCIL

Large-scale V-L PTMs undeniably serve as rich repositories of prior knowledge.
However, their application to 3D FSCIL tasks reveals unique challenges. First,
point cloud data, which consists of discrete points in 3D space, primarily
represents geometric features of objects. It inherently misses richer visual
nuances, such as color and texture. This deficiency is especially significant
given CLIP’s visual encoder, which inherently seeks and incorporates these
visual details into its semantic representation. Consequently, this could
jeopardize the precise classification of depth maps. Second, point clouds
derived from real-world scans often suffer from noise, which distorts the
extraction of depth-centric features. This problem is accentuated in multi-
view projection techniques, where a significant fraction of the points might
be occluded, thereby increasing the potential noise. Such disturbances can
upset the balance in few-shot incremental learning, leading to overfitting.
Addressing these challenges is imperative when tailoring PTMs for 3D FSCIL
tasks. In the ensuing sections, we aim to enhance SimpleCIL-3D with elucidate
solutions tailored to these specific hurdles, to obtain FILP-3D.

####  3.4.1 Redundant Feature Eliminator

We propose our design to mitigate the adverse effect of superfluous features
here. Our empirical observations indicate that redundant features typically
exhibit minimal inter-class distinction, leading to constrained variance. By
contrast, semantic features pertinent for classification distinctly express a
pronounced bias for each class, resulting in a more pronounced variance.
Chowdhury et al. [3], Zhu et al. [48] also yield empirical observations akin
to ours. Drawing upon this insight, we can discern between redundant and
semantic features based on their variance. Subsequently, we employ
Dimensionality Reduction (DR) techniques to preserve the semantic essence
while condensing the extraneous features.

Figure 3: The current feature space is generated by three principal
components. Each dimension’s one-hot vector represents a principal component.
The first two dimensions contain semantic information (high variance in the x
and y axes), while the third dimension serves as a redundant component (low
variance in the z-axis). By transforming feature vector (green) into the
feature space mentioned above, we can notice that projection can eliminate
redundant information, while normalization will improperly stretch the
semantic information.

Following the above discussions, we can obtain the following formulation for
$\widetilde{l}_{k}$, _i.e_., the logit after eliminating superfluous features.
For the redundant feature eliminated logit of the $k$-th class (the detail
discussion is in Sec. 1 of Suppl.), we have:

$\widetilde{l}_{k}=\frac{\widetilde{\mathbf{f}}^{d}\widetilde{\mathbf{F}}^{t^{\mathrm{T}}}_{k}}{\|\mathbf{f}^{d}\|\|\mathbf{F}^{t}_{k}\|}$
(3)

where $\widetilde{\mathbf{f}}^{d}=\mathbf{V}^{\mathrm{T}}\mathbf{f}^{d}$ and
$\widetilde{\mathbf{F}}^{t}_{k}=\mathbf{V}^{\mathrm{T}}\mathbf{F}^{t}_{k}$.
Here, $\mathbf{V}=[\mathbf{v}_{i}]_{M}$ consists of the principal components
extracted from the base task ($M$ is the number of principal components)

Note that, different from calculating the cosine similarity of
$\widetilde{\mathbf{f}}^{d}$ and $\widetilde{\mathbf{F}}^{t}_{k}$, the
_denominator_ in Eq. 3 is the norm of the original features $\mathbf{f}^{d}$
and $\mathbf{F}^{t}_{k}$. Principal features are directly dot multiplied after
normalizing with the original features, and such a procedure is named as the
renormalized cosine similarity (RCS). We can qualitatively compare the above
two similarity formulations: If the mode lengths of
$\widetilde{\mathbf{f}}^{d}$ and $\widetilde{\mathbf{F}}^{t}_{k}$ are
relatively small, it indicates that only a small portion of the features can
be accurately extracted. Performing normalization under such circumstances
will magnify the originally small features, resulting in incorrect
classification, as shown in Figure 3. Conversely, the dot method has no such
improper stretching operations.

####  3.4.2 Spatial Noise Compensator

SimpleCIL-3D unifies pre-aligned image and text encoders through comprehensive
data training. This ensures a substantial reservoir of prior knowledge,
minimizing the potential for information loss during any subsequent alignment
processes. Given these attributes, it stands as a markedly superior choice
when juxtaposed against partially-aligned 3D pre-trained models. Nonetheless,
the projection-based methodology inherent to SimpleCIL-3D exhibits heightened
sensitivity to both noise and viewpoint selection. When noise obscures the
clarity of depth maps, rendering them ineffective in delineating object
contours, there is a marked downturn in classification efficacy. As a
countermeasure, we propose the incorporation of a graph-based 3D model. This
addition is devised to augment the compromised information, bolstering our
model’s resilience against noise disturbances.

The 3D module yields features that are invariant to transformations, thereby
enhancing their robustness against noise. On the other hand, the multi-view
model, fortified by PTM, extracts semantically richer and more encompassing
information. To harness the strengths of both, we amalgamate the features
derived from the 3D module and the multi-view model. The global feature is
computed as:

$\displaystyle\mathbf{f}^{p}=f^{2}_{p}({\rm
ReLU}(f^{1}_{p}(\mathbf{f}^{3D})))$ (4)
$\displaystyle\mathbf{f}^{g}=\frac{1}{2}({\rm
max}(\mathbf{f}^{d},\mathbf{f}^{p})+{\rm avg}(\mathbf{f}^{d},\mathbf{f}^{p}))$
(5)

where $\mathbf{f}^{3D}$ is the feature extracted by 3D encoder from raw point
cloud data $\mathbf{x}_{i}^{b}$, and $\mathbf{f}^{g}$ represents new global
features used to replace $\mathbf{f}^{d}$ for further tasks. $f^{1}_{p}$,
$f^{2}_{p}$ are learnable MLPs, so that 3D features $\mathbf{f}^{p}$ can be
aligned to multi-view features $\mathbf{f}^{d}$.

Using this approach, any information inadequately captured due to viewpoint
selection is supplemented by the 3D channel using the ${\rm max}$ operation.
Concurrently, overly dominant features are moderated through the ${\rm
average}$ operation for balance.

###  3.5 Training Objective

The parameters of the 3D encoder and the multi-view render & encoder are fixed
during both base and incremental training stage. Only Merger (Eq. 1) for depth
feature fusion and Adapter (Eq. 4) for 3D feature alignment are trained. The
classification loss $\mathcal{L}^{b}_{cls}$ for $b$-th task can be calculated
as follows:

$\mathcal{L}^{b}_{cls}=\frac{1}{|\mathcal{D}^{b}|}\sum_{i=1}^{|\mathcal{D}^{b}|}\mathcal{L}_{ce}(\mathbf{p}^{b}_{i},y^{b}_{i})$
(6)

where $\mathcal{L}_{ce}$ is cross-entropy loss [4], $\mathbf{p}^{b}_{i}$
donates the predicted probability of the $i$-th sample of the $b$-th task.

Inspired by [38, 26], we also utilize contrastive learning [1] and data
augmentation in the training of 3D FSCIL models for continual learning.
Through the push and pull dynamics of contrastive learning, we optimize the
extracted features, bringing them closer to the appropriate prototype while
distancing them from erroneous ones. This strategy notably reduces ambiguity
throughout the continual learning phase. Concurrently, data augmentation not
only amplifies the efficacy of contrastive learning but also acts as a
safeguard against overfitting in few-shot scenarios.

Specifically, we employ random rotations along the coordinate axes and random
variations in the camera view distance as the augmentation function $f_{Aug}$.
For the $i$-th sample of the $b$-th task, we train using the corresponding
prototype $\mathbf{F}^{t}_{y^{b}_{i}}$ as a positive example and all other
visible prototypes $\mathbf{F}^{t}_{res^{b}_{i}}$ as negative examples. We
employ the InfoNCE loss [16] as contrastive learning loss. The contrastive
learning loss $\mathcal{L}^{b}_{cont}$ for the $b$-th task can be calculated
as follows:

$\mathcal{L}^{b}_{cont}=\frac{1}{|\mathcal{D}^{b}|}\sum_{i=1}^{|\mathcal{D}^{b}|}\sum_{j=1}^{N_{Aug}}\mathcal{L}_{InfoNCE}(\mathbf{f}^{g}_{i,j},\mathbf{F}^{t}_{y^{b}_{i}},\mathbf{F}^{t}_{res^{b}_{i}})$
(7)

where $N_{Aug}$ is the number of augmentations, and $\mathbf{f}^{g}_{i,j}$ is
the global feature encoded after replacing $x^{b}_{i}$ with
$f_{Aug}(x^{b}_{i})$.

The overall training loss $\mathcal{L}^{b}$ for $b$-th task can be calculated
as follows:

$\mathcal{L}^{b}=\mathcal{L}^{b}_{cls}+\alpha\mathcal{L}^{b}_{cont}$ (8)

##  4 Benckmark for 3D FSCIL task

Studies on 3D FSCIL benchmarks are still in early stage. The sole benchmark
introduced by Chowdhury et al. [3] has several notable limitations: 1) In the
synthetic data to synthetic data (S2S) task set forth by Chowdhury et al. [3],
the total number of classes is limited. This restricts the capacity of the
model for incremental learning and makes it challenging to evaluate its
effectiveness comprehensively. 2) For the synthetic data to real-scanned data
(S2R) task, many analogous classes have been removed without clear
justification. 3) Relying solely on accuracy as a metric means that
incremental classes may not receive the emphasis they warrant. In light of
these observations, we introduce a new benchmark: FSCIL3D-XL. Details of this
benchmark will be discussed in the subsequent sections.

###  4.1 Task Setting

FSCIL3D-XL comprises two series of incremental tasks: S2S and S2R. The S2S
task acts as a transitional and simulation-based task, primarily designed to
assess the model’s capability to mitigate issues like overfitting or
catastrophic forgetting. On the other hand, the S2R task is inherently more
complex and has broader applicability. Given its larger domain gap and
challenges with noise, it demands a more robust ability to generalize from
limited samples.

S2S Task. Chowdhury et al. [3]’s dataset is constructed using a single
dataset, encompassing only 55 classes. In our benchmark, we opt for ShapeNet
[2] as our base dataset and ModelNet40 [36] as the incremental dataset. Our
base task retains all the 55 classes from ShapeNet [2]. For the incremental
tasks, we exclude 16 classes from ModelNet40 [36] that overlap with classes in
the base task, and the remaining 24 unique classes from ModelNet40 are then
evenly distributed across 6 incremental tasks. In contrast to Chowdhury et al.
[3]’s work, our S2S task offers an increase in class count by over 40%,
introducing a heightened level of challenge.

S2R Task. Chowdhury et al. [3]’s benchmark unjustifiably excludes certain
classes that, while similar, are distinct in fact. For instance, semantically
similar classes like “handbag” and “bag”, or shape-analogous classes like
“ashcan” and “can”, are filtered out. By contrast, we retain all available
classes in our benchmark. Recognizing the subtle differences between such
similar classes is undoubtedly challenging, but we believe it is essential for
a comprehensive evaluation. We continue to use ShapeNet as the base dataset
and have selected CO3D [23] as the incremental dataset. From CO3D, we exclude
9 overlapping classes, resulting in 41 distinct classes designated for the
incremental tasks.

For a more detail of the task settings in FSCIL3D-XL, please refer to Sec. 3
of Suppl..

###  4.2 Evaluation metrics

3D datasets present challenges distinct from their 2D counterparts. Firstly,
there is a pronounced imbalance in the scale of different classes within 3D
datasets. For instance, in ShapeNet, the “chair” class boasts over 1,000
training samples, whereas “birdhouse” has a mere 15. Secondly, incremental
samples in 3D datasets, particularly those sourced from real-scanned data,
tend to be more intricate. These novel classes, given their complexity, demand
heightened attention in the FSCIL task. Relying solely on the accuracy metric,
denoted as ${\rm Acc}$, would inadequately addresses these challenges. As a
result, while we retain the 2D evaluation metrics ${\rm[Acc_{i}]_{B}}$
(accuracy of each session) and ${\Delta}$ [27] (relative accuracy dropping
rate, where ${\rm\Delta=\frac{|Acc_{B}-Acc_{1}|}{Acc_{1}}}$), we also
introduce new evaluation metrics specifically designed to address the
aforementioned issues.

Macro accuracy. Macro accuracy (${\rm MAcc}$) indicates the generalization
ability of the model, preventing overfitting in a small number of classes with
a relatively large number of samples. It can be calculated using the following
formula for $b$-th task:

${\rm MAcc_{b}}=\frac{1}{K_{b}}\sum_{i=1}^{K_{b}}Acc_{i}$ (9)

where $K_{b}=\sum_{i=1}^{b}|Y_{b}|$ donates the number of visible classes for
the $b$-th task, and $Acc_{i}$ donates the accuracy of the $i$-th class.

Novel class accuracy. Novel class accuracy (${\rm NCAcc}$) indicates the
ability of the model to learn new classes, preventing the model from over-
focusing on base classes with a large number of samples. It can be calculated
using the following formula for $b$-th task:

$\displaystyle{\rm
NCAcc_{b}}=\frac{|\\{p_{i}^{b}|p_{i}^{b}=y_{i}^{b},p_{i}^{b}\in
P_{b},y_{i}^{b}\in Y_{b}\\}|}{|\mathcal{D}^{b}|}$ (10) $\displaystyle{\rm
NCAcc}=\frac{1}{B}\sum_{b=1}^{B}{\rm NCAcc_{b}}$ (11)

where $P_{b}$ donates the predicted labels for test stage of $b$-th task and
$B$ donates the number of task.

F-Score. F-Score for FSCIL task (${\rm F_{FSCIL}}$, ${\rm F}$ for short): The
network needs to be plastic to learn new knowledge from the current task, and
it also needs to be stable to maintain knowledge learned from previous tasks.
To ensure that the model is not overly biased towards either of these aspects,
we refer to ${\rm F_{score}}$ and propose ${\rm F_{FSCIL}}$, which balances
the plasticity and stability of the evaluation network. It can be calculated
using the following formula:

${\rm F_{FSCIL}}=\frac{2\ {\rm Acc_{B}}\ {\rm NCAcc}}{{\rm Acc_{B}}+{\rm
NCAcc}}$ (12)

where ${\rm Acc_{B}}$ denotes the accuracy of the ${\rm B}$-th (final)
session.

Finally, the 3D dataset is still being refined, so a large number of new
datasets will likely continue to be produced. We take this into account and
design our benchmark to be more flexible and modular in adding datasets, i.e.,
the desired FSCIL dataset can be obtained by simply transmitting the
parameters and datasets to our generator. Additionally, The benchmark will be
open-resource.

##  5 Experiments

In this section, we first provide more detailed information about our models
and experiments. Then, we present and analyze the results of the comparison
experiments. Finally, we present our ablation experiment to verify the
components of our model.

Table 1: Quantitative results on the S2S task. For each set of results, the micro/marco average are presented at the top/bottom respectively. Bold donates the best performance, $joint$ serves solely as an upper reference limit in our model and is not involved in the comparison. Method | Pub. Year | Acc. in each session $\uparrow$ | Evaluation metrics
---|---|---|---
0 | 1 | 2 | 3 | 4 | 5 | 6 | ${\rm NCAcc}\uparrow$ | ${\rm\Delta}\downarrow$ | ${\rm F}\uparrow$
FACT [46] | CVPR’22 | 82.6 | 77.0 | 72.4 | 69.8 | 68.4 | 67.7 | 67.3 | 41.7 | 18.5 | 51.5
48.0 | 44.7 | 42.0 | 39.9 | 38.2 | 37.1 | 36.5 | 34.0 | 23.9 | 35.2
BiDist [45] | CVPR’23 | 89.6 | 87.7 | 86.2 | 84.7 | 83.8 | 83.6 | 82.3 | 35.0 | 8.1 | 49.1
82.5 | 80.3 | 77.3 | 75.2 | 72.5 | 71.2 | 69.3 | 38.0 | 16.0 | 49.1
Chowdhury et al.’s [3] | ECCV’22 | 86.9 | 84.6 | 82.8 | 78.3 | 78.5 | 71.5 | 68.6 | 50.8 | 21.1 | 58.4
73.0 | 62.8 | 65.2 | 62.8 | 60.9 | 57.9 | 55.1 | 45.5 | 24.5 | 49.8
SimpleCIL-3D (ours) | - | 90.4 | 88.0 | 85.6 | 85.0 | 84.1 | 78.4 | 80.1 | 68.2 | 11.4 | 73.7
78.9 | 76.1 | 72.8 | 71.1 | 70.8 | 70.2 | 69.7 | 68.8 | 11.7 | 69.2
FILP-3D (ours) | - | 90.6 | 89.0 | 86.7 | 84.2 | 83.2 | 81.8 | 82.2 | 79.3 | 9.3 | 80.7
80.0 | 76.8 | 74.9 | 72.8 | 71.2 | 70.9 | 70.7 | 77.0 | 11.6 | 73.7
$Joint$ FILP-3D | - | 90.6 | 89.1 | 88.5 | 87.8 | 87.4 | 87.6 | 86.8 | 79.9 | 4.2 | 83.2
80.0 | 78.5 | 79.0 | 77.2 | 77.5 | 77.7 | 77.3 | 79.9 | 3.4 | 78.6
Table 2: Quantitative results on the S2R task. For each set of results, the
micro/marco average are presented at the top/bottom respectively. Bold donates
the best performance, $joint$ serves solely as an upper reference limit in our
model and is not involved in the comparison.

Method | Acc. in each session $\uparrow$ | Evaluation metrics
---|---|---
0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | ${\rm NCAcc}\uparrow$ | ${\rm\Delta}\downarrow$ | ${\rm F}\uparrow$
FACT [46] | 82.4 | 77.2 | 74.5 | 73.1 | 71.3 | 70.4 | 67.2 | 65.2 | 63.8 | 61.8 | 59.9 | 59.8 | 26.2 | 27.4 | 36.4
48.6 | 41.4 | 39.7 | 36.8 | 35.5 | 33.6 | 31.2 | 29.5 | 28.4 | 27.2 | 25.8 | 25.9 | 30.8 | 46.7 | 28.1
BiDist [45] | 89.4 | 54.0 | 54.7 | 56.4 | 57.0 | 55.9 | 56.3 | 52.9 | 52.3 | 51.7 | 50.8 | 50.1 | 47.2 | 43.9 | 48.6
81.8 | 52.8 | 50.1 | 46.2 | 48.3 | 46.1 | 44.7 | 41.8 | 41.8 | 39.8 | 40.0 | 39.7 | 41.9 | 51.5 | 40.8
Chowdhury et al.’s [3] | 85.2 | 78.6 | 71.0 | 72.0 | 75.2 | 68.8 | 56.1 | 58.5 | 62.9 | 59.1 | 52.2 | 59.4 | 35.3 | 30.3 | 44.3
68.2 | 56.2 | 50.5 | 48.4 | 53.5 | 46.7 | 39.9 | 37.6 | 36.9 | 33.1 | 34.3 | 44.1 | 36.5 | 35.3 | 40.0
SimpleCIL-3D (ours) | 89.2 | 86.7 | 83.5 | 81.7 | 79.4 | 79.6 | 78.6 | 70.4 | 72.1 | 71.7 | 70.1 | 71.2 | 49.6 | 20.2 | 58.5
78.9 | 77.3 | 75.9 | 73.7 | 70.1 | 66.5 | 64.4 | 60.9 | 59.4 | 58.5 | 54.5 | 56.3 | 49.0 | 28.6 | 52.4
FILP-3D (ours) | 90.0 | 87.0 | 86.4 | 85.0 | 83.7 | 82.7 | 81.4 | 79.4 | 78.2 | 76.8 | 74.8 | 74.6 | 60.6 | 17.1 | 66.9
79.4 | 75.4 | 75.7 | 72.4 | 70.2 | 68.5 | 65.9 | 63.5 | 62.2 | 59.6 | 57.5 | 57.3 | 59.9 | 27.8 | 58.6
$Joint$ FILP-3D | 90.0 | 89.2 | 89.0 | 88.4 | 88.1 | 87.7 | 87.2 | 86.9 | 85.9 | 84.6 | 83.1 | 83.1 | 54.9 | 7.7 | 66.1
79.4 | 77.2 | 76.3 | 75.9 | 75.5 | 73.5 | 73.3 | 73.4 | 70.2 | 69.3 | 68.2 | 68.2 | 60.0 | 14.1 | 63.8

###  5.1 Implementation Details

We choose CLIP’s ViT-B/32 [21] as our pre-trained model, replace its visual
encoder with CLIP2Point’s pre-trained depth encoder [11] and adopt
CLIP2Point’s proposed rendering approach. As for the 3D encoder, we use DGCNN
[34] pre-trained on ShapeNet, which follows OcCo [32]. Dimensions of both
image features and text features are 512. We use SVD [9] as our dimensionality
reduction method. We extract principal components (242 out of 512) based on
the base task, which retains 95% energy of the principal components. We set
the temperature parameter of infoNCE $\tau$ and balance parameter $\alpha$ in
Eq. 8 to 0.1 and 1.0 respectively. For training, we use ADAM weight decay
optimizer [13]. We set the learning rate to $1\times 10^{-3}$ and the weight
decay to $1\times 10^{-4}$. Training in the base task and incremental take 10
epochs and 20 epochs respectively. For incremental tasks, we randomly select 5
samples/classes for training and allow 1 exemplar/class from previous tasks to
be used as memory. The batch size is 32.

###  5.2 Experimental Results

We conduct extensive experiments on FSCIL3D-XL’s S2S task (Table 1),
FSCIL3D-XL’s S2R task (Table 2), and Chowdhury et al. [3]’s benchmark (Sec. 6
of Suppl.). To make a comprehensive comparison, we choose the following
models: 1) $Joint$: Models are trained with samples from all currently visible
classes, which can represent their upper bound in FSCIL tasks. 2) One 3D SOTA
approach developed by Chowdhury et al. [3]. 3) Two 2D SOTA approaches FACT
[46] and BiDist [45]. FACT uses manifold-mixup [31] and virtual prototypes to
ensure forward compatibility. BiDist uses knowledge distillation to retain
knowledge. We use official codes to reproduce these methods, keeping all the
original parameters unchanged. The only difference is that we replace the CNN-
based network with CLIP2Point’s depth encoder to allow 3D application. For
visualization results, more comparison results (including ULIP [37] and
PointCLIP [43]) and effect of our proposed metrics, please refer to Sec. 4,7,8
of Suppl.

Experiments on the S2S task of FSCIL3D-XL. Table 1 reports the results of the
synthetic data to the synthetic data task in our benchmark. BiDist [45]
additionally employs a weighted sum of distillation loss for optimization,
along with a relatively higher loss weight assigned to it. Therefore, BiDist
tends to preserve existing knowledge and sacrifice the learning capacity for
new classes, leading to a relatively low $\rm NCAcc$. Moreover, the S2S task
has a limited proportion of incremental data, BiDist can achieve top $\rm Acc$
in over half of the tasks by keeping old knowledge. Contrary to it, FILP-3D
has a trade-off between the base classes and the novel classes. The results
also show that our metrics are reasonable for the evaluation of 3D FSCIL.

Experiments on the S2R task of FSCIL3D-XL. Table 2 reports the results of the
synthetic data to the real-scanned data task in our benchmark. Considering the
large domain gap between synthetic data and real-scanned data, it is even
harder to acquire knowledge from new classes based on previous knowledge of
synthetic shapes. We notice that BiDist witnesses a catastrophic performance
decline from task 1 to task 2, which further demonstrates that BiDist tends to
retain old knowledge and sacrifice the acquisition of new knowledge. In
contrast, SimpleCIL-3D has already outperformed the other three methods, which
proves that prior knowledge from pre-training models is effective for few-shot
incremental tasks. However, we notice that its $\rm NCAcc$ is much lower than
ours, indicating that SimpleCIL-3D performs poorly in real-scanned data. With
the Spatial Noise Compensator, our FILP-3D achieves a state-of-the-art
performance, almost close to its upper bound.

Table 3: Ablation studies on the S2R task of FSCIL3D-XL. For no contrastive
learning (CL) case, training loss $\mathcal{L}^{b}=\mathcal{L}^{b}_{cls}$

.  RFE SNC CL 0 11 ${\rm NCAcc}\uparrow$ ${\rm\Delta}\downarrow$ ${\rm
F}\uparrow$ ✗ ✗ ✗ 86.7 71.2 49.6 17.9 58.5 78.9 56.3 49.0 28.6 52.4 ✓ ✗ ✗ 90.4
75.6 50.8 16.3 60.8 80.0 58.3 51.0 27.1 54.4 ✗ ✓ ✗ 90.1 73.6 54.5 18.3 62.6
79.3 56.7 56.1 28.5 56.4 ✓ ✓ ✗ 89.8 74.0 57.8 17.6 64.9 78.7 56.7 58.0 28.0
57.3 ✓ ✓ ✓ 90.0 74.6 60.0 17.1 66.9 79.4 57.3 59.9 27.8 58.6

###  5.3 Ablation Studies

To evaluate the effectiveness of our proposed modules and components, we
conduct several ablation experiments. We verify the RCS, the Redundant Feature
Eliminator on the S2S task (Sec. 2 of Suppl.), and further verify the
Redundant Feature Eliminator, the Spatial Noise Compensator and contrastive
learning on the S2R task (Table 3).

Effect of Redundant Feature Eliminator. For no RFE case (line 1 and 3), we use
cosine similarity between $\mathbf{f}^{g}$ and ${\rm\mathbf{F}}^{t}$ to
calculate logits. One can notice that the model with RFE (line 2) can only
slightly outperform SimpleCIL-3D (line 1) in $\rm NCAcc$. We analyze the
reason as follows: The huge domain gap leads to some semantic features
available in synthetic data can not be extracted efficiently on real-scanned
data. Adding some real-scanned data samples in the principal component
extraction stage as prior knowledge may effectively alleviate this symptom.
Ablation study for RFE in the S2S task (Sec. 2 of Suppl.) can demonstrate a
more reasonable comparison and prove the effectiveness of our method in
solving redundant feature problems.

Effect of Spatial Noise Compensator. Let’s compare the results of the line 2
with the line 4. On the one hand, there is a slight decrease in $\rm Acc$,
which indicates that the 3D module is not as capable as the multi-view model
with 2D prior knowledge. It is suitable only for supplementing the information
in the 3D FSCIL task. On the other hand, there is a significant increase in
the incremental part of the evaluation metric, indicating that the 3D module
can complement the global information that the multi-view model lacks and
enhance the anti-interference ability.

##  6 Conclusion

In this paper, we proposed FILP-3D, a 3D few-shot class-incremental learning
framework with pre-trained V-L models. Specifically, we introduced a V-L pre-
trained model CLIP to the 3D FSCIL task. To guarantee that CLIP performs well
in the 3D FSCIL task, we proposed a Redundant Feature Eliminator to eliminate
redundant features without stretching semantic information and a Spatial Noise
Compensator to complement noise information. To comprehensively evaluate the
3D FSCIL task, we further proposed a new benchmark FSCIL3D-XL, which retains
all classes and introduces reasonable evaluation metrics. Extensive
experiments demonstrate that our FILP-3D achieves state-of-the-art performance
in all available benchmarks. Results from ablation studies further verify the
effectiveness of the proposed components.

## References

  * Caron et al. [2020] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin.  Unsupervised learning of visual features by contrasting cluster assignments.  _Advances in neural information processing systems_ , 33:9912–9924, 2020.
  * Chang et al. [2015] Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et al.  Shapenet: An information-rich 3d model repository.  _arXiv preprint arXiv:1512.03012_ , 2015.
  * Chowdhury et al. [2022] Townim Chowdhury, Ali Cheraghian, Sameera Ramasinghe, Sahar Ahmadi, Morteza Saberi, and Shafin Rahman.  Few-shot class-incremental learning for 3d point cloud objects.  In _European Conference on Computer Vision_ , pages 204–220. Springer, 2022.
  * De Boer et al. [2005] Pieter-Tjerk De Boer, Dirk P Kroese, Shie Mannor, and Reuven Y Rubinstein.  A tutorial on the cross-entropy method.  _Annals of operations research_ , 134:19–67, 2005.
  * Deitke et al. [2023] Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi.  Objaverse: A universe of annotated 3d objects.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 13142–13153, 2023.
  * Devlin et al. [2018] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_ , 2018.
  * Dong et al. [2022] Runpei Dong, Zekun Qi, Linfeng Zhang, Junbo Zhang, Jianjian Sun, Zheng Ge, Li Yi, and Kaisheng Ma.  Autoencoders as cross-modal teachers: Can pretrained 2d image transformers help 3d representation learning?  _arXiv preprint arXiv:2212.08320_ , 2022.
  * Dosovitskiy et al. [2020] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  _arXiv preprint arXiv:2010.11929_ , 2020.
  * Golub and Van Loan [1996] Gene H. Golub and Charles F. Van Loan.  _Matrix Computations_.  Johns Hopkins University Press, 1996.
  * He et al. [2022] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick.  Masked autoencoders are scalable vision learners.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 16000–16009, 2022.
  * Huang et al. [2022] Tianyu Huang, Bowen Dong, Yunhan Yang, Xiaoshui Huang, Rynson WH Lau, Wanli Ouyang, and Wangmeng Zuo.  Clip2point: Transfer clip to point cloud classification with image-depth pre-training.  _arXiv preprint arXiv:2210.01055_ , 2022.
  * Liu et al. [2022] Haotian Liu, Mu Cai, and Yong Jae Lee.  Masked discrimination for self-supervised learning on point clouds.  In _European Conference on Computer Vision_ , pages 657–675. Springer, 2022.
  * Loshchilov and Hutter [2017] Ilya Loshchilov and Frank Hutter.  Decoupled weight decay regularization.  _arXiv preprint arXiv:1711.05101_ , 2017.
  * Ma et al. [2022] Xu Ma, Can Qin, Haoxuan You, Haoxi Ran, and Yun Fu.  Rethinking network design and local geometry in point cloud: A simple residual mlp framework.  _arXiv preprint arXiv:2202.07123_ , 2022.
  * Mikolov et al. [2013] Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean.  Efficient estimation of word representations in vector space.  _arXiv preprint arXiv:1301.3781_ , 2013.
  * Oord et al. [2018] Aaron van den Oord, Yazhe Li, and Oriol Vinyals.  Representation learning with contrastive predictive coding.  _arXiv preprint arXiv:1807.03748_ , 2018.
  * Pang et al. [2022] Yatian Pang, Wenxiao Wang, Francis EH Tay, Wei Liu, Yonghong Tian, and Li Yuan.  Masked autoencoders for point cloud self-supervised learning.  In _European conference on computer vision_ , pages 604–621. Springer, 2022.
  * Qi et al. [2017a] Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas.  Pointnet: Deep learning on point sets for 3d classification and segmentation.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 652–660, 2017a.
  * Qi et al. [2017b] Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas.  Pointnet++: Deep hierarchical feature learning on point sets in a metric space.  _Advances in neural information processing systems_ , 30, 2017b.
  * Qin and Joty [2022] Chengwei Qin and Shafiq Joty.  Continual few-shot relation learning via embedding space regularization and data augmentation.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 2776–2789, 2022.
  * Radford et al. [2021a] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In _International conference on machine learning_ , pages 8748–8763. PMLR, 2021a.
  * Radford et al. [2021b] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In _International conference on machine learning_ , pages 8748–8763. PMLR, 2021b.
  * Reizenstein et al. [2021a] Jeremy Reizenstein, Roman Shapovalov, Philipp Henzler, Luca Sbordone, Patrick Labatut, and David Novotny.  Common objects in 3d: Large-scale learning and evaluation of real-life 3d category reconstruction.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 10901–10911, 2021a.
  * Reizenstein et al. [2021b] Jeremy Reizenstein, Roman Shapovalov, Philipp Henzler, Luca Sbordone, Patrick Labatut, and David Novotny.  Common objects in 3d: Large-scale learning and evaluation of real-life 3d category reconstruction.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 10901–10911, 2021b.
  * Snell et al. [2017] Jake Snell, Kevin Swersky, and Richard Zemel.  Prototypical networks for few-shot learning.  _Advances in neural information processing systems_ , 30, 2017.
  * Song et al. [2023] Zeyin Song, Yifan Zhao, Yujun Shi, Peixi Peng, Li Yuan, and Yonghong Tian.  Learning with fantasy: Semantic-aware virtual contrastive constraint for few-shot class-incremental learning.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 24183–24192, 2023.
  * Tan et al. [2022] Zhen Tan, Kaize Ding, Ruocheng Guo, and Huan Liu.  Graph few-shot class-incremental learning.  In _Proceedings of the Fifteenth ACM International Conference on Web Search and Data Mining_ , pages 987–996, 2022.
  * Tao et al. [2020] Xiaoyu Tao, Xiaopeng Hong, Xinyuan Chang, Songlin Dong, Xing Wei, and Yihong Gong.  Few-shot class-incremental learning.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 12183–12192, 2020.
  * Uy et al. [2019] Mikaela Angelina Uy, Quang-Hieu Pham, Binh-Son Hua, Thanh Nguyen, and Sai-Kit Yeung.  Revisiting point cloud classification: A new benchmark dataset and classification model on real-world data.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , pages 1588–1597, 2019.
  * Van der Maaten and Hinton [2008] Laurens Van der Maaten and Geoffrey Hinton.  Visualizing data using t-sne.  _Journal of machine learning research_ , 9(11), 2008.
  * Verma et al. [2019] Vikas Verma, Alex Lamb, Christopher Beckham, Amir Najafi, Ioannis Mitliagkas, David Lopez-Paz, and Yoshua Bengio.  Manifold mixup: Better representations by interpolating hidden states.  In _International conference on machine learning_ , pages 6438–6447. PMLR, 2019.
  * Wang et al. [2021] Hanchen Wang, Qi Liu, Xiangyu Yue, Joan Lasenby, and Matt J Kusner.  Unsupervised point cloud pre-training via occlusion completion.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , pages 9782–9792, 2021.
  * Wang et al. [2023] Runqi Wang, Xiaoyue Duan, Guoliang Kang, Jianzhuang Liu, Shaohui Lin, Songcen Xu, Jinhu Lü, and Baochang Zhang.  Attriclip: A non-incremental learner for incremental knowledge learning.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 3654–3663, 2023.
  * Wang et al. [2019] Yue Wang, Yongbin Sun, Ziwei Liu, Sanjay E Sarma, Michael M Bronstein, and Justin M Solomon.  Dynamic graph cnn for learning on point clouds.  _ACM Transactions on Graphics (tog)_ , 38(5):1–12, 2019.
  * Wu et al. [2023] Tong Wu, Jiarui Zhang, Xiao Fu, Yuxin Wang, Jiawei Ren, Liang Pan, Wayne Wu, Lei Yang, Jiaqi Wang, Chen Qian, et al.  Omniobject3d: Large-vocabulary 3d object dataset for realistic perception, reconstruction and generation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 803–814, 2023.
  * Wu et al. [2015] Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao.  3d shapenets: A deep representation for volumetric shapes.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 1912–1920, 2015.
  * Xue et al. [2023] Le Xue, Mingfei Gao, Chen Xing, Roberto Martín-Martín, Jiajun Wu, Caiming Xiong, Ran Xu, Juan Carlos Niebles, and Silvio Savarese.  Ulip: Learning a unified representation of language, images, and point clouds for 3d understanding.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 1179–1189, 2023.
  * Yoon et al. [2023] In-Ug Yoon, Tae-Min Choi, Young-Min Kim, and Jong-Hwan Kim.  Balanced supervised contrastive learning for few-shot class-incremental learning.  _arXiv preprint arXiv:2305.16687_ , 2023.
  * Yu et al. [2022] Xumin Yu, Lulu Tang, Yongming Rao, Tiejun Huang, Jie Zhou, and Jiwen Lu.  Point-bert: Pre-training 3d point cloud transformers with masked point modeling.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 19313–19322, 2022.
  * Zeng et al. [2023] Yihan Zeng, Chenhan Jiang, Jiageng Mao, Jianhua Han, Chaoqiang Ye, Qingqiu Huang, Dit-Yan Yeung, Zhen Yang, Xiaodan Liang, and Hang Xu.  Clip2: Contrastive language-image-point pretraining from real-world point cloud data.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 15244–15253, 2023.
  * Zhang et al. [2021] Chi Zhang, Nan Song, Guosheng Lin, Yun Zheng, Pan Pan, and Yinghui Xu.  Few-shot incremental learning with continually evolved classifiers.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 12455–12464, 2021.
  * Zhang et al. [2022a] Renrui Zhang, Ziyu Guo, Peng Gao, Rongyao Fang, Bin Zhao, Dong Wang, Yu Qiao, and Hongsheng Li.  Point-m2ae: multi-scale masked autoencoders for hierarchical point cloud pre-training.  _Advances in neural information processing systems_ , 35:27061–27074, 2022a.
  * Zhang et al. [2022b] Renrui Zhang, Ziyu Guo, Wei Zhang, Kunchang Li, Xupeng Miao, Bin Cui, Yu Qiao, Peng Gao, and Hongsheng Li.  Pointclip: Point cloud understanding by clip.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 8552–8562, 2022b.
  * Zhang et al. [2023] Renrui Zhang, Liuhui Wang, Yu Qiao, Peng Gao, and Hongsheng Li.  Learning 3d representations from 2d pre-trained models via image-to-point masked autoencoders.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 21769–21780, 2023.
  * Zhao et al. [2023] Linglan Zhao, Jing Lu, Yunlu Xu, Zhanzhan Cheng, Dashan Guo, Yi Niu, and Xiangzhong Fang.  Few-shot class-incremental learning via class-aware bilateral distillation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 11838–11847, 2023.
  * Zhou et al. [2022] Da-Wei Zhou, Fu-Yun Wang, Han-Jia Ye, Liang Ma, Shiliang Pu, and De-Chuan Zhan.  Forward compatible few-shot class-incremental learning.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 9046–9056, 2022.
  * Zhou et al. [2023] Da-Wei Zhou, Han-Jia Ye, De-Chuan Zhan, and Ziwei Liu.  Revisiting class-incremental learning with pre-trained models: Generalizability and adaptivity are all you need.  _arXiv preprint arXiv:2303.07338_ , 2023.
  * Zhu et al. [2023] Xiangyang Zhu, Renrui Zhang, Bowei He, Aojun Zhou, Dong Wang, Bin Zhao, and Peng Gao.  Not all features matter: Enhancing few-shot clip with adaptive prior refinement.  _arXiv preprint arXiv:2304.01195_ , 2023.

Supplementary Material

In the supplementary materials, we introduce a detailed discussion of RCS in
Section 7 and conduct ablation experiments to demonstrate the effectiveness of
RCS and RFE in Section 8 . We provide more details about the task settings of
our proposed benchmark FSCIL-3D XL in Section 9 as a supplement to our
proposed benchmark. We provide visualizations in Section 10 for a more
intuitive comparison between our model and other models (or ablation results).
We elaborate on the preprocessing details of the principal component
$\mathbf{V}$ in Section 11 to demonstrate the details of our model. We present
experiments conducted on Chowdhury et al.’s benchmark in Section 12 and
comparison experiments with more methods in Section 13 to demonstrate the
superiority of our model as well as the generality and effectiveness of our
proposed modules. Finally, by analyzing the experimental results, we
demonstrate how our proposed metrics offer a more comprehensive evaluation of
3D FSCIL models in Section 14.

##  7 Detail Discussion for RCS

$\mathbf{V}$ is the principal components extracted from the base task,
$\mathbf{v}_{i}$ is the $i$-th normalized principal component.
$\widetilde{\mathbf{f}}^{d}$ and $\widetilde{\mathbf{F}}^{t}_{k}$ are the
projection of $\mathbf{f}^{d}$ and $\mathbf{F}^{t}_{k}$ onto the $\mathbf{V}$
vector space, formally expressed as the following equation:

$\displaystyle\mathbf{f}^{d}=\sum_{i=1}^{M}\widetilde{\mathbf{f}}^{d}_{i}\mathbf{v}_{i}+\mathbf{R}^{d}$
(13)
$\displaystyle\mathbf{F}^{t}_{k}=\sum_{i=1}^{M}\widetilde{\mathbf{F}}^{t}_{k,i}\mathbf{v}_{i}+\mathbf{R}^{t}_{k}$
(14)

where $M$ is the number of principal components,
$\widetilde{\mathbf{f}}^{d}_{i}$ and $\mathbf{F}^{t}_{k,i}$ are the $i$-th
dimension of $\widetilde{\mathbf{f}}^{d}$ and
$\widetilde{\mathbf{F}}^{t}_{k}$, respectively, and $\mathbf{R}^{d}$ and
$\mathbf{R}^{t}_{k}$ are the components of $\mathbf{f}^{d}$ and
$\mathbf{F}^{t}_{k}$ which are not in the $\mathbf{V}$ vector space.
$\mathbf{R}^{d}$ and $\mathbf{R}^{t}_{k}$ are low-variance components (due to
the nature of principal component analysis) and thus are redundant components
according to discussions in Section 3.4.1.

Each $\mathbf{v}_{i}$ is an orthogonal vector and orthogonal to vectors
outside the $\mathbf{V}$ vector space, e.g., $\mathbf{R}^{d}$ and
$\mathbf{R}^{t}_{k}$. Therefore $\forall i,j,k,\ i\neq j$ The following
equation holds:

$\displaystyle\mathbf{v}_{i}\mathbf{v}^{T}_{j}=0,\
\mathbf{v}_{i}\mathbf{v}^{T}_{i}=1$ (15)
$\displaystyle\mathbf{R}^{d}\mathbf{v}_{i}^{T}=0,\
\mathbf{v}_{i}\mathbf{R}^{t^{T}}_{k}=0$ (16)

Then, we can derive the following equation:

$\displaystyle l_{k}$
$\displaystyle=\frac{\mathbf{f}^{d}\mathbf{F}^{t^{\mathrm{T}}}_{k}}{\|\mathbf{f}^{d}\|\|\mathbf{F}^{t}_{k}\|}$
$\displaystyle=\frac{(\sum_{i=1}^{M}\widetilde{\mathbf{f}}^{d}_{i}\mathbf{v}_{i}+\mathbf{R}^{d})(\sum_{i=1}^{M}\widetilde{\mathbf{F}}^{t}_{k,i}\mathbf{v}_{i}+\mathbf{R}^{t}_{k})^{T}}{\|\mathbf{f}^{d}\|\|\mathbf{F}^{t}_{k}\|}$
$\displaystyle=\frac{\sum_{i=1}^{M}\widetilde{\mathbf{f}}^{d}_{i}\mathbf{v}_{i}\sum_{j=1}^{M}\widetilde{\mathbf{F}}^{t}_{k,j}\mathbf{v}_{j}^{T}+\mathbf{R}^{d}\mathbf{R}^{t^{T}}_{k}}{\|\mathbf{f}^{d}\|\|\mathbf{F}^{t}_{k}\|}$
$\displaystyle+\frac{\sum_{i=1}^{M}\widetilde{\mathbf{f}}^{d}_{i}\mathbf{v}_{i}\mathbf{R}^{t^{T}}_{k}+\sum_{i=1}^{M}\widetilde{\mathbf{F}}^{t}_{k,i}\mathbf{R}^{d}\mathbf{v}_{i}^{T}}{\|\mathbf{f}^{d}\|\|\mathbf{F}^{t}_{k}\|}$
$\displaystyle=\frac{\sum_{i=1}^{M}\widetilde{\mathbf{f}}^{d}_{i}\widetilde{\mathbf{F}}^{t}_{k,i}+\mathbf{R}^{d}\mathbf{R}^{t^{T}}_{k}}{\|\mathbf{f}^{d}\|\|\mathbf{F}^{t}_{k}\|}$
$\displaystyle=\frac{\widetilde{\mathbf{f}}^{d}\widetilde{\mathbf{F}}^{t^{T}}_{k}+\mathbf{R}^{d}\mathbf{R}^{t^{T}}_{k}}{\|\mathbf{f}^{d}\|\|\mathbf{F}^{t}_{k}\|}$

Now we can observe that there are two terms in the numerator of this equation.
$\widetilde{\mathbf{f}}^{d}\widetilde{\mathbf{F}}^{t^{T}}_{k}$ represents
semantic information and should be retained, and
$\mathbf{R}^{d}\mathbf{R}^{t^{T}}_{k}$ is the multiplication of redundant
feature vectors that should be compressed to keep only semantic information.
Thus, the logit of the class $k$ can be calculated in a modified way as
follows to eliminate redundant features:

$\displaystyle\widetilde{l}_{k}=\frac{\widetilde{\mathbf{f}}^{g}\widetilde{\mathbf{F}}^{t^{T}}_{k}}{\|\mathbf{f}^{g}\|\|\mathbf{F}^{t}_{k}\|}.$
(17)

##  8 Extra Ablation Studies

Table 4: Ablation studies on the S2S task of FSCIL3D-XL DR | RCS | 0 | 6 | ${\rm NCAcc}\uparrow$ | ${\rm\Delta}\downarrow$ | ${\rm F}\uparrow$
---|---|---|---|---|---|---
✗ | ✗ | 90.4 | 80.1 | 68.2 | 11.4 | 73.7
78.9 | 69.7 | 68.8 | 11.7 | 69.2
✓ | ✗ | 90.5 | 81.2 | 70.9 | 10.2 | 75.7
79.7 | 69.7 | 70.9 | 12.5 | 70.3
✓ | ✓ | 90.6 | 82.2 | 79.3 | 9.3 | 80.7
80.0 | 70.7 | 77.0 | 11.6 | 73.7

To further demonstrate the effectiveness of our designs, we provide more
ablation results in Table 4. For no Dimensionality Reduction (DR) and no
renormalized cosine similarity (RCS) case (line 1), we use cosine similarity
between $\mathbf{f}^{d}$ and $\mathbf{F}^{t}$ to calculate logits. For with DR
and no renormalized cosine similarity (RCS) case (line 2), we use cosine
similarity between $\widetilde{\mathbf{f}}^{d}$ and
$\widetilde{\mathbf{F}}^{t}$ to calculate logits. For with DR and renormalized
cosine similarity (RCS) case (line 3), we use Eq.17 to calculate logits.

It can be observed that if DR is used without the RCS (line 2), The
improvement in performance is limited. Conversely, the combination of DR and
the RCS (line 3) leads to a significant improvement in performance,
particularly in terms of $\rm NCAcc$. We can thus conclude that, without using
the RCS, the process of dimensionality reduction leads to inappropriate
stretching, which ultimately distorts the semantic information when
eliminating redundant information. This phenomenon verifies the conclusion of
our analysis that redundant information affects the classification.

We can observe a significant improvement in line 3 compared to line 1 (both in
$\rm Acc$ and $\rm NCAcc$). This experimental result provides evidence that
redundancy within PTMs can have a pronounced impact on classification accuracy
and the effectiveness of RFE in mitigating this issue.

##  9 Details of FSCIL3D-XL

Dataset | Task | Name of Classes
---|---|---
ShapeNet | 1 | airplane, ashcan, bag, basket,
bathtub, bed, bench, birdhouse,
bookshelf, bottle, bowl, bus,
cabinet, camera, can, cap,
car, cellular telephone, chair, clock,
computer keyboard, dishwasher,
display, earphone, faucet, file,
guitar, helmet, jar, knife, lamp,
laptop, loudspeaker, mailbox,
microphone, microwave,
motorcycle, mug, piano, pillow,
pistol, pot, printer, remote control,
rifle, rocket, skateboard, sofa,
stove, table, telephone, tower,
train, vessel, washer
ModelNet | 2 | cone, cup, curtain, desk
3 | door, dresser, flower pot, glass box
4 | mantel, monitor, night stand, person
5 | plant, radio, range hood, sink
6 | stairs, stool, tent, toilet
7 | tv stand, vase, wardrobe, xbox
Table 5: Details of S2S task setup. Dataset | Task | Name of Classes
---|---|---
ShapeNet | 1 | airplane, ashcan, bag, basket,
bathtub, bed, bench, birdhouse,
bookshelf, bottle, bowl, bus,
cabinet, camera, can, cap,
car, cellular telephone, chair, clock,
computer keyboard, dishwasher,
display, earphone, faucet, file,
guitar, helmet, jar, knife, lamp,
laptop, loudspeaker, mailbox,
microphone, microwave,
motorcycle, mug, piano, pillow,
pistol, pot, printer, remote control,
rifle, rocket, skateboard, sofa,
stove, table, telephone, tower,
train, vessel, washer
CO3D | 2 | kite, keyboard, apple, plant
3 | toaster, pizza, donut, parkingmeter
4 | toybus, vase, baseballglove, couch
5 | broccoli, hydrant, bicycle, toilet
6 | toytrain, cup, banana, sandwich
7 | book, mouse, hotdog, cellphone
8 | baseballbat, umbrella,
toyplane, wineglass
9 | tv, orange, toytruck, ball
10 | stopsign, hairdryer,
backpack, remote
11 | carrot, frisbee, cake, handbag
12 | suitcase
Table 6: Details of S2R task setup.

###  9.1 S2S Task

We choose ShapeNet [2] as our base dataset and ModelNet40 [36] as the
incremental dataset. Our base task retains all 55 classes from ShapeNet [2],
with 42,001 training and 10,469 test samples. For the incremental tasks, we
choose 24 non-overlapping (with the base task) classes, consisting of 1,339
test instances. Table 5 shows the detailed class divisions.

###  9.2 S2R Task

We use ShapeNet [2] as our base dataset (similar to the S2S task, with 42,001
training and 10,469 test samples) and then choose CO3D [36] as the incremental
dataset. For the incremental tasks, we exclude 9 overlapping classes that
overlap with the base task, resulting in 41 distinct classes designated for
the incremental tasks, consisting of 2,928 test instances. Table 6 shows the
detailed class divisions.

##  10 Visualization and Analysis

Figure 4: Visualization of experimental results.

The visualization results of some classes in the test set are presented in the
first six figures of Figure 4. The first three figures present the
visualizations for Chowdhury et al. [3]’s model (1), ours w/o RFE (2), and
ours (3) in the S2S task. The next three figures similarly display the
visualizations for Chowdhury et al. [3]’s model (4), ours w/o SNC (5), and
ours (6) in the S2R task. The last two figures present the visualization
results of all classes in the S2R test set, with ours w/o SNC on (7) and ours
on (8). These visualizations are generated using t-SNE [30].

###  10.1 Comparison with Chowdhury et al. [3]’s

Comparing our model with Chowdhury et al. [3]’s, three observations can be
concluded: 1) In our model, the intra-class samples appear more compact,
indicating that samples within the same class are closer in the feature space.
In contrast, the visualizations of Chowdhury et al. [3]’s model, using chair
and bed as examples, exhibit a more scattered distribution. The lack of
distinct clustering centers suggests its inability to extract more
generalizable features. 2) In our model, there is a notable amount of blank
space, indicating that other classes have sufficient feature space for
representation. In contrast, the class distributions nearly fill the entire
space in the visualizations of Chowdhury et al. [3]’s model, indicating a
propensity for conflicts with unrepresented classes. 3) Considering the
displayed classes in Figure 4, our model can accurately distinguish between
bed and bus, chair and car, while Chowdhury et al. [3]’s model demonstrates
confusion as indicated by the dashed boxes.

Based on the above observations, we can conclude that the features of our
model surpass those extracted by Chowdhury et al. [3]’s model in aspects of
both generalizability and discriminability.

###  10.2 Effect of RFE

Observing Figure 4 (2), we can notice that the bed and chair classes remain
relatively dispersed compared to our model. This indicates that the model
struggles to extract more generalizable features under the influence of
redundant information, resulting in a more scattered distribution.

###  10.3 Effect of SNC

The distinction between ours w/o SNC (5) and ours (6) is not apparent in
specific classes. Nevertheless, by comparing ours w/o SNC (7) with ours (8) in
all classes, we can observe that noise greatly affects the discriminability of
features (7). The inclusion of supplementary information through SNC
effectively alleviates this issue (8).

##  11 Details of Pre-Processing

Figure 5: Overview of pre-processing

In the pre-processing stage, we render all training samples of the base task
$[\mathbf{x}^{1}_{i}]_{M}$ to generate their depth maps
$[{\rm\mathbf{D}}^{1}_{i}]_{M}$ and then embed them as depth features
${\rm\mathbf{F}^{2D}}\in\mathbb{R}^{MN\times C}$, where $M$ is the number of
training sample in task 1, $N$ is the number of viewpoint and $C$ is the
embedding dimension. We then use the depth features ${\rm\mathbf{F}^{2D}}$ to
calculate principal components $\mathbf{V}\in\mathbb{R}^{P\times C}$ through
SVD, as is shown in Figure 5, where $P$ is the number of principal component.

##  12 Experiments on Chowdhury et al. [3]’s benchmark.

Table 7: Quantitative results on Chowdhury et al. [3]’s ShapeNet2CO3D benchmark. Bold donates the best performance, $joint$ serves solely as an upper reference limit in Chowdhury et al. [3]’s work and is not involved in the comparison. Method |  Acc. in each session $\uparrow$ | ${\rm\Delta}\downarrow$
---|---|---
0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10
$Joint$ Chowdhury et al.’s [3] | 81.0 | 79.5 | 78.3 | 75.2 | 75.1 | 74.8 | 72.3 | 71.3 | 70.0 | 68.8 | 67.3 | 16.9
FACT [46] | 81.4 | 76.0 | 70.3 | 68.1 | 65.8 | 63.5 | 63.0 | 60.1 | 58.2 | 57.5 | 55.9 | 31.3
Chowdhury et al.’s [3] | 82.6 | 77.9 | 73.9 | 72.7 | 67.7 | 66.2 | 65.4 | 63.4 | 60.6 | 58.1 | 57.1 | 30.9
FILP-3D (ours) | 85.4 | 81.3 | 80.1 | 77.9 | 75.1 | 73.8 | 72.3 | 70.6 | 69.3 | 67.1 | 65.6 | 23.2
Table 8: Extra quantitative results on the S2R task of FSCIL3D-XL. For each
set of results, the micro/marco average are presented at the top/bottom
respectively. Bold donates the best performance, $joint$ serves solely as an
upper reference limit in our model and is not involved in the comparison.

Method |  Acc. in each session $\uparrow$ | Evaluation metrics
---|---|---
0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | ${\rm NCAcc}\uparrow$ | ${\rm\Delta}\downarrow$ | ${\rm F}\uparrow$
ULIP [37] | 86.3 | 85.6 | 81.7 | 74.0 | 71.7 | 68.1 | 67.6 | 64.5 | 59.5 | 58.4 | 55.2 | 57.5 | 56.4 | 33.4 | 56.9
85.4 | 82.0 | 78.2 | 70.4 | 66.9 | 63.3 | 61.9 | 58.1 | 54.8 | 52.1 | 49.2 | 50.3 | 57.0 | 41.1 | 53.4
PointCLIP [43] | 87.5 | 83.2 | 79.5 | 74.9 | 75.6 | 74.2 | 71.1 | 71.7 | 67.9 | 66.5 | 64.3 | 66.1 | 54.1 | 24.5 | 59.5
74.2 | 72.1 | 68.8 | 62.3 | 60.1 | 57.9 | 54.5 | 55.1 | 50.9 | 50.8 | 47.5 | 48.7 | 53.2 | 34.4 | 50.9
PointCLIP++ | 88.2 | 85.6 | 84.2 | 81.9 | 80.2 | 78.4 | 75.3 | 72.6 | 71.5 | 69.3 | 67.3 | 68.1 | 59.8 | 22.7 | 63.7
75.1 | 72.6 | 70.4 | 65.4 | 63.3 | 61.0 | 57.3 | 53.9 | 53.6 | 51.4 | 49.6 | 50.3 | 59.7 | 33.0 | 54.6
SimpleCIL-3D (ours) | 89.2 | 86.7 | 83.5 | 81.7 | 79.4 | 79.6 | 78.6 | 70.4 | 72.1 | 71.7 | 70.1 | 71.2 | 49.6 | 20.2 | 58.5
78.9 | 77.3 | 75.9 | 73.7 | 70.1 | 66.5 | 64.4 | 60.9 | 59.4 | 58.5 | 54.5 | 56.3 | 49.0 | 28.6 | 52.4
FILP-3D (ours) | 90.0 | 87.0 | 86.4 | 85.0 | 83.7 | 82.7 | 81.4 | 79.4 | 78.2 | 76.8 | 74.8 | 74.6 | 60.6 | 17.1 | 66.9
79.4 | 75.4 | 75.7 | 72.4 | 70.2 | 68.5 | 65.9 | 63.5 | 62.2 | 59.6 | 57.5 | 57.3 | 59.9 | 27.8 | 58.6

As shown in Table 7, FILP-3D outperforms the other two methods. After the
final task, our accuracy only drops to 65.6%, which is even close to the upper
bound of Chowdhury et al. [3]’s method. However, we note that the results in
this benchmark cannot distinctly indicate whether the performance degradation
is caused by the new class or the old classes. We cannot conclude more
observations based on it.

Unfortunately, Chowdhury et al. [3]’s work does not provide any data
partitions other than ShapeNet2CO3D. Consequently, we are unable to follow
them and conduct further experiments and comparisons, such as incremental
within a single dataset.

##  13 Extra Comparison Experiments

To explain why we choose 2D PTMs as the backbone rather than selecting a 3D
backbone, we compare our approach with ULIP [37] (PointMLP [14] backbone).
Besides, a considerable amount of research attempts to improve CLIP’s
classification ability on point clouds, such as PointCLIP [43]. We compare our
method with PointCLIP in the setting of 3D FSCIL and integrate our proposed
modules onto PointCLIP to validate the effectiveness and generality of our
proposed modules.

Comparing ULIP with our model, it’s evident that ULIP exhibits outstanding
learning capabilities in novel classes. However, the large number of trainable
parameters and severely inadequate data make it extremely challenging to avoid
overfitting during the FSCIL process, ultimately resulting in the highest
dropping rate $\rm\Delta$ and the worst performance, except the $\rm NCAcc$
metric. Consequently, we conclude that the combination of frozen 2D PTMs pre-
trained on massive data along with a simple MLP adapter is more suitable for
3D FSCIL. Conversely, ULIP, aligned only with CLIP on a small amount of data,
is not as suitable for the 3D FSCIL task.

Comparing PointCLIP with our model, we observe that the inter-view adapter
proposed by PointCLIP exhibits similarly inadequate performance due to its
complex structure. Moreover, to make a more fair comparison, we introduce
PointCLIP++ (PointCLIP backbone + RFE + SNC; the contrastive learning cannot
be similarly included due to PointCLIP’s inability to specify rendering
distances). Experimental results demonstrate that PointCLIP++ outperforms
PointCLIP significantly across various metrics, especially in $\rm NCAcc$.
This further validates the generality and effectiveness of our proposed
modules.

Table 9: Quantitative results on the S2S task. For each set of results, the micro/marco average are presented at the top/bottom respectively. Bold donates the best performance, $joint$ serves solely as an upper reference limit in our model and is not involved in the comparison. Method | Pub. Year |  Acc. in each session $\uparrow$ | Evaluation metrics
---|---|---|---
0 | 1 | 2 | 3 | 4 | 5 | 6 | ${\rm NCAcc}\uparrow$ | ${\rm\Delta}\downarrow$ | ${\rm F}\uparrow$
FACT [46] | CVPR’22 | 82.6 | 77.0 | 72.4 | 69.8 | 68.4 | 67.7 | 67.3 | 41.7 | 18.5 | 51.5
48.0 | 44.7 | 42.0 | 39.9 | 38.2 | 37.1 | 36.5 | 34.0 | 23.9 | 35.2
BiDist [45] | CVPR’23 | 89.6 | 87.7 | 86.2 | 84.7 | 83.8 | 83.6 | 82.3 | 35.0 | 8.1 | 49.1
82.5 | 80.3 | 77.3 | 75.2 | 72.5 | 71.2 | 69.3 | 38.0 | 16.0 | 49.1
Chowdhury et al.’s [3] | ECCV’22 | 86.9 | 84.6 | 82.8 | 78.3 | 78.5 | 71.5 | 68.6 | 50.8 | 21.1 | 58.4
73.0 | 62.8 | 65.2 | 62.8 | 60.9 | 57.9 | 55.1 | 45.5 | 24.5 | 49.8
SimpleCIL-3D (ours) | - | 90.4 | 88.0 | 85.6 | 85.0 | 84.1 | 78.4 | 80.1 | 68.2 | 11.4 | 73.7
78.9 | 76.1 | 72.8 | 71.1 | 70.8 | 70.2 | 69.7 | 68.8 | 11.7 | 69.2
FILP-3D (ours) | - | 90.6 | 89.0 | 86.7 | 84.2 | 83.2 | 81.8 | 82.2 | 79.3 | 9.3 | 80.7
80.0 | 76.8 | 74.9 | 72.8 | 71.2 | 70.9 | 70.7 | 77.0 | 11.6 | 73.7
$Joint$ FILP-3D | - | 90.6 | 89.1 | 88.5 | 87.8 | 87.4 | 87.6 | 86.8 | 79.9 | 4.2 | 83.2
80.0 | 78.5 | 79.0 | 77.2 | 77.5 | 77.7 | 77.3 | 79.9 | 3.4 | 78.6
Table 10: Quantitative results on the S2R task. For each set of results, the
micro/marco average are presented at the top/bottom respectively. Bold donates
the best performance, $joint$ serves solely as an upper reference limit in our
model and is not involved in the comparison.

Method |  Acc. in each session $\uparrow$ | Evaluation metrics
---|---|---
0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | ${\rm NCAcc}\uparrow$ | ${\rm\Delta}\downarrow$ | ${\rm F}\uparrow$
FACT [46] | 82.4 | 77.2 | 74.5 | 73.1 | 71.3 | 70.4 | 67.2 | 65.2 | 63.8 | 61.8 | 59.9 | 59.8 | 26.2 | 27.4 | 36.4
48.6 | 41.4 | 39.7 | 36.8 | 35.5 | 33.6 | 31.2 | 29.5 | 28.4 | 27.2 | 25.8 | 25.9 | 30.8 | 46.7 | 28.1
BiDist [45] | 89.4 | 54.0 | 54.7 | 56.4 | 57.0 | 55.9 | 56.3 | 52.9 | 52.3 | 51.7 | 50.8 | 50.1 | 47.2 | 43.9 | 48.6
81.8 | 52.8 | 50.1 | 46.2 | 48.3 | 46.1 | 44.7 | 41.8 | 41.8 | 39.8 | 40.0 | 39.7 | 41.9 | 51.5 | 40.8
Chowdhury et al.’s [3] | 85.2 | 78.6 | 71.0 | 72.0 | 75.2 | 68.8 | 56.1 | 58.5 | 62.9 | 59.1 | 52.2 | 59.4 | 35.3 | 30.3 | 44.3
68.2 | 56.2 | 50.5 | 48.4 | 53.5 | 46.7 | 39.9 | 37.6 | 36.9 | 33.1 | 34.3 | 44.1 | 36.5 | 35.3 | 40.0
SimpleCIL-3D (ours) | 89.2 | 86.7 | 83.5 | 81.7 | 79.4 | 79.6 | 78.6 | 70.4 | 72.1 | 71.7 | 70.1 | 71.2 | 49.6 | 20.2 | 58.5
78.9 | 77.3 | 75.9 | 73.7 | 70.1 | 66.5 | 64.4 | 60.9 | 59.4 | 58.5 | 54.5 | 56.3 | 49.0 | 28.6 | 52.4
FILP-3D (ours) | 90.0 | 87.0 | 86.4 | 85.0 | 83.7 | 82.7 | 81.4 | 79.4 | 78.2 | 76.8 | 74.8 | 74.6 | 60.6 | 17.1 | 66.9
79.4 | 75.4 | 75.7 | 72.4 | 70.2 | 68.5 | 65.9 | 63.5 | 62.2 | 59.6 | 57.5 | 57.3 | 59.9 | 27.8 | 58.6
$Joint$ FILP-3D | 90.0 | 89.2 | 89.0 | 88.4 | 88.1 | 87.7 | 87.2 | 86.9 | 85.9 | 84.6 | 83.1 | 83.1 | 54.9 | 7.7 | 66.1
79.4 | 77.2 | 76.3 | 75.9 | 75.5 | 73.5 | 73.3 | 73.4 | 70.2 | 69.3 | 68.2 | 68.2 | 60.0 | 14.1 | 63.8

##  14 Effect of our proposed metrics

###  14.1 S2S task

By observing the experiment results presented in Table 9, we notice a
substantial decline in Chowdhury et al.’s [3] performance in terms of $\rm
MAcc$ when learning incrementally, coupled with poor performance in the $\rm
NCAcc$ metric. This signifies their inability to effectively acquire new
knowledge. Solely relying on micro-Acc. and dropping rate $\Delta$, we are
unable to identify the aforementioned issues. Similarly, without $\rm NCAcc$,
Bidist’s inability to effectively learn new classes also can not be revealed.

###  14.2 S2R task

By observing the experiment results presented in Table 10, we notice a
catastrophic decline in FACT [46]’s performance in terms of $\rm MAcc$ when
learning real-scanned data. This may be because The feature space trained by
the manifold-mixup [31] method is relatively fragile in the aspect of
structure, thus it is easy to overfit the noise information during the
incremental process, especially for real-scanned data. Solely relying on
metrics proposed by Chowdhury et al., we are unable to identify the
aforementioned issues and then perform the analysis. Similarly, without $\rm
NCAcc$, Chowdhury et al.’s terrible performance in learning real scanned new
classes also can not be noticed.

Consequently, our proposed metrics, $\rm MAcc$ and $\rm NCAcc$, hold
significant importance in comprehensively evaluating and analyzing the 3D
FSCIL model. Moreover, ${\rm F_{FSCIL}}$, serving as a metric that balances
learning new classes and not forgetting old ones, can objectively evaluate the
performance of a continual learning model as much as possible.