# DynED: Dynamic Ensemble Diversification in Data Stream Classification

Soheil Abadifard<EMAIL_ADDRESS>0000-0002-2980-4251 Bilkent
UniversityAnkaraTurkey ,  Sepehr Bakhshi<EMAIL_ADDRESS>0000-0003-2292-6130 Bilkent UniversityAnkaraTurkey ,  Sanaz Gheibuni
<EMAIL_ADDRESS>0009-0005-4349-7568 Bilkent
UniversityAnkaraTurkey and  Fazli Can<EMAIL_ADDRESS>0000-0003-0016-4278 Bilkent UniversityAnkaraTurkey

(2023)

###### Abstract.

Ensemble methods are commonly used in classification due to their remarkable
performance. Achieving high accuracy in a data stream environment is a
challenging task considering disruptive changes in the data distribution, also
known as concept drift. A greater diversity of ensemble components is known to
enhance prediction accuracy in such settings. Despite the diversity of
components within an ensemble, not all contribute as expected to its overall
performance. This necessitates a method for selecting components that exhibit
high performance and diversity. We present a novel ensemble construction and
maintenance approach based on MMR (Maximal Marginal Relevance) that
dynamically combines the diversity and prediction accuracy of components
during the process of structuring an ensemble. The experimental results on
both four real and 11 synthetic datasets demonstrate that the proposed
approach (DynED) provides a higher average mean accuracy compared to the five
state-of-the-art baselines.

Data stream classification, Concept drift, Diversity adjustment, Ensemble
learning, Ensemble pruning, Maximal Marginal Relevance

††journalyear: 2023††copyright: cc††conference: Proceedings of the 32nd ACM
International Conference on Information and Knowledge Management; October
21–25, 2023; Birmingham, United Kingdom††booktitle: Proceedings of the 32nd
ACM International Conference on Information and Knowledge Management (CIKM
’23), October 21–25, 2023, Birmingham, United Kingdom††doi:
10.1145/3583780.3615266††isbn: 979-8-4007-0124-5/23/10††ccs: Information
systems Data stream mining††ccs: Computing methodologies Machine learning

##  1\. INTRODUCTION

The ability to extract meaningful insights out of an endless flow of incoming
data is crucial nowadays, and data stream classification methods have made
this task more feasible. As more organizations move towards a more technology-
driven environment, there has been an alarming increase in generated large
amounts of real-time information coming from different sources, such as social
media platforms, sensor-based systems, healthcare, etc. Data stream
classification techniques offer rapid processing capabilities. This allows
analysts to harness valuable insights at lightning speed. As a result,
decision-making processes can be carried out promptly to minimize risks while
improving performance.

Dealing with the dynamic nature of data streams is one of the main challenges
with data stream classification due to changes in the data distribution. This
phenomenon is known as concept drift (Tsymbal, 2004; Wang et al., 2011; Yuan
et al., 2022; Gulcan and Can, 2023; Bakhshi et al., 2023), and necessitates a
learning paradigm capable of handling it. Ensemble approaches combine multiple
possibly weak classifiers to improve model performance, robustness, resilience
distributivity, and redundancy (Vardi, 2020). These approaches have the
ability to adapt to the changes in data distributions while maintaining high
levels of accuracy (Wankhade et al., 2020; Wares et al., 2019; Bonab and Can,
2018).

The discrepancies in predictions provided by individual ensemble components
are referred to as diversity. In the ensemble learning setting, maintaining
diversity among the individual ensemble components is one of the main
challenges. Exposure of the data stream environment to various concept drifts
and the fast arrival rate of data items makes this challenge even harder.
High-diversity ensembles demonstrate better performance in the presence of
concept drift (Minku et al., 2010; Minku, 2011) even with fewer components
(Bonab and Can, 2019). The performance of ensemble components can decrease
drastically when concept drift occurs. To maintain high accuracy, it is
critical to detect the concept drift and update or replace the impacted
ensemble components (Minku, 2011).

Several approaches to handle the difficulties mentioned above have been
proposed. Leveraging Bagging (LevBag) (Bifet et al., 2010a) combines bagging’s
simplicity with additional randomization of component inputs and outputs. This
randomization can help individual components in an ensemble make different
predictions. Concerning diversity, the Adaptive Random Forest (ARF) (Gomes et
al., 2017) uses a local randomization strategy to retain diversity among
ensemble components. This method uses different random selections of features
for each component in the ensemble, encouraging diversity among individual
components.

Compared to the prior methods, the Streaming Random Patches (SRP) (Gomes et
al., 2019) combines random subspaces and online bagging to achieve competitive
prediction performance. As a result, they indirectly increase the diversity of
the ensemble components. Lastly, Kappa Updated Ensemble (KUE) (Cano and
Krawczyk, 2020) combines online and block-based methods and uses the Kappa
statistic to weigh and select classifiers dynamically. To increase the
diversity, each base learner is trained with a different subset of features.
Additionally, new instances are added to each base learner with a specific
probability based on the Poisson distribution.

The challenges of data stream classification and the efforts of the previous
solutions to increase diversity within their ensembles have motivated further
investigation. The aim is to determine how to add more variety and prune the
redundant or ineffective components (Shen and Liu, 2021; Elbasi et al., 2021;
Woźniak et al., 2023) in an ensemble to handle concept drift better, and
maintain high accuracy.

The following are the main contributions of this research. We

  * •

Propose a novel ensemble construction and maintenance approach, called DynED
(Dynamic Ensemble Diversification), based on the principles of the Maximal
Marginal Relevance (MMR) concept;

  * •

Adjust the diversity parameter dynamically to cope with the data stream to
have high diversity in case of severe drifts;

  * •

Experiment with 15 datasets with varying drift types and compare our results
with those of the state-of-the-art methods.

##  2\. PROPOSED APPROACH

Figure 1. Ensemble construction and maintenance using DynED. Stage 1:
Predicting, majority voting, and training. Stage 2: Detecting drifts, adding
new components, and updating the diversity parameter. Stage 3: Selecting new
components.

###  2.1. Using MMR in Data Stream Classification

Maximal Marginal Relevance (MMR) (Carbonell and Goldstein, 1998) is a
diversity-based ranking method that minimizes redundancy while maintaining the
relevance of a query in a document set. It is useful for text-document
summarization, response extraction (Mao et al., 2020; Adams et al., 2022), and
document re-ranking (Carbonell and Goldstein, 1998). Formally, the MMR method
follows Eq. 1 to rank the documents:

(1)
$\footnotesize\text{MMR}=\operatorname*{argmax}_{D_{i}\in\texttt{R\textbackslash{}S}}[\lambda\times\text{sim}(D_{i},q)-(1-\lambda)\times\max_{D_{j}\in
S}\text{sim}(D_{i},D_{j})]$

In Eq.1, $D_{i}$ represents a document, $R$ is a ranked list of the documents,
$S$ is a set of the selected documents, $\lambda$ is a parameter that balances
accuracy and redundancy, and $sim(D_{i},q)$ measures the relevance between
document $D_{i}$ and query $q$. When $\lambda=1$, MMR calculates the
relevance-ranked list; when $\lambda=0$, it calculates a ranking that
maximizes diversity among the documents in $R$. MMR optimizes a linear
combination of relevance and diversity criteria for values of $\lambda$
between 0 and 1.

Table 1. Symbols & default hyperparameter values of DynED Symbol | Meaning | Default value
---|---|---
$\xi_{slc}$ | Set of selected components | —
$I$ | Intensity of changes in the accuracy | —
$\theta$ | Threshold for the count of processed samples | 100
$\lambda$ | Controlling parameter of similarity and accuracy | 0.6
$\Delta_{\lambda}$ |  Variation in $\lambda$ parameter | 0.1
$win_{s}$ | Window size | 500
$init_{c}$ | Initial components count | 5
$init_{s}$ | Initial training sample size per component | 50
$pool_{s}$ | Maximum size of the component pool | 500
$add_{s}$ | # of components to add in each step | 5
$cls_{s}$ | # of components to select from each cluster | 10
$slc_{s}$ | # of active components for classification | 10
$error_{s}$ | Prediction list size to calculate similarity on | 50
$cls_{slc}$ | Set of selected components based on clustering | —
$cls_{rst}$ | Set of not selected components based on clustering | —

It is necessary to make changes in its definition in order to adapt the MMR
method for selecting and ranking ensemble components. In terms of ensemble
components, the first part of Eq. 1, which calculates the relevance of $D_{i}$
to query $q$, is replaced with the accuracy of each component. It is
represented as ”$\lambda\times\text{acc}(CS_{i},Ins)$”, where
$\xi=\\{CS_{1},CS_{2},...,CS_{m}\\}$ is the set of ensemble classifiers,
$CS_{i}$ represents each component, and $Ins$ are the previously seen
instances of the data stream.

The second part of the Eq. 1 determines a pairwise similarity between the
documents. However, evaluating the diversity of ensemble components is
difficult since there is no commonly agreed-upon formal definition of
diversity. Several methods are available for determining the pairwise
diversity of classifiers in terms of correct/incorrect (oracle) outputs, such
as Correlation Coefficient P (CP), double-fault measure (DF), Disagreement
Measure (DM), and $Q$ statistic (Tsymbal et al., 2005; Kuncheva and Whitaker,
2001, 2003). To adapt the second part of the Eq. 1 to the context of component
diversity, we replace it with the DF diversity measuring method (the procedure
of diversity measure selection is explained in section 3.2). Therefore, the
second part of the formal equation turns into
”$(1-\lambda)\times\max_{CS_{j}\in S}sim(CS_{i},CS_{j})$”. The final version
of the MMR method for our task is presented in Eq. 2:

(2)
$\footnotesize\centering\text{MMR}=\operatorname*{argmax}_{CS_{i}\in\texttt{R\textbackslash{}S}}[\lambda\times\text{acc}(CS_{i},Ins)-(1-\lambda)\times\max_{CS_{j}\in
S}sim(CS_{i},CS_{j})]\@add@centering$

The MMR method utilizes a measure of similarity, which can be derived from the
complement of a diversity measure.

###  2.2. Dynamic Ensemble Diversification: DynED

The working principles of our approach: DynED aims to dynamically ensure high
accuracy by increasing diversity in the presence of concept drift and
otherwise by decreasing it to reduce exposure to underperforming ensemble
components. The pseudo-code for DynED is provided in Algorithms 1 and 2.

Algorithm 1 DynED: Main flow.

0: $D$: data stream, $W$: sliding window, $DD$: drift detector, $init_{s}$,
$init_{c}$, $win_{s}$, $pool_{s}$, $error_{s}$, $add_{s}$, $\Delta_{\lambda}$

0: $\hat{y_{k}}$: prediction of each sample at time $t$

1: $\xi,\xi_{slc}\leftarrow\text{initialize }init_{c}\text{ components and
train on }init_{s}\text{ samples}$

2: while $D$ has more samples do

3: $\hat{y_{k}}\leftarrow\text{use selected components }\xi_{slc}\text{to
predict using majority voting}$

4: $W\leftarrow$ add sample to the sliding window

5: $DD\leftarrow\text{update drift detector using correct/incorrect
predictions}$

6: train $\xi_{slc}$ on sample

7: if $DD$ detects drift then

8: $\xi\leftarrow$ generate and train $add_{s}$ new component on data in $W$,
add to the pool

9: end if

10: if $\text{passed samples count}\geq\theta$ or new component is added then

11: $I\leftarrow$ calculate the intensity of accuracy change to update the
$\lambda$ parameter dynamically

12: if $I\geq 0$ then

13: increase $\lambda$ by $\Delta_{\lambda}$ until it reaches 1

14: else

15: decrease $\lambda$ by $\Delta_{\lambda}$ until it reaches 0

16: end if

17: $\xi_{slc}$, $\xi\leftarrow$ update $\xi_{slc}$ and $\xi$ using Algorithm
2

18: end if

19: end while

Stage 1: The primary process includes predicting new samples and training
selected components, which is outlined in Algorithm 1. As this method operates
online, line 3 of Algorithm 1 uses selected components to predict new samples
using Majority Voting (Leon et al., 2017; Dogan and Birant, 2019).
Stage 2: The Drift Detector, ADWIN (Bifet and Gavalda, 2007), is updated using
the correct/incorrect predictions. If drift is detected, a new classifier is
generated and trained on the last seen data available in the sliding window
and then added to the reserved component pool $\xi$ (lines 5-9 of Algorithm
1). Suppose a new component is added or the processed sample count passes the
$\theta$ threshold. In that case, the algorithm updates the $\lambda$
parameter to reflect the proper value of diversity based on the intensity of
accuracy changes (lines 10-16). Then Algorithm 2 is called to update
$\xi_{slc}$ and $\xi$. The intensity of changes in the accuracy is computed
using the formula ”$acc(t)-acc(t-i)\mathbin{/}i$” where $acc(t)$ denotes the
accuracy of the ensemble model at the tth sample and $i$ is equivalent to
$\theta$ in DynED.

Algorithm 2 DynED: Component selection.

0: $W$: data window, $\xi$, $\xi_{slc}$, $slc_{s}$, $\lambda$, $error_{s}$,
$pool_{s}$, $cls_{s}$

0: $\xi$, $\xi_{slc}$

1: $P\leftarrow\text{ combine }\xi\text{ and }\xi_{slc}$ sorted by accuracy

2: while $len(p)\geq pool_{s}$ do

3: remove the last component

4: end while

5: $PE\leftarrow$ obtain prediction error for all components recalling data
from $W$

6: $cls\leftarrow$ cluster components based on $PE$ ”(two clusters)”

7: $cls_{slc}\leftarrow$ select $cls_{s}$ components from each cluster based
on the accuracy

8: $cls_{rst}\leftarrow$ store rest of components

9: $\xi_{slc}$, $\xi_{rst}\leftarrow$ apply Eq. 2 on $cls_{slc}$ to obtain a
new set of components to continue prediction

10: $\xi\leftarrow cls_{rst}+\xi_{rst}$

11: return $\xi$, $\xi_{slc}$

Stage 3: Algorithm 2 selects a diverse set of components using an adapted MMR
method presented in Eq. 2. The algorithm combines previously selected
components with those in the reserve pool and maintains a fixed component
count by sorting them based on accuracy and removing poorly performing
components. In line 5 of Algorithm 2, prediction errors for all components on
previous samples held in sliding window are obtained. In the following line,
components are clustered using the K-means clustering algorithm (Hartigan and
Wong, 1979) into two groups based on these prediction errors to apply the
selection method effectively. After clustering, $cls_{s}$ high-performance
components are selected from each cluster, resulting in a total of $2\times
cls_{s}$ components out of $N$ where $N\leq pool_{s}$. In line 9 of Algorithm
2, an adapted MMR method is applied to perform the final selection step. This
algorithm outputs a new set of selected components to predict new incoming
samples of the stream actively and updates the reserved components pool as a
result. The way an ensemble structure is constructed and maintained by DynED
is illustrated in Figure 1.

(a) Poker.

(b) Agrawal-4567.

(c) Critical distance diagram.

Figure 2. (a, b): Prequential temporal accuracy results of DynED, and (c):
Critical distance diagram, CD= 1.946.

Table 2. Characteristics of the datasets, average interleaved-test-then-train
accuracy, and the rankings of the methods for each dataset. DT: Drift Type,
—X—: No. of features, —y—: No. of classes, —$\mathcal{D}$—: No. of Samples

| Name | DT | —X— | —y— | —$\mathcal{D}$— | DynED | KUE | SRP | ARF | SAM-kNN | LevBag
---|---|---|---|---|---|---|---|---|---|---|---
Real | Electricity (Harries et al., 1999) | U | 6 | 2 | 45,312 | 87.88 | 76.77 | 87.85 | 85.29 | 79.16 | 87.28
Poker (Dua and Graff, 2017) | U | 10 | 10 | 829,201 | 96.96 | 81.68 | 89.74 | 80.65 | 82.41 | 85.99
Cover Types | U | 54 | 7 | 581,012 | 94.66 | 84.30 | 94.07 | 90.06 | 93.87 | 90.94
Weather (Elwell and Polikar, 2011) | U | 8 | 2 | 18,152 | 78.17 | 72.74 | 78.79 | 78.28 | 78.24 | 77.87
Synthetic | Interchanging RBF (Losing et al., 2016a) | A | 2 | 15 | 200,000 | 93.70 | 71.77 | 97.44 | 97.15 | 94.19 | 95.34
LED-drift | A | 7 | 10 | 100,000 | 99.99 | 95.33 | 99.32 | 96.65 | 99.94 | 99.96
MG2C2D (Dyer et al., 2013) | I & G | 2 | 2 | 200,000 | 92.68 | 93.17 | 94.53 | 94.49 | 94.52 | 94.54
Moving Squares (Losing et al., 2016a) | I | 2 | 4 | 200,000 | 85.02 | 30.05 | 86.95 | 56.95 | 97.34 | 87.77
Rotating Hyperplane (Losing et al., 2016a) | I | 10 | 2 | 200,000 | 84.69 | 83.36 | 85.79 | 81.53 | 86.25 | 86.05
SEA-Abrupt-012 | A & R | 3 | 2 | 100,000 | 90.61 | 86.80 | 90.16 | 90.20 | 88.61 | 89.90
SEA-Abrupt-123 | A & R | 3 | 2 | 100,000 | 90.10 | 86.60 | 89.79 | 89.61 | 88.43 | 89.60
SEA-Gradual-012 | G & R | 3 | 2 | 100,000 | 80.33 | 77.28 | 79.83 | 79.77 | 77.20 | 79.76
SEA-Gradual-123 | G & R | 3 | 2 | 100,000 | 80.30 | 77.56 | 79.82 | 79.57 | 76.93 | 79.60
Agrawal-4567 | G & I | 9 | 2 | 100,000 | 89.95 | 75.41 | 84.29 | 73.64 | 73.91 | 74.77
| Mixed-12 | A & R | 4 | 2 | 100,000 | 98.47 | 89.45 | 93.44 | 97.02 | 98.20 | 94.71
| Average Mean | — | — | — | — | 89.57 | 78.82 | 88.78 | 84.72 | 87.28 | 87.60
| Rank | — | — | — | — | 2.20 | 5.46 | 2.44 | 4.00 | 3.73 | 3.20

###  2.3. Time Complexity Analysis of Component Selection

The time complexity of Algorithm 2, which employs the reformulated MMR method,
is as follows. In line 5 of Algorithm 2, the prediction errors of all
classifier components are obtained, the time complexity is $\mathcal{O}(p)$,
where $p$ represents the classifier component count and $p\leq pool_{s}$.
Lines 6, 7, and 8 of Algorithm 2 involve clustering the classifier components
based on their prediction errors and selecting $cls_{s}$ from each cluster,
where $cls_{s}<p$. The time complexity of these operations is
$\mathcal{O}(k\times p\times itr)$, where $itr$ is the number of iterations in
the clustering process, and $k$ is the number of clusters. $itr$ and $k$ are
considered as constant as they are not hyperparameters for DynED. In line 10,
which applies the reformulated MMR method, the time complexity can be broken
down as follows: calculating the pairwise similarity of classifier components
using any diversity measure has a time complexity of $\mathcal{O}(n^{2})$,
where $n$ represents the number of classifier components extracted by the
clustering step ($2\times cls_{s}$). Applying the reformulated MMR method
itself has a time complexity of $\mathcal{O}(n\times j)$, where $j=slc_{s}$.
Therefore, the overall time complexity of Algorithm 2 is
$\mathcal{O}(p+k\times p\times itr+n\times j+n^{2})$. The dominant term in
this time complexity analysis is $n^{2}$. Hence, the algorithm’s time
complexity can be approximated as $\mathcal{O}(n^{2})$.

##  3\. EXPERIMENTAL EVALUATION

###  3.1. Datasets

To assess the performance of our model, we conduct experiments using 15
datasets (Four real and 11 synthetic datasets) and compare them to the
baseline models. The datasets cover a wide range of concept drift scenarios.
Our experiments include all four types of drift: Gradual (G), Incremental (I),
Abrupt (A), Recurring (R), and (U) stands for Unknown drift type. The
synthetic datasets based on LED, SEA, Agrawal, and Mixed generators are
created using the scikit-multiflow library (Montiel et al., 2018) and MOA
framework (Bifet et al., 2010b). The LED dataset has seven drifting features
without noise. The Agrawal dataset uses four classification functions, and the
SEA dataset uses three classification functions to synthesize drift. The
description of the datasets is shown in Table 2.2.

###  3.2. Setup

In our study, we evaluate four diversity measures: Correlation Coefficient P
(CP), Double-Fault measure (DF), Disagreement Measure (DM), and $Q$ statistic.
We apply each measure to Eq. 2 across all datasets to determine the most
suitable diversity measure for DynED. Our results show that the DF has the
highest average mean accuracy compared to that of CP, DM, and $Q$-statistic
with respective average mean accuracies of 89.57, 88.14, 89.43, and 89.38.
Therefore, we choose DF as the diversity measure in DynED.

We evaluate the performance of DynED against five state-of-the-art baselines
including LevBag (Bifet et al., 2010a), SAM-kNN (Losing et al., 2016b), ARF
(Gomes et al., 2017), and SRP (Gomes et al., 2019). These baselines are
assessed using the Massive Online Analysis (MOA) (Bifet et al., 2010b)
framework with default hyperparameters. For KUE (Cano and Krawczyk, 2020), we
use the source code available on their GitHub for evaluation
111https://github.com/canoalberto/imbalanced-streams. DynED is implemented in
Python 3.8 using the scikit-multiflow (Montiel et al., 2018) library, with a
Hoeffding Tree as the base classifier, and split-confidence set to 9e-1 and
grace-period set to 50. All baseline models were evaluated using the
Interleaved-test-then-train approach. The codes and datasets for experiments
are publicly available, and all experiments and results are reproducible
222https://github.com/soheilabadifard/DynED.

The selection of appropriate hyperparameters is a critical aspect of all
machine learning methods, including DynED. After conducting tests with various
hyperparameters based on the grid search method, we determined the selected
values presented in Table 1. These values serve as the default hyperparameters
in DynED and are not tailored to any specific dataset. It should be noted that
the first 250 samples of each dataset are used as a warm-up, and they are not
involved in the accuracy calculations and final results presented in Table
2.2.

###  3.3. Results and Discussion

The overall accuracy of each method applied to each dataset is presented in
Table 2.2, with the highest scores emphasized in bold. A comparative analysis
reveals that DynED outperforms the baselines in 10 out of 15 datasets,
particularly in three out of four real and seven out of 11 synthetic datasets.
Furthermore, when the average mean accuracies are ranked in descending order,
DynED emerges as the top performer with an average rank of 2.20. A closer
examination of Table 2.2 and Figure 2.a and Figure 2.b indicates that DynED
provides robustness in the case of gradual and recurrent drift types,
outperforming the baselines in nearly all datasets that exhibit these drift
types. However, DynED’s performance declines in the presence of incremental
drift, struggling to maintain high accuracy levels throughout the stream.
Nonetheless, when confronted with abrupt drifts, DynED effectively captures
and addresses the drift by employing Eq. 2 to increase diversity among
components, resulting in enhanced performance as evidenced by the plots in
Figure 2. Overall, the results suggest that DynED is a promising method for
the online classification of data streams. To assess the statistical
significance of the employed methods, the Friedman Test is utilized in
conjunction with the Nemenyi post-hoc analysis (Demšar, 2006). We calculate
the Critical Distance as CD= 1.946. In our experiment, $\alpha=0.05$. The
statistical test analysis, as presented in Figure 2.c, reveals that DynED
statistically significantly outperforms KUE and achieves a better ranking
among the other baseline methods.

##  4\. CONCLUSION AND FUTURE WORK

This paper presents DynED, a novel ensemble construction and maintenance
method that combines diversity and prediction accuracy for data stream
classification tasks. It aims to increase the diversity among components in
the presence of concept drift in a data stream in order to handle drifts
better. The results show that DynED has higher average mean accuracy compared
to the baseline models. In real-world scenarios, data stream environments
often face the issue of label scarcity. As a part of future work, we aim to
enhance our study for the semi-supervised classification of data streams.

##  5\. ACKNOWLEDGMENTS

This study is partially supported by TÜBİTAK grant no. 122E271.

## References

  * (1)
  * Adams et al. (2022) David Adams, Gandharv Suri, and Yllias Chali. 2022\.  Combining state-of-the-art models with maximal marginal relevance for few-shot and zero-shot multi-document summarization.  _arXiv_ (2022).  https://doi.org/10.48550/arXiv.2211.10808
  * Bakhshi et al. (2023) Sepehr Bakhshi, Pouya Ghahramanian, Hamed Bonab, and Fazli Can. 2023.  A Broad ensemble learning system for drifting stream classification.  _IEEE Access_ (2023), 1–1.  https://doi.org/10.1109/ACCESS.2023.3306957
  * Bifet and Gavalda (2007) Albert Bifet and Ricard Gavalda. 2007.  Learning from time-changing data with adaptive windowing. In _Proceedings of the 2007 SIAM International Conference on Data Mining (SDM)_. SIAM, 443–448.  https://doi.org/10.1137/1.9781611972771.42
  * Bifet et al. (2010b) Albert Bifet, Geoff Holmes, Richard Kirkby, and Bernhard Pfahringer. 2010b.  MOA: Massive online analysis.  _The Journal of Machine Learning Research_ 11 (2010), 1601–1604.  https://doi.org/10.5555/1756006.1859903
  * Bifet et al. (2010a) Albert Bifet, Geoff Holmes, and Bernhard Pfahringer. 2010a.  Leveraging bagging for evolving data streams. In _Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2010, Barcelona, Spain, September 20-24, 2010, Proceedings, Part I 21_. Springer, 135–150.  https://doi.org/10.1007/978-3-642-15880-3_15
  * Bonab and Can (2019) Hamed Bonab and Fazli Can. 2019.  Less Is More: A Comprehensive framework for the number of components of ensemble classifiers.  _IEEE Transactions on Neural Networks and Learning Systems_ 30, 9 (2019), 2735–2745.  https://doi.org/10.1109/TNNLS.2018.2886341
  * Bonab and Can (2018) Hamed R Bonab and Fazli Can. 2018.  GOOWE: Geometrically optimum and online-weighted ensemble classifier for evolving data streams.  _ACM Transactions on Knowledge Discovery from Data (TKDD)_ 12, 2 (2018), 1–33.  https://doi.org/10.1145/3139240
  * Cano and Krawczyk (2020) Alberto Cano and Bartosz Krawczyk. 2020.  Kappa updated ensemble for drifting data stream mining.  _Machine Learning_ 109 (2020), 175–218.  https://doi.org/10.1007/s10994-019-05840-z
  * Carbonell and Goldstein (1998) Jaime Carbonell and Jade Goldstein. 1998.  The use of MMR, diversity-based reranking for reordering documents and producing summaries. In _Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval_. 335–336.  https://doi.org/10.1145/290941.291025
  * Demšar (2006) Janez Demšar. 2006\.  Statistical comparisons of classifiers over multiple data sets.  _The Journal of Machine Learning Research_ 7 (2006), 1–30.
  * Dogan and Birant (2019) Alican Dogan and Derya Birant. 2019.  A Weighted majority voting ensemble approach for classification. In _2019 4th International Conference on Computer Science and Engineering (UBMK)_. 1–6.  https://doi.org/10.1109/UBMK.2019.8907028
  * Dua and Graff (2017) Dheeru Dua and Casey Graff. 2017.  UCI Machine Learning Repository.  http://archive.ics.uci.edu/ml
  * Dyer et al. (2013) Karl B Dyer, Robert Capo, and Robi Polikar. 2013.  Compose: A Semisupervised learning framework for initially labeled nonstationary streaming data.  _IEEE Transactions on Neural Networks and Learning Systems_ 25, 1 (2013), 12–26.  https://doi.org/10.1109/TNNLS.2013.2277712
  * Elbasi et al. (2021) Sanem Elbasi, Alican Büyükçakır, Hamed Bonab, and Fazli Can. 2021.  On-the-Fly ensemble pruning in evolving data streams.  https://doi.org/10.48550/arXiv.2109.07611 arXiv:2109.07611 [cs.LG]
  * Elwell and Polikar (2011) Ryan Elwell and Robi Polikar. 2011.  Incremental learning of concept drift in nonstationary environments.  _IEEE Transactions on Neural Networks_ 22, 10 (2011), 1517–1531.  https://doi.org/10.1109/TNN.2011.2160459
  * Gomes et al. (2017) Heitor M Gomes, Albert Bifet, Jesse Read, Jean Paul Barddal, Fabrício Enembreck, Bernhard Pfharinger, Geoff Holmes, and Talel Abdessalem. 2017.  Adaptive random forests for evolving data stream classification.  _Machine Learning_ 106 (2017), 1469–1495.  https://doi.org/10.1007/s10994-017-5642-8
  * Gomes et al. (2019) Heitor Murilo Gomes, Jesse Read, and Albert Bifet. 2019\.  Streaming random patches for evolving data stream classification. In _2019 IEEE International Conference on Data Mining (ICDM)_. IEEE, 240–249.  https://doi.org/10.1109/ICDM.2019.00034
  * Gulcan and Can (2023) Ege Berkay Gulcan and Fazli Can. 2023.  Unsupervised concept drift detection for multi-label data streams.  _Artificial Intelligence Review_ 56, 3 (2023), 2401–2434.  https://doi.org/10.1007/s10462-022-10232-2
  * Harries et al. (1999) Michael Harries, New South Wales, et al. 1999\.  Splice-2 comparative evaluation: Electricity pricing.  (1999).
  * Hartigan and Wong (1979) John A Hartigan and Manchek A Wong. 1979.  Algorithm AS 136: A k-means clustering algorithm.  _Journal of the Royal Statistical Society Series C (Applied Statistics)_ 28, 1 (1979), 100–108.  https://doi.org/10.2307/2346830
  * Kuncheva and Whitaker (2001) Ludmila I Kuncheva and Chris J Whitaker. 2001.  Ten measures of diversity in classifier ensembles: Limits for two classifiers. In _A DERA/IEE Workshop on Intelligent Sensor Processing (Ref. No. 2001/050)_. IET, 10–1.  https://doi.org/c10.1049/ic:20010105
  * Kuncheva and Whitaker (2003) Ludmila I Kuncheva and Christopher J Whitaker. 2003.  Measures of diversity in classifier ensembles and their relationship with the ensemble accuracy.  _Machine learning_ 51, 2 (2003), 181.  https://doi.org/10.1023/A:1022859003006
  * Leon et al. (2017) Florin Leon, Sabina-Adriana Floria, and Costin Bădică. 2017.  Evaluating the effect of voting methods on ensemble-based classification. In _2017 IEEE International Conference on INnovations in Intelligent SysTems and Applications (INISTA)_. 1–6.  https://doi.org/10.1109/INISTA.2017.8001122
  * Losing et al. (2016a) Viktor Losing, Barbara Hammer, and Heiko Wersing. 2016a.  KNN Classifier with self adjusting memory for heterogeneous concept drift. In _2016 IEEE 16th International Conference on Data Mining (ICDM)_. 291–300.  https://doi.org/10.1109/ICDM.2016.0040
  * Losing et al. (2016b) Viktor Losing, Barbara Hammer, and Heiko Wersing. 2016b.  KNN classifier with self adjusting memory for heterogeneous concept drift. In _2016 IEEE 16th International Conference on Data Mining (ICDM)_. IEEE, 291–300.  https://doi.org/10.1109/ICDM.2016.0040
  * Mao et al. (2020) Yuning Mao, Yanru Qu, Yiqing Xie, Xiang Ren, and Jiawei Han. 2020.  Multi-document summarization with maximal marginal relevance-guided reinforcement learning. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_. Association for Computational Linguistics, Online, 1737–1751.  https://doi.org/10.18653/v1/2020.emnlp-main.136
  * Minku (2011) Leandro Lei Minku. 2011\.  _Online ensemble learning in the presence of concept drift_.  Ph. D. Dissertation. University of Birmingham.
  * Minku et al. (2010) Leandro L. Minku, Allan P. White, and Xin Yao. 2010.  The Impact of diversity on online ensemble learning in the presence of concept Drift.  _IEEE Transactions on Knowledge and Data Engineering_ 22, 5 (2010), 730–742.  https://doi.org/10.1109/TKDE.2009.156
  * Montiel et al. (2018) Jacob Montiel, Jesse Read, Albert Bifet, and Talel Abdessalem. 2018.  Scikit-multiflow: A multi-output streaming framework.  _The Journal of Machine Learning Research_ 19, 1 (2018), 2915–2914.
  * Shen and Liu (2021) Zixiong Shen and Xingcheng Liu. 2021.  A New ensemble pruning method based on margin and diversity. In _Mobile Multimedia Communications_ , Jinbo Xiong, Shaoen Wu, Changgen Peng, and Youliang Tian (Eds.). Springer International Publishing, Cham, 689–701.  https://doi.org/10.1007/978-3-030-89814-4_50
  * Tsymbal (2004) Alexey Tsymbal. 2004\.  The problem of concept drift: Definitions and related work.  _Computer Science Department, Trinity College Dublin_ 106, 2 (2004), 58\.  https://api.semanticscholar.org/CorpusID:8335940
  * Tsymbal et al. (2005) Alexey Tsymbal, Mykola Pechenizkiy, and Pádraig Cunningham. 2005\.  Diversity in search strategies for ensemble feature selection.  _Information Fusion_ 6, 1 (2005), 83–98.  https://doi.org/10.1016/j.inffus.2004.04.003
  * Vardi (2020) Moshe Y Vardi. 2020\.  Efficiency vs. resilience: What COVID-19 teaches computing.  , 9–9 pages.  https://doi.org/10.1145/3388890
  * Wang et al. (2011) Shenghui Wang, Stefan Schlobach, and Michel Klein. 2011\.  Concept drift and how to identify it.  _Journal of Web Semantics_ 9, 3 (2011), 247–265.  https://doi.org/10.1016/j.websem.2011.05.003 Semantic Web Dynamics Semantic Web Challenge, 2010.
  * Wankhade et al. (2020) Kapil K Wankhade, Snehlata S Dongre, and Kalpana C Jondhale. 2020.  Data stream classification: A review.  _Iran Journal of Computer Science_ 3 (2020), 239–260.  https://doi.org/10.1007/s42044-020-00061-3
  * Wares et al. (2019) Scott Wares, John Isaacs, and Eyad Elyan. 2019.  Data stream mining: Methods and challenges for handling concept drift.  _SN Applied Sciences_ 1 (2019), 1–19.  https://doi.org/10.1007/s42452-019-1433-0
  * Woźniak et al. (2023) Michał Woźniak, Paweł Zyblewski, and Paweł Ksieniewicz. 2023.  Active weighted aging ensemble for drifted data stream classification.  _Information Sciences_ 630 (2023), 286–304.  https://doi.org/10.1016/j.ins.2023.02.046
  * Yuan et al. (2022) Liheng Yuan, Heng Li, Beihao Xia, Cuiying Gao, Mingyue Liu, Wei Yuan, and Xinge You. 2022.  Recent advances in concept drift adaptation methods for deep learning. In _Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI-22_ , Lud De Raedt (Ed.). International Joint Conferences on Artificial Intelligence Organization, 5654–5661.  https://doi.org/10.24963/ijcai.2022/788 Survey Track.


# Intra-task Mutual Attention based Vision
Transformer for Few-Shot Learning

Weihao Jiang, Chang Liu, Kun He*  —, Weihao Jiang, Chang Liu and Kun He are
with the School of Computer Science, Huazhong University of Science and
Technology, Wuhan, China 430074. Corresponding author: Kun He, E-mail:
<EMAIL_ADDRESS>work was supported by National Natural Science
Foundation (U22B2017).

###### Abstract

Humans possess remarkable ability to accurately classify new, unseen images
after being exposed to only a few examples. Such ability stems from their
capacity to identify common features shared between new and previously seen
images while disregarding distractions such as background variations. However,
for artificial neural network models, determining the most relevant features
for distinguishing between two images with limited samples presents a
challenge. In this paper, we propose an intra-task mutual attention method for
few-shot learning, that involves splitting the support and query samples into
patches and encoding them using the pre-trained Vision Transformer (ViT)
architecture. Specifically, we swap the class (CLS) token and patch tokens
between the support and query sets to have the mutual attention, which enables
each set to focus on the most useful information. This facilitates the
strengthening of intra-class representations and promotes closer proximity
between instances of the same class. For implementation, we adopt the ViT-
based network architecture and utilize pre-trained model parameters obtained
through self-supervision. By leveraging Masked Image Modeling as a self-
supervised training task for pre-training, the pre-trained model yields
semantically meaningful representations while successfully avoiding
supervision collapse. We then employ a meta-learning method to fine-tune the
last several layers and CLS token modules. Our strategy significantly reduces
the number of parameters that require fine-tuning while effectively utilizing
the capability of pre-trained model. Extensive experiments show that our
framework is simple, effective and computationally efficient, achieving
superior performance as compared to the state-of-the-art baselines on five
popular few-shot classification benchmarks under the 5-shot and 1-shot
scenarios.

###### Index Terms:

Few-shot learning, Vision Transformer, mutual attention, token.

††publicationid: pubid:

##  I Introduction

There has been a substantial body of researches in the realm of few-shot
learning (FSL) [1, 2, 3, 4], whose goal is to imitate human’s proficiency of
acquiring new concepts, even with a small number of provided training
instances. FSL models aim to learn from a minimal set of examples and
effectively generalize the learned knowledge to diverse and previously unseen
images. Researches of FSL can be categorized into two main types. The first is
meta-learning, which utilizes samples from a base class and optimizes the
model for target new class, and metric-based approaches, that primarily
involves classifying images based on the metric values between support and
query features, stand as the most widely adopted methods. The second is
transfer learning, where a feature extractor is trained on a rich base class
which is subsequently used to predict weights of the new class classifier. All
these FSL methods involve a key component of feature extraction, using a
backbone of Convolutional Neural Networks (CNNs) and recently Vision
Transformer (ViT) models.

Early FSL methods are based on CNN models [5, 6, 7, 8, 9, 10, 11, 12, 13].
These CNN-based methods primarily utilize local feature maps as the
representation of images. To obtain more distinctive features, researchers [9,
10, 11, 12, 13] introduce external modules to generate weight masks to enhance
these features. However, these methods are collectively limited by their
dependence on local features. This limitation may arise from the convolution
operation employed by CNN in image processing, making CNN-based methods
relatively less adept at processing global information in images.

TABLE I: Comparison between our proposed IMAFormer and the counterparts. In the pursuit of acquiring more discriminative features, CNN-based models frequently incorporate an additional module, leveraging local feature maps to identify crucial local regions for image representation. On the other hand, while ViT-based models inherently possess the capability to capture both global and local features, existing methods often utilize them in isolation. In contrast, our approach adopts the ViT architecture and innovatively integrates global and local features through our intra-task mutual attention method, aiming to derive more discriminative features. Model | Feature utilization method | Backbone | External module
---|---|---|---
CTM [9], CAN [10], CTX [11], ATL-Net [12], ReNet [13] | Only operate local feature map | ResNet | Yes
PMF [14] | Only use class token | ViT | No
FewTURE [15] | Only use patch tokens | Yes
HCTransformer [16] | Independent use of class token and patch tokens | Yes
IMAformer (ours) | Cross-use class token and patch tokens | ViT | No

More recently, there is a growing body of FSL works based on Vision
Transformers [14, 15, 16]. Approaches based on ViT facilitates the acquisition
of both global feature (class token) and local feature (patch token) of an
image by partitioning it into multiple non-overlapping patches and leveraging
the attention mechanism. While ViT excels in extracting global features,
current methods predominantly employ it as an efficient feature extractor,
falling into three main categories. The first method [14] involves extracting
class tokens from support set images and query images using a pre-trained
model, then the similarity between these class tokens serves as a metric for
evaluation. The second method [15] focuses on extracting patch tokens from the
images and learning a weight generator based on the patch tokens of the
support set to enhance the feature representation. The third method [16]
utilizes both class token and patch tokens through the introduction of a
potential supervised propagation scheme. However, it is noteworthy that
despite combining these two features, they are utilized independently in the
method.

ViT-based approaches demonstrate significant performance improvements compared
to the CNN-based models. However, they still face the challenge of adequately
considering the support set and query set across the entire task. Despite the
capability of ViT-based approaches to capture both local and global features
of the images, they generate features independently for the support set and
the query set. Consequently, in the generation of global features for the
support set, the query set’s local features has not been taken into account.
Similarly, the generation of global features for the query set overlooks the
contribution of local features from the support set. This leads to the
inability to obtain a more discriminative representation for the task.

We observe that humans carefully examine known support images one by one to
identify similar features to the query, particularly the prominent local
features. In this way, humans can grasp both global information and local
information, as well as make two-way comparisons between the query and
support. Thereby, we introduce an intra-task mutual attention based on Vision
Transformer, called IMAFormer, where the query and support sets pay attention
to each other and reinforce common features of the same category.

Diverging from previous approaches, our method leverages a fusion of both
global and local features from support and query sets, without resorting to
external modules beyond the backbone architecture, as shown in Table I.
Specifically, we divide the support and query sets into patches and utilize
the first $L-1$ layers of the ViT model to obtain the CLS token and patch
tokens. These patch tokens are then exchanged between the query and support
sets. Through the attention mechanism, the query set pays attention to the
detailed features of the support set, absorbing and strengthening shared
characteristics between the same class. Likewise, the support set focuses on
what the query concerns, reinforcing features of the same category as the
query within the support set. Such process increases the similarity within the
same category between the query and support sets while decreasing the
similarity between different categories.

Our method employs a Transformer architecture, incorporates the concept of
intra-task mutual attention, utilizes the meta-learning method to fine-tune
specific parameters based on the self-supervised learning pre-trained model
training with Masked Image Modelling as pretext task, and leads to boosted
results in few-shot learning tasks. Our main contributions are summarized as
follows:

  * •

We propose a novel method of simultaneous utilizing both global and local
features from support and query data to enhance the features of valuable
targets for few-shot learning based on Vision Transformers.

  * •

We propose a novel few-shot learning framework that enables the model to learn
more discriminative features by incorporating an intra-task mutual attention
module to the Vision Transformer architecture.

  * •

Our method is simple, efficient and effective, based on self-supervised
learning pre-trained models, only need to fine-tune fewer parameters, and can
outperform existing methods without external module.

  * •

We provide the efficacy of our method by achieving new state-of-the-art
results on five popular public benchmarks, and do ablation study to show the
effectiveness of our intra-task mutual attention method.

##  II Related Works

In this section, we first summarize the categories of few-shot learning
methods and analyze the distinctions between our approach and existing
methods. Next, we introduce several ViT pre-training methods and pick Masked
Image Modeling as our pre-training approach. Finally, we present the
application of cross-attention, emphasizing the differences in our intra-task
mutual attention.

###  II-A Few-shot Learning

In recent years, there has been a significant body of research dedicated to
the few-shot learning task. Broadly, these efforts can be categorized into two
main branches: meta-learning and transfer learning. Certainly, most of the
existing few-shot learning methods rely on the meta-learning framework, which
can be roughly divided into two categories, optimization-based and metric-
based.

Optimization-based methods in FSL. Optimization-based methods aim to quickly
learn the model parameters that can be optimized when encountering new tasks.
As a representative work of this category, MAML [17] obtains good
initialization parameters through internal and external training in the meta-
training stage. MAML has inspired many follow-up efforts, such as ANIL [18],
BOIL [19], and LEO [20].

Metric-based methods in FSL. Metric-based methods embed support images and
query images into the same space and classify query images by calculating the
distance or similarity. Using different metric methods, various models have
been derived, such as Matching Networks [21], Prototypical Networks [22], MSML
[6] and DeepEMD [23]. Simultaneously, to garner more distinctive features,
researchers have started treating the entire task holistically and devising
external modules to enhance the features. CTM [9] introduces a Category
Traversal Module that traverses the entire support set at once, discerning
task-relevant features based on both intra-class commonality and inter-class
uniqueness in the feature space. CAN [10] generates cross-attention maps for
each pair of support feature map and query sample feature map to emphasize
target object regions, enhancing the discriminative nature of the extracted
features. CTX [11] proposes learning spatial and semantic alignment between
CNN-extracted query and support features using a Transformer-style attention
mechanism. ATL-Net [12] introduces episodic attention calculated by a local
relation map between the query image and the support set, adaptively selecting
important local patches within the entire task. RENet [13] combines self-
correlational representation within each image and cross-correlational
attention modules between images to learn relational embeddings.

Recently, researchers have started incorporating ViT into few-shot learning
scenarios. The PMF [14] method consists of three phases. Firstly, it utilizes
a pre-trained Transformer model with external unsupervised data. Then, it
simulates few-shot tasks for meta-training using base categories. Finally, the
model is fine-tuned using scarcely labeled data from test tasks. HCTransformer
[16] employs hierarchically cascaded Transformers as a robust meta feature
extractor for few-shot learning. It utilizes the DINO [24] learning framework
and maximizes the use of annotated data for training. Additionally, it
enhances data efficiency through attribute surrogates learning and spectral
tokens pooling. FewTURE [15] adopts a completely transformer-based
architecture and learns a token importance weight through online optimization
during inference.

FSL methods based on transfer learning. Few-shot learning methods based on the
transfer learning framework use a simple scheme to train a classification
model on the overall training set, and yields comparable results. During
testing, it removes the classification head and retains the feature extraction
part, then trains a new classifier based on the support set from testing data.
The representative jobs are Dynamic Classifier [25], Baseline++ [26] and RFS
[27].

Our proposed method falls into the category of metric-based meta-learning
methods. In contrast to existing metric-based learning approaches that extract
support and query sample features separately, our method exploits the semantic
correlation between support and query features to emphasize the target object.
While some methods [9, 10, 11, 12, 13] also consider the relationship between
support and query samples, they rely on local feature maps and complex
external module networks. In contrast, our approach achieves superior
performance without the need for additional computational overhead, utilizing
self-supervised pre-trained models. As demonstrated in the experimental
section, our method outperforms these approaches by a significant margin.

###  II-B Self-supervised Pretraining ViT

Vision Transformers have shown superior performance compared to traditional
architectures [28, 29]. However, their effectiveness heavily relies on a vast
amount of labeled data, which may not always be feasible or sustainable for
supervised training. Thus, implementing a self-supervised approach for Vision
Transformers can make them powerful and more applicable to downstream
problems. BEiT [30] introduces the concept of Masked Image Modeling as a self-
supervised training task for training ViT. SimMIM [31] suggests that randomly
masking image parts and increasing the resolution of image blocks can yield
satisfactory results. In contrast to BEiT, MAE [32] simplifies the training
logic by using random masks to process input image blocks and directly
reconstructing the masked image blocks for training. DINO [24] presents a
straightforward self-supervised approach for ViT, which can be viewed as a
form of knowledge distillation without the use of labels. The DINO framework
involves predicting the output of a teacher network, constructed with a
momentum encoder, by employing a standard cross-entropy loss.

In our approach, we leverage a Vision Transformer architecture as an encoder,
chosen for its patch-based nature. Simultaneously, to address the issue of
supervision collapse, we implement self-supervised training, employing Masked
Image Modeling as a pretext task. Moreover, a considerable number of
Transformer models pre-trained using self-supervised methods exist. To
expediently assess the extensibility of our model, we incorporated certain
pre-trained model parameters when utilizing the ViT-Base model, specifically
those obtained through pre-training with the MAE method.

###  II-C Cross Attention

Cross-attention was first introduced in the field of natural language
processing [33], and the attention mechanism used to process sequence data
applications in the field of computer vision. For image processing,
convolutional neural network is usually used to extract the local features of
the images. However, in order to better capture the global relationship,
cross-attention introduces a mechanism that allows the model to allocate
different attention when processing different parts of the input. CTX [11]
proposes learning spatial and semantic alignment between CNN-extracted query
and support features using a cross attention mechanism. CrossViT [34] propose
a dual-branch transformer to combine image patches through token fusion module
based on cross attention, which uses a single token for each branch as a query
to exchange information with other branches.

Different from conventional approaches, we engage in the interchange of global
features between the support image and the query image. Specifically, we
leverage the attention mechanism inherent in the Transformer encoder layer to
facilitate the cross-attention. This obviates the necessity for a distinct
cross-attention module, thereby mitigating the need to alter the model
structure.

##  III Methodology

This section outlines the framework of our proposed IMAformer model, and
provides a comprehensive study on our intra-task mutual attention method.

###  III-A The Framework

Figure 1: The main framework of our IMAformer model. We adopt a Transformer
architecture with the first $L-1$ layers be used to encode the patched input
images containing support and query. After obtaining the CLS token and patch
tokens, we exchange the patch tokens between support and query, and the new
combined tokens are sent to the last layer to get the final CLS token for the
inputs.

The framework of our IMAformer model, as illustrated in Figure 1, adopts a
Transformer architecture. Initially, the input images are divided into non-
overlapping patches. These patches, from both the support set and the query
set, are then encoded using the first $L-1$ layers of ViT [35], resulting in
CLS tokens and patch tokens. The CLS tokens capture global semantic
information of the entire image, whereas the patch tokens represent local
information specific to each patch. Then, we swap the patch tokens between the
support image and the query image, such that the CLS tokens and patch tokens
are crossed for the support set and query set. This allows the CLS token of
the support image to be combined with the patch tokens of the query image, and
vice versa. The merged tokens are then fed into the final layer of the ViT
model, where attention mechanisms are employed to select relevant patches. In
the end, we obtain the final CLS tokens for both the support images and the
query images, for which we calculate the cosine similarity score to determine
their similarity. We can then assign the query image an appropriate category
basing on this score.

Diverging from conventional approaches, we engage in the interchange of global
features between the support image and the query image. Subsequently, we
leverage the attention mechanism inherent in the Transformer encoder layer to
facilitate the cross-attention. This obviates the necessity for a distinct
cross-attention module, thereby mitigating the need to alter the model
structure. Our intra-task mutual attention method is parameter-free and can be
seamlessly integrated into the existing Transformer Encoder without the need
for architectural modifications or additional networks. It can be directly
plugged in, enhancing the capabilities of the model without introducing extra
parameters or complexity.

###  III-B Intra-task Mutual Attention

In typical single few-shot task, there are generally five classes of support
set. If there are $K$ shots in each category of support, take the average of
the token values as the prototype tokens. After encoding, five sets of CLS
token and patch token combinations can be obtained, namely:

Figure 2: Illustration of the IMAformer processing pipeline for 5-way 1-shot
task. The support and query images are firstly patched and then the first
$L-1$ layers of IMAformer are used to encode the input images. After getting
the CLS token and patch tokens, the patch tokens between support and query are
exchanged, and the new combination of tokens are sent to the last layer to get
the final CLS token of input images. The CLS token of the same class between
support and query will be strengthened, as marked by the dotted box.

$\displaystyle P_{i}$
$\displaystyle=[CLS_{i}^{p},patch_{i,1}^{p},patch_{i,2}^{p},...,patch_{i,M}^{p}],$
(1) $\displaystyle i\in[1,5],$

where $M$ is the number of patches.

For a query image, we have a combination of CLS token and patch tokens:

$\displaystyle Q=[CLS^{q},patch_{1}^{q},patch_{2}^{q},...,patch_{M}^{q}],$ (2)

where $M$ is the number of patches.

To ensure that the query pays attention to the specific details of all
categories in the support set, and to make the support set aware of the
details of the query, we combine the CLS token of the query with the patch
tokens of each category in the support set. Similarly, we combine the CLS
tokens of each category in the support set with the patch tokens of the query.
Such process is visualized in Figure 2.

For $i\in[1,5]$, the combined tokens

$\displaystyle P_{i}{}^{\prime}$
$\displaystyle=[CLS_{i}^{p},patch_{1}^{q},patch_{2}^{q},...,patch_{M}^{q}],$
(3)

$\displaystyle Q_{i}{}^{\prime}$
$\displaystyle=[CLS^{q},patch_{i,1}^{p},patch_{i,2}^{p},...,patch_{i,M}^{p}]$
(4)

are sent into the final layer, and the final CLS token is extracted as the
representation for both the support and query:

$\displaystyle CLS_{p,i}=FFN(Softmax\left(\frac{W_{q}(CLS_{i}^{p})\cdot
W_{k}(P_{i}{}^{\prime})}{\sqrt{d}}\right)$ (5) $\displaystyle\cdot
W_{v}(P_{i}{}^{\prime})),$ $\displaystyle
CLS_{q,i}=FFN(Softmax\left(\frac{W_{q}(CLS^{q})\cdot
W_{k}(Q_{i}{}^{\prime})}{\sqrt{d}}\right)$ (6) $\displaystyle\cdot
W_{v}(Q_{i}{}^{\prime})),$

where $W_{q}$, $W_{k}$ and $W_{v}$ are weights of the fully connected layer,
and $FFN$ is the feed forward network.

By performing the above operations, each category in the support set gains the
ability to perceive information of the query image, thereby reinforcing the
representation if the support and query images are of the same class.
Similarly, the query image obtains distinct feature representations by
observing the feature details of each category in the support set separately.
This process enhances the representations that are similar to the support set,
resulting in more discriminative extracted features. To calculate the
similarity between the query image and the support set, the following approach
can be employed:

$\displaystyle cos(q,p_{j})=\sum_{i=1}^{5}cos(CLS_{q,i},CLS_{p,j}).$ (7)

By strengthening the representation of the same class, the margin between the
similarity of instances within the same class and the similarity between
instances of different classes increases. It in turn leads to improved
accuracy in classifying queries.

###  III-C Classification using Enhanced CLS token

Similarly, a query point $\mathbf{x}$ is passed through the feature extractor
to obtain the corresponding feature representation. The cosine similarity
between $\mathbf{x}$ and the support set prototypes is calculated to obtain
the probability that $\mathbf{x}$ belongs to class $n$:

$\displaystyle
p(y=n|\mathbf{x})=\frac{exp(cos(q_{\mathbf{x}},p_{n}))}{\sum_{n=1}^{N}exp(cos(q_{\mathbf{x}},p_{n}))},$
(8)

where $cos(\cdot,\cdot)$ denotes the cosine similarity between two vectors. In
the end, the cross entropy $\mathcal{L}$ is used as the loss function.

TABLE II: Comparison with the state-of-the-art 5-way 1-shot and 5-way 5-shot performance with 95$\%$ confidence intervals on $mini$Imagenet and $tiered$Imagenet.  Model | Backbone | $\approx$Params |  $mini$ImageNet |  $tiered$ImageNet
---|---|---|---|---
1-shot | 5-shot | 1-shot | 5-shot
ProtoNet [22] | ResNet-12 | 12.4 M | 62.29±0.33 | 79.46±0.48 | 68.25±0.23 | 84.01±0.56
FEAT [36] | ResNet-12 | 12.4 M | 66.78±0.20 | 82.05±0.14 | 70.80±0.23 | 84.79±0.16
CAN [10] | ResNet-12 | 12.4 M | 63.85±0.48 | 79.44±0.34 | 69.89±0.51 | 84.23±0.37
CTM [9] | ResNet-18 | 11.2 M | 64.12±0.82 | 80.51±0.13 | 68.41±0.39 | 84.28±1.73
ReNet [13] | ResNet-12 | 12.4 M | 67.60±0.44 | 82.58 ±0.30 | 71.61±0.51 | 85.28±0.35
DeepEMD [23] | ResNet-12 | 12.4 M | 65.91±0.82 | 82.41±0.56 | 71.16±0.87 | 86.03±0.58
IEPT [37] | ResNet-12 | 12.4 M | 67.05±0.44 | 82.90±0.30 | 72.24±0.50 | 86.73±0.34
MELR [38] | ResNet-12 | 12.4 M | 67.40±0.43 | 83.40±0.28 | 72.14±0.51 | 87.01±0.35
FRN [39] | ResNet-12 | 12.4 M | 66.45±0.19 | 82.83±0.13 | 72.06±0.22 | 86.89±0.14
CG [40] | ResNet-12 | 12.4 M | 67.02±0.20 | 82.32±0.14 | 71.66±0.23 | 85.50±0.15
DMF [41] | ResNet-12 | 12.4 M | 67.76±0.46 | 82.71±0.31 | 71.89±0.52 | 85.96±0.35
InfoPatch [42] | ResNet-12 | 12.4 M | 67.67±0.45 | 82.44±0.31 | - | -
BML [43] | ResNet-12 | 12.4 M | 67.04±0.63 | 83.63±0.29 | 68.99±0.50 | 85.49±0.34
CNL [40] | ResNet-12 | 12.4 M | 67.96±0.98 | 83.36±0.51 | 73.42±0.95 | 87.72±0.75
Meta-NVG [44] | ResNet-12 | 12.4 M | 67.14±0.80 | 83.82±0.51 | 74.58±0.88 | 86.73±0.61
PAL [45] | ResNet-12 | 12.4 M | 69.37±0.64 | 84.40±0.44 | 72.25±0.72 | 86.95±0.47
COSOC [46] | ResNet-12 | 12.4 M | 69.28±0.49 | 85.16±0.42 | 73.57±0.43 | 87.57±0.10
Meta DeepBDC [47] | ResNet-12 | 12.4 M | 67.34±0.43 | 84.46±0.28 | 72.34±0.49 | 87.31±0.32
QSFormer [7] | ResNet-12 | 12.4 M | 65.24±0.28 | 79.96±0.20 | 72.47±0.31 | 85.43±0.22
LEO [20] | WRN-28-10 | 36.5 M | 61.76±0.08 | 77.59±0.12 | 66.33±0.05 | 81.44±0.09
CC+rot [48] | WRN-28-10 | 36.5 M | 62.93±0.45 | 79.87±0.33 | 70.53±0.51 | 84.98±0.36
FEAT [36] | WRN-28-10 | 36.5 M | 65.10±0.20 | 81.11±0.14 | 70.41±0.23 | 84.38±0.16
MetaQDA [49] | WRN-28-10 | 36.5 M | 67.83±0.64 | 84.28±0.69 | 74.33±0.65 | 89.56±0.79
OM [50] | WRN-28-10 | 36.5 M | 66.78±0.30 | 85.29±0.41 | 71.54±0.29 | 87.79±0.46
FewTURE [15] | ViT-Small | 22 M | 68.02±0.88 | 84.51±0.53 | 72.96±0.92 | 86.43±0.67
FewTURE [15] | Swin-Tiny | 29 M | 72.40±0.78 | 86.38±0.49 | 76.32±0.87 | 89.96±0.55
HCTransformer[16] | ViT-Small | 22 M | 74.74±0.17 | 85.66±0.10 | 79.67±0.20 | 89.27±0.13
IMAformer (ours) | ViT-Small | 22 M | 78.48±0.31 | 88.18±0.29 | 79.78±0.30 | 91.01±0.27
IMAformer (ours) | ViT-Base | 86 M | 85.68±0.66 | 93.28±0.42 | 83.10±0.57 | 93.10±0.55
TABLE III: Comparison with the state-of-the-art 5-way 1-shot and 5-way 5-shot performance with 95$\%$ confidence intervals on CIFAR-FS and FC100.  Model | Backbone | $\approx$Params | CIFAR-FS | FC100
---|---|---|---|---
1-shot | 5-shot | 1-shot | 5-shot
ProtoNet [22] | ResNet-12 | 12.4 M | - | - | 41.54±0.76 | 57.08±0.76
MetaOpt [51] | ResNet-12 | 12.4 M | 72.00±0.70 | 84.20±0.50 | 41.10±0.60 | 55.50±0.60
MABAS [52] | ResNet-12 | 12.4 M | 73.51±0.92 | 85.65±0.65 | 42.31±0.75 | 58.16±0.78
RFS [27] | ResNet-12 | 12.4 M | 73.90±0.80 | 86.90±0.50 | 44.60±0.70 | 60.90±0.60
BML [43] | ResNet-12 | 12.4 M | 73.45±0.47 | 88.04±0.33 | - | -
CG [40] | ResNet-12 | 12.4 M | 73.00±0.70 | 85.80±0.50 | - | -
Meta-NVG [44] | ResNet-12 | 12.4 M | 74.63±0.91 | 86.45±0.59 | 46.40±0.81 | 61.33±0.71
RENet [13] | ResNet-12 | 12.4 M | 74.51±0.46 | 86.60±0.32 | - | -
TPMN [53] | ResNet-12 | 12.4 M | 75.50±0.90 | 87.20±0.60 | 46.93±0.71 | 63.26±0.74
MixFSL [54] | ResNet-12 | 12.4 M | - | - | 44.89±0.63 | 60.70±0.60
QSFormer [7] | ResNet-12 | 12.4 M | - | - | 46.51±0.26 | 61.58±0.25
CC+rot [48] | WRN-28-10 | 36.5 M | 73.62±0.31 | 86.05±0.22 | - | -
PSST [55] | WRN-28-10 | 36.5 M | 77.02±0.38 | 88.45±0.35 | - | -
Meta-QDA [49] | WRN-28-10 | 36.5 M | 75.83±0.88 | 88.79±0.75 | - | -
FewTURE [15] | ViT-Small | 22 M | 76.10±0.88 | 86.14±0.64 | 46.20±0.79 | 63.14±0.73
FewTURE [15] | Swin-Tiny | 29 M | 77.76±0.81 | 88.90±0.59 | 47.68±0.78 | 63.81±0.75
HCTransformer [16] | ViT-Small | 22 M | 78.89±0.18 | 87.73±0.11 | 48.27±0.15 | 61.49±0.15
IMAformer (ours) | ViT-Small | 22 M | 81.46±0.73 | 92.06±0.63 | 46.38±0.53 | 65.58±0.65
IMAformer (ours) | ViT-Base | 86 M | 81.34±0.67 | 91.30±0.83 | 48.02±0.63 | 65.52±0.75

##  IV Experiments

This section presents the experimental results, where we assess the
characteristics of our model and compare the performance against several
baseline methods on five widely used benchmark datasets. We also conduct an
ablation study to demonstrate the effectiveness of our proposed method.

###  IV-A Datasets

In the standard few-shot classification task, there are five popular benchmark
datasets for few-shot classification, including $mini$ImageNet [56],
$tiered$ImageNet [57], CIFAR-FS [58], FC100 [59] and CUB-200-2011 [60].

The $mini$ImageNet dataset contains 100 categories sampled from ILSVRC-2012
[61]. Each category contains 600 images, and 60,000 images in total. In
standard setting, it is randomly partitioned into training, validation and
testing sets, each containing 64, 16 and 20 categories.

The $tiered$ImageNet dataset is also from ILSVRC-2012, but with a more
extensive data scale. There are 34 super-categories, partitioned into
training, validation and testing sets, with 20, 6 and 8 super-categories,
respectively. It contains 608 categories, corresponding to 351, 97 and 160
categories in each partitioned dataset.

Both CIFAR-FS and FC100 are derived from CIFAR-100, which comprises 100
classes and 600 images per class, respectively. CIFAR-FS and FC100 are
characterized by small-resolution images, with dimensions of 32 × 32 pixels.
Specifically, CIFAR-FS is randomly divided into 64 training classes, 16
validation classes, and 20 testing classes. On the other hand, FC100 contains
100 classes sourced from 36 super-classes in CIFAR100. These 36 super-classes
are further divided into 12 training super-classes (containing 60 classes), 4
validation super-classes (containing 20 classes), and 4 testing super-classes
(containing 20 classes).

The CUB-200-2011 (CUB) dataset is for fine-grained classification. It is
composed of 200 classes with a total of 11,788 images. We follow the original
setting [60], that randomly allocates 100 classes as the base set, 50 classes
for validation, and 50 classes for testing.

###  IV-B Implementation Details

Motivated by the scalability and effectiveness of pretraining techniques, we
employ an MIM-pretrained Vision Transformer as our model backbone.
Specifically, we utilize the ViT-based architecture with a patch size of 16.
In the pre-training phase, we employ the same strategy proposed by [15] to
pretrain our ViT-Small backbones and mostly stick to the hyperparameter
settings reported in their work. For the ViT-Base model, we employ the MAE-
pretrained [32] model directly as our pre-trained model. In the meta training
phase, we utilize the AdamW optimizer with default settings. The initial
learning rate is set to $1e-5$ and decays to $1e-6$ following a cosine
learning rate scheme. For the $mini$ImageNet and $tiered$ImageNet datasets, we
resize the images to a size of 224 and train for 100 epochs, with each epoch
consisting of 600 episodes. Similarly, for the CIFAR-FS and FC100 datasets, we
resize the images to a size of 224 and train for 90 epochs, with 600 episodes
each. During the fine-tuning, we focus on the layers before and after our
intra-task mutual attention method, as well as the CLS token. To be consistent
with prior studies, we employ the validation set to select the best performing
models. In addition, we apply standard data augmentation techniques, including
random resized loops and horizontal flips.

During the meta-testing phase, we randomly sample 1000 tasks, each containing
10 query images per class. We report the mean accuracy along with the
corresponding $95\%$ confidence interval.

###  IV-C Performance Comparison

TABLE IV: Comparison with the state-of-the-art 5-way 1-shot and 5-way 5-shot performance with 95$\%$ confidence intervals on fine-grained dataset CUB.  Model | Backbone | $\approx$Params | CUB
---|---|---|---
1-shot | 5-shot
ProtoNet [22] | Conv-4 | 0.1 M | 64.42±0.48 | 81.82±0.35
FEAT [36] | Conv-4 | 0.1 M | 68.87±0.22 | 82.90±0.15
MELR [38] | Conv-4 | 0.1 M | 70.26±0.50 | 85.01±0.32
HGNN [62] | Conv-4 | 0.1 M | 69.43±0.49 | 87.67±0.45
MatchNet [21] | ResNet-12 | 12.4 M | 71.87±0.85 | 85.08±0.57
ICI [63] | ResNet-12 | 12.4 M | 76.16 | 90.32
RENet [13] | ResNet-12 | 12.4 M | 79.49±0.44 | 91.11±0.24
MAML [17] | ResNet-18 | 11.2 M | 68.42±1.07 | 83.47±0.62
Baseline++ [26] | ResNet-18 | 11.2 M | 67.02±0.90 | 83.58±0.54
AA [64] | ResNet-18 | 11.2 M | 74.22±1.09 | 88.65±0.55
Neg-Cosine [65] | ResNet-18 | 11.2 M | 72.66±0.85 | 89.40±0.43
Meta DeepBDC [47] | ResNet-18 | 11.2 M | 83.55±0.40 | 93.82±0.17
STL DeepBDC [47] | ResNet-18 | 11.2 M | 84.01±0.42 | 94.02±0.24
IMAformer (ours) | ViT-Small | 22 M | 89.30±0.29 | 95.80±0.18
IMAformer (ours) | ViT-Base | 86 M | 86.57±0.56 | 95.25±0.35

Following the established conventions of few-shot learning, we perform
experiments on five popular few-shot classification benchmarks, and the
results are shown in Tables II, III and IV, respectively. From the results, it
is evident that our proposed IMAformer consistently achieves competitive
performance compared to the state-of-the-art methods on both the 5-way 1-shot
and 5-way 5-shot tasks.

Results on $mini$Imagenet and $tiered$Imagenet datasets. Table II presents a
comparison on the 1-shot and 5-shot performance of our method with the
baselines on the $mini$Imagenet and $tiered$Imagenet datasets. We achieve
significant improvements over existing SOTA results while utilizing fewer
learnable parameters. For instance, on the $mini$Imagenet dataset, with ViT-
Small as the backbone, our IMAformer surpasses the runner-up by 3.74% in the
1-shot setting and 1.80% in the 5-shot setting. For ViT-Base backbone, our
IMAformer surpasses the runner-up by 10.94% in the 1-shot setting and 6.90% in
the 5-shot setting. Similarly, on the $tiered$Imagenet dataset, with ViT-Small
as the backbone, IMAformer outperforms the runner-up by 0.12% and 1.05% in the
1-shot and 5-shot settings, respectively. For ViT-Base backbone, IMAformer
outperforms the runner-up by 3.43% and 3.14% in the 1-shot and 5-shot
settings, respectively.

Compared to FewTURE and HCTransformer, which require external module, our
method eliminates this need while achieving better performance and utilizing
fewer learnable parameters. The significant margin between our method and the
runner-up baselines further validates the contribution of our network
structure, which effectively captures inherent information in the data and
maintains good generalization capability.

Results on small-resolution datasets. To verify the adaptability of the model,
we conduct further experiments on two small-resolution datasets along with
comparisons to other methods. This allows us to evaluate the performance of
our method across various data scenarios, ensuring fair and comprehensive
performance analysis.

Table III demonstrates the 1-shot and 5-shot classification performance on the
small-resolution datasets CIFAR-FS and FC100. In the case of CIFAR-FS,
IMAformer outperforms the runner-up by 2.57% in the 1-shot setting and 3.16%
in the 5-shot setting. Similarly, for FC100, our method surpasses the runner-
up by 1.77% in the 5-shot setting.

Results on fine-grained dataset. Table IV showcases the 1-shot and 5-shot
classification results on the fine-grained dataset CUB. Our IMAformer achieves
improvements of 5.29% in the 1-shot setting and 1.78% in the 5-shot setting
over the baselines with ViT-Small as the backbone.

These results highlight the effectiveness of our method in both small-
resolution and fine-grained classification scenarios. Moreover, it can be
found that our method is effective regardless of whether it adopts ViT-Small
or ViT-Base. Our method achieves state-of-the-art performance across all
settings, with only one exception on the 1-shot task on FC100. In this
particular scenario, the performance of our method is slightly lower compared
to HCTransformer [16], which utilizes a stacked Transformers architecture with
a deeper backbone as the feature extractor.

The substantial enhancements observed affirm the superiority of our method,
which employs the intra-task mutual attention method to augment the
distinction between support and query sets. The effectiveness of this approach
is substantiated by the remarkable performance gains achieved across diverse
few-shot learning tasks.

###  IV-D Ablation Study

In this subsection, we conduct ablation experiments to analyze the impact of
each component on the performance of our method. We focus on the 5-way 5-shot
setting on the $mini$Imagenet dataset, utilizing the pre-trained Vision
Transformer backbone. By analyzing the performance changes resulting from
these ablations, we gain insights into the contribution of each component to
the overall effectiveness of our method. Specifically, we investigate the
influence of the following two components:

Intra-task Mutual Attention Method: We evaluate the performance of our model
without the intra-task mutual attention method, which is responsible for
enhancing the interaction and alignment between the support and query sets.

TABLE V: Comparison on 5-way 5-shot performance with 95$\%$ confidence intervals with the model without intra-task mutual attention method on $mini$ImageNet. ViT-Base is the backbone, with parameters of the last two layers and the CLS token be fine-tuned. Model | Backbone | Fine-tuning Layers | Accuracy
---|---|---|---
Vanilla | ViT-Small | no layer | 82.05±0.34
Vanilla | ViT-Small | last 6 layers | 75.50±0.51
Vanilla | ViT-Small | 12 layers | 75.16±0.51
IMAformer | ViT-Small | last 6 layers | 88.18±0.29
Vanilla | ViT-Base | last 2 layers+CLS | 75.68±0.81
IMAformer | ViT-Base | last 2 layers+CLS | 93.28±0.42

To verify the effectiveness of our intra-task mutual attention method, we
design a comparative model called Vanilla. In the Vanilla method, we do not
employ patch token switching, and the final CLS tokens of the support and
query sets are solely encoded by the backbone with the last six layers be
fine-tuned. As shown in Table V, the results clearly demonstrate the
significant improvement achieved by our IMAformer method compared to the
Vanilla method no matter on ViT-Small or ViT-Base.

Furthermore, an observation from the initial three rows in Table V indicates
that the absence of our intra-task mutual attention method leads to a
degradation in the performance of the fine-tuned model. This result confirms
the efficacy of our intra-task mutual attention method in improving the
performance of few-shot learning tasks.

$0$$2$$4$$6$$8$$10$$12$$86$$86.5$$87$$87.5$$88$$88.5$Fine-tuning
layersAccuracy on test set $(\%)$w/o CLSw/ CLS Figure 3: Comparison on 5-way
5-shot performance with the model under different fine-tuning layers on
$mini$ImageNet. ViT-Small is the backbone. ’w/o CLS’ denotes fine-tuning
excluding CLS and ’w/ CLS’ denotes fine-tuning including CLS.

CLS Token and Learnable Parameters: We conduct an examination of the impact of
various fine-tuning parameters on the performance of the model, including the
fine-tuning of the CLS token.

We conducted a more in-depth analysis to assess the influence of various fine-
tuning parameters on our method, utilizing the ViT-Small architecture as the
basis. The corresponding results are presented in Figure 3. Notably, as the
number of fine-tuning layers increases, there is a continuous improvement in
performance. However, beyond six fine-tuning layers, the model experiences
slight fluctuations. Further increasing the number of fine-tuning layers
results in an enhancement in model performance, but the improvement is not
substantial. Additionally, we observe that not fine-tuning the CLS token
yields higher accuracy rates than fine-tuning. Thereby, for ViT-Small, we
choose to fine-tune the last six layers.

TABLE VI: Comparison on 5-way 5-shot performance with 95$\%$ confidence intervals with the model under different fine-tuning parameters on $mini$ImageNet. ViT-Base is the backbone. Num | Fine-tuning Layers |  $\approx$Params | Accuracy
---|---|---|---
1 | no layer |  0 M | 38.82±0.74
2 | last 1 layer |  7 M | 90.71±0.48
3 | last 2 layers | 14 M | 92.66±0.44
4 | last 2 layers+CLS | 14 M | 93.28±0.42
5 | last 3 layers+CLS | 21 M | 92.83±0.45
6 | last 4 layers | 28 M | 86.48±0.58
7 | last 4 layers+CLS | 28 M | 87.85±0.55

Simultaneously, we conducted an analysis to assess the impact of different
fine-tuning parameters on our approach based on ViT-Base. The corresponding
results are presented in Table VI. Notably, when fine-tuning solely one layer,
the performance of the model exhibited improvements compared to the instance
where no fine-tuning is applied. Moreover, as the number of learnable
parameters increases, the model’s performance improves correspondingly.
However, note that there exists a trade-off between performance and the number
of learnable parameters. Specifically, the best performance was achieved when
fine-tuning the last two layers together with CLS token; beyond this point,
the accuracy began to decline as more parameters were fine-tuned.
Additionally, we observe that fine-tuning the CLS token yields higher accuracy
rates than no-fine-tuning.

###  IV-E Visualization Analysis

To qualitatively evaluate the proposed intra-task mutual attention mechanism,
we compare the query embedding visualization results of IMAformer to the model
without intra-task mutual attention method. As shown in Fig. 4, after the
mutual attention within the task, similar query features within the task are
more clustered such that the visual features are most discriminative for a
given task.

(a)

(b)

Figure 4: Qualitative visualization of model-based embedding before and after
using intra-task mutual attention method on test tasks. Each figure shows the
locations of PCA projected query embeddings (a) before and (b) after the
adaptation of IMAformer. Values below are the 5-way 15-shot few-shot task
before and after the adaptation. Obviously, the embedding adaptation step of
IMAformer pushes the query embeddings toward their own clusters, such that
they can better fits the test data of its categories.

This result once again demonstrates the effectiveness of our approach, that
is, through the mutual attention of global features and local features within
the task, the obtained query features and support features are more
distinguishable.

##  V Conclusion

We proposed a novel framework called IMAFormer that fully utilizes the global
feature and local feature in the context of few-shot learning. Our approach
involves segmenting the input image into smaller patches and encoding these
local patches using a pre-trained Vision Transformer architecture, achieved
through self-supervised learning. This segmentation allows us to obtain both
global information representation through the CLS token and local information
representation through the patch tokens. To enhance the representations of
support and query sets, we introduced an intra-task mutual attention method.
This method enables the support set focus on and leverage the detailed
features presented in the query set, thereby enhancing its representations and
promoting similarity between instances. Similarly, the query set can attend to
the detailed features of each class in the support set, leveraging similar
features to improve its representations and making the class features more
representative. Empirical evaluations demonstrated the effectiveness of our
proposed method, achieving advanced performance.

Our approach offers several advantages. Firstly, it significantly improves the
image representation through integrating class token and patch tokens.
Secondly, it allows for efficient utilization of the knowledge and
representations encoded in these pre-trained models learned from Masked Image
Modeling pre-training task. By building upon these pre-trained models, we can
potentially achieve competitive performance in few-shot learning tasks without
any external weight module.

## Acknowledgments

This work is supported by National Natural Science Foundation (U22B2017).

## References

  * [1] G. Koch, R. Zemel, R. Salakhutdinov _et al._ , “Siamese neural networks for one-shot image recognition,” in _ICML Deep Learning Workshop_ , 2015.
  * [2] B. M. Lake, R. Salakhutdinov, J. Gross, and J. B. Tenenbaum, “One shot learning of simple visual concepts,” in _Proceedings of the 33th Annual Meeting of the Cognitive Science Society_ , 2011.
  * [3] L. Zhang, S. Wang, X. Chang, J. Liu, Z. Ge, and Q. Zheng, “Auto-fsl: Searching the attribute consistent network for few-shot learning,” _IEEE Transactions on Circuits and Systems for Video Technology_ , vol. 32, no. 3, pp. 1213–1223, 2022.
  * [4] Z. Dang, M. Luo, C. Jia, C. Yan, X. Chang, and Q. Zheng, “Counterfactual generation framework for few-shot learning,” _IEEE Transactions on Circuits and Systems for Video Technology_ , vol. 33, no. 8, pp. 3747–3758, 2023.
  * [5] F. Hao, F. He, J. Cheng, and D. Tao, “Global-local interplay in semantic alignment for few-shot learning,” _IEEE Transactions on Circuits and Systems for Video Technology_ , vol. 32, no. 7, pp. 4351–4363, 2022.
  * [6] W. Jiang, K. Huang, J. Geng, and X. Deng, “Multi-scale metric learning for few-shot learning,” _IEEE Transactions on Circuits and Systems for Video Technology_ , vol. 31, no. 3, pp. 1091–1102, 2021.
  * [7] X. Wang, X. Wang, B. Jiang, and B. Luo, “Few-shot learning meets transformer: Unified query-support transformers for few-shot classification,” _IEEE Transactions on Circuits and Systems for Video Technology_ , vol. 33, no. 12, pp. 7789–7802, 2023.
  * [8] B. Jiang, K. Zhao, and J. Tang, “Rgtransformer: Region-graph transformer for image representation and few-shot classification,” _IEEE Signal Processing Letters_ , vol. 29, pp. 792–796, 2022.
  * [9] H. Li, D. Eigen, S. Dodge, M. Zeiler, and X. Wang, “Finding task-relevant features for few-shot learning by category traversal,” in _IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019_. Computer Vision Foundation / IEEE, 2019, pp. 1–10.
  * [10] R. Hou, H. Chang, B. Ma, S. Shan, and X. Chen, “Cross attention network for few-shot classification,” in _Advances in Neural Information Processing Systems, December 8-14, 2019, Vancouver, BC, Canada_ , H. M. Wallach, H. Larochelle, A. Beygelzimer, F. d’Alché-Buc, E. B. Fox, and R. Garnett, Eds., 2019, pp. 4005–4016.
  * [11] C. Doersch, A. Gupta, and A. Zisserman, “Crosstransformers: spatially-aware few-shot transfer,” in _Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual_ , H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, Eds., 2020.
  * [12] C. Dong, W. Li, J. Huo, Z. Gu, and Y. Gao, “Learning task-aware local representations for few-shot learning,” in _Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence_ , 2020.
  * [13] D. Kang, H. Kwon, J. Min, and M. Cho, “Relational embedding for few-shot classification,” in _2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021_. IEEE, 2021, pp. 8802–8813.
  * [14] S. X. Hu, D. Li, J. Stühmer, M. Kim, and T. M. Hospedales, “Pushing the limits of simple pipelines for few-shot learning: External data and fine-tuning make a difference,” in _IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022.
  * [15] M. Hiller, R. Ma, M. Harandi, and T. Drummond, “Rethinking generalization in few-shot classification,” in _Advances in Neural Information Processing Systems_ , 2022.
  * [16] Y. He, W. Liang, D. Zhao, H. Zhou, W. Ge, Y. Yu, and W. Zhang, “Attribute surrogates learning and spectral tokens pooling in transformers for few-shot learning,” in _IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022_. IEEE, 2022, pp. 9109–9119.
  * [17] C. Finn, P. Abbeel, and S. Levine, “Model-agnostic meta-learning for fast adaptation of deep networks,” in _Proceedings of the 34th International Conference on Machine Learning_ , 2017.
  * [18] A. Raghu, M. Raghu, S. Bengio, and O. Vinyals, “Rapid learning or feature reuse? towards understanding the effectiveness of MAML,” in _8th International Conference on Learning Representations_ , 2020.
  * [19] J. Oh, H. Yoo, C. Kim, and S. Yun, “BOIL: towards representation change for few-shot learning,” in _9th International Conference on Learning Representations_ , 2021.
  * [20] A. A. Rusu, D. Rao, J. Sygnowski, O. Vinyals, R. Pascanu, S. Osindero, and R. Hadsell, “Meta-learning with latent embedding optimization,” in _7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019_. OpenReview.net, 2019.
  * [21] O. Vinyals, C. Blundell, T. Lillicrap, k. kavukcuoglu, and D. Wierstra, “Matching networks for one shot learning,” in _Advances in Neural Information Processing Systems_ , 2016.
  * [22] J. Snell, K. Swersky, and R. S. Zemel, “Prototypical networks for few-shot learning,” in _Advances in Neural Information Processing Systems_ , 2017.
  * [23] C. Zhang, Y. Cai, G. Lin, and C. Shen, “Deepemd: Few-shot image classification with differentiable earth mover’s distance and structured classifiers,” in _IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020.
  * [24] M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, P. Bojanowski, and A. Joulin, “Emerging properties in self-supervised vision transformers,” in _2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021_. IEEE, 2021, pp. 9630–9640.
  * [25] S. Gidaris and N. Komodakis, “Dynamic few-shot visual learning without forgetting,” in _IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2018.
  * [26] W. Chen, Y. Liu, Z. Kira, Y. F. Wang, and J. Huang, “A closer look at few-shot classification,” in _7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019_. OpenReview.net, 2019.
  * [27] Y. Tian, Y. Wang, D. Krishnan, J. B. Tenenbaum, and P. Isola, “Rethinking few-shot image classification: A good embedding is all you need?” in _Computer Vision - ECCV 2020 - 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XIV_. Springer, 2020, pp. 266–282.
  * [28] W. Jiang, W. Zhou, and H. Hu, “Double-stream position learning transformer network for image captioning,” _IEEE Transactions on Circuits and Systems for Video Technology_ , vol. 32, no. 11, pp. 7706–7718, 2022.
  * [29] B. Tang, Z. Liu, Y. Tan, and Q. He, “Hrtransnet: Hrformer-driven two-modality salient object detection,” _IEEE Transactions on Circuits and Systems for Video Technology_ , vol. 33, no. 2, pp. 728–742, 2023.
  * [30] H. Bao, L. Dong, S. Piao, and F. Wei, “Beit: BERT pre-training of image transformers,” in _The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022_. OpenReview.net, 2022.
  * [31] Z. Xie, Z. Zhang, Y. Cao, Y. Lin, J. Bao, Z. Yao, Q. Dai, and H. Hu, “Simmim: a simple framework for masked image modeling,” in _IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022_. IEEE, 2022, pp. 9643–9653.
  * [32] K. He, X. Chen, S. Xie, Y. Li, P. Dollár, and R. B. Girshick, “Masked autoencoders are scalable vision learners,” in _IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022_. IEEE, 2022, pp. 15 979–15 988.
  * [33] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” in _Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA_ , 2017, pp. 5998–6008.
  * [34] C. R. Chen, Q. Fan, and R. Panda, “Crossvit: Cross-attention multi-scale vision transformer for image classification,” in _2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021_. IEEE, 2021, pp. 347–356.
  * [35] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby, “An image is worth 16x16 words: Transformers for image recognition at scale,” in _9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021_. OpenReview.net, 2021.
  * [36] H. Ye, H. Hu, D. Zhan, and F. Sha, “Few-shot learning via embedding adaptation with set-to-set functions,” in _IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020.
  * [37] M. Zhang, J. Zhang, Z. Lu, T. Xiang, M. Ding, and S. Huang, “IEPT: instance-level and episode-level pretext tasks for few-shot learning,” in _9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021_. OpenReview.net, 2021.
  * [38] N. Fei, Z. Lu, T. Xiang, and S. Huang, “MELR: meta-learning via modeling episode-level relationships for few-shot learning,” in _9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021_. OpenReview.net, 2021.
  * [39] D. Wertheimer, L. Tang, and B. Hariharan, “Few-shot classification with feature map reconstruction networks,” in _IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021.
  * [40] J. Zhao, Y. Yang, X. Lin, J. Yang, and L. He, “Looking wider for better adaptive representation in few-shot learning,” in _AAAI Conference on Artificial Intelligence_ , 2021.
  * [41] C. Xu, Y. Fu, C. Liu, C. Wang, J. Li, F. Huang, L. Zhang, and X. Xue, “Learning dynamic alignment via meta-filter for few-shot learning,” in _IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021_. Computer Vision Foundation / IEEE, 2021, pp. 5182–5191.
  * [42] C. Liu, Y. Fu, C. Xu, S. Yang, J. Li, C. Wang, and L. Zhang, “Learning a few-shot embedding model with contrastive learning,” in _Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021_. AAAI Press, 2021, pp. 8635–8643.
  * [43] Z. Zhou, X. Qiu, J. Xie, J. Wu, and C. Zhang, “Binocular mutual learning for improving few-shot classification,” in _2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021_. IEEE, 2021, pp. 8382–8391.
  * [44] C. Zhang, H. Ding, G. Lin, R. Li, C. Wang, and C. Shen, “Meta navigator: Search for a good adaptation policy for few-shot learning,” in _2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021_. IEEE, 2021, pp. 9415–9424.
  * [45] J. Ma, H. Xie, G. Han, S. Chang, A. Galstyan, and W. Abd-Almageed, “Partner-assisted learning for few-shot image classification,” in _2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021_. IEEE, 2021, pp. 10 553–10 562.
  * [46] X. Luo, L. Wei, L. Wen, J. Yang, L. Xie, Z. Xu, and Q. Tian, “Rectifying the shortcut learning of background for few-shot learning,” in _Advances in Neural Information Processing Systems, December 6-14, 2021, virtual_ , M. Ranzato, A. Beygelzimer, Y. N. Dauphin, P. Liang, and J. W. Vaughan, Eds., 2021, pp. 13 073–13 085.
  * [47] J. Xie, F. Long, J. Lv, Q. Wang, and P. Li, “Joint distribution matters: Deep brownian distance covariance for few-shot classification,” in _IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022_. IEEE, 2022, pp. 7962–7971.
  * [48] S. Gidaris, A. Bursuc, N. Komodakis, P. Pérez, and M. Cord, “Boosting few-shot visual learning with self-supervision,” in _2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 - November 2, 2019_. IEEE, 2019, pp. 8058–8067.
  * [49] X. Zhang, D. Meng, H. Gouk, and T. M. Hospedales, “Shallow bayesian meta learning for real-world few-shot recognition,” in _2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021_. IEEE, 2021, pp. 631–640.
  * [50] G. Qi, H. Yu, Z. Lu, and S. Li, “Transductive few-shot classification on the oblique manifold,” in _2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021_. IEEE, 2021, pp. 8392–8402.
  * [51] K. Lee, S. Maji, A. Ravichandran, and S. Soatto, “Meta-learning with differentiable convex optimization,” in _IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019_. Computer Vision Foundation / IEEE, 2019, pp. 10 657–10 665.
  * [52] J. Kim, H. Kim, and G. Kim, “Model-agnostic boundary-adversarial sampling for test-time generalization in few-shot learning,” in _Computer Vision - ECCV 2020 - 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part I_. Springer, 2020, pp. 599–617.
  * [53] J. Wu, T. Zhang, Y. Zhang, and F. Wu, “Task-aware part mining network for few-shot learning,” in _2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021_. IEEE, 2021, pp. 8413–8422.
  * [54] A. Afrasiyabi, J. Lalonde, and C. Gagné, “Mixture-based feature space learning for few-shot image classification,” in _2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021_. IEEE, 2021, pp. 9021–9031.
  * [55] Z. Chen, J. Ge, H. Zhan, S. Huang, and D. Wang, “Pareto self-supervised training for few-shot learning,” in _IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021_. Computer Vision Foundation / IEEE, 2021, pp. 13 663–13 672.
  * [56] S. Ravi and H. Larochelle, “Optimization as a model for few-shot learning,” in _5th International Conference on Learning Representations_ , 2017.
  * [57] M. Ren, E. Triantafillou, S. Ravi, J. Snell, K. Swersky, J. B. Tenenbaum, H. Larochelle, and R. S. Zemel, “Meta-learning for semi-supervised few-shot classification,” in _6th International Conference on Learning Representations_ , 2018.
  * [58] L. Bertinetto, J. F. Henriques, P. H. S. Torr, and A. Vedaldi, “Meta-learning with differentiable closed-form solvers,” in _7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019_. OpenReview.net, 2019.
  * [59] B. N. Oreshkin, P. R. López, and A. Lacoste, “TADAM: task dependent adaptive metric for improved few-shot learning,” in _Advances in Neural Information Processing Systems, December 3-8, 2018, Montréal, Canada_ , S. Bengio, H. M. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, Eds., 2018, pp. 719–729.
  * [60] C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie, “The caltech-ucsd birds-200-2011 dataset,” 2011.
  * [61] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” in _Advances in Neural Information Processing Systems_ , 2012.
  * [62] C. Chen, K. Li, W. Wei, J. T. Zhou, and Z. Zeng, “Hierarchical graph neural networks for few-shot learning,” _IEEE Transactions on Circuits and Systems for Video Technology_ , vol. 32, no. 1, pp. 240–252, 2022.
  * [63] Y. Wang, C. Xu, C. Liu, L. Zhang, and Y. Fu, “Instance credibility inference for few-shot learning,” in _2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020_. Computer Vision Foundation / IEEE, 2020, pp. 12 833–12 842.
  * [64] A. Afrasiyabi, J. Lalonde, and C. Gagné, “Associative alignment for few-shot image classification,” in _Computer Vision - ECCV 2020 - 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part V_. Springer, 2020, pp. 18–35.
  * [65] B. Liu, Y. Cao, Y. Lin, Q. Li, Z. Zhang, M. Long, and H. Hu, “Negative margin matters: Understanding margin in few-shot classification,” in _Computer Vision - ECCV 2020 - 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part IV_. Springer, 2020, pp. 438–455.

|  Weihao Jiang is currently pursuing a Ph.D. in School of Computer Science
and Technology, Huazhong University of Science and Technology, Wuhan, China.
His research focuses on deep learning, few-shot learning and meta-learning.
---|---
|  Chang Liu is currently pursuing a master degree in computer application
technology from the Huazhong University of Science and Technology, Wuhan,
China. His research mainly focuses on deep learning, few-shot learning, and
data augmentation.
---|---
|  Kun He (SM18) is currently a Professor in School of Computer Science and
Technology, Huazhong University of Science and Technology, Wuhan, P.R. China.
She received the Ph.D. degree in system engineering from Huazhong University
of Science and Technology, Wuhan, China, in 2006. She had been with the
Department of Management Science and Engineering at Stanford University in
2011-2012 as a visiting researcher. She had been with the de- partment of
Computer Science at Cornell University in 2013-2015 as a visiting associate
professor, in 2016 as a visiting professor, and in 2018 as a visiting
professor. She was honored as a Mary Shepard B. Upson visiting professor for
the 2016-2017 Academic year in Engineering, Cornell University, New York. Her
research interests include adversarial learning, representation learning,
social network analysis, and combinatorial optimization.
---|---