# Contrastive Learning for Interactive Recommendation in Fashion

Karin Sevegnani, Arjun Seshadri, Tian Wang, Anurag Beniwal, Julian McAuley,
Alan Lu, Gerard Medioni  AmazonSan FranciscoUSA<EMAIL_ADDRESS>sesarjun, wangtan, beanurag, jumcaule, alalu<EMAIL_ADDRESS>
(2022; 2022)

###### Abstract.

Recommender systems and search are both indispensable in facilitating
personalization and ease of browsing in online fashion platforms. However, the
two tools often operate independently, failing to combine the strengths of
recommender systems to accurately capture user tastes with search systems’
ability to process user queries. We propose a novel remedy to this problem by
automatically recommending personalized fashion items based on a user-provided
text request. Our proposed model, WhisperLite, uses contrastive learning to
capture user intent from natural language text and improves the recommendation
quality of fashion products. WhisperLite combines the strength of CLIP
embeddings with additional neural network layers for personalization, and is
trained using a composite loss function based on binary cross entropy and
contrastive loss. The model demonstrates a significant improvement in offline
recommendation retrieval metrics when tested on a real-world dataset collected
from an online retail fashion store, as well as widely used open-source
datasets in different e-commerce domains, such as restaurants, movies and TV
shows, clothing and shoe reviews. We additionally conduct a user study that
captures user judgements on the relevance of the model’s recommended items,
confirming the relevancy of WhisperLite’s recommendations in an online
setting.

††copyright: acmcopyright††journalyear: 2022††conference: Make sure to enter
the correct; July 15 2022; Madrid - Spain††copyright:
rightsretained††conference: ACM SIGIR Workshop on eCommerce; July 15, 2022;
Madrid, Spain††journalyear: 2022

##  1\. Introduction

Recommendation systems are at the core of e-commerce websites such as Amazon,
Netflix, and Ebay. These systems recommend products based on the user’s
historical preferences and purchases (Zhang et al., 2019; Ricci et al., 2015).
The goal is to recommend products to users that they might like and improve
the experience to make it enjoyable and satisfactory.

Text request: No dresses, skirts, shorts or heels please. I like Shoes like Dr. Martens, casual low to no heel. |
---|---
Text request: Show me some spring and summer wear. I’m a mom so please no crop tops |
Figure 1. Examples of personalized text requests written by the user browsing
a clothing retail online shop on the left, and an accordingly recommended item
on the right.

The goal of this work is to recommend fashion items focusing specifically on
the user’s free-form text request provided to a fashion stylist while using a
personalized styling subscription service. In our problem setting, we only
rely on the text request provided by the user, and do not assume knowledge of
user click-stream information or prior purchases. Although prior purchases are
often used to model user preferences (Schafer et al., 2007), the presence of
such information cannot be assumed for so-called ”cold start” users that are
using a service for the first time and have no historical purchase or click
stream information associated with them. Moreover, text requests often reveal
more nuanced aspects of user preferences at the time of placing a styling
request, making our work essential to improving existing recommender systems.

Our primary dataset consists of three parts: text requests from users of a
personalized styling service, items chosen by a human stylist for the user
based on those text requests, and items the user chooses to try from the
stylist’s recommended items. Figure 1 shows two stylized examples, where users
communicate a preference towards specific clothing items through a text
request, and professional fashion stylists respond with a product
recommendation pertaining to the customer request.

Our modeling task is challenging on several levels. First, text requests are
more elaborate than a search query, and contain both useful contextual
information that captures high-level user preferences and less helpful
information that is irrelevant for retrieving fashion items e.g. ”I am a
school teacher and I don’t wear yoga pants.” Second, each item in the
recommended set of items might only meet a subset of criteria specified by the
user in the text request, creating opportunities for model misattribution and
overfitting. Even the recommended set of items as a whole might not cover all
of the criteria mentioned by the user. Third, there are items in the dataset
selected by stylists that the user could potentially like despite the items
not having the exact attributes requested by the user, further exacerbating
the misattribution problem. Finally, there could be items that the user
selected despite being irrelevant to the text request, or sometimes even
contradictory to their initial text request. Each of these factors introduce
noise in the dataset labels, and together create a challenging modeling
problem.

The contributions of this paper can be summarized as follows:

  * •

Defining and addressing a real-world recommendation task for online fashion
retail in order to improve the user experience (Section 3.1),

  * •

Validating the multi-domain generalization of the system offline using a real-
world dataset and three other publicly available data sources (Section 4.3),

  * •

Validating the online performance of the system through a user study on Amazon
Mechanical Turk (Section 4.4).

##  2\. Related Work

The literature of recommender systems can be broadly segmented into
collaborative filtering (Su and Khoshgoftaar, 2009; Koren and Bell, 2015) and
content-based methods (Pazzani and Billsus, 2007; Balabanović and Shoham,
1997). While collaborative filtering techniques only exploit past user
purchases and product interactions to generate new recommendations (Herlocker
et al., 2000), content-based methods rely on information about the users or
the items (Basu et al., 1998). Recommendation systems for fashion retail
e-commerce, the domain of our work, frequently fall into the content-based
category, and are often paired with images and textual descriptions of the
products (Chakraborty et al., 2021; Kang et al., 2017; Chen et al., 2019; Yin
et al., 2019; Hwangbo et al., 2018; Tu and Dong, 2010; Ajmani et al., 2013).
Our work is similar in being content-based to other works in fashion retail
e-commerce, as it leverages textual user requests as the primary
representation of the user. Within the broader literature of recommendation
systems, our work most closely aligns with session-based recommendation
approaches, which focus on predicting the user’s immediate next actions using
explicit feedback of the customer during the current session. In our setting,
the user’s text request serves as the representation of the customer’s
preferences in the current session. We refer the reader to (Ludewig and
Jannach, 2018), which provides a comprehensive review of the session-based
recommendation literature.

##  3\. Methodology

###  3.1. Problem Formulation

Our work approaches the recommendation problem by trying to build affinity
between a natural language text query expressed by the user and an item
description. The natural language query $\textbf{x}=(x_{1},x_{2},...,x_{N})$
can consist of several sentences, and the item description
$\textbf{z}=(z_{1},z_{2},...,z_{M})$ can include several item attributes and
characteristics, where $N$ and $M$ are the number of tokens in the request and
the item description, respectively. The similarity between the request x and
the description z is determined by the cosine distance between the two. Once
similarity scores are computed, the various similarities are then ranked and
used to perform a retrieval task. The success of the task is evaluated with
different retrieval metrics, which we describe in Section 4.2.

We present two solutions to our problem setting:

  * •

The Whisper approach: training a model that refines word embeddings for text
notes and item descriptions in order to closely pair requests and products in
the word embeddings space and treat them as a match for recommendation
(Section 3.2); and

  * •

The WhisperLite approach: derive embeddings from a pre-trained system and
train a model only to learn which items to recommend using contrastive loss
(Section 3.3).

###  3.2. Whisper

XLNetBiLSTMBiLSTMIneedtopsandskirts467204953245029$cos(\theta)$womangreycroptop9315671045

Figure 2. Architecture of the Whisper model. The dashed lines mean that the
weights for XLNet are frozen when computing the embeddings for the product
description.

Large pre-trained transformers such as BERT (Devlin et al., 2018), RoBERTa
(Liu et al., 2019), or XLNet (Yang et al., 2019) have proven particularly
effective at the generation of contextual word embeddings. The architecture of
the Whisper model is planned and developed in a modular way. We initially
leverage one of the state-of-the-art large language models for generating word
embeddings. XLNet is the transformer of our choice, since it has been proven
to outperform BERT on a variety of tasks (Malon, 2021; Yang et al., 2019).

In order to prepare the initial text requests and item descriptions, they are
first split into tokens and assigned input ids based on the model tokenizer’s
built-in vocabulary. Moreover, we perform a keyword search for each text
request to highlight categorical tokens and strengthen the division of the
text requests into the main clothing categories.

The pre-processed inputs are then passed into XLNet, which generates
contextual word embeddings in an auto-regressive way. The embeddings of the
words are one-dimensional tensors retrieved through the last hidden state of
the language model. Then, the embeddings are fed into a bidirectional LSTM,
which produces one feature vector for each text request received as input. A
second bi-LSTM is used in order to generate different feature vectors for the
product descriptions in a two-tower fashion (Yang et al., 2020), since the
semantics of user requests are very different from those of product
descriptions.

As shown in Figure 2 of the Whisper architecture, the transformer’s task is to
generate contextual word embeddings for both the text request and the item
descriptions. The two are then separated and passed as input in the two LSTMs
as mentioned above, producing the two corresponding feature vectors. The model
is trained using Cosine Embedding Loss

(1) $\displaystyle loss(x,y)=\begin{cases}1-cos(x_{1},x_{2}),\text{if
$y=1$}\\\ max(0,cos(x_{1},x_{2})),\text{if $y=-1$ }\end{cases}$

###  3.3. WhisperLite

WhisperLite is a lighter version of the Whisper model described in the
previous section, where the text embeddings are calculated through the pre-
trained CLIP text-only model (Radford et al., 2021) as opposed to XLNet.
Moreover, WhisperLite uses MLPs instead of bi-LSTM layers, and is trained
using a customized loss function, combining binary cross entropy loss for
classification

(2) $\displaystyle\mathcal{L}_{BCE}=-(ylog(\hat{y})+(1-y)log(1-\hat{y}))$

with a contrastive loss. In (2), $\hat{y}$ is the predicted probability of
observation $o$ being in class $c$, and $y$ represents whether $o$ actually
belongs in $c$.

Additionally, we exchanged the initial cosine similarity to the dot product
between the request and the product description vectors since the two metrics
are comparable (Luo et al., 2018; Naveen et al., 2021). The CLIP parameters
are held frozen, and only the MLPs are trained, significantly reducing the
training time of WhisperLite.

##  4\. Experiments

###  4.1. Compared models

We consider two baseline models to compare with our proposed approach.

  * •

A random baseline, where the (request,item) pairs are assigned a random
ranking.

  * •

OkapiBM25, a function that ranks the item description based on its estimated
IDF to the text request that the user provides.

Since the task addressed in this project deals with query-to-item
recommendations without additional information about the user, to the best of
our knowledge there are no other baselines that fit the problem exactly. As a
result, our current baselines for comparison are elementary, and adapting
existing work to suit our problem setting is an important direction for future
work.

###  4.2. Datasets and Evaluation

| Text requests |  | Item
---
descriptions
Number |  $4.5$M |  $4.5$M
Unique |  $245$k |  $218$k
Sentiment (pos-neutr-neg) |  $74.26$% - $17.42$% - $8.32$% | N/A
Word length (min-max-avg) |  $3$ \- $221$ \- $27.57$ |  $9$ \- $52$ \- $20.68$
Word length (50th-90th perc.) |  $23$ \- $54$ |  $20$ \- $25$
Table 1. Data analysis of the WhisperD dataset.

The main dataset used in this work, WhisperD, contains text requests from a
personalized online fashion recommendation system, collected from real
interactions between users and fashion stylists. We additionally consider
adaptations of three open source datasets to demonstrate the broad
applicability of our methodology. The statistics of the datasets are shown in
Table 2.

####  4.2.1. WhisperD

The text requests in the dataset contain both positive preferences, e.g.,
_Spring and summer wear_ , as well as negative ones, e.g., _No dresses,
skirts_ , and encompass a wide variety of writing styles. Alongside every
request in the dataset is also a collection of fashion stylists’ item
recommendations catering to the request. These item recommendations are
categorized in three ways: items that the user decided to try on (TRYs), items
that the user bought (KEEPs), and items that the user did not even try
(NOTTRYs). Table 1 describes the details of the WhisperD dataset. All of the
items associated with a user’s request are chosen by a fashion stylist, and
are therefore classified as positive examples. On the other hand, the negative
examples are sampled randomly from the dataset.

####  4.2.2. Open-source Datasets

The open source datasets are obtained through slight modifications of the
existing Yelp (restaurants domain) and Amazon (movies&TV and clothing)
datasets. Whereas the original datasets contain customer reviews with binary
sentiment annotations, we instead transform the datasets to match customer
reviews to item product descriptions (Wang et al., 2022; Kang et al., 2017),
aligning the datasets to the modality of WhisperD. With this transformation,
given a review and a set of possible targets, the task is to pair the review
with the correct target description. As described in (Wang et al., 2022; Kang
et al., 2017), the content expressed through reviews can be interpreted as
personal user preferences.

####  4.2.3. Evaluation

The evaluation measures used on the aforementioned datasets are the standard
in recommender systems literature. The main goal is to determine whether the
items retrieved by the model satisfy the user’s interest based on their
initial query.

  * •

Precision@$k$ measures the number of recommended items in the top-$k$ that are
relevant,

  * •

Recall@$k$ calculates the number of relevant products that are in the top-$k$
recommended ones,

  * •

Normalized Discounted Cumulative Gain (NDCG) measures the relevancy of the
item based on its position in the list of recommended products.

For precision@$k$ and recall@$k$ we chose $k\in\\{1,2,3,4\\}$ because of the
varying number of relevant items for example.

Tasks | Train | Dev | Test
---|---|---|---
WhisperD |  $1.58$M |  $197$k |  $197$k
Yelp |  $11.3$M |  $1.4$M |  $1.4$M
Amazon Clothing |  $2.22$M |  $278$k |  $278$k
Amazon Movies |  $13.52$M |  $1.69$M |  $1.69$M
Table 2. Statistics of all datasets used for this work.

###  4.3. Results

|  | PREC@1 | PREC@2 | PREC@3 | PREC@4 | REC@1 | REC@2 | REC@3 | REC@4 | NDCG
---|---|---|---|---|---|---|---|---|---|---
WhisperD | Random | 0.5010 | 0.4667 | 0.4200 | 0.3697 | 0.2968 | 0.5250 | 0.6730 | 0.7571 | 0.6958
| OkapiBM25 | 0.5044 | 0.5069 | 0.5067 | 0.5072 | 0.0321 | 0.0646 | 0.0968 | 0.1293 | 0.7864
| Whisper | 0.5376 | 0.4917 | 0.4343 | 0.377 | 0.3217 | 0.5515 | 0.6916 | 0.7678 | 0.712
| WLite | 0.7633 | 0.6294 | 0.5111 | 0.4172 | 0.4707 | 0.6884 | 0.7817 | 0.8188 | 0.8088
Yelp | Whisper | 0.8532 | 0.4316 | 0.289 | 0.2173 | 0.8473 | 0.8514 | 0.8524 | 0.8528 | 0.8533
| WLite | 0.8554 | 0.4377 | 0.2943 | 0.2218 | 0.8440 | 0.8527 | 0.8547 | 0.8555 | 0.8562
Amazon movies&TV | Whisper | 0.7695 | 0.3881 | 0.2596 | 0.1951 | 0.7656 | 0.7683 | 0.7689 | 0.7692 | 0.7696
| WLite | 0.7761 | 0.3953 | 0.2652 | 0.1996 | 0.7682 | 0.7746 | 0.7759 | 0.7765 | 0.7768
Amazon clothes, shoes & jewelry | Whisper | 0.8793 | 0.4516 | 0.3043 | 0.2298 | 0.8648 | 0.8743 | 0.8765 | 0.8776 | 0.8794
| WLite | 0.8719 | 0.4587 | 0.3121 | 0.2368 | 0.845 | 0.8637 | 0.8685 | 0.8705 | 0.8732

Table 3. Results table.

Table 3 shows the results of the experiments on the previously presented
datasets. The WhisperLite model (WLite) substantially outperforms the other
baselines on the WhisperD data on almost all metrics. There is also a
considerable gap between the Whisper and WhisperLite models, which only
appears in the results on WhisperD. One explanation is the WhisperD dataset
consists of noisier text requests and label when compared to public datasets.
Higher noise settings could lead to a great degree of overfitting in Whisper
model relative to the WhisperLite model, as the Whisper model optimizes over
feature vectors for the text requests by users whereas the WhisperLite model
does not. One possible interpretation of such a difference could be that a
model trained on generating embeddings for ‘generic’ text requests, such as ‘
_Looking to update my wardrobe_ ’ or ‘ _Going to Vegas next month_ ’, diverts
its ability to generate meaningful feature vectors for the description of the
clothes that are paired with such requests. Instead, WLite exploits the
feature vectors generated by the pre-trained CLIP model to train only the two
perceptrons with a classification objective function, which could explain the
meaningful difference in performance on the WhisperD data.

On the open-source datasets the two models perform very similarly with a
generally higher performance on all metrics compared to the results on
WhisperD, hinting at the difficulty of the clothing recommendation task. The
results on the Yelp and Amazon clothing datasets are similar, whereas the
performance on WhisperD is closer to the Amazon movies dataset.

###  4.4. Human Evaluation

The evaluation compared the WhisperLite model with the random baseline through
a user study on Amazon Mechanical Turk. A total of $1500$ data points are
randomly sampled from the outputs of the two respective models on the WhisperD
dataset and submitted to the participants of the study for evaluation. Each
example is associated with $k$ recommended items, where $k\in\\{3,5,7\\}$. In
order to increase accountability and discard possible noise, every data point
has been rated by $3$ unique workers.

The set up of this study is as follows: users are shown one text request and
$k$ recommended products associated with it. They then answer the question ”
_How relevant are these items based on the text request?_ ” on a likert-scale
from $1$ to $5$, where $1$ represents ‘Definitely irrelevant’, $5$ means
‘Definitely relevant’, and $3$ is left as the option for ‘Not sure’. The
studies for WhisperLite and random are designed to be separate and not as a
comparison between the two, with the purpose of avoiding an introduction of
bias on workers completing the tasks. Furthermore, users were required
specific qualifications to participate in this study and ensure an acceptable
level of quality to the results, namely: being master workers, having
completed and submitted at least $500$ HITs on MTurk with an acceptance rate
greater than $80\%$, and being located in an English-speaking country. Table 4
shows the impact of both systems on real-world users. We first classified
‘negative’ ratings such as $1$ and $2$ as ‘$-1$’, and ‘positive’ ratings like
$4$ and $5$ as ‘$+1$’, whereas $0$ has been assigned to ‘neutral’ judgements.

For all three values of $k$, where $k$ items are recommended given a specific
request, users preferred the recommendations made by the WhisperLite model,
with an average rating much higher than the recommendations from the random
baseline. The error bars computed on the average scores show the variability
of the collected judgements from human judges. Instead, the error bars for the
random baseline indicate that the human ratings are less reliable than the
ones collected for WLite, as they are bigger (or almost as big, for $k=7$)
than the mean values. This shows a net preference towards the items
recommended by WLite.

|  | Average
---|---|---
$k=3$ | Random | $0.31\pm 0.37$
| WLite | $\textbf{0.52}\pm 0.28$
$k=5$ | Random | $0.31\pm 0.38$
| WLite | $\textbf{0.72}\pm 0.24$
$k=7$ | Random | $0.38\pm 0.35$
| WLite | $\textbf{0.59}\pm 0.27$
Table 4. Analysis of the human evaluation of the random baseline and the WLite
model.

##  5\. Conclusion and Future Work

In this work we describe recommending fashion items for a customer using a
single text request. We propose two different approaches, Whisper and
WhisperLite, that each address this session-based recommendation problem.
While both approaches are competitive in several open source datasets spanning
multiple domains, WhisperLite significantly outperforms all other baselines on
the WhisperD dataset. We attribute the difference in model performance to the
higher level of noise in the WhisperD dataset, where fine-tuning lower,
language-model layers of model is highly sub-optimal, as it can confuse the
system and negatively influence the generation of the features for the
products (see Section 4.3). A meaningful extension of the proposed
recommendation approach is to consider several other features like images of
the clothing items or the body shape of the customer, in order to recommend
items that could complement specific body shapes. Possible features that can
also be included are user behavioral data from previous browsing sessions or
historical data. We leave exploring the impact of these additional features as
future work.

## References

  * (1)
  * Ajmani et al. (2013) Stuti Ajmani, Hiranmay Ghosh, Anupama Mallik, and Santanu Chaudhury. 2013.  An ontology based personalized garment recommendation system. In _2013 IEEE/WIC/ACM International Joint Conferences on Web Intelligence (WI) and Intelligent Agent Technologies (IAT)_ , Vol. 3. IEEE, 17–20.
  * Balabanović and Shoham (1997) Marko Balabanović and Yoav Shoham. 1997.  Fab: content-based, collaborative recommendation.  _Commun. ACM_ 40, 3 (1997), 66–72.
  * Basu et al. (1998) Chumki Basu, Haym Hirsh, William Cohen, et al. 1998\.  Recommendation as classification: Using social and content-based information in recommendation. In _Aaai/iaai_. 714–720.
  * Chakraborty et al. (2021) Samit Chakraborty, Md Hoque, Naimur Rahman Jeem, Manik Chandra Biswas, Deepayan Bardhan, Edgar Lobaton, et al. 2021\.  Fashion Recommendation Systems, Models and Methods: A Review. In _Informatics_ , Vol. 8. Multidisciplinary Digital Publishing Institute, 49\.
  * Chen et al. (2019) Xu Chen, Hanxiong Chen, Hongteng Xu, Yongfeng Zhang, Yixin Cao, Zheng Qin, and Hongyuan Zha. 2019.  Personalized fashion recommendation with visual explanations based on multimodal attention network: Towards visually explainable recommendation. In _Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval_. 765–774.
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_ (2018).
  * Herlocker et al. (2000) Jonathan L Herlocker, Joseph A Konstan, and John Riedl. 2000\.  Explaining collaborative filtering recommendations. In _Proceedings of the 2000 ACM conference on Computer supported cooperative work_. 241–250.
  * Hwangbo et al. (2018) Hyunwoo Hwangbo, Yang Sok Kim, and Kyung Jin Cha. 2018\.  Recommendation system development for fashion retail e-commerce.  _Electronic Commerce Research and Applications_ 28 (2018), 94–101.
  * Kang et al. (2017) Wang-Cheng Kang, Chen Fang, Zhaowen Wang, and Julian McAuley. 2017.  Visually-aware fashion recommendation and design with generative image models. In _2017 IEEE International Conference on Data Mining (ICDM)_. IEEE, 207–216.
  * Koren and Bell (2015) Yehuda Koren and Robert Bell. 2015.  Advances in collaborative filtering.  _Recommender systems handbook_ (2015), 77–118.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_ (2019).
  * Ludewig and Jannach (2018) Malte Ludewig and Dietmar Jannach. 2018.  Evaluation of session-based recommendation algorithms.  _User Modeling and User-Adapted Interaction_ 28, 4 (2018), 331–390.
  * Luo et al. (2018) Chunjie Luo, Jianfeng Zhan, Xiaohe Xue, Lei Wang, Rui Ren, and Qiang Yang. 2018\.  Cosine normalization: Using cosine similarity instead of dot product in neural networks. In _International Conference on Artificial Neural Networks_. Springer, 382–391.
  * Malon (2021) Christopher Malon. 2021\.  Overcoming Poor Word Embeddings with Word Definitions.  _arXiv preprint arXiv:2103.03842_ (2021).
  * Naveen et al. (2021) L Naveen, Ananda Raj, S Rajalakshmi, et al. 2021\.  Abstractive Text Summarizer: A Comparative Study on Dot Product Attention and Cosine Similarity. In _2021 Fourth International Conference on Electrical, Computer and Communication Technologies (ICECCT)_. IEEE, 1–8.
  * Pazzani and Billsus (2007) Michael J Pazzani and Daniel Billsus. 2007.  Content-based recommendation systems.  In _The adaptive web_. Springer, 325–341.
  * Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021\.  Learning transferable visual models from natural language supervision.  _arXiv preprint arXiv:2103.00020_ (2021).
  * Ricci et al. (2015) Francesco Ricci, Lior Rokach, and Bracha Shapira. 2015\.  Recommender systems: introduction and challenges.  In _Recommender systems handbook_. Springer, 1–34.
  * Schafer et al. (2007) J Ben Schafer, Dan Frankowski, Jon Herlocker, and Shilad Sen. 2007.  Collaborative filtering recommender systems.  In _The adaptive web_. Springer, 291–324.
  * Su and Khoshgoftaar (2009) Xiaoyuan Su and Taghi M Khoshgoftaar. 2009.  A survey of collaborative filtering techniques.  _Advances in artificial intelligence_ 2009 (2009).
  * Tu and Dong (2010) Qingqing Tu and Le Dong. 2010.  An intelligent personalized fashion recommendation system. In _2010 International Conference on Communications, Circuits and Systems (ICCCAS)_. IEEE, 479–485.
  * Wang et al. (2022) Shoujin Wang, Chenlu Yang, Tong Qu, Kai Yang, and Wanggen Wan. 2022.  Learning Users’ Visual Preferences for Improving Recommendations.  _IEEE Access_ (2022).
  * Yang et al. (2020) Ji Yang, Xinyang Yi, Derek Zhiyuan Cheng, Lichan Hong, Yang Li, Simon Xiaoming Wang, Taibai Xu, and Ed H Chi. 2020\.  Mixed negative sampling for learning two-tower neural networks in recommendations. In _Companion Proceedings of the Web Conference 2020_. 441–447.
  * Yang et al. (2019) Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019\.  Xlnet: Generalized autoregressive pretraining for language understanding.  _Advances in neural information processing systems_ 32 (2019).
  * Yin et al. (2019) Ruiping Yin, Kan Li, Jie Lu, and Guangquan Zhang. 2019\.  Enhancing fashion recommendation with visual compatibility relationship. In _The World Wide Web Conference_. 3434–3440.
  * Zhang et al. (2019) Shuai Zhang, Lina Yao, Aixin Sun, and Yi Tay. 2019\.  Deep learning based recommender system: A survey and new perspectives.  _ACM Computing Surveys (CSUR)_ 52, 1 (2019), 1–38.


# Unsupervised Band Selection Using Fused HSI and LiDAR Attention Integrating
With Autoencoder

Judy X Yang, , Jun Zhou, , Jing Wang, , Hui Tian, , and Alan Wee Chung Liew

###### Abstract

Band selection in hyperspectral imaging (HSI) is critical for optimising data
processing and enhancing analytical accuracy. Traditional approaches have
predominantly concentrated on analysing spectral and pixel characteristics
within individual bands independently. These approaches overlook the potential
benefits of integrating multiple data sources, such as Light Detection and
Ranging (LiDAR), and is further challenged by the limited availability of
labeled data in HSI processing, which represents a significant obstacle. To
address these challenges, this paper introduces a novel unsupervised band
selection framework that incorporates attention mechanisms and an Autoencoder
for reconstruction-based band selection. Our methodology distinctively
integrates HSI with LiDAR data through an attention score, using a
convolutional Autoencoder to process the combined feature mask. This fusion
effectively captures essential spatial and spectral features and reduces
redundancy in hyperspectral datasets. A comprehensive comparative analysis of
our innovative fused band selection approach is performed against existing
unsupervised band selection and fusion models. We used data sets such as
Houston 2013, Trento, and MUUFLE for our experiments. The results demonstrate
that our method achieves superior classification accuracy and significantly
outperforms existing models. This enhancement in HSI band selection,
facilitated by the incorporation of LiDAR features, underscores the
considerable advantages of integrating features from different sources.

###### Index Terms:

Hyperspectral image, LiDAR, band selection, attention mechanism, fused mask,
Autoencoder, CNN, latent space

##  I Introduction

Hyperspectral imaging technology captures intricate spectral responses of
terrestrial objects across numerous narrow spectral bands [1, 2], finding
widespread application in various real-world scenarios such as precision
agriculture [3, 4], environmental monitoring [1, 5], and the prediction of
earth disasters [6]. The primary goal of band selection in the field of
hyperspectral imaging is to identify and isolate a concise subset of
hyperspectral bands [7]. This selection process is essential for eliminating
spectral redundancy and reducing computational demands while preserving
essential spectral information pertinent to terrestrial objects, thereby
ensuring the integrity and usefulness of the spectral data [8, 9].

The spectral richness captured by hyperspectral imaging (HSI) systems
introduces a high-dimensionality and significant redundancy in HSI datasets,
leading to computational and operational challenges. These include the Hughes
phenomenon, characterised by a decline in classifier performance due to an
excessively high feature-to-sample ratio [10], along with considerable
computational burdens, and inefficiencies in data storage and transmission[7].
To address these challenges, hyperspectral band selection serves as a vital
preprocessing step. The purpose of this system is to extract the most
informative spectral bands, removing redundant or unnecessary data.

Extensive research in the field of hyperspectral imaging has identified
various methodologies for band selection, dividing these approaches into five
primary frameworks: ranking-based, search-based, clustering-based, deep
learning-based and hybrid strategies [11, 12, 13]. Each framework offers a
unique strategy to mitigate the challenges of high dimensionality and
redundancy inherent in HSI data, highlighting the crucial role of band
selection as a foundational preprocessing step. This step is essential to
enable more precise and computationally efficient remote sensing analysis
[14].

These methodologies can be further classified based on the nature of the
processed data—whether labeled or not—into three categories: supervised, semi-
supervised, and unsupervised methods [15, 16]. Supervised and semi-supervised
approaches rely on labeled data to perform the selection process. However,
acquiring labeled remote sensing data is often prohibitively expensive [17],
rendering supervised and semi-supervised methods impractical for widespread
application. Consequently, this research prioritizes an unsupervised approach
to band selection, using deep learning techniques to navigate the complexities
of hyperspectral data without the need for labeled datasets [18, 19].

Recent advances have demonstrated the potential of combining deep learning
with clustering or ranking methods for band selection in hyperspectral imaging
(HSI), resulting in significant improvements [20, 21]. The success of these
integrations can be attributed to the adaptability of deep learning algorithms
to identify and learn the intricate, nonlinear spatial and spectral
relationships within HSI data [22]. This innovative blend represents a
substantial advancement in the field and combines the power of deep learning
to refine band selection processes.

In the realm of recent studies on band selection, some researchers have
redefined the task as one of reconstruction [23, 24]. This perspective
advocates for the sparse reconstruction of Hyperspectral Imaging (HSI) bands
using a carefully chosen subset of informative bands [25]. This approach
diverges from traditional unsupervised band selection methods by focusing on
identifying bands that offer the highest level of informativeness and
dissimilarity [26]. This strategy reflects a shift towards viewing
hyperspectral image processing as a feature extraction [27, 28, 7] and
selection challenge. The emphasis on maximizing the informativeness and
dissimilarity of the selected bands is consistent with previous findings [29],
which highlight the importance of these criteria in improving the accuracy and
efficiency of hyperspectral image analysis [27, 28].

To use the complementary information provided by the LiDAR data, numerous
studies have been carried out [30, 31]. These include the introduction of
Progressive Extinction Mechanisms (PEM) for the fusion of Hyperspectral
Imaging (HSI) and LiDAR data [32, 33],aiming to extract both spatial and
elevation information. This approach facilitates the creation of functional
classification maps and improves piecewise smoothness while preserving spatial
structure. Furthermore, Falahatnejad and Karami [34] have successfully
integrated the spectral and spatial content of HSI with the height detection
information from LiDAR through a 2D Convolutional Neural Network (2DCNN)
architecture, yielding promising results in classification improvement.
Subsequent research has expanded on this foundation, employing attention-
guided [31] and coupled adversarial learning for fusion [35], thus enhancing
the classification accuracy of hyperspectral images and LiDAR data. However,
there is a noticeable gap in the literature regarding the combination of
hyperspectral image band selection with LiDAR, with the aim of classification
enhancement.

Motivated by unsupervised band selection methods and inspired by a Lidar
guided fusion model [36] for supervised band selection, we intended to develop
a unsupervised band selection deep learning model which can take the advantage
of LiDAR attention scroes to improve hyperspectral image classification
performance. THerefor a new unsupervised learning framework is proposed, named
the Dual-Attention-Based Fused Mask Autoencoder. This framework is tailored to
improve the selection of the hyperspectral band by integrating HSI and LiDAR
data. It utilizes a novel dual-attention mechanism to analyze and pinpoint the
most informative bands, exploring both the spectral and spatial dimensions of
the data. This mechanism is bifurcated into two components: one focusing on
the spectral correlations among the HSI bands and the other on the spatial
correlations indicated by the LiDAR data. By synthesizing attention masks
derived from both HSI and LiDAR inputs, one guarantees a holistic band
selection. The framework culminates in a clustering phase, steered by an
innovative distance metric, to isolate the most indicative subset of bands
effectively. This method promises to significantly enhance band selection,
thus optimising spectral and spatial analysis capabilities essential for
remote sensing applications.

In this paper, we outline our contributions as follows:

  * •

Innovative Framework Employing LiDAR Data for Enhanced Band Selection: We
introduce a novel architectural framework that leverages LiDAR data to enhance
the feature representation of Hyperspectral Imaging (HSI) bands. This was
achieved by creating a fusion mask that combines spatial information from
LiDAR with spectral data from HSI, providing a richer and more detailed
representation for further analysis.

  * •

Fusion of Attention Mechanism with Autoencoder for Efficient Representation:
Our architecture integrates an attention mechanism with an Autoencoder,
ingeniously designed to capture a compact yet highly informative
representation of the merged HSI and LiDAR data. A key component of this
architecture is a construction loss that includes a term to induce sparsity,
ensuring that the model focuses on the most critical features of both types of
data.

  * •

Customised Distance Metric for Superior Band Selection: We propose a unique
distance metric specifically tailored to measure dissimilarity and pinpoint
bands with the highest information content. This metric cleverly incorporates
attention scores into its calculations, reducing the effective distance for
bands with higher attention scores to prioritize their selection during
clustering.

  * •

Targeted Band Selection via Hierarchical Clustering: Utilizing hierarchical
clustering and our bespoke distance metric, our method precisely selects bands
that are not only individually significant due to high attention scores, but
also exhibit minimal similarity to each other. This deliberate selection
strategy ensures that the chosen bands offer a comprehensive and varied
feature set, crucial to the intended analytical objectives.

The remainder of this paper is organised as follows. Section II reviews
related work in the literature. Our method is described in Section III,
followed by the experiments and results presented in Section IV. Finally,
Section V concludes this article.

##  II Related Work

This section provides a comprehensive overview of the unsupervised band
selection methods and their integration with LiDAR data in hyperspectral image
processing models. These methods, crucial for efficiently reducing data
dimensionality while preserving essential information, are categorized into
ranking-based, clustering-based, attention-based, and fusion techniques. This
overview not only outlines the methods but also explores their integration
with Light Detection and Ranging (LiDAR) data to enhance HSI processing
models.

###  II-A Unsupervised Band selection

Unsupervised band selection is essential for managing the complexity of
hyperspectral data, categorically split into ranking based and clustering
based approaches. Ranking-based strategies assign a significance score to each
band, selecting those with the highest scores through various methods,
including information [7] , similarity [29], sparse representation [37], and
distance-based criteria. These methods aim to prioritize bands based on their
informational content, similarity to other bands, sparse coding techniques, or
their distance/dissimilarity from other bands [38] , thereby reducing
redundancy and enhancing discriminative power.

Clustering-based methods, in contrast, group bands into clusters, selecting
the most representative band from each to best represent data subspaces [39],
high-density regions [40], or contribute to a sparse non-negative
representation of the data. This approach streamlines data by ensuring a broad
yet concise representation through selected bands [41].

The ranking-based methods further delves into advanced techniques employing
deep learning to refine band selection. These include methods like Similarity-
Based Ranking with Structural Similarity (SSIM) [42], Contrastive Learning for
Band Selection (ContrastBS) [43], deep reinforcement learning [18], and Band
Grouping with Multigraph Adaptive Constraint [44]. Each method applies
innovative criteria and learning models to enhance selection efficiency,
reduce redundancy, and improve classification accuracy by focusing on
structural similarities, mutual information maximization, sequential selection
for reward maximization, and spectral correlation, respectively. Despite their
effectiveness in improving analysis efficiency and accuracy, these methods
also present challenges, notably the computational demands of detailed image
generation required by some approaches. This comprehensive overview sets the
stage for further exploration of these techniques, especially in how they
integrate with other technologies like LiDAR to push the boundaries of
hyperspectral image processing..

The examination of clustering based methods within hyperspectral image (HSI)
analysis reveals their critical role in mitigating the challenges of data
dimensionality and redundancy. By dividing the full spectrum of bands into
distinct clusters and selecting a representative band from each, these methods
streamline the process of hyperspectral imaging, ensuring that the essential
discriminative information is retained. Recent innovations have introduced a
cluster-based framework leveraging optimal transport theory [13], as described
in select studies. This framework excels in producing optimal clustering
outcomes within given constraints and incorporates a cluster classification
approach along with an automated system for determining the optimal band count
[45]. Such advancements signify a move towards preserving critical information
more efficiently.

In essence, the evolution of clustering-based methods in hyperspectral imaging
is marked by increasingly sophisticated algorithms and parameter optimization
strategies. These developments not only augment the precision and efficiency
of band selection but also bolster the ability to extract and conserve crucial
data, thus markedly advancing hyperspectral image analysis.

Partition-based strategies in hyperspectral band selection are emerging as
effective solutions to the challenges of high dimensionality and data
redundancy. These methods aim to divide the hyperspectral image cube into sub-
cubes, selecting bands that encapsulate the most informative and distinct
features. Strategies like the Adaptive Subspace Partition Strategy (ASPS) [46]
excel by maximizing the distance ratio between pixel classes, selecting
minimally noisy bands, and thus reducing redundancy while conserving
information.

Moreover, methods like RLFFC [13] introduce fusion-based clustering sensitive
to regional features, employing superpixel segmentation to maintain spatial
data integrity in HSIs. This approach enhances band separability and
redundancy capture, using k-means clustering for band selection and showing
notable improvements on public datasets. The Curve Fitting Subspace
Partitioning (CFSP) [47] method segments hyperspectral bands into subcubes
based on maximum curvature points, grouping similar and adjacent bands to
minimize correlation among selected bands. It achieves high classification
accuracy with minimal bands across several datasets, illustrating the
robustness and efficiency of partition-based methods in hyperspectral band
selection.

Attention-based techniques in hyperspectral imaging utilize attention
mechanisms to identify the most informative bands, offering a novel approach
to band selection. Methods like the Attention-based Autoencoder (AAE)
[48]focus on sparsity and global band interdependencies, employing attention
modules for band selection and clustering, albeit with challenges related to
computational demands and method clarity.

Another approach involves using clustering attention masks for band selection,
demonstrating the capacity of attention-based methods to handle complex and
non-linear band relationships, reducing redundancy, and enhancing post-
training efficiency despite longer GPU training times [49].

In summary, attention-based methods present a promising avenue for
hyperspectral band selection, capable of addressing intricate band
interdependencies. Nonetheless, they confront obstacles in computational
efficiency, parameter tuning, and the clarity of the attention mechanisms
employed.

###  II-B HSI and LiDAR Fusion Models

Moving forward with the fusion model, we would like to investigate these
unspervised fusion models and supervised fusion model based on their
architecture and label dat used.

Regarding the supervised HSI and LiDAR fusion model, in particular,
architectures based on convolutional neural networks (CNN) have been
instrumental in the fusion of HSI and LiDAR data, offering robust spatial
feature extraction capabilities [50]. Falahatnejad et al. [34] used hybrid
3D/2D CNNs and attention modules to extract spatial spectral characteristics
of HSI and representative elevation characteristics of LiDAR.EndNet [51], an
encoder-decoder fusion network, addresses the limitations of single-modal
remote sensing data and uses a reconstruction strategy to activate neurons in
both modalities.

Regarding unsupervised learning, the collaborative contrastive learning method
(CCL) [52] uses a pre-training and fine-tuning strategy to extract features
from HSI and LiDAR data separately and produce a coordinated representation
and matching of features between the two-modal data without label samples. The
FusAtNet model [53] fuses self-attention and cross-attention modules,
highlighting the harmonious integration of the HSI and LiDAR data. The
extracted features were then integrated and classified using a softmax
classifier. In [54], authors introduce semantic understanding to dynamically
fuse data from two different sources, extract features of HSI and LiDAR
through different branches of the capsule network, and improve self-supervised
loss and random rigid rotation in the canonical capsule to a high-dimensional
situation.

Deep learning and attention-based fusion techniques take advantage of the
strengths of both types of data, providing comprehensive insight that is
crucial for various applications. However, there is a noticeable gap, as
current approaches tend to use full HSI bands and LiDAR data without
performing any band selection.

In summary, attention-based methods are a promising direction for band
selection in hyperspectral images, as they can capture complex and nonlinear
relationships between bands and reduce redundancy. However, they also face
some challenges, such as computational efficiency, parameter tuning, and
interpretability of the attention mechanism.

After investigation, when multiresource data is available, fewer researchers
considered using more external features for the selection of hyperspectral
image bands. Therefore, we plan to propose multi-data unsupervised band
selection to improve the hyperspectral image data band selection and increase
the downstream task. A cluster integration-dual-attention-band selection
method is proposed for hyperspectral image band selection with the aid of
LiDAR data.

##  III Methodology

This section describes our proposed methodology for the selection of
unsupervised hyperspectral bands, integrating a dual attention mechanism with
an Autoencoder framework. Our methodology is structured around two core
components: the generation of attention masks from both HSI and LiDAR data,
and the subsequent application of these fused masks to the combined HSI and
LiDAR inputs in an Autoencoder for band selection. This approach
conceptualises band selection as a reconstruction challenge, aiming to retain
a representative subset of bands that closely mirror the original
hyperspectral data, once fused with LiDAR information.

###  III-A Dual Fused Mask Attention-Autoencoder Framework for Band Selection
Overview

Dual Fused Mask Attention-Autoencoder Framework for Band Selection. The
architecture of our framework, shown in Fig. 1, describes the original
hyperspectral data post-fusion with LiDAR, as depicted in the corresponding
flow chart. At the heart of this framework lies a dual Attention Autoencoder
(AAE) network designed to process the fused HSI and LiDAR data inputs
effectively. Using an unsupervised convolutional neural network (CNN)
paradigm, the Autoencoder strives to replicate its input within its output
closely, facilitating the selection of the most informative and minimally
redundant bands. This selection is refined through a custom distance function
that guides the hierarchical clustering method to isolate the top ${k}$ bands
of interest.

Figure 1:  Flowchart of the HSI band Selection

Our framework’s uniqueness lies in its dual-attention module, embedded at the
foundational level to discern the most salient bands within the dataset. This
module bifurcates into the HSI attention mechanism, focusing on extracting
nonlinear correlational attention among hyperspectral bands, and the LiDAR
attention mechanism, which unravels spatial correlations, enriching spectral
analysis with spatial context.

Following attention analysis, each mechanism generates a sparse mask, the HSI
mask, and the LiDAR mask, which are then concatenated. This combined mask
undergoes a dot product operation with the input of the attention module, the
result of which is fed into the Autoencoder. For the final band selection
phase, the fused attention masks are subjected to a clustering algorithm,
steered by a novel distance metric designed to act as the selection criterion.
This metric is critical to select a band subset that effectively encapsulates
both the spectral and spatial attributes.

To facilitate understanding, we introduce the specific notation used
throughout this discussion. Let ${(X_{hsi}\in\mathbb{R}^{H\times W\times B})}$
be an HSI with $(B)$ bands and ${H\times W}$ pixels,
$X_{lid}\in\mathbb{R}^{H\times W}$. In the data pre-processing step, a square
window of size $p\times p$ slides across the original HSI and LiDAR to obtain
a sufficient number of spatial spectral HSI patches
$X_{p}\in\mathbb{R}^{p*p,B}$ as input to the dual attention network. Suppose
that the sample data obtained by flattening each sample from the original HSI
band into a vector is denoted as
${X_{hsi2D}=[x_{hsi1},x_{hsi2},\ldots,x_{B}]\in\mathbb{R}^{P\times B}}$, where
$B$ represents the number of patches. The objective of our method is to select
the desired subset of bands consisting of $k$ bands, and a candidate band
subset can be expressed as
$X_{s}=[x_{s(1)},x_{s(2)},\ldots,x_{s(k)}]\in\mathbb{R}^{P\times k}$.

Through this methodology, our aim is to introduce a novel and efficient
approach to hyperspectral band selection, leveraging the complementary
strengths of HSI and LiDAR data to enhance the accuracy and efficiency of
remote sensing data analysis.

###  III-B Proposed Methods Implementation

Fig. 2 shows the detailed dual attention and Autoencoder architecture.

Figure 2: Detailed Architecture of HSI and LiDAR MASK fusion based on
Attention Module for HSI band Selection and Autoencoder Reconstruction.

Given the architecture that fuses the HSI and light detection and positioning
(LiDAR) data, the objective function of training is to find a sparse fused
mask${M_{fused}}$ from HSI and LiDAR, respectively,
${M_{hsi}\in\mathbb{R}^{P\times B}}$ and ${M_{lid}\in\mathbb{R}^{P\times 1}}$.
where $P$ is the size of the patch and $B$ is the number of spectral bands.
The model aims to learn a mapping function $f$,

The transformer has revolutionised data processing by using self-attention
mechanisms to model long-range dependency and capture features in data [55].
Patch embedding is created based on divided patches with positional encoding,
which can ensure the capture of the important feature of the context and
retain the band position during data processing. Taking advantage of these
advances, dual self-attention attention modules are designed to generate HSI
and LiDAR masks, respectively. Our dual-attention module for HSI and LiDAR
consists of four fully connected layers. The first three layers are followed
by ELU activation functions, and the last layer is followed by a Sigmoid
function; detailed structure is illustrated in 2. Once HSI and LiDAR attention
masks are obtained, the LiDAR mask is multiplied by the HSI mask to enhance
the HSI mask attention, and then the fused HSI mask is multiplied by the raw
HSI in an element-wise manner to be the input of the Autoencoder. Equation 1
is the fused mask.

$X_{hsi}=f(X_{hsi}\odot M_{hsi})$ (1)

where ${\odot}$ is the elemental multiplication.

The mapping function $f$ uses the CNN Autoencoder. The Autoencoder takes the
masked HSI $X_{hsi}$ as input and outputs the reconstructed HSI. The
Autoencoder consists of four layers with filter (3, 3), pooling (2,2), and ElU
activation function. According to the description above of the dual attention
module and the Autoencoder module, the attention module captures the
interdependencies between bands and can obtain the attention mask of one
sample and batch samples. Taking into account the fusion of HSI and LiDAR, the
objective function 2 can be updated as follows through model training.

$\begin{split}J(\theta)=\frac{1}{2}\left\|f\left((X_{\text{HSI}}\odot
M_{\text{fused}})\right.\right.)-X_{\text{HSI}}|_{2}^{2}+\lambda\left\|M_{\text{fused}}\right\|_{2,1}\end{split}$
(2)

where

  * •

$X_{\text{HSI}}$ and $X_{\text{LiDAR}}$ are the input data from the HSI and
LiDAR sensors, respectively.

  * •

$M_{\text{HSI}}$ and $M_{\text{LiDAR}}$ are the sparse masks for the HSI and
LiDAR data.

  * •

$\odot$ denotes element-wise multiplication.

  * •

$f$ is the mapping function learned by the network.

  * •

The term $\left\|\cdot\right\|_{2}^{2}$ denotes the squared Euclidean norm,
indicating the reconstruction loss.

  * •

The term $\left\|\cdot\right\|_{2,1}$ denotes the $L_{2,1}$ norm which
promotes sparsity in masks.

  * •

$\lambda$ is a regularization parameter that balances the accuracy and
sparsity of the mask reconstruction.

  * •

$M_{fused}$ is $M_{hsi}$ and $M_{lidar}$ by adding the operation, which is
defined as the square root of the sum of the squares of the L1 norms of the
rows of the matrix M, which can be used to promote the sparcity of the vector
of rows M during the training of the objective function.

$\left\|M_{fused}\right\|_{2,1}=\sqrt{\sum_{i=1}^{n}\|\mathbf{M}_{i,:}\|_{1}^{2}}$
(3)

###  III-C Custom Score Function for Band Selection

In this study, we introduce a new custom distance function as a key criterion
for band selection in hyperspectral imaging. This function is meticulously
designed to integrate three core components: the normalised attention scores
for each band (referred to as $A_{norm}$ in Equation 6, the dissimilarity
measure derived from the Pearson correlation matrix (defined in
$D_{dissimilarity}$, Equation 7). These components collectively facilitate a
multifaceted assessment of band importance, inter-band similarity, and their
joint contribution to the overall information content.

Initially, the function computes a distance matrix that encapsulates two
fundamental types of information: the similarity between bands, as indicated
by the Pearson correlation matrix, and the significance of each band, as
reflected by its normalised attention score. The distance associated with each
band is inversely proportional to its attention score, highlighting the
importance of high-attention bands. On the contrary, the distance is directly
proportional to the dissimilarity, which is computed as one minus the
correlation coefficient, highlighting the uniqueness of each band.

To adeptly balance the influence of band importance (attention) and
distinctiveness (dissimilarity), the distance measure integrates these aspects
using adjustable parameters $\alpha$ and $\beta$ .Here, $\alpha$ modulates the
weight given to the attention-based component, while $\beta$ adjusts the
emphasis on dissimilarity. This balanced approach is crucial to tailoring band
selection to both data-driven requirements and task-oriented relevance
criteria. The bands thus selected are expected to be highly informative for
subsequent analyses in hyperspectral imaging contexts, embodying an optimal
blend of data representation and attention-derived significance.

The equations governing the custom distance function are delineated below.

####  III-C1 Aggregate Attention Values

Compute the mean attention value in all samples and spatial locations for each
band, assuming the batch size is (N) and the sample patch size is (p). The
computing equation is as follows:

$A_{i}=\frac{1}{N\times p\times
p}\sum_{n=1}^{N}\sum_{x=1}^{p}\sum_{y=1}^{p}M_{n,x,y,i}$ (4)

This results in a vector $A$ of length $B$, where each $A_{i}$ is the
aggregated attention score for the $i$-th band of $B$.

####  III-C2 Normalize Attention Values

Normalize the aggregated attention scores to ensure that they are on a
comparable scale:

$A_{\text{norm}}=\frac{A_{i}-\min(A)}{\max(A)-\min(A)}$ (5)

####  III-C3 Attention Distance Matrix

Based on normalised attention values, define the attention distance matrix
$D_{\text{attention}}$ where each element $D_{\text{attention},i,j}$
represents the combined attention distance for bands $i$ and $j$. This
attention matrix is computed using the outer product of the attention vector
as below.

$D_{\text{attention},i,j}=A_{\text{norm},i}\times A_{\text{norm},j}$ (6)

The attention matrix captures the relative attention-based separation between
each pair of bands, with higher values indicating high attention.

####  III-C4 Dissimilarity Distance Matrix

Let $D_{\text{dissimilarity}}$ be the dissimilarity matrix where each element
$D_{\text{dissimilarity},i,j}$ is defined based on the Pearson correlation
between bands $i$ and $j$:

$D_{\text{dissimilarity},i,j}=1-\text{Corr}_{i,j}$ (7)

where $\text{Corr}_{i,j}$ is the Pearson correlation coefficient between bands
$i$ and $j$. Bands with high correlation may provide redundant information,
leading to lower dissimilarity values.

####  III-C5 Combined Distance Matrix for Clustering

To facilitate the clustering process, we define $D_{\text{combined}}$ as the
matrix that represents the combined distance. This matrix integrates both
spectral and spatial distances, enabling a more nuanced band selection
process. The calculation of the combined distance is as follows:

$D_{\text{combined},i,j}=\alpha\times D_{\text{attention},i}+\beta\times
D_{\text{dissimilarity},i,j}$ (8)

where $\alpha$ and $\beta$ are adjustable parameters that control the
influence of the distance of attention, the distance of dissimilarity and the
weight matrix, respectively. These parameters should sum up to 1, allowing
flexibility in the band selection process.

Subsequently, the band selection process involves performing hierarchical
clustering using $D_{\text{combined}}$ to group bands into clusters. A
subsequent algorithm, k-means, is then applied to select the top $k$ bands of
these clusters. For each cluster, the band with the highest attention score
norm $A_{\text{norm}}$ is selected. This ensures that the selected bands are
representative of the variability of the data set and are deemed the most
important by the attention mechanism.

##  IV Experiments

This section covers the data set used in our experiments, the experimental
setup, detailed results of the tests and an in-depth analysis of the ablation
study. This thorough examination aims to provide a holistic understanding of
our method’s performance and its practical implications in the field of
hyperspectral imaging.

###  IV-A Data sets Collection and Pre-processing

We have carried out extensive experiments to assess the performance of the
proposed cross-attention band selection method. The experiments were run on
three paired HSI and LiDAR datasets, Houston 2013, Trento, and MUUFL.

Houston 2013 dataset contains hyperspectral and LiDAR data for the IEEE 2013
GRSS Data Fusion [56]. It includes a hyperspectral image (HSI) with 144
spectral bands (380-1050nm) and a LiDAR-derived digital surface model (DSM),
both at 2.5 m resolution. The HSI is calibrated to the radiance of the sensor,
while the DSM indicates elevation in meters above sea level. The data set
contains 15 types of land cover, making it ideal for evaluating band selection
and classification methods, despite urban complexity and challenges related to
noise in the HSI. In our experiments, we follow the number of standard
training data to perform the final classification test and comparison.

The Trento data set [57] was acquired in Trento, Italy, in 2015. The
hyperspectral image comprises 48 spectral bands (400-950 nm) with a one-meter
spatial resolution, whereas the LiDAR data present the DSM. The data set
covers approximately 100 hectares and includes six land cover classes.

The MUUFL Gulfport scene[58]contains hyperspectral and LiDAR data collected
over Gulfport, Mississippi, in 2010 and 2011. The data set includes four
subimages with different spatial resolutions and elevations and ground truth
information. There are 11 classes in this data set. For the experiments
consistently, we used the standard test samples in the final classification
and co-comparative experiments.

Given that our band selection method operates in an unsupervised manner, it
does not incorporate ground-truth labels during the extraction of training
samples. To validate classification performance, we employ the following
approach in comparative analysis with alternative methods to assess its
efficacy.

###  IV-B Competitors and Experiment Configurations

Competitors are categorised into two groups. The methods selected for group
one are band selection algorithms. The other group chosen is for fusion
models.

####  IV-B1 Competitors of Band Selection Methods

In our study, we employ an optimally trained model to perform a comprehensive
comparative analysis of various state-of-the-art unsupervised hyperspectral
band selection methods. Each comparison method is implemented and evaluated
according to the specific descriptions and parameters outlined in their
respective original publications. This approach ensures a fair and consistent
comparison across all methods.

SR-SSIM [42] uses structural similarity indices to analyse band relationships.
ASPS [46] employs an adaptive subspace partition strategy to improve the
discriminability of the band. TOF [20]focuses on optimal clustering with a
unique rank-on-cluster strategy for band selection. RRI [49] estimates the
representativeness of the band through a convolutional Autoencoder, balancing
redundancy and information content. RLFFC [13] uses superpixel segmentation
and latent feature fusion for hyperspectral images, improves spatial and
spectral data, and uses clustering to reduce band redundancy. These
unsupervised band selection methods are included in our band selection
comparative study. These selected bands from different methods are integrated
with LiDAR to have a comprehensive evaluation of classification performance.

####  IV-B2 Competitors of Fusion Models

In recent years, deep learning has been applied to build fusion models in the
field of remote sensing. In this selection, we explore the influence of the
application of the BS method integrated with the fusion model. Both the SVM
and 2DCNN deep supervised learning classifier is integrated with our
unsupervised BS method to have a comparison with other fusion models.

The fusion models selected for this comparative study include supervised deep
learning methods, the CNN-HSI model proposed by Yu et al. (2017) [59] and
further developed by Mohla et al. (2020) [53] , the CoupledCNN model of Hang
et al. (2020) [60], and a general multimodal deep learning framework (MDL-RS)
including early fusion, middle fusion, and late fusion models explored by Hong
et al. (2020) [61], additional comparisons made with EndNet (2020) [51].
Regarding unsupervised learning, include the collaborative convolutional
learning model (CCL) presented by Jia et al. (2023) [52], FuseAtNet as
detailed by Mohla et al. (2020) [53].

These state-of-the-art fusion models typically operate on full-band HSI
hyperspectral image data combined with LiDAR. All experimental comparisons
adhere to the same test settings as those defined in their rearch. Instead,
our method uses only 10 HSI bands from the unsupervised method and fuses them
with the LiDAR data.

####  IV-B3 Experimental Configuration and Parameters Setting

To evaluate the performance of our proposed band selection method (BSFMAA) and
other competitors, we chose two classifiers, including SVM and KNN. In the
experiment, the parameter K of KNN classifier is set to 5, and rbf is the
kernel of the Radial Basis Function (RBF) used in SVM. All tests are performed
on Google Colab TPU.

In detail, we systematically assess the classification efficacy of our
proposed band selection method from multiple perspectives. Our proposed model
consists of dual attentions, fused masks from dual attention architectures,
and four-layer convolutional networks for the encoder and decoder,
respectively. Hyperparameters for the training model include Epoch 50, SGD
Optimizer, batch size 32, the learning rate defined as 0.0001, and Optimiser
is SGD.

For band selection evaluation, we use consistent classification tools, namely
the Support Vector Machine (SVM) and the K-Nearest Neighbour (KNN). These
classifiers are applied to the same ratio of training and test data for the
assessment of band selection methods.

For further fusion model evaluation, we use 10 selected bands that integrate
SVM and 2DCNN as our fusion methods to compare these state-of-the-art fusion
models.

####  IV-B4 Accuracy Assessment

Key metrics for this evaluation include classification accuracy and the number
of bands required. Higher accuracy and fewer bands are indicative of improved
discrimination power and a more efficient reduction in the dimensionality of
hyperspectral images. The band number ranges from 1 to 50 and the interval is
5. Three common indexes, namely overall precision (OA), average precision
(AA), and Kappa coefficient (Kappa), are used to have a qualitative comparison
of classification performance based on different band selection methods. All
test and training samples are the same as our proposed method and companions
for suitability.

###  IV-C Band Selection Experimental Results and Analysis

We conducted extensive experiments using three different data sets to identify
the model that performs optimally with specific configurations. Our approach
integrates a dual-attention mechanism with a fused mask. We tested various
patch sizes (3, 5, 7, 9, 11, and 13) and hyperparameters to determine their
impact on performance. The combination of a patch size of 7, a batch size of
32, the SGD optimizer, and a learning rate of 0.0001 yielded the best results.
To ensure fairness in our evaluation, we used the SVM and KNN classifiers and
5 layers of 1DCNN to assess classification performance. Our baseline
comparison involves using standard SVM and KNN algorithms with full-band HSI
and LiDAR data as input.

####  IV-C1 UH2013 Experiments

Table I shows the 10 bands OA, AA and Kappa of all methods in the Houston 2013
data set. The outcome based on the data set revealed that our method improved
overall precision (OA), average precision (AA) and Kappa metric by
approximately 2%, 1. 5% and 1. 5%, respectively, in the SVM evaluations. In
the KNN assessments, we observed an improvement 2% in both OA and AA. The
results underscore the effectiveness of our method, highlighting its superior
performance against competing approaches, as detailed in the table below,
which compares the results across various methods using the UH2013 dataset.
Regarding the CNN assessment, it still shows that our selected bands are in
the best place 5.

TABLE I: UH 2013 Three Classification Test Results Based on Fewer Band Number
10.
Blue Represents the Best method, Red Represents the second place model. Method | Metrics | BaseLine(H+L) | SR_SSIM [42] | ASPS [46] | TOF [20] | RRI_BS [49] | RLFFC [13] | OurMethod
---|---|---|---|---|---|---|---|---
SVM | OA | 0.8960 | 0.8598 | 0.8614 | 0.8282 | 0.7635 | 0.9007 | 0.9228
AA | 0.9073 | 0.8807 | 0.8776 | 0.8515 | 0.7992 | 0.9162 | 0.9308
Kappa | 0.8872 | 0.8479 | 0.8496 | 0.8138 | 0.7443 | 0.8923 | 0.9162
KNN | OA | 0.8852 | 0.8540 | 0.8475 | 0.8083 | 0.7920 | 0.8737 | 0.8986
AA | 0.8633 | 0.8355 | 0.8327 | 0.7991 | 0.7848 | 0.8495 | 0.8755
Kappa | 0.8753 | 0.8414 | 0.8345 | 0.7921 | 0.7746 | 0.8629 | 0.8898
CNN | OA | 0.9790 | 0.9341 | 0.9341 | 0.9447 | 0.9490 | 0.9775 | 0.9887
AA | 0.9806 | 0.9418 | 0.9418 | 0.9489 | 0.9587 | 0.9826 | 0.9912
Kappa | 0.9772 | 0.9284 | 0.9284 | 0.9399 | 0.9447 | 0.9755 | 0.9878
Figure 3: SVM OA Comparison-Based on UH2013 Data set using Different Band
Selection Methods. Figure 4: KNN OA Comparison-Based on UH2013 Data set using
Different Band Selection Methods. Figure 5: CNN OA Comparison-Based on UH2013
Data set using Different Band Selection Methods.

Figs. 3 and 4 depict classification performance across bands 1 to 50 (in
intervals of 5) further illustrate our method’s consistency and exceptional
performance when evaluated using both SVM and KNN classifiers.

####  IV-C2 Trento Experiments

Similarly to the UH2013 results, our experiments with the Trento data set show
the efficacy of our method in Table II. It outperforms others by 0. 4% in OA,
1. 4% in AA, and 0. 8% in Kappa when using SVM. KNN results were even more
pronounced, with improvements of 2. 5% in OA and Kappa and 1% in AA over the
next best method. This demonstrates our approach’s robustness and reliability
across different datasets and classification techniques. Although the CNN
result has a slight increase, some AA is not in the best place, in the band
range 5-20, our method is the best 11.

TABLE II: Trento Test Result Based on Fewer Band Number 10.
Blue Represents the Best method, Red Represents the second place model Method | Metrics | BaseLine(H+L) | SR_SSIM [42] | ASPS [46] | TOF [20] | RRI_BS [49] | RLFFC [13] | OurMethod
---|---|---|---|---|---|---|---|---
SVM | OA | 0.9820 | 0.9831 | 0.9813 | 0.9804 | 0.9201 | 0.9815 | 0.9867
AA | 0.9712 | 0.9861 | 0.9726 | 0.9716 | 0.8953 | 0.9730 | 0.9872
Kappa | 0.9759 | 0.9773 | 0.9748 | 0.9737 | 0.8907 | 0.9751 | 0.9821
KNN | OA | 0.9323 | 0.9431 | 0.9519 | 0.9544 | 0.9227 | 0.9481 | 0.9706
AA | 0.9192 | 0.9456 | 0.9517 | 0.9448 | 0.9283 | 0.9525 | 0.9612
Kappa | 0.9103 | 0.9246 | 0.9360 | 0.9393 | 0.8981 | 0.9312 | 0.9608
CNN | OA | 0.9950 | 0.9958 | 0.9944 | 0.9955 | 0.9830 | 0.9949 | 0.9962
AA | 0.9912 | 0.9935 | 0.9918 | 0.9901 | 0.9805 | 0.9920 | 0.9942
Kappa | 0.9933 | 0.9944 | 0.9926 | 0.9940 | 0.9772 | 0.9933 | 0.9949
Figure 6: SVM OA Comparison-Based on Trento Data set using Different Band
Selection Methods. Figure 7: KNN OA Comparison-Based on Trento Data set using
Different Band Selection Methods. Figure 8: KNN OA Comparison-Based on Trento
Data set using Different Band Selection Methods.

Fig. 6 and Fig. 7 shows that the test results of our proposed method have
superior performance and stability in the selected bands range from 1 to 50
interval 5. For the outcome of the SVM, KNN and CNN OA test, our method is in
the best place between 1 and 10, and between 15 and 20, our method is slightly
lower $RRI_{BS}$, and then our method is resumed in the first place compared
to other methods than in most cases. Visual comparisons from the Trento
dataset further affirm our method’s superior classification accuracy and
stability across selected bands, with our approach consistently leading in
performance metrics.

####  IV-C3 MUUFL Experiments

In the exploration of the MUUFL dataset, particularly focusing on
classification performance with a reduced set of five bands, our findings
highlight a notable improvement in classification metrics. Our method shown in
Table IIIdemonstrates an enhancement of 0. 5% to 1. 5% on the overall accuracy
(OA), average accuracy (AA), and Kappa metrics for both the SVM and KNN
classifiers over the second best performing methods.

TABLE III: MUUFL Test Result Based on Fewer Band Number 10.
Blue Represents the Best method, Red Represents the second place model Method | Metrics | BaseLine(H+L) | SR_SSIM [42] | ASPS [46] | TOF [20] | RRI_BS [49] | RLFFC [13] | OurMethod
---|---|---|---|---|---|---|---|---
SVM | OA | 0.8477 | 0.8459 | 0.8150 | 0.8365 | 0.7594 | 0.8451 | 0.8591
AA | 0.8678 | 0.8596 | 0.8554 | 0.8728 | 0.8188 | 0.8680 | 0.8766
Kappa | 0.8008 | 0.7814 | 0.7648 | 0.7916 | 0.6974 | 0.8016 | 0.8189
KNN | OA | 0.7695 | 0.7684 | 0.7993 | 0.8042 | 0.7371 | 0.8035 | 0.8097
AA | 0.8047 | 0.8186 | 0.8300 | 0.8337 | 0.7658 | 0.8328 | 0.8353
Kappa | 0.7096 | 0.7300 | 0.7460 | 0.7521 | 0.6702 | 0.7514 | 0.7586
CNN | OA | 0.9017 | 0.8974 | 0.8989 | 0.8971 | 0.8679 | 0.8878 | 0.9031
AA | 0.9185 | 0.9064 | 0.8875 | 0.8970 | 0.8757 | 0.9025 | 0.9026
Kappa | 0.8725 | 0.8666 | 0.8679 | 0.8449 | 0.8275 | 0.8546 | 0.8739
Figure 9: SVM OA Comparison-Based on MUUFL Data set using Different Band
Selection Methods. Figure 10: KNN OA Comparison-Based on MUUFL Data set using
Different Band Selection Methods. Figure 11: CNN OA Comparison-Based on MUUFL
Data set using Different Band Selection Methods.

Fig. 9, Fig. 10, and Fig 11 show that our proposed method is clearly better
than the other methods. For SVM, our approach can achieve the highest scores
in OA outperforming the nearest competitor. Similarly, in the KNN and CNN
evaluations, our method continued to lead, underscoring its robustness and
effectiveness in handling hyperspectral image classification with a limited
number of bands.

In summary, the experimental results of the three data sets demonstrate the
efficacy of our proposed band selection method. Through rigorous testing and
analysis, we have shown that our approach excels not only in enhancing
classification metrics but also in ensuring stability and reliability across
various classifier configurations and patch sizes.

###  IV-D Fusion Models Experimental Results and Comparison

This section delves into the evaluation of fusion models that incorporate our
optimally selected bands with the SVM and a 5-layer CNN, respectively, forming
a straightforward but effective fusion framework. We benchmark our models
against supervised and unsupervised fusion approaches, with the fusion model
introduced by Mercier et al. [62]. serving as the primary baseline for
comparison. The configurations of these selected fusion models adhere strictly
to the definitions outlined in their respective studies.

####  IV-D1 Fusion Comparison based on UH2013 Data Set

A detailed comparison between the base model, our SVM fusion model, and our
CNN model, integrating the selected 10 bands, is presented with a focus on 15
classes, overall accuracy (OA), average accuracy (AA), and Kappa coefficient.
The SVM fusion model excels in 3 classes (C1, C4, C15), while the CNN fusion
model leads in 7 classes (C3, C7, C8, C9, C10, C11, C12), showcasing the
efficacy and precision of our fusion strategies across a broad spectrum of
classes.

Table IV illustrates a side-by-side performance metric comparison,
highlighting where our methods (SVM and CNN fusion models) stand in relation
to other advanced fusion methodologies. This clear delineation showcases our
fusion models’ competitiveness, particularly emphasising instances where our
approaches outperform others, thereby reinforcing the effectiveness of our
band selection and fusion technique.

TABLE IV: CLASSIFICATION PERFORMANCE OF DIFFERENT METHODS ON THE HOUSTON2013
DATASET
Our Methods are based on 10 bands with SVM and CNN
Blue Represents the Best method, Red Represents the second place model Class No | CNN-HSI | CoupledCNNs | EndNet | FusAtNet | Middle-Fusion | CCL | OurMethod1 | OurMethod2
---|---|---|---|---|---|---|---|---
| [59] | [60] | [51] | [53] | [61] | [52] | SVM | CNN
C1 | 0.9156 | 0.8770 | 0.8616 | 0.8150 | 0.8789 | 0.8961 | 0.9800 | 0.9760
C2 | 0.9253 | 0.9211 | 0.8908 | 0.8125 | 0.9068 | 0.9013 | 0.8600 | 0.9240
C3 | 0.8869 | 0.9689 | 0.9980 | 0.9588 | 0.9980 | 0.9972 | 0.9500 | 1.0000
C4 | 0.9410 | 0.9774 | 0.9203 | 0.9202 | 0.9556 | 0.9868 | 1.0000 | 0.9770
C5 | 0.9913 | 0.9920 | 0.9685 | 0.8904 | 0.9919 | 0.9958 | 0.9900 | 0.9991
C6 | 0.8951 | 0.9114 | 0.8724 | 0.7559 | 0.8667 | 0.9203 | 0.9200 | 0.9860
C7 | 0.6912 | 0.8688 | 0.8013 | 0.7267 | 0.8527 | 0.9490 | 0.9400 | 0.9930
C8 | 0.5554 | 0.7387 | 0.7062 | 0.6543 | 0.7363 | 0.8806 | 0.8400 | 1.0000
C9 | 0.7275 | 0.7547 | 0.7262 | 0.6152 | 0.7030 | 0.8347 | 0.9400 | 0.9970
C10 | 0.8052 | 0.8270 | 0.6424 | 0.5933 | 0.7085 | 0.9054 | 0.7800 | 0.9900
C11 | 0.7113 | 0.9054 | 0.7701 | 0.7211 | 0.8206 | 0.9098 | 0.8400 | 0.9820
C12 | 0.6406 | 0.7934 | 0.6665 | 0.6814 | 0.7760 | 0.8365 | 0.9100 | 0.9770
C13 | 0.8045 | 0.9540 | 0.4680 | 0.7946 | 0.9388 | 0.9663 | 0.8700 | 0.9210
C14 | 0.9741 | 0.9967 | 0.9737 | 0.9694 | 0.9919 | 0.9878 | 0.9000 | 0.9840
C15 | 0.9989 | 0.9975 | 0.9832 | 0.9218 | 0.9988 | 0.9955 | 1.0000 | 0.9980
OA | 0.8123 | 0.8835 | 0.8098 | 0.7693 | 0.8576 | 0.9215 | 0.9226 | 0.9887
AA | 0.8309 | 0.8989 | 0.8166 | 0.7887 | 0.8758 | 0.9306 | 0.9421 | 0.9912
Kappa | 0.7973 | 0.8741 | 0.7944 | 0.7508 | 0.8461 | 0.9151 | 0.9328 | 0.9878

Our analysis reveals that our fusion models, particularly when evaluated on
OA, AA, and Kappa metrics, demonstrate superior performance over the second-
place contender, FuseAtNet.

####  IV-D2 Fusion comparison based on Trento Data Set

Table V illustrates that our SVM fusion model with 10 selected bands has a
class C4 in the first place, our CNN fusion model with 10 selected bands has
five classes, C1, C2, C4, C5 and C6 in the first place, FuseAtNet has 2
classes in the first dimension, 2 classes in the second dimension.

TABLE V: CLASSIFICATION PERFORMANCE OF DIFFERENT METHODS ON THE TRENTO DATASET
Our Methods are based on 10 bands with SVM and CNN
Blue Represents the Best method, Red Represents the second place model Class No | CNN-HSI | CoupledCNNs | EndNet | FusAtNet | Middle-Fusion | CCL | OurMethod1 | OurMethod2
---|---|---|---|---|---|---|---|---
| [59] | [60] | [51] | [53] | [61] | [52] | SVM | CNN
C1 | 0.9579 | 0.9849 | 0.6035 | 0.9709 | 0.9928 | 0.9949 | 0.9700 | 0.9990
C2 | 0.8655 | 0.9504 | 0.9468 | 0.9519 | 0.9835 | 0.9760 | 0.9600 | 0.9890
C3 | 0.9718 | 0.9548 | 0.8631 | 0.9049 | 0.9588 | 0.9753 | 0.9700 | 0.9490
C4 | 0.9843 | 0.9974 | 0.9868 | 0.9959 | 0.9997 | 1.0000 | 1.0000 | 1.0000
C5 | 0.9678 | 0.9991 | 0.8505 | 0.9906 | 0.9945 | 0.9977 | 0.9900 | 1.0000
C6 | 0.8314 | 0.9306 | 0.8642 | 0.9188 | 0.9282 | 0.9608 | 0.9800 | 0.9930
OA | 0.9230 | 0.9841 | 0.8694 | 0.9770 | 0.9873 | 0.9917 | 0.9867 | 0.9969
AA | 0.8601 | 0.9695 | 0.8525 | 0.9555 | 0.9763 | 0.9841 | 0.9872 | 0.9942
Kappa | 0.8971 | 0.9788 | 0.8255 | 0.9693 | 0.9831 | 0.9890 | 0.9821 | 0.9949

Regarding OA, AA and Kappa, our selected 10 bands fused with a 5 layers CNN
are at the first place.

####  IV-D3 Fusion Comparison based on MUUFLE Data Set

MUUFL data contain 11 classes. Similarly to UH2013 and Trento, we continue to
use 10 selected bands to integrate LiDAR data as the input of SVM and 5 layers
CNN to conduct experiments. Table VI illustrates that our SVM fusion model
with 10 selected bands has a class C1 in the first place and a class C9 in the
second place. The CNN fusion model has 1 class C8 in the first place and 4
classes at the second place.

TABLE VI: CLASSIFICATION PERFORMANCE OF DIFFERENT METHODS ON THE MUUFL DATASET
Our Methods are based on 10 bands with SVM and CNN
Blue Represents the Best method, Red Represents the second place model Class No | CNN-HSI | CoupledCNNs | EndNet | FusAtNet | Middle-Fusion | CCL | OurMethod1 | OurMethod2
---|---|---|---|---|---|---|---|---
| [59] | [60] | [51] | [53] | [61] | [52] | SVM | CNN
C1 | 0.7278 | 0.8149 | 0.7788 | 0.7349 | 0.7966 | 0.8795 | 0.9900 | 0.9840
C2 | 0.8098 | 0.6844 | 0.7067 | 0.5400 | 0.6424 | 0.7425 | 0.6900 | 0.7790
C3 | 0.6769 | 0.5659 | 0.7354 | 0.5176 | 0.6597 | 0.6264 | 0.8400 | 0.8770
C4 | 0.7126 | 0.8910 | 0.7539 | 0.6512 | 0.8062 | 0.9041 | 0.7300 | 0.8590
C5 | 0.7599 | 0.7861 | 0.8804 | 0.8380 | 0.8331 | 0.8009 | 0.9700 | 0.9840
C6 | 0.9674 | 1.0000 | 0.9923 | 0.9825 | 0.9982 | 1.0000 | 0.7100 | 0.6880
C7 | 0.9453 | 0.8935 | 0.8938 | 0.7269 | 0.9246 | 0.8125 | 0.5100 | 0.7790
C8 | 0.7375 | 0.8789 | 0.8815 | 0.8539 | 0.8829 | 0.8533 | 0.9400 | 0.9330
C9 | 0.5174 | 0.4741 | 0.5478 | 0.3939 | 0.4980 | 0.4792 | 0.6600 | 0.8860
C10 | 0.7984 | 0.6532 | 0.7637 | 0.4618 | 0.5653 | 0.6156 | 0.1500 | 0.7830
C11 | 0.7361 | 0.7625 | 0.7032 | 0.7498 | 0.8591 | 0.8655 | 0.8000 | 0.8610
OA | 0.7384 | 0.7743 | 0.7915 | 0.7076 | 0.7806 | 0.8111 | 0.8591 | 0.9031
AA | 0.7626 | 0.7640 | 0.7852 | 0.6773 | 0.7697 | 0.7800 | 0.8766 | 0.9026
Kappa | 0.6748 | 0.7133 | 0.7366 | 0.6337 | 0.7217 | 0.7562 | 0.8189 | 0.8739

As for OA, AA, and Kappa of the classification performance based on MUUFL data
set, although our method OA and are at the second place, the AA is at the
first place.

Three different datasets have validated our proposed method, showing that our
method consistently outperforms traditional and contemporary band selection
techniques. By utilising both spatial and spectral information, our method
adeptly navigates the complex data landscape, ensuring that the selected bands
are not only relevant, but also conducive to higher classification accuracy.

The experiments underscore the importance of feature separability in achieving
high classification performance. Future research could investigate the
development of more advanced metrics for evaluating feature separability,
beyond traditional distance criteria. Such metrics could account for the
complex interplay between bands and the multidimensional nature of
hyperspectral data, offering more nuanced assessments of band importance and
redundancy.

##  V Conclusions

In our study, we propose a novel unsupervised approach to band selection in
hyperspectral image (HSI) processing. Our method, distinct from conventional
techniques, incorporates an attention module within an Autoencoder
architecture, adeptly harnessing both the spatial and spectral attributes of
HSIs alongside the spatial features of LiDAR data. This strategy effectively
counters the issue of hyperspectral redundancy by selecting a concise yet
highly informative set of bands, thus optimising the efficiency of subsequent
analytical tasks and reducing computational overhead.

At the heart of our proposed solution is the generation of a focus-fused
feature mask synthesising input from both HSI and LiDAR data, which undergoes
processing via a convolutional Autoencoder. This innovative step ensures a
thorough encapsulation of spatial and spectral relevance within the imagery.
To facilitate the selection of informative bands, we devise a unique
combination distance metric, merging attention and dissimilarity distances,
which, alongside a clustering algorithm, aids in the meticulous selection of
band subsets.

Our evaluation spanned three comprehensive datasets: Houston 2013, Trento and
MUUFL, where our method demonstrated superior performance against existing
unsupervised hyperspectral band selection techniques in classification
accuracy. This evaluation, carried out through the implementation and
evaluation of each compared method according to its foundational
specifications, ensured a rigorous and equitable comparison.

The core of our proposed method lies in the generation of a focus-fused
feature mask derived from both HSI and LiDAR inputs, which is then processed
through a convolutional Autoencoder. This novel approach allows for a
comprehensive capture of both spatial and spectral significance within the
hyperspectral imagery. To efficiently select informative bands, we introduced
a custom combination distance comprising the attention distance and the
dissimilarity distance, which serves as the criterion for the selection of
small-band sets aided by a clustering algorithm.

Conclusively, our approach constitutes a significant contribution to the field
of hyperspectral imaging analysis, particularly beneficial in situations
marked by limited labeled data and the integration of auxiliary data types.
Although our results are encouraging, further exploration into incorporating
diverse auxiliary data, refining the attention mechanism, and broadening the
application spectrum of our methodology in remote sensing tasks remains a
promising direction for future research. Our method’s adaptability and
efficiency signify a substantial stride towards enhancing hyperspectral image
processing capabilities, heralding advancements in this rapidly progressing
area.

## References

  * [1] M. J. Khan, H. S. Khan, A. Yousaf, K. Khurshid, and A. Abbas, “Modern trends in hyperspectral image analysis: A review,” _IEEE Access_ , vol. 6, pp. 14 118–14 129, 2018.
  * [2] M. Ahmad, S. Shabbir, S. K. Roy, D. Hong, X. Wu, J. Yao, A. M. Khan, M. Mazzara, S. Distefano, and J. Chanussot, “Hyperspectral image classification—traditional to deep models: A survey for future prospects,” _IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing_ , vol. 15, pp. 968–999, 2022.
  * [3] M. Teke, H. S. Deveci, O. Haliloğlu, S. Z. Gürbüz, and U. Sakarya, “A short survey of hyperspectral remote sensing applications in agriculture,” in _2013 6th International Conference on Recent Advances in Space Technologies (RAST)_ , 2013, pp. 171–176.
  * [4] P. K. Sethy, C. Pandey, Y. K. Sahu, and S. K. Behera, “Hyperspectral imagery applications for precision agriculture-a systemic survey,” _Multimedia Tools and Applications_ , pp. 1–34, 2022.
  * [5] M. B. Stuart, A. J. McGonigle, and J. R. Willmott, “Hyperspectral imaging in environmental monitoring: A review of recent developments and technological advances in compact field deployable systems,” _Sensors_ , vol. 19, no. 14, p. 3071, 2019.
  * [6] S.-E. Qian, “Hyperspectral satellites, evolution, and development history,” _IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing_ , vol. 14, pp. 7032–7056, 2021.
  * [7] W. Sun and Q. Du, “Hyperspectral band selection: A review,” _IEEE Geoscience and Remote Sensing Magazine_ , vol. 7, no. 2, pp. 118–139, 2019.
  * [8] O. S. Chander Goud, T. H. Sarma, and C. S. Bindu, “Optimal band selection in hyperspectral images using improved k-means clustering with spectral similarity measures,” in _2023 IEEE International Conference on Artificial Intelligence in Engineering and Technology (IICAIET)_ , 2023, pp. 268–271.
  * [9] P. Ribalta Lorenzo, L. Tulczyjew, M. Marcinkiewicz, and J. Nalepa, “Hyperspectral band selection using attention-based convolutional neural networks,” _IEEE Access_ , vol. 8, pp. 42 384–42 403, 2020.
  * [10] M. J. Mashala, T. Dube, B. T. Mudereri, K. K. Ayisi, and M. R. Ramudzuli, “A systematic review on advancements in remote sensing for assessing and monitoring land use and land cover changes impacts on surface water resources in semi-arid tropical environments,” _Remote Sensing_ , vol. 15, no. 16, p. 3926, 2023.
  * [11] C. Yu, S. Zhou, M. Song, and C.-I. Chang, “Semisupervised hyperspectral band selection based on dual-constrained low-rank representation,” _IEEE Geoscience and Remote Sensing Letters_ , vol. 19, pp. 1–5, 2021.
  * [12] S. S. Sawant and P. Manoharan, “Unsupervised band selection based on weighted information entropy and 3d discrete cosine transform for hyperspectral image classification,” _International Journal of Remote Sensing_ , vol. 41, no. 10, pp. 3948–3969, 2020.
  * [13] J. Wang, C. Tang, Z. Li, X. Liu, W. Zhang, E. Zhu, and L. Wang, “Hyperspectral band selection via region-aware latent features fusion based clustering,” _Information Fusion_ , vol. 79, pp. 162–173, 2022.
  * [14] W. Sun and Q. Du, “Hyperspectral band selection: A review,” _IEEE Geoscience and Remote Sensing Magazine_ , vol. 7, no. 2, pp. 118–139, 2019.
  * [15] X. Bai, Z. Guo, Y. Wang, Z. Zhang, and J. Zhou, “Semisupervised hyperspectral band selection via spectral–spatial hypergraph model,” _IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing_ , vol. 8, no. 6, pp. 2774–2783, 2015.
  * [16] J. Feng, L. Jiao, F. Liu, T. Sun, and X. Zhang, “Mutual-information-based semi-supervised hyperspectral band selection with high discrimination, high information, and low redundancy,” _IEEE Transactions on Geoscience and Remote Sensing_ , vol. 53, no. 5, pp. 2956–2969, 2015.
  * [17] X. Wang, L. Chen, T. Ban, D. Lyu, Y. Guan, X. Wu, X. Zhou, and H. Chen, “Accurate label refinement from multiannotator of remote sensing data,” _IEEE Transactions on Geoscience and Remote Sensing_ , vol. 61, pp. 1–13, 2023.
  * [18] L. Mou, S. Saha, Y. Hua, F. Bovolo, L. Bruzzone, and X. X. Zhu, “Deep reinforcement learning for band selection in hyperspectral image classification,” _IEEE Transactions on Geoscience and Remote Sensing_ , vol. 60, pp. 1–14, 2022.
  * [19] M. Zeng, Y. Cai, Z. Cai, X. Liu, P. Hu, and J. Ku, “Unsupervised hyperspectral image band selection based on deep subspace clustering,” _IEEE Geoscience and Remote Sensing Letters_ , vol. 16, no. 12, pp. 1889–1893, 2019.
  * [20] Q. Wang, F. Zhang, and X. Li, “Optimal clustering framework for hyperspectral band selection,” _IEEE Transactions on Geoscience and Remote Sensing_ , vol. 56, no. 10, pp. 5910–5922, 2018.
  * [21] C. O. Ayna, R. Mdrafi, Q. Du, and A. C. Gurbuz, “Learning-based optimization of hyperspectral band selection for classification,” _Remote Sensing_ , vol. 15, no. 18, p. 4460, 2023.
  * [22] F. Feng, S. Wang, C. Wang, and J. Zhang, “Learning deep hierarchical spatial–spectral features for hyperspectral image classification based on residual 3d-2d cnn,” _Sensors_ , vol. 19, no. 23, p. 5276, 2019.
  * [23] F. Zhang, Q. Wang, and X. Li, “Optimal neighboring reconstruction for hyperspectral band selection,” in _IGARSS 2018 - 2018 IEEE International Geoscience and Remote Sensing Symposium_ , 2018, pp. 4709–4712.
  * [24] Q. Wang, F. Zhang, and X. Li, “Hyperspectral band selection via optimal neighborhood reconstruction,” _IEEE Transactions on Geoscience and Remote Sensing_ , vol. 58, no. 12, pp. 8465–8476, 2020.
  * [25] Y. Liu, X. Li, Z. Xu, and Z. Hua, “Bsformer: Transformer-based reconstruction network for hyperspectral band selection,” _IEEE Geoscience and Remote Sensing Letters_ , vol. 20, pp. 1–5, 2023.
  * [26] D. Bao, G. Tuxworth, and J. Zhou, “Similarity-based hyperspectral band selection using deep reinforcement learning,” in _2022 12th Workshop on Hyperspectral Imaging and Signal Processing: Evolution in Remote Sensing (WHISPERS)_ , 2022, pp. 1–5.
  * [27] S. Feng, Y. Itoh, M. Parente, and M. F. Duarte, “Hyperspectral band selection from statistical wavelet models,” _IEEE Transactions on Geoscience and Remote Sensing_ , vol. 55, no. 4, pp. 2111–2123, 2017.
  * [28] N. Jeenath Shafana, J. K T, and R. Divagar Iyyappan, “Optimal band selection and scale based feature selection for hyper spectral image classification using hybrid neural network,” in _2022 3rd International Conference on Smart Electronics and Communication (ICOSEC)_ , 2022, pp. 1515–1519.
  * [29] Q. Du and H. Yang, “Similarity-based unsupervised band selection for hyperspectral image analysis,” _IEEE Geoscience and Remote Sensing Letters_ , vol. 5, no. 4, pp. 564–568, 2008.
  * [30] M. Feng, F. Gao, J. Fang, and J. Dong, “Hyperspectral and lidar data classification based on linear self-attention,” in _2021 IEEE International Geoscience and Remote Sensing Symposium IGARSS_. IEEE, 2021, pp. 2401–2404.
  * [31] J. Huang, Y. Zhang, F. Yang, and L. Chai, “Attention-guided fusion and classification for hyperspectral and lidar data,” _Remote Sensing_ , vol. 16, no. 1, p. 94, 2023.
  * [32] B. Rasti, P. Ghamisi, and R. Gloaguen, “Hyperspectral and lidar fusion using extinction profiles and total variation component analysis,” _IEEE Transactions on Geoscience and Remote Sensing_ , vol. 55, no. 7, pp. 3997–4007, 2017.
  * [33] Y. Tang, S. Song, S. Gui, W. Chao, C. Cheng, and R. Qin, “Active and low-cost hyperspectral imaging for the spectral analysis of a low-light environment,” _Sensors_ , vol. 23, no. 3, p. 1437, 2023.
  * [34] S. Falahatnejad and A. Karami, “Deep fusion of hyperspectral and LiDAR images using attention-based CNN,” _SN Computer Science_ , vol. 4, no. 1, p. 1, 2022.
  * [35] T. Lu, K. Ding, W. Fu, S. Li, and A. Guo, “Coupled adversarial learning for fusion classification of hyperspectral and lidar data,” _Information Fusion_ , vol. 93, pp. 118–131, 2023.
  * [36] J. X. Yang, J. Zhou, J. Wang, H. Tian, and W. C. Liew, “Lidar-guided cross-attention fusion for hyperspectral band selection and image classification,” 2024, submitted on 5 April 2024. [Online]. Available: https://arxiv.org/abs/2404.03883
  * [37] S. Li and H. Qi, “Sparse representation based band selection for hyperspectral images,” in _2011 18th IEEE International Conference on Image Processing_ , 2011, pp. 2693–2696.
  * [38] G. Morales, J. Sheppard, R. Logan, and J. Shaw, “Hyperspectral band selection for multispectral image classification with convolutional networks,” in _2021 International Joint Conference on Neural Networks (IJCNN)_. IEEE, 2021, pp. 1–8.
  * [39] W. Sun, J. Peng, G. Yang, and Q. Du, “Fast and latent low-rank subspace clustering for hyperspectral band selection,” _IEEE Transactions on Geoscience and Remote Sensing_ , vol. 58, no. 6, pp. 3906–3915, 2020.
  * [40] X. Luo, R. Xue, and J. Yin, “Information-assisted density peak index for hyperspectral band selection,” _IEEE Geoscience and Remote Sensing Letters_ , vol. 14, no. 10, pp. 1870–1874, 2017.
  * [41] J.-m. Li and Y.-t. Qian, “Clustering-based hyperspectral band selection using sparse nonnegative matrix factorization,” _Journal of Zhejiang University SCIENCE C_ , vol. 12, no. 7, pp. 542–549, 2011.
  * [42] B. Xu, X. Li, W. Hou, Y. Wang, and Y. Wei, “A similarity-based ranking method for hyperspectral band selection,” _IEEE Transactions on Geoscience and Remote Sensing_ , vol. 59, no. 11, pp. 9585–9599, 2021.
  * [43] X. Li, Y. Liu, Z. Hua, and S. Chen, “An unsupervised band selection method via contrastive learning for hyperspectral images,” _Remote Sensing_ , vol. 15, no. 23, p. 5495, 2023.
  * [44] M. You, X. Meng, Y. Wang, H. Jin, C. Zhai, and A. Yuan, “Hyperspectral band selection via band grouping and adaptive multi-graph constraint,” _Remote Sensing_ , vol. 14, no. 17, p. 4379, 2022.
  * [45] M. Caron, I. Misra, J. Mairal, P. Goyal, P. Bojanowski, and A. Joulin, “Unsupervised learning of visual features by contrasting cluster assignments,” _ArXiv_ , vol. abs/2006.09882, 2020. [Online]. Available: https://api.semanticscholar.org/CorpusID:219721240
  * [46] Q. Wang, Q. Li, and X. Li, “Hyperspectral band selection via adaptive subspace partition strategy,” _IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing_ , vol. 12, no. 12, pp. 4940–4950, 2019.
  * [47] S. Li, Z. Wang, L. Fang, and Q. Li, “An efficient subspace partition method using curve fitting for hyperspectral band selection,” _IEEE Geoscience and Remote Sensing Letters_ , vol. 21, pp. 1–5, 2024.
  * [48] Z. Dou, K. Gao, X. Zhang, H. Wang, and L. Han, “Band selection of hyperspectral images using attention-based autoencoders,” _IEEE Geoscience and Remote Sensing Letters_ , vol. 18, no. 1, pp. 147–151, 2020.
  * [49] Y. Liu, X. Li, Z. Hua, C. Xia, and L. Zhao, “A band selection method with masked convolutional autoencoder for hyperspectral image,” _IEEE Geoscience and Remote Sensing Letters_ , vol. 19, pp. 1–5, 2022.
  * [50] R. Hang, Z. Li, P. Ghamisi, D. Hong, G. Xia, and Q. Liu, “Classification of hyperspectral and LiDAR data using coupled CNNs,” _IEEE Transactions on Geoscience and Remote Sensing_ , vol. 58, no. 7, pp. 4939–4950, 2020.
  * [51] D. Hong, L. Gao, R. Hang, B. Zhang, and J. Chanussot, “Deep encoder-decoder networks for classification of hyperspectral and LiDAR data,” _IEEE Geoscience and Remote Sensing Letters_ , vol. 19, pp. 1–5, 2020.
  * [52] S. Jia, X. Zhou, S. Jiang, and R. He, “Collaborative contrastive learning for hyperspectral and LiDAR classification,” _IEEE Transactions on Geoscience and Remote Sensing_ , vol. 61, pp. 1–14, 2023.
  * [53] S. Mohla, S. Pande, B. Banerjee, and S. Chaudhuri, “FusAtNet: Dual attention based spectrospatial multimodal fusion network for hyperspectral and LiDAR classification,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops_ , 2020, pp. 92–93.
  * [54] Q. Xu, Y. Tang, and Y. She, “Unsupervised multi-branch capsule for hyperspectral and lidar classification,” in _Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series_ , vol. 12617, 2023, p. 126170A.
  * [55] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” _Advances in neural information processing systems_ , vol. 30, 2017.
  * [56] X. Zhou, M. Cui, A. Singhania, and D. Fernández Díaz, Juan Carlos, “Houston 2013 data set,” National Center for Airborne Laser Mapping (NCALM), University of Houston, Jun. 2013, dataset collected over the University of Houston campus and pre-processed with the assistance of the authors. [Online]. Available: https://hyperspectral.ee.uh.edu/?page_id=459
  * [57] University of Trento, “Theses of the university of trento,” [Data set], 2022, original work published 2020. [Online]. Available: http://data.europa.eu/88u/dataset/theses-of-the-university-of-trento
  * [58] X. Du and A. Zare, “Technical report: Scene label ground truth map for MUUFL Gulfport data set,” University of Florida, Gainesville, FL, Tech. Rep. 20170417, Apr. 2017, available: http://ufdc.ufl.edu/IR00009711/00001.
  * [59] S. Yu, S. Jia, and C. Xu, “Convolutional neural networks for hyperspectral image classification,” _Neurocomputing_ , vol. 219, pp. 88–98, 2017.
  * [60] R. Hang, Z. Li, P. Ghamisi, D. Hong, G. Xia, and Q. Liu, “Classification of hyperspectral and LiDAR data using coupled CNNs,” _IEEE Transactions on Geoscience and Remote Sensing_ , vol. 58, no. 7, pp. 4939–4950, 2020.
  * [61] D. Hong, L. Gao, N. Yokoya, J. Yao, J. Chanussot, Q. Du, and B. Zhang, “More diverse means better: Multimodal deep learning meets remote-sensing imagery classification,” _IEEE Transactions on Geoscience and Remote Sensing_ , vol. 59, no. 5, pp. 4340–4354, 2020.
  * [62] G. Mercier and M. Lennon, “Support vector machines for hyperspectral image classification with spectral-based kernels,” in _IGARSS 2003. 2003 IEEE International Geoscience and Remote Sensing Symposium. Proceedings (IEEE Cat. No. 03CH37477)_ , vol. 1. IEEE, 2003, pp. 288–290.