Sharpend Cosine Similarity based Neural Network for Hyperspectral Image
  Classification
Hyperspectral Image Classification (HSIC) is a difficult task due to high
inter and intra-class similarity and variability, nested regions, and
overlapping. 2D Convolutional Neural Networks (CNN) emerged as a viable network
whereas, 3D CNNs are a better alternative due to accurate classification.
However, 3D CNNs are highly computationally complex due to their volume and
spectral dimensions. Moreover, down-sampling and hierarchical filtering (high
frequency) i.e., texture features need to be smoothed during the forward pass
which is crucial for accurate HSIC. Furthermore, CNN requires tons of tuning
parameters which increases the training time. Therefore, to overcome the
aforesaid issues, Sharpened Cosine Similarity (SCS) concept as an alternative
to convolutions in a Neural Network for HSIC is introduced. SCS is
exceptionally parameter efficient due to skipping the non-linear activation
layers, normalization, and dropout after the SCS layer. Use of MaxAbsPool
instead of MaxPool which selects the element with the highest magnitude of
activity, even if it's negative. Experimental results on publicly available HSI
datasets proved the performance of SCS as compared to the convolutions in
Neural Networks.
Muhammad Ahmad
# Sharpenal Cosine Similarity based Neural Network for Hyperspectral Image Classification

###### Abstract

Hyperspectral Image Classification (HSIC) is a difficult task due to high inter and intra-class similarity and variability, nested regions, and overlapping. 2D Convolutional Neural Networks (CNN) emerged as a viable network whereas, 3D CNNs are a better alternative due to accurate classification. However, 3D CNNs are highly computationally complex due to their volume and spectral dimensions. Moreover, down-sampling and hierarchical filtering (high frequency) i.e., texture features need to be smoothed during the forward pass which is crucial for accurate HSIC. Furthermore, CNN requires tons of tuning parameters which increases the training time. Therefore, to overcome the aforesaid issues, Sharpened Cosine Similarity (SCS) concept as an alternative to convolutions in a Neural Network for HSIC is introduced. SCS is exceptionally parameter efficient due to skipping the non-linear activation layers, normalization, and dropout after the SCS layer. Use of MaxAbsPool instead of MaxPool which selects the element with the highest magnitude of activity, even if it's negative. Experimental results on publicly available HSI datasets proved the performance of SCS as compared to the convolutions in Neural Networks.

 Cosine Similarity; Convolutional Neural Network (CNN); Hyperspectral Images Classification (HSIC).

## I Introduction

Hyperspectral Image Classification (HSIC) has been an extensively studied area of research for decades for many applications [1, 2, 3, 4, 5]. HSIC task requires assigning a unique class label to a set of pixels according to the information presented in HSI [6, 7, 8]. Early HSIC approaches are based on handcrafted features which are extracted from spectral (color, intensity, etc.) or spatial (texture, shape, etc.) information or their combination. Among the handcrafted features, HoG, texture, and color features are widely employed. However, the classification performance largely deteriorated since these approaches are not able to extract rich semantic features from HSI.

To overcome the aforesaid limitations, several unsupervised feature-learning techniques have been proposed. Such techniques focus on learning a set of basis functions used for feature encoding for an image [6]. These approaches can learn more discriminative information thus suitable for representing the information. However, the discriminative performance is still limited as these techniques do not make use of class information, hence, do not well represent distinguishable features among different classes [9].

To overcome the aforesaid limitations, deep learning approaches have been proposed, especially Convolutional Neural Network (CNN) [10, 11, 12, 13]. CNNs are able to use class information to learn discriminative features, thus the features learned by CNN are more robust and discriminative for HSIC. However, due to the down-sampling and hierarchical filtering processes, the high frequency (i.e., texture features) gradually needs to be smoothened during the forward processes [14]. Thus, to some extent, CNN architecture can be considered a low-frequency pathway for HSIC. Moreover, traditional CNN models require tons of parameters to train the model which increases the training time. Therefore, to overcome the aforesaid issues, this work proposed the use of the Sharpened Cosine Similarity (SCS) concept as an alternative to convolutions in a neural network. More specifically, the following benefits can be obtained using SCS instead of convolutions in neural networks. SCS appears to be parameter efficiency and architecture simplicity due to the nonexistence of nonlinear activation layers, dropout, and normalization layers, like batch normalization or layer normalization, after SCS layers. Use AbsMaxpool instead of Maxpool. It selects the element with the highest magnitude of activity, even if it's negative.

## II Literature Review

High-level earth observation using HSI has become cutting-edge research. Hence HSIC has drawn widespread attention and brought several state-of-the-art methodologies since its emergence [15]. However, most of the early approaches only considered shallow processing which requires feature engineering prior to the classification. Moreover, shallow-level processing requires extensive parameter tuning, settings, and experience which affects the learning capabilities. Therefore, during the past decade, deep learning such as CNNs and similar methodologies have been proposed.

The works [16, 17] proposed an HSIC method based on deep pyramid residual and capsule networks. Similarly, the work [18] proposed a spectral-spatial residual network in which the residual blocks were used for identity mapping to connect 3D CLs whereas the work [19] proposed an unsupervised HSIC method to better explore the residual CLs. The work [20] proposed a 3D CNN network for both spatial-spectral feature learning and classification. The work [21] proposed a mini-batch Graph CNN (miniGCN) which addresses the complexity of graph computation and feature fusion extracted by CNN and miniGCN as an end-to-end network.

From the above discussion, one can conclude that features learned by CNN are more robust and discriminative for HSIC as compared to the other methods. However, due to the down-sampling and hierarchical filtering process, the high-frequency features gradually need to be smoothened during the forward
pass. To some extent, these features are considered crucial for accurate classification, therefore, CNN architecture is referred to as a low-frequency pathway. Moreover, parameter explosion is yet another issue that increases the computational time. To overcome the aforesaid issues, this work implemented an SCS as an alternative to Convolutional Layers (CLs) for HSIC. There are a number of benefits that can be obtained using SCS instead of CLs in neural networks.

For instance, SCS appears to be parameter efficient, and its architecture simplicity. It's not about the accuracy records but it's killing in parameter efficiency leaderboard. Skip the non-linear activation layers, dropout layers, and normalization layers after SCS. SCS uses Abs Max-pool instead of Max-pool which selects the element with the highest magnitude of activity even if it's negative. Thus, in a nutshell, the SCS works better in terms of the computational cost requires to train a DNN as compared to deep CNNs.

## III Methodology

Let us consider an HSI \(\textbf{X}\in\mathcal{R}^{(M\times N)\times B}\) where \((M\times N)\) represents the spatial region over the Earth's surface and \(B\) refers to be the number of spectral bands exists in HSI. All \(x_{ij}\in\textbf{X}\) pixels of HSI are classified into \(C\) disjoint land-cover classes denoted as \(Y=(y_{1},y_{2},\ldots,y_{n})\). Each \(x_{ij}\in\textbf{X}\) (where \(i=1,2,\ldots,M\) and \(j=1,2,\ldots,N\)) define a land-cover pixel as a spectral vector \(x_{ij}=[x_{i,j,1},x_{i,j,2},\ldots,x_{i,j,B}]\in\textbf{X}\) contain \(B\) number of values.

Moreover, to process the spatial information, the patch extraction process is carried out as a preprocessing step where the HSI cube \(\textbf{x}_{i,j}\in\mathcal{R}^{(k\times k)\times B}\) with the neighboring regions of size \(k\times k\) centered at targeted pixel \((i,j)\). joint spectral-spatial features can increase the discriminative power of any model, thus the spectral-spatial cubes \(\textbf{x}_{i,j}\in\mathcal{R}^{(k\times k)\times B}\) are extracted from raw data and stacked into \(X\) before feature extraction. Finally, the training and test samples were selected for each class.

Convolution in Neural Network is a sliding dot product between the patch of the image and the kernel aligned which may not be a good similarity metric and thus may skip some important information which makes it an inappropriate feature detector. However, normalizing both to a magnitude of 1, thus it will turn to cosine similarity. The cosine of two non-zero vectors can be derived by using the Euclidean distance as \(k\cdot x_{i}=||k||\ ||x_{i}||cos\theta\), given \(k\) and \(x_{i}\), the cosine similarity \(cos(\theta)\) can be computed as \(cos(\theta)=\frac{k\cdot x_{i}}{||k||\ ||x_{i}||}=\frac{\sum_{i=1}^{n}k_{i}\ x_{i}}{ \sqrt{\sum_{i=1}^{n}k_{i}^{2}\sqrt{\sum_{i=1}^{n}x_{i}^{2}}}}\).

The resulting similarity ranges from \(-1\) to \(1\), where \(-1\) means that the kernel and the image patch are exactly opposite and \(0.5\) means intermediate similarity or dissimilarity among the kernel and image, whereas, \(1\) means that the kernel and the patch are aligned. The main issue in cosine similarity is a very small magnitude between the kernel and the image patch which in principle desired to be magnitude in variance. However, keeping it to an extreme may end up gathering background or noise rather than foreground information. Thus, extra parameters can help to overcome the aforesaid issues.

Therefore, the SCS was initiated, which is a stridden operation similar to convolution that extracts features from an image patch. SCS is similar to the convolutional process, however, have some important differences. In practice, the convolution is a stridden dot product among the kernel \(k\) and the image patch \(x_{i}\) as \(k\cdot x_{i}\), whereas in cosine similarity, the image patch and kernel are normalized to have a magnitude of 1 before the dot product is taken. It is so named because, in two dimensions, it gives the cosine of the angle between the image patch and the kernel as \(\frac{k\cdot x_{i}}{||k||\ ||x_{j}||}\). Therefore, the cosine can be sharpened by raising the magnitude of the result to a power \(p\) while maintaining the sign as \(sign(k\cdot x_{i})\left|\frac{k\cdot x_{i}}{||k||\ ||x_{i}||}\right|^{p}\). This measure can become numerically unstable if ever the magnitude of the image patch or kernel gets too close to zero. In practice, the kernel magnitude doesn't get too small and doesn't need \(q\), however, maybe a good practice to have a small constraint on it as well, i.e., \(sign(k\cdot x_{i})\left|\frac{k\cdot x_{i}}{||k||+q||\ ||x_{i}||}\right|^{p}\).

\[SCS(k,x_{i})=sign(k\cdot x_{i})\bigg{(}\frac{k\cdot x_{i}}{(||k||+q)(||x_{i} +q||)}\bigg{)} \tag{1}\]

Similar to the traditional convolutional process in Deep Learning, SCS is also a stride operation that takes features as an output from an input image patch. However, before calculating the straight dot product, the kernel and image patch are adjusted to have a magnitude of 1 that produce the sharpened cosine similarity or some literature refers to it as sharpened cosine normalization. SCS operation extracts feature from image patches for which sharpening exponents must be learned for each unit which can be sharpened by raising it to a power of an exponent. The aforementioned procedure outperformed and is faster than a convolutional process in any neural network by 10 to 100 times due to the fewer number of required parameters, without the normalization and activation function.

Apart from the SCS process, as compared to the pooling that is used to downsample the data, absolute max-pooling is used to update the filter in backpropagation until an appropriate value of features is obtained from the given patch. Moreover, as compared to simple pooling, absolute maxpooling chooses the highest magnitude even if a certain element has a negative value.

The overall model is trained using 250 epochs, 256 batch sizes, and a 0.001 learning rate. The learning rate has a strong influence on the pace of learning any Deep Model that determines the number of movements required to minimize the loss of function value whereas the momentum is used to enhance the accuracy and model's training speed. The RMS and momentum prop optimizer combined model is trained using the Adam optimizer. TensorFlow-based Keras inner library is called to fine-tune the hyperparameters for SCS-Net which ultimately makes the overall model robust and comparative as compared to the other state-of-the-art models. The motivation behind using the Adam optimizer is its efficiency in consuming less memory and computational efficiency. The complete architecture is presented in Figure 1.
## IV Experimental Results and Discussion

This section discusses the experimental datasets used to evaluate the proposed pipeline along with the comparative methods proposed in recent years. Moreover, this section presents the details regarding the experimental settings, results, and discussion together with the experimental results as compared to the state-of-the-art methods proposed in recent years. For experimental evaluation, two publicly available HSI datasets have been used. These datasets were acquired at different times, locations, and sensors.

The works published in the literature present comprehensive results to highlight the pros and cons as compared to state-of-the-art works however, to some extent, all the works may have different experimental settings such as, the number or percentage of training, validation, and test samples may remain the same but their geographical locations may differ (due to the random selection nature) as these methods may have run in different times or may have executed on different machines. Therefore, to make the comparison fair among different works, one must need to have the same number/percentage and geographical locations of training, validation, and test samples. Thus, in this work, the first experimental settings, i.e., the percentage of training, validation, and test samples along with their geographical locations remain the same as all the comparative methods along with the SCS pipeline are executed in one single run.

The experimental results presented in this section have been obtained using Google Colab with a Graphical Process Unit with 358\(+\) GB of storage and \(25GB\) of RAM. For all the experimental results presented in this section the training, validation, and test samples are randomly selected as 40%/30%/30%. For fair comparative analysis, all three models have been executed at once with one-time randomly selected samples. The presented results are obtained using \(15\times 15\) patch size along with the \(15\) most informative bands selected through PCA. As far as the training parameters for 3D CNN and Hybrid models, the weights are initially randomized and later optimized using backpropagation with Adam optimizer and softmax loss function. The overall training and validation loss and accuracy for all three models are presented in Figure 2.

The experimental results presented in Figure 3 and Table I for both IP and SA datasets are obtained using the overall, average accuracy, and Kappa coefficient. Kappa computes the mutual information regarding a strong agreement among the classification and ground-truth maps whereas Average and Overall accuracy compute the average class-wise classification performance and the correctly classified samples out of the total test samples, respectively.
easy. Therefore, all the comparative methods must need to be evaluated on the same set of samples rather than different samples. In this work, the training and test samples are selected once only and used to train and test all the models for the same samples.

Here the SCS method is compared with theal-layered Perceptron (MLP) [22], Multinomial Logistic Regression (MLR) [23], Random Forest (RF) [24], Support Vector Machine (SVM) [25], 1D CNN [11], 2D CNN [26], 3D CNN [13], Hybrid CNN [27], Bayesian CNN (1D, 2D, and 3D BCNN) [12]. All these methods were implemented as per the parameters mentioned in their respective works. The detailed experimental results are presented in Table II for the Salains dataset (due to the page length, other datasets' results are skipped). Looking into Table II, one can conclude that the pixel-based classification methods i.e., MLR and RF under-performed SVM, however, 1D CNNs performance is superior to other pixel-based spectral classification methods. Whereas, the spatial and spectral-spatial classification methods such as 2D, 3D, and Hybrid methods, respectively outperformed spectral methods. Moreover, SCS performance is similar to spectral-spatial methods' performance while requiring significantly fewer training parameters, i.e., SCS requires \(5624\) training parameters whereas, 3D and Hybrid CNN require \(127,104\) training parameters, which is significant.

## VI Conclusion

Hyperspectral Image Classification is a difficult task due to high inter and intra-class similarity and variability, nested regions, and overlapping. 2D Convolutional Neural Networks (CNN) emerged as a viable network whereas, 3D CNNs are a better alternative due to accurate classification. However, 3D CNNs are highly computationally complex due to their volume and spectral dimensions. Moreover, down-sampling and hierarchical filtering (high frequency) i.e., texture features need to be smoothed during the forward pass which is crucial for accurate classification. Furthermore, CNN requires tons of tuning parameters which increases the training time. To overcome the aforesaid issues, this work presented a Sharpened Cosine Similarity (SCS) as an alternative to Convolution in neural networks. SCS is exceptionally parameter efficient due to the following reasons: Skip nonlinear activation layer (ReLU, Sigmoid, etc.), Skip normalization and dropout layers, Use of Abs Max-pooling instead of MaxPool which selects the elements with the highest magnitude of activity even if its negative. Several experimental results proved the efficiency of SCS as compared to Convolutional layers.
