Operational Neural Networks for Parameter-Efficient Hyperspectral
  Single-Image Super-Resolution
Hyperspectral Imaging is a crucial tool in remote sensing which captures far
more spectral information than standard color images. However, the increase in
spectral information comes at the cost of spatial resolution. Super-resolution
is a popular technique where the goal is to generate a high-resolution version
of a given low-resolution input. The majority of modern super-resolution
approaches use convolutional neural networks. However, convolution itself is a
linear operation and the networks rely on the non-linear activation functions
after each layer to provide the necessary non-linearity to learn the complex
underlying function. This means that convolutional neural networks tend to be
very deep to achieve the desired results. Recently, self-organized operational
neural networks have been proposed that aim to overcome this limitation by
replacing the convolutional filters with learnable non-linear functions through
the use of MacLaurin series expansions. This work focuses on extending the
convolutional filters of a popular super-resolution model to more powerful
operational filters to enhance the model performance on hyperspectral images.
We also investigate the effects that residual connections and different
normalization types have on this type of enhanced network. Despite having fewer
parameters than their convolutional network equivalents, our results show that
operational neural networks achieve superior super-resolution performance on
small hyperspectral image datasets. Our code is made available on Github:
https://github.com/aulrichsen/SRONN.
Alexander Ulrichsen, Paul Murray, Stephen Marshall, Moncef Gabbouj, Serkan Kiranyaz, Mehmet Yamac, Nour Aburaed
# Operational Neural Networks for Efficient Hyperspectral Single-Image Super-Resolution

###### Abstract

Hyperspectral Imaging is a crucial tool in remote sensing which captures far more spectral information than standard color images. However, the increase in spectral information comes at the cost of spatial resolution. Super-resolution is a popular technique where the goal is to generate a high-resolution version of a given low-resolution input. The majority of modern super-resolution approaches use convolutional neural networks. However, convolution itself is a linear operation and the networks rely on the non-linear activation functions after each layer to provide the necessary non-linearity to learn the complex underlying function. This means that convolutional neural networks tend to be very deep to achieve the desired results. Recently, self-organized operational neural networks have been proposed that aim to overcome this limitation by replacing the convolutional filters with learnable non-linear functions through the use of MacLaurin series expansions. This work focuses on extending the convolutional filters of a popular super-resolution model to more powerful operational filters to enhance the model performance on hyperspectral images. We also investigate the effects that residual connections and different normalization types have on this type of enhanced network. Despite having fewer parameters than their convolutional network equivalents, our results show that operational neural networks achieve superior super-resolution performance on small hyperspectral image datasets.

 Hyperspectral Imaging, Super-Resolution, Operational Neural Networks

## I Introduction

Hyperspectral imaging is a key component in remote sensing applications as the additional spectral information offers insights into the materials within the image that standard color images cannot provide. However, such increased spectral resolution comes at the cost of spatial resolution due to the sensors' limitations. Automated image processing tasks such as image segmentation, object detection and classification can improve the efficiency of remote sensing systems. However, the reduction in spatial resolution can be detrimental to their performance. To recover the lost spatial resolution and help improve the performance of image processing tasks on the resulting hyperspectral image (HSI), one option is to apply single image super-resolution (SISR) to enhance the spatial resolution of the given low-resolution hyperspectral image.

Most modern super-resolution (SR) approaches use convolutional neural networks (CNNs) to produce an image-to-image mapping operator which converts the input low-resolution image to a high-resolution image [1, 2, 3, 4]. These operators are of a complex non-linear nature and part of the reason that CNNs have had so much success in this field is due to their capacity to learn complex non-linear operators. However, the sole non-linear elements of a CNN come from the activation functions after each layer, meaning that CNNs often require many layers to have the necessary non-linear capacity and diversity to learn the desired operator. Recently, operational neural networks (ONNs) [5, 6] and their new variants, self-organised operational neural networks (Self-ONNs) [7], have been proposed to overcome this limitation by using the generative neuron model that can customize the optimal non-linear function during training for each kernel element. To accomplish this, each kernel element is extended with MacLaurin series expansions and the terms of the series are made learnable. This means that each kernel element can learn to approximate any non-linear function and thus similar theoretical non-linear capacity of a deep CNN can be achieved in a much shallower Self-ONN which is more computationally efficient. In this paper, we take the popular SR network, SRCNN [3], and extend it for use on hyperspectral images. We also make a Self-ONN equivalent model by replacing the convolutional layers with operational layers. Furthermore, we make a Self-ONN version with a reduced number of filters to demonstrate the non-linear capacity of operational layers over convolutional layers. We train our models on the publicly available Pavia University, Cuprite, Salinas, and Urban datasets [8, 9] and show that Self-ONNs can provide an HSI SR performance improvement of over 0.5 dB PSNR even when it has fewer parameters than a CNN.

Furthermore, this study investigates the effects residual connections and various normalization types have on Self-ONN performance, as, to the best of our knowledge, this has
not been previously investigated.

The contributions of this work can be summarised as follows:

* We propose novel Self-ONN-based efficient neural networks that have structures similar to that of the well known lightweight CNN SR network, SRCNN [3].
* We present our novel findings on the effect residual connections and various normalization types have on hyperspectral image super-resolution performance in our Self-ONN models.

## II Related Work

### _Hyperspectral Imaging_

Hyperspectral imaging is a valuable tool in remote sensing applications such as material classification, mineral exploration, environmental monitoring, and more [10]. The effectiveness of these tasks is dependent on the resolution of the captured image. However, due to sensor limitations, it is difficult to obtain a high-quality hyperspectral image (HSI) with both high spectral and spatial resolution [11] and thus the increased spectral resolution generally comes at the cost of decreased spatial resolution [12]. It is therefore desirable to be able to recover the lost spatial resolution to improve the performance of post-processing tasks.

### _Super-Resolution_

Most modern approaches to super-resolution use convolutional neural networks (CNNs) in either a supervised or unsupervised manner [13, 14, 15, 16]. Supervised training involves training a model on a dataset consisting of low-resolution and high-resolution image pairs. One of the first papers to adopt this approach was [3] where they proposed their CNN model named SRCNN for the task of single image super-resolution.

SRCNN is a fairly shallow CNN by modern standards, consisting of only 3 layers, so the authors of [17] proposed a much deeper CNN to perform supervised SISR. The deeper network provides more learning capacity but is also more difficult to train due to the vanishing gradient problem. To overcome this issue, they proposed a residual connection which sums the input of the model directly to the output so that instead of learning the direct input-to-output image mapping, the model learns the residual between the input and output which improved results and greatly decreased training times.

Since then, many other deep CNN models have been proposed for supervised single image super-resolution [18, 19, 20, 13, 21]. However, the main challenge of this approach is acquiring the dataset. Ideally, perfectly aligned images would be captured with a low-resolution and a high-resolution sensor, but this is impractical to perform in many situations. What is more commonly done is a dataset of high-resolution images is acquired and the low-resolution image pairs are then synthetically generated by blurring and downsampling the high-resolution images and then adding noise.

To overcome this limitation, unsupervised methods using Generative Adversarial Networks (GANs) [22] have been proposed which utilize datasets of unpaired real high-resolution and low-resolution images through the use of generator and discriminator models. The generator produces high-resolution versions of the low-resolution images and the discriminator aims to distinguish between the true high-resolution images and the generated high-resolution images. Over time, the generator learns to produce realistic high-resolution outputs of the input low-resolution images which match the distribution of the high-resolution image dataset. Thus, the model is more likely to learn the true low-resolution to high-resolution image mapping function. Many researchers have achieved impressive results using this approach [1, 23, 24, 15, 25, 26]. However, the unsupervised nature of this approach means that it is inherently more difficult to train as the generator learns from feedback provided by the discriminator and the discriminator has no prior knowledge of the objective. In addition, it is also challenging to measure the performance of a GAN objectively as typical image quality metrics such as peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM) [27] require a target image to be evaluated. To overcome these problems it typically requires a lot of training data to produce a realistic GAN [28] which is not always available, particularly in the case of hyperspectral imagery.

Attempts have been made to improve upon human-perceived super-resolution quality. [29] introduces a perceptual loss function generated through a fixed loss network to create visually pleasing results. Their method produces visually pleasing results but at the cost of PSNR and SSIM, indicating that their per-pixel accuracy is lower. [15] introduces a perceptual loss function to train a GAN, which again focuses on learning mappings that are perceptually pleasing to humans, rather than pixel-to-pixel accuracy. These approaches improve how pleasing super-resolution outputs may be to a human observer, but do not necessarily provide any performance improvement for post-processing tasks to be done on the resulting images.

It has been shown that super-resolution performance can be improved by utilising multiple images captured in quick succession [30]. However, this approach is impractical when it comes to HSIs due to the slow acquisition times. Data fusion techniques can be applied to HSIs [31, 32]. However, these approaches rely on the availability of a high-resolution multispectral image of the same scene.

Transformers [33] are gaining popularity in the vision community and some researchers have utilized them for super-resolution [34]. However, this approach suffers the same problem as the unsupervised GAN methods in that that they require very large amounts of data to be trained. Furthermore, the use of these techniques is also known to be computationally expensive during the inference process.

Given the limited availability of training data makes the use of transformers, modern deep GANs, and CNNs difficult to apply to HSI SR problems. Furthermore, we generally aim to improve the quality of the hyperspectral data before inference tasks, which means that an efficient SR network that can operate in real-time is preferred. The large amount of data to be processed in hyperspectral imaging presents a challenge to using deep networks, so we propose a highly efficient SR neural network structure based on a new paradigm, self-operational neural filtering.
### _Operational Neural Networks_

Recent advances in deep learning have resulted in CNNs dominating many computer vision fields, including super-resolution. Part of the reason for their success is their ability to learn complex non-linear operators. However, convolution itself is a linear operation and the non-linear components of the networks are solely provided by the activation functions used after each convolutional layer in the network. This means that CNNs often have to be very deep in order to have the necessary non-linear capacity and diversity to learn the complex function of the learning problem.

Recently, Operational Neural Networks (ONNs) [5, 6] were proposed to address this issue by incorporating non-linear nodal and pooling functions that replace the sole convolution operation with any non-linear operator, which adds significantly more non-linear components to the network than a traditional CNN. However, these additional non-linear operations are hard coded and thus cannot be changed during training. This means that the functions need to be searched for, which is computationally expensive, and the search space is limited to the function set, which may not contain the optimal function(s).

The authors of [5] then addressed these limitations by proposing self-organized operational neural networks (Self-ONNs) [7] which aim to make the linear filters of a standard CNN non-linear through the use of MacLaurin series expansions, rather than applying hard-coded functions. Such non-linear filters for each kernel element are learnable during training, and thus, eliminate the need for an exhaustive search to find the optimal functions. Furthermore, any function can theoretically be approximated using MacLaurin series expansions, which means that a Self-ONN is not limited to a specified function set, allowing for an enhanced non-linear search space. These improvements mean that Self-ONNs are far more computationally efficient than their standard ONN counterparts, with greater theoretical non-linear capacity than both their ONN and CNN counterparts. This additional complexity comes at the cost of each filter requiring more parameters. However, the network size of a Self-ONN can be much smaller than a CNN to have the same or increased theoretical non-linear capacity, allowing for the overall model to have fewer parameters than a CNN despite each individual filter containing more parameters. In many applications [35, 36, 37, 38, 39, 40] Self-ONNs outperformed the deeper and more complex CNNs whilst achieving an elegant computational efficiency.

## III Methodology

We take the super-resolution model SRCNN [3] and modify it for use on hyperspectral images by extending the number of input and output channels of the model from 3 (for RGB images) to the required number for the relevant HSI depending on the number of wavelength bands it contains. SRCNN, shown in Figure 1, is a relatively compact model consisting of 3 convolutional layers followed by ReLU activation functions, except for the output layer, where no activation function is used.

We propose a novel Self-ONN model, SRONN, that shares the same configuration as SRCNN as shown in Figure 2. A key aspect of Self-ONNs is that data passed between layers must be bounded between -1 and 1 in order to prevent exponentially large values due to the non-linear nature of the model. We, therefore, use _tanh_ activation functions after the first and second operational layers in our SRONN model instead of the ReLU activation functions of SRCNN.

Self-ONNs gain their additional non-linear complexity through the use of MacLaurin series expansions:

\[f(x)=\sum_{n=0}^{\infty}\frac{f^{(n)}(0)}{n!}x^{n} \tag{1}\]

In practice, the 0th term in the expansion is the bias. Therefore, the 0th term can be disregarded from the filter approximation. The order of the polynomial should be finite in practice so the number of terms is supplied to the network by a parameter \(Q\). This makes the expansion for an ONN as follows:

\[f(x)=\sum_{n=1}^{Q}\frac{f^{(n)}(0)}{n!}x^{n} \tag{2}\]

Note that when the \(Q\) value is 1, it is the exact equivalent of a standard convolutional layer. Higher \(Q\) values yield more accurate function approximations but at the cost of additional parameters as the \(Q\) value directly equates to the multiplication in parameters over a standard convolutional filter. The number of parameters in the convolutional layers of a CNN can be calculated using the following equation:

\[\text{\# parameters}=\sum_{l=0}^{l-1}(n_{l}\times m_{l}\times f_{l}+1)\times f _{l+1} \tag{3}\]

where \(l\) is the number of layers, \(n_{l}\), \(m_{l}\) is the number of rows and columns in the convolutional filters at layer \(l\), \(f\)

Fig. 1: SRCNN model representation consisting of 3 convolutional layers with filter sizes f1 x f1, f2 x f2, and f3 x f3.

Fig. 2: SRONN model representation consisting of 3 convolutional layers with filter sizes (f1 x f1 x Q), (f2 x f2 x Q), and (f3 x f3 x Q).
the number of filters and the constant 1 accounts for the bias for each filter. Note, that on the first layer, i.e. \(l=0\), the number of filters from the previous layer (\(l-1\)) is given by the number of channels of the input image. To compute the number of parameters of a Self-ONN, we simply multiply this by \(Q\):

\[\text{\# parameters}=\sum_{l=0}^{l-1}(n_{l}\times m_{l}\times f_{l}\times Q+1) \times f_{l+1} \tag{4}\]

We select a \(Q\) value of 3 for all our experiments as we found this to be a good balance between sufficient approximation accuracy and the number of parameters. However, this means that each of our SRONN models will have approximately three times as many parameters as the SRCNN models. For a fair comparison, we also propose another Self-ONN model again with the same number of layers as SRCNN, but four times fewer filters per layer. As a result, this model has between 26.5% and 28.2% fewer parameters than SRCNN depending on the number of input and output channels required by the target dataset. We name this model small SRONN, or sSRONN.

### _Normalization and Residual Connections_

Due to their recent proposal, techniques commonly applied to CNNs to improve results have been studied little on ONNs. We study the effects of incorporating various normalization layer types into our ONN models after each Tanh activation function including L1, L2, instance [41], and batch [42] normalization. We also study the effects of adding a residual connection to connect the output of the model directly to the input of the model so that the model learns the residual rather than the direct mapping as performed in [20]. To the best of our knowledge, this is the first work to study the effects of these techniques on Self-ONNs.

The proposed Self-ONN model is illustrated in Figure 3.

## IV Training Details

Each model was trained for 50000 epochs to guarantee network convergence, and the weights from the epoch which produced the highest SSIM validation score were used for testing. We use the Adam optimizer [43] with default parameters except for the learning rate. Each model was initially trained with a learning rate of \(10^{-4}\) which was decreased by a factor of 10 at epochs 5000 and 40000. Two following runs were then completed where the starting learning rate and the epoch milestones - where the learning rate was decreased by a factor of 10 - were manually adjusted in an attempt to improve the performance. We use mean squared error as our loss function. We initialize our models' weights with a normal distribution with a gain of 0.02. All training LR tiles are fed to the model in a single batch on each epoch.

### _Datasets_

We evaluate our models on four different HSI datasets: Pavia University; Salinas; Cuprite; Urban. Details for each dataset [9, 8] can be seen in Table I.

We use the standard approach to generating a low-resolution image pair from a given high-resolution target image by using Eq. (5):

\[I_{LR}=(I_{HR}*k)\downarrow_{s}+n \tag{5}\]

where \(k\in R^{2}\) is a 2D degradation kernel, * is a spatial convolution, \(\downarrow_{s}\) is a decimation operation with a stride s, and n is a noise term. We use Gaussian blur with a sigma value of 0.8943 for k, 2x bicubic interpolation for \(\downarrow_{s}\). We do not add any noise so the parameter n is ignored. Each generated LR tile was then bilinearly interpolated back up to the size of the original tile so the model could perform super-resolution by recovering the information at the desired output resolution. The model would then be trained with the LR tile as input and the original HR tile as the target.

Each dataset was preprocessed with min-max normalization and then divided into 64x64 pixel tiles, maintaining the entire wavelength spectrum. We utilize 70% of the tiles for training, 15% for validation and reserve 15% for testing.

## V Results

We train each model combination 3 times, i.e. the combination of the base model, the normalization type used, and whether or not there was a residual connection. The first iteration uses the default parameters of \(10^{-4}\) learning rate, and the learning rate is decreased by a factor of 10 on epochs 5000 and 40000. For the remaining two iterations, the parameters were manually adjusted in an attempt to improve the performance from the default first iteration. For all experiments, the entire training dataset was forward propagated through the model at once so there was no need to adjust the batch size.

We only apply normalization to the Self-ONN models, since normalization has been widely studied on CNNs. Therefore,

Fig. 3: General model architecture. C represents the number of channels in the hyperspectral image. Values in brackets represent the number of filters in the compact sSRONN model. SRCNN and SRONN variants have Cx128, 128x64, and 64xC filters in each respective layer. sSRONN variant has Cx32, 32x16, and 16xC filters in each respective layer. The normalization type depends on the experiment and in some experiments, there is no normalization, in which case the normalization layers are skipped. The residual connection is also removed in experiments where it is not applied.
we first compare the SRCNN models against the SRONN and sSRONN models without normalization for a fair comparison. The results without a residual connection can be seen in Table II with example outputs from the Pavia University dataset in Figure 4 and the results without a residual connection can be seen in Table III with example outputs from the Pavia University dataset in Figure 7.

For the three training iterations of each model on each dataset, we report only the results from the best iteration.

### _Normalization_

We present the results from adding various normalization types to the Self-ONN models in separate tables for each dataset. Results for the Cuprite dataset are shown in Table IV, Pavia University in Table V, Salinas in Table VI and Urban in Table VII.

## VI Discussion

The results from Table II show that the SRONN models tend to offer a slight improvement over the SRCNN model, but this is not always the case. For example, on the Salinas dataset, the SRCNN model outperformed the SRONN models across all metrics. We hypothesise that this is due to the Self-ONN models having a more complex search space to navigate and optimise, and therefore have more difficulty converging than the simpler SRCNN model.

Fig. 4: Output of Models with no Residual Connection or Normalization on slice 80 of the Pavia University dataset. The original HSI is shown on the left. LR, predictions, original and spatial absolute difference between prediction and HR shown on the right.
### _Effect of Residual Connection_

The results from Table III show that adding a residual connection provides significant improvement to both Self-ONN models, resulting in both the SRONN and sSRONN models outperforming the SRCNN models across all metrics on all datasets. The addition of a residual connection improved all metrics across all datasets for both sSRONN and SRONN except for PSNR on the Urban dataset for the SRONN model where a slight decrease was observed. The residual connection has a lesser impact on the results of SRCNN, only offering improvement in some cases, which is likely due to the model not being complex enough to see any consistent performance improvement from a residual connection.

The improvement seen in the Self-ONN models when a residual connection is added supports our convergence hypothesis. It could also be indicative that Self-ONNs may suffer more from vanishing gradients than CNNs. Interestingly, the sSRONN model generally saw greater performance improvements from the addition of a residual connection than the SRONN model, which is counterintuitive as the sSRONN optimization search space is significantly smaller than the search space of the SRONN model. One explanation for this could be that the sSRONN model might be slightly under-parameterized for direct image-to-image mapping. However, it may have sufficient parameters to learn the residual, resulting in a bigger performance improvement when the residual connection is

Fig. 5: Output of Models with a Residual Connection on slice 80 of the Pavia University dataset. The original HSI is shown on the left. LR, predictions, original and spatial absolute difference between prediction and HR shown on the right.
added to the model. The larger SRONN model, which may be well-parameterised for image-to-image mapping but slightly over-parameterised for residual learning, does not see as much of a performance improvement as the smaller sSRONN model.

Since both SRONN and sSRONN outperform SRCNN, this demonstrates the power of the non-linear filters over the standard linear convolutional filters. The non-linear filters provide the operational layer with an enhanced ability to produce sharper edges and thus produce sharper contrast between pixels resulting in a more detailed output image, which is evident in the resulting images shown in Figure 7.

### _Effects of Normalization_

Our results in Tables IV, V, VI, and VII show the effects of incorporating normalization layers into our SRONN and sSRONN models are largely varied and highly dataset dependent. It appears that normalization has a greater impact on the datasets with larger spatial dimensions. We found L2 normalization to be the most effective, providing a slight performance boost to the SRONN model across all metrics on the Cuprite, Pavia University and Urban datasets while boosting the SAM on the Salinas dataset. For the sSRONN model, the performance improvement from adding L2 normalization is less significant, providing only a performance boost to SSIM and SAM on the Cuprite dataset, PSNR on the Pavia University dataset and SSIM on the Urban dataset. No performance improvement was provided by using L2 normalization over no normalization on the Salinas dataset.

Our results show that normalization is generally more effective when utilized in conjunction with a residual connection. This is likely due to the fact the normalization layers will normalize the data around a zero mean which makes it more difficult for the models without a residual connection to map the zero mean feature maps to the true mean of the output. However, when a residual connection is introduced, the model learns the residual between the input and the target, which should have a mean near zero. Therefore, normalization may offer a greater benefit in this scenario as it assists the model in transforming the data to the target mean, rather than moving it away from the target mean.

Interestingly, we found instance normalization to be especially detrimental to all results. This could be because instance normalization normalizes each channel individually which may have an adverse effect on the channel dependencies.

## VII Conclusion

We show that Self-ONNs outperform equivalent well known CNNs in the task of HSI SR, even when the Self-ONN models

Fig. 6: True Super-Resolution Output of Models with a Residual Connection on slice 80 of the Pavia University dataset. The original HSI is shown on the left. Test tiles bilinearly interpolated up to 2x their original size, and super-resolution results on interpolated tiles are shown on the right.
have a lower number of parameters than the CNNs. The Self-ONN results produced sharper images and contained more detail which is likely a direct result of the enhanced non-linear filters.

We found that adding a residual connection to our SRONN and sSRONN models provided a significant performance improvement and greatly increased convergence times. We hypothesize that Self-ONNs suffer more from the vanishing gradient problem than CNNs due to their more complex search spaces and thus the residual connection helps mitigate this issue, even in relatively shallow models.

We examined the effects of adding a residual connection and various normalization layers to our ONN models. Our results show that L2 normalization layers in ONNs can offer a moderate performance improvement when used in conjunction with a residual connection, but the benefit of normalization appears to be highly dependent on the dataset.

We show that the superior non-linear capabilities of ONNs compared to CNNs allow for sharper and more detailed HSI SR results. This indicates that ONNs may a better solution than the standard CNN in such image-to-image mapping tasks.

## VIII Acknowledgments

This work was supported by the Engineering and Physical Sciences Research Council [grant number EP/T517938/1] and Peacock Technology Limited.
