# Multi-image Super-resolution via Quality Map Associated Attention Network

Minji Lee Minji Lee is with the School of Computing, KAIST, Daejeon, South
Korea (e-mail: haewon_lee@kaist.ac.kr).

###### Abstract

Multi-image super-resolution, which aims to fuse and restore a high-resolution
image from multiple images at the same location, is crucial for utilizing
satellite images. The satellite images are often occluded by atmospheric
disturbances such as clouds, and the position of the disturbances varies by
the images. Many radiometric and geometric approaches are proposed to detect
atmospheric disturbances. Still, the utilization of detection results, _i.e._
, quality maps in deep learning was limited to pre-processing or computation
of loss. In this paper, we present a quality map-associated attention network
(QA-Net), an architecture that fully incorporates QMs into a deep learning
scheme for the first time. Our proposed attention modules process QMs
alongside the low-resolution images and utilize the QM features to distinguish
the disturbances and attend to image features. As a result, QA-Net has
achieved state-of-the-art results in the PROBA-V dataset.

###### Index Terms:

Multi-image super-resolution (MISR), Remote sensing, Self attention

##  I Introduction

Super-resolution (SR) aims to reconstruct a high-resolution (HR) image from
either single or multiple low-resolution (LR) images. Due to the expensive
cost of high-precision sensors, super-resolution is an extensively studied
subject in remote sensing. Recent single-image super-resolution methods [1, 2,
3, 4, 5] have shown outstanding performances, utilizing deep neural networks
to represent the mapping function from the LR image to the target HR image.
However, the high-frequency details in a single image are limited, and the
lost details are hard to be recovered.

Multi-image super-resolution (MISR) exploits multiple LR images to generate an
HR image. MISR has gained massive interest due to its use in Earth observation
satellite images, as the satellite can capture images of the same location
over time. Note that MISR differs from video super-resolution (VSR), although
they both employ multiple LR frames. VSR aims to reconstruct a high-resolution
video of a changing scene, and the frame sequence has a very short time
interval. The input images in MISR do not have an explicit temporal relation
and capture the scenery of the same location.

Figure 1: Illustration of the proposed method. The quality map features are
repetitively exploited to attend the low-resolution image features during
feature extraction and fusion.

There are two main concerns about the MISR of satellite imagery: (i)
misalignments such as rotation and shift between the LR images, and (ii)
atmospheric disturbances such as clouds and shadows degrading the quality of
the images. Previous works [6, 7, 8, 9, 10] have mainly concentrated on the
alignment and the fusion of the LR images, and the local deviation of image
quality due to atmospheric disturbances has been overlooked. Even though
comparing images is crucial to distinguish atmospheric disturbances, many MISR
methods compute each image separately.

We argue that it is necessary to discriminate the region hindered by the
disturbances and concentrate on the reliable features to generate a better
super-resolved image. It can be accomplished by encouraging communication
among the images during feature extraction and effectively using the _quality
maps_. A quality map indicates the area of the remotely sensed image corrupted
by the atmospheric disturbances and can be generated using algorithms [11]
comparing the retrieved reflectances of different bands. QMs provide helpful
information on the reliability of the features, but previous deep learning-
based MISR methods only used QMs for image selection, registration, and as the
mask to compute loss.

Figure 2: Model architecture of QA-Net. We omitted ReLU activation functions
for simplicity. LR image encoding modules (LEM) and QM encoding modules (QEM)
are repeated and for each iteration the output of QEM are used in the quality
map-associated module of LEM.

In this paper, we explore the usage of QMs further and propose the quality-map
associated attention network (QA-Net). Our model introduces QMs in both the
feature extraction of the LR images and their fusion, as depicted in Fig. 1.
QA-Net simultaneously processes the LR image and QM pairs in separate repeated
modules, and the processed QM features are utilized to attend the features
from the LR images for every iteration. With the guidance of QMs, QA-Net can
locally concentrate on the features of the less-disturbed images and fully
exploit the advantages of utilizing multiple LR images. As a result, our QA-
Net achieves state-of-the-art performance on the PROBA-V dataset.

##  II Related Works

MISR aims to effectively align and fuse multiple images captured in the same
location to reconstruct a higher resolution image. In MISR, LR images are
first registered, _i.e.,_ aligned against sub-pixel shifts and rotations. Some
methods [6, 12] select or generate reference images and register LR images to
the reference image using convolutional layers. Others [7, 13, 10] register
images in the Fourier domain using masked normalized cross-correlation (NCC),
_i.e._ masked FFT NCC [14], and optical flow estimation.

As well to the aforementioned registration methods, many deep learning-based
fusion methods are proposed. HighRes-net [6] fuses images using recursive
convolutional layers, and MISR-GRU [9] fuses images using a recurrent neural
network. However, such methods that process images in the separate
convolutional or recurrent module are incapable of learning relations between
images. Recently, TR-MISR [10] used a pixel-wise fusion method to learn the
relation between features of images.

##  III Methodology

This section presents QA-Net, a novel end-to-end neural network for multi-
image super-resolution. QA-Net consists of five parts: shallow feature
extraction, LR image encoding modules (LEMs), QM encoding modules (QEMs),
weighted mean, and upsampling. Let the low-resolution images of a scene and
their quality maps, $\text{LR}_{i}$, $\text{QM}_{i}$, respectively, where
$i=1,2,\cdots,T$ and $T$ is the number of the images. $\text{QM}_{i}$ is a
binary map where the value of 0 indicates the disturbances such as clouds,
shadows, ice, or water, and the value of 1, _i.e._ , clear pixel, indicates
the area of interest. QA-Net uses a shared convolutional layer
$\text{SF}_{\theta}$ to extract a shallow feature from each LR image and QM:

$\displaystyle L^{0}_{i}$
$\displaystyle=\text{SF}_{\theta}(\text{LR}_{i}),\quad
Q^{0}_{i}=\text{SF}_{\theta}(\text{QM}_{i}).$ (1)

$\\{L^{0}_{i}\\}^{T}_{i=1},\\{Q^{0}_{i}\\}^{T}_{i=1}$ pass through
$\text{LEM}_{j}$ and $\text{QEM}_{j}$ respectively, where $j=1,2,\cdots,N$ and
$N$ denotes the number of encoding modules:

$\displaystyle\\{L^{j}_{i}\\}^{T}_{i=1}$
$\displaystyle=\text{LEM}_{j}(\\{L^{j-1}_{i}\\}^{T}_{i=1},\\{Q^{j-1}_{i}\\}^{T}_{i=1}),$
(2) $\displaystyle\\{Q^{j}_{i}\\}^{T}_{i=1}$
$\displaystyle=\text{QEM}_{j}(\\{Q^{j}_{i}\\}^{T}_{i=1}).$ (3)

After the encoding, the shallow feature $L_{i}^{0}$ are added to the last LR
features $L_{i}^{N}$ and are accumulated with a weighted mean using the QM
features as the weight, resulting in a fused feature $f$ for every pixel
$(x,y)$:

$\displaystyle
f(x,y)=\dfrac{\sum_{i=1}^{T}(L_{i}^{0}(x,y)+L_{i}^{N}(x,y))\cdot
Q_{i}^{N}(x,y)}{\sum_{i=1}^{T}Q_{i}^{N}(x,y)}$ (4)

Finally, the super-resolved image SR is achieved via upsampling $f$ with a 2D
convolutional layer and pixel shuffle.

###  III-A LR image encoding modules (LEMs)

LEM mainly consists of self-attention [15] module (SA), QM-associated
attention module (QA), and multiple residual channel attention blocks (RCABs)
[5]. SA learns the relation between LR features, while the QA computes the
attention map based on features of QM and LR to let the network focus more on
LR features with better quality.

LR features $\\{L^{j}_{i}\\}_{i=1}^{T}$ and QM features
$\\{Q^{j}_{i}\\}_{i=1}^{T}$ are divided to $H\times W$ pixels
$l_{(x,y)}=\\{l_{i,(x,y)}\\}_{i=1}^{T},q_{(x,y)}=\\{q_{i,(x,y)}\\}_{i=1}^{T}\in\mathbb{R}^{T\times
C}$ where $C$ denotes the number of channels. Here, we omit $j$ since
attention modules are repeated for every LEM. Self-attention compares
$l_{(x,y)}$ along the image dimension $T$ to attend clear feature using
learnable parameter matrices $W_{Q},W_{K}\in\mathbb{R}^{C\times T}$, resulting
a self-attention map $\text{SAM}\in\mathbb{R}^{T\times T}$:

$\displaystyle\text{SAM}(l_{(x,y)})=\text{Softmax}\bigg{(}\dfrac{l_{(x,y)}\;W_{Q}\;W_{K}^{T}\;l_{(x,y)}^{T}}{\sqrt{T}}\bigg{)}.$
(5)

SAM is multiplied by $l_{(x,y)}$ to weigh the clear features. Attended
features are further projected using learnable parameter matrices
$W_{V}\in\mathbb{R}^{C\times T},W_{0}\in\mathbb{R}^{T\times C}$:

$\displaystyle\text{SA}(l_{(x,y)})=\text{SAM}(l_{(x,y)})\;l_{(x,y)}W_{V}W_{0}.$
(6)

Similarly, QA compares $l_{(x,y)},q_{(x,y)}$ along the image dimension $T$
using learnable parameter matrices $V_{Q},V_{K}\in\mathbb{R}^{C\times T}$,
resulting a QM-associated attention map $\text{QAM}\in\mathbb{R}^{T\times T}$:

$\displaystyle\text{QAM}(l_{(x,y)},q_{(x,y)})=\text{Softmax}\bigg{(}\dfrac{q_{(x,y)}\;V_{Q}\;V_{K}^{T}\;l_{(x,y)}^{T}}{\sqrt{T}}\bigg{)}.$
(7)

By multiplying QAM by $l_{(x,y)}$, the LR feature with a better corresponding
QM feature is weighted. Then QM-associated attention is completed by
projecting learnable parameter matrices $V_{V}\in\mathbb{R}^{C\times
T},V_{0}\in\mathbb{R}^{T\times C}$:

$\displaystyle\text{QA}(l_{(x,y)},q_{(x,y)})=\text{QAM}(l_{(x,y)},q_{(x,y)})\;l_{(x,y)}V_{V}V_{0}.$
(8)

Altogether, repeated LEMs allow the network to compare features of multiple
images during the feature extraction process.

Algorithm 1 Quality-map associated image selection

$\text{Corresponding QMs }\text{QM}=\\{\text{QM}_{1},\cdots,\text{QM}_{N}\\}$

$\text{Number of input images }T$

$\text{Number of pixels to compare }p$

$\mathcal{D}\leftarrow\\{\\},\mathcal{Q}\leftarrow\\{\\}$

for $t\in[0,\text{Number of iterations}]$ do

$d\leftarrow$ Set of randomly sampled $T$ indices

$A\leftarrow\displaystyle\sum_{i\in d}\text{QM}_{i}$

$\mathcal{D}\leftarrow\mathcal{D}\cup d$

$\mathcal{Q}\leftarrow\mathcal{Q}\cup\displaystyle\sum(p$ minimum values of
$A$)

end for

$m\leftarrow\text{argmin}\mathcal{Q}$

return $\mathcal{D}_{m}$

###  III-B QM encoding modules (QEMs)

Compared to LRs, QMs are processed using a relatively shallow module composed
of two convolutional layers with a skip connection. Since the features are
used as the query of the attention module, we used the sigmoid activation
function.

###  III-C Image selection

Since the number of LR images can vary by location, $T$ LR images are selected
according to the Algorithm 1 in test time. For training, $T$ images are
randomly selected according to the probability proportional to the number of
clear pixels in the corresponding QM. Quality-map-associated image selection
algorithm leads to uniform distribution of clear pixels throughout the images,
thus leveraging the attention process.

TABLE I: Quantitative comparison between different models on the PROBA-V validation set. Model | RED | NIR
---|---|---
cPSNR (dB) | cSSIM | cPSNR (dB) | cSSIM
Bicubic | 47.34 | 0.9846 | 45.44 | 0.9767
IBP [16] | 48.21 | 0.9865 | 45.96 | 0.9796
RCAN [5] | 48.22 | 0.9870 | 45.66 | 0.9798
HighRes-net [6] | 49.75 | 0.9904 | 47.55 | 0.9855
DeepSUM [12] | 50.00 | 0.9908 | 47.84 | 0.9858
MISR-GRU [9] | 50.11 | 0.9910 | 47.88 | 0.9861
RAMS [7] | 50.17 | 0.9913 | 48.23 | 0.9875
TR-MISR [10] | 50.67 | 0.9921 | 48.54 | 0.9882
PIUnet [13] | 50.62 | 0.9921 | 48.72 | 0.9883
QA-Net (ours) | 50.82 | 0.9976 | 48.81 | 0.9964

##  IV Experimental Settings

QA-Net is trained and tested on the PROBA-V dataset [17], which is composed of
Top-Of-Atmosphere reflectances for red visible (RED) and Near IR (NIR)
spectral bands at 300-m (128$\times$128 grayscale image) and 100-m
(384$\times$384 grayscale image) resolution. We aim to reconstruct 100-m
resolution (HR) images from multiple 300-m resolution (LR) images.

###  IV-A Implementation details

LR images are registered using masked FFT NCC [14] and cropped to 32$\times$32
patches before the training process. We set the number of RCABs in LEM $M=8$,
LEMs and QEMs $N=12$, input LR and QM pairs $T=9$, and the pixels to compare
$p=50$. The kernel size of the 2D convolution is set to 3$\times$3\. We used
the Adam optimizer with the initial learning rate set to 1e-4 and the weight
decay set to 1e-5. The network is trained 520 epochs and the learning rate is
multiplied by 0.8 at 120, 220, 300, 360, 400, 440, and 480 epochs.

###  IV-B Evaluation metric

Assuming that pixel intensities are real numbers ranging from 0 to 1, the
quality of a super-resolved image is evaluated with the clear peak signal-to-
noise ratio (cPSNR) via the following calculations. First, we calculate the
bias in brightness:

$\displaystyle b=\frac{1}{|P|}\sum_{(x,y)\in
P}(\text{HR}(x,y)-\text{SR}(x,y)),$ (9)

where $P$ represents a set of clear pixels in a high-resolution image. Then we
calculate the clear mean squared error (cMSE) and cPSNR as follows.

cMSE $\displaystyle=\frac{1}{|P|}\sum_{(x,y)\in
P}(\text{HR}(x,y)-\text{SR}(x,y)+b)^{2},$ (10) cPSNR
$\displaystyle=-10\cdot\log_{10}(\text{cMSE}).$ (11)

We used cPSNR between the ground truth and the output as a loss function.
Similarly, we define clear structural similarity index measure (cSSIM) as SSIM
calculated on clear pixels. In the test phase, the super-resolved images are
cropped by a 3-pixel border, and the highest cPSNR among corresponding patches
around the center of the ground-truth images is used as the score to
compensate for pixel shifts.

Figure 3: QEM output and attention map visualization on NIR imgset1121. (a)
Input QMs. Red boundaries denote that the pixel (120, 120) is disturbed. (b)
Normalized outputs of the last QEM. (c)-(e) QM-associated attention map (QAM)
of the pixel (120, 120) of first, fourth, and eighth LEMs, respectively.

##  V Results

###  V-A Quantitative comparisons

Table I shows the comparison of cPSNR and cSSIM with several methods on the
PROBA-V validation set. Since the ground truth of the test set is not publicly
available, we compared models on the validation set. Our proposed method
outperforms the previous methods by 0.20 dB in the RED spectral band and
0.09dB in the NIR spectral band in PSNR. Also, our method achieves the best
cSSIM results in both spectral bands with a significant performance gap.

###  V-B Qualitative comparisons

As shown in the example of Fig. 4, the region in the yellow box is captured
clearly in only a few LR images, _e.g._ , LR000. Previous methods cannot
recover the region accurately since the images are hindered by atmospheric
disturbances in other LR images, _e.g._ , LR003. On the contrary, our QA-Net
recovered the region properly with the aid of QMs and communication among LR
images. Our quality-map-associated attention allows the network to unveil the
disturbed feature of an LR image using the clear feature from another LR
image.

##  VI Discussion

Fig. 3 (a)-(b) shows the QMs and normalized feature of their last QEM output,
respectively. As the convolution operations in QEMs increase the receptive
field and the number of channels for each pixel, QEM outputs have richer
information about the atmospheric disturbances than the binary values of QMs.
Fig 3 (c)-(e) shows the QM-associated attention map (QAM) of the pixel (120,
120). Attention is computed through:

$\displaystyle\\{\text{QAM}_{k,1}\;l_{1,(120,120)}+\cdots+\text{QAM}_{k,9}\;l_{9,(120,120)}\\}^{9}_{k=1}.$
(12)

The most weighted vectors are $l_{1,(120,120)},l_{8,(120,120)}$ in the first
QAM, $l_{2,(120,120)},l_{3,(120,120)},l_{4,(120,120)}$ in the fourth QAM, and
$l_{1,(120,120)},l_{4,(120,120)}$ in the eighth QAM.
$l_{5,(120,120)},l_{7,(120,120)},l_{9,(120,120)}$, is disturbed heavily
(marked with red box in Fig. 3 (a)) so are never highly weighted. However, the
highly weighted vectors differ by every LEM, which implies that each LEM
focuses on different features, similar to the convolutional filters in CNN.

##  VII Conclusion

In this paper, we introduced a novel architecture that can distinguish
atmospheric disturbances in satellite imagery named QA-Net. QA-Net is the
first multi-image super-resolution method to exploit quality maps to attend
low-resolution images in feature extraction and fusion processes. Quantitative
and qualitative evaluations on the PROBA-V dataset show the outstanding
performance of our method, thus highlighting the importance of utilizing
quality maps in deep learning-based MISR, and the comparison and attention
across images.

Figure 4: Visual comparison between different models on NIR imgset0658.

## References

  * [1] C. Dong, C. C. Loy, K. He, and X. Tang, “Image super-resolution using deep convolutional networks,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 38, no. 2, pp. 295–307, 2015.
  * [2] J. Kim, J. K. Lee, and K. M. Lee, “Accurate image super-resolution using very deep convolutional networks,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 1646–1654.
  * [3] N. Ahn, B. Kang, and K.-A. Sohn, “Fast, accurate, and lightweight super-resolution with cascading residual network,” in _Proceedings of the European conference on computer vision (ECCV)_ , 2018, pp. 252–268.
  * [4] Y. Tai, J. Yang, and X. Liu, “Image super-resolution via deep recursive residual network,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 3147–3155.
  * [5] Y. Zhang, K. Li, K. Li, L. Wang, B. Zhong, and Y. Fu, “Image super-resolution using very deep residual channel attention networks,” in _Proceedings of the European conference on computer vision (ECCV)_ , 2018, pp. 286–301.
  * [6] M. Deudon, A. Kalaitzis, M. R. Arefin, I. Goytom, Z. Lin, K. Sankaran, V. Michalski, S. E. Kahou, J. Cornebise, and Y. Bengio, “Highres-net: Multi-frame super-resolution by recursive fusion,” 2019.
  * [7] F. Salvetti, V. Mazzia, A. Khaliq, and M. Chiaberge, “Multi-image super resolution of remotely sensed images using residual attention deep neural networks,” _Remote Sensing_ , vol. 12, no. 14, p. 2207, 2020.
  * [8] G. Bhat, M. Danelljan, L. Van Gool, and R. Timofte, “Deep burst super-resolution,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 9209–9218.
  * [9] M. R. Arefin, V. Michalski, P.-L. St-Charles, A. Kalaitzis, S. Kim, S. E. Kahou, and Y. Bengio, “Multi-image super-resolution for remote sensing using deep recurrent networks,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops_ , 2020, pp. 206–207.
  * [10] T. An, X. Zhang, C. Huo, B. Xue, L. Wang, and C. Pan, “Tr-misr: Multiimage super-resolution based on feature fusion with transformers,” _IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing_ , vol. 15, pp. 1373–1388, 2022.
  * [11] G. Lisens, P. Kempencers, F. Fierens, and J. Van Rensbergen, “Development of cloud, snow, and shadow masking algorithms for vegetation imagery,” in _IGARSS 2000. IEEE 2000 International Geoscience and Remote Sensing Symposium. Taking the Pulse of the Planet: The Role of Remote Sensing in Managing the Environment. Proceedings (Cat. No. 00CH37120)_ , vol. 2. IEEE, 2000, pp. 834–836.
  * [12] A. B. Molini, D. Valsesia, G. Fracastoro, and E. Magli, “Deepsum: Deep neural network for super-resolution of unregistered multitemporal images,” _IEEE Transactions on Geoscience and Remote Sensing_ , vol. 58, no. 5, pp. 3644–3656, 2019.
  * [13] D. Valsesia and E. Magli, “Permutation invariance and uncertainty in multitemporal image super-resolution,” _IEEE Transactions on Geoscience and Remote Sensing_ , vol. 60, pp. 1–12, 2021.
  * [14] D. Padfield, “Masked object registration in the fourier domain,” _IEEE Transactions on image processing_ , vol. 21, no. 5, pp. 2706–2718, 2011.
  * [15] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” _Advances in neural information processing systems_ , vol. 30, 2017.
  * [16] M. Irani and S. Peleg, “Improving resolution by image registration,” _CVGIP: Graphical models and image processing_ , vol. 53, no. 3, pp. 231–239, 1991.
  * [17] S. Sterckx, I. Benhadj, G. Duhoux, S. Livens, W. Dierckx, E. Goor, S. Adriaensen, W. Heyns, K. Van Hoof, G. Strackx _et al._ , “The proba-v mission: Image processing and calibration,” _International journal of remote sensing_ , vol. 35, no. 7, pp. 2565–2588, 2014.