# BrailleBand: Blind Support Haptic Wearable Band for Communication using
Braille Language

Savindu H.P.1, Iroshan K.A., Panangala C.D., Perera W.L.D.W.P., De Silva A.C.
(MIEEE) *This work was supported by the University of Moratuwa, Faculty of
Engineering.1Savindu H.P. is with the Department of Electronic and
Telecommunication Engineering, Faculty of Engineering, University of Moratuwa,
10400 Katubedda, Sri Lanka<EMAIL_ADDRESS>
###### Abstract

Visually impaired people are neglected from many modern communication and
interaction procedures. Assistive technologies such as text-to-speech and
braille displays are the most commonly used means of connecting such visually
impaired people with mobile phones and other smart devices. Both these
solutions face usability issues, thus this study focused on developing a user
friendly wearable solution called the ‘BrailleBand’ with haptic technology
while preserving affordability. The ‘BrailleBand’ enables passive reading
using the Braille language. Connectivity between the BrailleBand and the smart
device (phone) is established using Bluetooth protocol. It consists of six
nodes in three bands worn on the arm to map the braille alphabet, which are
actuated to give the sense of touch corresponding to the characters. Three
mobile applications were developed for training the visually impaired and to
integrate existing smart mobile applications such as navigation and short
message service (SMS) with the device BrailleBand. The adaptability, usability
and efficiency of reading was tested on a sample of blind users which
reflected progressive results. Even though, the reading accuracy depends on
the time duration between the characters (character gap) an average Character
Transfer Rate of 0.4375 characters per second can be achieved with a character
gap of 1000 ms.

##  I INTRODUCTION

The global completely blind population is estimated to be 40 to 45 million and
nearly 135 million are estimated to have low vision. The cause for blindness
for youth is mainly due to birth defects in the brain or the eye, and
uncorrected refractive errors. While visual impairment and blindness due to
infections have significantly reduced with the rapid progress of health care
services, there is a notable increase in blind and visually impaired people
over 65 years of age due to long life expectancy. Unfortunately, the blind
population is expected to double by 2020 [9].

Sight is the main human sense which possess the main influence on perception
of all sensations, in collaboration with other senses such as hearing.
Therefore, the lack of sight is the greatest challenge the blind face in
performing their daily tasks such as navigation, information access,
interpersonal interactions and safety. Hence, the blind are unemployed and
deprived of the privilege of education under normal circumstances.
Approximately 75% of the blind are unemployed while only 10% of the blind
children receive special braille education [1]. In the context of the modern
society a blind person and his/her family faces many socioeconomic problems.
Consequently, the need for assistive technologies which enable the blind to
live independent, productive and better lives emerged as investing on nursing
homes, blind welfare, health care and blind care specialists were perceived to
be costly and unsustainable solutions [11].

###  I-A Existing technologies and their drawbacks

Many research projects have been conducted focusing on independent
communication for the blind, the deaf and the deaf-blind communities. Wearable
systems such as mechanical hands for automated fingerspelling and
communication glove systems have been developed using various alphabets [5].
The Lorm glove uses the Lorm alphabet, a form of communication for deaf and
blind. It translates the hand-touch Lorm alphabet to text and vice versa using
flat fabric sensors embedded on the glove [7]. DB-HAND is a similar wearable
device with Malossi alphabet interface for the deaf-blind community. This
glove is equipped with sensors and actuators enabling two way communication
with pressing of tactile sensors enabling the inputs to the device [3]. In
another braille based mobile communication and translation glove, uses
capacitive touch sensors and actuators to enable two way communication by
translating text to braille patterns and vice versa [4]. Wearabraille is a
keyboard for the deaf and blind which uses finger mount accelerometers to
identify the tapping action of fingers in a similar fashion to typing in a
traditional keyboard [8].

However, the main drawback of glove systems is that they limit the use of
hands for other activities since the sense of touch for communication is given
to the palm or the fingers and it gets interrupted if hands are being used for
another activity.

Text-to-speech is another mode of assisting the blind. However, it possesses
usability issues such as privacy issues, disturbs hearing which is the main
sense of blind people etc. Even if a pair of headsets is used it permanently
disconnects the visually impaired person from the surrounding environment
increasing the chance of meeting with accidents. Moreover, Braille terminals
and displays carry an exorbitant cost, hence unaffordable.

###  I-B Proposed solution

The hypothesis of the proposed solution is to use the sense of touch of the
arm to transfer information to a blind person using a wearable band having six
vibration nodes corresponding to the standardized braille dot code. The system
developed to test this hypothesis integrated with a smart device was named as
the ‘BrailleBand’.

Wearing a band on the arm is similar to wearing a wristwatch, hence does not
hinder the use of the hand for other essential tasks. Moreover, wearable
assistive solutions are preferred than the portable assitive solutions such as
braille displays and mobile phones (text to speech) as they provide hands-free
convenient interactions.

Assistive technologies basically provide the ability to disabled people to
work independently without having a need for someone else attending. The main
focus areas of assistive technologies are [2]:

  1. 1.

Information transmission

Issues with regard to information transmission are reading, character
identification and interpreting two and three-dimensional graphics.The most
successful and most popular blind reading tool is braille dot code which is
the same dot code used in the BrailleBand making it readily usable for the
blind community for information transmission. Tactile displays and braille
displays enable character and graphic recognition by transferring information
to tangible forms to be felt by the blind.

  2. 2.

Mobility assistance/Navigation

Mobility and navigation involves information of the immediate dynamic
environment and obstacle avoidance. The basic functionality of all electronic
travel aids (ETAs) is scanning the surrounding environment and transferring
the gathered information through other senses to the user. BrailleBand can
transfer directional data through sense of touch to the user and give feedback
on the traveling directions.

  3. 3.

Computer and smart device accessibility

With the increased use of computers and smart devices the need of the visually
impaired to access smart devices emerged. The solutions currently leveraged
are text-to-speech and braille displays. BrailleBand can transfer information
on simple computer screens to blind users using the vibrating braille dot
patterns and subsequent to the development of the replying system two way
communication will be established with computers and smart devices.

The product BrailleBand caters the needs of all the above three areas and its
scope of applications are broader and can be customized according to diverse
user needs (refer to Fig. 1).

The study showed promising results with blind users being able to read mobile
phone text messages and tweets using the BrailleBand. Also, the study revealed
increase in reading speed and accurate character identification as the usage
increased.

Figure 1: The BrailleBand

##  II METHODOLOGY

###  II-A Hardware implementation

To represent nodes of the universal Braille dot code, six haptic motors (8mm
Vibration Motor - 2mm Type from Precision MicrodrivesTM) along with haptic
motor controllers were used to generate the vibration sensation to be felt at
the forearm corresponding to the characters to be read. Refer to Fig. 3 for a
few samples of Braille dot code.

The Fig. 2 summaries the hardware implementation of the Braileband. A
Rechargeable 1200 mAh Li-Ion battery was used as the source of power to the
whole system which through a 5V Voltage booster circuit. The battery was
charged through the USB type-C port using a Battery Management 0.8A single
input battery charger.

The microcontroller used was ATMEGA328P-AU which interacts with the Bluetooth
module and I2C multiplexers to generate the required signals corresponding to
the information received from the smart device via Bluetooth.

Haptic motor controllers are I2C controllable. However, they have the same
static I2C address and therefore I2C multiplexers were used.

The circuit was designed using AltiumTM and the enclosure of the prototype
product was developed using SolidworksTM which was realized through three-
dimensional (3D) printing.

Figure 2: Interconnections and the functionality of the modules Figure 3:
Braille alphabet and braille numbers (www.pharmabraille.com)

###  II-B Vibration motor placement and sense of touch

When testing the hypothesis of this study, placement of vibration motors on
the forearm was a critical task. Main aspects considered were:

  1. 1.

Two-point discrimination threshold (TPDT)

  2. 2.

Axes of vibration

TPDT is a measure of the minimum distance between two pressure points on the
skin to be felt as two distinct points. TPDT varies along body due to uneven
distribution of mechanoreceptors. As BrailleBand is worn on the forearm the
TPDT on the forearm which is 40mm is critical for design specifications [6].

The axis of vibration of the motors were horizontal and Eccentric Rotating
Mass (ERM) haptic motors were used to generate vibrations.

Even though the sense of touch is a supplementary sensory modality to a
healthy-sighted person, it is the primary sensory modality for a blind person
for non-audible information. Braille is a prime example of the success of
tactile mode of transferring information.

The skin is responsible for the sense of touch through three types of sensory
modalities:

  1. 1.

Thermoreceptors - for thermal sensations

  2. 2.

Nociceptors - pain sensing

  3. 3.

Mechanoreceptors - mechanical sensations due to external forces

There are four types of mechanoreceptors in the human skin and out of them
Meissner senses touch while Pacini corpuscles sense vibration. Since the
BrailleBand stimulus is vibration our interest was on Meissner and Pacini
sensors [10].

  * •

Meissner - high amplitude low frequency stimuli

  * •

Pacini corpuscles - low amplitude high frequency stimuli

The frequencies and the amplitudes of the haptic motors were determined in
such a manner that they stimulate both Pacini and Meissner vibration sensors
in the skin to enhance the sense of vibration.

###  II-C Software implementation

####  II-C1 Connectivity

Mobile applications which are connected with the device are developed using
android. By selecting the Media Access Control (MAC) address of the Bluetooth
module of the device (HC06), application can establish a socket connection
between the BrailleBand and the smart phone. After the establishing the
connection, a thread is created in the application which handles the socket
connection. When the thread is started, the input stream and the output stream
is set with the BrailleBand. The gaps between characters and words are
controlled by changing the threads to postDelayed state. The program
continuously read the input stream and control haptic motors depending on the
received character.

####  II-C2 Mobile applications

Three mobile applications were developed for the BrailleBand:

  1. 1.

BrailleBand Teacher

Used to get the blind person familiarized with the device and to teach the
method of reading braille from BrailleBand.

  2. 2.

BrailleBand Messenger

Enables the user to read text messages received to the mobile phone through
the BrailleBand.

  3. 3.

BrailleBand Navigator

This application can navigate a blind person to a destination by giving haptic
feedback about the direction and the distance to travel through the
BrailleBand. However, the accuracy of navigation still needs to be developed.

The existing social media and utility mobile applications such as facebook,
twitter, linkedin and google maps can be integrated with the BrailleBand
application and is readily usable with the BrailleBand device since
information from these applications can be conveyed to the blind user through
the device. Refer to Fig.4.

An android library was also developed and is readily available at
https://github.com/BrailleBand/BrailleBand-mobile-app to facilitate other
developers to develop mobile applications customized to different user
requirements. Hence, the product BrailleBand is open for innovations.

Figure 4: BrailleBand mobile applications and integration of existing
applications with product BrailleBand

####  II-C3 Microcontroller programming

The microcontroller is programmed using Arduino software to convert the
character which is being received from the bluetooth module to the respective
braille pattern. Then using the I2C communication protocol the microcontroller
addresses the desired haptic motors and activate them.

###  II-D Testing and verification

Three experiments were conducted to test and validate the hypothesis which
resulted in the following parameters:

  1. 1.

Reading speed

  2. 2.

Character transfer rate (CTR)

  3. 3.

Device usability score

A sample of 10 blind subjects (7 male and 3 female) were chosen from the Blind
School, Rathmalana, Sri Lanka within an age range of 16 to 19 years based on
the criteria of having experience with Braille language for more than 5 years.
No initial training was given to the students but 15 minutes were taken to
familiarize the students with the device using the BrailleBand Teacher mobile
application. The sample was tested with the device where 3 character words
were sent with different character gaps. The critical parameter which governs
the Reading Speed and the CTR is the Character gap and hence we focused on
reducing the character gap and identifying the minimum character gap which
does not significantly differ statistically with the character gap which gives
the best average reading accuracy (90.1%) using Analysis of Variance (ANOVA),
Bonferroni and Holm tests. It was found that the minimum character gap
satisfying the above conditions is 1000 ms. Refer to Table I.

$CTR=\frac{number\,of\,characters\,sent}{time\,taken}$ (1)

An exact CTR cannot be calculated since the number of dots to be vibrated vary
with characters. Hence, a minimum (for “q”) and a maximum (for “a”) CTR was
calculated and a average CTR was derived by using (2).

$CTR_{average}=\frac{{CTR_{maximum}}+{CTR_{minimum}}}{2}$ (2)

The subject was given a score ranging from 0 to 10 to rate the usability of
the BrailleBand based on the preference to adopt and use this device for
communication purposes.

##  III RESULTS

###  III-A Development of the BrailleBand

BrailleBand is a blind support wearable which is light weight, portable and
easy to use. Moreover, it mitigates the usability and affordability issues
associated with the current solutions available. Blind support wearable
solutions are still progressing in the experimental stages and hence the
market is open with minimum competition. BrailleBand caters a broader scope of
user needs such as information transmission, navigation and mobility
assistance, accessibility to computers and smart devices.

###  III-B Reading speed test

The sample was tested with random words consisting of three characters and the
character gap was reduced as in Table I and the following reading accuracies
were noted. With decreasing character gap the reading speed increases as
characters are being transmitted faster.

TABLE I: Reading speed test results Character gap(ms) | Average reading accuracy(%) | Standard deviation
---|---|---
2000 | 90.1 | 10.94
1500 | 90.1 | 6.24
1200 | 87.9 | 6.24
1000 | 87.9 | 12.11
800 | 69.1 | 18.03
500 | 53.2 | 21.02
400 | 46.4 | 22.84

It is notable that with practice the reading speed can be improved to a
greater extent.

###  III-C Character Transfer Rate (CTR)

A dot in the braille cell is vibrated for a period of 300 ms and the gap
between the vibration of two consecutive dots is also 300 ms. Therefore, for a
single dot to be vibrated it will take 600 ms.

An ANOVA test can be done to check whether the average reading accuracies
obtained in Table I are statistically significantly different with a
confidence level of 95%.

TABLE II: One-way ANOVA for 7 independent character gaps source | sum of squares | df | mean square | F statistic | p-value
---|---|---|---|---|---
treatment | 21,168.37 | 6 | 3,528.06 | 15.1239 | 0
error | 14,696.5 | 63 | 233.28 | - | -
total | 35,864.87 | 69 | - | - | -

The conclusion of the ANOVA test as tabulated in Table II is that the p-value
corresponding to the F-statistic of one-way ANOVA is lower than 0.05 (95%
confidence), suggesting that the sample data of one or more treatments
(character gaps) are significantly different to the others. Therefore, a
multiple comparison method should be used to assess the statistical
significance of the differences between the means of the sample data under
different treatments.

Hence, Bonferroni and Holm multiple comparison tests follow to test only pairs
relative to a certain treatment (character gap) to verify statistical
significant differences. Since the character gap 1500 ms gives the “highest
reading accuracy (90.1%)” while having the least standard deviation (refer to
Table I), we choose character gap = 1500 ms as the treatment against which
other treatments are compared for statistical significant differences.

TABLE III: Bonferroni and Holm results: only pairs relative to Character gap=1500 ms simultaneously compared Treatment pairs |  Bonferroni and Holm T-statistic |  Bonferroni inference |  Holm inference
---|---|---|---
1500 Vs 2000 |  0.00 |  Insignificant |  Insignificant
1500 Vs 1200 |  0.32 |  Insignificant |  Insignificant
1500 Vs 1000 |  0.32 |  Insignificant |  Insignificant
1500 Vs 800 |  3.07 |  Insignificant |  Significant
1500 Vs 500 |  5.40 |  Significant |  Significant
1500 Vs 400 |  6.40 |  Significant |  Significant

From the above results table of Bonferroni and Holm multiple comparison tests,
we can observe that there is statistically insignificant difference between
the sample data under Character Gap=1500 ms, Character Gap=2000 ms, Character
Gap=1200 ms, and Character Gap=1000 ms.

Therefore, we can use the character gap of 1000 ms for Character Transfer Rate
(CTR) calculations as the sample data do not show a statistically significant
difference with the treatment which gives the highest reading accuracy of
90.1% (Character Gap=1500 ms).

Therefore, we can calculate the maximum CTR which is corresponding to the
character ”a” and minimum CTR corresponding to the character ”q” using
Equation 1.

$CTR_{maximum}=\frac{1}{0.6+1}=0.625\,characters\,s^{-1}$

$CTR_{minimum}=\frac{1}{0.6\times 5+1}=0.25\,characters\,s^{-1}$

Therefore, from Equation 2,

$CTR_{average}=\frac{0.625+0.25}{2}=0.4375\,characters\,s^{-1}$

It is observed that the average CTR is 0.4375 characters per second and this
also can be improved with practise.

###  III-D Usability score

All the subjects were requested to rate the usability of the BrailleBand out
of 10 and the resultant average usability score was 8.7.

##  IV DISCUSSION

BrailleBand caters a wider range of blind support needs and it has many
applications such as reading, navigation and smart device accessibility. It is
a complete package of the device and the mobile applications for smart phones.

However, the replying system of the BrailleBand is still under development.
The capacitive touch pads and capacitive touch controllers are the
constituents of the replying system which allows the user to reply back to
messages using the same braille dot pattern. Refer to Fig. 2.

Additionally, further research should be done on the health and safety aspect
of the product as well. The go-to-market plan should include a market research
to identify the additional features demanded by the users and to identify the
price the users are willing to pay. Moreover, long term sustainable usability
of the product should also be verified as BrailleBand will be a lifetime
product for a blind person focusing on robustness of the device and biological
effects as well (for example: nerve adaptations for vibrations).

The reading speed and the CTR can be further improved with practise. Moreover,
from the device end we can reduce the dot vibration time which is currently at
300 ms to a lesser value and transmit characters faster. But the compromise
here is the effect of the vibrating sensation to the user.

As blind people usually lack financial strength to afford sophisticated
devices. Therefore, the BrailleBand will be a better solution. Yet, the price
of BrailleBand should be subsidized through corporate and government funding
to reach the wider blind community.

Wearable assistive technology solutions for blind are still at an emerging
level with lot of research and development being done around the world.
BrailleBand integrates haptics in developing an assistive device for blind
support.

##  V CONCLUSIONS

The blind community uses the sense of hearing and sense of touch to interact
and understand the surrounding environment. Hence, the sense of touch becomes
the primary sensory modality to communicate non-audible information to and
from a blind person.

The ‘BrailleBand’ haptic wearable blind support device connected to smart
phone applications helps the blind community to lead an independent quality
life. This assistive technology enables information transmission, navigation
and smart device accessibility through the sense of touch. With the
development of the BrailleBand we have successfully implemented a new mode of
communication to the blind using haptic technology.

The product feasibility tests conducted showed promissing results with regard
to reading speed and reading accuracy which improved with practice. Based on
experimental results an average CTR of 0.4375 characters per second was
achieved.

## ACKNOWLEDGMENT

We extend our gratitude to Dr. Ajith Pasqual and all the academic staff of
Department of Electronic and Telecommunication Engineering, University of
Moratuwa for veteran assistance and guidance given in developing the product
BrailleBand. We acknowledge the Blind School, Rathmalana, Sri Lanka for
accommodating the testing and verification process by granting patient access.

## References

  * [1] Blind World Magazine (2006).  Breaking the chains of paternalism [online].  Available online at: http://home.earthlink.net/~blindworld/NEWS/6-06-14-02.htm.
  * [2] J Brabyn, KD Seelman, and S Panchang.  Aids for people who are blind or visually impaired.  An Introduction to Rehabilitation Engineering, ed. RA Cooper, H. Ohnabe, and DA Hobson, pages 287–313, 2007.
  * [3] Nicholas Caporusso.  A wearable malossi alphabet interface for deafblind people.  In Proceedings of the working conference on Advanced visual interfaces, pages 445–448. ACM, 2008.
  * [4] Tanay Choudhary, Saurabh Kulkarni, and Pradyumna Reddy.  A braille-based mobile communication and translation glove for deaf-blind people.  In Pervasive Computing (ICPC), 2015 International Conference on, pages 1–4. IEEE, 2015.
  * [5] Ben AG Elsendoorn.  Assistive technology for the hearing-impaired, deaf and deafblind, by marion e. hersch and michael a. johnson, eds.  Technology and Disability, 16(2):111–113, 2004.
  * [6] SR Geiger.  Handbook of physiology, Section 1: The nervous system.  American Physiological Society, 1984.
  * [7] Ulrike Gollner, Tom Bieling, and Gesche Joost.  Mobile lorm glove: introducing a communication device for deaf-blind people.  In Proceedings of the Sixth International Conference on Tangible, Embedded and Embodied Interaction, pages 127–130. ACM, 2012.
  * [8] JA Miele.  Wearabraille: development of a wireless, virtual, braille keyboard.  in 25th Annu. Int. Technology & Persons with Disabilities Conf. (2010).
  * [9] World Health Organization et al.  Visual impairment and blindness. fact sheet no. 282. updated august 2014, 2014.
  * [10] R. Sekuler and R. Blake.  Perception.  Number v. 1 in Perception. McGraw-Hill, 2002.
  * [11] Ramiro Velázquez.  Wearable assistive devices for the blind.  In Wearable and autonomous biomedical devices and systems for smart environment, pages 331–349. Springer, 2010.


# LMR: A Large-Scale Multi-Reference Dataset for Reference-based Super-
Resolution

Lin Zhang
CASIA Work done during an internship at Baidu Inc. Xin Li
Baidu Inc  Dongliang He
Baidu Inc  Fu Li
Baidu Inc  Errui Ding
Baidu Inc  Zhaoxiang Zhang
CASIA

###### Abstract

It is widely agreed that reference-based super-resolution (RefSR) achieves
superior results by referring to similar high quality images, compared to
single image super-resolution (SISR). Intuitively, the more references, the
better performance. However, previous RefSR methods have all focused on
single-reference image training, while multiple reference images are often
available in testing or practical applications. The root cause of such
training-testing mismatch is the absence of publicly available multi-reference
SR training datasets, which greatly hinders research efforts on multi-
reference super-resolution. To this end, we construct a large-scale, multi-
reference super-resolution dataset, named LMR. It contains 112,142 groups of
300$\times$300 training images, which is 10$\times$ of the existing largest
RefSR dataset. The image size is also much larger. More importantly, each
group is equipped with 5 reference images with different similarity levels.
Furthermore, we propose a new baseline method for multi-reference super-
resolution: MRefSR, including a Multi-Reference Attention Module (MAM) for
feature fusion of an arbitrary number of reference images, and a Spatial Aware
Filtering Module (SAFM) for the fused feature selection. The proposed MRefSR
achieves significant improvements over state-of-the-art approaches on both
quantitative and qualitative evaluations. https://github.com/wdmwhh/MRefSR

##  1 Introduction

Figure 1: Visual comparison of single-reference training RefSR method
$C^{2}$-Matching [12] and our multi-reference training MRefSR. Our MRefSR can
more fully utilize arbitrary number of multiple reference images to achieve
the best results. This figure is best viewed by zoom-in.

Single image super-resolution (SISR) is to restore a degraded low-resolution
(LR) image to a texture-realistic high-resolution (HR) image [11]. SISR has a
wide range of applications in surveillance [39], astronomy [8], medical
imaging [7], film and television [23, 14], and other industries [26, 37, 33].
With the development of deep learning, SISR has made great progress over these
years [4, 5, 16, 18, 13, 40, 31, 19, 3, 22]. Compared with SISR, reference-
based super-resolution (RefSR) can leverage textures from additional similar
HR reference images, so it often achieves better performance.

Because of promising results shown by recent RefSR methods [28, 36, 42, 41,
27, 29, 12, 20, 34], it attracts more and more research interest. However, all
these previous RefSR methods have focused on using a single reference image
for training, but there are often multiple reference images available for
testing or practical applications. To the best of our knowledge, the only
RefSR training dataset currently available is CUFED5 [41, 32], which has only
11,871 image pairs with a small resolution of 160$\times$160\. More
importantly, there is only one reference image for each LR input image.
However, in practical applications, multiple reference images are often
encountered. For example, testing set of CUFED5 has 126 input images and each
has 5 reference images with different similarity levels. Similarly, we can
also easily find multiple reference images for any real test case. Due to the
limitation of the only available training dataset, previous RefSR methods do
not make good use of multiple reference images in testing or practical
applications. The previous RefSR methods usually stitch together several
reference images to get a large resolution image as one reference image to fit
the models trained with only one reference image. Nevertheless, if the
resolution of the reference images is too large, this way of testing will
exhaust the GPU memory. Furthermore, the relationship among multiple reference
images is not modeled effectively. So this is certainly much worse than a
method designed specifically for multiple reference images. Therefore, a
multi-reference RefSR training dataset and a simple but effective multi-
reference RefSR method are needed.

In this paper, we propose a large-scale, multi-reference RefSR dataset, named
LMR. The training set of LMR consists of 112,142 groups of 300$\times$300
training images, each group containing 5 reference images of different
similarity levels. LMR training dataset has 10 times images compared to CUFED5
and the image size is also much larger. Such a sufficiently large training
dataset will be beneficial for improving the generalization ability of models.
We believe this training dataset will greatly facilitate the RefSR research as
it is the first RefSR training dataset with multiple reference images.
Meanwhile, the testing set of LMR has 142 groups of images and each group with
2$\sim$6 reference images. The side length of the testing images ranges from
800 to 1600.

With the help of LMR, we propose a new RefSR baseline method for multiple
reference RefSR, named MRefSR. First, we develop a Multi-Reference Attention
Module (MAM) for feature fusion from an arbitrary number of reference images.
We treat the LR input feature as query, and candidate keys and values are
generated from the aligned reference features corresponding to different
reference images. Then, attention across different aligned reference features
is conducted to fuse features from different reference images. Second, since
not all LR feature points can well match the reference features, we use
Spatial Aware Filtering Module (SAFM) for fused feature selection. As shown in
Figure 1, our MRefSR effectively utilizes information from multiple reference
images to produce visually pleasing details. In summary, our contributions are
threefold:

  * •

We contribute the first multi-reference RefSR dataset, named LMR, which
contains 112,142 groups of 300$\times$300 training images and each group has 5
reference images for the input image. This dataset will enable RefSR research
from single-reference to multi-reference images and largely promote the
development of the RefSR research field.

  * •

We propose a novel multi-reference baseline RefSR method MRefSR, using a
multi-reference attention module for feature fusion of an arbitrary number of
reference images, and a spatial aware filtering module for the fused feature
selection. Our method effectively learns the relationship among multiple
references and makes the best use of them, this is also thanks to the multi-
reference dataset LRM.

  * •

We conduct extensive experiments which demonstrate the superiority of the
proposed LMR and the potential of multi-reference RefSR methods. Our method
achieves significant improvements over state-of-the-art approaches on both
quantitative and qualitative evaluations.

##  2 Related Work

###  2.1 Reference-based Image Super-Resolution

RefSR is gradually becoming an emerging research field. Compared with SISR,
RefSR is more advantageous because it can utilize the information of
additional HR reference images with similar contents. SRNTT [41] proposed an
end-to-end network structure that performs multi-scale adaptive texture
transfer from the reference image to recover the SR image. Subsequently, TTSR
[35] applied a cross-scale feature integration method to merge multi-scale
reference features. MASA [20] designed a coarse-to-fine patch matching scheme
to reduce the computational complexity. Consequently, $C^{2}$-Matching [12]
got more accurate pre-offsets of reference features to LR features by a
teacher-student correlation distillation and a dynamic DCN [2, 43] aggregation
module. AMSA [34] made an incremental extension of $C^{2}$-Matching by
introducing multi-scale aggregation and coarse-to-fine patch matching. Huang
_et al_. [10] also used the $C^{2}$-Matching model, but added an additional
SISR network to decouple the texture transfer and the super-resolution, which
made the network parameters much larger and the inference much slower.
Recently, RRSR [38] and DATSR [1] also introduce reciprocal learning and
transformers to boost the performance. Although previous methods have made
great progress, all of the above methods focus on research exploration using
only one single reference image due to the limitation of the only available
training dataset, CUFED5.

###  2.2 RefSR Datasets

Target |  Ref-1 |  Ref-2 |  Ref-3 |  Ref-4 |  Ref-5
---|---|---|---|---|---

Figure 2: Two groups of sample images from our LMR training dataset. From left
to right, there is one target image, one high-similarity (H) reference image,
two medium-similarity (M) reference images, and two low-similarity (L)
reference images.

To the best of our knowledge, there are five datasets commonly used in RefSR
research: Sun80 [28], Urban100 [9], Manga109 [21], WR-SR [12] and CUFED5 [41,
32]. However, the first four are all testing sets. The Sun80 dataset contains
80 natural images, each with 20 web-search reference images, but these
reference images are not very similar to the corresponding LR input, so it is
not suitable as a testing set for RefSR. The Urban100 dataset contains 100
building images, lacking references. Because of self-similarity in the
building image, the corresponding LR image is usually treated as the reference
image. The Manga109 dataset contains 109 manga images without references.
Since all the images in Manga109 are the same category (manga cover), the
previous methods randomly use one HR image in the dataset as a reference
image. The WR-SR dataset with more diverse categories, contains 80 image
pairs, each target image accompanied by a web-searching reference image.
CUFED5 [41, 32] is the only dataset with a training set, which has 11,871
image pairs with a small resolution of 160$\times$160 and only one reference
image for the LR input in each image pair. CUFED5 testing set has 126 input
images and each has 5 reference images with different similarity levels.
Recently, Wang _et al_. [30] proposed a new dataset named CameraFusion for
dual-camera super-resolution with 131 training image pairs and 15 testing
image pairs. However, the image pairs captured by the dual-camera is too ideal
for the RefSR task, and the number of dataset is too small. In this paper, to
better meet the demands of RefSR research, we propose LMR, a large-scale
multi-reference RefSR dataset.

##  3 Approach

In this section, we first introduce the proposed Large-scale Multi-reference
RefSR dataset LMR in Sec. 3.1. Subsequently, we detail a new baseline RefSR
method MRefSR using multiple references in Sec. 3.2.

###  3.1 Construction of LMR

Figure 3: The proposed Multi-Reference Attention Module (left) for the multi-
reference feature fusion and the Spatial Aware Filtering Module (right) for
the fused feature selection. Both modules perform pixel-wise functions.

The MegaDepth [17] dataset was originally proposed for single-view depth
prediction. They used a large number of Internet images from overlapping
viewpoints to obtain the dense depth by COLMAP, a state-of-the-art SfM system
[24] (for reconstructing camera poses and sparse point clouds) and MVS system
[25] (for generating dense depth maps). The generated dense depth maps of the
COLMAP are used as the supervised targets for single-view depth prediction
model training. MegaDepth contains 1,070,468 internet photos of landmarks
around the world and reconstructs 196 3D landmark models from these photos.
Each photo of the same landmark varies widely in viewpoint, scene extent, and
focused buildings. The scene of finding Internet images from overlapping
viewpoints for 3D reconstruction is very similar to finding reference images
for target images to do reference-based super-resolution. Inspired by this,
the image groups in the off-the-shelf MegaDepth dataset are very suitable for
making a RefSR dataset. Consequently, we propose a new large-scale multi-
reference RefSR dataset, dubbed LMR.

To construct the LMR training image patch groups, we first perform the
following preprocessing steps on the original MegaDepth dataset to obtain
similar image pairs.

  * •

Firstly, the PSNR of the target image and the candidate reference images
should be lower than $30$dB to filter duplicate images.

  * •

Secondly, the candidate reference images and the target image should have some
similar contents, and we achieve this filtering by controlling the overlap
ratio $R_{olp}$ of matched keypoints in the sparse 3D point clouds.

  * •

Thirdly, the size ratio $R_{s}$ of the same object in the reference image and
the target image cannot be too small, otherwise the reference image cannot
provide enough detailed texture information.

We calculate $R_{s}$ and $R_{olp}$ following the existing code in D2-Net [6],
a method for image matching and 3D reconstruction.

Further, we define three similarity levels for these image pairs, that is high
similarity ($H$), medium similarity ($M$) and low similarity ($L$). A image
pair is categorized as $H$ if the overlap ratio $R_{olp}$ is greater than
$30\%$ and the size ratio $R_{s}$ is larger than $0.9$, $M$ if $R_{olp}$ is
greater than $10\%$ and the $R_{s}$ is larger than $0.66$, otherwise $L$.

Through the above operations, we can obtain a large number of image groups,
each containing one target image and multiple reference images. However, due
to GPU memory limitation, it is often not possible to use the entire large
image to train the network. For SISR, it is common to randomly crop a patch
from the image for training. While in the case of RefSR, it is better to crop
corresponding patches with similar contents in the reference images and the
target image, e.g. CUFED5 cropped 11,871 paired 160$\times$160 patches as the
training set. For the multi-reference dataset LMR, we first randomly crop a
patch from the target image. Then, we map the center point of the cropped
patch into 3D sparse point cloud and pick up 5 keypoints near the mapped
point, which are from 5 reference images with different similarities (one H,
two M, two L). Next, we take the selected keypoints as centers and crop the
corresponding patches. In this way, we collect a total of 112,142 groups of
300$\times$300 patches as the training set, which is ten times larger than
CUFED5 and the image size is much larger too. More importantly, each group has
5 reference image patches of different similarities. Some representative
samples are presented in Figure 2. As shown in Sec. 4, the model trained on
the LMR dataset shows good generalization performance on other RefSR datasets,
demonstrating the effectiveness of the LMR.

In addition to the LMR training set, we also prepare a testing set for multi-
reference RefSR testing. We remove the images containing target or reference
patches that appeared in the training set. From the remaining image pairs, we
construct a testing set consisting of 142 groups, each containing a target
image and 2$\sim$6 reference images with image side lengths between
800$\sim$1600.

###  3.2 Multi-Reference RefSR network

Armed with the LMR dataset, we propose a multi-reference RefSR network to make
good use of multiple reference images, dubbed MRefSR. Our MRefSR is based on
$C^{2}$-Matching [12] as it is currently the open source method with best
performance and easy to get started. Note that other RefSR frameworks such as
TTSR [35] are also applicable since we aims to exploit multi-reference
features instead of single-reference feature transfer. As $C^{2}$-Matching
did, a _Content Extractor_ (CE) is used to extract features $F_{LR}$ from $LR$
image. Multi-scale ($1\times$, $2\times$ and $4\times$) reference features
$F^{s}_{Ref_{i}}$ are extracted by a _VGG_ extractor, where $s=1,2,4$ and
$i\in\\{1,2,...,N\\}$, $N$ is the number of reference images. For the sake of
brevity, the $s$ in the following are omitted, and $F_{Ref_{i}}$ is used
instead of $F^{s}_{Ref_{i}}$. A pretrained _Contrastive Correspondence
Network_ (CCN) is used to obtain the relative target offsets $O_{i}$ of the LR
input and the corresponding multiple reference images. Afterwards, as shown in
Figure 3, we develop a Multi-Reference Attention Module (MAM) for the multi-
reference feature fusion and a Spatial Aware Filtering Module (SAFM) for the
fused feature selection.

_Dynamic Aggregation Module_ in $C^{2}$-Matching is used to get the aligned
features $F_{a_{i}}$ from the reference features $F_{Ref_{i}}$ by the
corresponding pre-offsets $O_{i}$. After that, we introduce MAM to fuse the
aligned features from different reference images. In detail, at each feature
scale, we first generate corresponding $N$ attention maps for the aligned
features of $N$ reference images:

$\displaystyle att_{i}(x,y)$ $\displaystyle=softmax(\left\langle
Q(x,y),K_{i}(x,y)\right\rangle)$ (1) $\displaystyle=\frac{exp(\left\langle
Q(x,y),K_{i}(x,y)\right\rangle)}{\sum_{j=1}^{N}exp(\left\langle
Q(x,y),K_{j}(x,y)\right\rangle)}.$

We use inner product to measure the similarity between the features $Q(x,y)$
and $K_{i}(x,y)$ at the point $(x,y)$, where query $Q$ is obtained from the LR
input feature $F_{LR}$, key $K_{i}$ and value $V_{i}$ are obtained from the
$i$-th reference image aligned feature $F_{a_{i}}$:

$\displaystyle Q$ $\displaystyle=conv_{q}(F_{LR}),$ (2) $\displaystyle K_{i}$
$\displaystyle=conv_{k}(F_{a_{i}}),$ $\displaystyle V_{i}$
$\displaystyle=conv_{v}(F_{a_{i}}),$

where $conv_{q}$, $conv_{k}$ and $conv_{v}$ are convolutions with kernel size
3$\times$3 and stride 1. Then, we get fused reference feature $F_{fref}$ from
all reference images:

$\displaystyle F_{fref}(x,y)=\sum_{i=1}^{N}(att_{i}(x,y)\cdot V_{i}(x,y)).$
(3)

The proposed MAM enables MRefSR to handle an arbitrary number of reference
images during training and testing phases, making the MRefSR more flexible for
practical applications.

Since not all LR feature pixels can be well matched with reference features,
we use the proposed SAFM for the selection of fused reference features
$F_{fref}$. As shown in Figure 3, we get two masks $M_{mul}$ and $M_{add}$
from the concatenated feature of $F_{LR}$ and $F_{fref}$ and a $sigmoid$
function is used to limit the range of the $M_{mul}$.

$\displaystyle M_{mul}$ $\displaystyle=sigmoid(f_{1}(F_{LR}\|F_{fref}))\cdot
2,$ (4) $\displaystyle M_{add}$ $\displaystyle=f_{2}(F_{LR}\|F_{fref}),$

where $f_{1}$ and $f_{2}$ are nonlinear mapping functions consisting of
convolution and leaky ReLU layers. At last, the $M_{mul}$ and $M_{add}$ are
used for the final selected reference features $F_{sref}$:

$\displaystyle F_{sref}=F_{fref}\odot M_{mul}+M_{add},$ (5)

where $\odot$ denote element-wise multiplication.

In the end, a restoration module $\mathcal{G}$ takes the LR features $F_{LR}$
and the selected reference features $F_{sref}$ to reconstruct the target
image:

$X_{SR}=\mathcal{G}(F_{LR},F_{sref}).$ (6)

###  3.3 Implementation Details

We train and evaluate our MRefSR in a scale factor $4\times$. In detail, we
train the network for 255K iterations using Adam optimizer [15] with
parameters $\beta_{1}=0.9$, $\beta_{2}=0.999$, and constant learning rate of
1e-4. Each mini-batch includes 48 groups of image patches, each consisting of
an LR input patch with size 40$\times$40 and five reference HR patches with
size 160$\times$160\. We use three commonly used loss functions to train our
model, including reconstruction loss $L_{rec}$, perceptual loss $L_{per}$ ,
and adversarial loss $L_{adv}$, referring to supplementary material for the
network training loss details. The weight coefficients for $L_{rec}$,
$L_{per}$ and $L_{adv}$ are set to 1, 1e-4 and 1e-6. The network is first
trained with $L_{rec}$ only and then finetuned with all losses. During
training, we augment the training data by randomly horizontally flipping and
vertically flipping, and random 90° rotation. Following the standard protocol,
we generate all LR images by bicubically downsampling the HR images with a
scale factor of $4\times$. All experiments run in parallel on 4 NVIDIA V100
GPUs. For the quantitative comparison, we train MRefSR without GAN loss and
perceptual loss as other methods did. Benefiting from the large-resolution
training images of LMR, we get an LPF (Large Patch Finetuning) version of the
model, which is finetuned using the large-patch training images.

##  4 Experiments

Table 1: We report PSNR/SSIM on Y channel of YCbCR space to compare among
different SR methods on the testing set of LMR, CUFED5 [41, 32], Sun80 [28],
and WR-SR [12]. Methods are grouped by SISR methods (top) and reference-based
methods (bottom). The best results are marked in bold and with underlines. The
second best and the third best results are marked in bold and with underlines,
respectively. $C^{2}$-Matching-LMR means $C^{2}$-Matching-$rec$ is trained on
the LMR dataset and the Ours-$rec$-LPF indicates that the model was finetuned
using large patch size (300$\times$300) training images.

Method |  Training Dataset | LMR | CUFED5 [41, 32] | Sun80 [28] | WR-SR [27]
---|---|---|---|---|---
PSNR$\uparrow$ / SSIM$\uparrow$ | PSNR$\uparrow$ / SSIM$\uparrow$ | PSNR$\uparrow$ / SSIM$\uparrow$ | PSNR$\uparrow$ / SSIM$\uparrow$
SRCNN [4] | CUFED5 | - |  $25.33$ / $0.745$ |  $28.26$ / $0.781$ |  $27.27$ / $0.767$
EDSR [18] | CUFED5 | - |  $25.93$ / $0.777$ |  $28.52$ / $0.792$ |  $28.07$ / $0.793$
RCAN [40] | CUFED5 | - |  $26.33$ / $0.781$ |  $29.97$ / $0.814$ |  $27.91$ / $0.793$
RRDB [31] | CUFED5 | - |  $26.41$ / $0.783$ |  $29.99$ / $0.814$ |  $27.96$ / $0.793$
RCAN [40] | LMR |  $29.63$ / $0.841$ |  $26.58$ / $0.785$ |  $\mathbf{30.36}$ / $\mathbf{\underline{0.821}}$ |  $28.24$ / $0.798$
RRDB [31] | LMR |  $29.68$ / $0.842$ |  $26.61$ / $0.786$ |  $\mathbf{\underline{30.37}}$ / $\mathbf{\underline{0.821}}$ |  $28.25$ / $0.798$
Landmark [36] | CUFED5 | - |  $24.91$ / $0.718$ |  $27.68$ / $0.776$ | -
CrossNet [42] | CUFED5 | - |  $25.48$ / $0.764$ |  $28.52$ / $0.793$ | -
SRNTT-$rec$ [41] | CUFED5 | - |  $26.24$ / $0.784$ |  $28.54$ / $0.793$ |  $27.59$ / $0.780$
TTSR-$rec$ [35] | CUFED5 |  $29.13$ / $0.832$ |  $27.09$ / $0.804$ |  $30.02$ / $0.814$ |  $27.97$ / $0.792$
MASA-$rec$ [20] | CUFED5 |  $29.42$ / $0.837$ |  $27.54$ / $0.814$ |  $30.15$ / $0.815$ |  $28.19$ / $0.796$
$C^{2}$-Matching-$rec$ [12] | CUFED5 |  $30.01$ / $0.856$ |  $28.40$ / $0.846$ |  $30.18$ / $0.817$ |  $28.32$ / $0.801$
AMSA-$rec$ [34] | CUFED5 | - |  $28.50$ / $0.849$ |  $30.29$ / $0.819$ | -
TDF-$rec$ [10] | CUFED5 | - |  $28.64$ / $0.850$ |  $30.31$ / $\underline{0.820}$ |  $\underline{28.52}$ / $\mathbf{\underline{0.807}}$
$C^{2}$-Matching-LMR | LMR |  $\underline{30.64}$ / $\underline{0.869}$ |  $\underline{28.65}$ / $\underline{0.853}$ |  $30.31$ / $0.819$ |  $\mathbf{28.53}$ / $\mathbf{\underline{0.807}}$
Ours-$rec$ | LMR |  $\mathbf{31.81}$ / $\mathbf{0.895}$ |  $\mathbf{28.94}$ / $\mathbf{0.860}$ |  $30.28$ / $0.819$ |  $\underline{28.52}$ / $0.806$
Ours-$rec$-LPF | LMR |  $\mathbf{\underline{31.98}}$ / $\mathbf{\underline{0.898}}$ |  $\mathbf{\underline{29.05}}$ / $\mathbf{\underline{0.862}}$ |  $\underline{30.32}$ / $0.819$ |  $\mathbf{\underline{28.59}}$ / $\mathbf{\underline{0.807}}$

LR input |  Ref-1 |  Ref-2 |  Ref-3 |  Ref-4 |  Ref-5
---|---|---|---|---|---
Target HR |  ESRGAN |  MASA |  $C^{2}$-Matching |  $C^{2}$-Matching-LMR |  Ours

Figure 4: Qualitative comparisons on the testing set of CUFED5 (the top four
examples) and LMR (the bottom four examples). We compare our results with
ESRGAN, MASA, $C^{2}$-Matching, $C^{2}$-Matching-LMR. All these methods are
trained with GAN loss. Our method reconstructs sharper details than other
methods.  Figure 5: User study results. Values on Y-axis denote the voting
percentage of users favoring our method. Figure 6: Visual comparisons of
ablation study on MAM and SAFM.

###  4.1 Datasets and Metrics

We train our network on the proposed LMR training set and evaluate it on the
testing set of LMR, CUFED5 [41, 32], Sun80 [28] and WR-SR [12]. As mentioned
earlier, LMR and CUFED5 are two real multi-reference testing sets. Although
Sun80 has multiple reference images, these reference images are not very
similar to the corresponding target images. WR-SR is a single-reference
testing set. According to previous practice, due to the relatively small
images of CUFED5 testing set (300$\times$500), when testing other single-
reference RefSR methods on CUFED5, we stitch multiple reference images into
one large reference image for testing. However, on the LMR and Sun80, due to
the large image resolution of the testing sets and the limitation of GPU
memory, other RefSR methods cannot be tested by the stitching reference image
together and can only use a single reference image. With the multi-reference
attention module (MAM), our MRefSR can utilize multiple reference images for
prediction on the LMR, CUFED5 and Sun80 testing sets. we adopt two
quantitative metrics, PSNR and SSIM, both calculated on Y channel in the
transformed YCbCr color space. To evaluate the results qualitatively, we show
the visual results of different methods and conduct a user study for
subjective visual quality comparison.

###  4.2 Comparison with State-of-the-Art Methods

We compare the proposed MRefSR with previous state-of-the-art SISR methods and
single-reference RefSR methods. SISR methods include SRCNN [4], EDSR [18],
RCAN [40], RRDB [31] and ESRGAN [31]. As for single-reference RefSR methods,
Landmark [36], CrossNet [42], SRNTT [41], TTSR [35], MASA [20],
$C^{2}$-Matching [12], AMSA [34] and TDF [10] are included. For fair
comparison, we retrain three high-performance SISR methods RCAN, RRDB and
ESRGAN, and one open-sourced top-performing single-reference RefSR method
$C^{2}$-Matching on the training set of LMR.

Quantitative evaluation. As shown in Table 1, our MRefSR outperforms other
methods by a large margin on two real multiple reference datasets, CUFED5 and
LMR. On the most commonly used CUFED5 benchmark, MRefSR outperforms the
retrained $C^{2}$-Matching-LMR by 0.29dB. Models trained on LMR can achieve
better performance on CUFED5, which also demonstrates the generalization
ability and effectiveness of LMR. What’s more, MRefSR shows a significant
improvement of 1.15 dB over the second best method on the LMR testing set. The
above two results demonstrate the superiority of learning the interaction
among multiple references, further manifesting the necessity of the LMR
dataset that enables multi-reference RefSR training. On Sun80, SISR methods
RRDB and RCAN get the best two results. The results gap of the top RefSR
methods AMSA-rec, TDF-rec, $C^{2}$-Matching-LMR and MRefSR are less than 0.04
dB, which further proves the reference image and its target image in Sun80 are
not very similar. On the WR-SR benchmark, since there is only one reference
image per LR, our results are very close to $C^{2}$-matching-LMR.

Qualitative evaluation. As shown in Figure 4, we compare the results of
ESRGAN, MASA, $C^{2}$-Matching, $C^{2}$-Matching-LMR and our MRefSR. The top
four examples are from CUFED5, and the models trained on LMR generalize well
on the CUFED5 testing set, demonstrating the effectiveness of the proposed
LMR. What’s more, the results of our MRefSR trained with multiple references
are much better than those trained with a single reference image. We also show
four examples from the LMR testing set, and MRefSR can recover more texture
details than other methods.

Besides, we perform a user study to compare with some typical methods
including ESRGAN, MASA and $C^{2}$-Matching. Specifically, in each test, we
present paired super-resolution results, one of which is generated by our
MRefSR, and ask the users to choose the one with higher visual quality. As
shown in Figure 5, the users prefer our results over the others.

###  4.3 Ablation Study

In this section, we verify the effectiveness of Multi-reference Attention
Module (MAM) and Spatial Aware Filtering Module (SAFM). Besides, we
demonstrate the benefit of large-resolution training images of LMR. At last,
we also investigate the impact of number of reference images.

Table 2: Ablation study on the influence of MAM, SAFM and LPF.

Method | LMR | CUFED5
---|---|---
PSNR$\uparrow$ / SSIM$\uparrow$ | PSNR$\uparrow$ / SSIM$\uparrow$
Baseline($C^{2}$-Matching-LMR) |  $30.64$ / $0.869$ |  $28.65$ / $0.853$
Baseline+MAM |  $31.70$ / $0.894$ |  $28.85$ / $0.859$
Baseline+MAM+SAFM |  $31.81$ / $0.895$ |  $28.94$ / $0.860$
Baseline+MAM+SAFM+LPF |  $31.98$ / $0.898$ |  $29.05$ / $0.862$

The effectiveness of MAM and SAFM. As shown in Table 2, with $C^{2}$-Matching-
LMR as the baseline, our MAM achieves a PSNR improvement of 1.06 dB on LMR and
0.20 dB on CUFED5. The reason why the improvement on LMR is larger than that
on CUFED5 is that $C^{2}$-Matching-LMR cannot use multiple reference images on
LMR due to the limitation of GPU memory, and MAM greatly solves this problem.
More importantly, our MAM supports an arbitrary number of reference images,
making it more flexible and practical. On the basis of MAM, SAFM is used to
adjust the fused reference features and the PSNR scores on LMR and CUFED5
increase to 31.81 dB and 28.94 dB, respectively. Figure 6 shows their
influence. Furthermore, thanks to the larger image size of the LMR training
data, MRefSR with large-patch (300$\times$300) finetuning strategy (LPF) can
consistently improve the performance by roughly 0.1 dB on LMR and CUFED5. This
result reflects the advantage of the large training images of the LMR dataset.

Table 3: The effect of different number of reference images on CUFED5.

$\\#Num$ |  $C^{2}$-Matching-LMR | Ours
---|---|---
$n=1$ | $28.474$ | $28.663$
$n=2$ |  $28.615$ ($+0.141$) |  $28.869$ ($+0.206$)
$n=3$ |  $28.651$ ($+0.036$) |  $28.920$ ($+0.051$)
$n=4$ |  $28.649$ ($-0.002$) |  $28.932$ ($+0.012$)
$n=5$ |  $28.650$ ($+0.001$) |  $28.935$ ($+0.003$)
$\;\;\;\Delta$ | $+0.176$ | $+0.272$

The effect of the number of reference images. To study the influence of number
of reference images, we conduct experiments on the testing set of CUFED5, in
which each LR input image has five reference images. As shown in Table 3, as
the number of reference images increases, although $C^{2}$-Matching-LMR has a
slight improvement with the stitching testing strategy, the gap is still
smaller than the improvement of MRefSR. What’s more, when the number of
reference images is greater than 3, the results are worse than the case of 3
reference images, which indicates that the stitching testing strategy neglects
the interaction among references, so the information from the fourth reference
doesn’t explore with that from the first three reference effectively in this
case. In contrast, it can be seen that with the increase of reference images,
MRefSR has a stable positive gain. Last but no least, MRefSR with five
references has a PSNR increase of 0.272 dB than that with one reference,
whereas $C^{2}$-Matching-LMR only has a PSNR increase of 0.176 dB, which
further demonstrates the superiority of modeling the relationship among
multiple references.

###  4.4 Computational Cost

Here, we present the computational cost comparisons between the proposed
MRefSR and previous single-reference RefSR methods, including MASA [20] and
$C^{2}$-Matching [12]. The computational cost is computed on CUFED5 [41, 32]
using one NVIDIA V100 GPU. In specific, for the single-reference RefSR methods
on CUFED5, we stitch five reference images into a 2500$\times$500 image as the
reference image for testing. Certainly, our MRefSR can directly utilize all
the reference images for testing. Table 4 reports the GPU memory, runtime and
performance for each method. Our MRefSR consumes the least GPU memory and
achieves the best performance with acceptable runtime.

Table 4: Computational cost and performance comparisons on the testing set of
CUFED5. $C^{2}$-Matching-LMR means $C^{2}$-Matching-$rec$ is trained on our
LMR dataset.

Model | MASA-$rec$ |  $C^{2}$-Matching-$rec$ |  $C^{2}$-Matching-LMR | MRefSR-$rec$
---|---|---|---|---
GPU Memory (GB) | 21.98 | 8.37 | 8.37 | 3.42
Runtime (s) | 0.417 | 2.29 | 2.29 | 0.875
PSNR$\uparrow$ | $27.54$ | $28.40$ | $28.65$ | $28.94$
SSIM$\uparrow$ | $0.814$ | $0.846$ | $0.853$ | $0.860$

##  5 Conclusion

In this paper, we propose a large-scale multi-reference RefSR dataset: LMR.
Unlike CUFED5, the only training RefSR dataset available before, LMR has 5
reference images for each LR input image. What’s more, LMR contains 112,142
groups of 300$\times$300 training images, 10 times the number of CUFED5, and
the image size is also much larger than CUFED5. Besides, we propose a new
multi-reference baseline RefSR method, named MRefSR. We use a multi-reference
attention module (MAM) for feature fusion of an arbitrary number of reference
images, and a spatial aware filtering module (SAFM) for the fused feature
selection. With LMR enabling multi-reference RefSR training, our method
effectively models the relationship among multiple references, thus achieving
significant improvements over state-of-the-art approaches on both quantitative
and qualitative evaluations. And our method solves the mismatch problem of
previous methods using a single reference image for training but testing with
multiple reference images.

## References

  * [1] Jiezhang Cao, Jingyun Liang, Kai Zhang, Yawei Li, Yulun Zhang, Wenguan Wang, and Luc Van Gool.  Reference-based image super-resolution with deformable attention transformer.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 325–342, 2022.
  * [2] Jifeng Dai, Haozhi Qi, Yuwen Xiong, Yi Li, Guodong Zhang, Han Hu, and Yichen Wei.  Deformable convolutional networks.  In ICCV, pages 764–773, 2017.
  * [3] Tao Dai, Jianrui Cai, Yongbing Zhang, Shu-Tao Xia, and Lei Zhang.  Second-order attention network for single image super-resolution.  In CVPR, pages 11065–11074, 2019.
  * [4] Chao Dong, Chen Change Loy, Kaiming He, and Xiaoou Tang.  Learning a deep convolutional network for image super-resolution.  In ECCV, pages 184–199, 2014.
  * [5] Chao Dong, Chen Change Loy, Kaiming He, and Xiaoou Tang.  Image super-resolution using deep convolutional networks.  IEEE TPAMI, 38(2):295–307, 2015.
  * [6] Mihai Dusmanu, Ignacio Rocco, Tomas Pajdla, Marc Pollefeys, Josef Sivic, Akihiko Torii, and Torsten Sattler.  D2-net: A trainable cnn for joint description and detection of local features.  In CVPR, pages 8092–8101, 2019.
  * [7] Hayit Greenspan.  Super-resolution in medical imaging.  The computer journal, 52(1):43–63, 2009.
  * [8] Seamus J Holden, Stephan Uphoff, and Achillefs N Kapanidis.  Daostorm: An algorithm for high-density super-resolution microscopy.  Nature methods, 8(4):279–280, 2011.
  * [9] Jia-Bin Huang, Abhishek Singh, and Narendra Ahuja.  Single image super-resolution from transformed self-exemplars.  In CVPR, pages 5197–5206, 2015.
  * [10] Yixuan Huang, Xiaoyun Zhang, Yu Fu, Siheng Chen, Ya Zhang, Yan-Feng Wang, and Dazhi He.  Task decoupled framework for reference-based super-resolution.  In CVPR, pages 5931–5940, 2022.
  * [11] Michal Irani and Shmuel Peleg.  Improving resolution by image registration.  Graphical models and image processing, 53(3):231–239, 1991.
  * [12] Yuming Jiang, Kelvin CK Chan, Xintao Wang, Chen Change Loy, and Ziwei Liu.  Robust reference-based super-resolution via c2-matching.  In CVPR, pages 2103–2112, 2021.
  * [13] Justin Johnson, Alexandre Alahi, and Li Fei-Fei.  Perceptual losses for real-time style transfer and super-resolution.  In ECCV, pages 694–711, 2016.
  * [14] Yongwoo Kim, Jae-Seok Choi, and Munchurl Kim.  2x super-resolution hardware using edge-orientation-based linear mapping for real-time 4k uhd 60 fps video applications.  IEEE Transactions on Circuits and Systems II: Express Briefs, 65(9):1274–1278, 2018.
  * [15] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  In ICLR, 2015.
  * [16] Christian Ledig, Lucas Theis, Ferenc Huszár, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, and Wenzhe Shi.  Photo-realistic single image super-resolution using a generative adversarial network.  In CVPR, pages 4681–4690, 2017.
  * [17] Zhengqi Li and Noah Snavely.  Megadepth: Learning single-view depth prediction from internet photos.  In CVPR, pages 2041–2050, 2018.
  * [18] Bee Lim, Sanghyun Son, Heewon Kim, Seungjun Nah, and Kyoung Mu Lee.  Enhanced deep residual networks for single image super-resolution.  In CVPRW, pages 136–144, 2017.
  * [19] Ding Liu, Bihan Wen, Yuchen Fan, Chen Change Loy, and Thomas S Huang.  Non-local recurrent network for image restoration.  In NeurIPS, 2018.
  * [20] Liying Lu, Wenbo Li, Xin Tao, Jiangbo Lu, and Jiaya Jia.  Masa-sr: Matching acceleration and spatial adaptation for reference-based image super-resolution.  In CVPR, pages 6368–6377, 2021.
  * [21] Yusuke Matsui, Kota Ito, Yuji Aramaki, Azuma Fujimoto, Toru Ogawa, Toshihiko Yamasaki, and Kiyoharu Aizawa.  Sketch-based manga retrieval using manga109 dataset.  Multimedia Tools and Applications, 76(20):21811–21838, 2017.
  * [22] Yiqun Mei, Yuchen Fan, and Yuqian Zhou.  Image super-resolution with non-local sparse attention.  In CVPR, pages 3517–3526, 2021.
  * [23] Andrew J Patti, M Ibrahim Sezan, and A Murat Tekalp.  Superresolution video reconstruction with arbitrary sampling lattices and nonzero aperture time.  IEEE TIP, 6(8):1064–1076, 1997.
  * [24] Johannes L Schönberger and Jan-Michael Frahm.  Structure-from-motion revisited.  In CVPR, pages 4104–4113, 2016.
  * [25] Johannes L Schönberger, Enliang Zheng, Jan-Michael Frahm, and Marc Pollefeys.  Pixelwise view selection for unstructured multi-view stereo.  In ECCV, pages 501–518, 2016.
  * [26] Vida Fakour Sevom, Esin Guldogan, and Joni-Kristian Kämäräinen.  360 panorama super-resolution using deep convolutional networks.  In Int. Conf. Computer Vision Theory and Applications, 2018.
  * [27] Gyumin Shim, Jinsun Park, and In So Kweon.  Robust reference-based super-resolution with similarity-aware deformable convolution.  In CVPR, pages 8425–8434, 2020.
  * [28] Libin Sun and James Hays.  Super-resolution from internet-scale scene matching.  In IEEE Int. Conf. Computational Photography, pages 1–12, 2012\.
  * [29] Yang Tan, Haitian Zheng, Yinheng Zhu, Xiaoyun Yuan, Xing Lin, David Brady, and Lu Fang.  Crossnet++: Cross-scale large-parallax warping for reference-based super-resolution.  IEEE TPAMI, 43(12):4291–4305, 2020.
  * [30] Tengfei Wang, Jiaxin Xie, Wenxiu Sun, Qiong Yan, and Qifeng Chen.  Dual-camera super-resolution with aligned attention modules.  In International Conference on Computer Vision (ICCV), 2021.
  * [31] Xintao Wang, Ke Yu, Shixiang Wu, Jinjin Gu, Yihao Liu, Chao Dong, Yu Qiao, and Chen Change Loy.  Esrgan: Enhanced super-resolution generative adversarial networks.  In ECCVW, 2018.
  * [32] Yufei Wang, Zhe Lin, Xiaohui Shen, Radomir Mech, Gavin Miller, and Garrison W Cottrell.  Event-specific image importance.  In CVPR, pages 4810–4819, 2016.
  * [33] Yuzhuo Wei, Li Chen, Rong Xie, Li Song, Xiaoyun Zhang, and Zhiyong Gao.  Fpga based video transcoding system with 2k-4k super-resolution conversion.  In IEEE Visual Communications and Image Processing, pages 1–2, 2019\.
  * [34] Bin Xia, Yapeng Tian, Yucheng Hang, Wenming Yang, Qingmin Liao, and Jie Zhou.  Coarse-to-fine embedded patchmatch and multi-scale dynamic aggregation for reference-based super-resolution.  In AAAI, 2022.
  * [35] Fuzhi Yang, Huan Yang, Jianlong Fu, Hongtao Lu, and Baining Guo.  Learning texture transformer network for image super-resolution.  In CVPR, pages 5791–5800, 2020.
  * [36] Huanjing Yue, Xiaoyan Sun, Jingyu Yang, and Feng Wu.  Landmark image super-resolution by retrieving web images.  IEEE TIP, 22(12):4865–4878, 2013.
  * [37] Kaipeng Zhang, Zhanpeng Zhang, Chia-Wen Cheng, Winston H Hsu, Yu Qiao, Wei Liu, and Tong Zhang.  Super-identity convolutional neural network for face hallucination.  In ECCV, pages 183–198, 2018.
  * [38] Lin Zhang, Xin Li, Dongliang He, Fu Li, Yili Wang, and Zhaoxiang Zhang.  Rrsr: Reciprocal reference-based image super-resolution with progressive feature alignment and selection.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 648–664, 2022.
  * [39] Liangpei Zhang, Hongyan Zhang, Huanfeng Shen, and Pingxiang Li.  A super-resolution reconstruction algorithm for surveillance images.  Signal Processing, 90(3):848–859, 2010.
  * [40] Yulun Zhang, Kunpeng Li, Kai Li, Lichen Wang, Bineng Zhong, and Yun Fu.  Image super-resolution using very deep residual channel attention networks.  In ECCV, pages 286–301, 2018.
  * [41] Zhifei Zhang, Zhaowen Wang, Zhe Lin, and Hairong Qi.  Image super-resolution by neural texture transfer.  In CVPR, pages 7982–7991, 2019.
  * [42] Haitian Zheng, Mengqi Ji, Haoqian Wang, Yebin Liu, and Lu Fang.  Crossnet: An end-to-end reference-based super resolution network using cross-scale warping.  In ECCV, pages 88–104, 2018.
  * [43] Xizhou Zhu, Han Hu, Stephen Lin, and Jifeng Dai.  Deformable convnets v2: More deformable, better results.  In CVPR, pages 9308–9316, 2019.