# Exploring Separable Attention for Multi-Contrast MR Image Super-Resolution

Chun-Mei Feng, Yunlu Yan, Kai Yu, Yong Xu, , Jian Yang, ,
Ling Shao, and Huazhu Fu The work was supported by Shenzhen Science and
Technology Innovation Committee (NO. GJHZ20210705141812038), A*STAR Career
Development Fund (IHPC/CI/G22-003), and AME Programmatic Fund
(A20H4b0141).C.-M. Feng is with the Shenzhen Key Laboratory of Visual Object
Detection and Recognition, Harbin Institute of Technology (Shenzhen), 518055,
China, and also with the Institute of High Performance Computing, A*STAR,
Singapore 138632. (Email: strawberry.feng0304@gmail.com)Y. Yan, and Y. Xu are
with the Shenzhen Key Laboratory of Visual Object Detection and Recognition,
Harbin Institute of Technology (Shenzhen), 518055, China. (Email:
yongxu@ymail.com).J. Yang is with the PCA Laboratory, Key Laboratory of
Intelligent Perception and Systems for High-Dimensional Information of
Ministry of Education, Nanjing University of Science and Technology, Nanjing
210094, China, and also with the Jiangsu Key Laboratory of Image and Video
Understanding for Social Security, School of Computer Science and Engineering,
Nanjing University of Science and Technology, Nanjing 210094, China (e-mail:
csjyang@njust.edu.cn).L. Shao is with Terminus Group, China. (Email:
ling.shao@ieee.org).K. Yu and H. Fu is with the Institute of High Performance
Computing, A*STAR, Singapore 138632. (E-mail: hzfu@ieee.org).Corresponding
author: Yong Xu and Huazhu Fu.

###### Abstract

Super-resolving the Magnetic Resonance (MR) image of a target contrast under
the guidance of the corresponding auxiliary contrast, which provides
additional anatomical information, is a new and effective solution for fast MR
imaging. However, current multi-contrast super-resolution (SR) methods tend to
concatenate different contrasts directly, ignoring their relationships in
different clues, e.g., in the high-intensity and low-intensity regions. In
this study, we propose a separable attention network (comprising high-
intensity priority attention and low-intensity separation attention), named
SANet. Our SANet could explore the areas of high-intensity and low-intensity
regions in the “forward” and “reverse” directions with the help of the
auxiliary contrast, while learning clearer anatomical structure and edge
information for the SR of a target-contrast MR image. SANet provides three
appealing benefits: (1) It is the first model to explore a separable attention
mechanism that uses the auxiliary contrast to predict the high-intensity and
low-intensity regions regions, diverting more attention to refining any
uncertain details between these regions and correcting the fine areas in the
reconstructed results. (2) A multi-stage integration module is proposed to
learn the response of multi-contrast fusion at multiple stages, get the
dependency between the fused representations, and boost their representation
ability. (3) Extensive experiments with various state-of-the-art multi-
contrast SR methods on fastMRI and clinical in vivo datasets demonstrate the
superiority of our model.

###### Index Terms:

MR imaging, multi-contrast, super-resolution, high-intensity and low-intensity
regions.

##  I Introduction

Since Magnetic Resonance (MR) imaging has the advantages of being non-
invasive, non-radiative and carried out in vivo, it has become one of the most
widely used imaging methods for disease diagnosis and treatment planning [1,
2, 3]. However, due to the physical nature of the MR data acquisition process,
the scan time can take up to tens of minutes [4, 5]. Further, the quality of
MR images obtained in clinical practice may be inadequate due to the patient’s
involuntary physiological movement (e.g., heart beating and breathing) during
the acquisition process [6]. This is especially an issue when the scans are
taken following multiple protocols that require a long echo time (TE) or
repetition time (TR). These scans thus often yield poor diagnostic quality,
since medical images must be examined at high resolution (HR) with rich
structural details. If we could reconstruct HR images from low-resolution (LR)
inputs, we would have the potential to achieve better spatial resolution, in
shorter scan times [7, 8].

Figure 1: Comparison between single-contrast method SMORE [9]), multi-contrast
method Lyu et al. [10], and our SANet on in vivo datasets. We effectively
reconstruct the target contrast image by exploring the complementary
information between the clear anatomical structure and boundary details
provided by the auxiliary contrast from the separable attention. The visual
comparison displays that our method can effectively recover the anatomical
information. (Zoom in for best view.)

Super-resolution (SR) technology, which could enhance the image resolution
without any hardware updates [11, 12, 13], has been extensively studied to
overcome the challenge of obtaining HR MR image scans [14]. However, the two
most basic methods of SR interpolation [13, 15, 16], i.e., bicubic and
b-spline interpolation, inevitably lead to blurred edges and blocking
artifacts. To solve this problem, several methods have been proposed including
traditional and deep learning based methods [17, 18, 19, 20, 21, 22, 23, 24,
25, 26]. However, most methods work on mono-contrast images to restore HR
images, resulting in the multi-contrast information being ignored and bad
image quality. (see the single-contrast method SMORE in Fig. 1).

In clinical practice, different MR imaging settings will produce various
multi-contrast images, e.g., T1 and T2 weighted images (T1WIs and T2WIs), as
well as proton density and fat-suppressed proton density weighted images
(PDWIs and FS-PDWIs), which can provide complementary information to each
other [27, 28]. For the same test case, these different-contrast images will
have the same anatomical structure if they are aligned. However, they have
different properties and acquisition times. For example, morphological and
structural information can be provided by T1WIs, while edema and inflammation
can be described by T2WIs. Further, PDWIs provide information on structures
such as articular cartilage, and display high signal-to-noise ratios (SNRs)
for tissues that have little difference in their PDWIs. Meanwhile, FS-PDWIs
can inhibit fat signals and highlight the contrast between tissue structures,
such as cartilage ligaments [29]. During the acquisition process, T2WIs
usually have a longer repetition time (TR) and echo time (TE) than T1WIs,
while PDWIs usually have a shorter TR and TE than FS-PDWI. It should be noted
that in the actual acquisition process, multiple contrasts are often acquired
together to facilitate the comprehensive diagnosis of disease. Thus, an
intuitive idea is to use the contrast with the shorter acquisition time as
additional information to assist the contrast with the slower imaging speed,
the overall acquisition time and difficulty will be greatly reduced [10, 30].
For instance, additional information from T1WIs or PDWIs can be used as
auxiliary contrasts to help the generation of target contrasts, e.g., T2WIs or
FS-PDWIs [31, 32, 33].

Assisted acquisition of difficult contrasts through easily accessible
contrasts has been proven to reduce clinical acquisition time and difficulty
[32, 34, 10, 30]. They aim to restore the target contrast which with lower
resolution through the auxiliary one while with full resolution. For example,
Zheng et al. estimate the statistical information to restore a target SR image
with the help of another contrast image [32, 34]. In work [10], a deep
convolutional neural network (CNN) is utilized to produce the multi-contrast
SR images, where the one contrast applies an HR image to assist another one
[10]. A progressive network is introduced in [30], which is combined with the
HR auxiliary contrast images for the target contrast SR.

However, though significant progress has been achieved in multi-contrast MR
image SR, existing methods still have the following problems: (1) Current
methods typically restore the entire HR image, but the low-intensity area is
often filled with zeros or contains noise that is meaningless for clinical
diagnosis. Nevertheless, exploring the relationship between the high-intensity
and low-intensity regions can improve the performance, because inaccurate
reconstruction results are usually produced on the structural edges between
the two regions. This has never been considered in previous studies (see the
area indicated by the yellow arrow in Fig. 1). Fig. 1 (a) and (b) show the
results of the multi-modal MR image SR methods, but the recovery of anatomical
boundary details of these methods is significantly lower than our method
(Fig. 1 (c)). Therefore, exploring the relationship between the high-
intensity and low-intensity regions can improve the performance. (2) Current
multi-contrast approaches ignore the interaction between different modalities
at various stages. However, during the target-contrast restoration, if the
feature maps of different layers are treated equally, some detailed anatomical
textures will be lost in the reconstructed image. Therefore, we speculate that
MR image SR is still possible to achieve significant improvement by
considering both (1) and (2).

With this insight, we take a further step towards exploring the boundary
details between high-intensity anatomical structure and low-intensity regions
of an MR image separately for SR from the “forward” and “reverse”
perspectives. The terms “forward” and “backward” denote refinement from the
object to the high-intensity region and from the low-intensity region to the
object, respectively. Thereby, the network can concentrate on the uncertain
details between the two regions thanks to the exploration of two opposing
directions. Specifically, we propose a separable attention network to fuse MR
images of different contrasts and generate stronger target features. Our
motivation stems from the fact that during the SR process, the anatomical
contours are typically learned first, while the boundary details of some
structures are more difficult to learn [35]. These detailed areas are usually
important for the diagnosis of lesions [36, 37]. Therefore, the recovery of
anatomical boundary details (see our SANet in Fig. 1) is the main goal of MR
image SR. With this mind, the primary focus of our work is to explore the
high-intensity and low-intensity regions knowledge of the auxiliary contrast
MR image separately, so as to shift more attention towards refining the
uncertain details between the two regions. This has never yet been explored
within our knowledge. In addition, the features of each stage are interacted
and weighted to yield a comprehensive feature, which can guide the learning of
the target features. Overall, our main contributions are three-fold:

  * •

A novel multi-contrast fusion network, named SANet, is designed for multi-
contrast MR image SR. It uses separable attention to assist the target
contrast in recovering clear anatomical information (e.g.,bones, tissues,
blood vessels, and muscles) by exploring the uncertain details between the
high-intensity and low-intensity regions provided by the auxiliary contrast.

  * •

A multi-stage integration module is developed to explore the response of the
multi-contrast fusion at different stages. This allows us to mine the
dependency relationship between the fused features and boost their
representation ability.

  * •

Extensive experiments on fastMRI and two clinical datasets demonstrate the
significant performance improvements of SANet against current state-of-the-art
methods.111The code is released at https://github.com/chunmeifeng/SANet.

Compared with the original conference version [38], in this paper, we provide
several important extensions. (1) We generalize the architecture to a new
version (i.e., SANet). This is achieved by our new separable attention
compensation module, which aims to help the target contrast restore clear
anatomical information by exploring the uncertain details between high-
intensity and low-intensity regions provided by the auxiliary contrast
(§III-B). (2) We also propose a high-intensity priority attention to gradually
refine the anatomical structure in a “forward” manner, and a low-intensity
separation attention to strengthen the edge information of the two regions in
a “reverse” manner. (3) We provide more insightful experiments to evaluate the
mechanism of our separable attention module in Sections §III-B and §IV.

##  II Related Work

###  II-A Deep Learning for MR Image SR

SR techniques are widely used in MR imaging as a form of post-processing to
enhance image quality without changing the acquisition hardware. However,
traditional methods, such as low rank [17], iterative deblurring algorithms
[18, 19], and dictionary learning methods [20, 39], provide very limited
information within a single image. To achieve better restoration, some
techniques use additional training datasets to capture the structural
information between training images [40, 41, 42, 43]. Further, as CNNs delve
deeper into computer vision tasks, many SR techniques employing deep learning
for MR imaging have been developed [44, 45]. For example, Oktay et al.
computed the mapping between LR and HR cardiac MR images using CNNs to restore
heart volumes [21]. Pham et al. also applied SR technology based on deep
learning, using patches of other HR scans to enhance fetal brain MR images
[22]. McDonagh et al. used a residual CNN architecture with context-sensitive
upsampling to generate an improved image with sharp edges and anatomical
details [23]. 3D residual networks have also been used to generate thin-slice
MR images [25].

To restore an enhanced image with sharp details from a single degraded one,
Chen et al. applied a densely connected SR network [24]. More recently,
generative adversarial network (GAN) frameworks [46] used for CT denoising
have been transferred to MR image SR [26]. For instance, a fast and accurate
GAN-based network was proposed in [47] to explore the SR of multi-level dense
connections. Subsequently, more powerful discriminators, such as the pyramid
pooling discriminator, have been proposed to recover details at different
scales for the SR of MR images [48]. However, the above methods focus on a
single contrast, such as a T1WI or T2WI, when restoring HR images, ignoring
the complementary multi-contrast information of the MR image data.

###  II-B Multi-Contrast MR Image Representation

In clinical practice, MR images are usually acquired with multiple contrasts
for comprehensive diagnosis. Each contrast reflects a different type of
tissue, but with the same anatomical structures. Benefiting from this, multi-
contrast methods have been proposed to learn powerful representations for
various MR image tasks, such as segmentation and SR. For example, Huo et al.
used T1WI and T2WI scans together to improve the performance of MR image
segmentation [49]. Since multi-contrast images provide richer information than
single-contrast ones, various MR image segmentation tasks, such as CNN-based
and GAN-based methods, are built on these [50, 51, 52]. Different from the
multi-contrast MR image segmentation task, in the SR task the images are
divided into auxiliary and target contrasts. The auxiliary contrast, which has
a shorter acquisition time, is used to guide the restoration of the target
contrast. For example, Rousseau et al. used the anatomical priors from a
paired auxiliary image to super-resolve the target image [27]. Manjón et al.
restored the LR T2WI with the help of the auxiliary T1WI of the same subject
[53]. Meanwhile, similar anatomical structures from the auxiliary contrast can
be used to obtain an SR image from its corresponding LR image [32, 34, 33,
28]. Recently, CNN-based methods, e.g., UNet and GAN-based methods, have been
used to super-resolved the target contrast image from the help of auxiliary
one [10, 30]. However, previous methods simply use the auxiliary contrast as
the prior data, without exploring the potential correlation between the
different contrasts. In addition, the LR MR image scans in these works are
simulated by downsampling the magnitude images, which violates the imaging
principle of MR images, leaving the effectiveness to be verified. Besides,
current various computer vision works based on attention mechanisms have
obtained outstanding results [54, 55, 56]. For example, they aggregate multi-
modal features in modal and channel-wise ways [54], quantify the contribution
of a specific motion by measuring the consistency between the human motion and
the overall activity [55], and capture spatial coherence among joints using a
skeleton-joint co-attention mechanism [56]. This motivates us to enhance the
aggregation of both modalities in the same way.

In this work, we build our model by simulating an actual LR image to
demonstrate the appealing properties of SANet for the SR of multi-contrast MR
images. Based on this, we propose a separable attention network that learns
the uncertainty between the high-intensity and low-intensity regions of the
auxiliary contrast to improve the restoration performance of the target
contrast.

##  III Methodology

###  III-A Architecture Overview

$\mathbf{x}_{aux}$$\mathbf{y}_{tar}$$\hat{{\mathbf{x}}}_{aux}$$\hat{{\mathbf{x}}}_{tar}$$\mathcal{R}_{aux}^{1}$$\mathcal{R}_{aux}^{2}$$\mathcal{R}_{aux}^{l}$$\mathcal{R}_{tar}^{1}$$\mathcal{R}_{tar}^{2}$$\mathcal{R}_{tar}^{l}$$\mathcal{S}^{1}$$\mathcal{S}^{2}$$\mathcal{S}^{l}$$\mathcal{M}_{\text{Att}}$$\mathcal{M}_{\text{Att}}$$\mathcal{M}_{\text{Int}}$$\mathbf{G}_{aux}$$\mathbf{G}_{tar}$$\mathbf{H}$$\mathbf{F}$$\mathbf{F}_{tar}^{0}$$\mathbf{F}_{tar}^{1}$$\mathbf{F}_{aux}^{1}$$\mathbf{F}_{tar}^{2}$$\mathbf{F}_{aux}^{2}$$\mathbf{F}_{tar}^{L}$$\mathbf{F}_{aux}^{L}$$\mathbf{F}_{aux}^{0}$$\mathbf{F}_{tar}^{0}$$\mathbf{F}_{tar}^{L}$$\mathbf{F}_{aux}^{L}$$\mathcal{S}^{1}$$\mathcal{M}_{\text{Att}}$$\mathcal{M}_{\text{Int}}$

Figure 2: Illustration of our SANet framework. Our model is made up of two
branches: a target contrast branch and an auxiliary contrast branch. We input
the features $\mathbf{x}_{aux}$ and $\mathbf{x}_{tar}$ of two contrasts to the
residual groups $\mathcal{R}_{aux}^{l}$ and $\mathcal{R}_{tar}^{l}$ to learn a
multi-stage feature representation. In each stage, we aggregate the features
of the two contrasts with separable attention $\mathcal{S}^{l}$. Then we use
the channel-spatial attention module $\mathcal{M}_{\text{Att}}$ to enhance
them, and a multi-stage integration module $\mathcal{M}_{\text{Int}}$ to learn
the correlation among different stages.

Let $\mathbf{x}\in\mathbb{R}^{H\times W}$ represent a fully sampled HR MR
image. We can obtain the corresponding LR image
$\mathbf{y}\in\mathbb{R}^{\frac{H}{s}\times\frac{W}{s}}$ ($s$ is the scale
factor) by truncating the outer part of the fully sampled $k$-space according
to the desired scale factor and then applying a 2D fast Fourier transform
(FFT) $\mathcal{F}$ [57]. Thus, the down-scaled LR image can be obtained. This
process mimics the real image acquisition process, making it different from
current SR methods for MR images [57], which simply add Gaussian blurs or
noise to the downsampled amplitude image to obtain the LR one [22]. Therefore,
this approach can accurately evaluate the effectiveness of the SR models. The
SR process for MR images aims to learn a CNNs that can restore an SR image
$\hat{{\mathbf{x}}}$ that is similar to its corresponding HR image
$\mathbf{x}$. Current efforts aim to solve this problem by directly super-
resolve $\hat{\mathbf{x}}$ from $\mathbf{x}$, followed by the existing multi-
contrast works [32, 34, 10, 30], we try to solve it from a multi-contrast
perspective by adding an additional contrast image with the same anatomical
structure, which yields higher quality details.

In Fig. 2, we employ a two-branch structure with an HR auxiliary contrast to
enhance the qualitiy of the LR target contrast. Both the HR auxiliary contrast
$\mathbf{x}_{aux}$ and LR target contrast $\mathbf{y}_{tar}$ are accepted as
the input of our SANet. We send the two contrasts to two different branches to
learn a deep combination. To mine the complementary information of the two
different contrast, we explore the high-intensity and low-intensity cues in
both directions under the guidance of the auxiliary contrast, so that the
target branch can learn the edges and fine structure of the image from the
auxiliary contrast. Further, to explore the dependencies of multi-contrast
features in different stages, we propose a multi-stage integration module that
aims to yield a more powerful holistic feature representation for SR.

####  III-A1 Feature Extraction

We employ the target and auxiliary contrast information in two streams.
Specifically, we use a $3\times 3$ convolutional layer to the auxiliary stream
to yield an initial feature $\mathbf{F}_{aux}^{0}$. Similarly, the target
stream is processed by a convolutional layer and a sub-pixel convolution [58]
$\operatorname{U}_{\uparrow}$ to obtain the enlarged feature
$\mathbf{F}_{tar}^{0}$:

$\displaystyle\mathbf{F}_{aux}^{0}$
$\displaystyle=\operatorname{Conv}\left(\mathbf{x}_{aux}\right),$ (1)
$\displaystyle\mathbf{F}_{tar}^{0}$
$\displaystyle=\operatorname{U}_{\uparrow}\left({\operatorname{Conv}\left(\mathbf{y}_{tar}\right)}\right).$
(2)

We use this pre-upsampling strategy because it provides two advantages [58]:
(1) The CNNs only need to refine the coarse HR image, which reduces the
difficulty of learning. (2) The features in both streams being of identical
size allows the target contrast to inherit the clear structure and edge
information from the bidirectional high-intensity and low-intensity cues in
the auxiliary contrast.

####  III-A2 Multi-Stage Feature Representation

Unlike previous works [32, 34, 10, 30], we explore the cross-contrast fusion
of auxiliary and target features at multiple stages, rather than fusing them
in an early stage or the input. Specifically, the multi-stage feature
representations are learned through a cascade of residual groups $\mathcal{R}$
[59]. In particular, in the $l^{th}$ stage, we obtain the intermediate
features as follows:

$\displaystyle\mathbf{F}_{aux}^{l}$
$\displaystyle=\mathcal{R}_{aux}^{l}(\mathbf{F}_{aux}^{l}),l\in\\{1,\ldots,L\\}$
(3) $\displaystyle\mathbf{F}_{tar}^{l}$
$\displaystyle=\mathcal{R}_{tar}^{l}(\mathcal{S}^{l}(\mathbf{F}_{aux}^{l},\mathbf{F}_{tar}^{l})).$
(4)

Here, $\mathcal{R}_{aux}^{l}$ and $\mathcal{R}_{tar}^{l}$ represent the
residual groups in the $l^{th}$ stage of each respective branch.
$\mathcal{S}^{l}$ represents the separable attention module in the $l^{th}$
stage, which will be introduced in §III-B. Through Eq. (4), our model uses the
clear structural features of the auxiliary contrast to bidirectionally explore
the high-intensity and low-intensity cues. Instead of the current method which
uses only the representation of the final stage for SR [30, 60], we try to
store the features of multiple stages:

$\mathbf{F}=[\mathbf{F}_{aux}^{1},\mathbf{F}_{tar}^{1},\mathbf{F}_{aux}^{2},\mathbf{F}_{tar}^{2},\cdots\mathbf{F}_{aux}^{L},\mathbf{F}_{tar}^{L}].$
(5)

Here, $\mathbf{F}\\!\in\mathbb{R}^{2L\times\\!H\times\\!W\times\\!C}$ is a
multi-stage feature representation obtained by concatenating all the
intermediate features, and $L$ represents the number of residual groups, $C$
is the number of channels.

####  III-A3 Multi-Contrast Feature Enhancement

Eq. (5) aggregates the fusion features of multiple stages and has a more
comprehensive representation. However, each feature is independent, and the
relationship between them has not been mined. Thus, we design a multi-stage
integration module $\mathcal{M}_{\text{Int}}$ (§III-C), which learns the
relationships between the features of each stage and uses them to modulate
each stage:

$\mathbf{H}=\mathcal{M}_{\text{Int}}(\mathbf{F}),$ (6)

where $\mathbf{H}$ denotes the enriched representation integrated from various
stages. To further enhance the feature representation, we design a channel-
spatial attention module $\mathcal{M}_{\text{Att}}$ to modulate the feature
$\mathbf{F}_{tar}^{L}$ as
$\mathbf{G}_{tar}\\!=\\!\mathcal{M}_{\text{Att}}(\mathbf{F}_{tar}^{L})$. Based
on the spatial [61] and channel attention mechanisms [59], the
$\mathcal{M}_{\text{Att}}$ reveals the responses from all dimensions of the
feature maps. As can be seen from Fig. 4 (b), we send the
$\mathbf{F}_{tar}^{L}$ to a 3D convolutional layer. To this end, an attention
map $\mathbf{A}_{tt}$ is generated by capturing joint channels and spatial
features. Formally, we have

$\mathbf{G}_{tar}=\sigma\left(\mathbf{A}_{tt}\right)\operatorname*{\text{\raisebox{1.07639pt}{\scalebox{0.6}{$\bigotimes$}}}}\mathbf{F}_{tar}^{L}+\mathbf{F}_{tar}^{L},$
(7)

where $\sigma$ represents the sigmoid operation, and
$\operatorname*{\text{\raisebox{1.07639pt}{\scalebox{0.6}{$\bigotimes$}}}}$ is
element-wise multiplication. Thus, Eq. (7) mechanism makes
$\mathbf{F}_{tar}^{L}$ enhanced by the context in the feature. We also use a
residual layer to preserve information from the original feature. Similarly,
the enhanced auxiliary contrast features $\mathbf{G}_{aux}$ can also be
calculated using Eq. (7).

$\mathbf{F}_{aux}^{l}$$\mathbf{F}_{tar}^{l}$$\hat{\mathbf{F}}^{l}$$\mathbf{A}_{H}^{l}$$\mathbf{A}_{L}^{l}$$\mathbf{R}^{l}$

Figure 3: Illustration of the proposed separable attention compensation
module. $\mathbf{E}$ is a matrix in which all the elements are 1.
$\mathbf{A}_{H}^{l}$ and $\mathbf{A}_{L}^{l}$ are the high-intensity priority
attention (HP) map and the low-intensity attention (LS) map, respectively.
$\mathbf{F}_{aux}^{l}$ and $\mathbf{F}_{tar}^{l}$ are the intermediate
features of the two contrasts, $\hat{\mathbf{F}}^{l}$ is the fused feature of
the $l^{th}$ stage, and $\mathbf{R}^{l}$ is the output of the separable
attention module. The features obtained from the HP provide more anatomical
structure (zoom in for best view), while the features obtained from the LS
have more edge details (see the blue box). In this way, our method can achieve
significantly improvement in anatomical boundaries of the soft tissue (see the
yellow arrows in Fig. 5 for more details).

$\mathbf{F}\\!\in\\!\mathbb{R}^{2L\\!\times\\!H\times\\!W\\!\times\\!C}$$\hat{\mathbf{F}}\\!\in\\!\mathbb{R}^{2L\\!\times\\!HWC}$$\hat{\mathbf{F}}^{\top}\\!\in\\!\mathbb{R}^{2L\\!\times\\!HWC}$$\mathbf{S}\\!\in\\!\mathbb{R}^{2L\\!\times\\!2L}$$\mathbf{S}\hat{\mathbf{F}}\\!\in\\!\mathbb{R}^{2L\\!\times\\!HWC}$$\mathbf{H}\\!\in\\!\mathbb{R}^{H\\!\times
W\\!\times\\!2LC}$$\mathbf{R}{\mathbf{F}}\\!\in\\!\mathbb{R}^{2L\times\\!H\times\\!W\\!\times\\!C}$$\mathbf{F}_{tar,aux}^{L}$$\mathbf{G}_{tar,aux}^{L}$

Figure 4: (a) Illustration of the proposed multi-stage integration module,
where $\mathbf{F}$ is the multi-stage feature representation, $\mathbf{S}$ is
the affinity matrix, and $\mathbf{H}$ is the comprehensive and holistic output
feature representation. (b) Illustration of the proposed channel-spatial
attention module, where $\mathbf{F}_{tar,aux}^{L}$ is the input feature output
by the residual group, and $\mathbf{G}_{tar,aux}^{L}$ are the output feature
after a 3D convolutional layer.

####  III-A4 Super-Resolved Image

Because our method is a form of pre-upsampling SR, a convolutional layer
$\operatorname{Conv}_{tar}$ is used to convolve $\mathbf{G}_{tar}$ and
$\mathbf{H}$ with residual compensation to obtain the final restored image
$\hat{{\mathbf{x}}}_{tar}$:

$\displaystyle\hat{{\mathbf{x}}}_{tar}$
$\displaystyle=\operatorname{Conv}_{tar}(\mathbf{F}_{tar}^{0}\oplus\mathbf{G}_{tar}\oplus\mathbf{H}),$
(8) $\displaystyle\hat{{\mathbf{x}}}_{aux}$
$\displaystyle=\operatorname{Conv}_{aux}({\mathbf{G}_{aux}}),$

where $\oplus$ is the element-wise summation. Besides, as can be seen from
Fig. 2, we also restore the auxiliary contrast $\hat{{\mathbf{x}}}_{aux}$
using a convolutional layer with input $\mathbf{G}_{aux}$. Note that this
reconstruction procedure of the auxiliary contrast is only used to maintain
consistency with the input.

####  III-A5 Loss Function

Here, the $L_{1}$ loss is employed to evaluate the results of the target and
auxiliary contrast. The overall loss function can be expressed as

$L=\frac{1}{N}\sum_{n=1}^{N}\alpha\left\|\hat{{\mathbf{x}}}_{tar}^{n}-{{\mathbf{x}}}_{tar}^{n}\right\|_{1}+(1-\alpha)\left\|\hat{{\mathbf{x}}}_{aux}^{n}-{{\mathbf{x}}}_{aux}^{n}\right\|_{1},$
(9)

where $\alpha$ weights the trade-off between the reconstruction of the two
contrasts, and $N$ is the number of training samples.

###  III-B Separable Attention Compensation

Since simple fusion of HR auxiliary contrast to assist the SR of target
contrast cannot meet current requirements, we further explore the relationship
between multi-contrast here [32, 34, 10, 30]. Given an HR auxiliary contrast,
in this paper we aim to refine and reconstruct the target contrast using the
clear structural information from the auxiliary contrast. The distributions of
the high-intensity and low-intensity regions in MR images are quite different.
For example, the low-intensity region is usually all zero or contains some
noise data, while the high-intensity region contains all the structural and
diagnostic information. We design a separable attention module to learn the
high-intensity and low-intensity features from two directions at each stage,
and then jointly refine the uncertain details between the two regions. As can
be seen in Fig. 3, we compute the high-intensity priority attention (HP) map
$\mathbf{A}_{H}^{l}$ using a sigmoid operation, while the low-intensity
separation attention (LS) map $\mathbf{A}_{L}^{l}$ is generated by subtracting
the HP attention map from matrix $\mathbf{E}$:

$\displaystyle\mathbf{A}_{H}^{l}=\sigma\left(\mathbf{F}_{aux}^{l}\right),$
(10)
$\displaystyle\mathbf{A}_{L}^{l}=\mathbf{E}-\sigma\left(\mathbf{F}_{aux}^{l}\right),$

where $\mathbf{E}$ is a matrix in which all the elements are 1. Then, we
obtain the fused feature $\hat{\mathbf{F}}^{l}$ as follows:

$\hat{\mathbf{F}}^{l}=\operatorname{Conv_{Re}}[\mathbf{F}_{aux}^{l},\mathbf{F}_{tar}^{l}],$
(11)

where $\operatorname{Conv_{Re}}$ is a 1$\times$1 convolution with 32 output
channels to reduce the computational cost. Then, we use the HP and LS, as well
as the residual component, to weight the separate features in the two streams,
which can be expressed as

$\mathcal{R}^{l}=\mathcal{Q}_{R}\left(\left[\mathcal{Q}\left(\hat{\mathbf{F}}^{l}\operatorname*{\text{\raisebox{1.07639pt}{\scalebox{0.6}{$\bigotimes$}}}}\mathbf{A}_{H}^{l}\right),\mathcal{Q}\left(\hat{\mathbf{F}}^{l}\operatorname*{\text{\raisebox{1.07639pt}{\scalebox{0.6}{$\bigotimes$}}}}\mathbf{A}_{L}^{l}\right)\right]\right)+\mathbf{F}_{tar}^{l},$
(12)

where $[,]$ denotes the concatenation operation, and $\mathcal{Q}$ represents
the 3$\times$3 convolutional layer and a ReLU layer for extracting features.
Finally, we use a prediction layer $\mathcal{Q}_{R}$ with the same operation
as $\mathcal{Q}$ to output a single-channel residual map. It is obvious that
the features obtained from the HP provide more anatomical structure, while the
features obtained from the LS have more edge details (see the blue box in
Fig. 3).

###  III-C Multi-Stage Integration Module

Although our separable attention aggregates the features in the intermediate
stages, it would be interesting if we can explore the correlation among
different stages. If we regard the different contrasts as different
categories, then features of different stages can be considered as as
responses to specific categories. To this end, we propose a multi-stage
integration module $\mathcal{M}_{\text{Int}}$. As can be seen from Fig. 4
(a), the network can assign different weights to the features of different
contrasts and stages according to the dependence between the auxiliary
contrast features of each stage and other stages. As a result, the network not
only focuses on the more informative stages of the target branch, but also
emphasizes the stages that guide the SR of the target contrast.

The $\mathbf{F}\\!\in\mathbb{R}^{2L\times\\!H\times\\!W\times\\!C}$ in Eq. (5)
has concatenates all intermediate representations together. To enable our
multi-stage integration module to understand the correlation between each pair
of multi-contrast features. First, we flatten $\mathbf{F}$ into a matrix
representation $\hat{\mathbf{F}}\in\mathbb{R}^{2L\times HWC}$ for
computational convenience. Then, we compute the correspondence between each
pair of features $\hat{\mathbf{F}}_{i}\in\mathbb{R}^{HWC}$ and
$\hat{\mathbf{F}}_{j}\in\mathbb{R}^{HWC}$ using the following bilinear model:

$\displaystyle\small\mathbf{S}$
$\displaystyle=\texttt{softmax}(\hat{\mathbf{F}}\hat{\mathbf{F}}^{\top})$ (13)
$\displaystyle=\texttt{softmax}([\hat{\mathbf{{F}}}_{1},\cdots,\hat{\mathbf{F}}_{2L}][\hat{\mathbf{F}}_{1},\cdots,\hat{\mathbf{F}}_{2L}]^{\top})\in[0,1]^{2L\times
2L}.$

Here, $\mathbf{S}$ represents the affinity matrix to store the similarity
scores corresponding to each pair of features in $\hat{\mathbf{F}}$, i.e., the
$(i,j)^{th}$ element of $\hat{\mathbf{F}}$ gives the similarity between
$\hat{\mathbf{F}}_{i}$ and $\hat{\mathbf{F}}_{j}$. We use
$\texttt{softmax}(\cdot)$ to normalize each row of the input. Then, attention
summaries are computed as $\mathbf{S}\hat{\mathbf{F}}\in\mathbb{R}^{2L\times
HWC}$ and used the following form to generate an enhanced representation:

$\mathbf{H}=\texttt{reshape}(\mathbf{R}{\mathbf{F}}+\mathbf{F})\in\mathbb{R}^{H\times
W\times 2LC},$ (14)

where $\mathbf{R}{\mathbf{F}}$ is reshaped by $\mathbf{S}\hat{\mathbf{F}}$. In
this way, our model is able to learn a more comprehensive and holistic feature
representation $\mathbf{H}$ by exploring the relations between features of
multi-contrast images at multiple stages. Thus, with the help of multi-
contrast related features at multiple stages, our model can capture a more
holistic and comprehensive feature representation $mathbf{H}$.

##  IV Experiments

###  IV-A Datasets

Since the raw $k$-space data can better verify the validity of the experiment
than the amplitude image, we use three $k$-space raw datasets to evaluate the
effectiveness of our approach.

fastMRI: This dataset is currently the largest open-access MRI dataset [62].
Because there are no multi-modal data description on the official fastMRI
website [63], we select out $227$ pairs samples from the training set and $24$
samples pairs for validation set, including PD and FS-PDWI volumes which have
been aligned. Since this dataset does not release the test set, we just report
the results on the validation set.

SMS: We collected this dataset from 155 patients using a 3T Siemens Magnetom
Skyra system. Each sample contains a T1WI and a T2WI with full $k$-space
sampling, resulting in 2D volumes of size 320$\times$320 with 20 slices for
each patient. The detailed acquisition parameters are as follows: TRT1 =
2001ms, slice thickness = 5 mm, field of view (FOV) = 230$\times$200 mm2, TET1
= 10.72 ms, TET2 = 112.86 ms, TRT2 = 4511 ms.

uMR: We collected this dataset from 50 patients with a 3T whole body scanner
uMR 790 222Provided by United Imaging Healthcare, Shanghai, China.. Each
sample contains a T1WI and a T2WI with full $k$-space sampling, resulting in
2D volumes of size 320$\times$320 with 19 slices for each patient. The
detailed acquisition parameters are as follows: TRT1 = 2001 ms, FOV =
220$\times$250 mm2, slice thickness = 4 mm, matrix = 320$\times$320$\times$19,
TRT1 = 4511 ms, TET1 = 10.72 ms, TET2 = 112.86 ms.

Each dataset is aligned through the simple affine registration
algorithm333https://github.com/Slicer/Slicer-RigidRegistration. before our
experiments. SMS and uMRI are split patient-wise with a ratio of 7:1:2 for the
train/validation/test sets. Finally, we obtain the LR image by truncating the
outer part of the $k$-space [57] according to the enlargement scale factor,
see §III-A for more details.

###  IV-B Baselines and Training Details

We use PSNR and SSIM to evaluate the performance of SANet. The results are
compared with the following previous works, including single-contrast and
multi-contrast SR methods. The hyperparameter optimization for each method is
performed by cross-validation. Single-contrast SR including EDSR [60], a
classical SR method, the learning rate is initialized as $10^{-4}$ and Adam is
adopted as the optimizer; SMORE [9], an anti-aliasing method for MR image SR.
Multi-contrast SR including Zeng et al. [10], a CNN-based method which uses
the T1WI as the prior information to guide the restoration of the target
contrast, where the learning rate is initialized as $10^{-3}$ and Adam is
adopted as the optimizer; Lyu et al. [30], a progressive network combine the
auxiliary contrast in the high-level feature space for the restoration of the
target contrast, where the learning rate is initialized as $10^{-5}$, the
hyperparameter $\lambda_{1}$, $\lambda_{2}$, and $\lambda_{3}$ are set as 1.0,
0.1 and 0.01, respectively, and Adam is adopted as the optimizer; McMRSR [64],
a multi-scale contextual matching and aggregation network for multi-contrast
MR image super-resolution, where the learning rate is initialized as $10^{-4}$
and Adam is adopted as the optimizer; MINet [38], our conference version,
which trained with Adam optimizer and learning rate of $10^{-5}$, the
parameters $\alpha$ and $L$ are empirically set to 0.7 and 6, respectively.
Our model is trained using PyTorch with two NVIDIA Tesla V100 GPUs and 32GB of
memory per card. We use the Adam optimizer with a learning rate of $10^{-5}$
for 50 epochs to train the model. The parameter $\alpha$ is empirically set to
0.7, the number of residual groups $L$ are set to 6.

TABLE I: Quantitative comparison results with standard deviation on three
datasets with 2$\times$ enlargement. $P<$ 0.001 was considered as a
statistically significant level.

| fastMRI | SMS | uMRI
---|---|---|---
Method |  PSNR$\uparrow$ |  SSIM$\uparrow$ |  $P$ values |  PSNR$\uparrow$ |  SSIM$\uparrow$ |  $P$ values |  PSNR$\uparrow$ |  SSIM$\uparrow$ |  $P$ values
EDSR [60] | 26.7$\pm$1.11 | 0.512$\pm$0.06 |  $<0.001$/$<0.001$ | 36.4$\pm$1.04 | 0.962$\pm$0.06 |  $<0.001$/$<0.001$ | 35.4$\pm$1.55 | 0.965$\pm$0.07 |  $<0.001$/$<0.001$
SMORE [9] | 28.3$\pm$1.00 | 0.667$\pm$0.07 |  $<0.001$/$<0.001$ | 38.1$\pm$1.08 | 0.973$\pm$0.05 |  $<0.001$/$<0.001$ | 36.4$\pm$1.30 | 0.971$\pm$0.05 |  $<0.001$/$<0.001$
Zeng et al. [10] | 28.9$\pm$1.05 | 0.670$\pm$0.05 |  $<0.001$/$<0.001$ | 38.2$\pm$1.10 | 0.973$\pm$0.05 |  $<0.001$/$<0.001$ | 36.4$\pm$1.20 | 0.971$\pm$0.03 |  $<0.001$/$<0.001$
Lyu et al. [30] | 29.5$\pm$0.80 | 0.682$\pm$0.04 |  $<0.001$/$<0.001$ | 39.2$\pm$0.94 | 0.978$\pm$0.05 |  $<0.001$/$<0.001$ | 37.1$\pm$1.09 | 0.977$\pm$0.03 |  $<0.001$/$<0.001$
McMRSR [64] | 31.5$\pm$1.33 | 0.699$\pm$0.04 |  $<0.001$/$<0.001$ | 40.0$\pm$1.20 | 0.963$\pm$0.06 |  $<0.001$/$<0.001$ | 37.6$\pm$1.00 | 0.946$\pm$0.03 |  $<0.001$/$<0.001$
MINet [38] | 31.8$\pm$0.83 | 0.709$\pm$0.03 |  $<0.001$/$<0.001$ | 40.5$\pm$0.69 | 0.983$\pm$0.04 |  $<0.001$/$<0.001$ | 38.0$\pm$1.00 | 0.980$\pm$0.03 |  $<0.001$/$<0.001$
SANet | 32.0$\pm$0.71 | 0.716$\pm$0.03 |  $<0.001$/$<0.001$ | 42.1$\pm$0.59 | 0.997$\pm$0.03 |  $<0.001$/$<0.001$ | 39.9$\pm$0.90 | 0.990$\pm$0.03 |  $<0.001$/$<0.001$

TABLE II: Quantitative comparison results with standard deviation on three
datasets with 4$\times$ enlargement. $P<$ 0.001 was considered as a
statistically significant level.

| fastMRI | SMS | uMRI
---|---|---|---
Method |  PSNR$\uparrow$ |  SSIM$\uparrow$ |  $P$ values |  PSNR$\uparrow$ |  SSIM$\uparrow$ |  $P$ values |  PSNR$\uparrow$ |  SSIM$\uparrow$ |  $P$ values
EDSR [60] | 18.4$\pm$1.02 | 0.208$\pm$0.06 |  $<0.001$/$<0.001$ | 31.5$\pm$1.19 | 0.886$\pm$0.07 |  $<0.001$/$<0.001$ | 31.2$\pm$1.29 | 0.907$\pm$0.07 |  $<0.001$/$<0.001$
SMORE [9] | 21.8$\pm$0.95 | 0.476$\pm$0.07 |  $<0.001$/$<0.001$ | 32.1$\pm$1.00 | 0.901$\pm$0.06 |  $<0.001$/$<0.001$ | 32.0$\pm$1.17 | 0.918$\pm$0.05 |  $<0.001$/$<0.001$
Zeng et al. [10] | 23.3$\pm$0.91 | 0.507$\pm$0.06 |  $<0.001$/$<0.001$ | 32.5$\pm$1.03 | 0.912$\pm$0.04 |  $<0.001$/$<0.001$ | 31.9$\pm$1.22 | 0.921$\pm$0.06 |  $<0.001$/$<0.001$
Lyu et al. [30] | 28.2$\pm$0.86 | 0.574$\pm$0.06 |  $<0.001$/$<0.001$ | 33.7$\pm$1.00 | 0.931$\pm$0.04 |  $<0.001$/$<0.001$ | 32.2$\pm$1.02 | 0.929$\pm$0.03 |  $<0.001$/$<0.001$
McMRSR [64] | 29.9$\pm$1.05 | 0.609$\pm$0.05 |  $<0.001$/$<0.001$ | 34.6$\pm$0.89 | 0.945$\pm$0.04 |  $<0.001$/$<0.001$ | 34.0$\pm$1.12 | 0.951$\pm$0.03 |  $<0.001$/$<0.001$
MINet [38] | 29.8$\pm$0.90 | 0.601$\pm$0.04 |  $<0.001$/$<0.001$ | 35.0$\pm$0.92 | 0.948$\pm$0.03 |  $<0.001$/$<0.001$ | 34.2$\pm$1.01 | 0.956$\pm$0.03 |  $<0.001$/$<0.001$
SANet | 30.4$\pm$0.83 | 0.622$\pm$0.04 |  $<0.001$/$<0.001$ | 35.6$\pm$0.70 | 0.957$\pm$0.03 |  $<0.001$/$<0.001$ | 35.3$\pm$0.79 | 0.963$\pm$0.03 |  $<0.001$/$<0.001$

Figure 5: Qualitative comparison of various methods on the three datasets,
with 2$\times$ and 4$\times$ enlargement. SR images and error maps are
presented with their PSNR and SSIM values.

###  IV-C Quantitative Evaluation

Here, we use the average SSIM and PSNR results calculated for the fully
sampled images as the ground-truths. Table I reports the target contrast
restoration evaluations under 2$\times$ enlargement. Since the diverse
machines used for each dataset, there are some discrepancies in their results.
Notably, the best results come from our method both on the three datasets. It
indicate that our method can effectively guide the restoration of the target
contrast. However, since the lack of auxiliary contrast information in the
single-mode methods, the results of EDSR [60] and SMORE [9] are far less
effective than other methods. similarly, Zeng et al. [10] and Lyu et al. [30],
do not employ a deep fusion mechanism, e.g., the dependencies between
different contrast at each stage and the high/low-intensity attention, they
are less effective than our model. Although McMRSR [64] adopts a multi-scale
context matching and aggregation scheme, it still has some shortcomings in
capturing the structure edge details. Notably, MINet is also obviously
superior to any comparison method. Nevertheless, with the help of the
separable attention, our results are further enhanced. For example, we improve
the PSNR from 29.5 dB to 31.8 dB with MINet and to 32.0 dB with SANet, and the
SSIM from 0.682 to 0.709 with MINet and to 0.716 with SANet on the fastMRI
dataset, as compared to the previous best method, i.e., Lyu et al. [30].

Table II reports the target contrast restoration evaluations under 4$\times$
enlargement. As can be seen, compared to the PSNR of SMORE [9] (21.8/32.1/32.0
dB), which dose not use the auxiliary contrast, our MINet and SANet achieve
significantly better results. Compared to the previous state-of-the-art
methods that do use the auxiliary contrast, i.e., Zeng et al. [10] and Lyu et
al. [30], we improve the results from 28.2/33.7/32.2 dB to 30.4/35.6/35.3 dB.
Our multi-contrast method with its multi-stage integration module is thus
clearly effective for target-contrast image restoration, while the separable
attention greatly improves the final results. Because of the powerful multi-
contrast fusion capability of our method, it is superior to previous methods
in both 2$\times$ and 4$\times$ enlargement scenarios. To substantiate our
results, we used a paired Student’s t-test to evaluate the significant
difference between the two methods [65]. $P<$ 0.001 was considered as a
statistically significant level. As can be seen from the $P$ values in Tables
I and Table II, our method has a statistically significant improvement over
the various baselines.

###  IV-D Qualitative Evaluation

The qualitative comparison results on the different enlarge scales of SR
images and their corresponding error maps for the three datasets are shown
Fig. 5. The texture on an error map represents the restoration error; the
smoother the texture, the better the restoration. As shown in this figure, the
error of multi-contrast methods is lower than that of single-contrast models.
With the increase in magnification, the difficulty of image restoration also
increases. It is worth noting that the images recovered by our MINet and SANet
have less of a checkerboard effect and less structural loss. In particular,
SANet significantly reduces the error on the key organizational and structural
margins (see the yellow arrows in Fig. 5), thanks to the separable attention.

Figure 6: Ablation study of the key parts in our method, where $Ab_{1}$,
$Ab_{2}$, $Ab_{3}$, and $Ab_{4}$ are the derived models with different key
components.

###  IV-E Ablation Studies

We discuss our method through various aspects, including an analysis of the
key components, a discussion of the auxiliary modality, a comparison of the
number of blocks, parameter analysis, and the analysis of multi-contrast for
joint super-resolution.

TABLE III: Ablation study regarding the key components of our SANet on the SMS
dataset with 2$\times$ enlargement. $P<$ 0.001 was considered as a
statistically significant level.

Variant |  $\mathcal{F}_{\text{Aux}}$ |  $\mathcal{M}_{\text{Int}}$ |  $\mathcal{M}_{\text{Att}}$ |  $\mathcal{S}^{l}$ |  PSNR |  SSIM |  NMSE |  $P$ valus
---|---|---|---|---|---|---|---|---
$Ab_{1}$ | ✗ | ✗ | ✗ | ✗ | 38.6$\pm$0.015 | 0.970$\pm$0.009 | 0.004$\pm$0.012 |  $<0.001$/$<0.001$/$<0.001$
$Ab_{2}$ | ✗ | ✗ | ✓ | ✗ | 39.2$\pm$0.108 | 0.978$\pm$0.025 | 0.003$\pm$0.002 |  $<0.001$/$<0.001$/$<0.001$
$Ab_{3}$ | ✓ | ✗ | ✓ | ✗ | 39.2$\pm$0.001 | 0.979$\pm$0.037 | 0.002$\pm$0.036 |  $<0.001$/$<0.001$/$<0.001$
$Ab_{4}$ | ✓ | ✓ | ✓ | ✗ | 40.5$\pm$0.014 | 0.983$\pm$0.009 | 0.002$\pm$0.004 |  $<0.001$/$<0.001$/$<0.001$
SANet | ✓ | ✓ | ✓ | ✓ | 41.0$\pm$0.038 | 0.990$\pm$0.010 | 0.002$\pm$0.002 |  $<0.001$/$<0.001$/$<0.001$

####  IV-E1 Analysis of the key components.

In this section, we first evaluate the performance of the key components of
our model. The performance of four important components is evaluated under
2$\times$ enlargement. The components evaluated include
$\mathcal{F}_{\text{Aux}}$, which is the auxiliary contrast in our model,
$\mathcal{M}_{\text{Int}}$, which is our multi-stage integration module,
$\mathcal{M}_{\text{Att}}$, which is our multi-contrast feature enhancement
module, and $\mathcal{S}^{l}$, which is our separable attention compensation
module. Table III summarizes the component analysis under 2$\times$
enlargement on the SMS dataset, where $Ab_{1}$, $Ab_{2}$, $Ab_{3}$, and
$Ab_{4}$ are the four derived ablation models. From this table, we observe
that $Ab_{2}$ improves the performance over the single-contrast version,
$Ab_{1}$, by 0.6 dB, demonstrating the importance of the auxiliary contrast.
Combining the auxiliary contrast, $\mathcal{M}_{\text{Int}}$ and
$\mathcal{M}_{\text{Att}}$ achieve the largest boost in performance, e.g.,
38.6 dB to 40.5 dB, as compared to the baseline. This supports our conclusion
that these two modules can provide deep information to complement the SR of
the target contrast. Our SANet, equipped with all components, produces the
best restoration results, improving the PSNR from 38.6 dB to 41.0 dB. Overall,
all four components help enhance the performance of SANet, providing it with a
powerful capability of extract important information to guide the SR of
target-contrast. It should be noted that although $\mathcal{S}^{l}$ can
further improve the reconstruction results than other fusion scheme,
$\mathcal{S}^{l}$ is also built on the auxiliary contrast, demonstrating that
our separable attention mechanism is an effective multi-contrast assisted
imaging scheme.

####  IV-E2 Analysis of the Separable Attention Compensation Module

Here, we will focus on the analysis of why our separable attention
compensation module is useful for MR image restoration. We have mentioned that
it can explore high-intensity and low-intensity areas in both forward and
reverse directions, making the network pay more attention to the uncertain
boundary details as well as anatomical details. We have shown the high-
intensity priority and low-intensity separation feature maps in Fig. 3. It is
obvious that the two feature maps are focused on the anatomical structure and
edge details, respectively (see the blue box). To better understand the
separable attention compensation module, we show the high-intensity priority
(HP) attention map and the low-intensity separation (LS) attention map of
fastMRI dataset in Fig. 7. The brighter the color, the higher the network’s
attention. As can be seen from this figure, HP focuses on the high-intensity
structure, while LS focuses on the low-intensity region. In other words, such
reverse attention mechanism can joint inferring to produce bidirectional
enhanced visual features. More directly, compared the error map of MINet with
SANet, we observed that SANet can recover the boundary and organizational
details of the image, such as the yellow arrow area, with the help of the
separable attention compensation module.

Figure 7: Separable attention visualization, where the first line represents
high-intensity priority (HP) attention map, and the second line represents
low-intensity separation (LS) attention map. The brighter the color, the
higher the network’s attention.

####  IV-E3 Discussion of the auxiliary modality.

To investigate the influence of weighting the trade-off between the two
modalities, we report the SR results of our method on the SMS dataset with
2$\times$ enlargement in Fig. 8. The weights of both the target and auxiliary
streams are determined by the value of $\alpha$. The bigger the value of
$\alpha$, the smaller the influence of the auxiliary modality in our model.
From this figure, we can see that our model has the best PSNR and SSIM results
at $\alpha$ = 0.7, which demonstrates that the auxiliary contrast plays an
important role in our model. However, when $\alpha$ is smaller than 0.3, the
PSNR and SSIM values quickly degrades. This is likely because too small a
value of $\alpha$, results in insufficient target contrast information.

Figure 8:  Ablation study on the ratio of the two contrasts $\alpha$ in terms
of PSNR and SSIM. The abscissa value is $\alpha$. Figure 9: Ablation study on
using different numbers of residual groups. The abscissa value is the number
of residual groups.

####  IV-E4 Comparison of the number of blocks.

As the number of network parameters grows with the number of residual groups,
we need to select an appropriate number of blocks to balance the accuracy and
computational complexity of the network. In Fig. 9, we evaluate the PSNR and
SSIM scores of different numbers groups on the SMS dataset with 2$\times$
enlargement. As shown in this figure, the SR performance, measured by PSNR and
SSIM, increases monotonically when the number of residual groups increases,
becoming stable when the number of groups is 6. However, although it uses
fewer numbers of groups, our method still generates higher PSNR and SSIM
scores than other baselines (see Table I). This study reveals the
effectiveness of our proposed separable attention and multi-stage integration
modules.

####  IV-E5 Parameter analysis.

We also find that our method requires the lowest number of parameters. For
example, SANet requires 5M parameters when residual groups = 2, and 11M when
residual groups = 6, where the separable attention module requires 0.2M
parameters and the FLOPs of this module are 17.77G. The FLOPs of SANet are
307.02G and the execution time is 2.20s/it. However, EDSR requires 43M
parameters, and both SMORE [9] and Zeng et al. [10] were built on the
framework of EDSR, among which SMORE is composed of two EDSR framework. Lyu et
al. [30] adopted progressive model which requires 7M parameters, but the
performance worse than our method.

####  IV-E6 Analysis of Multi-Contrast for Joint Super-Resolution.

To reveal the effectiveness of our SANet for multi-contrast fusion, we record
the values where the two contrasts need to be super-resolved simultaneously.
The results of SANet are PSNR = 31.7.2 dB and SSIM = 0.700 in the target
contrast, PSNR = 32.3 dB and SSIM = 0.705 in the auxiliary contrast on the
fastMRI data with 2$\times$ enlargement. These results are significantly
higher than the baseline, as shown in Table I. Thus, SANet is a powerful
multi-contrast fusion mechanism, which can be used to restore images of
multiple contrasts simultaneously.

####  IV-E7 Discussion of the Loss Function.

In Table IV, we record the 2$\times$ enlargement results of two commonly used
loss on the SMS dataset, e.g., $L_{1}$ and $L_{2}$. The results in this table
show that there is no significant difference between the restoration results
under the two losses [66]. Since the convergence speed of $L_{1}$ is faster
than $L_{2}$ loss [60], the $L_{1}$ loss is adopted in our model.

TABLE IV: Discussion of the loss function on SMS datasets with 2$\times$
enlargement.

Loss | PSNR | SSIM |  $P$ values
---|---|---|---
$L_{1}$ | 42.10$\pm$0.59 | 0.997$\pm$0.03 |  $<0.001$/$<0.001$
$L_{2}$ | 42.13$\pm$0.62 | 0.998$\pm$0.05 |  $<0.001$/$<0.001$

##  V Conclusion

In this work, we aim to explore the high-intensity and low-intensity
relationships in the auxiliary contrast to assist the super-resolution of the
target contrast. For this purpose, we propose a separable attention mechanism,
named SANet, for multi-contrast MR image super-resolution, which can
effectively restore the target contrast under the guidance of the auxiliary
one. In this method, a high-intensity priority attention and a low-intensity
separation attention are used to jointly reconstruct the image from the
complementary information of the high-intensity and low-intensity regions in
the auxiliary contrast. In this way, more attention is diverted to refining
the uncertain details between the two regions, yielding a higher-quality
target-contrast image. In addition, we design a multi-stage integration module
to explore the responses of the multi-contrast fusion at different stages and
improve their representation ability. We conduct numerous experiments on
various datasets, demonstrating the superiority of our model over the most
advanced methods for multi-contrast MR image super-resolution.

## References

  * [1] S. Wang, Z. Su, L. Ying, X. Peng, S. Zhu, F. Liang, D. Feng, and D. Liang, “Accelerating magnetic resonance imaging via deep learning,” in _2016 IEEE 13th international symposium on biomedical imaging (ISBI)_. IEEE, 2016, pp. 514–517.
  * [2] S. Wang, H. Cheng, L. Ying, T. Xiao, Z. Ke, H. Zheng, and D. Liang, “Deepcomplexmri: Exploiting deep residual network for fast parallel mr imaging with complex convolution,” _Magnetic Resonance Imaging_ , vol. 68, pp. 136–147, 2020.
  * [3] C.-M. Feng, S. Wang, Y. Yan, Y. Xu, L. Shao, and H. Fu, “Specificity-preserving federated learning for mr image reconstruction,” _IEEE Transactions on Medical Imaging_ , 2022.
  * [4] C.-M. Feng, Z. Yang, H. Fu, Y. Xu, J. Yang, and L. Shao, “Donet: Dual-octave network for fast mr image reconstruction,” in _IEEE Transactions on Neural Networks and Learning Systems_ , 2021.
  * [5] C.-M. Feng, Z. Yang, G. Chen, Y. Xu, and L. Shao, “Dual-octave convolution for accelerated parallel mr image reconstruction,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 35, no. 1, 2021, pp. 116–124.
  * [6] C.-M. Feng, Y. Yan, H. Fu, L. Chen, and Y. Xu, “Task transformer network for joint mri reconstruction and super-resolution,” in _International Conference on Medical Image Computing and Computer-Assisted Intervention_. Springer, 2021, pp. 307–317.
  * [7] C.-M. Feng, Y. Yan, G. Chen, Y. Xu, Y. Hu, L. Shao, and H. Fu, “Multi-modal transformer for accelerated mr imaging,” _IEEE Transactions on Medical Imaging_ , 2022.
  * [8] C.-M. Feng, H. Fu, T. Zhou, Y. Xu, L. Shao, and D. Zhang, “Multi-modal aggregation network for fast mr imaging,” _arXiv preprint arXiv:2110.08080_ , 2021.
  * [9] C. Zhao, A. Carass, B. E. Dewey, J. Woo, J. Oh, P. A. Calabresi, D. S. Reich, P. Sati, D. L. Pham, and J. L. Prince, “A deep learning based anti-aliasing self super-resolution algorithm for mri,” in _International Conference on Medical Image Computing and Computer-Assisted Intervention_. Springer, 2018, pp. 100–108.
  * [10] K. Zeng, H. Zheng, C. Cai, Y. Yang, K. Zhang, and Z. Chen, “Simultaneous single-and multi-contrast super-resolution for brain mri images based on a convolutional neural network,” _Computers in biology and medicine_ , vol. 99, pp. 133–141, 2018.
  * [11] K. Zhang, L. V. Gool, and R. Timofte, “Deep unfolding network for image super-resolution,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 3217–3226.
  * [12] K. Zhang, W. Zuo, and L. Zhang, “Learning a single convolutional super-resolution network for multiple degradations,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 3262–3271.
  * [13] K. Zhang, W. Zuo, D. Meng, and L. Zhang, “Deep plug-and-play super-resolution for arbitrary blur kernels,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 1671–1681.
  * [14] M. Jiang, M. Zhi, L. Wei, X. Yang, J. Zhang, Y. Li, P. Wang, J. Huang, and G. Yang, “Fa-gan: Fused attentive generative adversarial networks for mri image super-resolution,” _Computerized Medical Imaging and Graphics_ , vol. 92, p. 101969, 2021.
  * [15] K. Zhang, Y. Li, W. Zuo, L. Zhang, L. Van Gool, and R. Timofte, “Plug-and-play image restoration with deep denoiser prior,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2021.
  * [16] K. Zhang, W. Zuo, and L. Zhang, “Ffdnet: Toward a fast and flexible solution for cnn-based image denoising,” _IEEE Transactions on Image Processing_ , vol. 27, no. 9, pp. 4608–4622, 2018.
  * [17] F. Shi, J. Cheng, L. Wang, P.-T. Yap, and D. Shen, “Lrtv: Mr image super-resolution with low-rank and total variation regularizations,” _IEEE transactions on medical imaging_ , vol. 34, no. 12, pp. 2459–2466, 2015\.
  * [18] R. Hardie, “A fast image super-resolution algorithm using an adaptive wiener filter,” _IEEE Transactions on Image Processing_ , vol. 16, no. 12, pp. 2953–2964, 2007.
  * [19] S. Tourbier, X. Bresson, P. Hagmann, J.-P. Thiran, R. Meuli, and M. B. Cuadra, “An efficient total variation algorithm for super-resolution in fetal brain mri with adaptive regularization,” _NeuroImage_ , vol. 118, pp. 584–597, 2015.
  * [20] K. K. Bhatia, A. N. Price, W. Shi, J. V. Hajnal, and D. Rueckert, “Super-resolution reconstruction of cardiac mri using coupled dictionary learning,” in _2014 IEEE 11th International Symposium on Biomedical Imaging (ISBI)_. IEEE, 2014, pp. 947–950.
  * [21] O. Oktay, W. Bai, M. Lee, R. Guerrero, K. Kamnitsas, J. Caballero, A. de Marvao, S. Cook, D. O’Regan, and D. Rueckert, “Multi-input cardiac image super-resolution using convolutional neural networks,” in _International conference on medical image computing and computer-assisted intervention_. Springer, 2016, pp. 246–254.
  * [22] C.-H. Pham, A. Ducournau, R. Fablet, and F. Rousseau, “Brain mri super-resolution using deep 3d convolutional networks,” in _2017 IEEE 14th International Symposium on Biomedical Imaging (ISBI 2017)_. IEEE, 2017, pp. 197–200.
  * [23] S. McDonagh, B. Hou, A. Alansary, O. Oktay, K. Kamnitsas, M. Rutherford, J. V. Hajnal, and B. Kainz, “Context-sensitive super-resolution for fast fetal magnetic resonance imaging,” in _Molecular Imaging, Reconstruction and Analysis of Moving Body Organs, and Stroke Imaging and Treatment_. Springer, 2017, pp. 116–126.
  * [24] Y. Chen, Y. Xie, Z. Zhou, F. Shi, A. G. Christodoulou, and D. Li, “Brain mri super resolution using 3d deep densely connected neural networks,” in _2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018)_. IEEE, 2018, pp. 739–742.
  * [25] A. S. Chaudhari, Z. Fang, F. Kogan, J. Wood, K. J. Stevens, E. K. Gibbons, J. H. Lee, G. E. Gold, and B. A. Hargreaves, “Super-resolution musculoskeletal mri using deep learning,” _Magnetic resonance in medicine_ , vol. 80, no. 5, pp. 2139–2154, 2018.
  * [26] Q. Lyu, C. You, H. Shan, and G. Wang, “Super-resolution mri through deep learning,” _arXiv preprint arXiv:1810.06776_ , 2018.
  * [27] F. Rousseau, A. D. N. Initiative _et al._ , “A non-local approach for image super-resolution using intermodality priors,” _Medical image analysis_ , vol. 14, no. 4, pp. 594–605, 2010.
  * [28] K. Jafari-Khouzani, “Mri upsampling using feature-based nonlocal means approach,” _IEEE transactions on medical imaging_ , vol. 33, no. 10, pp. 1969–1985, 2014.
  * [29] W. Chen, J. Zhao, Y. Wen, B. Xie, X. Zhou, L. Guo, L. Yang, J. Wang, Y. Dai, and D. Zhou, “Accuracy of 3-t mri using susceptibility-weighted imaging to detect meniscal tears of the knee,” _Knee Surgery, Sports Traumatology, Arthroscopy_ , vol. 23, no. 1, pp. 198–204, 2015.
  * [30] Q. Lyu, H. Shan, C. Steber, C. Helis, C. T. Whitlow, M. Chan, and G. Wang, “Multi-contrast super-resolution mri through a progressive network,” _IEEE Transactions on Medical Imaging_ , 2020.
  * [31] L. Xiang, Y. Chen, W. Chang, Y. Zhan, W. Lin, Q. Wang, and D. Shen, “Deep-learning-based multi-modal fusion for fast mr reconstruction,” _IEEE Transactions on Biomedical Engineering_ , vol. 66, no. 7, pp. 2105–2114, 2018.
  * [32] H. Zheng, X. Qu, Z. Bai, Y. Liu, D. Guo, J. Dong, X. Peng, and Z. Chen, “Multi-contrast brain magnetic resonance image super-resolution using the local weight similarity,” _BMC medical imaging_ , vol. 17, no. 1, p. 6, 2017\.
  * [33] X. Lu, Z. Huang, and Y. Yuan, “Mr image super-resolution via manifold regularized sparse learning,” _Neurocomputing_ , vol. 162, pp. 96–104, 2015\.
  * [34] H. Zheng, K. Zeng, D. Guo, J. Ying, Y. Yang, X. Peng, F. Huang, Z. Chen, and X. Qu, “Multi-contrast brain mri image super-resolution with gradient-guided edge enhancement,” _IEEE Access_ , vol. 6, pp. 57 856–57 867, 2018.
  * [35] E. Van Reeth, I. W. Tham, C. H. Tan, and C. L. Poh, “Super-resolution in magnetic resonance imaging: a review,” _Concepts in Magnetic Resonance Part A_ , vol. 40, no. 6, pp. 306–325, 2012.
  * [36] G. Almeida and J. M. R. Tavares, “Deep learning in radiation oncology treatment planning for prostate cancer: A systematic review,” _Journal of medical systems_ , vol. 44, no. 10, pp. 1–15, 2020.
  * [37] D. Mishra, S. Chaudhury, M. Sarkar, and A. S. Soin, “Ultrasound image segmentation: a deeply supervised network with attention to boundaries,” _IEEE Transactions on Biomedical Engineering_ , vol. 66, no. 6, pp. 1637–1648, 2018.
  * [38] C.-M. Feng, H. Fu, S. Yuan, and Y. Xu, “Multi-contrast mri super-resolution via a multi-stage integration network,” in _International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI)_ , 2021.
  * [39] Z. Yue, H. Yong, D. Meng, Q. Zhao, Y. Leung, and L. Zhang, “Robust multiview subspace learning with nonindependently and nonidentically distributed complex noise,” _IEEE transactions on neural networks and learning systems_ , vol. 31, no. 4, pp. 1070–1083, 2019.
  * [40] A. Rueda, N. Malpica, and E. Romero, “Single-image super-resolution of brain mr images using overcomplete dictionaries,” _Medical image analysis_ , vol. 17, no. 1, pp. 113–132, 2013.
  * [41] Y.-H. Wang, J. Qiao, J.-B. Li, P. Fu, S.-C. Chu, and J. F. Roddick, “Sparse representation-based mri super-resolution reconstruction,” _Measurement_ , vol. 47, pp. 946–953, 2014.
  * [42] D. Zhang, J. He, Y. Zhao, and M. Du, “Mr image super-resolution reconstruction using sparse representation, nonlocal similarity and sparse derivative prior,” _Computers in biology and medicine_ , vol. 58, pp. 130–145, 2015\.
  * [43] Y. Jia, Z. He, A. Gholipour, and S. K. Warfield, “Single anisotropic 3-d mr image upsampling via overcomplete dictionary trained from in-plane high resolution slices,” _IEEE journal of biomedical and health informatics_ , vol. 20, no. 6, pp. 1552–1561, 2015.
  * [44] G. Litjens, T. Kooi, B. E. Bejnordi, A. A. A. Setio, F. Ciompi, M. Ghafoorian, J. A. Van Der Laak, B. Van Ginneken, and C. I. Sánchez, “A survey on deep learning in medical image analysis,” _Medical image analysis_ , vol. 42, pp. 60–88, 2017.
  * [45] Y. Yang, J. Sun, H. Li, and Z. Xu, “Admm-csnet: A deep learning approach for image compressive sensing,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 42, no. 3, pp. 521–538, 2018.
  * [46] H. Zhang, I. Goodfellow, D. Metaxas, and A. Odena, “Self-attention generative adversarial networks,” in _International conference on machine learning_. PMLR, 2019, pp. 7354–7363.
  * [47] Y. Chen, A. G. Christodoulou, Z. Zhou, F. Shi, Y. Xie, and D. Li, “Mri super-resolution with gan and 3d multi-level densenet: smaller, faster, and better,” _arXiv preprint arXiv:2003.01217_ , 2020.
  * [48] J. Wang, Y. Chen, Y. Wu, J. Shi, and J. Gee, “Enhanced generative adversarial network for 3d brain mri super-resolution,” in _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_ , 2020, pp. 3627–3636.
  * [49] Y. Huo, Z. Xu, S. Bao, C. Bermudez, H. Moon, P. Parvathaneni, T. K. Moyo, M. R. Savona, A. Assad, R. G. Abramson _et al._ , “Splenomegaly segmentation on multi-modal mri using deep convolutional networks,” _IEEE transactions on medical imaging_ , vol. 38, no. 5, pp. 1185–1196, 2018.
  * [50] I. Despotovic, E. Vansteenkiste, and W. Philips, “Brain volume segmentation in newborn infants using multi-modal mri with a low inter-slice resolution,” in _2010 Annual International Conference of the IEEE Engineering in Medicine and Biology_. IEEE, 2010, pp. 5038–5041.
  * [51] V. V. Valindria, N. Pawlowski, M. Rajchl, I. Lavdas, E. O. Aboagye, A. G. Rockall, D. Rueckert, and B. Glocker, “Multi-modal learning from unpaired images: Application to multi-organ segmentation in ct and mri,” in _2018 IEEE winter conference on applications of computer vision (WACV)_. IEEE, 2018, pp. 547–556.
  * [52] A. K. Mondal, J. Dolz, and C. Desrosiers, “Few-shot 3d multi-modal medical image segmentation using generative adversarial learning,” _arXiv preprint arXiv:1810.12241_ , 2018.
  * [53] J. V. Manjón, P. Coupé, A. Buades, D. L. Collins, and M. Robles, “Mri superresolution using self-similarity and image priors,” _International journal of biomedical imaging_ , vol. 2010, 2010.
  * [54] X. Shu, J. Yang, R. Yan, and Y. Song, “Expansion-squeeze-excitation fusion network for elderly activity recognition,” _IEEE Transactions on Circuits and Systems for Video Technology_ , 2022.
  * [55] J. Tang, X. Shu, R. Yan, and L. Zhang, “Coherence constrained graph lstm for group activity recognition,” _IEEE transactions on pattern analysis and machine intelligence_ , 2019.
  * [56] X. Shu, L. Zhang, G.-J. Qi, W. Liu, and J. Tang, “Spatiotemporal co-attention recurrent neural networks for human-skeleton motion prediction,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 44, no. 6, pp. 3300–3315, 2021.
  * [57] Y. Chen, Y. Xie, Z. Zhou, F. Shi, A. G. Christodoulou, and D. Li, “Brain mri super resolution using 3d deep densely connected neural networks,” in _2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018)_. IEEE, 2018, pp. 739–742.
  * [58] Z. Wang, J. Chen, and S. C. Hoi, “Deep learning for image super-resolution: A survey,” _IEEE transactions on pattern analysis and machine intelligence_ , 2020.
  * [59] Y. Zhang, K. Li, K. Li, L. Wang, B. Zhong, and Y. Fu, “Image super-resolution using very deep residual channel attention networks,” in _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2018, pp. 286–301.
  * [60] B. Lim, S. Son, H. Kim, S. Nah, and K. Mu Lee, “Enhanced deep residual networks for single image super-resolution,” in _Proceedings of the IEEE conference on computer vision and pattern recognition workshops_ , 2017, pp. 136–144.
  * [61] S. Woo, J. Park, J.-Y. Lee, and I. So Kweon, “Cbam: Convolutional block attention module,” in _Eur. Conf. Comput. Vis._ , 2018, pp. 3–19.
  * [62] J. Zbontar, F. Knoll, A. Sriram, M. J. Muckley, M. Bruno, A. Defazio, M. Parente, K. J. Geras, J. Katsnelson, H. Chandarana _et al._ , “fastmri: An open dataset and benchmarks for accelerated mri,” _arXiv preprint arXiv:1811.08839_ , 2018.
  * [63] K. Xuan, S. Sun, Z. Xue, Q. Wang, and S. Liao, “Learning mri k-space subsampling pattern using progressive weight pruning,” in _International Conference on Medical Image Computing and Computer-Assisted Intervention_. Springer, 2020, pp. 178–187.
  * [64] G. Li, J. Lv, Y. Tian, Q. Dou, C. Wang, C. Xu, and J. Qin, “Transformer-empowered multi-scale contextual matching and aggregation for multi-contrast mri super-resolution,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 20 636–20 645.
  * [65] C. Ye, X. Li, and J. Chen, “A deep network for tissue microstructure estimation using modified lstm units,” _Medical image analysis_ , vol. 55, pp. 49–64, 2019.
  * [66] C. Ledig, L. Theis, F. Huszár, J. Caballero, A. Cunningham, A. Acosta, A. Aitken, A. Tejani, J. Totz, Z. Wang _et al._ , “Photo-realistic single image super-resolution using a generative adversarial network,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 4681–4690.

|  Chun-Mei Feng received her M.S. degree at QuFu Normal University, China, in
2018. She is currently a Ph.D. student of the school of Information Science
and Technology, Harbin Institute of Technology Shenzhen, China. Her research
interests include medical imaging and bioinformatics.
---|---
|  Yunlu Yan received his Bachelor’s degree at NanChang University, China, in
2020. He is currently a M.S. student of the school of Information Science and
Technology, Harbin Institute of Technology Shenzhen, China. His research
interests include medical imaging and federated learning.
---|---
|  Kai Yu is a Scientist at the Institute of High Performance Computing,
A*STAR, Singapore. He received his Ph.D. from Soochow University in 2020 and
was an assistant researcher at the Children’s Hospital of Zhejiang University
School of Medicine for two years. His research interests include computer
vision, medical image analysis, and heart sound signal analysis.
---|---
|  Yong Xu (Senior Member, IEEE) received his B.S. and M.S. degrees at Air
Force Institute of Meteorology (China) in 1994 and1997, respectively. He then
received his Ph.D. degree in pattern recognition and intelligence system at
the Nanjing University of Science and Technology in 2005. From May 2005 to
April 2007, he worked at Harbin Institute of Technology Shenzhen as a post-
doctoral research fellow. Now he is a professor at HIT Shenzhen. His current
interests include pattern recognition, machine learning, and bioinformatics.
---|---
|  Jian Yang (Member, IEEE) received the PhD degree from Nanjing University of
Science and Technology (NUST), on the subject of pattern recognition and
intelligence systems in 2002. In 2003, he was a Postdoctoral researcher at the
University of Zaragoza. From 2004 to 2006, he was a Postdoctoral Fellow at
Biometrics Centre of Hong Kong Polytechnic University. From 2006 to 2007, he
was a Postdoctoral Fellow at Department of Computer Science of New Jersey
Institute of Technology. Now, he is a Chang-Jiang professor in the School of
Computer Science and Technology of NUST. He is the author of more than 200
scientific papers in pattern recognition and computer vision. His papers have
been cited more than 6000 times in the Web of Science, and 18000 times in the
Scholar Google. His research interests include pattern recognition, computer
vision and machine learning. Currently, he is/was an associate editor of
Pattern Recognition, Pattern Recognition Letters, IEEE Trans. Neural Networks
and Learning Systems, and Neurocomputing. He is a Fellow of IAPR.
---|---
|  Ling Shao (Fellow, IEEE) is the Chief Scientist and President of Terminus
International, China. He is an Associate Editor of the IEEE TRANSACTIONS ON
IMAGE PROCESSING, the IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING
SYSTEMS, the IEEE TRANSACTIONS ON CIRCUITS ANDSYSTEM FOR VIDEO TECHNOLOGY, and
several other journals. His research interests include Computer Vision,
Machine Learning and Medical Imaging. He is a Fellow of the IEEE, the IAPR,
the IET and the BCS.
---|---
|  Huazhu Fu (SM’18) is a Senior Scientist at the Institute of High
Performance Computing, A*STAR, Singapore. He received his Ph.D. from Tianjin
University in 2013, and was a Research Fellow at Nanyang Technological
University for two years. From 2015 to 2018, he was a Research Scientist in
Institute for Infocomm Research at Agency for Science, Technology and
Research. His research interests include computer vision, machine learning,
and medical image analysis. He currently serves as an Associate Editor of IEEE
Transactions on Medical Imaging, IEEE Journal of Biomedical and Health
Informatics, and IEEE Access. He also serves as a co-chair of OMIA workshop
and co-organizer of ocular image series challenge (i-Challenge).
---|---