# Seeking Common Ground While Reserving Differences: Multiple Anatomy
Collaborative Framework for Undersampled MRI Reconstruction

Jiangpeng Yan, Chenghui Yu, Hanbo Chen, Zhe Xu, Junzhou Huang, , Xiu Li, , and
Jianhua Yao Manuscript received XXX; revised XXX; accepted XXX. Date of
publication XXX; date of current version XXX. This research was partly
supported by the XXX.J. Yan, H. Yu and H. Chen have equal contributions to the
work. X. Li and J. Yao are corresponding authors.J. Yan, H. Yu and X. Li are
with Tsinghua Shenzhen International Graduate School, Tsinghua University,
Shenzhen 518055, China. (E-mail<EMAIL_ADDRESS>li.xiu@sz.tsinghua.edu.cn).H. Chen and J. Yao are with AI healthcare in
Tencent AI Lab, Shenzhen 518055, China. (E-mail<EMAIL_ADDRESS>jianhuayao@tencent.com).Z. Xu is with Department of Biomedical Engineering,
The Chinese University of Hong Kong, Shatin, NT, Hong Kong, China. (E-mail:
jackxz@link.cuhk.edu.hk).J. Huang is with the Department of Computer Science
and Engineering, University of Texas at Arlington. (E-mail: jzhuang@uta.edu).

###### Abstract

Recently, deep neural networks have greatly advanced undersampled Magnetic
Resonance Image (MRI) reconstruction, wherein most studies follow the one-
anatomy-one-network fashion, i.e., each expert network is trained and
evaluated for a specific anatomy. Apart from inefficiency in training multiple
independent models, such convention ignores the shared de-aliasing knowledge
across various anatomies which can benefit each other. To explore the shared
knowledge, one naive way is to combine all the data from various anatomies to
train an all-round network. Unfortunately, despite the existence of the shared
de-aliasing knowledge, we reveal that the exclusive knowledge across different
anatomies can deteriorate specific reconstruction targets, yielding overall
performance degradation. Observing this, in this study, we present a novel
deep MRI reconstruction framework with both anatomy-shared and anatomy-
specific parameterized learners, aiming to “seek common ground while reserving
differences” across different anatomies. Particularly, the primary anatomy-
shared learners are exposed to different anatomies to model flourishing shared
knowledge, while the efficient anatomy-specific learners are trained with
their target anatomy for exclusive knowledge. Four different implementations
of anatomy-specific learners are presented and explored on the top of our
framework in two MRI reconstruction networks. Comprehensive experiments on
brain, knee and cardiac MRI datasets demonstrate that three of these learners
are able to enhance reconstruction performance via multiple anatomy
collaborative learning.

###### Index Terms:

MRI Reconstruction, Neural Network, Collaborative Learning

##  I Introduction

Figure 1: Comparison of (a) One-Anatomy-One-Network, (b) Multiple-Anatomy-One-
Network, and (c) Our Proposed Multiple-Anatomy-Parameterized-Network paradigm.

Magnetic Resonance Images (MRIs), which can provide detailed anatomical
information without invasion, are widely used for various image-guided
therapies. As a typical compressed-sensing [1] based inverse problem,
reconstructing high-quality MRIs from undersampled k-space signals plays an
important role in accelerating the imaging process [2, 3]. Recently, deep
convolutional neural networks (CNNs) have proven soaring successes in
efficiently reconstructing high-quality de-aliased anatomy MRIs [4, 5] with
their stronger ability of automatically learning high-level anatomical
representation, wherein most efforts [6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16]
follow the one-anatomy-one-network (OAON) fashion, i.e., each expert network
is trained and evaluated for specific anatomy, as shown in Fig. 1 (a).
However, training multiple independent networks for different anatomies
greatly increases the deployment difficulty since tens of anatomies are
examined with MRIs in clinical practice. Moreover, apart from inefficiency in
deployment, such convention ignores the shared de-aliasing knowledge across
various anatomies, which can benefit each other for reconstruction.

Therefore, it is natural to ask: whether a deep network can simultaneously
learn from mixed anatomy data for all-round MRI reconstruction? Intuitively,
one naive solution is to adopt a multiple-anatomy-one-network (MAON) fashion
that directly introduces all the data from various anatomies to train an all-
round network, as shown in Fig. 1 (b). Unfortunately, despite the existence of
the shared de-aliasing knowledge and better generalization ability, we reveal
that the exclusive knowledge across different anatomies (e.g., unique
intensity distributions and structure/location priors) can mislead the network
and deteriorate specific reconstruction targets, yielding overall decreased
reconstruction quality.

Most relevantly, Liu et al. [17] introduced a universal undersampled MRI
reconstruction network trained by distilling knowledge from pre-trained expert
models into one universal network and the anatomy-specific instance
normalization was proposed to adapt the various intensity distributions for
multiple anatomies. Given the knowledge distilling as a complicated process
where pre-trained expert models are still required, here we lean to
disentangle knowledge via different parameterized learners. In terms of the
anatomy-specific normalization, we make a step further than [17] to use the
parameterized learners for capturing anatomy-specific information with a novel
unified framework, where anatomy-specific normalization can be regarded as one
of the possible implementations.

In this study, we propose to seek common ground while reserving differences
across different anatomies by redesigning a novel deep MRI reconstruction
framework constructed with anatomy-shared and anatomy-specific parameterized
learners, named as multiple-anatomy-parameterized-network (MAPN) framework
shown in Fig. 1 (c). Particularly, such design echos the multi-domain visual
classification learning [18, 19] and allows the MRI reconstruction of each
anatomy to enjoy both (i) the flourishing shared de-aliasing knowledge by
training the primary anatomy-shared learners with mixed anatomies, and (ii)
the exclusive knowledge reserved for it in the tailored anatomy-specific
learners. In fact, the OAON and MAON fashion are the special cases of our MAPN
where there exist either anatomy-specific or anatomy-shared learners. Under
our proposed unified framework, we explore four types of anatomy-specific
learners on top of our framework, including anatomy-specific normalization
[17], channel attention, series learners and paralleled learners, and
integrated them into two different MRI reconstruction networks. Comprehensive
experiments on brain, knee and cardiac MRI datasets demonstrate the
effectiveness of our new “all-round” network with various insightful findings.
The contributions of this work can be summarized as follows:

  * •

We unify current one-anatomy-one-network and multiple-anatomy-one-network
solutions under a novel multiple-anatomy-parameterized-network framework where
we can both seek common ground and reserve differences across anatomies.

  * •

We implement four possible designs of the compact anatomy-specific
parameterized learners as the early attempts under the proposed framework and
three of them show the ability to enhance multiple anatomy collaborative
learning with two different reconstruction networks, providing insightful
findings on how to better capture anatomy-specific information with our
framework.

  * •

Extensive experiments are performed on brain, knee and cardiac MRI datasets
demonstrating the effectiveness of our strategy for improving different
anatomy reconstruction quality with different reconstruction networks.

##  II Related Work

###  II-A Deep Learning Based Undersampled MRI Reconstruction

Since Wang et al.[6] firstly introduced deep networks for undersampled MRI
reconstruction, researchers have developed various approaches which can be
roughly categorized into the data-driven [6, 20, 21, 22, 11], the model-driven
[9, 10, 23, 7, 8, 16, 24]. Among these works, DCCNN [7] has gained much
attention with a series of variants [12, 13, 14, 15, 16] where researchers
explored how to design neural networks for better representation learning.
However, these methods all followed the OAON fashion as mentioned above, and
there only existed a few previous works discussing the anatomy generalization
ability in MRI reconstruction. Yan et al. [14] used neural architecture search
technology to search different network architectures for knee and brain,
proving that different anatomies prefer different architectures. Transfer
learning in MRI reconstruction is discussed in [25, 26, 27], where researchers
pre-trained a network on a large pool of data and then transferred the learned
weights for one anatomy reconstruction by fine-tuning, while our aim is to
learn and reconstruct multiple anatomies simultaneously in one network. Guo et
al. [28] introduced the distribution shifting problem of MRIs from different
institutions and proposed a federated learning-based solution but only for
brain MRI reconstruction tasks. Few attempts were made to explore how
different anatomies can be reconstructed by one network.

As mentioned above, Liu et al. [17] made the first step to introduce a
universal undersampled MRI reconstruction network with knowledge distilling
and the anatomy-specific instance normalization strategy. In our framework, we
lean to disentangle knowledge via different parameterized learners without the
need of difficult knowledge distilling process. Since the idea of anatomy-
specific instance normalization can be regarded as a setting of the anatomy-
specific learners, we re-implemented [17] as the anatomy-specific
normalization settings without knowledge distilling to make a fair comparison
with other settings in our experiments.

###  II-B Multi-domain Visual Classification Learning

Our method is closely related to recent works in multi-domain visual
classification learning [29, 18, 19, 30], where researchers aim to train a
single network to perform image classification tasks across different domains
(animals, flowers, digits, …). Bilen and Vedaldi [29] used parameters in batch
and instance normalization layers to model different domains for task-specific
classification. Rebuffi et al. extended [29] in [18, 19] to parameterize the
standard residual network architecture into a network family to capture
different domains. Mallya et al. [30] proposed to adapt a single network to
different classification tasks by masking weights of different convolution
kernels.

Distinct from the above works, our aim is to make MRI reconstruction model
jointly and better learn from multiple anatomies data and we make the early
attempt to introduce the idea in [18, 19] for building an all-round MRI
reconstruction model with a parameterized network where the compact anatomy-
specific learners focus on exclusive anatomy prior while the major anatomy-
shared learners enjoy common de-aliasing knowledge. Our experiments
demonstrate that such setting can help different anatomies collaborate with
each other in the simultaneous learning process.

##  III Methodology

Figure 2: Illustration of (a) Anatomy parameterization reconstruction
framework for multiple anatomy collaborative learning. Our framework can be
generalized to both data-driven and model-driven models. (b) Details of four
anatomy parameterization settings. (c) Notes on shading boxes. (d) Anatomy-
specific parameterization example with paralleled learners setting (d-i) and
its anatomy-shared variant (d-ii) for the ablation study.

Without loss of generalization ability, we study multiple anatomy undersampled
MRI reconstruction problems under single-coil reconstruction scenario
following [12, 13, 17]. We firstly introduce the problem formulation and two
baseline solutions: the data-driven U-Net [31] based reconstruction model [20,
22, 11] and the model-driven deep cascaded network (DCCNN) based
reconstruction model [7, 12, 13, 14, 17]. Then, we illustrate the details of
the proposed MAPN solution with four anatomy-specific learner designs. The
overall framework is shown in Fig. 2.

###  III-A Problem Formulation and Baseline Solutions

The aim of MRI reconstruction is to recover the fully-sampled MR image $x$
from the under-sampled k-space signals $s$, which can be formulated as:

$s=F_{u}x,$ (1)

where $F_{u}$ is the under-sampling Fourier encoding matrix and $x$ can be
obtained by solving the following inverse problem:

$\min_{x}\lVert s-F_{u}x\rVert_{2}^{2}+\mathcal{R}(x),$ (2)

where $\mathcal{R}(x)$ is the regularization term. By integrating CNN-derived
constraints into Eq. 2, this problem can be formulated as:

$\min_{x}\lVert s-F_{u}x\rVert_{2}^{2}+\lambda\lVert
x-\mathcal{N}\left(x,\omega)\right\rVert^{2},$ (3)

where $\mathcal{N}$ represents the deep CNN network with learn-able weights
$\omega$.

For data-driven solutions, researchers usually solve Eq. 3 by directly
learning a mapping between between paired under-sampled MRI $x_{u}$ and fully-
sampled MRI $x$. Typically, U-Net [31] is adapted in [20, 22, 11] as
$\mathcal{N}$ and we have:

$y=\mathcal{N}(x_{u},\omega).$ (4)

Then, $\mathcal{N}$ is trained to reconstruction $x$ from $x_{u}$ by
minimizing the objective function $\mathcal{L}$ as:

$\mathop{{\min}}\limits_{{\omega}}\mathcal{L}(\mathcal{N}(x_{u},\omega)-x),$
(5)

where $\mathcal{L}$ is the pixel-wise loss, e.g., L1 or L2 loss.

For model-driven solutions, researchers tend to regard $\lVert
s-F_{u}x\rVert_{2}^{2}$ as the data consistence (DC) term between the image
and k-space data, then unroll Eq. 3 with the alternating minimization steps
[7, 12, 13, 14, 17]. Schlemper et al. [7] firstly proposed to use the data
consistency (DC) modules and re-formulated Eq. 3 by cascading $n$ sub-networks
as the following two steps:

$y^{(n)}=\mathcal{N}^{(n)}(x^{(n)},\omega^{(n)}),$ (6)
$x^{(n+1)}=\arg\min_{x}\lambda\lVert s_{u}-F_{u}x\rVert_{2}^{2}+\lVert
x-y^{(n)}\rVert^{2},$ (7)

where $x^{(0)}=x_{u}$. The problem of Eq. 7 is given with a closed-form
solution with the DC layer, which can be regarded as a plug-in module in the
deep network to align current reconstructed k-space of $y^{(n)}$ with the
ground truth sampled signals in $s$, given $\lambda\rightarrow\infty$. The
problem of Eq. 6 is similar to Eq. 4. A super network $\mathcal{N}$ is built
with $n$-fold cascaded sub-CNNs $\mathcal{N}^{(n)}$ and DC modules, and
trained by minimizing the objective function $\mathcal{L}$ to solve Eq. 6.

Figure 3: Illustration of Anatomy Parameterization based U-Net [20] (a) and
DCCNN [7] (b) for multiple anatomy collaborative learning.

For better understanding, we visualize U-Net based reconstruction network in
Fig.3 (a), where we follow the setting in [20] with 4 down/up-sampling
operations, and DCCNN based reconstruction network in Fig.3 (b) with our
proposed anatomy parameterization strategy. At a higher level, we can regard
the U-Net based reconstruction solution as a sub-problem of the DCCNN based
one, and U-Net [31] can be directly used as the sub-CNNs $\mathcal{N}^{(n)}$
in DCCNN. Here, we independently treat these two solutions to demonstrate the
generalization ability of our anatomy parameterization framework. What’s more,
in original DCCNN [7], the researchers only use plain convolutional layers to
build their network. Comparing to [7], the U-Net includes 4 down-sampling
blocks with convolution layers and 4 up-sampling blocks with de-convolution
layers. For simplicity and to keep consistent with DCCNN, we replace all the
convolution layers in U-Net with anatomy-shared/specific learners and leave
the de-convolution layers as anatomy-shared learners.

Notably, most existed works follow the OAON setting, where one independent
$\mathcal{N}$ is trained from scratch and only tested for one target
reconstruction anatomy. In this work, we benchmark above baseline solutions
with multiple anatomy reconstruction task.

###  III-B Multiple Anatomy Parameterized Framework

Revisiting Eq. 4 and Eq. 6 for multiple-anatomy reconstruction scenario, it is
a natural idea to disentangle $\omega$ with the whole network $\mathcal{N}$
for the shared and exclusive knowledge via rewriting Eq. 4 and Eq. 6 as:

$y=\mathcal{N}(x_{u},\omega_{s},\omega_{e^{i}}),$ (8)

where $\omega_{s}$ is trained with all anatomies to learn shared de-aliasing
knowledge and $\omega_{e^{i}}$ is from a parametric set
$\Omega_{e}=\\{\omega_{e^{1}},\omega_{e^{2}},...,\omega_{e^{n}}\\}$ in which
each element is corresponding to a specific anatomy. Examples of detailed
implementations of $\omega_{s}$ and $\Omega_{e}$ are visualized in Fig. 2 (d)
for better understanding, termed as anatomy-shared/specific parameterization.
With Eq. 8 as the basis of our MAPN framework, we then need to answer: how to
properly design $\Omega_{e}$?

The design of $\Omega_{e}$ needs to consider the following two criteria: (1)
$\omega_{e^{i}}$ should be more efficient compared to $\omega_{s}$, otherwise
it brings in more learning parameters; (2) the $\omega_{e^{i}}$ should be easy
to plug into and out from the network with $\omega_{s}$ such that they can be
easily switched for different organs and enjoy the shared knowledge.

With the above two criteria, when we further investigate the convolution
blocks shown in Fig. 2 (b) PN0, we can find that an intuitive setting is to
use the $3\times 3$ convolutional layers as $\omega_{s}$, while the common
Batch Normalization (BN) [32] layers can be regarded as an anatomy-specific
setting of $\omega_{e^{i}}$ shown in Fig. 2 (b) PN1 noted by Anatomy-specific
Normalization strategy. It is obvious that MRIs of different anatomies have
significant feature distribution shifting in terms of means and variances.
Given the input feature map as $m$, the normalization layers utilized to
calibrate the feature map can be formulated as:

$n=\gamma\cdot\frac{m-\mu(m)}{\sigma(m)}+\beta,$ (9)

where scale weight $\gamma$, shift bias $\beta$ in BN layers. To make BN
layers better adapt to one specific anatomy with its unique intensity
distribution, one independent set of normalization layers is assigned with
unique learning parameters $[\gamma,\beta]$ by an anatomy switch operation in
the network for each anatomy, as indicting in Fig. 2 (d-i). It is reminded
that the anatomy-specific instance normalization used in [17] is similar to
PN1 setting with $[\mu(m),\sigma(m)]$ calculated differently and a complicated
knowledge distillation step. With a fair comparison protocol comparing to
following other settings, we think PN1 is our enhanced re-implemented version
of [17], given codes/models of [17] are not open source.

With Anatomy-specific Normalization as a good starting point, another
direction to design $\Omega_{e}$ is to plug some additive compact tailored
anatomy-specific structure into the convolution block for exclusive knowledge
learning. We propose to use the channel attention mechanism, which was
introduced in Squeeze-and-Excited (SE) Network [33], to model anatomy
knowledge by re-calibrating channel-wise feature responses with fully-
connected layers shown in Fig. 2 (b) PN2 noted by Anatomy-specific Channel
Attention strategy.

So far, the previous two solutions do not change the major graphic structure
of the network. In order to leverage features from different levels and
inspired by successes in multi-domain natural image classification [18, 19],
we use $1\times 1$ convolutional layers as residual connections to adjust
features from different levels and then obtain the setting in Fig. 2 (b) PN3
and PN4, namely Anatomy-specific Series and Parallel Learners,
correspondingly, according to where the learners locate. The $1\times 1$
convolutional operations are chosen due to parameter efficiency.

The above four settings are explored as possible implementations under our
proposed framework to prove its effectiveness and how to design the optimal
anatomy-specific learners remains an open problem. We explore these
implementations to shed the light in a new direction for multiple anatomy
collaborative reconstruction.

##  IV Experiments

###  IV-A Implementation Details

####  IV-A1 Environment

All the experiments were carried out on an NVIDIA GeForce RTX 3090 GPU with
the Pytorch.

####  IV-A2 Dataset Preprocessing and Evaluation Metrics

We conducted our experiments on three public MRI datasets of different
anatomies with Institutional Review Board approval: Knee: we randomly sampled
800 knee MR slices from 25 subjects in the fastMRI dataset [20] provided by
New York University School of Medicine. Brain: we randomly sampled 800 brain
MR slices from 50 subjects in the fastMRI dataset [20]. Cardiac: we randomly
sampled 800 cardiac MR slices from 90 cardiac subjects in the ACDC dataset
[34] obtained from the University Hospital of Dijon. Noteworthily, we removed
some slices where no biological structure exists to make three anatomies have
equal slices for fairness. It is reminded that the above three datasets are
acquired with different protocols, and we follow the setting in [17] to treat
them as the same single-coil slice-level MR image for reconstruction.
Following [12, 13, 17], we adopted these data pre-processing steps: the MRIs
were cropped into $320\times 320$, then normalized in magnitude, and clipped
to [-6, 6] with zero phases to simulate 2-channel complex-valued images. All
the models were trained with under-sampled 1D Cartesian masks and evaluated
with $4\times$ and $6\times$ under-sampled reconstruction tasks. We report
PSNR and SSIM [35] between reconstructed and target fully-sampling MRIs as the
quantitative metrics. The final results are calculated on a 5-fold cross-
validation setting by dividing the above three datasets according to subjects
with a ratio of 4:1 for the training and validation.

####  IV-A3 Network Setting

To implement DCCNN reconstruction network, we used 5 stacked convolutional
blocks to build each sub-CNN and cascaded the sub-CNNs and DC modules for 5
times to build the baseline DCCNN. Each block includes cascading Conv.-BN-
LeakyReLU operations, as shown in Fig. 2 (b) PN0, with 64 channels. We
followed the setting in [20] to build a U-Net based reconstruction network but
modified the input and output as 2 channels to keep consistent with the DCCNN.
To perform anatomy parameterization settings, we replaced all the
convolutional operations, i.e. PN0, in U-Net/DCCNN by PN1-4, as shown in Fig.
2 (b).

####  IV-A4 Training Strategies

To implement the OAON baselines, we trained U-Net/DCCNN based reconstruction
models independently on three datasets for 50 epochs. To implement the
multiple-anatomy-related baseline, we fed a batch of data from the same
anatomy in a round-robin fashion in each iteration for $3\times 50=150$ epochs
as we used 3 anatomies in this study. For the MAPN methods, the anatomy-
specific learners were optimized only once in each iteration for their target
anatomy, while the anatomy-shared $3\times 3$ convolution layers were
optimized in every mini-batch across anatomies. We also adopted a warm-up
strategy for anatomy parameterized learners to dig out exclusive knowledge: we
loaded and fixed the weight of $3\times 3$ convolution layers obtained from
the MAON baseline for the first 30 epochs to better initialize the additive
anatomy-specific modules, then the whole network was fully optimized for the
next 120 epochs. The L1 loss was adopted for supervision and the Adam
optimizer with the same learning rate of 0.01 was adopted for all the methods.

###  IV-B Experiments of U-Net based Reconstruction

Here, we report the reconstruction performance of the anatomy parameterization
based U-Net as shown in Fig. 3 (a).

####  IV-B1 Quantitative Results

We summarized the quantitative results in Table. I, which can be divided into
three parts under $4\times$ and $6\times$ under-sampling ratio: (i) OAON
setting where knee/brain/cardiac expert models are trained independently, (ii)
MAON setting where all-round models are trained on the mixture of three
datasets, (iii) four different MAPN settings. Though the overall performance
with $6\times$ under-sampling ratio is lower than that with $4\times$ ratio,
findings are quite consistent between them, suggesting the robustness of
proposed method. Intuitively, we can find that MRI reconstruction results can
not benefit from direct multiple anatomy data mixed training. Expert for brain
reconstruction with $4\times$ ratio, the knee/brain/cardiac expert models all
outperform the MAON model although with unsatisfied anatomy generalization
ability. Our proposed MAPN solutions can learn from different anatomy to
better reconstruct MRIs compared to the naive MAON solution. By integrating
anatomy-specific batch normalization (PN1), we can find that the network can
enjoy the shared knowledge learned in $3\times 3$ convolutional layers to
outperform the MAON baseline in all three tasks but can not provide better
results than OAON experts in terms of the cardiac reconstructions. After
integrating anatomy-specific learners with PN2-4 settings, the MAPN network
can reconstruct all three anatomies better than both OAON expert models and
MAON baselines from collaborative learning. Notably, the PN2 setting achieves
comparable results with PN4 settings in U-Net based reconstruction.

####  IV-B2 Qualitative Results

We also visualize reconstruction performance of all settings in U-Net with
$4\times$ ratio in Fig. 4. Similar to the qualitative results, we can find
that networks with our parameterized learners can reconstruct images with
clearer details and reduced error maps.

TABLE I: Quantitative reconstruction results of U-Net based reconstruction on
5-fold cross validation of different anatomy parameterization strategies with
the metric of PSNR(dB) and SSIM(%). ‘Und. Rat.’ represents Under-sampling
Ratio. ‘K’,‘B’, and ‘C’ are short for ‘Knee’, ‘Brain’ and ‘Cardiac’ at the 3rd
column for compactness. ‘OAON’ is short for ‘One-Anatomy-One-Network’. ‘MAON’
is short for ‘Multiple-Anatomy-One-Network’. ‘MAPN’ is short for ‘Multiple-
Anatomy-Parameterized-Network’. The best results are addressed in bold.

Und. Rat. |  Learning Fashion | Model |  Training Anatomy | Knee Reconstruction | Brain Reconstruction | Cardiac Reconstruction
---|---|---|---|---|---|---
PSNR | SSIM |  $\Delta$ PSNR/SSIM | PSNR | SSIM |  $\Delta$ PSNR/SSIM | PSNR | SSIM |  $\Delta$ PSNR/SSIM
4x | OAON | U-Net | K | 34.82 | 82.09 | 0.51/2.24 | 34.45 | 87.24 | -2.11/-3.92 | 34.99 | 85.74 | -1.56/-3.90
U-Net | B | 31.12 | 70.22 | -3.19/-9.63 | 36.50 | 90.83 | -0.06/-0.33 | 33.57 | 83.85 | -2.98/-5.79
U-Net | C | 33.32 | 75.64 | -0.99/-4.21 | 35.54 | 89.36 | -1.02/-1.80 | 37.17 | 90.74 | 0.62/1.10
MAON | U-Net | K+B+C | 34.31 | 79.85 | 0/0 | 36.56 | 91.16 | 0/0 | 36.55 | 89.64 | 0/0
MAPN | U-Net+PN1 [17] | K+B+C | 34.86 | 81.91 | 0.55/2.06 | 37.00 | 91.50 | 0.44/0.34 | 37.18 | 90.67 | 0.63/1.03
U-Net+PN2 | K+B+C | 35.30 | 83.18 |  0.99/3.33 | 37.59 | 92.65 | 1.03/1.49 | 37.29 | 90.83 | 0.74/1.19
U-Net+PN3 | K+B+C | 35.22 | 82.77 | 0.91/2.92 | 37.37 | 91.93 | 0.81/0.77 | 37.28 | 90.83 | 0.73/1.19
U-Net+PN4 | K+B+C | 35.36 | 83.15 | 1.05/3.30 | 37.66 | 92.10 | 1.10/0.94 | 37.30 | 90.86 | 0.75/1.22
6x | OAON | U-Net | K | 33.58 | 77.65 | 0.27/1.32 | 32.92 | 84.21 | -1.25/-3.34 | 33.70 | 83.24 | -0.89/-2.96
U-Net | B | 30.70 | 68.15 | -2.61/-8.18 | 34.29 | 87.83 | 0.12/0.28 | 32.67 | 81.76 | -1.92/-4.44
U-Net | C | 32.90 | 74.02 | -0.41/-2.31 | 34.04 | 87.11 | -0.13/-0.44 | 35.46 | 88.11 | 0.87/1.91
MAON | U-Net | K+B+C | 33.31 | 76.33 | 0/0 | 34.17 | 87.55 | 0/0 | 34.59 | 86.20 | 0/0
MAPN | U-Net+PN1 [17] | K+B+C | 33.66 | 77.76 | 0.35/1.43 | 34.78 | 88.66 | 0.61/1.11 | 35.25 | 87.71 | 0.66/1.51
U-Net+PN2 | K+B+C | 34.25 | 79.30 | 0.94/2.97 | 35.86 | 90.44 | 1.69/2.89 | 35.80 | 88.58 | 1.21/2.38
U-Net+PN3 | K+B+C | 33.93 | 78.47 | 0.62/2.14 | 35.08 | 89.17 | 0.91/1.62 | 35.53 | 88.21 | 0.94/2.01
U-Net+PN4 | K+B+C | 34.12 | 79.01 | 0.81/2.68 | 35.66 | 89.91 | 1.49/2.36 | 35.64 | 88.40 | 1.05/2.20

Figure 4: Qualitative comparison of different anatomy parameterization U-Net
reconstruction settings with $4\times$ under-sampling ratio. Original MRIs are
normalized to [0, 1] for visualization and their intensity distributions are
shown in the first column. From the second column, reconstructed image (top)
and its absolute error map (bottom) are shown. PSNR (dB) is listed at the
right bottom of each error map with the highest one highlighted in red. Values
$>0.1$ in error maps are clipped to 0.1 for better visual comparison. Green
boxes address where significant performance difference can be observed.

###  IV-C Experiments of DCCNN based Reconstruction

Here, we report the reconstruction performance of the anatomy parameterization
based DCCNN as shown in Fig. 3 (b). Experiments show that DCCNN’s performance
is superior to U-Net in MRI reconstruction, which is consistent with previous
works [12, 13, 14]. Therefore, we perform more ablation experiments with
anatomy parameterization based DCCNN.

####  IV-C1 Quantitative Results

The quantitative results are summarized in Table. II with the same structure
as Table. I. Similar to the results of U-Net based reconstruction, we can
conclude that networks trained with MAON setting are confused by different
anatomies and can not reconstruct MRI as better as the corresponding OAON
experts. Our proposed MAPN solutions can learn from different anatomy to
better reconstruct MRIs compared to the naive MAON solution or the MAON model
with more learning parameters (DCCNN+PN4) - will be discussed later in the
following ablation study. Most differently, anatomy-specific channel attention
learners (PN2) fail to dig out useful information in most cases in DCCNN based
reconstruction. We assume that this is due to the different network
architectures between DCCNN and U-Net. Since U-Net’s encoder will double the
size of feature map after pooling while DCCNN’s feature map size is kept
constant, the anatomy-specific channel attention may have a better chance in
capturing useful information and thus generates better results. The optimal
reconstruction performance is obtained by anatomy-specific parallel learners
(PN4), which also performs robustly in U-Net based reconstruction results.

TABLE II: Quantitative reconstruction results of DCCNN based reconstruction on
5-fold cross validation of different anatomy parameterization strategies with
the metric of PSNR(dB) and SSIM(%). ‘Und. Rat.’ represents Under-sampling
Ratio. ‘K’,‘B’, and ‘C’ are short for ‘Knee’, ‘Brain’ and ‘Cardiac’ at the 3rd
column for compactness. ‘OAON’ is short for ‘One-Anatomy-One-Network’. ‘MAON’
is short for ‘Multiple-Anatomy-One-Network’. ‘MAPN’ is short for ‘Multiple-
Anatomy-Parameterized-Network’. The best results are addressed in bold.

Und. Rat. |  Learning Fashion | Model |  Training Anatomy | Knee Reconstruction | Brain Reconstruction | Cardiac Reconstruction
---|---|---|---|---|---|---
PSNR | SSIM |  $\Delta$ PSNR/SSIM | PSNR | SSIM |  $\Delta$ PSNR/SSIM | PSNR | SSIM |  $\Delta$ PSNR/SSIM
4x | OAON | DCCNN | K | 36.08 | 84.76 | 0.29/1.05 | 36.54 | 90.26 | -2.14/-2.68 | 37.73 | 90.76 | -1.62/-2.37
DCCNN | B | 33.45 | 76.14 | -2.34/-7.57 | 39.19 | 93.38 | 0.51/0.44 | 36.99 | 88.94 | -2.36/-4.19
DCCNN | C | 34.75 | 80.47 | -1.04/-3.24 | 37.74 | 91.79 | -0.94/-1.15 | 40.30 | 94.25 | 0.95/1.12
MAON | DCCNN | K+B+C | 35.79 | 83.71 | 0/0 | 38.68 | 92.94 | 0/0 | 39.35 | 93.13 | 0/0
DCCNN+PN4 | K+B+C | 35.49 | 82.91 | -0.30/-0.80 | 38.25 | 92.61 | -0.43/-0.33 | 38.82 | 92.41 | -0.53/-0.72
MAPN | DCCNN+PN1 [17] | K+B+C | 35.95 | 84.33 | 0.16/0.62 | 38.91 | 93.19 | 0.23/0.25 | 39.57 | 93.36 | 0.21/0.23
DCCNN+PN2 | K+B+C | 35.75 | 84.14 |  -0.04/0.43 | 38.39 | 92.82 | -0.29/-0.12 | 38.92 | 92.61 | -0.43/-0.52
DCCNN+PN3 | K+B+C | 36.31 | 85.35 | 0.52/1.64 | 39.86 | 93.92 | 1.18/0.98 | 40.31 | 94.25 | 0.96/1.12
DCCNN+PN4 | K+B+C | 36.50 | 85.82 | 0.71/2.11 | 40.15 | 94.32 | 1.47/1.38 | 40.49 | 94.45 | 1.14/1.32
6x | OAON | DCCNN | K | 34.13 | 78.77 | 0.19/0.78 | 33.78 | 85.47 | -1.39/-3.06 | 35.02 | 86.18 | -1.17/-2.61
DCCNN | B | 32.12 | 71.49 | -1.82/-6.50 | 35.64 | 89.34 | 0.47/0.81 | 34.78 | 85.07 | -1.41/-3.72
DCCNN | C | 33.39 | 75.95 | -0.55/-2.04 | 34.72 | 87.61 | -0.45/-0.92 | 36.85 | 90.05 | 0.66/1.26
MAON | DCCNN | K+B+C | 33.94 | 77.99 | 0/0 | 35.17 | 88.53 | 0/0 | 36.19 | 88.79 | 0/0
MAPN | DCCNN+PN1 [17] | K+B+C | 34.05 | 78.48 | 0.11/0.49 | 35.45 | 89.07 | 0.28/0.54 | 36.48 | 89.29 | 0.29/0.50
DCCNN+PN2 | K+B+C | 34.11 | 78.71 |  0.17/0.72 | 35.23 | 88.83 | 0.06/0.30 | 36.10 | 88.66 | -0.09/-0.13
DCCNN+PN3 | K+B+C | 34.28 | 79.24 | 0.34/1.25 | 36.17 | 90.21 | 1.00/1.68 | 36.87 | 90.06 | 0.68/1.27
DCCNN+PN4 | K+B+C | 34.47 | 79.75 | 0.53/1.76 | 36.51 | 90.77 | 1.34/2.24 | 37.07 | 90.39 | 0.88/1.60

Figure 5: Qualitative comparison of different anatomy parameterization DCCNN
reconstruction settings with $4\times$ under-sampling ratio. Original MRIs are
normalized to [0, 1] for visualization and their intensity distributions are
shown in the first column. From the second column, reconstructed image (top)
and its absolute error map (bottom) are shown. PSNR (dB) is listed at the
right bottom of each error map with the highest one highlighted in red. Values
$>0.1$ in error maps are clipped to 0.1 for better visual comparison. Green
boxes address where significant performance difference can be observed.

####  IV-C2 Qualitative Results

We also visually compare the reconstruction performance of all settings in
DCCNN with $4\times$ ratio in Fig. 5. We can also find that the visual quality
of final reconstructed MRIs are improved with our strategies.

Figure 6: Visualization of learned parameters in the 3rd CNN of DCCNN+PN4
model. The average weight values in BN of 5 blocks are shown in (a), and the
average weight values of $1\times 1$ convolutional layers are reported in (b).
The weights in additive learning modules are anatomy-sensitive. The weights of
BN can re-scale different distributions for different anatomies with simple
value shifting. The weights of anatomy-specific $1\times 1$ convolutional
layers vary more significantly between each other and help to improve the
final results. TABLE III: Comparison of learning parameter number between
different anatomy parameterization settings with DCCNN. Note that the
parameter sum of MAON+DCCNN+PN4 is not exactly equal to MAPN+DCCNN+PN4 due to
anatomy-shared BN settings.

| Learning
---
Fashion
Model |  | Anatomy-shared
---
Parameters (K)
| Anatomy-specific
---
Parameters (K)
| Three Antomy
---
Param. Sum (K)
OAON | DCCNN | 0.00 | 569.64 | 1708.92
MAON | DCCNN | 569.64 | 0.00 | 569.64
DCCNN+PN4 | 757.80 | 0.00 | 757.80
MAPN | DCCNN+PN1 [17] | 564.48 | 5.16 | 579.96
DCCNN+PN2 | 564.48 | 87.10 | 825.78
DCCNN+PN3 | 564.48 | 89.68 | 833.52
DCCNN+PN4 | 564.48 | 67.88 | 768.12

####  IV-C3 Comparison on the Increment of Learning Parameters

There is a remaining question that whether the improvement is brought by the
increment of learning parameters. We re-draw the MAPN+PN4 in Fig.2 (d-i) as
MAON+PN4 (d-ii), where 3 paralleled $1\times 1$ convolutional layers are
added, and we trained the DCCNN+PN4 with the MAON setting with the
quantitative results reported in Table. II with $4\times$ ratio. Here, we do
not use 3 paralleled BN layers because such setting seems to be non-sense. It
can be concluded that MAON+DCCNN+PN4 results in worse reconstruction than
MAPN+DCCNN+PN4, suggesting that the performance gain achieved by MAPN+PN4 is
not due to more parameters nor the parallel convolution with different
receptivity fields. The learning parameter numbers of all the involved anatomy
parameterization settings are summarized in Table. III. For a fair comparison,
OAON setting needs 3 expert models with $3\times 569.64$K = $1708.92$K
parameters in total to outperform the MAON baseline with $569.64$K parameters
for three anatomy reconstruction tasks. While our MAPN+PN4 models contains
$768.12$K parameters, which can save $>50\%$ parameters with better results
than OAON. The MAON+PN4 model has $757.80$K parameters, which are not exactly
equal to MAPN+PN4 without 3 paralleled BN in every convolution blocks.

####  IV-C4 Comparison on the Inference Time

We also evaluate the inference time for different methods, and PN1-4 show
little difference from the original DCCNN due to PN1-4 are parameter-
efficient. It takes $<0.6s$ to reconstruct per slice for all the methods with
an NVIDIA GeForce RTX 3090 GPU and Pytorch backend.

####  IV-C5 Visualization of the Unique Information Captured by Anatomy-
specific Learners

Here, we try to discuss what the simple anatomy-specific learners (BN and
$1\times 1$ convolution layers) capture for better reconstruction. However,
since MRI reconstruction is a low-level signal reconstruction task, to
visualize and explain the feature maps inside the network remains an unsolved
problem itself. So, we visualize the final learned parameters in the anatomy-
specific learners.

For simplicity, we focus on the parameters in DCCNN+PN4, which achieves the
best performance in our experiments. Without loss of generalization ability,
we visualize the learned parameters of the 3rd CNN by the following strategy:
we calculate and present the average value of weights in anatomy-specific BN
and $1\times 1$ convolutional operations across 5 blocks, and show them in
Fig.6 (a) and (b).

We can find that the weights in additive learning modules are anatomy-
sensitive. The weights of BN can re-scale different distributions for
different anatomies with simple value shifting. The weights of anatomy-
specific $1\times 1$ convolutional layers vary more significantly between each
other compared to anatomy-specific BN layers, indicating that paralleled
$1\times 1$ learners do learn something more unique than BN layers. As a
result, the integrated paralleled learners can then effectively guide the
reconstruction for their targeted anatomy and finally contribute to the
overall improvement.

##  V Conclusion and Discussion

In this work, we focus on an under-explored problem that how to make different
anatomy to collaborate with each other in one network for better MRI
reconstruction results. We reformulate the common deep MRI reconstruction
network with anatomy-shared and anatomy-specific learners and build a novel
Multiple-Anatomy-Parameterized-Network framework. Under our proposed
framework, we explore four possible implementations of anatomy-specific
learners and evaluate their performance on three anatomies with U-Net or DCCNN
based reconstruction networks. Experimental results prove that our framework
can help to boost the final performance in multiple anatomy reconstruction
tasks in general for different networks. It is surprising that anatomy-
specific learners built with simple BN layers and $1\times 1$ convolution can
capture useful information, which is evident in not only the reconstruction
results but also the visualization of learned parameters inside these
learners. However, under our proposed MAPN framework, how to design the
optimal anatomy-specific learners is still an open problem, and we believe
this work sheds the light in a new direction towards more robust MRI
reconstruction for multiple anatomy scene.

Our work is closely related to previous works [14] and [17]. In [14],
researchers found that the optimal network architectures obtained via neural
architecture searching are different for brain and knee reconstruction tasks.
However, it is difficult and impractical to design or search an anatomy-
specific architecture for every anatomy. In this work, we solve this problem
by disentangling the whole network into anatomy-share and anatomy-specific
parts and using the flexible anatomy-specific learners to model unique
knowledge instead of changing the entire architecture. We also reveal that the
same anatomy-specific learners, especially PN2, perform differently when they
are integrated into different reconstruction network architectures (U-Net and
DCCNN). Based on this observation, one potential future extension of our work
is to search the architectures of the flexible anatomy-specific learners via
neural architecture searching. Another important work towards a universal MRI
reconstruction is [17], where the researchers also used the anatomy-specific
normalization settings (PN1). However, they mainly focused on using knowledge
distillation in DCCNN to enhance the final performance, where well pre-trained
expert networks are required before training the universal model. In contrast,
our framework does not require any pre-trained expert models, and we further
enhance anatomy-specific normalization with more advanced designed learners.
Experiments show that our settings can robustly handle different networks.

Despite the good results, current experiments are still early attempts of our
proposed framework, and there exist some limitations. First, current
experiments are only conducted in single-coil MRI reconstruction scenario for
demonstration purpose. As multi-coil MRI reconstruction [36, 16] is more
popular in clinical practice, extending this work to multi-coil reconstruction
will be an important future work. Since we have evaluated our framework with
DCCNN [7], it is easy to extend DCCNN [10] to MoDL [10] framework by modifying
the network slightly and using conjugate gradient method to solve the Eq. 7
for multi-coil MRIs. Another limitation lies in the objective loss function.
In this work, we only adopted L1 loss to supervise the learning process of
deep networks for a fair and initial comparison without considering the real
distribution of the noise. It will be an important future work to introduce
more robust objective loss functions [22, 37, 38] to supervise the designed
learners for better modeling the reconstruction process in practice.

## Acknowledgment

The authors would like to thank the fastMRI [20] project for making their
codes and datasets accessible online.

## References

  * [1] D. L. Donoho, “Compressed sensing,” _IEEE Transactions on Information Theory_ , vol. 52, no. 4, pp. 1289–1306, 2006.
  * [2] D. Liang, B. Liu, J. Wang, and L. Ying, “Accelerating sense using compressed sensing,” _Magnetic Resonance in Medicine_ , vol. 62, no. 6, pp. 1574–1584, 2009.
  * [3] J. Huang, S. Zhang, and D. Metaxas, “Efficient mr image reconstruction for compressed mr imaging,” _Medical Image Analysis_ , vol. 15, no. 5, pp. 670–679, 2011.
  * [4] D. Liang, J. Cheng, Z. Ke, and L. Ying, “Deep magnetic resonance image reconstruction: Inverse problems meet neural networks,” _IEEE Signal Processing Magazine_ , vol. 37, no. 1, pp. 141–151, 2020.
  * [5] F. Knoll, K. Hammernik, C. Zhang, S. Moeller, T. Pock, D. K. Sodickson, and M. Akcakaya, “Deep-learning methods for parallel magnetic resonance imaging reconstruction: A survey of the current approaches, trends, and issues,” _IEEE Signal Processing Magazine_ , vol. 37, no. 1, pp. 128–140, 2020.
  * [6] S. Wang, Z. Su, L. Ying, X. Peng, S. Zhu, F. Liang, D. Feng, and D. Liang, “Accelerating magnetic resonance imaging via deep learning,” in _2016 IEEE 13th International Symposium on Biomedical Imaging_. IEEE, 2016, pp. 514–517.
  * [7] J. Schlemper, J. Caballero, J. V. Hajnal, A. N. Price, and D. Rueckert, “A deep cascade of convolutional neural networks for dynamic mr image reconstruction,” _IEEE Transactions on Medical Imaging_ , vol. 37, no. 2, pp. 491–503, 2017.
  * [8] C. Qin, J. Schlemper, J. Caballero, A. N. Price, J. V. Hajnal, and D. Rueckert, “Convolutional recurrent neural networks for dynamic mr image reconstruction,” _IEEE Transactions on Medical Imaging_ , vol. 38, no. 1, pp. 280–290, 2018.
  * [9] K. Hammernik, T. Klatzer, E. Kobler, M. P. Recht, D. K. Sodickson, T. Pock, and F. Knoll, “Learning a variational network for reconstruction of accelerated mri data,” _Magnetic Resonance in Medicine_ , vol. 79, no. 6, pp. 3055–3071, 2018.
  * [10] H. K. Aggarwal, M. P. Mani, and M. Jacob, “Modl: Model-based deep learning architecture for inverse problems,” _IEEE Transactions on Medical Imaging_ , vol. 38, no. 2, pp. 394–405, 2018.
  * [11] D. Lee, J. Yoo, S. Tak, and J. C. Ye, “Deep residual learning for accelerated mri using magnitude and phase networks,” _IEEE Transactions on Biomedical Engineering_ , vol. 65, no. 9, pp. 1985–1995, 2018.
  * [12] L. Sun, Z. Fan, Y. Huang, X. Ding, and J. Paisley, “Compressed sensing mri using a recursive dilated network,” in _Thirty-Second AAAI Conference on Artificial Intelligence_ , 2018.
  * [13] H. Zheng, F. Fang, and G. Zhang, “Cascaded dilated dense network with two-step data consistency for mri reconstruction,” in _Advances in Neural Information Processing Systems_ , 2019.
  * [14] J. Yan, S. Chen, Y. Zhang, and X. Li, “Neural architecture search for compressed sensing magnetic resonance image reconstruction,” _Computerized Medical Imaging and Graphics_ , vol. 85, p. 101784, 2020.
  * [15] A. Sriram, J. Zbontar, T. Murrell, A. Defazio, C. L. Zitnick, N. Yakubova, F. Knoll, and P. Johnson, “End-to-end variational networks for accelerated mri reconstruction,” in _International Conference on Medical Image Computing and Computer Assisted Intervention_. Springer, 2020, pp. 64–73.
  * [16] Y. Jun, H. Shin, T. Eo, and D. Hwang, “Joint deep model-based mr image and coil sensitivity reconstruction network (joint-icnet) for fast mri,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 5270–5279.
  * [17] X. Liu, J. Wang, F. Liu, and S. K. Zhou, “Universal undersampled mri reconstruction,” in _International Conference on Medical Image Computing and Computer Assisted Intervention_. Springer, 2021, pp. 211–221.
  * [18] S.-A. Rebuffi, H. Bilen, and A. Vedaldi, “Learning multiple visual domains with residual adapters,” in _Advances in Neural Information Processing Systems_ , 2017.
  * [19] ——, “Efficient parametrization of multi-domain deep neural networks,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 8119–8127.
  * [20] J. Zbontar, F. Knoll, A. Sriram, M. J. Muckley, M. Bruno, A. Defazio, M. Parente, K. Geras, J. Katsnelson, H. Chandarana, Z. Zhang, M. Drozdzal, A. Romero, M. G. Rabbat, P. Vincent, J. Pinkerton, D. Wang, N. Yakubova, E. Owens, C. L. Zitnick, M. P. Recht, D. K. Sodickson, and Y. W. Lui, “fastmri: An open dataset and benchmarks for accelerated mri,” _ArXiv_ , vol. abs/1811.08839, 2018.
  * [21] B. Zhu, J. Z. Liu, S. F. Cauley, B. R. Rosen, and M. S. Rosen, “Image reconstruction by domain-transform manifold learning,” _Nature_ , vol. 555, no. 7697, pp. 487–492, 2018.
  * [22] G. Yang, S. Yu, H. Dong, G. Slabaugh, P. L. Dragotti, X. Ye, F. Liu, S. Arridge, J. Keegan, Y. Guo _et al._ , “Dagan: deep de-aliasing generative adversarial networks for fast compressed sensing mri reconstruction,” _IEEE Transactions on Medical Imaging_ , vol. 37, no. 6, pp. 1310–1321, 2017.
  * [23] M. Akçakaya, S. Moeller, S. Weingärtner, and K. Uğurbil, “Scan-specific robust artificial-neural-networks for k-space interpolation (raki) reconstruction: Database-free deep learning for fast imaging,” _Magnetic Resonance in Medicine_ , vol. 81, no. 1, pp. 439–453, 2019.
  * [24] Y. Zhang, J. Lyu, and X. Bi, “A dual-task dual-domain model for blind mri reconstruction,” _Computerized Medical Imaging and Graphics_ , vol. 89, p. 101862, 2021.
  * [25] M. Arshad, M. Qureshi, O. Inam, and H. Omer, “Transfer learning in deep neural network based under-sampled mr image reconstruction,” _Magnetic Resonance Imaging_ , vol. 76, pp. 96–107, 2021.
  * [26] J. Lv, G. Li, X. Tong, W. Chen, J. Huang, C. Wang, and G. Yang, “Transfer learning enhanced generative adversarial networks for multi-channel mri reconstruction,” _Computers in Biology and Medicine_ , vol. 134, p. 104504, 2021.
  * [27] S. U. H. Dar, M. Özbey, A. B. Çatlı, and T. Çukur, “A transfer-learning approach for accelerated mri using deep neural networks,” _Magnetic Resonance in Medicine_ , vol. 84, no. 2, pp. 663–685, 2020.
  * [28] P. Guo, P. Wang, J. Zhou, S. Jiang, and V. M. Patel, “Multi-institutional collaborations for improving deep learning-based magnetic resonance image reconstruction using federated learning,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 2423–2432.
  * [29] H. Bilen and A. Vedaldi, “Universal representations: The missing link between faces, text, planktons, and cat breeds,” _arXiv preprint arXiv:1701.07275_ , 2017.
  * [30] A. Mallya, D. Davis, and S. Lazebnik, “Piggyback: Adapting a single network to multiple tasks by learning to mask weights,” in _Proceedings of the European Conference on Computer Vision_ , 2018, pp. 67–82.
  * [31] O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” in _International Conference on Medical image computing and computer Assisted intervention_. Springer, 2015, pp. 234–241.
  * [32] S. Ioffe and C. Szegedy, “Batch normalization: Accelerating deep network training by reducing internal covariate shift,” in _International Conference on Machine Learning_. PMLR, 2015, pp. 448–456.
  * [33] J. Hu, L. Shen, and G. Sun, “Squeeze-and-excitation networks,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 7132–7141.
  * [34] O. Bernard, A. Lalande, C. Zotti, F. Cervenansky, X. Yang, P.-A. Heng, I. Cetin, K. Lekadir, O. Camara, M. A. G. Ballester _et al._ , “Deep learning techniques for automatic mri cardiac multi-structures segmentation and diagnosis: Is the problem solved?” _IEEE Transactions on Medical Imaging_ , vol. 37, no. 11, pp. 2514–2525, 2018.
  * [35] A. Hore and D. Ziou, “Image quality metrics: Psnr vs. ssim,” in _2010 20th International Conference on Pattern Recognition_. IEEE, 2010, pp. 2366–2369.
  * [36] A. Sriram, J. Zbontar, T. Murrell, C. L. Zitnick, A. Defazio, and D. K. Sodickson, “Grappanet: Combining parallel imaging with deep learning for multi-coil mri reconstruction,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 14 315–14 322.
  * [37] J. Cheng, Z.-X. Cui, W. Huang, Z. Ke, L. Ying, H. Wang, Y. Zhu, and D. Liang, “Learning data consistency and its application to dynamic mr imaging,” _IEEE Transactions on Medical Imaging_ , vol. 40, no. 11, pp. 3140–3153, 2021\.
  * [38] Y. Zhou, H. Wang, Y. Liu, D. Liang, and L. Ying, “Accelerating mr parameter mapping using nonlinear compressive manifold learning and regularized pre-imaging,” _IEEE Transactions on Biomedical Engineering_ , 2022.


# Dynamic Open Vocabulary Enhanced Safe-landing with Intelligence (DOVESEI)

Haechan Mark Bong∗, Rongge Zhang∗, Ricardo de Azambuja∗, Giovanni Beltrame∗
∗MISTLab, École Polytechnique Montréal, Montréal, CanadaContact:
<EMAIL_ADDRESS>work was supported by the National Research
Council Canada (NRC).

###### Abstract

Drawing inspiration from the UNIX culture of “Do One Thing And Do It Well“,
this work targets what we consider to be the foundational step for urban
airborne robots, a safe landing. Furthermore, our attention is directed toward
what we deem the most crucial aspect of the safe landing perception stack:
segmentation. In order to tackle the challenge of ensuring safe landings, we
present a streamlined reactive UAV system that employs visual servoing by
harnessing the capabilities of contemporary open vocabulary image
segmentation. This proposed approach can adapt to various scenarios with
minimal adjustments to parameters, bypassing the necessity for extensive data
accumulation for refining internal models, thanks to its open vocabulary
methodology. Given the limitations imposed by local authorities (e.g.
altitude, noise levels), our primary focus centers on operations originating
from altitudes of 100 meters. This choice is deliberate, as numerous preceding
works have dealt with altitudes up to 30 meters, aligning with the
capabilities of small stereo cameras. Consequently, we leave the remaining 20m
to be navigated using conventional 3D path planning methods.

Utilizing monocular cameras and image segmentation, our findings demonstrate
the system’s capability to successfully execute landing maneuvers at altitudes
as low as 20 meters, all the while ensuring collision avoidance. However, this
approach is vulnerable to intermittent and occasionally abrupt fluctuations in
the terrain segmentation between frames in a video stream. To address this
challenge, we enhance the image segmentation output by introducing what we
call a dynamic focus: a masking mechanism that self adjusts according to the
current landing stage. This dynamic focus guides the control system to avoid
regions beyond the drone’s safety radius projected onto the ground, thus
mitigating the problems with fluctuations. Through the implementation of this
supplementary layer, our experiments have reached improvements in the landing
success rate of almost tenfold when compared to global segmentation. All the
source code is open source and available online. 111
https://github.com/MISTLab/DOVESEI

###### Index Terms:

SLZ, UAV, lightweight, Segmentation, Visual Servoing

##  I Introduction

Logistics stand as a pivotal element across diverse sectors, ranging from
e-commerce operations to complex military undertakings. Application of
Uncrewed [1, 2] Aerial Vehicles (UAVs) is becoming part of research and
industrial interest. Within this context, autonomous robots have emerged as an
extensively sought-after solution. Notably, in modern urban environments,
aerial robots are being explored as a compelling avenue to enhance last-mile
delivery efficiency and reduce carbon footprint. However, challenges
concerning safety have significantly hindered the widespread adoption of
flying robots in more densely populated areas. When not adequately designed
and operated, they can represent a possible threat to structures, vehicles and
the public in general, especially if problems arise with their geolocation and
other sensory information such that it could impede safe landing. Therefore,
our aim is to achieve secure landings without the need for external
communication, relying solely on onboard computational capabilities and
perceptual abilities of compact, lightweight cameras.

Many previous systems dealt with automatic UAV landing, but they would limit
the maximum altitude [3, 4, 5, 6, 7] (e.g. the OAK-D Lite [8] has a small
baseline, 75mm, limiting its maximum depth range to less than 19m), use more
expensive and heavier sensors [9, 6, 10] (e.g. 3D LiDARs), or scan and create
a map of the environment using Simultaneous Localization and Mapping (SLAM)
[3, 4, 11] before deciding where to land. Finally, past use of machine
learning models meant those systems would not tolerate domain shifts as they
were trained and tested for very specific scenarios. Safe landing zone
detection through segmentation needs to work in different scenarios. We
proposed to use an “open vocabulary“ based model [12, 13], which allows to
fine tune any internal model by changing only its prompt, without needing
extensive collection of data. As the system in emergency landing scenario may
not be able to receive or send data, external positioning like a Global
Navigation Satellite System (GNSS), Ultra-Wideband Positioning (UWB), laser or
radio guidance, or external markers (e.g. APRIL, ArUco) will not be available.
On the other hand, it is expected the UAV can be controlled by an extra
onboard system through velocity commands and it is capable of rudimentary 3D
position hold (e.g. barometer, range finder, optical flow or equivalent
sensor) to avoid major drifting. Consequently, our method does not use
odometry or a map of the environment. An inherent advantage of the proposed
methodology lies in its adaptability across diverse scenarios. By requiring
only minimal parameter adjustments, this approach can cater to varying
environments and operational conditions without necessitating extensive data
collection or recalibration.

Our motivation is to study a minimum viable system, capable of running even
with only a monocular RGB camera, that can “dinamically focus“, by masking the
received raw segmentation according to the system’s current state, and
leverage open vocabulary models to allow it to be easily “tuned“ only using
language without extensive data collection. Moreover, according to the onboard
sensors and computational capacity, such a system can be easily used in
parallel to other methods that offer odometry and 3D mapping for further
improvements. Our setup is a small, lightweight, and low-power system that can
be adapted to commercial drones, and it aims to find and land on a safe
landing zone (SLZ) (ground level, relatively flat, grass, open
field/area/park, avoiding pedestrians, vehicles, certain structures, etc.)
that is accessible for recovery by its operator in case of localization (e.g.
GPS) or remote-control communication failure. In sum, we envision achieving
the overarching objective of developing a compact, lightweight, onboard
external controller that can be affixed to commercial UAVs, enabling them to
execute safe landings even in scenarios involving internal navigational and
sensory system issues, be it due to accidents, technical malfunctions, or
attacks.

##  II System Design

###  II-A Main System Architecture

Figure 1: Our safe-landing system was implemented in ROS 2 and it is composed
of three main blocks: UAV (flight controller, sensors), landing heatmap
generation (receives a RGB image and produces a heatmap of the best places to
land), and main processing node (orchestrates the data exchange with the UAV,
sends velocity of landing commands).

The proposed system architecture, implemented as a ROS 2 [14] package,
encompasses two discrete yet interconnected processes. These processes are the
Landing Heatmap Generation Service and the Main Processing Node (Fig. 1).

###  II-B Landing Heatmap Generation Service

The heatmap generator receives as inputs a RGB image and textual prompts, and
it uses a pre-trained semantic segmentation model to generate (zero-shot) a
comprehensive heatmap. This heatmap offers crucial insights into optimal
(referred throughout this work as “best“) landing positions within the context
of the current image frame.

The identification of these optimal landing locations (pixels) is executed
through the utilization of the open vocabulary segmentation model CLIPSeg
[13], which is rooted in CLIP [12], a model trained on a private dataset with
an estimated 400 million image-text pairs. To provide some context on CLIP’s
training dataset scale, the widely recognized ImageNet [15] dataset contains
_only_ 1.2 million images. We use CLIPSeg with a collection of words that was
strategically chosen to align with the environment, such as rendered
simulations and real-world satellite images. The outcomes of the segmentation
process are combined to create the initial raw safe landing heatmap.

###  II-C Main Processing Node

The main node is responsible for the high level control of the whole system
and it’s directly interconnected to the UAV flight controller. Its
functionality can be summarised within three core components: the main state
machine, raw heatmap post-processing, and dynamic focus.

####  II-C1 State Machine

The state machine controls the dynamic behavior of our system and its main
states are:  i) Searching: coarse search for a landing spot from a safe
(collision free) altitude.  ii) Aiming: refined search to better align the UAV
with a safe landing spot.  iii) Landing: descend while checking for dynamic
obstacles.  iv) Waiting: stop and wait if any obstacles where detected after
it started landing.  v) Climbing: climb back to the safe altitude if the
waiting phase triggered a failure.  vi) Restarting: restart the coarse search
by moving to a new starting position.

####  II-C2 Raw Heatmap Post-processing

Figure 2: The visual servoing system takes as input raw segmentation heatmaps
(pixels with classes considered good to land on), averages them over time
(UAV’s max. bank angle is limited, constraining its max. horizontal speed),
creates a distance map, applies our dynamical focus masking method, and
finally the objective function, Eq. (1), to decide on the best pixel.

The raw segmentation received from the Landing Heatmap Generation Service
(II-B) is a binary mask and it alone is not enough to allow the selection of
the next movement direction. To compute this direction, it’s essential to
choose the optimal pixel for landing, which involves passing the raw
segmentation through the pipeline depicted in Fig. 2. A final “best“ pixel
position is found using the objective function (the higher the value, the
better) below:

$P_{best}=\frac{Area}{Perimeter}\cdot{\left(C_{dist}+1\right)}^{-1}$ (1)

where $P_{best}$ is the objective function value defining the best pixel to
land on (see Fig. 2), $Area$ and $Perimeter$ refer to the continuous
segmentation patch where the pixel is in (after the Distance Map, see Fig. 2),
and $C_{dist}$ is the distance to the centre of the image (the UAV’s virtual
position).

####  II-C3 Dynamic Focus

Figure 3: The focus mask radius (R in the illustration above) continuously
varies (Eq. 2), expanding or shrinking, according to the current state of the
system. Its minimum size is limited by the UAV’s projection on the ground
(multiplier factor 6X for Aiming and 2X for Landing), while its upper limit is
when the image is inscribed in the circle.

The dynamic focus regulates the extent to which the raw heatmap is processed
by the Raw Heatmap Post-processing module (II-C2). It “focuses“, i.e. applies
a binary mask that covers portions of the input, on the most important areas
according to the current system state (II-C1) and its operation is illustrated
in Fig. 3. The dynamic (instantaneous) radius of the mask is define by the
equation below:

$R_{focus}=R_{focus}+(S-R_{focus})\lambda$ (2)

where $R_{focus}$ is the dynamic focus radius (the R in Fig. 3), $S$ is the
UAV’s safe-radius (user-defined parameter for maintaining a minimum clearance
from obstacles) projection on the ground multiplied by a factor depending on
the current state (6X for Aiming and 2X for Landing), and $\lambda$ is
constant (empirically set to 0.1) that controls its decay or increase speed.

##  III Experimental Setup

All experiments were conducted employing high-resolution satellite images from
Paris, France, sourced from our open-source specialized ROS 2
package222https://github.com/ricardodeazambuja/ROS2-SatelliteAerialViewSimulator.
We can’t guarantee, but we don’t expect these specific images, matched with
our prompts, were ever seen by CLIP during training, therefore we consider
this setting as zero-shot. A total of 50 experiments were carried out for each
configuration: one with the implementation of our dynamic focus masking
approach, and the other without (_No Focus_). The simulated UAV always starts
at an altitude of 100m and in a random location uniformly sampled from the
rectangle formed by the latitude and longitude coordinates pairs
(48.8810503932738, 2.2926285019834536) and (48.837230391660526,
2.389960496086875). This bounding box is illustrated in Fig. 4.

Figure 4: Satellite image of Paris showing the latitude and longitude bounding
box used to uniformly sample the 50 starting positions for our experiments
(red box, dashed) [16]

##  IV Results

To validate our initial hypothesis that the incorporation of the Dynamic Focus
(II-C3) into the open vocabulary visual servoing system for safe landings was
effective, we conducted tests on both systems, labeled as _Dynamic Focus_ and
_No Focus_ , utilizing identical randomly sampled latitude and longitude
starting positions (III). Therefore, besides the use or not of the Dynamic
Focus, all other parameters were kept the same.

TABLE I: Aggregated results from our landing experiments | Dynamic Focus | No Focus
---|---|---
Total Successful Runs | 29 | 3
Average Horizontal Distance (m) | 74.40 | 81.77
Average Time Spent (s) | 843.98 | 943.43

As outlined earlier, our success criteria involve achieving an altitude of 20m
over a suitable landing area, thus enabling low-cost, lightweight stereo
camera and conventional 3D path planning techniques to be applied. Hence, goal
positions aren’t set but instead manually determined through visual
inspection. Experiments exceeding the maximum allowed time (1200s) are
terminated.

Figure 5: Example of a successful landing approach (from lat. / lon.
48.83948619062335 / 2.296169442158885 to 48.83922328688019 /
2.2948090593103774, red circle to blue star). From right to left: initial
UAV’s view (alt. 100m), zoom out with alt. 300m (trajectory in red, yellow
dashes initial view), final location (alt. 100m) and final UAV’s view.

From a total of 50 experiments for each configuration, the setup using Dynamic
Focus reached the defined safe-landing goal in 29 flights versus 3 flights
without it. We also measured the total time and horizontal distance travelled
in the successful trials where Dynamic Focus was, on average, 10.6% faster and
travelled a distance 9.0% smaller (Table I).

Figure 6: Paths generated for landing experiments starting at the same
location as presented in Fig. 5, where stars indicate where each trajectory
reached at the end of the experiment. Without the dynamic focus, the (global)
semantic segmentation noise pushed the system away from a successful landing
spot (dashed line, blue).

For a clearer depiction of a typical successful run and due to space
constraints in this publication, Fig. 5 displays the satellite images
alongside the UAV’s path for one run using the dynamic focus. The paths for
both configurations, and the same starting location, is presented in the Fig.
6. While the UAV is navigating only in the Searching state( II-C1), both
configurations present the same behaviour, as expected. The dynamic focus
allows the system to ignore “distractions“ in the global semantic segmentation
after it has found a good landing place candidate.

##  V Discussions and Conclusions

The primary motivation behind this work was to explore an alternative
approach. Rather than fully relying on conventional solutions rooted in 3D
SLAM and path planning, this study aimed on investigating the feasibility of
transferring much of the system’s intelligence to an open-vocabulary semantic
segmentation model that uses only a RGB monocular camera, thus leaving only
the final metres to be handled by a small stereo camera and traditional 3D
path planing. The reliance on semantic segmentation and the observed
susceptibility of such systems to fluctuations even with slight alterations in
input images [17] prompted the development of our dynamic focus masking
mechanism.

While relatively straightforward to be implemented in code, our dynamic focus
functions as a selective time-varying mask significantly enhancing the UAV’s
ability to select suitable landing sites for safe descents. Although the
number of successful runs for the “no focus“ setup was limited, with only 3
out of 50 runs, it’s important to acknowledge that the inclusion of our
dynamic focus yielded a substantial increase in successful outcomes, with 29
successes out of 50 runs. Although the sample size for successful runs in the
“no focus“ setup might not be statistically significant, the impact of
introducing the dynamic focus seems evident in the improved success rate.

The fact that most previous works focused on low initial altitudes [3, 4, 5,
6, 7] or equipment with higher complexity [6, 10] and weight that would not
fit in a small UAV [9], makes us confident in the potential of our approach,
underscoring the significance of further exploration and expansion of this
research in the future.

##  VI Future Works

Since the objective of our work is safe landing, robustness on segmentation is
paramount. Such systems achieve high scores in zero-shot tasks and, therefore,
offer improved generalisation capabilities over models that use a closed
vocabulary (e.g. a model trained only on MS COCO [18] classes). More efforts
are needed to advance prompt engineering optimized for aerial images together
with the understanding of what is considered safe given an environment.

Furthermore, our simulated UAV stopped at an altitude of 20m without exploring
the final metres where small stereo vision sensors would have enough range to
allow the use of 3D based path planning.

Such efforts would be advantageous in the context of detecting secure landing
zones.

## References

  * [1] S. Garber, “Style guide for nasa history authors and editors,” _National Aeronautics and Space Administration_ , 2012, accessed: 2022-02-17. [Online]. Available: https://history.nasa.gov/styleguide.html
  * [2] M. Koren, “The outdated language of space travel,” Jul 2019, accessed: 2022-02-17. [Online]. Available: https://www.theatlantic.com/science/archive/2019/07/manned-spaceflight-nasa/594835/
  * [3] M. K. Mittal, A. Valada, and W. Burgard, “Vision-based autonomous landing in catastrophe-struck environments,” _ArXiv_ , vol. abs/1809.05700, 2018.
  * [4] E. Chatzikalymnios and K. Moustakas, “Landing site detection for autonomous rotor wing uavs using visual and structural information,” _Journal of Intelligent & Robotic Systems_, vol. 104, 02 2022.
  * [5] C. Forster, M. Faessler, F. Fontana, M. Werlberger, and D. Scaramuzza, “Continuous on-board monocular-vision-based elevation mapping applied to autonomous landing of micro aerial vehicles,” in _2015 IEEE International Conference on Robotics and Automation (ICRA)_ , 2015, pp. 111–118.
  * [6] M. Rabah, A. Rohan, M. Talha, K.-H. Nam, and S. Kim, “Autonomous vision-based target detection and safe landing for uav,” _International Journal of Control, Automation and Systems_ , 10 2018.
  * [7] J. Park, Y. Kim, and S. Kim, “Landing site searching and selection algorithm development using vision system and its application to quadrotor,” _IEEE Transactions on Control Systems Technology_ , vol. 23, no. 2, pp. 488–503, 2015.
  * [8] “Luxonis oak-d lite documentation,” https://docs.luxonis.com/projects/hardware/en/latest/pages/DM9095/, accessed: 2023-08-15.
  * [9] A. Cesetti, E. Frontoni, A. Mancini, P. Zingaretti, and S. Longhi, “A vision-based guidance system for uav navigation and safe landing using natural landmarks,” _Journal of Intelligent and Robotic Systems_ , vol. 57, pp. 233–257, 01 2010.
  * [10] B. E. Cohanim, J. A. Hoffman, and T. Brady, “Onboard and self-contained landing site selection for planetary landers/hoppers,” in _2014 IEEE Aerospace Conference_ , 2014, pp. 1–13.
  * [11] L. Bartolomei, Y. Kompis, L. Teixeira, and M. Chli, “Autonomous emergency landing for multicopters using deep reinforcement learning,” in _2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , 2022, pp. 3392–3399.
  * [12] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark _et al._ , “Learning transferable visual models from natural language supervision,” in _International conference on machine learning_. PMLR, 2021, pp. 8748–8763.
  * [13] T. Lüddecke and A. Ecker, “Image segmentation using text and image prompts,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2022, pp. 7086–7096.
  * [14] D. Thomas, W. Woodall, and E. Fernandez, “Next-generation ROS: Building on DDS,” in _ROSCon Chicago 2014_. Mountain View, CA: Open Robotics, sep 2014. [Online]. Available: https://vimeo.com/106992622
  * [15] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in _2009 IEEE Conference on Computer Vision and Pattern Recognition_ , 2009, pp. 248–255.
  * [16] “Google maps,” https://www.google.com/maps/, accessed: 2023-08-19.
  * [17] A. Azulay and Y. Weiss, “Why do deep convolutional networks generalize so poorly to small image transformations?” _Journal of Machine Learning Research_ , vol. 20, no. 184, pp. 1–25, 2019.
  * [18] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick, “Microsoft coco: Common objects in context,” in _Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13_. Springer, 2014, pp. 740–755.