# Multi-Modality Cardiac Image Analysis with Deep Learning

Lei Li# , Fuping Wu#, Sihang Wang#, Xiahai Zhuang# # All authors contribute
equally to this article.Lei Li is with Department of Engineering Science,
University of Oxford, Oxford, UK. Fuping Wu, Sihan Wang and Xiahai Zhuang are
with School of Data Science, Fudan University, Shanghai, China. Xiahai Zhuang
is the corresponding author<EMAIL_ADDRESS>
###### Abstract

Accurate cardiac computing, analysis and modeling from multi-modality images
are important for the diagnosis and treatment of cardiac disease. Late
gadolinium enhancement magnetic resonance imaging (LGE MRI) is a promising
technique to visualize and quantify myocardial infarction (MI) and atrial
scars. Automating quantification of MI and atrial scars can be challenging due
to the low image quality and complex enhancement patterns of LGE MRI.
Moreover, compared with the other sequences LGE MRIs with gold standard labels
are particularly limited, which represents another obstacle for developing
novel algorithms for automatic segmentation and quantification of LGE MRIs.
This chapter aims to summarize the state-of-the-art and our recent advanced
contributions on deep learning based multi-modality cardiac image analysis.
Firstly, we introduce two benchmark works for multi-sequence cardiac MRI based
myocardial and pathology segmentation. Secondly, two novel frameworks for left
atrial scar segmentation and quantification from LGE MRI were presented.
Thirdly, we present three unsupervised domain adaptation techniques for cross-
modality cardiac image segmentation.

_Multi-Modality, Cardiac Image, Deep Learning, Domain Adaptation_

##  1 Introduction

Multi-modality cardiac images are widely utilized to assist the diagnosis and
treatment management of patients. Cardiac computed tomography (CT) is
generally fast, low cost and with high quality for cardiac imaging. Magnetic
resonance imaging (MRI) can provide important anatomical and function
information of the heart, and different sequences of MRI can further capture
different information. For example, the LGE MRI sequence has been widely used
to visualize myocardial infarction (MI) as well as left atrial (LA) fibrosis
and scars; the balanced-steady state free procession (bSSFP) cine sequence can
present clear cardiac boundaries and captures cardiac motions in different
phases; the T2-weighted MRI sequence can display the acute injury and ischemic
regions. Combining multi-modality/sequence cardiac images is a promising
research direction in the literature, such as multi-modality registration [1],
multi-modality/sequence fusion [2, 3], and domain adaptation [4].

In this chapter, we will present three topics related to the multi-modality
cardiac image processing. Section 2 presents two challenges for multi-sequence
cardiac MRI based myocardial and pathology segmentation. Section 3 presents
two novel frameworks, namely LearnGC and AtrialJSQnet, for left atrial scar
segmentation and quantification from LGE MRI (with the assist of an additional
non-enhanced MRI). Section 4 introduces three unsupervised domain adaptation
algorithms for cross-modality cardiac image segmentation.

##  2 Multi-sequence cardiac MRI based myocardial and pathology segmentation

###  2.1 Introduction

Cardiovascular disease is one of the leading causes of global death, among
which MI is the most acute and deadly one [5]. Early diagnosis as well as
prompt treatment are the key to prevent the poor prognosis. Among the imaging
modalities available in clinical routines for MI, cardiac magnetic resonance
(CMR) imaging becomes the gold-standard technique. The functional cardiac
indexes, i.e., left ventricular (LV), ejection fraction (EF), LV volumes,
myocardium mass as well as precise location of lesion could be obtained from
the analysis of MRI. However, one MRI sequence could provide limited clinic
information. In clinic practice, multi-modality would be simultaneously
utilized, since the complementary information from multi sequences can assist
the diagnosis and location of lesion. Whereas, the misalignment among multi
modalities caused by the different scanning orientations, the various
sequences characteristic and the low tissue contrast in special modalities
(such as T2 and LGE CMR) make it challenging to handle multi-modality tasks.

To boost the study in this field, in recent years, we have organized two
challenges, providing a fair and unified platform and open benchmark dataset
for researches around the world. We first held MS-CMRSeg challenge event for
LGE CMR image segmentation with multi-sequence CMR available, segmenting the
anatomical structure (i.e., left ventricle, right ventricle and myocardium ) .
Fig. 1 provides an example of multi-sequence images from one subject,
including bSSFP, T2 and LGE CMR. One can see that there exist misalignment,
vast image intensity distribution gap and various pathology region shape among
them. These differences are the main obstacles for image analysis. As an
extension, we then organized myocardial pathology segmentation (MyoPS)
challenge event for the myocardial pathology segmentation, including scar and
edema. In the following, we summarize the top-performed methods in the two
challenges.

Figure 1: Visualization of myocardial pathology segmentation combing multi-
sequence CMR images acquired from the same patient. Image adopted from the
website of MyoPS 2020 challenge:
http://www.sdspeople.fudan.edu.cn/zhuangxiahai/0/myops20/.

###  2.2 Methodology summary for challenge events

####  2.2.1 MS-CMRSeg Challenge: Cardiac Segmentation on Late Gadolinium
Enhancement MRI

We organized multi-sequence cardiac MR segmentation (MS-CMRSeg) challenge, in
conjunction with MICCAI 2019. The challenge mainly focused on the anatomical
structure (ventricles and myocardium) segmentation of LGE sequence with
complementary information from the other two sequences (bSSFP and T2
sequence). Sixty-five teams from all over the world participated in the
challenge, twenty three of which submitted results for evaluation before the
submission deadline. To keep the chapter concise, only the top nine methods
will be discussed. These nine methods could roughly be divided into two
categories. The first one is training without LGE annotation, namely an
unsupervised domain adaptation (UDA) problem, and three teams focused on this
task; the other six teams trained their models with a few LGE ground-truth,
considered as supervised methods.

For UDA methods, two teams, i.e., ICL [6] and INRIA [7], adopted image
translation scheme via the style-transfer. ICL leveraged multi-modal
unsupervised image-to-image translation network [8] to implement style
transfer. In their framework, the encoder extracts style information and
structure information from image separately, while decoders reconstruct the
image using the extracted features. INRIA adopts conventional style
transformation, i.e., histogram matching towards LGE, and also employs some
data augmentation schemes including adaptive histogram equalization and
intensity inversion. As for the network structure, ICL adopts two stage
cascaded network to extract ”coarse to fine” features, while INRIA utilizes a
dual U-Net [9]. Instead of using image translation, XMU [10] performs feature
alignment to achieve UDA optimization. The alignment is achieved via a
discriminator, which can minimize the distribution discrepancy between the two
domains in both feature and output spaces.

Among the supervised-learning based methods, UB [11] and HIT [12] mainly
adopted several date augmentation strategies to avoid over-fitting, while the
other teams try to design effective model structures for LGE MRI segmentation.
For data augmentation, UB leverages CycleGAN strategy [13] to convert bSSFP
images into LGE-like ones and utilizes region rotation of scars, while HIT
uses histogram matching for augmentation. For model structure, only NVIDIA [7]
leverages conventional multi-atlas segmentation (MAS) framework, while the
other teams, i.e., FAU [14], SUST [15] and SCU [16] employs deep neural
networks. SCU [16] redesigns the baseline U-Net with the squeeze-and-
excitation residual module [17] and the selective kernel module [18] to
recalibrate channel-wise feature responses and adjust the size of its
receptive field, respectively. FAU [14] uses transfer-learning to facilitate
modeling ability with complementary modality information for bSSFP and T2.
SUST utilizes generator and discriminator to generate segmentation masks from
LGE CMR image.

####  2.2.2 MyoPS: Myocardial Pathology Segmentation from Multi-Sequence
Cardiac MRI

We further organized MyoPS challenge as the extension of MS-CMRSeg challenge,
in conjunction with MICCAI 2020, to focus on the pathology segmentation
instead of the anatomical structure segmentation. Up to now, MyoPS challenge
has received seventy-six requests of registration around world, among which
twenty-three teams submitted results for evaluation. As mentioned before, one
of the major challenges of multi-modal image analysis is the misalignment
between different modalities. To alleviate this, in this challenge we pre-
processed the MS-CMR dataset via the MvMM algorithm [19] for inter-sequence
registration. In this section, we will summarize useful training strategies
and trends observed from 15 approaches submitted to MyoPS, in terms of
preprocessing, data augmentation, segmentation model architecture and post-
processing.

Preprocessing. Preprocessing is a crucial technique to mitigate undesirable
variations from raw data and reduce modeling complexity. Since the region of
interest (edema and scar) of CMR is relatively small compared to background,
all approaches crops the training images as a preprocessing. Most of them
roughly crops the training images to generate the center of heart. For
example, USTB [20] simply crops the images into the ROI with 256 $\times$ 256
pixels. Besides, several teams employs prior segmentation network to
automatically localize the position of LV and myocardium for ROI extraction.
For example, UBA [21] leverages U-Net to segment myocardium and crops the
smallest bounding box containing the prediction. In this way, one could obtain
an ROI and dramatically reduce the useless information for the following
pathology segmentation. Moreover, data normalization is also a vital data
preprocessing technique to reduce data variations, due to the large domain
shift among different modalities. The majority of algorithms adopts the
z-score normalization, while several teams simply scales the value to $[0,1]$.

Data augmentation. Data augmentation schemes are widely used to facilitate
model generalization ability. The augmentation strategies utilized in the
challenge methods could be roughly categorized into two parts, online
transformation and offline data augmentation. Online transformation mainly
contains several conventional augmentation schemes, such as randomly rotation,
scaling, shifting, brightness, non-rigid transformations and contrast
adjustment. For example, USTB [20] transforms the original images non-rigidly
with elastic-transform, grid-distortion and optical-distortion. Offline
augmentation mainly refers to data generation. UBA [21] adopts spatially-
adaptive normalization [22] for style transfer, pathology rotation and
dilation/ erosion.

Model architectures. The design of model architecture is another determinant
for the prediction results. The most prevailing backbone utilized by these
methods is U-Net, which includes elegant symmetric encoder and decoder
structure with multi-scale feature and skip-connection strategy. UESTC [23]
utilizes U-Net for both coarse and fine segmentation networks. Moreover, dense
connection and attention are useful techniques. UBA [21] utilized an U-Net
variant, termed as BCDU-Net [24], which reuses feature maps via dense
convolutions. USTB [20] adopts channel and space attention modules at the
basis of U-Net. As for loss functions, most teams employ the Dice loss and the
cross entropy loss. Also, FZU [25] utilizes boundary loss to enforce the model
to focus on the boundary regions.

Post-processing. Among all teams, only four of them utilized post-processing
to refine the predictions. UBA [21] adopts the most complicated post-
processing scheme. They first constrains the myocardium into a ring shape, and
then calculates the distances of the pixels which are predicted as non
background to the Myo. In addition, NPU [26] and USTB [20] simply removes the
isolated regions. UOA [27] solely retains the largest connected component of
predicted LV, and then fills the holes.

###  2.3 Data and results

####  2.3.1 Data

The datasets of two challenges were collected from the same patient group, but
with different annotations. The datasets includes multi-CMR sequences (bSSFP,
T2 and LGE), taken from 45 subjects who underwent cardiomyopathy, and were
annotated by at least three experts. They had been collected with
institutional ethics approval and underwent anonymization. Moreover, multi-
sequences were pre-aligned by MvMM algorithm [19] in the MyoPS challenge. The
details of three sequences are shown in Table 1.

Table 1: Image acquisition parameters of the original multi-sequence data. ED: end-diastolic [28]. Sequence | Imaging type | No. slices | TR/ TE (ms) | Slice thickness | In-plane resolution
---|---|---|---|---|---
LGR CMR | T1-weighted | 10-18 | 3.6/1.8 | 5 mm |  0.75 $\\!\times\\!$ 0.75 mm2
T2 CMR | T2-weighted | 3-7 | 2000/90 | 12-20 mm |  1.35 $\\!\times\\!$ 1.35 mm2
bSSFP CMR | Cine sequence | 8-12 | 2.7/1.4 | 8-13 mm |  1.25 $\\!\times\\!$ 1.25 mm2

####  2.3.2 Evaluation metrics

For evaluation, Dice score was applied in both MS-CMRSeg and MyoPS challenges,

$\displaystyle Dice(V_{seg},V_{GD})=\frac{2|V_{seg}\And
V_{GD}|}{|V_{seg}|+|V_{GD}|},$ (1)

where $V_{GD}$ and $V_{seg}$ denote the gold standard and automatic
segmentation, respectively. Moreover, MS-CMRSeg used average surface distance
(ASD) and Hausdorff distance (HD) as supplementary metrics, which can be
defined as,

$\displaystyle HD(X,Y)=max[\sup_{x\in X}\inf_{y\in Y}d(x,y),\sup_{y\in
Y}\inf_{x\in X}d(x,y)],$ (2)
$\text{ASD}(X,Y)=\frac{1}{2}\left(\frac{\sum_{x\in X}\min_{y\in
Y}d(x,y)}{\sum_{x\in X}1}+\frac{\sum_{y\in Y}\min_{x\in X}d(x,y)}{\sum_{y\in
Y}1}\right),$ (3)

where X and Y represent two sets of contour points, and $d(x,y)$ indicates the
distance between the two points x and y.

Figure 2: The average Dice score of the nine evaluated methods on each slice
of the 40 test LGE CMR images. Image adopted from Zhuang et al. [28]

.

####  2.3.3 Results from MS-CMRSeg challenge event

Table 2 and Table 3 present the quantitative results of the nine evaluated
algorithms. The mean Dice scores of Myo, LV and RV segmentation were $0.766\pm
0.104$, $0.891\pm 0.056$ and $0.822\pm 0.116$ respectively, and the mean
volumetric HD values were $16.37\pm 12.27$ mm (LV Endo), $18.06\pm 12.18$ mm
(LV Epi) and $19.35\pm 9.587$ mm respectively. Interestingly, the unsupervised
domain adaptation models performed comparably to supervised ones, thanks to
the style-translation data synthesis. As shown in Table 2, the Dice scores of
LV were evidently better than that of RV and Myo. Similarly, the HD values of
myocardium including LV Endo and Epi, were generally better than that of RV
Endo. One reasonable explanation is that the variance of the shape of RV is
larger than that of LV, which commonly presents a ring-like structure.
Moreover, the segmentation results were related to spatial position. Fig. 2
presents the accuracy of different slices. One can see that the accuracy on
apical slices were generally worse than that on midden and basal slices, which
may be due to the shape variance.

Table 2:  Dice score of the evaluated algorithms on the LGE MRI segmentation [28]. UDA: unsupervised domain adaptation; Teams updating their results after the challenge deadline are indicated with an asterisk (*), and teams using the unlabeled LGE images ($I^{\mathrm{unl}}$) for training are indicated with a dagger (${\dagger}$).  Teams | Volumetric Dice | Training
---|---|---
Myo | LV | RV
ICL† [6] | $0.826\pm 0.035$ | $0.919\pm 0.026$ | $0.875\pm 0.050$ | UDA
XMU† [10] | $0.796\pm 0.059$ | $0.896\pm 0.047$ | $0.846\pm 0.086$ | UDA
INRIA∗[29] | $0.705\pm 0.115$ | $0.870\pm 0.051$ | $0.762\pm 0.150$ | UDA
SCU∗[16] | $0.843\pm 0.048$ | $0.926\pm 0.028$ | $0.890\pm 0.044$ | Supervised
UB† [11] | $0.810\pm 0.061$ | $0.898\pm 0.045$ | $0.866\pm 0.050$ | Supervised
FAU [14] | $0.789\pm 0.073$ | $0.912\pm 0.034$ | $0.833\pm 0.084$ | Supervised
NVIDIA† [7] | $0.780\pm 0.047$ | $0.890\pm 0.043$ | $0.844\pm 0.063$ | Supervised
HIT† [12] | $0.751\pm 0.119$ | $0.884\pm 0.070$ | $0.791\pm 0.165$ | Supervised
SUSTech [15] | $0.610\pm 0.102$ | $0.824\pm 0.068$ | $0.710\pm 0.135$ | Supervised
Average | $0.775\pm 0.093$ | $0.895\pm 0.047$ | $0.828\pm 0.114$ | UDA
$0.764\pm 0.109$ | $0.889\pm 0.060$ | $0.822\pm 0.117$ | Supervised
$0.766\pm 0.104$ | $0.891\pm 0.056$ | $0.822\pm 0.116$ | All
Inter-Ob | $0.764\pm 0.069$ | $0.881\pm 0.064$ | $0.816\pm 0.084$ |

Table 3:  HD of the evaluated algorithms on the LGE MRI segmentation [28].  Teams | Volumetric HD (mm) | Training
---|---|---
LV Endo | LV Epi | RV Endo
ICL†[6] | $10.28\pm 3.376$ | $12.45\pm 3.142$ | $15.38\pm 6.942$ | UDA
XMU†[10] | $13.59\pm 5.206$ | $15.70\pm 5.814$ | $15.21\pm 6.327$ | UDA
INRIA∗[29] | $41.74\pm 7.696$ | $42.79\pm 13.26$ | $34.38\pm 8.065$ | UDA
SCU∗[16] | $9.748\pm 3.280$ | $11.65\pm 4.002$ | $13.34\pm 4.615$ | Supervised
UB†[11] | $10.78\pm 4.066$ | $11.96\pm 3.620$ | $15.91\pm 6.895$ | Supervised
FAU [14] | $11.29\pm 4.559$ | $12.54\pm 3.379$ | $17.11\pm 6.141$ | Supervised
NVIDIA† [7] | $11.58\pm 7.524$ | $16.25\pm 6.336$ | $18.12\pm 9.262$ | Supervised
HIT† [12] | $14.30\pm 8.170$ | $14.75\pm 7.823$ | $17.87\pm 9.322$ | Supervised
SUSTech [15] | $23.69\pm 14.66$ | $24.62\pm 12.66$ | $23.46\pm 7.596$ | Supervised
Average | $21.87\pm 15.23$ | $23.65\pm 16.07$ | $21.66\pm 11.49$ | UDA
$13.56\pm 9.316$ | $15.30\pm 8.389$ | $17.64\pm 8.092$ | Supervised
$16.37\pm 12.27$ | $18.06\pm 12.18$ | $19.35\pm 9.587$ | All
Inter-Ob | $12.03\pm 4.443$ | $14.32\pm 5.164$ | $21.53\pm 9.460$ |

####  2.3.4 Results from MyoPS challenge event

The mean Dice scores of the scar and edema were $0.634\pm 0.225$ and $0.665\pm
0.146$ respectively, and the best Dice scores for both scar and edema were
achieved by ESTC. The results indicated that the Dice scores of edema are
higher than that of scars, maybe due to the larger extent percentage of edema.
Moreover, the segmentation precise is also related with slice position, as we
found that the accuracy of midden and basal slices are relatively higher than
that of apical slices. One possible reason is that the large variation of both
anatomical and pathological shapes on apical slices. To figure out the
correlation between the accuracy and pathology regional position, we also
generated the bulls-eye maps to visualize the segmentation accuracy per
region. The maps show that the inaccurate segmentation of scars and edema
mainly occurs at the basal and inferior regions.

###  2.4 Discussion and conclusion

Intrinsically, the aforementioned works are trying to explore and analyze the
same dataset progressively. Firstly, the inherent challenge for the multi-
sequence task is that the images obtained from different sequences or even
from the same center are generally not well-aligned. MS-CMRSeg challenge
focused on the anatomical structure segmentation, while the MyoPS challenge
was an extent of the MS-CMRSeg for the pathology segmentation. Observed from
the evaluated methods, scar and edema segmentation will be benefited from
cardiac structure information, hence the majority of methods leveraged “coarse
to fine” strategy. Note that the MS-CMRSeg and MyoPS dataset and corresponding
evaluation tool are still available for researchers upon request and
registration via the challenge homepage
https://zmiclab.github.io/index.htmlhttps://zmiclab.github.io/index.html.

##  3 LGE MRI based left atrial scar segmentation and quantification

###  3.1 Introduction

Atrial fibrillation (AF) is the most common type of arrhythmia and a severe
public health concern. The identification of fibrosis and scarring region in
left atrium (LA) is important for AF diagnosis, treatment and prognosis. Late
gadolinium enhancement magnetic resonance imaging (LGE MRI) has been shown to
be an non-invasive tool for left atrial (LA) fibrosis and scar assessment and
quantification. However, LGE MRI usually has poor image quality, mainly due to
its residual respiratory motion, variability in the heart rate and gadolinium
wash-out during the long acquisition time. Moreover, LA wall is thin (mean
thickness: $1.89\pm 0.48~{}mm$ [30]) with regional wall thickness variations.
Various shapes of LA and pulmonary veins (PV) introduce additional challenges
for scar segmentation of LGE MRI. Figure 3 visualizes the major challenges.

Most methods for scar segmentation and quantification mainly based on
thresholding [31]. Recently, with the development of deep learning (DL) in
medical image computing, several DL-based algorithms have been proposed for
automatic scar segmentation and quantification [32, 33]. One could refer to
the review papers [31, 34] for the literature of LA and scar segmentation from
LGE MRI. Next, we will introduce our two state-of-the-art framework for LA LGE
MRI computing. Our first work mainly aim to solve the challenging of thin
thickness in scar quantification, and we proposed a surface projection scheme
(see Section 3.2.1). The second work focuses on the joint optimization of LA
segmentation and scar quantification, and we designed a multi-task learning
based network (see Section 3.2.2).

###  3.2 Method

####  3.2.1 LearnGC: Atrial Scar Segmentation via Potential Learning in the
Graph-Cut Framework

Figure 4 presents the proposed LearnGC framework. First, we employ the multi-
atlas segmentation (MAS) algorithm for whole heart segmentation, and then
extract LA segmentation as an initialization. Then, we generate a surface mesh
based on the LA endocardium to perform the scar quantification on it via
graph-cuts algorithm. The edge weights of graph-cuts are predicted by the
proposed multi-scale convolution network (MS-CNN). Note that the scar
quantification is performed on the generated surface mesh, to avoid the
challenging segmentation task, i.e., thin LA wall segmentation. Moreover, it
could be effective to reduce the computational cost.

Figure 3: The challenges of automatic scar quantification from LGE MRI: (a)
various LA and pulmonary vein (PV) shapes; (b) two typical LGE MRIs with poor
quality; (c) thin atrial walls highlighted using bright white color; (d)
surrounding enhanced regions pointed out by the arrows, where (1) and (2)
indicate the enhanced walls of descending and ascending aorta, respectively;
and (3) denotes the enhanced walls of right atrium. Images adopted from Li et
al. [34].

LA segmentation via MAS. As LGE MRI covers the whole heart, we propose to
employ multi-atlas whole heart segmentation (MA-WHS) to obtain the geometrical
information of the LA. MAS algorithm can be separated into two steps:
registration between atlases and the target image; label fusion of transformed
atlases. Considering the poor image quality of LGE MRI, we employ an non-
enhanced MRI from the same patient to assist its segmentation. The non-
enhanced MRI normally has higher quality, and shares the anatomical structures
with LGE MRI. With the WHS results, we employ marching cube algorithm to
obtain a surface mesh of the LA endocardium.

Projection and graph formulation. In clinic, the location and extent of scars
are the main concern. Inspired by this, we project the LA endocardium onto a
surface mesh, and then performed the scar quantification on the surface via
graph-cuts. Moreover, the projection mitigates the effect of LA wall thickness
and inaccurate LA segmentation, and also reduces dramatically the
computational complexity. The projection is equidistant to preserve the
geodesic distances between two nodes in the graph-cuts framework. Instead of
using single pixel on the surface, we extract a profile via multi-scale patch
to incorporate both global and local texture information of scars.

The scar quantification is formulated as an energy minimization problem in the
graph-cuts framework. The edge weights of graph include the regional term
$E_{R}$ and the boundary term $E_{B}$, which encodes the intensity
distributions of two classes and ensures the continuity between neighbors. One
can denote the graph as $G=\\{\mathcal{X},\mathcal{N}\\}$, where
$\mathcal{X}=\\{x_{i}\\}$ is the set of graph nodes and
$\mathcal{N}=\\{<x_{i},x_{j}>\\}$ indicates the set of edges connecting graph
nodes. Each graph has two terminals, which denote the scars (foreground) and
normal myocardium (background), respectively. There are two kinds of edges,
i.e., t-link that connects graph nodes to the terminals and n-link connecting
neighboring nodes [35]. Therefore, the segmentation energy can be defined as
follows,

$\begin{array}[]{l@{\ }l}E(l)&=E_{R}(l)+\lambda E_{B}(l)\\\
&=\displaystyle\sum_{x_{i}\in\mathcal{X}}W_{x_{i}}^{\textit{t-link}}(l_{x_{i}})+\lambda\sum_{(x_{i},x_{j})\in\mathcal{N}}W_{\\{x_{i},x_{j}\\}}^{\textit{n-link}}(l_{x_{i}},l_{x_{j}}),\end{array}$
(4)

where $W_{x_{i}}^{\textit{t-link}}$ and
$W_{\\{x_{i},x_{j}\\}}^{\textit{n-link}}$ are the t-link and n-link weight,
respectively; $l_{x_{i}}\in\\{0,1\\}$ is the label value of $x_{i}$; and
$\lambda$ is a balancing parameter. Different from conventional graph-based
segmentation, we directly predict the t/n-link weights for the regional and
boundary terms. In this way, we can represent each graph node using a multi-
scale patch (MSP), and then learn the weighs using the proposed MS-CNN. These
patches are defined along the normal direction of the LA endocardial surface
with an elongate shape. We further adjust the sample spacing to generate MSPs
on the LGE MRI with different resolutions.

Edge weight prediction via MS-CNN. To predict the edge weights in Equation 4,
we design two networks, i.e., $T$-NET and $N$-NET. $T$-NET is designed to
predict the t-link weights, i.e., the probabilities of a node belonging to
scarring and normal regions respectively; $N$-NET aims to calculate the n-link
weights, defined based on the similarity of two neighbour nodes and their
distance. To embed the MSP into the networks, we adopt parallel convolutional
pathways for multi-scale training, namely MS-CNN. In the training phase of MS-
CNN, we adopt a weight sampling strategy to mitigate the class imbalance
problem. Specifically, instead of extracting the patches of all nodes for
training, we randomly select the similar number of nodes from the normal wall
and scars. Besides, we use a random shift strategy when extracting the MSPs to
mitigate the effect of inaccurate LA segmentation. Note that the weight
sampling and random shift strategy are not required in the testing phase.
After obtain the weights, one can obtain the scar quantification result by
optimizing the graph-cuts framework.

Figure 4: The proposed LearnGC framework for the scar quantification. The
images modified from Li et al. [32, 33].

####  3.2.2 AtrialJSQnet: A New Framework for Joint Segmentation and
Quantification of Left Atrium and Scars Incorporating Spatial and Shape
Information

Figure 5 presents the overview of the proposed joint segmentation and
quantification framework, i.e., AtrialJSQnet. One can see that AtrialJSQnet is
a two-task network consisting of two decoders for LA segmentation and scar
quantification, respectively. We proposed a spatial encoding (SE) loss as to
learn the spatial information of the LA cavity and scars. Moreover, we employ
a shape attention (SA) scheme to utilize the spatial relationship between LA
and scar. The SA scheme is also helpful to achieve an end-to-end scar
projection.

Spatially encoded constraint in the AtrialJSQnet framework. We introduce a
novel SE loss, which incorporates spatial information in the pipeline without
any modifications of networks. The SE loss is designed based on the distance
transform maps (DTM), a continuous representation of the target label. The
signed DTM can be defined as,

$\phi(x_{i})=\begin{cases}-d^{\beta}&x_{i}\in\Omega_{in}\\\ 0&x_{i}\in S\\\
d^{\beta}&x_{i}\in\Omega_{out}\end{cases}$ (5)

where $\Omega_{in}$ and $\Omega_{out}$ indicate the region inside and outside
the target label, respectively; $S$ denotes the surface boundary, $d$
represents the distance from pixel $x_{i}$ to the nearest point on $S$, and
$\beta$ is a hyperparameter.

For LA segmentation, the SE loss is defined as,

$\mathcal{L}_{LA}^{SE}=\sum_{i=1}^{N}(\hat{y}(x_{i};\theta)-T_{LA})\cdot\phi(x_{i}),$
(6)

where $\hat{y}$ and $y$ ($y\in\\{0,1\\}$) are the prediction of LA and its
ground truth, respectively; $N$ is the number of pixels, $T_{LA}$ is the
threshold for LA segmentation, and $\cdot$ denotes the dot product. One can
see the main idea of $\mathcal{L}_{LA}^{SE}$ is to assign different penalties
to false classification of each pixel based on its distance to the target
boundary, i.e., the DTM value. The final loss for LA segmentation can be
defined as,

$\mathcal{L}_{LA}=\mathcal{L}_{LA}^{BCE}+\lambda_{LA}\mathcal{L}_{LA}^{SE},$
(7)

where $\mathcal{L}_{LA}^{BCE}$ is the conventional binary cross entropy (BCE)
loss, and $\lambda_{LA}$ is a balancing parameter.

For scar quantification, we first obtain the DTM of scars and normal wall, and
then calculate the corresponding probability maps based on DTMs, i.e.,
$p=exp^{-|\phi(x)|}$ and $p=[p_{normal},p_{scar}]$. Therefore, the SE loss for
scar quantification is defined as,

$\mathcal{L}_{scar}^{SE}=\sum_{i=1}^{N}\|\hat{p}(x_{i};\theta)-p(x_{i})\|^{2}_{2},$
(8)

where $\hat{p}$ ($\hat{p}=[\hat{p}_{normal},\hat{p}_{scar}]$) is the predicted
probability maps of normal wall and scars. Note that here we did not consider
the probability map of background (pixels not belong to scars neither normal
wall), as we quantify the scars on the LA surface. However, the predicted LA
surface can be inaccurate, so we did not employ a fixed threshold on
$\hat{p}_{normal}$ or $\hat{p}_{scar}$. Instead, we propose to compare the
probabilities of each pixel belonging to scars and normal wall for final scar
quantification results.

Figure 5: The proposed AtrialJSQnet framework for the simultaneous LA
segmentation, scar projection onto LA surface, and scar quantification. The
images modified from Li et al. [33].

End-to-end trainable shape attention via multi-task learning. For joint
optimization, we develop a multi-task network where segmentation and
quantification of LGE MRI are simultaneously achieved. To further learn the
spatial relationship between LA and scars, we introduce an attention mask,
which is represented by the LA boundary from the predicted LA segmentation.
The attention mask not only can alleviate the class imbalance problem, but
also contributes to the end-to-end surface projection. To achieve this, we
design a SA loss which is defined as follows,

$\mathcal{L}^{SA}_{scar}=\sum_{i=1}^{N}(M\cdot(\nabla\hat{p}(x_{i};\theta)-\nabla
p(x_{i})))^{2},$ (9)

where $\nabla\hat{p}=\hat{p}_{normal}-\hat{p}_{scar}$, $\nabla
p=p_{normal}-p_{scar}$, and $M=\left\\{M_{1},M_{2}\right\\}$ is the attention
mask. Here, $M_{1}$ and $M_{2}$ refer to the gold standard LA wall
segmentation and the predicted LA boundary, respectively.

Therefore, we can combine these loss functions for the final optimization of
AtrialJSQnet as follows,

$\mathcal{L}=\mathcal{L}_{LA}+\lambda_{scar}\mathcal{L}_{scar}^{SE}+\lambda_{M_{1}}\mathcal{L}^{SA}_{scarM_{1}}+\lambda_{M_{2}}\mathcal{L}^{SA}_{scarM_{2}},$
(10)

where $\lambda_{scar}$, $\lambda_{M_{1}}$ and $\lambda_{M_{2}}$ are balancing
parameters.

###  3.3 Data and results

####  3.3.1 Data acquisition

For the proposed methods, we evaluated them via two different dataset,
separately. For LearnGC, we employed 58 post-ablation LGE MRIs from
longstanding persistent AF patients. All images were acquired from 1.5T
Siemens Magnetom Avanto scanner (Siemens Medical Systems, Erlangen, Germany),
using an inversion prepared segmented gradient echo sequence (TE/TR 2.2/5.2
ms), and were acquired 15 min after gadolinium administration. The acquisition
resolution is (1.4-1.5) $\\!\times\\!$ (1.4-1.5) $\\!\times\\!$ 4 mm, and is
reconstructed to (0.7-0.75) $\\!\times\\!$ (0.7-0.75) $\\!\times\\!$ 2 mm. For
AtrialJSQnet, we adopted a public data from the MICCAI2018 Atrial Segmentation
Challenge [36], which released 60 post-ablation LGE MRIs with manual LA
segmentation results. The public LGE MRIs were acquired from 1.5T Siemens
Avanto or 3T Siemens Vario (Siemens Medical Solutions, Erlangen, Germany),
with a resolution of 1.25 $\\!\times\\!$ 1.25 $\\!\times\\!$ 2.5 mm. All
images were acquired approximately 20–25 min after gadolinium administration,
using a 3D respiratory navigated, inversion recovery prepared gradient echo
pulse sequence (TE/TR 2.3/5.4 ms).

####  3.3.2 Gold standard and evaluation

All the images were manually segmented by a well-trained expert, which were
regarded as the gold standard for evaluation. For LA segmentation, we employed
Dice score, ASD and HD for evaluation. For scar quantification evaluation, we
firstly projected manual and (semi-) automatic scar segmentation results onto
the surface of manual LA segmentation. Then, we used the Accuracy, Dice score
and generalized Dice (GDice) score for the evaluation of scar quantification.

$\textit{Accuracy}=\frac{TP+TN}{TP+FP+FN+TN},$ (11)

where $TP$, $TN$, $FN$ and $FP$ stand for the number of true positives, true
negatives, false negatives and false positives, respectively.

$\text{GDice}=\frac{2\sum_{k=0}^{N_{k}-1}\left|{S}_{k}^{\textit{auto}}\cap{S}_{k}^{\textit{manual}}\right|}{\sum_{k=0}^{N_{k}-1}(\left|S_{k}^{\textit{auto}}\right|+\left|S_{k}^{\textit{manual}}\right|)},$
(12)

where $S_{k}^{\textit{auto}}$ and $S_{k}^{\textit{manual}}$ indicate the
segmentation results of label $k$ from the automatic method and manual
delineation on the LA surface, respectively, $N_{k}$ is the number of labels,
and $N_{k}=2$ here to represent scarring ($k=1$) and normal wall ($k=0$)
regions.

####  3.3.3 Performance of the proposed method

Here, we will present the results for testing LearnGC and AtrialJSQnet
algorithms, respectively.

LearnGC. Table 4 presents the LearnGC quantification results, which includes
the results of both comparison and ablation study. The proposed method is
LA${}_{\text{auto}}$ \+ LearnGC, where the LA segmentation was performed via
MA-WHS, the weights of the graph were predicted using MS-CNN, and the
balancing parameter $\lambda$ was set to 0.4. One can see that proposed
LearnGC method obtained the best performance compared to other comparison
methods. It indicated that the proposed method can obtain the state-of-the-art
performance, and also proved the effect of the proposed schemes, such as MS-
CNN, random shift, and graph-cuts.

Figure 7 presents the 3D visualization of scar quantification results by the
nine methods. One can see that the 3D visualization results were consistent
with the above quantitative analysis. The predicted scars based on
LA${}_{\text{M}}$ and LA${}_{\text{auto}}$ were projected onto two surfaces,
i.e., GT${}_{\text{M}}$ and GT${}_{\text{auto}}$. However, one can see that
GT${}_{\text{M}}$ and GT${}_{\text{auto}}$ have similar scar distribution.
Compared to other methods, the proposed LearnGC obtained the most accurate and
smooth scar quantification results. Figure 7 visualizes the axial view of
three representative cases. The illustration further proved that the proposed
method could obtain promising scar quantification results, though with some
minor errors. The mis-classification indicates the major challenges of
automatic scar quantification, contributed to the major errors of scar
quantification.

Table 4:  Summary of the quantitative evaluation results of scar quantification. Here, LA${}_{\text{M}}$ denotes the manual LA segmentation, while LA${}_{\text{auto}}$ refers to the automatically segmented LA using MA-WHS. 0 means that the methods did not employ random shift scheme ($\gamma$=0). MS-CNN refers to the learning based method only using the two t-link weights estimated from $T$-NET to classify scars. The asterisk (∗) in column Dice${}_{\text{scar}}$ indicates the methods performed statistically poorer ($p<0.01$) compared to the proposed LA${}_{\text{auto}}$ \+ LearnGC.  Method |  Accuracy |  Dice${}_{\text{scar}}$ |  $G$Dice
---|---|---|---
LA${}_{\text{M}}$ \+ 2SD [37] | $0.809\pm 0.074$ | $0.275\pm 0.091^{*}$ | $0.758\pm 0.098$
LA${}_{\text{M}}$ \+ Otsu [38] | $0.763\pm 0.188$ | $0.396\pm 0.090^{*}$ | $0.726\pm 0.207$
LA${}_{\text{M}}$ \+ MGMM [39] | $0.708\pm 0.160$ | $0.545\pm 0.101^{*}$ | $0.716\pm 0.190$
LA${}_{\text{M}}$ \+ MGMM + GC | $0.716\pm 0.162$ | $0.562\pm 0.102^{*}$ | $0.721\pm 0.192$
LA${}_{\text{M}}$ \+ U-Net [40] | $0.832\pm 0.046$ | $0.568\pm 0.083^{*}$ | $0.826\pm 0.052$
LA${}_{\text{M}}$ \+ MS-CNN0 | $0.798\pm 0.051$ | $0.615\pm 0.083^{*}$ | $0.811\pm 0.047$
LA${}_{\text{auto}}$ \+ MS-CNN0 | $0.806\pm 0.052$ | $0.631\pm 0.080^{*}$ | $0.814\pm 0.047$
LA${}_{\text{auto}}$ \+ MS-CNN | $0.846\pm 0.032$ | $0.692\pm 0.069^{*}$ | $0.851\pm 0.030$
LA${}_{\text{auto}}$ \+ LearnGC | $0.856\pm 0.033$ | $0.702\pm 0.071$ | $0.859\pm 0.031$

Figure 6: 3D visualization of the LA scar classification results using the
nine methods. This is the median case selected from the test set in terms of
Dice score of scars by the proposed method. The scarring areas are red-colored
on the LA surface mesh, which can be constructed either from $LA_{M}$ (LA
surface in white) or from $LA_{auto}$ (LA surface in light yellow). Image
adopted from Li et al. [32] Figure 7: Axial view of the images, the ground
truth scar segmentation and the results by the proposed method. The red and
white color labels represent the scar and normal wall, respectively. Arrow
(1), (2) and (3) indicate the major classification errors of the proposed
method caused by the surrounding enhanced regions, respectively from the right
atrium wall, ascending aorta wall and descending aorta wall; arrow (4) shows
an error from the misalignment between the automatic LA segmentation and the
ground truth; arrow (5) illustrates that the proposed method can still perform
well, even though the automatic LA segmentation contains obvious errors.

AtrialJSQnet. Table 5 and Table 6 present the LA segmentation and scar
quantification results of different methods, respectively. For the LA
segmentation, the proposed SE loss performed better than the conventional
losses, such as BCE and Dice loss. For the scar quantification, the SE loss
also obtained promising performance compared to the conventional losses in
terms of Dicescar. The three (semi-) automatic scar quantification methods
generally obtained acceptable results, but relied on an accurate
initialization of LA. LearnGC had a similar result compared to MGMM in
Dicescar, but its _Accuracy_ and $G$Dice were higher. The proposed method
performed statistically significant better than all the automatic methods in
terms of Dicescar ($p\leq 0.001$). Both the LA segmentation and scar
quantification benefited from the proposed joint optimization scheme comparing
to separately optimize the two tasks. After introducing the new SA loss, the
results were further improved in terms of Dicescar ($p\leq 0.001$), but with a
slightly worse _Accuracy_ ($p\leq 0.001$) and $G$Dice ($p>0.1$). Moreover,
with the SA loss some small and discrete scars could be detected, and an end-
to-end scar quantification and projection were achieved.

Table 5:  Summary of the quantitative evaluation results of LA segmentation. Here, U-Net${}_{\text{LA}}$ uses the original U-Net architecture for LA segmentation; BCE, SE, SA and SESA refer to the different loss functions. The proposed method is denoted as AJSQnet-SESA. Method |  Dice${}_{\text{LA}}$ | ASD (mm) | HD (mm)
---|---|---|---
U-Net${}_{\text{LA}}$-BCE | $0.889\pm 0.035$ | $2.12\pm 0.797$ | $36.4\pm 23.6$
U-Net${}_{\text{LA}}$-Dice | $0.891\pm 0.049$ | $2.14\pm 0.888$ | $35.0\pm 17.7$
U-Net${}_{\text{LA}}$-SE | $0.880\pm 0.058$ | $2.36\pm 1.49$ | $25.1\pm 11.9$
AJSQnet-BCE | $0.890\pm 0.042$ | $2.11\pm 1.01$ | $28.5\pm 14.0$
AJSQnet-SE | $0.909\pm 0.033$ | $1.69\pm 0.688$ | $22.4\pm 9.80$
AJSQnet-SESA | $0.913\pm 0.032$ | $1.60\pm 0.717$ | $20.0\pm 9.59$
Inter-ob | $0.894\pm 0.011$ | $1.81\pm 0.272$ | $17.0\pm 5.50$

Table 6:  Summary of the quantitative evaluation results of scar quantification. Here, LA${}_{\text{M}}$ denotes that scar quantification is based on the manually segmented LA, while LA${}_{\text{U\mbox{-}Net}}$ indicates that it is based on the U-Net${}_{\text{LA}}$-BCE segmentation; U-Net${}_{\text{scar}}$ is the scar segmentation directly based on the U-Net architecture with different loss functions; The inter-observer variation (Inter-Ob) is calculated from randomly selected twelve subjects.  Method |  Accuracy |  Dice${}_{\text{scar}}$ |  $G$Dice
---|---|---|---
LA${}_{\text{M}}$+Otsu [41] | $0.750\pm 0.219$ | $0.420\pm 0.106$ | $0.750\pm 0.188$
LA${}_{\text{M}}$+MGMM [39] | $0.717\pm 0.250$ | $0.499\pm 0.148$ | $0.725\pm 0.239$
LA${}_{\text{M}}$+LearnGC [32] | $0.868\pm 0.024$ | $0.481\pm 0.151$ | $0.856\pm 0.029$
LA${}_{\text{U\mbox{-}Net}}$+Otsu | $0.604\pm 0.339$ | $0.359\pm 0.106$ | $0.567\pm 0.359$
LA${}_{\text{U\mbox{-}Net}}$+MGMM | $0.579\pm 0.334$ | $0.430\pm 0.174$ | $0.556\pm 0.370$
U-Net${}_{\text{scar}}$-BCE | $0.866\pm 0.032$ | $0.357\pm 0.199$ | $0.843\pm 0.043$
U-Net${}_{\text{scar}}$-Dice | $0.881\pm 0.030$ | $0.374\pm 0.156$ | $0.854\pm 0.041$
U-Net${}_{\text{scar}}$-SE | $0.868\pm 0.026$ | $0.485\pm 0.129$ | $0.863\pm 0.026$
AJSQnet-BCE | $0.887\pm 0.023$ | $0.484\pm 0.099$ | $0.872\pm 0.024$
AJSQnet-SE | $0.882\pm 0.026$ | $0.518\pm 0.110$ | $0.871\pm 0.024$
AJSQnet-SESA | $0.867\pm 0.032$ | $0.543\pm 0.097$ | $0.868\pm 0.028$
Inter-Ob | $0.891\pm 0.017$ | $0.580\pm 0.110$ | $0.888\pm 0.022$

###  3.4 Conclusion and future work

In this sub-chapter, we present two approaches for scar quantification from
LGE MRI. The first approach combines graph-cuts and MS-CNN, referred to as
LearnGC, which integrates the multi-scale information of scars and ensures a
smooth segmentation results. More importantly, LearnGC is trying to ignore the
wall thickness and project the extracted scars onto the LA surface. Therefore,
it converted the challenging volume-based scar segmentation problem into the
relatively easy surface-based scar quantification problem. However, the pixel-
wise quantification on the surface only includes limited information, and
tends to be effected by the misalignment between the predicted endocardial
surface and the corresponding ground truth. Therefore, the proposed random
shift scheme and MS-CNN are effective to improve the robustness of the
proposed method against the LA segmentation errors. A major limitation of the
LearnGC method is the lack of an end-to-end training scheme, i.e., MS-CNN and
graph-cuts were separately achieved. We therefore proposed another approach,
i.e., AtrialJSQnet, which can simultaneously achieved LA segmentation and scar
quantification. To eliminate the effect of inaccurate LA segmentation, we
learn the spatial information of each pixel on the surface via a newly
designed SE loss. The SE loss and joint optimization were both proved to be
effective by observing our experimental results.

A limitation of the two works is that the gold standard was constructed from
the manual delineation of only one expert. Besides, the subjects included in
this study are only post-ablation AF patients. In future work, we will combine
multiple experts to construct the gold standard, and consider both pre- and
post-ablation data. Moreover, we will collect multi-center LGE MRI to explore
the generalization ability of LGE MRI segmentation and quantification models.

##  4 Domain adaptation for cross-modality cardiac image segmentation

###  4.1 Introduction

The capacity of model generalization is essential for the application of
computer-aided-diagnosis (CAD) system on cardiac image analysis. In practice,
a cardiac segmentation model trained on a specific modality could perform
poorly on images from other modalities [42]. The reason is that there exists
nonnegligible gap between the distributions of test and training data, which
is known as domain shift [43]. How to transfer the learned anatomical
knowledge from one domain to others without labeling new data is an
interesting and open problem. An important research direction is domain
adaptation, which aims to reduce the domain discrepancy between the labeled
source and unlabeled target data [44].

To date, many domain adaptation approaches have been proposed for cardiac
image segmentation. Most of them learned modality-invariant features via
adversarial training [45, 46]. For example, Dou et al.[42] designed a dilated
fully convolutional network (denoted as PnP-AdaNet), which consists of a plug-
and-play domain adaptation module to map two domains into a common space. It
adopted the training scheme of domain adversarial neural networks (DANN) [47].
The method was validated on MRI-CT cross-modality cardiac segmentation for 2D
images. While PnP-AdaNet extracted domain-invariant latent features from
middle layers and achieved promising results, Chen et al. [48] proposed to
implement domain adaptation on both feature and image levels. Their method,
referred to as SIFA, translated images between two domains, and used the
cycle-consistency loss for model constrain. This work was validated to
outperform peer methods on cardiac cross-modality segmentation. In addition,
Ouyang et al.[49] introduced a VAE-based feature prior matching to further
adapt their features, and proposed a data efficient method for multi-domain
medical image segmentation.

Although adversarial training has shown great potential in domain adaptation,
especially for image translation, there still exist drawbacks which degrade
its effectiveness and efficiency. First, the extracted domain-invariant
features (DIFs) may not be pure. They could contain specific domain
information and lead to biased results [50]. For image translation, no
attention has been paied on domain-specific features (DSFs), which might be
useful to improve the quality of the reconstructed images. Second, adversarial
training reduces the discrepancy implicitly. It suffers from problems
originated from the generative adversarial network (GAN) [51], such as the
extra discriminators, complex training process and difficulty of obtaining the
Nash equilibrium point [52]. While many explicit measurements for distribution
discrepancy have been designed, such as the Maximum Mean Discrepancy (MMD)
[53] and the moment distance [54], all of them were proposed for
classification tasks, and no work has been validated for segmentation, to the
best of our knowledge. What kind of explicit metric is efficient for cross-
modality segmentation remains an important and open problem.

To tackle these issues, we proposed three domain adaptation frameworks for
cardiac segmentation, which are termed as DDFSeg [55], CFDNet [56] and VarDA
[57], respectively. Specifically, we first studied feature disentanglement for
domain adaptation, and constrained DIFs and DSFs by introducing self-attention
and zero-loss. It uses adversarial training for model optimization, thus can
be categorized into Type I, as Figure 8 (a) illustrated. Next, we studied the
effectiveness of explicit discrepancy metrics for domain adaptation. A new
metric based on the distance between characteristic functions is proposed, and
it was validated to be effective in cardiac segmentation tasks. This metric is
denoted as CF distance, and its minimization leads to the reduction of domain
discrepancy and the extraction of domain-invariant features. This method
avoids adversarial training, and has a simpler training process and faster
model convergence. As Figure 8 (a) illustrated, it can be classified as type
II approaches. For both type I and II methods, the domains were mapped into a
common latent feature variable $z$. The domain discrepancy was then either
reduced by adversarial training or explicitly minimization of discrepancy
metrics. While type II methods were validated to be useful for cross-modality
cardiac segmentation, especially using the proposed CFDNet, we found that the
computation of these metrics are complex. In practice, we calculate these
metrics with marginal distributions instead of joint ones. This substitution
weakens the constraint for domain-invariant features. Based on this
consideration, we further proposed another type of methods. As illustrated in
Figure 8 (b), method of Type III drives two domains towards a common
parameterized distribution, i.e., $q_{\phi}(z)$, in a latent feature $z$. As
$q_{\phi}$ can be set to independent among its element variables, such as
Gaussian distributions, the effect of the aforementioned substitution for
metric calculation could be alleviated significantly. We achieve this
approximation using variational auto-encoders, and thus denote the proposed
framework as VarDA. In the following, we will describe the three methods in
detail, and present their performances on the cardiac segmentation.

(a) The framework of previous researches of domain adaptation in a latent
feature space.

(b) The idea of the proposed variational approximation method for domain
adaptation.

Figure 8: Illustration of the difference among three types of methods for
domain adaptation. The proposed three frameworks refer to the three types,
respectively. Images adopted from Wu et al. [57]

###  4.2 Method

Let $X_{S}=\\{x_{S}^{i}\\}_{i=1}^{N_{S}}$ be the set of labeled source data,
which are independent and identically distributed ($i.i.d.$) samples collected
from source domain $x_{S}$ with distribution $p_{\theta_{S}}(x)$, and
$\theta_{S}$ is model parameter. $Y_{S}=\\{y_{S}^{i}\\}_{i=1}^{N_{S}}$ denotes
the corresponding label of $X_{S}$. With $X_{S}$ and $Y_{S}$, we can learn a
segmentation model for this source domain. We further collect target data
samples from target domain $x_{T}$, denoted by
$X_{T}=\\{x_{T}^{i}\\}_{i=1}^{N_{T}}$, from a different distributions
$p_{\theta_{T}}(x)$. The goal of domain adaptation is to transfer the
knowledge from the source data, and train a segmentation model for the target
data.

####  4.2.1 DDFSeg: Disentangle Domain Features for Domain Adaptation and
Segmentation

We first study domain adaptation via image translation. Taking DSFs into
account, we propose a new framework. As Figure 9 illustrated, we use four
encoders, i.e., $E^{styl}_{S}$ and $E^{cont}_{S}$ for source domain,
$E^{styl}_{T}$ and $E^{cont}_{T}$ for target domain, to disentangle each
domain into their DIFs and DSFs. These features are denoted as
$z^{c}_{d}=E^{c}_{d}(x_{d})$, where $c\in\\{styl,cont\\}$ and $d\in\\{S,T\\}$.
These features are then swapped and decoded into images with the anatomical
structures maintained and the style exchanged, using two decoders $D_{S}$ and
$D_{T}$. Mathematically, the new generated images can be expressed as
$x_{S}^{fake}=D_{S}(z^{styl}_{S},z^{cont}_{T})$, and
$x_{T}^{fake}=D_{T}(z^{styl}_{T},z^{cont}_{S})$. $x_{S}^{fake}$ and
$x_{T}^{fake}$ are further encoded and decoded again for image reconstruction,
which can be seen as a modified version of CycleGAN [58]. The reconstructed
images are denoted as $x_{s}^{recon}$ and $x_{t}^{recon}$, with
$x_{S}^{recon}=D_{S}(E_{S}^{styl}(x_{S}^{fake}),E_{T}^{cont}(x_{T}^{fake}))$
and
$x_{T}^{recon}=D_{T}(E_{T}^{styl}(x_{T}^{fake}),E_{S}^{cont}(x_{S}^{fake}))$.
To enhance the DIFs and DSFs, we introduce the techniques of self-attention
and zero-loss . In addition, extra discriminator is used to constrain the
anatomical shape of segmentation output. Hence, the total loss function
consists of three parts, i.e., image translation loss, zero-loss, and
segmentation loss.

Figure 9: (a) Overview of the framework. (b) Illustration of different domain
images translation. (c) Illustration of the segment with target domain image
at test time. Images adopted from Pei et al. [55]

For image translation, we adopt cycle consistency loss to achieve image
reconstruction, and use several discriminators to force the generated images
to be as real as possible. The cycle consistency loss is defined as follows,

$\displaystyle\mathcal{L}_{cyc}\left(E_{S}^{cont},E_{S}^{styl},E_{T}^{cont},E_{T}^{styl},D_{S},D_{T}\right)$
(13) $\displaystyle=$ $\displaystyle\mathbb{E}_{x_{S},x_{T}\sim
P\left(X_{S},X_{T}\right)}\left\|x_{S}^{recon}-x_{S}\right\|_{1}+\mathbb{E}_{x_{S},x_{T}\sim
P\left(X_{S},X_{T}\right)}\left\|x_{T}^{recon}-x_{T}\right\|_{1}.$

To force $x_{S}^{fake}$ and $x_{T}^{fake}$ to be real, we introduce two
discriminators $Dis_{S}$ and $Dis_{T}$ for source and target domains,
respectively. The objective functions for adversarial training can be
formulated as follows,

$\displaystyle\min_{\left(E_{S}^{cont},E_{T}^{styl},D_{T}\right)}\max_{Dis_{T}}\mathcal{L}_{adv}^{T}=$
$\displaystyle\mathbb{E}_{x_{T}\sim P\left(X_{T}\right)}\left[\log
Dis_{T}\left(x_{T}\right)\right]+$ (14)
$\displaystyle\mathbb{E}_{x_{S},x_{T}\sim
P\left(X_{S},X_{T}\right)}\left[\log\left(1-Dis_{T}\left(x_{T}^{fake}\right)\right)\right],$

$\displaystyle\min_{\left(E_{T}^{cont},E_{S}^{styl},D_{S}\right)}\max_{Dis_{S}}\mathcal{L}_{adv}^{S}=$
$\displaystyle\mathbb{E}_{x_{S}\sim P\left(X_{S}\right)}\left[\log
Dis_{S}\left(x_{S}\right)\right]+$ (15)
$\displaystyle\mathbb{E}_{x_{S},x_{T}\sim
P\left(X_{S},X_{T}\right)}\left[\log\left(1-Dis_{S}\left(x_{S}^{fake}\right)\right)\right].$

Moreover, to further enhance the extracted DIFs to be domain-invariant, we add
an auxiliary task to the source discriminator $Dis_{S}$ to differentiate
$x_{s}^{fake}$ and $x_{s}^{recon}$. The objective function is defined as
follows,

$\displaystyle\min_{E_{T}^{cont}}\max_{Dis_{S}}\mathcal{L}_{adv}^{S.aux}=$
$\displaystyle\mathbb{E}_{x_{S}^{recon}\sim
P\left(x_{S}^{recon}\right)}\left[\log
Dis_{S}\left(x_{S}^{recon}\right)\right]+$ (16)
$\displaystyle\mathbb{E}_{x_{S}^{fake}\sim
P\left(x_{S}^{fake}\right)}\left[\log\left(1-Dis_{S}\left(x_{S}^{fake}\right)\right)\right].$

The zero-loss is used to to constrain the encoders $E_{S}^{styl}$ and
$E_{T}^{styl}$ to force the extracted information from target and source
images to be zero. Hence, the losses can be formulated as follows,

$\displaystyle\mathcal{L}_{\text{zero}}^{T}\left(E_{T}^{styl}\right)=\mathbb{E}_{x_{S}\sim
P\left(X_{S}\right)}\left[\left\|E_{T}^{styl}\left(x_{S}\right)\right\|_{1}\right].$
(17)

and

$\displaystyle\mathcal{L}_{\text{zero}}^{S}\left(E_{S}^{styl}\right)=\mathbb{E}_{x_{T}\sim
P\left(X_{T}\right)}\left[\left\|E_{S}^{styl}\left(x_{T}\right)\right\|_{1}\right].$
(18)

Combining (17) (18), we have the total zero-loss,

$\displaystyle\mathcal{L}_{\text{zero}}=\mathcal{L}_{\text{zero}}^{T}+\mathcal{L}_{\text{zero}}^{S}.$
(19)

The third types of losses is designed for semantic segmentation. The
segmentation module predicts the labels from the latent features
$z^{cont}_{S}$ and $z^{cont}_{T}$. The first segmentation loss can be
formulated as follows,

$\displaystyle\mathcal{L}_{seg}^{x_{S}}\left(E_{S}^{cont},\mathcal{S}_{seg}\right)=\mathbb{E}_{x_{S},y_{S}\sim
P\left(X_{S},Y_{S}\right)}\left[\mathcal{C}\left(y_{S},\widehat{y}_{S}\right)+\alpha\cdot
Dice\left(y_{S},\widehat{y}_{S}\right)\right],$ (20)

where$\widehat{y}_{S}$ is the prediction of $x_{S}$,
$\widehat{y}_{S}=\mathcal{S}_{seg}(z_{s}^{cont})$,
$\mathcal{C}\left(y_{S},\widehat{y}_{S}\right)$ is the cross-entropy loss,
$Dice\left(y_{S},\widehat{y}_{S}\right)$ the Dice loss, and $\alpha$ is the
hyper-parameter.

We further use the generated images $x_{t}^{fake}$, which contains the same
anatomical information as $x_{S}$, to train the segmentation module. The loss
is then formulated as follows,

$\displaystyle\mathcal{L}_{seg}^{x_{T}^{fake}}\left(E_{T}^{cont},\mathcal{S}_{seg}\right)=\mathbb{E}_{x_{T}^{fake},y_{S}\sim
P\left(X_{T}^{fake},Y_{S}\right)}\left[\mathcal{C}\left(y_{S},\widehat{y}_{S}^{fake}\right)+\alpha\cdot
Dice\left(y_{S},\widehat{y}_{S}^{fake}\right)\right],$ (21)

where $\widehat{y}_{S}^{fake}$ is the prediction of $x_{T}^{fake}$.

We have the total segmentation loss as follows,

$\displaystyle\mathcal{L}_{seg}=\mathcal{L}_{seg}^{x_{S}}+\mathcal{L}_{seg}^{x_{T}^{fake}}.$
(22)

In addition, we introduce another discriminator $Dis^{seg}$ to constrain the
shape of segmentation output of target images to be similar to that of source
images. The objective function is defined as follows,

$\displaystyle\min_{\left(E_{T}^{cont},\mathcal{S}_{seg}\right)}\max_{Dis^{seg}}\mathcal{L}_{adv}^{seg}=$
$\displaystyle\mathbb{E}_{x_{T}^{fake}\sim
P\left(x_{T}^{fake}\right)}\left[\log
Dis^{seg}\left(\mathcal{S}_{seg}\left(E_{T}^{cont}\left(x_{T}^{fake}\right)\right)\right)\right]+$
(23) $\displaystyle\mathbb{E}_{x_{T}\sim
P\left(X_{T}\right)}\left[\log\left(1-Dis^{seg}\left(\mathcal{S}_{seg}\left(E_{T}^{cont}\left(x_{T}\right)\right)\right)\right)\right].$

Combining all the aforementioned losses, we have the total loss as follows,

$\displaystyle\mathcal{L}=$
$\displaystyle\lambda_{1}\mathcal{L}_{adv}^{t}\left(E_{s}^{cont},E_{t}^{styl},D_{t},Dis_{t}\right)+$
(24)
$\displaystyle\lambda_{2}\mathcal{L}_{adv}^{s}\left(E_{t}^{cont},E_{s}^{styl},D_{s},Dis_{s}\right)+$
$\displaystyle\lambda_{3}\mathcal{L}_{cyc}\left(E_{s}^{cont},E_{s}^{styl},E_{t}^{cont},E_{t}^{styl},D_{s},D_{t}\right)+$
$\displaystyle\lambda_{4}\mathcal{L}_{adv}^{s.aux}\left(E_{t}^{cont},Dis_{s}\right)+\lambda_{5}\mathcal{L}_{\text{zero}}\left(E_{t}^{styl},E_{s}^{styl}\right)+$
$\displaystyle\lambda_{6}\mathcal{L}_{seg}\left(E_{s}^{cont},E_{t}^{cont},S\right)+\lambda_{7}\mathcal{L}_{adv}^{seg}\left(E_{t}^{cont},S,Dis^{seg}\right).$

In experiments, we set $\lambda_{adv}^{1}=1.0$, $\lambda_{adv}^{2}=1.0$,
$\lambda_{3}=1.0$, $\lambda_{4}=0.1$, $\lambda_{5}=0.01$, $\lambda_{6}=0.1$,
and $\lambda_{7}=0.1$.

####  4.2.2 CFDNet: Characteristic Function Distance for Unsupervised Domain
Adaptation

Beside the adversarial training, we further study the effectiveness of
explicit metric for domain adaptation. We propose a new metric which measures
the distance between the characteristic functions of the latent features from
source and target domains. Based on this CF distance, we propose a framework
for cardiac segmentation, denoted as CFDNet. Figure 10 illustrated the whole
framework. The encoder extracts the latent features $z_{S}\in\mathbb{R}^{n}$
and $z_{T}\in\mathbb{R}^{n}$ respectively from the source and target data. The
segmentor module outputs the segmentation results from the latent features.
The reconstructor module reconstructs the target images. The prior matching
module regularizes the prior distributions of $z_{S}$ and $z_{T}$ to be close
to $\mathcal{N}(0,I)$. The explicit adaptation module computes the domain
discrepancy explicitly. Next we describe each module in detail.

Figure 10: Framework of the proposed domain adaptation method for medical
image segmentation. Images adopted from Wu et al. [56]

Calculating the discrepancy between the distributions of two domains in the
latent feature space, i.e., $p_{z_{S}}(z)$ and $p_{z_{T}}(z)$, is difficult,
because they are unknown. We instead estimate the distance of their CFs,
$f_{z_{S}}(\vec{t})$ and $f_{z_{T}}(\vec{t})$. The loss function is given by,

$\mathcal{D}_{CF}(z_{S},z_{T};U)=\int_{-U}^{U}||f_{z_{S}}(\vec{t})-f_{z_{T}}(\vec{t})||^{2}dt.$
(25)

where $U\in\mathbb{R_{+}}^{n}$.

In practice, we solely compute the loss for a single point
$U_{a}=[a,\cdots,a]\in\mathbb{R_{+}}^{n}$ using the mini-batch of samples. The
CF distance is then estimated by,

$\displaystyle\mathcal{D}_{CF}(z_{S},z_{T};U_{a})\approx$
$\displaystyle\frac{1}{M^{2}}\sum_{p=1}^{M}\sum_{q=1}^{M}k(z_{S}^{p},z_{S}^{q})+$
$\displaystyle\frac{1}{M^{2}}\sum_{p=1}^{M}\sum_{q=1}^{M}k(z_{T}^{p},z_{T}^{q})-\frac{2}{M^{2}}\sum_{p=1}^{M}\sum_{q=1}^{M}k(z_{S}^{p},z_{T}^{q}),$
(26)

where
$k(z_{S},z_{T})$=$\prod_{k=1}^{n}\\!\\!\frac{2\sin[(z_{S_{k}}-z_{T_{k}})a]}{z_{S_{k}}-z_{T_{k}}}$,
for $\forall z_{S},z_{T}\in\mathbb{R}^{n}$; $z_{S_{k}}$ is the $k$-th element
of $z_{S}$, and likewise for $z_{T_{k}}$; $M$ is the number of samples.

To simplify the calculation, we adopt a sliced version of CFD distance as a
substitution, which is defined as follows,

$\displaystyle\mathcal{L}_{SCF}(z_{S},z_{T};U)\approx\frac{1}{n}\sum_{i=1}^{n}\mathcal{D}_{CF}(z_{S_{i}},z_{T_{i}};U).$
(27)

We further introduce the mean value matching to enforce domain adaptation. The
mean loss is defined as follows,

$\displaystyle\mathcal{L}_{mean}=\parallel\mathbb{E}_{p_{z_{S}}(z)}(z)-\mathbb{E}_{p_{z_{S}}(z)}(z)\parallel^{2}.$
(28)

Combining the sliced CF distance with the mean loss, we have the explicit
domain discrepancy loss,

$\displaystyle\mathcal{L}_{explicit}=\mathcal{L}_{CFD}=\beta_{1}\mathcal{L}_{SCF}+\beta_{2}\mathcal{L}_{mean},$
(29)

where $\beta_{1}$ and $\beta_{2}$ are hyperparameters.

Moreover, we introduce the technique of prior matching. We use a variational
auto-encoder to map two domains into a common latent space $z$, with posterior
distributions being subject to
$q_{\phi_{S/T}}(z|x)=\mathcal{N}(u_{S/T},\Sigma_{S/T})$, where
$u_{S/T}=(u_{S/T}^{1},\cdots,u_{S/T}^{n})\in\mathbb{R}^{n}$,
$\Sigma_{S/T}={\rm diag}(\lambda_{S/T}^{1},$
$\cdots,\lambda_{S/T}^{n})\in\mathbb{R}^{n\times n}$. The prior matching loss
is fornulated as follows,

$\displaystyle\mathcal{L}_{prior}=E_{p_{x_{S}}(x)}[KL(q_{\phi_{S}}(z|x)||\mathcal{N}(\bm{0},I))]+E_{p_{x_{T}}(x)}[KL(q_{\phi_{T}}(z|x)||\mathcal{N}(\bm{0},I))],$
(30)

where $KL(\cdot||\cdot)$ denotes the Kullback-Leibler ($KL$) divergence, and
$\bm{0}\in\mathbb{R}^{n}$ is a zero vector.

To constrain the feature of target images, we add a reconstruction loss,
denoted as $\mathcal{L}_{recon}$. We use the cross entropy loss for the
segmentation loss from the labeled source domain, and denote it as
$\mathcal{L}_{seg}(\widehat{y}_{S},y_{S})$. Then, the total loss of the
proposed CFDNet is formulated as follows,

$\mathcal{L}=\alpha_{1}\mathcal{L}_{seg}+\alpha_{2}\mathcal{L}_{prior}+\alpha_{3}\mathcal{L}_{recon}+\alpha_{4}\mathcal{L}_{explicit},$
(31)

where $\alpha_{1}$, $\alpha_{2}$, $\alpha_{3}$ and $\alpha_{4}$ are the
parameters.

####  4.2.3 VarDA: Domain Adaptation via Variational Approximation

Although the proposed CF distance is validated to be effective for domain
adaptation, the substitution of its calculation using marginal distributions
leads to a weaker constraint on the features. Moreover, the prior matching
technique is not very useful as expected, due to the two-step sampling for
estimation of the CF distance. Based on these observation, we further proposed
another domain adaptation framework, which drives two domains towards a common
parameterized distribution via variational approximation.

As Figure 11 illustrated, the proposed VarDA framework consists of three
modules, i.e., two VAE for each domains, and a module to compute the
regularization term on domain discrepancy. The objective functions of the two
VAEs are denoted as $LB_{VAE}(\theta_{S},\phi_{S})$ and
$LB_{VAE}(\theta_{T},\phi_{T})$, respectively. The regularization term for
$q_{\phi_{S}}(z)$ and $q_{\phi_{T}}(z)$ is denoted as
$Loss_{discrepancy}(q_{\phi_{S}},q_{\phi_{T}})$. The total loss function of
VarDA is then formulated by,

$\begin{array}[]{l@{\
}l}Full\,Loss(\omega)=&-\alpha_{1}LB_{VAE}(\theta_{S},\phi_{S})-\alpha_{2}LB_{VAE}(\theta_{T},\phi_{T})\\\
&+\alpha_{3}Loss_{discrepancy}(q_{\phi_{S}},q_{\phi_{T}}),\end{array}$ (32)

where $\omega=(\theta_{S},\phi_{S},\theta_{T},\phi_{T})$ are the parameters to
be optimized, and $\alpha_{1}$, $\alpha_{2}$, $\alpha_{3}$ are the trade-off
parameters.

Figure 11: Framework of the proposed VarDA. Image adopted from Wu et al. [57]

Similar to the original VAE, we have the following objective function for the
source domain,

$\displaystyle
LB_{VAE}(\theta_{S},\phi_{S})=-D_{KL}(q_{\phi_{S}}(z|x)\parallel
p_{\theta_{S}}(z))\quad$ $\displaystyle+E_{\log
q_{\phi_{S}}(z|x)}[p_{\theta_{S}}(x|y,z)]+E_{q_{\phi_{S}}(z|x)}[\log
p_{\theta_{S}}(y|z)],$ (33)

where $D_{KL}(q||p)$ is the $KL$ divergence of $q$ and $p$. The second term
$E_{q_{\phi_{S}}(z|x)}[\log$ $p_{\theta_{S}}(x|y,z)]$ can be modeled by the
image reconstruction. The third term $E_{q_{\phi_{S}}(z|x)}[\log$
$p_{\theta_{S}}(y|z)]$ is modeled by the segmentor.

We estimate $LB_{VAE}(\theta_{S},\phi_{S})$ using mini-batch samples as
follows,

$\displaystyle\widetilde{\mathcal{L}}_{S}(\theta_{S},$
$\displaystyle\phi_{S};x^{i},y^{i})=-D_{KL}(q_{\phi_{S}}(z^{i}|x^{i})\parallel
p_{\theta_{S}}(z^{i}))$ $\displaystyle+\frac{1}{L}\sum_{l=1}^{L}\big{[}\log
p_{\theta_{S}}(x^{i}|y^{i},z^{(i,l)})+\log
p_{\theta_{S}}(y^{i}|z^{(i,l)})\big{]},$ (34)

where $z^{(i,l)}=g_{\phi_{S}}(\epsilon^{(i,l)},x^{i})$, with
$\epsilon^{(i,l)}\sim p(\epsilon)$, $L$ is the number of samples, and
$z=g_{\phi_{S}}(\epsilon,x)$ is a differentiable transformation with
$\epsilon\sim p(\epsilon)$.

Similarly, we have the variational lower bound for the target domain as
follows,

$\displaystyle
LB_{VAE}(\theta_{T},\phi_{T})=-D_{KL}(q_{\phi_{T}}(z|x)\parallel
p_{\theta_{T}}(z))$ $\displaystyle+E_{q_{\phi_{T}}(z|x)}[\log
p_{\theta_{T}}(x|\widehat{y},z)]+E_{q_{\phi_{T}}(z|x)}[\log
p_{\theta_{T}}(\widehat{y}|z)].$ (35)

Finally, we force the approximations $q_{\phi_{S}}(z)$ and $q_{\phi_{T}}(z)$
to be the same one, and thus have the regularization term using $l_{2}$ norm
of their distance as follows,

$\displaystyle D$
$\displaystyle(\phi_{S},\phi_{T})=\int[q_{\phi_{S}}(z)-q_{\phi_{T}}(z)]^{2}{\rm
d}z$
$\displaystyle\approx\int\Big{[}\frac{1}{M}\sum_{i=1}^{M}q_{\phi_{S}}(z|x_{S}^{i})-\frac{1}{M}\sum_{j=1}^{M}q_{\phi_{T}}(z|x_{T}^{i})\Big{]}^{2}{\rm
d}z$
$\displaystyle=\frac{1}{M^{2}}\sum_{i=1}^{M}\sum_{j=1}^{M}\left[k(x_{S}^{i},x_{S}^{j})+k(x_{T}^{i},x_{T}^{j})-2k(x_{S}^{i},x_{T}^{j})\right],$
(36)

where $k(x_{S}^{i},x_{T}^{j})=\int q_{\phi_{S}}(z|x_{S}^{i})\cdot
q_{\phi_{T}}(z|x_{T}^{j}){\rm d}z$. Let $q_{\phi_{S}}(z|x_{S}^{i})$ and
$q_{\phi_{T}}(z|x_{T}^{j})$ subject to $N(u_{S}^{i},\Sigma_{S}^{i})$ or
$N(u_{T}^{j},\Sigma_{T}^{j})$, one can obtain that

$k(x_{S}^{i},x_{T}^{j})=\frac{e^{-\frac{1}{2}\sum_{l=1}^{n}\frac{(u_{S_{l}}^{i}-u_{T_{l}}^{j})^{2}}{\lambda_{S_{l}}^{i}+\lambda_{T_{l}}^{j}}}}{(2\pi)^{\frac{n}{2}}\cdot(\prod_{l=1}^{n}(\lambda_{S_{l}}^{i}+\lambda_{T_{l}}^{j}))^{\frac{1}{2}}},$
(37)

where $u_{S_{l}}^{i}$ is the $l$-th element of $u_{S}^{i}$.

As the computation for this regularization term is complex, similar to the
sliced CF distance, we use the marginal distributions of $z_{S}$ and $z_{T}$
to calculate the distance. The substitution is as follows,

$\displaystyle\widetilde{\mathcal{D}}(\phi_{S},\phi_{T})=\sum_{i=1}^{n}\int[q_{\phi_{S}}(z_{i})-q_{\phi_{T}}(z_{i})]^{2}{\rm
d}z.$ (38)

Based on the three loss functions discussed above, we have the total loss of
VarDA as follows,

$\displaystyle\widetilde{\mathcal{H}}(\omega)=$
$\displaystyle-\alpha_{1}\cdot\widetilde{\mathcal{L}}_{S}(\theta_{S},\phi_{S};X_{S},Y_{S})$
$\displaystyle-\alpha_{2}\cdot\widetilde{\mathcal{L}}_{T}(\theta_{T},\phi_{T};X_{T},\widehat{Y}_{T})+\alpha_{3}\cdot\widetilde{\mathcal{D}}(\phi_{S},\phi_{T}).$
(39)

###  4.3 Data and results

In this section, we present performances of the three proposed frameworks on
two cardiac segmentation tasks, i.e., CT-MR cross modality cardiac
segmentation, and C0-LGE multi-sequence CMR segmentation.

####  4.3.1 Data

We used two datasets from two public challenges, i.e., the CT-MR dataset from
MM-WHS challenge [59, 60], and the the bSSFP and LGE CMR images from MS-CMRSeg
challenge 111http://www.sdspeople.fudan.edu.cn/zhuangxiahai/0/mscmrseg19/. The
former dataset was from different subjects, while the later was paired images
from the same subjects and we shuffled them to be unpaired.

CT-MR dataset: This dataset consists of 52 cardiac CT images and 45 MR images,
of which 20 CT images and 20 MR images were from the MM-WHS challenge, and the
others were from an open data source [61]. For each 3D image, 16 slices from
the long-axis view around the center of left ventricular cavity were selected,
cropped with size of $192\times 192$ pixel around the center of heart. All
methods were validated on these 2D slices for the segmentation of left
ventricular cavity (LV) and left ventricular myocardium (MYO).

MS-CMRSeg dataset: This dataset consists of 45 paired bSSFP CMR and LGE CMR
images, among which 5 LGE CMR images were provided with labels for validation,
and the ground truths of other 40 images were not available. The target is to
learn knowledge from the labeled bSSFP CMR, and transfer it to LGE CMR images
for the prediction of LV, MYO and RV.

####  4.3.2 Comparison study for DDFSeg

We compared DDFSeg with other four methods: (1) Unet(supervised): A U-net was
trained with labeled target data in a supervised manner. (2) Unet(NoAdapt): A
U-net was trained with the source data, and then applied directly on the
target images. (3) CycleSeg: We used CycleGAN [58] for image translation. The
generated fake target images were then used for model training, and (4) SIFA
[48].

Table 7: Comparison results for DDFSeg on the CT-MR cardiac dataset in both
directions of domain adaptation. This table adopted from Pei et al. [55]

CT$\longrightarrow$MR
---
Method | MYO | RV | LV | Mean
Dice(%) | ASD(mm) | Dice(%) | ASD(mm) | Dice(%) | ASD(mm) | Dice(%) | ASD(mm)
Unet(supervised) |  77.1$\pm$10.2 |  8.6$\pm$5.4 |  86.8$\pm$11.2 |  4.9$\pm$3.1 |  90.3$\pm$8.4 |  2.5$\pm$5.4 |  84.8$\pm$11.4 |  5.3$\pm$4.5
Unet(NoAdapt) |  23.8$\pm$24.1 |  17.2$\pm$8.5 |  64.7$\pm$22.1 |  12.6$\pm$7.9 |  72.0$\pm$19.7 |  8.7$\pm$5.1 |  53.4$\pm$30.6 |  12.8$\pm$8.1
CycleSeg |  53.2$\pm$17.1 |  11.8$\pm$5.1 |  79.2$\pm$13.1 |  8.9$\pm$4.7 |  81.3$\pm$11.8 |  6.6$\pm$3.6 |  71.2$\pm$19.1 |  9.1$\pm$5.0
SIFA |  67.3$\pm$11.4 | 8.2$\pm$5.3 | 84.2$\pm$11.5 |  5.3$\pm$2.8 |  87.6$\pm$8.9 |  4.6$\pm$2.3 |  79.6$\pm$13.9 | 6.0$\pm$4.0
DDFseg | 71.3$\pm$10.6 |  9.7$\pm$5.7 |  83.2$\pm$11.7 | 4.6$\pm$2.4 | 87.7$\pm$10.4 |  3.8$\pm$1.9 | 80.7$\pm$12.9 |  6.0$\pm$4.5
MR$\longrightarrow$CT
Method | MYO | RV | LV | Mean
Dice(%) | ASD(mm) | Dice(%) | ASD(mm) | Dice(%) | ASD(mm) | Dice(%) | ASD(mm)
Unet(supervised) |  84.1$\pm$5.0 |  3.2$\pm$2.4 |  89.2$\pm$6.7 |  3.9$\pm$2.3 |  90.6$\pm$10.6 |  3.9$\pm$3.3 |  88.0$\pm$8.3 |  3.6$\pm$2.7
Unet(NoAdapt) |  10.6$\pm$9.1 |  22.2$\pm$8.0 |  56.0$\pm$12.4 |  18.3$\pm$7.0 |  56.2$\pm$16.7 |  17.0$\pm$5.0 |  40.9$\pm$25.1 |  19.2$\pm$7.1
CycleSeg |  51.3$\pm$15.4 |  6.6$\pm$3.8 | 83.3$\pm$7.7 |  8.4$\pm$2.9 |  79.3$\pm$15.3 |  8.3$\pm$3.9 |  71.3$\pm$19.5 |  7.8$\pm$3.7
SIFA |  56.6$\pm$12.4 | 6.8$\pm$3.8 |  80.0$\pm$8.3 |  8.0$\pm$2.7 |  82.6$\pm$12.6 | 7.8$\pm$3.0 |  73.1$\pm$16.3 |  7.5$\pm$3.3
DDFseg | 66.9$\pm$11.0 |  6.8$\pm$4.6 |  79.1$\pm$6.7 | 6.6$\pm$3.9 | 83.5$\pm$16.0 |  8.3$\pm$4.2 | 76.5$\pm$13.8 | 7.3$\pm$4.3

Table 7 presents the comparison results on CT-MR cross modality segmentation.
One can see that U-Net (NoAdapt) performed poorly on both tasks because of the
domain shift. When MR images were taken as the target data with CT as the
source domain, DDFseg achieved the best Dice and ASD values among all UDA
methods on both tasks. Particularly on MYO, when compared to SIFA, it obtained
more than 4% higher Dice score for MR segmentation and more than 10% for CT
segmentation.

Table 8: Comparison results for DDFSeg on LGE CMR segmentation. This table
adopted from Pei et al. [55]

Method | MYO | RV | LV | Mean
---|---|---|---|---
Dice(%) | ASD(mm) | Dice(%) | ASD(mm) | Dice(%) | ASD(mm) | Dice(%) | ASD(mm)
Unet(supervised) |  74.4$\pm$10.0 |  2.0$\pm$1.8 |  78.6$\pm$12.0 |  1.8$\pm$1.0 |  87.1$\pm$9.0 |  1.1$\pm$0.5 |  80.0$\pm$11.6 |  1.7$\pm$1.3
Unet(NoAdapt) |  29.6$\pm$19.5 |  5.4$\pm$5.3 |  48.1$\pm$20.2 |  3.5$\pm$1.9 |  62.7$\pm$18.0 |  3.4$\pm$1.8 |  46.8$\pm$23.5 |  4.1$\pm$3.5
CycleSeg |  57.1$\pm$14.9 |  2.6$\pm$1.7 |  75.7$\pm$15.3 |  2.5$\pm$2.1 |  82.2$\pm$9.3 |  2.2$\pm$1.2 |  71.1$\pm$17.1 |  2.4$\pm$1.7
SIFA |  68.1$\pm$15.0 |  2.2$\pm$1.9 |  73.6$\pm$18.7 |  1.7$\pm$1.2 |  83.5$\pm$13.0 |  1.6$\pm$0.8 |  75.1$\pm$16.9 |  1.8$\pm$1.4
CFDnet |  69.5$\pm$9.2 |  2.5$\pm$1.8 |  77.6$\pm$8.8 |  1.9$\pm$1.4 |  86.4$\pm$5.6 |  1.9$\pm$0.9 |  77.8$\pm$10.6 |  2.1$\pm$1.4
DDFseg | 75.0$\pm$7.3 | 1.4$\pm$1.3 | 84.5$\pm$7.0 | 1.3$\pm$0.8 | 88.6$\pm$5.0 | 1.4$\pm$0.9 | 82.7$\pm$8.6 | 1.3$\pm$1.0

For LGE CMR segmentation, we presents the comparison results in Table 8. One
can see that U-Net (NoAdapt) failed on this task, especially on myocardium.
This indicates the large domain shift between the two domains. Among all the
three UDA methods, DDFSeg achieved the best results, with 7.6% higher average
Dice score than SIFA. Cycleseg and SIFA obtained comparable Dice scores on RV
and LV, but the Dice scores were much lower than that of DDFSeg. The reason
could be that their performances were heavily dependent on the generated fake
images. However, they did not pay attention on the DSFs to enhance the
translation process, while we introduced the zero-loss, which led to better
image disentanglement and higher image quality of translated images.

Table 9: Performance comparisons for CFDNet on the CT-MR cross-modality
cardiac segmentation task. This table adopted from Wu et al. [56]

Segmentation task | methods | LV | MYO
---|---|---|---
Dice(%) | ASSD(mm) | Dice(%) | ASSD(mm)
Target: MR seg | NoAdapt |  44.4$\pm$13.9 |  19.1$\pm$6.52 |  24.4$\pm$9.11 |  17.3$\pm$2.70
Source: CT | PnP-AdaNet |  86.2$\pm$6.46 | 2.74$\pm$1.04 |  57.9$\pm$8.43 |  2.46$\pm$0.661
| AdvLearnNet |  83.8$\pm$10.3 |  5.76$\pm$6.07 |  61.9$\pm$15.2 |  3.79$\pm$2.23
| CORALnet | 88.4$\pm$7.11 |  3.02$\pm$2.45 | 67.7$\pm$10.8 | 3.18$\pm$1.65
| MMDnet |  86.7$\pm$8.65 |  3.64$\pm$3.47 |  64.4$\pm$12.0 |  3.85$\pm$2.39
| CFDnet | 88.7$\pm$10.6 | 2.99$\pm$2.79 | 67.9$\pm$8.62 |  3.40$\pm$2.75
Target: CT seg | NoAdapt |  30.3$\pm$27.7 | N/A |  0.140$\pm$0.130 | N/A
Source: MR | PnP-AdaNet | 78.3$\pm$18.4 | 3.88$\pm$4.09 | 62.8$\pm$8.24 | 3.09$\pm$1.59
| AdvLearnNet |  77.7$\pm$18.0 |  4.56$\pm$3.68 |  54.7$\pm$9.51 |  3.65$\pm$1.31
| CORALnet |  76.1$\pm$16.9 |  12.2$\pm$10.2 |  58.1$\pm$10.9 |  5.89$\pm$2.97
| MMDnet |  77.7$\pm$18.2 |  5.62$\pm$4.86 |  57.1$\pm$12.0 |  4.13$\pm$1.70
| CFDnet | 81.9$\pm$18.2 | 3.64$\pm$3.94 | 62.9$\pm$10.9 | 3.16$\pm$1.18

Table 10: Performance comparison for CFDNet on LGE CMR images with bSSFP CMR
as the source domain. This table adopted from Wu et al. [56]

methods | Dice (%) | ASSD (mm)
---|---|---
MYO | LV | RV | MYO | LV | RV
NoAdapt |  14.5$\pm$20.12 |  34.5$\pm$31.6 |  31.1$\pm$26.3 |  21.6$\pm$19.4 |  11.3$\pm$13.1 |  14.5$\pm$17.3
PnP-AdaNet |  64.6$\pm$16.4 |  78.4$\pm$16.2 |  72.6$\pm$19.0 |  4.64$\pm$6.41 |  13.8$\pm$10.3 |  5.30$\pm$5.33
AdvLearnNet |  65.5$\pm$13.7 |  84.6$\pm$8.26 | 75.2$\pm$16.5 |  2.68$\pm$1.23 |  3.70$\pm$2.33 | 4.08$\pm$2.65
CORALnet | 68.0$\pm$10.4 | 85.2$\pm$6.41 |  73.8$\pm$11.7 | 2.30$\pm$0.831 |  3.43$\pm$1.66 |  5.44$\pm$2.55
MMDnet |  67.0$\pm$9.83 |  84.8$\pm$6.26 |  72.3$\pm$11.4 | 2.32$\pm$0.664 | 3.26$\pm$1.27 |  5.74$\pm$2.46
CFDnet | 69.1$\pm$9.69 | 86.4$\pm$5.62 | 76.0$\pm$10.9 |  2.46$\pm$0.840 | 3.07$\pm$1.66 | 4.50$\pm$2.13

Figure 12: Visualization of 2D LGE CMR slices and segmentation results from
comparison study for CFDNet. These are extracted from the test subject with
median Dice score by CFDnet. Image adopted from Wu et al. [56]

####  4.3.3 Comparison study for CFDNet

We compared the the proposed CFDnet with five other methods: (1) NoAdapt, (2)
PnP-AdaNet, (3) AdvLearnNet: We used the same network via adversarial
training, (4) CORALnet: This method is the same as CFDNet except using the
distance between the second-order statistics (covariances) of the source and
target features as the explicit metric, which was proposed by [54], and (5)
MMDnet: This method is the same as CFDNet except using MMD with Gaussian
kernel function as the explicit metric.

Table 9 presents the comparison results on CT-MR cross-modality cardiac
segmentation. When we took MR images as the target domain and CT as the source
one, CFDNet achieved comparable results with PnP-AdaNet, though worse in ASSD
values. When tested on CT images with MR as the source domain, CFDnet obtained
better in Dice scores, especially significantly better on LV ($p<0.01$). These
results indicated that the proposed CF distance was effective for domain
adaptation on segmentation tasks, and could achieve no worse prediction than
the conventional adversarial training methods.

For LGE CMR segmentation, as shown in Table 10, CFDNet obtained much better
accuracies on all structures in all metrics compared to PnP-AdaNet and
AdvLearnNet. This results further demonstrated the effectiveness of explicit
metric for domain adaptation. Compared to other explicit metrics, CF distance
obtained higher score on RV, and comparable results on LV and MYO. This might
be due to the substitution process for the computation of CF distance, which
used the marginal distributions instead of joint ones. This substitution
leaded to a weaker constraint for feature extraction. We further provided
their visual comparison in Figure 12. One can see that CFDNet can achieve
better prediction, while the shapes of the segmentation results were not
satisfactory as expected.

Table 11: Performance comparison for VarDA on LGE CMR images. This table
adopted from Wu et al. [57]

methods | Dice (%) | ASSD (mm)
---|---|---
MYO | LV | RV | MYO | LV | RV
NoAdapt |  14.50$\pm$20.18 |  34.51$\pm$31.62 |  31.10$\pm$26.30 |  21.6$\pm$19.4 |  11.3$\pm$13.1 |  14.5$\pm$17.3
PnP-AdaNet |  64.64$\pm$16.41 |  78.43$\pm$16.24 |  72.66$\pm$19.04 |  4.64$\pm$6.41 |  13.8$\pm$10.3 |  5.30$\pm$5.33
CFDnet |  69.1$\pm$9.69 |  86.4$\pm$5.62 |  76.0$\pm$10.9 |  2.46$\pm$0.840 |  3.07$\pm$1.66 |  4.50$\pm$2.13
SIFA |  70.66$\pm$9.689 |  84.62$\pm$7.760 |  83.99$\pm$6.821 |  2.40$\pm$1.22 |  2.68$\pm$1.14 | 2.05$\pm$1.19
VarDA | 73.03$\pm$8.316 | 88.06$\pm$4.832 |  78.47$\pm$14.86 |  1.73$\pm$0.560 |  2.55$\pm$1.18 |  3.51$\pm$2.24

Figure 13:  Visualization of 2D LGE MR cardiac image segmentation results for
comparison study for VarDA. The cardiac structure of MYO, LV, RV are indicated
in blue, red and green color, respectively. Note that VarDA is the same as
DAVAEVarDA. Image adopted from Wu et al. [57]

####  4.3.4 Comparison study for VarDA

We compared the proposed VarDA with three state-of-the-art methods, i.e., PnP-
AdaNet (also denoted as PnP-Ada for short) [42], SIFA [48] and CFDNet.

Table 11 presents the comparison results on LGE CMR segmentation. The
segmentation was done slice-by-slice in a 2D manner. Figure 13 provides the
visualization of the segmentation results of a subject, which was the median
case of VarDA according to the average Dice score. One can see that the
proposed VarDA performed much better the CFDNet, which also used an explicit
metric for domain adaptation. The reason could be that it used variational
approximation, which forced the correlations between the elements of the
latent features to be weak, and thus leaded to a reasonable substitution of
the metric calculation. When compared to SIFA, VarDA obtained better
accuracies on MYO and LV segmentation but worse on RV segmentation.

###  4.4 Conclusion

The three proposed domain adaptation frameworks investigated different
aspects. DDFSeg studied the technique of feature disentanglement by paying
more attention on DSFs, and demonstrated that DSFs are useful for image
translation and thus for domain adaptation. CFDNet proposed a new metric for
domain discrepancy, and was effective in cross-modality segmentation, with
comparable results as the conventional adversarial training methods. Based on
CFDNet, we further proposed VarDA, which improved the effectiveness and
performance of the explicit metric for domain adaptation.

## References

  * [1] A. Collignon, F. Maes, D. Delaere, D. Vandermeulen, P. Suetens, G. Marchal, Automated multi-modality image registration based on information theory, in: Information processing in medical imaging, Vol. 3, 1995, pp. 263–274.
  * [2] B. H. Menze, A. Jakab, S. Bauer, J. Kalpathy-Cramer, K. Farahani, J. Kirby, Y. Burren, N. Porz, J. Slotboom, R. Wiest, et al., The multimodal brain tumor image segmentation benchmark (BRATS), IEEE transactions on medical imaging 34 (10) (2014) 1993–2024.
  * [3] T. Zhou, S. Ruan, S. Canu, A review: Deep learning for medical image segmentation using multi-modality fusion, Array 3 (2019) 100004.
  * [4] N. Tajbakhsh, L. Jeyaseelan, Q. Li, J. N. Chiang, Z. Wu, X. Ding, Embracing imperfect datasets: A review of deep learning solutions for medical image segmentation, Medical Image Analysis 63 (2020) 101693.
  * [5] K. Thygesen, J. S. Alpert, H. D. White, Universal definition of myocardial infarction, Journal of the American College of Cardiology 50 (22) (2007) 2173–2195.
  * [6] C. Chen, C. Ouyang, G. Tarroni, J. Schlemper, H. Qiu, W. Bai, D. Rueckert, Unsupervised multi-modal style transfer for cardiac MR segmentation, in: International Workshop on Statistical Atlases and Computational Models of the Heart, Springer, 2019, pp. 209–219.
  * [7] H. Roth, W. Zhu, D. Yang, Z. Xu, D. Xu, Cardiac segmentation of LGE MRI with noisy labels, in: International Workshop on Statistical Atlases and Computational Models of the Heart, Springer, 2019, pp. 228–236.
  * [8] X. Huang, M.-Y. Liu, S. Belongie, J. Kautz, Multimodal unsupervised image-to-image translation, in: Proceedings of the European conference on computer vision (ECCV), 2018, pp. 172–189.
  * [9] S. Jia, A. Despinasse, Z. Wang, H. Delingette, X. Pennec, P. Jaïs, H. Cochet, M. Sermesant, Automatically segmenting the left atrium from cardiac images using successive 3D U-nets and a contour loss, in: International Workshop on Statistical Atlases and Computational Models of the Heart, Springer, 2018, pp. 221–229.
  * [10] J. Wang, H. Huang, C. Chen, W. Ma, Y. Huang, X. Ding, Multi-sequence cardiac MR segmentation with adversarial domain adaptation network, in: M. Pop, M. Sermesant, O. Camara, X. Zhuang, S. Li, A. Young, T. Mansi, A. Suinesiaputra (Eds.), Statistical Atlases and Computational Models of the Heart. Multi-Sequence CMR Segmentation, CRT-EPiggy and LV Full Quantification Challenges, Springer International Publishing, Cham, 2020, pp. 254–262.
  * [11] V. M. Campello, C. Martín-Isla, C. Izquierdo, S. E. Petersen, M. A. G. Ballester, K. Lekadir, Combining multi-sequence and synthetic images for improved segmentation of late gadolinium enhancement cardiac MRI, in: M. Pop, M. Sermesant, O. Camara, X. Zhuang, S. Li, A. Young, T. Mansi, A. Suinesiaputra (Eds.), Statistical Atlases and Computational Models of the Heart. Multi-Sequence CMR Segmentation, CRT-EPiggy and LV Full Quantification Challenges, Springer International Publishing, Cham, 2020, pp. 290–299.
  * [12] Y. Liu, W. Wang, K. Wang, C. Ye, G. Luo, An automatic cardiac segmentation framework based on multi-sequence MR image, in: International Workshop on Statistical Atlases and Computational Models of the Heart, Springer, 2019, pp. 220–227.
  * [13] J.-Y. Zhu, T. Park, P. Isola, A. A. Efros, Unpaired image-to-image translation using cycle-consistent adversarial networks, in: Proceedings of the IEEE international conference on computer vision, 2017, pp. 2223–2232.
  * [14] S. Vesal, N. Ravikumar, A. Maier, Automated multi-sequence cardiac MRI segmentation using supervised domain adaptation, in: International workshop on statistical atlases and computational models of the heart, Springer, 2019, pp. 300–308.
  * [15] J. Chen, H. Li, J. Zhang, B. Menze, Adversarial convolutional networks with weak domain-transfer for multi-sequence cardiac MR images segmentation, in: M. Pop, M. Sermesant, O. Camara, X. Zhuang, S. Li, A. Young, T. Mansi, A. Suinesiaputra (Eds.), Statistical Atlases and Computational Models of the Heart. Multi-Sequence CMR Segmentation, CRT-EPiggy and LV Full Quantification Challenges, Springer International Publishing, Cham, 2020, pp. 317–325.
  * [16] X. Wang, S. Yang, M. Tang, Y. Wei, X. Han, L. He, J. Zhang, SK-Unet: An improved U-net model with selective kernel for the segmentation of multi-sequence cardiac MR, in: M. Pop, M. Sermesant, O. Camara, X. Zhuang, S. Li, A. Young, T. Mansi, A. Suinesiaputra (Eds.), Statistical Atlases and Computational Models of the Heart. Multi-Sequence CMR Segmentation, CRT-EPiggy and LV Full Quantification Challenges, Springer International Publishing, Cham, 2020, pp. 246–253.
  * [17] X. Wang, S. Yang, M. Tang, Y. Wei, X. Han, L. He, J. Zhang, SK-Unet: an improved U-net model with selective kernel for the segmentation of multi-sequence cardiac MR, in: International Workshop on Statistical Atlases and Computational Models of the Heart, Springer, 2019, pp. 246–253.
  * [18] J. Hu, L. Shen, G. Sun, Squeeze-and-excitation networks, in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 7132–7141.
  * [19] X. Zhuang, Multivariate mixture model for myocardial segmentation combining multi-source images, IEEE transactions on pattern analysis and machine intelligence 41 (12) (2018) 2933–2946.
  * [20] H. Yu, S. Zha, Y. Huangfu, C. Chen, M. Ding, J. Li, Dual attention U-Net for multi-sequence cardiac MR images segmentation, in: Myocardial Pathology Segmentation Combining Multi-Sequence CMR Challenge, Springer, 2020, pp. 118–127.
  * [21] C. Martín-Isla, M. Asadi-Aghbolaghi, P. Gkontra, V. M. Campello, S. Escalera, K. Lekadir, Stacked BCDU-Net with semantic CMR synthesis: Application to myocardial pathology segmentation challenge, in: Myocardial Pathology Segmentation Combining Multi-Sequence CMR Challenge, Springer, 2020, pp. 1–16.
  * [22] T. Park, M.-Y. Liu, T.-C. Wang, J.-Y. Zhu, Semantic image synthesis with spatially-adaptive normalization, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 2337–2346.
  * [23] S. Zhai, R. Gu, W. Lei, G. Wang, Myocardial edema and scar segmentation using a coarse-to-fine framework with weighted ensemble, in: Myocardial Pathology Segmentation Combining Multi-Sequence CMR Challenge, Springer, 2020, pp. 49–59.
  * [24] M. Tan, Q. Le, Efficientnet: Rethinking model scaling for convolutional neural networks, in: International Conference on Machine Learning, PMLR, 2019, pp. 6105–6114.
  * [25] Z. Zhang, C. Liu, W. Ding, S. Wang, C. Pei, M. Yang, L. Huang, Multi-modality pathology segmentation framework: Application to cardiac magnetic resonance images, in: Myocardial Pathology Segmentation Combining Multi-Sequence CMR Challenge, Springer, 2020, pp. 37–48.
  * [26] J. Zhang, Y. Xie, Z. Liao, J. Verjans, Y. Xia, Efficientseg: A simple but efficient solution to myocardial pathology segmentation challenge, in: Myocardial Pathology Segmentation Combining Multi-Sequence CMR Challenge, Springer, 2020, pp. 17–25.
  * [27] X. Zhang, M. Noga, K. Punithakumar, Fully automated deep learning based segmentation of normal, infarcted and edema regions from multiple cardiac MRI sequences, in: Myocardial Pathology Segmentation Combining Multi-Sequence CMR Challenge, Springer, 2020, pp. 82–91.
  * [28] X. Zhuang, J. Xu, X. Luo, C. Chen, C. Ouyang, D. Rueckert, V. M. Campello, K. Lekadir, S. Vesal, N. RaviKumar, et al., Cardiac segmentation on late gadolinium enhancement MRI: a benchmark study from multi-sequence cardiac MR segmentation challenge, arXiv preprint arXiv:2006.12434.
  * [29] B. Ly, H. Cochet, M. Sermesant, Style data augmentation for robust segmentation of multi-modality cardiac MRI, in: M. Pop, M. Sermesant, O. Camara, X. Zhuang, S. Li, A. Young, T. Mansi, A. Suinesiaputra (Eds.), Statistical Atlases and Computational Models of the Heart. Multi-Sequence CMR Segmentation, CRT-EPiggy and LV Full Quantification Challenges, Springer International Publishing, Cham, 2020, pp. 197–208.
  * [30] R. Beinart, S. Abbara, A. Blum, M. Ferencik, K. Heist, J. Ruskin, M. Mansour, Left atrial wall thickness variability measured by CT scans in patients undergoing pulmonary vein isolation, Journal of cardiovascular electrophysiology 22 (11) (2011) 1232–1236.
  * [31] G. Pontecorboli, R. M. Figueras i Ventura, A. Carlosena, E. Benito, S. Prat-Gonzales, L. Padeletti, L. Mont, Use of delayed-enhancement magnetic resonance imaging for fibrosis detection in the atria: a review, EP Europace 19 (2) (2017) 180–189.
  * [32] L. Li, F. Wu, G. Yang, L. Xu, T. Wong, R. Mohiaddin, D. Firmin, J. Keegan, X. Zhuang, Atrial scar quantification via multi-scale cnn in the graph-cuts framework, Medical image analysis 60 (2020) 101595.
  * [33] L. Li, V. A. Zimmer, J. A. Schnabel, X. Zhuang, AtrialJSQnet: a new framework for joint segmentation and quantification of left atrium and scars incorporating spatial and shape information, arXiv preprint arXiv:2008.04729.
  * [34] L. Li, V. A. Zimmer, J. A. Schnabel, X. Zhuang, Medical image analysis on left atrial LGE MRI for atrial fibrillation studies: A review, arXiv preprint arXiv:2106.09862.
  * [35] Y. Y. Boykov, M.-P. Jolly, Interactive graph cuts for optimal boundary & region segmentation of objects in nd images, in: IEEE International Conference on Computer Vision, Vol. 1, 2001, pp. 105–112.
  * [36] J. Zhao, Z. Xiong, MICCAI 2018: Atrial segmentation challenge, http://atriaseg2018.cardiacatlas.org/ (2018).
  * [37] R. Karim, R. J. Housden, M. Balasubramaniam, Z. Chen, D. Perry, A. Uddin, Y. Al-Beyatti, E. Palkhi, P. Acheampong, S. Obom, et al., Evaluation of current algorithms for segmentation of scar tissue from late gadolinium enhancement cardiovascular magnetic resonance of the left atrium: an open-access grand challenge, Journal of Cardiovascular Magnetic Resonance 15 (1) (2013) 105–105.
  * [38] N. Otsu, A threshold selection method from gray-level histograms, IEEE transactions on systems, man, and cybernetics 9 (1) (1979) 62–66.
  * [39] J. Liu, X. Zhuang, L. Wu, D. An, J. Xu, T. Peters, L. Gu, Myocardium segmentation from DE MRI using multicomponent Gaussian mixture model and coupled level set, IEEE Transactions on Biomedical Engineering 64 (11) (2017) 2650–2661.
  * [40] O. Ronneberger, P. Fischer, T. Brox, U-net: Convolutional networks for biomedical image segmentation, in: International Conference on Medical image computing and computer-assisted intervention, Springer, 2015, pp. 234–241.
  * [41] D. Ravanelli, E. C. dal Piaz, M. Centonze, G. Casagranda, M. Marini, M. Del Greco, R. Karim, K. Rhode, A. Valentini, A novel skeleton based quantification and 3-D volumetric visualization of left atrium fibrosis using late gadolinium enhancement magnetic resonance imaging, IEEE transactions on medical imaging 33 (2) (2013) 566–576.
  * [42] Q. Dou, C. Ouyang, C. Chen, H. Chen, B. Glocker, X. Zhuang, P. Heng, PnP-AdaNet: Plug-and-play adversarial domain adaptation network at unpaired cross-modality cardiac segmentation, IEEE Access 7 (2019) 99065–99076.
  * [43] H. Shimodaira, Improving predictive inference under covariate shift by weighting the log-likelihood function, Journal of statistical planning and inference 90 (2) (2000) 227–244.
  * [44] G. Csurka, A Comprehensive Survey on Domain Adaptation for Visual Applications, Springer International Publishing, Cham, 2017, pp. 1–35.
  * [45] A. Chartsias, G. Papanastasiou, C. Wang, S. Semple, D. E. Newby, R. Dharmakumar, S. A. Tsaftaris, Disentangle, align and fuse for multimodal and zero-shot image segmentation, CoRR abs/1911.04417.
  * [46] Q. Dou, L. Yu, H. Chen, Y. Jin, X. Yang, J. Qin, P.-A. Heng, 3D deeply supervised network for automated segmentation of volumetric medical images, Medical image analysis 41 (2017) 40–54.
  * [47] Y. Ganin, E. Ustinova, H. Ajakan, P. Germain, H. Larochelle, F. Laviolette, M. Marchand, V. Lempitsky, Domain-adversarial training of neural networks, The Journal of Machine Learning Research 17 (1) (2016) 2096–2030.
  * [48] C. Chen, Q. Dou, H. Chen, J. Qin, P. A. Heng, Unsupervised bidirectional cross-modality adaptation via deeply synergistic image and feature alignment for medical image segmentation, IEEE Transactions on Medical Imaging 39 (7) (2020) 2494–2505.
  * [49] C. Ouyang, K. Kamnitsas, C. Biffi, J. Duan, D. Rueckert, Data efficient unsupervised domain adaptation for cross-modality image segmentation, in: Medical Image Computing and Computer Assisted Intervention - MICCAI 2019 - 22nd International Conference, Shenzhen, China, October 13-17, 2019, Proceedings, Part II, 2019, pp. 669–677.
  * [50] R. Cai, Z. Li, P. Wei, J. Qiao, K. Zhang, Z. Hao, Learning disentangled semantic representation for domain adaptation, in: IJCAI: proceedings of the conference, Vol. 2019, NIH Public Access, 2019, p. 2060.
  * [51] I. Goodfellow, et al., Generative adversarial nets, in: Advances in neural information processing systems, 2014, pp. 2672–2680.
  * [52] M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, S. Hochreiter, GANs trained by a two time-scale update rule converge to a local nash equilibrium, in: Advances in Neural Information Processing Systems 30, Curran Associates, Inc., 2017, pp. 6626–6637.
  * [53] E. Tzeng, J. Hoffman, N. Zhang, K. Saenko, T. Darrell, Deep domain confusion: Maximizing for domain invariance, CoRR abs/1412.3474.
  * [54] B. Sun, K. Saenko, Deep CORAL: Correlation alignment for deep domain adaptation, in: European Conference on Computer Vision, Springer, 2016, pp. 443–450.
  * [55] C. Pei, F. Wu, L. Huang, X. Zhuang, Disentangle domain features for cross-modality cardiac image segmentation, Medical Image Analysis 71 (2021) 102078\.
  * [56] F. Wu, X. Zhuang, CF distance: A new domain discrepancy metric and application to explicit domain adaptation for cross-modality cardiac image segmentation, IEEE Transactions on Medical Imaging 39 (12) (2020) 4274–4285.
  * [57] F. Wu, X. Zhuang, Unsupervised domain adaptation with variational approximation for cardiac segmentation, IEEE Transactions on Medical Imaging (2021) 1–13.
  * [58] J.-Y. Zhu, P. Taesung, I. Phillip, E. A. A, Unpaired image-to-image translation using cycle-consistent adversarial networks, in: ICCV, 2017, pp. 2223–2232.
  * [59] X. Zhuang, J. Shen, Multi-scale patch and multi-modality atlases for whole heart segmentation of MRI, Medical image analysis 31 (2016) 77–87.
  * [60] X. Zhuang, et al., Evaluation of algorithms for multi-modality whole heart segmentation: An open-access grand challenge, Medical Image Analysis 58 (2019) 101537–101550.
  * [61] M. Schaap, et al., Standardized evaluation methodology and reference database for evaluating coronary artery centerline extraction algorithms, Medical image analysis 13 (5) (2009) 701–714.