# TricycleGAN: Unsupervised Image Synthesis and Segmentation Based on Shape
Priors

Umaseh Sivanesan  Department of Surgery at McMaster University, Hamilton,
Ontario, Canada  Luis H. Braga  Department of Surgery at McMaster University,
Hamilton, Ontario, Canada  Division of Paediatric Urology, McMaster Children’s
Hospital, Hamilton, Ontario, Canada  Ranil R. Sonnadara  Department of Surgery
at McMaster University, Hamilton, Ontario, Canada  Research and High
Performance Computing, McMaster University, Hamilton, Ontario, Canada  Vector
Instutite for Artificial Intelligence, Toronto, Ontario, Canada  Kiret Dhindsa
Corresponding Author<EMAIL_ADDRESS>Department of Surgery at McMaster
University, Hamilton, Ontario, Canada  Research and High Performance
Computing, McMaster University, Hamilton, Ontario, Canada  Vector Instutite
for Artificial Intelligence, Toronto, Ontario, Canada

###### Abstract

Medical image segmentation is routinely performed to isolate regions of
interest, such as organs and lesions. Currently, deep learning is the state of
the art for automatic segmentation, but is usually limited by the need for
supervised training with large datasets that have been manually segmented by
trained clinicians. The goal of semi-superised and unsupervised image
segmentation is to greatly reduce, or even eliminate, the need for training
data and therefore to minimze the burden on clinicians when training
segmentation models. To this end we introduce a novel network architecture for
capable of unsupervised and semi-supervised image segmentation called
TricycleGAN. This approach uses three generative models to learn translations
between medical images and segmentation maps using edge maps as an
intermediate step. Distinct from other approaches based on generative
networks, TricycleGAN relies on shape priors rather than colour and texture
priors. As such, it is particularly well-suited for several domains of medical
imaging, such as ultrasound imaging, where commonly used visual cues may be
absent. We present experiments with TricycleGAN on a clinical dataset of
kidney ultrasound images and the benchmark ISIC 2018 skin lesion dataset.

Keywords: Biomedical imaging, machine learning, image segmentation, computer
vision, data augmentation, generative models

##  1 Introduction

In vivo medical imaging is an important and commonly used technique in a wide
variety of clinical applications. However, analysis is often made challenging
due to the poor quality of images obtained with current imaging technologies.
In order to take advantage of these technologies, current medical practice
relies heavily on clinicians who are highly specialized in interpreting
medical images [1, 2].

Segmentation, or identifying and isolating regions of interest (ROIs) from an
image, is often a key step in medical image analysis. For both human and
machine readers, segmentation aids in the extraction of clinically important
features by identifying relevant tissues and/or objects, such as an organ or a
lesion, without interference from non-relevant tissues captured in the image
[3]. Since manual segmentation, i.e., having appropriately trained clinicians
segment images by hand, is time-consuming, expensive, and subjective,
significant effort has been put into developing algorithms that can
automatically provide accurate and reliable segmentations for downstream
analysis [4]. This has lead to the growth of the subfield of semantic
segmentation, which is image segmentation via the association of pixels with
classification labels (e.g., kidney vs. not kidney).

Here we present a novel approach to image segmentation that can be trained
unsupervised or semi-supervised, thus greatly reducing, or even eliminating,
the need for manually segmented images. Inspired by the way trained clinicians
use anatomical knowledge of the shapes of ROIs and their internal structures
when performing manual segmentation, we rely on shape priors to generate
synthetic labelled data that can be used to train a segmentation model. Shape
priors are built in by generating synthetic medical images from ROI shape
templates, making it possible to learn mappings between the generated
segmentation-medical image pairs. We expand upon the rationale and methodology
for this approach in Section 4.

Our new architecture, TricycleGAN, is comprised of three Generative
Adversarial Networks (GANs), specifically, three CycleGANs [5] that have been
chained together. Each CycleGAN learns one of three mappings: a mapping
between original medical images and edge maps derived from those medical
images, a mapping between edge maps and segmentation maps for the regions of
interest contained in those medical images, and a mapping between segmentation
maps and medical images. As a whole, TricycleGAN unifies these three
subnetworks using an adapted cycle consistency loss and thus learns to segment
medical images by learning to generate realistic image-edge-segmentation
triplets. As is explained further in Section 4, the inclusion of edge maps
into the process provides an intermediate step that introduces sources of
variance and complexity that allows TricycleGAN to translate simplistic ROI
templates into realistic medical images while retaining the necessary shape
information needed to predict a segmentation map.

To demonstrate the value of this novel approach, we evaluate our method using
two distinct datasets: a dataset of ultrasound images for which the task is to
segment the kidney, and the ISIC 2018 Skin Lesion Analysis competition
dataset, for which the task is to segment skin lesions in dermoscopic images.

##  2 Related Work

Prior to deep learning approaches to automated image segmentation, techniques
based on edge detection [6], region growing [7], contour modelling [8], and
texture analysis [9] relied on detecting statistical difference among objects
in an image using explicitly defined constraints and criteria. These methods
can often perform pooly in medical imaging applications because typically used
metrics, such as contrast gradients, are not robust enough to overcome the
challenges imposed by noisy images, e.g., in ultrasound imaging, and
artifacts, e.g., surrounding tissues.

A current strategy for image segmentation aims to overcome such limitations
using supervised deep neural networks that can learn to identify distinct
objects and ROIs in images by discovering combinations of discriminative
features given exposure to numerous labelled samples. Convolutional neural
networks (CNNs) have been particularly successful for semantic segmentation
and a wide variety of other computer vision applications [10, 11, 12, 13].
Notably, Fully Convolutional Networks (FCNs) omit the fully connected layers
used to group pixels by class label in standard CNNs in favour of
deconvolution layers used to estimate segmentation probability maps for entire
images [14]. Encoder-decoder networks, first introduced by deconvolving a
pretrained VGG16 CNN architecture [15], are based on a similar principle. CNNs
have also been paired with Conditional Random Fields (CRFs) to better take
advantage of the spatial correlations among pixels belonging to the same
object [16].

Variations on these segmentation-oriented CNNs have also played an important
role in clinical applications of medical imaging [17, 18, 19, 20, 21, 22]. The
U-net [23], a type of FCN, has performed well in a variety of medical image
segmentation tasks (e.g., [24]), including for 3D segmentation tasks [25].
Various extensions have been developed, including to incorporate an attention
mechanism [26], or by altering the initialization of the model [27].

While successful in many areas, CNN-based approaches can be prone to poor
performance under certain conditions that are often observed in medical
imaging applications. One issue is that obtaining spatially and semantically
contiguous solutions usually requires significant post-processing of the
resulting segmentation maps. In the presence of artifacts and distractors,
such as the presence of multiple salient organs when only one is the desired
ROI, such methods can result in large errors. Fortunately, recent work has
begun to address this issue [28, 29, 30]. However, another significant
limitation for real-world use is that deep learning approaches in general
usually require large amounts of training data, which usually means hundreds
or thousands of ground-truth segmentations that must be hand-drawn by
radiologists outside of their clinical duties. Although some success has been
seen with unsupervised CNN approaches under some conditions, e.g., with W-net
[31] (typically only successful for segmentation tasks with distinct non-
overlapping objects) or DeepCo3 [32] (relying on feature similarity among
multiple instances of same-class objects in an image), unsupervised solutions
for some medical applications may require an altogether different approach.

The use of GANs has been proposed as one way to address the need for ground
truth segmentation labels. GANs learn to perform image-to-image translation by
training with pairs of images [33], and therefore can be appropriated for
segmentation tasks by training them with image-segmentation map pairs in
supervised [34, 35, 36, 37, 17] or semi-supervised [38] fashions. A useful
function of GANs is their ability to augment training datasets by generating
realistic synthetic data, thus requiring relatively smaller amounts of
labelled training data [39, 40, 41]. To adapt GANs to learn segmentation in an
unsupervised manner, recomposition approaches have been used. For example,
SEIGAN can be used to segment foreground objects by exploiting distinct
feature statistics between background and foreground objects and similar
feature statistics between similar backgrounds [42]. Another recent approach,
ReDO, uses scene decomposition to separate objects in an image based on the
assumption that each object is distinct with respect to specific properties,
such as colour and texture [43].

Unsupervised approaches for image segmentation make reasonable assumptions
about ways in which the desired ROI is distinct from other objects in an
image. However, typically relied upon features, such as colour, texture, or
brightness, may not carry over well to many medical imaging domains. In the
case of organ segmentation, as illustrated with the kidney ultrasound dataset
used in this study, a clinician must rely primarily on a priori anatomical
knowledge and experience in order to estimate the contours of the kidney when
a clear boundary is not visible. This task is made especially difficult by the
fact that the kidney may not even be the most salient object in the image,
leading even clinicians from other domains to find the task very challenging.

To fill the gap left by previous methods, TricycleGAN is aimed at overcoming
these challenges by incorporating a shape prior for the ROI to generate
synthetic image-segmentation map pairs using only unlabelled medical images,
simulating the use of a priori anatomical knowledge that a radiologist would
use to perform similar segmentation tasks. TricycleGAN is an extention of
CycleGAN [5] that uses three generators: the first translates between
segmentation maps and edge maps, the second translates between edge maps and
medical images, and the third uses the cycle consistence loss to translate
medical images back into segmentation maps. An illustration of the TricycleGAN
pipeline is given in Figure 1, and examples showing the output at every major
step of the pipeline using the ISIC 2018 skin lesion dataset are shown in
Figure 2.

##  3 Data and Preparation

###  3.1 Dataset 1: Renal Ultrasound Images

We use a dataset of renal ultrasound images developed for prenatal
hydronephrosis, a congenital kidney disorder marked by excessive and
potentially dangerous fluid retention in the kidneys. The dataset consists of
2492 2D sagittal kidney ultrasound images from 773 patients across multiple
hospital visits.The evaluation set consists of 438 images that have been
manually segmented by a trained surgical urologist. After removing training
images taken from the same patients represented in the evaluation set, 918
unlabelled images remain for training. During training, random samples of 20%
of the images were used for validation. Each grade of hydronephrosis is
represented in the evaluation set approximately evenly.

This is a difficult dataset for image segmentation due to poor image quality,
unclear contours of the kidneys, and the large variation introduced by
different degrees of the kidney disorder called hydronephrosis (see
Supplementary Figure S1). In addition, a major challenge of this dataset is
that the two most salient boundaries are the outer ultrasound cone inherent to
ultrasound imaging with a probe, and the dark inner region of the kidney,
which is caused by fluid retention in hydronephrosis. As neither of these are
the desired ROI, both are misleading with respect to segmenting the kidney.
Further details concerning this dataset can be found in [44, 45, 46].

###  3.2 Dataset 2: Skin Lesion Segmentation

We use the ISIC 2018 Lesion Boundary Segmentation Challenge dataset [47, 48]
to more directly compare TricycleGAN with other approaches. The provided
training set of 2075 images, 100 validation images, and 519 evaluation images
are used for the intended purposes in this study. By showing that TricycleGAN
is also successful on this benchmark dataset, we show that it is not limited
to only one domain and imaging modality. In addition, we show one method by
which TricycleGAN can be adapted to also take advantage of commonly used
features, such as colour.

###  3.3 Image Preprocessing

We follow a similar methodology used for preprocessing renal ultrasound images
described in [44]. We crop the images to remove white borders, despeckle them
to remove speckle noise caused by interference with the ultrasound probe
during imaging [49], and re-scale to 256$\times$256 pixels for consistency. We
remove text annotations made by clinicians using the pre-trained Efficient and
Accurate Scene Text Detector (EAST) [50]. We then normalize the pixel
intensity of each image to be from 0 to 1 after trimming the pixel intensity
from the 2nd percentile to the 98th percentile of the original pixel intensity
across the image. In addition, we enhance the contrast of each image using
Contrast Limited Adaptive Histogram Equalization with a clip limit of 0.03
[51]. Finally, we normalize the images by the mean and standard deviation of
the training set during cross-validation.

We perform no preprocessing for the ISIC skin lesion images other than to
resize them to 256 $\times$ 256 pixels.

##  4 TricycleGAN

###  4.1 Rationale

Analogously to how other image features are exploited for semantic
segmentation, we assume that the shape of an ROI for a particular image
segmentation task is statistically more similar for within-class objects than
it is for background objects. In order to make use of this assumption, we
additionally assume that the boundary of the ROI is sufficiently prominent in
the image to appear in its corresponding edge map (though as mentioned
previously in Section 3.1, this does not require that the ROI boundary is the
most prominent boundary in the image, and does not preclude partial
occlusion). Based on these two assumptions, we make use of a basic template
shape that is appropriate for representing the ROIs of a given class of
objects. For example, both kidneys and skin lesion ROIs are roughly
elliptical. Since TricycleGAN adds the required complexity to make the
resulting synthetic images realistic on its own, constructing a template can
be as simple as drawing an ellipse with randomized shape, size, and location.
Details of this part of the process are given in Section 4.2.

Note that by constructing simple ROIs explicitly and generating synthetic
images from them, we also provide our model with a ground truth segmentation
label for the synthetic images. It is for this reason that we do not generate
the underlying ground truth with a generative model. Instead, the level of
variance and complexity required to generate realistic images in the
application domain are introduced by the downstream generators in TricycleGAN.

In addition to the potentially large variation in ROI morphologies within a
class, a common challenge with medical imaging segmentation is the presence of
artifacts and distractors (i.e., objects or anatomical features that also
appear prominently in edge maps, but which are not part of the intended ROI).
We overcome both of these challenges simultaneously by using a generative
model to learn the statistics of edge maps extracted from real images, which
can then be used to construct realistic edge maps from generated template ROIs
(see Section 4.4). By then constructing synthetic images in the application
domain from the synthetic edge maps, TricycleGAN generates segmentation map -
edge map - medical image triplets that can be used for segmentation training.

The reason for using edge maps as an intermediary step is that edge maps can
be easily extracted from real training data using known statistical methods,
providing a training set of real image - edge map pairs that can be used to
learn a mapping with supervision. Since segmentation maps cannot be extracted
from the real images, a direct mapping cannot be learned. Instead, as
described in Section 4.4, we introduce a patch-occlusion method for learning
the statistics of real edge maps and use this to generate realistic edge maps
from synthetic segmentation maps, thus completing the loop from synthetic
segmentation map to synthetic edge map to synthetic image and back to a
predicted segmentation map with the required complexity.

Figure 1: The TricycleGAN training pipeline. Figure 2: Examples showing the
output of each major step of the program, including synthetic segmentation map
construction and the output of each generator.

###  4.2 Constructing Synthetic Segmentation Maps

The primary concern when construction ROI templates, or synthetic segmentation
maps, to be used as ground truth segmentation maps is to capture the
distribution of shapes and locations that can be expected in the real data.
For many applications this is relatively simple to do.

To generate an ROI, we create a randomized ellipse with a random origin
(horizontally and vertically offset from the center by up to 1/8 of the image
size), rotation (any angle), and major and minor axes (major axis of length
between 1/4 and 1/2 of the image length, and minor axis between 0.5 and 0.9
times the length of the major axis). The ellipse itself serves as the ground
truth segmentation mask.

For the ultrasound dataset we require the addition of the ultrasound cone, as
its outline is even more prominent than the desired kidney ROI. The cone is
also randomly generated by creating a randomized bottom curve (a partially
complete ellipse) and a triangle extending beyond the upper boundary of the
image (so that it is cut off at the top). All pixels of the generated kidney
ROI not lying within the generated ultrasound cone is removed to simulate the
common occurrence of only partially captured kidneys. Similar adjustments in
constructing ROI templates can be made for other applications where prominant
shapes may serve as distractors.

###  4.3 Extracting Edge Maps from Real Images

Edge maps that are extracted from real images are used to train the synthetic
edge map generator. We extract edge maps from real medical images using the
VGG16 model [15] with Richer Convolutional Features pretrained as in Liu et
al. [52]. As recommended for this approach, we further fine-tune the output
edge maps using non-maximum suppression with Structured Forests for edge
thinning [53].

###  4.4 Generator 1: Segmentation Map to Edge Map Translation

To generate images with structural variation resembling that observed in the
real data, we first convert the synthetic segmentation maps into realistic
edge maps, complete with artifacts and natural irregularities, using the first
generative model in TricycleGAN, $G^{s2e}$. As with all of the generators used
in TricycleGAN, $G^{s2e}$ is based on the _pix2pix_ architecture defined in
[33].

To train generator $G^{s2e}$ to produce the appropriate variation seen in the
real data, we frame this problem as an image completion problem (e.g., [54])
and make use of our previously extracted real edge maps. However, since
$G^{s2e}$ takes as input the very simplistic synthetic segmentation maps
without any corresponding realistic edge maps (since the edge maps extracted
from real images do not have paired segmentation maps), on its own it learns
to produce very simple edge maps. This is partially because the use of the
cycle consistency loss used to train TricycleGAN (see Section 4.7 below)
encourages less complex edge maps. We therefore incorporate an additional
generator based on the _pix2pix_ architecture, denoted $G^{pf}$ (a patch-
filling generator), which enhances the complexity of the edge maps produced by
$G^{s2e}$ by adding artifacts and a greater degree of irregularity. Therefore,
edge maps produced by $G^{s2e}$ are patch-occluded and then passed into
$G^{pf}$ for added complexity.

We pretrain $G^{pf}$ on the _edges2handbags_ dataset [55] before further
training it on the edge maps extracted from the training set of real images
after partial occlusion using randomly generated square masks. For each
extracted edge map, up to 10 non-overlapping square masks with length between
1/8 and 1/2 of the image length are used. Once this is done, the weights of
$G^{pf}$ are frozen allowing it to produce the artifacts and necessary
complexity to emulate realistic edge maps, which $G^{s2e}$ does not encounter,
without loss of complexity due to the cycle consistency loss. Additionally,
this allows us to use $G^{pf}$ to help train $G^{s2e}$ by serving as a
regularizer. Specifically, the $L_{1}$-loss used to regularize $G^{s2e}$ is
computed by taking the difference between the output generated by $G^{s2e}$
and $G^{pf}$, thereby encouraging $G^{s2e}$ to increase the complexity of its
output to provide $G^{pf}$ a better starting point for enhancement, and
countering some of the simplicity encouraged by the cycle consistency loss.

A natural question is why TricycleGAN cannot simply use $G^{pf}$ without
$G^{s2e}$. In practice, $G^{pf}$ requires its weights to be frozen in order to
maintain its ability to generate artifacts. Therefore, it does not learn to
contribute to minimizing the overall loss of the network, thus breaking
unification of the overall model and ultimately causing TricycleGAN to perform
suboptimally. If the weights of $G^{pf}$ are allowed to change, the cycle
consistency loss causes it to gradually simplify its edge maps until the
desired complexity is no longer produced. Alternatively, using $G^{s2e}$
without influence from $G^{pf}$ results in overly simplistic edge maps that
fail to provide enough challenge to the later generators of TricycleGAN. The
required balance of complexity and optimal training for TricycleGAN is
achieved by using $G^{pf}$ both as an augmenter of the output of $G^{s2e}$ and
as a regularizer for its training.

In practice, while $G^{s2e}$ undergoes training with $G^{pf}$, we begin by
using simple Sobel filtering to convert segmentation maps into edge maps so
that meaningful outputs can be generated by downstream generators. After 250
epochs of training TricycleGAN, the probability of using the output of
$G^{s2e}$ versus Sobel filtering is linearly ramped down until it reaches zero
at epoch 500. From there, Sobel filtering is discontinued and $G^{pf}$ takes
only the output of $G^{s2e}$ as input. In this way, the downstream generators
of TricycleGAN can still learn to produce meaningful outputs while $G^{s2e}$
learns to initially be able to produce edge maps, and TricycleGAN as a whole
can continue to improve as $G^{s2e}$ learns to produce edge maps with greater
complexity than provided by Sobel filtering.

Finally, $G^{s2e}$ is ready to be used to construct realistic edge maps from
the synthetic segmentation maps with artifacts and added complexity
contributed by $G^{pf}$. Figure 2 provides examples of generator outputs at
each stage of the pipeline comparing $G^{pf}$ ouputs during the early stages
of training when Sobel filtering is used to later stages of training when
$G^{s2e}$ is used.

###  4.5 Generator 2: Edge Map to Medical Image Translation

The second generator in TricycleGAN, $G^{e2m}$ translates edge maps into
realistic images. As before, we use a _pix2pix_ model here. $G^{e2m}$ is
trained using the extracted edge map and real image pairs obtained via our
training data while also training through the cycle consistency loss as part
of TricycleGAN when synthetic images are used as input. Once trained, the
generator is used to translate the edge maps previously constructed from
$G^{s2e}$ into realistic synthetic medical images.

###  4.6 Generator 3: Medical Image to Segmentation Map Translation

The final generator in TricycleGAN, $G^{m2s}$ performs image segmentation by
translating a real or synthetic image into its corresponding segmentation map.
Again, we use the _pix2pix_ architecture here. The model is trained using the
synthetic segmentation maps and their corresponding synthetic medical images
created by $G^{e2m}$. Since no real image - segmentation map pairs are
available for the training data, $G^{m2s}$ is only trained using synthetic
images. Thus, it undergoes considerably less training than the other
generators. The specific loss functions used to train each generator are given
below.

###  4.7 Training Scheme and Loss Functions

####  4.7.1 Training Pipeline

TricycleGAN is trained using a combination of unlabelled real images and
labelled synthetic images. Since we extract edge maps from the real images,
image - edge map pairs are available. Using the patch occlusion method, the
real edge maps are used to pretrain $G^{pf}$, as described in Section 4.4.
When training TricycleGAN with the weights of $G^{pf}$ frozen, the output of
$G^{pf}$ is used to compute the $L_{1}$ loss for $G^{s2e}$. The real image -
edge map pairs are also used to train $G^{e2m}$. However, with no segmentation
map available, no training signal is computable for $G^{m2s}$ when TricycleGAN
trains with real images. Instead, every 20th image is generated using a
synthetic segmentaiton map that is passed through the full network, meaning
that $G^{m2s}$ trains at a slower rate compared to the other generators.

To ensure that generator $G^{m2s}$ learns to generate accurate segmentation
maps, we constrain the output space of the network by incorporating the cycle
consistency loss first introduced into adversarial learning with CycleGAN.
This pushes $G^{m2s}$ to translate an image back into the original image input
into the network during training, i.e., the synthetic segmentation map [5]. A
second motivating factor for using the cycle consistency loss is that it does
not assume that the embedding spaces of the input and output are the same (in
our case, segmentation maps are almost guaranteed to lie in a lower-
dimensional embedding space than the medical images themselves), which
provides some desired flexibility in TricycleGAN.

When training using synthetic images, the cycle consistency losses for each
trainable generator is computed after the initial segmentation map is passed
through the entire network, which provides a training signal for $G^{m2s}$.
The predicted segmentation map generated by $G^{m2s}$ is then looped back to
the start of the network and passed through each generator again, and the
difference between their outputs on the second pass versus their outputs on
their first pass are used to compute the cycle consistency loss (with the
caveat that $G^{s2e}$ uses the output of $G^{pf}$). This allows TricycleGAN to
take as much advantage of the real data as possible, while using synthetic
data to train the chain of generators to function as a cohesive network that
serves a specific purpose. Since TricycleGAN uses three trainable generators
instead of CycleGAN’s two, we use an adapted cycle consistency loss that is
split into three parts; one for each generator. The training pipeline is
illustrated in Figure 1, and further details of how each generator is trained
as part of the network and how the adapted cycle consistency loss is defined
are provided below.

####  4.7.2 Training Generator $G^{s2e}$

To train generator $G^{s2e}$, we start with the adversarial loss defined in
[33]:

$\min_{\theta_{G}}\max_{\theta_{D}}\mathcal{L}_{GAN}(\theta_{G},\theta_{D})+\gamma\mathcal{L}_{L_{1}}(\theta_{G}).$
(1)

Here, $\theta_{G}$ and $\theta_{D}$ are parameters for a generator $G$ and a
discriminator $D$ paired in a GAN.

$\min_{\theta_{G}}\max_{\theta_{D}}\mathcal{L}_{GAN}(\theta_{G},\theta_{D})=\mathbb{E}_{x\sim
P_{X}}\left[log(D(x))\right]+\mathbb{E}_{z\sim
P_{Z}}\left[log(1-D(G(z)))\right],$ (2)

is the conventional adversarial loss [56], where $x$ is a real image from the
training set $X$ with unknown distribution $P_{X}$, and $z$ is a random noise
vector from a Gaussian distribution $P_{Z}$, and

$\mathcal{L}_{L_{1}}(\theta_{G})=\mathbb{E}_{x\sim P_{X},z\sim P_{Z}}\lVert
x-G(z)\rVert_{1}$ (3)

is simply the $L_{1}$ loss between the generated image and the target image.
When training with real images, the target image for $G^{s2e}$ is defined as
the real extracted edge map, and the generated image is $G^{pf}(G^{s2e}(x))$,
as described in Section 4.4. Full details of the construction of this loss
function and the _pix2pix_ network architecture can be found at [33].

To train the generators in TricycleGAN to work together as a cohesive network,
we also include a cycle consistency loss for each generator. For an edge map
$x_{1}$, cycle consistency is satisfied for $G^{s2e}$ if $x_{1}\rightarrow
G^{pf}(G^{s2e}(x_{1}))\rightarrow G^{e2m}(G^{pf}(G^{s2e}(x_{1})))\rightarrow
G^{m2s}(G^{e2m}(G^{pf}(G^{s2e}(x_{1}))))\rightarrow
G^{pf}(G^{s2e}(G^{m2s}(G^{e2m}(G^{pf}(G^{s2e}(x_{1}))))))\approx x_{1}$. The
cycle consistency loss portion for $G^{s2e}$ is defined as

$\mathcal{L}_{Cycle}^{s2e}(G^{s2e},G^{e2m},G^{m2s})=\mathbb{E}_{x_{1}\sim
P_{X_{1}}}\lVert G^{pf}(G^{s2e}(G^{m2s}(G^{e2m}(x_{1}))))-x_{1}\rVert_{1}.\ $
(4)

${L}_{Cycle}^{s2e}$ is computed by passing the generated edge map through the
generators in TricycleGAN to produce an image with $G^{e2m}$, leading to a
predicted segmentation map generated by $G^{m2s}$, which is then used as input
to produce another edge map with $G^{s2e}$. The original real edge map and the
newly generated edge map produced after a cycle through TricycleGAN are used
to compute the cycle consistency loss for $G^{s2e}$.

####  4.7.3 Training Generator $G^{e2m}$

Generator $G^{e2m}$ is trained similarly to $G^{e2m}$. The adversarial loss
and the $L_{1}$ loss are defined as before, except the target image for
$G^{e2m}$ is the medical image domain. As $G^{e2m}$ starts at a different
point in TricycleGAN’s cycle, it contributes the second part of the total
cycle consistency loss:

$\mathcal{L}_{Cycle}^{e2m}(G^{s2e},G^{e2m},G^{m2s})=\mathbb{E}_{x\sim
P_{X}}\lVert G^{e2m}(G^{pf}(G^{s2e}(G^{m2s}(x_{2}))))-x_{2}\rVert_{1},$ (5)

where $x_{2}$ is a real image from the application domain and $y$ is the
attempted reproduction of that image generated by $G^{e2m}$.

####  4.7.4 Training Generator $G^{m2s}$

$G^{m2s}$ can only undergo training when a ground truth segmentation map is
available, and thus only receives a training signal when a synthetic
segmentation map is passed through TricycleGAN. Importantly, $G^{m2s}$ is not
part of a GAN, and therefore there is no adversarial loss. Instead, it acts as
a segmentation model, and thus is trained using its portion of the cycle
consistency loss, the binary cross-entropy loss, and the Tversky loss.

For $G^{m2s}$, we can define the cycle consistency loss between a ground truth
segmentation $x_{3}$ and its predicted segmentation as

$\mathcal{L}_{Cycle}^{m2s}(G^{s2e},G^{e2m},G^{m2s})=\mathbb{E}_{x\sim
P_{X}}\lVert G^{m2s}(G^{e2m}(G^{pf}(G^{s2e}(x_{3}))))-x_{3}\rVert_{1}.$ (6)

The binary cross-entropy is the sum of the cross-entropy loss for all $i$
pixels is defined as

$\mathcal{L}_{BCE}=-\sum_{i}(y_{i}\log(\hat{y}_{i})+(1-y_{i})\log(1-\hat{y}_{i}))$
(7)

for pixel-wise ground truth labels $y_{i}$ and model predictions
$\hat{y}_{i}$.

Finally, we add the Tversky loss, which is a generlization of the dice loss
[57], to promote segmentation accuracy. Given the Tversky similarity index for
a two-class problem

$TI=\frac{\sum P_{g}(i)P_{y}(i)+\epsilon}{\sum P_{g}(i)P_{y}(i)+\alpha\sum
Q_{g}(i)+\beta\sum Q_{y}(i)+\epsilon},$ (8)

where

$P_{g}{i}=\begin{cases}1,&\text{if pixel $i$ is in the ground truth ROI}\\\
0,&\text{otherwise},\end{cases}$ (9)

$Q_{g}(i)=(1-P_{g}(i))P_{y}(i)$, and $Q_{y}(i)=P_{g}(i)(1-P_{y}(i))$.

The Tversky loss is simply $\mathcal{L}_{T}=1-TI$. The parameters $\alpha$ and
$\beta$ allow for shifting emphasis towards minimizing false positives or
false negatives, depending on the class imbalance exhibited in the data. Here
we use $\alpha=\beta=0.5$ to maintain a balanced emphasis.

####  4.7.5 Total Loss for Training TricycleGAN

To train generator $G^{s2e}$, we add together the cycle consistency losses for
$G^{s2e}$ (Eq. 4), and $G^{e2m}$ (Eq. 5), along with its adversarial loss (Eq.
2) and its $L_{1}$ loss (Eq. 3) to obtain

$\mathcal{L}^{s2e}=\mathcal{L}_{GAN}(\theta_{G^{s2e}},\theta_{D^{s2e}})+\lambda_{1}\mathcal{L}_{L_{1}}(\theta_{G^{s2e}})+\mathcal{L}_{Cycle}^{s2e}(G^{s2e},G^{e2m},G^{m2s})+\mathcal{L}_{Cycle}^{e2m}(G^{s2e},G^{e2m},G^{m2s}).\
$ (10)

Note that the sum of $\mathcal{L}^{s2e}_{Cycle}$ and
$\mathcal{L}^{e2m}_{Cycle}$ is analagous to the sum of the forward and
backward cycle consistency in CycleGAN.

Similarly for $G^{e2m}$, the total loss is

$\mathcal{L}^{e2m}=\mathcal{L}_{GAN}(\theta_{G^{e2m}},\theta_{D^{e2m}})+\lambda_{1}\mathcal{L}_{L_{1}}(\theta_{G^{e2m}})+\mathcal{L}_{Cycle}^{s2e}(G^{s2e},G^{e2m},G^{m2s})+\mathcal{L}_{Cycle}^{e2m}(G^{s2e},G^{e2m},G^{m2s}).\
$ (11)

To train $G^{m2s}$, we combine the the cycle consistency losses for each
trainable generator (Eq. 4, Eq. 5, and Eq. 6), the binary cross-entropy loss
(Eq. 7) and the Tversky loss (Eq. 8) to obtain

$\mathcal{L}^{m2s}=\mathcal{L}_{Cycle}^{e2m}(G^{s2e},G^{e2m},G^{m2s})+\mathcal{L}_{Cycle}^{e2m}(G^{s2e},G^{e2m},G^{m2s})+\mathcal{L}_{Cycle}^{m2s}(G^{s2e},G^{e2m},G^{m2s})+\mathcal{L}_{BCE}+\mathcal{L}_{T}.\
$ (12)

For each of the above combined losses, $\lambda_{1}=100$ as recommended in
[33] and $\lambda_{2}=10$.

####  4.7.6 Training Parameters

All generators were trained using the Adam optimizer with a learning rate of
$2^{-4}$ and a batch size of $32$. Training was considered complete when
$\mathcal{L}_{m2s}$ did not improve for 20 consecutive epochs.

####  4.7.7 Data Augmentation

During training, images are altered slightly to introduce additional variance,
thereby allowing the resulting models to learn additional robustness and
improve performance. The alterations for both datasets are as follows: a
random translation of up to 30 pixels along both axes, and a horizontal flip
with a probability of 0.5. The following additional alterations are applied to
the ISIC 2018 training data: a random rotation of
$r\in\\{0,\pi/2,\pi,3\pi/2\\}$, up to a 20% change in brightness, up to a 50%
change in contrast, up to a 5% change in hue, and up to a 50% change in
saturation.

We also use style transfer to provide additional data augmentation for the
ISIC 2018 dataset. This allows the generators to produce a greater variety of
skin lesions, particularly to better capture the statistics of different skin
colours and artifacts (see Figure S2). This allows TricycleGAN to take
advantage of colour features in addition to the usual shape features upon
which it relies. Rather than treating each style variation as separate
training images, styles were incorporated as an additional input channel (or
dimension), enabling the networks to more effectively learn to produce the
same segmentation for the various style variants of an image.

##  5 Evaluation

We evaluate our model using five standard metrics computed using the withheld
evaluation images: the F1 score (also called the dice coefficient),
specificity, sensitibity, intersection over union (IoU; also called the
Jaccard Index), and pixel-wise classification accuracy (pACC).

###  5.1 Semi-Supervised Models

In addition to unsupervised segmentation, TricycleGAN can be used for semi-
supervsed segmentation. We implement semi-supervised learning by fine-tuning
the unsupervised models with increasing amounts of labelled images. The
results of this experiment are given in Figure 5.

###  5.2 U-Net Comparison

We use the commonly used and widely accessible U-net [23] to train a
supervised segmentation model for the ultrasound dataset. As with our U-net
embedded in TricycleGAN, we train the U-net using the sum of the pixel-wise
binary cross-entropy and the dice coefficient as the loss function. We use
Adam for optimization with a batch size of 1. Finally, we perform data
augmentation with horizontal flips (50% probability) and horizontal and
vertical translations of up to 26 pixels (10%).

###  5.3 Mask-RCNN Comparison

The Mask-RCNN [58] was the segmentation model that performed best during the
ISIC 2018 competition [59], therefore it is used here as a direct comparison.
We use anchor sizes of $2^{i},i\in\\{3,4,6,7,8\\}$ and 32 training ROIs per
image, and other hyperparameters were kept to recommended values. We perform
data augmentation with both horizontal and vertical flips (50% probability),
rotation of $\pi/2$ or $3\pi/2$, and a Gaussian blur of up to 5 standard
deviations. Note that is not the same as the winning ISIC 2018 model that is
desribed in Section 5.5, which includes an additional encoder-decoder model
for segmentation.

###  5.4 W-net Comparison

To compare TricycleGAN with a popular unsupervised approach, we train W-net
with the soft normalized cut term in the loss function [31]. In addition, we
perform the recommended post-processing of the W-net generated segmentation
maps using a fully-connected CRF for edge recovery, and hierarchical image
segmentation for contour grouping [60].

###  5.5 Comparison with ISIC Competition Winner

The winning method for the ISIC 2018 segmentation task competition [59] uses
Mask-RCNN for supervised detection of the ROI. The ROI bounding box output by
Mask-RCNN is then flipped and rotated by different angles to create four
copies of the ROI. Each of these is then segmented with a custom encoder-
decoder network, and the average segmentation map is taken as the final
predicted mask.

###  5.6 SegCM Comparison

We test a recent method for unsupervised medical image segmentation based on
the local centre of mass on our kidney dataset [61]. We perform nested 5-fold
cross-validation to tune the alpha and power hyperparameters (alpha
$\in\\{250,500,750,1000\\}$, power $\in\\{1,2,3\\}$) with 240 training
samples, 60 validation samples, and 138 test samples. Since this method
provides a segmentation label for every discovered region of an image, we
measure performance by taking the label that maximizes the Dice score on a per
image basis, and accumulate these scores across all images on the test set.

##  6 Results

###  6.1 Kidney Segmentation Performance

In Figure 3 we show the kidney segmentation masks produced by TricycleGAN
compared with the clinician-provided ground truth for randomly selected images
in the test set. In Table 1 we show the corresponding segmentation performance
metrics. TricycleGAN trained in a semi-supervised manner using synthetic
images and 45 real images and results obtained by first training the model on
synthetic data and then fine-tuning it on 45 real images are listed under
semi-supervised results TricycleGAN and TricycleGAN+10 respectively (with
“+10” denoting 10% of the labelled data used for fine-tuning; the amount
required before performance plateaued).

Figure 3: Kidney segmentation masks comparingTrycicleGAN (blue) to the clinician-provided ground truth labels (red) for a random subset of test images. Table 1: Performance metrics for ultrasound kidney segmentation. | Model | F1 | Specificity | Sensitivity | IoU | pACC
---|---|---|---|---|---|---
Unsup. | TricycleGAN | 0.81 (0.10) | 0.93 (0.08) | 0.84 (0.14) | 0.69 (0.13) | 0.90 (0.06)
SegCM | 0.48 | 0.31 | 0.91 | 0.31 | 0.47
W-net | 0.46 (0.10) | 0.20 (0.05) | 0.98 (0.02) | 0.30 (0.12) | 0.41 (0.07)
Semi-Sup. | TricycleGAN | 0.87 (0.11) | 0.97 (0.04) | 0.86 (0.13) | 0.78 (0.13) | 0.93 (0.05)
TricycleGAN+10 | 0.88 (0.08) | 0.97 (0.03) | 0.88 (0.09) | 0.80 (0.11) | 0.94 (0.04)
Sup. | U-net | 0.91 (0.09) | 0.97 (0.04) | 0.90 (0.10) | 0.84 (0.10) | 0.95 (0.03)

###  6.2 Skin Lesion Segmentation Performance

Performance metrics on the ISIC 2018 dataset using TricycleGAN are shown in
Table 2 along with results obtained by the competition winner [59] and current
top submission. Here we use the metrics given by the online submission system,
which includes a thresholded IoU (th-IoU). This metric sets all per-image IoU
scores that are less than 0.65 to 0 before computing the mean IoU. Examples of
the output masks on randomly selected test images are shown in Figure 4. As
with the kidney dataset, TricycleGAN+15 denotes model performance after
supervised fine-tuning of the unsupervised model with 15% of the labelled
validation data. However, the submission page for model evaluation on the test
set no longer provides metrics other than the th-IoU, so other metrics are not
presented for this model.

Table 2: Performance metrics for ISIC 2018 skin lesion boundary segmentation. | Model | th-IoU
---|---|---
Unsup. | TricycleGAN | 0.691
Semi-Sup. | TricycleGAN+15 | 0.759
Sup. | Mask-RCNN | 0.763
Winner [59] | 0.802
Current Top | 0.836
Figure 4: Skin lesion segmentation masks comparing unsupervised TricycleGAN
(blue) to a semi-supervised TricycleGAN+15 (red). Images are a random subset
taken from the ISIC 2018 test set, for which ground truth labels are not
available for download.

###  6.3 Supervised Fine-Tuning

We evaluate the effect of fine-tuning TricycleGAN with increasing amounts of
labelled data. The results presented in Figure 5 suggest that only 5-10% of
the validation images are needed for optimal performance on the kidney
ultrasound dataset, corresponding to 22-43 images. However, segmentation
performance continues to improve on the ISIC 2018 dataset with additional
data.

---

(a) Kidney Data

(b) ISIC 2018 Data

Figure 5: Result of fine-tuning the unsupervised models with increasing
amounts of the labelled validation data.

##  7 Discussion

We present TricycleGAN as way of creating labelled synthetic medical images
from unlabelled data to enable unsupervised image segmentation. TricycleGAN
takes advantage of recent advances in image synthesis and generative modelling
by making use of shape priors that can be used to construct templates of an
object of interest. This is accomplished by extending the recently developed
CycleGAN architecture to a network comprised of three generators that
translate between images, their edge maps, and their segmentation maps.

TricycleGAN performs better than alternative unsupervised methods that do not
require colour and texture features. For example, W-net performs poorly on the
kidney segmentation task because it only identifies the ultrasound cone
itself, rather than the kidney. We also show that our approach performs nearly
as well as supervised methods for most images, though overall performance is
diminished. Importantly, we show that with just a few training examples for
supervised fine-tuning (here, only 1̃0% of the data used for the supervised
models), TricycleGAN performs close to the level of supervised models.

## References

  * [1] M. Ravi and R. S. Hegadi, “Pathological medical image segmentation: A quick review based on parametric techniques,” _Medical Imaging: Artificial Intelligence, Image Recognition, and Machine Learning Techniques_ , p. 207, 2019\.
  * [2] N. Tajbakhsh, L. Jeyaseelan, Q. Li, J. Chiang, Z. Wu, and X. Ding, “Embracing imperfect datasets: A review of deep learning solutions for medical image segmentation,” _arXiv preprint arXiv:1908.10454_ , 2019.
  * [3] Y. Guo, Y. Liu, T. Georgiou, and M. S. Lew, “A review of semantic segmentation using deep neural networks,” _International journal of multimedia information retrieval_ , vol. 7, no. 2, pp. 87–93, 2018.
  * [4] C. L. Chowdhary and D. Acharjya, “Segmentation and feature extraction in medical imaging: a systematic review,” _Procedia Computer Science_ , vol. 167, pp. 26–36, 2020.
  * [5] J.-Y. Zhu, T. Park, P. Isola, and A. A. Efros, “Unpaired image-to-image translation using cycle-consistent adversarial networks,” in _Proceedings of the IEEE international conference on computer vision_ , 2017, pp. 2223–2232.
  * [6] J. Canny, “A computational approach to edge detection,” in _Readings in computer vision_. Elsevier, 1987, pp. 184–203.
  * [7] R. Adams and L. Bischof, “Seeded region growing,” _IEEE Transactions on pattern analysis and machine intelligence_ , vol. 16, no. 6, pp. 641–647, 1994\.
  * [8] M. Kass, A. Witkin, and D. Terzopoulos, “Snakes: Active contour models,” _International journal of computer vision_ , vol. 1, no. 4, pp. 321–331, 1988\.
  * [9] B. Manjunath and R. Chellappa, “Unsupervised texture segmentation using markov random field models,” _IEEE Transactions on Pattern Analysis & Machine Intelligence_, vol. 5, pp. 478–482, 1991.
  * [10] A. Voulodimos, N. Doulamis, A. Doulamis, and E. Protopapadakis, “Deep learning for computer vision: A brief review,” _Computational intelligence and neuroscience_ , vol. 2018, 2018.
  * [11] H.-J. Yoo, “Deep convolution neural networks in computer vision,” _IEEE Transactions on Smart Processing & Computing_, vol. 4, no. 1, pp. 35–43, 2015\.
  * [12] A. Garcia-Garcia, S. Orts-Escolano, S. Oprea, V. Villena-Martinez, and J. Garcia-Rodriguez, “A review on deep learning techniques applied to semantic segmentation,” _arXiv preprint arXiv:1704.06857_ , 2017.
  * [13] L. Lu, Y. Zheng, G. Carneiro, and L. Yang, “Deep learning and convolutional neural networks for medical image computing,” _Advances in Computer Vision and Pattern Recognition; Springer: New York, NY, USA_ , 2017.
  * [14] J. Long, E. Shelhamer, and T. Darrell, “Fully convolutional networks for semantic segmentation,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2015, pp. 3431–3440.
  * [15] K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” _arXiv preprint arXiv:1409.1556_ , 2014.
  * [16] G. Lin, C. Shen, A. Van Den Hengel, and I. Reid, “Efficient piecewise training of deep structured models for semantic segmentation,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2016, pp. 3194–3203.
  * [17] P. Moeskops, J. M. Wolterink, B. H. van der Velden, K. G. Gilhuijs, T. Leiner, M. A. Viergever, and I. Išgum, “Deep learning for multi-task medical image segmentation in multiple modalities,” in _International Conference on Medical Image Computing and Computer-Assisted Intervention_. Springer, 2016, pp. 478–486.
  * [18] W. Zhang, R. Li, H. Deng, L. Wang, W. Lin, S. Ji, and D. Shen, “Deep convolutional neural networks for multi-modality isointense infant brain image segmentation,” _NeuroImage_ , vol. 108, pp. 214–224, 2015.
  * [19] J. Kleesiek, G. Urban, A. Hubert, D. Schwarz, K. Maier-Hein, M. Bendszus, and A. Biller, “Deep MRI brain extraction: a 3D convolutional neural network for skull stripping,” _NeuroImage_ , vol. 129, pp. 460–469, 2016.
  * [20] Q. Dou, H. Chen, L. Yu, L. Zhao, J. Qin, D. Wang, V. C. Mok, L. Shi, and P.-A. Heng, “Automatic detection of cerebral microbleeds from MR images via 3D convolutional neural networks,” _IEEE transactions on medical imaging_ , vol. 35, no. 5, pp. 1182–1195, 2016.
  * [21] T. Brosch, L. Y. Tang, Y. Yoo, D. K. Li, A. Traboulsee, and R. Tam, “Deep 3D convolutional encoder networks with shortcuts for multiscale feature integration applied to multiple sclerosis lesion segmentation,” _IEEE transactions on medical imaging_ , vol. 35, no. 5, pp. 1229–1239, 2016.
  * [22] D. C. Cireşan, A. Giusti, L. M. Gambardella, and J. Schmidhuber, “Mitosis detection in breast cancer histology images with deep neural networks,” in _International Conference on Medical Image Computing and Computer-assisted Intervention_. Springer, 2013, pp. 411–418.
  * [23] O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” in _International Conference on Medical image computing and computer-assisted intervention_. Springer, 2015, pp. 234–241.
  * [24] P. F. Christ, M. E. A. Elshaer, F. Ettlinger, S. Tatavarty, M. Bickel, P. Bilic, M. Rempfler, M. Armbruster, F. Hofmann, M. D’Anastasi _et al._ , “Automatic liver and lesion segmentation in CT using cascaded fully convolutional neural networks and 3D conditional random fields,” in _International Conference on Medical Image Computing and Computer-Assisted Intervention_. Springer, 2016, pp. 415–423.
  * [25] Ö. Çiçek, A. Abdulkadir, S. S. Lienkamp, T. Brox, and O. Ronneberger, “3D U-Net: learning dense volumetric segmentation from sparse annotation,” in _International conference on medical image computing and computer-assisted intervention_. Springer, 2016, pp. 424–432.
  * [26] O. Oktay, J. Schlemper, L. L. Folgoc, M. Lee, M. Heinrich, K. Misawa, K. Mori, S. McDonagh, N. Y. Hammerla, B. Kainz _et al._ , “Attention u-net: learning where to look for the pancreas,” _arXiv preprint arXiv:1804.03999_ , 2018.
  * [27] V. Iglovikov and A. Shvets, “Ternausnet: U-net with vgg11 encoder pre-trained on imagenet for image segmentation,” _arXiv preprint arXiv:1801.05746_ , 2018\.
  * [28] M. Havaei, A. Davy, D. Warde-Farley, A. Biard, A. Courville, Y. Bengio, C. Pal, P.-M. Jodoin, and H. Larochelle, “Brain tumor segmentation with deep neural networks,” _Medical image analysis_ , vol. 35, pp. 18–31, 2017.
  * [29] K. Kamnitsas, C. Ledig, V. F. Newcombe, J. P. Simpson, A. D. Kane, D. K. Menon, D. Rueckert, and B. Glocker, “Efficient multi-scale 3d cnn with fully connected crf for accurate brain lesion segmentation,” _Medical image analysis_ , vol. 36, pp. 61–78, 2017.
  * [30] S. Pereira, A. Pinto, V. Alves, and C. A. Silva, “Brain tumor segmentation using convolutional neural networks in mri images,” _IEEE transactions on medical imaging_ , vol. 35, no. 5, pp. 1240–1251, 2016.
  * [31] X. Xia and B. Kulis, “W-net: A deep model for fully unsupervised image segmentation,” _arXiv preprint arXiv:1711.08506_ , 2017.
  * [32] K.-J. Hsu, Y.-Y. Lin, and Y.-Y. Chuang, “Deepco3: Deep instance co-segmentation by co-peak search and co-saliency detection,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 8846–8855.
  * [33] P. Isola, J.-Y. Zhu, T. Zhou, and A. A. Efros, “Image-to-image translation with conditional adversarial networks,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 1125–1134.
  * [34] P. Luc, C. Couprie, S. Chintala, and J. Verbeek, “Semantic segmentation using adversarial networks,” _arXiv preprint arXiv:1611.08408_ , 2016.
  * [35] Y. Xue, T. Xu, H. Zhang, L. R. Long, and X. Huang, “SegAN: Adversarial network with multi-scale L1 loss for medical image segmentation,” _Neuroinformatics_ , vol. 16, no. 3-4, pp. 383–392, 2018.
  * [36] J. Son, S. J. Park, and K.-H. Jung, “Retinal vessel segmentation in fundoscopic images with generative adversarial networks,” _arXiv preprint arXiv:1706.09318_ , 2017.
  * [37] D. Yang, D. Xu, S. K. Zhou, B. Georgescu, M. Chen, S. Grbic, D. Metaxas, and D. Comaniciu, “Automatic liver segmentation using an adversarial image-to-image network,” in _International Conference on Medical Image Computing and Computer-Assisted Intervention_. Springer, 2017, pp. 507–515.
  * [38] S. Hong, H. Noh, and B. Han, “Decoupled deep neural network for semi-supervised semantic segmentation,” in _Advances in neural information processing systems_ , 2015, pp. 1495–1503.
  * [39] H.-C. Shin, N. A. Tenenholtz, J. K. Rogers, C. G. Schwarz, M. L. Senjem, J. L. Gunter, K. P. Andriole, and M. Michalski, “Medical image synthesis for data augmentation and anonymization using generative adversarial networks,” in _International Workshop on Simulation and Synthesis in Medical Imaging_. Springer, 2018, pp. 1–11.
  * [40] N. Souly, C. Spampinato, and M. Shah, “Semi supervised semantic segmentation using generative adversarial network,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2017, pp. 5688–5696.
  * [41] J. T. Guibas, T. S. Virdi, and P. S. Li, “Synthetic medical images from dual generative adversarial networks,” _arXiv preprint arXiv:1709.01872_ , 2018\.
  * [42] P. Ostyakov, R. Suvorov, E. Logacheva, O. Khomenko, and S. I. Nikolenko, “Seigan: Towards compositional image generation by simultaneously learning to segment, enhance, and inpaint,” _arXiv preprint arXiv:1811.07630_ , 2019\.
  * [43] M. Chen, T. Artières, and L. Denoyer, “Unsupervised object segmentation by redrawing,” _arXiv preprint arXiv:1905.13539_ , 2019.
  * [44] K. Dhindsa, L. C. Smail, M. McGrath, L. H. Braga, S. Becker, and R. R. Sonnadara, “Grading prenatal hydronephrosis from ultrasound imaging using deep convolutional neural networks,” in _2018 15th Conference on Computer and Robot Vision (CRV)_. IEEE, 2018, pp. 80–87.
  * [45] U. Sivanesan, L. H. Braga, R. R. Sonnadara, and K. Dhindsa, “Unsupervised medical image segmentation with adversarial networks: From edge diagrams to segmentation maps,” _arXiv preprint arXiv:1911.05140_ , 2019.
  * [46] L. C. Smail, K. Dhindsa, L. H. Braga, S. Becker, and R. R. Sonnadara, “Using deep learning algorithms to grade hydronephrosis severity: Toward a clinical adjunct,” _Frontiers in Pediatrics_ , vol. 8, 2020.
  * [47] N. Codella, V. Rotemberg, P. Tschandl, M. E. Celebi, S. Dusza, D. Gutman, B. Helba, A. Kalloo, K. Liopyris, M. Marchetti _et al._ , “Skin lesion analysis toward melanoma detection 2018: A challenge hosted by the international skin imaging collaboration (isic),” _arXiv preprint arXiv:1902.03368_ , 2019.
  * [48] P. Tschandl, C. Rosendahl, and H. Kittler, “The ham10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions,” _Scientific data_ , vol. 5, p. 180161, 2018.
  * [49] P. C. Tay, C. D. Garson, S. T. Acton, and J. A. Hossack, “Ultrasound despeckling for contrast enhancement,” _IEEE Transactions on Image Processing_ , vol. 19, no. 7, pp. 1847–1860, 2010.
  * [50] X. Zhou, C. Yao, H. Wen, Y. Wang, S. Zhou, W. He, and J. Liang, “East: an efficient and accurate scene text detector,” in _Proceedings of the IEEE conference on Computer Vision and Pattern Recognition_ , 2017, pp. 5551–5560.
  * [51] S. M. Pizer, E. P. Amburn, J. D. Austin, R. Cromartie, A. Geselowitz, T. Greer, B. ter Haar Romeny, J. B. Zimmerman, and K. Zuiderveld, “Adaptive histogram equalization and its variations,” _Computer vision, graphics, and image processing_ , vol. 39, no. 3, pp. 355–368, 1987.
  * [52] Y. Liu, M.-M. Cheng, X. Hu, K. Wang, and X. Bai, “Richer convolutional features for edge detection,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 3000–3009.
  * [53] P. Dollár and C. L. Zitnick, “Fast edge detection using structured forests,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 37, no. 8, pp. 1558–1570, 2014.
  * [54] P. Liu, X. Qi, P. He, Y. Li, M. R. Lyu, and I. King, “Semantically consistent image completion with fine-grained details,” _arXiv preprint arXiv:1711.09345_ , 2017.
  * [55] J.-Y. Zhu, P. Krähenbühl, E. Shechtman, and A. A. Efros, “Generative visual manipulation on the natural image manifold,” in _European conference on computer vision_. Springer, 2016, pp. 597–613.
  * [56] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial nets,” in _Advances in neural information processing systems_ , 2014, pp. 2672–2680.
  * [57] N. Abraham and N. M. Khan, “A novel focal tversky loss function with improved attention u-net for lesion segmentation,” in _2019 IEEE 16th International Symposium on Biomedical Imaging (ISBI 2019)_. IEEE, 2019, pp. 683–687.
  * [58] K. He, G. Gkioxari, P. Dollár, and R. Girshick, “Mask r-cnn,” in _Proceedings of the IEEE international conference on computer vision_ , 2017, pp. 2961–2969.
  * [59] C. Qian, T. Liu, H. Jiang, Z. Wang, P. Wang, M. Guan, and B. Sun, “A detection and segmentation architecture for skin lesion segmentation on dermoscopy images,” _arXiv preprint arXiv:1809.03917_ , 2018.
  * [60] P. Arbelaez, M. Maire, C. Fowlkes, and J. Malik, “Contour detection and hierarchical image segmentation,” _IEEE Trans. Pattern Anal. Mach. Intell._ , vol. 33, no. 5, pp. 898–916, May 2011.
  * [61] I. Aganj, M. G. Harisinghani, R. Weissleder, and B. Fischl, “Unsupervised medical image segmentation based on the local center of mass,” _Scientific reports_ , vol. 8, no. 1, pp. 1–8, 2018.

## Supplementary Data

|
---|---
|

(a) Grade 1

(b) Grade 2

(c) Grade 3

(d) Grade 4

Figure S1: Examples from the kidney ultrasound dataset with different
hydronephrosis severity grades, from 1 (low severity) to 4 (severe
hydronephrosis).  Figure S2: Examples of generated skin lesions. From left to
right: synthetic segmentations, generated edge maps, followed by 12 style
variations generated by applying 12 different styles with TricycleGAN
generator $G^{e2m}$.