DKE-21-38 Artificial Intelligence Dr. E. Hortal Quesada
Dr. S. Asteriadis
Dr. I. Zhovannik

# Exploring constraints on CycleGAN-based CBCT enhancement for adaptive
radiotherapy

B. S. Pai

###### Abstract

Research exploring CycleGAN-based synthetic image generation has recently
accelerated in the medical community, as it is able to leverage unpaired
datasets effectively. However, clinical acceptance of these synthetic images
pose a significant challenge as they are subject to strict evaluation
protocols. A commonly established drawback of the CycleGAN, the introduction
of artifacts in generated images is unforgivable in the case of medical
images. In an attempt to alleviate this drawback, we explore different
constraints of the CycleGAN along with investigation of adaptive control of
these constraints. The benefits of imposing additional constraints on the
CycleGAN, in the form of structure retaining losses is also explored. A
generalized frequency loss inspired by Jiang et al., (2020) that preserves
content in the frequency domain between source and target is investigated and
compared with existing losses such as the MIND loss (Yang et al.,, 2018).
Synthetic images generated from our methods are quantitatively and
qualitatively investigated and outperform the baseline CycleGAN and other
approaches. Furthermore, no observable artifacts or loss in image quality is
found, which is critical for acceptance of these synthetic images. The
synthetic medical images thus generated are also evaluated using domain-
specific evaluation and using segmentation as a downstream task, in order to
clearly highlight their applicability to clinical workflows.

###### Contents

  1. 1 Introduction
    1. 1.1 Medical Imaging and Adaptive Radiotherapy
      1. 1.1.1 Significance of CT and CBCT imaging
    2. 1.2 Image quality improvement for adaptive radiotherapy
      1. 1.2.1 Deep Learning approaches
    3. 1.3 Cycle-consistent Generative Adversarial Networks
      1. 1.3.1 Generative Adversarial Networks
      2. 1.3.2 Image-to-Image translation using GANs
      3. 1.3.3 Constraints enforced on a CycleGAN
    4. 1.4 Evaluation of CycleGAN-based methods
    5. 1.5 Clinical value and impact
    6. 1.6 Contributions
  2. 2 Methods and Materials
    1. 2.1 Dataset
      1. 2.1.1 Map $\leftrightarrow$ aerial photo Dataset
      2. 2.1.2 Radiotherapy treatment dataset with CBCT and CT images
    2. 2.2 Data Pre-processing
      1. 2.2.1 Map $\leftrightarrow$ aerial photo  Dataset
      2. 2.2.2 CBCT and CT Scans
    3. 2.3 Adaptive Control of CycleGAN Constraints
    4. 2.4 Structure-consistency Losses
      1. 2.4.1 Modality Independent Neighbourhood Descriptor  (MIND) Loss
      2. 2.4.2 Registration Metric Losses
      3. 2.4.3 Generalized Frequency Loss
    5. 2.5 Evaluation
      1. 2.5.1 Image Similarity Metrics
      2. 2.5.2 Qualitative Inspection
      3. 2.5.3 Out-of-distribution Analysis
      4. 2.5.4 Domain-specific Evaluation
      5. 2.5.5 Validation and Selection of Best Checkpoints
  3. 3 Experiments and Results
    1. 3.1 Experimental Setup
      1. 3.1.1 Network Architectures
    2. 3.2 Exploring Default CycleGAN Constraints
      1. 3.2.1 Results
    3. 3.3 Adapting Constraints For Improved Translation
      1. 3.3.1 Results
    4. 3.4 Data-driven Constraints for CBCT Image Quality Enhancement
      1. 3.4.1 Results
    5. 3.5 Domain-specific Evaluation
      1. 3.5.1 Histogram and Line Profiles
      2. 3.5.2 Automated Segmentation Task
  4. 4 Discussion
  5. 5 Conclusion

##  Chapter 1 Introduction

###  1.1 Medical Imaging and Adaptive Radiotherapy

Medical imaging plays a key role in providing a simulation of the human body
through imaging techniques for aiding multiple medical procedures such as
diagnosis, prognosis and treatment. Many different imaging techniques form a
part of medical imaging aiding radiologists/clinicians to gain additional
information that may not be available solely with the naked eye. This
additional information can provide key insight to a radiologist for decision
making (Doi,, 2006). More specifically for oncology, medical imaging is
essential in order to detect cancer earlier and provide improved outcomes to
the patient.

Several medical imaging techniques (modalities) are used for diagnosis of
cancer such as Computed Tomography (CT), Positron Emission Tomograpy (PET),
Magnetic Resonance Imaging (MRI), etc. Each of these modalities have unique
properties and provide specific information due to the physics of their
acquisition. These can then be leveraged accordingly to support different
stages of decision-making. Medical Imaging also plays a key role in treatment
of cancer where different modalities can be used to design a course of action.
Surgery, chemotherapy and radiotherapy can all benefit by being driven through
imaging (Frangioni,, 2008).

##### Radiation Therapy

Radiotherapy is a method of treatment of cancer where doses of radiation are
targeted at the tumour in order to stop its growth and kill tumour cells.
While delivering dose to the tumour, healthy organs need to be spared in order
to avoid causing damage to them which makes the design of radiotherapy
treatment a challenging task. Radiotherapy can be either

  1. 1.

External Beam Radiotherapy (EBRT): A machine placed externally delivers beams
of radiation targeted at the tumour.

  2. 2.

Internal Radiotherapy (IRT): A radioactive source is placed inside the body
and delivers radiation internally.

EBRT consists of a large set of techniques that can rely on different types of
beams such as proton or photon beams. 3D conformal radiotherapy relies on
medical imaging to deliver radiation to the tumour by targeting beams at it
from multiple directions. Intensity Modulated Radiotherapy (IMRT) is a more
advanced technique where said beams are shaped more precisely to conform to
the shape of a tumour. In addition to precisely shaping beams, radiation
therapy can also be improved by leveraging medical images not just while
planning but also during treatment delivery in a method called Image Guided
Radiotherapy (IGRT). In an IGRT workflow, images are used during treatment
delivery to position the patient accurately or to change the radiation dose.
IGRT can help mitigate errors induced during initial imaging during planning
and account for treating sites where motion might be common such as the lung
(Ling et al.,, 2006). More recently, the capturing of these guidance images
have allowed adapting radiotherapy plans during the process of treatment.

##### Adaptive Radiotherapy

Adaptive Radiotherapy (ART) is a set of emerging data-driven techniques that
aims to administer more accurate treatment plans that account for changes
observed during the process of treatment planning and delivery. Multiple
factors can lead to changes being observed during the treatment process such
as,

  * •

Geometrical uncertainties:
These are uncertainties are caused by both systemic and random errors.
Systemic errors such as delineation differences, setup errors and organ motion
can cause reduction in accuracies of the RT process.

  * •

Anatomical changes:
Tumour regression along with changes in the size/shape of the different organs
over the course of treatment can lead to differences in ideal dose versus the
one estimated during initial planning.

  * •

Biological response:
Changes in the biology of the tumour can also occur during treatment which may
contribute to changes in toxicities for surrounding organs at risk.

Sonke and Belderbos, (2010) discuss the above change inducing factors in
detail. They propose that adaptive methods in RT can help account for these
changes to a certain extent by repetitive imaging and analysis.

##### ART Process Workflow

Figure 1.1 shows a high level overview of the treatment planning and delivery
process with integration of adaptive radiotherapy. The treatment planning
process begins by the acquisition of a CT scan, followed by contouring of the
target (tumour, lymph nodes and other malignant locations) and organs-at-risk
(OAR) by radiation therapists. The CT scan along with contours are used to
generate treatment plans and dose distributions that can then be delivered
using a linear accelerator. The treatment plan is generally designed, such
that, it is administered across different fractions or sittings. At each
fraction, a fractional dose is delivered to the patient. When IGRT-based
methods are leveraged, on-board imaging such as CBCT or MRI is used to
position the patient accurately before delivering the prescribed dose. These
modalities are preferred as they provide either minimal or no radiation to the
patient. This process is repeated across multiple fractions. Adaptive
radiotherapy, as shown in the green block in Figure 1.1, suggests using on-
board imaging data to adapt/re-plan treatment as required. This allows
improving treatment outcomes by providing minimal overdosing to healthy
tissues while concentrating maximum dose to the tumour. However, on-boarding
imaging

Figure 1.1: ART Process Workflow showing different processes of treatment
planning, treatment delivery and adaptive radiotherapy.

####  1.1.1 Significance of CT and CBCT imaging

Computed Tomography (CT) was the first method that provided non-invasive
imaging of the human body without being biased by overlapping structures owing
to its 3D nature (Buzug,, 2011). CT imaging uses X-ray radiation aimed at the
body in a fan-like shape falling on a line detector to generate 3D cross-
sectional images. The intensities within these cross-sectional images are
dependent on the density of bone, tissues and organs within the body and
therefore allow the observing the make-up of the body when stacked. These
densities are represented as Hounsfield Units (HU) in a CT scan. In addition
to providing beneficial visual information, CT scans can be used for planning
treatment as the modelling of densities allow gauging how the same objects
might respond to radiation.

Cone-beam Computed Tomography (CBCT) imaging uses a cone-shaped X-ray beam
with a flat panel detector. Unlike conventional CT, CBCT generates 3D images
by projecting cone-beams at different angles and reconstructing a 3D image
using projection geometry. This makes the image more susceptible to scattering
and noise when compared to a CT. However, due to smaller footprint and lower
dose exposure of CBCT scanning systems, they have been integrated quite
successfully as onboard imaging on linear accelerators (Sterzing et al.,,
2011). Both CT and CBCT images are crucial to the process of radiation
therapy. CT allows generating treatment plans via the acquistion of a planning
CT while CBCT allows efficient delivery by more accurate positioning and
correction through IGRT. CBCT can also play a key role in adaptive
radiotherapy providing better outcomes for the patient. Figure 1.2 shows the
acquisition physics of CBCT and CT scanners. A sample of resulting scans from
these scanners are show in Figure 1.3.

Figure 1.2: Acquisition physics of fan-beam and cone-beam CT. In fan beam CT,
the source emits a fan-shaped X-ray beam on the object which is then collected
by a line-detector. The source moves along the vertical axes capturing cross
sectional imagery. In cone-beam CT, the source emits a cone-shaped beam that
falls on a flat panel detector. The source revolves around the object and
captures projections at different angles. Figure 1.3: Mid-axial views of a
patient scanned using cone-beam CT (CBCT) and fan-beam CT (CT). A rough glance
at these images shows that the cone-beam image is much noisier with streaks
dominant across the image. The Hounsfield Units (HU) of the cone-beam CT scan
are not standardized either. Water, which would have a standard HU value of 0
in a fan-beam CT might not present the same value in a cone-beam CT.

###  1.2 Image quality improvement for adaptive radiotherapy

CBCT images are are acquired with about an order of magnitude lesser radiation
than conventional fan-beam CTs (Schulze et al.,, 2011). This renders them more
susceptible to sources of noise that already affect CT imaging (Lechuga and
Weidlich,, 2016). In addition, physics of CBCT acquisition introduce sources
of noise that fan-beam CTs generally do not face such as increased scatter due
to usage of flat panel detectors. Combining both these factors, CBCT images
often present with lower quality. Schulze et al., (2011) present a
comprehensive overview of the different artefacts that affect the quality of a
CBCT scan. Along with affecting the visual quality, these artefacts can also
contribute to the dosimetric accuracy of using a CBCT scan.

Posiewnik and Piotrowski, (2019) discuss CBCT image quality and highlight two
groups of issues that can prove adaptive radiotherapy with CBCT challenging.
The first group consists of image quality degradation that manifests as
reduction of contrast, obscuring structures and impairment of regions of
interest. This is mainly due to the various artefacts outlined by Schulze et
al., (2011). The second group of issues pertain to parameters that determine
the CBCT acquisition. Standardized guidelines and QA procedures can lead to
acceptable CBCT image quality whereas deviation from these can cause adverse
effects.

Dose calculation capability or dosimetric quality of CBCT imaging is integral
in determining its value in ART. Dosimetric quality is defined as the ability
of the image to produce accurate dose distributions when used for treatment
planning. Therefore, dosimetric quality similar to fan-beam CT imaging would
be ideal with increasing deviations affecting the efficacy of ART procedures.
Schulze et al., (2011) highlight multiple studies that show that dose
calculated on CBCT vs CT present unacceptable deviations when HU calibration
is done through look-up tables.

Considerable research has been presented on the benefit of processing CBCT
images to improve their quality and reduce artefacts with traditional methods
such as scatter correction (Jin et al.,, 2010), density overrides (Dunlop et
al.,, 2015), CT number calibration (Dunlop et al.,, 2015), deformable image
registration (Landry et al.,, 2015) and certain model-based methods (Zhao et
al.,, 2016). With the democratization of machine learning and deep learning,
multiple studies have also presented ML/DL-based methods to post-process CBCT
images. These methods are often much quicker and less cumbersome in contrast
to traditional methods and have found wide acceptance in the medical imaging
research community. However, in-terms of clinical implementation, the
vulnerability of ML/DL-based methods in dealing with out-of-distribution data
has been a limiting factor. In the section below, a short review of some deep
learning approaches along with their benefits and limitations is discussed.

####  1.2.1 Deep Learning approaches

Deep learning-based approaches for CBCT improvement have mostly revolved
around encoder-decoder networks or generative-adversarial networks where voxel
regression is performed with the original CBCT image as input. The vast
majority of such approaches can be categorized into either paired or unpaired.

##### Paired approaches

Paired approaches require input images with ground truth that denotes what the
ideal output would be. This is generally a highly matching CT image as CBCT
improvement is synonymous with translation to a CT like quality. A CBCT image
along with its highly matching CT can be collected in various ways - planning
CT and first positioning CBCT might provide required correspondences. CTs can
also be taken during the treatment delivery process, called rescanned CTs, and
generally correspond well with CBCTs taken for positioning around that time.
The CBCT and its matching CT image need to be co-registered to align anatomies
accounting for differences in scan parameters and movement. Upon obtaining
images with GT pairs, these can then be used in a fully-supervised fashion.

Paired approaches using encoder-decoder networks, more specifically using
UNets (Ronneberger et al.,, 2015) have been presented by multiple studies
(Kida et al.,, 2018; Landry et al.,, 2019; Yuan et al.,, 2020; Thummerer et
al.,, 2020). Kida et al., (2018) train their model with 20 patients in a 2D
fashion using a 39-layer UNet architecture. They show that their methods
improve the image and dosimetric quality through better quantitative scores
for metrics such as Structural Similarity Index (Wang et al.,, 2004), Power-
to-Signal Noise Ratio, ROI Mean values and Spatial Non-Uniformity. Landry et
al., (2019) compare three UNets trained with different input representations —
in projection and image spaces — and aim to identify the best approach for
dosimetric accuracy. They use L2 norm between the target and predicted images
in contrast to MAE (Kida et al.,, 2018). They show promising results in both
projection and image spaces concluding that both are feasible methods for CBCT
correction. Yuan et al., (2020) present an approach using three cross-
sectional slices as 3-channel inputs to a 2 dimensional UNet. The authors
extensively validate their approaches through group-based cross-validation and
show large improvements in image quality across all their test studies.
Thummerer et al., (2020) train each 2D network considering different planes —
axial, sagittal and coronal — of the 3D CBCT-CT scan pair followed by
aggregation during test-time. Although they show the efficacy of their methods
compared to traditional approaches, they do not present comparisons of single
plane vs their multi-planar approach.

Image-to-image translation community placed a lot of importance on designing
handcrafted losses between target and predicted image spaces in order to
integrate domain knowledge and preserve relevant features in the predicted
image. With the introduction of Pix2pix GAN (Isola et al.,, 2016), a
discriminator was designed to substitute complex loss design and preserve high
frequency information. CBCT to CT translation was also benefit by addition of
a discriminator. Zhang et al., (2020) compare multiple deep learning
approaches from UNet encoder-decoders to CycleGANs and show that their pix2pix
GAN outperforms other methods which shows a clear benefit of the pix2pix
framework. Do note that their CycleGAN was trained using paired data and does
not leverage unpaired data. Another interesting work by Dahiya et al., (2021)
show how physics based data-augmentation can be used to create paired data
which can then be leveraged in a pix2pix framework. In addition to improved
CBCT, they also generate orgam segmentations for the CBCT image.

Analysis of the above studies shows that paired approaches are feasible
candidates for image quality improvement. However, the pre-processing needed
to obtain paired data might pose a hindrance in utilizing all available data
efficiently. During treatment, multiple CBCTs are acquired but, generally,
only a single pair can be formed when matching with a planning CT. Usage of
rescanned CTs need additional scans to be taken which might be inconvenient
for the patient due to increased cost, radiation and time. Moreover, the
process of pairing the data might also introduce biases such as dependence on
method/quality of registration chosen. These impeding factors along with the
emergence of unpaired approaches such as CycleGAN has lead the research
community to lean toward unpaired approaches.

##### Unpaired approaches

These approaches are designed to handle data where matching characteristics
may be hard to obtain. For example, in the task of converting horses to
zebras, it is almost impossible to have a known ground truth of what a horse
might look like if it were a zebra. We can consider a set of horse images to
form the source data space and a set of zebra images to form the target data
space. This is where unpaired approaches enter and leverage the properties of
these data spaces and allow translation between them without enforcing strict
matches.

The CycleGAN framework introduced by (Zhu et al.,, 2017) is one of the most
consistently used unpaired approaches for image-to-image translation. Kurz et
al., (2019) present an approach using a 2D CycleGAN where co-registered slices
are used as inputs. Their methods show very good correspondence with an
existing CBCT correction method, both image and dosimetry-wise, while being
much faster. Maspero et al., (2020) use limited field-of-view CBCTs and
rescanned CTs in a purely unpaired fashion across three different anatomical
sites. A single network trained on all sites along with individually trained
networks are explored in their work. Both approaches perform similarly and
show a large improvement in image similarity metrics when evaluated with a set
of ground truth pairs. They additionally show that the improved CBCT is of
sufficient dosimetric quality through dose differences and gamma analysis. Liu
et al., (2020) demonstrate the use of attention gates in CycleGANs and show
that their introduction improves smoothness and reduces artifacts when
compared with a UNet and a vanilla CycleGAN. Various other studies by Harms et
al., (2019); Eckl et al., (2020); Kida et al., (2020) show the benefit of
CycleGAN approaches in CBCT improvement for both visual and dosimetric tasks.

###  1.3 Cycle-consistent Generative Adversarial Networks

In this section, an overview of Generative Adversarial Networks (GAN) is
provided followed by an in-depth description of a variant of GAN for image-to-
image translation in unpaired settings - the Cycle-consistent Generative
Adversarial Network or CycleGAN. Constraints that are set in a CycleGAN are
discussed and studies that discuss additional constraints are presented.

####  1.3.1 Generative Adversarial Networks

GANs are a category of generative models that are trained in an adversarial
fashion lending it the name. The generative aspect of a GAN is through a
generative network called the generator ($G$) while the adversarial aspect is
through a discriminative network called the disciminator ($D$). In the
original paper (Goodfellow et al.,, 2014), the authors define $G$ as a
function with parameters $\theta_{g}$ and construct a mapping
$G(z;\theta_{g})$ where $z$ is a sampled from a noise distribution. The
discriminator $D$ has parameters $\theta_{d}$ and applies the mapping
$D(x;\theta_{d})$ to an input $x$ providing a scalar output.$x$ comes from
either the data distribution or the generated distribution. The goal of a GAN
is to train both $D$ and $G$ simultaneously such that $D$ learns to maximize
the probability of identifying whether an input $x$ comes from data
distribution or the generated distribution while G tries to minimize this
probability while generating samples. This is formally presented as,

$\displaystyle L_{adversarial}(G,D)$ $\displaystyle=\mathbb{E}_{x\sim
p_{data}}[log(D(x))]$ (1.1) $\displaystyle+\quad\mathbb{E}_{z\sim
p_{generated}}[log(1-D(G(z)))]$

Equation 1.1 shows the combined objective for the generator and discriminator
where $p_{data}$ and $p_{generated}$ are the data and generated distributions
respectively. The generator tries to minimize this objective while the
discriminator tries to maximize it, which is formulated as,

$\displaystyle G^{*}=\underset{G}{min}\underset{D}{max}L_{adversarial}(G,D)$
(1.2)

####  1.3.2 Image-to-Image translation using GANs

Generative-adversarial methods have also been extended for conditional data —
where a generated distribution is conditioned on additional information (Mirza
and Osindero,, 2014). Several modes of information such as text, labels and
images have been used as conditional information. Pix2pix (Isola et al.,,
2016) uses images as input in a conditional fashion to generate translations
of those images. Here the sampled noise $z$ is conditioned with an input $x$.
In addition to the adversarial loss $L_{adversarial}$, a L1 loss between the
generated and input conditional image is proposed. The combined loss is
presented as,

$\displaystyle L_{pix2pix}(G,D)$ $\displaystyle=L_{adversarial}(G,D)$ (1.3)
$\displaystyle+\quad\lambda\mathbb{E}_{x,y,z}[||y-G(x,z)||_{1}]$

The authors interestingly show that the noise $z$ does not affect the GAN and
can be completely eliminated leading to providing only $x$ sampled from the
real data distribution as input. The concept of conditional GANs for image-to-
image translation is extended to unpaired settings through the CycleGAN
framework.

##### CycleGAN

The CycleGAN architecture consists of two sets of generator and discriminator
networks. Given images belonging to two domains $X$ and $Y$, the CycleGAN
attempts to learn a mapping from $X\rightarrow Y$ through a network $G$.
Discriminator $D_{Y}$ learns to differentiate if an image belongs to domain
$Y$ or not and drives the training of $G$. The concept of cycle-consistency is
enforced by learning the inverse mapping from $Y\rightarrow X$ through a
network $F$. Similar to $D_{Y}$, $D_{X}$ exists for the inverse mapping. After
mapping $X\rightarrow Y$ and $Y\rightarrow X$, the generated image is compared
with the original by means of a cycle-consistency loss that ensures accurate
reconstruction of the original image through the two mappings.

Figure 1.4 shows a diagram of the CycleGAN architecture for learning a mapping
from domain $X\rightarrow Y$. The adversarial loss is similar to Equation 1.1
but with image $x$ from domain $X$ as input instead of a noise vector $z$. The
cycle-consistency loss in the standard CycleGAN is a L1 loss between the input
and reconstructed image. In the original paper (Zhu et al.,, 2017), the
combined loss is formulated as,

$\displaystyle L_{CycleGAN}(G,F,D_{X},D_{Y})$
$\displaystyle=L_{adversarial}(G,D_{X})+L_{adversarial}(F,D_{Y})$ (1.4)
$\displaystyle+\quad\lambda_{A}\mathbb{E}_{x\sim
p_{data(x)}}[||F(G(x))-x||_{1}]$
$\displaystyle+\quad\lambda_{B}\mathbb{E}_{y\sim
p_{data(y)}}[||G(F(y))-y||_{1}]$

Figure 1.4: Block diagram of a CycleGAN architecture..

####  1.3.3 Constraints enforced on a CycleGAN

The authors of CycleGAN motivate the addition of a cycle-consistency loss by
stating that there might be infinitely many mappings G that can produce an
output distribution that is identified as belonging to the target domain. They
also note that mode-collapse is quite common when only adversarial mechanics
are used for training. They propose addition of structure to their training
objective through a cycle-consistency loss as formulated in Equation 1.4. The
nature of the cycle-consistency constraint and other additional constraints
that have been proposed in literature, are described in the following
sections.

##### Cycle consistency constraints

Adding a cycle-consistency objective constraints optimization such that
$F(G(x))\approx x$ and $G(F(y))\approx y$. This objective is balanced against
the adversarial objective through introduction of $\lambda_{A}$ and
$\lambda_{B}$ as seen in Equation 1.4. The balance between the adversarial
objective and cycle-consistency objective can promote different
characteristics during translation of the image. Increasing $\lambda$ can lead
to stronger optimization of the cycle-consistency leading to better
reconstructed images while the translated images might worsen as the
adversarial objective tends to be weaker and vice-versa. Determining an ideal
value of $\lambda$ generally involves a hyper-parameter search and can become
unfeasible for training large, time consuming 3D networks as is the case with
CBCT improvement. This begs the question – can these $\lambda$ values be
determined automatically? And can they be adapted during training by observing
different training metrics and use them as proxy controls? In the first part
of the thesis, we try to answer these questions in a general manner that might
be relevant across multiple tasks.

##### Structural constraints

Zhu et al., (2017) conduct ablation studies across various datasets to
demonstrate that stable performance is obtained when both cycle-consistency
and adversarial constraints are retained. Additional structural constraints
may be imposed on the CycleGAN such as constraints between input and
translated images similar to Shrivastava et al., (2017) where they term it as
a regularization loss. Equation 1.5 shows the modification made to the
original CycleGAN objective shown in Equation 1.4 upon addition of the
structure loss.

$\displaystyle L_{CycleGAN+structure}(G,F,D_{X},D_{Y})$
$\displaystyle=L_{CycleGAN}(G,F,D_{X},D_{Y})$ (1.5)
$\displaystyle+\quad\lambda_{struct}\mathbb{E}_{x\sim p_{data(x)}}[d(G(x),x)]$
$\displaystyle+\quad\lambda_{struct}\mathbb{E}_{y\sim p_{data(y)}}[d(F(y),y)]$

where $d(.,.)$ is the distance metric or the specific loss function used
between the images in source and target domain and $\lambda_{struct}$ is the
weight associated with the structure loss. Figure 1.5 shows an extension of
Figure 1.4 where the structure loss/ constraint is applied between the real
image and generated image.

Figure 1.5: Design of structure loss for a CycleGAN architecture

Yang et al., (2018) also present a structure-constrained CycleGAN and show
clear benefits of the additional structural constraints for the use-case of
MRI to CT synthesis. Another research question posed as a part of the thesis
is — can we benefit through the addition of structural constraints to a
CycleGAN? How do some of the existing methods, such as, the MIND loss
(Heinrich et al.,, 2012) as presented in the work of (Yang et al.,, 2018)
fare? Can simplified and potentially generalized losses also prove to be
effective? We aim to explore and provide answers to the questions posed above
as the second part of the thesis.

###  1.4 Evaluation of CycleGAN-based methods

Due to its unpaired nature, evaluation of CycleGAN-based translation poses
quite a challenge. Fortunately, for the use-case of CBCT enhancement, we have
the benefit of being able to construct some weak pairs consisting of CBCT
images and their matching CT images. This process will be described in detail
in Section 2.2. The set of weak pairs is split into validation and testing
allowing evaluation of the translation during training as well as on unseen
data post-training. Standard image similarity metrics are used to evaluate
image translation quantitatively along with qualitative evaluation.
Qualitative evaluation looks at multiple standpoints such as the introduction
of artifacts, translation quality and resolution.

##### Out of Distribution Analysis

Deep Learning approaches are susceptible to drastic performance drops when
shown data that is out of the training distribution. In the case of generative
approaches like the CycleGAN, presenting out of distribution data might induce
unexpected artifacts during the translation putting into question the
robustness of these methods. In order to test the generalizability of methods
in this thesis, phantoms are used as out-of-distribution data. Phantoms are
used to calibrate CBCT and CT machines and have different properties when
compared to patient data and are described in Section 2.1.2. Moreover, unlike
weak pairs formed with patient data, phantoms can be used as perfectly matched
pairs as their composition is static.

##### Domain-specific analysis

Enhancement of CBCT scans can play a key role in adaptive radiotherapy,
however, improvement in image similarity metrics might not solely show their
validity for this purpose. Domain-specific analysis of the improved CBCT
through comparing HU value distribution and line profiles with a matching CT
image can provide more clinical relevant evaluation in this case. An automated
lung segmentation model is also applied on the original and improved CBCT and
compared with ground truth contours. Through these comparisons, we aim to show
that improvement in CBCT quality benefits various down-stream applications of
CBCT.

###  1.5 Clinical value and impact

CBCT acquisition through onboard imaging during treatment delivery improves
outcomes for treatment through more accurate positioning. Since acquiring this
CBCT image is convenient and less intrusive compared to obtaining a full CT
scan, improvements in quality of the CBCT, by making it CT-like can add a lot
of value to the treatment process. For proton therapy, rescanned CTs are
captured at certain intervals to reassess the treatment plan as proton-based
plans are highly susceptible to changes observed during treatment. By having
CBCTs similar to CT quality, rescanning may be reduced or avoided saving time,
costs and improving patient convenience. For photon therapy, more accurate
plans can be generated through these improved CBCT images and adapting
treatment plans. However, clinical adoption of deep learning-based generative
methods are challenging due to their black-box nature and inability to offer
guarantees on new data. Evaluating methods on sufficiently large test sets,
out of distribution analysis and qualitative inspection can offer more insight
into their robustness. In addition to evaluation, deploying these models in
clinical workflows with well described model cards (Mitchell et al.,, 2019)
and data sheets describing the expected and potentially unexpected operational
scenarios can help with their acceptance. Clinicians can then evaluate these
independently on large cohorts of data and determine if they are suitable for
clinical use.

###  1.6 Contributions

The thesis and its various investigations aim to answer the following research
questions,

  1. 1.

Are default CycleGAN constraints sufficient to produce clinically viable
enhanced CBCT images, if not, what are their failure modes?

  2. 2.

Where does one start adapting constraints? What constraints of the CycleGAN
affect generated image quality the most?

  3. 3.

Can informed and data-driven constraints be developed for CBCT image
enhancement?

  4. 4.

In order to evaluate image quality, what quantitative metrics can be leveraged
effectively in such a semi-supervised setting?

One of the research questions that proposed to investigate if constraints that
work well for CBCT image enhancement generalize well to other medical imaging
tasks was dropped in favour of deeper investigation of data-driven constraints
and domain-specific evaluation. As future work, another medical image dataset
will be incorporated to demonstrate generalizability of our methods.

##  Chapter 2 Methods and Materials

###  2.1 Dataset

This section describes the data used to answer the questions this thesis
proposes. Two datasets, one comprising of natural images and the other of
medical images, are considered. Natural image dataset allows for quick
experimentation and also helps demonstrate if the methods explored in the
thesis transfer between natural and medical images.

####  2.1.1 Map $\leftrightarrow$ aerial photo Dataset

Figure 2.1: Two pairs of matching images for the map $\leftrightarrow$ aerial
photo dataset

The map $\leftrightarrow$ aerial photo dataset was introduced by Isola et al.,
(2016) in their pix2pix paper and contains 2194 images scraped from Google
Maps. Since map data and satellite data have one-to-one correspondence, they
were used in a paired form in the original paper. Zhu et al., (2017) use this
dataset in an unpaired form to evaluate the CycleGAN and additionally, compare
it against the pix2pix. Figure 2.1 shows two sample pairs from the dataset.
The map $\leftrightarrow$ aerial photo dataset was chosen as it contains
certain geometrical structures that are preserved between source and target
domains. Structures such as clusters of houses, road/highway patterns, and
boundaries between land and water are retained across both domains. In
contrast to the original paper, we use aerial as the source domain and map as
the target domain in order to be able to better evaluate the generated images
as map images are visually simpler.

####  2.1.2 Radiotherapy treatment dataset with CBCT and CT images

The dataset chosen for the use-case of CBCT enhancement, comprises of CBCT and
CT scans acquired during proton beam radiation therapy. Seventy-two lung
cancer patients were considered for treatment with the treatment delivery
conducted over multiple fractions. At each fraction, a CBCT image was captured
resulting in a total of 774 CBCT scans across all patients. In addition to
planning CTs, rescanned CTs are also collected to verify/adapt treatment
plans, leading to a total of 257 CT scans. Figure 1.3 shows the CBCT and CT
scan from a randomly selected patient. The CBCT images are acquired through
Mevion CBCT scanners which are suspectible to a significant amount of noise
(sources of which are highlighted in Section 1.2) while reconstructing the
images. Figure 2.2 shows images of the CBCT and CT using a soft-tissue window
that helps focus on values within the heart, skeletal muscle, etc. Compared to
the CT, the CBCT image has very different intensity values (bad HU
calibration) along with streaks in the image (due to scattering and motion
artefacts). Table 2.1 shows the datasheet for this dataset filled with
information that is made available to us. It is designed in such a manner that
it should allow a layperson to understand more about the dataset and its
nature.

Figure 2.2: Mid-axial views of the CBCT, CT and their difference image for a
patient. The difference image is shown in the HU scale capped between -350 and
350.

CBCT-CT Dataset For what purpose was the dataset created? Radiation therapy
treatment Who created the dataset and on behalf of which entity Maastro Clinic
What do the instances that comprise the dataset represent? Patients undergone
proton-beam radiotherapy How many instances are there in total? 72 Does the
dataset contain all possible instances or is it a sample of instances from a
larger set? Contains all possible instances from the treatment process What
data does each instance consist of? Each instance consists of multiple CBCTs
used for positioning and multiple repeat CTs used to recalculate the treatment
plan Is there a label or target associated with each instance? Each instance
consists of annotated organs and tumours that could be used as targets for
segmentation Are relationships between individual instances made explicit?
Yes. No relationships exist between the patients Are there recommended data
splits? Are there any errors, sources of noise, or redundancies in the
dataset? Redudancies in the form of multiple CT scans per patient might exist
Is the dataset self-contained, or does it link to or otherwise rely on
external resources Self-contained Does the dataset contain data that might be
considered confidential Yes, senstive patient data Does the dataset contain
data that, if viewed directly, might be offensive, insulting, threatening, or
might otherwise cause anxiety? Yes, dataset consists of patients with cancer
Does the dataset relate to people? Yes Does the dataset identify any
subpopulations? Not directly, but information from the scans may be used to
identify gender, age, weight etc. Does the dataset contain data that might be
considered sensitive in any way? Yes How was the data associated with each
instance acquired? For each instance, CBCTs were acquired using Mevion CBCT
scanners and CTs were acquired using Philips Sensation 16 CT scanner Were any
ethical review processes conducted? Approval through an Institutional Review
Board before internal access to data Was any preprocessing/cleaning/labeling
of the data done? Not in its original form Has the dataset been used for any
tasks already? Yes, for treatment planning. In terms of deep learning based
approaches, \- Used for CUT implementation \- Comparison between different
models for unpaired translation Will the dataset be distributed to third
parties outside of the entity (e.g., company, institution, organization) on
behalf of which the dataset was created? No, it is currently an internal
Maastro dataset

Table 2.1: Datasheet for the CBCT-CT dataset with relevant information to
allow easier understanding as per Gebru et al., (2020).

##### Phantoms

Imaging phantoms are objects designed with a known geometrical and physical
composition which are used for quality assurance and evaluation of CT machines
(in Medicine,, 1977). In addition to the patient data, we also have access to
phantoms scanned on the same CBCT and CT machines as the patients. Due to
their known composition, phantoms can be used as true ground truth pairs,
termed as strong pairs, in contrast to weak pairs that were mentioned earlier.
This is because weak pairs can contain differences due to motion, setup and
biological changes while phantoms are not subject to these changes. Figure 2.3
shows the anthropomorphic phantom available as a part of the dataset.

Figure 2.3: Anthropomorphic phantom shown with front and tilted views. A bone-
window view is also shown where intensities are windowed to expose only the
bones. As can be seen, the phantom replicates a human-like anatomy.

###  2.2 Data Pre-processing

Data pre-processing is a key element in the machine/deep learning pipeline
converting existing representations to representations that are best suited
for optimization. This is especially important for medical images where
inconsistent representations of data can lead to biases that propagate through
the modelling process (de Raad et al.,, 2021). In addition to preprocessing
the data, the splits used for training, validation and test for each of the
datasets are also mentioned.

####  2.2.1 Map $\leftrightarrow$ aerial photo  Dataset

The dataset is randomly split into 1085, 9 and 1098 images for train,
validation and test, respectively. A small number of images are chosen for
validation to allow extensive manual inspection of translation properties.
During training, the dataset is resized to $256\times 256$, and augmented with
random horizontal flipping of the images. The data is then normalized between
$(-1,1)$. For validation and testing, only resizing and normalization are
performed.

####  2.2.2 CBCT and CT Scans

The general standard for capture and communication of medical images, is the
DICOM (Digital Imaging and Communications in Medicine) format. DICOM is a rich
format providing extensive metadata about the captured images in the form of
sets of data that contain various attributes within. In addition to the
storage of CT and CBCT scans, the format also allows storage of annotated
contours, treatment plans and dose distributions. All the data used for this
thesis was made available in the DICOM format. Multiple steps of data
preparation and processing need to be undertaken to convert the scans from
DICOM to images that can be fed to deep learning models. This section
highlights steps that are taken for processing the data starting from format
conversion and resampling to split specific processing.

##### Data Conversion, Resampling and Stratification

The DICOM files for CBCT and CT scans are first converted to an NRRD (Nearly
Raw Raster Data) format. The NRRD format retains only lightweight metadata
that is needed to determine the orientation and spacing of the 3D image in a
global coordinate system. Patient, scanner and process-specific information is
thus, discarded. DICOM stores annotated contours as points on a plane in a
format called RTSTRUCT (RT Structure Sets). Along with the CBCT/CT scans, the
RTSTRUCTS are converted to binary labelmaps stored in NRRD format, as well.

The CT scans in the dataset are acquired with different field-of-view at the
discretion of the radiation therapist. Due to constant size of the line
detector, increase in field-of-view leads to a larger pixel sizes and
therefore, increased spacing between them. This needs to be taken care of
during pre-processing, as without this the image representations can be
misleading. Figure 2.4 shows an example of the same CT scan with different
spacings.

Figure 2.4: Difference in images rendered by an imaging software that does not
account for pixel spacing. The same scan appears very different for the two
spacings. For the spacings, the first term denotes the x-axis, second term the
y-axis and third term the z-axis. Note that the pixel sizes of the images are
different with the first and second images having a size of $259\times
259\times 59$ and $1003\times 670\times 59$ respectively.

In order to minimize interpolation, the resampling is done as follows,

  1. 1.

Get frequency counts of spacing of all scans in the dataset.

  2. 2.

Sort spacings in ascending order and rank all spacings based on their
frequency counts.

  3. 3.

Select the smallest rank starting from bottom of the list.

On following the procedure above, $1.2695\times 1.2695\times 3$ is determined
as the ideal spacing. The CBCT scans are also resampled to the resolution of
the CT as we expect the CBCT to be a replacement for the CT and hence, possess
similar resolution. All of the above are performed using the SimpleITK library
(Lowekamp et al.,, 2013) in python.

The dataset is stratified at the patient-level with 50 patients in the train
set, 4 in the validation set and 18 in the test set. Do note that due to
availability of more than a single pair per patient, the instances available
for train and val are larger than the number of patients. These are
specifically mentioned in the pre-processing subsections.

##### Train Set Pre-processing

In order to leverage the full benefits of unpaired data, training is done by
selecting a random CBCT and CT scan from all the available set of scans. Since
the selection is completely random, a CT scan from one patient can be paired
with a CBCT scan from the same patient or another patient. We hypothesize that
this allows for learning more generalized properties and also allows balancing
training instances in cases where a particular patient may have fewer CT or
CBCT scans. A total of 736 CBCT scans and 219 CT scans, from the 50 patients,
are used during training. A set of online  pre-processing steps are followed
during training , as shown in Algorithm 1.

1 function process_CT_CBCT(CTpath, CBCTpath):

// Load CT and CBCT from NRRD files

2 CT, CBCT = load_image(CTpath), load_image(CBCTpath)

3

// Generate body mask and apply it to the CT and CBCT images

4 CT, CBCT = apply_body_mask(CT), apply_body_mask(CBCT)

5

// Limit the axial range of the CT to match the CBCT

6 CT = truncate_CT(CT, CBCT)

7

// Sample patches of a given size from the CT and CBCT

8 CT_patch, CBCT_patch = get_patch_pair(CT, CBCT);

9 return CT_patch, CBCT_patch

Algorithm 1 Steps for processing CT/CBCT data in an online manner during
training

Each of the functions in the pre-processing steps are described below,

  * •

apply_body_mask: A body mask based on HU value thresholding is generated for
both the CBCT and CT scans. This is done in order to minimize the effect of
out-of-body structures during training. The body masking algorithm takes in a
CBCT/CT scan and binarizes the image based on a threshold. After binarization,
for each slice in the 3D scan, the largest contour is found. This contour is
filled and finally a smoothing kernel is applied over the contour points in
order to have smooth masks.

  * •

truncate_CT: Truncation of the CT aims to limit the field-of-view (FOV
represents the amount of physical space captured in the image) of the CT scan
in the z-axis. This is done so that the FOV of the CBCT and CT are roughly
matching and correspond to similar anatomical regions. A registration is
performed using SimpleITK where CBCT is the fixed image and CT is the moving
image. The resulting transformation is used to truncate the CT scan at
matching coordinate locations of the CBCT scan.

  * •

get_patch_pair: Loading full volumes during training is limited by memory size
and computational time requirements. To tackle this, training is usually done
in a patch-based manner. This function takes in a CBCT and CT scan and samples
patches from random locations in the CT. For CBCT, the CT location is used as
a starting point with small perturbations added to allow more randomness
during training.

##### Validation and Test Set Pre-processing

For each of the patients in the validation and test set, multiple CBCTs and
rescanned CTs are available that are acquired through the treatment process.
Rescanned CTs acquired at a time point close to the CBCT generally have good
anatomical correspondences although there may be differences due to setup and
random errors. These correspondences can be leveraged in order to form weak
pairs, which can then be used to evaluate translation quantitatively. The
process followed to generate these weak pairs is highlighted below,

  1. 1.

Select the rescanned CT and the CBCT closest to it. The maximum time-
difference between the two is limited to one day, so that scans with
potentially larger anatomical changes are ignored.

  2. 2.

The rescanned CT is registered to the CBCT through deformable registration
using parameters from the SimpleElastix library (Marstal et al.,, 2016).

  3. 3.

Apply the registration transform to the rescanned CT and available contours
(only available on the test set).

A sample of the weak pair formed from the process above, is shown in Figure
2.5. Out of the four patients in the validation set, 20 pairs are formed that
are used for evaluating model checkpoints. The 18 patients in the test set are
chosen more restrictively by manually ensuring matches between the CBCT and
rescanned CTs, and therefore, contain only 18 pairs used during testing. The
manual inspection is done by medical physicists at Maastro Clinic.

Figure 2.5: Registered CBCT, CT and overlay image. In the overlay image, the
green and pink areas in the image show differences between the CT and CBCT.
Green shows values that are greater in the CT while pink shows values that are
greater in the CBCT.

###  2.3 Adaptive Control of CycleGAN Constraints

The constraints of a CycleGAN that aid in its translation, namely, adversarial
and cycle-consistency loss, are generally balanced statically. Well-performing
values for $\lambda$ (both $\lambda_{A}$ and $\lambda_{B}$) can generally be
chosen through hyper-parameter search. However, this is resource-intensive and
may not transfer across different datasets and model configurations.
Furthermore, training may benefit by using more dynamic balancing strategies,
similar to dynamic augmentation proposed by Karras et al., (2020). In the
paper, Karras et al., (2020) propose using discriminator outputs as a control
signal to determine if augmentation would benefit the training. If these
control signals capture training dynamics effectively, then they could also
prove useful for a dynamic adaptation of constraints. Thus, inspiration from
this work was taken to develop the following control signals in an adaptive
form of the CycleGAN called AdaptiveCycleGAN,

##### Discriminator-based Control Signals

The discriminator-based control signal is very similar to the one used by
Karras et al., (2020). It looks at the outputs of the discriminator and
formulates the parameter, $r_{t}$.

$r_{t}=\mathbb{E}[sign(D_{train})]$ (2.1)

where $D_{train}$ represents the discriminator output and the expectation is
computed over four mini-batches of 64 images each. If $r_{t}>0.6$, then the
adaptive augmentation strategy is triggered.

In the case of the CycleGAN, the discriminator is PatchGAN-based and outputs a
3D patch. To account for that, Equation 2.1 is modified such that $D_{train}$
corresponds to the mean value over all voxels of the output patch. After some
initial experiments, it was determined that $D_{generated}$ is more responsive
to training dynamics than $D_{train}$ and was used henceforth. The number of
images that $r_{t}$ is computed over is set to 256, although, the number of
mini-batches vary based on the batch-size selected. Finally, the adaptive
lambda strategy is triggered at certain frequency intervals, as it was seen
with preliminary experiments that adapting too often leads to early
divergence.

When the adaptive $\lambda$ strategy is triggered, the $\lambda$ values are
changed by a percentage of 0.05. It is increased if $r_{t}>0.6$ as the
discriminator is largely predicting generated images as real. This generally
means that the generator is overpowering the discriminator and can focus less
on the adversarial loss. Similarly $\lambda$ is decreased when $r_{t}<0.6$.

##### Cycle-consistency-based Control Signals

The intuition for cycle-consistency-based control signals is derived from
various initial experiments that were run to understand training dynamics,
interpretation of loss curves and their effect on the translations. The
following observations were made, (1) for well-performing configurations, the
cycle-consistency loss decayed smoothly while the adversarial loss had a high
variance but with a mostly constant mean value; and (2) decaying cycle-
consistency loss was generally a good indicator of strong reconstructions
while high-variance adversarial loss was a good indicator of being able to
capture the domain style. Based on this, the cycle-consistency loss curves are
used as a control signal for adaptive $\lambda$ strategy. Cycle-consistency-
based control signal starts monitoring the cycle-consistency loss after a
certain number of iterations of training have completed, termed generally as
warmup iterations. Following this, a patience-based strategy similar to the
one used in learning rate schedulers is leveraged, where, at a certain
iteration a set of previous samples are retained and compared with the current
value. If the current value is greater than all of the samples, then the
adaptive $\lambda$ strategy is triggered.

The adaptive $\lambda$ strategy for this signal, operates in two modes,
threshold-based and relative. The threshold-based version increases/decreases
$\lambda$ by a constant change rate while the relative version changes it by a
rate that is equal to the relative difference between the current and the
minimum value among previous samples.

###  2.4 Structure-consistency Losses

The second part of the thesis investigates methods that add additional
constraints to the CycleGAN in the form of structure loss. As described in
Figure 1.5 and Section 1.3.3, the structure loss is added between the real
image and the generated image in order to restrict the set of possible
mappings between the source and target domains. This is not a new concept in
the CycleGAN framework but its investigation has been fairly limited,
especially for medical images. A challenge posed in the wider applicability of
such structural losses, is that these tend to be domain-specific or rely on
hand-crafted functions, while one of the primary benefits of the CycleGAN is
that it is able to perform well in general as well as domain-specific settings
without much configuration required. To address this problem, a generalized
frequency loss, inspired by Jiang et al., (2020), is implemented that requires
no configuration on the part of a user. However, to reap its full benefits, a
preliminary check is advised to see if the loss would benefit a particular
use-case. This is done through implementation of a script that uses a pre-
existing dataloader and provides a report. Before the generalized frequency
loss is described, some existing structure losses are described and
subsequently investigated.

####  2.4.1 Modality Independent Neighbourhood Descriptor
(MIND) Loss

The MIND loss was used for MR-to-CT synthesis as a part of a structure-
constrained CycleGAN (Yang et al.,, 2018). Images from source and target
domain are mapped to a common feature domain, which is done using the MIND
descriptor. The MIND descriptor was originally developed for MR to CT
registration and is therefore capable of handling cases where images are of
different modalities. The MIND feature for an image $I$ at voxel $x$ is
formulated as,

$F_{x}^{(\alpha)}(I)=\frac{1}{Z}exp(-\frac{D_{P}(I,x,\alpha)}{V(I,x)})$ (2.2)

where $\alpha$ is such that $x+\alpha$ is within a region called the non-local
region. $Z$ is a normalization constant and $D_{P}$ is formulated as

$D_{P}(I,x,\alpha)=C*(I-I^{\prime}(\alpha))^{2}$ (2.3)
$V(I,x)=\frac{1}{4}\sum_{n\in N}D_{p}(I,x,n)$ (2.4)

where $I^{\prime}(\alpha)$ represents the image $I$ translated by $\alpha$;
$C$ is a gaussian kernel with $\sigma=2$; and $N$ is the four-neighbourhood of
voxel $x$. For each value of $\alpha$, a scalar value is computed resulting in
a vector $F_{x}$, with 81 elements. The number of elements is given by the
size of the non-local region which is $9\times 9$. The MIND descriptor is used
as MIND loss through the formulation below,

$d(G(x),x)=\frac{1}{N}\sum_{x}||F_{x}(G(x))-F_{x}(x)||_{1}$ (2.5)

where $G(x)$ is the predicted image by the generator, and $x$ is the real
image. The difference in MIND descriptor representations is summed over all
voxels in the images. $d(G(x),x)$ is used in Equation 1.5 to formulate the
full structure loss. For more details, the original paper by Yang et al.,
(2018) can be referred to.

####  2.4.2 Registration Metric Losses

Since the MIND loss was originally based on a registration similarity metric,
a less computation and memory intensive metric was tested in this thesis. The
metric chosen was the Local Normalized Cross Correlation (LNCC) which is used
by the DeepReg toolkit. An implementation by the Monai library (Ma et al.,,
2021) was chosen as it operates directly on pytorch tensors and therefore, is
more efficient. Similar to the MIND loss, the loss values are interpretable on
the image space and can allow easy debugging.

Figure 2.6 shows the MIND and registration metric in the image space for mid-
axial slices for the same patient as well as for different patients. Larger
values are expected for different patients as their anatomies differ. The
losses are also expected penalize deviations in anatomy when used during
training. Registration metric shows larger values for different patients in
the bony areas and body contours. For the MIND representation, it is seen that
the intensities are not directly correlated with anatomical properties.

Figure 2.6: Absolute differences in MIND (shown in green) and registration
metric (shown in red) representations between CBCT and CT images. The
differences are shown on the mid-axial slice for CBCT and CT scans taken from
the same patient and CBCT and CT scans taken from different patients.

####  2.4.3 Generalized Frequency Loss

Frequency spectrum representations of images can allow capturing patterns
within the image, that may not be easy to identify in their spatial domain
representations. In its frequency spectrum, the image is broken down into its
sinusoidal components that can then be further analyzed by looking at their
amplitudes and phase. Converting an image to its frequency spectrum
representation involves a three-dimensional DFT transform, which is stated as
follows,

$F(u,v,w)=\sum_{x=0}^{L-1}\sum_{y=0}^{M-1}\sum_{z=0}^{N-1}I(x,y,z)e^{-j2\pi(ux/L+vy/M+wz/N)}$
(2.6)

where $L$, $M$ and $N$ are dimensions of the CT and CBCT scans. For the
development of the generalized frequency loss, the frequency representation is
then normalized as below,

$F^{{}^{\prime}}(u,v,w)=\frac{1}{\sqrt{LMN}}F(u,v,w)$ (2.7)

where $F^{{}^{\prime}}(u,v,w)$ is then shifted such that zero frequency lies
at the center of the image. Following the shift, only magnitude spectrum of
the frequency spectrum is taken as $F_{mag}(u,v,w)=|F^{{}^{\prime}}(u,v,w)|$.
On the magnitude spectrum, a tanh non-linearity is applied as,

$F_{rep}(u,v,w)=tanh(F_{mag}(u,v,w))$ (2.8)

The $tanh$ non-linearity is applied in order to scale all values to the range
of 0 to 1. This was done to combat the differences in scale of frequency
domain representations across different image sets. Alternatively, a careful
strategy to ensure that images in the dataset are in similar scales while
generating their frequency representations can be designed. However, this
becomes extremely data-specific, and by no means, is generalized. Another more
general solution is, computation of a log transformed magnitude spectrum, but
this led to very different scales across different images tested. This would
be unsuitable for direct comparison in the frequency domain between the two
images.

Addition of the $tanh$ makes several assumptions about the importance of
different intensities in the magnitude spectrum, as it leads to the following
effects: (1) values greater than $0.5$ are subdued and (2) as values increase
in intensity, their rate of change is also dampened. However, the hypothesis
is that the distribution of values, in the magnitude spectrum and not the
intensities itself, are of primary importance. Note that intensity of values
are still captured but only higher intensities are dampened. Due to ortho-
normalization of the FFT, the values are not generally large to begin with.

Through detailed analysis on both the datasets used in the thesis, it was seen
that the generalized representation captured differences in domains
appropriately. A small difference in representation was seen for corresponding
images of different domains, while a larger difference was seen for non-
corresponding images. Figure 2.7 shows the generalized frequency
representations along with absolute and squared difference between these
representations for different sets of images.

Figure 2.7: Generalized frequency representation of CBCT scan taken from one
patient compared against CT scan taken from the following, (1) same patient,
(2) same patient but offset by 10 slices and; (3) a different patient. The
absolute and squared differences between representations are shown.

Upon obtaining the generalized frequency representation, the generalized
frequency loss is represented as,

$d(G(x),x)=\sum_{u,v,w}||F_{rep}(u,v,w)(x)-F_{rep}(u,v,w)(G(x))||_{1}$ (2.9)

where $G(x)$ is the predicted image by the generator, and $x$ is the real
image. The difference in frequency representations is summed over all voxels
in the images. $d(G(x),x)$ is used in Equation 1.5 to formulate the full
structure loss. The methods and evaluation implemented as a part of this
thesis was built upon the ganslate (Hadzic et al.,, 2021) codebase.

###  2.5 Evaluation

Evaluation of unpaired translation methods is a non-trivial task and generally
relies on a combination of quantitative and qualitative criteria. Domain-
specific evaluation can also often be leveraged in order to determine if the
generated images are suitable for downstream tasks.

####  2.5.1 Image Similarity Metrics

Image similarity metrics common in image translation and quantitative IQA
(Image Quality Assessment) such as Mean Absolute Error (MAE), Mean Squared
Error (MSE), Normalized Mean Squared Error (NMSE), Power-to-Signal Noise Ratio
(PSNR) and Structural Similarity Index Measure (SSIM) are used to
quantitatively evaluate different methods. These metrics are outlined below,

  * •

Mean Absolute Error (MAE)

$MAE(ref,pred)=\frac{1}{N}\sum_{i=1}^{N}|ref(i)-pred(i)|$ (2.10)

where $N=$total number of voxels in the image. $ref$, in our work, is the CT
image while $pred$, is the generated image.

  * •

Mean Squared Error (MSE)

$MSE(ref,pred)=\frac{1}{N}\sum_{i=1}^{N}|ref(i)-pred(i)|^{2}$ (2.11)

The mean-squared error largely penalizes deviations from the reference image
due to the difference being squared.

  * •

Normalized Mean Squared Error (NMSE)

$NMSE(ref,pred)=\sqrt{\frac{{\sum_{i=1}^{N}|CT(i)-CBCT(i)|^{2}}}{\sum_{i=1}^{N}|CT(i)|^{2}}}$
(2.12)

The $NMSE$ gives the mean squared error while also factoring in the signal
power.

  * •

Power-to-Signal Noise Ratio (PSNR)

$\begin{split}PSNR(ref,pred)&=20\times log_{10}(ref_{max})\\\ &\quad-10\times
log_{10}MSE(ref,pred)\end{split}$ (2.13)

$ref_{max}$ refers to the maximum value of the $ref$ image. $MSE(ref,pred)$ is
computed as described in Equation 2.11

  * •

Structural Similarity Index Metric (SSIM)
$SSIM(ref,pred)$ is computed using the formula presented below, with $ref$
denoted as $x$, and $pred$ as $y$.

$SSIM(x,y)=\frac{(2\mu_{x}\mu_{y}+c_{1})(2\sigma_{xy}+c_{2})}{(\mu_{x}^{2}+\mu_{y}^{2}+c_{1})(\sigma_{x}^{2}+\sigma_{y}^{2}+c_{2})}$
(2.14)

where $\mu$ and $\sigma$ represent the mean and variance respectively. $c_{1}$
and $c_{2}$ are variables used to stabilize division.

####  2.5.2 Qualitative Inspection

Qualitative criteria are incorporated into the evaluation procedure, mainly
due to the limitations of existing quantitative criteria in capturing
undesired effects of unpaired translation. Effects such as checkerboard
patterns, addition of artifacts, modification of anatomies are not directly
captured by metrics. For example, consider a model that translates images very
well and offers good quantitative scores across all metrics. However, this
model adds small artifacts such as air pockets that were not present in the
original image. As a result, no matter how good the metric score, this model
will not be accepted clinically. Therefore, qualitative evaluation and
analysis, is inescapable.

Structured qualitative inspection can allow comparing models in a more
systematic manner. Through analyzing translations from various experiments, a
set of criteria for qualitative inspection are formulated,

  1. 1.

Closeness to the target domain: We attempt to determines how visually similar
the translation looks to the domain of the target. For CBCT-CT, we do this by
inspecting the images using soft-tissue windows. A soft-tissue window filters
HU values outside the range of $(-135,215)$ and allows better visual
examination of soft-tissue boundaries. As a result, within these windows, it
is possible to determine if different regions such as the heart, skeletal
muscle, fat, etc, correspond visually between the translated image and target.
We inspect specifically, (1) bony regions, (2) heart and lung, and (3)
skeletal muscles. In addition to visual correspondence, we also check the
reduction of CBCT artefacts in the translated image by comparing it with a
region in the CBCT where artefacts were largely present. For maps
$\leftrightarrow$ aerial photo, we look at how well the translated image
reproduces water bodies, highway roads, among other structures present in the
maps image.

  2. 2.

Presence of artifacts or undesirable elements: Induction of artifacts are an
established drawback of generative modelling with GAN-based methods (Zhang et
al.,, 2019). Such artifacts are hard to identify using pixel-based
quantitative metrics and no other metric that fully captures the range of
possible artifacts in a CycleGAN is available yet. To this end, we inspect
images manually to check for artifacts or any undesirable elements such as
localized checkerboard artifacts that appear randomly.

  3. 3.

Quality of image in terms of resolution: This criteria aims to identify
reduction in perceived image quality for translated images. Some very commonly
seen phenomena in CycleGAN translations are blurry images, aliasing-like
effects and bright spots in parts of the image. Ideally, a reader study would
be performed to analyze these factors. However, that is beyond the scope of
this thesis.

All the criteria mentioned above are inspected using mid-axial, sagittal and
coronal views. Visualizing the entire 3D scan for 18 patients in the test set
for multiple trained models would be very exhausting and impractical. Looking
at three cross-sectional planes allows us to make a good judgement on the
overall quality of the image.

####  2.5.3 Out-of-distribution Analysis

Phantoms, as described in Section 2.1.2, are used to test models on out-of-
distribution data. Image-similarity metrics are computed on the phantom with
an available body mask to generate quantitative metrics. Qualitative
inspection of the phantom is done similar to the patient data, as mentioned in
Section 2.5.2. Special attention is paid to the translation of the tumour in
the phantom as it is hypothesized to be a potential source of failure for the
translation.

####  2.5.4 Domain-specific Evaluation

Apart from the quantitative and qualitative evaluations highlighted above, we
would like to understand if the methods proposed as a part of the thesis
benefit the use-case of adaptive radiotherapy, for which we chose to design
such methods. In order to establish this, we do a short analysis of HU value
distributions between the original, target and the improved CBCT (translated
scan). We also compare these scans based on line profiles that demonstrate HU
values observed when a line passing through the heart, lung, skeletal muscles
and bones, is drawn in the axial plane.

Additionally, the improved CBCT can also be used to generate RT contours
through incorporation of automated segmentation methods. To check if the CBCT
improvement benefits this task, we compare the difference in segmentation
contours obtained between original and generated images. An automated lung
segmentation model111https://github.com/JoHof/lungmask (Hofmanninger et al.,,
2020), trained on a large and diverse dataset, is used to segment the CT/CBCT
scan into left and right lung. Since the test data contains ground truth
contours, we generate automated contours for the CT, original CBCT and
improved CBCT and compare each with the ground truth using the Dice score.

####  2.5.5 Validation and Selection of Best Checkpoints

Unpaired GAN-based training can offering varying translations across different
iterations. To pick the best performing model (quantitatively and
qualitatively), a well designed validation method is needed. In this thesis, a
validation method that analyzes translation at different checkpoints
quantitatively, followed by visual inspection is employed. Checkpoints are
ranked in terms of their metric scores on the patients in the validation set.
The top-5 ranked checkpoints are taken as candidates. From this list of
candidates, the top checkpoint is popped and if it satisfies visual
inspection, is chosen as the best checkpoint. If it fails the inspection, it
is discarded and the next checkpoint is considered. For visual inspection,
criteria that are outlined in Section 2.5.2 are used. Figure 2.8 shows a
flowchart diagram describing this process of checkpoint selection.

Figure 2.8: Process of selection of best checkpoint using the validation data.
For each of the inspection boxes, a reference and window is outlined that
represents what it was compared against and the viewing window used for
comparison

##  Chapter 3 Experiments and Results

In this section, experimental configurations that are used to investigate the
research questions posed as a part of this thesis are outlined. Each
experiment is performed as an ablation study with only the parameter that is
under-investigation being changed. However, in some cases, when this is not
possible, for example, due to hardware and other limitations, its implications
on the experiment are discussed.

###  3.1 Experimental Setup

Firstly, two base configurations are determined,

  1. 1.

For models trained with the maps $\leftrightarrow$ aerial photo data, called
the 2D base configuration

  2. 2.

For models trained with the CBCT-CT data, called the 3D base configuration

Table 3.1 shows the parameters for the two base configurations in detail.

| 2D Configuration | 3D Configuration
---|---|---
Generator | UNet 2D | VNet 3D
Discriminator | PatchGAN 2D | PatchGAN 3D
LR | D: 0.0001, G:0.0002 | D: 0.0002, G:0.0004
Batch-size | 4* | 1
LR schedule | Fixed for 50%, Linear decay for 50%
Optimizer | Adam $(\beta_{1}=0.5,\ \beta_{2}=0.999)$
| Lambda
---
($\lambda_{A},\lambda_{B}$)
10 | 5
Input size | $(256,256)$ | $(16,320,320)$
Normalization | Instance normalization
Training iterations | 219.2k | 30k

∗ Batch-size of 4 is used as initial experiments with higher batch-size did
not show any improvement.

Table 3.1: Base configurations used for experiments performed across the two
datasets

The 2D models are trained on a GTX1080Ti with 4GB of memory for 219.2k
iterations (200 * size of dataset) while the 3D models are trained on Tesla
V100 with 16GB memory for 30k iterations. All experiments are run with mixed
precision enabled, unless mentioned otherwise, in order to efficiently use
available GPU memory.

####  3.1.1 Network Architectures

In this section, we describe in detail the network architectures used for 2D
and 3D configurations. Specifically, the generator and discriminator
architectures along with their parameters are discussed.

##### 2D Configuration

The 2D base configuration consists of a 2D UNet (Ronneberger et al.,, 2015) as
generator and PatchGAN2D (Isola et al.,, 2016) as discriminator. The 2D UNet
consists of a down-sampling path with 7 convolutional blocks, each with
convolutions of kernel size 4 and stride 2. Following the convolution, a
LeakyReLU and instance norm is applied (except for the terminal levels where
LeakyReLU is applied for the input level and ReLU is applied for the output
level following the convolution). This is followed by an up-sampling path that
consists of the same number of blocks as the down-sampling. Each of the blocks
consist of transposed convolutions of the same kernel size and stride as the
convolutions. After the transposed convolution, instance norm and ReLU layers
follow (except for the last block where a $tanh$ follows the transposed
convolution). Between each of the down-sampling and up-sampling blocks of
matching feature sizes, a skip-connection is made which concatenates their
feature maps. Figure 1 in the Appendix shows a block diagram of the 2D UNet
generator. The PatchGAN2D consists of 5 convolutional blocks with instance
norm and LeakyReLU after each of the convolutions with kernel size 4 and
stride 2 (except for the input block where no instance norm is applied and the
output block where neither instance norm nor LeakyReLU is applied). For the
given input size used for our 2D configuration, the PatchGAN2D outputs a patch
of size $30\times 30$.

##### 3D Configuration

The 3D base configuration consists of a 3D VNet (Milletari et al.,, 2016) as
generator and PatchGAN3D as discriminator. The 3D VNet structure consists of
an input block, 4 down-sampling blocks, 4 up-sampling blocks and an out block.
The input block consists of a 3D convolutional block of kernel size 5 followed
by instance norm and PReLU. The four down-sampling blocks consists of 1, 2, 3
and 2 convolutional blocks, respectively, with varying kernel sizes and
strides. The four up-sampling blocks consists of 2, 2, 1, 1 convolutional
blocks. The output block contains 2 convolutional blocks, the first followed
by an instance norm and PReLU and the second followed by a $tanh$. This
configuration is arrived at based on previous experiments conducted on other
medical imaging data with promising performance. Skip connections similar to
the UNet 2D are also seen in the 3D VNet. Figure 2 in the Appendix shows a
block diagram of the 3D VNet generator. The PatchGAN 3D is a 3D version of the
PatchGAN 2D, obtained by replacing the the 2D convolutions with 3D
convolutions.

###  3.2 Exploring Default CycleGAN Constraints

In this experiment, the significance of $\lambda$ parameter, that determines
the balance between cycle-consistency and adversarial losses, is investigated.
The authors of CycleGAN show that excluding the cycle-consistency distorts the
translated images with hallucinated structures and patterns. However, they do
not discuss the impact of different values of lambda on translation. We
conduct experiments with different values of lambda, specifically $\lambda=1$
and $\lambda=50$, with the end goal of gauging their impact on translation
performance. These values are chosen as extremes as the default value of
$\lambda=10$ is chosen by the authors. As we expect to gain an intuition of
$\lambda$ values and their impact on translation in a general sense, we run
the experiments only on maps $\leftrightarrow$ aerial photo dataset with the
2D configuration.

####  3.2.1 Results

Table 3.2 and Figure 3.1 show the quantitative and qualitative results of
different $\lambda$ parameters.

Model | MAE | MSE | NMSE | PSNR | SSIM
---|---|---|---|---|---
$\lambda=1$ | 0.113 | 0.030 | 0.054 | 17.036 | 0.213
$\lambda=10$ | 0.111 | 0.030 | 0.053 | 16.549 | 0.242
$\lambda=50$ | 0.563 | 0.363 | 0.551 | 4.656 | 0.043
Table 3.2: Image similarity metrics on the test set for experiments with
different values for the $\lambda$ parameter on the maps $\leftrightarrow$
aerial photo dataset Figure 3.1: Translated images for different values of
$\lambda$. For each parameter, two sets of images are shown. An image set
consists of the source image in the first column, translated image in the
second and ground truth in the third.

$\lambda=10$ performs the best quantitatively, providing best scores for all
metrics except PSNR. $\lambda=1$ provides the best PSNR score. Looking at
Figure 3.1, we see that $\lambda=10$ also performs the best qualitatively by
estimating several components of the image satisfactorily. However, it is far
from being geometrically accurate to the structures in the target image.
Translations from $\lambda=1$ undergo mode collapse with different input
images being mapped to the same output translation. Mode collapse is a common
occurence in GANs and avoiding it is an active area of research. $\lambda=50$
shows structures that correspond best with the target image, but do not
identify as being the maps domain. A lot of high-frequency components are seen
in the translated image.

CBCT-CT Dataset The baseline model, with $\lambda=10$, is run on the CBCT-to-
CT dataset and the quantitative results can be seen in Table 3.3 while the
translated images for 2 patients, chosen randomly, can be seen in Figure 3.2.
Only soft-tissue windows are shown as they are easier to inspect for quality
of translation. However, it is harder to inspect addition of GAN-artifacts or
anatomy modifications in this window range. Since no major artifacts were
observed on the baseline, we present only the windowed images.

Figure 3.2: Mid-axial, sagittal and coronal views for CBCT, CT and baseline
model on two different patients from the test, shown using the soft-tissue
window. The two patients are separated by a white space.

The baseline offers satisfactory translation and looks closer to the CT domain
than the CBCT domain. It retains the structure of the original scan, unlike
what was seen on the aerial imagery. However, zooming in on the images shows
that there are checkerboard artifacts, especially visible in the heart region
on the axial view (left). We also see that CBCT scattering artefacts are still
retained in the translated images. This can be seen properly in the sagittal
views (center).

###  3.3 Adapting Constraints For Improved Translation

This experiment is designed in order to address the possibility of dynamically
adapting $\lambda$ values based on training dynamics. The CycleGAN using
adaptive $\lambda$ strategies is called the AdaptiveCycleGAN, for ease of
reference. As mentioned in Section 2.3, it is possible to use different
control signals to evaluate the state of training in a CycleGAN. A set of
experiments are designed to investigate if these two control signals can be
used for adaptive training. In these experiments, for (1) discriminator-based
control signal, different change rates and $D_{train}$ vs $D_{generator}$ is
experimented with and for (2) cycle-consistency loss, change modes and rates
are varied. The configurations for the best performing models (chosen based on
the image-similarity metric scores on the validation set) from these initial
experiments can be found in Table 3.3.

|  AdaptiveCycleGAN discriminator-based |  AdaptiveCycleGAN cycle-consistency based
---|---|---
Change rate | 0.01 | 0.01
Change mode | - | Threshold-based
Frequency of update | 2000 | 1000
Sample Size | 256 | -
$r_{t}$ | 0.6 | -
Patience | - | 100
Warmup iterations | - | 5000
Table 3.3: Best performing configurations determined across different
AdaptiveCycleGAN experiments.

The full range of experiments can be found as a Weights and Biases workspace
111https://wandb.ai/surajpai/aerial_to_maps. The best performing run across
the two control signals are then run on the CBCT-CT dataset.

####  3.3.1 Results

Table 3.4 and Figure 3.3 shows the quantitative and qualitative results of the
two control signals investigated for adaptive $\lambda$ strategies on the maps
$\leftrightarrow$ aerial photo dataset.

Figure 3.3: Source, translated and target images for baseline, discriminator-based and cycle-consistency based adaptive $\lambda$ strategies. For each experiment, two sets of images are shown. An image set consists of the source image in the first column, translated image in the second and ground truth in the third. Model | MAE | MSE | NMSE | PSNR | SSIM
---|---|---|---|---|---
Baseline | 0.111 | 0.030 | 0.053 | 16.549 | 0.242
AdaptiveCycleGAN discriminator-based | 0.122 | 0.034 | 0.058 | 15.759 | 0.197
AdaptiveCycleGAN cycle-consistency based | 0.110 | 0.030 | 0.052 | 16.531 | 0.239
Table 3.4: Image similarity metrics on the test set for experiments to
determine the best control signal for adaptive $\lambda$ strategies on the
maps $\leftrightarrow$ aerial photo dataset.

The baseline model achieves the best PSNR and SSIM, while the cycle-
consistency based AdaptiveCycleGAN obtains the best MAE and NMSE. Both of them
obtain very similar MSE values. The discriminator-based AdaptiveCycleGAN
performs much worse compared to baseline and cycle-consistency control signal.
Qualitative analysis of the translated images shows very comparable properties
between baseline and cycle-consistency based AdaptiveCycleGAN. Specific
attributes that maybe favoured from either model exist in the translated
images. For instance, the baseline model, provides less blurrier additions
(blur added in 2nd image set for the cycle-consistency AdaptiveCycleGAN) to
the image while the cycle-consistency AdaptiveCycleGAN adds lesser
hallucinated structures (as seen in 2nd image set for the baseline). The
discriminator-based AdaptiveCycleGAN has mode collapsed with meaningless
representations.

CBCT-CT Dataset The cycle-consistency based AdaptiveCycleGAN is tested on the
CBCT-CT dataset, and for simplicity, it is addressed as AdaptiveCycleGAN
henceforth. Table 3.5 and Figure 3.5 show the quantitative and qualitative
results respectively.

Model | MAE | MSE | NMSE | PSNR | SSIM
---|---|---|---|---|---
Baseline | 88.846 | 24244.151 | 0.03086 | 29.365 | 0.9349
AdaptiveCycleGAN | 101.94 | 24899.015 | 0.03171 | 29.193 | 0.9276
Table 3.5: Image similarity metrics on the test set for the AdaptiveCycleGAN
experiment run on the CBCT-CT dataset. Note that all values with units are in
a CT number scale (between 0 to 3000) Figure 3.4: Mid-axial, sagittal and
coronal views for CBCT, CT , baseline and AdaptiveCycleGAN generated images
for a patient chosen randomly from the test set. Two different HU windows are
used to since the soft-tissue window (-135, 215) does not sufficiently show
addition of artifacts.

The AdaptiveCycleGAN obtains better MSE and NMSE while the baseline obtains
better MAE, PSNR and SSIM. Do note that baseline trumps the AdaptiveCycleGAN
in MAE and SSIM by a considerable margin. This can also be seen visually in
Figure 3.5 where the AdaptiveCycleGAN is closer to the CBCT (source) rather
than the CT (target). However, a lot of noise and CBCT artefacts are still
present. No GAN-induced artifacts are observed (a checkpoint @ 14k iterations
was chosen as later checkpoints generated plenty of artifacts). In contrast,
the baseline gets much closer to the CT distribution and suppresses CBCT
artefact in many regions across, especially in the heart and skeletal muscles.

###  3.4 Data-driven Constraints for CBCT Image Quality Enhancement

After seeking to adapt existing constraints, the next set of experiments look
at adding more constraints to the CycleGAN framework. This is with the
hypothesis that existing constraints still have issues with maintaining
structure between the two domains. Section 2.4 describes the different data-
driven constraints, in the form of structure losses, investigated as a part of
the thesis.

For the structure losses, experimental configurations that differ from the
base configuration or that may be of specific interest are mentioned below,

MIND loss: A few changes were made in the experiment configuration for the
MIND loss, (1) the authors propose a weight of $\lambda_{structure}=5$, this
is changed to $\lambda_{structure}=50$ through scale-matching with other
losses, (2) a patch-size of $(16,192,192)$ is used for the MIND loss due to
memory restrictions, as a direct consequence of (3) Mixed-precision being
unsuitable for this experiment as it led to early divergence.
Generalized Frequency loss: Two different distance metrics were used for
generalized frequency loss, shown in Equation 2.9, the $L_{1}$ distance
between the frequency representations and the $L_{2}$ distance between the
frequency representations. Other distance metrics such as $L_{p}$ distances
may also offer interesting properties but are not experimented with in this
thesis due to time and resource constraints.
Registration loss: As the values of the registration loss measure was not
bound, different methods of normalization of the losses were tried, namely,
(1) $tanh$ squashing, (2) loss-clipping and (3) full range (not normalized).
Combined Loss: A combination loss consisting of the Frequency L1 loss with the
MIND loss is investigated. The losses have $\lambda_{struct}$ values
consistent with their individual experiments, and are summed to obtain the
combined loss. This is trained with a patch size of $(16,320,320)$.

A total of six different models are compared in this experiment, (1) Baseline,
(2) MIND, (3) Frequency L1, (4) Frequency L2, (5) MIND + Frequency L1 and (6)
Registration loss

####  3.4.1 Results

Table 3.6 and Figure 3.5 present the different image similarity metrics and
visuals of a patient scan from different experiments. Note that the
registration loss experiment is not shown due to its complete failure to
converge to a meaningful translation. Visuals of its failure modes are
presented in Figure 3.6 and subsequently discussed.

Model | MAE | MSE | NMSE | PSNR | SSIM
---|---|---|---|---|---
Baseline | 88.846 | 24244.151 | 0.03086 | 29.365 | 0.9349
MIND | 85.911 | 25604.077 | 0.03249 | 29.265 | 0.9441
Frequency loss $L_{1}$ | 85.503 | 20433.408 | 0.02606 | 30.016 | 0.935
Frequency loss $L_{2}$ | 85.966 | 20247.973 | 0.02577 | 30.121 | 0.9387
MIND + Frequency loss | 86.632 | 21125.905 | 0.02696 | 29.88 | 0.9346
Table 3.6: Quantitative metrics obtained on the test set for experiments with
various structure losses run on the CBCT-CT dataset. Note that all values with
units are in a CT number scale (between 0 to 3000) Figure 3.5: Mid-axial,
sagittal and coronal views for CBCT, CT, and generated images from models with
different data-driven constraints for a patient chosen randomly from the test
set. Two different HU windows are used to since the soft-tissue window (-135,
215) does not sufficiently show addition of artifacts. Qualitative
observations discussed in-text are marked with red dotted ellipses and
numbered.

The baseline model is one of the worst performing models with the worst MAE
and second worst SSIM, MSE, NMSE and PSNR. MIND loss provides the best SSIM
but gives the worst MSE, NMSE and PSNR among all the models investigated. In-
terms of MSE, NMSE and PSNR, the Frequency loss with $L_{2}$ performs the
best. Inspecting the scans visually, a sample of which is shown in Figure 3.5,
we make the following observations based on criteria highlighted in Section
2.5.2, 222The observations are also pointed out in the Figure 3.5 using red
dotted ellipses and numbering. The numbering scheme in the list corresponds
with numbers in the image

  1. 1.

Air pockets that are present in the original scan are closed by the baseline
model.

  2. 2.

For the baseline model, decrease in the quality of the translated image is
observed through addition of checkerboard-like patterns.

  3. 3.

MIND loss adds unexplained artifacts in the form of black density reduction
fields.

  4. 4.

Frequency $L_{2}$ also closes air pockets similar to the baseline model.

  5. 5.

Frequency $L_{2}$ provides a shift in density as we move down to the
diaphragm, as seen on the sagittal view.

  6. 6.

MIND + Frequency $L_{1}$ causes a random drop in density across a particular
region.

The above observations are made across multiple patients from the test
dataset. For brevity, only features from a single patient are described.
Although the MIND loss seems to visually be closest to the CT, it adds
significant artifacts. The next closest candidate where no artifacts or image
quality drops are observed, is the Frequency $L_{1}$.

##### Failure of Registration Loss

Multiple configurations of registration loss, as mentioned in Section 3.4,
were tested along with hyper-parameter searches. None of them seemed to
provide any meaningful translations. Figure 3.6 shows the registration loss at
two checkpoints during training.

Figure 3.6: Failure modes of the registration loss shown on a patient from the
validation set at different iterations during training. The columns show
source, generated and target images respectively.

From Figure 3.6, it is evident that the training is divergent at 10k
iterations and gets worse with adding ‘bony‘ structures toward the end at 30k
iterations. Note that all other configurations performed similar to the
figure, if not, worse. The registration loss was not able to be utilized
directly in the CycleGAN framework and also failed with simple modifications.
Further analysis on how this loss may be incorporated into the framework
without divergence needs to be conducted.

##### Out-of-distribution Analysis

Table 3.7 and Figure 3.7 show metric scores and visuals of phantoms as
outlined in Section 2.5.3.

Model | MAE | MSE | NMSE | PSNR | SSIM
---|---|---|---|---|---
Baseline | 72.161 | 16207.693 | 0.0243 | 34.551 | 0.9761
MIND | 62.743 | 11303.564 | 0.01694 | 36.116 | 0.9852
Frequency loss $L_{1}$ | 71.387 | 16878.252 | 0.0253 | 34.375 | 0.9757
Frequency loss $L_{2}$ | 63.651 | 12046.446 | 0.01806 | 35.84 | 0.9826
MIND + Frequency loss $L_{1}$ | 75.337 | 17723.257 | 0.02657 | 34.163 | 0.9747
Table 3.7: Image similarity metrics on the phantom for experiments with
various structure losses. Note that all values with units are in a CT number
scale (between 0 to 3000) Figure 3.7: Mid-axial views of the CBCT and CT scans
of the phantom shown along with generated images from models with different
structure losses. Red dotted circles are used to highlight qualitative
observations that are discussed in-text.

MIND loss shows the best scores across all metrics. However, when looking at
the visuals for the phantom mid-axial slice, MIND loss does not correct values
very accurately as can be seen with the black region in the center of the
simulated heart. Frequency $L_{2}$ and baseline models do not correct tumour
values well (as indicated by the red circles in the CT, Baseline and Frequency
$L_{2}$ generated samples). Moreover, they also add checkerboard patterns,
which can be observed by zooming in on the image. The MIND + Frequency shows
regions that are much darker than the CT, as shown by the red circle.
Frequency $L_{1}$, similar to observations on the patient data, provides
robust translations on the phantom with neither GAN-induced artifacts nor loss
of quality. It is interesting to note that the Frequency $L_{1}$ metric values
are among the worst across all models.

###  3.5 Domain-specific Evaluation

In this section we present the results of domain-specific evaluation criteria
as described in Section 2.5.4. The Frequency loss with $L_{1}$, which is
chosen as the best-performing model, due to its robust performance across both
patient and phantom data, is used for subsequent evaluation. The translated
scan generated using this model is termed as the sCT (Synthetic CT) and will
be used to refer to it henceforth.

####  3.5.1 Histogram and Line Profiles

Figure 3.8 shows the histogram of HU intensity values between $(-500,500)$ on
the full scan for CT, CBCT and sCT. In addition, it also shows the line
profiles for the same set of scans. The line chosen for the profile is drawn
in red over the scans.

Figure 3.8: Histogram of HU intensities and line-profiles shown for CT, CBCT
and sCT on two patients chosen randomly from the test set. The line chosen for
profiling is highlighted by a red line passing the axial view of the images.

For both the patients, the sCT calibrates well with the CT in in terms of
distribution of HU values in the soft-tissue region, made easily observable
through the windowing. The sCT also matches the CT line profiles better
compared to the CBCT. This behaviour extends across all the patients in the
test set.

####  3.5.2 Automated Segmentation Task

The sCT is also evaluated on a downstream task of lung segmentation as
described in 2.5.4. The CT, CBCT and sCT are contoured for left and right lung
and compared with ground truth contours for the same, available in the
dataset. Figure 3.9 shows the box-plot of Dice scores obtained across all
patients along with ground truth and automated contours generated on a
randomly selected patient. Table 3.8 shows the mean Dice scores.

Figure 3.9: First row shows box plot showing Dice scores for segmentation of the right and left lung using CT, CBCT and sCT images as input. Second row shows visuals of segmentations generated by using different inputs to the automated segmentation pipeline, namely CT, CBCT and sCT. The ground truth segmentation, contoured by a clinician, is also shown. Note that all segmentations are shown on the CT as the original segmentations were drawn on the CT. | Left Lung | Right Lung
---|---|---
CT | 0.910984 | 0.913132
CBCT | 0.898207 | 0.902951
sCT | 0.900434 | 0.914695
Table 3.8: Mean Dice scores for left and right lung segmentations for the CT,
CBCT and sCT images as input to the automated segmentation model.

From Table 3.8, we see that the sCT segmentations, on average, provide
improved Dice scores when compared to CBCT segmentations. The mean Dice on the
sCT, even improves over the CT for the right lung. The visualization of
segmentation contours in Figure 3.9 shows a sample case where the CBCT is
worse than the sCT (highlighted in red). Note that similar behaviour, where
CBCT misses/adds parts of the contour, is seen in contours generated across
multiple patients in the test set.

##  Chapter 4 Discussion

In this section, we discuss the results obtained from various experiments
conducted and attempt to connect them to provide answers to the research
questions posed in Section 1.6.

Section 3.2 presents results for experimenting with different values of
existing CycleGAN parameters and aims to answer the first research question of
sufficiency of existing constraints. For map $\leftrightarrow$ aerial photo,
we identified that the default parameters provide the best results. However,
the results are underwhelming as they fail to retain complex structure present
in the aerial imagery. It seems that unpaired methods, in their existing
state, may have trouble capturing such structural complexities. This is also
corroborated through results from the original CycleGAN paper (Zhu et al.,,
2017). On the use-case of CBCT to CT, the structural complexity is vastly
reduced in comparison to aerial imagery, and much more acceptable results are
obtained using the default CycleGAN parameters. However, we identify failure
modes such as loss of quality through addition of checkerboard artifacts and
remnants of scattering artefacts in the generated images. Although it remains
to be evaluated as suitable for clinical use, it does not offer the CT-like
properties, especially in terms of smoothness.

Experiments and results associated with adaptation of constraints were shown
in Section 3.3. For map $\leftrightarrow$ aerial photo dataset, clear benefits
were not observed by adapting constraints, while it was even detrimental for
the CBCT-CT dataset. This is likely due to difficulty in associating control
signals to a strategy that effectively adapts constraints. As described in
Section 2.3, multiple strategies were explored to go from sensing control
signals to using that information to adapt $\lambda$ values. This was
extremely challenging due to the large space of possibilities on when and how
to adapt the parameters. This association is even more challenging for 3D
models as they are trained for lesser number of iterations which might lead to
design of more aggressive adapting strategies. The discriminator-based control
signal completely failed in driving $\lambda$ changes while the cycle-
consistency control signal was better, possibly due to the direct relationship
between the cycle-consistency control signal and the constraints. As
experimentally validated with the discriminator-based control signal, a single
control signal may not always directly advise a change in constraints,
therefore, better ways to indirectly use them to adapt parameters need to be
investigated. To address this, looking at more than one control signal at a
time was a potential idea. For instance, looking at the cycle-consistency
signal along with variance of adversarial losses as a second control signal.
However, it was dropped due to the large number of experiment configurations
that would need to be investigated. More appropriately, multi-objective
optimization methods such as hyper-volume optimization may prove to work well
for these methods since they deal with balancing constraints in a more optimal
way than simple additive combination.

Various data-driven constraints and their results on translation are shown in
Section 3.4. We demonstrated that losses built for other medical imaging tasks
may not work too well when introduced into the CycleGAN framework. Although
MIND performed satisfactorily on quantitative scores, it rendered the images
unusable due to large modifications made to patient anatomy. Registration
losses, that seem to offer properties that would benefit structure retention,
as presented in Section 2.4.2, failed to converge across multiple settings
when combined with the rest of the constraints in the CycleGAN. In contrast,
simple frequency-based losses seem to combine well with existing constraints
in the CycleGAN and provide translations with desirable qualitative and
quantitative scores. Combination of MIND with frequency-based loss seemed to
get rid of the GAN-induced artifacts but the combination performed poorly on
quantitative scores and showed other qualitative issues.

Another major observation is the insufficiency of solely relying on
quantitative analysis in choosing the best model. This is observed even with
out-of-distribution data where strong pairs were formed. For instance,
Frequency $L1$ provides one of the worst scores on the phantom but is superior
to the other models as it does not induce artifacts or result in reduction in
quality that all other models were susceptible to. This puts forward the
question if existing image similarity metrics can be relied on fully to
evaluate such methods. Research into evaluation methods that can sufficiently
capture these properties in generated images would push the field closer
toward general and clinical acceptance. Gragnaniello et al., (2021) present a
review of existing methods for synthetic image detection and propose potential
research areas for the future. These methods could also help in quantitatively
determining undesirable additions such as artifacts in the generated images.

Domain-specific methods of evaluation can provide good insight into the
clinical usability of a particular set of methods. As seen in Section 3.5,
synthetic CT generated from the best-performing model provided HU intensity
distribution and line profiles inline with real CTs. Automated segmentation on
the synthetic CT showed performance on par with real CT (even better for the
right lung) and improvements from the original CBCT. Given the simplicity of
the translation process, it can be integrated into existing clinical workflows
to improve the quality of the CBCT. The improved CBCT can be useful for
multiple downstream tasks, from improving auto-contouring to adapting
treatment plans.

##  Chapter 5 Conclusion

In our work, we investigated various constraints that affect CycleGAN-based
CBCT enhancement along with supporting methodology and evaluation. We began by
exploring existing CycleGAN constraints to determine if they are sufficient
for generating high-quality images, in order to answer our first research
question. Through our qualitative and quantitative evaluations, we conclude
that although an improvement in calibration is observed when compared to the
original, the image quality is impaired through the addition of minor
artifacts and decreased resolution. As this is undesirable for clinical
acceptability, we looked at possible solutions through answering the
subsequent questions that the thesis proposed. In order to answer our second
research question, we designed adaptive control strategies to dynamically
balance cycle-consistency and adversarial losses,to gauge their impact on
generating improved CBCT images. However, this approach was not successful in
improving image quality, demonstrated both qualitatively and quantitatively.
We then reported the complexity of designing such strategies, due to the large
space of control signals and adaptive strategy choices that need to be made,
and subsequently proposed more optimal multi-objective methods as future work.
Finally, we looked at data-driven constraints, in the form of structure-
consistency losses, as part of investigating our third research question. This
investigation proved to be most successful, with the best method, the
generalized frequency loss, improving MAE, MSE, NMSE, and PSNR, by 4%, 20%,
18%, and 3%, respectively, compared to the baseline. The generalized frequency
loss, implemented as a part of this thesis, proved to not only improve over
the baseline, but also outperform existing methods, such as the MIND loss
(Yang et al.,, 2018). This was done at a much lesser memory and compute cost.
More importantly, in terms of qualitative comparison, we observed the best
performance, with no drops in image quality or any addition of artifacts. As a
part of answering our fourth research question, all our evaluations were
conducted using image-similarity based quantitative evaluation and structured
qualitative evaluation. We leveraged weak pairs of patient data and a strong
pair of phantom data obtained through image-registration. The phantom data was
also used as out-of-distribution data. Efficacy of our methods were
corroborated by subjecting them to out-of-distribution data, where they
demonstrated sufficient accuracy and did not provide unexpected results.

One of the core goals of this thesis was providing a more reliable and robust
method for CBCT improvement. This improved CBCT can benefit various adaptive
radiotherapy workflows in the clinic such as auto-contouring, image
registration, and dosimetry. Improvements in these worflows can save
clinicians’ valuable time and effort along with reduction of costs associated
with repeated imagery. Improved CBCT can not only benefit the clinic, but also
the patient as improved quality CBCTs can mean lesser CT scans and therefore,
lesser radiation exposure. In an extension to the fourth research question, we
implemented clinically motivated evaluations such as HU intensity distribution
comparisons and line profiles, where we demonstrated that the improved CBCT
matches the fan-beam CT very well. Furthermore, the value of improved CBCTs in
down-stream tasks was shown through a comparison of contours generated through
lung auto-segmentation. Mean Dice scores of contours on the improved CBCT were
comparable to the fan-beam CT and surpassed the CBCT. We point out that a full
dosimetric evaluation would help better establish clinical applicability of
our methods. However, due to complexity of designing clinically acceptable
treatment plans, we exclude this in the thesis and propose it as future work.
Another promising line of future work is the development of quantitative
methods that can capture artifacts in generated images so as to decrease the
reliance on subjective qualitative inspection, which is heavily utilized in
this thesis.

## References

  * Buzug, (2011) Buzug, T. M. (2011).  Computed Tomography, pages 311–342.  Springer Berlin Heidelberg, Berlin, Heidelberg.
  * Dahiya et al., (2021) Dahiya, N., Alam, S. R., Zhang, P., Zhang, S., Yezzi, A. J., and Nadeem, S. (2021).  Multitask 3d cbct-to-ct translation and organs-at-risk segmentation using physics-based data augmentation.  CoRR, abs/2103.05690.
  * de Raad et al., (2021) de Raad, K., van Garderen, K., Smits, M., van der Voort, S., Incekara, F., Oei, E., Hirvasniemi, J., Klein, S., and Starmans, M. (2021).  The effect of preprocessing on convolutional neural networks for medical image segmentation.  In 2021 IEEE 18th International Symposium on Biomedical Imaging (ISBI), pages 655–658.
  * Doi, (2006) Doi, K. (2006).  Diagnostic imaging over the last 50 years: research and development in medical imaging science and technology.  Physics in Medicine and Biology, 51(13):R5—-R27.
  * Dunlop et al., (2015) Dunlop, A., McQuaid, D., Nill, S., Murray, J., Poludniowski, G., Hansen, V. N., Bhide, S., Nutting, C., Harrington, K., Newbold, K., and Oelfke, U. (2015).  Vergleich unterschiedlicher CT-Kalibrierungsmethoden zur Dosisberechnung auf Basis der Kegelstrahlcomputertomographie.  Strahlentherapie und Onkologie, 191(12):970–978.
  * Eckl et al., (2020) Eckl, M., Hoppen, L., Sarria, G. R., Boda-Heggemann, J., Simeonova-Chergou, A., Steil, V., Giordano, F. A., and Fleckenstein, J. (2020).  Evaluation of a cycle-generative adversarial network-based cone-beam CT to synthetic CT conversion algorithm for adaptive radiation therapy.  Physica Medica, 80(November):308–316.
  * Frangioni, (2008) Frangioni, J. V. (2008).  New Technologies for Human Cancer Imaging.  26(24).
  * Gebru et al., (2020) Gebru, T., Morgenstern, J., Vecchione, B., Vaughan, J. W., Wallach, H., au2, H. D. I., and Crawford, K. (2020).  Datasheets for datasets.
  * Goodfellow et al., (2014) Goodfellow, I. J., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. (2014).  Generative adversarial networks.
  * Gragnaniello et al., (2021) Gragnaniello, D., Cozzolino, D., Marra, F., Poggi, G., and Verdoliva, L. (2021).  Are GAN generated images easy to detect? A critical analysis of the state-of-the-art.  CoRR, abs/2104.02617.
  * Hadzic et al., (2021) Hadzic, I., Pai, S., Rao, C., and Teuwen, J. (2021).  ganslate-team/ganslate: a simple and extensible gan image-to-image translation framework.
  * Harms et al., (2019) Harms, J., Lei, Y., Wang, T., Zhang, R., Zhou, J., Tang, X., Curran, W. J., Liu, T., and Yang, X. (2019).  Paired cycle-GAN-based image correction for quantitative cone-beam computed tomography.  Medical Physics, 46(9):3998–4009.
  * Heinrich et al., (2012) Heinrich, M., Jenkinson, M., Bhushan, M., Matin, T., Gleeson, F., Brady, M., and Schnabel, J. (2012).  MIND: Modality Independent Neighbourhood Descriptor for Multi-Modal Deformable Registration.  Medical image analysis, 16:1423–1435.
  * Hofmanninger et al., (2020) Hofmanninger, J., Prayer, F., Pan, J., Röhrich, S., Prosch, H., and Langs, G. (2020).  Automatic lung segmentation in routine imaging is primarily a data diversity problem, not a methodology problem.  European Radiology Experimental, 4(1).
  * in Medicine, (1977) in Medicine, A. A. o. P. (1977).  Phantoms for performance evaluation and quality assurance of CT scanners.
  * Isola et al., (2016) Isola, P., Zhu, J., Zhou, T., and Efros, A. A. (2016).  Image-to-image translation with conditional adversarial networks.  CoRR, abs/1611.07004.
  * Jiang et al., (2020) Jiang, L., Dai, B., Wu, W., and Loy, C. C. (2020).  Focal frequency loss for image reconstruction and synthesis.  arXiv preprint, arXiv:2012.12821.
  * Jin et al., (2010) Jin, J.-Y., Ren, L., Liu, Q., Kim, J., Wen, N., Guan, H., Movsas, B., and Chetty, I. J. (2010).  Combining scatter reduction and correction to improve image quality in cone-beam computed tomography (CBCT).  Medical Physics, 37(11):5634–5644.
  * Karras et al., (2020) Karras, T., Aittala, M., Hellsten, J., Laine, S., Lehtinen, J., and Aila, T. (2020).  Training generative adversarial networks with limited data.
  * Kida et al., (2020) Kida, S., Kaji, S., Nawa, K., Imae, T., Nakamoto, T., Ozaki, S., Ohta, T., Nozawa, Y., and Nakagawa, K. (2020).  Visual enhancement of Cone-beam CT by use of CycleGAN.  Medical Physics, 47(3):998–1010.
  * Kida et al., (2018) Kida, S., Nakamoto, T., Nakano, M., Nawa, K., Haga, A., Kotoku, J., Yamashita, H., and Nakagawa, K. (2018).  Cone Beam Computed Tomography Image Quality Improvement Using a Deep Convolutional Neural Network.  Cureus, 10(4).
  * Kurz et al., (2019) Kurz, C., Maspero, M., Savenije, M. H., Landry, G., Kamp, F., Pinto, M., Li, M., Parodi, K., Belka, C., and Van Den Berg, C. A. (2019).  CBCT correction using a cycle-consistent generative adversarial network and unpaired training to enable photon and proton dose calculation.  Physics in Medicine and Biology, 64(22).
  * Landry et al., (2019) Landry, G., Hansen, D., Kamp, F., Li, M., Hoyle, B., Weller, J., Parodi, K., Belka, C., and Kurz, C. (2019).  Corrigendum: Comparing Unet training with three different datasets to correct CBCT images for prostate radiotherapy dose calculations (Physics in Medicine and Biology (2019) 64 (035011) DOI: 10.1088/1361-6560/aaf496).  Physics in Medicine and Biology, 64(8).
  * Landry et al., (2015) Landry, G., Nijhuis, R., Dedes, G., Handrack, J., Thieke, C., Janssens, G., de Xivry, J., Reiner, M., Kamp, F., Wilkens, J. J., Paganelli, C., Riboldi, M., Baroni, G., Ganswindt, U., Belka, C., and Parodi, K. (2015).  Investigating CT to CBCT image registration for head and neck proton therapy as a tool for daily dose recalculation.  Medical Physics, 42(3):1354–1366.
  * Lechuga and Weidlich, (2016) Lechuga, L. and Weidlich, G. A. (2016).  Cone Beam CT vs. Fan Beam CT: A Comparison of Image Quality and Dose Delivered Between Two Differing CT Imaging Modalities.  Cureus, 8(9).
  * Ling et al., (2006) Ling, C. C., Yorke, E., and Fuks, Z. (2006).  From IMRT to IGRT: Frontierland or Neverland?  Radiotherapy and Oncology, 78(2):119–122.
  * Liu et al., (2020) Liu, Y., Lei, Y., Wang, T., Fu, Y., Tang, X., Curran, W. J., Liu, T., Patel, P., and Yang, X. (2020).  CBCT-based synthetic CT generation using deep-attention cycleGAN for pancreatic adaptive radiotherapy.  Medical Physics, 47(6):2472–2483.
  * Lowekamp et al., (2013) Lowekamp, B., Chen, D., Ibanez, L., and Blezek, D. (2013).  The Design of SimpleITK.  Frontiers in Neuroinformatics, 7:45.
  * Ma et al., (2021) Ma, N., Li, W., Brown, R., Wang, Y., Behrooz, Gorman, B., Johnson, H., Yang, I., Kerfoot, E., charliebudd, Li, Y., Adil, M., Y.-T. H., Aggarwal, A., Trentz, C., adam aji, masadcv, Graham, M., Murray, B., Daroach, G., Tudosiu, P.-D., myron, McCormick, M., Ambros, Balamurali, Baker, C., Sellner, J., Fidon, L., and cgrain (2021).  Project-monai/monai: 0.5.3.
  * Marstal et al., (2016) Marstal, K., Berendsen, F., Staring, M., and Klein, S. (2016).  Simpleelastix: A user-friendly, multi-lingual library for medical image registration.  In 2016 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pages 574–582.
  * Maspero et al., (2020) Maspero, M., Houweling, A. C., Savenije, M. H., van Heijst, T. C., Verhoeff, J. J., Kotte, A. N., and van den Berg, C. A. (2020).  A single neural network for cone-beam computed tomography-based radiotherapy of head-and-neck, lung and breast cancer.  Physics and Imaging in Radiation Oncology, 14(April):24–31.
  * Milletari et al., (2016) Milletari, F., Navab, N., and Ahmadi, S.-A. (2016).  V-net: Fully convolutional neural networks for volumetric medical image segmentation.
  * Mirza and Osindero, (2014) Mirza, M. and Osindero, S. (2014).  Conditional generative adversarial nets.
  * Mitchell et al., (2019) Mitchell, M., Wu, S., Zaldivar, A., Barnes, P., Vasserman, L., Hutchinson, B., Spitzer, E., Raji, I. D., and Gebru, T. (2019).  Model cards for model reporting.  Proceedings of the Conference on Fairness, Accountability, and Transparency.
  * Posiewnik and Piotrowski, (2019) Posiewnik, M. and Piotrowski, T. (2019).  A review of cone-beam CT applications for adaptive radiotherapy of prostate cancer.  Physica Medica, 59(October 2018):13–21.
  * Ronneberger et al., (2015) Ronneberger, O., Fischer, P., and Brox, T. (2015).  U-net: Convolutional networks for biomedical image segmentation.  CoRR, abs/1505.04597.
  * Schulze et al., (2011) Schulze, R., Heil, U., Gro$\beta$, D., Bruellmann, D. D., Dranischnikow, E., Schwanecke, U., and Schoemer, E. (2011).  Artefacts in CBCT: a review.  Dentomaxillofacial Radiology, 40(5):265–273.
  * Shrivastava et al., (2017) Shrivastava, A., Pfister, T., Tuzel, O., Susskind, J., Wang, W., and Webb, R. (2017).  Learning from simulated and unsupervised images through adversarial training.
  * Sonke and Belderbos, (2010) Sonke, J.-J. and Belderbos, J. (2010).  Adaptive Radiotherapy for Lung Cancer.  Seminars in Radiation Oncology, 20(2):94–106.
  * Sterzing et al., (2011) Sterzing, F., Engenhart-Cabillic, R., Flentje, M., and Debus, J. (2011).  Optionen der bildgestützten bestrahlung: Eine neue dimension in der radioonkologie.  Deutsches Arzteblatt, 108(16):274–280.
  * Thummerer et al., (2020) Thummerer, A., Zaffino, P., Meijers, A., Marmitt, G. G., Seco, J., Steenbakkers, R. J., Langendijk, J. A., Both, S., Spadea, M. F., and Knopf, A. C. (2020).  Comparison of CBCT based synthetic CT methods suitable for proton dose calculations in adaptive proton therapy.  Physics in Medicine and Biology, 65(9).
  * Wang et al., (2004) Wang, Z., Bovik, A. C., Sheikh, H. R., and Simoncelli, E. P. (2004).  Image quality assessment: from error visibility to structural similarity.  IEEE transactions on image processing, 13(4):600–612.
  * Yang et al., (2018) Yang, H., Sun, J., Carass, A., Zhao, C., Lee, J., Xu, Z., and Prince, J. (2018).  Unpaired brain mr-to-ct synthesis using a structure-constrained cyclegan.
  * Yuan et al., (2020) Yuan, N., Dyer, B., Rao, S., Chen, Q., Benedict, S., Shang, L., Kang, Y., Qi, J., and Rong, Y. (2020).  Convolutional neural network enhancement of fast-scan low-dose cone-beam CT images for head and neck radiotherapy.  Physics in Medicine and Biology, 65(3).
  * Zhang et al., (2019) Zhang, X., Karaman, S., and Chang, S.-F. (2019).  Detecting and simulating artifacts in gan fake images.
  * Zhang et al., (2020) Zhang, Y., Yue, N., Su, M.-Y., Liu, B., Ding, Y., Zhou, Y., Wang, H., Kuang, Y., and Nie, K. (2020).  Improving CBCT quality to CT level using deep learning with generative adversarial network.  Medical Physics, n/a(n/a).
  * Zhao et al., (2016) Zhao, W., Vernekohl, D., Zhu, J., Wang, L., and Xing, L. (2016).  A model-based scatter artifacts correction for cone beam CT.  Medical Physics, 43(4):1736–1753.
  * Zhu et al., (2017) Zhu, J. Y., Park, T., Isola, P., and Efros, A. A. (2017).  Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks.  In Proceedings of the IEEE International Conference on Computer Vision, volume 2017-Octob, pages 2242–2251.

Figure 1: Block diagram of the 2D base configuration architecture. It consists
of 7 down and up convolutional blocks with skip connections as described in
Section 3.1.1. Figure 2: Block diagram of the 3D base configuration
architecture. It comprises of input blocks, output blocks, up blocks and down
blocks along with skip connections as described in Section 3.1.1.