# T2V-DDPM: Thermal to Visible Face Translation using Denoising Diffusion
Probabilistic Models

Nithin Gopalakrishnan Nair and Vishal M. Patel
Department of Electrical and Computer Engineering, Johns Hopkins University,
Baltimore, USA This research was supported by NSF CAREER award 2045489.

###### Abstract

Modern-day surveillance systems perform person recognition using deep
learning-based face verification networks. Most state-of-the-art facial
verification systems are trained using visible spectrum images. But, acquiring
images in the visible spectrum is impractical in scenarios of low-light and
nighttime conditions, and often images are captured in an alternate domain
such as the thermal infrared domain. Facial verification in thermal images is
often performed after retrieving the corresponding visible domain images. This
is a well-established problem often known as the Thermal-to-Visible (T2V)
image translation. In this paper, we propose a Denoising Diffusion
Probabilistic Model (DDPM) based solution for T2V translation specifically for
facial images. During training, the model learns the conditional distribution
of visible facial images given their corresponding thermal image through the
diffusion process. During inference, the visible domain image is obtained by
starting from Gaussian noise and performing denoising repeatedly. The existing
inference process for DDPMs is stochastic and time-consuming. Hence, we
propose a novel inference strategy for speeding up the inference time of
DDPMs, specifically for the problem of T2V image translation. We achieve the
state-of-the-art results on multiple datasets. The code and pretrained models
are publically available at http://github.com/Nithin-GK/T2V-DDPM

##  I INTRODUCTION

Many surveillance systems include sensors to capture images at multiple
wavelengths to accommodate day and nighttime settings. Under low-light
settings, the images captured by visible spectrum cameras fail to capture
visual details in the scene due to the increasing amount of additive Poisson
noise on the captured image. Hence, the use of infrared cameras to capture an
additional thermal image is quite prominent. The images captured by these
surveillance systems are further used for person recognition using facial
recognition algorithms, and this falls under the broad area of research called
Heterogeneous Face Recognition (HFR) [12, 13, 14]. Deep Convolutional Neural
Network (CNN) based algorithms have produced state-of-the-art results for
facial recognition with almost perfect accuracy in multiple benchmarks [1,
18].

Although the existing facial verification algorithms work remarkably well
during daytime, the performance of these algorithms fall drastically under
low-light settings, this is due to the large domain discrepancy between
thermal and visible images. One solution is to retrain a network for facial
recognition using just thermal images. But this is impractical since the
normal CNN-based facial recognition networks [1] often require large amounts
of face images to obtain good performance and there does not exist many
publicaly available large scale thermal images to train such networks. Hence
the problem of HFR is very relevant and has multiple practical implications.

(a) $T=100$

(b) $T=86$

(c) $T=72$

(d) $T=58$

(e) $T=44$

(f) $T=29$

(g) $T=15$

(h) $T=0$

Figure 1: Visualization of time re-scaled diffusion process during inference.

The recent rise of conditional generative models [5] has enabled an alternate
approach to address this problem through a two-step process. First, the
problem is simplified to an image-to-image translation problem from thermal to
visible domains. After the corresponding visible images are obtained, facial
recognition algorithms can be directly applied to them. Multiple works utilize
GANs for tackling this problem [16, 2, 8]. Conditional GANs attempt to learn
the translation from the thermal domain to the visible domain through a min-
max approach. Although with sufficient training, GAN-based T2V translation
networks can produce good results, training of GANs is a tedious process and
may result in phenomena like mode collapse. Also, given limited data, the
convergence of GANs for accurate T2V translation is not guaranteed.

Recently, Denoising Diffusion Probabilistic Models (DDPMs) [7] have gained a
significant attention due to their ability to generate high-quality images.
Like Variational Autoencoders [11], DDPMs attempt to learn the variational
lower bound of the log-likelihood of the data distribution. DDPMs have already
beaten GANs in the task of image generation. Multiple methods have proposed
DDPMs for low-level vision tasks like image super resolution, colorization,
deblurring, and denoising [22, 21, 10, 15]. These methods have performed much
better in terms of quality of the output produced based on realness metrics
like Fréchet Inception Distance (FID) [6] and structural similarity metrics
like Structural Similarity Index (SSIM). Moreover, there are works connecting
the ability of DDPM to learn the conditional distribution to the optimal
transport theory of the most efficient transformation [24]. Despite its
immense potential in modeling conditional distributions, there are no works
yet published for the problem of T2V face image translation. In this paper, we
propose a solution for T2V face translation using DDPMs. Since the model
attempts to learn the transformation, very few image pairs are required for
training the model to achieve good results. We show this in Figure 4 where we
can clearly see the performance of DDPM-based models with just a few training
image pairs.

One disadvantage of DDPMs over conventional CNNs is that the inference process
is quite slow because of the underlying Markovian chain, which requires
multiple forward passes through a single neural network. But through our
experiments we have observed that the initial steps of the DDPM inference
process aim to recover the high-level image details, like structure and color.
This information can be easily captured from the corresponding thermal image;
hence, rather than starting from isotropic Gaussian noise, we start with the
noised thermal image. Thereby skipping some steps and speeding up the
inference process.

Infrared cameras capture images in different wavelengths within the infrared
spectrum based on the application. Two common modalities are the Near Infrared
(NIR) and Long wave Infrared (LWIR). The choice between NIR and LWIR is based
on whether it is a short-range or long-range surveillance application. NIR
images consist of images close to the visible spectrum; hence the resolution
of these images is naturally high and can capture finer details in the facial
image captured. LWIR images are often used for long-range surveillance
applications, but these images are often shot in low-resolution and fail to
capture essential facial details. This could be clearly seen when we compare
the facial recognition accuracies when HFR algorithms are applied to these
images. NIR face datasets [3, 4] produce accuracies close to $99\%$ but those
of LWIR datasets [8, 19] are significantly lower. In this paper, we mainly
focus on HFR for LWIR datasets.

In short, this paper makes the following contributions

  * •

We propose a DDPM-based solution for the HFR problem. We focus on the image
translation problem from LWIR thermal images to their corresponding visible
images.

  * •

We introduce a novel inference strategy for DDPMs mainly applicable to the T2V
face translation problem. Through this we achieve upto $(2\times)$ speed up on
the inference times without any drop in performance.

  * •

We evaluate our method qualitatively and quantitatively on real-world datasets
and show that it performs better than the existing methods on T2V face
translation problem.

Figure 2: An overview of the proposed method. During training the thermal
image is conditioned along with the noisy visible image. During inference, we
skip some timesteps by starting with a coarse image consisting of the
background colour and the coarse features.

##  II Proposed Method

###  II-A Denoising Diffusion Probabilistic Models

Denoising Diffusion Probabilistic models [23, 7] belong to a class of
generative models where the models learn the distribution of data through a
Markovian process. DDPMs consist of a forward process and a reverse process.
The joint distribution of the data is represented by the reverse process. The
forward process is a Markovian process where the next state is obtained by
sampling from a Gaussian distribution whose mean depends on the current state
of the system, a predefined variance schedule as well as the time $t$. The
sampling operation for the forward step is given by

$q(x_{t}|x_{t-1})=\mathcal{N}(x_{t};\sqrt{1-\beta_{t}}x_{t-1},\beta_{t}\bm{I}),$
(1)

where $\\{\beta_{t}\\}$ is a predefined variance schedule. In practical
scenarios, $t$ usually takes values between $10^{-4}$ to $10^{-2}$. This could
also be considered as a noising operation, where the next state is obtained
from the current state by adding a small Gaussian noise with variance schedule
$\\{\beta_{t}\\}$. The state at timestep $t$ can also be computed from the
initial state $x_{0}$. the distribution of the state at a particular timestep
$t$ given $x_{0}$ is,

$q(x_{t}|x_{0})=\mathcal{N}(x_{t};\sqrt{\bar{\alpha}_{t}}x_{0},(1-\bar{\alpha}_{t})\bm{I}).$
(2)

Equivalently,

$x_{t}=\bar{\alpha}_{t}x_{0}+(1-\bar{\alpha}_{t})\epsilon,\epsilon\sim\mathcal{N}(0,I),$
(3)

where, $\bar{\alpha}_{t}=\prod_{s=1}^{t}\alpha_{s}$ and
$\alpha_{t}=1-\beta_{t}$. As can be seen from Eq. 2, for large $t$,
$\bar{\alpha}_{t}$ becomes zero. Hence it results in a standard isotropic
Gaussian. The reverse process denotes a generation step where we start from a
standard Gaussian and iteratively perform denoising through $t$ timesteps to
generate an image corresponding to the training distribution. [23] has shown
that when the number of timesteps is large and the increment in
$\\{\beta_{i}\\}$ is small, then the reverse distribution could be also
approximated by a Gaussian. Each step in the reverse process is performed by
sampling from a distribution whose parameters are modelled using a neural
network with parameters $\theta$ and each reverse step is defined by,

$p_{\theta}(x_{t-1}|x_{t})=\mathcal{N}(\mu_{\theta}(x_{t},t),\Sigma_{\theta}(x_{t},t)).$
(4)

The parameters $\theta$ are obtained by minimizing the variational lower bound
of the negative log-likelihood of the data distribution defined by

$E[-log(p(x_{0})]\leq E_{q}[\frac{p_{\theta}(x_{0:T})}{q(x_{1:T}|x_{0})}]=L.$
(5)

Further simplifying,

$L=\mathbb{E}_{q}\bigg{[}\underbrace{D_{KL}(q(\mathbf{x}_{T}|\mathbf{x}_{0})\,||\,p(\mathbf{x}_{T}))}_{L_{T}}\underbrace{-\text{log}\,p_{\theta}(\mathbf{x}_{0}|\mathbf{x}_{1})}_{L_{0}}\\\
+\sum_{t>1}\underbrace{D_{KL}(q(\mathbf{x}_{t-1}|\mathbf{x}_{t},\mathbf{x}_{0})\,||\,p(\mathbf{x}_{t-1}|\mathbf{x}_{t}))}_{L_{t-1}}\bigg{]}.$
(6)

Recent works [7] have shown the variance schedule for the reverse step could
be kept the same as the forward process and only the means need to be learned
using a neural network. Furthermore using the input at the current timestep,
the mean of the reverse step could be reparametrized according to

$\mu_{\theta}(\mathbf{x}_{t},t)=\frac{1}{\sqrt{\alpha_{t}}}\bigg{(}\mathbf{x}_{t}-\frac{\beta_{t}}{\sqrt{1-\bar{\alpha}_{t}}}\epsilon_{\theta}(\mathbf{x}_{t},t)\bigg{)}.$
(7)

Also given an $x_{0}$, $x_{t}$, the mean of the distribution
$q(\mathbf{x}_{t-1}|\mathbf{x}_{t},\mathbf{x}_{0})$ can be written as

$\mu(\mathbf{x}_{t},t)=\frac{1}{\sqrt{\alpha_{t}}}\bigg{(}\mathbf{x}_{t}-\frac{\beta_{t}}{\sqrt{1-\bar{\alpha}_{t}}}\epsilon(\mathbf{x}_{t},t)\bigg{)}.$
(8)

Equating the means of the distributions, the training objective could be
further[7] simplified to,

$L_{simple}=\mathbb{E}_{t,\mathbf{x}_{0},\epsilon}\big{[}\|\epsilon-\epsilon_{\theta}(\mathbf{x}_{t},t)\|^{2}\big{]}.$
(9)

Once the data distribution is learned, new samples can be derived from the
data distribution, starting from a Gaussian random sample and following a
Markovian process.

###  II-B Conditional Diffusion Models for T2V Generation

Saharia et al. [22] proposed a method for conditional generation using DDPM
where the images are generated based on a constraint, and the model learns the
conditional distribution given a condition $y$. Learning the conditional
distribution rather than the unconditional distribution enables the use of
DDPMs for low-level vision problems like image restoration. To make the model
learn the conditional distribution, we condition the neural network with the
thermal image $y$ at all timesteps $t$. Here the effective training objective
is defined by,

$L_{simple}=\mathbb{E}_{t,\mathbf{x}_{0},\epsilon}\big{[}\|\epsilon-\epsilon_{\theta}(\mathbf{x}_{t},y,t)\|^{2}\big{]}.$
(10)

Once the conditional distribution is learnt, the model can be used for
inference, i.e., given a thermal image, we can sample the corresponding
visible image by starting with pure Gaussian noise and sampling from a Markov
chain of $T$ steps. The corresponding mean of the conditional distribution for
the transition $p_{\theta}(x_{t-1}|x_{t},y)$ is

$\mu_{\theta}(\mathbf{x}_{t},t)=\frac{1}{\sqrt{\alpha_{t}}}\bigg{(}\mathbf{x}_{t}-\frac{\beta_{t}}{\sqrt{1-\bar{\alpha}_{t}}}\epsilon_{\theta}(\mathbf{x}_{t},y,t)\bigg{)}.$
(11)

and the variance schedule remains the same as that during training.

###  II-C Speeding up conditional diffusion models

From our experiments we made a key observation that forms the basis for this
section. As can be seen from Figure 3, which shows how the generated image
varies with time, the low-level semantics of the image is learnt much further
into the diffusion process. The initial steps are aimed at learning coarse
features like the shape of the underlying face and background color. This
means if we have an image with coarse facial features but lacks finer details,
it can possibly be utilized to speed up the reconstruction process. Motivated
by this empirical observation, we generate an image with coarse features
through the following process. Given the thermal image $y$ and the visible
image, $x$ normalized in the range $[0,1]$, we binarize the thermal image to
generate a mask $m$ through the following operation

$m=\begin{cases}1&\text{if }x>\epsilon\\\ 0&\text{otherwise}.\end{cases}$ (12)

(a) $y$

(b) $y^{c}$

(c) $y^{c}_{t1}$

(d) $x_{0}^{c}$

(e) $x^{c}_{50}$

(f) $x^{c}_{0}$

(g) $x$

Figure 3: (a) Thermal Image, (b) Masked thermal image (c) Noise masked thermal
image(T=50) (d) Reconstructed masked thermal image(T=100) (e) Intermediate
diffusion process result image starting from Gaussian noise visualized at
(T=50) (f) Reconstructed image starting from (T=0). (g) Visible Image

Since the background colour is constant for almost all images, we sample the
background colour from a random image in the training dataset and form a grid
$c$ of the size of the image. The coarse image $y^{c}$ is created according to

$y^{c}=m\cdot y+(1-m)\cdot c.$ (13)

Using $y^{c}$, the noisy version of $y^{c}$ denoted by $y^{c}_{T_{r}}$
corresponding to time $T_{r}$ is created. Figure 3 shows a visualization of
$y^{c}$, $y^{c}_{T_{r}}$ as well as $x_{T_{r}}$ starting from an isotropic
Gaussian noise. From the visualization, one can note that $x_{T_{r}}$ and
$y^{c}_{T_{r}}$ for $T_{r}=50$ does not differ much semantically. Moreover the
SNR values of both the images are almost the same.

Algorithm 1 Training

1:Thermal image and visible image pairs
$P=\\{(\mathbf{y}^{k},\mathbf{x}^{k})\\}^{K}_{k=1}$

2:repeat

3: $(\mathbf{y},\mathbf{x})\sim P$

4: $t\sim\text{Uniform}(\\{1,\ldots,T\\})$

5: $\epsilon\sim\mathcal{N}(\mathbf{0},\mathbf{I})$

6:
$\mathbf{x}_{t}=\sqrt{\bar{\alpha_{t}}\mathbf{x}_{c}+\sqrt{1-\bar{\alpha_{t}}}\epsilon}$

7: Gradient descent step on
$\nabla_{\theta}\,||\epsilon-\epsilon_{\theta}(\mathbf{x}_{t},\mathbf{y},t)||^{2}$

8:until converged

Algorithm 2 Inference

1:Thermal image $\mathbf{x}$, background colour $c$

2:Create course image $x^{c}$ according to Equation 13

3:Sample $x^{c}_{T_{r}}\sim q(x^{c}_{T_{r}}|x^{c})$

4:for $t=T_{r},\ldots,1$ do

5: sample $\mathbf{z}\sim\mathcal{N}(\mathbf{0},\mathbf{I})$ if $t>1$ else
$\mathbf{z}=0$

6: compute
$\mathbf{x}^{c}_{t-1}=\frac{1}{\sqrt{\alpha_{t}}}\Big{(}\mathbf{x}^{c}_{t}-\frac{1-\alpha_{t}}{\sqrt{1-\bar{\alpha_{t}}}}\epsilon_{\theta}(\mathbf{x}^{c}_{t},\mathbf{x},t)\Big{)}+\sigma_{t}\mathbf{z}$

7:end for

8:return $x^{c}_{0}$

Algorithm 1 summarises the training procedure of the proposed method. The
thermal image and their corresponding noisy visible images are used for
training a DDPM model for $T$ timesteps. During inference, we reduce the
inference time of the diffusion process by starting from a noisy coarse image
than isotropic Gaussian noise. The inference procedure is given in Algorithm
2.

(a) THERMAL

(b) PIX2PIX[9]

(c) SAGAN[2]

(d) GANVFS[26]

(e) HIFACEGAN[25]

(f) AXIALGAN[8]

(g) OURS

(h) VISIBLE

Figure 4: Qualitative results for the thermal to visible translation tasks for
facial images on a challenging dataset (VIS-TH dataset). Due to the low number
of training image pairs, existing methods fail to reconstruct facial images
accuractely, see coloumns(1,2,3,4) in the results and doesn’t generalize well
over unseen poses. On contrast Our method is able to generalize well given a
low number of training image pairs.

##  III Experiments

In our experiments, we focus on the translation problem from thermal images of
dimension $128\times 128$ to visible images of size $128\times 128$. The main
emphasis of our experiments is on improving the facial recognition accuracy of
the reconstructed images, but we also perform comparisons with different
metrics. Until now, there are no standardized baselines for T2V face
translation. Hence we make use of the datasets used in the work [16] for our
experiments. We perform experiments on three different datasets for T2V face
translation. More details about the individual datasets and their evaluation
criteria are presented in this section.

VIS-TH dataset [16]:- The VIS-TH dataset consists of facial images
corresponding to 50 different identities. Images in the VIS-TH dataset are
captured through a dual-sensor camera in Long Wave Infrared (LWIR) modality
and are aligned. The facial images in the dataset consist of various poses. We
create the train-test split for the dataset in the same fashion as followed by
[16, 8]. We use all the images corresponding to 40 randomly selected
identities as the training set and the remaining images as the test set. VIS-
TH is quite a challenging dataset owing to the low number of identities and
the diversity of the dataset.

ARL-VTF dataset[20]:- Like the VIS-TH dataset, the RL-VTF dataset also
consists of facial images captured in the LWIR modality. The dataset also
provides the image capture settings for aligning the faces. Visible images in
the ARL-VTF dataset are severely overexposed. Hence we correct this
overexposure through exposure matching with the VIS-TH dataset. We create a
subset of the original ARL-VTF dataset for all our experiments and choose 100
identities with different expressions as the training dataset, and data
corresponding to 40 identities as the testing set. In total, there are 3,200
training pairs and 985 testing pairs.

Evaluation metrics: For evaluating the effectiveness of our method, we utilize
two different schemes like [3, 16]. We evaluate the facial verification
performance of the reconstructed images by using and comparing our method with
the existing methods in terms of the Rank-1 accuracy, Verification Rate (VR) @
False Accept Rate (FAR)=$1\%$ and<EMAIL_ADDRESS>All Facial verification
experiments are conducted using pretrained ArcFace facial recongittion
system[1]. For evaluating the quality of the reconstructed outputs, we use the
following metrics: Learned Perceptual Image Patch Similarity (LPIPS) [27], Deg
(cosine distance between LightCNN features), Peak Signal to Noise ratio (PSNR)
of the underlying grayscale image and Structural Similarity Index (SSIM).

Training settings: For the diffusion model, we use the same parameters as for
the iamgeNet superresolution model as used in improved diffusion [17]. We
initialize the model with the imageNet pretrained weights. The model is
trained for $T=1000$ timesteps. During inference, we use timestep rescaling as
in [17] and reduce the number of inference steps to $100$. $T_{r}=60$,
$\epsilon$ is set equal to $0.1$ for all experiments

Comparison methods: We evaluate our method by comparing with different
generative model based approaches for image-to-image translation. The
following methods are used for comparison: Pixel2Pixel[9], Self-Attention GAN
(SAGAN)[2], GANVFS[26], HIFaceGAN[25] and AxialGAN[8].

(a) THERMAL

(b) PIX2PIX[9]

(c) SAGAN[2]

(d) GANVFS[2]

(e) HIFACEGAN[25]

(f) AXIALGAN[8]

(g) OURS

(h) VISIBLE

Figure 5: Qualitative results on the ARL-VTF-dataset. TABLE I: Verification
results on VIS-TH dataset

Method | Rank-1 | VR@FAR=1% |<EMAIL_ADDRESS>
---|---|---|---
Pix2Pix[9] | 20.88 | 1.73 | 0.0
SAGAN[2] | 13.04 | 0.0 | 0.0
GANVFS[26] | 30.98 | 3.43 | 0.0
HiFaceGAN[25] | 68.48 | 37.14 | 17.14
AxialGAN[8] | 58.15 | 23.43 | 6.86
T2V-DDPM | 70.5 | 45.5 | 23.5

TABLE II: Image Quality metrics results on VIS-TH dataset

Method | LPIPS($\downarrow$) | Deg.($\uparrow$) | PSNR($\uparrow$) | SSIM($\uparrow$)
---|---|---|---|---
TH | 0.5780 | 0.3635 | 6.3516 | 0.1841
Pix2Pix[9] | 0.4847 | 0.2794 | 13.45 | 0.353
SAGAN[2] | 0.4678 | 0.3367 | 14.14 | 0.4183
GANVFS[26] | 0.4116 | 0.3292 | 13.83 | 0.4186
HiFaceGAN[25] | 0.1950 | 0.6268 | 19.16 | 0.6963
AxialGAN[8] | 0.1756 | 0.6157 | 22.80 | 0.7458
T2V-DDPM | 0.1718 | 0.6441 | 19.419 | 0.7178

###  III-A Results on the VIS-TH dataset

Figure 4 shows the qualitative results on the VIS-TH dataset for four
different poses. As we can see, all the existing methods except HIFaceGAN fail
to reconstruct attributes like hats and sunglasses. Whereas our method, as
well as HIFaceGAN work well even with these occlusions. If we have a close
look at the last row of Figure 4 we can see that when the frontal pose
changes, HiFaceGAN fails to reconstruct the face properly, but our method
works well even with different attributes as well as poses. For evaluating the
methods quantitatively, we use two criteria as mentioned earlier. The
quantitative results can be found in Table II and I. For the rank-1 accuracy,
we can see that we gain an improvement by $2\%$. For<EMAIL_ADDRESS>and
VR@FAR=$1\%$, we gain performance improvements of $8.3\%$ and $6.3\%$
respectively. For the image quality metrics, in terms of the perceptual
similarity metrics (LPIPS and Deg), we gain an improvement of $0.0038$ and
$0.03$, respectively. Our values for PSNR and SSIM are a bit lower than
AxialGAN. However, note that PSNR often does not often represent the amount of
facial details present in the reconstructed images since a blurry image could
have a higher PSNR compared to a much sharper image.

###  III-B Results on the ARL-VTF dataset

Figure 5 shows the qualitative results on the ARL-VTF dataset. This dataset
consists of more identities and much more training and testing images. As we
can see in Fig. 5, only our method can properly reconstruct the salient
underlying facial features, and it can create realistic features that have
coarse features corresponding to the thermal image. Further quantitative
evaluations can be found in Table IV and III.

TABLE III: Verification results on the ARL-VTF dataset

Method | Rank-1 | VR@FAR=1% |<EMAIL_ADDRESS>
---|---|---|---
Pix2Pix[9] | 18.88 | 5.09 | 0.33
SAGAN[2] | 13.46 | 5.42 | 0.33
GANVFS[26] | 21.84 | 12.97 | 2.63
HiFaceGAN[25] | 65.35 | 41.33 | 20.89
AxialGAN[8] | 66.67 | 42.86 | 18.62
T2V-DDPM | 75.37 | 43.51 | 19.87

TABLE IV: Image quality metrics results on the ARL-VTF dataset

Method | LPIPS($\downarrow$) | Deg.($\uparrow$) | PSNR($\uparrow$) | SSIM($\uparrow$)
---|---|---|---|---
TH | 0.5551 | 0.4201 | 5.674 | 0.1095
Pix2Pix[9] | 0.4467 | 0.3489 | 13.12 | 0.3804
SAGAN[2] | 0.4044 | 0.4244 | 14.25 | 0.4490
GANVFS[26] | 0.3924 | 0.3935 | 13.68 | 0.4313
HiFaceGAN[25] | 0.1937 | 0.6563 | 19.62 | 0.6937
AxialGAN[8] | 0.2123 | 0.6699 | 20.04 | 0.7179
T2V-DDPM | 0.2010 | 0.6606 | 19.70 | 0.6775

###  III-C Ablation study: Performance variation on changing $T_{r}$:

To note the performance variation on changing $T_{r}$, we vary $T_{r}$ in
steps of 10 from $T_{r}=40,...,100$ on the VIS-TH dataset, where $T_{r}=100$
denotes starting from pure Gaussian noise. The corresponding results are shown
in Table V. As can be seen, the facial verification performance doesn’t vary
much even for $T_{r}=40$. This clearly validates our claim and observation
that in the initial steps of the diffusion process, only the coarse features
are learnt and if this could be inputted properly to the diffusion model, we
can achieve major boost for the inference time.

TABLE V: Ablation study on the TH-VIS dataset. The number of inference
timesteps are varied here and the performance is noted.

$T_{r}$ | Rank-1 | VR@FAR=1% |<EMAIL_ADDRESS>
---|---|---|---
40 | 66.5 | 34.0 | 18.0
50 | 70.5 | 45.5 | 23.5
60 | 71 | 42.5 | 31.0
70 | 70.5 | 40.0 | 21.0
80 | 68.5 | 39.0 | 23.5
90 | 72.0 | 37.0 | 22.0
100 | 70.0 | 40.5 | 18.5

##  IV Conclusion

We presented a solution for V2T face translation using DDPMs by treating it as
a conditional image generation problem. We find that with the immense power in
modeling probabilistic distributions, DDPMs prove as an ideal solution for
generating samples from the conditional distribution of visible images given
thermal images. We also introduce a novel sampling strategy to reduce the
inference time for DDPMs. Our experiments on multiple datasets show that DDPMs
perform better than GANs for the T2V face translation problem. To the best of
our knowledge, this is the first work utilizing DDPMs to reconstruct visible
facial images from thermal images.

##  V ACKNOWLEDGMENTS

This research was supported by NSF CAREER award 2045489.

## References

  * [1] J. Deng, J. Guo, N. Xue, and S. Zafeiriou.  Arcface: Additive angular margin loss for deep face recognition.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4690–4699, 2019.
  * [2] X. Di, B. S. Riggan, S. Hu, N. J. Short, and V. M. Patel.  Polarimetric thermal to visible face verification via self-attention guided synthesis.  In 2019 International Conference on Biometrics (ICB), pages 1–8. IEEE, 2019.
  * [3] B. Duan, C. Fu, Y. Li, X. Song, and R. He.  Cross-spectral face hallucination via disentangling independent factors.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7930–7938, 2020.
  * [4] C. Fu, X. Wu, Y. Hu, H. Huang, and R. He.  Dual variational generation for low shot heterogeneous face recognition.  Advances in Neural Information Processing Systems, 32, 2019.
  * [5] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio.  Generative adversarial nets.  Advances in neural information processing systems, 27, 2014.
  * [6] M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter.  Gans trained by a two time-scale update rule converge to a local nash equilibrium.  Advances in neural information processing systems, 30, 2017.
  * [7] J. Ho, A. Jain, and P. Abbeel.  Denoising diffusion probabilistic models.  Advances in Neural Information Processing Systems, 33:6840–6851, 2020.
  * [8] R. Immidisetti, S. Hu, and V. M. Patel.  Simultaneous face hallucination and translation for thermal to visible face verification using axial-gan.  In 2021 IEEE International Joint Conference on Biometrics (IJCB), pages 1–8. IEEE, 2021.
  * [9] P. Isola, J.-Y. Zhu, T. Zhou, and A. A. Efros.  Image-to-image translation with conditional adversarial networks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1125–1134, 2017.
  * [10] B. Kawar, M. Elad, S. Ermon, and J. Song.  Denoising diffusion restoration models.  arXiv preprint arXiv:2201.11793, 2022.
  * [11] D. P. Kingma and M. Welling.  Auto-encoding variational bayes.  arXiv preprint arXiv:1312.6114, 2013.
  * [12] S. Z. Li, R. Chu, S. Liao, and L. Zhang.  Illumination invariant face recognition using near-infrared images.  IEEE Transactions on pattern analysis and machine intelligence, 29(4):627–639, 2007.
  * [13] S. Z. Li, Z. Lei, and M. Ao.  The hfb face database for heterogeneous face biometrics research.  In 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, pages 1–8. IEEE, 2009.
  * [14] S. Liu, D. Yi, Z. Lei, and S. Z. Li.  Heterogeneous face image matching using multi-scale features.  In 2012 5th IAPR International Conference on Biometrics (ICB), pages 79–84. IEEE, 2012.
  * [15] A. Lugmayr, M. Danelljan, A. Romero, F. Yu, R. Timofte, and L. Van Gool.  Repaint: Inpainting using denoising diffusion probabilistic models.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11461–11471, 2022.
  * [16] Y. Mei, P. Guo, and V. M. Patel.  Escaping data scarcity for high-resolution heterogeneous face hallucination.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18676–18686, 2022.
  * [17] A. Q. Nichol and P. Dhariwal.  Improved denoising diffusion probabilistic models.  In International Conference on Machine Learning, pages 8162–8171. PMLR, 2021.
  * [18] O. M. Parkhi, A. Vedaldi, and A. Zisserman.  Deep face recognition.  2015\.
  * [19] N. Peri, J. Gleason, C. D. Castillo, T. Bourlai, V. M. Patel, and R. Chellappa.  A synthesis-based approach for thermal-to-visible face verification.  In 2021 16th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2021), pages 01–08. IEEE, 2021.
  * [20] D. Poster, M. Thielke, R. Nguyen, S. Rajaraman, X. Di, C. N. Fondje, V. M. Patel, N. J. Short, B. S. Riggan, N. M. Nasrabadi, et al.  A large-scale, time-synchronized visible and thermal face dataset.  In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 1559–1568, 2021.
  * [21] C. Saharia, W. Chan, H. Chang, C. A. Lee, J. Ho, T. Salimans, D. J. Fleet, and M. Norouzi.  Palette: Image-to-image diffusion models.  arXiv preprint arXiv:2111.05826, 2021.
  * [22] C. Saharia, J. Ho, W. Chan, T. Salimans, D. J. Fleet, and M. Norouzi.  Image super-resolution via iterative refinement.  arXiv preprint arXiv:2104.07636, 2021.
  * [23] J. Sohl-Dickstein, E. Weiss, N. Maheswaranathan, and S. Ganguli.  Deep unsupervised learning using nonequilibrium thermodynamics.  In International Conference on Machine Learning, pages 2256–2265. PMLR, 2015.
  * [24] X. Su, J. Song, C. Meng, and S. Ermon.  Dual diffusion implicit bridges for image-to-image translation.  arXiv preprint arXiv:2203.08382, 2022.
  * [25] L. Yang, S. Wang, S. Ma, W. Gao, C. Liu, P. Wang, and P. Ren.  Hifacegan: Face renovation via collaborative suppression and replenishment.  In Proceedings of the 28th ACM International Conference on Multimedia, pages 1551–1560, 2020.
  * [26] H. Zhang, B. S. Riggan, S. Hu, N. J. Short, and V. M. Patel.  Synthesis of high-quality visible faces from polarimetric thermal faces using generative adversarial networks.  International Journal of Computer Vision, 127(6):845–862, 2019\.
  * [27] R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang.  The unreasonable effectiveness of deep features as a perceptual metric.  In CVPR, 2018.


LEA participants had higher average MD scores than community members and
technical participants. However, the data’s higher standard deviation and
lower median for LEA suggest more variability, with some finding the task to
be mentally challenging. This reflects LEA’s perception of the task as being
important regarding understanding the map to allocate patrol cars effectively.

We considered temporal demand (TD) as the second parameter, assessing time
pressure during the task. Similar patterns to mental demand (MD) in
descriptive statistics are also observed in TD. The correlation between mental
demand (MD) and temporal demand (TD) suggests that the patterns and reasoning
in mental demand scores apply here to the observations of temporal demand.

The third parameter we considered is performance (PF) which gauges workers’
success and satisfaction. In Figure 8, community members have medium PF scores
(10-29), while technical participants and LEAs show somewhat high scores
(30-49) (NASA, ). This suggests greater confidence in results for technical
participants and LEAs compared to community members. We explain in Sections
4.1.1 and 4.1.2 that LEAs and technical participants are more familiar with
such algorithmic systems than community members. Despite technical
participants having data science expertise (average PF score of 45.6), LEAs,
due to prior experience of working with algorithmic crime maps, exhibit
similar confidence levels in their outputs.

The fourth parameter we considered is frustration level (FT), which assesses
whether the job is frustrating or mentally demeaning such that the participant
is left feeling insecure, desperate, offended, or disturbed when doing the
task. The average FT is medium (10-29) for community members and low (0-9) for
technical participants and LEAs. As we highlight in the qualitative results,
community members were more frustrated because they wanted to generate and
compare different maps. Notably, LEAs were less frustrated than technical
participants, likely because they didn’t generate new maps, relying on the
first map and connecting it with their street-level domain knowledge. Through
this analysis, we can fairly say that the mental workload regarding the task
is not very high which implies it is feasible to use such AI applications as
boundary objects that can facilitate deeper conversations about algorithm
design and use with a variety of stakeholders.


# Dosimetric comparison of the BNCT treatment planning performances when using
a nnU-NET to automatically segment Glioblastoma Multiforme

Cristina Pezzi  Francesco Morosato  Barbara Marcaccio  Silva Bortolussi
Ricardo Luis Ramos  Valerio Vercesi  Ian Postuma  Setareh Fatemi

###### Abstract

This work presents a preliminary evaluation of the use of the convolutional
neural network nnU-NET to automatically contour the volume of Glioblastoma
Multiforme in medical images of patients. The goal is to assist the
preparation of the Treatment Planning of patients who undergo Boron Neutron
Capture Therapy (BNCT). BNCT is a binary form of radiotherapy based on the
selective loading of a suitable 10-boron concentration into the tumour and on
subsequent low-energy neutron irradiation. The selectivity of the therapeutic
effect is based on the capacity of the boron drug to target preferentially
cancer cells, thus triggering the neutron capture only in the tumour and
depositing there a lethal dose. Even if the tailoring of the beam to the
tumour volume is less crucial for BNCT than for other radiation therapies, a
proper delimitation of the tumour volume is needed to assess a safe and
effective dosimetry. In clinical application the contour must be manually
decided by the physician, however, a tool to automatically define important
structures such as the Gross Tumour Volume (GTV) and the Organs At Risk (OAR)
would be beneficial to enable medical physicists assessing preliminary
positioning and simulated dosimetry before the approval or possible changes
introduced by the radiotherapist. Moreover, an initial contouring may speed up
the work of the physician. The nnU-NET was trained and tested and its
performance was evaluated through different parameters such as the Dice
Coefficient. To assess a more meaningful evaluation for BNCT, for the first
time, this work analyzed the difference of the clinical dosimetry in 16
patients using the manual and the automatic contoured images. Results show
that the neural network performs well in assisting BNCT dose calculation,
predicting minimum, maximum and mean dose compatible with those obtained in
the manual-obtained contours. The work also highlights some limitations,
especially due to the relative low number of clinical cases used for the
training in this study. Discussion poses the bases for the next work necessary
to fully exploit the potential of such a strategy.

###### keywords:

Deep Learning , BNCT , TPS , isoeffective dose

††journal: Arxiv

[inst1]organization=Università diPavia,addressline=via A. Bassi 6, city=Pavia,
postcode=27100, state=Italy,

[inst2]organization=Istituto Nazionale di Fisica Nucleare, Unit of
Pavia,addressline=via A. Bassi 6, city=Pavia, postcode=27100, state=Italy,
[inst3]organization=Universidad Nacional de San Martin,addressline= 25 de Mayo
y Francia, city=Buenos Aires, postcode=1650, state=Argentina

##  1 Introduction

Boron Neutron Capture Therapy (BNCT) is a binary therapy based on the thermal
neutron capture reaction by Boron-10 (${}^{10}B$). The high capture cross
section of ${}^{10}B$ for thermal neutrons renders the therapy feasible when
the element is loaded in the tumour target with a high ratio respect to the
normal tissue. The products of this capture reaction are an alpha particle and
a ${}^{7}Li$-nucleus, both have high Linear Energy Transfer and a combined
range in soft tissue comparable to the cell dimension. Therefore, all the
energy carried by these two particles is deposited in the target cell ensuring
high efficacy to the therapy [1].
In recent years many accelerator-based BNCT (ab-BNCT) clinical facilities are
operative in various countries of the world, e.g. China, South Korea, Japan
and Finland [2, 3, 4, 5] and others are under development in many other
countries including Italy[6]. The most commonly treated tumours are
Glioblastoma Multiforme (GBM) and Head and Neck Tumours (H&N). Treatment of
H&N tumours is now part of the National Japanese Health System [4].
Currently Glioblastoma Multiforme is the most common malignant primary tumour
of the brain and is one of the most aggressive, the median survival for GBM
patients is around 15 months [7]. BNCT has been proposed as a treatment method
for GBM and numerous studies have been carried out [8, 9].
In this framework this study focuses on GBM cases to apply Artificial
Intelligence Methods to accurately and automatically segment the Gross Tumour
Volume (GTV).
Artificial Intelligence methods and in particular Deep Learning Models (DL)
have been successfully applied to image segmentation, in particular U-NET
models have become the gold standard to automatically segment medical images
[10]. The U-NET shown in Fig.1 is a Fully Convolutional Network (FCN), because
it does not have any dense layer, but only convolutional ones. It is composed
by two parts, the Contracting path or Encoder on the left and the Expansive
path or Decoder on the right. Every path is the repetition of the same
architecture module repeated with different characteristics.

Figure 1: U-NET architecture (example for $32\times 32$ pixels in the lowest
resolution). Each blue box corresponds to a multi-channel image. The number of
channels is denoted on top of the box. The x-y-size is provided at the lower
left edge of the box. White boxes represent copied feature maps, that are
concatenated to the feature maps represented by the blue box. The arrows
denote the different operations. Taken from [11].

The innovation of the U-NET consists in how it retrieves the lost spatial
information of the input image, that is the expanding path. In this study, the
U-NET used is the nnU-NET [12], a DL method for segmentation that autonomously
configures itself on the whole segmentation task pipeline for any new
datasets. This includes decisions about data pre-processing, NN architecture,
training and data post-processing for any task. This model is the state of the
art in the segmentation field and it has shown better performances than other
specialized NNs during the international biomedical segmentation competitions
[12].
The automatically segmented GTV’s using the nnU-NET have been compared to the
segmentation performed by a radiologist. However, to obtain relevant
information for the BNCT community, this study had the aim to evaluate the
performances and the limitations of using automatically segmented images from
the dosimetric point of view.

##  2 Materials and Methods

The images used in this work have been obtained from a public database: The
Cancer Imaging Archive (TCIA) [13]. From all datasets that can be found on
TCIA related to Glioblastoma Multiforme cases, only one was chosen for this
study. The GLIS-RT dataset [14] is the only GBM datsaset provided with CT
images and RT_STRUCT files, which are sets of anatomical structures identified
on radiological images for the planning and delivery of radiotherapy. This
collection consists of 230 cases of GBM and low-grade Glioma patients treated
with surgery and adjuvant radiotherapy at Massachusetts General Hospital. All
images are provided with the radiotherapy targets, the GTV and the clinical
target volume (CTV) manually delineated by the radiation oncologist. Of the
230 cases displayed, 198 are GBM patients and these were used for the training
and the testing of the nnU-NET.
To be suitable for the training and testing of the neural network the DICOM
images of the dataset were converted in the NIFTI format. Moreover, to focus
on the tumour and reduce computational time, a boundary box algorithm was used
to reduce the image dimensions to include only the head of the patient.
The images were then used for the nnU-NET, the network architecture was
configured to suit the characteristics of CT imaging, while the training and
the validation stages were closely monitored adjusting weights and
hyperparameters as necessary. The 198 cases have been automatically sorted by
the network, allocating 152 (76.7%) for the training and validation processes
and the remaining 46 (23.3%) for the testing phase.
The evaluation of the performances of the network involves the following
metrics and indexes to assess the segmentation accuracy and to compare the
given ground-truth volume (G) and the NN predicted volume (S).

  * (i)

Dice Coefficient (DC) which measures the similarity between two volumes. It
ranges between 0 and 1, where 0 is obtained when there is no similarity, and 1
means perfect superposition.

$DC=\frac{2\cdot(G\cap S)}{G+S}$

  * (ii)

Geometrical Miss Index (GMI), which corresponds to the fraction of the ground-
truth volume G that is not predicted. Also GMI ranges between 0 and 1, where a
higher GMI indicates a larger geometric discrepancy between the real and the
predicted volumes.

$GMI=\frac{G-(G\cap S)}{G}$

  * (iii)

Discordance Index (DI), which is the fraction of the predicted volume S that
does not belong to ground-truth volume G. It measures the over-contouring of
the NN respect to the manual segmented image.

$DI=\frac{S-(G\cap S)}{S}$

Of the 46 cases used for the testing of the algorithm, 16 were selected for
this analysis. This subset comprised cases with both low and high Dice Index
and with variable tumour sizes as shown in Tab.1.

Patient | DC | GMI | DI |  | Volume true
---
($cm^{3}$)
| Volume NN
---
($cm^{3}$)
GBM_0071 | 0.296 | 0.002 | 0.774 | 33.681 | 32.832
GBM_0057 | 0.369 | 0.038 | 0.734 | 29.267 | 128.957
GBM_0137 | 0.417 | 0.044 | 0.725 | 31.274 | 113.013
GBM_0202 | 0.427 | 0.105 | 0.574 | 11.446 | 39.766
GBM_0096 | 0.577 | 0.391 | 0.078 | 124.71 | 261.85
GBM_0217 | 0.688 | 0.097 | 0.333 | 253.558 | 152.465
GBM_0001 | 0.767 | 0.078 | 0.306 | 75.297 | 101.889
GBM_0020 | 0.777 | 0.039 | 0.348 | 30.747 | 42.127
GBM_0157 | 0.777 | 0.176 | 0.237 | 107.539 | 158.435
GBM_0192 | 0.793 | 0.231 | 0.150 | 34.354 | 37.096
GBM_0225 | 0.807 | 0.199 | 0.152 | 202.057 | 182.820
GBM_0022 | 0.824 | 0.137 | 0.177 | 61.243 | 57.871
GBM_0129 | 0.843 | 0.073 | 0.214 | 134.324 | 140.839
GBM_0144 | 0.851 | 0.093 | 0.173 | 89.635 | 105.703
GBM_0060 | 0.853 | 0.076 | 0.100 | 104.607 | 114.451
GBM_0128 | 0.912 | 0.042 | 0.842 | 207.673 | 213.312
Table 1: Volumetric dimensions and values of DC, GMI, and DI for each analyzed
patient. Volumes are referenced for both manually segmented GTVs and those
artificially generated by the the neural network.

To evaluate the performances of the neural network segmentation, in addition
to the indexes listed above, a dosimetric comparison was performed to test the
suitability of automatic contouring in BNCT. Dose distributions and values
were calculated in the segmentations made by the radiation oncologist and in
the masks obtained from the neural network.
To simulate the Treatment Planning we used the IT_STARTS Treatment Planning
System (TPS) developed by INFN (publication in preparation). For the
simulation of a BNCT treatment, it is necessary to position the patient model
inside the irradiation room, in the most advantageous orientation with respect
to the neutron beam port.
For the irradiation simulation, the neutron source designed for the
Radiofrequency Quadrupole accelerator (RFQ) designed and built in Italy by
INFN, delivering a 5 MeV, 30 mA proton beam [15]. A facility consisting of the
RFQ, a beryllium target for neutron production and a beam shaping assembly for
moderation and collimation [16], will be built in Caserta (Italy) in the frame
of the PNC-PNRR ANTHEM project. In order to achieve an optimal dose
distribution, several strategies can be applied, and medical considerations
must be taken into account regarding the protection of sensitive structures.
In this work, we minimised the distance of the centre of mass of the GTV
segmented by the radiation oncologist from the beam-port. This corresponds to
maximising the irradiation of the GTV, regardless possible limitations of the
surrounding structures. This choice is justified because the goal of this
study is not the evaluation of BNCT effectiveness and therapeutic potential in
specific cases, but rather to evaluate the relative differences of treatment
plans, once fixed the irradiation position. Figure 2 shows the geometry of the
beam port and the voxelized patient model oriented to comply with the
criterion described above.

Figure 2: Simulation of the treatment planning of one of the cases analyzed in
this work. Green:air, purple, pink, grey:materials of the Beam Shaping
Assemply, as described in [16].

For photon-equivalent dose calculation, IT_STARTS allows the input of
radiobiological data for both the RBE-weighted dose model and the photon
isoeffective dose model [17]. The absorbed dose in the OAR is calculated
applying the RBE-weighted dose model and assuming the RBE (Relative Biological
Effectiveness) and CBE (Compound Biological Effectiveness) parameters given by
Coderre [18], following the protocol applied in clinics, i.e. $CBE=1.35$,
$RBE_{th}=RBE_{f}=3.2$, $RBE_{\gamma}=1$. The dose in the GTV is calculated
using the isoeffective dose model that was proven to better express the dose-
effect relation in BNCT, expressing the mixed-field dose in photon equivalent
units with higher predictive power [19]. The radiobiological parameters to
implement the model were taken from dose-survival studies using the human
Glioblastoma U87 cell line. Monolayer cell cultures were irradiated with
photons, neutrons and neutrons in presence of boron, at the TRIGA Mark II
reactor of the Pavia University (data to be published).
The calculation of the treatment based on the radiation oncologists’
segmentation was then compared to the one obtained using the mask generated by
the nnU-NET. The simulations produced the Dose Volume Histograms in the GTVs
and it was thus possible to extract the minimum absorbed dose (the dose
absorbed by 98% of the GTV volume), the mean dose and the maximum absorbed
dose (the dose absorbed by 2% of the volume).

##  3 Results

The Treatment Planning simulation was conducted on the 16 available cases of
GBM patients. Table 2 lists the irradiation time obtained in the two
segmentations, fixing the patient position by using as a reference the tumour
segmented by the radiation oncologists. Moreover, the value of minimum, mean
and maximum dose to the GTV are reported in both cases. Figure 3 shows a
graphical representation of a patient imaging overlaid with the manually
segmented GTV (green) and the one segmented by the neural network (red). In
this case, the Dice Coefficient was high, proving a good superposition of the
manual and the automatic contouring.

Figure 3: CT imaging of a representative case in which the manually segmented
GTV (green) and the contouring performed by the neural network (red) are
superposed.

Figure 4 shows the box-violin plots of the minimum, mean and maximum dose for
both manual and automatic segmentation. These plots show the mean value and
the median for each parameter taking into account the whole dataset.

Patient |  | tirr
---
(min)
| tirr
---
(min)
| D98%
---
(Gy)
| D98%
---
(Gy)
| D50%
---
(Gy)
| D50%
---
(Gy)
| D2%
---
(Gy)
| D2%
---
(Gy)
| True | NN | True | NN | True | NN | True | NN
| GBM
---
0071
29.60 | 29.58 | 18.3 | 18.6 | 20.7 | 21.0 | 23.0 | 23.4
| GBM
---
0057
32.06 | 34.77 | 21.7 | 20.9 | 26.6 | 26.2 | 29.0 | 30.2
| GBM
---
0137
39.20 | 38.30 | 25.4 | 22.5 | 29.1 | 26.7 | 30.9 | 30.2
| GBM
---
0202
30.92 | 31.67 | 27.1 | 24.0 | 29.3 | 28.7 | 30.5 | 30.7
| GBM
---
0096
32.58 | 35.35 | 11.5 | 9.0 | 19.0 | 18.2 | 26.7 | 28.3
| GBM
---
0217
36.98 | 36.35 | 13.3 | 16.5 | 21.8 | 23.5 | 30.0 | 29.8
| GBM
---
0001
36.42 | 35.65 | 20.2 | 18.8 | 27.4 | 26.0 | 31.1 | 30.7
| GBM
---
0020
36.09 | 36.09 | 25.1 | 23.4 | 29.0 | 28.4 | 31.3 | 31.3
| GBM
---
0157
39.93 | 39.81 | 19.9 | 17.6 | 26.6 | 26.0 | 30.9 | 30.8
| GBM
---
0192
30.71 | 30.79 | 24.0 | 25.5 | 27.6 | 27.9 | 29.4 | 29.4
| GBM
---
0225
33.83 | 33.86 | 15.5 | 15.5 | 22.8 | 24.1 | 30.5 | 30.7
| GBM
---
0022
37.30 | 38.05 | 23.3 | 23.1 | 28.0 | 28.4 | 30.7 | 31.1
| GBM
---
0129
35.58 | 35.39 | 20.5 | 20.5 | 26.4 | 26.4 | 30.8 | 30.7
| GBM
---
0144
33.50 | 33.94 | 22.1 | 21.9 | 26.8 | 26.6 | 29.7 | 29.9
| GBM
---
0060
31.37 | 31.37 | 20.9 | 22.3 | 26.6 | 26.5 | 29.8 | 29.7
| GBM
---
0128
36.96 | 36.96 | 14.4 | 14.6 | 22.7 | 23.0 | 30.6 | 30.6
Table 2: Irradiation time, minimum, mean, and maximum dose values for the 16
analyzed patients. Values were calculated for both manually segmented GTVs
(True) and those artificially generated by the the neural network (NN).

The variation of such parameters was analyzed considering the volume ratio
obtained between the medically segmented GTV and the neural network segmented
volume. To such purpose we also used the Dice coefficient which represents the
degree of similarity between the two segmented volumes.
A first consideration is that in AI-based GTVs the values of minimum and mean
absorbed dose are lower. This tendency to underestimate the minimum and the
mean doses seems a direct consequence of the different volume value and a
different tumor depth extension predicted by the network. In fact, the deeper
the GTV develops, and therefore farther from the beam-port, the lower the
minimum absorbed dose is. This difference is less important regarding maximum
dose values, which all remain slightly above 30 Gy(IsoE). The maximum absorbed
dose, in fact, is located closer to the patient’s surface and is more closely
related to the prescription, the irradiation time and dose limits, set equally
for all patients under examination.

Figure 4: Box-violin plot of the distribution of dose values in the GVTs. From
left to right: values of minimum, mean, and maximum doses deposited in the
GTVs. The mean value is the green triangle and the median is graphically
represented by the orange line.

In Figure 5 the minimum dose ratio, which is the value of the minimum dose
obtained in the automatic segmentation divided by the minimum dose obtained in
the manual segmentation, is shown in respect to the Dice Coefficient. The plot
shows that the cases where the ratio of minimum doses tends to 1 are mainly
situated in the region of high Dice Coefficient and low Geometrical Miss Index
and Discordance Index. The reason is strictly related to the definition of
these parameters. Higher DC as well as lower GMI and DI are attributable to a
good correspondence and overlapping between the two GTVs, it is thus
reasonable to assume that the distributions of doses in the two volumes are
similar. As expected, a better overlap and a proper volume reconstruction lead
to similar minimum doses. Focusing more on the relationship between the ratio
of minimum doses and the Dice Coefficient, two different behaviors are
observed in Figure 5 depending on the Dice.

Figure 5: Plot of the trend of the minimum dose ratio as a function of the
Dice Coefficient for the 16 patients analysed.

##  4 Discussion

These findings support the conclusion that the training of a neural network
nnU-NET assuming the Dice Coefficient as the Loss Function can be efficient in
achieving adequate segmented medical images. However, it is also observed
that, for low Dice values, predicting correct dose distributions becomes more
challenging. Clearly, additional factors come into play, such as, for example,
the difference in depth-distribution of the GTVs. This is typical of BNCT,
where the distribution of the total dose varies significantly as the neutron
beam penetrates the patient’s body. Moreover, some of these complications
arise from the limited number of clinical cases used to train the nnU-NET.
This work was important to point out that it is challenging for the neural
network to generalize the obtained results, to take into account the
complexity of the problem, and to achieve accurate outcomes with this number
of cases analyzed. However, even with these limitations, this study has
demonstrated, for the first time, the possibility to apply Artificial
Intelligence algorithms for the BNCT treatment of patients affected by brain
tumors. The artificial segmentation can indeed be incorporated into a
treatment plan, and a corresponding dose distribution can be efficiently
generated.
Improvements in GTV contouring with ANN could come from defining alternative
loss functions, different from the Dice similarity index. New loss functions
may include more evaluation metrics, leading GTV prediction to ponder over
multiple coefficients, increasing the ANN learning ability. Metrics that may
be added to the loss function are GMI and DI.
This study represents an initial approach in applying AI algorithms in the
field of BNCT, opening the way to further research in this sector, and to the
optimization of the network performance when applied to clinical cases. One of
the first considerations is that it is difficult to ascertain whether the
differences in minimum, mean and maximum dose are clinically significant. More
meaningful figures of merit such as a Tumour Control Probability (TCP) should
be calculated to inter-compare the treatment planning results on the basis of
its clinical effect. To this end, we are developing a model of TCP for GBM and
understanding how to use it to assess the AI segmentation performance.
Moreover, this work assumed the manually contoured structures as ground truth.
However, manual contouring itself is subject to uncertainties and inter-
observer variability. Contouring performed by a different physician would
produce a different segmented GTV, resulting in a different dose distribution.
It is thus interesting to compare the variability between different manual
contours and the discrepancy observed between ground truth and AI to draw more
robust conclusions on the significance of the difference between dose
distributions obtained in the manual and in the automatic contouring.

Artificial Intelligence may be an important tool for the development of a more
efficient and reliable BNCT and its use is being explored for dosimetry and
treatment planning [20] and in other aspects of the research such as boron
concentration evaluation in biological samples [21]. However, its potential is
still to be explored in this field and a huge amount of scientific work is
expected in the future. This contribution represents the starting point to
insert an AI module into the software for dose determination in patients.

##  5 Conclusions

The aim of this work was to assess the suitability of AI algorithms in
contouring medical images for BNCT treatment of Glioblastoma Multiforme (GBM).
The comparison between manually segmented CT images of GBM for dosimetry
purposes and those segmented with the assistance of Artificial Intelligence
provides valuable insights into the effectiveness and reliability of automated
segmentation techniques in radiotherapy planning. Automatic contouring will
not substitute the work by medical doctors, but it can be a valuable aid in
BNCT treatment planning, bringing positive effects both in research and
clinical settings. In research, more patients can be studied and evaluated,
and computational dosimetry can become more refined and robust, towards a
better description of the dose distribution in the tissues. New tumour targets
can be studied as potential candidates for BNCT from the point of view of dose
distribution and therapeutic advantage. In clinics, a more efficient treatment
planning procedure would enlarge the number of patients, assisting the
physician decision process and alowing for a rapid re-evaluation of dose
distributions. In fact, the time-consuming phase of BNCT treatment planning is
the positioning and Monte Carlo transport; once the dose values have been
obtained they can be re-calculated in new masks, if necessary. This, in turn,
would stimulate new research and a general improvement of the quality of the
therapy for enhanced clinical applicability in the future.

## Acknowledgements

This work was funded by the National Scientific Committee 5 of National
Institute of Nuclear Physics (INFN), project AI_MIGHT. This work was also
supported by the National Plan for NRRP Complementary Investments (PNC,
established with the decree-law 6 May 2021, n. 59, converted by law n. 101 of
2021) in the call for the funding of research initiatives for technologies and
innovative trajectories in the health and care sectors (Directorial Decree n.
931 of 06-06-2022) - project n. PNC0000003 - AdvaNced Technologies for Human-
centrEd Medicine (project acronym: ANTHEM). This work reflects only the
authors’ views and opinions, neither the Ministry for University and Research
nor the European Commission can be considered responsible for them.

## References

  * [1] Advances in Boron Neutron Capture Therapy, TECDOC Series, INTERNATIONAL ATOMIC ENERGY AGENCY, Vienna, 2023.
  * [2] Z. Zhang, Y. Chong, Y. Liu, J. Pan, C. Huang, Q. Sun, Z. Liu, X. Zhu, Y. Shao, C. Jin, T. Liu, A review of planned, ongoing clinical studies and recent development of bnct in mainland of china, Cancers 15 (16) (2023).
  * [3] D. Bae, Ys.and Kim, H. e. a. Seo, Advances of linac-based boron neutron capture therapy in korea., AAPPS Bull. 32 (34) (2022).
  * [4] A. Matsumura, T. Asano, K. Hirose, H. Igaki, S. Kawabata, H. Kumada, Initiatives toward clinical boron neutron capture therapy in japan, Cancer Biother. Radiopharm. 38 (3) (2023) 201–207.
  * [5] L. Porra, L. Wendland, T. Seppälä, H. Koivunoro, H. Revitzer, J. Tervonen, L. Kankaanranta, A. Anttonen, M. Tenhunen, H. Joensuu, From nuclear reactor-based to proton accelerator-based therapy: The finnish boron neutron capture therapy experience, Cancer Biotherapy and Radiopharmaceuticals 38 (3) (2023) 184–191.
  * [6] S. Bortolussi, Boron neutron capture therapy: a technological journey to bring neutrons in hospital, il Nuovo Saggiatore 39 (3-4) (2023).
  * [7] S. Grochans, A. M. Cybulska, D. Simińska, J. Korbecki, K. Kojder, D. Chlubek, I. Baranowska-Bosiacka, Epidemiology of glioblastoma multiforme;literature review, Cancers 14 (10) (2022).  doi:10.3390/cancers14102412.
URL https://www.mdpi.com/2072-6694/14/10/2412

  * [8] S. Kawabata, M. Suzuki, K. Hirose, H. Tanaka, T. Kato, H. Goto, Y. Narita, S.-I. Miyatake, Accelerator-based bnct for patients with recurrent glioblastoma: a multicenter phase ii study, Neuro-oncology advances 3 (1) (2021).
  * [9] S.-I. Miyatake, M. Wanibuchi, N. Hu, K. Ono, Boron neutron capture therapy for malignant brain tumors, Journal of neuro-oncology 149 (2020) 1–11.
  * [10] R. Azad, E. K. Aghdam, A. Rauland, Y. Jia, A. H. Avval, A. Bozorgpour, S. Karimijafarbigloo, J. P. Cohen, E. Adeli, D. Merhof, Medical image segmentation review: The success of u-net (2022).  arXiv:2211.14830.
  * [11] O. Ronneberger, P. Fischer, T. Brox, U-net: Convolutional networks for biomedical image segmentation, Medical Image Computing and Computer-Assisted Intervention – MICCAI 2015 160 (3) (2015) 234–241.
  * [12] F. Isensee, P. F. Jaeger, S. A. Kohl, J. Petersen, K. H. Maier-Hein, nnu-net: a self-configuring method for deep learning-based biomedical image segmentation, Nature methods 18 (2) (2021) 203–211.
  * [13] F. W. Prior, K. Clark, P. Commean, J. Freymann, C. Jaffe, J. Kirby, S. Moore, K. Smith, L. Tarbox, B. Vendt, G. Marquez, Tcia: An information resource to enable open science, in: 2013 35th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), 2013, pp. 1282–1285.  doi:10.1109/EMBC.2013.6609742.
  * [14] N. Shusharina, T. Bortfeld, Glioma image segmentation for radiotherapy: Rt targets, barriers to cancer spread, and organs at risk [data set], The Cancer Imaging Archive (2021).  doi:10.7937/TCIA.T905-ZQ20.
  * [15] C. Magni, I. Postuma, M. Ferrarini, N. Protti, S. Fatemi, C. Gong, U. Anselmi-Tamburini, V. Vercesi, G. Battistoni, S. Altieri, S. Bortolussi, Design of a bnct irradiation room based on proton accelerator and beryllium target, Applied Radiation and Isotopes 165 (2020) 109314.  doi:10.1016/j.apradiso.2020.109314.
  * [16] I. Postuma, S. González, M. S. Herrera, L. Provenzano, M. Ferrarini, C. Magni, N. Protti, S. Fatemi, V. Vercesi, G. Battistoni, et al., A novel approach to design and evaluate bnct neutron beams combining physical, radiobiological, and dosimetric figures of merit, Biology 10 (3) (2021) 174.
  * [17] S. J. González, G. A. S. Cruz, The photon-isoeffective dose in boron neutron capture therapy, Radiation research (2012).
  * [18] J. A. Coderre, G. M. Morris, The radiation biology of boron neutron capture therapy, Radiation Research Society 151 (1999) 1–18.
  * [19] S. González, E. C. C. Pozzi, A. M. Hughes, L. Provenzano, H. Koivunoro, D. G. Carando, S. I. Thorp, M. Casal, S. Bortolussi, V. A. Trivillin, et al., Photon iso-effective dose for cancer treatment with mixed field radiation based on dose–response assessment from human and an animal model: clinical application to boron neutron capture therapy for head and neck cancer, Physics in Medicine & Biology 62 (20) (2017) 7938.
  * [20] F. Tian, S. Zhao, C. Geng, C. Guo, R. Wu, X. Tang, Use of a neural network-based prediction method to calculate the therapeutic dose in boron neutron capture therapy of patients with glioblastoma, Medical Physics 50 (5) (2023) 3008–3018.
  * [21] J. Viglietti, M. Espain, R. Díaz, L. Nieto, M. Szewc, G. Bernardi, M. Rodríguez, D. Fregenal, G. Saint Martin, A. Portu, From nuclear track characterization to machine learning based image classification in neutron autoradiography for boron neutron capture therapy, Plos One (2023).