# Comparative Analysis of Deep Convolutional Neural Networks
for Detecting Medical Image Deepfakes

Abdel Rahman Alsabbagh1,2 Omar Al-Kadi1
1University of Jordan, Jordan
2King Abdullah University of Science and Technology, Saudi Arabia
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Generative Adversarial Networks (GANs) have exhibited noteworthy advancements
across various applications, including medical imaging. While numerous state-
of-the-art Deep Convolutional Neural Network (DCNN) architectures are renowned
for their proficient feature extraction, this paper investigates their
efficacy in the context of medical image deepfake detection. The primary
objective is to effectively distinguish real from tampered or manipulated
medical images by employing a comprehensive evaluation of 13 state-of-the-art
DCNNs. Performance is assessed across diverse evaluation metrics, encompassing
considerations of time efficiency and computational resource requirements. Our
findings reveal that ResNet50V2 excels in precision and specificity, whereas
DenseNet169 is distinguished by its accuracy, recall, and F1-score. We
investigate the specific scenarios in which one model would be more favorable
than another. Additionally, MobileNetV3Large offers competitive performance,
emerging as the swiftest among the considered DCNN models while maintaining a
relatively small parameter count. We also assess the latent space separability
quality across the examined DCNNs, showing superiority in both the DenseNet
and EfficientNet model families and entailing a higher understanding of
medical image deepfakes. The experimental analysis in this research
contributes valuable insights to the field of deepfake image detection in the
medical imaging domain111TECHNICAL REPORT, UNIVERSITY OF JORDAN, ARTIFICIAL
INTELLIGENCE DEPARTMENT, TR-01,23..

##  1 Introduction

In the realm of medical imaging, the advent of generative modeling marks a
transformative era. Traditional data augmentation techniques, effective in
many domains, encounter limitations when applied to medical images like
Computed Tomography (CT) scans or Magnetic Resonance (MR) scans. Geometric
transformations, such as random flipping, cropping, rotation, or translation,
prove inadequate, failing to enhance neural network generalization
capabilities beyond the initial training population and often resulting in the
generation of highly correlated samples [1].

Recognizing these challenges, recent strides have been made in leveraging
Generative Adversarial Networks (GANs) as a solution [2, 3]. GANs contribute
by generating authentic-looking medical images, augmenting datasets, and
positively impacting model accuracy. This not only simplifies data synthesis
within the medical imaging domain but also offers a cost-effective
alternative. However, the application of GANs introduces challenges, such as
incorporating intentional manipulation, forgery, and tampering in the medical
images, potentially giving rise to future apprehensions among clinicians
considering the integration of AI in the field of medicine. Authenticating the
generated images is crucial, given the potential consequences of
misinterpretation regardless of the intent behind the application. To this
end, this paper revolves around the utilization of state-of-the-art Deep
Convolutional Neural Network (DCNN) architectures to discern between authentic
and synthetic CT scan images, generated by the CT-GAN [4]. The hypothesis
driving our investigation is rooted in the critical need for a reliable
approach to authenticate medical images, especially in scenarios where the
visual realism of GAN-generated images poses challenges to accurate diagnosis.
Our contributions are:

  1. (a)

Conducting a set of extensive experiments on the most prominent DCNNs known to
be used by the machine learning community for medical image deepfake detection
tasks;

  2. (b)

Analyzing time complexity and model efficiency for a finer understanding of
DCNNs in the medical domain;

  3. (c)

Exploring the embeddings of various DCNNs after training on the medical image
deepfake detection task, and examining the latent space separability quality.

Through this paper, we present a comprehensive overview of the study and
related work in section 2. Subsequently, we detail the dataset preprocessing,
methodology, and results analysis in subsequent sections (3, 4, and 5),
providing valuable insights into the potential of DCNNs in addressing
authenticity challenges in medical imaging. Finally, section 6 further discuss
and summarize our findings and conclusion.

##  2 Overview

The field of deepfake detection has experienced a surge in research activity,
driven by milestones like the introduction of StyleGAN [5]. This Generative
Adversarial Network (GAN) marked a transformative shift by excelling in the
creation of highly realistic images, surpassing its predecessors. Researchers
have responded to the challenge of detecting forgery in such images through a
diverse array of approaches. These range from machine learning-based methods,
such as tree-based methods [6] claiming enhanced interpretability, to the
utilization of vanilla Convolutional Neural Networks (CNNs) [7].

The adoption of well-established or novel Deep Convolutional Neural Network
(DCNN) architectures has been a prevalent theme in recent research efforts.
For instance, [8] utilized VGG16 [9], MobileNetV2 [10], Xception [11], and
InceptionV3 [12] to detect deepfakes on generic data, [13] employed a
combination of ensemble learning techniques with CNNs, and [14] developed a
lightweight deepfake detector using the successive subspace learning
principle. Some researchers have explored hand-crafted features, considering
biological signals within images [15].

Beyond image manipulation, the creation of deepfakes has extended to videos
and audio, necessitating the application of Recurrent Neural Networks (RNNs)
to model forgery in sequential data. [16] addressed audio deepfakes using a
large margin cosine loss function and frequency masking, [17] used recurrent
neural networks for video deepfake detection, and [18] and [19] developed
methods to detect joint audio-visual deepfakes. While Transformers have
revolutionized natural language processing [20] and extended to images [21],
attention-based blocks have been introduced in various deepfake detection
techniques. [22], for instance, utilized the Transformer to detect multimodal
deepfakes on multiple scales, and [23] assembled a hybrid Transformer-CNN
architecture to capture both local and global contexts within an image.

As the field of deepfake detection matures, it finds novel applications in the
medical imaging domain, addressing unique challenges. So far, [24] employed
convolutional reservoir networks, and [25] conducted a comparative study on a
limited set of machine learning and deep learning architectures. Our work is
inspired by this evolving landscape, with a specific focus on the capabilities
of DCNNs. Unlike previous works, our concentration on DCNNs aims to discern
differentiating features in medical images, providing a deeper understanding
of their capabilities in capturing complex features within the medical imaging
domain.

In the following sections, we delve into the methodologies employed, dissect
encountered obstacles and vulnerabilities, and analyze the solutions proposed
to overcome challenges in our exploration of DCNNs for detecting deepfakes in
medical imaging. This comprehensive overview critically examines state-of-the-
art approaches, laying the foundation for subsequent sections in our work.

Figure 1: Sequential scan slices of three patients denoted as (a), (b), and
(c) on each row, where the column Original denotes the label provided, and
columns $[\pm\textit{i}]$ denote the adjacent slices within a range of i of
the Original slice. (a) shows that the tumor starts and ends exactly within
the range $[-5,+5]$, (b) shows that the tumor starts and ends within the range
$[-3,+4]$, and finally (c) shows that the tumor starts and ends out of the
range $[-5,+5]$.

##  3 Data Preprocessing

In this section, we will discuss both the datasets used and how we
preprocessed the data before employing it in our models.

###  3.1 Datasets

In constructing a model capable of discerning between real and fake medical
images, the utilization of diverse and representative datasets is imperative.
For the real medical image dataset, we employed the Lung Image Database
Consortium image collection (LIDC-IDRI), a comprehensive repository of lung CT
scans encompassing data from a thousand distinct patients [26]. This dataset
serves as a benchmark for authentic medical images, providing a diverse and
well-established foundation for our model’s understanding of genuine medical
scans.

Conversely, to introduce the dimension of fake medical images, we leveraged
tampered data generated by the CT-GAN. Notably, the CT-GAN utilized the LIDC-
IDRI dataset during its training phase, making it an intriguing source for
simulated or manipulated medical images. This dual dataset approach,
incorporating both authentic and manipulated medical images, enables our model
to learn discriminative features essential for distinguishing between real and
fake instances. Through this balanced and diverse dataset selection, we aim to
enhance the robustness and generalization capabilities of our deep learning
model in the domain of medical image deepfake detection.

###  3.2 Preprocessing

Both datasets contain files in Digital Imaging and Communications in Medicine
(DICOM) format, and it is a widely used standard for storing and transmitting
medical images. The format includes not only the image itself but also the
metadata of the scan such as the diagnosis of the scan and the patient’s
information. A medical scan folder contains a stack of 2D image slices in
DICOM format that can be reconstructed into a 3D volume. The CT-GAN dataset
has a total of a hundred scans, each scan is a series of $512\times 512$
images, and the series ranges from $100-300$ slices long. In the metadata file
provided by [4], there were only 113 slices labeled as false malignant and
false benign, i.e. tampered / fake. It appears to suffer from severe
imbalance, but the CT-GAN model injects and removes nodules in a 3D volume and
not only one slice of the whole scan. So in principle, the neighboring slices
should also extend the nodule of a labeled slice, as this is how the 3D
pix2pix network works [27].

To examine our preprocessing approach, we visualized the labeled slices and
the adjacent ones within different ranges. Figure 1 shows three separate False
Malignant scans with their tumor region with a range of $[-5,+5]$ of the
original label. We can see that indeed the tumor does not show up only on the
labeled slice, but also on the neighbor slices, which means in return that
there are more than 113 tampered images. At this stage, there were two choices
to get a hold of as much tampered image data as possible: a) visualize all
labeled slices and their adjacent slices and select manually all of the slices
that include a tumor at the region of interest, or b) automatically select the
labeled slices and their adjacent slices within a range of $[-5,+5]$ of the
labeled slice. The initial option is deemed impractical; therefore, the latter
is chosen. However, it is not a strict rule that all slices within the range
of $[-5,+5]$ of the labeled slice are consistently tampered, as illustrated in
Figure 1. We acknowledge the potential for errors associated with this choice.
Nevertheless, in a significant majority of cases, nearly all adjacent slices
within the range $[-5,+5]$ do exhibit tampering. After this preprocessing
step, we ended up having $1,243$ images for the fake class.

The LIDC-IDRI dataset contains scans of a thousand patients with precisely
$244,385$ scan slices for all patients. To maintain a balanced overall dataset
with a $50{:}50$ real:fake ratio, we down-sampled the LIDC-IDRI dataset to
have $1,243$ images, which was achieved by choosing slices at random, making
the overall dataset that will be used in training a total of $2,486$ images.

##  4 Methodology

To solve the medical deepfake detection task, we experimented with a diverse
set of DCNN architectures, these include ConvNeXtTiny [28], DenseNet121,
DenseNet169, DenseNet201 [29], EfficientNetB4 [30], EfficientNetV2S [31],
InceptionV3 [12], MobileNetV3Large [32], RegNetX040, RegNetY040 [33],
ResNet50V2 [34], VGG19 [9], and Xception [11]. The rationale behind selecting
these models particularly is detailed in Appendix A.

We modified the DCNNs by appending a global average pooling layer at the end,
followed by a couple of fully connected layers as additional feature
extractors tailored to the binary classification problem at hand and for the
later latent space visualization. The full architecture is depicted in Figure
2.

Figure 2: Full architecture used for our experiments, which consists of a Deep Convolutional Neural Network (DCNN), a global average pooling layer, a Dense (fully-connected) layer of size 512 followed by a size of 256, and finally a Sigmoid activation function indicating a probability of how much an image is real. Table 1: Results of different Deep Convolutional Neural Networks (1 run) Model | Accuracy | Precision | Recall/Sensitivity | Specificity | F1-Score | AUC
---|---|---|---|---|---|---
ConvNeXtTiny | 0.9544 | 0.9444 | 0.9605 | 0.9490 | 0.9524 | 0.9968
DenseNet121 | 0.9678 | 0.9412 | 0.9944 | 0.9439 | 0.9671 | 0.9988
DenseNet169 | 0.9786 | 0.9669 | 0.9887 | 0.9694 | 0.9777 | 0.9985
DenseNet201 | 0.9571 | 0.9448 | 0.9661 | 0.9490 | 0.9553 | 0.9971
EfficientNetB4 | 0.9678 | 0.9661 | 0.9661 | 0.9694 | 0.9661 | 0.9980
EfficientNetV2S | 0.9678 | 0.9609 | 0.9718 | 0.9643 | 0.9663 | 0.9980
InceptionV3 | 0.9517 | 0.9344 | 0.9661 | 0.9388 | 0.9500 | 0.9968
MobileNetV3Large | 0.9598 | 0.9451 | 0.9718 | 0.9490 | 0.9583 | 0.9975
RegNetX040 | 0.9491 | 0.9593 | 0.9322 | 0.9643 | 0.9456 | 0.9949
RegNetY040 | 0.9544 | 0.9348 | 0.9718 | 0.9388 | 0.9529 | 0.9945
ResNet50V2 | 0.9732 | 0.9883 | 0.9548 | 0.9898 | 0.9713 | 0.9982
VGG19 | 0.9517 | 0.9492 | 0.9492 | 0.9541 | 0.9492 | 0.9964
Xception | 0.9651 | 0.9659 | 0.9605 | 0.9694 | 0.9632 | 0.9977
Table 2: Results of top-performing Deep Convolutional Neural Networks (3 runs) Model | Accuracy | Precision | Recall/Sensitivity | Specificity | F1-Score | AUC
---|---|---|---|---|---|---
DenseNet121 |  0.9705 $\pm$0.0047 |  0.9547 $\pm$0.0153 |  0.9849 $\pm$0.0214 |  0.9592 $\pm$0.0216 |  0.9694 $\pm$0.0050 | 0.9984 $\pm$0.0004
DenseNet169 | 0.9732 $\pm$0.0047 |  0.9500 $\pm$0.0147 | 0.9962 $\pm$0.0007 |  0.9567 $\pm$0.0180 | 0.9738 $\pm$0.0055 |  0.9982 $\pm$0.0003
EfficientNetB4 |  0.9642 $\pm$0.0040 |  0.9507 $\pm$0.0137 |  0.9755 $\pm$0.0086 |  0.9541 $\pm$0.0135 |  0.9623 $\pm$0.0040 |  0.9940 $\pm$0.0062
EfficientNetV2S |  0.9615 $\pm$0.0086 |  0.9502 $\pm$0.0109 |  0.9699 $\pm$0.0086 |  0.9541 $\pm$0.0102 |  0.9599 $\pm$0.0090 |  0.9976 $\pm$0.0005
ResNet50V2 |  0.9723 $\pm$0.0016 | 0.9746 $\pm$0.0288 |  0.9680 $\pm$0.0279 | 0.9762 $\pm$0.0281 |  0.9710 $\pm$0.0001 |  0.9981 $\pm$0.0001
Xception |  0.9418 $\pm$0.0295 |  0.9246 $\pm$0.0538 |  0.9579 $\pm$0.0029 |  0.9274 $\pm$0.0560 |  0.9404 $\pm$0.0284 |  0.9936 $\pm$0.0066

All experiments were conducted using Google Colab’s NVIDIA T4 GPU. We loaded
the DCNNs directly from the TensorFlow [35] and initiated them with pre-
trained weights from the ImageNet dataset [36], keeping the base models frozen
while allowing the remaining architecture to be trainable. The input image
size was fixed at $512\times 512$ pixels, and we uniformly trained all models
for 80 epochs, saving models with the lowest validation loss. A batch size of
8 was used, and we employed Binary Cross-Entropy (BCE) as the loss function as
follows:

$\text{BCE}=-\frac{1}{N}\sum_{i=1}^{N}\left[y_{i}\log(\hat{y}_{i})+(1-y_{i})\log(1-\hat{y}_{i})\right]$
(1)

where $N$ is the number of samples in the dataset, $y_{i}$ is the true binary
label for the $i$-th sample, and $\hat{y}_{i}$ is the predicted probability
that the $i$-th sample belongs to the real class.
We used Adam [37] as an optimizer with a learning rate of $10^{-4}$,
$\beta_{1}$ set to 0.9, and $\beta_{2}$ set to 0.999. Following the initial
training, we unfroze the base models and fine-tuned them for an additional 20
epochs with a learning rate of $10^{-5}$. Once again, we saved the models with
the least validation loss, following the same protocol as in the initial
training phase. To evaluate the performance of our model, we used several
evaluation metrics, including accuracy, precision, recall, specificity,
F1-score, and area under the curve (AUC). Extensive details about the metrics
are in Appendix B.

##  5 Results & Discussion

The experiments involved the execution of a binary classification task
designed to discriminate between medical images categorized as real and fake
across a collection of 13 distinct models. In the initial phase, we conducted
a single round of evaluation, wherein we computed the evaluation metrics
(defined in Appendix B), after which we calculated the harmonic mean of
prominent performance metrics, including accuracy, F1-Score, and specificity,
for each of the models. Subsequently, we employed this assessment to identify
the top-performing six models, subjecting them to two additional evaluation
iterations. This way we not only expedited the process of model selection but
also facilitated a more comprehensive investigation of their stability
characteristics.

#### ResNet50V2 Exhibits Distinctive Performance

With consistent robustness in comparison to other models. This distinction is
evident in the evaluation metrics detailed in Appendix B, summarized in both
Table 1 and Table 2. ResNet50V2 particularly excels, achieving the highest
precision and recall, both around 99%.

Figures 5 and 6 visually highlight ResNet50V2’s unique characteristics, as its
outcomes exhibit a notable lack of substantial correlation with those of other
models. This is measured using Pearson’s correlation coefficient, defined as:

$\rho=\frac{\sum(x_{i}-\bar{x})(y_{i}-\bar{y})}{\sqrt{\sum(x_{i}-\bar{x})^{2}(y_{i}-\bar{y})^{2}}}$
(2)

where $x$ and $y$ represent the results of each model in vector form,
constructed as
$[\text{Accuracy}_{m},\text{Precision}_{m},\text{Recall}_{m},\text{Specificity}_{m},\text{F1-Score}_{m},$
$\text{AUC}_{m}]^{T}$.

This distinct performance of ResNet50V2 is attributed to the intrinsic
characteristics of the residual blocks within the ResNet family. These blocks
effectively preserve features from the input image and consistently integrate
them across subsequent layers, contributing to the model’s robustness. It is
noteworthy to observe that all DCNNs demonstrated commendable proficiency in
capturing differentiating features, as illustrated in Figure 7.

Table 3: Number of parameters and mean inference step time of different Deep Convolutional Neural Networks (10 runs)  Model | Parameters (M) | Inference step (ms)
---|---|---
ConvNeXtTiny | 28.3 |  582.1 $\pm 18.6$
DenseNet121 | 7.7 |  334.9 $\pm 72.4$
DenseNet169 | 13.6 |  396.9 $\pm 67.2$
DenseNet201 | 19.4 |  409.2 $\pm 51.8$
EfficientNetB4 | 18.7 |  442.9 $\pm 52.7$
EfficientNetV2S | 21.1 |  382.6 $\pm 65.7$
InceptionV3 | 23.0 |  380.7 $\pm 86.6$
MobileNetV3Large | 3.6 | 325.4 $\pm$44.7
RegNetX040 | 21.7 |  457.2 $\pm 56.1$
RegNetY040 | 20.3 |  446.3 $\pm 86.3$
ResNet50V2 | 24.7 |  368.3 $\pm 87.2$
VGG19 | 20.4 |  490.2 $\pm 45.1$
Xception | 22.0 |  395.3 $\pm 76.9$
Figure 3: Time recorded per inference step after ten inferences on various
Deep Convolutional Neural Networks Figure 4: Latent space separability quality
between real and fake examples, showing penultimate layer embedding of each
Deep Convolutional Neural Network (DCNN). (minor areas of collision circled in
green)

#### DenseNet169 Consistently Excels Across Metrics

Achieving the highest levels of accuracy, recall, and F1-score, as observed
over three iterative runs. This consistent excellence underscores its
stability and efficacy in addressing the non-uniform distribution of tumor
types and nodules within CT scan images. Comparatively, DenseNet121
demonstrates a performance comparable to DenseNet169, achieving similar
results with nearly half the number of parameters and quicker inference times.
These findings are summarized in Table 3 and visually presented in Figure 3.
Notably, this suggests that DenseNet121 offers a favorable trade-off between
model complexity and performance. However, DenseNet201 exhibits a degradation
in performance compared to its counterparts in the DenseNet family. This
departure from the expected trend suggests that the conclusions drawn in [38]
may not readily extend to our specific case. We attribute this discrepancy to
potential overfitting, likely exacerbated by the increased complexity of
DenseNet201’s architecture.

Figure 5: Pearson’s correlation matrix showing interrelationships among Deep
Convolutional Neural Networks (DCNNs) results. Figure 6: Pearson’s correlation
matrix for the top-performing Deep Convolutional Neural Networks results.

#### Comparable Efficiency of MobileNetV3Large

Exhibiting remarkable efficiency, and boasting minimal parameters and
inference time while delivering similar results to other DCNNs. With
approximately 6.81 times and 3.78 times fewer parameters than ResNet50V2 and
DenseNet169, respectively, MobileNetV3Large maintains a maximum difference of
only around 4% in comparison to the highest values across all evaluation
metrics. This establishes MobileNetV3Large as a promising option, particularly
when facing hardware limitations. Conversely, VGG19 demonstrates fair
stability in its inference time, aligning with the consistency observed in
ConvNeXtTiny and MobileNetV3Large, as illustrated in Figure 3. On the
contrary, ResNet50V2 exhibits notable inconsistency in inference time.

#### Distinguished Latent Space Separability of DenseNet169

Showing high quality separability between real and fake data points as seen in
Figure 4 with visualization of the second-to-last layer embeddings. Similarly,
this is seen in other members of the DenseNet family, particularly in
DenseNet121. Interestingly, EfficientNetB4, EfficientNetV2S, and ResNet50V2
have a likewise outstanding ability to separate the different classes. The
DCNNs mentioned above all share a common property in their latent space which
is the dense concentration across the embeddings of the same class. This trait
is not seen in the other DCNNs, which tells us that these models are less
appropriate for the task of medical image deepfake detection. Moreover, there
is a phenomenon seen across all DCNNs, which is what we like to call the areas
of collision, in which examples from opposing classes overlap at different
regions in the latent space.

Figure 7: Confusion matrices of Deep Convolutional Neural Networks.

Our exploration of various DCNN architectures has unveiled nuanced differences
in their performance, prompting the question: In what context is a particular
DCNN most suitable?

In medical image deepfake detection, the relationship between model
architecture, complexity, and performance is intricate. For maintaining robust
performance resilient to noise and out-of-distribution changes, ResNet50V2
emerges as a top choice. Its near-perfect precision and specificity are
attributed to its residual connections, which mitigate the vanishing gradients
problem, facilitate deep network training, encourage feature reuse, and
enhance model expressiveness. This outstanding property is seen in Figures 5
and 6, where the Pearson correlation’s activations are merely shown in
ResNet50V2. However, considering ResNet50V2’s shortcomings in other evaluation
measures, the DenseNet family, especially DenseNet169, proves valuable. Its
superior accuracy, recall, and F1-score indicate a comprehensive and sensitive
approach to identifying relevant cases and addressing imbalanced class
distributions effectively. The unique dense connectivity in DenseNet,
promoting feature reuse and efficient gradient flow, is a key factor behind
its improved performance.

Despite the distinctive strengths of ResNet50V2 and DenseNet169, they face
challenges in meeting time and hardware constraints. MobileNetV3Large
addresses these limitations by offering comparable performance while executing
the fastest. Designed for optimal performance on mobile phone CPUs,
MobileNetV3Large utilizes a blend of hardware-aware network architecture
search (NAS) and the NetAdapt [39] algorithm, contributing to its efficient
execution. One thing is that important to mention is that the time complexity
is not merely dependent on the architecture and number of parameters,
implementation efficiency also plays a pivotal role, which can be the reason
why a DCNN like ConvNeXtTiny has a substantially higher inference time while
having a little increase of parameters over ResNet50V2.

These trade-offs between efficiency, stability, and performance offer valuable
insights for selecting models tailored to specific medical image deepfake
detection tasks under varying restrictions. One intriguing prospect is the
development of hybrid architectures that combine DCNNs capturing different
semantics, aiming to maintain collective efficacy when prioritizing reliable
performance, albeit with a less complex model and a subtle loss of
performance.

Moreover, effectively learning class-differentiating features is a complicated
task that all DCNNs failed to fully succeed in as previously mentioned. We can
infer from the areas of collision (see Figure 4) that DCNNs can either have
intrinsic difficulty in learning dissimilar features that map each data point
to its label or that the deepfakes are sufficiently real-looking, even to the
machine. Another reason can be due to the data mislabeling probability as
mentioned in Section 3 or because of the embedding size that can be inadequate
to fully capture all the features within a given image.

##  6 Conclusion

In this paper, we addressed the pivotal challenge of authenticating medical
images amidst the complexities introduced by deepfakes. Employing a diverse
range of DCNN architectures, we utilized a dataset that amalgamates CT images
generated as fake through CT-GAN with real CT images sourced from the LIDC-
IDRI dataset. Our training of 13 distinct DCNN models, guided by the
minimization of binary cross-entropy loss, allowed us to uncover distinctive
strengths inherent to these models.

Remarkably, ResNet50V2 emerged as the preeminent model, distinguished by
exceptional precision and specificity. Its consistent and reliable performance
profile signifies its efficacy in the context of medical image authentication,
addressing the challenges posed by GAN-generated images. In contrast,
DenseNet169 showcased notable excellence in accuracy, recall, and F1-score,
highlighting its stability and effectiveness in navigating the nuanced
distribution within CT scan images. Furthermore, our findings unveiled that
MobileNetV3Large, while offering comparable performance to leading models,
stands out due to its minimal parameters and swift inference time. For future
work, we plan to further our experiments with a larger set of DCNNs and
explore the effects of hyperparameter choice. We are also interested in
investigating the causality behind the DCNN choice to select a certain label
over another and have a strong explainability framework.

These insights collectively lay the groundwork for future research endeavors
within the field of DCNN applications in medical imagery, particularly in the
authentication of CT scans. Additionally, they provide a valuable resource for
selecting an appropriate DCNN model tailored to the unique attributes of a
given medical imaging scenario, aligning with the critical need for reliable
approaches in the face of GAN-generated challenges.

## References

  * [1] Fabio Garcea, Alessio Serra, Fabrizio Lamberti, and Lia Morra.  Data augmentation for medical imaging: A systematic literature review.  Computers in Biology and Medicine, 152:106391, 1 2023.
  * [2] Richard Osuala, Kaisar Kushibar, Lidia Garrucho, Akis Linardos, Zuzanna Szafranowska, Stefan Klein, Ben Glocker, Oliver Diaz, and Karim Lekadir.  Data synthesis and adversarial networks: A review and meta-analysis in cancer imaging.  Medical Image Analysis, 84:102704, 2023.
  * [3] Alaa Abu-Srhan, Israa Almallahi, Mohammad AM Abushariah, Waleed Mahafza, and Omar S Al-Kadi.  Paired-unpaired unsupervised attention guided gan with transfer learning for bidirectional brain mr-ct synthesis.  Computers in Biology and Medicine, 136:104763, 2021.
  * [4] Yisroel Mirsky, Tom Mahler, Ilan Shelef, and Yuval Elovici.  Ct-gan: Malicious tampering of 3d medical imagery using deep learning.  In Proceedings of the 28th USENIX Conference on Security Symposium, SEC’19, page 461–478, USA, 2019. USENIX Association.
  * [5] Tero Karras, Samuli Laine, and Timo Aila.  A style-based generator architecture for generative adversarial networks.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4401–4410, 2019.
  * [6] Md. Shohel Rana, Beddhu Murali, and Andrew H. Sung.  Deepfake detection using machine learning algorithms.  In 2021 10th International Congress on Advanced Applied Informatics (IIAI-AAI), pages 458–463, 2021.
  * [7] Anuj Badale, Lionel Castelino, Chaitanya Darekar, and Joanne Gomes.  Deep fake detection using neural networks.  In 15th IEEE international conference on advanced video and signal based surveillance (AVSS), 2018.
  * [8] Nishika Khatri, Varun Borar, and Rakesh Garg.  A comparative study: Deepfake detection using deep-learning.  In 2023 13th International Conference on Cloud Computing, Data Science & Engineering (Confluence), pages 1–5, 2023.
  * [9] Karen Simonyan and Andrew Zisserman.  Very deep convolutional networks for large-scale image recognition.  arXiv preprint arXiv:1409.1556, 2014.
  * [10] Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen.  Mobilenetv2: Inverted residuals and linear bottlenecks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4510–4520, 2018.
  * [11] François Chollet.  Xception: Deep learning with depthwise separable convolutions.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1251–1258, 2017.
  * [12] Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna.  Rethinking the inception architecture for computer vision.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2818–2826, 2016.
  * [13] Md. Shohel Rana and Andrew H. Sung.  Deepfakestack: A deep ensemble-based learning technique for deepfake detection.  In 2020 7th IEEE International Conference on Cyber Security and Cloud Computing (CSCloud)/2020 6th IEEE International Conference on Edge Computing and Scalable Cloud (EdgeCom), pages 70–75, 2020.
  * [14] Hong-Shuo Chen, Mozhdeh Rouhsedaghat, Hamza Ghani, Shuowen Hu, Suya You, and C.-C. Jay Kuo.  Defakehop: A light-weight high-performance deepfake detector.  In 2021 IEEE International Conference on Multimedia and Expo (ICME), pages 1–6, 2021.
  * [15] Umur Aybars Ciftci, Ilke Demir, and Lijun Yin.  Fakecatcher: Detection of synthetic portrait videos using biological signals.  IEEE transactions on pattern analysis and machine intelligence, 2020.
  * [16] Tianxiang Chen, Avrosh Kumar, Parav Nagarsheth, Ganesh Sivaraman, and Elie Khoury.  Generalization of audio deepfake detection.  In Odyssey, pages 132–137, 2020.
  * [17] David Güera and Edward J Delp.  Deepfake video detection using recurrent neural networks.  In 2018 15th IEEE international conference on advanced video and signal based surveillance (AVSS), pages 1–6. IEEE, 2018.
  * [18] Yipin Zhou and Ser-Nam Lim.  Joint audio-visual deepfake detection.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 14800–14809, 2021.
  * [19] Trisha Mittal, Uttaran Bhattacharya, Rohan Chandra, Aniket Bera, and Dinesh Manocha.  Emotions don’t lie: An audio-visual deepfake detection method using affective cues.  In Proceedings of the 28th ACM international conference on multimedia, pages 2823–2832, 2020.
  * [20] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
  * [21] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  arXiv preprint arXiv:2010.11929, 2020.
  * [22] Junke Wang, Zuxuan Wu, Wenhao Ouyang, Xintong Han, Jingjing Chen, Yu-Gang Jiang, and Ser-Nam Li.  M2tr: Multi-modal multi-scale transformers for deepfake detection.  In Proceedings of the 2022 International Conference on Multimedia Retrieval, ICMR ’22, page 615–623, New York, NY, USA, 2022. Association for Computing Machinery.
  * [23] Shreyan Ganguly, Aditya Ganguly, Sk Mohiuddin, Samir Malakar, and Ram Sarkar.  Vixnet: Vision transformer with xception network for deepfakes based video and image forgery detection.  Expert Systems with Applications, 210:118423, 12 2022.
  * [24] Rajat Budhiraja, Manish Kumar, M.K. Das, Anil Singh Bafila, and Sanjeev Singh.  Medifaked: Medical deepfake detection using convolutional reservoir networks.  In 2022 IEEE Global Conference on Computing, Power and Communication Technologies (GlobConPT), pages 1–6, 2022.
  * [25] Siddharth Solaiyappan and Yuxin Wen.  Machine learning based medical image deepfake detection: A comparative study.  Machine Learning with Applications, 8:100298, 6 2022.
  * [26] Armato iii, s. g., mclennan, g., bidaut, l., mcnitt-gray, m. f., meyer, c. r., reeves, a. p., zhao, b., aberle, d. r., henschke, c. i., hoffman, e. a., kazerooni, e. a., macmahon, h., van beek, e. j. r., yankelevitz, d., biancardi, a. m., bland, p. h., brown, m. s., engelmann, r. m., laderach, g. e., max, d., pais, r. c. , qing, d. p. y. , roberts, r. y., smith, a. r., starkey, a., batra, p., caligiuri, p., farooqi, a., gladish, g. w., jude, c. m., munden, r. f., petkovska, i., quint, l. e., schwartz, l. h., sundaram, b., dodd, l. e., fenimore, c., gur, d., petrick, n., freymann, j., kirby, j., hughes, b., casteele, a. v., gupte, s., sallam, m., heath, m. d., kuhn, m. h., dharaiya, e., burns, r., fryd, d. s., salganicoff, m., anand, v., shreter, u., vastagh, s., croft, b. y., clarke, l. p. (2015). Data From LIDC-IDRI [data set]. the cancer imaging archive.
  * [27] Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A. Efros.  Image-to-image translation with conditional adversarial networks.  In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 5967–5976, 2017.
  * [28] Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie.  A convnet for the 2020s.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11976–11986, 2022.
  * [29] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger.  Densely connected convolutional networks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4700–4708, 2017.
  * [30] Mingxing Tan and Quoc Le.  Efficientnet: Rethinking model scaling for convolutional neural networks.  In International conference on machine learning, pages 6105–6114. PMLR, 2019.
  * [31] Mingxing Tan and Quoc Le.  Efficientnetv2: Smaller models and faster training.  In International conference on machine learning, pages 10096–10106. PMLR, 2021.
  * [32] Andrew Howard, Mark Sandler, Grace Chu, Liang-Chieh Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, et al.  Searching for mobilenetv3.  In Proceedings of the IEEE/CVF international conference on computer vision, pages 1314–1324, 2019.
  * [33] Ilija Radosavovic, Raj Prateek Kosaraju, Ross Girshick, Kaiming He, and Piotr Dollár.  Designing network design spaces.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10428–10436, 2020.
  * [34] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Identity mappings in deep residual networks.  In Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part IV 14, pages 630–645. Springer, 2016.
  * [35] Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dandelion Mané, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng.  TensorFlow: Large-scale machine learning on heterogeneous systems, 2015.  Software available from tensorflow.org.
  * [36] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009.
  * [37] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980, 2014.
  * [38] Pranav Rajpurkar, Jeremy Irvin, Kaylie Zhu, Brandon Yang, Hershel Mehta, Tony Duan, Daisy Ding, Aarti Bagul, Curtis Langlotz, Katie Shpanskaya, et al.  Chexnet: Radiologist-level pneumonia detection on chest x-rays with deep learning.  arXiv preprint arXiv:1711.05225, 2017.
  * [39] Tien-Ju Yang, Andrew Howard, Bo Chen, Xiao Zhang, Alec Go, Mark Sandler, Vivienne Sze, and Hartwig Adam.  Netadapt: Platform-aware neural network adaptation for mobile applications.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 285–300, 2018.

## Appendix

##  Appendix A Model Selection

Based on our analysis, as highlighted in the CheXNet model presented by [38],
the performance of architectures that have demonstrated effectiveness on
ImageNet [36] does not necessarily translate to superior results with CheXNet.
CheXNet, specifically trained on chest X-ray images, challenges the assumption
that models excelling on general datasets, such as ImageNet, will perform
equally well on domain-specific data like medical images. Therefore, we
derived that existing DCNNs that perform greatly with generic data might not
also perform as well in our case. As a result, we have decided to explore
alternative well-established DCNN architectures to address this challenge.

To this end, we determined that the models we chose to experiment with are
most reasonable to satisfy two conditions: a) experiment with various model
families by choosing the model with the largest size from each family, but
that is restricted by $2$. The model we choose should not exceed $30$M
parameters due to computational constraints. Given that the CheXNet paper
claimed intra-family generalization, we decided to experiment with all of the
models in the DenseNet family to test the claim.

To choose which models we will perform 3 runs to validate their stability, we
will need to calculate the harmonic mean of accuracy, F1-Score, and
specificity as mentioned before. The formula for the harmonic mean H given N
numbers is:

$\text{H}=\frac{n}{\sum_{i=1}^{n}\frac{1}{x_{i}}}$ (3)

For the case of our experiment, the formula is:

$\text{H}=\frac{3}{\frac{1}{\text{Accuracy}}+\frac{1}{\text{F1-Score}}+\frac{1}{\text{Specificity}}}$
(4)

Table A1 shows the result of H for all models. Given the results of H and
after reducing the number of models, we will proceed with experimenting with
two more runs with the winner models, i.e. DenseNet121, DenseNet169,
EfficientNetB4, EfficientNetV2S, ResNet50V2, and Xception, then calculate the
mean and standard deviations of the result, consequently analyzing their
results.

Table A1: Results of the harmonic mean of the accuracy, F1-Score, and specificity on the evaluation metrics results of the proposed models after one run. The best 6 models are in bold. Model | H
---|---
ConvNeXtTiny | 0.9519
DenseNet121 | 0.9595
DenseNet169 | 0.9752
DenseNet201 | 0.9538
EfficientNetB4 | 0.9678
EfficientNetV2S | 0.9661
InceptionV3 | 0.9468
MobileNetV3Large | 0.9557
RegNetX040 | 0.9529
RegNetY040 | 0.9486
ResNet50V2 | 0.9780
VGG19 | 0.9517
Xception | 0.9659

##  Appendix B Evaluation Metrics

To calculate the evaluation metrics, we describe the four possible outcomes of
our models, namely, True Positives (TPs), True Negatives (TNs), False
Positives (FPs), and False Negatives (FNs).
We get a TP when the model correctly predicts the positive class, a TN when
the model correctly predicts the negative class, an FP when the model
incorrectly predicts the positive class, and finally, an FN when the model
incorrectly predicts the negative class.

Accuracy. Accuracy measures the proportion of correct predictions made by the
model out of all the predictions made. In other words, accuracy tells us how
often the model’s predictions were correct. The formula for accuracy is:

$\text{Accuracy}=\frac{TP+TN}{TP+TN+FP+FN}$ (5)

Precision. Precision measures the proportion of true positive predictions out
of all the positive predictions made by the model. In other words, precision
tells us how many of the positive predictions made by the model were correct.
The formula for precision is:

$\text{Precision}=\frac{TP}{TP+FP}$ (6)

Recall. Recall measures the proportion of true positive predictions out of all
the actual positive instances in the dataset. In other words, recall tells us
how many of the positive instances in the dataset were correctly identified by
the model. Recall is also called sensitivity or true positive rate (TPR). The
formula for recall is:

$\text{Recall}=\frac{TP}{TP+FN}$ (7)

Specificity. Specificity measures the proportion of true negative predictions
out of all the actual negative instances in the dataset. In other words,
specificity tells us how many of the negative instances in the dataset were
correctly identified by the model as negative. Specificity is also known as
selectivity or true negative rate (TNR). The formula for specificity is:

$\text{Specificity}=\frac{TN}{TN+FP}$ (8)

F1-score. F1-score is the harmonic mean of precision and recall, which
provides a single measure of a model’s accuracy by balancing the trade-off
between precision and recall. The formula for F1-score is:

$\text{F1-Score}=\frac{2\times\text{Precision}\times\text{Recall}}{\text{Precision}+\text{Recall}}$
(9)

Fall-out. Fall-out measures the proportion of actual negative instances that
were incorrectly classified as positive by the model. Fall-out is also known
as the false positive rate. The formula for fall-out is:

$\text{Fall-out}=\frac{FP}{FP+TN}$ (10)

We will not directly mention the fall-out in the results table, but we will
rather use it to calculate the area under the curve (AUC).
AUC. AUC stands for the Area Under the Curve, specifically the ROC curve. The
ROC curve (Receiver Operating Characteristic curve) is a plot of the true
positive rate (sensitivity) against the false positive rate (fall-out) for
various classification thresholds. AUC represents the area under the ROC
curve, which ranges from 0 to 1, with higher values indicating better
performance of the model. The formula for AUC is:

$\text{AUC}=\int_{0}^{1}\text{ROC}(x)\,dx$ (11)


# Learning to compose 6-DoF omnidirectional videos using multi-sphere images

###### Abstract

Omnidirectional video is an essential component of Virtual Reality. Although
various methods have been proposed to generate content that can be viewed with
six degrees of freedom (6-DoF), existing systems usually involve complex depth
estimation, image in-painting or stitching pre-processing. In this paper, we
propose a system that uses a 3D ConvNet to generate a multi-sphere images
(MSI) representation that can be experienced in 6-DoF VR. The system utilizes
conventional omnidirectional VR camera footage directly without the need for a
depth map or segmentation mask, thereby significantly simplifying the overall
complexity of the 6-DoF omnidirectional video composition. By using a newly
designed weighted sphere sweep volume (WSSV) fusing technique, our approach is
compatible with most panoramic VR camera setups. A ground truth generation
approach for high-quality artifact-free 6-DoF contents is proposed and can be
used by the research and development community for 6-DoF content generation.

Index Terms—  Omnidirectional video composition, multi-sphere images, 6-DoF VR

##  1 Introduction

As a result of the rapidly increasing popularity of omnidirectional videos on
video-sharing platforms such as YouTube and Veer, professional content
producers are striving to produce better viewing experiences with higher
resolution, new interaction mechanisms, as well as more degrees of viewing
freedom, using playback platforms such as the Google Welcome to Light Field,
where the viewer has the freedom to move along three axes with motion
parallax. A body of research has been dedicated to composing 6-DoF from
footage captured by VR cameras, extending toolsets for the professional
content generators to produce more immersive contents. However, most of the
proposed systems involve complex procedures such as depth estimation and in-
painting, that are both time- and resource- consuming and require tedious
hand-optimizations.

Recently, Attal et al.proposed MatryODShka [1], which used a convolutional
neural network to predict multi-sphere images (MSIs) that can be viewed in
6-DoF. This approach significantly simplified the overall pipeline complexity
while producing promising visual results. However, the system requires
omnidirectional stereo (ODS) inputs that cannot be acquired directly from
widely available VR cameras, as ODS must be produced from raw panoramic VR
footage using stitching with annoying visual artifacts. In addition, as in the
case for any system design, 6-DoF content production also requires a large
volume of high-quality content that can serve as the ground-truth, for both
training purposes and performance evaluation. However, because 6-DoF contents
are difficult to produce, there is no widely available high-quality 6-DoF
content that can be used as a ground-truth dataset for the community.

The contribution of this paper is therefore two-fold: First, we propose a
system that can produce large volumes of high-quality artifact-free 6-DoF data
for training and performance evaluation that serves as the basis for
evaluation of the system proposed in this paper, as well as for future
development by the community. Second, we propose an algorithm for predicting
MSI from VR camera footage directly, using a modified weighted sphere sweep
volume fusing scheme with 3D ConvNet without explicit depth map or
segmentation mask, thereby improving quality while lowering complexity.

Fig. 1: Our proposed 6-DoF omnidirectional video composition framework. Fig.
2: An example of conventional 6-DoF content generation method

##  2 Related Works

Substantial research has been carried out on the reconstruction and
representation of omnidirectional contents [2][3][4][5][6][7]. In this
section, we focus on recent research aimed at 6-DoF reconstruction and view
synthesis.

Six degrees of freedom content generation requires detailed scene depth
information. Dynamic 3D reconstruction and content playback have been
extensively studied in the context of free-viewpoint video, with many
approaches achieving real-time performance [8][9]. Using the conventional
multi-view stereo method, Google Welcome to Light field [10] and Facebook
manifold system [11] both achieved realistic high-quality 6-DoF content
composition, with hardware systems that are substantially more complex than VR
cameras widely available on the market. Lately, studies that used
convolutional neural networking (CNN) show promising results for depth
estimation and view synthesis [12][13][14]. CNN achieves excellent results for
predicting multi-plane images (MPI) and representing the non-Lambertian
reflectance [15][16][17][18][19]. Many recent approaches adopt and extend
these studies to generate omnidirectional contents [1][20][21]. Broxton et al.
[21] designed a half-sphere camera rig with GoPro cameras and used the method
in [16] to generate pieces of MPIs and later converted it into 360 layered
mesh representation for viewing. Lin et al. [20] generated multiple MPIs and
formed them into a multi-depth panorama using the MPI predicting techniques in
[17]. Lai et al. [22] proposed to generate panorama depth map for ODS images
6-DoF synthesization. More recently, MartyODShka [1] archived real time
performance when using the method in [15] to synthesize MSI from ODS content.
In spite of such successes, ODS images relaid by these approaches still
require stitching of original VR camera footage, which by itself is still a
problem that has not been sufficiently solved. Many existing neural network
based approaches were also designed with a fixed number of input images that
is not configurable.

In this paper, we propose an approach without stitching pre-processing or
depth map estimation. In our approach, a weighted sphere sweep volume is fused
directly from camera footage using spherical projection, eliminating artifacts
introduced by stitching. By utilizing 3D ConvNet, our framework is applicable
to different VR camera designs with different numbers of MSI layers. We also
propose a high-quality 6-DoF dataset generation approach using UnrealCV [23]
and Facebook Replica [24] engine, so that our, and future systems for 6-DoF
content composition can be designed and evaluated qualitatively and
quantitatively using our data generation scheme.

##  3 Omnidirectional 6-DoF Data Generation

Commercially available VR camera models provide a range of selection for
omnidirectional content acquisition. A great amount of footage from these VR
cameras is captured by professional photographers and enthusiasts. However, it
is hard to render a ground truth from these footage, as content captured by
such cameras need to be stitched, which by itself is still a problem that has
not been completely solved. As a result, it has been extremely difficult to
find artifact-free high-quality 360 video datasets in this literature, which
is required for the continued development and evaluation of VR and/or 6-DoF
content. The few datasets produced by companies like Google or Facebook are
limited in size, content scenarios, resolutions etc., as they were constrained
by the various factors imposed by the time and equipment used for producing
such datasets.

It is therefore highly desirable to be able to generate any number of test
clips of any use cases and different resolutions and to continue this test
data generation process as new cameras with higher resolutions become
available. In this study, we use two CG rendering frameworks UnrealCV [23] and
Replica [24] to generate high-quality VR datasets for both training and
evaluation. The UnrealCV engine can render realistic images with lighting
changes and reflections on handcrafted models. On the other hand, Replica
engine uses indoor reconstruction models, and produces rendered images with a
similar distribution to real-world textures and dynamic ranges. We use the
combination of two datasets to mimic real-world challenges.

To compose an omnidirectional image using the above mentioned CG engines, we
first generate six ${120}^{\circ}$ pin-hole images with different poses
towards the engine’s x-y-z axes and their inverse directions. The optical
centers of these virtual cameras are located at the same point, to avoid any
parallax between cameras. We then project six pin-hole images to
equirectangular projection (ERP) and blend the overlapping area to avoid
aliasing. To simulate camera footage from a n-sensor VR camera, we render ERP
footage on the poses of each sensor and mask off the external details
according to the lens field of view. In our experiments, we selected 2000
locations to generate 6-DoF contents for network training and evaluation. The
generated datasets were split into training subsets of 1600 locations and
evaluation subsets with 400 locations. The split was implemented based on
virtual camera locations with no overlap across the training and evaluation
subsets. Contents generated from UnrealCV and Replica were mixed together
during training procedure but evaluated separately. We rendered two
resolutions (640$\times$320 and 400$\times$200) on each location for training
with different numbers of MSI layers due to GPU memory constraints. Upon that,
we rendered two fields of view (${190}^{\circ}$ and ${220}^{\circ}$) to
simulate fisheye images from different VR camera modules.

##  4 A Simplified System for Producing 6-DoF Content from Panoramic VR
Footage

Our process of turning panoramic VR camera footage into 6-DoF playable multi-
sphere images involves three steps as shown in Fig. 1. We first project the
fisheye images onto concentric spheres using the weighted sphere sweep method
and generate weighted sphere sweep volume (WSSV) as described in Sec. 4.2.
This 4D volume is input into the 3D ConvNet detailed in Sec. 4.3 to predict
the $\alpha$ channel. We combine the predicted $\alpha$ channel with WSSV to
form the MSI representation. Later inside the render engine, the MSI is
calculated following Equ. 2 to infer per-eye views in VR.

###  4.1 Multi-Sphere Images

Fig. 3: Illustration of an MSI representation. Blue lines show the opacity at
different area of the concentric spheres.

Inspired by multi-plane image (MPI) and its performance in view synthesis
applications, multi-sphere images (MSIs) is generated for omnidirectional
content representation. Following the design of MPI, the MSI representation
used in our work consists of $N$ images to represent $N$ concentric spheres.
These concentric spheres are warped into planes using equirectangular
projection (ERP) for efficient processing. Each image inside an MSI contains 3
channels of color and an additional $\alpha$ channel of transparency
information. This character inherited from MPI empowers the MSI to represent
scene occlusion and non-Lambertian reflectance. The form of $RGB\alpha$ data
representation also allows MSIs to be compressed using standard image
compression algorithms.

The differentiable rendering scheme for the MPI also applies on the MSI. As
described in Equ. 1, in MPI differentiable rendering, the position
$\\{p_{i}\\}_{i=1}^{N}$ where a ray intersects with each layer
$\\{L_{i}\\}_{i=1}^{N}$ is calculated and RGB value
$\\{c_{p_{i}}\\}_{i=1}^{N}$ and $\alpha$ value
$\\{\alpha_{p_{i}}\\}_{i=1}^{N}$ are interpolated and then used to calculate
the output color $c$

$c=\sum_{i=1}^{N}c_{p_{i}}\cdot{\alpha}_{p_{i}}\cdot\prod_{j=1}^{i-1}(1-{\alpha}_{p_{i}}).$
(1)

The rendering procedure for viewing MSI in 6-DoF is similar, as described in
Equ. 2. To render a ray inside of the inmost sphere of MSI with viewing angle
$(\theta,\phi)$ from camera location $(x,y,z)$, the intersection point $s_{i}$
with $i$-th layers of MSI is determined by ray-plane intersection function
$q(\cdot)$ in graphic engines, where ${d}_{i}$ is the radius of $i$-th sphere.
The color of the rendered ray $c_{x,y,z,\theta,\phi}$ is computed by color
values $\\{c_{s_{i}}\\}_{i=1}^{N}$ and transparency values
$\\{\alpha_{s_{i}}\\}_{i=1}^{N}$ in each layer following:

$\begin{split}&s_{i}=q(x,y,z,\theta,\phi,d_{i})\\\
&{c}_{x,y,z,\theta,\phi}=\sum_{i=1}^{N}c_{s_{i}}\cdot{\alpha}_{s_{i}}\cdot\prod_{j=1}^{i-1}(1-{\alpha}_{s_{i}}).\end{split}$
(2)

This procedure can be carried out efficiently on a common CG engine such as
Unity or OpenGL.

Fig. 4: Left: A fisheye image with ${220}^{\circ}$ FOV. Right: A fisheye image
reprojected to ERP format

###  4.2 Weighted Sphere Sweep Volume Construction

A weighted sphere sweep volume is constructed with $N$ layers of concentric
spheres. The radius of these spheres is chosen uniformly in the reciprocal
space between the closest object distance and the farthest distance. As images
from VR cameras are often captured using fisheye lenses that compresses the
field of view and causes chroma aberration on the edges of images, we propose
a weighted sphere sweep method to reduce such optical defects.

To this end, we first warp these input fisheye images into the ERP form as
shown in Fig. 4. Then $M$ input ERP images are projected onto $N$ layers of
sphere in ERP form using the intrinsic and extrinsic parameters of the camera.
In each layer, we fuse $M$ input images together on the overlapping area
following :

${c}_{{p}}=\sum_{i=1}^{Q}\frac{{e}^{{\gamma}_{p,i}}\cdot{c}_{p,i}}{\sum_{j=1}^{Q}{e}^{{\gamma}_{p,j}}},$
(3)

where $Q$ is the number of overlapping images at position $p$, and ${c}_{p,i}$
is the color of the $i$-th image on position $p$. The parameter $\gamma$ is
the optical distortion value. Here we use ${\gamma}_{p}=1-r_{p}$, where
$r_{p}$ is the distance of pixel $p$ to the optical center normalized to
$[0,1]$. We can also use the lens MTF data to replace $\gamma$ values Then,
$N$ projected ERP images are stacked to form a 4D volume so that the first
three dimensions are ERP image height (H), width (H) and number of layers (N),
while the 4-th dimension is the color channel. As each ERP image contains 3
channels of RGB color, the constructed WSSV has a shape of [H, W, N, 3]. The
WSSV construction can be represented by Equ. 4, where the weighted and warp
function ${\cal{W}}(\cdot)$ takes in a camera pose
${\\{\omega_{j}\\}}_{j=1}^{M}$ and the camera center pose $\omega_{c}$, then
projects an input image from ${\\{I_{j}\\}}_{j=1}^{M}$ to a set of concentric
spheres with predefined radius ${\\{d_{i}\\}}_{i=1}^{N}$. The
${\cal{S}}(\cdot)$ function stacks the warped results along 4-th dimension to
form the weighted sphere sweep volume:

$WSSV={\cal{S}}_{j=1}^{N}({\cal{W}}_{i=1}^{M}(I_{i},\omega_{i},\omega_{c},{d}_{j})).$
(4)

As each layer of sphere sweep volume is formed by multiple input images
projected onto the same sphere, the combination of these images is similar to
the result of light field refocusing. Because multiple images of the same
object are projected to a layer whose radius approximately equals to the
distance of that object, the average color of such a combination of the
projected images will appear in-focus. The 3D ConvNet in our framework is
trained to distinguish such properties and predict the transparency values,
which are further combined with WSSV to generate MSI.

Layer | s | d | n | depth | in | out | input
---|---|---|---|---|---|---|---
conv1_1 | 1 | 1 | 8 | 32/32 | 1 | 1 | WSSV
conv1_2 | 2 | 1 | 16 | 32/16 | 1 | 2 | conv1_1
conv2_1 | 1 | 1 | 16 | 16/16 | 2 | 2 | conv1_2
conv2_2 | 2 | 1 | 32 | 16/8 | 2 | 4 | conv2_1
conv3_1 | 1 | 1 | 32 | 8/8 | 4 | 4 | conv2_2
conv3_2 | 1 | 1 | 32 | 8/8 | 4 | 4 | conv3_1
conv3_3 | 2 | 1 | 64 | 8/4 | 4 | 8 | conv3_2
conv4_1 | 1 | 2 | 64 | 4/4 | 8 | 8 | conv3_3
conv4_2 | 1 | 2 | 64 | 4/4 | 8 | 8 | conv4_1
conv4_3 | 1 | 2 | 64 | 4/4 | 8 | 8 | conv4_2
nnup_5 |  |  |  | 4/8 | 8 | 4 | conv3_3+conv4_3
conv5_1 | 1 | 1 | 32 | 8/8 | 4 | 4 | nnup_5
conv5_2 | 1 | 1 | 32 | 8/8 | 4 | 4 | conv5_1
conv5_3 | 1 | 1 | 32 | 8/8 | 4 | 4 | conv5_2
nnup_6 |  |  |  | 8/16 | 4 | 2 | conv2_2+conv5_3
conv6_1 | 1 | 1 | 16 | 16/16 | 2 | 2 | nnup_6
conv6_2 | 1 | 1 | 16 | 16/16 | 2 | 2 | conv6_1
nnup_7 |  |  |  | 16/32 | 2 | 1 | conv1_2+conv6_2
conv7_1 | 1 | 1 | 8 | 32/32 | 1 | 1 | nnup_7
conv7_2 | 1 | 1 | 8 | 32/32 | 1 | 1 | conv7_1
conv7_3 | 1 | 1 | 1 | 32/32 | 1 | 1 | conv7_2

Table 1: Our proposed network architecture. The size of all convolution
kernels are set to 3. Column s,d,n denote the stride, kernel dilation and
number of output filters respectively. depth column shows the number of layer
input/output depth (we use 32 input depth layers as an example), while in and
out are the accumulated stride of layer input/output. input names with + mean
concatenation along the depth dimension and Layer names starting with “nnup”
perform 2x nearest neighbor upsampling.

###  4.3 Network Architecture

Our 3D ConvNet architecture is a slight variation of Middle et al.’s work
[17]. In our design, we modify the network to suit the WSSV input described in
Sec. 4.2.

The output vector of the neural network has a shape of [H, W, D, 1], which is
converted into the $\alpha$ channel of an MSI by a ReLu function. This
$\alpha$ channel is then stacked with WSSV RGB volume to form the MSI. As
color information of an MSI comes directly from WSSV, which is the combination
of projected input footage, during training, the network is not learning to
simulate the correct color but to select the correct layer by inferring the
corresponding weight of the $\alpha$ channel to imply an object’s distance.

The training objective is :

$\begin{split}&{MSI}_{\theta}={f}_{\theta}(WSSV)\\\
&{\cal{L}}={\cal{L}}_{L1}+{\lambda}_{VGG}{\cal{L}}_{VGG}\\\
&{\operatorname*{argmin}}_{\theta}~{}\sum_{i=1}^{M}{\cal{L}}(Render({MSI}_{\theta},\omega_{c},\omega_{i}),{\cal{W}}_{ERP}(I_{i})),\end{split}$
(5)

where the goal is to minimize the difference between the rendered MSI result
and ground truth. In Equ. 5 the 3D ConvNet $f_{\theta}(\cdot)$ takes in $WSSV$
and predicts ${MSI}_{\theta}$. The $Render(\cdot)$ function follows the
differentiable rendering scheme in Equ. 1, where the ${\cal{W}}_{ERP}(\cdot)$
is the fisheye to ERP warp function. The training loss $\cal{L}(\cdot)$ is a
weighted combination of L1 loss ${\cal{L}}_{L1}(\cdot)$ and VGG loss
${\cal{L}}_{VGG}(\cdot)$ proposed in [25]. We refer readers to Tab. 1 for
detailed network architecture.

N | Dataset | Resolution | PSNR | SSIM
---|---|---|---|---
N=32 | UnrealCV | $400\times 200$ |  28.28$\pm$2.45 |  0.90$\pm$0.03
$640\times 320$ |  28.79$\pm$2.40 |  0.90$\pm$0.03
N=64 | $400\times 200$ |  28.52$\pm$2.51 |  0.90$\pm$0.03
$640\times 320$ |  29.23$\pm$2.43 |  0.91$\pm$0.03
N=32 | Replica | $400\times 200$ |  33.48$\pm$3.72 |  0.95$\pm$0.03
$640\times 320$ |  33.40$\pm$3.87 |  0.95$\pm$0.03
N=64 | $400\times 200$ |  33.59$\pm$3.74 |  0.95$\pm$0.03
$640\times 320$ |  33.80$\pm$3.92 |  0.95$\pm$0.03
Table 2: Evaluation results. We examine our framework on the combination of
two different number of MSI layers (N=32 and N=64), two input resolution
(400$\times$200 and 640$\times$320) and two data generation engine (UnrealCV
and Replica).

##  5 Experiments

We trained and evaluated our network performance by generating MSI
representations and rendering them in different input positions and computing
peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM)
quality scores against ground truth generated using our proposed approach in
Sec 3. In this section, we describe network implementation details, and
experiment results.

###  5.1 Implementation Details

We implement our proposed 3D ConvNet using TensorFlow API following the
description in Tab. 1. During training, Automatic Mixed Precision feature was
applied to utilize GPU memory by using half precision (FP16) in computation.
The network was optimized with an SGD optimizer with the learning rate set to
$2\times{10}^{-4}$. We employed the VGG loss [25] as the perceptual loss with
weight $\lambda_{VGG}=200$. Using the data generation method described in Sec.
3, we synthesized a set of 6-sensors VR camera footage on 2000 locations and
masked the field of view to ${190}^{\circ}$ and ${220}^{\circ}$ randomly. The
entire dataset was split into 1600 locations for training subsets and the
reset for evaluation. The network was trained on 640$\times$320 resolution
with 32 layers of MSI and 400$\times$200 resolution with 64 layers of MSI
simultaneously for 400k iterations on an Nvidia RTX 2080Ti GPU.

Fig. 5: Illustration of system input and output. Left column: Input fisheye
imags. Right column: Render result from MSI predicted by our system.

###  5.2 Evaluation

We examined the performance of our approach with the evaluation subsets that
contains 400 locations with two different resolutions and corresponding color
ground truth. We generated MSIs on evaluation subsets using our proposed
method and rendered each MSI to novel viewpoints using input poses, then
computed PSNR and SSIM of these novel views with the ground truth color. These
quantitative results are reported in Tab. 2 and some selected qualitative
results are shown in Fig 5.

As demonstrated in Tab. 2, our system can generate high-quality 6-DoF contents
from VR camera footage. Comparing with the ground truth color, results
rendered on input views achieve an average PSNR over 31 dB. After carefully
examining the quantitative results in Tab. 2, we notice a minor performance
difference between two datasets. A small variance between two render engines
could be the cause, as UnrealCV engine renders reflectance that varies from
different angles while Replica uses static reconstructed object texture
acquired from real-world scenes. Overall, results rendered from MSIs show
plausible visual details on complex textures like leaves , books, floor
textures as shown in Fig. 5. We also notice there is a performance gain by
increasing the number of layers in MSI. As each MSI is a set of concentric
spheres, a denser set of spheres can represent more detailed depth variances
among scene objects. In our experiments, we also confirmed that our network
can be trained and utilized among variable image resolution and number of
layers of an MSI. We used 640x320 with 32-layer and 400x200 with 64-layer
configuration on training and evaluated the network on the combination of both
resolution and numbers of layers. Experiment results show that the network is
not overfitting on one particular configuration.

##  6 Conclusions

In this paper, we present an end-to-end deep-learning framework to compose
6-DoF omnidirectional with multi-sphere images. We use weighted sphere sweep
volume to unify inputs from various panoramic camera setups into one constant
volume size, solving the compatibility issue in previous work. Combined with
our proposed 3D ConvNet architecture, we can process camera footage directly
and reduce the systematic artifacts introduced in the ODS stitching process.
We propose a high-quality 6-DoF dataset generation method using UnrealCV and
Facebook Replica engines for training and quantitive performance evaluations.
A series of experiments were conducted to verify our system. Experiment
results show our system can operate on variable image resolution and MSI
layers, as well as producing high-quality novel views that contain correct
occlusion and detailed textures.

## References

  * [1] Benjamin Attal, Selena Ling, Aaron Gokaslan, Christian Richardt, and James Tompkin,  “Matryodshka: Real-time 6dof video view synthesis using multi-sphere images,”  in European Conference on Computer Vision. Springer, 2020, pp. 441–459.
  * [2] Venkata N Peri and Shree K Nayar,  “Generation of perspective and panoramic video from omnidirectional video,”  in Proc. DARPA Image Understanding Workshop. Citeseer, 1997, vol. 1, pp. 243–245.
  * [3] Richard Szeliski,  “Image alignment and stitching: A tutorial,”  Foundations and Trends® in Computer Graphics and Vision, vol. 2, no. 1, pp. 1–104, 2006.
  * [4] Robert Anderson, David Gallup, Jonathan T Barron, Janne Kontkanen, Noah Snavely, Carlos Hernández, Sameer Agarwal, and Steven M Seitz,  “Jump: virtual reality video,”  ACM Transactions on Graphics (TOG), vol. 35, no. 6, pp. 1–13, 2016\.
  * [5] Minhao Tang, Jiangtao Wen, Yu Zhang, Jiawen Gu, Philip Junker, Bichuan Guo, Guansyun Jhao, Ziyu Zhu, and Yuxing Han,  “A universal optical flow based real-time low-latency omnidirectional stereo video system,”  IEEE Transactions on Multimedia, vol. 21, no. 4, pp. 957–972, 2018\.
  * [6] Tobias Bertel, Mingze Yuan, Reuben Lindroos, and Christian Richardt,  “Omniphotos: casual 360° vr photography,”  ACM Transactions on Graphics (TOG), vol. 39, no. 6, pp. 1–12, 2020\.
  * [7] Jisheng Li, Ziyu Wen, Sihan Li, Yikai Zhao, Bichuan Guo, and Jiangtao Wen,  “Novel tile segmentation scheme for omnidirectional video,”  in 2016 IEEE International Conference on Image Processing (ICIP). IEEE, 2016, pp. 370–374.
  * [8] Alvaro Collet, Ming Chuang, Pat Sweeney, Don Gillett, Dennis Evseev, David Calabrese, Hugues Hoppe, Adam Kirk, and Steve Sullivan,  “High-quality streamable free-viewpoint video,”  ACM Transactions on Graphics (ToG), vol. 34, no. 4, pp. 1–13, 2015\.
  * [9] Mingsong Dou, Sameh Khamis, Yury Degtyarev, Philip Davidson, Sean Ryan Fanello, Adarsh Kowdle, Sergio Orts Escolano, Christoph Rhemann, David Kim, Jonathan Taylor, et al.,  “Fusion4d: Real-time performance capture of challenging scenes,”  ACM Transactions on Graphics (TOG), vol. 35, no. 4, pp. 1–13, 2016\.
  * [10] Ryan S Overbeck, Daniel Erickson, Daniel Evangelakos, and Paul Debevec,  “Welcome to light fields,”  in ACM SIGGRAPH 2018 Virtual, Augmented, and Mixed Reality, pp. 1–1. 2018.
  * [11] Albert Parra Pozo, Michael Toksvig, Terry Filiba Schrager, Joyce Hsu, Uday Mathur, Alexander Sorkine-Hornung, Rick Szeliski, and Brian Cabral,  “An integrated 6dof video camera and system design,”  ACM Transactions on Graphics (TOG), vol. 38, no. 6, pp. 1–16, 2019\.
  * [12] Clément Godard, Oisin Mac Aodha, and Gabriel J Brostow,  “Unsupervised monocular depth estimation with left-right consistency,”  in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017, pp. 270–279.
  * [13] Nima Khademi Kalantari, Ting-Chun Wang, and Ravi Ramamoorthi,  “Learning-based view synthesis for light field cameras,”  ACM Transactions on Graphics (TOG), vol. 35, no. 6, pp. 1–10, 2016\.
  * [14] Pratul P Srinivasan, Tongzhou Wang, Ashwin Sreelal, Ravi Ramamoorthi, and Ren Ng,  “Learning to synthesize a 4d rgbd light field from a single image,”  in Proceedings of the IEEE International Conference on Computer Vision, 2017, pp. 2243–2251.
  * [15] Tinghui Zhou, Richard Tucker, John Flynn, Graham Fyffe, and Noah Snavely,  “Stereo magnification: Learning view synthesis using multiplane images,”  in SIGGRAPH, 2018.
  * [16] John Flynn, Michael Broxton, Paul Debevec, Matthew DuVall, Graham Fyffe, Ryan Overbeck, Noah Snavely, and Richard Tucker,  “Deepview: View synthesis with learned gradient descent,”  in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019, pp. 2367–2376.
  * [17] Ben Mildenhall, Pratul P Srinivasan, Rodrigo Ortiz-Cayon, Nima Khademi Kalantari, Ravi Ramamoorthi, Ren Ng, and Abhishek Kar,  “Local light field fusion: Practical view synthesis with prescriptive sampling guidelines,”  ACM Transactions on Graphics (TOG), vol. 38, no. 4, pp. 1–14, 2019\.
  * [18] Pratul P Srinivasan, Richard Tucker, Jonathan T Barron, Ravi Ramamoorthi, Ren Ng, and Noah Snavely,  “Pushing the boundaries of view extrapolation with multiplane images,”  in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019, pp. 175–184.
  * [19] Inchang Choi, Orazio Gallo, Alejandro Troccoli, Min H Kim, and Jan Kautz,  “Extreme view synthesis,”  in Proceedings of the IEEE International Conference on Computer Vision, 2019, pp. 7781–7790.
  * [20] Kai-En Lin, Zexiang Xu, Ben Mildenhall, Pratul P. Srinivasan, Yannick Hold-Geoffroy, Stephen DiVerdi, Qi Sun, Kalyan Sunkavalli, and Ravi Ramamoorthi,  “Deep multi depth panoramas for view synthesis,”  in European Conference on Computer Vision (ECCV). 2020, vol. 12358, pp. 328–344, Springer.
  * [21] Michael Broxton, John Flynn, Ryan Overbeck, Daniel Erickson, Peter Hedman, Matthew Duvall, Jason Dourgarian, Jay Busch, Matt Whalen, and Paul Debevec,  “Immersive light field video with a layered mesh representation,”  ACM Transactions on Graphics (TOG), vol. 39, no. 4, pp. 86–1, 2020\.
  * [22] Po Kong Lai, Shuang Xie, Jochen Lang, and Robert Laqaruère,  “Real-time panoramic depth maps from omni-directional stereo images for 6 dof videos in virtual reality,”  in 2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR). IEEE, 2019, pp. 405–412.
  * [23] Weichao Qiu and Alan Yuille,  “Unrealcv: Connecting computer vision to unreal engine,”  in European Conference on Computer Vision. Springer, 2016, pp. 909–916.
  * [24] Julian Straub, Thomas Whelan, Lingni Ma, Yufan Chen, Erik Wijmans, Simon Green, Jakob J Engel, Raul Mur-Artal, Carl Ren, Shobhit Verma, et al.,  “The replica dataset: A digital replica of indoor spaces,”  arXiv preprint arXiv:1906.05797, 2019.
  * [25] Qifeng Chen and Vladlen Koltun,  “Photographic image synthesis with cascaded refinement networks,”  in Proceedings of the IEEE international conference on computer vision, 2017, pp. 1511–1520.