# Calibrating Ensembles for Scalable Uncertainty Quantification in Deep
Learning-based Medical Segmentation

Thomas Buddenkotte  Department of Applied Mathematics and Theoretical Physics,
Univeristy of Cambridge, United Kingdom  Department of Radiology, University
of Cambridge, United Kingdom  Lorena Escudero Sanchez  Department of
Radiology, University of Cambridge, United Kingdom  Cancer Research UK
Cambridge Centre, University of Cambridge, United Kingdom  Mireia Crispin-
Ortuzar  Cancer Research UK Cambridge Centre, University of Cambridge, United
Kingdom  Cancer Research UK Cambridge Institute, University of Cambridge,
United Kingdom  Department of Oncology, University of Cambridge, Cambridge,
United Kingdom  Ramona Woitek  Department of Radiology, University of
Cambridge, United Kingdom  Cancer Research UK Cambridge Centre, University of
Cambridge, United Kingdom  Medical Image Analysis & Artificial Intelligence
(MIAAI), Department of Medicine, Danube Private University, Krems, Austria
Cathal McCague  Department of Radiology, University of Cambridge, United
Kingdom  Cancer Research UK Cambridge Centre, University of Cambridge, United
Kingdom  James D. Brenton  Cancer Research UK Cambridge Centre, University of
Cambridge, United Kingdom  Cancer Research UK Cambridge Institute, University
of Cambridge, United Kingdom  Department of Oncology, University of Cambridge,
Cambridge, United Kingdom  Ozan Öktem  Department of Mathematics, KTH Royal
Institute of Technology, Sweden  Evis Sala  Department of Radiology,
University of Cambridge, United Kingdom  Cancer Research UK Cambridge Centre,
University of Cambridge, United Kingdom  Leonardo Rundo  Department of
Radiology, University of Cambridge, United Kingdom  Cancer Research UK
Cambridge Centre, University of Cambridge, United Kingdom  Department of
Information and Electrical Engineering and Applied Mathematics, University of
Salerno, Fisciano (SA), Italy

###### Abstract

Uncertainty quantification in automated image analysis is highly desired in
many applications. Typically, machine learning models in classification or
segmentation are only developed to provide binary answers; however,
quantifying the uncertainty of the models can play a critical role for example
in active learning or machine human interaction. Uncertainty quantification is
especially difficult when using deep learning-based models, which are the
state-of-the-art in many imaging applications. The current uncertainty
quantification approaches do not scale well in high-dimensional real-world
problems. Scalable solutions often rely on classical techniques, such as
dropout, during inference or training ensembles of identical models with
different random seeds to obtain a posterior distribution. In this paper, we
show that these approaches fail to approximate the classification probability.
On the contrary, we propose a scalable and intuitive framework to calibrate
ensembles of deep learning models to produce uncertainty quantification
measurements that approximate the classification probability. On unseen test
data, we demonstrate improved calibration, sensitivity (in two out of three
cases) and precision when being compared with the standard approaches. We
further motivate the usage of our method in active learning, creating pseudo-
labels to learn from unlabeled images and human-machine collaboration.

##  1 Introduction

Uncertainty quantification (UQ) in image classification and segmentation is an
actively studied area of research [1]. Many applications across various
imaging tasks can benefit from having access to the certainty of a machine
learning model instead of just considering the binary answer of the machine
learning system. For example, the work in [2] used a machine learning system
that classified cities from images of famous sights and calibrated it for
improved human machine interaction. A second important application of UQ in
machine learning is active learning [3]. The ultimate goal is efficient use of
manual labelling resources [4]. Active learning aims at finding images from a
pool of unlabeled images that give the largest performance increase when being
labelled and added to the training data. Typically, model uncertainty and
sample diversity are the main factors considered in such selection processes.
UQ can also be applied to directly learn from unlabeled images as suggested in
[5, 6].

Since the invention of the U-Net architecture [7] and the nnU-Net framework
[8], medical segmentation problems have been dominated by deep learning-based
algorithms [9]. Currently, deep learning models based on the U-Net
architecture or the nnU-Net framework are the state-of-the-art across a large
amount of medical segmentation tasks. However, the labelling of large 3D scans
can take up to hours of expert’s time per scan. Consequently, the
corresponding datasets typically only have a few hundred samples instead of
thousands or millions as in other imaging tasks. Naturally it is desirable to
integrate active learning [3, 4], human-machine collaboration [2, 10] or
learning from unlabeled images [5, 6, 11, 12, 13, 14] into the development
cycle of deep learning-based segmentation systems and large annotated
datasets.

The main challenge in applying UQ to deep learning is that deep neural
networks tend to produce overconfident predictions. It is tempting to
interpret the output of the softmax layer as a probability, however it has
been shown that these are overconfident and do not represent the certainty of
the model well [15]. This is especially prevalent in medical segmentation
tasks as here state-of-the-art models often use a combination of entropy-based
and overlap-based loss functions [9], which force the networks to produce
almost binary predictions. Many alternatives—such as variational inference,
Bayesian backpropagation, or Bayesian neural networks—have been suggested for
the use in UQ tasks [1], however most of these methods are difficult to apply
in large scale 3D networks [16]. The two most popular approaches that can be
employed in these problems are: (i) to use Dropout both during training and
inference, and (ii) to train ensembles of identical models with different
random seeds [16].

In this paper, we propose a scalable and intuitive framework for UQ in 3D
medical image segmentation. The framework simply suggests training models
ranging from highly sensitive to highly precise and combine their outputs with
appropriate weightings. We demonstrate that our framework is superior in
estimating classification probabilities when being compared to well-
established methods, such as inference Dropout and the traditional ensembling.
We further motivate its adoption in active learning [3, 4] and creating
pseudo-labels to learn from unlabeled images [11, 12] and human-machine
collaboration [2, 10].

This manuscript is organized as follows. Section 2 motivates and introduces
our suggested approach. Section 3 describes the datasets and deep learning
models used throughout the manuscript. Section 4 shows results on comparing
our method to two established approaches. Section 5 motivates and describes
potential ways of applying this method in a clinical setting or in creating
improved deep learning algorithms. Section 6 summarizes the paper and
discusses potential next steps.

##  2 Calibration of Ensembles for Uncertainty Quantification

Let us consider a one-dimensional classification task as a motivational
example. Consider an equal number of samples from the two distributions
$p_{0}=\mathcal{N}(-1,1)$ and $p_{1}=\mathcal{N}(1,1)$ are given and the task
to determine whether some sample $x$ came from the distribution $p_{0}$ or
$p_{1}$. Naturally one would suggest the function:

$c(x)=\left\\{\begin{array}[]{ll}0&x\leq 0\\\ 1&\,\textrm{otherwise}\\\
\end{array},\right.$

as a classifier that achieves maximal accuracy. In this problem, it is
expected that the uncertainty of the classification is high close to the
decision boundary in $x=0$. As in this problem the densities of the two
distributions are known, the classification probability can be computed
analytically via

$\mathbf{P}(x\in C_{1})=\frac{p_{1}(x)}{p_{0}(x)+p_{1}(x)}.$

A trained classifier for this problem would not place the decision boundary
exactly at $x=0$, but only close by due to the early stopping of the training.
As a consequence, ensembles of such models would only express uncertainty
close around $x=0$, but would not approximate the classification accuracy well
as shown in Figure 1. These plots suggest why ensembles of identical models
trained with different random seeds are not suited to approximate the
classification probability. All decision boundaries are distributed close
around 0, where a step function is also required to have jumps further away
from 0 to approximate the classification probability well. Thus, it is
necessary to train classifiers with shifted decision boundaries. Additionally,
it is necessary to find suitable weightings for the different models to adjust
the step heights and to finally produce a good approximation of the
classification probability.

Figure 1: A minimal example of classification functions and classification
probability for a one-dimensional feature space. The example demonstrates why
training identical models with different random seeds fails to approximate the
classification probability.

We suggest creating models with a large variety of decision boundaries by
altering the importance of sensitivity and precision throughout the ensemble.
We achieve this by using parametric loss functions for the calibration of
models in the ensemble. For this, we modify the widely used sum of cross
entropy and soft dice loss in the following way:

$\displaystyle L_{w}(y,p)$ $\displaystyle=L^{ce}_{w}(y,p)+L^{DSC}_{w}(y,p)$
$\displaystyle L^{ce}_{w}(y,p)$
$\displaystyle=\frac{1}{N}\sum_{i=1}^{N}y_{0,i}\log(p_{0,i})+\sum_{c=1}^{C}\exp(-w_{c})y_{c,i}\log(p_{c,i})$
(1) $\displaystyle L^{DSC}_{w}(y,p)$
$\displaystyle=\frac{1}{C}\sum_{c=1}^{C}2\frac{TP}{TP+\sigma(w_{c})FP+\sigma(-w_{c})FN},$

where $C$ is the number of foreground classes, $y$ is the ground truth given
in one hot encoding, $p$ is the softmax output of the neural network, $\sigma$
is the sigmoid function and $TP$, $FP$, and $FN$ are the number of true
positives, false positives and false negatives. It is to be noted that the
original sum of cross entropy and dice loss can be obtained by choosing $w=0$.
In this formulation negative weights promote sensitivity and positive weights
promote precision.

To obtain meaningful estimations for the classification probability, one needs
to compute a weighted sum of the models trained with differing loss function
weights $w$. However, in contrast to the minimal example from Figure 1, we
cannot compute the classification probability analytically and need to rely on
other ways to determine meaningfulness. For this reason, we suggest to compare
the estimated probabilities with the prevalence of foreground voxels. The
rational is that these two quantities should match. For example, if 20% of the
voxels marked with a probability of 0.2 contain foreground, it can be argued
that 0.2 was a reasonable estimate for the classification probability. In the
calibrated ensemble we suggest achieving this by matching the two quantities
on the training set. In a first step we train the model following a cross-
validation procedure to obtain one (non-overfitted) prediction for each image
in the training set. Next, we setup up a linear system to determine the
coefficients in the linear combination. For this, let $s_{w_{k}}$ be the
segmentation models of the ensemble with loss weights $w_{k}$. Since only
binary segmentation models are considered in this approach, the vector
$(s_{w_{0}}(x)_{i},\dots,s_{w_{n}}(x)_{i})$ has only binary entries for each
image $x$ and voxel $i$. Given the set $B$ of all possible binary vectors and
a set of images $X$, the linear system

$\displaystyle\sum_{x\in X}\sum_{i}g_{i}$
$\displaystyle\mathbb{I}((s_{w_{0}}(x)_{i},\dots,s_{w_{n}}(x)_{i})=b)=$ (2)
$\displaystyle\sum_{k}a_{k}b_{k}\sum_{x\in
X}\sum_{i}\mathbb{I}((s_{w_{0}}(x)_{i},\dots,s_{w_{n}}(x)_{i})=b)\quad b\in B$

can be used to determine the coefficients $a_{k}$ of the calibrated ensemble,
where $\mathbb{I}$ is the indicator function (1 if the expression is true, 0
otherwise). The heatmap, which approximates the classification probability in
each pixel, can finally be computed as

$h=\sum_{k}a_{k}s_{w_{k}}.$ (3)

##  3 Materials and Methods

###  3.1 Datasets

The experiments were conducted on two independent datasets: (i) a dataset of
CT images of high grade serous ovarian cancer patients, and (ii) a public
dataset of CT images of kidney tumors111https://kits21.kits-
challenge.org/[17].

The ovarian cancer data consisted of four datasets in total. All scans were
collected retrospectively after receiving the patient’s consent for the usage
in research. For training we used a dataset of 276 scans from 157 patients
that were referred from various clinics to Addenbrooke’s hospital in
Cambridge, UK, for diagnosis and treatment. From all patients a pre-treatment
scan was available and for those treated with neoadjuvant chemotherapy and
additional post-treatment scan was available as well. The test data was
obtained by merging three datasets. The first one contained 104 scans from 53
patients, which received neo-adjuvant chemotherapy, obtained at St.
Bartholomew’s (Barts) hospital in London, UK. The second dataset was obtained
by the Apollo network in the US and consisted of 51 scans from 51 patients
which received immediate primary surgery. The last dataset consisted of 20
scans from 20 patients from the cancer imaging archive (TCIA), which were also
acquired in the US from patients who also received immediate primary surgery.

All scans were either manually segmented by a trainee radiologist and
subsequently reviewed and corrected by a consultant radiologist, or directly
segmented by the consultant radiologist. For the 104 scans obtained at Barts
hospital in London, both the segmentations reviewed and corrected by the
consultant radiologist (ground truth) and the unreviewed segmentations
produced by the trainee radiologist (trainee segmentations) were available.
While this disease is regularly expressed in many different sites across the
whole abdomen, we focus only on the main two disease sites which are the
pelvis/ovaries and the omentum, as they are by far the largest and most
frequent sites.

Additionally, we used the Kidney Tumor Segmentation 2021 (KiTS21)
dataset222https://kits21.kits-challenge.org/ [17]. The dataset consists of 300
publicly available scans to be used for model training and a hidden test
dataset. Both kidneys and all prevalent kidney tumors and cysts were segmented
by three radiologists independently. At least one tumor was visible in each
scan. We decided to focus only on the kidney tumors for simplification and
used the majority vote of the three prevalent masks for training and testing.
The publicly available data was randomly split in 60% training (n=180) and 40%
test scans (n=120).

###  3.2 Deep learning model

The implementation is based on PyTorch 1.9 and was coded from scratch, i.e.,
not relying on public open libraries. However, our code design and hyper-
parameter closely follow the implementation in the nnU-Net framework [8]. All
scans were first interpolated to a uniform voxel spacing of $0.8\times
0.8\times 5\text{mm}^{3}$, which resulted in higher DSC compared to the median
voxel spacing, which was suggested by nnU-Net. The resizing was followed by
windowing the image by the 99.5 and 0.5 percentile of the gray values of
voxels that contain foreground and Z-score normalization. The augmentation was
applied as suggested by nnU-Net [8]. The architecture was chosen to be U-Net
with a ResNet [18, 19] decoder using 1, 2, 6, and 3 blocks per stage and 32,
63, 128, and 256 filters per stage. For the ovarian cancer segmentation we
used a single multi-class U-Net to segment both disease sites simultaneously.
For the kidney data we used a single-class U-Net for only the tumors.
Additionally, we only sampled patches centered at randomly chosen voxels
inside the kidney and excluded voxels outside the of kidneys by using a mask
in the loss function. The training was performed with a standard stochastic
gradient optimizer using Nesterov’s momentum of factor 0.98, a weight decay of
$10^{-4}$ and a linear ascent plus cosine decay learning rate schedule with
maximum rate 0.02. We used a batch size of 4 for all experiments where one
sample of each batch was forced to be centered at a randomly chosen foreground
voxel. Full 3D volumes were predicted using the Sliding Window approach with
an overlap of 0.5 patches and a Gaussian weighting as suggested in [8]. The
post-processing was performed by simply applying the argmax function to the
softmax outputs of the network.

To reduce the training cost, we applied progressive learning as suggested in
[20]. Here the training is split in four equally long stages. The amount of
voxels contained in a sample and the magnitude of the augmentations was
reduced by a factor of 4, 3, and 2 in the first, second, and third stage of
the training. This alone reduced the training time by roughly 40%. However,
preliminary experiments also showed that full retraining from scratch is not
needed when changing the loss weight $w_{k}$ in the loss function. In this
case it is sufficient to restart the training after the third stage of the
progressive learning and finish the training with the new choice of $w_{k}$.

For the calibrated ensemble, we aimed at sampling the loss weights $w$
symmetrically around the parameter $w^{DSC}$ that maximizes the mean DSC.
Using cross-validation we determined $w^{DSC}$ (-0.5, -1.5, and -1.0 for
pelvic/ovarian disease, omental disease and kidney tumors respectively) and
chose the loss weights of the calibrated ensembles to be
$\\{w^{DSC}-k|k=-3,-2,\dots,3\\}$. For each of these loss function weights a
full cross-validation was trained to determine the coefficients $a_{k}$ as
described in equation 2. Finally, for each $w_{k}$ ensembling over all models
in the cross-validation predictions was applied on the test set before
computing the heatmap as suggested by equation 3.

As a baseline to compare our approach with, we trained an (uncalibrated)
ensemble seven models trained with random seeds and a network using Dropout
with drop probability $p=0.1$ after each convolution (except for the logits).
Both were trained using the full training data instead of relying on cross-
validation splits, embracing the weighted loss function from Equation 1 with
weight $w^{DSC}$. The Dropout model was evaluated seven times on the test data
with Dropout in training mode. The heatmaps were computed as the pixel-wise
mean over the seven predictions for both the uncalibrated ensembles and the
Dropout model.

The implementation is available at https://github.com/ThomasBudd/ovseg. Please
follow the documentation and manual to learn how to use the library for new
data and apply the suggested method to your segmenation problem.

##  4 Results

To check the meaningfulness of the heatmaps the predicted probabilities were
compared with the prevalence of foreground voxels in the test set. As
previously described in Section 2, a probability estimation can be considered
meaningful if these two quantities match. If the prevalence of foreground
voxels is higher/lower than the predicted probability, the uncertainty
quantification approach underestimates/overestimates the risk. We used Parzen
Windowing [21] with a linear kernel to compute the pixel-wise classification
probability as the ratio of ground truth voxels per bin. The results are shown
in Figure 2. It is clearly visible that the calibrated ensemble has a better
calibration on external test for all three segmentation tasks, when being
compared to uncalibrated ensembles and dropout networks. The latter two have
instead a tendency for overestimation in regions of high probability and
underestimation in regions of low probability.

Figure 2: Calibration curves of the different uncertainty quantification
approaches (dashed lines) compared with perfect calibration (solid line). The
area above and below the solid line correspond to underestimating and
overestimating probabilities respectively as marked in the middle plot.

Figure 3 shows heatmaps of the three different approaches. The heatmaps
produced by uncalibrated ensembles and Dropout models express uncertainty in
relatively few voxels compared to our suggested approach. Instead, some voxels
were falsely included or excluded with very high certainty. In contrast to
this, the calibrated ensemble captures almost all foreground, even if with low
probability in some cases. Voxels outside the ground truth segmentation were
typically annotated with a low probability. To quantify this, we computed the
sensitivity of the union and the precision of the intersection of all binary
segmentation masks for each approach. The results can be found in Table 1.
Overall, the calibrated ensemble achieves the highest values except for the
sensitivity for the omental disease, where the Dropout model achieves a
marginally higher score.

Figure 3: Visual comparison of the heatmaps created by different approaches. The uncalibrated ensemble and Dropout heatmaps are almost binary and falsely include and exclude voxels with high certainty. This is considerably improved in the heatmaps of the calibrated ensemble. |  | Uncalibrated | Calibrated | Dropout
---|---|---|---|---
Disease | Metric | ensemble | ensemble | model
Pelvic/ovarian | Sens | 89.0 | 84.6 | 86.9
disease | Prec | 90.4 | 81.7 | 82.3
Omental | Sens | 75.5 | 72.4 | 75.6
disease | Prec | 91.6 | 75.9 | 74.7
Kidney | Sens | 95.5 | 94.3 | 94.9
tumor | Prec | 98.1 | 91.6 | 89.9
Table 1: Sensitivity of the union and precision of the intersection of all
binary segmentation masks produced by the different UQ approaches. The
calibrated ensemble achieves overall the highest scores except for the
sensitivity of the omental disease.

##  5 Applications to Active Learning and Processing of Unlabeled Images

In the following, we will motivate and discuss potential applications of our
method in active learning, creating pseudo-labels for training on unlabeled
datasets and human-machine collaboration. In this section we used the majority
vote of the uncalibrated ensemble as a prediction on the test sets as
suggested in [8].

Figure 4: Motivational examples for usage in other areas of research. The
first, second and third row motivate the usage in active learning, creating
pseudo-labels on unlabeled images and human-machine collaboration
respectively. The top row compares the estimated DSC, computed from different
heatmaps, with the DSC computed from ground truth segmentations. Our approach
demonstrates a better agreement between the two compared to uncalibrated
ensembles and dropout-based methods. The middle row compares the estimated
probabilities with the relative frequency of ground truth and predicted
foreground. The large deviation of the two curves demonstrates the
suboptimality of the prediction. A loss function that promotes similarity of
the estimated probability and the relative frequency of predicted foreground
might be applied on unlabeled datasets to solve this issue. The predictions
were chosen to be a majority vote of predictions from the uncalibrated
ensemble as these resulted in the higher DSCs compared to using calibrated
ensembles or Dropout predictions for these tasks. The bottom row considers
human-machine collaboration by checking the probabilities our approach
computed in voxels where a trainee radiologist segmentation deviated from the
ground truth.

###  5.1 Active learning

Applying active learning to the process of manually labelling new datasets can
bring huge advantages. It is either possible to directly reduce the manual
labelling time by integrating a continuously updated model in the labelling
process [4], or to select a subset of scans that contribute most to the models
performance [3].

To motivate the adoption of our suggested approach in the area of active
learning we aimed at estimating the DSC using only a heatmap $h$ and prevalent
binary predictions $p$. First, we compute the DSC of some prediction $p$ and
the ground truth segmentation $s$ on the test set using the standard
formulations of the DSC:

$\displaystyle TP(s,p)$ $\displaystyle=\sum_{i}s_{i}p_{i}$ $\displaystyle
FP(s,p)$ $\displaystyle=\sum_{i}(1-s_{i})p_{i}$ $\displaystyle FN(s,p)$
$\displaystyle=\sum_{i}s_{i}(1-p_{i})$ $\displaystyle DSC(s,p)$
$\displaystyle=100\times\frac{2TP(s,p)}{2TP(s,p)+FP(s,p)+FN(s,p)}.$

Following the hypothesis that the heatmaps created by our suggested approach
do approximate the classification probability, we can compute the expected
number of true positives, false positives and false negatives by simply
replacing $s$ with $h$ in the formulas above. Hence, the DSC can be estimated
with this technique without having access to the ground truth. The top row of
Figure 4 compares DSCs from the different heatmaps with the DSCs computed from
the ground truth. It can be observed that the heatmaps created by uncalibrated
ensembles and inference Dropout tend to overestimate the true DSC, while using
the heatmaps created by our approach results in a more accurate DSC. The
overestimation of the DSC is matches the observation from Figure 3 that
suggested that uncalibrated ensembles and Dropout models tend to produce false
positives and negatives with high confidence.

###  5.2 Creation of pseudo-labels for learning on unlabeled datasets

Next, we investigated the usage of our method in the creation of pseudo-labels
for training on unlabeled datasets. Including unlabeled images in the training
process can be of significant advantage compared to standard supervised
methods, for example when using self-supervised methods as pretraining [13,
14] or using pseudo-labels in semi-supervised learning [11, 12].

As demonstrated in Figure 2, comparing prevalence of foreground voxels and
predicted probability can be used to validate heatmaps generated by
uncertainty quantification approaches. However, given a heatmap where
prevalence of foreground voxels and predicted probability are reasonably well
calibrated, this comparison can also be used to validate deep neural network
predictions. Figure 2 shows that only the calibration curve of the calibrated
ensemble was relatively close to perfect calibrated on the test set, which is
why we focused solely on this approach for this application. If the prevalence
of predicted foreground voxels does not match the probabilities generated by
the heatmap, the prediction cannot be perfect. The second row of Figure 4
demonstrates that this is not the case for our predictions which were trained
in a fully supervised fashion. The current predictions tend to annotate too
much foreground in regions of large certainty. Additionally for the
pelvic/ovarian lesions and kidney tumors it can be observed that the model
tends to annotate too little foreground in regions of low certainty.

It should be noted that even though a prediction method produces segmentations
where the prevalence of foreground voxels and the predicted probability of a
heatmap match, it must match with the ground truth. For example, a
segmentation method that produces a checkerboard pattern in voxels with a
probability of 0.5 would fulfill the optimality condition while producing
biologically implausible segmentations. This is why methods cannot be trained
only basing on this pseudo-label approach. Instead, we believe that it can
either augment existing supervised approaches in a semi-supervised framework,
or serve as a pre-training objective followed by supervised fine-tuning.

###  5.3 Human-machine collaboration

Lastly, we want to motivate the potential usage in human-machine
collaboration. In this dicispline it is researched how and if humans can be
partnered with machine learning systems to improve overall performance [2, 10]

For this we used the unreviewed trainee radiologist segmentations that were
available on a subset of 104 scans of the test set of the ovarian cancer data.
More specifically, we wanted to understand if the AI can be used to
potentially flag false positives or false negatives in the trainee’s work. The
last row of Figure 4 shows boxplots of the estimated probabilities, generated
by the calibrated ensemble, in voxels that were falsely annotated by the
trainee radiologist as positive/negative. The plots show that the heatmap
actually annotated a large number of false positive voxels with a very low
probability. In fact, 50% and 75% of the false positive annotations of the
trainee were annotated with a probability of less than 0.25 by the heatmap for
pelvic/ovarian and omental disease respectively. Additionally, more than 25%
of the false negative annotations of the trainee were marked with a
probability of greater than 0.7 by the heatmap. For the omental disease, more
than 75% of the false negatives were annotated with a probability of less than
0.5, indicating that the majority these mistakes by the trainee were also
difficult to detect for the deep learning system.

It should be noted that simply removing trainee annotations with low
probabilities can be harmful as also true positives of the trainee might be
removed. Instead, such system could be used to flag potential false positives
and negatives to the trainee and suggest to rethink the decision made.

##  6 Discussion

In this paper, we presented a simple, yet effective, approach for calibrating
ensembles of deep learning models to approximate the classification
probability in 3D medical image segmentation, while many other approaches do
not scale well in these high-dimensional problems. We motivated why ensembles
of identical models trained with different random seeds fail to approximate
the classification probability and why models with differing sensitivity and
precision are needed to do so. We demonstrated that it is feasible to apply
the approach in complex segmentation problems using large 3D networks and that
the resulting heatmaps approximate the classification probability. We also
showed that the traditional approaches tend to produce false positives and
negatives with high confidence. Considering the intersection/union of all
segmentation produced by the different approaches, ours had the highest
precision/sensitivity in all except one case. We further motivated how our
method can be used in active learning [3, 4], computing pseudo-labels to learn
from unlabeled data [11, 12, 13, 14] and human-machine collaboration [2, 10].

Our main limitations are the following. While our method only considers the
estimation of the classification probability, it does not contain any out-of-
distribution detection [22]. However, we believe that such approaches can be
combined with ours due to the simplicity of our suggested framework. Next, our
currently suggested framework introduces high computational cost during
inference making it unattractive for clinical deployment.

For future work we suggest investigating the combination of our approach with
other UQ approaches such as out of distribution detection [22]. For this, we
believe that it is possible to generalize our approach to consider also soft
segmentations by adapting Equation 2 adequately. To further improve the
sensitivity and precision of the intersection/union of segmentations in the
calibrated ensemble, it might be useful to adapt the choices of loss weights
$w$ accordingly. We also believe that knowledge distillation might be of great
help to reduce inference cost and facilitate clinical deployment. For this we
believe that theoretical considerations and clinical implementations of our
approach are needed to determine the value of UQ based methods like ours in
human-machine collaboration. Furthermore, it would be interesting to see how
our suggested method of pretraining on unlabeled datasets using pseudo-labels
performs when being compared to other pretraining approaches like self-
supervised learning [13, 14]. We also believe that our method could be
extended to incorporate the impact of the uncertainty in the segmentation into
Radiomics-based classification models using such segmentations, or even be
extended from segmentation problems to classification problems like Radiomics-
based classification.

To conclude, we proposed a simple and intuitive approach to develop calibrated
ensembles that approximate classification probabilities in 3D medical
segmentation problems.

## Acknowledgement

The authors would like to thank everyone involved in the data collection and
labeling process of the analyzed ovarian cancer datasets. In particular, the
authors thank Lucian Beer, Vlad Bura, Hilal Sahin, Roxana Pintican and Marta
Zerunian for assisting the manual labeling process. The authors also thank
Naveena Singh, Anju Sahdev for collecting the patient data at the Barts
Hospital, London, UK. Additionally, the authors thank Kathleen Barcy, Larry G.
Maxwell, Nicholas W. Bateman, Thomas P. Conrads and John B. Freymann for the
collection of the Apollo dataset, as well as Iris Allajbeu for the assistance
in collecting the scans available from the Cancer Imaging Archive.

This work was partially supported by the Wellcome Trust Innovator Award
[RG98755], The Mark Foundation for Cancer Research and Cancer Research UK
Cambridge Centre [C9685/A25177], and the CRUK National Cancer Imaging
Translational Accelerator (NCITA) [C42780/A27066]. Additional support was also
provided by the National Institute of Health Research (NIHR) Cambridge
Biomedical Research Centre (BRC-1215-20014). RW was supported by the Austrian
Science Fund (FWF) [J4025-B26]. MC-O was funded by the EPSRC Tier-2 capital
grant EP/P020259/1. The views expressed are those of the authors and not
necessarily those of the NHS, the NIHR, or the Department of Health and Social
Care.

The work by Öktem was supported by the Swedish Foundation of Strategic
Research under Grants AM13-0049.

Microsoft Radiomics was provided to the Addenbrooke’s Hospital (Cambridge
University Hospitals NHS Foundation Trust, Cambridge, UK) by the Microsoft
InnerEye project.

## References

  * [1] Moloud Abdar, Farhad Pourpanah, Sadiq Hussain, Dana Rezazadegan, Li Liu, Mohammad Ghavamzadeh, Paul Fieguth, Xiaochun Cao, Abbas Khosravi, U. Rajendra Acharya, Vladimir Makarenkov, and Saeid Nahavandi.  A review of uncertainty quantification in deep learning: Techniques, applications and challenges.  Information Fusion, 76:243–297, 2021.
  * [2] Kailas Vodrahalli, Tobias Gerstenberg, and James Zou.  Uncalibrated models can improve human-ai collaboration, 2022.
  * [3] Pengzhen Ren, Yun Xiao, Xiaojun Chang, Po-Yao Huang, Zhihui Li, Xiaojiang Chen, and Xin Wang.  A survey of deep active learning.  ACM Computing Surveys (CSUR), 54:1 – 40, 2022.
  * [4] Jakob Wasserthal, Manfred Meyer, Hanns-Christian Breit, Joshy Cyriac, Shan Yang, and Martin Segeroth.  Totalsegmentator: robust segmentation of 104 anatomical structures in ct images, 2022.
  * [5] Lequan Yu, Shujun Wang, Xiaomeng Li, Chi-Wing Fu, and Pheng-Ann Heng.  Uncertainty-aware self-ensembling model for semi-supervised 3d left atrium segmentation.  In Dinggang Shen, Tianming Liu, Terry M. Peters, Lawrence H. Staib, Caroline Essert, Sean Zhou, Pew-Thian Yap, and Ali Khan, editors, Medical Image Computing and Computer Assisted Intervention – MICCAI 2019, pages 605–613, Cham, 2019. Springer International Publishing.
  * [6] Lequan Yu, Shujun Wang, Xiaomeng Li, Chi-Wing Fu, and Pheng-Ann Heng.  Uncertainty-aware self-ensembling model for semi-supervised 3d left atrium segmentation.  In Dinggang Shen, Tianming Liu, Terry M. Peters, Lawrence H. Staib, Caroline Essert, Sean Zhou, Pew-Thian Yap, and Ali Khan, editors, Medical Image Computing and Computer Assisted Intervention – MICCAI 2019, pages 605–613, Cham, 2019. Springer International Publishing.
  * [7] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-net: Convolutional networks for biomedical image segmentation.  In Nassir Navab, Joachim Hornegger, William M. Wells, and Alejandro F. Frangi, editors, Medical Image Computing and Computer-Assisted Intervention – MICCAI 2015, pages 234–241, Cham, 2015. Springer International Publishing.
  * [8] Fabian Isensee, Paul F. Jaeger, Simon A. A. Kohl, Jens Petersen, and Klaus H. maier Hain.  nnu-net: a self-configuring method for deep learning-based biomedical image segmentation.  Nature Methods, 18, 2021.
  * [9] Jun Ma.  Cutting-edge 3d medical image segmentation methods in 2020: Are happy families all alike?  ArXiv, abs/2101.00232, 2021.
  * [10] Thomas Schaffter, Diana S. M. Buist, Christoph I. Lee, Yaroslav Nikulin, Dezső Ribli, Yuanfang Guan, William Lotter, Zequn Jie, Hao Du, Sijia Wang, Jiashi Feng, Mengling Feng, Hyo-Eun Kim, Francisco Albiol, Alberto Albiol, Stephen Morrell, Zbigniew Wojna, Mehmet Eren Ahsen, Umar Asif, Antonio Jimeno Yepes, Shivanthan Yohanandan, Simona Rabinovici-Cohen, Darvin Yi, Bruce Hoff, Thomas Yu, Elias Chaibub Neto, Daniel L. Rubin, Peter Lindholm, Laurie R. Margolies, Russell Bailey McBride, Joseph H. Rothstein, Weiva Sieh, Rami Ben-Ari, Stefan Harrer, Andrew Trister, Stephen Friend, Thea Norman, Berkman Sahiner, Fredrik Strand, Justin Guinney, Gustavo Stolovitzky, , and the DM DREAM Consortium.  Evaluation of Combined Artificial Intelligence and Radiologist Assessment to Interpret Screening Mammograms.  JAMA Network Open, 3(3):e200265–e200265, 03 2020.
  * [11] Eric Arazo, Diego Ortego, Paul Albert, Noel E. O’Connor, and Kevin McGuinness.  Pseudo-labeling and confirmation bias in deep semi-supervised learning, 2020.
  * [12] Rihuan Ke, Angelica I. Aviles-Rivero, Saurabh Pandey, Saikumar Reddy, and Carola-Bibiane Schönlieb.  A three-stage self-training framework for semi-supervised semantic segmentation.  IEEE Transactions on Image Processing, 31:1805–1815, 2022.
  * [13] Shekoofeh Azizi, Laura Culp, Jan Freyberg, Basil Mustafa, Sebastien Baur, Simon Kornblith, Ting Chen, Patricia MacWilliams, S. Sara Mahdavi, Ellery Wulczyn, Boris Babenko, Megan Wilson, Aaron Loh, Po-Hsuan Cameron Chen, Yuan Liu, Pinal Bavishi, Scott Mayer McKinney, Jim Winkens, Abhijit Guha Roy, Zach Beaver, Fiona Ryan, Justin Krogue, Mozziyar Etemadi, Umesh Telang, Yun Liu, Lily Peng, Greg S. Corrado, Dale R. Webster, David Fleet, Geoffrey Hinton, Neil Houlsby, Alan Karthikesalingam, Mohammad Norouzi, and Vivek Natarajan.  Robust and efficient medical imaging with self-supervision, 2022.
  * [14] Priya Goyal, Mathilde Caron, Benjamin Lefaudeux, Min Xu, Pengchao Wang, Vivek Pai, Mannat Singh, Vitaliy Liptchinsky, Ishan Misra, Armand Joulin, and Piotr Bojanowski.  Self-supervised pretraining of visual features in the wild, 2021.
  * [15] Dennis Ulmer and Giovanni Cinà.  Know your limits: Monotonicity & softmax make neural classifiers overconfident on OOD data.  CoRR, abs/2012.05329, 2020.
  * [16] Evaluating Scalable Bayesian Deep Learning Methods for Robust Computer Vision, 2020.
  * [17] Nicholas Heller, Fabian Isensee, Klaus H. Maier-Hein, Xiaoshuai Hou, Chunmei Xie, Fengyi Li, Yang Nan, Guangrui Mu, Zhiyong Lin, Miofei Han, Guang Yao, Yaozong Gao, Yao Zhang, Yixin Wang, Feng Hou, Jiawei Yang, Guangwei Xiong, Jiang Tian, Cheng Zhong, Jun Ma, Jack Rickman, Joshua Dean, Bethany Stai, Resha Tejpaul, Makinna Oestreich, Paul Blake, Heather Kaluzniak, Shaneabbas Raza, Joel Rosenberg, Keenan Moore, Edward Walczak, Zachary Rengel, Zach Edgerton, Ranveer Vasdev, Matthew Peterson, Sean McSweeney, Sarah Peterson, Arveen Kalapara, Niranjan Sathianathen, Nikolaos Papanikolopoulos, and Christopher Weight.  The state of the art in kidney and kidney tumor segmentation in contrast-enhanced ct imaging: Results of the kits19 challenge.  Medical Image Analysis, 67:101821, 2021.
  * [18] Deep Residual Learning for Image Recognition, 2016.
  * [19] Andrew Brock, Soham De, Samuel L. Smith, and Karen Simonyan.  High-performance large-scale image recognition without normalization.  CoRR, abs/2102.06171, 2021.
  * [20] Marina Meila and Tong Zhang, editors.  EfficientNetV2: Smaller Models and Faster Training, volume 139 of Proceedings of Machine Learning Research. PMLR, 18–24 Jul 2021.
  * [21] E. Parzen.  Mathematical considerations in the estimation of spectra.  Technometrics, 3:167–190, 1961.
  * [22] Distance-based detection of out-of-distribution silent failures for covid-19 lung lesion segmentation, 2022.


# Defect-seeded lateral epitaxy and exfoliation on graphene-terminated
surfaces

Sebastian Manzo  Patrick J. Strohbeen  Zheng-Hui Lim  Vivek Saraswat  Michael
S. Arnold  Jason K. Kawasaki  Materials Science and Engineering, University of
Wisconsin-Madison, Madison, WI 53706

###### Abstract

We exploit defects in a 2D barrier layer to drive the seeded lateral epitaxy
of atomically smooth, exfoliatable, single-crystalline membranes. This growth
mode offers many of the advantages of the recently discovered remote epitaxy,
including novel strain relaxation pathways and the ability to exfoliate free-
standing membranes, plus additional advantages of (1) tolerance for imperfect
graphene/substrate interfaces and (2) the ability to engineer the growth by
patterning the 2D graphene layer. We show that GaSb films grow on graphene-
terminated GaSb (001) via a seeded lateral epitaxy mechanism, in which pinhole
defects in the graphene serve as selective nucleation sites, followed by
lateral epitaxy and coalescence into a continuous film. Importantly, the small
size of the pinholes permits exfoliation of a continuous, free-standing GaSb
membrane. By combining molecular beam epitaxy with in-situ electron
diffraction and photoemission, plus ex-situ atomic force microscopy and Raman
spectroscopy, we track the graphene defect generation and GaSb growth
evolution a few monolayers at a time. Our discovery provides a highly tunable
method to engineer single-crystalline 3D materials on precisely tuned 2D
barriers.

##  I Introduction

Remote epitaxy of three-dimensional materials on graphene-terminated surfaces
promises to circumvent many of the limitations of conventional epitaxy. For
example, the weak van der Waals interaction between the film and graphene
creates new strain relaxation pathways for highly mismatched growth Bae et al.
(2020); Jiang et al. (2019) and enables mechanical exfoliation of free-
standing crystalline membranes for flexible devices Kim et al. (2017); Guo et
al. (2019) and tunign properties via strain in novel geometries Du et al.
(2021). Since the first demonstrations on compound semiconductors Kim et al.
(2017); Kong et al. (2018), epitaxy and exfoliation from graphene have been
demonstrated for a variety of other materials including transition metal
oxides Kum et al. (2020); Guo et al. (2019), simple metals Lu et al. (2018),
halide perovskites Jiang et al. (2019), and intermetallic compounds Du et al.
(2021). These films are typically thought to grow via a remote epitaxy
mechanism, in which epitaxial registry between film and substrate is achieved
via remote interactions through the graphene Kim et al. (2017); Kong et al.
(2018).

Here we discover a new growth mode that delivers many of the practical
advantages of remote epitaxy, and offers additional advantages of (1) the
possibility to engineer the growth mode via patterning of the 2D material
barrier and (2) relaxed requirements for cleanliness of the graphene/substrate
interface. We demonstrate the seeded lateral epitaxy of atomically smooth,
exfoliatable GaSb films on a graphene-terminated GaSb (001) substrate. Pinhole
defects in the graphene, which are created by the native oxide desorption from
the substrate, serve as nucleation sites for GaSb, followed by lateral epitaxy
and coalescence. Since the pinholes are small ($\sim 10$ nm diameter, up to
300 nm), films grown by seeded lateral epitaxy can be exfoliated to yield
smooth, free-standing single-crystalline membranes. The fast surface diffusion
on graphene and high chemical selectivity for nucleation at pinholes are
essential in driving this mechanism. We track the evolution of GaSb film
growth on graphene-terminated GaSb(001), a few atomic layers at a time, by
combining molecular beam epitaxy (MBE) with in-situ electron diffraction and
photoemission spectroscopy (XPS), plus ex-situ atomic force microscopy (AFM)
and scanning electron microscopy (SEM). Our experiments demonstrate that the
introduction of defects provides a powerful knob for tuning epitaxial growth
on 2D materials surfaces.

##  II Results and Discussion

###  II.1 Epitaxy and exfoliation of GaSb films

Figure 1: Epitaxy and exfoliation of GaSb on graphene-terminated GaSb (001).
(a-d) GaSb epitaxy on a GaSb (001) substrate that is half covered with
graphene. (a) Reflection high energy electron diffraction (RHEED) patterns
along a $[1\bar{1}0]$ azimuth, tracking direct epitaxy on the exposed GaSb
region. Top: native oxide desorption at 530 ∘C. Bottom: GaSb film growth on
the exposed GaSb substrate. Bulk reflections are marked by arrows. (b) RHEED
patterns tracking oxide desorption and growth on the graphene-terminated half
of the substrate. (c) Schematic of the heterostructure. (g) X-ray diffraction
pattern of the GaSb film grown on the graphene-terminated side of GaSb (001).
(e-g) GaSb membrane exfoliation using a Ni stressor layer. (e) SEM image of
the under side of the exfoliated GaSb membrane. (f) SEM images of the
substrate after exfoliation. (g) X-ray diffraction patterns of the $004$
reflection for the exfoliated GaSb membrane (blue) and the GaSb substrate
(black).

We first demonstrate the epitaxy and exfoliation of single crystalline GaSb
films on a graphene-terminated GaSb (001) substrate. Fig. 1(a,b) shows
reflection high energy electron diffraction (RHEED) patterns tracking the MBE-
growth of GaSb directly on GaSb (001) and on graphene-terminated GaSb (001).
On this sample, we layer transfer graphene on half of the GaSb (001) substrate
using a wet transfer procedure similar to the one that has previously been
used for remote epitaxy of GaAs, including an HCl etch to remove some of the
native oxides from the substrate Kim et al. (2017) (Methods). The other half
of the sample has an exposed GaSb surface for direct epitaxy.

Immediately prior to growth we follow the standard procedure of annealing at
520 ∘C in an Sb2/Sb1 flux to desorb the native oxides from the substrate. On
the bare GaSb side, the native oxide removal is confirmed by the appearance of
bright semi-streaky spots in the RHEED pattern. The graphene-terminated side
looks qualitatively similar, although the pattern is not as bright and there
is diffuse intensity around the specular reflection. We attribute these
differences in part to scattering through monolayer graphene that is randomly
oriented in-plane.

After 75 nm of GaSb film growth, on both sides of the sample we observe a
distinct ring of spots RHEED pattern with $3\times$ superstructure reflections
corresponding to the expected $c(2\times 6)$ reconstruction of GaSb (001)
[Fig. 1(a,b), bottom]. This indicates that a well-ordered, smooth epitaxial
film has grown on both the bare substrate side and on the graphene-terminated
side. X-ray diffraction measurements confirm epitaxial GaSb growth with no
impurity phases (Fig. 1d). Closer inspection of the $006$ reflection reveals a
subtle shift of the film reflection to lower angle compared to the substrate,
or a larger out-of-plane lattice parameter. We speculate that this shift may
arise from strain induced by the underlying graphene, e.g. due to the large
thermal expansion mismatch between graphene and GaSb. Alternatively, the shift
may be due to slight wrinkling of the GaSb film since it resting on a graphene
interlayer.

Our GaSb films can be mechanically exfoliated from the graphene-terminated
side of the sample, using a Ni stressor layer (Methods). Figs. 1(e,f) show
scanning electron micrographs of the top side of the substrate and the bottom
side of the exfoliated membrane, after exfoliation. The large scale SEM images
are comparable to previous demonstrations of remote epitaxy of other III-V
materials, including GaAs and InP Kong et al. (2018); Kim et al. (2017), and
show minimal spalling marks or tears. X-ray diffraction measurements of the
exfoliated membranes (Fig. 1g) confirm that the membrane itself is single
crystalline.

###  II.2 Holes created by native oxide desorption

We now show that GaSb films grow on graphene-terminated GaSb (001) via a
lateral epitaxy, seeded at pinhole defects. We first examine the creation of
pinhole defects in graphene, which form due to native oxide desorption from
the substrate immediately prior to GaSb film growth.

Fig 2b shows two models of the graphene/GaSb substrate interface, immediately
after the graphene layer transfer but before heating the sample to the film
growth temperature. For an idealized transfer, the graphene/GaSb interface
would be chemically clean and atomically sharp. This allows the lattice
potential of the crystalline substrate to permeate through monolayer graphene
and serve as the template for subsequent film growth in a remote epitaxy mode.
However, such clean interfaces can be difficult to fabricate in practice,
especially for III-V semiconductor substrates like GaSb since III-Vs are
typically terminated with an amorphous $\sim 3$ nm thick native oxide. This
oxide must be removed before film growth in order to reveal the underlying
crystalline template for an epitaxial film. Although we adopt an HCl etching
procedure to remove the substrate native oxides prior to forming the
graphene/substrate interface (Methods), some residual oxides may remain.

Figure 2: Native oxides desorption from the graphene/GaSb interface. (a) In-
situ photoemission spectra of the Sb $3d$ and O $1s$ core levels, tracking the
desorption of native oxides by annealing. (b) Schematics of idealized and real
interfaces, after a chemical etch but before thermal oxide desorption.

Our in-situ photoemission spectroscopy measurements reveal that some
interfacial oxides remain after the etch procedure. Fig. 2(a) shows in-situ
x-ray photoemission spectra (XPS) of the O 1s and Sb 3d core levels for a
graphene-terminated GaSb(001) sample, as a function of annealing. These
measurements are performed in an interconnected MBE + XPS chamber, such that
samples are annealed and measured without removing them from ultrahigh vacuum
($p<5\times 10^{-10}$ Torr). After a 350 ∘C ultrahigh vacuum anneal to clean
surface adsorbates, we observe a significant O 1s intensity at binding energy
of 531 eV. In addition, we observe a satellite to the Sb $3d_{3/2}$ core level
at binding energy of 540 eV, indicative of Sb-oxides.

Further annealing removes most of the oxides, despite the presence of the
graphene overlayer. After annealing at $515^{\circ}$C with an Sb2/Sb1 flux, no
oxide components are detected in the Sb 3d or Ga 3p core levels, suggesting
that the GaSb-oxides have fully desorbed. We also observe a finite O 1s core
level intensity that decreases as a result of the $515^{\circ}$C anneal, and
decreases further after the 540∘C anneal. At this point the chemical state of
the remaining oxygen above $515^{\circ}$C is unclear, since no oxide
components are detected in the Ga 3p, Sb 3d, or C 1s core levels, indicating
that neither the GaSb substrate nor the graphene itself is oxidized
(Supplemental Fig. S-1). This indicates that the remaining oxygen at high
temperature must be weakly coordinated to the graphene or GaSb, yet remain
trapped in the near surface region of the sample. Further experiments are
needed to understand the oxygen that remains after nominal GaSb-oxide
desorption. We find similar results for the native oxide desorption from
graphene-terminated GaAs (001) substrates (Supplemental Fig. S-2). Variations
on the anneal, including annealing in a H2/Ar forming gas, also revealed
residual oxygen and Sb-oxides after light annealing (Supplemental Fig. S-3).

Figure 3: Holes created by native oxide desorption. (a) SEM images and RHEED
patterns tracking the native oxide desorption and creation of large $300$ nm
diameter holes. (b) AFM height images. Pinholes with diameter $\sim 10$ nm
appear at the onset of native oxide desorption, $450^{\circ}$C. (c) AFM phase
contrast images. The phase contrast arises from differences in elastic modulus
between the graphene and the holes with exposed GaSb. (d) Schematic of the
oxide desorption process leading to the creation of holes with different
sizes. (e) AFM height and phase line profiles extracted from the
$450^{\circ}$C sample. The corresponding line cut is marked by the blue line
in the AFM images. (g) Raman spectra at various stages during oxide
desorption. A strong $D$ peak appears coincides with the appearance of large
holes after annealing at $540^{\circ}$C.

Combined RHEED, AFM, SEM and Raman spectroscopy reveal that the interfacial
oxides desorb by creating pinholes in the graphene. Fig. 3 tracks the
evolution of surface morphology during native oxide desorption. The sample
lightly annealed at 350∘C appears qualitatively smooth with no tears or holes
observed in the SEM images (Fig. 3a). AFM topography reveals a bumpy
morphology with height variations of $\sim 2$ nanometers and hints of the
underlying GaSb step and terrace morphology, with terrace length $\sim 100$ nm
(Fig. 3c, left). We attribute the slightly bumpy morphology to the presence of
some interfacial oxides. Raman spectroscopy at this stage shows sharp $G$ and
$2D$ bands with no detectable $D$ peak, which is commonly used as a metric to
indicate minimal defects in the graphene.

After annealing at $450^{\circ}$C, the onset of native oxide desorption,
pinholes with diameter $\sim 10$ nm are observed in the AFM phase and height
images (Fig. 3b,c, middle column and Fig. 3e). The distinct contrast in the
AFM phase suggests a different elastic modulus in the holes (exposed GaSb)
compared to away from the pinholes (graphene). We interpret these holes to be
created by the native oxide desorption process, as illustrated in Fig. 3(d).
These pinholes are difficult to observe at SEM length scales (Fig. 3(a)) and
are not readily detected by Raman spectroscpy since the sample does not have
an observable $D$ peak (Fig. 3f). The RHEED signature of the beginnings of
oxide desorption is a sharpening of the specular reflection (Fig. 3a, middle
insert). With further annealing up to $540^{\circ}$C, the density and diameter
of the pinholes increases as observed in the AFM images. Additionally, we
observe larger $\sim 300$ nm diameter holes in the SEM images. After the
$540^{\circ}$C anneal, a significant $D$ band is present in the Raman
spectrum, which is indicative of graphene defects. We observe similar pinhole
formation from native oxide desorption on graphene-terminated GaAs substrates
(Supplemental Fig. S-2).

###  II.3 Seeded lateral epitaxy in graphene holes

Figure 4: Seeded lateral epitaxy at the graphene pinholes. (a) AFM images
tracking the evolution of GaSb nucleation at pinholes in the graphene ($<2$
monolayers, ML), followed by lateral overgrowth (5 ML) and coalescence into an
atomically stepped continuous film (200 ML). One monolayer is defined as half
the thickness of a unit cell $a/2=3$ Angstrom, corresponding to an areal
density of $5.4\times 10^{14}$ atoms/cm2 each of Ga and Sb. The GaSb growth
temperature was $500^{\circ}$C and the oxide desorption temperatures were
$520-540^{\circ}$C. Inserts show RHEED patterns at each stage of growth. (b)
SEM images showing the nucleation at both small and large holes. The islands
begin to coalesce around 50-100 ML, and full coalescence is observed after 200
ML film growth. (c) Schematic of lateral epitaxy seeded at holes in the
graphene.

These holes serve as nucleation sites for direct epitaxial growth. Fig. 4
tracks the evolution of GaSb film growth on the graphene/GaSb (001) surface
after native oxide desorption. After 0.5 monolayers (ML) of GaSb growth, AFM
and SEM measurements show that the GaSb islands selectively nucleate in the
pinholes ($\sim 10$ nm, Fig. 4a) and in the larger holes (300 nm, Fig. 4b),
rather than on the graphene. We attribute the selective nucleation at pinholes
to the fact that exposed holes to the substrate are more chemically reactive
than the relatively inert graphene. After 2 ML of GaSb growth, nearly all of
the holes have nucleated GaSb islands. After 5 ML growth, the islands begin to
grow laterally (Fig. 4a). At 50-100 ML, the islands begin to coalesce (Fig.
4b). Finally, after 200 ML a smooth GaSb film surface is recovered, with an
atomic step-and-terrace morphology observed by AFM. Our results indicate that
the holes created by native oxide desorption serve as sites for the
nucleation, lateral epitaxy, and coalescence of GaSb films (Fig. 4c).

Figure 5: Exfoliation of GaSb films that are seeded at pinholes. (a) SEM image
of the substrate side after exfoliation, showing $\sim 300$ nm diameter
spalling marks and clusters of spalling marks. The size and spatial
distribution of these spalling marks matches the large holes observed in Fig.
3a. (b) SEM images showing a relatively smooth region of the substrate after
exfoliation. No large 300 nm spalling marks are observed. (c) Cartoon of the
exfoliation. (d) Magnified SEM image of the relatively smooth region.
Particles with diameter $\sim 10$ nm are observed. We attribute these to
nucleation in the $\sim 10$ nm graphene pinholes.

Despite the presence of holes, GaSb films grown by a seeded lateral epitaxy
mode can be exfoliated, as shown in the large scale SEM images of Fig. 1e,f.
To understand this exfoliation further, in Fig. 5 we show higher magnification
SEM images of the substrate after GaSb membrane exfoliation. In Fig 5(a) we
observe elongated clusters of spalling marks with width $\sim 300$ nm and
length ranging from 300 nm to several microns, which have approximately the
same size and areal density as the 300 nm diameter holes observed in Fig.
3(a). We attribute these spalling marks to direct epitaxy in the 300 nm holes,
where the film remains stuck to the GaSb substrate rather than being peeled
away with the film/membrane. Other regions of the substrate appear relatively
smooth at this length scale (Fig. 5(b)), since the original 300 nm holes from
oxide desorption are sparsely distributed across the sample. Zooming into this
relatively smooth region, we observe smaller scale $\sim 10$ nm islands (Fig.
5(d)), which we attribute to the film sticking at the $\sim 10$ nm graphene
pinholes. These pinholes are typically evenly distributed across the sample
after oxide desorption.

##  III Discussion

Our experiments reveal that pinhole defects drive the seeded lateral epitaxy
of GaSb on graphene-terminated GaSb (001). These pinholes are created by
native oxide desorption from the substrate. The resulting GaSb films have
similar characteristics as films grown by remote epitaxy Kim et al. (2017);
Kong et al. (2018); Jiang et al. (2019); Du et al. (2021), namely, epitaxial
alignment to the underlying substrate, smooth atomically stepped surfaces, and
the ability to exfoliate macroscopically smooth films. We understand the
growth in terms of a balance between several atomic scale processes, as
illustrated in Fig. 6. This balance dictates whether the growth is in a seeded
lateral epitaxy regime or a remote epitaxy regime.

Seeded lateral epitaxy occurs in the graphene/GaSb system because (1) graphene
is chemically inert and (2) the surface diffusion length $\lambda$ is long
compared to the average spacing $L$ between pinhole defects ($\lambda>>L$).
Previous experiments on confinement heteroepitaxy demonstrate that the
effective surface diffusion length on graphene is of order 10 microns, since
metal adatoms preferentially nucleate at graphene defects over length scales
of order 10 microns Briggs et al. (2020). Our preliminary experiments on
lithographically defined graphene stripes reveals a similar picture
(Supplemental Fig. S-4). We find that during the early stages of nucleation
for GaAs films on patterned graphene/Ge(001), the GaAs preferentially
nucleates at exposed regions of the Ge substrate, even when the spacing
between graphene openings is 10 microns. This $\lambda\sim 10$ micron
effective diffusion length is much larger than the $L\sim 100$ nm spacing
between pinhole defects on graphene/GaSb(001). Therefore, adatoms can sample
enough of the surface to nucleate at pinhole defects, rather than on the inert
graphene. If the spacing between defects were larger or if the graphene were
defect free ($\lambda<L$), then the system would be closer to a remote epitaxy
regime.

The long effective surface diffusion length arises from a combination of long
surface diffusion and a high adatom desorption rate (Fig. 6a), since the
sticking coefficient of Ga and Sb adatoms on inert masks at $500^{\circ}$C is
less than unity. Similar behavior has been observed for selective area epitaxy
of III-V films on patterned dielectric masks, such as SiO2 and SiNx Nishinaga
et al. (1988).

We compare energy scales associated with different sites using the potential
landscape shown in Fig 6b. On graphene-terminated regions, the combined
potential modulations from the graphene itself and from the substrate that
permeate through graphene are expected to be weak. Density functional theory
calculations suggest these modulations are of order $\Delta\phi\sim 15$ meV
Kong et al. (2018), which is smaller than the thermal energy $k_{B}T\sim 70$
meV at growth temperature $500^{\circ}$C. An estimate of the free carrier
screening from graphene also suggests this potential should be weak. For a
free electron gas with density $10^{19}-10^{20}$ cm-3, the same density as
graphene (graphite) Klein and Straub (1961), the Thomas-Fermi screening length
is approximately $1/k_{0}\sim 2$ Å. This is less than the film/substrate layer
spacing of $\sim 5$ ÅKim et al. (2017), suggesting that the screened potential
above the graphene is exponentially weak. In contrast, the potential
modulation at a graphene pinhole is expected to be much larger,
$\Delta\phi\sim 1$ eV, since this involves adatoms making chemical bonds to
the graphene or to the underlying substrate. The pinholes provide the low
energy sites for nucleation and growth. Epitaxial registry is obtained because
the pinholes are locations of direct epitaxy to the substrate. Given these
factors that favor seeded lateral epitaxy when there is a finite density of
graphene defects, it is possible that some previous demonstrations of remote
epitaxy can alternatively be explained by a seeded lateral epitaxy mechanism.

Figure 6: Seeded lateral epitaxy. (a) Pinholes are created by the native oxide
desorption, which then serve as nucleation sites for epitaxial growth. (b)
Schematic potential $\phi$ for adatoms diffusing on the graphene-terminated
surface, with diffusion length $\lambda$. Seeded lateral epitaxy is expected
if $\lambda>>L$.

Seeded lateral epitaxy retains many of the most promising features of remote
epitaxy. Since these pinholes are small, the films can still be exfoliated.
Additionally, because the majority of the film/substrate interface is still
decoupled by the monolayer graphene barrier, we expect similar slippery strain
relaxation pathways to apply for highly mismatched growth Bae et al. (2020).

The seeded lateral epitaxy mode has two promising additional features.
Firstly, seeded lateral epitaxy is tolerant to an imperfect graphene/substrate
interface. Despite the oxides present at the graphene/GaSb substrate interface
after graphene transfer, epitaxial growth can still be achieved by desorbing
the native oxide to create pinholes. This tolerance is crucial for translating
epitaxy on graphene-terminated surfaces to wafer scale, since it is difficult
to perform native oxide-free graphene layer transfers at wafer scale and
difficult to grow graphene directly on arbitrary substrates. Secondly, the
controlled patterning of nanoscale openings in graphene provides a route to
precisely engineer the size, location, and spatial distribution of nucleation
sites (Supplemental Fig. S-4), and resulting properties of the films. For
example, controlled openings may provide selective anchoring points to control
the properties of films that undergo solid-solid phase transitions, e.g.
ferroelectric materials and shape memory alloys.

##  IV Methods

Graphene synthesis and transfer. Our graphene synthesis and transfer procedure
is a modified polymer-assisted wet transfer of CVD-grown graphene Park et al.
(2018), similar to the transfer recipe from previous demonstrations of remote
epitaxy Kim et al. (2017); Kong et al. (2018). Graphene was grown by thermal
chemical vapor deposition (CVD) of ultra high purity CH4 at 1050 $\degree$C on
Cu foil, as described in Refs. Du et al. (2021); Jacobberger and Arnold
(2013). The graphene/Cu foils were then cut and and flattened using clean
glass slides to match the dimensions of the semiconductor substrate.
Approximately 200 nm of 950K C2 PMMA (Chlorobenzene base solvent, 2% by wt.,
Kayaku Advanced Materials, Inc.) was spin coated on the graphene/Cu foil at
2000 RPM for 2 minutes and left to cure at room temperature for 24 hours.
Graphene on the backside of the Cu foil was removed via reactive ion etching
using 90 W O2 plasma at a pressure of 100 mTorr for 30 s. The Cu foil was then
etched by placing the PMMA/graphene/Cu foil stack on the surface of an etch
solution containing 1-part ammonium persulfate (APS-100, Transene) and 3-parts
H2O. After 16 hours of etching at room temperature, the floating PMMA/graphene
membrane was scooped up with a clean glass slide and sequentially transferred
onto five 5-minute water baths to rinse any etch residuals.

GaSb (001) substrates were etched in 38% HCl solution for 5 minutes followed
by a 2-propanol rinse to remove some of the native oxides. We then used the
etched GaSb substrate to scoop the PMMA/graphene stack from the final water
bath. To remove water at the graphene/substrate interface, the samples were
baked in air at 50°C for 15 minutes, then at 150°C for another 15 minutes. The
PMMA was dissolved by submerging the sample in an acetone bath at 80°C for 3
hours. This is followed by an IPA rinse and nitrogen drying. The sample is
indium bonded onto a molybdenum puck and outgassed at 150°C for 2 hours in a
loadlock at a pressure $p<$ $5\times{10^{-7}}$ Torr before introduction to the
MBE growth chamber. Finally, the sample is annealed at $\sim 350^{\circ}$C for
1 hour in the MBE chamber to desorb organic residuals.

MBE growth of GaSb. GaSb films were grown in a custom molecular beam epitaxy
chamber (Mantis Deposition) using an effusion cell for Ga and a thermal
cracker cell for Sb (MBE Komponenten). Temperatures were measured using a
pyrometer that is calibrated to the native oxide desorption temperature of
GaAs. In-situ reflection high energy electron diffraction (RHEED) measurements
were performed using a Staib RHEED gun at 15 keV.

In-situ XPS. X-ray photoemission spectroscopy (XPS) measurements were
performed in an XPS chamber that is connected to the MBE via an ultrahigh
vacuum ($p<5\times 10^{-10}$ Torr) transfer chamber, such that samples are
transferred in vacuum and are not exposed to air. We use a non-monochromated
Al $K_{\alpha}$ x-ray source (1486.6 eV) and an Omicron EA125 hemispherical
analyzer with an energy resolution of 1.08eV. Samples were annealed in the MBE
and measured in the XPS at room temperature.

AFM, SEM, and Raman spectroscopy. Field emission scanning electron microscopy
(SEM) was performed using a Zeiss GeminiSEM 450. Raman spectroscopy was
performed using a 532 nm wavelength laser (Thermo Scientific DXR Raman
Microscope). The laser power is kept below 5 mW in order to prevent damage to
the graphene. In order to acquire representative raman spectra, 2600 spectra
where captured and averaged over a 2500 $\mu m^{2}$ area. Atomic force
microscopy (AFM) measurements were performed using a Bruker Dimension Icon in
tapping mode.

GaSb exfoliation. To exfoliate the GaSb epilayer, a 100 nm Ni stressor layer
was deposited on the GaSb / graphene / GaSb (001) heterostructure at room
temperature using a metal evaporator (Angstrom Engineering Inc.) in high-
vacuum ($p<2\times 10^{-6}$ Torr). Thermal release tape (Revalpha 3195H,
Semiconductor Equipment Corp.) was then stamped onto the sample, and the film
can then be readily exfoliated by carefully peeling away the tape from the
substrate.

##  V Acknowledgments

GaSb film growth was supported primarily by the National Science Foundation
(DMR-1752797). Graphene transfers and III-V growth on patterned graphene were
supported by the Defense Advanced Research Projects Administration (DARPA
Grant number D19AP00088). Graphene synthesis and characterization are
supported by the U.S. Department of Energy, Office of Science, Basic Energy
Sciences, under award no. DE-SC0016007. We gratefully acknowledge the use of
Raman and electron microscopy facilities supported by the NSF through the
University of Wisconsin Materials Research Science and Engineering Center
under Grant No. DMR-1720415.

## References

  * Bae et al. (2020) S.-H. Bae, K. Lu, Y. Han, S. Kim, K. Qiao, C. Choi, Y. Nie, H. Kim, H. S. Kum, P. Chen, et al., Nature nanotechnology 15, 272 (2020).
  * Jiang et al. (2019) J. Jiang, X. Sun, X. Chen, B. Wang, Z. Chen, Y. Hu, Y. Guo, L. Zhang, Y. Ma, L. Gao, et al., Nature communications 10, 1 (2019).
  * Kim et al. (2017) Y. Kim, S. S. Cruz, K. Lee, B. O. Alawode, C. Choi, Y. Song, J. M. Johnson, C. Heidelberger, W. Kong, S. Choi, et al., Nature 544, 340 (2017).
  * Guo et al. (2019) Y. Guo, X. Sun, J. Jiang, B. Wang, X. Chen, X. Yin, W. Qi, L. Gao, L. Zhang, Z. Lu, et al., Nano letters 20, 33 (2019).
  * Du et al. (2021) D. Du, S. Manzo, C. Zhang, V. Saraswat, K. T. Genser, K. M. Rabe, P. M. Voyles, M. S. Arnold, and J. K. Kawasaki, Nature Communications 12, 1 (2021).
  * Kong et al. (2018) W. Kong, H. Li, K. Qiao, Y. Kim, K. Lee, Y. Nie, D. Lee, T. Osadchy, R. J. Molnar, D. K. Gaskill, et al., Nature materials 17, 999 (2018).
  * Kum et al. (2020) H. S. Kum, H. Lee, S. Kim, S. Lindemann, W. Kong, K. Qiao, P. Chen, J. Irwin, J. H. Lee, S. Xie, et al., Nature 578, 75 (2020).
  * Lu et al. (2018) Z. Lu, X. Sun, W. Xie, A. Littlejohn, G.-C. Wang, S. Zhang, M. A. Washington, and T.-M. Lu, Nanotechnology 29, 445702 (2018).
  * Briggs et al. (2020) N. Briggs, B. Bersch, Y. Wang, J. Jiang, R. J. Koch, N. Nayir, K. Wang, M. Kolmer, W. Ko, A. D. L. F. Duran, et al., Nature materials 19, 637 (2020).
  * Nishinaga et al. (1988) T. Nishinaga, T. Nakano, and S. Zhang, Japanese journal of applied physics 27, L964 (1988).
  * Klein and Straub (1961) C. A. Klein and W. D. Straub, Physical Review 123, 1581 (1961).
  * Park et al. (2018) H. Park, C. Lim, C.-J. Lee, J. Kang, J. Kim, M. Choi, and H. Park, Nanotechnology 29, 415303 (2018).
  * Jacobberger and Arnold (2013) R. M. Jacobberger and M. S. Arnold, Chemistry of Materials 25, 871 (2013).

Figure S-1: Extended in-situ photoemission spectra of graphene on GaSb (001)
tracking the native oxide desorption as a function of annealing. (a) Sb 3d and
O 1s core levels. The Sb-oxide intensity goes to zero after the $515^{\circ}$C
anneal, but residual atomic O 1s remains. (b) C 1s. After the $515^{\circ}$C
anneal we observe a shift in the C 1s core level towards lower binding energy,
which we attribute to de-oxidation. (c) Ga 3p. Ga-oxide components are not
resolved in these spectra. Figure S-2: Native oxide desorption from a
graphene/GaAs (001) sample. The CVD-grown graphene was wet transferred to GaAs
(001) after an HCl etch to remove some of the GaAs native oxides. (a)
Phtoemission spectra of the As 3p core level. Arsenic (As) oxides are detected
after the 150 C anneal (red curve) and are desorbed after the 625 C anneal
(black curve). (b) There is a corresponding change in the C 1s lineshape. (c)
Ga 3p core level. (d) O 1s core level. (e) SEM image of the graphene/GaAs
(001) surface after the 625C anneal, showing pinholes. (f) Cartoon of the
pinholes created by native oxide desorption. Figure S-3: Forming gas annealing
of graphene/GaSb(001). Photoemission spectra of forming gas annealed
graphene/GaSb (001) (green curve) compared to vacuum annealed graphene/GaSb
(001) (red curve). Both spectra show significant Sb-oxides and O 1s. The
forming gas annealed sample was annealed in a tube furnace at $350^{\circ}$C
for two hours and then transferred through air to the XPS system. The vacuum
annealed sample was annealed in ultrahigh vacuum in the XPS chamber, and thus
did not get exposed to air after the anneal. Figure S-4: GaAs selectivity on
patterned graphene/Ge. The graphene was grown directly on the Ge (001) by CVD,
thus avoiding the damage and interfacial oxides typically present for graphene
layer transfers. We use photolithography and an oxygen plasma etch to define
10 micron wide stripes of alternating graphene and exposed Ge substrate. GaAs
films were then grown on the patterned graphene substrate by MBE. (a) SEM
image of GaAs nucleation on patterned graphene on a Ge (001) substrate. We
observe selective nucleation of GaAs (light color) on the exposed Ge regions
of the substrate, and no GaAs nucleation on the graphene. (b) Cartoon of the
sample. (c) Raman spectra on the graphene masked region (black curves) and on
the GaAs nucleation region (blue). Representative areas for these spectra are
marked by the circles in the SEM image (a).