# Tiny Deep Ensemble: Uncertainty Estimation in Edge AI Accelerators via
Ensembling Normalization Layers with Shared Weights

Soyed Tuhin Ahmed, Michael Hefenbrock, Mehdi B. Tahoori
Karlsruhe Institute of Technology, Revoai GmbH, Germany
<EMAIL_ADDRESS>
###### Abstract

The applications of artificial intelligence (AI) are rapidly evolving, and
they are also commonly used in safety-critical domains, such as autonomous
driving and medical diagnosis, where functional safety is paramount. In AI-
driven systems, uncertainty estimation allows the user to avoid overconfidence
predictions and achieve functional safety. Therefore, the robustness and
reliability of model predictions can be improved. However, conventional
uncertainty estimation methods, such as the deep ensemble method, impose high
computation and accordingly hardware (latency and energy) overhead because
they require the storage and processing of multiple models. Alternatively,
Monte Carlo dropout (MC-dropout) methods, although having low memory overhead,
necessitate numerous ($\sim 100$) forward passes, leading to high
computational overhead and latency. Thus, these approaches are not suitable
for battery-powered edge devices with limited computing and memory resources.
In this paper, we propose the Tiny-Deep Ensemble approach, a low-cost approach
for uncertainty estimation on edge devices. In our approach, only
normalization layers are ensembled $M$ times, with all ensemble members
sharing common weights and biases, leading to a significant decrease in
storage requirements and latency. Moreover, our approach requires only one
forward pass in a hardware architecture that allows batch processing for
inference and uncertainty estimation. Furthermore, it has approximately the
same memory overhead compared to a single model. Therefore, latency and memory
overhead are reduced by a factor of up to $\sim M\times$. Nevertheless, our
method does not compromise accuracy, with an increase in inference accuracy of
up to $\sim 1\%$ and a reduction in RMSE of $17.17\%$ in various benchmark
datasets, tasks, and state-of-the-art architectures.

###### Index Terms:

Deep Ensemble, BatchEnsemble, TinyML, Uncertainty Estimation, MC-Dropout

##  I Introduction

Recent advances in deep learning models, such as neural networks (NNs), have
shown superior performance in various domains [1]. Consequently, they are
widely adopted in different sectors, including critical ones such as
automotive, health care, and industrial control. However, training and
inference of modern NN models require a tremendous amount of computational
power and memory. Therefore, they are suitable for the cloud computing
paradigm due to their “unlimited” storage capacity and computing resources
[2], but they are challenging for edge AI accelerators. Edge AI acceleration
provides privacy and real-time processing, but they have limited computational
and memory resources. Numerous industries may expect significant
transformations due to AI-powered edge computing [3] and their the market is
estimated to be worth $\$3.5$ billion by 2027 [4].

Numerous contributions have been made in the field of TinyML [5], where the
emphasis is on the running of NNs on hardware with extremely low power,
memory, and computational resources while still maintaining reasonable
accuracy. Nevertheless, research on predictive uncertainty estimation with
tiny NN models is lacking.

Uncertainty estimation in prediction is crucial in safety-critical
applications where NNs operate on real-time data, e.g., from sensory inputs.
During NN deployment, the underlying data distribution may shift or the data
may become corrupted due to sensor noise [6]. To address this, predictive
uncertainty can supplement model predictions and enable informed decision-
making. As a result, unreliable predictions can be prevented from reaching the
end user and reviewed by a human expert.

Figure 1: a) Deep Ensemble [7] with $M$ ensemble members , b) BatchEnsemble
[8], proposed Tiny-DE model with $M$ normalization layers with _a single
shared convolutional layer_ in c) serial mode, and d) parallel mode.

Among the numerous uncertainty estimation methods [9], the Deep Ensemble [7]
is considered a “gold standard” for uncertainty estimation [10]. In the Deep
Ensemble, $M$ ensemble members $1,\cdots,M$ are trained independently and
stored in hardware. During inference in edge AI accelerators, the input is
processed by each model in $M$ forward passes (see Fig. 1 (a)). Subsequently,
the outputs of all models are combined to obtain the predictive distribution.
Therefore, the cost in terms of latency, power, and memory for training,
storage, and processing of $M$ ensemble members is challenging for edge AI
accelerators.

To reduce computational and memory overhead in ensemble methods, several
studies exist. Monte Carlo dropout (MC-dropout) can be interpreted as
”implicit” ensembles that can create an exponential number of weight-sharing
sub-networks for uncertainty estimates [11]. Although MC-dropout requires
training and storage of a single model, inference involves $M$ forward passes
through a dropout-enabled network. Here, $M$ varies with tasks, and the
topology can be as large as $94$ even on a small (six-layer) fully
convolutional network [12]. Furthermore, MC-dropout has sampling latency and
chip area overhead for the dropout module implementation [13]. To reduce
inference latency, the work [14] proposed to ensemble only deeper
convolutional layers while the shared backbone is computed only once and
cached. However, in convolutional NNs (CNNs), deeper convolutional layers have
significantly larger parameter counts than other layers, as shown in Fig 2
(a). Also, this approach only works if dropout is applied only to deeper
convolutional layers rather than to all layers. Another domain-specific group
of work, binarized the MC-Dropout to reduce memory overhead by $32\times$,
improve latency by accelerating them in Spintronics-based computation-in-
memory (CIM) architecture, and reduce sampling latency by reducing the total
number of Dropout modules [13, 15, 16, 17]. In contrast, the BatchEnsemble [8]
approach also shares weights but introduces two sets of $M$ rank-1 matrices to
generate $M$ ensemble members. Their approach is not scalable to the AI
accelerator architecture that does not allow batch processing. Additionally,
it introduces additional computation at the input and output of a layer, as
shown in Fig. 1 (b).

Figure 2: a) Number of parameters in each layer and b) Share of parameter
groups with respect to the total number of parameters in ResNet-32.

We observed that parameters other than weights and biases in a NN consume only
$\sim 1\%$ of all parameters, as shown in Fig. 2 (b). Therefore, we propose to
ensemble only normalization layers with shared weight and biases. The
normalization layer is commonly used in NNs, as it speeds up the training and
improves performance [18]. Our approach is scalable 1) in any AI accelerator
architecture, 2) in any NN topologies, such as CNN and recurrent neural
network (RNN), 3) in tasks, and 4) in datasets. Furthermore, our approach is
parallelizable during training and inference within an AI accelerator
architecture. Consequently, all ensemble members can be updated concurrently
for a given mini-batch, and inference requires a single forward pass, allowing
for _single-shot training and inference_.

Our contributions can be summarized as follows:

  * •

Ensembling normalization layers with shared weights and biases for low-cost
uncertainty estimation, tailored for edge AI accelerators.

  * •

Tiny-DE network topology that is scalable to existing NN topologies, AI
accelerator architectures, and NN tasks.

  * •

Single-Shot training and inference in hardware architecture that allows batch
processing.

  * •

EnsembleNorm layer for normalizing all ensemble members in a single shot.

  * •

Substantial reduction in computational and storage requirements without
sacrificing accuracy and quality of uncertainty estimates, as evidenced by
extensive empirical evaluation.

The rest of the paper is organized as follows: Section II reviews the related
work, Section III details the proposed methodology, Section IV provides
experimental results, and Section V concludes the paper.

##  II Preliminary

###  II-A Uncertainty In Deep Learning

In deep learning, uncertainty estimation is crucial for evaluating the
reliability and robustness of model predictions. It offers vital information
about confidence in these predictions. This is especially crucial in
supporting decision-making in safety-critical applications, such as autonomous
driving and automatic medical diagnostics.

Deep learning models are usually deployed in dynamic and uncertain
environments where the distribution of inference data can change over time.
Therefore, the model can receive input data that is unseen during training and
its distribution is completely different. For example, a model trained on the
MNIST (handwritten digit recognition) dataset can receive corrupted data
during inference due to sensor noise or domain shift. Such data are referred
to as out-of-distribution (OoD) data or sometimes called out-of-training-
distribution data points [19, 20]. Uncertainty in prediction arises primarily
due to the tendency of the model to give overconfident predictions for unknown
data. For example, in a classification task, the model will predict that the
unseen OoD data belong to one of the classes with close to $100\%$ confidence
[13]. In such scenarios, quantifying the uncertainty in the prediction allows
the user to make informed decisions and avoid catastrophic failures.

A reliable uncertainty estimation method should demonstrate low uncertainty in
data similar to what it has been trained on, in distribution (ID) data, and
high uncertainty on unseen or OoD data. In a fine-grained method, an incorrect
prediction should show high uncertainty and a correct prediction should show
low uncertainty.

Note that there is a difference between generalizing on the same data, i.e.,
inference accuracy, and OoD data. Inference accuracy refers to prediction
accuracy with data that have the same distribution as training data but are
unseen during training, e.g., validation data. An ideal uncertainty estimation
method, during inference, is expected to generalize well on the same data
distribution and provide interpretable uncertainty estimates on OoD data.

###  II-B Normalization Approaches

In modern-deep learning topologies, normalization layers are essential to
improve training stability, speed, convergence, and performance [18]. In
general, normalization layers standardize its input $\bm{y}$, as follows:

$\bar{\bm{y}}=\frac{\bm{y}-\bm{\mu}}{\sqrt{\bm{\sigma^{2}}+\epsilon}}\times\bm{\gamma}+\bm{\beta}.$
(1)

where, the mean $\bm{\mu}$ and standard deviation $\bm{\sigma}$ are calculated
across a specific dimension (batch, feature map, channel groups) depending on
the type of normalization method. For instance, batch normalization (BN) [18]
normalizes activations across a mini-batch, layer normalization (LN) [21]
normalizes across all features of a single example, Instance Normalization
(IN) [22] normalizes independently within each channel of a single example,
and Group Normalization (GN) [23] normalizes across groups of channels.
Furthermore, $\bm{\gamma}$ and $\bm{\beta}$ are learnable parameters and
$\epsilon$ is a small constant for numerical stability.

###  II-C Model Ensemble and Related works

As stated earlier, in the literature, several methods for uncertainty
estimates are proposed. Among them, the model ensemble method is highly
successful due to its high inference accuracy and quality uncertainty
estimates.

Model ensemble involves combining predictions from multiple individual models
(see Fig. 1(a)) to improve overall performance and estimate uncertainty.
During training, $M$ models are trained independently or collaboratively using
techniques such as bagging or boosting. These models can be trained with
different architectures, initializations, or subsets of data to encourage
diversity. During inference, predictions from different models are aggregated
using methods such as averaging or weighted averaging to obtain the final
prediction. Since training, storage, and processing of $M$ full models is
required, the hardware cost, e.g., memory, latency, and power is a concern.

In Section I, related studies on model ensembles for uncertainty estimates and
related works for cost reduction were discussed. Nevertheless, the ensemble of
models has been extensively studied to improve model performance [24, 25, 26].
Even in this case, there are several methods to reduce the cost of inference.
For example, the work in [27] proposed a model compression technique to
compress large and complex models into smaller and faster ones. Similarly,
[28] introduced the knowledge distillation method, which distills model
ensembles into a single neural network.

Since ensembles require training $M$ models, several studies aim to reduce
their cost at training time. For example, [29] proposed the Snapshot ensemble
method, which encourages a single model to visit multiple local minima by
training it using cyclic learning rates [30]. This method encourages the
exploration of numerous local minima, which are then used as ensemble members.

In contrast, our approach aims to optimize performance, training, and
inference costs collectively with AI accelerator architectures in mind.

##  III Tiny Deep Ensemble (Tiny-DE)

As mentioned previously, a naive ensemble approach incurs significant memory
and computational overhead. Here, the proposed Tiny Deep Ensemble approach is
discussed, a low-cost ensemble method for uncertainty estimation in deep
neural networks.

###  III-A Core Idea

In Tiny-DE, only the normalization layers are ensembled, which overall have
the smallest amount of parameters in the network, differ between the ensemble
members, while all other weights are shared. We denote the normalization
layers of layer index $l$ by $N_{0}^{l},N_{1}^{l},\dots,N_{M-1}^{l}$ in the
following. The normalization layers can be Batch Normalization, Layer
Normalization, Instance Normalization, and Group Normalization with learnable
parameters $\beta\in\mathbb{R}^{n}$ and $\gamma\in\mathbb{R}^{n}$. Therefore,
compared to the deep ensemble approach [7] and BatchEnsemble [8], our approach
requires a $M\times$ lower weight matrix storage and a $2M\times$ lower rank-1
matrix computation (see Figs.1 (a) and (b)).

###  III-B Operation Modes

Depending on the batch processing capabilities of the hardware architecture,
Tiny-DE can operate in either sequential or parallel modes. In hardware
architectures where batch processing is challenging, such as the memristor-
based computation-in-memory (CIM) architecture [31, 32, 33], a sequential
processing NN architecture should be used. Here, ”sequential” refers to
sequential in time rather than signal flow through the ensembles. In contrast,
in parallel mode, _single-shot uncertainty estimation_ can be done using
vectorization in hardware architectures such as edge tensor processing units
(TPUs), field-programmable gate arrays (FPGAs), and graphics processing units
(GPUs) [34, 35]. Both methods are described in detail in the following.

####  III-B1 Sequential Inference

The sequential inference of Tiny-DE utilizes a counter variable $c$ and router
to dynamically select a normalization layer for each forward pass. Depending
on the state of the counter $c$, the output of the $l$-th layer $y^{l}$ is
directed through one of the $M$ normalization layers, as shown in Fig. 1 c).
The activation function such as the ReLU function is applied to the processed
output as is normally done.

The counter $c$ is an unsigned integer and it is updated cyclically in each
layer as follows:

$c\leftarrow(c+1)\mod M,$ (2)

where $c$ is initialized to $0$ at the start of the inference process. The
mechanism ensures that the output of each layer sequentially passes through
each normalization layer in a cyclic order. For example, if $c=0$, the output
$y^{l}$ is processed by $N_{0}$. In the next forward pass, $c$ becomes $1$,
routing the output $y^{l}$ through $N_{1}$, and this process is repeated until
the $M$-th forward pass. After that, $c$ resets to $0$. Note that, due to the
global signal routing and synchronization challenge, the counter variable is
updated locally in each layer.

This cyclic routing mechanism allows each input of the NN to experience every
normalization setting, providing diverse internal-state manipulations within a
single inference cycle, which is crucial for enhancing the ensemble’s ability
to generalize and generate output distribution for uncertainty estimation.

Furthermore, the proposed Tiny-DE can be generalized to all existing NN
architectures by making minor modifications, as shown in Fig. 3. For popular
architectures, such as ResNet and VGG, a router can be inserted after the
convolutional layer.

Figure 3: Sketch of proposed Tiny-DE architecture based on popular CNN
architectures ResNet [36] and VGG [37]. We only show the four signature layers
of a specific topology. Our proposed topology is generalizable across existing
topologies, with only the addition of a router before the normalization
layers. In the case of our proposed approach in batch mode, no change is
required in the topology.

##### Router Implementation

In CIM architectures, the router can be implemented digitally at the periphery
using a demultiplexer (DeMux). The DeMux takes the $Q$-bit unsigned counter
$c$ as the control signal, allowing for up to $2^{Q}$ possible routing paths,
each corresponding to one of the normalization layers (ensemble members).
Since a typical DeMux expects a bit-wise control signal, the DeMux for our
purpose is designed to interpret the control signal $c$ in binary
representation. This can be expressed as:

$\operatorname{binary}(c)=b_{Q-1}b_{Q-2}\cdots b_{0},$ (3)

where $b_{Q-1}$ to $b_{0}$ are the bits of the binary sequence representing
$c$.

Our approach requires only changes to the CiM periphery since the router is
implemented in the digital domain with some logic hardware. Specifically, the
Multiply-Accumulate (MAC) operation of a layer is computed in a memristor-
based crossbar structure (analog domain) and the result is digitized by an
analog-to-digital converter (ADC) operation. Following that the router selects
the parameter for normalization and the normalization is performed. In the
following, non-linear activation is performed and a digital-to-analog (DAC)
converts the results of the activation function for MAC operation (in the
analog domain) of the subsequent layer. The overall algorithm for our proposed
approach in sequential inference mode is depicted in Algorithm 1.

Algorithm 1 Sequential inference mode of Tiny-DE in CiM

1:Input: Controller $c$, number of ensembles $M$, input to the network
$\bm{x}$, number of layers $L$

2:for $m=1,\dots,M$ do $\triangleright$ sequential inference

3: for $l=1,\dots,L$ do $\triangleright$ single forward pass

4: Digital-to-analog conversion

5: MAC operation in memristor-based crossbar array

6: Analog-to-digital conversion

7: Router selects parameters of normalization layer

8: Perform normalization

9: Non-linear activation

10: end for

11: Increment counter

12:end for

####  III-B2 Single-Shot Uncertainty Estimation

By manipulating the computations for a mini-batch, the computations of the
Tiny-DE approach are parallelizable within a hardware architecture that allows
batched processing such as FPGAs, GPUs, and TPUs. Therefore, only a _single
forward pass_ with respect to multiple ensemble members in parallel is
required to estimate uncertainty. Here, an input to the convolution or linear
layer is repeated $M$ times to generate a mini-batch of size $M$ to obtain the
batched output $\bm{Y}^{l}$. However, if the batch size of the inference
inputs is more than one, e.g., $B$, by repeating the input similarly $M$
times, an effective batch size of $M\cdot B$ can be created. Therefore, a
single forward pass is required for the convolution or linear layer.

However, to still allow a single forward pass through all ensemble members, we
propose _EnsembleNorm_. In EnsembleNorm, the input dimension and the
parameters are modified across the batch dimension so that they independently
apply normalization to each input of the batch. Specifically, the input of the
shape $[M\cdot B,C,H,W]$ is reshaped as $[M,B,C,H,W]$. Here, $C$, $H$, and $W$
represent the channel, height, and width, respectively. Similarly, the
learnable parameters expanded to $\beta\in\mathbb{R}^{M\times n}$ and
$\gamma\in\mathbb{R}^{M\times n}$. That means that the parameters are not only
channel-specific, but also unique to each ensemble member. The mean and
variance are also calculated in the respective dimensions. That means that
each ensemble member can have its own specific mean $\bm{\mu}_{m,c}$ and
variance $\bm{\sigma}^{2}_{m,c}$. Furthermore, each ensemble member can be
scaled and shifted by its own unique parameters, $\bm{\gamma}_{m,c}$ and
$\bm{\beta}_{m,c}$.

Subsequently, the normalized output $\bar{\bm{Y}}^{l}$ is reshaped again to
$[M\cdot B,C,H,W]$ before applying the non-linear activation function. The
PyTorch implementation of EnsembleNorm with other implementations will appear
in 111will be open-sourced upon acceptance.

Consequently, all ensemble members can compute the output in a single forward
pass, eliminating the need to calculate the output of each ensemble member
sequentially. Therefore, the computational latency is reduced to a minimum.

###  III-C Training

The training procedure of Tiny-DE also depends on the operating mode. The
sequential mode involves two main phases, but the parallel mode allows single-
shot training. Both methods are described in detail in the following.

##### Sequential Mode

As stated earlier, the overall training of the $M$ ensembles requires two main
phases. Initially, the full model is trained with all parameters (weights and
biases) being updated. After this, the parameters of the model, e.g., weights
and biases are frozen, and the normalization layers are re-initialized. Here,
”frozen” means that they are not updated using backpropagation. In each
subsequent training, only the normalization layers are updated. The training
is stopped once a comparable accuracy to the full model is achieved. All
trained parameters of the normalization layer are accumulated in a list to
allow for ensemble learning as described earlier.

Since the full model is only trained once, the training overhead and
complexity are significantly lower compared to [7] and [8], respectively. The
decoupling of parameters allows for effective ensemble learning without the
overhead of training multiple distinct models from scratch. In addition, it
allows one to _obtain $M$ ensemble members from a single pre-trained model_.

##### Single-Shot Training

In the batched processing mode, replacing the normalization layer with the
proposed EnsembleNorm layers along with manipulating the dimension as
discussed earlier section, all the ensemble members can be trained together.

Here, the effective batch size for training may need to be reduced due to the
memory overflow issue in GPUs. However, since training is typically done in
the cloud, it is not an issue for edge inference.

###  III-D Prediction and Uncertainty Estimation

The input for inference is forward-passed through the Tiny-DE to get the
predictive distribution. The final prediction of Tiny-DE is obtained from the
average predictions of all ensemble members.

To obtain uncertainty in the prediction, we explore different methods
depending on the task. For classification tasks, the predictive entropy is
commonly used, but we also measure the maximum disagreement among the outputs,
as shown in Algorithm 2.

Algorithm 2 Maximum Disagreement

1:Input: output samples of $\bm{y}$ of shape $(M,B,K)$

2:Initialize Max Disagreement (MD) with zeros of shape $(B,K)$

3:for $m=1,\dots,M-1$ do

4: for $m^{\prime}=m+1,\dots M$ do

5: Calculate absolute difference $m^{\prime}$ and $m$ output

6: Calculate the maximum across the class dimension

7: Update Max Disagreement

8: end for

9:end for

The Maximum Disagreement metric quantifies uncertainty by calculating the
maximum absolute difference in output distributions for each class, across all
models in the ensemble. Since it is computed directly from SoftMax output,
this metric ranges from 0 to 1. A low maximum disagreement value (closer to 0)
indicates low uncertainty, and a high value (closer to 1) indicates high
uncertainty.

Furthermore, in semantic segmentation and time series prediction tasks,
uncertainty is quantified by the variance in predictions of different ensemble
members. Lastly, for regression tasks, the uncertainty is estimated using the
negative log-likelihood (NLL) of the prediction.

###  III-E Diversity Improvement Among Ensemble Members

Diverse predictions among ensemble members are advantageous as they offer
complementary perspectives, potentially improving performance and enhancing
uncertainty estimates. For our approach, diversity can be improved by a) using
different kinds of normalization layers in each member, b) training each
ensemble member with different data augmentations, and c) creating multiple
bootstrap samples (random samples with replacement) from the training data and
training each ensemble member on each sample.

##  IV Results

###  IV-A Experimental Setup

To show scalability on deep learning tasks, we have evaluated our method on
four different tasks: image classification, regression, autoregressive time
series forecast, and semantic segmentation. To further show scalability on
datasets and NN topologies, we have evaluated each task on several state-of-
the-art (SOTA) NN topologies (including CNN and RNN) and datasets.

For image classification, we used the CIFAR-10 and CIFAR-100 benchmark
datasets on the VGG-19, ResNet-56, ShuffleNet-V2, RepVGG-A1, and TinyML
compatible MobileNet-V2 CNN topologies. Furthermore, for the regression task,
we have used 10 UCI datasets with a topology and setting as [11].
Specifically, each dataset except for the protein and Year Prediction MSD, is
split into 20 train-test folds. Five train-test splits were used for the
protein dataset, and a single train-test split was used for the Year
Prediction MSD dataset. The NN has 2-hidden layers with ReLU6 nonlinearity
followed by a 1D batch normalization layer. The number of neurons is 50 for
the smaller datasets and 100 for the larger protein and Year Prediction MSD
datasets, making the network compatible with edge AI accelerators. All the
dataset was trained for 40 epochs and we have used 5 ensemble members (M=5).

On the other hand, for the time-series forecast, an NN with an LSTM layer and
a classifier layer was used for the Mauna Loa CO2 concentrations dataset.
Lastly, for Semantic segmentation tasks, we have considered binary as well as
multi-class segmentation datasets and two safety-critical scenarios, for
biomedical and automotive. For biomedical image segmentation, we have used the
Kvasir-SEG [38] dataset which contains medically obtained gastrointestinal
polyps images on the Feature Pyramid Network (FPN) [39]. For automotive scene
understanding we used the CamVid [40] dataset which consists of road scene
images and involves segmenting each pixel into one of the 12 classes on the
UNet++ topology [41]. We have further evaluated the generalized scene
understanding task with the Pascal VOC dataset with the fully convolutional
network (FCN). The encoder network for each topology is shown in brackets in
Table. III.

Note that the semantic segmentation task is known to be more challenging than
other tasks due to its finer granularity. That is, it involves segmenting an
image into multiple sections and assigning each pixel with its corresponding
class label.

The performance of the classification task is evaluated on inference accuracy,
time series, and regression on root-mean-square-error (RMSE), and semantic
segmentation on pixel accuracy and mean intersection-over-union (mIoU)
metrics.

In terms of uncertainty estimation, classification tasks are evaluated on data
distribution shift and out-of-domain data as OoD data. Specifically, for data
distribution shift, images are corrupted by $90^{\circ}$ rotation and Gaussian
noise, a subset of the CIFAR-C dataset [6]. Furthermore, SVHN (Street View
House Numbers) and STL-10 datasets are used for out-of-domain data which
refers to data that significantly deviates from the distribution of the
training data. The predictive entropy distribution is calculated from the mean
of 250 batch samples and is subsequently modeled as a normal distribution.

TABLE I: Results on regression benchmark datasets of the proposed approach and
related works Probabilistic back-propagation (PBP) [42], MC-Dropout [11], Deep
Ensembles [7] comparing RMSE and NLL. Dataset size ($N$) and input
dimensionality ($Q$) are also given.

|  Avg. Test RMSE and Std. Errors $\downarrow$ |  Avg. Test LL and Std. Errors $\downarrow$ |
---|---|---|---
Dataset | $N$ | $Q$ | PBP | MC-Dropout | Deep Ensemble | Proposed | PBP | MC-Dropout | Deep Ensemble | Proposed |
Boston Housing | 506 | 13 |  3.01 $\pm$ 0.18 |  2.97 $\pm$ 0.85 |  3.28 $\pm$ 1.00 | 2.97 $\pm$0.46 |  2.57 $\pm$ 0.09 | 2.46 $\pm$ 0.25 | 2.41 $\pm$ 0.25 |  4.92 $\pm$1.03 |
Concrete Strength | 1,030 | 8 |  5.67 $\pm$ 0.09 |  5.23 $\pm$ 0.53 |  6.03 $\pm$ 0.58 | 5.51 $\pm$0.41 |  3.16 $\pm$ 0.02 | 3.04 $\pm$ 0.09 |  3.06 $\pm$ 0.18 |  5.02 $\pm$0.62 |
Energy Efficiency | 768 | 8 |  1.80 $\pm$ 0.05 |  1.66 $\pm$ 0.19 |  2.09 $\pm$ 0.29 | 1.53 $\pm$0.38 |  2.04 $\pm$ 0.02 |  1.99 $\pm$ 0.09 | 1.38 $\pm$ 0.22 | 1.41 $\pm$0.46 |
Kin8nm | 8,192 | 8 |  0.10 $\pm$ 0.00 |  0.10 $\pm$ 0.00 |  0.09 $\pm$ 0.00 | 0.07 $\pm$0.00 |  -0.90 $\pm$ 0.01 |  -0.95 $\pm$ 0.03 | -1.20 $\pm$ 0.02 |  -0.95 $\pm$0.01 |
Naval Propulsion | 11,934 | 16 |  0.01 $\pm$ 0.00 |  0.01 $\pm$ 0.00 |  0.00 $\pm$ 0.00 | 0.00 $\pm$0.00 |  -3.73 $\pm$ 0.01 |  -3.80 $\pm$ 0.05 | -5.63 $\pm$ 0.05 |  -3.81 $\pm$0.08 |
Power Plant | 9,568 | 4 |  4.12 $\pm$ 0.03 | 4.02 $\pm$ 0.18 |  4.11 $\pm$ 0.17 |  4.48 $\pm$0.18 |  2.84 $\pm$ 0.01 |  2.80 $\pm$ 0.05 | 2.79 $\pm$ 0.04 |  2.95 $\pm$0.05 |
Protein Structure | 45,730 | 9 |  4.73 $\pm$ 0.01 |  4.36 $\pm$ 0.04 |  4.71 $\pm$ 0.06 | 3.92 $\pm$0.03 |  2.97 $\pm$ 0.00 |  2.89 $\pm$ 0.01 | 2.83 $\pm$ 0.02 |  5.05 $\pm$0.52 |
Wine Quality Red | 1,599 | 11 |  0.64 $\pm$ 0.01 |  0.62 $\pm$ 0.04 |  0.64 $\pm$ 0.04 | 0.64 $\pm$0.05 |  0.97 $\pm$ 0.01 |  0.93 $\pm$ 0.06 |  0.94 $\pm$ 0.12 |  1.28 $\pm$0.33 |
Yacht Hydrodynamics | 308 | 6 | 1.02 $\pm$ 0.05 |  1.11 $\pm$ 0.38 |  1.58 $\pm$ 0.48 |  3.22 $\pm$1.59 |  1.63 $\pm$ 0.02 |  1.55 $\pm$ 0.12 | 1.18 $\pm$ 0.21 |  1.37 $\pm$0.43 |
Year Prediction MSD | 515,345 | 90 |  8.88 $\pm$ NA |  8.85 $\pm$ NA |  8.89 $\pm$ NA | 8.53 $\pm$NA |  3.60 $\pm$ NA |  3.59 $\pm$ NA | 3.35 $\pm$ NA |  7.63 $\pm$ NA |

###  IV-B Evaluation of Regression on Real-World UCI Datasets

The result of the regression task is depicted in Table I. Our approach is
compared with Bayesian [42], implicit ensemble (MC-Dropout) [11], and ensemble
[7] methods. As can be seen, our method outperforms or is competitive with
existing methods in terms of RMSE and NLL. Specifically, our method
outperforms other methods in 8 out of the 10 datasets in terms of RMSE. In
some datasets, we observe that our method is slightly worse in terms of NLL.
We believe that this is due to the fact that our method optimizes for RMSE
instead of NLL (which captures predictive uncertainty). We found that there is
a trade-off between RMSE and NLL. Optimizing for NLL instead reduces RMSE.
Also, we did not perform hyperparameters optimization, unlike [11] which
performed grid search.

###  IV-C Evaluation of Classification

In classification tasks with various topologies, it can be observed that our
method improves inference accuracy by up to $0.81\%$ or is comparable with the
single model, as shown in Table II.

In terms of uncertainty estimates in the OoD data, Fig. 4 shows the predictive
uncertainty of the ResNet-32 model trained on clean CIFAR-10. It can be
observed that the predictive entropy is low in clean CIFAR-10, that is, ID
data. However, if the model receives OoD data, e.g., rotated, SVHN, or STL-10
data, the predictive entropy increases from baseline. Importantly, the
relative change in the predictive entropy is significantly higher for our
proposed Tiny-DE approach. Here, the relative change in the uncertainty
estimates signifies better capabilities in the uncertainty estimates.
Furthermore, the change in predictive entropy becomes greater as the number of
ensembles increases, which is an ideal behavior.

In contrast, the CIFAR-100 model is evaluated on the max disagreement metric,
as shown in Fig. 5. In ID data, our approach shows finer granularity in
uncertainty estimates. Specifically, the uncertainty is low for correctly
classified images and high for incorrectly classified images. On corrupted
(rotated and noisy) images, OoD data, the model can still predict some images
correctly. Our approach shows a similar uncertainty distribution for correctly
and incorrectly predicted images. In addition, the relative change from the
baseline distribution is also high. In domain-changed data (SVHN and STL-10),
our approach shows high uncertainty with distributions concentrated toward the
right. Furthermore, the distributions shift more toward the right as the
number of ensembles increases.

TABLE II: Performance of Tiny-DE with CIFAR-10 and CIFAR-100 dataset trained on various topologies with up to 15 ensemble members. Topology | Dataset | Number of ensembles
---|---|---
1 | 5 | 10 | 15
VGG-19 | CIFAR-10 | 93.91 | 93.86 | 93.79 | 93.80
ResNet-56 | 94.37 | 94.28 | 94.14 | 94.38
ShuffleNet-V2 | 93.3 | 93.27 | 93.44 | 93.67
RepVGG-A1 | 94.93 | 94.56 | 94.84 | 94.62
MobileNet-V2 | 94.05 | 93.67 | 93.92 | 94.01
VGG-19 | CIFAR-100 | 73.87 | 74.21 | 74.56 | 74.68
ResNet-56 | 72.63 | 72.64 | 72.85 | 72.82
ShuffleNet-V2 | 72.58 | 72.75 | 73.54 | 73.11
RepVGG-A1 | 76.44 | 75.77 | 74.67 | 75.21
MobileNet-V2 | 74.29 | 74.41 | 74.67 | 75.21
Figure 4: Uncertainty distributions for the Tiny-DE approach on CIFAR-10,
including ID CIFAR-10, and OOD datasets such as rotated CIFAR-10, SVHN, and
STL. Notably, larger ensembles show increased relative change of uncertainty
distribution from ID compared to a single model (M = 1).  Figure 5: ID and OoD
Max Disagreement distributions for the Tiny-DE approach trained on clean
CIFAR-100 (ID). Notably, larger ensembles show increased relative change of
uncertainty distribution from ID.

###  IV-D Evaluation of Time-Series Prediction

Figure 6: Auto-regressive time series prediction of atmospheric CO2 of a
single model and our proposed Tiny-DE model with up to 10 ensemble members.
The shaded region shows the uncertainty around prediction.

The performance of our proposed approach on autoregressive time series
prediction is shown in Fig. 6. As can be seen, the prediction curve is closer
to the ground truth for our approach compared to the single model.
Furthermore, the curve approaches ground truth as the number of ensemble
members increases. Specifically, the single model achieves an RMSE score of
$0.1119$. In contrast, our proposed Tiny-DE method achieves an RMSE score of
$0.0943$ for 5 ensemble members, which is reduced to $0.0921$ for 10 members.
That translates into a $17.7\%$ reduction in the RMSE score. In general, all
models follow the same trend as the ground truth.

###  IV-E Evaluation of Semantic Segmentation

Similarly, in semantic segmentation tasks with several challenging datasets
and SOTA models, our approach performs comparably or outperforms the baseline
model, as shown in Table III. Two qualitative examples of each dataset are
shown in Fig. 7. As can be seen, the predictions are close to the ground
truth, with only incorrect predictions around the edges of segments or in
uncommon classes. Here, uncommon classes refer to classes that occur
infrequently or are less represented in the dataset.

In terms of uncertainty estimates, our proposed approach can estimate
uncertainty accurately. In an ideal case, misclassified pixels should have
high uncertainty around them, and correctly classified pixels should have low
uncertainty. As shown in Fig. 7 our approach captured this behavior
effectively.

TABLE III: Pixel accuracy and mean intersection over union (IoU) of the single
model and our proposed Tiny-DE (M = 5) with different datasets and SOTA
models.

Topology | Dataset | Single Model | Proposed (M=5)
---|---|---|---
|  | Pixel Acc | mIoU | Pixel Acc | mIoU
UNet++ (ResNet-34) | CamVid | 91.65 | 63.95 | 91.52 | 63.99
FPN (ResNet-18) | KvaSir | 95.95 | 74.62 | 95.89 | 74.57
FCN (ResNet-50) | CIFAR-10 | 87.78 | 69.63 | 87.71 | 68.58

Figure 7: Qualitative results for several semantic segmentation tasks and
associated uncertainty estimates. The correctness map is a binary diagram
indicating correct and incorrect predictions in white and black, respectively.

###  IV-F Comparison with Related Works

In the presence of OoD data, the higher the relative change in predictive
entropy with respect to ID distribution, the better the method. Compared to
related uncertainty estimation methods with model ensemble [7, 11, 8], the
relative predictive entropy of our Tiny-DE approach is much higher, as shown
in Fig. 8. This further underscores the robustness of our approach. Here, the
validation is done on the ResNet-32 topology on the CIFAR-10 dataset, but we
found that this translates to other topologies and datasets.

Figure 8: Relative change in predictive entropy on OoD data of Tiny-DE (ours)
in comparison to Deep Ensemble [7], MC-Dropout [11], and BatchEnsemble [8].

###  IV-G Improving Diversity

As mentioned in Section III-E, more diversity among the prediction of the
ensembling members can lead to better performance and uncertainty estimates.
Therefore, we have performed another set of experiments in which each ensemble
member is trained with different data augmentations. We found that by
improving diversity with different random data augmentation to train each
ensemble member, the uncertainty estimates increase on OoD data. For example,
as shown in Fig. 9 the uncertainty maps around incorrect pixels become
stronger compared to Fig. 7 when each ensemble member is trained using
different data augmentations. Furthermore, pixel accuracy and mIoU increased
to $88.67\%$ and $72.48\%$, respectively.

Figure 9: Results for Pascal VOC with improved diversity in ensemble members
using different random data augmentation.

###  IV-H Hardware Overhead

Figure 10: The inference cost in terms of memory and latency of our and
related approaches w.r.t the ensemble size. The results are relative to a
single model cost. The testing time cost and memory cost of the naive ensemble
are plotted in blue.

Figs. 10 show the relative cost in terms of memory and latency of our approach
and related approaches for the ResNet-32 topology. In terms of memory
overhead, our approach has approximately the same overhead as the
BatchEnsemble [8] and MC-Dropout [11] methods but significantly outperforms
branch ensemble [14] and Deep Ensemble [7] methods. The memory overhead of the
deep ensemble increases linearly with the size of the ensemble. In the branch
ensemble method, the last two convolutional and final classifier layers are
ensembled. Since the last two layers consume $\sim 75\%$ of the total
parameters, ensembling them leads to a high memory overhead. Specifically, if
batch normalization is used, our method has slightly more overhead compared to
BatchEnsemble due to the requirements of running mean and variance vector
storage. However, for other normalization layers that do not calculate running
mean and variance, the memory overhead is the same.

In terms of latency, our approach has the same latency as the single model, as
no additional computation is required relative to the single model. Therefore,
the latency is the same as the branch ensemble method. However, BatchEnsemble
has additional computation requirements in the input and output of
convolutional layers, leading to as much as $2\times$ latency as our method.
The latency of the deep ensemble and the MC-dropout increases linearly with
the size of the ensembles.

In general, our approach provides a good balance between memory and latency.
In parallel mode, our approach requires _one forward-passes and has
approximately the same memory overhead_ relative to a single model (_an ideal
case_). Consequently, our approach has up to $\sim M\times$ reduction in
overhead.

##  V Conclusion

In this paper, we present a cost-effective ensembling method for edge AI
accelerators. We introduce the Tiny-DE topology, where only normalization
layers are ensembled and all ensemble members share the weights and biases.
Our approach is scalable in terms of AI accelerators, datasets, NN topologies,
and tasks. With an expensive evaluation, we show that our approach can
estimate uncertainty effectively with up to $\sim 1\%$ improvement in
accuracy, and a $17.7\%$ reduction in RMSE score on various tasks.
Furthermore, our approach has up to $\sim M\times$ reduction in hardware
overhead.

## References

  * [1] Y. LeCun, Y. Bengio, and G. Hinton, “Deep learning,” _nature_ , vol. 521, no. 7553, pp. 436–444, 2015.
  * [2] M. Carroll, A. Van Der Merwe, and P. Kotze, “Secure cloud computing: Benefits, risks and controls,” in _2011 information security for South Africa_. IEEE, 2011, pp. 1–9.
  * [3] L. Lovén, T. Leppänen, E. Peltonen, J. Partala, E. Harjula, P. Porambage, M. Ylianttila, and J. Riekki, “Edgeai: A vision for distributed, edge-native artificial intelligence in future 6g networks,” _6G Wireless Summit, March 24-26, 2019 Levi, Finland_ , 2019.
  * [4] Y. Hu, W. Pang, X. Liu, R. Ghosh, B. Ko, W.-H. Lee, and R. Govindan, “Rim: Offloading inference to the edge,” in _Proceedings of the International Conference on Internet-of-Things Design and Implementation_ , 2021, pp. 80–92.
  * [5] Y. Abadade, A. Temouden, H. Bamoumen, N. Benamar, Y. Chtouki, and A. S. Hafid, “A comprehensive survey on tinyml,” _IEEE Access_ , vol. 11, pp. 96 892–96 922, 2023.
  * [6] D. Hendrycks and T. Dietterich, “Benchmarking neural network robustness to common corruptions and perturbations,” _arXiv preprint arXiv:1903.12261_ , 2019.
  * [7] B. Lakshminarayanan, A. Pritzel, and C. Blundell, “Simple and scalable predictive uncertainty estimation using deep ensembles,” _Advances in neural information processing systems_ , vol. 30, 2017.
  * [8] Y. Wen, D. Tran, and J. Ba, “Batchensemble: an alternative approach to efficient ensemble and lifelong learning,” _arXiv preprint arXiv:2002.06715_ , 2020.
  * [9] M. Abdar, F. Pourpanah, S. Hussain, D. Rezazadegan, L. Liu, M. Ghavamzadeh, P. Fieguth, X. Cao, A. Khosravi, U. R. Acharya _et al._ , “A review of uncertainty quantification in deep learning: Techniques, applications and challenges,” _Information fusion_ , vol. 76, pp. 243–297, 2021.
  * [10] A. G. Wilson and P. Izmailov, “Bayesian deep learning and a probabilistic perspective of generalization,” _Advances in neural information processing systems_ , vol. 33, pp. 4697–4708, 2020.
  * [11] Y. Gal and Z. Ghahramani, “Dropout as a bayesian approximation: Representing model uncertainty in deep learning,” in _international conference on machine learning_. PMLR, 2016, pp. 1050–1059.
  * [12] A. Mobiny, P. Yuan, S. K. Moulik, N. Garg, C. C. Wu, and H. Van Nguyen, “Dropconnect is effective in modeling uncertainty of bayesian deep networks,” _Scientific reports_ , vol. 11, no. 1, p. 5458, 2021.
  * [13] S. T. Ahmed, K. Danouchi, C. Münch, G. Prenat, L. Anghel, and M. B. Tahoori, “Spindrop: Dropout-based bayesian binary neural networks with spintronic implementation,” _IEEE Journal on Emerging and Selected Topics in Circuits and Systems_ , vol. 13, no. 1, pp. 150–164, 2023.
  * [14] J. Rock, T. Azevedo, R. de Jong, D. Ruiz-Muñoz, and P. Maji, “On efficient uncertainty estimation for resource-constrained mobile applications,” _arXiv preprint arXiv:2111.09838_ , 2021.
  * [15] S. Tuhin Ahmed, K. Danouchi, C. Münch, G. Prenat, A. Lorena, and M. B. Tahoori, “Binary bayesian neural networks for efficient uncertainty estimation leveraging inherent stochasticity of spintronic devices,” in _Proceedings of the 17th ACM International Symposium on Nanoscale Architectures_ , 2022, pp. 1–6.
  * [16] S. T. Ahmed, K. Danouchi, M. Hefenbrock, G. Prenat, L. Anghel, and M. B. Tahoori, “Scale-dropout: Estimating uncertainty in deep neural networks using stochastic scale,” _arXiv preprint arXiv:2311.15816_ , 2023.
  * [17] ——, “Spatial-spindrop: Spatial dropout-based binary bayesian neural network with spintronics implementation,” _arXiv preprint arXiv:2306.10185_ , 2023.
  * [18] S. Ioffe and C. Szegedy, “Batch normalization: Accelerating deep network training by reducing internal covariate shift,” in _International conference on machine learning_. pmlr, 2015, pp. 448–456.
  * [19] C. Guo, G. Pleiss, Y. Sun, and K. Q. Weinberger, “On calibration of modern neural networks,” in _International conference on machine learning_. PMLR, 2017, pp. 1321–1330.
  * [20] L. V. Jospin, H. Laga, F. Boussaid, W. Buntine, and M. Bennamoun, “Hands-on bayesian neural networks—a tutorial for deep learning users,” _IEEE Computational Intelligence Magazine_ , vol. 17, no. 2, pp. 29–48, 2022.
  * [21] J. L. Ba, J. R. Kiros, and G. E. Hinton, “Layer normalization,” _arXiv preprint arXiv:1607.06450_ , 2016.
  * [22] D. Ulyanov, A. Vedaldi, and V. Lempitsky, “Instance normalization: The missing ingredient for fast stylization,” _arXiv preprint arXiv:1607.08022_ , 2016.
  * [23] Y. Wu and K. He, “Group normalization,” in _Proceedings of the European conference on computer vision (ECCV)_ , 2018, pp. 3–19.
  * [24] L. K. Hansen and P. Salamon, “Neural network ensembles,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 12, no. 10, pp. 993–1001, 1990.
  * [25] T. G. Dietterich, “Ensemble methods in machine learning,” in _International workshop on multiple classifier systems_. Springer, 2000, pp. 1–15.
  * [26] D. Opitz and R. Maclin, “Popular ensemble methods: An empirical study,” _Journal of artificial intelligence research_ , vol. 11, pp. 169–198, 1999.
  * [27] C. Buciluǎ, R. Caruana, and A. Niculescu-Mizil, “Model compression,” in _Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining_ , 2006, pp. 535–541.
  * [28] G. Hinton, O. Vinyals, and J. Dean, “Distilling the knowledge in a neural network,” _arXiv preprint arXiv:1503.02531_ , 2015.
  * [29] G. Huang, Y. Li, G. Pleiss, Z. Liu, J. E. Hopcroft, and K. Q. Weinberger, “Snapshot ensembles: Train 1, get m for free,” _arXiv preprint arXiv:1704.00109_ , 2017.
  * [30] I. Loshchilov and F. Hutter, “Sgdr: Stochastic gradient descent with warm restarts,” _arXiv preprint arXiv:1608.03983_ , 2016.
  * [31] S. Hamdioui, L. Xie, H. A. Du Nguyen, M. Taouil, K. Bertels, H. Corporaal, H. Jiao, F. Catthoor, D. Wouters, L. Eike _et al._ , “Memristor based computation-in-memory architecture for data-intensive applications,” in _2015 Design, Automation & Test in Europe Conference & Exhibition (DATE)_. IEEE, 2015, pp. 1718–1725.
  * [32] S. Yu, H. Jiang, S. Huang, X. Peng, and A. Lu, “Compute-in-memory chips for deep learning: Recent trends and prospects,” _IEEE circuits and systems magazine_ , vol. 21, no. 3, pp. 31–56, 2021.
  * [33] O. Mutlu, S. Ghose, J. Gómez-Luna, and R. Ausavarungnirun, “Processing data where it makes sense: Enabling in-memory computation,” _Microprocessors and Microsystems_ , vol. 67, pp. 28–41, 2019.
  * [34] N. P. Jouppi, C. Young, N. Patil, D. Patterson, G. Agrawal, R. Bajwa, S. Bates, S. Bhatia, N. Boden, A. Borchers _et al._ , “In-datacenter performance analysis of a tensor processing unit,” in _Proceedings of the 44th annual international symposium on computer architecture_ , 2017, pp. 1–12.
  * [35] T. Posewsky and D. Ziener, “Efficient deep neural network acceleration through fpga-based batch processing,” in _2016 International Conference on ReConFigurable Computing and FPGAs (ReConFig)_. IEEE, 2016, pp. 1–8.
  * [36] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 770–778.
  * [37] K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” _arXiv preprint arXiv:1409.1556_ , 2014.
  * [38] D. Jha, P. H. Smedsrud, M. A. Riegler, P. Halvorsen, T. de Lange, D. Johansen, and H. D. Johansen, “Kvasir-seg: A segmented polyp dataset,” in _MultiMedia Modeling: 26th International Conference, MMM 2020, Daejeon, South Korea, January 5–8, 2020, Proceedings, Part II 26_. n.d.: Springer, 2020, pp. 451–462.
  * [39] T.-Y. Lin, P. Dollár, R. Girshick, K. He, B. Hariharan, and S. Belongie, “Feature pyramid networks for object detection,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 2117–2125.
  * [40] G. J. Brostow, J. Fauqueur, and R. Cipolla, “Semantic object classes in video: A high-definition ground truth database,” _Pattern Recognition Letters_ , 2009.
  * [41] Z. Zhou, M. M. R. Siddiquee, N. Tajbakhsh, and J. Liang, “Unet++: Redesigning skip connections to exploit multiscale features in image segmentation,” _IEEE transactions on medical imaging_ , vol. 39, no. 6, pp. 1856–1867, 2019.
  * [42] J. M. Hernández-Lobato and R. Adams, “Probabilistic backpropagation for scalable learning of bayesian neural networks,” in _International conference on machine learning_. PMLR, 2015, pp. 1861–1869.


# Hole-Like Fermi Surface in the Overdoped Non-Superconducting
Bi1.8Pb0.4Sr2CuO6+δ

T. Valla<EMAIL_ADDRESS>Condensed Matter Physics and Materials Science
Department, Brookhaven National Lab, Upton, New York 11973, USA
P. Pervan  Institute of Physics, Bijenička 46, HR-10000 Zagreb, Croatia
Faculty of Science, Department of Physics, University of Split, Rudera
Boškovića 33, HR-21000 Split, Croatia
I. Pletikosić  Condensed Matter Physics and Materials Science Department,
Brookhaven National Lab, Upton, New York 11973, USA
Department of Physics, Princeton University, Princeton, NJ 08544, USA  I. K.
Drozdov  Asish K. Kundu  Zebin Wu  G. D. Gu  Condensed Matter Physics and
Materials Science Department, Brookhaven National Lab, Upton, New York 11973,
USA

###### Abstract

In high-temperature cuprate superconductors, the anti-ferromagnetic spin
fluctuations are thought to have a very important role in naturally producing
an attractive interaction between the electrons in the $d$-wave channel. The
connection between superconductivity and spin fluctuations is expected to be
especially consequential at the overdoped end point of the superconducting
dome. In some materials, that point seems to coincide with a Lifshitz
transition, where the Fermi surface changes from the hole-like centered at
($\pi,\pi$) to the electron-like, centered at the $\Gamma$ point causing a
loss of large momentum anti-ferromagnetic fluctuations. Here, we study the
doping dependence of the electronic structure of Bi1.8Pb0.4Sr2CuO6+δ in angle-
resolved photoemission and find that the superconductivity vanishes at lower
doping than at which the Lifshitz transition occurs. This requires a more
detailed re-examination of a spin-fluctuation scenario.

###### pacs:

71.18.+y, 74.72.-h, 74.25.Jb

##  I Introduction

Even before the discovery of high temperature superconductors (HTSC), it was
realized that antiferromagnetic (AF) fluctuations give rise to a $d$-wave
pairing from short ranged repulsion of electrons on a lattice Hirsch (1985);
Scalapino et al. (1986). Additionally, for two-dimensional materials with
saddle points in the electronic structure, the Van Hove singularities enhance
the pairing, if the Fermi level is near the saddle point and the Fermi surface
(FS) is not nested Hirsch and Scalapino (1986). Furthermore, depending on the
FS shape, the spin susceptibility will transform from an anti-ferromagnetic,
peaked at $q=(\pi,\pi)$ to a ferromagnetic, centered at $q=(0,0)$ at a filling
corresponding to the Van Hove singularity (VHS). This has been considered as
the main reason for the loss of pairing in the $d$-wave channel and appearance
of ferromagnetic fluctuations Kopp et al. (2007); Kurashima et al. (2018) in
the strongly hole-overdoped cuprates when the FS undergoes a Lifshitz
transition and becomes electron-like, centered at the zone center Maier and
Scalapino (2020); Maier et al. (2020). Similar tendency towards the
ferromagnetism at the overdoped end of superconducting dome was also very
recently observed in the $n$-type cuprates Sarkar et al. (2020). The numerical
studies of hole-doped cuprates suggest that the pseudogap too is tightly
related to the FS topology, existing only for the hole-like FS, with the
Lifshitz transition effectively representing its upper phase boundary Wu et
al. (2018).

Figure 1: Evolution of the Fermi surface of Bi1.8Pb0.4Sr2CuO6+δ with doping.
(a) Fermi surface of a vacuum annealed bulk sample with the transition
temperature $T_{c}=20$ K. (b) Fermi surface of as grown, non-superconducting
sample ($T_{c}=0$). (c) Fermi surface of as-grown sample after annealing in
ozone. (d) Fermi surface of overdoped, non-superconducting Bi2212 ($T_{c}=0$),
for comparison. (e) and (f) illustrate the cases of hole- and electron-like
FSs. Spectral intensity in (a-d), represented by the gray-scale contours, is
integrated within $\pm 3$ meV around the Fermi level. Solid lines represent
the states obtained from tight-binding approximation that best fit the
experimental data. The area enclosed by the TB lines that best represent the
experimental data is calculated and used for the determination of the doping
parameter $p$. All the maps were recorded at 22 K (normal state).

Experimentally, the coincidental loss of superconductivity and the Lifshitz
transition have been observed in some materials at the overdoped end of the
superconducting dome, reviving the interest in this matter Kondo et al.
(2004); Drozdov et al. (2018); Ding et al. (2019); Valla et al. (2020); Maier
and Scalapino (2020). In the case of the single Cu-O plane cuprates, the
coincidence could be considered as a clear-cut evidence that the spin-fermion
model is at play at the overdoped endpoint of the doping range Kondo et al.
(2004); Ding et al. (2019). However, the recently established connection
between the loss of superconductivity and the Lifshitz transition that was
observed in the bilayer system, Bi2Sr2CaCu2O8+δ (Bi2212) raises more questions
Drozdov et al. (2018); Valla et al. (2020). In Bi2212, the Lifshitz transition
affects only the anti-bonding FS, but not the bonding one. Another cuprate
that does not follow this Lifshitz driven scenario is Tl2Ba2CuO6+δ (Tl2201).
The Tl2201 is a clean system showing quantum oscillations and a large FS in
the overdoped regime which undergoes the Lifshitz transition at much higher
doping than at which the superconductivity disappears Rourke et al. (2010);
Putzke et al. (2019). Also, in the case of La2-xSrxCuO4 and La2-xBaxCuO4 the
two transitions, although close, are not simultaneous, indicating that this
correlation might not be universal for all cuprates Ino et al. (2002); Yoshida
et al. (2006); Valla et al. (2006, 2007); Valla (2012).

These discrepancies call for a reexamination of the spin-fluctuation scenario
in which the Lifshitz transition triggers the loss of $d$-wave
superconductivity. Here, we study the electronic structure of
Bi1.8Pb0.4Sr2CuO6+δ (Bi2201), a single layer cousin of Bi2212, in angle-
resolved photoemission spectroscopy (ARPES) and find that the
superconductivity in this system is lost sooner than the doping at which the
Lifshitz transition occurs. This would suggest that the picture in which the
topology of the FS alone is the driving force for the loss of
superconductivity in the overdoped cuprates is too simplistic and requires
other ingredients.

##  II Methods

The experiments were done in an experimental facility that integrates oxide-
MBE with ARPES and SI-STM capabilities within a common vacuum system Kim et
al. (2018); Drozdov et al. (2018); Valla et al. (2020). The starting samples
were synthesized by the traveling floating zone method and the superconducting
transition temperature ($T_{c}$) was measured with a SQUID down to 1.8 K. The
as-grown Bi2201 samples were not superconducting and they required vacuum
annealing to achieve superconductivity with the maximal $T_{c}\approx 20-28$
K. The as-grown and vacuum annealed samples were clamped to the sample holder,
cleaved with Kapton tape and studied by ARPES. After the initial measurements,
some samples were additionally annealed in vacuum or in ozone, to reduce or
increase the hole-doping, respectively. The ARPES experiments were carried out
on a Scienta SES-R4000 electron spectrometer with the monochromatized HeI
(21.22 eV) radiation (VUV-5k). The total instrumental energy resolution was
$\sim$ 5 meV. Angular resolution was better than $\sim 0.15^{\circ}$ and
$0.4^{\circ}$ along and perpendicular to the slit of the analyzer,
respectively.

##  III Results and Discussion

Figure 1 shows the photoemission intensity from the narrow energy window
around the Fermi level ($\pm$3 meV), representing the Fermi surfaces of the
as-grown Bi2201 sample, panel (b), the vacuum annealed Bi2201 (a) and the as-
grown Bi2201 sample after annealing in ozone (c). For comparison, the ozone
annealed, non-superconducting Bi2212 sample is also shown in panel (d). The
panels (e) and (f) represent the FS that allows the $d$-wave pairing in the
antiferromagnetic spin-fluctuation scenario and the one that does not,
respectively Maier and Scalapino (2020); Maier et al. (2020). The tight
binding (TB) contours, representing the best fits to the experimental FSs are
also shown Markiewicz et al. (2005). The bare in-plane band structure is
approximated by the tight-binding formula:

$E_{\pm}(k)=\mu-2t(\cos k_{x}+\cos k_{y})+4t^{\prime}\cos k_{x}\cos
k_{y}-2t^{\prime\prime}(\cos 2k_{x}+\cos 2k_{y})\pm t_{\perp}(\cos k_{x}-\cos
k_{y})^{2}/4$

For Bi2212, $t_{\perp}\neq 0$ and $\pm$ is for anti-bonding (bonding) state;
$\mu$ is chemical potential. The hopping parameters that best describe the FSs
of the selected measured samples are given in Table 1.

Figure 2: Van Hove singularity in Bi1.8Pb0.4Sr2CuO6+δ. (a-d) Dispersion at
$k_{x}=\pi/a$ along the $Y-M-X$ line corresponding to the Bi2201 and Bi2212
samples from Fig.1. The TB dispersions used to fit the Fermi surfaces in Fig.1
are also shown, following the same color scheme as in Fig.1.

The Bi2201 FS consists of a single sheet, while the Bi2212 has two Fermi
sheets, the bonding (marked by the solid TB contour) and the antibonding one
(dashed TB contour). The “shadow” FS, shifted by $(\pi,\pi)$ relative to the
intrinsic one is also visible. In addition, in the Bi2212 and in the ozone
annealed Bi2201, the supermodulation replicas are also visible. Both the
“shadow” and the supermodulation replicas are structural in origin and were
shown to only minimally affect the electrons in the Cu-O planes Norman et al.
(1995); Ding et al. (1996); Mans et al. (2006); Valla et al. (2019); Gao et
al. (2020). The re-appearance of the supermodulation replicas in the ozone
annealed Bi2201 is probably related to the segregation of Pb-O and Bi-O in the
surface layer. If recleaved, the surface of ozone annealed crystal does not
show the supermodulation. For each sample, the number of doped carriers is
obtained directly from the Luttinger count of the area enclosed by the Fermi
contour, $p=2A_{FS}-1$. For the Bi2212, both the bonding and the antibonding
states are counted, $A_{FS}=(A_{-}+A_{+})/2$, originating from the two Cu-O
planes per unit cell.

From Fig. 1(b) it is clear that the Bi2201 FS remains hole-like, centered at
$(\pi,\pi)$ at $p=0.32$, even when the superconductivity is already lost. We
were able to induce the Lifshitz transition and turn the FS into an electron-
like, centered at (0, 0) by pushing additional holes into the Cu-O planes by
ozone annealing. This, however, only happens beyond $p\approx 0.43$, far away
from the doping at which superconductivity is lost. It would appear then that
in Bi2201 the two transitions are not bound and that a simple picture where
the Lifshitz transition and the loss of $d$-wave pairing occur simultaneously,
is not entirely correct. This is similar to the Tl2201 case where the
superconductivity ceases to exist at $p=0.31$, while the Lifshitz transition
is projected to occur at $p\approx 0.5$ Rourke et al. (2010); Putzke et al.
(2019).

The Lifshitz transition related to the filling of VHS at the antinodal points
of the Brillouin zone is further evidenced in Fig. 2. The VHS is clearly below
the Fermi level for the two less doped samples $(p=0.23$ and 0.32) and only
moves slightly above the Fermi level for the ozone-annealed sample ($p=0.45$).
For the latter, the state visible at $k_{y}\approx 0.2\pi/a$ is a
supermodulation replica of the intrinsic electronic structure, as previously
noted (see Fig. 1(c)).

Table 1: Tight-binding parameters for Bi2201 and Bi2212 samples from Fig.1 Sample |  $\mu$ (eV) |  $t$ (eV) |  $t^{\prime}$ (eV) |  $t^{\prime\prime}$ (eV) |  $t_{\perp}$ (eV)
---|---|---|---|---|---
Bi2201(a) | 0.42 | 0.36 | 0.1 | 0.036 | 0
Bi2201(b) | 0.49 | 0.36 | 0.1 | 0.036 | 0
Bi2201(c) | 0.545 | 0.37 | 0.1 | 0.036 | 0
Bi2212(d) | 0.467 | 0.36 | 0.108 | 0.036 | 0.108

It is interesting that in the Bi2212, the Lifshitz transition in one of the
Fermi sheets (antibonding) coincides with the loss of superconductivity, while
the other Fermi sheet (bonding) remains hole-like (Fig. 1(d) and 2(d)). Thus,
at the end of superconducting dome in Bi2212, we have a situation illustrated
in both panels (e) and (f) of Fig. 1 simultaneously. Assuming the simple spin-
fluctuation picture were correct, this would imply that the antibonding state
is somehow more important for superconductivity than the bonding state.

Figure 3: Phase diagram of Bi1.8Pb0.4Sr2CuO6+δ and Bi2Sr2CaCu2O8+δ.
Superconducting transition temperature, $T_{c}$, for 2212 (black circles) and
2201 (gray triangles) plotted versus experimentally determined doping, $p$.
The experimentally determined Bi2212 superconducting dome Drozdov et al.
(2018) is represented by black dashed line. The putative dome for 2201 is
represented by gray dashed line. The Lifshitz transition in Bi2201 is marked
by the gray arrow and the one in the antibonding FS of Bi2212 is marked by the
green arrow.

The main points of this study are summarized in Fig. 3. We show the doping
phase diagram with the superconducting phases for both Bi2212 from Drozdov et
al Drozdov et al. (2018) and the Bi2201 from this study. We also indicate the
position of the Lifshitz transition where the hole like FS turns into the
electron-like. In Bi2212 the transition is achievable only in the antibonding
Fermi sheet (indicated by the green arrow) and coincides with the end of the
superconducting dome. The transition in Bi2201 (gray arrow) occurs far from
the superconducting zone boundary.

We note that our results on Bi2201 are in contrast with previous ARPES studies
Kondo et al. (2004); Ding et al. (2019), probably due to differences in
chemical composition of the samples. Kondo et al Kondo et al. (2004) report
the superconducting phase up to $p\sim 0.4$ in Bi2201 co-doped with both Pb
and La. In Ding et al Ding et al. (2019) the sample composition is similar to
ours and the superconducting phase does not extend that far. Both studies,
however, suggest much closer correlation between the loss of superconductivity
and the Lifshitz transition. We emphasize that our as grown Bi2201 samples
were not superconducting down to 1.8 K, but were very stable both in the
ambient conditions and in UHV. They also gave very consistent ARPES results,
with the hole-like FS and $p=0.32\pm 0.015$ at every cleave ($>10$ cleaves).
Therefore, already our as grown sample makes the strong case for the title
claim of this study.

With the observed irregularities in the coincidence of the Lifshitz transition
and the loss of $d$-wave superconductivity in Bi2201 and in different families
of cuprate superconductors, a simple picture relating the FS topology with the
presence or absence of superconductivity requires a more detailed re-
examination. One problem could be that a certain level of three-dimensional
(3D) character in the electronic structure ($k_{z}$ warping) might spread the
VHS and place its apparent position in ARPES at the point that depends on the
experimental parameters, i.e. the used photon energy. This might be
particularly important in La2-xSrxCuO4 that shows a higher level of 3D
character Nakamura and Uchida (1993); Horio et al. (2018), unlike Bi2201 and
Tl2201 that are known to be extremely two-dimensional Manako et al. (1991);
Ono and Ando (2003). Another reason for the two transitions occurring non-
simultaneously could be that the $T_{c}$ is reduced much faster than the AF
spin-fluctuations due to some other effect. A plausible reason would be a
large pair-breaking scattering of some sort that would push the system into
the dirty limit and terminate superconductivity before the pairing
interactions are gone Rourke et al. (2010); Lee-Hone et al. (2017); Pelc et
al. (2019); Lee-Hone et al. (2020); Maier et al. (2020). A direct comparison
between Bi2201 and Bi2212 in Fig. 1 and 2 indeed points to the remarkably
broader electronic states in Bi2201 for similar doping levels. That clearly
indicates a significantly higher level of scattering in the Bi2201 system that
could explain a large mismatch between the Lifshitz transition and the loss of
superconductivity in that system. However, the Tl2201 case would argue against
this scenario, as this material is clean enough to display quantum
oscillations in transport and yet, it shows an even larger mismatch Rourke et
al. (2010); Putzke et al. (2019); Lee-Hone et al. (2020).

There is also a possibility that the curvature of the FS in the antinodal
region might play a significant role in cuprate superconductivity. The
straight antinodal segments would normally favor nesting and a formation of
density waves. However, in the highly overdoped regime near the end point of
the superconducting dome, the density waves can be discarded, as evidenced by
the lack of any gaps or reconstructions that they should produce. These
segments might be ineffective in forming mobile Cooper pairs because the group
velocity of these segments lacks the component that should connect them from
$\mathbf{k_{F}}$ to $-\mathbf{k_{F}}$ into singlet pairs. The Bi2212 (Fig.
1(d)) seems to be a good example of a large discrepancy in curvature of
antinodal segments in bonding and antibonding Fermi sheets and an indication
that this scenario might be at play in cuprates.

Finally, we speculate that the apparent coincidence of the onset of the
ferromagnetism and the demise of superconductivity in the $n$-type cuprates
Sarkar et al. (2020), is also dictated by the same fundamental principles that
are expected to be at play in the heavily overdoped $p$-type cuprates. In the
superconducting electron doped cuprates, the underlying FS is always hole-
like, centered at ($\pi,\pi$), but reconstructed at the “hot spots” by the AF
spin fluctuations into an electron and a hole pocket. However, at the end
point of the superconducting phase in $n$-type cuprates, the FS does not
suddenly turn into an electron-like one, as in the case of the hole doped
cuprates discussed here. What seems to happen is that the ($\pi,\pi$) centered
hole FS becomes so small that it is not touching the AF zone boundary anymore.
With the lack of AF “hot spots” the FS would not be reconstructed anymore and
it should turn into a single hole-like one centered at ($\pi,\pi$) He et al.
(2019). The spin susceptibility is then expected to shift from an
antiferromagnetic, with the weight centered at ($\pi,\pi$), to a
ferromagnetic, peaked at (0,0). This should be correlated with the loss of
$d$-wave pairing, similarly to what happens in the hole-doped cuprates.

##  IV Summary

In summary, we have explored the correlation between the loss of
superconductivity and the Lifshitz transition occurring in the heavily
overdoped side of the phase diagram of the hole doped cuprates. Although the
two transitions are not perfectly coincidental in all the cuprates, there
seems to be a general trend where they are relatively close but the exact
positions in the doping phase diagram may be influenced by other factors such
as the amount of elastic scattering and the curvature of the FS. In addition,
the energy scales related to electrons and the details of energy and momentum
distribution of spin susceptibility might also play a role in fine tuning of
doping positions of these two transitions. This implies that the $d$-wave
superconductivity in cuprates is very strongly coupled to the AF spin
fluctuations that, when present, likely mediate the pairing in these
materials.

###### Acknowledgements.

This work was supported by the US Department of Energy, Office of Basic Energy
Sciences, contract no. DE-SC0012704. I.P. was supported by ARO MURI program,
grant W911NF-12-1-0461.

## References

  * Hirsch (1985) J. E. Hirsch, Physical Review Letters 54, 1317 (1985), ISSN 0031-9007, URL https://link.aps.org/doi/10.1103/PhysRevLett.54.1317.
  * Scalapino et al. (1986) D. J. Scalapino, E. Loh, and J. E. Hirsch, Physical Review B 34, 8190 (1986), ISSN 0163-1829, URL https://link.aps.org/doi/10.1103/PhysRevB.34.8190.
  * Hirsch and Scalapino (1986) J. E. Hirsch and D. J. Scalapino, Physical Review Letters 56, 2732 (1986), ISSN 0031-9007, URL https://link.aps.org/doi/10.1103/PhysRevLett.56.2732.
  * Kopp et al. (2007) A. Kopp, A. Ghosal, and S. Chakravarty, Proceedings of the National Academy of Sciences of the United States of America 104, 6123 (2007), ISSN 0027-8424, URL http://www.ncbi.nlm.nih.gov/pubmed/17404239http://www.pubmedcentral.nih.gov/articlerender.fcgi?artid=PMC1851066.
  * Kurashima et al. (2018) K. Kurashima, T. Adachi, K. M. Suzuki, Y. Fukunaga, T. Kawamata, T. Noji, H. Miyasaka, I. Watanabe, M. Miyazaki, A. Koda, et al., Physical Review Letters 121, 057002 (2018), ISSN 0031-9007, URL https://link.aps.org/doi/10.1103/PhysRevLett.121.057002.
  * Maier and Scalapino (2020) T. A. Maier and D. J. Scalapino, Journal of Superconductivity and Novel Magnetism 33, 15 (2020), ISSN 1557-1939, URL http://link.springer.com/10.1007/s10948-019-05366-4.
  * Maier et al. (2020) T. A. Maier, S. Karakuzu, and D. J. Scalapino (2020), eprint 2004.13120, URL http://arxiv.org/abs/2004.13120.
  * Sarkar et al. (2020) T. Sarkar, D. S. Wei, J. Zhang, N. R. Poniatowski, P. R. Mandal, A. Kapitulnik, and R. L. Greene, Science 368, 532 (2020), ISSN 0036-8075, URL https://science.sciencemag.org/content/368/6490/532.
  * Wu et al. (2018) W. Wu, M. S. Scheurer, S. Chatterjee, S. Sachdev, A. Georges, and M. Ferrero, Physical Review X 8, 021048 (2018), ISSN 2160-3308, URL https://link.aps.org/doi/10.1103/PhysRevX.8.021048.
  * Kondo et al. (2004) T. Kondo, T. Takeuchi, T. Yokoya, S. Tsuda, S. Shin, and U. Mizutani, Journal of Electron Spectroscopy and Related Phenomena 137-140, 663 (2004), ISSN 0368-2048, URL https://www.sciencedirect.com/science/article/pii/S036820480400129X.
  * Drozdov et al. (2018) I. K. Drozdov, I. Pletikosić, C.-K. Kim, K. Fujita, G. D. Gu, J. C. S. Davis, P. D. Johnson, I. Božović, and T. Valla, Nature Communications 9, 5210 (2018), ISSN 2041-1723, URL http://www.nature.com/articles/s41467-018-07686-w.
  * Ding et al. (2019) Y. Ding, L. Zhao, H.-T. Yan, Q. Gao, J. Liu, C. Hu, J.-W. Huang, C. Li, Y. Xu, Y.-Q. Cai, et al., Chinese Physics Letters 36, 017402 (2019), ISSN 0256-307X, URL https://iopscience.iop.org/article/10.1088/0256-307X/36/1/017402.
  * Valla et al. (2020) T. Valla, I. K. Drozdov, and G. D. Gu, Nature Communications 11, 569 (2020), URL http://www.nature.com/articles/s41467-020-14282-4.
  * Rourke et al. (2010) P. M. C. Rourke, A. F. Bangura, T. M. Benseman, M. Matusiak, J. R. Cooper, A. Carrington, and N. E. Hussey, New Journal of Physics 12, 105009 (2010), ISSN 1367-2630, URL https://iopscience.iop.org/article/10.1088/1367-2630/12/10/105009.
  * Putzke et al. (2019) C. Putzke, S. Benhabib, W. Tabis, J. Ayres, Z. Wang, L. Malone, S. Licciardello, J. Lu, T. Kondo, T. Takeuchi, et al. (2019), eprint 1909.08102, URL http://arxiv.org/abs/1909.08102.
  * Ino et al. (2002) A. Ino, C. Kim, M. Nakamura, T. Yoshida, T. Mizokawa, A. Fujimori, Z.-X. Shen, T. Kakeshita, H. Eisaki, and S. Uchida, Physical Review B 65, 094504 (2002), ISSN 0163-1829, URL https://link.aps.org/doi/10.1103/PhysRevB.65.094504.
  * Yoshida et al. (2006) T. Yoshida, X. J. Zhou, K. Tanaka, W. L. Yang, Z. Hussain, Z.-X. Shen, A. Fujimori, S. Sahrakorpi, M. Lindroos, R. S. Markiewicz, et al., Physical Review B 74, 224510 (2006), ISSN 1098-0121, URL http://prb.aps.org/abstract/PRB/v74/i22/e224510.
  * Valla et al. (2006) T. Valla, A. V. Fedorov, J. Lee, J. C. Davis, and G. D. Gu, Science (New York, N.Y.) 314, 1914 (2006), ISSN 1095-9203, URL http://www.ncbi.nlm.nih.gov/pubmed/17110536.
  * Valla et al. (2007) T. Valla, T. Kidd, W.-G. Yin, G. Gu, P. Johnson, Z.-H. Pan, and A. Fedorov, Physical Review Letters 98, 167003 (2007), ISSN 0031-9007, URL http://link.aps.org/doi/10.1103/PhysRevLett.98.167003.
  * Valla (2012) T. Valla, Physica C: Superconductivity 481, 66 (2012), ISSN 0921-4534, URL https://www.sciencedirect.com/science/article/pii/S0921453412001839?via{%}3Dihub.
  * Kim et al. (2018) C. K. Kim, I. K. Drozdov, K. Fujita, J. C. S. Davis, I. Božović, and T. Valla, https://doi.org/10.1016/j.elspec.2018.07.003 (2018), eprint 1805.04811, URL https://doi.org/10.1016/j.elspec.2018.07.003.
  * Markiewicz et al. (2005) R. S. Markiewicz, S. Sahrakorpi, M. Lindroos, H. Lin, and A. Bansil, Physical Review B 72, 054519 (2005), ISSN 1098-0121, URL https://link.aps.org/doi/10.1103/PhysRevB.72.054519.
  * Norman et al. (1995) M. R. Norman, M. Randeria, H. Ding, and J. C. Campuzano, Physical Review B 52, 615 (1995), ISSN 0163-1829, URL https://link.aps.org/doi/10.1103/PhysRevB.52.615.
  * Ding et al. (1996) H. Ding, A. F. Bellman, J. C. Campuzano, M. Randeria, M. R. Norman, T. Yokoya, T. Takahashi, H. Katayama-Yoshida, T. Mochiku, K. Kadowaki, et al., Physical Review Letters 76, 1533 (1996), ISSN 0031-9007, URL https://link.aps.org/doi/10.1103/PhysRevLett.76.1533.
  * Mans et al. (2006) A. Mans, I. Santoso, Y. Huang, W. K. Siu, S. Tavaddod, V. Arpiainen, M. Lindroos, H. Berger, V. N. Strocov, M. Shi, et al., Physical Review Letters 96, 107007 (2006), ISSN 0031-9007, URL https://link.aps.org/doi/10.1103/PhysRevLett.96.107007.
  * Valla et al. (2019) T. Valla, I. Pletikosić, I. K. Drozdov, and G. D. Gu, Physical Review B 100, 241112 (2019), ISSN 2469-9950, URL https://link.aps.org/doi/10.1103/PhysRevB.100.241112.
  * Gao et al. (2020) Q. Gao, H. Yan, J. Liu, P. Ai, Y. Cai, C. Li, X. Luo, C. Hu, C. Song, J. Huang, et al., Phys. Rev. B 101, 014513 (2020), URL https://link.aps.org/doi/10.1103/PhysRevB.101.014513.
  * Nakamura and Uchida (1993) Y. Nakamura and S. Uchida, Physical Review B 47, 8369 (1993), URL https://journals.aps.org/prb/abstract/10.1103/PhysRevB.47.8369.
  * Horio et al. (2018) M. Horio, K. Hauser, Y. Sassa, Z. Mingazheva, D. Sutter, K. Kramer, A. Cook, E. Nocerino, O. Forslund, O. Tjernberg, et al., Physical Review Letters 121, 077004 (2018), ISSN 0031-9007, URL https://link.aps.org/doi/10.1103/PhysRevLett.121.077004.
  * Manako et al. (1991) T. Manako, Y. Shimakawa, Y. Kubo, and H. Igarashi, Physica C: Superconductivity 185-189, 1327 (1991), ISSN 0921-4534, URL https://www.sciencedirect.com/science/article/pii/092145349191887A.
  * Ono and Ando (2003) S. Ono and Y. Ando, Physical Review B 67, 104512 (2003), URL https://journals.aps.org/prb/abstract/10.1103/PhysRevB.67.104512.
  * Lee-Hone et al. (2017) N. R. Lee-Hone, J. S. Dodge, and D. M. Broun, Physical Review B 96, 024501 (2017), ISSN 2469-9950, URL http://link.aps.org/doi/10.1103/PhysRevB.96.024501.
  * Pelc et al. (2019) D. Pelc, P. Popčević, M. Požek, M. Greven, and N. Barišić, Science Advances 5, eaau4538 (2019), ISSN 2375-2548, URL https://advances.sciencemag.org/lookup/doi/10.1126/sciadv.aau4538.
  * Lee-Hone et al. (2020) N. R. Lee-Hone, H. U. Özdemir, V. Mishra, D. M. Broun, and P. J. Hirschfeld, Physical Review Research 2, 013228 (2020), ISSN 2643-1564, URL https://link.aps.org/doi/10.1103/PhysRevResearch.2.013228.
  * He et al. (2019) J. He, C. R. Rotundu, M. S. Scheurer, Y. He, M. Hashimoto, K.-J. Xu, Y. Wang, E. W. Huang, T. Jia, S. Chen, et al., Proceedings of the National Academy of Sciences of the United States of America 116, 3449 (2019), ISSN 1091-6490, URL http://www.ncbi.nlm.nih.gov/pubmed/30808739%****␣2201.tex␣Line␣600␣****http://www.pubmedcentral.nih.gov/articlerender.fcgi?artid=PMC6397544.