https://doi.org/10.1109/ACCESS.2022.3163270

Rohitash Chandra: First A. Author (rohitash.chandra@sydney.edu.au).

# Revisiting Bayesian autoencoders with MCMC

ROHITASH CHANDRA1,3  MAHIR JAIN 2, MANAVENDRA MAHARANA , JR.1,3  Pavel N.
Krivitsky 1  Transitional Artificial Intelligence Research Group, School of
Mathematics and Statistics, UNSW Sydney, NSW 2052, Australia  Manipal
Institute of Technology, Karnataka, India  UNSW Data Science Hub, UNSW Sydney,
NSW 2052, Australia

###### Abstract

Autoencoders gained popularity in the deep learning revolution given their
ability to compress data and provide dimensionality reduction. Although
prominent deep learning methods have been used to enhance autoencoders, the
need to provide robust uncertainty quantification remains a challenge. This
has been addressed with variational autoencoders so far. Bayesian inference
via Markov Chain Monte Carlo (MCMC) sampling has faced several limitations for
large models; however, recent advances in parallel computing and advanced
proposal schemes have opened routes less traveled. This paper presents
Bayesian autoencoders powered by MCMC sampling implemented using parallel
computing and Langevin-gradient proposal distribution. The results indicate
that the proposed Bayesian autoencoder provides similar performance accuracy
when compared to related methods in the literature. Furthermore, it provides
uncertainty quantification in the reduced data representation. This motivates
further applications of the Bayesian autoencoder framework for other deep
learning models.

###### Index Terms:

Bayesian deep learning; MCMC; Langevin dynamics; Autoencoders; Parallel
tempering; Deep learning.

=-15pt

##  I Introduction

Autoencoders are a family of unsupervised learning methods that use neural
network architectures and learning algorithms to learn a lower-dimensional
(reduced) representation which can then be used to reconstruct a
representation close to the original data. They thus facilitate dimensionality
reduction for prediction and classification problems [1, 2]. Autoencoders have
been successfully applied to image classification [3, 4], face recognition [5,
6], geoscience and remote sensing [7], speech-based emotion recognition [8],
and data generation [9]. Autoencoders have been prominent in deep neural
network architectures and also for transfer learning tasks [10, 11]. Recent
developments include regularized autoencoders [12], variational autoencoders
[13, 2, 14, 15, 16], adversarial autoencoders [17, 18, 8, 19, 20], variational
graph autoencoders [21, 22], and convolutional autoencoders [10].

Bayesian neural networks incorporate alternative training via Bayesian
inference in which the model parameters (weights and biases) are (jointly)
represented by a probability distribution rather than point estimates given by
conventional gradient-based training methods [23]. The _posterior_ probability
distribution naturally accounts for uncertainty in parameter estimates, which
is further propagated into the decision-making process [24, 25]. Bayes’
theorem is used as the foundation for inference in Bayesian neural networks.
Markov Chain Monte Carlo (MCMC) sampling methods [26] implement Bayesian
inference by estimating the posterior distribution, which is often referred to
as sampling from the posterior distribution. Variational inference [27]
provides another way to approximate the posterior distribution, which
approximates an intractable posterior distribution by a tractable one. This
makes it particularly suited to large data sets and models, and so it has been
popular for autoencoders and neural networks [13, 28]. Variational
autoencoders [13] employ variational inference [27] as a form of
regularisation of the model parameters (weights) during training [2, 14, 15,
16].

A deep learning model can feature tens of thousands to millions of parameters
that are typically optimized by gradient-based algorithms. The major challenge
of MCMC methods for training (sampling) deep learning models is the
computational time required for constructing a good approximation of the
posterior distribution, incurring heavy computational costs. However, progress
in incorporating gradient-based proposals into MCMC sampling has been inspired
by earlier works [29, 30] for the development of Hamiltonian and Langevin-
based MCMC methods [25, 24, 31]. Langevin-gradient proposal distribution
demonstrated to be effective for Bayesian neural learning [32, 33] and
combining it with parallel computing via tempered MCMC resulted in further
improvements [32], which motivates their application for deep autoencoders
that can feature thousands of model parameters.

This paper presents Bayesian autoencoders powered by tempered MCMC sampling
that incorporate parallel computing and Langevin-gradient proposal
distribution. We demonstrate the effectiveness of Bayesian autoencoders with
benchmark datasets that involve small-scale to large-scale models featuring a
maximum of about a million parameters. We also investigate the effect of two
prominent gradient-based training methods and develop an adaptive Langevin-
gradient proposal distribution for tempered MCMC. We finally compare our
results with the literature where gradient-based methods have been used for
the same datasets. The Bayesian autoencoder would provide a principled
approach to uncertainty quantification in reduced data representation using
MCMC. The Bayesian autoencoder can be used to reconstruct an ensemble of the
reduced dataset from the posterior distribution rather than a single one given
by single-point estimates given by gradient-descent learning in conventional
autoencoders. Hence, once the posterior distribution is obtained, different
estimators for the Bayesian autoencoder’s parameters are possible such as the
maximum a posteriori probability (MAP) estimate [34].

The rest of the paper is organized as follows. In Section II, we present a
background and literature review of related methods. Section III presents the
proposed methodology, followed by experiments and results in Section IV.
Section V provides a discussion, and Section VI concludes the paper with
directions of future work.

##  II Related work

###  II-A Autoencoders

Autoencoders compress data by providing a reduced feature set that may be used
for supervised learning. Autoencoders have been used with supervised learning
methods such as simple neural networks [35], recurrent neural networks [36],
long-short term memory (LSTM) networks [37], and graph neural networks [38].
Depending on the dataset, the features from other deep learning methods are
used to augment the canonical autoencoder. For instance, a deep convolutional
autoencoder incorporates pooling and convolutional layers used in
convolutional neural networks for unsupervised learning tasks [39]. The
applications of autoencoders in machine learning problems include data
classification, dimensionality reduction, pattern recognition, image
denoising, anomaly detection, and recommender systems [40, 41, 42, 43, 44,
45]. Autoencoders have been successfully applied to a number of real-world
applications such as cyber-security [46], medical imaging [47], and biometrics
[36]. Furthermore, a review of autoencoders has been done for geoscience and
remote sensing with a focus on hyperspectral images [7].

Next, we review some of the key autoencoder architectures from the literature
[48]. Sparsity autoencoders [49] are used to capture the latent, rather than
redundant, representations of the data using a loss function that penalizes
activation of the hidden layer; hence, enforcing the sparsity constraint.
Denoising autoencoders [50] involve adding artificial noise to the data, after
which the autoencoder attempts to reconstruct the original data from noisy
data. This prevents the autoencoder from capturing an overly-complete
representation of the data which is common to many conventional autoencoders.
Contractive autoencoder [51] makes the reduced representation less sensitive
to small variations in the input samples; this is achieved by adding a
regularizer to the loss function which reduces the reduced representation’s
sensitivity to the input. Convolutional autoencoders [52, 53] create a reduced
representation of image data which is helpful for supervised machine learning.

###  II-B Bayesian deep learning

The major limitation of conventional autoencoders is the lack of uncertainty
quantification in the estimation of model parameters and the challenge of
tuning hyperparameters. Bayesian neural networks address these by estimating
the parameters of the model (weights and biases) as random variables, in
contrast to single point estimates by backpropagation learning that employs
gradient-based methods [26, 23]. The progress in neural networks has led to a
revolution in deep learning models; however, progress in Bayesian neural
networks and Bayesian deep learning has been relatively slow due to certain
challenges [54, 55]. Bayesian inference implemented with canonical MCMC
methods used to face major difficulties in handling the large number of model
parameters. Variational inference provides an alternative Bayesian inference
approach with successful implementations in variational autoencoders [56],
generative adversarial networks (GANs) [57], variational convolutional neural
networks (CNNs) [58, 59, 60, 61], variational recurrent neural networks via
LSTM networks [62], and variational graph neural networks [63, 21, 64].
Further details about their applications are given in [54]. We note that
variational autoencoders also use a probabilistic representation of model
parameters, as opposed to a single value representation in conventional
autoencoders, facilitating uncertainty quantification [65, 66, 67]. Finally,
the importance weighted autoencoder extends the variational autoencoder to use
multiple samples to approximate more complex posteriors [68].

The limitations of MCMC sampling have been addressed with better computational
resources and advanced proposal distributions, incorporating gradients [31,
24, 25]. Hamiltonian MCMC sampling methods have been used for Bayesian neural
networks [69] with enhanced computation strategies [70]. Langevin-based MCMC
methods have been successfully used in the implementation of Bayesian neural
networks for pattern classification and time series prediction problems [71].
A major challenge in MCMC sampling has been in addressing big data and
computationally expensive models; hence, surrogate-assisted estimation, in
which a low-cost surrogate model provides an approximation of the likelihood,
has been used to address this issue [72, 73]. Langevin-based MCMC methods have
also been used in Bayesian neural networks for transfer learning given
multiple sources of data [74]. In previous research, simple Bayesian neural
networks have been used that at most had several thousand model parameters
[72, 74]; hence, the challenge lies in adapting them for autoencoder-based
deep learning models that can feature up to a million parameters.

##  III Methodology

###  III-A Model and Priors

An autoencoder consists of two major parts: an encoder function
$f_{\phi}(\mathbf{x})$ and a decoder function $\bar{f}_{\psi}(\mathbf{h})$,
where $\mathbf{x}$ is the data that represents a set of features, and
$\mathbf{h}$ is a set of reduced (latent) features. The autoencoder is
assessed by how well the decoder can reconstruct the data from the encoder
using a loss function $R_{\text{loss}}$ as given

$R_{\text{loss}}=\operatorname*{arg\,min}_{\phi,\psi}\,\lvert\mathbf{x}-(\bar{f}_{\psi}(f_{\phi}(\mathbf{x}))\rvert^{2}$
(1)

where $\phi$ and $\psi$ represent the parameters (weights and biases) of the
encoder and decoder, respectively.

In particular, $z(\mathbf{x},\theta)$ represents a feedforward neural network,
with $\theta$ being its set of weights and biases. An encoder-decoder pair is
then constructed from it, as given below

$\displaystyle\mathbf{h}$
$\displaystyle=f_{\phi}(\mathbf{x})=z(\mathbf{x},\phi)$ (2)
$\displaystyle\mathbf{x}$
$\displaystyle=\bar{f}_{\psi}(\mathbf{h})=z(\mathbf{h},\psi).$

The encoder extracts the essential features into a reduced representation
while the decoder is used to reconstruct the input from the reduced
representation as shown in Figure 1. The challenge is in determining the
optimal $\theta=(\phi,\psi)$ and typically gradient-based learning algorithms
are used to minimize $R_{\text{loss}}$ [1, 2]. In case the data are labeled,
another way to measure the performance of the autoencoder is by training
another neural network model using the features from the reduced
representation ($\mathbf{h}$), and comparing it with performance when trained
with original data. In contrast to principal component analysis (PCA),
autoencoders are capable of nonlinear data reduction and can detect repetitive
structures in the data [75].

Our likelihood function compares the original data $\mathbf{x}$ with decoder
output $\mathbf{x}^{\prime}=z(\mathbf{x},\theta)=z(z(\mathbf{x},\phi),\psi)$.
We incorporate the $\tau^{2}$ parameter in the model which denotes the
variance of data, $\tau^{2}$ is also estimated; hence, our combined set of
parameters becomes $\Theta=(\theta,\tau^{2})$. We only consider continuous
data for the Bayesian autoencoder, where mean-squared-error (MSE) is used as
the loss function [76]. Hence, we assume the data to have Gaussian
distribution and use a Gaussian likelihood given by

$P(\mathbf{x}|\Theta)=\frac{1}{(2\pi\tau^{2})^{N/2}}\exp\left(-\frac{1}{2\tau^{2}}\sum_{t=1}^{N}\left(y_{t}-\bar{E}(x_{t}|{\bf
x}^{\prime}_{t})\right)^{2}\right),$ (3)

where $\bar{E}(x_{t}|{\bf x^{\prime}}_{t})$ is the output of the neural
network model for $x_{t}$ input features denoted by $t$ instances in the
training data. $N$ is the total number of instances in the training data. The
priors are based on Gaussian distribution for $\theta$ and inverse gamma
distribution for $\tau^{2}$, respectively. Hence, the autoencoder weights and
biases ($\theta$) are independent _a priori_ with a normal distribution with
zero mean and variance $\sigma^{2}$. If $\theta$ features $L$ model parameters
in total, its joint prior with $\tau^{2}$ is

$\bar{P}(\Theta)\propto\frac{1}{(2\pi\sigma^{2})^{L/2}}\exp\Bigg{\\{}-\frac{1}{2\sigma^{2}}\bigg{(}\sum_{l=1}^{L}\theta_{l}^{2}\bigg{)}\Bigg{\\}}\\\
\times\tau^{2(1+\nu_{1})}\exp\left(\frac{-\nu_{2}}{\tau^{2}}\right),$ (4)

where, $\nu_{1}$ and $\nu_{2}$ are user chosen constants. We update $\tau^{2}$
via random-walk proposal distribution and use the log-scale to avoid numerical
instabilities in computing the prior and likelihood.

Figure 1: The autoencoder features the encoder, the decoder which are
highlighted.

###  III-B Langevin-gradient Metropolis–Hastings

We now present the MCMC sampler for training the model parameters (posterior)
of the Bayesian autoencoder. We use a combination of methods: 1.) efficient
proposal distribution that uses Langevin-gradients, 2.) parallel computing,
3.) efficient multimodal sampling via tempered MCMC (parallel tempering). We
utilize the Langevin-gradient proposal distribution that incorporates Gaussian
noise with gradients for a single iteration (sample). We note that Langevin-
gradient proposal distribution has been effective for novel Bayesian neural
networks in relatively small and large neural network architectures of up to
several thousand model parameters [32, 77, 78, 79]; however, in this work we
will experiment with deep autoencoders that feature up to a million model
parameters.

The Langevin-gradient (LG) proposal distribution for a given sample ($n$)
proposes to update the vector of weights and biases $\theta_{n}$ from a biased
multivariate normal distribution

$\theta_{n}^{\star}\sim\mathcal{N}(\theta_{n}+\nu_{3}\times\nabla E(\theta,\bf
x),\nu_{4}^{2}I_{L})$ (5)

where $\nu_{3}$ (learning rate) and $\nu_{4}^{2}$ are user-defined tuning
parameters. $I_{L}$ is an $L\times L$ identity matrix used to generate the
stochastic noise. We take into account the loss function $E(\theta,\bf x)$,
for the autoencoder model $f()$ that features data $\bf x$ as shown below

$E(\theta,\bf x)=\sum_{t\in{\bf x}}(\bf x_{t}-f({\bf x}_{t},\theta))^{2}.$ (6)

Hence, the gradient $\nabla E(\theta,\bf x)$ is given by

$\nabla E(\theta,\bf
x)=\left(\frac{\partial{E}}{\partial{\theta_{1}}},\ldots,\frac{\partial{E}}{\partial{\theta_{L}}}\right).$
(7)

The proposal attempts to “explore” the posterior density, given our prior
$\bar{P}(\Theta)$ and likelihood $P(\mathbf{x}|\Theta)$ defined in Equations
(4) and (3). Furthermore, with
$\Theta^{\star}_{n}=(\theta^{\star}_{n},\tau^{2})$, we either accept or reject
the proposal using the standard Metropolis-Hastings criterion

$\alpha=\min\bigg{\\{}1,\frac{P(\textbf{x}|\Theta_{n}^{\star})\bar{P}(\Theta_{n}^{\star})Q(\Theta_{n}|\Theta_{n}^{\star})}{P(\textbf{x}|\Theta_{n})\bar{P}(\Theta_{n})Q(\Theta_{n}^{\star}|\Theta_{n})}\bigg{\\}},$
(8)

with $Q(\Theta_{n}^{\star}|\Theta_{n})=P(\Theta_{n}^{\star}|\Theta_{n})$, the
conditional proposal density and vice versa. In general,
$\nabla_{\theta}\log\\{\bar{P}(\Theta_{n})P(\mathbf{x}|\Theta_{n})\\}\neq\nabla_{\theta}\log\\{\bar{P}(\Theta_{n}^{\star})P(\mathbf{x}|\Theta_{n}^{\star})\\}$
and so $Q(\Theta_{n}^{\star}|\Theta_{n})\neq\
Q(\Theta_{n}|\Theta_{n}^{\star})$ – an asymmetric proposal, and thus they do
not cancel in Equation (8).

###  III-C Adaptive Langevin-gradient proposal distribution

In practice, in our trial experiments, we found that using the LG proposal
distribution directly produces relatively slow mixing leading to inferior
results for the case or large neural networks such as autoencoders. Hence, we
formulate a new proposal distribution, borrowing ideas from the popular Adam
optimizer [80]. The Adam-based weight update in the conventional learning
paradigm is expressed as follows

$\displaystyle\theta_{k}$ $\displaystyle=\theta_{k-1}-\alpha\times\hat{g}$ (9)
$\displaystyle\hat{g}$
$\displaystyle=\frac{\sqrt{1-\beta_{2}^{k}}}{\sqrt{1-\beta_{1}^{k}}}\cdot\frac{\eta_{k}}{\sqrt{\mu_{k}}+\epsilon}$

where

$\eta_{k}=\beta_{1}\eta_{k-1}+(1-\beta_{1})\times\nabla E(\theta,\bf x)_{k}$
$\mu_{k}=\beta_{2}\mu_{k-1}+(1-\beta_{2})\times\nabla
E(\theta,\bf{x})_{k}^{2}$

where, $\beta_{1}$ and $\beta_{2}$ are first and second moment estimates,
respectively. Moreover, $\epsilon$ is a small scalar used to prevent division
by 0. $\alpha$ is the user defined learning rate and typically, the default
values are $\beta_{1}=0.99$, $\beta_{2}=0.999$, $\alpha=10^{-3}$, and
$\epsilon=10^{-8}$ (implementation in PyTorch
111https://pytorch.org/docs/stable/optim.html).

In our previous work [79], Adam-based gradients have been utilized in adaptive
Langevin gradient (adapt-LG) proposal distributions for convolutional graph
neural networks, which produced better results when compared to LG proposal
distribution. We note that the learning rate is a user-chosen hyperparameter
that is fixed during sampling in the case of LG and adapt-LG proposal
distributions. Stochastic gradient descent (SGD) does not adapt the gradients
while Adam uses a heuristic to adapt the gradients during optimization. Hence,
Adam-based updates make use of earlier values of the gradient, which makes the
Markov transition kernel in MCMC time-inhomogeneous. In our Bayesian
autoencoder framework, we present an enhanced version known as optimization
adapt-LG* which only makes use of adapt-LG in the burn-in phase and switches
to LG afterwards. In this way, we address the inhomogeneous nature of adapt-
LG. Hence, we use an enhanced form of gradients $\hat{E}(\theta,\bf x)$ to
obtain the adapt-LG proposal distribution as follows

$\hat{\theta}_{n}\sim\mathcal{N}(\theta_{n}+\nu_{3}\times\hat{g},\nu_{4}^{2}I_{L})$
(10)

where $\nu_{3}$ is user-defined learning rate previously denoted as $\alpha$.

###  III-D Tempered MCMC Framework

Figure 2: Bayesian autoencoder framework highlighting tempered MCMC utilizing
parallel computing and autoencoder neural network.

The development of _tempered MCMC_ (also called _parallel tempering_ and
_replica exchange MCMC_) has been motivated by the cooling behaviour of
certain materials in the field of thermodynamics [81, 82, 83]. In tempered
MCMC, an ensemble of $M$ replicas $\Omega=[R_{1},R_{2},\dotsc,R_{M}]$, is
executed with a distinct temperature value $T=[1,\dotsc,T_{\text{max}}]$,
where $T_{\text{max}}$ defines the extent of exploration. A replica defined by
temperature $t$, samples from an attenuated posterior distribution
$P_{t}(\Theta_{n}^{t}|\textbf{x})\propto
P(\textbf{x}|\Theta_{n}^{t})^{1/t}\bar{P}(\Theta_{n}^{t})$. This serves to
“flatten” the distribution, giving replicates with high $t$ a generally higher
probability of accepting proposals. The ensemble of replicas facilitates
enhanced exploration capabilities, in particular, it enables a given replica
to escape from a local maximum, allowing multi-modal and discontinuous
posteriors to be explored [84, 85].

Inferentially, this can be viewed as augmenting the sampler state $\Theta_{n}$
into $\Theta_{n}^{t}=(\theta,\tau^{2},t)$ and sampling from them jointly with
those realisations, given $t=1$ features the target distribution. Therefore,
periodically, a proposal can exchange with the states of the neighbouring
replicas, so that $\Theta_{n}^{t\star}=\Theta_{n}^{t^{\prime}}$ and
$\Theta_{n}^{t^{\prime}\star}=\Theta_{n}^{t}$, with $t^{\prime}=t+1$, which is
accepted with the probability

$\beta=\min\bigg{\\{}1,\frac{P(\textbf{x}|\Theta_{n}^{t})^{1/t^{\prime}}P(\textbf{x}|\Theta_{n}^{t^{\prime}})^{1/t}}{P(\textbf{x}|\Theta_{n}^{t^{\prime}})^{1/t^{\prime}}P(\textbf{x}|\Theta_{n}^{t})^{1/t}}\bigg{\\}},$
(11)

since the proposal is symmetric and the priors cancel.

We note that in the temperature ladder $T$, not only the $T_{\text{max}}$ is
important, but also the distribution of the temperature values. Specifically,
for the geometric path between prior and posterior, ideally one should take
temperatures such that the sum of the Kullback-Leibler (KL) divergences
between successive tempered distributions is minimized. However, we use a
geometric temperature ladder that worked well for Bayesian neural networks
from our previous research [77, 79].

###  III-E Bayesian Autoencoder

We now present the details for implementing a tempered MCMC framework for
Bayesian autoencoders that utilize parallel computing. Figure 2 presents the
framework that features the autoencoder model, tempered MCMC replica ensemble,
and the inter-process communication for the exchange of neighboring replica
states.

We present further details in Algorithm 1, where we first define the
autoencoder topology by specifying the number of inputs, hidden layers, and
outputs for the encoder, and the decoder (in reversed order). Secondly, we
define the hyperparameters for the tempered MCMC sampler, such as the number
of replicas ($M$), geometric temperature ladder - maximum temperature
($T_{\text{max}}$), replica swap interval ($R_{\text{swap}}$) that determines
how often to propose an exchange between neighboring replicas, and the maximum
number of samples (iterations) for each replica ($R_{\text{max}}$). In
parallel processing, it is difficult to determine when to stop sampling, and
hence this is done by checking several alive (active) replica processes.
Therefore, we need to set the number of replicas in the ensemble as $alive$;
$alive=M$. These are the key steps in initialization as shown in Stage 0 of
Algorithm 1. The algorithm uses tempered MCMC in its first phase, then
switches to canonical MCMC in its second phase, which is defined by several
samples that needs to switch $R_{\text{switch}}$. During the switch, all the
temperature values in the ladder are changed to 1 as done in our earlier works
[32, 86]. The algorithm implements adapt-LG* proposal distribution (LG in the
first phase, and adapt-LG in the second phase) using $R_{\text{switch}}$.

We then begin within replica sampling (Stage 1.1) by iterating through the
respective replicas, where the manager process executes and manages parallel
processes. In Stage 1.2, each replica sample creates a proposal using either
random-walk or Langevin-gradient (either LG or adapt-LG*) (Equation (5)) and
computes the likelihood (Stage 1.3) and uses Metropolis-Hastings condition in
Stage 1.4 to check if the proposal is good enough to be part of the posterior
distribution. In Stage 1.5, the algorithm checks if it needs to use tempered
MCMC or needs to transform into canonical MCMC where temperature values of all
the replicas in the ensemble are set to 1. Hence, tempered MCMC employs adapt-
LG and canonical MCMC employs LG proposal distribution. Note that the samples
obtained by tempered MCMC are later discarded as part of the burn-in period.
In Stage 2.1, the algorithm computes the likelihood of the replica-exchange
which is similar to the way it’s computed in Equation (11), and then it
exchanges the neighboring replica using the Metropolis-Hastings condition
(Stage 2.2). Note that we represent the likelihood in log-scale to avoid
numerical instabilities.

In Stage 3, the algorithm checks if ($R_{\text{max}}$) has been reached, and
if so, the number of replicas alive is decremented to eventually stop the
entire sampling process. Finally, we reach the end in Stage 4 where the
sampling process ends, and we combine the respective replica posterior
distributions of the Bayesian autoencoder for further analysis.

Rather than swapping the entire replica state (weights and biases in the
autoencoder), it is possible to swap the respective neighboring replica
temperature values, which is cheaper computationally. This is done by simply
swapping neighing replicate temperature values, which is useful in parallel
computing implementation, where inter-process communication can be memory
intensive and computationally costly.

##  IV Experiments and Results

###  IV-A Dataset Description

We select benchmark datasets commonly used in the literature of autoencoders
and data reduction methods. The datasets feature distinct properties in terms
of size, features, and application. These include Madelon [87], Coil-2000 [88]
and Swiss Roll [89] which are summarised in Table I.

Madelon is a synthetic dataset that features data points grouped in 32
clusters, each on a vertex of a five-dimensional hypercube. The clusters are
randomly labeled +1 or -1. In addition to these 5 coordinates, 15 linear
combinations have been added for a total of 20 (redundant) informative
features, as well as 480 random distractors. Coil-2000 has 86 variables
containing customer information, including product usage data and socio-
demographic data derived from postal codes, and whether the customer has a
caravan insurance policy. Lastly, the synthetic Swiss Roll dataset features
data points in the shape of a Swiss Roll given in three dimensions. It is a
popular dataset for visualisation of data reduction methods [90, 91, 92].

TABLE I:  Overview of dataset used for dimensional reduction. Dataset | No. Features | No. Train | No. Test
---|---|---|---
Madelon | 500 | 2000 | 1800
Coil-2000 | 85 | 5822 | 4000
Swiss Roll | 3 | 3750 | 1250

The respective datasets were normalized before model training in the range of
[0,1]. This was done to ensure that all features had equal importance in the
model.

###  IV-B Implementation

Table II provides the Bayesian autoencoder (adapt-LG*) topology that is used
in the experiments described in this section. In the respective datasets, the
input feature size and the feature size of the compressed data are different.
Table II highlights the exact dimensions of the data at the input,
intermediate, and output layers. Table II also demonstrates the sheer volume
in the number of parameters being dealt with and its exponential rise related
to the increase in the number of input features from the data.

We evaluate some of the critical parameters in MCMC sampling to observe the
effects on the sampling performance, computational time, and reconstruction
accuracy. The Bayesian autoencoder is designed to reduce the Madelon dataset
from 500 features to 300. In the case of the Coil 2000 dataset, the Bayesian
autoencoder reduces 85 features to 50 features. In the Swiss Roll dataset, the
Bayesian autoencoder reduces 3 features to 2 features, essentially
“unwrapping” the Swiss Roll, as shown in Figure 3.

TABLE II: Bayesian autoencoder (adapt-LG*) topology showing the total number of parameters (weights and biases). Data | Input Layer | Hidden Layer 1 | Hidden Layer 2 | Hidden Layer 3 | Hidden Layer 4 | Output Layer | Total Parameters
---|---|---|---|---|---|---|---
| (Input–Output) | (Input–Output) | (Input–Output) | (Input–Output) | (Input–Output) | (Input–Output) |
Madelon | 500–450 | 450–400 | 400–300 | 300–400 | 400–450 | 450–500 | 1053701
Coil 2000 | 85–70 | 70–60 | 60–50 | 50–60 | 60–70 | 70–85 | 27037
Swiss Roll | 3–10 | 10–5 | 5–2 | 2–5 | 5–10 | 10–3 | 224

In all the experiments, we use the tempered MCMC sampler hyper-parameters as
follows. In random-walk proposals and the random component of the Langevin
proposal, the Gaussian noise is added with a standard deviation
$\nu_{4}=0.005$, centered at 0. We use $\nu_{1}=0$ and $\nu_{2}=3$ as shown in
the prior distribution given in Equation (4). We use replica temperature
ladder maximum $T_{\text{max}}=2$, and swap interval $R_{\text{swap}}=5$
samples for all experiments. We use $M=8$ replicas with $R_{\text{max}}=6000$
MCMC samples per replica. After $R_{\text{switch}}=3000$ samples, tempered
MCMC converts to canonical MCMC with neighborhood exchange by setting the
replica’s temperature to 1. Note that we chose $R_{\text{max}}$ in trial
experiments by observing the trend and selecting a value, where the accuracy
and likelihood do not have major changes. In terms of $R_{\text{swap}}$, we
selected a value from trial experiments that ensured that a good mixing of the
replicas take place.

Finally, we compare our results with several common classification algorithms:
$K$-nearest neighbors (KNN) [93]; support vector machines (SVM) [94]; random
subspaces ensemble (RSE) [95] method which uses ensemble of decision trees;
and naive Bayes [96]. We implement the Bayesian autoencoder using the pyTorch
library 222pyTorch library: https://pytorch.org/ with Python multi-processing
library 333Python multi-processing library:
https://docs.python.org/3/library/multiprocessing.html for parallel tempered
MCMC replica processes.

###  IV-C Preliminary investigation: parameter tuning

We first carry out a preliminary investigation to understand the effect of
certain parameters on the tempered MCMC sampling and show a visualization of
the results using the Swiss Roll dataset. Table III gives the accuracy for
different Bayesian autoencoder configurations in MCMC sampling with adapt-LG*
proposal distribution for the Swiss Roll dataset. We show the trend of the
reconstruction accuracy (MSE) and acceptance percentage as we change the MCMC
hyper-parameters, i.e. step-size (variance) in proposal distribution, and
learning rate for Langevin-gradients. The highlighted configuration in Table
III is visualised in Figure 3. We also find that the accuracy for the chosen
configuration is close to the canonical autoencoder. However, we note that the
step size of 0.03 with learning rate of 0.04 provides the best performance,
since they provide a balance between accuracy (0.025) and acceptance
performance (19.25 %). In MCMC sampling, only the accuracy is not desired, a
good acceptance rate is also needed to ensure that the sampler has well
explored the posterior distribution. We note that in some models an acceptance
of around 23.40 % [97] is known to be optimum for MCMC sampling.

Figure 3 gives a visual representation of the Swiss Roll dataset for 2500 data
points. It shows the 3D and 2D views of the various stages of the
visualization process. Panel (3(a)) shows the original dataset; Panel (3(b))
shows the reconstructed dataset after reducing and then reconstructing it
through a canonical autoencoder; and Panel (3(c)) uses the same architecture
but the Bayesian autoencoder. The visualization and analysis (Table III, and
Figure 3) demonstrates the effectiveness of the Bayesian autoencoder and its
applicability to larger datasets.

(a) Original

(b) Canonical Autoencoder (MSE=0.012)

(c) Bayesian Autoencoder (MSE=0.010)

Figure 3: Swiss Roll Visualization for different autoencoders  TABLE III: Reconstruction accuracy (MSE) with different settings of tempered MCMC parameters for the two instances of the Swiss Roll dataset using Topology shown in Table II. Method | Topology | MSE | Acceptance Percentage | Step-size | Learning-rate
---|---|---|---|---|---
Canonical Autoencoder (Adam) | Swiss Roll | 0.012 | - | - | -
Bayesian Autoencoder (adapt-LG*) | Swiss Roll | 0.025 | 19.250 | 0.030 | 0.040
Bayesian Autoencoder (adapt-LG*) | Swiss Roll | 0.010 | 0.125 | 0.005 | 0.010
Bayesian Autoencoder (adapt-LG*) | Swiss Roll | 0.104 | 31.630 | 0.080 | 0.090
TABLE IV: Reconstruction accuracy (MSE) as a function of the adapt-LG* rate for the Madelon dataset.  adapt-LG* rate | Train MSE | Test MSE | Swap % | Acceptance % | Time (mins.)
---|---|---|---|---|---
| Best (Mean,Std) | Best (Mean,Std) |  |  |
0 | 0.946 (1.908,0.952) | 0.937 (1.899,0.951) | 78.14 | 1.20 | 184
0.25 | 0.019 (0.024,0.004) | 0.019 (0.025,0.004) | 77.23 | 15.78 | 242
0.5 | 0.011 (0.027,0.004) | 0.013 (0.027,0.004) | 78.39 | 22.68 | 302
0.75 | 0.012 (0.021, 0.007) | 0.013 (0.023,0.006) | 71.57 | 34.64 | 359
1 | 0.012 (0.019, 0.005) | 0.012 (0.021,0.002) | 76.85 | 40.25 | 435
TABLE V: Effect of the number of replicas on Madelon dataset using adapt-LG* proposal distribution. Replicas | Train MSE | Test MSE | Swap % | Acceptance % | Time (mins.)
---|---|---|---|---|---
| Best (Mean,Std) | Best (Mean,Std) |  |  |
2 | 0.012 (0.018, 0.009) | 0.013 (0.019, 0.006) | 27.29 | 32.0 | 564
4 | 0.013 (0.020, 0.01) | 0.015 (0.023, 0.008) | 42.67 | 32.5 | 527
6 | 0.013 (0.014,0.008) | 0.014 (0.016, 0.007) | 56.93 | 32.0 | 483
8 | 0.012 (0.021, 0.007) | 0.013 (0.023,0.006) | 71.57 | 34.64 | 359

###  IV-D Results

We evaluate specific aspects of our proposed Bayesian autoencoder framework
using the Madelon dataset. The Bayesian autoencoder reduces the number of the
features from 500 to 300 features (encoder) and then reconstructs it, i.e.
from the 300 features into the 500 features (decoder). We report the accuracy
between the reconstructed dataset and the original dataset. We first present
results in terms of computational efficiency, the effect of adapt-LG* rate,
and several replicas ($M$) for the Madelon dataset. We note that the Langevin-
gradient is for one iteration or epoch which is defined by a single forward
and backward pass via backpropagation.

Table IV shows the effect of adapt-LG* rate ($g_{\text{prob}}$) on the
performance of the Bayesian autoencoder given by training and testing accuracy
(MSE) in terms of best, mean, and standard deviation (std). Since all of these
are sampling from the same posterior, the accuracy should be the same as well.
However, we find that without the adapt-LG* (when the rate is 0), the sampler
cannot find the posterior mode. On the other hand, adapt-LG* proposals are
much more computationally costly as given by the time in minutes (mins.).

Next, we evaluate the effect of the number of replicas ($M$) using the
Bayesian autoencoder using adapt-LG* proposal distribution. Table V summarises
the results where we observe a lower computational time with a similar
accuracy (MSE) as the number of replicas increases, suggesting that parallel
processing is useful in this case.

Figure 4 shows trace plots over the samples with training and testing accuracy
for one of the selected replicas of the Bayesian autoencoder (adapt-LG*),
demonstrating convergence to the neighborhood of a posterior mode. We present
a quantitative assessment of convergence in Table VI in the form of
Gelman–Rubin [98] diagnostics. This diagnostic works by comparing the variance
between replicates to the variance within them. A ratio sufficiently close to
1 indicates convergence to the MCMC sampler’s stationary distribution – the
posterior [98]. Figure 5 presents the log-likelihood of selected replicas
(temperature level of 1.0) of the Bayesian autoencoder (adapt-LG*) for the
respective problems. Figure 6 shows the trace plots and posterior distribution
for the selected weights of the Bayesian autoencoder (adapt-LG*) with the
convergence diagnostic given in Table VI.

The data compression by autoencoders is typically bench-marked by applying
machine learning methods on the reduced dataset. We trained the Bayesian
autoencoder and obtain a reduced dataset. Next, we train another machine
learning model for the task of classification with reduced data obtained from
the Bayesian autoencoder as done in previous studies [99, 100, 95, 101, 96,
76]. In this way, we can compare the performance by training the machine
learning model on the reduced dataset with that of the original dataset.

Table VII presents a comparison of the results to selected methods from the
literature that use the compressed datasets for classification tasks. We note
that caution should be exercised when making direct comparisons, because the
autoencoder architectures may vary somewhat between papers. To the extent that
these comparisons are valid, our Bayesian autoencoder combined with SVM
outperforms the classification accuracy given in most methods from literature
for the respective datasets. This shows the effectiveness of the Bayesian
autoencoder as an alternative for dimensionality reduction and motivates the
use of autoencoders as a possible data pre-processing step to increase
classification accuracy.

Table VIII compares the canonical autoencoder (using SGD and Adam optimizer)
with the Bayesian autoencoder (using LG and adapt-LG*). We show the best,
mean, and the standard deviation (std) of the reconstruction accuracy (MSE).
As evident in Table VII, we find that the canonical autoencoder with SGD
optimizer does not perform well for the respective datasets. We also find that
the Bayesian autoencoder with adapt-LG* performs better than LG for all three
datasets.

TABLE VI: Gelman–Rubin MCMC convergence diagnostics for the weight chains in parallel tempering Dataset | ID-0 | ID-50 | ID-100 | ID-150 | ID-2000 | ID-3000 | ID-4000 | ID-6000 | ID-9000 | ID-10,000
---|---|---|---|---|---|---|---|---|---|---
Madelon | 1.15 | 1.14 | 1.15 | 1.21 | 1.16 | 1.13 | 1.18 | 1.25 | 1.13 | 1.25
Coil 2000 | 1.14 | 1.16 | 1.18 | 1.23 | 1.15 | 1.17 | 1.11 | 1.24 | 1.12 | 1.13
Swiss Roll | 1.14 | 1.24 | 1.13 | 1.16 | - | - | - | - | - | -
TABLE VII:  Comparison of Bayesian autoencoder with established methods from literature.  Method | Algorithm | Madelon | Coil 2000
---|---|---|---
|  | Best (Mean, Std) | Best (Mean, Std)
Autoencoder based classifier [99] | kNN | 0.547 (-,-) | 0.929 (-,-)
Ensemble Classification using Dimensionality Reduction [95] | RSE | 0.5565 (- , 0.0263) | -
Autoencoder inspired unsupervised feature selection [101] | kNN | 0.71 (-,-) | -
Multi-objective Evolutionary Approach [96] | Naive Bayes | - | 0.93 ( -, 0.014)
Multi-objective Evolutionary Approach [96] | SVM | - | 0.948 ( -, 0.002)
Classifier (without compression) | SVM | 0.6 (0.56, 0.02 ) | 0.96 (0.95, 0.005)
Bayesian-Autoencoder (LG) | kNN | 0.49 (0.45, 0.01 ) | 0.89 (0.81, 0.05)
Bayesian-Autoencoder (adapt-LG*) | kNN | 0.554 (0.523, 0.018 ) | 0.958 (0.92, 0.015 )
Bayesian-Autoencoder (adapt-LG*) | SVM | 0.589 (0.554, 0.021 ) | 0.96 (0.94, 0.009 )
TABLE VIII:  Autoencoder reconstruction accuracy (MSE) for the three datasets.  Method | Swiss Roll | Madelon | Coil 2000
---|---|---|---
| Best (Mean, Std) | Best (Mean, Std) | Best (Mean, Std)
Canonical Autoencoder (SGD) | 0.084 (0.087, 0.01) | 0.0389 (0.042,0.02) | 0.044 (0.0469, 0.03)
Canonical Autoencoder (Adam) | 0.012 (0.019, 0.005) | 0.0192 (0.0199, 0.007) | 0.0117 (0.014,0.002)
Bayesian Autoencoder (LG) | 0.020 (0.03,0.016) | 0.050 (0.060,0.003) | 0.033 (0.036,0.001)
Bayesian Autoencoder (adapt-LG*) | 0.010 (0.014,0.063) | 0.013 (0.023,0.006) | 0.016 (0.023,0.002)

(a) Madelon

(b) Coil 2000

(c) Swiss Roll

Figure 4: Reconstruction accuracy (MSE) over time (samples) for selected
replica (temperature=1).

(a) Madelon

(b) Coil 2000

(c) Swiss Roll

Figure 5: Log-Likelihood for replica with temperature=1

(a) Madelon

(b) Coil 2000

(c) Swiss Roll

Figure 6: Posterior and trace plot for selected weights for the respective
problems.

##  V Discussion

Our primary goal was to incorporate Bayesian inference via MCMC sampling into
autoencoders. The proposed Bayesian autoencoder provides an approach for
uncertainty quantification in the reduced dataset. We can reconstruct an
ensemble of reduced datasets from the posterior distribution, rather than a
single one, given by single-point estimates. The results show that the
Bayesian autoencoder framework can efficiently sample the posterior
distribution of the autoencoder’s weights and biases, which are compared very
well with canonical methods in terms of accuracy (such as SGD and Adam). We
also found that our approach is competitive with the other methods found in
the literature.

It is unusual to use MCMC methods for deep learning, which often features
models with tens of thousands of parameters; however, the field is slowly
gaining momentum [71]. We have shown that with parallel computing and
sophisticated gradient-based proposal distributions in MCMC, it is feasible
for them to be used, nonetheless. We finally demonstrated how MCMC diagnostics
apply to this problem (Table 6).

One limitation is that convergence analyses for MCMC samplers, such as the
Gelman-Rubin diagnostics are generally designed for modest numbers of
parameters. More work is needed to develop convergence diagnostic methods that
are more suited for deep learning models that feature tens of thousands to
millions of parameters. Furthermore, in large deep learning models, not all
nodes and weights are equally valuable. We note that a significant portion of
these model parameters (weights) are not contributing, and hence they can be
pruned during or post sampling (training). Optimization and evolutionary
computation methods have been used for pruning weights of large deep learning
models; hence, there is potential for Bayesian inference methods to be used in
model pruning. The future research can focus on determining and
differentiating essential weights from non-essential ones using specific
metrics to determine which weights have a major effect on the performance of
the model. Furthermore, a Bayesian framework can be developed to prune the
non-essential weights that do not affect the performance of the model to build
a robust and compact Bayesian autoencoder.

We further highlight that we utilized uncorrelated multivariate Gaussian
distributions as prior distribution for the autoencoder model parameters
(weights and biases). Although we consider previous research for selecting
this prior distribution, we note that our previous research used different
neural network architectures, i.e. simple neural networks [32], and graph CNNs
[79]. The recent developments on prior elicitation in Bayesian deep learning
have spread the interest in more general choices [102]. The choice of prior is
a fundamental issue when the interest lies in performing Bayesian model
selection [103]. Further extensions of the proposed Bayesian autoencoder can
consider current developments of priors in the field.

##  VI Conclusions

We presented a framework that employs tempered MCMC sampling for the Bayesian
autoencoder which is an alternative to the variational autoencoder. We
addressed some of the known limitations of MCMC methods for Bayesian deep
learning with the power of parallel computing and enhanced proposal
distributions. Our results indicate that the Bayesian framework is as good as
related methods from the literature in terms of performance accuracy, with the
additional feature of robust model uncertainty quantification for the
generation of reduced datasets. The paper motivates further applications of
the Bayesian framework for other deep learning models, such as the generative
adversarial networks. Furthermore, there is also scope for domain-specific
applications such as remote sensing and geoscience applications where robust
uncertainty quantification is vital.

## Code and Data

We provide the code as open-source software via our GitHub repository
444https://github.com/sydney-machine-learning/BayesianAutoencoders.

## References

  * [1] P. Baldi, “Autoencoders, unsupervised learning, and deep architectures,” in _Proceedings of ICML workshop on unsupervised and transfer learning_ , 2012, pp. 37–49.
  * [2] C. Doersch, “Tutorial on variational autoencoders,” _arXiv preprint arXiv:1606.05908_ , 2016.
  * [3] W. Luo, J. Li, J. Yang, W. Xu, and J. Zhang, “Convolutional sparse autoencoders for image classification,” _IEEE transactions on neural networks and learning systems_ , vol. 29, no. 7, pp. 3289–3294, 2018.
  * [4] P. Zhou, J. Han, G. Cheng, and B. Zhang, “Learning compact and discriminative stacked autoencoder for hyperspectral image classification,” _IEEE Transactions on Geoscience and Remote Sensing_ , vol. 57, no. 7, pp. 4823–4833, 2019.
  * [5] S. Gao, Y. Zhang, K. Jia, J. Lu, and Y. Zhang, “Single sample face recognition via learning deep supervised autoencoders,” _IEEE transactions on information forensics and security_ , vol. 10, no. 10, pp. 2108–2118, 2015.
  * [6] M. Kan, S. Shan, H. Chang, and X. Chen, “Stacked progressive auto-encoders (spae) for face recognition across poses,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2014, pp. 1883–1890.
  * [7] G. Dong, G. Liao, H. Liu, and G. Kuang, “A review of the autoencoder and its variants: A comparative perspective from target recognition in synthetic-aperture radar images,” _IEEE Geoscience and Remote Sensing Magazine_ , vol. 6, no. 3, pp. 44–68, 2018.
  * [8] S. Sahu, R. Gupta, G. Sivaraman, W. AbdAlmageed, and C. Espy-Wilson, “Adversarial auto-encoders for speech based emotion recognition,” _arXiv preprint arXiv:1806.02146_ , 2018.
  * [9] A. Ranjan, T. Bolkart, S. Sanyal, and M. J. Black, “Generating 3d faces using convolutional mesh autoencoders,” in _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2018, pp. 704–720.
  * [10] J. Masci, U. Meier, D. Cireşan, and J. Schmidhuber, “Stacked convolutional auto-encoders for hierarchical feature extraction,” in _International conference on artificial neural networks_. Springer, 2011, pp. 52–59.
  * [11] J. Deng, Z. Zhang, E. Marchi, and B. Schuller, “Sparse autoencoder-based feature transfer learning for speech emotion recognition,” in _2013 Humaine Association Conference on Affective Computing and Intelligent Interaction_. IEEE, 2013, pp. 511–516.
  * [12] G. Alain and Y. Bengio, “What regularized auto-encoders learn from the data-generating distribution,” _The Journal of Machine Learning Research_ , vol. 15, no. 1, pp. 3563–3593, 2014.
  * [13] D. P. Kingma and M. Welling, “Auto-encoding variational bayes,” _arXiv preprint arXiv:1312.6114_ , 2013.
  * [14] C. K. Sønderby, T. Raiko, L. Maaløe, S. K. Sønderby, and O. Winther, “Ladder variational autoencoders,” in _Advances in neural information processing systems_ , 2016, pp. 3738–3746.
  * [15] I. Tolstikhin, O. Bousquet, S. Gelly, and B. Schoelkopf, “Wasserstein auto-encoders,” _arXiv preprint arXiv:1711.01558_ , 2017.
  * [16] D. Liang, R. G. Krishnan, M. D. Hoffman, and T. Jebara, “Variational autoencoders for collaborative filtering,” in _Proceedings of the 2018 World Wide Web Conference_ , 2018, pp. 689–698.
  * [17] J. Zhao, Y. Kim, K. Zhang, A. Rush, and Y. LeCun, “Adversarially regularized autoencoders,” in _International Conference on Machine Learning_. PMLR, 2018, pp. 5902–5911.
  * [18] L. Beggel, M. Pfeiffer, and B. Bischl, “Robust anomaly detection in images using adversarial autoencoders,” in _Joint European Conference on Machine Learning and Knowledge Discovery in Databases_. Springer, 2019, pp. 206–222.
  * [19] S. Pidhorskyi, R. Almohsen, and G. Doretto, “Generative probabilistic novelty detection with adversarial autoencoders,” in _Advances in neural information processing systems_ , 2018, pp. 6822–6833.
  * [20] A. Creswell and A. A. Bharath, “Denoising adversarial autoencoders,” _IEEE transactions on neural networks and learning systems_ , vol. 30, no. 4, pp. 968–984, 2018.
  * [21] T. N. Kipf and M. Welling, “Variational graph auto-encoders,” _arXiv preprint arXiv:1611.07308_ , 2016.
  * [22] Y. Ding, L.-P. Tian, X. Lei, B. Liao, and F.-X. Wu, “Variational graph auto-encoders for mirna-disease association prediction,” _Methods_ , vol. 192, 08 2020.
  * [23] D. J. MacKay, “Probable networks and plausible predictions—a review of practical Bayesian methods for supervised neural networks,” _Network: Computation in Neural Systems_ , vol. 6, no. 3, pp. 469–505, 1995.
  * [24] R. M. Neal, _Bayesian learning for neural networks_. Springer Science & Business Media, 2012, vol. 118.
  * [25] R. M. Neal _et al._ , “MCMC using Hamiltonian dynamics,” _Handbook of Markov chain Monte Carlo_ , vol. 2, no. 11, pp. 113–124, 2011\.
  * [26] D. J. MacKay, “A practical Bayesian framework for backpropagation networks,” _Neural computation_ , vol. 4, no. 3, pp. 448–472, 1992.
  * [27] G. E. Hinton and D. Van Camp, “Keeping the neural networks simple by minimizing the description length of the weights,” in _Proceedings of the sixth annual conference on Computational learning theory_. ACM, 1993, pp. 5–13.
  * [28] A. Graves, “Practical variational inference for neural networks,” in _Advances in neural information processing systems_ , 2011, pp. 2348–2356.
  * [29] G. O. Roberts and R. L. Tweedie, “Exponential convergence of langevin distributions and their discrete approximations,” _Bernoulli_ , pp. 341–363, 1996.
  * [30] G. O. Roberts and J. S. Rosenthal, “Optimal scaling of discrete approximations to langevin diffusions,” _Journal of the Royal Statistical Society: Series B (Statistical Methodology)_ , vol. 60, no. 1, pp. 255–268, 1998.
  * [31] M. Welling and Y. W. Teh, “Bayesian learning via stochastic gradient langevin dynamics,” in _Proceedings of the 28th International Conference on Machine Learning (ICML-11)_ , 2011, pp. 681–688.
  * [32] R. Chandra, K. Jain, R. V. Deo, and S. Cripps, “Langevin-gradient parallel tempering for Bayesian neural learning,” _Neurocomputing_ , vol. 359, pp. 315 – 326, 2019.
  * [33] R. Chandra and A. Kapoor, “Bayesian neural multi-source transfer learning,” _Neurocomputing_ , vol. 378, pp. 54 – 64, 2020.
  * [34] R. Bassett and J. Deride, “Maximum a posteriori estimators as a limit of bayes estimators,” _Mathematical Programming_ , vol. 174, no. 1, pp. 129–144, 2019\.
  * [35] B. L. Betechuoh, T. Marwala, and T. Tettey, “Autoencoder networks for hiv classification,” _Current Science_ , pp. 1467–1473, 2006.
  * [36] D. D’Avino, D. Cozzolino, G. Poggi, and L. Verdoliva, “Autoencoder with recurrent neural networks for video forgery detection,” _Electronic Imaging_ , vol. 2017, no. 7, pp. 92–99, 2017.
  * [37] F. Zhao, J. Feng, J. Zhao, W. Yang, and S. Yan, “Robust lstm-autoencoders for face de-occlusion in the wild,” _IEEE Transactions on Image Processing_ , vol. 27, no. 2, pp. 778–790, 2017.
  * [38] A. Taheri, “Learning graph representations with recurrent neural network autoencoders,” 2018.
  * [39] H. Pardede, E. Suryawati, R. Sustika, and V. Zilvan, “Unsupervised convolutional autoencoder-based feature learning for automatic detection of plant diseases,” 11 2018, pp. 158–162.
  * [40] F.-N. Yuan, L. Zhang, J.-T. Shi, X. Xia, and G. Li, “Theories and applications of auto-encoder neural networks: A literature survey,” _Jisuanji Xuebao/Chinese Journal of Computers_ , vol. 42, pp. 203–230, 01 2019.
  * [41] X. Zhao, M. Jia, and M. Lin, “Deep laplacian auto-encoder and its application into imbalanced fault diagnosis of rotating machinery,” _Measurement_ , vol. 152, p. 107320, 2020.
  * [42] Z. Ji, Y. Zhao, Y. Pang, and X. Li, “Cross-modal guidance based auto-encoder for multi-video summarization,” _Pattern Recognition Letters_ , vol. 135, pp. 131 – 137, 2020.
  * [43] C. Zihao, L. Xiao-li, and L. Fei, “Near-infrared fault detection based on stacked regularized auto-encoder network,” _Chemometrics and Intelligent Laboratory Systems_ , vol. 204, p. 104101, 2020.
  * [44] B. Zhang, H. Zhang, G. Zhao, and J. Lian, “Constructing a PM2.5 concentration prediction model by combining auto-encoder with Bi-LSTM neural networks,” _Environmental Modelling & Software_, vol. 124, p. 104600, 2020.
  * [45] R. Li, X. Wang, W. Quan, Y. Song, and L. Lei, “Robust and structural sparsity auto-encoder with L21-norm minimization,” _Neurocomputing_ , vol. 425, 02 2020.
  * [46] M. Yousefi-Azar, V. Varadharajan, L. Hamey, and U. Tupakula, “Autoencoder-based feature learning for cyber security applications,” in _2017 International joint conference on neural networks (IJCNN)_. IEEE, 2017, pp. 3854–3861.
  * [47] L. Gondara, “Medical image denoising using convolutional denoising autoencoders,” in _2016 IEEE 16th international conference on data mining workshops (ICDMW)_. IEEE, 2016, pp. 241–246.
  * [48] G. Dong, G. Liao, H. Liu, and G. Kuang, “A review of the autoencoder and its variants: A comparative perspective from target recognition in synthetic-aperture radar images,” _IEEE Geoscience and Remote Sensing Magazine_ , vol. 6, pp. 44–68, 09 2018.
  * [49] M. Lingheng, S. Ding, and Y. Xue, “Research on denoising sparse autoencoder,” _International Journal of Machine Learning and Cybernetics_ , vol. 8, 10 2017\.
  * [50] V. Kumar, G. Nandi, and R. Kala, “Static hand gesture recognition using stacked denoising sparse autoencoders,” 08 2014, pp. 99–104.
  * [51] S. Rifai, P. Vincent, X. Muller, X. Glorot, and Y. Bengio, “Contractive auto-encoders: Explicit invariance during feature extraction,” 2011, pp. 833–840.
  * [52] S. Ryu, H. Choi, H. Lee, and H. Kim, “Convolutional autoencoder based feature extraction and clustering for customer load analysis,” _IEEE Transactions on Power Systems_ , vol. PP, pp. 1–1, 08 2019.
  * [53] S. Wang, H. Chen, L. Wu, and J. Wang, “A novel smart meter data compression method via stacked convolutional sparse auto-encoder,” _International Journal of Electrical Power & Energy Systems_, vol. 118, p. 105761, 2020.
  * [54] H. Wang and D.-Y. Yeung, “A survey on Bayesian deep learning,” _ACM Computing Surveys (CSUR)_ , vol. 53, no. 5, pp. 1–37, 2020.
  * [55] N. G. Polson, V. Sokolov _et al._ , “Deep learning: A Bayesian perspective,” _Bayesian Analysis_ , vol. 12, no. 4, pp. 1275–1304, 2017\.
  * [56] M. J. Kusner, B. Paige, and J. M. Hernández-Lobato, “Grammar variational autoencoder,” in _International Conference on Machine Learning_. PMLR, 2017, pp. 1945–1954.
  * [57] L. Mescheder, S. Nowozin, and A. Geiger, “Adversarial variational Bayes: Unifying variational autoencoders and generative adversarial networks,” _arXiv preprint arXiv:1701.04722_ , 2017.
  * [58] Y. Gal and Z. Ghahramani, “Bayesian convolutional neural networks with bernoulli approximate variational inference,” _arXiv preprint arXiv:1506.02158_ , 2015.
  * [59] C. Zhao, B. Ni, J. Zhang, Q. Zhao, W. Zhang, and Q. Tian, “Variational convolutional neural network pruning,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 2780–2789.
  * [60] J.-T. Chien, K.-T. Kuo _et al._ , “Variational recurrent neural networks for speech separation,” in _18th Annual Conference of the International Speech Communication Association (INTERSPEECH 2017), Volume 1-6: Situated Interaction_ , 2017, pp. 1193–1197.
  * [61] X. Zhou, Y. Hu, W. Liang, J. Ma, and Q. Jin, “Variational lstm enhanced anomaly detection for industrial big data,” _IEEE Transactions on Industrial Informatics_ , vol. PP, pp. 1–1, 09 2020.
  * [62] J. Su, S. Wu, D. Xiong, Y. Lu, X. Han, and B. Zhang, “Variational recurrent neural machine translation,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 32, no. 1, 2018.
  * [63] S. Bonner, A. Atapour-Abarghouei, P. T. Jackson, J. Brennan, I. Kureshi, G. Theodoropoulos, A. S. McGough, and B. Obara, “Temporal neighbourhood aggregation: Predicting future links in temporal graphs via recurrent variational graph convolutions,” in _2019 IEEE International Conference on Big Data (Big Data)_. IEEE, 2019, pp. 5336–5345.
  * [64] M. Qu, Y. Bengio, and J. Tang, “GMNN: Graph Markov neural networks,” in _International conference on machine learning_. PMLR, 2019, pp. 5241–5250.
  * [65] D. Kingma and M. Welling, “An introduction to variational autoencoders,” _Foundations and Trends® in Machine Learning_ , vol. 12, pp. 307–392, 01 2019.
  * [66] C. Xu, Y. Dai, R. Lin, and S. Wang, “Deep clustering by maximizing mutual information in variational auto-encoder,” _Knowledge-Based Systems_ , vol. 205, p. 106260, 2020.
  * [67] J. Zhu, H. Shi, B. Song, Y. Tao, and S. Tan, “Information concentrated variational auto-encoder for quality-related nonlinear process monitoring,” _Journal of Process Control_ , vol. 94, pp. 12 – 25, 2020.
  * [68] Y. Burda, R. Grosse, and R. Salakhutdinov, “Importance weighted autoencoders,” _arXiv preprint arXiv:1509.00519_ , 2015.
  * [69] D. Levy, M. D. Hoffman, and J. Sohl-Dickstein, “Generalizing Hamiltonian Monte Carlo with neural networks,” _arXiv preprint arXiv:1711.09268_ , 2017\.
  * [70] A. D. Cobb and B. Jalaian, “Scaling Hamiltonian Monte Carlo inference for Bayesian neural networks with symmetric splitting,” _arXiv preprint arXiv:2010.06772_ , 2020.
  * [71] R. Chandra, K. Jain, R. V. Deo, and S. Cripps, “Langevin-gradient parallel tempering for Bayesian neural learning,” _Neurocomputing_ , vol. 359, pp. 315–326, 2019.
  * [72] R. Chandra, K. Jain, A. Kapoor, and A. Aman, “Surrogate-assisted parallel tempering for Bayesian neural learning,” _Engineering Applications of Artificial Intelligence_ , vol. 94, p. 103700, 2020.
  * [73] F. Llorente, L. Martino, J. Read, and D. Delgado, “A survey of Monte Carlo methods for noisy and costly densities with application to reinforcement learning,” _arXiv preprint arXiv:2108.00490_ , 2021.
  * [74] R. Chandra and A. Kapoor, “Bayesian neural multi-source transfer learning,” _Neurocomputing_ , vol. 378, pp. 54–64, 2020.
  * [75] Y. Wang, H. Yao, and S. Zhao, “Auto-encoder based dimensionality reduction,” _Neurocomputing_ , vol. 184, pp. 232–242, 2016.
  * [76] Y. Bozkurt, T. Bereau, and J. Rudzinski, “Interpretable embeddings from molecular simulations using gaussian mixture variational autoencoders,” _Machine Learning: Science and Technology_ , vol. 1, 03 2020.
  * [77] R. Chandra and A. Kapoor, “Bayesian neural multi-source transfer learning,” _Neurocomputing_ , vol. 378, pp. 54–64, 2020.
  * [78] R. Chandra, K. Jain, A. Kapoor, and A. Aman, “Surrogate-assisted parallel tempering for Bayesian neural learning,” _Eng. Appl. Artif. Intell._ , vol. 94, p. 103700, 2020.
  * [79] R. Chandra, A. Bhagat, M. Maharana, and P. Krivitsky, “Bayesian graph convolutional neural networks via tempered MCMC,” _IEEE Access_ , vol. PP, pp. 1–1, 09 2021.
  * [80] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” _arXiv preprint arXiv:1412.6980_ , 2014.
  * [81] R. H. Swendsen and J.-S. Wang, “Replica Monte Carlo simulation of spin-glasses,” _Physical review letters_ , vol. 57, no. 21, p. 2607, 1986\.
  * [82] K. Hukushima and K. Nemoto, “Exchange Monte Carlo method and application to spin glass simulations,” _Journal of the Physical Society of Japan_ , vol. 65, no. 6, pp. 1604–1608, 1996.
  * [83] U. H. Hansmann, “Parallel tempering algorithm for conformational studies of biological molecules,” _Chemical Physics Letters_ , vol. 281, no. 1-3, pp. 140–150, 1997.
  * [84] M. K. Sen and P. L. Stoffa, “Bayesian inference, Gibbs’ sampler and uncertainty estimation in geophysical inversion,” _Geophysical Prospecting_ , vol. 44, no. 2, pp. 313–350, 1996.
  * [85] M. Maraschini and S. Foti, “A Monte Carlo multimodal inversion of surface waves,” _Geophysical Journal International_ , vol. 182, no. 3, pp. 1557–1566, 2010.
  * [86] R. Chandra, R. D. Müller, D. Azam, R. Deo, N. Butterworth, T. Salles, and S. Cripps, “Multicore parallel tempering Bayeslands for basin and landscape evolution,” _Geochemistry, Geophysics, Geosystems_ , vol. 20, no. 11, pp. 5082–5104, 2019.
  * [87] I. Guyon, S. Gunn, A. Ben-Hur, and G. Dror, “Result analysis of the NIPS 2003 feature selection challenge,” _Advances in neural information processing systems_ , vol. 17, 2004.
  * [88] P. V. D. Putten and M. V. Someren, “A bias-variance analysis of a real world learning problem: The coil challenge 2000,” _Machine Learning_ , vol. 57, no. 1/2, p. 177–195, 2004.
  * [89] D. Surendran, “Swiss roll dataset,” 2004. [Online]. Available: http://people.cs.uchicago.edu/~dinoj/manifold/swissroll.html
  * [90] S. Das and N. R. Pal, “Nonlinear dimensionality reduction for data visualization: An unsupervised fuzzy rule-based approach,” _IEEE Transactions on Fuzzy Systems_ , 2021. [Online]. Available: https://doi.org/10.1109/TFUZZ.2021.3076583
  * [91] L. Van Der Maaten, E. Postma, and J. Van den Herik, “Dimensionality reduction: a comparative review,” _J Mach Learn Res_ , vol. 10, pp. 66–71, 2009.
  * [92] D. Peluffo, J. Salazar Castro, and J. Alvarado Pérez, “Interactive interface for efficient data visualization via a geometric approach,” 09 2015.
  * [93] Z. Zhang, “Introduction to machine learning: K-nearest neighbors,” _Annals of Translational Medicine_ , vol. 4, pp. 218–218, 06 2016.
  * [94] M. A. Hearst, S. T. Dumais, E. Osuna, J. Platt, and B. Scholkopf, “Support vector machines,” _IEEE Intelligent Systems and their applications_ , vol. 13, no. 4, pp. 18–28, 1998.
  * [95] A. Schclar, L. Rokach, and A. Amit, “Ensembles of classifiers based on dimensionality reduction,” _Intelligent Data Analysis_ , vol. 21, 05 2013\.
  * [96] D. Singh, D. S. Sisodia, and P. Singh, “Multi-objective evolutionary approach for the performance improvement of learners using ensembling feature selection and discretization technique on medical data,” 2020.
  * [97] M. Bédard, “Optimal acceptance rates for metropolis algorithms: Moving beyond 0.234,” _Stochastic Processes and their Applications_ , vol. 118, no. 12, pp. 2198–2222, 2008.
  * [98] D. Vats and C. Knudson, “Revisiting the Gelman–Rubin diagnostic,” _Statistical Science_ , vol. 36, no. 4, pp. 518–529, 2021.
  * [99] F. Pulgar, F. Charte, A. Rivera Rivas, and M. J. Del Jesus, “Aeknn: An autoencoder knn–based classifier with built-in dimensionality reduction,” _International Journal of Computational Intelligence Systems_ , vol. 12, 02 2018.
  * [100] P. Rauber, A. Falcão, and A. Telea, “Projections as visual aids for classification system design,” _Information Visualization_ , vol. 17, p. 147387161771333, 06 2017.
  * [101] K. Han, Y. Wang, C. Zhang, and C. Li, “Autoencoder inspired unsupervised feature selection,” 04 2018, pp. 2941–2945.
  * [102] V. Fortuin, “Priors in Bayesian deep learning: A review,” _arXiv preprint arXiv:2105.06868_ , 2021.
  * [103] F. Llorente, L. Martino, D. Delgado, and J. Lopez-Santiago, “Marginal likelihood computation for model selection and hypothesis testing: an extensive review,” _arXiv preprint arXiv:2005.08334_ , 2020.


10ptrevtex4-2

# The Hund’s Superconductor Li(Fe,Co)As

H. Miao<EMAIL_ADDRESS>Materials Science and Technology Division, Oak Ridge
National Laboratory, Oak Ridge, Tennessee 37831, USA  Y. L. Wang  Condensed
Matter Physics and Materials Science Department, Brookhaven National
Laboratory, Upton, New York 11973, USA  J.-X. Yin  Laboratory for Topological
Quantum Matter and Advanced Spectroscopy (B7), Department of Physics,
Princeton, New Jersey 08544, USA  J. Zhang  Materials Science and Technology
Division, Oak Ridge National Laboratory, Oak Ridge, Tennessee 37831, USA  S.
Zhang  Laboratory for Topological Quantum Matter and Advanced Spectroscopy
(B7), Department of Physics, Princeton, New Jersey 08544, USA  M. Z. Hasan
Laboratory for Topological Quantum Matter and Advanced Spectroscopy (B7),
Department of Physics, Princeton, New Jersey 08544, USA  R. Yang  Laboratorium
f$\ddot{u}$r Festk$\ddot{o}$rphysik ETH-Z$\ddot{u}$rich, 8093 Z$\ddot{u}$rich,
Switzerland  X. C. Wang  Beijing National Laboratory for Condensed Matter
Physics, Institute of Physics, Chinese Academy of Sciences, Beijing 100190,
China  School of Physical Sciences, University of Chinese Academy of Sciences,
Beijing 100190, China  C. Q. Jin  Beijing National Laboratory for Condensed
Matter Physics, Institute of Physics, Chinese Academy of Sciences, Beijing
100190, China  School of Physical Sciences, University of Chinese Academy of
Sciences, Beijing 100190, China  T. Qian  Beijing National Laboratory for
Condensed Matter Physics, Institute of Physics, Chinese Academy of Sciences,
Beijing 100190, China  School of Physical Sciences, University of Chinese
Academy of Sciences, Beijing 100190, China  H. Ding  Beijing National
Laboratory for Condensed Matter Physics, Institute of Physics, Chinese Academy
of Sciences, Beijing 100190, China  School of Physical Sciences, University of
Chinese Academy of Sciences, Beijing 100190, China  H.-L. Lee  Materials
Science and Technology Division, Oak Ridge National Laboratory, Oak Ridge,
Tennessee 37831, USA  G. Kotliar  Physics and Astronomy Department, Rutgers
University, Piscataway, New Jersey 08854, USA  Condensed Matter Physics and
Materials Science Department, Brookhaven National Laboratory, Upton, New York
11973, USA

(August 27, 2024)

###### Abstract

We combine transport, angle-resolved photoemission spectroscopy and scanning
tunneling spectroscopy to investigate several low energy manifestations of the
Hund coupling in a canonical FeSC family Li(Fe,Co)As. We determine the doping
dependence of the coherent-incoherent crossover temperature and the quasi-
particle effective mass enhancement in the normal state. Our tunnelling
spectroscopy result in the superconducting state supports the idea that
superconductivity emerging from Hund’s metal state displays a universal
maximal superconducting gap vs transition temperature
(2$\Delta_{max}/k_{B}T_{c}$) value, which is independent of doping level and
$T_{c}$.

A decade after the first discovery of the high-$T_{c}$ iron-based
superconductors (FeSCs), the pairing mechanism and the interplay between the
normal and superconducting state in these materials remain elusive Kamihara
_et al._ (2008); Hirschfeld  _et al._ (2011); Scalapino (2012). While the
phase diagram of the FeSCs shares some similarities with the cuprate
high-$T_{c}$ superconductors, FeSCs are multi-orbital semimetals rather than
doped Mott insulators with an effective single orbital Zhang and Rice (1988).
The increased orbital degree of freedom is found to qualitatively change the
underlying electronic dynamics Misawa  _et al._ (2012); Ikeda  _et al._
(2010). In particular, dynamical mean-field theory (DMFT) studies have shown
that strong electron-electron correlations in the FeSCs originate from the
inter-orbital Hund’s coupling, $J_{H}$, rather than the on-site intra-orbital
Coulomb interaction, $U$ Werner  _et al._ (2008); Haule and Kotliar (2009);
Yin  _et al._ (2011a, b); Werner  _et al._ (2012); Georges  _et al._ (2013);
de’ Medici  _et al._ (2014); Stadler  _et al._ (2015); Walter  _et al._
(2020). Consequently, FeSCs are often dubbed as Hund’s metals Yin  _et al._
(2011a); Georges  _et al._ (2013).

Evidences of Hund’s metals have been found in the paramagnetic state (PM) of
several FeSC families, including doping dependent effective mass de’ Medici
_et al._ (2014), and orbital differentiations Yi  _et al._ (2013); Miao  _et
al._ (2016); Yin  _et al._ (2014); Li  _et al._ (2016). More recently,
theoretical analysis of the Hundness on superconductivity is linked to the
ratio 2$\Delta_{max}/k_{B}T_{c}$ in FeSCs Miao  _et al._ (2018); Lee  _et al._
(2018), where $\Delta_{max}$ is the largest superconducting gap in the
momentum space. Despite these important experimental and theoretical
progresses, justification of Hund’s superconductivity requires a synergetic
transport and spectroscopy studies of the Hundness that covers the whole
superconducting dome as well as their normal state. Here, using transport,
angle-resolved photoemission spectroscopy (ARPES), scanning tunneling
spectroscopy (STM) and numerical calculations, we demonstrate that the
canonical FeSC, Li(Fe,Co)As, is a Hund’s superconductor, where the Hundness
controls the entire temperature ($T$) vs doping ($x$) phase diagram. Our
results place strong experimental evidence to support a pairing mechanism
driven by the Hundness-induced critical spin fluctuations for the FeSCs Lee
_et al._ (2018).

Figure 1: Doping dependent coherent-incoherent crossover. (a) Normalized
resistivity of LiFe0.6Co0.4As (Co40), LiFe0.75Co0.25As (Co25),
LiFe0.88Co0.12As (Co12), LiFeAs (Li111), LiFe0.9V0.1As (V10), Ba0.6K0.4Fe2As2
(K40) and KFe2As2 (K122). Resistivity (red curve) of K122, LiFeAs, Co12 and
Co40 are shown in (b)-(e). Dashed lines are polynomial fittings,
$\rho_{FIT}(T)=\sum_{n=0}^{9}a_{n}T^{n}$, of the resistivity. Cyan curves in
(b)-(e) are the first derivative ($\frac{d\rho_{FIT}}{dT}$) of the fitted
resistivity. The dashed vertical line marks the maximum of
$\frac{d\rho_{FIT}}{dT}$, which is defined as the coherent-incoherent
crossover temperature, $T^{*}$ Haule and Kotliar (2009). (f) Extracted
$n$-dependent $T^{*}$ is remarkably consistent with theoretical predictions
for Hund’s metal Haule and Kotliar (2009); Werner  _et al._ (2008, 2012). Here
$n$ is converted from the Co concentration x (%) Sup . The background colors
in (b)-(f) represent the electronic coherence determined by the resisitivity
curvature.

We choose Li(Fe,Co)As for this study as its phase diagram is not intervened by
any magnetic or nematic long range orders Sup . This condition is crucial as
the Hund’s metal physics is essentially a description of the PM phase. Another
reason to study Li(Fe,Co)As is that the Hund’s coupling induced characteristic
orbital differentiation in the dynamical charge and magnetic excitations are
observed in the parent compound LiFeAs Miao  _et al._ (2016); Yin  _et al._
(2014); Li  _et al._ (2016).

We first explore the PM state of electron and hole doped LiFeAs via transport
measurements (see ref. Sup  for experimental and computational methods).
Figure 1a shows the normalized resistivity of Li(Fe,TM)As (TM=V, Co) that
cover a wide doping range, corresponding to the occupation number, $n$, from
5.97 to 6.4 Shi  _et al._ (2017); Sup . For comparison, we also include the
normalized resistivity of Ba0.6K0.4Fe2As2 (K40) and KFe2As2 (K122) that
correspond to $n$ = 5.8 and 5.5, respectively Ding  _et al._ (2008); Fang  _et
al._ (2015). From the electron-doped side to the hole-doped side, the
resistivity curvature undergoes a qualitative change: it remains positive in
the entire temperature range in LiFe0.6Co0.4As (Co40) but changes sign at high
temperature in LiFeAs. Previous DMFT studies have shown that the curvature
change in resistivity represents a coherent-incoherent crossover of the
electronic system Haule and Kotliar (2009). To quantify this change, we show
the resistivity of K122, LiFeAs, LiFe0.88Co0.12As (Co12) and Co40 in Fig.
1b-e. First of all, we notice that the room temperature resistivity
$\rho_{300}$ decreases by a factor of four from K122 to Co40 despite a larger
number of impurities in Co40, suggesting reduced electronic interaction via
electron doping. We then fit the resistivity curves above $T_{c}$ with 9th
order polynomial and use the maximum of the first derivative to characterize
the coherent-incoherent crossover temperature, $T^{*}$ (Fig. 1b-e). Figure 1f
summarizes $T^{*}$ as function of $n$, where the $T^{*}$ increases from 80 K
in K122 to room temperature in Co40, remarkably consistent with previous
LDA+DMFT calculations Haule and Kotliar (2009); Werner  _et al._ (2012) and in
agreement with our resistivity calculations for LiFeAs (see supplementary
materials Sup ).

Figure 2: Doping dependent band structure of Li(Fe,Co)As. The ARPES intensity
plot near the $\Gamma$ (a-c) and M (d-f) point. Dashed lines are parabolic
fitting of the band dispersion (peak intensity). The extracted doping
dependent effective mass of the $\alpha^{\prime}$, $\beta$, $\gamma$ and
$\delta$ band are shown in (g). (h) shows the $n$-dependent bandwidth of
$\beta$ band, determined by the band top of the $\beta$ band and band bottom
of the $\gamma$ band. Since the $\beta$ and $\gamma$ band are degenerate at
the M point, the energy difference between the top of $\beta$ and the bottom
of $\gamma$g is equivalent to the band width of the $\beta$ band, which is
broadened by a factor of two from LiFeAs to Co40, consistent with the doping
dependent effective mass of $\beta$ band in (g).

The evolution of coherent-incoherent crossover in resistivity mirrors the
$n$-dependent electronic interactions. To directly show this trend, we plot
the doping-dependent electronic structure measured by ARPES at the $\Gamma$
and M point in Fig. 2a-f. The dashed lines are parabolic fittings of the band
structure with their colors representing the main orbital characters. Figure
2g summarizes the extracted band and orbital-resolved effective mass, $m^{*}$,
as function of $n$. We find that $m^{*}$ of all bands are reduced by a factor
of two from LiFeAs to Co40. This is in agreement with the bandwidth evolution
of the $\beta$ band (Fig. 2h), which is determined by the energy difference
between the top of the $\beta$ band and the bottom of the $\gamma$ band.

The profound doping-dependent $T^{*}$ and $m^{*}$ strongly indicate that the
entire superconducting dome ($6<n<6.16$) of Li(Fe,Co)As is emerging from the
Hund’s metal normal state. To uncover the Hundness in the superconducting
state, we extract the doping-dependent SC gap from scanning tunneling
microscopy/spectroscopy (STM/STS) measurements of Li(Fe,Co)As at 400 mK, deep
in the superconducting state Yin  _et al._ (2019). Figure 3b shows a typical
atomically resolved STM topography image of LiFe0.99Co0.01As. The spatially
averaged STS spectra of eight different doping levels with $T_{c}$ ranging
from 18 to 4 K are shown in Fig.3a. The dashed lines mark the zero-intensity
value for each doping. The extracted $\Delta_{max}$ as a function of doping is
plotted on top of the $T_{c}$ vs $n$ diagram in Fig. 3c. The perfect overlap
of these two plots demonstrates a universal 2$\Delta_{max}/k_{B}T_{c}\sim 7.7$
scaling in the entire Li(Fe,Co)As superconducting phase. This value is nicely
consistent with recent model study, where a universal
2$\Delta_{max}/k_{B}T_{c}\sim 7.2$ is derived by assuming the Cooper pairs are
“glued” by the Hundness induced local spin fluctuations with a characteristic
$\chi_{sp}^{{}^{\prime\prime}}(\omega)\propto\omega^{-6/5}$ scaling Lee  _et
al._ (2018); Sup ; Wang  _et al._ (2020).
$\chi_{sp}^{{}^{\prime\prime}}(\omega)$ is the imaginary part of the local
dynamical susceptibility. The nearly perfect agreement between the theory and
experiment in the entire phase diagram firmly establishes the Hund’s
superconductivity in Li(Fe,Co)As.

Figure 3: Doping-dependent SC gap and universal 2$\Delta_{max}/k_{B}T_{c}$
scaling. (a) Averaged STS spectra measured at $T$=400 mK. The dashed lines
centered the zero bias voltage represent zero $dI/dV$. (b) The STM topography
of LiFe0.99Co0.01As. (c) The $T_{c}$ and extracted $\Delta_{max}$ as function
of doping are shown in red circles and blue squares. Data shown in (a) and (c)
are adapted from ref. Yin  _et al._ (2019). (d) shows the universal
2$\Delta_{max}/k_{B}T_{c}$ scaling of FeSCs Miao  _et al._ (2018). The black
and green dashed lines correspond to Hund’s superconductor and BCS
superconductor, respectively. The red dashed line is a fit to
2$\Delta_{max}/k_{B}T_{c}$ extracted from FeSe monolayer.

Our results have significant implications on the pairing mechanism of FeSCs.
Figure 3d shows the 2$\Delta_{max}$ vs $k_{B}T_{c}$ plot of a large number of
FeSCs including Li(Fe,Co)As. The ubiquitous 2$\Delta_{max}/k_{B}T_{c}$ scaling
over a large set of FeSCs suggests a common pairing mechanism for all FeSCs.
The consistent 2$\Delta_{max}/k_{B}T_{c}$ value between the experimental data
and model calculation strongly indicates that the leading pairing interaction
is the Hundness-induced local spin fluctuations. This conclusion explains
early experimental observations where the SC gap function $\Delta(k)$ follows
simple local form factors and involves electronic state away from the Fermi
energy $E_{F}$ Miao  _et al._ (2012); Hu and Ding (2012); Miao  _et al._
(2015). It should be noted, however, the theoretically derived
2$\Delta_{max}/k_{B}T_{c}$ is based on a simplified model that neglects
several material specific details such as the proximity to electronic
nematicity in Ba(Fe,Co)2As2 and bulk FeSe Yi  _et al._ (2011); Fernandes  _et
al._ (2014); Sprau  _et al._ (2017) and strong electron-phonon coupling in
FeSe monolayer Lee  _et al._ (2014). In addition, the presence of non-local
interaction can induce moderate gap variations on the Fermi surface Umezawa
_et al._ (2012); Borisenko  _et al._ (2010); Bhattacharyya  _et al._ (2020).
However, the large number of FeSCs with a similar 2$\Delta_{max}/k_{B}T_{c}$
ratio suggests that the Hundness is robust even in the presence of various
low-energy modifications.

In summary, using transport and spectroscopic techniques, we synergistically
uncovered signatures of Hundness from the normal to SC state of Li(Fe,Co)As.
Our result establishes the Hund’s superconductivity in canonical FeSC
Li(Fe,Co)As and indicates that the Hundness-induced critical spin fluctuation
is the leading pairing interaction in FeSCs.

This research at Oak Ridge National Laboratory (ORNL) was sponsored by the
Laboratory Directed Research and Development Program of ORNL, managed by UT-
Battelle, LLC, for the U. S. Department of Energy (ARPES) and by the U.S.
Department of Energy, Office of Science, Basic Energy Sciences, Materials
Sciences and Engineering Division (transport). Y.W. was supported by the US
Department of energy, Office of Science, Basic Energy Sciences as a part of
the Computational Materials Science Program through the Center for
Computational Design of Functional Strongly Correlated Materials and
Theoretical Spectroscopy. GK was supported by NSF DMR-1733071 Work at the IOP
is supported by grants from the National Natural Science Foundation of China
(11888101, 11674371), and the Ministry of Science and Technology of China
(2016YFA0401000). Work at Princeton University was supported by the Gordon and
Betty Moore Foundation (GBMF4547/ Hasan) and the United States Department of
energy (US DOE) under the Basic Energy Sciences program (Grant No. DOE/BES DE-
FG-02-05ER46200). H.M., Y.L.W and J.-X.Y are contributed equally to this work.

## References

  * Kamihara  _et al._ (2008) Y. Kamihara, T. Watanabe, M. Hirano, and H. Hosono, Journal of the American Chemical Society 130, 3296 (2008).
  * Hirschfeld  _et al._ (2011) P. J. Hirschfeld, M. M. Korshunov, and I. I. Mazin, Reports on Progress in Physics 74, 124508 (2011).
  * Scalapino (2012) D. J. Scalapino, Rev. Mod. Phys. 84, 1383 (2012).
  * Zhang and Rice (1988) F. C. Zhang and T. M. Rice, Phys. Rev. B 37, 3759 (1988).
  * Misawa  _et al._ (2012) T. Misawa, K. Nakamura, and M. Imada, Phys. Rev. Lett. 108, 177007 (2012).
  * Ikeda  _et al._ (2010) H. Ikeda, R. Arita, and J. Kuneš, Phys. Rev. B 82, 024508 (2010).
  * Werner  _et al._ (2008) P. Werner, E. Gull, M. Troyer, and A. J. Millis, Phys. Rev. Lett. 101, 166405 (2008).
  * Haule and Kotliar (2009) K. Haule and G. Kotliar, New Journal of Physics 11, 025021 (2009).
  * Yin  _et al._ (2011a) Z. P. Yin, K. Haule, and G. Kotliar, Nat Mater 10, 932 (2011a).
  * Yin  _et al._ (2011b) Z. P. Yin, K. Haule, and G. Kotliar, Nat Phys 7, 294 (2011b).
  * Werner  _et al._ (2012) P. Werner, M. Casula, T. Miyake, F. Aryasetiawan, A. J. Millis, and S. Biermann, Nat Phys 8, 331 (2012).
  * Georges  _et al._ (2013) A. Georges, L. de’ Medici, and J. Mravlje, Annual Review of Condensed Matter Physics 4, 137 (2013).
  * de’ Medici  _et al._ (2014) L. de’ Medici, G. Giovannetti, and M. Capone, Phys. Rev. Lett. 112, 177001 (2014).
  * Stadler  _et al._ (2015) K. M. Stadler, Z. P. Yin, J. von Delft, G. Kotliar, and A. Weichselbaum, Phys. Rev. Lett. 115, 136401 (2015).
  * Walter  _et al._ (2020) E. Walter, K. M. Stadler, S.-S. B. Lee, Y. Wang, G. Kotliar, A. Weichselbaum, and J. von Delft, Phys. Rev. X 10, 031052 (2020).
  * Yi  _et al._ (2013) M. Yi, D. H. Lu, R. Yu, S. C. Riggs, J.-H. Chu, B. Lv, Z. K. Liu, M. Lu, Y.-T. Cui, M. Hashimoto, S.-K. Mo, Z. Hussain, C. W. Chu, I. R. Fisher, Q. Si, and Z.-X. Shen, Phys. Rev. Lett. 110, 067003 (2013).
  * Miao  _et al._ (2016) H. Miao, Z. P. Yin, S. F. Wu, J. M. Li, J. Ma, B.-Q. Lv, X. P. Wang, T. Qian, P. Richard, L.-Y. Xing, X.-C. Wang, C. Q. Jin, K. Haule, G. Kotliar, and H. Ding, Phys. Rev. B 94, 201109 (2016).
  * Yin  _et al._ (2014) Z. P. Yin, K. Haule, and G. Kotliar, Nat Phys 10, 845 (2014).
  * Li  _et al._ (2016) Y. Li, Z. Yin, X. Wang, D. W. Tam, D. L. Abernathy, A. Podlesnyak, C. Zhang, M. Wang, L. Xing, C. Jin, K. Haule, G. Kotliar, T. A. Maier, and P. Dai, Phys. Rev. Lett. 116, 247001 (2016).
  * Miao  _et al._ (2018) H. Miao, W. H. Brito, Z. P. Yin, R. D. Zhong, G. D. Gu, P. D. Johnson, M. P. M. Dean, S. Choi, G. Kotliar, W. Ku, X. C. Wang, C. Q. Jin, S.-F. Wu, T. Qian, and H. Ding, Phys. Rev. B 98, 020502 (2018).
  * Lee  _et al._ (2018) T.-H. Lee, A. Chubukov, H. Miao, and G. Kotliar, Phys. Rev. Lett. 121, 187003 (2018).
  * (22) See Supplemental Material at [URL will be inserted by publisher] for experimental and calculation methods .
  * Shi  _et al._ (2017) X. Shi, Z.-Q. Han, X.-L. Peng, P. Richard, T. Qian, X.-X. Wu, M.-W. Qiu, S. C. Wang, J. P. Hu, Y.-J. Sun, and H. Ding, Nature Communications 8, 14988 (2017).
  * Ding  _et al._ (2008) H. Ding, P. Richard, K. Nakayama, K. Sugawara, T. Arakane, Y. Sekiba, A. Takayama, S. Souma, T. Sato, T. Takahashi, Z. Wang, X. Dai, Z. Fang, G. F. Chen, J. L. Luo, and N. L. Wang, EPL (Europhysics Letters) 83, 47001 (2008).
  * Fang  _et al._ (2015) D. Fang, X. Shi, Z. Du, P. Richard, H. Yang, X. X. Wu, P. Zhang, T. Qian, X. Ding, Z. Wang, T. K. Kim, M. Hoesch, A. Wang, X. Chen, J. Hu, H. Ding, and H.-H. Wen, Phys. Rev. B 92, 144513 (2015).
  * Yin  _et al._ (2019) J.-X. Yin, S. S. Zhang, G. Dai, Y. Zhao, A. Kreisel, G. Macam, X. Wu, H. Miao, Z.-Q. Huang, J. H. J. Martiny, B. M. Andersen, N. Shumiya, D. Multer, M. Litskevich, Z. Cheng, X. Yang, T. A. Cochran, G. Chang, I. Belopolski, L. Xing, X. Wang, Y. Gao, F.-C. Chuang, H. Lin, Z. Wang, C. Jin, Y. Bang, and M. Z. Hasan, Phys. Rev. Lett. 123, 217004 (2019).
  * Wang  _et al._ (2020) Y. Wang, E. Walter, S.-S. B. Lee, K. M. Stadler, J. von Delft, A. Weichselbaum, and G. Kotliar, Phys. Rev. Lett. 124, 136406 (2020).
  * Miao  _et al._ (2012) H. Miao, P. Richard, Y. Tanaka, K. Nakayama, T. Qian, K. Umezawa, T. Sato, Y.-M. Xu, Y. B. Shi, N. Xu, X.-P. Wang, P. Zhang, H.-B. Yang, Z.-J. Xu, J. S. Wen, G.-D. Gu, X. Dai, J.-P. Hu, T. Takahashi, and H. Ding, Phys. Rev. B 85, 094506 (2012).
  * Hu and Ding (2012) J. Hu and H. Ding, Scientific Reports 2, 381 (2012).
  * Miao  _et al._ (2015) H. Miao, T. Qian, X. Shi, P. Richard, T. K. Kim, M. Hoesch, L. Y. Xing, X.-C. Wang, C.-Q. Jin, J.-P. Hu, and H. Ding, Nat Commun 6 (2015).
  * Yi  _et al._ (2011) M. Yi, D. Lu, J.-H. Chu, J. G. Analytis, A. P. Sorini, A. F. Kemper, B. Moritz, S.-K. Mo, R. G. Moore, M. Hashimoto, W.-S. Lee, Z. Hussain, T. P. Devereaux, I. R. Fisher, and Z.-X. Shen, Proceedings of the National Academy of Sciences 108, 6878 (2011).
  * Fernandes  _et al._ (2014) R. M. Fernandes, A. V. Chubukov, and J. Schmalian, Nature Physics 10, 97 (2014).
  * Sprau  _et al._ (2017) P. O. Sprau, A. Kostin, A. Kreisel, A. E. Böhmer, V. Taufour, P. C. Canfield, S. Mukherjee, P. J. Hirschfeld, B. M. Andersen, and J. C. S. Davis, Science 357, 75 (2017).
  * Lee  _et al._ (2014) J. J. Lee, F. T. Schmitt, R. G. Moore, S. Johnston, Y.-T. Cui, W. Li, M. Yi, Z. K. Liu, M. Hashimoto, Y. Zhang, D. H. Lu, T. P. Devereaux, D.-H. Lee, and Z.-X. Shen, Nature 515, 245 (2014).
  * Umezawa  _et al._ (2012) K. Umezawa, Y. Li, H. Miao, K. Nakayama, Z.-H. Liu, P. Richard, T. Sato, J. B. He, D.-M. Wang, G. F. Chen, H. Ding, T. Takahashi, and S.-C. Wang, Phys. Rev. Lett. 108, 037002 (2012).
  * Borisenko  _et al._ (2010) S. V. Borisenko, V. B. Zabolotnyy, D. V. Evtushinsky, T. K. Kim, I. V. Morozov, A. N. Yaresko, A. A. Kordyuk, G. Behr, A. Vasiliev, R. Follath, and B. Büchner, Phys. Rev. Lett. 105, 067002 (2010).
  * Bhattacharyya  _et al._ (2020) S. Bhattacharyya, K. Björnson, K. Zantout, D. Steffensen, L. Fanfarillo, A. Kreisel, R. Valentí, B. M. Andersen, and P. J. Hirschfeld, Phys. Rev. B 102, 035109 (2020).