#  CARD: Classification and Regression
Diffusion Models

Xizewen Han, Huangjie Zheng††footnotemark: , and Mingyuan Zhou
The University of Texas at Austin
Austin, TX 78712
{xizewen.han<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
Equal contribution.

###### Abstract

Learning the distribution of a continuous or categorical response variable
${\bm{y}}$ given its covariates ${\bm{x}}$ is a fundamental problem in
statistics and machine learning. Deep neural network-based supervised learning
algorithms have made great progress in predicting the mean of ${\bm{y}}$ given
${\bm{x}}$, but they are often criticized for their ability to accurately
capture the uncertainty of their predictions. In this paper, we introduce
classification and regression diffusion (CARD) models, which combine a
denoising diffusion-based conditional generative model and a pre-trained
conditional mean estimator, to accurately predict the distribution of
${\bm{y}}$ given ${\bm{x}}$. We demonstrate the outstanding ability of CARD in
conditional distribution prediction with both toy examples and real-world
datasets, the experimental results on which show that CARD in general
outperforms state-of-the-art methods, including Bayesian neural network-based
ones that are designed for uncertainty estimation, especially when the
conditional distribution of ${\bm{y}}$ given ${\bm{x}}$ is multi-modal.

##  1 Introduction

A fundamental problem in statistics and machine learning is to predict the
response variable ${\bm{y}}$ given a set of covariates ${\bm{x}}$. Generally
speaking, ${\bm{y}}$ is a continuous variable for regression analysis and a
categorical variable for classification. Denote $f({\bm{x}})\in\mathbb{R}^{C}$
as a deterministic function that transforms ${\bm{x}}$ into a $C$ dimensional
output. Denote $f_{c}({\bm{x}})$ as the $c$-th dimension of $f({\bm{x}})$.
Existing methods typically assume an additive noise model: for regression
analysis with ${\bm{y}}\in\mathbb{R}^{C}$, one often assumes
${\bm{y}}=f({\bm{x}})+\bm{\epsilon},~{}\bm{\epsilon}\sim\mathcal{N}(0,\bm{\Sigma})$,
while for classification with $y\in\\{1,\ldots,C\\}$, one often assumes
$y=\operatorname*{arg\,max}\big{(}f_{1}({\bm{x}})+\bm{\epsilon}_{1},\ldots,f_{C}({\bm{x}})+\bm{\epsilon}_{C}\big{)}$,
where $\bm{\epsilon}_{c}\stackrel{{\scriptstyle
iid}}{{\sim}}\mbox{EV}_{1}(0,1)$, a standard type-1 extreme value
distribution. Thus we have the expected value of ${\bm{y}}$ given ${\bm{x}}$
as $\mathop{\mathbb{E}}[{\bm{y}}\,|\,{\bm{x}}]=f({\bm{x}})$ in regression and
$P(y=c\,|\,{\bm{x}})=\mathbb{E}[y=c\,|\,{\bm{x}}]=\mbox{softmax}_{c}\big{(}f({\bm{x}})\big{)}=\frac{\exp\left({f_{c}({\bm{x}})}\right)}{\sum_{c^{\prime}=1}^{C}\exp\left({f_{c^{\prime}}({\bm{x}})}\right)}$
in classification.

These additive-noise models are primarily focusing on accurately estimating
the conditional mean $\mathbb{E}[{\bm{y}}\,|\,{\bm{x}}]$, while paying less
attention to whether the noise distribution can accurately capture the
uncertainty of ${\bm{y}}$ given ${\bm{x}}$. For this reason, they may not work
well if the distribution of ${\bm{y}}$ given ${\bm{x}}$ clearly deviates from
the additive-noise assumption. For example, if $p({\bm{y}}\,|\,{\bm{x}})$ is
multi-modal, which commonly happens when there are missing categorical
covariates in ${\bm{x}}$, then $\mathbb{E}[{\bm{y}}\,|\,{\bm{x}}]$ may not be
close to any possible true values of ${\bm{y}}$ given that specific
${\bm{x}}$. More specifically, consider a person whose weight, height, blood
pressure, and age are known but gender is unknown, then the testosterone or
estrogen level of this person is likely to follow a bi-modal distribution and
the chance of developing breast cancer is also likely to follow a bi-modal
distribution. Therefore, these widely used additive-noise models, which use a
deterministic function $f({\bm{x}})$ to characterize the conditional mean of
${\bm{y}}$, are inherently restrictive in their ability for uncertainty
estimation.

In this paper, our goal is to accurately recover the full distribution of
${\bm{y}}$ conditioning on ${\bm{x}}$ given a set of $N$ training data points,
denoted as $\mathcal{D}=\\{({\bm{x}}_{i},{\bm{y}}_{i})\\}_{1,N}$. To realize
this goal, we consider the diffusion-based (a.k.a. score-based) generative
models (Song and Ermon, 2019; Ho et al., 2020; Song and Ermon, 2020; Song et
al., 2021) and inject covariate-dependence into both the forward (inference)
and reverse (generative) diffusion chains. Our method can model the
conditional distribution of both continuous and categorical ${\bm{y}}$
variable, and the algorithms developed under this method will be collectively
referred to as Classification And Regression Diffusion (CARD) models.

Diffusion-based generative models have received significant recent attention
due to not only their ability to generate high-dimensional data, such as high-
resolution photo-realistic images, but also their training stability. They can
be understood from the perspective of score matching (Hyvärinen and Dayan,
2005; Vincent, 2011) and Langevin dynamics (Neal, 2011; Welling and Teh,
2011), as pioneered by Song and Ermon (2019). They can also be understood from
the perspective of diffusion probabilistic models (Sohl-Dickstein et al.,
2015; Ho et al., 2020), which first define a forward diffusion to transform
the data into noise and then an inverse diffusion to regenerate the data from
noise.

These previous methods mainly focus on unconditional generative modeling.
While there exist guided-diffusion models (Song and Ermon, 2019; Song et al.,
2021; Dhariwal and Nichol, 2021; Nichol et al., 2021; Ramesh et al., 2022)
that target on generating high-resolution photo-realistic images that match
the semantic meanings or content of the label, text, or corrupted-images, we
focus on studying diffusion-based conditional generative modeling at a more
fundamental level. In particular, our goal is to thoroughly investigate
whether CARD can help accurately recovery
$p({\bm{y}}\,|\,{\bm{x}},\mathcal{D})$, the predictive distribution of
${\bm{y}}$ given ${\bm{x}}$ after observing data
$\mathcal{D}=\\{({\bm{x}}_{i},{\bm{y}}_{i})\\}_{i=1,N}$. In other words, our
focus is on regression analysis of continuous or categorical response
variables given their corresponding covariates.

We summarize our main contributions as follows: 1) We show CARD, which injects
covariate-dependence and a pre-trained conditional mean estimator into both
the forward and reverse diffusion chains to construct a denoising diffusion
probabilistic model, provides an accurate estimation of
$p({\bm{y}}\,|\,{\bm{x}},\mathcal{D})$. 2) We provide a new metric to better
evaluate how well a regression model captures the full distribution
$p({\bm{y}}\,|\,{\bm{x}},\mathcal{D})$. 3) Experiments on standard benchmarks
for regression analysis show that CARD achieves state-of-the-art results,
using both existing metrics and the new one.

##  2 Methods and Algorithms for CARD

###  2.1 Problem Statement

Given the ground-truth response variable ${\bm{y}}_{0}$ and its covariates
${\bm{x}}$, and assuming a sequence of intermediate uncertain prediction
${\bm{y}}_{1:T}$ made by the diffusion model, the goal of supervised learning
is to learn a good model such that the log-likelihood is maximized by
optimizing the following ELBO:

$\displaystyle\log p_{{\mathbf{\theta}}}({\bm{y}}_{0}\,|\,{\bm{x}})=\log\int
p_{{\mathbf{\theta}}}({\bm{y}}_{0:T}\,|\,{\bm{x}})d{\bm{y}}_{1;T}\geq\mathbb{E}_{q({\bm{y}}_{1:T}|{\bm{x}})}\left[\log\frac{p_{{\mathbf{\theta}}}({\bm{y}}_{0:T}\,|\,{\bm{x}})}{q({\bm{y}}_{1:T}\,|\,{\bm{y}}_{0},{\bm{x}})}\right],$
(1)

where $q({\bm{y}}_{1:T}\,|\,{\bm{y}}_{0},{\bm{x}})$ is called the forward
process or diffusion process in the concept of diffusion models (Ho et al.,
2020). Denoting $D_{\mathrm{KL}}(q||p)$ as the Kullback–Leibler (KL)
divergence from distributions $p$ to $q$. The above objective can be rewritten
as

$\displaystyle{\mathcal{L}}_{\text{ELBO}}({\bm{x}})$
$\displaystyle\coloneqq{\mathcal{L}}_{0}({\bm{x}})+\sum_{t=2}^{T}{\mathcal{L}}_{t-1}({\bm{x}})+{\mathcal{L}}_{T}({\bm{x}}),$
(2) $\displaystyle{\mathcal{L}}_{0}({\bm{x}})$
$\displaystyle\coloneqq\mathbb{E}_{q}\left[-\log
p_{\theta}({\bm{y}}_{0}\,|\,{\bm{y}}_{1},{\bm{x}})\right],$ (3)
$\displaystyle{\mathcal{L}}_{t-1}({\bm{x}})$
$\displaystyle\coloneqq\mathbb{E}_{q}\left[D_{\mathrm{KL}}\left({q({\bm{y}}_{t-1}\,|\,{\bm{y}}_{t},{\bm{y}}_{0},{\bm{x}})}||{p_{\theta}({\bm{y}}_{t-1}\,|\,{\bm{y}}_{t},{\bm{x}})}\right)\right],$
(4) $\displaystyle{\mathcal{L}}_{T}({\bm{x}})$
$\displaystyle\coloneqq\mathbb{E}_{q}\left[D_{\mathrm{KL}}\left({q({\bm{y}}_{T}\,|\,{\bm{y}}_{0},{\bm{x}})}||{p({\bm{y}}_{T}|{\bm{x}})}\right)\right].$
(5)

Here we follow the convention to assume ${\mathcal{L}}_{T}$ does not depend on
any parameter and it will be close to zero by carefully diffusing the observed
response variable ${\bm{y}}_{0}$ towards a pre-assumed distribution
$p({\bm{y}}_{T}\,|\,{\bm{x}})$. The remaining terms will make sure the model
$p_{\theta}({\bm{y}}_{t-1}\,|\,{\bm{y}}_{t},{\bm{x}})$ predicts at different
diffusion steps and we hope this gives us different scales in uncertainty
measure. Different from the vanilla diffusion model, we assume the endpoint of
the diffusion as

$\displaystyle p({\bm{y}}_{T})=\mathcal{N}(f_{\phi}({\bm{x}}),{\bm{I}}),$ (6)

where $f_{\phi}({\bm{x}})$ is the pre-knowledge of the relation between
${\bm{x}}$ and ${\bm{y}}_{0}$, e.g., pre-trained with $\mathcal{D}$ to
approximate $\mathbb{E}[{\bm{y}}\,|\,{\bm{x}}]$, or $\mathbf{0}$ if we assume
the relation is unknown. With a diffusion schedule
$\\{\beta_{t}\\}_{t=1:T}\in(0,1)^{T}$, we specify the forward diffusion
process conditional distributions in a similar fashion as Pandey et al.
(2022), but for all timesteps including $t=1$:

$\displaystyle
q\big{(}{\bm{y}}_{t}\,|\,{\bm{y}}_{t-1},f_{\phi}({\bm{x}})\big{)}=\mathcal{N}\big{(}{\bm{y}}_{t};\sqrt{1-\beta_{t}}{\bm{y}}_{t-1}+(1-\sqrt{1-\beta_{t}})f_{\phi}({\bm{x}}),\beta_{t}{\bm{I}}\big{)},$
(7)

which admits a closed-form sampling distribution with arbitary timestep $t$:

$\displaystyle
q\big{(}{\bm{y}}_{t}\,|\,{\bm{y}}_{0},f_{\phi}({\bm{x}})\big{)}=\mathcal{N}\big{(}{\bm{y}}_{t};\sqrt{\bar{\alpha}_{t}}{\bm{y}}_{0}+(1-\sqrt{\bar{\alpha}_{t}})f_{\phi}({\bm{x}}),(1-\bar{\alpha}_{t}){\bm{I}}\big{)},$
(8)

where $\alpha_{t}:=1-\beta_{t}$ and $\bar{\alpha}_{t}:=\prod_{t}\alpha_{t}$.
Note that Eq. (7) can be viewed as an interpolation between true data
${\bm{y}}_{0}$ and the predicted conditional expectation $f_{\phi}({\bm{x}})$,
which gradually changes from the former to the latter throughout the forward
process.

Such formulation corresponds to a tractable forward process posterior:

$\displaystyle
q({\bm{y}}_{t-1}\,|\,{\bm{y}}_{t},{\bm{y}}_{0},{\bm{x}})=q\big{(}{\bm{y}}_{t-1}\,|\,{\bm{y}}_{t},{\bm{y}}_{0},f_{\phi}({\bm{x}})\big{)}=\mathcal{N}\Big{(}{\bm{y}}_{t-1};\tilde{{\bm{\mu}}}\big{(}{\bm{y}}_{t},{\bm{y}}_{0},f_{\phi}({\bm{x}})\big{)},\tilde{\beta_{t}}{\bm{I}}\Big{)},$
(9)

where

$\displaystyle\tilde{{\bm{\mu}}}$
$\displaystyle\coloneqq\underbrace{\frac{\beta_{t}\sqrt{\bar{\alpha}_{t-1}}}{1-\bar{\alpha}_{t}}}_{\gamma_{0}}{\bm{y}}_{0}+\underbrace{\frac{(1-\bar{\alpha}_{t-1})\sqrt{\alpha_{t}}}{1-\bar{\alpha}_{t}}}_{\gamma_{1}}{\bm{y}}_{t}+\underbrace{\Bigg{(}1+\frac{(\sqrt{\bar{\alpha}_{t}}-1)(\sqrt{\alpha_{t}}+\sqrt{\bar{\alpha}_{t-1}})}{1-\bar{\alpha}_{t}}\Bigg{)}}_{\gamma_{2}}f_{\phi}({\bm{x}}),$
$\displaystyle\tilde{\beta_{t}}$
$\displaystyle\coloneqq\frac{1-\bar{\alpha}_{t-1}}{1-\bar{\alpha}_{t}}\beta_{t}.$

We provide the derivation in Appendix A.1. The labels under the terms are used
in Algorithm 2.

###  2.2 CARD for Regression

For regression problems, the goal of the reverse diffusion process is to
gradually recover the distribution of the noise term, the aleatoric or local
uncertainty inherent in the observations (Kendall and Gal, 2017; Wang and
Zhou, 2020), enabling us to generate samples that match the true conditional
$p({\bm{y}}\,|\,{\bm{x}})$.

Following the reparameterization introduced by denoising diffusion
probabilistic models (DDPM) (Ho et al., 2020), we construct
$\bm{\epsilon}_{{\mathbf{\theta}}}\big{(}{\bm{x}},{\bm{y}}_{t},f_{\phi}({\bm{x}}),t\big{)}$,
which is a function approximator parameterized by a deep neural network that
predicts the forward diffusion noise $\bm{\epsilon}$ sampled for
${\bm{y}}_{t}$. The training and inference procedure can be carried out in a
standard DDPM manner.

1: Pre-train $f_{\phi}({\bm{x}})$ that predicts
$\mathbb{E}({\bm{y}}\,|\,{\bm{x}})$ with MSE

2: repeat

3: Draw $y_{0}\sim q(y_{0})$

4: Draw $t\sim\text{Uniform}(\\{1\dots T\\})$

5: Draw $\bm{\epsilon}\sim\mathcal{N}(\mathbf{0},{\bm{I}})$

6: Compute noise estimation loss
$\mathcal{L}_{\bm{\epsilon}}=||\bm{\epsilon}-\bm{\epsilon}_{{\mathbf{\theta}}}({\bm{x}},\sqrt{\bar{\alpha}_{t}}{\bm{y}}_{0}+\sqrt{1-\bar{\alpha}_{t}}\bm{\epsilon}+\sqrt{1-\bar{\alpha}_{t}}f_{\phi}({\bm{x}}),t)||^{2}$

7: Take numerical optimization step on:
$\nabla_{\theta}\mathcal{L}_{\bm{\epsilon}}$

8: until Convergence

Algorithm 1 Training (Regression)

1: ${\bm{y}}_{T}\sim\mathcal{N}(f_{\phi}({\bm{x}}),{\bm{I}})$

2: for $t=T$ to $1$ do

3: Draw ${\bm{z}}\sim\mathcal{N}(\bm{0},{\bm{I}})$ if $t>1$, else set
${\bm{z}}=\bm{0}$

4: Calculate reparameterized
$\hat{{\bm{y}}}_{0}=\frac{1}{\sqrt{\bar{\alpha}_{t}}}\big{(}{\bm{y}}_{t}-(1-\sqrt{\bar{\alpha}_{t}})f_{\phi}({\bm{x}})-\bm{\epsilon}_{{\mathbf{\theta}}}\big{(}{\bm{x}},{\bm{y}}_{t},f_{\phi}({\bm{x}}),t\big{)}\sqrt{1-\bar{\alpha}_{t}}\big{)}$

5: Let
${\bm{y}}_{t-1}=\gamma_{0}\hat{{\bm{y}}}_{0}+\gamma_{1}{\bm{y}}_{t}+\gamma_{2}f_{\phi}({\bm{x}})+{\bm{z}}\tilde{\beta}_{t}$

6: end for

7: return ${\bm{y}}_{0}$

Algorithm 2 Inference (Regression)

###  2.3 CARD for Classification

We formulate the classification tasks in a similar fashion as in Section 2.2,
where we:

  1. 1.

Replace the continuous response variable with a one-hot encoded label vector
for ${\bm{y}}_{0}$;

  2. 2.

Replace the mean estimator with a pre-trained classifier that outputs softmax
probabilities of the class labels for $f_{\phi}({\bm{x}})$.

This construction no longer assumes ${\bm{y}}_{0}$ to be drawn from a
categorical distribution, but instead treating each one-hot label as a class
prototype. The sampling procedure would output reconstructed ${\bm{y}}_{0}$ in
the range of real numbers for each dimension, instead of a vector in the
probability simplex. We convert such output to a probability vector in a
softmax form of a temperature-weighted the Brier score (Brier, 1950), which
computes the squared error between the prediction and a vector of $1$s with
length $C$, the number of classes. Mathematically, the probability of
predicting the $k^{th}$ class $y=k$ and the final point prediction $\hat{y}$
can be expressed as

$\displaystyle\text{Pr}(y=k)=\frac{\exp(-({\bm{y}}_{0}-\bm{1}_{C})^{2}_{k}/\tau\big{)}}{\sum_{i=1}^{C}\exp(-({\bm{y}}_{0}-\bm{1}_{C})^{2}_{i}/\tau\big{)}};~{}\hat{y}=\operatorname*{arg\,max}_{k}\big{(}-({\bm{y}}_{0}-\bm{1}_{C})^{2}_{k}\big{)}.$
(10)

where $\tau>0$ is the temperature parameter, and
$({\bm{y}}_{0}-\bm{1}_{C})^{2}_{k}$ indicates the $k^{th}$ dimension of the
vector of element-wise square error between ${\bm{y}}_{0}$ and $\bm{1}_{C}$,
i.e., $({\bm{y}}_{0}-\bm{1}_{C})^{2}_{k}=\|{\bm{y}}_{0_{k}}-1\|^{2}$.
Intuitively, this construction would assign the class whose raw output in the
sampled ${\bm{y}}_{0}$ is closest to the true class, encoded by the value of
$1$ in the one-hot label, with the highest probability.

Given the same covariates ${\bm{x}}$, the stochasticity of the generative
model would give us a different class prototype reconstruction after each
reverse process sampling, which enables us to construct predicted probability
intervals for all class labels. Such stochastic reconstruction is in a similar
fashion as DALL-E $2$ (Ramesh et al., 2022) that applies a diffusion prior to
reconstruct the image embedding by conditioning on the text embedding during
the reverse diffusion process, which is a key step in the diversity of
generated images.

##  3 Experiments

For the hyperparameters of CARD in both regression and classification tasks,
we set the number of timesteps as $T=1000$, a linear noise schedule with
$\beta_{1}=10^{-4}$ and $\beta_{T}=0.02$, same as Ho et al. (2020). We provide
a more detailed walk-through of the experimental setup, including training and
network architecture, in Appendix A.4.

###  3.1 Regression

Putting aside its statistical interpretation, the word regress indicates a
direction opposite to progress, suggesting a less developed state. Such
semantics in fact translates well into the statistical domain, in the sense
that traditional regression analysis methods often only focus on estimating
$\mathbb{E}({\bm{y}}\,|\,{\bm{x}})$, while leaving out all the other details
about $p({\bm{y}}\,|\,{\bm{x}})$. In recent years, Bayesian neural networks
(BNNs) have emerged as a class of models that aims at estimating the
uncertainty (Hernández-Lobato and Adams, 2015; Gal and Ghahramani, 2016;
Lakshminarayanan et al., 2017; Tomczak et al., 2021), providing a more
complete picture of $p({\bm{y}}\,|\,{\bm{x}})$. The metric that they use to
quantify uncertainty estimation, negative log-likelihood (NLL), is computed
with a Gaussian density, implying their assumption such that the conditional
distributions $p({\bm{y}}\,|\,{\bm{x}}=x)$ for all $x$ are Gaussian. However,
this assumption is very difficult to verify for real-world datasets: the
covariates can be arbitrarily high-dimensional, making the feature space
increasingly sparse with respect to the number of collected observations.

To accommodate the need for uncertainty estimation without imposing such
restriction for the parametric form of $p({\bm{y}}\,|\,{\bm{x}})$, we apply
the following two metrics, both of which are designed to empirically evaluate
the level of similarity between the learned and the true conditional
distributions:

  1. 1.

Prediction Interval Coverage Probability (PICP);

  2. 2.

Quantile Interval Coverage Error (QICE).

PICP has been described in Yao et al. (2019), whereas QICE is a new metric
proposed by us. We describe both of them in what follows.

####  3.1.1 PICP and QICE

The PICP is computed as

$\displaystyle\text{PICP}\coloneqq\frac{1}{N}\sum_{n=1}^{N}\mathbb{1}_{y_{n}\geq\hat{y}_{n}^{\text{low}}}\cdot\mathbb{1}_{y_{n}\leq\hat{y}_{n}^{\text{high}}},$
(11)

where $\hat{y}_{n}^{\text{low}}$ and $\hat{y}_{n}^{\text{high}}$ represents
the low and high percentiles, respectively, of our choice for the predicted
${\bm{y}}$ outputs given the same ${\bm{x}}$ input. This metric measures the
proportion of true observations that fall in the percentile range of the
generated ${\bm{y}}$ samples given each ${\bm{x}}$ input. Intuitively, when
the learned distribution represents the true distribution well, this
measurement should be close to the difference between the selected low and
high percentiles. In this paper, we choose the $2.5^{th}$ and $97.5^{th}$
percentile, thus an ideal PICP value for the learned model should be $95\%$.

Meanwhile, there is a caveat for this metric: for example, imagine a situation
where the $2.5^{th}$ to $97.5^{th}$ percentile of the learned distribution
happens to cover the data between the $1^{st}$ and $96^{th}$ percentiles from
the true distribution. Given enough samples, we shall still obtain a PICP
value close to $95\%$, but clearly there is a mismatch between the learned
distribution and the true one.

Based on such reasoning, we propose a new empirical metric QICE, which by
design can be viewed as PICP with finer granularity. To compute QICE, we first
generate enough ${\bm{y}}$ samples given each ${\bm{x}}$, and divide them into
$M$ bins with roughly equal sizes. We would obtain the corresponding quantile
values at each boundary. In this paper, we set $M=10$, and obtain the
following $10$ quantile intervals (QIs) of the generated ${\bm{y}}$ samples:
below the $10^{th}$ percentile, between the $10^{th}$ and $20^{th}$
percentiles, $\dots$, between the $80^{th}$ and $90^{th}$ percentiles, and
above the $90^{th}$ percentile. Optimally, when the learned conditional
distribution is identical to the true one, given enough samples from both
learned and true distribution we shall observe about $10\%$ of true data
falling into each of these $10$ QIs.

We define QICE to be the mean absolute error between the proportion of true
data contained by each QI and the optimal proportion, which is $1/M$ for all
intervals:

$\displaystyle\text{QICE}\coloneqq\frac{1}{M}\sum_{m=1}^{M}\bigg{|}r_{m}-\frac{1}{M}\bigg{|},\text{
where
}r_{m}=\frac{1}{N}\sum_{n=1}^{N}\mathbb{1}_{y_{n}\geq\hat{y}_{n}^{\text{low}_{m}}}\cdot\mathbb{1}_{y_{n}\leq\hat{y}_{n}^{\text{high}_{m}}}.$
(12)

Intuitively, under optimal scenario with enough samples, we shall obtain a
QICE value of $0$. Note that each $r_{m}$ is indeed the PICP for the
corresponding QI with boundaries at $\hat{y}_{n}^{\text{low}_{m}}$ and
$\hat{y}_{n}^{\text{high}_{m}}$. Since the true ${\bm{y}}$ for each ${\bm{x}}$
is guaranteed to fall into one of these QIs, we are thus able to overcome the
mismatch issue described in the above example for PICP: fewer true instances
falling into one QI would result in more instances captured by another QI,
thus increasing the absolute error for both QIs.

QICE is similar to NLL in the sense that it also utilizes the summary
statistics of the samples from the learned distribution conditional on each
new ${\bm{x}}$ to empirically evaluate how well the model fits the true data.
Meanwhile, it does not assume any parametric form on the conditional
distribution, making it a much more generalizable metric to measure the level
of distributional match between the learned and the underlying true
conditional distributions, especially when the true conditional distribution
is known to be multi-modal. We will demonstrate this point through the
regression toy examples.

####  3.1.2 Toy Examples

To demonstrate the effectiveness of CARD in regression tasks for not only
learning the conditional mean $\mathbb{E}({\bm{y}}\,|\,{\bm{x}})$, but also
recreating the ground truth data generating mechanism, we first apply CARD on
$8$ toy examples, whose data generating functions are designed to possess
different statistical characteristics: some have a uni-modal symmetric
distribution for their error term (linear regression, quadratic regression,
sinusoidal regression111The data generating function was originally proposed
in (Bishop, 1994).), others have heteroscedasticity (log-log linear
regression, log-log cubic regression) or multi-modality (inverse sinusoidal
regression222Swap $x$ and the generated $y$ from the sinusoidal regression
task., $8$ Gaussians333We set the coordinates of $8$ modes at $(\sqrt{2},0)$,
$(-\sqrt{2},0)$, $(0,\sqrt{2})$, $(0,-\sqrt{2})$, $(1,1)$, $(1,-1)$, $(-1,1)$,
$(-1,-1)$, and add a different noise sample to $x$ and $y$ coordinate of a
mode to generate one instance., full circle). We show that CARD can generate
samples that are visually indistinguishable from the true response variables
of the new covariates, as well as quantitatively match the true distribution
in terms of some summary statistics.

The $8$ toy examples are summarized by Table 13 in Appendix A.6. For each
task, we create the dataset by sampling $10,240$ data points from the data
generating function, and randomly split them into training and test set with a
$80\%/20\%$ ratio.

We first examine the performance of CARD visually, by making the scatter plots
of both true test data and generated data conditional on test ${\bm{x}}$ for
all $8$ tasks. For tasks with uni-modal conditional distribution, we fill the
region between the $2.5^{th}$ and $97.5^{th}$ percentile of the generated
${\bm{y}}$’s, and plot the true $\mathbb{E}({\bm{y}}\,|\,{\bm{x}})$ along with
the sample means. We observe that for all $8$ tasks, the generated samples
(blue dots) have blended remarkably well with the true test instances (red
dots), suggesting the CARD model is capable of reconstructing the underlying
data generation mechanism. Note that we include all ${\bm{y}}$ samples for the
computation of each conditional mean, instead of discarding any samples that
might be deemed as extreme.

Figure 1: Regression toy example scatter plots. (Top) left to right: linear
regression, quadratic regression, log-log linear regression, log-log cubic
regression; (Bottom) left to right: sinusoidal regression, inverse sinusoidal
regression, 8 Gaussians, full circle.

To quantitatively evaluate the performance of CARD, we generate $1000$
${\bm{y}}$ samples for each ${\bm{x}}$ in the test set, and compute the
corresponding metrics. We conduct such procedure over $10$ runs, each applying
a different random seed to generate the dataset, and report the mean and
standard deviation over all runs for each metric. For all tasks regardless of
the form of $p({\bm{y}}\,|\,{\bm{x}})$, we compute PICP and QICE. For tasks
with uni-modal $p({\bm{y}}\,|\,{\bm{x}})$ distributions, we summarize the
$1000$ samples for each test ${\bm{x}}$ by computing their mean, as an
unbiased estimator to $\mathbb{E}({\bm{y}}\,|\,{\bm{x}})$, and compute the
root mean squared error (RMSE) between the estimated and true conditional
means. For all tasks, we obtain a mean PICP very close to the optimal $95\%$,
and most of the tasks have a mean QICE value far less than $0.01$ except log-
log cubic regression, which also has a mean RMSE noticeably larger by an order
of magnitude among cases with uni-modal conditional distributions. Note that
the ${\bm{y}}$ samples here have a much wider range: as ${\bm{x}}$ increases
from $0$ to $10$, ${\bm{y}}$ increases from $0$ to over $1200$, resulting in a
much more difficult task. Therefore, the metrics reported here can be viewed
with relativity, and combined with the qualitative conclusions from Figure 1.
The metrics of all tasks are recorded to Table 14 in Appendix A.6.

####  3.1.3 UCI Regression Tasks

We continue to investigate our model through experiments on real-world
datasets. We adopt the same set of $10$ UCI regression benchmark datasets (Dua
and Graff, 2017) as well as the experimental protocol proposed in Hernández-
Lobato and Adams (2015) and followed by Gal and Ghahramani (2016) and
Lakshminarayanan et al. (2017). The dataset information in terms of their size
and number of features are summarized in Table 1.

Table 1: Dataset size ($N$ observations, $P$ features) of UCI regression
tasks.

Dataset | Boston | Concrete | Energy | Kin8nm | Naval | Power | Protein | Wine | Yacht | Year
---|---|---|---|---|---|---|---|---|---|---
$(N,P)$ | $(506,13)$ | $(1030,8)$ | $(768,8)$ | $(8192,8)$ | $(11934,16)$ | $(9568,4)$ | $(45730,9)$ | $(1599,11)$ | $(308,6)$ | $(515345,90)$

We apply multiple train-test splits with $90\%/10\%$ ratio in the same way
($20$ folds for all datasets except $5$ for Protein and $1$ for Year), and
report the metrics by their mean and standard deviation across all splits. We
compare our method to all aforementioned BNN frameworks: PBP, MC Dropout, and
Deep Ensembles, as well as another deep generative model that estimates a
conditional distribution sampler, GCDS (Zhou et al., 2021). Similar to BNNs,
we evaluate the accuracy and predictive uncertainty estimation of CARD by
reporting RMSE and NLL. Furthermore, we also report QICE for all methods to
evaluate distributional matching. Since this new metric was not applied in
previous methods, we re-ran the experiments for all BNNs and obtained
comparable or slightly better results reported in their literature. Further
details about the experiment setup for these models can be found in the
Appendix A.5. The experiment results with corresponding metrics are shown in
Tables 2, 4, and 4, with the number of times each model achieves the best
corresponding metric reported in the last row.

Table 2: RMSE of UCI regression tasks. For both 1Kin8nm and 2Naval dataset, we
multiply the response variable by $100$ to match the scale of others.

Dataset |  |  | RMSE $\downarrow$ |  |
---|---|---|---|---|---
| PBP | MC Dropout | Deep Ensembles | GCDS | CARD (ours)
Boston | $2.89\pm 0.74$ | $3.06\pm 0.96$ | $3.17\pm 1.05$ | $2.75\pm 0.58$ | $\bm{2.62\pm 0.63}$
Concrete | $5.55\pm 0.46$ | $5.09\pm 0.60$ | $\bm{4.91\pm 0.47}$ | $5.39\pm 0.55$ | $4.92\pm 0.48$
Energy | $1.58\pm 0.21$ | $1.70\pm 0.22$ | $2.02\pm 0.32$ | $0.64\pm 0.09$ | $\bm{0.52\pm 0.06}$
Kin8nm1 | $9.42\pm 0.29$ | $7.10\pm 0.26$ | $8.65\pm 0.47$ | $8.88\pm 0.42$ | $\bm{6.40\pm 0.19}$
Naval2 | $0.41\pm 0.08$ | $0.08\pm 0.03$ | $0.09\pm 0.01$ | $0.14\pm 0.05$ | $\bm{0.02\pm 0.00}$
Power | $4.10\pm 0.15$ | $4.04\pm 0.14$ | $4.02\pm 0.15$ | $4.11\pm 0.16$ | $\bm{3.91\pm 0.16}$
Protein | $4.65\pm 0.02$ | $4.16\pm 0.12$ | $4.45\pm 0.02$ | $4.50\pm 0.02$ | $\bm{3.71\pm 0.01}$
Wine | $0.64\pm 0.04$ | $\bm{0.62\pm 0.04}$ | $0.63\pm 0.04$ | $0.66\pm 0.04$ | $\bm{0.62\pm 0.04}$
Yacht | $0.88\pm 0.22$ | $0.84\pm 0.27$ | $1.19\pm 0.49$ | $0.79\pm 0.26$ | $\bm{0.56\pm 0.20}$
Year |  $8.86\pm$ NA |  $8.77\pm$ NA |  $8.79\pm$ NA |  $9.20\pm$ NA |  $\bm{8.69\pm}$ NA
# best | $0$ | $1$ | $1$ | $0$ | $\bm{9}$

Table 3: NLL of UCI regression tasks.

Dataset |  |  | NLL $\downarrow$ |  |
---|---|---|---|---|---
| PBP | MC Dropout | Deep Ensembles | GCDS | CARD (ours)
Boston | $2.53\pm 0.27$ | $2.46\pm 0.12$ | $2.35\pm 0.16$ | $18.66\pm 8.92$ | $\bm{2.32\pm 0.17}$
Concrete | $3.19\pm 0.05$ | $3.21\pm 0.18$ | $\bm{2.93\pm 0.12}$ | $13.64\pm 6.88$ | $2.99\pm 0.12$
Energy | $2.05\pm 0.05$ | $1.50\pm 0.11$ | $1.40\pm 0.27$ | $1.46\pm 0.72$ | $\bm{0.94\pm 0.28}$
Kin8nm | $-0.83\pm 0.02$ | $-1.14\pm 0.05$ | $-1.06\pm 0.02$ | $-0.38\pm 0.36$ | $\bm{-1.31\pm 0.02}$
Naval | $-3.97\pm 0.10$ | $-4.45\pm 0.38$ | $-5.94\pm 0.10$ | $-5.06\pm 0.48$ | $\bm{-7.29\pm 0.03}$
Power | $2.92\pm 0.02$ | $2.90\pm 0.03$ | $2.89\pm 0.02$ | $2.83\pm 0.06$ | $\bm{2.82\pm 0.02}$
Protein | $3.05\pm 0.00$ | $2.80\pm 0.08$ | $2.89\pm 0.02$ | $2.81\pm 0.09$ | $\bm{2.47\pm 0.01}$
Wine | $1.03\pm 0.03$ | $0.93\pm 0.06$ | $0.96\pm 0.06$ | $6.52\pm 21.86$ | $\bm{0.87\pm 0.08}$
Yacht | $1.58\pm 0.08$ | $1.73\pm 0.22$ | $1.11\pm 0.18$ | $\bm{0.61\pm 0.34}$ | $0.89\pm 0.07$
Year |  $3.69\pm$ NA |  $3.42\pm$ NA |  $3.44\pm$ NA |  $3.43\pm$ NA |  $\bm{3.34\pm}$ NA
# best | $0$ | $0$ | $1$ | $1$ | $\bm{8}$

Table 4: QICE (in $\%$) of UCI regression tasks.

Dataset |  |  | QICE $\downarrow$ |  |
---|---|---|---|---|---
| PBP | MC Dropout | Deep Ensembles | GCDS | CARD (ours)
Boston | $3.50\pm 0.88$ | $3.82\pm 0.82$ | $\bm{3.37\pm 0.00}$ | $11.73\pm 1.05$ | $3.50\pm 0.89$
Concrete | $2.52\pm 0.60$ | $4.17\pm 1.06$ | $2.68\pm 0.64$ | $10.49\pm 1.01$ | $\bm{2.38\pm 0.61}$
Energy | $6.54\pm 0.90$ | $5.22\pm 1.02$ | $\bm{3.62\pm 0.58}$ | $7.41\pm 2.19$ | $5.39\pm 0.89$
Kin8nm | $1.31\pm 0.25$ | $1.50\pm 0.32$ | $1.17\pm 0.22$ | $7.73\pm 0.80$ | $\bm{0.90\pm 0.28}$
Naval | $4.06\pm 1.25$ | $12.50\pm 1.95$ | $6.64\pm 0.60$ | $5.76\pm 2.25$ | $\bm{3.70\pm 0.33}$
Power | $\bm{0.82\pm 0.19}$ | $1.32\pm 0.37$ | $1.09\pm 0.26$ | $1.77\pm 0.33$ | $0.85\pm 0.18$
Protein | $1.69\pm 0.09$ | $2.82\pm 0.41$ | $2.17\pm 0.16$ | $2.33\pm 0.18$ | $\bm{0.75\pm 0.04}$
Wine | $\bm{2.22\pm 0.64}$ | $2.79\pm 0.56$ | $2.37\pm 0.63$ | $3.13\pm 0.79$ | $3.68\pm 0.82$
Yacht | $6.93\pm 1.74$ | $10.33\pm 1.34$ | $7.22\pm 1.41$ | $\bm{5.01\pm 1.02}$ | $8.17\pm 0.85$
Year |  $2.96\pm$ NA |  $2.43\pm$ NA |  $2.56\pm$ NA |  $1.61\pm$ NA |  $\bm{0.56\pm}$ NA
# best | $2$ | $0$ | $2$ | $1$ | $\bm{5}$

We observe that CARD outperforms existing methods, often by a considerable
margin (especially on larger datasets), in all metrics for most of the
datasets, and is competitive with the best method for the remaining ones: we
obtain state-of-the-art results in $9$ out of $10$ datasets in terms of RMSE,
$8$ out of $10$ for NLL, and $5$ out of $10$ for QICE. Note that although we
do not explicitly optimize our model by MSE or by NLL, we still obtain better
results than models trained with these objectives.

###  3.2 Classification

Similar to Lakshminarayanan et al. (2017), our motivation for classification
is not to achieve the state-of-the-art performance in terms of mean accuracy
on the benchmark datasets, which is strongly related to network architecture
design. Our goal is two-fold:

  1. 1.

We aim to solve classification problems via a generative model, emphasizing
its capability to improve the performance of a base classifier with
deterministic outputs;

  2. 2.

We intent to introduce the idea of uncertainty to classification predictions
at the instance level.

As another type of supervised learning problems, classification is different
than regression mainly for the response variable being discrete class labels
instead of continuous values. Conventionally, the output from a classification
algorithm is a point estimate, a value being casted as between $0$ and $1$,
with the intention to align with the human cognitive intuition of probability
for each class label (Cosmides and Tooby, 1996). In other words, that point
estimate should express a level of confidence, or the likelihood of
correctness, in that prediction by the model. Metrics like Expected
Calibration Error (ECE) and Maximum Calibration Error (MCE) have been proposed
to evaluate the alignment between classifier predictions and the true
correctness likelihood, and calibration methods like Platt scaling and
isotonic regression have been developed to improve such alignment (Guo et al.,
2017), but all are based on point estimate predictions. In the next section,
we introduce an alternative in accessing model confidence, at the level of
individual instances, specifically tailored for models with stochastic output.

####  3.2.1 Estimating Instance Level Prediction Confidence via Generative
Models

We utilize the stochasticity of generative models to construct our framework
of evaluating prediction uncertainty at the instance level. As introduced in
Section 2.3, we view each one-hot label as a class prototype, and we use the
generative model to reconstruct this prototype in a stochastic fashion. The
intuition is that if the learned model knows well about which class a
particular instance belongs to, it would precisely reconstruct the original
prototype vector; otherwise if the model is unsure about a new instance based
on the learned patterns, different reconstructions tend to have more
variations: under the context of denoising diffusion models, given different
samples from the prior distribution at timestep $T$, the class prototype
reconstructions would appear rather different from each other.

We propose the following framework to access model confidence: during the test
time, we sample $N$ class prototype reconstructions for each instance,
converting them into probability scale with Eq. (10), and make the following
two computations:

  1. 1.

We calculate the prediction interval width (PIW) between the $2.5^{th}$ and
$97.5^{th}$ percentiles of the $N$ predicted probabilities for the true class
associated with each instance;

  2. 2.

We apply paired two-sample $t$-test as an uncertainty estimation method
proposed in Fan et al. (2021): we obtain the most and second most predicted
classes for each instance, and test whether the difference in their mean
predicted probability is statistically significant.

####  3.2.2 Classification with Prediction Uncertainty Evaluation

We demonstrate our experimental results on the FashionMNIST dataset. We
contextualize the performance of CARD through the mean accuracy of other BNNs
with the LeNet CNN (LeCun et al., 1998) architecture in Table 5, where the
metrics were reported in (Tomczak et al., 2021). Following the recipe in
Section 2.3, we first pre-train a deterministic classifier with the same LeNet
architecture, and achieve a test accuracy of $91.12\%$. After training CARD,
we obtain our instance prediction through majority vote, i.e., the most
predicted class label among its $N$ samples, and achieve an improved test
accuracy of $91.79\%$, showing its ability to improve the prediction accuracy
from the base classifier. In addition, we apply a different network
architecture for the pre-trained classifier $f_{\phi}$, ResNet-18, and achieve
a test accuracy of $91.19\%$. The trained CARD model conditioned on such
$f_{\phi}$ achieves an improved test accuracy of $91.71\%$.

Table 5: Comparison of mean accuracy for FashionMNIST classification task with
other BNNs and the corresponding pre-trained classifier $f_{\phi}$.

Model | CMV-MF-VI | CM-MF-VI | CV-MF-VI | CM-MF-VI OPT | MF-VI | MAP | MC Dropout | MF-VI EB | ResNet-18 | ResNet-18+CARD | LeNet | LeNet+CARD
---|---|---|---|---|---|---|---|---|---|---|---|---
Accuracy | $91.10\%$ | $90.95\%$ | $88.53\%$ | $90.67\%$ | $87.04\%$ | $88.06\%$ | $87.99\%$ | $87.04\%$ | $91.19\%$ | $\bm{91.71\%}$ | $91.12\%$ | $\bm{91.79\%}$

In Table 6, we report the mean PIW among both correct and incorrect
predictions, as well as the mean accuracy among both groups rejected and not-
rejected by the paired two-sample $t$-test ($\alpha=0.01$). We report these
metrics for all test instances and for each class label along with their group
accuracy.

Table 6: PIW (in $\%$) and $t$-test results for FashionMNIST classification
task.

Class | Accuracy | PIW | Accuracy by $t$-test Status
---|---|---|---
|  | Correct | Incorrect | Rejected | Not-Rejected (Count)
All | $91.79\%$ | $0.67$ | $3.20$ | $92.07\%$ |  $55.84\%$ (77)
1 | $88.10\%$ | $0.96$ | $3.40$ | $88.45\%$ |  $61.54\%$ (13)
2 | $98.50\%$ | $0.39$ | $2.08$ | $98.60\%$ |  $66.67\%$ (3)
3 | $87.70\%$ | $0.84$ | $3.42$ | $88.00\%$ |  $50.00\%$ (8)
4 | $91.10\%$ | $0.76$ | $2.97$ | $91.59\%$ |  $53.85\%$ (13)
5 | $87.90\%$ | $0.89$ | $2.91$ | $88.40\%$ |  $33.33\%$ (9)
6 | $97.20\%$ | $0.41$ | $2.89$ | $97.29\%$ |  $66.67\%$ (3)
7 | $74.80\%$ | $1.37$ | $3.26$ | $74.90\%$ |  $70.00\%$ (20)
8 | $97.40\%$ | $0.49$ | $1.60$ | $97.50\%$ |  $50.00\%$ (2)
9 | $98.40\%$ | $0.34$ | $1.93$ | $98.50\%$ |  $0.00\%$ (1)
10 | $96.80\%$ | $0.46$ | $5.59$ | $97.09\%$ |  $40.00\%$ (5)

We observe from Table 6 that under the scope of the entire test set, mean PIW
of the true class label among the correct predictions is narrower than that of
the incorrect predictions by an order of magnitude, indicating that CARD is
much more confident when making correct predictions; on the other hand, CARD
also seems to be aware of what it does not know. More revealing observations
can be made when comparing mean PIWs across different classes: a class with
more accurate predictions tends to have a sharper contrast between correct and
incorrect predictions; additionally, the metric values of both types of
predictions tend to be larger in a less accurate class. Furthermore, it is
worth mentioning that if we make the prediction merely by the class with the
narrowest PIW for each instance, we can already obtain a test accuracy of
$85.33\%$, suggesting a strong correlation between the model confidence and
its prediction correctness. Meanwhile, we observe that the accuracy of test
instances rejected by the $t$-test is generally much higher than that of the
not-rejected ones, both among the entire set and within each class.

We point out that these metrics can thus reflect how sure CARD is about its
predictions, and can be used as an important indicator of whether the model
prediction can be trusted or not. Therefore, it has the potential to be
further applied in the human-machine collaboration domain (Madras et al.,
2018; Raghu et al., 2019; Wilder et al., 2020; Gao et al., 2021), such that
one can apply such uncertainty measurement to decide if we can directly accept
the model prediction, or we need to allocate the instance to humans for
further evaluation.

##  4 Conclusion

In this paper, we propose Classification And Regression Diffusion (CARD)
models, a class of conditional generative models that approaches supervised
learning problems from a conditional generation perspective. Without training
with objectives directly related to the evaluation metrics, we achieve state-
of-the-art results on benchmark regression tasks. Furthermore, CARD exhibits a
strong ability to represent the conditional distribution with multiple density
modes. We also propose a new metric Quantile Interval Coverage Error (QICE),
which can be viewed as a generalized version of negative log-likelihood in
evaluating how well the model fits the data. Lastly, we introduce a framework
to evaluate prediction uncertainty at the instance level for classification
tasks.

## References

  * Bishop (1994) Christopher Bishop.  Mixture density networks.  In _Aston University Neural Computing Research Group Report_ , 1994\.
  * Blundell et al. (2015) Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra.  Weight uncertainty in neural network.  In _Proceedings of the 32nd International Conference on Machine Learning_ , 2015.
  * Brier (1950) Glenn W. Brier.  Verification of forecasts expressed in terms of probability.  In _Monthly weather review_ , 1950.
  * Cosmides and Tooby (1996) Leda Cosmides and John Tooby.  Are humans good intuitive statisticians after all? Rethinking some conclusions from the literature on judgment under uncertainty.  In _cognition_ , volume 58(1), pages 1–73, 1996.
  * Dhariwal and Nichol (2021) Prafulla Dhariwal and Alexander Quinn Nichol.  Diffusion models beat GANs on image synthesis.  In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, _Advances in Neural Information Processing Systems_ , 2021.  URL https://openreview.net/forum?id=AAWuCvzaVt.
  * Dua and Graff (2017) Dheeru Dua and Casey Graff.  UCI Machine Learning Repository, 2017.  URL http://archive.ics.uci.edu/ml.
  * Fan et al. (2021) Xinjie Fan, Shujian Zhang, Korawat Tanwisuth, Xiaoning Qian, and Mingyuan Zhou.  Contextual dropout: An efficient sample-dependent dropout module.  In _International Conference on Learning Representations_ , 2021.
  * Gal and Ghahramani (2016) Yarin Gal and Zoubin Ghahramani.  Dropout as a Bayesian approximation: Representing model uncertainty in deep learning.  In _Proceedings of the 33rd International Conference on Machine Learning_ , 2016.
  * Gal et al. (2017) Yarin Gal, Jiri Hron, and Alex Kendall.  Concrete dropout.  In _Advances in Neural Information Processing Systems 30_ , 2017.
  * Gao et al. (2021) Ruijiang Gao, Maytal Saar-Tsechansky, Maria De-Arteaga, Ligong Han, Min Kyung Lee, and Matthew Lease.  Human-AI collaboration with bandit feedback.  In _International Joint Conferences on Artificial Intelligence_ , 2021\.
  * Guo et al. (2017) Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger.  On calibration of modern neural networks.  In _Proceedings of the 34th International Conference on Machine Learning_ , 2017.
  * Hernández-Lobato and Adams (2015) José Miguel Hernández-Lobato and Ryan P. Adams.  Probabilistic backpropagation for scalable learning of Bayesian neural networks.  In _Proceedings of the 32nd International Conference on International Conference on Machine Learning_ , 2015.
  * Ho et al. (2020) Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models.  In _Advances in Neural Information Processing Systems_ , 2020.
  * Hyvärinen and Dayan (2005) Aapo Hyvärinen and Peter Dayan.  Estimation of non-normalized statistical models by score matching.  _Journal of Machine Learning Research_ , 6(4), 2005.
  * Joachims (2021) Per Joachims.  nnuncert: Uncertainty quantification with BNNs, 2021.  URL https://github.com/nnuncert/nnuncert.
  * Kendall and Gal (2017) Alex Kendall and Yarin Gal.  What uncertainties do we need in Bayesian deep learning for computer vision?  In _31st Conference on Neural Information Processing System_ , 2017\.
  * Kingma and Ba (2015) Diederik P. Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  In _Proceedings of the 3rd International Conference on Learning Representations_ , 2015.
  * Lakshminarayanan et al. (2017) Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell.  Simple and scalable predictive uncertainty estimation using deep ensembles.  In _Proceedings of the 31st Conference on Neural Information Processing Systems_ , 2017.
  * LeCun et al. (1998) Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner.  Gradient-based learning applied to document recognition.  _Proceedings of the IEEE_ , 86(11):2278–2324, 1998.  doi: 10.1109/5.726791.
  * Loshchilov and Hutter (2017) Ilya Loshchilov and Frank Hutter.  SGDR: Stochastic gradient descent with warm restarts.  In _Proceedings of the 5th International Conference on Learning Representations_ , 2017.
  * Madras et al. (2018) David Madras, Toniann Pitassi, and Richard Zemel.  Predict responsibly: Improving fairness and accuracy by learning to defer.  In _Proceedings of the 32nd Conference on Neural Information Processing Systems_ , 2018.
  * Mukhoti and Gal (2018) Jishnu Mukhoti and Yarin Gal.  Evaluating Bayesian deep learning methods for semantic segmentation.  In _arXiv preprint arXiv: 1811.12709_ , 2018.
  * Neal (2011) Radford M Neal.  MCMC using Hamiltonian dynamics.  _Handbook of Markov Chain Monte Carlo_ , page 113, 2011.
  * Nichol et al. (2021) Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen.  GLIDE: Towards photorealistic image generation and editing with text-guided diffusion models.  _arXiv preprint arXiv:2112.10741_ , 2021.
  * Pandey et al. (2022) Kushagra Pandey, Avideep Mukherjee, Piyush Rai, and Abhishek Kumar.  DiffuseVAE: Efficient, controllable and high-fidelity generation from low-dimensional latents.  _arXiv preprint arXiv:2201.00308_ , 2022.
  * Raghu et al. (2019) Maithra Raghu, Katy Blumer, Greg Corrado, Jon Kleinberg, Ziad Obermeyer, and Sendhil Mullainathan.  The algorithmic automation problem: Prediction, triage, and human effort.  In _arXiv preprint arXiv:1903.12220_ , 2019.
  * Ramesh et al. (2022) Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen.  Hierarchical text-conditional image generation with CLIP latents.  _arXiv preprint arXiv:2204.06125_ , 2022.
  * Reddi et al. (2018) Sashank J. Reddi, Satyen Kale, and Sanjiv Kumar.  On the convergence of Adam and beyond.  In _Proceedings of the 6th International Conference on Learning Representations_ , 2018.
  * Ren et al. (2019) Hongyu Ren, Shengjia Zhao, and Stefano Ermon.  Adaptive antithetic sampling for variance reduction.  In _Proceedings of the 36th International Conference on Machine Learning_ , 2019.
  * Sohl-Dickstein et al. (2015) Jascha Sohl-Dickstein, Eric A. Weiss, Niru Maheswaranathan, and Surya Ganguli.  Deep unsupervised learning using nonequilibrium thermodynamics, 2015.
  * Song and Ermon (2019) Yang Song and Stefano Ermon.  Generative modeling by estimating gradients of the data distribution.  In _Advances in Neural Information Processing Systems_ , pages 11918–11930, 2019.
  * Song and Ermon (2020) Yang Song and Stefano Ermon.  Improved techniques for training score-based generative models.  _arXiv preprint arXiv:2006.09011_ , 2020.
  * Song et al. (2021) Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole.  Score-based generative modeling through stochastic differential equations.  In _International Conference on Learning Representations_ , 2021.  URL https://openreview.net/forum?id=PxTIG12RRHS.
  * Srivastava et al. (2014) Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov.  Dropout: A simple way to prevent neural networks from overfitting, 2014\.
  * Tomczak et al. (2021) Marcin B. Tomczak, Siddharth Swaroop, Andrew Y. K. Foong, and Richard E. Turner.  Collapsed variational bounds for Bayesian neural networks.  In _Proceedings of the 35th Conference on Neural Information Processing Systems_ , 2021.
  * Vincent (2011) Pascal Vincent.  A connection between score matching and denoising autoencoders.  _Neural computation_ , 23(7):1661–1674, 2011\.
  * Wang and Zhou (2020) Zhendong Wang and Mingyuan Zhou.  Thompson sampling via local uncertainty.  In _International Conference on Machine Learning_ , pages 10115–10125. PMLR, 2020.
  * Welling and Teh (2011) Max Welling and Yee W Teh.  Bayesian learning via stochastic gradient Langevin dynamics.  In _Proceedings of the 28th international conference on machine learning (ICML-11)_ , pages 681–688. Citeseer, 2011.
  * Wilder et al. (2020) Bryan Wilder, Eric Horvitz, and Ece Kamar.  Learning to complement humans.  In _International Joint Conferences on Artificial Intelligence_ , 2020\.
  * Xiao et al. (2021) Zhisheng Xiao, Karsten Kreis, and Arash Vahdat.  Tackling the generative learning trilemma with denoising diffusion GANs.  _arXiv preprint arXiv:2112.07804_ , 2021.
  * Yao et al. (2019) Jiayu Yao, Weiwei Pan, Soumya Ghosh, and Finale Doshi-Velez.  Quality of uncertainty quantification for Bayesian neural network inference.  In _ICML 2019 Workshop on Uncertainty and Robustness in Deep Learning_ , 2019.
  * Zheng et al. (2022) Huangjie Zheng, Pengcheng He, Weizhu Chen, and Mingyuan Zhou.  Truncated diffusion probabilistic models.  _arXiv preprint arXiv:2202.09671_ , 2022.
  * Zhou et al. (2021) Xingyu Zhou, Yuling Jiao, Jin Liu, and Jian Huang.  A deep generative approach to conditional sampling.  _Journal of the American Statistical Association_ , pages 1–28, 2021\.

##  Appendix A Theoretical Derivations and Additional Experiment Results

###  A.1 Theoretical Derivations

##### Derivation for Forward Process Posteriors:

In this section, we derive the mean and variance of the forward process
posteriors $q({\bm{y}}_{t-1}\,|\,{\bm{y}}_{t},{\bm{y}}_{0},{\bm{x}})$ in Eq.
9, which are tractable when conditioned on ${\bm{y}}_{0}$:

$\displaystyle
q({\bm{y}}_{t-1}\,|\,{\bm{y}}_{t},{\bm{y}}_{0},{\bm{x}})\propto\underbrace{q\big{(}{\bm{y}}_{t}\,|\,{\bm{y}}_{t-1},f_{\phi}({\bm{x}})\big{)}}_{\text{Eq.
}(\ref{eq:form_1_forward_one_step})}\underbrace{q\big{(}{\bm{y}}_{t-1}\,|\,{\bm{y}}_{0},f_{\phi}({\bm{x}})\big{)}}_{\text{Eq.
}(\ref{eq:form_1_forward_t_steps})}$ (13)
$\displaystyle\propto\exp\Bigg{(}-\frac{1}{2}\bigg{(}\frac{\big{(}{\bm{y}}_{t}-(1-\sqrt{\alpha_{t}})f_{\phi}({\bm{x}})-\sqrt{\alpha_{t}}{\bm{y}}_{t-1}\big{)}^{2}}{\beta_{t}}$
$\displaystyle+\frac{\big{(}{\bm{y}}_{t-1}-\sqrt{\bar{\alpha}_{t-1}}{\bm{y}}_{0}-(1-\sqrt{\bar{\alpha}_{t-1}})f_{\phi}({\bm{x}})\big{)}^{2}}{1-\bar{\alpha}_{t-1}}\bigg{)}\Bigg{)}$
(14)
$\displaystyle\propto\exp\Bigg{(}-\frac{1}{2}\bigg{(}\frac{\alpha_{t}{\bm{y}}_{t-1}^{2}-2\sqrt{\alpha_{t}}\big{(}{\bm{y}}_{t}-(1-\sqrt{\alpha_{t}})f_{\phi}({\bm{x}})\big{)}{\bm{y}}_{t-1}}{\beta_{t}}$
$\displaystyle+\frac{{\bm{y}}_{t-1}^{2}-2\big{(}\sqrt{\bar{\alpha}_{t-1}}{\bm{y}}_{0}+(1-\sqrt{\bar{\alpha}_{t-1}})f_{\phi}({\bm{x}})\big{)}{\bm{y}}_{t-1}}{1-\bar{\alpha}_{t-1}}\bigg{)}\Bigg{)}$
(15)
$\displaystyle=\exp\Bigg{(}-\frac{1}{2}\bigg{(}\Big{(}\underbrace{\frac{\alpha_{t}}{\beta_{t}}+\frac{1}{1-\bar{\alpha}_{t-1}}}_{\leavevmode\hbox
to11.75pt{\vbox to11.75pt{\pgfpicture\makeatletter\hbox{\hskip
5.87407pt\lower-5.87407pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{
{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{
}{{}{{{}}}{{}}{}{}{}{}{}{}{}{}{}{{}\pgfsys@moveto{5.67407pt}{0.0pt}\pgfsys@curveto{5.67407pt}{3.13374pt}{3.13374pt}{5.67407pt}{0.0pt}{5.67407pt}\pgfsys@curveto{-3.13374pt}{5.67407pt}{-5.67407pt}{3.13374pt}{-5.67407pt}{0.0pt}\pgfsys@curveto{-5.67407pt}{-3.13374pt}{-3.13374pt}{-5.67407pt}{0.0pt}{-5.67407pt}\pgfsys@curveto{3.13374pt}{-5.67407pt}{5.67407pt}{-3.13374pt}{5.67407pt}{0.0pt}\pgfsys@closepath\pgfsys@moveto{0.0pt}{0.0pt}\pgfsys@stroke\pgfsys@invoke{
} }{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-1.75pt}{-2.25555pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\hbox{{1}}
}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} }
\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope{{{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}}\Big{)}{\bm{y}}_{t-1}^{2}$
$\displaystyle-2\Big{(}\underbrace{\frac{\sqrt{\bar{\alpha}_{t-1}}}{1-\bar{\alpha}_{t-1}}{\bm{y}}_{0}+\frac{\sqrt{\alpha_{t}}}{\beta_{t}}{\bm{y}}_{t}+\big{(}\frac{\sqrt{\alpha_{t}}(\sqrt{\alpha_{t}}-1)}{\beta_{t}}+\frac{1-\sqrt{\bar{\alpha}_{t-1}}}{1-\bar{\alpha}_{t-1}}\big{)}f_{\phi}({\bm{x}})}_{\leavevmode\hbox
to11.75pt{\vbox to11.75pt{\pgfpicture\makeatletter\hbox{\hskip
5.87407pt\lower-5.87407pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{
{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{
}{{}{{{}}}{{}}{}{}{}{}{}{}{}{}{}{{}\pgfsys@moveto{5.67407pt}{0.0pt}\pgfsys@curveto{5.67407pt}{3.13374pt}{3.13374pt}{5.67407pt}{0.0pt}{5.67407pt}\pgfsys@curveto{-3.13374pt}{5.67407pt}{-5.67407pt}{3.13374pt}{-5.67407pt}{0.0pt}\pgfsys@curveto{-5.67407pt}{-3.13374pt}{-3.13374pt}{-5.67407pt}{0.0pt}{-5.67407pt}\pgfsys@curveto{3.13374pt}{-5.67407pt}{5.67407pt}{-3.13374pt}{5.67407pt}{0.0pt}\pgfsys@closepath\pgfsys@moveto{0.0pt}{0.0pt}\pgfsys@stroke\pgfsys@invoke{
} }{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-1.75pt}{-2.25555pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\hbox{{2}}
}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} }
\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope{{{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}}\Big{)}{\bm{y}}_{t-1}\bigg{)}\Bigg{)},$
(16)

where

$\displaystyle\leavevmode\hbox to14.18pt{\vbox
to14.18pt{\pgfpicture\makeatletter\hbox{\hskip 7.09111pt\lower-7.09111pt\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{
{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{
}{{}{{{}}}{{}}{}{}{}{}{}{}{}{}{}{{}\pgfsys@moveto{6.89111pt}{0.0pt}\pgfsys@curveto{6.89111pt}{3.8059pt}{3.8059pt}{6.89111pt}{0.0pt}{6.89111pt}\pgfsys@curveto{-3.8059pt}{6.89111pt}{-6.89111pt}{3.8059pt}{-6.89111pt}{0.0pt}\pgfsys@curveto{-6.89111pt}{-3.8059pt}{-3.8059pt}{-6.89111pt}{0.0pt}{-6.89111pt}\pgfsys@curveto{3.8059pt}{-6.89111pt}{6.89111pt}{-3.8059pt}{6.89111pt}{0.0pt}\pgfsys@closepath\pgfsys@moveto{0.0pt}{0.0pt}\pgfsys@stroke\pgfsys@invoke{
} }{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-2.5pt}{-3.22221pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\hbox{{1}}
}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} }
\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope{{{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}=\frac{\alpha_{t}(1-\bar{\alpha}_{t-1})+\beta_{t}}{\beta_{t}(1-\bar{\alpha}_{t-1})}=\frac{1-\bar{\alpha}_{t}}{\beta_{t}(1-\bar{\alpha}_{t-1})},$
(17)

and we have the posterior variance

$\displaystyle\tilde{\beta}_{t}=\frac{1}{\leavevmode\hbox to14.18pt{\vbox
to14.18pt{\pgfpicture\makeatletter\hbox{\hskip 7.09111pt\lower-7.09111pt\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{
{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{
}{{}{{{}}}{{}}{}{}{}{}{}{}{}{}{}{{}\pgfsys@moveto{6.89111pt}{0.0pt}\pgfsys@curveto{6.89111pt}{3.8059pt}{3.8059pt}{6.89111pt}{0.0pt}{6.89111pt}\pgfsys@curveto{-3.8059pt}{6.89111pt}{-6.89111pt}{3.8059pt}{-6.89111pt}{0.0pt}\pgfsys@curveto{-6.89111pt}{-3.8059pt}{-3.8059pt}{-6.89111pt}{0.0pt}{-6.89111pt}\pgfsys@curveto{3.8059pt}{-6.89111pt}{6.89111pt}{-3.8059pt}{6.89111pt}{0.0pt}\pgfsys@closepath\pgfsys@moveto{0.0pt}{0.0pt}\pgfsys@stroke\pgfsys@invoke{
} }{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-2.5pt}{-3.22221pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\hbox{{1}}
}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} }
\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope{{{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys<EMAIL_ADDRESS>(18)

Meanwhile, the following coefficients of the terms in the posterior mean
through dividing each coefficient in 2 by 1:

$\displaystyle\gamma_{0}=\frac{\sqrt{\bar{\alpha}_{t-1}}}{1-\bar{\alpha}_{t-1}}/\leavevmode\hbox
to14.18pt{\vbox to14.18pt{\pgfpicture\makeatletter\hbox{\hskip
7.09111pt\lower-7.09111pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{
{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{
}{{}{{{}}}{{}}{}{}{}{}{}{}{}{}{}{{}\pgfsys@moveto{6.89111pt}{0.0pt}\pgfsys@curveto{6.89111pt}{3.8059pt}{3.8059pt}{6.89111pt}{0.0pt}{6.89111pt}\pgfsys@curveto{-3.8059pt}{6.89111pt}{-6.89111pt}{3.8059pt}{-6.89111pt}{0.0pt}\pgfsys@curveto{-6.89111pt}{-3.8059pt}{-3.8059pt}{-6.89111pt}{0.0pt}{-6.89111pt}\pgfsys@curveto{3.8059pt}{-6.89111pt}{6.89111pt}{-3.8059pt}{6.89111pt}{0.0pt}\pgfsys@closepath\pgfsys@moveto{0.0pt}{0.0pt}\pgfsys@stroke\pgfsys@invoke{
} }{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-2.5pt}{-3.22221pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\hbox{{1}}
}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} }
\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope{{{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}=\frac{\sqrt{\bar{\alpha}_{t-1}}}{1-\bar{\alpha}_{t}}\beta_{t},$
(19)
$\displaystyle\gamma_{1}=\frac{\sqrt{\alpha_{t}}}{\beta_{t}}/\leavevmode\hbox
to14.18pt{\vbox to14.18pt{\pgfpicture\makeatletter\hbox{\hskip
7.09111pt\lower-7.09111pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{
{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{
}{{}{{{}}}{{}}{}{}{}{}{}{}{}{}{}{{}\pgfsys@moveto{6.89111pt}{0.0pt}\pgfsys@curveto{6.89111pt}{3.8059pt}{3.8059pt}{6.89111pt}{0.0pt}{6.89111pt}\pgfsys@curveto{-3.8059pt}{6.89111pt}{-6.89111pt}{3.8059pt}{-6.89111pt}{0.0pt}\pgfsys@curveto{-6.89111pt}{-3.8059pt}{-3.8059pt}{-6.89111pt}{0.0pt}{-6.89111pt}\pgfsys@curveto{3.8059pt}{-6.89111pt}{6.89111pt}{-3.8059pt}{6.89111pt}{0.0pt}\pgfsys@closepath\pgfsys@moveto{0.0pt}{0.0pt}\pgfsys@stroke\pgfsys@invoke{
} }{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-2.5pt}{-3.22221pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\hbox{{1}}
}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} }
\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope{{{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}=\frac{1-\bar{\alpha}_{t-1}}{1-\bar{\alpha}_{t}}\sqrt{\alpha_{t}},$
(20)

and

$\displaystyle\gamma_{2}$
$\displaystyle=\bigg{(}\frac{\sqrt{\alpha_{t}}(\sqrt{\alpha_{t}}-1)}{\beta_{t}}+\frac{1-\sqrt{\bar{\alpha}_{t-1}}}{1-\bar{\alpha}_{t-1}}\bigg{)}/\leavevmode\hbox
to14.18pt{\vbox to14.18pt{\pgfpicture\makeatletter\hbox{\hskip
7.09111pt\lower-7.09111pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{
{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{
}{{}{{{}}}{{}}{}{}{}{}{}{}{}{}{}{{}\pgfsys@moveto{6.89111pt}{0.0pt}\pgfsys@curveto{6.89111pt}{3.8059pt}{3.8059pt}{6.89111pt}{0.0pt}{6.89111pt}\pgfsys@curveto{-3.8059pt}{6.89111pt}{-6.89111pt}{3.8059pt}{-6.89111pt}{0.0pt}\pgfsys@curveto{-6.89111pt}{-3.8059pt}{-3.8059pt}{-6.89111pt}{0.0pt}{-6.89111pt}\pgfsys@curveto{3.8059pt}{-6.89111pt}{6.89111pt}{-3.8059pt}{6.89111pt}{0.0pt}\pgfsys@closepath\pgfsys@moveto{0.0pt}{0.0pt}\pgfsys@stroke\pgfsys@invoke{
} }{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-2.5pt}{-3.22221pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\hbox{{1}}
}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} }
\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope{{{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}$
$\displaystyle=\frac{\alpha_{t}-\bar{\alpha}_{t}-\sqrt{\alpha_{t}}(1-\bar{\alpha}_{t-1})+\beta_{t}-\beta_{t}\sqrt{\bar{\alpha}_{t-1}}}{1-\bar{\alpha}_{t}}$
$\displaystyle=1+\frac{(\sqrt{\bar{\alpha}_{t}}-1)(\sqrt{\alpha_{t}}+\sqrt{\bar{\alpha}_{t-1}})}{1-\bar{\alpha}_{t}},$
(21)

which together give us the posterior mean

$\displaystyle\tilde{{\bm{\mu}}}\big{(}{\bm{y}}_{t},{\bm{y}}_{0},f_{\phi}({\bm{x}})\big{)}=\gamma_{0}{\bm{y}}_{0}+\gamma_{1}{\bm{y}}_{t}+\gamma_{2}f_{\phi}({\bm{x}}).$

##### Derivation for Forward Process Sampling Distribution with Arbitrary
Timesteps:

For completeness, we include the derivation for the parameters of the forward
diffusion process sampling distribution with arbitrary $t$ steps.

The expectation term is based on Eqs. $(36)$–$(38)$ in Pandey et al. [2022].
From Eq. (7), we have that for all $t=1,\dots,T$,

$\displaystyle{\bm{y}}_{t}=\sqrt{1-\beta_{t}}{\bm{y}}_{t-1}+(1-\sqrt{1-\beta_{t}})f_{\phi}({\bm{x}})+\sqrt{\beta_{t}}\bm{\epsilon},\text{
where }\bm{\epsilon}\sim\mathcal{N}(\bm{0},{\bm{I}}).$ (22)

Taking expectation of both sides, we have

$\displaystyle\mathbb{E}({\bm{y}}_{t})$
$\displaystyle=\sqrt{1-\beta_{t}}\mathbb{E}({\bm{y}}_{t-1})+(1-\sqrt{1-\beta_{t}})f_{\phi}({\bm{x}})$
(23)
$\displaystyle=\sqrt{1-\beta_{t}}\big{(}\sqrt{1-\beta_{t-1}}\mathbb{E}({\bm{y}}_{t-2})+(1-\sqrt{1-\beta_{t-1}})f_{\phi}({\bm{x}})\big{)}+(1-\sqrt{1-\beta_{t}})f_{\phi}({\bm{x}})$
$\displaystyle=\sqrt{(1-\beta_{t})(1-\beta_{t-1})}\mathbb{E}({\bm{y}}_{t-2})+\big{(}1-\sqrt{(1-\beta_{t})(1-\beta_{t-1})}\big{)}f_{\phi}({\bm{x}})$
(24) $\displaystyle\vdots$
$\displaystyle=\sqrt{\prod_{t=2}^{t}(1-\beta_{t})}\mathbb{E}({\bm{y}}_{1})+\Bigg{(}1-\sqrt{\prod_{t=2}^{t}(1-\beta_{t})}\Bigg{)}f_{\phi}({\bm{x}})$
(25)
$\displaystyle=\sqrt{\prod_{t=2}^{t}(1-\beta_{t})}\big{(}\sqrt{1-\beta_{1}}{\bm{y}}_{0}+(1-\sqrt{1-\beta_{1}})f_{\phi}({\bm{x}})\big{)}+\Bigg{(}1-\sqrt{\prod_{t=2}^{t}(1-\beta_{t})}\Bigg{)}f_{\phi}({\bm{x}})$
(26)
$\displaystyle=\sqrt{\prod_{t=1}^{t}(1-\beta_{t})}{\bm{y}}_{0}+\Bigg{(}1-\sqrt{\prod_{t=1}^{t}(1-\beta_{t})}\Bigg{)}f_{\phi}({\bm{x}})$
(27)
$\displaystyle=\sqrt{\bar{\alpha}_{t}}{\bm{y}}_{0}+(1-\sqrt{\bar{\alpha}_{t}})f_{\phi}({\bm{x}}).$
(28)

Meanwhile, since the addition of two independent Gaussians with different
variances, $\mathcal{N}(\bm{0},\sigma_{a}^{2}{\bm{I}})$ and
$\mathcal{N}(\bm{0},\sigma_{b}^{2}{\bm{I}})$, is
$\mathcal{N}(\bm{0},(\sigma_{a}^{2}+\sigma_{b}^{2}){\bm{I}})$, we can derive
the variance term accordingly,

$\displaystyle\sigma^{2}({\bm{y}}_{t})=(1-\bar{\alpha}_{t}){\bm{I}}.$ (29)

###  A.2 Patch Accuracy vs Patch Uncertainty (PAvPU)

We also consider an additional uncertainty evaluation metric, _Patch Accuracy
vs Patch Uncertainty_ (PAvPU) [Mukhoti and Gal, 2018], which measures the
proportion of predictions that are either correct while the model is confident
about them, or incorrect while the model is ambiguous. Based on the $t$-test
results, we can easily compute PAvPU, which is defined as

$\displaystyle\text{PAvPU}\coloneqq\frac{n_{ac}+n_{iu}}{n_{ac}+n_{au}+n_{ic}+n_{iu}},$
(30)

where $n_{ac},n_{au},n_{ic},n_{iu}$ represent the number of accurate (correct)
predictions when the model is certain (confident) about them, accurate but
uncertain, inaccurate but certain, as well as inaccurate and uncertain. We
apply the $t$-test results as the proxy for the model’s confidence level on
each of its predictions. A higher value indicates that the model tends to be
correct when being confident, and to make mistakes when being vague — a
characteristic that we want our model to possess. In the following we provide
additional experimental results including this metric.

Class | Accuracy (%) | PAvPU (%)
---|---|---
All | 91.79 | 91.70

We report PAvPU along with the mean accuracy across all classes for the
FashionMNIST classification experiment from Sec. 3.2.2 in the right table.
Note that accuracy can be computed by replacing $n_{iu}$ with $n_{au}$ in Eq.
30. CARD with a pre-trained LeNet classifier obtains a PAvPU of $91.70\%$,
implying the model can express both what it knows and what it does not know.
On the other hand, if the PAvPU value is low, the model could either be making
mistakes when confident about its predictions, or happened to guess the right
label, or both. Below we provide more experiment justification of CARD’s
classification performance with the addition of this metric.

###  A.3 Classification on Noisy MNIST dataset

To further demonstrate the effectiveness of the CARD model for classification,
especially in expressing model confidence, we run additional experiments on
Noisy MNIST dataset (adding a Gaussian noise with mean $0$ and variance $1$ to
each pixel). Besides reporting PIW and $t$-test results similar to Table 6, we
also compute PAvPU, and compare the results with baseline models appearing in
Fan et al. [2021], i.e., MC Dropout [Gal and Ghahramani, 2016], Gaussian
Dropout [Srivastava et al., 2014], Concrete Dropout [Gal et al., 2017], Bayes
by Backprop [Blundell et al., 2015], and variants of Contextual Dropout [Fan
et al., 2021].

Following the same experimental settings in Fan et al. [2021], we apply a
batch size of $128$ for model training, and adopt the same MLP architecture
with two hidden layers of $300$ and $100$ hidden units, respectively, each
followed by a ReLU non-linearity, as the network architecture for the pre-
trained classifier $f_{\phi}({\bm{x}})$. We simplify the diffusion network
architecture (detailed in Section A.4) by reducing feature dimension from
$2\,048$ to $128$ to match the model parameter scale for a fair comparison. We
pre-train $f_{\phi}({\bm{x}})$ for $100$ epochs, and obtain a test set
accuracy of $85.50\%$. We train the diffusion model for $1000$ epochs. Unlike
on FashionMNIST, we do not apply data normalization on the noisy MNIST
dataset, following the settings in Fan et al. [2021].

Similar to Table 6, we report the mean PIW among both correct and incorrect
predictions, and the mean accuracy among both groups rejected and not-rejected
by the paired two-sample $t$-test, in Table 7. Same as Fan et al. [2021], we
set $\alpha=0.05$. We report these metrics for all test instances and for each
class label along with their group accuracy.

Table 7: PIW (in $\%$) and $t$-test ($\alpha=0.05$) results for Noisy MNIST
classification task.

Class | Accuracy | PIW | Accuracy by $t$-test Status
---|---|---|---
|  | Correct | Incorrect | Rejected | Not-Rejected (Count)
All | $88.26\%$ | $39.76$ | $76.25$ | $91.24\%$ |  $43.04\%$ (618)
1 | $94.69\%$ | $15.93$ | $68.67$ | $96.63\%$ |  $35.48\%$ (31)
2 | $96.48\%$ | $20.48$ | $74.73$ | $97.66\%$ |  $46.15\%$ (26)
3 | $88.47\%$ | $29.92$ | $74.98$ | $91.25\%$ |  $44.26\%$ (61)
4 | $87.23\%$ | $41.38$ | $78.58$ | $90.85\%$ |  $45.68\%$ (81)
5 | $85.34\%$ | $48.87$ | $78.61$ | $87.65\%$ |  $53.73\%$ (67)
6 | $85.76\%$ | $51.35$ | $76.43$ | $89.85\%$ |  $40.54\%$ (74)
7 | $92.28\%$ | $22.92$ | $74.04$ | $94.49\%$ |  $28.13\%$ (32)
8 | $86.97\%$ | $39.80$ | $74.42$ | $90.17\%$ |  $37.10\%$ (62)
9 | $80.90\%$ | $62.48$ | $77.73$ | $84.25\%$ |  $45.88\%$ (85)
10 | $83.25\%$ | $76.92$ | $76.67$ | $87.69\%$ |  $42.42\%$ (99)

We are able to draw similar conclusions from Table 7 as in Table 6: across the
entire test set, mean PIW of the true class label among the correct
predictions is much narrower than that of the incorrect predictions, implying
that CARD is confident in its correct predictions, and more likely to make
mistakes when being vague. When comparing mean PIWs at class level, we observe
that a more accurate class is inclined to have a larger difference between
correct and incorrect predictions. Meanwhile, we have almost $10$ times of
not-rejected cases as those for FashionMNIST task shown in Table 6. Similarly,
the accuracy of test instances rejected by the $t$-test is much higher than
that of the not-rejected ones, both across the entire test set and at the
class level. This result could have a more significant practical impact when
applying human-machine collaboration: we are able to identify more than $6\%$
of the data with a mean accuracy of less than $50\%$. If we pass these cases
to human agents, we would be able to remarkably improve the classification
accuracy, while still enjoying the automation by the machine classifier in the
vast majority.

Furthermore, we contextualize the performance of CARD by reporting accuracy,
PAvPU and NLL, along with those from the baseline models mentioned at the
beginning of Section A.3, in Table 8. The metrics of the other models are from
Table 1 in Fan et al. [2021].

Table 8: PAvPU (in $\%$) along with accuracy (in $\%$) and NLL for Noisy MNIST
classification task. The relative difference between accuracy and PAvPU is
noted right next to the PAvPU value, and marked in green if PAvPU is higher
than accuracy.

Method | Accuracy | PAvPU $(0.05)$ | NLL
---|---|---|---
MC - Bernoulli | $86.36$ | $85.63$ | $1.72$
MC - Gaussian | $86.31$ | $85.64$ | $1.72$
Concrete | $86.52$ | $86.77$ | $1.68$
Bayes by Backprop | $86.55$ | $87.13$ | $2.30$
Contextual Gating | $86.20$ | - | $1.81$
Contextual Gating + Dropout | $86.70$ | $87.01$ | $1.71$
Bernoulli Contextual | $87.43$ | $87.81$ | $1.41$
Gaussian Contextual | $87.35$ | $87.72$ | $1.43$
CARD (ours) | $\bm{88.26}$ | $\bm{89.12}$ | $\bm{0.39}$

We observe that CARD obtains an accuracy of $88.26\%$ (improves from $85.50\%$
by the pre-trained classifier $f_{\phi}$), and a PAvPU of $89.12\%$, both are
the best among all models. This implies that our model is making not only
accurate classifications, but also high-quality predictions in terms of model
confidence. Lastly, we apply temperature scaling in Guo et al. [2017] to
calibrate the predicted probability for the computation of NLL, where the
temperature parameter is tuned with the training set, and again obtain the
best metric among all models.

###  A.4 General Experiment Setup Details

In this section, we provide the experimental setup for the CARD model in both
regression and classification tasks.

Training: As mentioned at the beginning of Section 3, we set the number of
timesteps to $1\,000$, and adopt a linear $\beta_{t}$ schedule same as Ho et
al. [2020]. We set the learning rate to $0.001$ for all tasks. We use the
AMSGrad [Reddi et al., 2018] variant of the Adam optimizer [Kingma and Ba,
2015] for all regression tasks. We use the Adam optimizer for the
classification task on FashionMNIST dataset, and adopt cosine learning rate
decay [Loshchilov and Hutter, 2017]. We use EMA on model parameters with a
decay factor of $0.9999$. We adopt antithetic sampling [Ren et al., 2019] to
draw correlated timesteps during training. We set the number of epochs at
$5\,000$ by default, to sufficiently cover the timesteps with $T=1000$. The
batch size for each UCI regression task are reported in Table 10. We also
apply a batch size of $256$ for all toy regression tasks, as well as the
FashionMNIST classification task. For all UCI regression tasks, we follow the
convention in Hernández-Lobato and Adams [2015] and standardize both the input
features and the response variable to have zero mean and unit variance, and
remove the standardization to compute the metrics. For all toy regression
tasks, we do not standardize the input feature; we only standardize the
response variable on the log-log cubic regression task. For classification on
FashionMNIST, we normalize the dataset with $0.5$ mean and $0.5$ standard
deviation.

Network Architecture: For the diffusion model, we adopt a simpler network
architecture than [Xiao et al., 2021, Zheng et al., 2022], by first changing
the Transformer sinusoidal position embedding to linear embedding for the
timestep. As the network
$\bm{\epsilon}_{{\mathbf{\theta}}}\big{(}{\bm{x}},{\bm{y}}_{t},f_{\phi}({\bm{x}}),t\big{)}$
has $3$ other inputs besides the timestep $t$, we integrate them in different
ways for regression and classification tasks:

  * •

For regression, we first concatenate ${\bm{x}}$, ${\bm{y}}_{t}$, and
$f_{\phi}({\bm{x}})$, then send the resulting vector through $3$ fully-
connected layers, all with an output dimension of $128$. We perform Hadamard
product between each of the output vector with the corresponding timestep
embedding, followed by a Softplus non-linearity, before sending the resulting
vector to the next fully-connected layer. Lastly, we apply a $4$-th fully-
connected layer to map the vector to one with a dimension of $1$, as the
output forward diffusion noise prediction. We summarize the architecture in
Table 9 (a).

  * •

For classification on FashionMNIST, we first apply an encoder on the flattened
input image (originally $32\times 32\times 1$) to obtain a representation with
$2048$ dimensions. The encoder consists of $3$ fully-connected layers with an
output dimension of $2048$. Meanwhile, we concatenate ${\bm{y}}_{t}$ and
$f_{\phi}({\bm{x}})$, and apply a fully-connected layer to obtain an output
vector of $2048$ dimensions. We perform a Hadamard product between such vector
and a timestep embedding to obtain a response embedding conditioned on the
timestep. We then perform Hadamard product between image embedding and
response embedding to integrate these variables, and send the resulting vector
through $2$ more fully-connected layers with $2\,048$ output dimensions, each
would first followed by a Hadamard product with a timestep embedding, and
lastly a fully-connected layer with an output dimension of $1$ as the noise
prediction. Note that all fully-connected layers are also followed by a batch
normalization layer and a Softplus non-linearity, except the output layer. We
summarize the architecture in Table 9 (b).

Table 9: CARD $\bm{\epsilon}_{{\mathbf{\theta}}}$ network architecture. We
denote concatenation as $\oplus$, Hadamard product as $\odot$, and Softplus
non-linearity as $\sigma$. We also denote a fully-connected layer as $g$, and
a hidden layer output as $l$, with subscripts to differentiate them.

(a) Regression network architecture. [5pt]  input:
${\bm{x}},{\bm{y}}_{t},f_{\phi}({\bm{x}}),t$
$l_{1}=\sigma\Big{(}g_{1,a}\big{(}{\bm{x}}\oplus{\bm{y}}_{t}\oplus
f_{\phi}({\bm{x}})\big{)}\odot g_{1,b}(t)\Big{)}$
$l_{2}=\sigma\big{(}g_{2,a}(l_{1})\odot g_{2,b}(t)\big{)}$
$l_{3}=\sigma\big{(}g_{3,a}(l_{2})\odot g_{3,b}(t)\big{)}$ output:
$g_{4}(l_{3})$

(b) Classification network architecture. [5pt]  input:
${\bm{x}},{\bm{y}}_{t},f_{\phi}({\bm{x}}),t$
$l_{1,x}=\sigma\Big{(}\text{BN}\big{(}g_{1,x}({\bm{x}})\big{)}\Big{)}$
$l_{2,x}=\sigma\Big{(}\text{BN}\big{(}g_{2,x}({\bm{x}})\big{)}\Big{)}$
$l_{3,x}=\text{BN}\big{(}g_{1,x}({\bm{x}})\big{)}$
$l_{1,y}=\sigma\bigg{(}\text{BN}\Big{(}g_{1,y}\big{(}{\bm{y}}_{t}\oplus
f_{\phi}({\bm{x}})\big{)}\odot g_{1,b}(t)\Big{)}\bigg{)}$ $l_{1}=l_{3,x}\odot
l_{1,y}$ $l_{2}=\sigma\Big{(}\text{BN}\big{(}g_{2,a}(l_{1})\odot
g_{2,b}(t)\big{)}\Big{)}$
$l_{3}=\sigma\Big{(}\text{BN}\big{(}g_{3,a}(l_{2})\odot
g_{3,b}(t)\big{)}\Big{)}$ output: $g_{4}(l_{3})$

For the pre-trained model $f_{\phi}({\bm{x}})$, we adjust the functional form
and training scheme based on the task. For regression, we adopt a feed-forward
neural network with two hidden layers, each with $100$ and $50$ hidden units,
respectively. We apply a Leaky ReLU non-linearity with a $0.01$ negative slope
after each hidden layer. In practice, we find that when the dataset size is
small (most of the UCI tasks have less than $10,000$ data points, and several
around or less than $1000$), a deep neural network $f_{\phi}({\bm{x}})$ is
prone to overfit. We thus set the default number of epochs to $100$, and adopt
early stopping with a patience of $30$ epochs, i.e., we terminate the training
if there is no improvement for $30$ consecutive epochs. For classification, we
apply a pre-trained ResNet-18 network with the input and output layer modified
for the FashionMNIST dataset. We only train the model with $10$ epochs, as
more would lead to overfit. We apply the Adam optimizer for
$f_{\phi}({\bm{x}})$ in all tasks, and we only pre-train the model and freeze
it during the training of the diffusion model, instead of fine-tuning it.

###  A.5 UCI Baseline Model Experiment Setup Details

In this section, we provide the experimental setup for the UCI regression
baseline models (PBP, MC Dropout, Deep Ensembles, and GCDS), including
learning rate, batch size, network architecture, number of epochs, etc. We
applied the GitHub repo [Joachims, 2021] to run BNN models, and implemented
our own version of GCDS since the code has not been published.

Overall, we apply the same learning rate of $0.001$ for all models on all
datasets, except for PBP on the Boston dataset, which is $0.1$. We also apply
the Adam optimizer for all experiments. We follow the convention in Hernández-
Lobato and Adams [2015] and standardize both the input features and response
variable for training, and remove the standardization for evaluation.

For batch size, we adjust on a case-by-case basis, taking the running time and
dataset size into consideration. We provide the batch size in Table 10. For
network architecture, we use ReLU non-linearities for all $3$ BNNs, and Leaky
ReLU with a $0.01$ negative slope for GCDS; we choose the number of hidden
layers with the number of hidden units per layer from the following $3$
options: a) $1$ hidden layer with $50$ hidden units; b) $1$ hidden layer with
$100$ hidden units; c) $2$ hidden layers with $100$ and $50$ hidden units,
respectively. We show the choice of hidden layer architecture in Table 11. For
the number of training epochs, we also vary on a case-by-case basis: PBP and
Deep Ensembles both applied $40$ epochs, but we observed in many experiments
that the model has not converged. We show the number of epochs in Table 12.
Lastly, dataset Boston, Energy and Naval all contain one or more categorical
variables, thus we ran the experiments both with and without conducting one-
hot encoding on the data. We found that except the Naval dataset with PBP, all
other cases had worse metrics when one-hot encoding was applied.

Table 10: Batch size settings of UCI regression tasks across different models.

| PBP | MC Dropout | Deep Ensembles | GCDS | CARD (ours)
---|---|---|---|---|---
Boston | $32$ | $32$ | $32$ | $32$ | $32$
Concrete | $32$ | $32$ | $32$ | $32$ | $32$
Energy | $32$ | $32$ | $32$ | $32$ | $32$
Kin8nm | $64$ | $32$ | $64$ | $64$ | $64$
Naval | $64$ | $32$ | $64$ | $64$ | $64$
Power | $64$ | $64$ | $64$ | $64$ | $64$
Protein | $100$ | $256$ | $100$ | $256$ | $256$
Wine | $32$ | $32$ | $32$ | $32$ | $32$
Yacht | $32$ | $32$ | $32$ | $32$ | $32$
Year | $256$ | $256$ | $100$ | $256$ | $256$

Table 11: Network hidden layer architecture options of UCI regression tasks
across different models.

| PBP | MC Dropout | Deep Ensembles | GCDS
---|---|---|---|---
Boston | a | c | a | c
Concrete | a | c | c | c
Energy | a | b | a | c
Kin8nm | a | c | a | a
Naval | a | c | a | a
Power | a | b | a | a
Protein | b | c | b | c
Wine | a | c | a | c
Yacht | a | a | a | a
Year | b | c | b | b

Table 12: Number of training epochs of UCI regression tasks across different
models.

| PBP | MC Dropout | Deep Ensembles | GCDS
---|---|---|---|---
Boston | $100$ | $1000$ | $100$ | $500$
Concrete | $100$ | $500$ | $40$ | $500$
Energy | $100$ | $500$ | $100$ | $500$
Kin8nm | $100$ | $500$ | $100$ | $500$
Naval | $100$ | $500$ | $100$ | $500$
Power | $100$ | $500$ | $100$ | $500$
Protein | $100$ | $500$ | $100$ | $500$
Wine | $100$ | $500$ | $100$ | $500$
Yacht | $100$ | $500$ | $100$ | $500$
Year | $100$ | $100$ | $100$ | $500$

We reiterate that we re-ran the experiments with the baseline BNN models to
compute the new metric QICE along with the conventional metrics RMSE and NLL.
We carefully tuned the hyperparameters to obtain results better than or
comparable with the ones reported in the original papers.

###  A.6 Regression Toy Example Data Generation Functions and CARD Metrics

In this section, we display Table 13 containing the data generating functions
for the $8$ regression toy examples, as well as Table 14 with the metrics
including RMSE, QICE, and PICP. Note that QICE has been converted to
percentage scale as we report two significant figures for all metrics.

Table 13: Regression toy examples.

Regression Task | Data Generating Function | ${\bm{x}}$ | $\bm{\epsilon}$
---|---|---|---
Linear | ${\bm{y}}=2{\bm{x}}+3+\bm{\epsilon}$ | $\text{U}(-5,5)$ | $\mathcal{N}(0,2^{2})$
Quadratic | ${\bm{y}}=3{\bm{x}}^{2}+2{\bm{x}}+1+\bm{\epsilon}$ | $\text{U}(-5,5)$ | $\mathcal{N}(0,2^{2})$
Log-Log Linear | ${\bm{y}}=\exp\big{(}\log({\bm{x}})+\bm{\epsilon}\big{)}$ | $\text{U}(0,10)$ | $\mathcal{N}(0,0.15^{2})$
Log-Log Cubic | ${\bm{y}}=\exp\big{(}3\log({\bm{x}})+\bm{\epsilon}\big{)}$ | $\text{U}(0,10)$ | $\mathcal{N}(0,0.15^{2})$
Sinusoidal | ${\bm{y}}={\bm{x}}+0.3\sin(2\pi{\bm{x}})+\bm{\epsilon}$ | $\text{U}(0,1)$ | $\mathcal{N}(0,0.08^{2})$
Inverse Sinusoidal | swap ${\bm{x}}$ and ${\bm{y}}$ from Sinusoidal | — | —
8 Gaussians |  $8$ modes | — | $\mathcal{N}(0,0.1^{2})$
Full Circle | ${\bm{y}}=(10+\bm{\epsilon})\big{(}\cos(2\pi{\bm{x}})+\sin(2\pi{\bm{x}})\big{)}$ | $\text{U}(0,1)$ | $\mathcal{N}(0,0.5^{2})$

For all uni-modal cases as well as the full circle task, the ${\bm{x}}$
variable is sampled from a uniform distribution. The noise variable
$\bm{\epsilon}$ is sampled from a Gaussian distribution. The dataset of the
inverse sinusoidal task is created by simply swapping ${\bm{x}}$ and
${\bm{y}}$ variable of the sinusoidal task (so that we have multi-modality
when the new ${\bm{x}}$ is roughly between $0.25$ and $0.75$), thus the name
of the task.

Table 14: Regression toy example RMSE, QICE (in $\%$), and PICP (in $\%$).

Regression Task | RMSE $\downarrow$ | QICE $\downarrow$ | PICP
---|---|---|---
Linear | $0.09\pm 0.01$ | $0.55\pm 0.14$ | $95.26\pm 0.59$
Quadratic | $0.21\pm 0.03$ | $0.55\pm 0.12$ | $95.12\pm 0.55$
Log-Log Linear | $0.07\pm 0.01$ | $0.55\pm 0.15$ | $95.17\pm 0.62$
Log-Log Cubic | $5.54\pm 1.38$ | $2.10\pm 0.35$ | $96.08\pm 0.62$
Sinusoidal | $0.01\pm 0.00$ | $0.48\pm 0.11$ | $94.81\pm 0.54$
Inverse Sinusoidal | — | $0.71\pm 0.18$ | $95.89\pm 0.52$
8 Gaussians | — | $0.66\pm 0.19$ | $95.92\pm 0.46$
Full Circle | — | $0.72\pm 0.15$ | $95.46\pm 0.46$

The results in Table 14 implicitly suggest that our proposed metric QICE is
reasonable: when RMSE is low (way below $1$) and PICP is close to $95\%$,
implying that CARD is performing well, QICE is also low (far less than $1\%$);
for the most difficult task, log-log cubic regression, as RMSE is above $5$
and PICP deviates most from $95\%$, QICE also has the largest value (alove
$2\%$).

###  A.7 Test for Normality Assumption of Paired Two-Sample t-test

To access the normality assumption of the paired two-sample $t$-test, we
inspect the Q-Q plots of the differences in probability between the most and
the second most predicted classes within each test instance. We include $16$
instances in Figure 2, each with $100$ samples. We observe that in all plots,
the points align closely with the $45$-degree line, indicating that the
normality assumption is valid.

Figure 2: Q-Q plots for the differences in probability between the most and
$2^{nd}$ most predicted class.

##  Appendix B Broader Impact and Limitations

We believe that our model has a practical impact in various industrial
settings, where supervised learning methods have been increasingly applied to
facilitate the decision making process. For regression tasks, e.g.,
understanding the relationship between drug dosage and biographical features
of a patient in the medical domain, and evaluating player performance given
various on-court measurements, where the actual distribution of the real-world
data is complicated and unknown, we are able to compute summary statistics
with minimal amount of assumptions, as we do not assume the parametric form of
the conditionals $p({\bm{y}}\,|\,{\bm{x}})$. The analytical results from our
framework thus have the potential to reach a broader audience, who in the past
might be hindered by the sheer amount of jargon due to the lack of statistical
training. For classification tasks, as mentioned in both of the experiments,
we could easily adopt a human-machine collaboration framework: since our model
is capable of conveying the prediction confidence along with the prediction
itself, we could pass the cases where the model is less assertive to humans
for further evaluation. This trait is especially valuable for classification
tasks with exceptionally imbalanced data, e.g., fraud detection and ad click-
through rate prediction, where the volume of one class could be orders of
magnitude more than the other. False negative errors for these applications
are usually quite expensive, and simply adjusting classification threshold
would often put too many positive predictions for human agent evaluation. As
demonstrated in our experiments, CARD is capable of providing uncertain cases
with a very reasonable ratio, keeping the workload for human agents at a
sensible level.

Meanwhile, since CARD is capable of modeling multi-modality, we are concerned
that it could be devised for malicious purposes, like revealing personalised
information of the patient through reverse engineering the prediction results:
as an extension to the medical example of breast cancer given in the
introduction section, one could tell with high confidence the gender of the
patient based on the predicted mode. For research purpose, in this work we
only consider Gaussian diffusion and the reverse denoising in CARD, while
there could be much more options when given different data. For example, in
classification we can optimize the classification likelihood with cross-
entropy instead of using simple MSE loss; we only take a pre-trained neural
network as $f_{\phi}$ while there could be more possibilities like combining
BNN methods with CARD. Moreover, the computation efficiency of CARD may also
be further investigated if the dataset size becomes larger. We encourage
researchers in our community to further study those potential safety concerns
and the potential improvements in order to develop more mature supervised
learning tools.

##  Appendix C Computational Resources

Our models are trained and evaluated with a single Nvidia GeForce RTX 3090
GPU. We use PyTorch 1.10.0 as the deep learning framework. Our CARD model
trains between $100$ and $200$ steps per second at batch size specified in
Table 10 for regression tasks, and at $44$ steps per second at batch size
$256$ for classification on FashionMNIST dataset. The sampling for a batch of
$250$ test instances, each with $100$ samples, takes around $1.05$ seconds at
most for the classification task on FashionMNIST. The computation time could
vary if we apply different network architectures for regression and
classification tasks, and the architecture details are provided in Appendix
A.4.