# Uncertainty Visualization via Low-Dimensional Posterior Projections

Yair Omer
Technion - Israel Institute of Technology
Haifa, Israel
<EMAIL_ADDRESS>
Elias Nehme
Technion - Israel Institute of Technology
Haifa, Israel
<EMAIL_ADDRESS>Tomer Michaeli
Technion - Israel Institute of Technology
Haifa, Israel
<EMAIL_ADDRESS>

###### Abstract

In ill-posed inverse problems, it is commonly desirable to obtain insight into
the full spectrum of plausible solutions, rather than extracting only a single
reconstruction. Information about the plausible solutions and their
likelihoods is encoded in the posterior distribution. However, for high-
dimensional data, this distribution is challenging to visualize. In this work,
we introduce a new approach for estimating and visualizing posteriors by
employing energy-based models (EBMs) over low-dimensional subspaces.
Specifically, we train a conditional EBM that receives an input measurement
and a set of directions that span some low-dimensional subspace of solutions,
and outputs the probability density function of the posterior within that
space. We demonstrate the effectiveness of our method across a diverse range
of datasets and image restoration problems, showcasing its strength in
uncertainty quantification and visualization. As we show, our method
outperforms a baseline that projects samples from a diffusion-based posterior
sampler, while being orders of magnitude faster. Furthermore, it is more
accurate than a baseline that assumes a Gaussian posterior.

##  1 Introduction

Figure 1: Informed uncertainty visualization. Point estimation methods receive
a distorted image and output only a single solution, _e.g.,_ the MMSE
estimator (top). NPPC [49] complements MMSE estimators with input-adaptive
uncertainty directions (principal components of the posterior) without
modeling output likelihood (middle). Our method (bottom) learns the input-
adaptive projected posterior distribution, facilitating a likelihood-informed
uncertainty visualization.

Interpreting and communicating prediction uncertainty plays a key role in
advancing trustworthy models that could assist decision-makers. In the context
of imaging, many practical problems are ill-posed, so that a range of
plausible explanations exist for any given input. In such cases, it is
beneficial to provide the user with tools to efficiently explore and visualize
the set of all admissible solutions. This is especially crucial in safety-
critical domains such as scientific and medical image analysis [52, 9, 55,
16], where mistaken predictions could influence human life.

The information about the plausible solutions and their likelihoods is fully
encoded in the posterior distribution. However, high-dimensional posteriors
are hard to estimate and practically impossible to visualize. One way to
visualize uncertainty is to settle with generating samples from the posterior
[68, 51, 30, 72, 10, 33, 39, 59, 58, 5]. However, navigating many samples per
input is highly inefficient and often an impractical way of communicating
uncertainty [12]. Indeed, in complex domains and high levels of uncertainty,
users may need to examine hundreds of posterior samples to confidently confirm
or refute their suspicion about the unobserved ground-truth image. Several
works proposed to manipulate the sampling process to produce a small set of
meaningful samples highlighting posterior diversity [43, 75, 63, 12].
Nonetheless, methods that are based on state-of-the-art (diffusion based)
samplers are unacceptably slow due to their iterative sampling process.

Another way to summarize uncertainty in image restoration problems is via per-
pixel heatmaps [31, 1, 71, 2, 24]. However, such maps ignore the correlations
between pixels in the recovered image and thus typically result in non-
semantic and unnecessarily inflated uncertainty estimates that are of limited
practical use.

Recently, several works proposed to visualize posterior uncertainty by
traversing along the principal components (PCs) of the posterior [49, 4].
These visualizations shed light on the main modes of variation along which the
solution could vary, given the input, and have thus been demonstrated to be
natural for signals with strong correlations, like images. These methods are
also mathematically appealing since one-dimensional statistics of the
projected posterior along the PCs (like variances or quantiles) provide a
tighter and more accurate description of the underlying uncertainty [4].
Nonetheless, methods in this category do not provide estimates for the
likelihoods of the projected solutions within the low-dimensional subspace.
This hinders the user’s ability to interpret and visualize the effective
support of the projected posterior.

In this work, we propose to model the projected posterior using a conditional
energy-based model (EBM) that receives a degraded measurement and an affine
subspace (parameterized by an origin point and a set of directions) and can
output the projection of the posterior probability for any queried point
within that subspace. This allows us to visualize the projected posterior
within any low-dimensional space (_e.g.,_ the one spanned by the dominant
posterior PCs), facilitating informed navigation of posterior uncertainty in
both one and two dimensions (Fig. 1). Our method, which we coin _projected
posterior distribution estimation_ (PPDE), is a general technique for
uncertainty visualization that is seamlessly transferrable across tasks and
datasets. Here, we illustrate the method with an MSE-optimized point
prediction and the posterior PCs provided by NPPC [49]. However, it is not
constrained to this specific choice and can wrap around any point estimate and
an accompanying set of directions.

As we illustrate, our proposed visualization approach cannot be achieved with
methods that explicitly learn the high-dimensional posterior distribution,
like conditional invertible models (_e.g.,_ SRFlow) or score-based models that
enable the calculation of the exact likelihood using the probability flow ODE
[66, 67]. In particular, the naive approach of taking a 2D or 1D slice through
the high-dimensional distribution gives meaningless results because it almost
always misses the high-probability regions in space (see Fig. 3 and the
explanation in Sec. 3.1). By contrast, our approach corresponds to
_projecting_ the distribution onto the 2D or 1D slice (_i.e.,_ integrating
over all other dimensions), an effect that is practically impossible to
achieve with models that output the density in the high-dimensional space.

We demonstrate the practical benefit of our method on multiple inverse
problems in imaging. We quantitatively compare our learned posteriors to a
Gaussian approximation obtained from [49], as well as to a baseline that
projects samples generated by a diffusion-based posterior sampler and applies
kernel-density estimation (KDE) in the low-dimensional space. In both cases,
we show a significant improvement in sample log-likelihood across tasks and
datasets. This illustrates the benefit offered by our method compared to works
that approximate the posterior with a Gaussian [14, 47, 44, 50] or attempt to
rely on posterior samplers [4], setting forth a new approach to proper
uncertainty visualization.

##  2 Related Work

In the deep learning literature (_e.g.,_[17, 31]), predictive uncertainty is
often decomposed into two main sources: (i) epistemic/model uncertainty [48,
61, 7, 32, 37, 54, 53, 41, 27], which stems from imperfect knowledge of model
parameters and can be reduced by acquiring additional training data, and (ii)
aleatoric/data uncertainty [31, 71, 51, 2, 30, 72], which is inherent to the
task (_e.g.,_ due to the degradation and measurement noise), and cannot be
reduced even with infinite data. We focus on the latter.

Until recently, the majority of methods for uncertainty quantification in
imaging problems focused on per-pixel estimates _e.g.,_ in the form of
variance heatmaps [31] or confidence intervals [2]. However, these techniques
ignore inter-pixel correlations, which are very important in visual data.
Distribution-free methods such as risk-controlling prediction sets (RCPS) [3]
treat all pixels jointly by aggregating and controlling their risk (_e.g.,_ in
image segmentation). However, RCPS only provides upper and lower bounds on the
solution set without the ability to navigate within-set possibilities, and it
also requires an extra data split which may not be readily available in data-
scarce settings. A follow-up work [62], employed a similar idea in the latent
space of StyleGAN, demonstrating some informative visualizations of
uncertainty. However, this work is limited in its ability to treat real images
and it requires previously identified disentangled latent directions.

Another alternative for exploring data uncertainty is posterior sampling using
conditional generative models such as [38, 51, 68, 29, 30, 10, 33, 39, 58, 59,
5, 72, 11], with score-based/diffusion models [26, 64, 65, 23] pulling ahead
in the last two years. Posterior sampling can in principle offer a large
solution set for any given input, which can, in turn, be summarized to useful
uncertainty estimates _e.g.,_ using PCA [4]. However, this strategy is
extremely slow with unbearable run times for modern state-of-the-art models,
despite promising recent efforts to speed them up [60, 40, 45, 69].

Some works proposed to approximate posteriors with more simple distributions
that account for pixel correlations (_e.g.,_ correlated Gaussians [14, 47, 44,
50]), which are fast to infer. Most recently, two methods were proposed to
output the top PCs of the posterior distribution directly without imposing a
distributional assumption [49, 42]. Specifically, NPPC [49] was shown to
provide useful uncertainty estimates while being extremely fast. Here we use
it as a building block to demonstrate our method.

A visualization of projected posterior distributions, as we propose here, can
theoretically be achieved by generating many posterior samples with some
stochastic inverse-problem solver (_e.g.,_[72]), projecting them onto the
desired subspace, and estimating the distribution of the low-dimensional
projections using _e.g.,_ kernel density estimation (KDE). However, this
approach is impractical for real-world applications as the computational
burden of generating a sufficient number of samples is very high with state-
of-the-art posterior samplers. Furthermore, as we show, our method outperforms
this approach in terms of average negative log-likelihood (NLL) even when
using a large sample.

Another way to obtain an approximation of the projected posterior was recently
proposed in the concurrent work of [42]. Specifically, this work presents a
training-free method for estimating the one-dimensional projected posterior
distribution along any desired direction in the task of Gaussian denoising.
Our approach offers two key advantages over this work: (i) It is applicable to
arbitrary inverse problems and not only to Gaussian denoising; (ii) It is
applicable to arbitrary subspace dimensions (_e.g.,_ 2D) and not constrained
to only 1D projections.

##  3 Method

Figure 2: Notations illustration. An image $\bm{x}$ is projected onto a
measurement-adaptive affine subspace $\mathcal{A}(\bm{y})$, and represented by
projection coefficients $\bm{v}=[v_{1},v_{2}]^{\top}$. The restored image
within the subspace is given by $\bm{x}_{\text{restored}}$.

###  3.1 Projecting the posterior distribution

Our goal is to visualize the uncertainty when predicting a signal $\bm{x}$
based on measurements $\bm{y}$. In the context of imaging, $\bm{y}$ often
represents a degraded version of $\bm{x}$ (_e.g.,_ noisy, blurry). We assume
that $\bm{x}$ and $\bm{y}$ are realizations of random vectors $\mathbf{x}$ and
$\mathbf{y}$, respectively. The desired uncertainty we aim to explore is
therefore encapsulated in the posterior distribution
$p_{\mathbf{x}|\mathbf{y}}(\bm{x}|\bm{y})$. As mentioned earlier, this
distribution is typically defined over a high-dimensional space, and therefore
visualizing it requires projection onto a lower dimensional manifold (_e.g.,_
1D or 2D).

It is important to note that the manifold that best suits uncertainty
visualization may generally depend on the observation $\bm{y}$. Here we choose
the manifold to be an input-dependent affine subspace of the form

$\left\\{\bm{x}:\bm{x}=\bm{x}_{0}(\bm{y})+\bm{W}(\bm{y})\bm{v},\;\bm{v}\in\mathbb{R}^{K}\right\\},$
(1)

where $\bm{x}_{0}(\bm{y})$ is an origin point and $\bm{W}(\bm{y})$ is a matrix
with $K$ orthonormal columns,

$\bm{W}(\bm{y})=\begin{bmatrix}|&|&&|\\\
\bm{w}_{1}(\bm{y})&\bm{w}_{2}(\bm{y})&\dots&\bm{w}_{K}(\bm{y})\\\
|&|&&|\end{bmatrix}.$ (2)

For conciceness, we denote the pair $\\{\bm{x}_{0}(\bm{y}),\bm{W}(\bm{y})\\}$
by

$\mathcal{A}(\bm{y})=\left\\{\bm{x}_{0}(\bm{y}),W(\bm{y})\right\\}$ (3)

and with slight abuse of terminology, refer to $\mathcal{A}(\bm{y})$ as our
affine subspace (or just subspace). Our method can wrap around any model that
outputs a subspace $\mathcal{A}(\bm{y})$ for each input $\bm{y}$. Here we use
the NPPC method [49] (see Sec. 3.2).

For any given input $\bm{y}$, our goal is to visualize the posterior
distribution of the coordinates of the projection of $\mathbf{x}$ onto the
subspace $\mathcal{A}(\bm{y})$, which we denote by

$\mathbf{v}=\bm{W}(\bm{y})^{\top}(\mathbf{x}-\bm{x}_{0}(\bm{y})).$ (4)

Namely, we are interested in the _projected posterior distribution_ (PPD)
$p_{\mathbf{v}|\mathbf{y}}(\bm{v}|\bm{y})$. When exploring this distribution,
a user can visually inspect any particular $\bm{v}$ by projecting it back into
pixel space as

$\bm{x}_{\text{restored}}=\bm{x}_{0}(\bm{y})+\bm{W}(\bm{y})\bm{v}.$ (5)

Figure 2 summarizes our notations.

(a) Schematic illustration.

(b) Digit inpainting.

Figure 3: Posterior slicing vs. projection. (a) Schematic illustrating the
difference between the slice and the projection of a high-dimensional manifold
onto a low-dimensional subspace. (b) Sliced and projected posterior comparison
for a 1D affine subspace defined by the posterior mean and the first PC in the
task of digit inpainting. The sliced posterior has been computed using an EBM.

It is instructive to note the difference between the PDD and the sliced
posterior distribution (_i.e.,_ the posterior evaluated on
$\mathcal{A}(\bm{y})$). Any $\bm{x}$ can be expressed in terms of its
projection onto $\mathcal{A}(\bm{y})$ and its projection onto the orthogonal
complement of $\mathcal{A}(\bm{y})$ as
$\bm{x}=\bm{x}_{0}(\bm{y})+\bm{W}(\bm{y})\bm{v}+\bm{W}^{\perp}(\bm{y})\bm{u}$.
Here, $\bm{W}^{\perp}(\bm{y})$ is a matrix with orthonormal columns that span
$\text{Range}^{\perp}\\{\bm{W}(\bm{y})\\}$, and $\bm{u}$ corresponds to the
coordinates within that subspace. Now, the sliced posterior distribution
corresponds to evaluating the posterior at $\bm{u}=0$ in this decomposition.
Namely, it is given by111The function $f^{\text{sliced}}$ is not a density
function as it is not normalized.

$\displaystyle
f^{\text{sliced}}(\bm{v}|\bm{y})=p_{\mathbf{x}|\mathbf{y}}\left(\bm{x}_{0}(\bm{y})+\bm{W}(\bm{y})\bm{v}\middle|\bm{y}\right).$
(6)

In contrast, the PDD at a point $\bm{v}$ corresponds to the integral of the
posterior over all points whose projection onto $\mathcal{A}(\bm{y})$ has
coordinates $\bm{v}$, or in other words, it is the integral over $\bm{u}$ in
this decomposition,

$\displaystyle p_{\mathbf{v}|\mathbf{y}}(\bm{v}|\bm{y})$ $\displaystyle=\int
p_{\mathbf{x}|\mathbf{y}}\left(\bm{x}_{0}(\bm{y})+\bm{W}(\bm{y})\bm{v}+\bm{W}^{\perp}(\bm{y})\bm{u}\middle|\bm{y}\right)d\bm{u}.$
(7)

There exist settings in which $f^{\text{sliced}}$ equals
$p_{\mathbf{v}|\mathbf{y}}$ (up to a normalization constant). This is the
case, for example, when $\mathbf{u}$ and $\mathbf{v}$ are statistically
independent given $\mathbf{y}$. However, in imaging inverse problems, this
seems to rarely be the case. Indeed, the distribution of natural images is
known to be concentrated near a low-dimensional manifold within the high-
dimensional ambient space. Therefore, any low-dimensional subspace almost
always misses the high-density regions of the distribution. On the other hand,
the PDD provides meaningful information regardless of whether the slice passes
through the high-density regions of the posterior or not. This is illustrated
in Fig. 3

Figure 4: Architecture overview. The degraded image $\bm{y}$ is first fed to a
pre-selected subspace extractor that outputs an input-adaptive subspace
$\mathcal{A}(\bm{y})=\\{\bm{W}(\bm{y}),\bm{x}_{0}(\bm{y})\\}$. Afterwards, the
degraded image $\bm{y}$ and the extracted subspace are fed to a feature
extractor that outputs a feature vector $h(\bm{y})$. The resulting $h(\bm{y})$
modulates a lightweight MLP that outputs $p(\bm{v}|\bm{y})$ for any query
$\bm{v}$. The resulting projected distribution can then be navigated to
visualize posterior uncertainty.

(a) $p_{\mathbf{x}}(\bm{x})$ and a sample
$({\color[rgb]{0,0,1}\bm{x}_{i}},{\color[rgb]{1,0,0}\bm{y}_{i}})$.

(b) $p_{\mathbf{x}|\mathbf{y}}(\bm{x}|\bm{y})$ and $\mathcal{A}(\bm{y})$.

(c) $f^{\text{sliced}}(\bm{v}|\bm{y})$ vs.
$p_{\mathbf{v}|\mathbf{y}}(\bm{v}|\bm{y})$.

(d) Learned vs. GT $p_{\mathbf{v}|\mathbf{y}}(\bm{v}|\bm{y})$.

Figure 5: Gaussian mixture denoising. (a) Prior distribution
$p_{\mathbf{x}}{(\bm{x})}$, and a sample from the joint distribution
$p_{\mathbf{x},\mathbf{y}}{(\bm{x},\bm{y})}$. (b) Full posterior distribution
$p_{\mathbf{x}|\mathbf{y}}{(\bm{x}|\bm{y})}$, and the selected 1D subspace
$\mathcal{A}(\bm{y})=\\{\bm{x}_{0}(\bm{y}),\bm{W}(\bm{y})\\}$. (c) Sliced
$f^{\text{sliced}}(\bm{v}|\bm{y})$ (gray) vs projected
$p_{\mathbf{v}|\mathbf{y}}(\bm{v}|\bm{y})$ (green) posterior distribution
along the 1D subspace (dashed green line in (b)). (d) Comparison of the GT and
the learned projected posterior $p_{\mathbf{v}|\mathbf{y}}(\bm{v}|\bm{y})$.

###  3.2 Choosing the subspace

Selecting an appropriate subspace is crucial for producing a meaningful PPD
visualization. The ideal subspace should be either 1D or 2D to facilitate
plotting. Moreover, the origin $\bm{x}_{0}(\bm{y})$ and directions
$\bm{W}(\bm{y})$ should be chosen such that they disclose as much posterior
variance as possible, hopefully in a semantically meaningful manner. A natural
choice is to take the origin to be the minimum mean square error (MMSE)
prediction,
$\bm{x}_{0}(\bm{y})\approx\mathbb{E}[\mathbf{x}|\mathbf{y}=\bm{y}]$, and the
directions $\bm{w}_{1}(\bm{y}),\dots,\bm{w}_{K}(\bm{y})$ to be the top PCs of
the posterior. NPPC [49] outputs both $\bm{x}_{0}(\bm{y})$ and
$\bm{w}_{1}(\bm{y}),\dots,\bm{w}_{K}(\bm{y})$ in a single forward pass,
rendering it a suitable candidate for fast training and inference. In Fig. 3,
we exemplify this choice for the task of image inpainting where moving about
the origin $\bm{x}_{0}(\bm{y})$ along the selected direction
$\bm{w}_{1}(\bm{y})$ changes digit identity.

###  3.3 Architecture

Here we propose to learn the conditional distribution
$p_{\mathbf{v}|\mathbf{y}}(\bm{v}|\bm{y})$ using a conditional EBM. Unlike
normalizing flows, EBMs do not require specialized architectures, and are thus
more flexible in their design. EBMs are usually trained with variants of
contrastive divergence (CD) [22, 8, 6, 70] or noise contrastive estimation
(NCE) [19], with various recent efforts to stabilize and improve training [18,
15, 76]

As depicted in Fig. 4, the architecture we use has two parts: A (heavy)
feature extractor with a classifier-style architecture and a lightweight small
MLP with a few linear layers. The feature extractor receives the measurement
$\bm{y}$ and the set of vectors defining the subspace $\mathcal{A}(\bm{y})$,
and outputs a 1D feature vector $\bm{h}(\bm{y})$. This feature vector is then
fed along with a queried point $\bm{v}$ into the second part that outputs
$p_{\mathbf{v}|\mathbf{y}}(\bm{v}|\bm{y})$.

This segregation of the architecture serves the purpose of lowering
computational complexity. The rationale behind it is the following: For a
given measurement $\bm{y}$ we would like to query
$p_{\mathbf{v}|\mathbf{y}}(\bm{v}|\bm{y})$ for many values of $\bm{v}$. This
is true both during training when running MCMC chains to produce contrastive
samples, and during testing when evaluating the PPD on a grid of $\bm{v}$’s
for plotting. Therefore, for a given measurement $\bm{y}$ and subspace
$\mathcal{A}(\bm{y})$, this inner separation of the architecture allows us to
evaluate the feature extractor only once while the lightweight MLP can be
queried dozens of times incurring only a minor computational burden. The
conditioning mechanism we adopt here is similar to the one employed for
timestep encoding in diffusion models [23], with adaptive shifting and scaling
of features reminiscent of AdaIN [25].

To train our conditional EBM, we use a variant of CD [22]. In its basic form,
CD fits a given parametric function $E(\bm{v};\bm{\theta})$ to the
(unnormalized) negative log distribution function
$-\log\tilde{p}_{\mathbf{v}}(\bm{v})$. In each training step, the MCMC process
(_e.g.,_ Langevin dynamics) is applied to a batch of data samples $\bm{v}$ to
produce contrastive samples $\bm{v}_{\text{neg}}$. The model parameters
$\bm{\theta}$ are then updated as

$\bm{\theta}_{t+1}=\bm{\theta}_{t}-\nabla_{\bm{\theta}}[E(\bm{v};\bm{\theta})-E(\bm{v}_{\text{neg}};\bm{\theta})]\lvert_{\bm{\theta}_{t}}.$
(8)

This process gradually increases the likelihoods that the model assigns to the
samples in the training set and decreases the likelihoods that the model
assigns to their contrastive counterparts, until convergence. At test time,
the resulting model can output the probability density
$p_{\mathbf{v}}{(\bm{v})}$ (up to an unknown normalization constant) for any
queried $\bm{v}$.

The only adjustment needed for using CD to learn a _conditional_ distribution,
is to add the feature vector $\bm{h}(\bm{y})$ as an additional input to the
model, so that
$E(\bm{v},\bm{h}\left(\bm{y}\right);\bm{\theta})=-\log\tilde{p}_{\mathbf{v}|\mathbf{y}}(\bm{v}|\bm{y})$.
Here, we use the CD variant from [74], which learns a series of distributions
by employing multiple levels of additive noise, similarly to diffusion models.

(a) Inpainting

(b) Inpainting

(c) Denoising

Figure 6: MNIST inpainting and denoising. (a),(b) Application of PPDE to image
inpainting from only the 8 bottom rows. The areas within the contours
correspond to $50\%,80\%,90\%,98\%$ of the total probability mass. Note how
the PPD reveals posterior multi-modality (_e.g.,_ it shows the digit is either
a “7” or a “9”). Similarly, it also reveals intra-digit variations such as the
circular part of the “6”. (c) Application of PPDE to image denoising with an
extreme noise level of $\sigma_{\varepsilon}=1$. PPDE reveals a similar
posterior multi-modality (_e.g.,_ the digit is either an “8” or a “5”).

##  4 Experiments

We now illustrate our method on several tasks and datasets. In all our
experiments with visual data, the subspace $\mathcal{A}(\bm{y})$ corresponds
to the top two PCs of the posterior, obtained from NPPC [49]. Full details
regarding the architectures and the per-task settings are in the Appendix.

###  4.1 Toy example in 2D

Figure 5 illustrates the application of PPDE to a 2D toy example with 1D
projections. Here, $\mathbf{x}$ is sampled from a mixture of six Gaussians
(arranged as a face), and $\mathbf{y}$ is a noisy version of $\mathbf{x}$. The
prior distribution $p_{\mathbf{x}}(\bm{x})$ and an exemplar sample from the
joint distribution $(\bm{x},\bm{y})\sim p_{\mathbf{x},\mathbf{y}}$ are
presented in Fig. 5(a). For this simple case, the posterior distribution can
be calculated analytically (see Appendix for the derivation) and is also a
mixture of Gaussians (Fig. 5(b)). Note that for this illustration the
posterior itself is two-dimensional, and therefore can be plotted and
visualized in full. However, as explained earlier, in the general high-
dimensional case, this is precisely the problem our method aims to solve by
projection. Here, this example serves as a sanity check enabling us to
benchmark our results against a known ground truth.

To demonstrate our method, we selected an arbitrary 1D subspace on which we
project and plot the posterior. The origin point and a direction defining this
subspace are shown in Fig. 5(b). As can be seen in Fig. 5(c), the PPD on the
selected 1D subspace (green line) can provide us with insights regarding the
behavior of the full posterior, such as the fact that it is composed of three
modes with different weights. In addition, Fig. 5(c) also shows the
probability density of the high dimensional (2D) posterior along the 1D line,
which we refer to as the sliced posterior (gray line). Comparing both plots,
it is easy to see why the sliced posterior is less informative as not all
posterior modes appear in the slice projection.

###  4.2 Handwritten Digits

Figures 6 demonstrate PPDE on inpainting and denoising of handwritten digits
from the MNIST dataset. In the inpainting task, we used a mask that covers the
top $70\%$ of the image, and in denoising we added noise of standard deviation
$\sigma_{\varepsilon}=1$. As can be seen, for both tasks, PPDE reveals the
inherent bimodality of the posterior with two different possible digits being
likely given the measurement $\bm{y}$. In addition, note that when the digit
identity is easier to infer from $\bm{y}$, PPDE reveals uncertainty
corresponding to intricate intra-digit variations. More examples are available
in the Appendix.

(a) Inpainting.

(b) Super-resolution 8$\times$.

Figure 7: CelebA inpainting and 8$\times$ super-resolution. (a) Application of
PPDE to eye inpainting on the CelebA-HQ dataset. Regions of higher likelihood
correlate with more natural eyes reflecting a positive correlation between
$p_{\mathbf{v}|\mathbf{y}}{(\bm{v}|\bm{y})}$ and perceived plausibility. (b)
Similarly, in super-resolution sharper images (_e.g.,_ B) are more likely than
blurry ones (_e.g.,_ D).

(a)

(b)

(c)

(d)

Figure 8: Colorization. (a) Application of PPDE to face colorization on the
CelebA-HQ dataset. The PPD is comprised of multiple connected modes along the
first PC revealing background color ambiguity. Additionally, implausible skin
colors (such as D) are significantly less likely along the second PC. (b)-(d)
Application of PPDE to natural image colorization on the ImageNet dataset.
Here, PPDE reveals typical grass and dog colors, where again implausible
colorizations are significantly less likely (_e.g.,_ D).

###  4.3 Faces

To test PPDE on faces, we trained on the CelebA-HQ dataset using the original
split inherited from celebA [35], resulting in 24183 images for training, 2993
images for validation, and 2824 images for testing. Figure 7 presents results
on $256\times 256$ face images. Here, we demonstrate PPDE on the task of image
inpainting on CelebA-HQ, as well as on $8\times$ super-resolution with a
bicubic downsampling filter on a center cropped version of the full celebA
adopted from SRFlow [38]. As can be seen, likely solutions in the PPD are more
semantically coherent, corresponding to more meaningful variations of
$\bm{x}_{0}(\bm{y})$. Fig. 8 also shows the application of PPDE to face image
colorization. Here, the PPD reveals the unique structure of the posterior,
manipulating the background color and reflecting dataset bias. Note the
plausibility of the most likely projected image (_i.e.,_ the maximum
a-posteriori estimate), which can be easily obtained with our method. More
results and examples are provided in the Appendix.

###  4.4 Natural images

Finally, we tested our approach on the large-scale ImageNet 1K dataset [57]
for the task of image colorization. Fig. 8 shows that the resulting PPD
reveals typical colors of grass and dogs. The resulting distribution is not
necessarily multimodal, however, far from being an axis-aligned Gaussian,
showing the practical benefit of the PPD. This validates the generality of
PPDE on a large-scale dataset with rich posteriors. More examples are
available in the Appendix.

Figure 9: PPD comparison on MNIST. On the left we plot the input measurement
$\bm{y}$, and the input-adaptive subspace
$\mathcal{A}(\bm{y})=\\{\bm{x}_{0}(\bm{y}),\bm{w}_{1}(\bm{y}),\bm{w}_{2}(\bm{y})\\}$.
In the middle, we plot the estimated PPDs obtained from a Gaussian
approximation using NPPC, from the KDE approach, and from our PPDE. The
projected ground truth posterior sample $\bm{x}$ is marked by a blue cross,
and its likelihood under the estimated PPD is shown on the bottom left of the
corresponding plot. On the right, we show a few posterior samples out of the
100 used in the KDE approach. These were obtained using an EBM trained on
MNIST. Figure 10: PPD comparisons on CelebA-HQ. The layout is the same as Fig.
9. The posterior samples on the right were obtained using DDNM.

###  4.5 Quantitative comparisons

#### Comparison to NPPC

NPPC outputs the posterior mean $\bm{x}_{0}(\bm{y})$, alongside the top $K$
posterior PCs $\bm{W}(\bm{y})$ and variances
$\\{\sigma_{k}^{2}(\bm{y})\\}_{k=1}^{K}$ for a given input image $\bm{y}$.
While it was originally proposed as a distribution-free method, the predicted
variances can be used to construct a 2D Gaussian approximation of the
projected posterior covariance. Let $\bm{\Sigma}(\bm{y})$ denote the diagonal
matrix that has $[\sigma_{1}^{2}(\bm{y}),\sigma_{2}^{2}(\bm{y})]^{\top}$ along
its main diagonal, the projected posterior distribution can be approximated as

$p_{\mathbf{v}|\mathbf{y}}(\bm{v}|\bm{y})\approx\mathcal{N}(\bm{v};0,\bm{\Sigma}(\bm{y})).$
(9)

We compare our results to this baseline in Table 1, showing a significant
improvement in sample NLL in favor of PPDE.

Table 1: Negative log likelihood comparison. NPPC and our method were computed on the entire test set. The KDE approach was computed for 100 test images due to computational constraints. Task | NPPC | KDE | Ours
---|---|---|---
| (full test) | (100 samples) | (full test)
MNIST |  |  |
Inpainting | 4.19 ± 0.02 | 3.9 ± 0.09 | 3.72 ± 0.01
Denoising | 3.41 ± 0.02 | 3.26 ± 0.07 | 3.04 ± 0.01
CelebA |  |  |
Inpainting Eyes | 6.38 ± 0.08 | 49.51 ± 6.9 | 5.37 ± 0.03
Colorization | 10.3 ± 0.2 | 7.7 ± 0.3 | 7.39 ± 0.03
Super-resolution | 5.12 ± 0.3 | - | 2.84 ± 0.1
ImageNet |  |  |
Colorization | 10.15 ± 0.06 | - | 6.86 ± 0.01

#### Comparsion to sampling + KDE

The aim of our density estimator is to estimate the PPD in one or two
dimensions. Given a method that can sample from the posterior (_e.g.,_ DDNM
[72]), a possible baseline approach to achieve this goal is as follows: (i)
Generate $N$ samples from the posterior $\\{\bm{x}_{i}\\}$; (ii) Compute PCA
using $\\{\bm{x}_{i}\\}$ and project the samples onto the space spanned by the
top two PCs; (iii) Estimate the PPD in 2D from the projected samples
$\\{\bm{v}_{i}\\}$ using standard techniques such as _kernel density
estimation_ (KDE). Comparisons against this baseline are reported in Table 1.
The posterior samplers used for this experiment are a conditional EBM for
MNIST denoising/inpainting, MAT [33] for face inpainting, and DDNM for face
colorization. For each task/dataset we randomly chose 100 test images and
sampled 100 posterior samples per image. Compared to this (impractical)
baseline, our technique is orders of magnitude faster as this approach
requires costly repeated posterior sampling to estimate the projected density
with enough accuracy. Figures 9 and 10 present visual examples of the
approximated PPD with NPPC, KDE, and PPDE. PPDE exhibits superior NLL on
average while being extremely faster than KDE, and only slightly slower than
NPPC.

##  5 Discussion and conclusion

We presented a new method for _informed_ uncertainty visualization and
demonstrated its practical benefit across various tasks and datasets. Compared
to the proposed baselines, our method brought a significant advantage both
qualitatively and quantitatively in terms of improved sample NLL. The main
limitation of our method lies in the fact that visualization beyond two or
three dimensions becomes very difficult. This limits our supported analysis to
2 or 3 PCs, which may be insufficient to faithfully represent rich posteriors.
Our method is particularly suited for tasks where posteriors have a
spectrally-concentrated covariance, such as in image colorization. However,
for severely ill-posed inverse problems where a large number of PCs is
required to project the posterior in an informative manner, our strategy
becomes impractical, warranting further research of condensed alternative
representations of uncertainty.

##  Appendix A Interactive examples, code, and pretrained models

A website containing interactive examples and a link to the code and the
trained models can be found at https://anonymous-monkey.github.io/PPDE/

##  Appendix B Toy model

###  B.1 Problem setting

In the main text, we demonstrated PPDE on a 2D denoising task, where samples
$\bm{x}_{i}$ from a Gaussian mixture model were distorted by additive white
Gaussian noise $\bm{n}_{i}$ with a standard deviation of $\sigma_{n}$ to
result in noisy measurements $\bm{y}_{i}$. The prior distribution
$p_{\mathbf{x}}{(\bm{x})}$ was comprised of $L$ Gaussians defined by the
parameters
$\left\\{\bm{\mu}_{\ell},\bm{\Sigma}_{\ell},\pi_{\ell}\right\\}_{\ell=1}^{L}$,
namely,

$p_{\mathbf{x}}(\bm{x})=\sum_{\ell=1}^{L}\pi_{\ell}\cdot\varphi(\bm{x};\bm{\mu}_{\ell},\bm{\Sigma}_{\ell}),$
(10)

where $\varphi(\bm{x};\bm{\mu},\bm{\Sigma})$ is the PDF of a multivariate
normal distribution with mean $\bm{\mu}$ and covariance matrix $\bm{\Sigma}$.
Specifically, in the 2D example of Fig. 5 from the main text, we used $L=6$
Gaussians with the following parameters:

$\begin{array}[]{lll}\bm{\mu}_{1}=(-1,1)^{T}&\Sigma_{1}=\begin{pmatrix}0.05&-0.01\\\
-0.01&0.025\end{pmatrix}&\pi_{1}=0.15\\\
\bm{\mu}_{2}=(1,1)^{T}&\Sigma_{2}=\begin{pmatrix}0.05&0.01\\\
0.01&0.025\end{pmatrix}&\pi_{2}=0.15\\\
\bm{\mu}_{3}=(0,0)^{T}&\Sigma_{3}=\begin{pmatrix}0.02&0\\\
0&0.03\end{pmatrix}&\pi_{3}=0.15\\\
\bm{\mu}_{4}=(-0.7,-1)^{T}&\Sigma_{4}=\begin{pmatrix}0.15&-0.04\\\
-0.04&0.04\end{pmatrix}&\pi_{4}=0.15\\\
\bm{\mu}_{5}=(0,-1.2)^{T}&\Sigma_{5}=\begin{pmatrix}0.15&0\\\
0&0.025\end{pmatrix}&\pi_{5}=0.15\\\
\bm{\mu}_{6}=(0.7,-1)^{T}&\Sigma_{6}=\begin{pmatrix}0.15&0.04\\\
0.04&0.04\end{pmatrix}&\pi_{6}=0.15.\\\ \end{array}$ (11)

The standard deviation of the additive noise was $\sigma_{n}=0.4$.

###  B.2 Analytical posterior distribution

Assuming the setting in Sec. B.1, here we derive the analytical posterior
distribution. Specifically, we show that the posterior is also a GMM, and that
it has the form

$p_{\mathbf{x}|\mathbf{y}}(\bm{x}|\bm{y})=\sum_{\ell=1}^{L}\tilde{\pi}_{\ell}(\bm{y})\cdot\varphi(\bm{x};\tilde{\bm{\mu}}_{\ell}(\bm{y}),\tilde{\bm{\Sigma}}_{\ell}(\bm{y})),$
(12)

where

$\displaystyle\tilde{\bm{\mu}}_{\ell}\left(\bm{y}\right)$
$\displaystyle=\bm{\mu}_{\ell}+\bm{\Sigma}_{\ell}(\bm{\Sigma}_{\ell}+\sigma_{n}^{2}\bm{I})^{-1}\left(\bm{y}-\bm{\mu}_{\ell}\right),$
$\displaystyle\tilde{\bm{\Sigma}}_{\ell}(\bm{y})$
$\displaystyle=\bm{\Sigma}_{\ell}-\bm{\Sigma}_{\ell}(\bm{\Sigma}_{\ell}+\sigma_{n}^{2}\bm{I})^{-1}\bm{\Sigma}_{\ell},$
$\displaystyle\tilde{\pi}_{\ell}(\bm{y})$
$\displaystyle=\frac{\pi_{\ell}\varphi(\bm{y};\bm{\mu}_{\ell},\bm{\Sigma}_{\ell}+\sigma_{n}^{2}\bm{I})}{\sum_{\ell^{\prime}=1}^{L}\pi_{\ell^{\prime}}\varphi(\bm{y};\bm{\mu}_{\ell^{\prime}},\bm{\Sigma}_{\ell^{\prime}}+\sigma_{n}^{2}\bm{I})}.$
(13)

To derive this result, we start by invoking the law of total probability,
marginalizing over an auxiliary random variable c that selects one of the $L$
distributions with probabilities $\left\\{\pi_{1},\dots,\pi_{L}\right\\}$,

$p_{\mathbf{x}}(\bm{x})=\sum_{\ell=1}^{L}p_{\mathbf{x}|\textnormal{c}}(\bm{x}|\ell)p_{\textnormal{c}}(\ell).$
(14)

The posterior distribution can then be written as

$p_{\mathbf{x}|\mathbf{y}}(\bm{x}|\bm{y})=\sum_{\ell=1}^{L}p_{\mathbf{x}|\mathbf{y},\textnormal{c}}(\bm{x}|\bm{y},\ell)p_{\textnormal{c}|\mathbf{y}}(\ell|\bm{y}).$
(15)

The term $p_{\textnormal{c}|\mathbf{y}}(\ell|\bm{y})$ in Eq. (15) can be
written using Bayes’ rule as

$\displaystyle\tilde{\pi}_{\ell}(\bm{y})$ $\displaystyle\triangleq
p_{\textnormal{c}|\mathbf{y}}(\ell|\bm{y})$
$\displaystyle=\frac{p_{\mathbf{y}|\textnormal{c}}(y|\ell)p_{\textnormal{c}}(\ell)}{p_{\mathbf{y}}(\bm{y})}$
$\displaystyle=\frac{p_{\mathbf{y}|\textnormal{c}}(\bm{y}|\ell)p_{\textnormal{c}}(\ell)}{\sum_{\ell^{\prime}=1}^{L}p_{\mathbf{y}|\textnormal{c}}(\bm{y}|\ell^{\prime})p_{\textnormal{c}}(\ell^{\prime})}$
$\displaystyle=\frac{\pi_{\ell}\varphi(\bm{y};\bm{\mu}_{\ell},\bm{\Sigma}_{\ell}+\sigma_{n}^{2}\bm{I})}{\sum_{\ell^{\prime}=1}^{L}\pi_{\ell^{\prime}}\varphi(\bm{y};\bm{\mu}_{\ell^{\prime}},\bm{\Sigma}_{\ell^{\prime}}+\sigma_{n}^{2}\bm{I})}.$
(16)

Moreover, to explicitly express
$p_{\mathbf{x}|\mathbf{y},\textnormal{c}}(\bm{x}|\bm{y},\ell)$ in Eq. (15), we
note that $\mathbf{x}$ and $\mathbf{y}$ are jointly Gaussian given c, because
under a particular choice $\textnormal{c}=\ell$, the random vector y is the
sum of two independent Gaussian vectors, $\mathbf{x}$ and $\mathbf{n}$. Hence,
conditioned on the event $\textnormal{c}=\ell$, the joint distribution of
$\mathbf{x}$ and $\mathbf{y}$ is given by

$\displaystyle p_{\mathbf{x},\mathbf{y}|\textnormal{c}}(\bm{x},\bm{y}|\ell)$
$\displaystyle=\mathcal{N}\left(\begin{pmatrix}\bm{\mu}_{\ell}\\\
\bm{\mu}_{\ell}\end{pmatrix},\begin{pmatrix}\bm{\Sigma}_{\mathbf{x}\mathbf{x}|\textnormal{c}}(\ell)&\bm{\Sigma}_{\mathbf{x}\mathbf{y}|\textnormal{c}}(\ell)\\\
\bm{\Sigma}_{\mathbf{x}\mathbf{y}|\textnormal{c}}(\ell)&\bm{\Sigma}_{\mathbf{y}\mathbf{y}|\textnormal{c}}(\ell)\end{pmatrix}\right)$
$\displaystyle=\mathcal{N}\left(\begin{pmatrix}\bm{\mu}_{\ell}\\\
\bm{\mu}_{\ell}\end{pmatrix},\begin{pmatrix}\bm{\Sigma}_{\ell}&\bm{\Sigma}_{\ell}\\\
\bm{\Sigma}_{\ell}&\bm{\Sigma}_{\ell}+\sigma_{n}^{2}\bm{I}\end{pmatrix}\right).$
(17)

From the joint distribution in Eq. (17), the conditional distribution of
$\mathbf{x}$ given $\mathbf{y}$ and c is derived as

$p_{\mathbf{x}|\mathbf{y},\textnormal{c}}(\bm{x}|\bm{y},\ell)=\varphi(\bm{x};\tilde{\bm{\mu}}_{\ell}(\bm{y}),\tilde{\bm{\Sigma}}_{\ell}(\bm{y})),$
(18)

with $\tilde{\bm{\mu}}_{\ell}(\bm{y})$ and
$\tilde{\bm{\Sigma}}_{\ell}(\bm{y})$ defined as in Eq. (B.2).

###  B.3 Projected posterior distribution

Given that the posterior distribution is a GMM as in Eq. (12), it is
straightforward to derive the _projected posterior distribution_ (PPD) over
any arbitrary 1D subspace defined by a center $\bm{x}_{0}$ and a normalized
direction $\bm{w}$. This is done by projecting each of the Gaussians
individually, _i.e.,_ the PPD for $v=\bm{w}^{T}(\bm{x}-\bm{x}_{0})$ is given
by

$p_{\textnormal{v}|\mathbf{y}}(v|\bm{y})=\sum_{\ell=1}^{L}\tilde{\pi}_{\ell}(\bm{y})\cdot\varphi(v;\bm{w}^{T}(\tilde{\bm{\mu}}_{\ell}(\bm{y})-\bm{x}_{0}),\bm{w}^{T}\tilde{\bm{\Sigma}}_{\ell}(\bm{y})\bm{w}).$
(19)

During both training and testing, we took $\bm{x}_{0}$ to be the posterior
mean for a given $\bm{y}$,

$\bm{x}_{0}(\bm{y})=\sum_{\ell=1}^{L}\tilde{\bm{\mu}}_{\ell}(\bm{y}),$ (20)

and $\bm{w}$ to be a random normalized direction.

##  Appendix C PPD learning with an EBM

For the purpose of training the conditional EBM, we used a variant of
contrastive divergence (CD), proposed in [74]. This method attempts to model
the log distribution of the dataset by employing a series of distributions
that gradually transition between the distribution of the dataset to some
reference Gaussian distribution with known parameters. We refer to this method
here as _multilevel CD_ (MCD) and review it only briefly for completeness. For
a more detailed description please refer to [74].

At the core of MCD lies a series of distributions defined by (i) coefficients
$\left\\{\alpha_{t}\right\\}_{t=1}^{T}$ gradually decreasing from
$\alpha_{1}=1$ to $\alpha_{T}=0$, (ii) a discrete random variable
$\textnormal{t}\in\left\\{1,\dots,T\right\\}$, with some chosen prior
distribution $p_{\textnormal{t}}$, and (iii) a reference Gaussian vector
$\mathbf{n}$. During training, a new random variable $\tilde{\mathbf{x}}$ is
defined as a linear mixture of $\mathbf{x}$ and $\mathbf{n}$ with random
linear coefficients associated with $\alpha_{\mathbf{t}}$ (note that
$\mathbf{t}$ is a random variable),

$\tilde{\mathbf{x}}=\alpha_{\mathbf{t}}\mathbf{x}+\sqrt{1-\alpha_{\mathbf{t}}^{2}}\,\mathbf{n}.$
(21)

The surrogate objective of MCD is to model the mixed distribution of the pair
$(\tilde{\mathbf{x}},\textnormal{t})$. This is done using a neural network
$\bm{f}(\tilde{\bm{x}};\bm{\theta})$ with parameters $\bm{\theta}$, which
receives $\tilde{\bm{x}}$ as input, and outputs $T$ scalars, where the $t$-th
output of the network models the log-probability

$f_{t}(\tilde{\bm{x}};\bm{\theta})\approx\log
p_{\tilde{\mathbf{x}},\textnormal{t}}(\tilde{\bm{x}},t)=\log
p_{\tilde{\mathbf{x}}|\textnormal{t}}(\tilde{\bm{x}}|t)+\log
p_{\textnormal{t}}(t).$ (22)

The distribution of $\mathbf{x}$ can then be extracted from the model at $t=1$
as

$f_{1}(\bm{x};\bm{\theta})\approx\log p_{\mathbf{x}}(\bm{x})+\log
p_{\textnormal{t}}(1).$ (23)

The motivation for modeling the pair $(\tilde{\mathbf{x}},\textnormal{t})$, as
opposed to just modeling $\mathbf{x}$, is to improve the accuracy of CD in
modeling low-density regions of $\log p_{\mathbf{x}}{(\bm{x})}$. During
training, the model is only exposed to samples $\bm{x}$ coming from regions of
high probability. Therefore, without further modifications, it usually fails
to extrapolate the density function beyond these regions correctly. Training
on pairs $(\tilde{\mathbf{x}},\textnormal{t})$ generated by sampling $\bm{n}$
and $t$ from their known distribution and $\bm{x}$ from the dataset, MCD can
better overcome this challenge.

The update step of MCD combines the standard CD update at any fixed $t$ with a
classification update for $t$ given $\tilde{\bm{x}}$. The CD update at a fixed
$t$ is derived by applying the classical CD algorithm to the $t$-th output of
the network, which models $\log p_{\tilde{\mathbf{x}}|\textnormal{t}}$ (up to
the known term $\log p_{\textnormal{t}}(t)$). At each training step, the
modeled $\log p_{\tilde{\mathbf{x}}|\textnormal{t}}$ is used to produce an
MCMC process starting at $\tilde{\bm{x}}$ and ending at a contrastive sample
$\tilde{\bm{x}}_{\text{neg}}$. The used MCMC process was Langevin dynamics
[73] with a Metropolis-Hastings rejection step similar to [46, 20]. The
parameters of the model were then updated according to the CD algorithm,
following the negative gradient of
$f_{t}(\tilde{\bm{x}}_{\text{neg}};\bm{\theta})-f_{t}(\tilde{\bm{x}};\bm{\theta})$.

As for the classification update, it was derived by viewing the network as a
classifier, minimizing the cross-entropy loss. This holds since a
$\mathrm{Softmax}$ operation on the network outputs results in the estimation
of the conditional distribution $p_{\textnormal{t}|\tilde{\mathbf{x}}}$

$\mathrm{Softmax}(\bm{f}(\tilde{\bm{x}};\bm{\theta}))_{t}\approx\frac{p_{\tilde{\mathbf{x}},\textnormal{t}}(\tilde{\bm{x}},t)}{\sum_{t^{\prime}=1}^{T}p_{\tilde{\mathbf{x}},\textnormal{t}}(\tilde{\bm{x}},t^{\prime})}=p_{\textnormal{t}|\tilde{\mathbf{x}}}(t|\tilde{\bm{x}}).$
(24)

The combined update rule of MCD was therefore given by

$\displaystyle\bm{\theta}^{(k+1)}=\bm{\theta}^{(k)}-\eta\nabla_{\bm{\theta}}$
$\displaystyle\left[f_{t}(\tilde{\bm{x}}_{\text{neg}};\bm{\theta})-f_{t}(\tilde{\bm{x}};\bm{\theta})\right.$
$\displaystyle\qquad\left.+\mathrm{Softmax}(\bm{f}(\tilde{\bm{x}};\bm{\theta}))_{t}\right],$
(25)

where $\eta$ is the algorithm’s learning rate.

Note that thus far the derivation of MCD assumed an unconditional setting
where our goal was to model $\log p_{\mathbf{x}}(\bm{x})$. To adopt this
training scheme for modeling the PPD, we need to replace $\bm{x}$ with its
projection $\bm{v}=\bm{W}^{T}(\bm{x}-\bm{x}_{0})$ and make the EBM depend on
the distorted sample $\bm{y}$ and the selected subspace
$\mathcal{A}(\bm{y})=\left\\{\bm{x}_{0}(\bm{y}),\bm{W}(\bm{y})\right\\}$. As
described in Sec. 3.3 of the main text, this dependency is achieved using a
separate feature extractor which outputs a feature vector $\bm{h}$
conditioning the layers of the EBM through feature normalization.

##  Appendix D Datasets and restoration tasks

We evaluated our method on a range of datasets and tasks spanning common
scenarios in low-level vision. Specifically, we used the MNIST dataset [13]
($28\times 28$), the CelebA dataset [36] (cropped and resized to $160\times
160$ as in SRFlow [38]), the CelebA-HQ dataset [28] ($256\times 256$), and the
ImageNet 1K dataset [57] ($128\times 128$). The tasks we experimented with
were:

  * •

MNIST inpainting. Recovering digit images from only the bottom 8 rows.

  * •

MNIST denoising. Denoising digit images contaminated with additive white
Gaussian noise with $\sigma_{n}=1$ and clipping.

  * •

CelebA-HQ inpainting. Recovering a rectangular area of size $70\times 175$
around the eyes in CelebA-HQ images of size $256\times 256$.

  * •

CelebA-HQ colorization. Recovering RGB images from grayscale images obtained
by averaging color channels.

  * •

CelebA super-resolution. 8$\times$ super-resolution of face images downsampled
with a bicubic kernel to $20\times 20$ pixels as described in [38].

  * •

ImageNet colorization. Recovering RGB images from grayscale images obtained by
averaging color channels.

##  Appendix E Architectures and training hyperparameters

The estimation of the projected posterior distribution (PPD) requires the
selection of a subspace for each distorted image on which the posterior is to
be projected. As described in Sec. 3.2 of the main text, we selected in this
work to use a subspace that contains the minimum MSE (MMSE) estimator and is
spanned by the first two principal components (PCs) of the posterior. For the
purpose of predicting these values we trained two auxiliary networks. Overall,
we have sequentially trained three networks for each of the evaluated tasks.

The first network was trained using an MSE loss to output the MMSE estimate
for a given distorted image $\bm{y}$. The second network was trained using
NPPC [49] to predict the first five principal components of the posterior
given both the distorted image and the MMSE estimate (predicted by the first
network). The third network was trained using the method described in this
paper to model the PPD.

Each network required a different architecture depending on the task at hand.
Overall, for every task, we used a specific subset of the following
architectures:

  * •

U-Net-S. A small version of the original U-Net architecture from [56] with
three encoder and decoder levels containing a single convolution layer each.
The number of channels in each level is $32$, $64$ and $128$, respectively.
The bottleneck has two convolutions of $256$ channels. In addition, we used a
group-norm layer after each convolution, LeakyReLU as a non-linearity layer,
and a nearest-neighbor interpolation for upsampling.

  * •

U-Net-L. A U-Net architecture similar to the one used in DDPM [23] for the
CelebA generative task, but with half the number of channels. We also removed
all network parts related to the time index dependency.

  * •

U-Net-M. A network similar to U-net-L with one level less in the encoder and
decoder (the top level, in terms of the number of channels).

  * •

EDSR. An architecture similar to the EDSR architecture described in [34] using
16 residual blocks and a width of 64 channels.

  * •

cEBM-S. This network is comprised of two parts, as described in Sec. 3.3 in
the main text. The first part is a feature extractor based on a ResNet18
architecture [21], fitted for smaller images. We removed the stride in the
input convolution and the following pooling layer and reduced the number of
channels by 2$\times$. In addition, we replaced the BatchNorm layers with
GroupNorm layers and performed the downsampling using average pooling layers
(instead of stride convolutions). The output of this network is a feature
vector of length 256. The second part of the network is a conditional MLP with
6 hidden layers of width 128. We used SiLU as the non-linearity followed by an
AdaIN [25] layer that performs adaptive scaling and shifting. The scaling and
shifting parameters are extracted from the feature vector using a linear
layer.

  * •

cEBM-L. This network is similar to the cEBM-S except that the architecture of
the feature extractor is replaced by the encoder part of the U-Net-L, using a
global average pooling layer at the end of the bottleneck to produce a feature
vector of length 512.

In general, the U-Net architectures were used for the MMSE estimator and the
PCs, and the conditional EBMs was used for modeling the PPD. The only
exceptions were the networks in the super-resolution task, in which the
distorted image was smaller than all other images and therefore was fed into
the EDSR architecture either to directly produce the desired output (as in the
case of the MMSE) or as a preprocessing step before concatenating it with the
other inputs.

All networks were trained using the Adam optimizer, stopping the training
process once it reached a minimal value of the objective over a dedicated
validation set. For the training of the PC predictor and the PPD model, the
learning rate was kept fixed. On the other hand, for training the MMSE
estimator, the learning rate was reduced by half every 5000 steps. Table 2
summarizes the architectures and training hyperparameters used for the
networks in each task/dataset.

Table 2: Architectures, learning rates, and number of steps used per task/dataset. | MMSE | MMSE | MMSE | PCs | PCs | PCs | PPD | PPD | PPD
---|---|---|---|---|---|---|---|---|---
Task | Arch. | LR | # of steps | Arch. | LR | # of steps | Arch. | LR | # of steps
MNIST |  |  |  |  |  |  |  |  |
Inpainting | U-Net-S | $10^{-4}$ | 3,510 | U-Net-S | $10^{-4}$ | 5,000 | cEBM-S | $10^{-4}$ | 36,504
Denoising | U-Net-S | $10^{-3}$ | 57,330 | U-Net-S | $10^{-4}$ | 408,000 | cEBM-S | $10^{-3}$ | 1,026,558
CelebA |  |  |  |  |  |  |  |  |
Inpainting Eyes | U-Net-L | $10^{-4}$ | 7,555 | U-Net-L | $10^{-5}$ | 22,500 | cEBM-L | $10^{-4}$ | 94,000
Colorization | U-Net-L | $3\cdot 10^{-5}$ | 9,066 | U-Net-L | $3\cdot 10^{-5}$ | 38,500 | cEBM-L | $10^{-4}$ | 44,500
Super-resolution | EDSR | $10^{-4}$ | 287,651 |  EDSR \+ U-Net-L | $10^{-5}$ | 113,500 |  EDSR \+ cEBM-L | $10^{-4}$ | 203,250
ImageNet |  |  |  |  |  |  |  |  |
Colorization | U-Net-M | $3\cdot 10^{-4}$ | 187,200 | U-Net-M | $10^{-5}$ | 37,600 | cEBM-L | $10^{-4}$ | 102,500

##  Appendix F Additional Results

###  F.1 PPDs

Figures 11-16 provide more examples of PPDs captured by PPDE for the datasets
and tasks presented in the main text.

Figure 11: More examples of MNIST inpainting PPDs.

Figure 12: More examples of MNIST denoising.

Figure 13: More examples of CelebA-HQ inpainting.

Figure 14: More examples of CelebA-HQ colorization.

Figure 15: More examples of CelebA 8$\times$ super-rsolution.

Figure 16: More examples of ImageNet colorizaion.

###  F.2 Comparisons to NPPC and KDE

Figure 17: More comparisons of MNIST inpainting PPDs. The posterior samples on
the right were obtained using an EBM trained on MNIST.

Figure 18: More comparisons of MNIST denoising. The posterior samples on the
right were obtained using an EBM trained on MNIST.

Figure 19: More comparisons of CelebA-HQ inpainting. The posterior samples on
the right were obtained using MAT [33].

Figure 20: More comparisons of CelebA-HQ colorization. The posterior samples
on the right were obtained using DDNM [72].

Figure 21: More comparisons of CelebA 8$\times$ super-resolution.

Figure 22: More comparisons of ImageNet colorization.

Figures 17-22 provide further negative log-likelihood comparisons of PPDE with
the respective baselines.

## References

  * Amini et al. [2020] Alexander Amini, Wilko Schwarting, Ava Soleimany, and Daniela Rus.  Deep evidential regression.  _Advances in Neural Information Processing Systems_ , 33:14927–14937, 2020.
  * Angelopoulos et al. [2022] Anastasios N Angelopoulos, Amit P Kohli, Stephen Bates, Michael I Jordan, Jitendra Malik, Thayer Alshaabi, Srigokul Upadhyayula, and Yaniv Romano.  Image-to-image regression with distribution-free uncertainty quantification and applications in imaging.  In _International Conference on Machine Learning_ , 2022.
  * Bates et al. [2021] Stephen Bates, Anastasios Angelopoulos, Lihua Lei, Jitendra Malik, and Michael Jordan.  Distribution-free, risk-controlling prediction sets.  _Journal of the ACM (JACM)_ , 68(6):1–34, 2021\.
  * Belhasin et al. [2023] Omer Belhasin, Yaniv Romano, Daniel Freedman, Ehud Rivlin, and Michael Elad.  Principal uncertainty quantification with spatial correlation for image restoration problems.  _arXiv preprint arXiv:2305.10124_ , 2023.
  * Bendel et al. [2022] Matthew Bendel, Rizwan Ahmad, and Philip Schniter.  A regularized conditional gan for posterior sampling in inverse problems.  _arXiv preprint arXiv:2210.13389_ , 2022.
  * Bengio and Delalleau [2009] Yoshua Bengio and Olivier Delalleau.  Justifying and generalizing contrastive divergence.  _Neural computation_ , 21(6):1601–1621, 2009\.
  * Blundell et al. [2015] Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra.  Weight uncertainty in neural network.  In _International conference on machine learning_ , pages 1613–1622. PMLR, 2015.
  * Carreira-Perpinan and Hinton [2005] Miguel A Carreira-Perpinan and Geoffrey Hinton.  On contrastive divergence learning.  In _International workshop on artificial intelligence and statistics_ , pages 33–40. PMLR, 2005.
  * Christiansen et al. [2018] Eric M Christiansen, Samuel J Yang, D Michael Ando, Ashkan Javaherian, Gaia Skibinski, Scott Lipnick, Elliot Mount, Alison O’neil, Kevan Shah, Alicia K Lee, et al.  In silico labeling: predicting fluorescent labels in unlabeled images.  _Cell_ , 173(3):792–803, 2018.
  * Chung et al. [2022] Hyungjin Chung, Jeongsol Kim, Michael Thompson Mccann, Marc Louis Klasky, and Jong Chul Ye.  Diffusion posterior sampling for general noisy inverse problems.  In _The Eleventh International Conference on Learning Representations_ , 2022.
  * Chung et al. [2023] Hyungjin Chung, Suhyeon Lee, and Jong Chul Ye.  Fast diffusion sampler for inverse problems by geometric decomposition.  _arXiv preprint arXiv:2303.05754_ , 2023.
  * Cohen et al. [2023] Noa Cohen, Hila Manor, Yuval Bahat, and Tomer Michaeli.  From posterior sampling to meaningful diversity in image restoration.  _arXiv preprint arXiv:2310.16047_ , 2023.
  * Deng [2012] Li Deng.  The mnist database of handwritten digit images for machine learning research.  _IEEE Signal Processing Magazine_ , 29(6):141–142, 2012.
  * Dorta et al. [2018] Garoe Dorta, Sara Vicente, Lourdes Agapito, Neill DF Campbell, and Ivor Simpson.  Structured uncertainty prediction networks.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 5477–5485, 2018.
  * Du et al. [2021] Yilun Du, Shuang Li, Joshua B Tenenbaum, and Igor Mordatch.  Improved contrastive divergence training of energy based models.  In _Energy Based Models Workshop-ICLR 2021_ , 2021.
  * Falk et al. [2019] Thorsten Falk, Dominic Mai, Robert Bensch, Özgün Çiçek, Ahmed Abdulkadir, Yassine Marrakchi, Anton Böhm, Jan Deubner, Zoe Jäckel, Katharina Seiwald, et al.  U-net: deep learning for cell counting, detection, and morphometry.  _Nature methods_ , 16(1):67–70, 2019.
  * Gal and Ghahramani [2016] Yarin Gal and Zoubin Ghahramani.  Dropout as a bayesian approximation: Representing model uncertainty in deep learning.  In _international conference on machine learning_ , pages 1050–1059. PMLR, 2016.
  * Gao et al. [2020] Ruiqi Gao, Yang Song, Ben Poole, Ying Nian Wu, and Diederik P Kingma.  Learning energy-based models by diffusion recovery likelihood.  In _International Conference on Learning Representations_ , 2020.
  * Gutmann and Hyvärinen [2010] Michael Gutmann and Aapo Hyvärinen.  Noise-contrastive estimation: A new estimation principle for unnormalized statistical models.  In _Proceedings of the thirteenth international conference on artificial intelligence and statistics_ , pages 297–304. JMLR Workshop and Conference Proceedings, 2010.
  * Hastings [1970] W Keith Hastings.  Monte carlo sampling methods using markov chains and their applications.  1970\.
  * He et al. [2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 770–778, 2016.
  * Hinton [2002] Geoffrey E Hinton.  Training products of experts by minimizing contrastive divergence.  _Neural computation_ , 14(8):1771–1800, 2002\.
  * Ho et al. [2020] Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models.  _Advances in neural information processing systems_ , 33:6840–6851, 2020.
  * Horwitz and Hoshen [2022] Eliahu Horwitz and Yedid Hoshen.  Conffusion: Confidence intervals for diffusion models.  _arXiv preprint arXiv:2211.09795_ , 2022.
  * Huang and Belongie [2017] Xun Huang and Serge Belongie.  Arbitrary style transfer in real-time with adaptive instance normalization.  In _Proceedings of the IEEE international conference on computer vision_ , pages 1501–1510, 2017.
  * Hyvärinen and Dayan [2005] Aapo Hyvärinen and Peter Dayan.  Estimation of non-normalized statistical models by score matching.  _Journal of Machine Learning Research_ , 6(4), 2005.
  * Izmailov et al. [2020] Pavel Izmailov, Wesley J Maddox, Polina Kirichenko, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson.  Subspace inference for bayesian deep learning.  In _Uncertainty in Artificial Intelligence_ , pages 1169–1179. PMLR, 2020.
  * Karras et al. [2017] Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen.  Progressive growing of gans for improved quality, stability, and variation.  _arXiv preprint arXiv:1710.10196_ , 2017.
  * Kawar et al. [2021] Bahjat Kawar, Gregory Vaksman, and Michael Elad.  Snips: Solving noisy inverse problems stochastically.  _Advances in Neural Information Processing Systems_ , 34:21757–21769, 2021.
  * Kawar et al. [2022] Bahjat Kawar, Michael Elad, Stefano Ermon, and Jiaming Song.  Denoising diffusion restoration models.  In _Advances in Neural Information Processing Systems_ , 2022.
  * Kendall and Gal [2017] Alex Kendall and Yarin Gal.  What uncertainties do we need in bayesian deep learning for computer vision?  _Advances in neural information processing systems_ , 30, 2017.
  * Lakshminarayanan et al. [2017] Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell.  Simple and scalable predictive uncertainty estimation using deep ensembles.  _Advances in neural information processing systems_ , 30, 2017.
  * Li et al. [2022] Wenbo Li, Zhe Lin, Kun Zhou, Lu Qi, Yi Wang, and Jiaya Jia.  Mat: Mask-aware transformer for large hole image inpainting.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 10758–10768, 2022.
  * Lim et al. [2017] Bee Lim, Sanghyun Son, Heewon Kim, Seungjun Nah, and Kyoung Mu Lee.  Enhanced deep residual networks for single image super-resolution.  In _Proceedings of the IEEE conference on computer vision and pattern recognition workshops_ , pages 136–144, 2017.
  * Liu et al. [2015a] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang.  Deep learning face attributes in the wild.  In _Proceedings of the IEEE International Conference on Computer Vision_ , pages 3730–3738, 2015a.
  * Liu et al. [2015b] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang.  Deep learning face attributes in the wild.  In _Proceedings of International Conference on Computer Vision (ICCV)_ , 2015b.
  * Louizos and Welling [2017] Christos Louizos and Max Welling.  Multiplicative normalizing flows for variational bayesian neural networks.  In _International Conference on Machine Learning_ , pages 2218–2227. PMLR, 2017.
  * Lugmayr et al. [2020] Andreas Lugmayr, Martin Danelljan, Luc Van Gool, and Radu Timofte.  Srflow: Learning the super-resolution space with normalizing flow.  In _ECCV_ , 2020.
  * Lugmayr et al. [2022] Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, and Luc Van Gool.  Repaint: Inpainting using denoising diffusion probabilistic models.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 11461–11471, 2022.
  * Luhman and Luhman [2021] Eric Luhman and Troy Luhman.  Knowledge distillation in iterative generative models for improved sampling speed.  _arXiv preprint arXiv:2101.02388_ , 2021.
  * Malinin and Gales [2018] Andrey Malinin and Mark Gales.  Predictive uncertainty estimation via prior networks.  _Advances in neural information processing systems_ , 31, 2018.
  * Manor and Michaeli [2023] Hila Manor and Tomer Michaeli.  On the posterior distribution in denoising: Application to uncertainty quantification.  _arXiv preprint arXiv:2309.13598_ , 2023.
  * Mao et al. [2019] Qi Mao, Hsin-Ying Lee, Hung-Yu Tseng, Siwei Ma, and Ming-Hsuan Yang.  Mode seeking generative adversarial networks for diverse image synthesis.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 1429–1437, 2019.
  * Meng et al. [2021] Chenlin Meng, Yang Song, Wenzhe Li, and Stefano Ermon.  Estimating high order gradients of the data distribution by denoising.  _Advances in Neural Information Processing Systems_ , 34:25359–25369, 2021.
  * Meng et al. [2022] Chenlin Meng, Ruiqi Gao, Diederik P Kingma, Stefano Ermon, Jonathan Ho, and Tim Salimans.  On distillation of guided diffusion models.  In _NeurIPS 2022 Workshop on Score-Based Methods_ , 2022.
  * Metropolis et al. [1953] Nicholas Metropolis, Arianna W Rosenbluth, Marshall N Rosenbluth, Augusta H Teller, and Edward Teller.  Equation of state calculations by fast computing machines.  _The journal of chemical physics_ , 21(6):1087–1092, 1953.
  * Monteiro et al. [2020] Miguel Monteiro, Loïc Le Folgoc, Daniel Coelho de Castro, Nick Pawlowski, Bernardo Marques, Konstantinos Kamnitsas, Mark van der Wilk, and Ben Glocker.  Stochastic segmentation networks: Modelling spatially correlated aleatoric uncertainty.  _Advances in neural information processing systems_ , 33:12756–12767, 2020.
  * Neal [2012] Radford M Neal.  _Bayesian learning for neural networks_.  Springer Science & Business Media, 2012.
  * Nehme et al. [2023] Elias Nehme, Omer Yair, and Tomer Michaeli.  Uncertainty quantification via neural posterior principal components.  In _Thirty-seventh Conference on Neural Information Processing Systems_ , 2023.
  * Nussbaum et al. [2022] Frank Nussbaum, Jakob Gawlikowski, and Julia Niebling.  Structuring uncertainty for fine-grained sampling in stochastic segmentation networks.  _Advances in Neural Information Processing Systems_ , 35:27678–27691, 2022.
  * Ohayon et al. [2021] Guy Ohayon, Theo Adrai, Gregory Vaksman, Michael Elad, and Peyman Milanfar.  High perceptual quality image denoising with a posterior sampling cgan.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 1805–1813, 2021.
  * Ounkomol et al. [2018] Chawin Ounkomol, Sharmishtaa Seshamani, Mary M Maleckar, Forrest Collman, and Gregory R Johnson.  Label-free prediction of three-dimensional fluorescence images from transmitted-light microscopy.  _Nature methods_ , 15(11):917–920, 2018.
  * Pearce et al. [2018] Tim Pearce, Alexandra Brintrup, Mohamed Zaki, and Andy Neely.  High-quality prediction intervals for deep learning: A distribution-free, ensembled approach.  In _International conference on machine learning_ , pages 4075–4084. PMLR, 2018.
  * Ritter et al. [2018] Hippolyt Ritter, Aleksandar Botev, and David Barber.  A scalable laplace approximation for neural networks.  In _6th International Conference on Learning Representations, ICLR 2018-Conference Track Proceedings_. International Conference on Representation Learning, 2018.
  * Rivenson et al. [2019] Yair Rivenson, Hongda Wang, Zhensong Wei, Kevin de Haan, Yibo Zhang, Yichen Wu, Harun Günaydın, Jonathan E Zuckerman, Thomas Chong, Anthony E Sisk, et al.  Virtual histological staining of unlabelled tissue-autofluorescence images via deep learning.  _Nature biomedical engineering_ , 3(6):466–477, 2019.
  * Ronneberger et al. [2015] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-net: Convolutional networks for biomedical image segmentation.  In _Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18_ , pages 234–241. Springer, 2015.
  * Russakovsky et al. [2015] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, and Fei-Fei Li.  Imagenet large scale visual recognition challenge.  _International journal of computer vision_ , 115:211–252, 2015.
  * Saharia et al. [2022a] Chitwan Saharia, William Chan, Huiwen Chang, Chris Lee, Jonathan Ho, Tim Salimans, David Fleet, and Mohammad Norouzi.  Palette: Image-to-image diffusion models.  In _ACM SIGGRAPH 2022 Conference Proceedings_ , pages 1–10, 2022a.
  * Saharia et al. [2022b] Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David J Fleet, and Mohammad Norouzi.  Image super-resolution via iterative refinement.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2022b.
  * Salimans and Ho [2022] Tim Salimans and Jonathan Ho.  Progressive distillation for fast sampling of diffusion models.  _arXiv preprint arXiv:2202.00512_ , 2022.
  * Salimans et al. [2015] Tim Salimans, Diederik Kingma, and Max Welling.  Markov chain monte carlo and variational inference: Bridging the gap.  In _International conference on machine learning_ , pages 1218–1226. PMLR, 2015.
  * Sankaranarayanan et al. [2022] Swami Sankaranarayanan, Anastasios Nikolas Angelopoulos, Stephen Bates, Yaniv Romano, and Phillip Isola.  Semantic uncertainty intervals for disentangled latent spaces.  In _Advances in Neural Information Processing Systems_ , 2022.
  * Sehwag et al. [2022] Vikash Sehwag, Caner Hazirbas, Albert Gordo, Firat Ozgenel, and Cristian Canton.  Generating high fidelity data from low-density regions using diffusion models.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 11492–11501, 2022.
  * Sohl-Dickstein et al. [2015] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli.  Deep unsupervised learning using nonequilibrium thermodynamics.  In _International conference on machine learning_ , pages 2256–2265. PMLR, 2015.
  * Song and Ermon [2019] Yang Song and Stefano Ermon.  Generative modeling by estimating gradients of the data distribution.  _Advances in neural information processing systems_ , 32, 2019.
  * Song et al. [2020] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole.  Score-based generative modeling through stochastic differential equations.  _arXiv preprint arXiv:2011.13456_ , 2020.
  * Song et al. [2021a] Yang Song, Conor Durkan, Iain Murray, and Stefano Ermon.  Maximum likelihood training of score-based diffusion models.  _Advances in Neural Information Processing Systems_ , 34:1415–1428, 2021a.
  * Song et al. [2021b] Yang Song, Liyue Shen, Lei Xing, and Stefano Ermon.  Solving inverse problems in medical imaging with score-based generative models.  In _International Conference on Learning Representations_ , 2021b.
  * Song et al. [2023] Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever.  Consistency models.  _arXiv preprint arXiv:2303.01469_ , 2023.
  * Sutskever and Tieleman [2010] Ilya Sutskever and Tijmen Tieleman.  On the convergence properties of contrastive divergence.  In _Proceedings of the thirteenth international conference on artificial intelligence and statistics_ , pages 789–795. JMLR Workshop and Conference Proceedings, 2010.
  * Wang et al. [2019] Guotai Wang, Wenqi Li, Michael Aertsen, Jan Deprest, Sébastien Ourselin, and Tom Vercauteren.  Aleatoric uncertainty estimation with test-time augmentation for medical image segmentation with convolutional neural networks.  _Neurocomputing_ , 338:34–45, 2019.
  * Wang et al. [2023] Yinhuai Wang, Jiwen Yu, and Jian Zhang.  Zero-shot image restoration using denoising diffusion null-space model.  _The Eleventh International Conference on Learning Representations_ , 2023.
  * Welling and Teh [2011] Max Welling and Yee W Teh.  Bayesian learning via stochastic gradient langevin dynamics.  In _Proceedings of the 28th international conference on machine learning (ICML-11)_ , pages 681–688, 2011.
  * Yair and Michaeli [2022] Omer Yair and Tomer Michaeli.  Thinking fourth dimensionally: Treating time as a random variable in ebms.  _https://openreview.net/forum?id=m0fEJ2bvwpw_ , 2022.
  * Yu et al. [2020] Ning Yu, Ke Li, Peng Zhou, Jitendra Malik, Larry Davis, and Mario Fritz.  Inclusive gan: Improving data and minority coverage in generative models.  In _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXII 16_ , pages 377–393. Springer, 2020.
  * Zhao et al. [2020] Yang Zhao, Jianwen Xie, and Ping Li.  Learning energy-based generative models via coarse-to-fine expanding and sampling.  In _International Conference on Learning Representations_ , 2020.