# Free will belief as a consequence of model-based reinforcement learning

Erik M. Rehn
Independent Researcher
Malmö, Sweden.
<EMAIL_ADDRESS>

(November 14, 2021)

###### Abstract

The debate on whether or not humans have free will has been raging for
centuries. Although there are good arguments based on our current
understanding of the laws of nature for the view that it is not possible for
humans to have free will, most people believe they do. This discrepancy begs
for an explanation. If we accept that we do not have free will, we are faced
with two problems: (1) while freedom is a very commonly used concept that
everyone intuitively understands, what are we actually referring to when we
say that an action or choice is “free" or not? And, (2) why is the belief in
free will so common? Where does this belief come from, and what is its
purpose, if any? In this paper, we examine these questions from the
perspective of reinforcement learning (RL). RL is a framework originally
developed for training artificial intelligence agents. However, it can also be
used as a computational model of human decision making and learning, and by
doing so, we propose that the first problem can be answered by observing that
people’s common sense understanding of freedom is closely related to the
information entropy of an RL agent’s normalized action values, while the
second can be explained by the necessity for agents to model themselves as if
they could have taken decisions other than those they actually took, when
dealing with the temporal credit assignment problem. Put simply, we suggest
that by applying the RL framework as a model for human learning it becomes
evident that in order for us to learn efficiently and be intelligent we need
to view ourselves as if we have free will.

_Keywords_ free will belief $\cdot$ reinforcement learning $\cdot$ model-based
$\cdot$ credit assignment

##  1 Introduction

Although, according to the current understanding of the laws of nature, there
are good arguments in support of the view that it is not possible for humans
to have free will, so-called free will belief is widely spread across cultures
(Wisniewski et al., 2019). This discrepancy between the materialist view held
by large parts of the scientific community, on the one hand, and the
subjective experience most laypeople have of their own freedom, on the other
hand, begs for an explanation. If we assume that the materialists are right
(i.e., that we lack free will), we must ask ourselves the following questions:

  1. 1.

What does “freedom" mean when used in everyday language? Freedom is a commonly
used concept but what are we actually referring to when we say that an action
or choice is “free" or not?

  2. 2.

Why is free will belief so common? Even the most determined determinist feels
as if he or she can make free choices. Where does this experience come from,
and what is its purpose, if any?

In this paper, we attempt to address these questions from the perspective of
an agent trained with model-based reinforcement learning (RL) (Silver et al.,
2017; Levine and Koltun, 2013; Deisenroth and Rasmussen, 2011). Reinforcement
learning is a framework for training artificial intelligence (AI) agents in
settings where there is a delay between the actions of the agent and the
evaluation of the objective of the agent. The purpose of an RL agent is to
learn to perform sequences of actions that maximize the long-term reward it
receives from its environment. Where reward is received when the objective
assigned to the agent is fulfilled. A classic example is games like chess,
where an AI should learn to make a sequence of turns that lead to a win.

Importantly, reward maximization through reinforcement learning has been
proposed not only as a method for building AI systems, but also as a
computational theory of human learning and decision-making (Silver et al.,
2021). Although the human brain most definitely mixes many different kinds of
algorithms and objectives, reward maximization is suggested as a general
framework for how to express the multitude of objectives. The pursuit of
reward allows humans to optimize for long-term success, which is subjectively
defined through the reward system of the brain—a system that has evolved to
balance different needs and desires and preserve the homeostasis of the body.
Accordingly, reward is hypothesized to be a “common currency" that allows for
most (if not all) aspects of intelligence and behavior to be prioritized and
integrated.

In the literature, it has frequently been argued that the problem of free will
is a semantic issue. What do we actually mean with free will? (e.g., see the
debate between Dennett (2017) and Harris (2012)). In this paper, we agree that
this is indeed the case. The assumption that humans are reward-maximizing RL
agents offers a possibility to resolve the conceptual confusion. We argue
that, whenever we talk about freedom, we are conflating two different
concepts. The first of these concepts—which we will refer to here as physical
freedom—is what incompatibilists have in mind when they conclude that the
currently known laws of nature do not permit one to have free will in any
meaningful way. This kind of freedom is essentially randomness. It is the
possibility for things to happen without a cause, which is in physics a
consequence of quantum mechanics, or at least according to some
interpretations thereof. Therefore, it makes perfect sense to conclude, as
hard incompatibilists do, that physical freedom is not something we can
possess, since if something is random, it is by definition not under our
control (Pereboom, 2001).

However, as we will show, the RL perspective allows us to talk about and
formally define the freedom of an agent in a different yet meaningful way. We
refer to this other type of freedom as value freedom. In this term, “value"
does not refer to values as in moral values, but rather to the estimated
values of actions available to an RL agent given its current knowledge of the
environment. The value freedom of an agent is defined as the information
entropy of its action selection process and thus can, at least in some sense,
be viewed as a measure of unpredictability, regardless of whether the
selection process is fully deterministic or somewhat stochastic. Accordingly,
we suggest that the concept of value freedom bridges the everyday use of the
word free, the compatibilist view that it is meaningful to talk about free
will even under strict determinism, and the incompatibilist view according to
which our understanding of causality does not permit us to have free will in
the physical sense.

Furthermore, we also suggest that value freedom is beneficial for agents due
to its connection to the exploitation vs. exploration dilemma faced by all
agents trying to learn from delayed rewards (Thompson, 1933; Lai and Robbins,
1985). High freedom states are desirable because they signal low-risk
exploration and provide room for potential policy improvements.

Next, we argue that the RL perspective sheds light on why humans so strongly
believe they have free will and that they can make decisions independently of
causes outside of themselves. Central to efficient RL is a solution to the so-
called temporal credit assignment problem (Minsky, 1963). Since rewards are in
general sparse and delayed, an RL agent has to figure out which of all the
actions leading up to the reward were actually important for the outcome.
However, at present, most commonly used RL algorithms rely only on temporal
proximity heuristics where the actions closer in time to the reward are simply
assigned more credit than those farther away. This trick obviously breaks down
in more complex environments with long temporal dependencies, as well as when
different tasks are interleaved with each other. In such complex settings, in
order to reinforce the correct behaviors, agents must figure out which actions
were actually important, and why the reward was received. Therefore, credit
assignment becomes a problem of counterfactual reasoning using a causal world
model. Agents need to answer questions like “If I did A instead of B, would I
still receive reward C?" In other words, agents must be able to simulate or
imagine having made choices they did not in fact make. We suggest that this
necessary—albeit, from a deterministic point of view, flawed—self-model can
explain the human experience of being free. We tend to think of ourselves as
being capable of breaking the causal structure of our environment and making
free choices, and this imagined freedom—despite being, in all probability,
just a figment of imagination—is necessary for efficient learning and thus a
vital part of our intelligence.

In what follows, we start with a description of model-based RL and the
challenges that learning agents, including humans, face when trying to
maximize reward. Then, based on the formulated definition of RL, we proceed to
define the value freedom of an agent and, through a series of examples,
demonstrate that this quantity closely relates to the concept of freedom, as
used in everyday language. In the next section, we discuss why the credit
assignment problem makes it necessary for efficient model-based RL agents to
believe that—or at least act as if—they have free will that defies causality.

##  2 Reinforcement learning

To define the value freedom of an agent, we start by giving a formulation of
RL. In RL, an agent is imagined to exist in an environment with a high-
dimensional and time-evolving state, $\mathbf{s}_{t}$. However, in the general
case, this state is only partially observed by the agent with observations
denoted by $\mathbf{o}_{t}$. The agent also has a number of potential actions,
$\mathbf{a}_{t}$, it can choose from at each time step. These actions alter
the observed state by changing both the environment and the part of the
environment that is observed, hence
$\mathbf{o}_{t+1}=f(\mathbf{s}_{t},a_{t})$. Furthermore, the environment emits
a scalar reward/penalty to the agent at each step,
$\mathbf{r}_{t}\in\mathbb{R}^{1}$, with
$\mathbf{r}_{t+1}=f(\mathbf{s}_{t},a_{t})$. An RL agent’s objective is to find
an action selection rule, or a so-called policy, $a_{t}=\pi(\mathbf{o}_{t})$,
such that it maximizes the expected long-term reward the agent receives:

$V^{*}(\mathbf{s})=\max_{\pi}\operatorname{E}[r|\mathbf{s},a,\mathbf{\pi}]\in\mathbb{R}^{1}$
(1)

where $V^{*}(\mathbf{s})$ is the expected reward or value received by the
agent when starting in state, $\mathbf{s}$, and choosing a sequence of
actions, $a_{1...t}$, according to the optimal policy $\pi^{*}$. Hence, the
goal of RL is to find a policy, $\pi$, which as close as possible result in
the same expected reward as $\pi^{*}$.

To break down the problem further, we also define the action value of an
action in state s, as follows:

$Q^{*}(\mathbf{s},a)=\operatorname{E}[r|\mathbf{s},a,\mathbf{\pi}^{*}]\in\mathbb{R}^{1}$
(2)

which is the expected reward received over time if an action, $a$, is chosen
according to $\pi^{*}$, starting in state $\mathbf{s}$. Hence, one way to
maximize $V^{*}(\mathbf{s})$ is to try to estimate $Q^{*}(\mathbf{o},a)$ and
then choose the actions with the highest value at each time step. If
$Q^{\pi}=Q^{\pi}(\mathbf{o},a)$ is the agent’s estimated action values given
its currently observed state $\mathbf{o}$, and its policy is
$a=\pi(\mathbf{o})=\text{argmax}(Q^{\pi}(\mathbf{o},a))$, RL comes down to
learning to estimate $Q^{\pi}$ as closely as possible to the optimal action
values, $Q^{*}$.

As a side note, observe that the reward signal, $r_{t}$, can be viewed as a
special dimension of $\mathbf{s}_{t}$ that attains its status from the
objective of RL to maximize the expected reward. In the case of humans, the
reward system of the brain can thus be viewed as a special part of the
environment whose output the learning system of the brain tries to maximize.

The challenge of RL arises in environments where the reward signal is sparse,
i.e., mostly zero, which is generally the case. For instance, in a game of
chess, you do not receive a reward until you have performed a long sequence of
actions that result in a win or loss. Owing to this delay between actions and
rewards, RL presents a learning agent with two central challenges: reward
prediction and credit assignment.

If an agent can predict how its behavior will alter its environment and what
future states will result in reward, it can use these predictions to better
choose its actions, i.e., estimate $Q^{\pi}$ as accurately as possible. Less
trial-and-error is needed if an agent, when faced with a new situation, is
capable of seeing into the future to make predictions of the outcome of its
actions.

In contrast to planning that requires an RL agent to predict the future, the
problem of credit assignment requires it to understand the past. If a sequence
of actions leads to a successful outcome, the agent needs to figure out which
actions were actually important for the success, and which were not.
Accordingly, the agent has to assess which behaviors shall be reinforced,
making them more likely to be repeated in similar situations in the future,
and which shall be suppressed.

Model-based RL (Silver et al., 2017; Levine and Koltun, 2013; Deisenroth and
Rasmussen, 2011) is an approach to tackle both the problem of reward
prediction through planning and/or that of credit assignment by equipping the
agent with some form of generative model of its environment, i.e., a world
model. Specifically, reward prediction can be achieved by running the world
model forward, allowing the agent to search for favorable action sequences
without really performing them, while credit assignment is attained by
employing the world model to evaluate which past actions were actually
important for the received reward, without needing to go back and test
alternative sequences. Accordingly, credit assignment can be viewed as a
problem of causal inference (Pearl, 2010), where actions causally necessary
for reaching the rewarding state are identified and assigned credit. Both
prospection and retrospection through a causal world model are thus crucial
for effective RL.

##  3 What is freedom?

Similar to many other concepts used in everyday language, freedom and
voluntary vs. involuntary action are not precisely defined. However, this does
not make them less useful. For instance, the United Nations (UN) Universal
Declaration of Human Rights (UN, 1948) is sprinkled with the words free and
freedom.

Yet, if we seriously consider the perspective that humans are RL agents, a
problem emerges. Namely, what does it mean for an RL agent to make free
choices? An RL agent is just an algorithm or a collection thereof, be it
deterministic or somewhat stochastic, or equipped with a world model or not.
An RL agent is still just a mapping between states of the environment and
actions. It is difficult to talk about algorithms having free will.

We suggest that the solution to this problem starts by noting that the action
value estimates, $Q^{\pi}(\mathbf{o},a)$, represent something close to the
“will" of an agent, since in every situation an agent selects the action with
the highest expected value. Therefore, action values represent what an agent
wants to do, i.e., what outcomes it expects to be rewarding or least costly,
and how effective it believes the available actions are when trying to reach
those outcomes. For instance, if the agent is an imprisoned human, he or she
might really want to escape, but, at the same time, realizes that the
probability for success is minuscule and thus estimates the action value of
escaping to be rather small. Therefore, after an agent has considered the
probability of success, the resulting action values can be viewed as the will
of that agent.

Based on the aforementioned definition of “will", we suggest that it is also
possible to quantify how free an agent is. To this end, we define the action
selection probability as:

$P(a_{i})=\text{softmax}(Q^{\pi}(a_{i}))=\frac{\exp(Q^{\pi}(a_{i}))}{\sum_{j}\exp(Q^{\pi}(a_{j}))}$
(3)

where the softmax function normalizes111The exact method of normalization in
not important. We here ignore the problem that the softmax function is not
scale invariant. the estimated action values into a distribution over actions.
We then use this to define the freedom of an agent with an observed state as
the information entropy of the action selection distribution:

$H_{a}=-\sum_{i}P(a_{i})log_{2}(P(a_{i})).$ (4)

This quantity, which we refer to as value freedom, can thus be interpreted as
the uncertainty of an agent’s action selection if the policy samples action
weighted according to P(a). It is high in situations where the agent’s
estimated action values, $Q^{\pi}(\mathbf{o},a)$ , are uniform, and low when
the value of one or a few actions appears to be markedly higher than those of
all other actions.

However, note that for this treatment of RL it does not matter whether the
policy is stochastic or deterministic, as long as we can define the relative
values of actions given a state, $Q^{\pi}(\mathbf{o},a)$, and normalize these
into an action selection distribution, $P(a)$. For instance, the policy,
$\pi(\mathbf{o})$, can be greedy as above and simply select the most valuable
action, or stochastic and randomly sample actions according to P(a) somehow.
Stochasticity can also enter through noise in the estimation of
$Q^{\pi}(\mathbf{o},a)$ rather than in the action selection process.

To demonstrate that value freedom, $H_{a}$ , is indeed a quantity that closely
maps to what, in everyday language, is referred to as freedom of choice, let
us imagine two idealized scenarios where one could consider a binary choice to
be free or not (here we switch to a second-person perspective to allow the
reader to better imagine themselves in these situations):

  1. 1.

You are sitting on a lush green lawn. Someone asks you in a friendly way to
raise your left arm if you like, with no consequences whether you do it or
not.

  2. 2.

You are standing at the edge of a precipice with a deep dark ocean crashing
against the sharp cliffs beneath you. Someone sneaks up behind you and
threatens to push you off the ledge if you do not raise your left arm.

Let us also assume you can be very certain that what is promised will also
happen, and that you are not suicidal.

When comparing these scenarios, one could argue that you are not free to
choose what to do in either case, because you are just an algorithm and, as
such, have no free will. Whether you raise your arm or not is just a result of
the initial conditions of the situation and some deterministic or stochastic
machinery in your head. However, in the common-sense meaning of being free, it
is obvious that you are freer to choose what to do in the first scenario.
Nothing is forcing you to do anything and, if you raise your arm, it seems as
if you were free to not do it. So what is the difference between these two
situations?

The central argument here is that the reason behind our perception of these
two cases as very different is not that you are actually more physically free
in one case or the other, but rather that you estimate the action values,
$Q^{\pi}(\mathbf{o},a))$, differently. Your goal (or that of any other RL
agent) is to maximize the reward you receive over time. To this end, you need
to continuously estimate the future reward of every available action and
select the action with the highest estimate. Hence, you always perform the
action that you subjectively estimate to be the best, i.e., the one that, in
your assessment, has the highest estimated action value.

From this perspective, the difference between the two scenarios lies in how
you reasonably would estimate the future reward of raising your arm or not. In
the first scenario, raising your arm has no consequences; therefore, doing so
or not has basically equal values (if we disregard that the tiny bit of
physical exercise probably would do you good…). In contrast, in the second
scenario, your fear of death produces a very large estimated negative reward
for the option of not raising your arm.

For more examples of how various value estimates, in binary choice situations,
result in different levels of value freedom, see Figure 1 and Table 1.

Note that you could also imagine that your policy is random, and that you do
not always select the action with the highest value, but instead somehow
sample your actions weighted according to the estimated rewards, since your
action selection process might be more or less noisy. Nevertheless,
intuitively, the “freedom" of your policy is large in the first scenario and
very small in the second scenario, because, in the former case, there is
simply less difference in value between available options. Indeed, there is
empirical evidence showing that a person indifferent to the consequences of
his/her choice is seen by lay people as maximally free (Deutschländer, 2017).

Figure 1: (A) How most people would reasonably estimate the reward and
penalty, i.e., the value, for the option of raising their left arm as
requested in scenarios 1 and 2. In scenario 1, both alternatives would lead to
a zero reward. In scenario 2, not raising the arm would likely lead to a very
large penalty (death). The exact scale of the reward is not important here.
(B) The action values converted into probability distributions. These can be
viewed as probabilities for raising the arm or not, although the actual action
selection is deterministic. (C) The Shannon entropy for the probability
distributions in B. For a uniform distribution with only two options, the
maximum entropy is 1 bit. If there is a 99% chance of raising the arm in
scenario 2, the entropy is approximately 0.1 bit.

Value of 1st choice, $Q^{\pi}(\mathbf{o},a_{1})$ Value of 2nd choice,
$Q^{\pi}(\mathbf{o},a_{2})$ Value freedom, $H_{a}$ Interpretation 0 0 High You
are indifferent to $a_{1}$ or $a_{2}$ 0 + Low a2  seems to result in a reward
you cannot afford to not choose (e.g., money, or pleasure). 0 - Low a2 seems
to result in a penalty (e.g., physical violence (pain)). + + High All choices
seem good. - - High Bad state to be in, but you are still free to choose
between plague or cholera. We often view these states as unfree, since we
imagine that there is a third neutral option (no plague and no cholera). + -
Very low $a_{1}$is undoubtedly and massively better than $a_{2}$. $a_{2}$ is
probably not even considered as an option.

Table 1: Table 1: How possible estimated action values result in different
levels of value freedom in a binary choice situation. “+" and “-" refer to
large positive or large negative action values.

At this point, one might rightfully object that, thus far, this seems like a
very narrow definition of freedom. Isn’t it possible to be very sure about
what you want to do, but still feel free to do so or not? When we talk about
freedom, are we really referring to uncertainty in our action selection? For
instance, let us imagine the following scenario:

You sit down in a restaurant and are handed the menu. You can pick anything
from it. You are free to choose your dinner: you have no allergies, and you do
not have any particular preferences this evening. After casually skimming
through the menu at random, you decide to have meatballs with mashed potatoes.
You sip on your drink and think about something else. When several minutes
later the waiter comes over to take the order, he does not accept your choice
right away but instead suggests you should try the fried herring. It is
supposedly much better. Immediately, you feel the annoyance grow inside. Who
would ever want fish when they could have meatballs?!

In the above scenario, you could argue that you feel completely free to make
whatever decision you want, but you still prefer one particular option over
the others; hence, what we previously defined as value freedom cannot be what
our understanding of freedom is based on. However, we would like to argue that
the reason for this apparent contradiction is two-fold.

First, you feel free to choose whatever you want to have for dinner because it
really does not matter much. Even if you are disappointed when you cannot have
meatballs, you know that the relative reward between having herring,
meatballs, or any other well-cooked dish is marginal. In contrast, in a
scenario where, after starving for a week, you have the choice between having
meatballs and no dinner at all, your freedom is quite limited, to say the very
least. Hence, the value freedom is actually relatively high in the original
scenario.

Second, the value freedom was indeed very high when you first sat down and
started skimming through the menu. Since you had no initial preferences, your
memory of the decision is that you were free to pick anything. However, once
taken, your decision somehow changed your preferences, and suddenly meatballs
seemed much more rewarding than what else was on the menu. Accordingly, when
the waiter tried to change your mind, your value freedom decreased, since you
were now less probable to choose the herring or anything else. In this way,
our sense of being free to make decisions is based not only on our current
value freedom, but also on our memory of the causal chain of decisions we
made, and the value freedom in those situations. Note that, for an agent
without any memory of past choices, this would not be the case.

The purpose of the examples provided above is to show that our common-sense
understanding of freedom comes from how we (i.e., our reward system and world
model) subjectively estimate the values of available actions, rather than from
how free we are in a physical sense. It is this type of freedom that we refer
to in everyday language when we say that someone is free to choose or forced
to do something. If someone is threatened with violence, they tend to do what
they are told and, therefore, have low value freedom since their actions
become predictable.

However, there is one additional aspect that needs to be accounted for when we
assess if someone’s actions were free or not. First, note that the value
freedom can be defined for any system— be it a chess computer or an
insect—that evaluates different options and somehow selects between them.
Furthermore, the value freedom per se does not depend on the quality of the
evaluation. A weak chess computer that plays on a beginner’s level can on
average have the same value freedom as the one playing like a grandmaster.
Similarly, the intoxicated you can have the same value freedom as the sober
you.

Therefore, when judging the freedom that we or someone else possess when faced
with a decision, we do so taking into account not only the relative estimated
values of available options but also the quality of the estimates.
Accordingly, our common-sense understanding of when a person makes a voluntary
action according to his or her free will relies on both the assumption that
the value freedom is high and that the person’s competence to judge the
relative value of available options is roughly equal to what can be expected
from an adult human being in a similar situation and culture. Using our world
models and counterfactual reasoning, we try to understand how the person would
have predicted his or her future rewards, given that the person, for example,
was not intoxicated or suffering from any other condition that could have
reduced the quality of his or her predictive power. Other common reasons for
us to not judge someone’s actions as free are deception and young age.
Similarly, we analyze our memory of our own actions and their causes and might
feel that we did not do something out of free will, since we misjudged the
situation due to some external factors beyond our control.

Another exception from only considering the momentary value freedom is how the
future outcome of actions influence our perception of freedom. A consequence
of quantifying freedom as entropy is that splitting the action space into more
actions with the same or very similar outcome increases the freedom. An agent
faced with choosing between two actions that result in the same state is
obviously not free. In such situations one must consider not only the value of
actions but also the diversity of their outcomes. However, this is subjective
to the agent. Two agents can have completely different opinions on whether
outcomes are similar or not depending on their reward function. And, further
note that outcome is not equivalent to received reward. Actions can have the
same estimated value but still result in very different future states. For
instance, imagine choosing between two vacation destinations. You might
estimate both as equally likely to give you pleasant experiences, but you
still know that when you come home these experiences will have been vastly
different.

In summary, we argue that the freedom of an agent can be quantified by looking
at the uniformity of $Q^{\pi}$. The amplitudes of $Q^{\pi}$ naturally
translate to the will of the agent, since they represent how beneficial each
action appears to be to the agent, while the uniformity of $Q^{\pi}$, here
suggested to be measured through the information entropy, Ha, corresponds to
how “free" the will is. In other words, value freedom could also be viewed as
the “entropy of the will". And as such, its unit, bits, corresponds to how
much information is gained by observing the choice of an agent. Since the
decisions where the value freedom is high involve more uncertainty concerning
what the agent will do, the information content is higher than for low freedom
decisions.

In addition to the momentary value freedom, we also suggest that our common-
sense understanding of whether an agent’s action given a state is free is also
based on the following factors:

  1. 1.

The historic value freedom: How high was the value freedom in situations
leading up to the current state? A decision is rarely made in isolation.

  2. 2.

The quality of the action values: Does the agent estimate action values, and
thus the value freedom, as well as one can expect?

  3. 3.

The diversity of the outcomes: Does the agent consider the outcomes of
available actions different enough for the actions to be seen as separate
choices?

Note that all these factors are highly subjective, from the perspective of
both the agent itself and other agents judging someone else’s freedom. Value
freedom is inherently subjective, while physical freedom is objective.

##  4 Why do we believe we are free?

Now that we have a way to quantify and think about our subjective
understanding of freedom through the concept of value freedom, we will address
the other questions we set out to answer: Why do we feel that we have free
will, and what is the purpose of this feeling? Two questions which we suggest
can be translated to: (1) Why do we tend to mix up value freedom with physical
freedom? And, (2) why is high value freedom desirable?

We hypothesize that the answer to (2) can be found in the need for RL agents
to deal with the famous exploration vs. exploitation dilemma (Thompson, 1933;
Lai and Robbins, 1985). High value freedom indicates to an agent that the risk
of exploration is low, because, in such situations, it looks as if there is
not much to lose from exploring. Since several alternatives have similar
predicted values, choosing one that is not known to be optimal is perceived as
less of a risk. From this perspective, freedom can be seen as room for
learning. When value freedom is high, an agent has the opportunity to test new
approaches that hopefully will improve its world model, leading to an improved
policy. Hence, high value freedom signals potential low-risk policy
improvements.

To exemplify, let us reuse the scenario where you sit down at a restaurant and
are about to choose what to have for dinner. If you do not have any particular
preferences this evening, you are free to try something new, which may turn
out to be your new favorite dish. However, if you are on a strict diet that
limits your options, the likelihood of you discovering something new is
obviously much lower. Hence, less value freedom presents fewer opportunities
for learning.

But, note that the value freedom does not say anything about how good your
options are in an absolute sense—rather, it only shows how you subjectively
value them relative to each other. Therefore, if all dishes on the menu look
equally disgusting to your refined taste, then you are still bad off, even if
your freedom is high. The inductive bias of “freedom desire" is thus balanced
towards other goals, like having good food, through the common currency of
reward. We are all familiar with the fact that people are often willing to
give up some of their freedom in exchange for the fulfillment of other more
basic needs.

We hypothesize that, rather than directly giving an agent a higher reward,
striving for freedom makes the policy more robust and increases the venues for
improving it in the future. Similarly to curiosity (the intrinsic reward we
get when we learn something new, i.e., improve our world model) (Schmidhuber,
1991; Deepak et al., 2017), our sense of freedom can thus be viewed as an
inductive bias that evolution has given us to increase our learning
efficiency. On this view, freedom is also intimately related to curiosity.
Only a free learning agent can afford to be curious.

Intuitively, this makes sense. If you are always forced to do the same thing
over and over again, there is no room to try anything new, and you will never
learn. However, an important distinction to be made here is that the lack of
freedom does not necessarily mean the lack of exploration. We all have been in
situations where our exploration was guided by a teacher, and where we trusted
our teacher to show us the best path forward. When we follow a course or a
curriculum, this is what happens. We suggest that supervised exploration can
be seen as a middle-ground between imitation learning and reinforcement
learning, where the teacher does not explicitly demonstrate what the correct
answer or solution is, but instead points the learner in the right direction
by supplying problems of suitable difficulty. In this mode of learning,
although the learner’s value freedom can be low, exploration and learning
still happen, and such learning can often be more efficient than when the
learner relies only on unsupervised exploration. This is why schools were
invented. During supervised exploration, the teacher employs its model of the
learning domain and the learner and, through authority or trust, skews the
learner’s reward estimates to make some actions look more preferable than
others.

Now that we have discussed a potential motivation for the importance of
(value) freedom to learning agents like humans, let us turn to the other
question: Why do we tend to mix up value freedom with physical freedom?

The short answer is: Because we constantly imagine that we have physical
freedom. This fantasy, although arguably rather wishful, fills a very
important purpose in the learning algorithm that makes us intelligent.

Remember that temporal credit assignment is a challenge faced by all agents
that need to learn from delayed rewards. The problem is essentially to figure
out what actions leading up to a reward or penalty were responsible for the
outcome and reinforce or weaken similar behavior in the future. In most work
on RL, this problem is dealt with by relying on a temporal proximity heuristic
where the actions closer in time to the reward state are straightforwardly
assigned more credit than those farther away, i.e., the credit is somehow
discounted based on the temporal proximity to the reward. A notable example of
this is the board game playing system AlphaGo and its descendants (Silver et
al., 2017; Silver et al., 2018) that use a hardcoded model of the world (the
rules of the game) for planning and, when doing credit assignment, simply
assign the same credit to all moves of a game leading up to a win or loss.
This works in board games like Go or chess which have little causal invariance
(most actions/turns influence the end state of the game), and where it is easy
to generate a lot of independent action trajectories (e.g., chess games), but
fails when actions have very different importance, and you cannot afford to
fail multiple times, or in multi-task settings where it is not obvious which
action influenced which reward state. For instance, imagine a Wall Street
investor who assigns credit to his daily choice of a tie when he reaches the
office in the morning and finds out that the value of one of his assets has
crashed overnight. Superstition can in many cases be viewed as a failure of
credit assignment.

Instead of relying only on temporal proximity, agents become more efficient
learners if they are capable of figuring out what actions were actually
important for a received reward or penalty. They must understand which actions
had a causal influence on the reward. To this end, agents need to be able to
imagine choosing other actions than those that were actually chosen. It comes
down to answering counterfactual questions, such as “What would have happened
if I had not done X? Would Y still happen?" Accordingly, to figure out what
actions caused a received reward or penalty, agents need to imagine
alternative realities. Along with planning by simulating the future, the
simulation of such alternative histories is a key benefit of having a world
model that encodes causal relationships between variables. So-called hindsight
credit assignment has been investigated (Harutyunyan et al., 2019, Mesnard et
al., 2020; Guez et al., 2020), albeit, to the best of our knowledge, only for
model-free RL, i.e. RL when there is no explicit world model.

One can argue that not only credit assignment but also planning involves
imagining taking actions independently of causal influence from the
environment. When simulating future scenarios, agents need to evaluate the
consequence of different decisions and can imagine being free to do virtually
anything as long as it can be represented by their world model. For instance,
Deery (2014) argued that prospection, i.e., our experience of different future
scenarios through mental simulation, is the cause for people’s free will
belief. However, since the action of planning lies in the causal path to the
future, it is less clear that the imagination of being free in the future
contradicts a deterministic model of the world and the agent itself. For
instance, you can easily imagine doing something obviously stupid with a very
low action value tomorrow, and hence imagine doing something that goes against
your deterministic decision process. But if you actually go ahead and do this
thing tomorrow, it is because reading this sentence and then thinking about
what to do have skewed your value estimates to make it seem as if that thing
is worth doing, and thus you have not escaped determinism.

Since planning involves modeling the future, at least in its basic form, it
does not require you to think about yourself as non-deterministic and having
the ability to do differently than you actually did, i.e., no counterfactuals
are needed. Although, it is not uncommon for humans to figure out what to do
by imagining alternative versions of the present. One way to evaluate whether
it is a good idea for us as a society to build an artificial general
intelligence is to imagine an alternate reality where this has already been
done.

In summary, we argue that, in order to efficiently learn from our successes or
mistakes, we need to imagine that we are free. This imagined freedom is
physical freedom in the simulated reality of our world model. When planning
and performing credit assignment by counterfactual reasoning, we break the
causal structure of our virtual reality and basically decide that something
that did not in fact happen actually happened. In other words, we decide that
events should occur without causes. The real causes for events in our mental
simulation lie outside the simulation. In the frame of reference of the
simulation, they are, to use physics jargon, hidden variables. Therefore,
according to our imagination, we have physical freedom. Our ability to
mentally break the chains of determinism makes us believe that—or at least
feel as if—we really have physical freedom and that we can make decisions that
are not ultimately caused by events external to ourselves.

##  5 Discussion

In this paper, we argued that centuries-long philosophical debate about free
will is a result of a necessary, but flawed aspect of our self-concept. Our
models of ourselves say that, when offered a choice, we could have chosen to
do something that we did not in fact do. This model has a good purpose.
Without it, we would be unable to apply counterfactual reasoning to the
problem of credit assignment and efficiently learn from our successes and
failures. To learn in complex situations, we need to imagine that we could
have made other choices than those we actually made. And, interestingly, this
model of ourselves is not something we seem free to get rid of. On some level,
even the most determined determinists act as if they have free will. While
they might intellectually know that they have no physical freedom, if they
truly thought as if they did not, they would fail to learn efficiently.

Furthermore, we argue that this contradiction can be resolved by applying a
reward maximization perspective on human decision-making. It allows us to
formalize what free and will actually refer to and realize that we do not
really need to believe that we have physical freedom. Instead, what we refer
to here as value freedom suffices. When we reason about our own freedom, write
laws, or communicate with each other in general, this is the kind of freedom
we are actually referring to. Therefore, our freedom is not based on our
ability to act independently from the world—rather, it emerges from our
brain’s reward systems and the predictions of our world models. And since
freedom tells us something about how much room for exploration and learning we
have, it becomes, in Dennett’s (1984) parlance, “worth wanting"—for humans,
and for learning agents in general.

Of note, our definition of value freedom does not rely on RL being a complete
and accurate theory of human learning and decision-making. There is no
requirement for any learning to occur, and rewards do not necessarily need to
be scalar. The only requirement is that agents should be capable of selecting
actions based on their expected relative benefit. Such a selection process
appears to be a condition for agency itself. To choose is to weigh different
options against each other, and then make a decision. While the nature of this
weighting can considerably vary, agency postulates the need for some kind of
ordering of alternatives. Similarly, it is clear that humans are capable of
performing complex credit assignments using some sort of causal model of the
world.

In some sense, by clearly defining what freedom is, regardless of whether or
not determinism holds, our conceptualization of value freedom allows for a
compatibilist view on free will. We are not disagreeing with the classic
incompatibilist argument (McKenna $\&$ Coates, 2021) that determinism does not
allow for free will; rather, what we propose is that the kind of freedom lay
people actually refer to in everyday language is not the same kind of freedom
incompatibilists reject. This distinction between what we here call physical
freedom vs. value freedom allows for several potentially interesting
perspectives.

For instance, when the UN declares that freedom of opinion is a human right,
this does not imply that humans shall be able to choose their opinions
independently from their environment. Rather, this means that people shall be
able to do so without risking being punished for “wrong" choices, or bribed
into making “correct" ones. When people vote to select their leaders, they
should not perceive huge differences in estimated rewards between candidates.
Although some differences, seemingly more or less important, must obviously
exist, voters should know that making one choice or the other will not
massively change their outlook. They will not get jail time, nor will they get
a promotion, regardless of how they cast their votes. Said differently,
freedom of opinion means that, when forming opinions, agents should not
experience any extremes in the action value distribution.

From this perspective, political liberty should be seen not only as freedom
from oppression, but also as freedom from excessively strong positive
incentives. Since freedom stems from reward prediction, it has a dual nature
where it can be restricted from both a negative and a positive direction. This
duality comes in many well-known pairs, such as whips or carrots, fines or
bribes, rape or prostitution. That we frequently consider only the negative
part of this spectrum as potential restrictions of freedom is presumably due
to the asymmetry between suffering and pleasure—an asymmetry that, on a
fundamental level, likely arises from the irreversibility of death. Beyond
suffering lies the risk of death, while pleasure has no such ultimate
consequence. Beyond pleasure lies only more or less pleasure. It is very
difficult to imagine an experience equally positive as the negative experience
of slowly dying of physical torture. Therefore, to reduce freedom to the same
extent as moderate penalties do, a reward must be very large.

Furthermore, the RL perspective on human decision-making also suggests that
laws, be they written or moral, do not exist because they reflect some
metaphysical justice or ideal, but rather because they are ways for societies
of learning agents to teach each other how to act. Nothing is morally good or
bad in itself, since the only value judgments that exist are action value
estimates made by RL agents. Instead, the reason for societies to have rules
is to shape the action value estimates of their members. Accordingly, the
classic incompatibilist position—the one that submits that, without free will,
no one can be morally responsible for their actions (McKenna $\&$ Coates,
2021)—is based on a misunderstanding of the purpose of assigning
responsibility. Moral responsibility must be forward-looking (McKenna $\&$
Coates, 2021). The main reason for punishing members of a society, or any
other group, for something they have done, is to discourage similar behavior
in the future, and it makes little difference whether they were physically
free to do what they did or not. From an RL perspective, what is important is
to reduce the likelihood of someone making the same choice again in a similar
situation by skewing future action value estimates. The “responsibility
assignment" that societies have created courts for can be viewed as analogous
to the credit assignment single RL agents need to deal with when learning what
behaviors to suppress and what behaviors to reinforce. In this view, courts
are important components of the multi-agent learning system we call society.

There has been some empirical research on the connection between
counterfactual thinking and free will belief. For instance, in a series of
experiments, Alquist et al. (2015) found the belief in free will to be linked
to increased counterfactual thinking. The authors then hypothesized that
“individuals and society as a whole may have benefitted from free will beliefs
and the counterfactual simulations they stimulate". Here we show why this is
the case and demonstrate not only that individuals and societies benefit from
counterfactual thinking, but also how and why such thinking is essential for
learning on a fundamental level through its role in efficient credit
assignment.

Another observation is the relation between freedom and indifference, and how
the latter is central to certain religious beliefs. In line with empirical
accounts of laypeople’s intuition about freedom (Deutschländer et al., 2017),
an indifferent agent has high value freedom because it assigns equal value to
multiple available options. However, indifference also means that the
estimated action values are close to zero, while high freedom only implies
that several options have approximately equal value. If an agent assigns a
very high value to multiple options (but not all), that agent cannot be said
to be indifferent, as it still thinks selecting one of these options is
important. This relation between freedom and indifference has interesting
philosophical analogies. According to the definition of value freedom, a
maximally free agent is the one that always estimates all actions to have
equal value. Such an agent never wants anything more or less than anything
else, and obviously, would never learn anything and be utterly useless. One
can question if it would be an agent at all. Applied to humans, such an agent
is a person who completely lacks ego—an individual without any needs, drives,
or desires. Interestingly, this view that the destruction of the ego is the
ultimate path to freedom closely resembles a common theme in Eastern
philosophy. According to Buddha’s Four Noble Truths, to end personal
suffering, stop the cycle of rebirth, and reach spiritual liberation, nirvana,
we must let go of all desires. It is only when we have no desires—not even the
desire to rid ourselves of all desires—that we become completely free.
Accordingly, nirvana is described as a state of complete emptiness. It is when
we experience ultimate indifference, and end experience itself. Nirvana is
thus a state of eternal maximum value freedom.

A venue for future research is to empirically investigate whether information
entropy of the action selection distribution is an accurate measure of how
people estimate freedom in simple decision scenarios. One can assume that it
is more complex than that, and a considerable inter-individual variation can
be expected. However, if the hypothesis proposed in this paper is correct,
people’s freedom estimates should strongly correlate with the uniformity of
their action value estimates.

##  6 References

Alquist, J. L., Ainsworth, S. E., Baumeister, R. F., Daly, M., $\&$ Stillman,
T. F. (2015). The Making of Might-Have-Beens: Effects of Free Will Belief on
Counterfactual Thinking. Personality and Social Psychology Bulletin, 41(2),
268–283. doi:10.1177/0146167214563673

Coggan, M. (2004), ‘Exploration and exploitation in reinforcement learning’,
Research supervised by Prof. Doina Precup, CRA-W DMP Project at McGill
University.

Deepak Pathak, Pulkit Agrawal, Alexei A. Efros and Trevor Darrell. (2017)
Curiosity-driven Exploration by Self-supervised Prediction. In ICML 2017.

Deery, O. (2014). Why people believe in indeterminist free will. Philosophical
Studies, 172(8), 2033–2054. doi:10.1007/s11098-014-0396-7

Deutschländer R, Pauen M, Haynes JD. (2017), Probing folk-psychology: Do
Libet-style experiments reflect folk intuitions about free action? Conscious
Cogn. 2017 Feb;48:232-245. doi: 10.1016/j.concog.2016.11.004. Epub 2016 Dec
23. PMID: 28013177.

Deisenroth, M. and Rasmussen, C. E. (2011). PILCO: A model-based and data-
efficient approach to policy search. In Proceedings of the 28th International
Conference on machine learning (ICML-11), pages 465–472.

Dennett, D. C. (1984). Elbow room : the varieties of free will worth wanting.
Cambridge, Mass.: MIT Press. ISBN 0-262-04077-8.

Dennett, D. C. (2017), Reflections on Sam Harris’ "Free Will". Rivista
internazionale di Filosofia e Psicologia, [S.l.], v. 8, n. 3, p. 214-230, dic.
2017. ISSN 2239-2629.

Guez, A., Fabio Viola, Théophane Weber, Lars Buesing, Steven Kapturowski,
Doina Precup, David Silver, $\&$ Nicolas Heess. (2020). Value-driven Hindsight
Modelling.

Harutyunyan, A., Dabney, W., Mesnard, T., Gheshlaghi Azar, M., Piot, B.,
Heess, N., Hasselt, H., Wayne, G., Singh, S., Precup, D., $\&$ Munos, R.
(2019). Hindsight Credit Assignment. In Advances in Neural Information
Processing Systems.

Harris, S. (2012). Free will. Free Press.

Lai, T. and Robbins, H. (1985). Asymptotically efficient adaptive allocation
rules. Adv. Appl. Math., 6(1):4–22. 14

Levine, S. and Koltun, V. (2013). Guided policy search. In International
Conference on Machine Learning, pages 1–9.

McKenna, M., Coates, D.J. (2021), "Compatibilism", The Stanford Encyclopedia
of Philosophy (Fall 2021 Edition), Edward N. Zalta (ed.)

Mesnard, T., Théophane Weber, Fabio Viola, Shantanu Thakoor, Alaa Saade, Anna
Harutyunyan, Will Dabney, Tom Stepleton, Nicolas Heess, Arthur Guez, Marcus
Hutter, Lars Buesing, $\&$ Rémi Munos. (2020). Counterfactual Credit
Assignment in Model-Free Reinforcement Learning.

Minsky, M. L. (1963). Steps toward artificial intelligence. In E. A.
Feigenbaum $\&$ J. Feldman (Eds.), Computers And Thought (pp. 406-450). New
York, NY: McGraw-Hill.

Pearl J. (2010). An introduction to causal inference. The international
journal of biostatistics, 6(2), 7. https://doi.org/10.2202/1557-4679.1203

Pereboom, D. (2001). Living without Free Will. Cambridge: Cambridge University
Press. ISBN 9780521029964.

Schmidhuber, J. (1991). A possibility for implementing curiosity and boredom
in model-building neural controllers. In J. A. Meyer and S. W. Wilson,
editors, Proc. of the International Conference on Simulation of Adaptive
Behavior: From Animals to Animats, pages 222-227. MIT Press/Bradford Books

Silver, D., Schrittwieser, J., Simonyan, K., Antonoglou, I., Huang, A., Guez,
A., Hubert, T., Baker, L., Lai, M., Bolton, A., et al. (2017a). Mastering the
game of go without human knowledge. Nature, 550(7676):354.

Silver, D., Hubert, T., Schrittwieser, J., Antonoglou, I., Lai, M., Guez, A.,
$\ldots$ Hassabis, D. (2018). A general reinforcement learning algorithm that
masters chess, shogi, and Go through self-play. Science, 362(6419), 1140–1144.
doi:10.1126/science.aar6404

Silver, D., Singh, S., Precup, D., Sutton, R.S. (2021), Reward is enough,
Artificial Intelligence, Volume 299, 2021, 103535, ISSN 0004-3702.
doi:10.1016/j.artint.2021.103535

Thompson, W. R. (1933a). On the likelihood that one unknown probability
exceeds another in view of the evidence of two samples. Biometrika,
25(3/4):285–294. 14

Wisniewski D, Deutschländer R, Haynes JD (2019) Free will beliefs are better
predicted by dualism than determinism beliefs across different cultures. PLOS
ONE 14(9): e0221617.

United Nations (1948) Universal declaration of human rights.


v#1

# BayesDLL: Bayesian Deep Learning Library

Minyoung Kim1
1Samsung AI Center Cambridge, UK
<EMAIL_ADDRESS>&Timothy Hospedales1,2
2University of Edinburgh, UK
<EMAIL_ADDRESS>
###### Abstract

We release a new Bayesian neural network library for PyTorch for large-scale
deep networks. Our library implements mainstream approximate Bayesian
inference algorithms: variational inference, MC-dropout, stochastic-gradient
MCMC, and Laplace approximation. The main differences from other existing
Bayesian neural network libraries are as follows: 1) Our library can deal with
very large-scale deep networks including Vision Transformers (ViTs). 2) We
need virtually zero code modifications for users (e.g., the backbone network
definition codes do not neet to be modified at all). 3) Our library also
allows the pre-trained model weights to serve as a prior mean, which is very
useful for performing Bayesian inference with the large-scale foundation
models like ViTs that are hard to optimise from scratch with the downstream
data alone. Our code is publicly available at:
https://github.com/SamsungLabs/BayesDLL111A mirror repository is also
available at: https://github.com/minyoungkim21/BayesDLL..

##  1 Bayesian Neural Networks: Overview

The followings are the list of approximate Bayesian inference algorithms
implemented in the library:

  * •

Variational Inference (Sec. 2.1)

  * •

MC-Dropout (Sec. 2.2)

  * •

SG-MCMC (SGLD) (Sec. 2.3)

  * •

Laplace Approximation (Sec. 2.4)

The Bayesian neural network (BNN) is a Bayesian model where we treat the
parameters of the deep neural network (e.g., weights and biases) as random
variables that are endowed with some distribution a priori (prior
distribution). Like other Baysian models, there is a likelihood model that
assigns the compatibility score to the observation given the network
parameters.

Formally we use the following notations:

  * •

$\theta=$ Network parameters (weights $\&$ biases) of the underlying deep
model.

  * •

$\overline{\theta}=$ The most reasonable parameter values before observing any
evidence. In typical situations, we can have $\overline{\theta}=0$ meaning
that we have no prior information, or $\overline{\theta}$ can take pre-trained
model parameters on some base datasets, often called the upstream datasets.
For simplicity, we assume a Gaussian prior model in which case
$\overline{\theta}$ becomes the prior mean. More specifically, the prior
distribution is written as:

$\displaystyle p(\theta)=\mathcal{N}(\theta;\overline{\theta},\sigma^{2}I),$
(1)

where $\sigma^{2}$ is the prior variance (isotropic Gaussian) chosen by the
users. Obviously, the prior mean $\overline{\theta}$ and variance $\sigma^{2}$
are fixed constants.

  * •

$D=$ Given evidence. Typically $D$ is a supervised dataset ($D=\\{(x,y)\\}$
where $x\in\mathcal{X}$ is the input and $y\in\mathcal{Y}$ is the target
label, either class-valued or real-valued). As conventional practice we have
i.i.d. samples $(x,y)\in D$, which form the likelihood model,

$\displaystyle p(D|\theta)=\prod_{(x,y)\in D}p(y|x,\theta),\ \ \ \
p(y|x,\theta)\propto\exp(-l(x,y;\theta)/\tau),$ (2)

where $l(x,y;\theta)=l(y,f_{\theta}(x))$ is the conventional deep learning
loss (e.g., cross entropy or $L_{2}$ distance),
$f_{\theta}(x)\\!\in\\!\mathcal{Y}$ is the prediction of the deep network with
parameters $\theta$ and input $x$, and $\tau$ is the scaling hyperparameter
(e.g., temperature for cross entropy in classification cases or variance of
the output noise in regression cases).

#### Neural Network Learning

The main task is posterior inference, the task of inferring the posterior
distribution $p(\theta|D)$ of the weights given the evidence $D$. That is,

$\displaystyle p(\theta|D)=\frac{p(\theta)\ p(D|\theta)}{\int p(\theta)\
p(D|\theta)\ d\theta}.$ (3)

The denominator does not in general admit closed-form expression, it is even
infeasible to evaluate it exactly. Thus one has to resort to approximation,
and several well-known approximate inference algorithms are listed in the
beginning of this section, detailed in the next section, and implemented in
our BayesDLL library.

#### Neural Network Inference

Once the posterior inference is done, at test time we can use the posterior to
derive the test predictive distribution $p(y^{*}|x^{*},D)$ where $x^{*}$ is
the test input. In principle,

$\displaystyle p(y^{*}|x^{*},D)=\int p(y^{*}|x^{*},\theta)\ p(\theta|D)\
d\theta.$ (4)

However, the integration in (4) is in general intractable to compute exactly.
Instead one can approximate it by the Monte Carlo estimation. If we have a
finite number ($S$) of samples from the posterior, then the posterior
predictive distribution can be approximated as:

$\displaystyle
p(y^{*}|x^{*},D)\approx\frac{1}{S}\sum_{i=1}^{S}p(y^{*}|x^{*},\theta^{(i)}),\
\ \textrm{where}\ \ \theta^{(i)}\\!\sim\\!p(\theta|D),\ \ i=1,\dots,S.$ (5)

#### BayesDLL Usages (Pseudocodes)

The above two steps are implemented in our BayesDLL. For the four inference
methods to be described in the next section, we highlight the pseudocodes in
Fig. 1, which shows how to use BayesDLL to do posterior inference and test
prediction.

Figure 1: BayesDLL usage pseudocodes for the four inference methods. For each
method, we have pseudocodes showing how to perform posterior inference
(training) and how to obtain a test predictive distribution (test evaluation).

##  2 Approximate Inference Algorithms

###  2.1 Variational Inference (aka Bayes-by-Backprop [1])

In the variational inference we typically adopt the following Gaussian222
Perhaps this assumption/restriction of the tractable density family is one of
the main caveats of the variational inference. On the other hand, the MCMC
algorithms (e.g., SGLD in Sec. 2.3) do not require such an assumption, thus
being highly flexible. The only requirement for SGLD is that we can easily
(e.g., analytically) compute the gradients of the log-prior $\log p(\theta)$
and the log-likelihood $\log p(y|x,\theta)$ with respect to $\theta$. Sec. 2.3
for details.  densities for both prior and variational posterior:

  * •

Prior: $p(\theta)=\mathcal{N}(\theta;\overline{\theta},\sigma^{2}I)$.

  * •

Variational posterior: $q(\theta)=\mathcal{N}(\theta;m,s^{2})$ where $s$ is
the vectorized standard deviations (of the same shape as $\theta$), embedded
in a diagonal matrix (and $s^{2}$ means elementwise squaring). Both $m$ and
$s$ are the variational parameters to be estimated. Due to the positivity
constraint for $s$, we consider positive linking $s=g(\tilde{s})$ where
$\tilde{s}$ is free (unconstrained) optimization variables, and $g(\cdot)$ can
be typically the exponential function ($s=\exp(\tilde{s})$), the soft-plus
function ($s=\log(1+\exp(\tilde{s}))$), or a simple hinge function
($s=\max(\tilde{s},s_{\min})$ where $s_{\min}$ is a small positive constant
such as $10^{-8}$).

The negative ELBO loss function, in the data size normalized and the unbiased
minibatch stochastic estimate version, can be written as (here, $B$ denotes a
minibatch):

$\displaystyle\textrm{loss}\ =\ \frac{1}{|B|}\sum_{(x,y)\in
B}l(x,y;\theta\\!=\\!m\\!+\\!\epsilon\\!\odot\\!s)\ +\
\frac{1}{|D|}\sum_{i}\frac{1}{2}\bigg{(}\log\frac{\sigma^{2}}{s_{i}^{2}}-1+\frac{s_{i}^{2}}{\sigma^{2}}+\frac{(m_{i}-\overline{\theta}_{i})^{2}}{\sigma^{2}}\bigg{)},$
(6)

where $\epsilon\sim\mathcal{N}(0,I)$ and $\odot$ is elementwise product. The
loss gradient can be easily derived using the chain rule:

$\displaystyle\frac{\partial\textrm{loss}}{\partial m}\ $ $\displaystyle=\
\frac{1}{|B|}\sum_{(x,y)\in B}\frac{\partial l(x,y;\theta)}{\partial\theta}\
+\ \frac{1}{\sigma^{2}|D|}(m-\overline{\theta}),$ (7)
$\displaystyle\frac{\partial\textrm{loss}}{\partial\tilde{s}}\ $
$\displaystyle=\ \frac{1}{|B|}\sum_{(x,y)\in B}\frac{\partial
l(x,y;\theta)}{\partial\theta}\odot\epsilon\odot s\ +\
\frac{1}{|D|}\bigg{(}\frac{s^{2}}{\sigma^{2}}-1\bigg{)},$ (8)

where in this case we assumed the exponential positive linking function
($s=\exp(\tilde{s})$).

Once $m$ and $s$ (that is, $\tilde{s}$) are learned, at test time we can
sample $\theta\\!\sim\\!q(\theta)\\!=\\!\mathcal{N}(\theta;m,s^{2})$. If we
consider $S$ samples, then the posterior predictive distribution becomes:

$\displaystyle p(y|x,D)=\frac{1}{S}\sum_{i=1}^{S}p(y|x,\theta^{(i)}),\ \
\textrm{where}\ \ \theta^{(i)}\\!\sim\\!\mathcal{N}(\theta;m,s^{2}),\ \
i=1,\dots,S.$ (9)

###  2.2 MC-Dropout

In this section we describe our formulation for the MC-Dropout approximate
inference algorithm. This is slightly different from the original version [6]
in the following aspects: 1) We allow the Gaussian prior mean is either set to
be $0$ (original version) if no prior information is available, or set to be
some known values $\overline{\theta}$ to incorporate the prior knowledge
(typically pre-trained network parameters); 2) Whereas the original version
dropouts the inputs to layers, we dropout the network parameters instead;
Since the former usually requires modification of the network definition codes
in order to insert dropout layers, the main advantage of the parameter dropout
is that the code modification is not necessary; 3) In the original version the
bias parameters take Gaussian posteriors, being different from the mixture of
two spiky Gaussian posteriors for the weight parameters; We consider both the
Gaussian posterior and the spiky mixture posterior for bias parameters, which
is offered as an option for users to select; 4) Moreover, sometimes it is
conventional practice not imposing prior for the bias parameters at all, and
we incorporate this option as well for the sake of user’s convenience. Now we
discuss the detailed formulations for our implementation.

  * •

Prior: $p(\theta)=\mathcal{N}(\theta;\overline{\theta},\sigma^{2}I)$.

  * •

Variational posterior:
$q(\theta)=\prod_{i}\big{(}(1\\!-\\!p)\cdot\mathcal{N}(\theta_{i};m_{i},\epsilon^{2})+p\cdot\mathcal{N}(\theta_{i};\overline{\theta}_{i},\epsilon^{2})\big{)}$,
where $\epsilon$ is negligibly small (making two components spiky),
$p\in[0,1]$ corresponds to the dropout probability, and $m$ (of the same
shape/size as $\theta$) is the only variational parameters to be estimated.

  * •

Bias options: The bias parameters, denoted by $\theta_{b}$, can take different
prior and/or variational posterior distributions depending on user’s option
choice. The first option is not imposing prior on $\theta_{b}$ at all, more
precisely imposing the uninformative prior $p(\theta_{b})\\!\propto\\!1$, in
which case we set $q(\theta_{b})$ as a delta function, or virtually equivalent
to $q(\theta_{b})=\mathcal{N}(\theta_{b};m_{b},\epsilon^{2}I)$, and the
consequence is that in the loss function (negative ELBO) we can simply ignore
the corresponding KL term; The second option is to place the Gaussian prior
$p(\theta_{b})=\mathcal{N}(\overline{\theta}_{b},\sigma^{2}I)$ and Gaussian
posterior $q(\theta_{b})=\mathcal{N}(\theta_{b};m_{b},\epsilon^{2}I)$, which
is exactly the option taken by the original MC-Dropout [6]. This can be
implemented by treating the dropout probability $p$ separately for biases
(denoted by $p_{b}$) and non-biases (denoted by $p$), and setting $p_{b}=0$;
And of course the last (default) option is to treat biases in the same way as
weights, in which case we use exactly the above prior and variational
posterior.

The negative ELBO loss function, in the data size normalized and the unbiased
minibatch stochastic estimate version, is comprised of the expected negative
log-likelihood (ENLL) $\frac{1}{|B|}\mathbb{E}_{q}[-\log p(B|\theta)]$ and the
KL term $\frac{1}{|D|}\textrm{KL}(q(\theta)||p(\theta))$, where $B$ and $D$
are the minibatch and the whole training set, respectively. The ENLL term is
estimated by Monte Carlo, using the reparametrization trick. We first sample
keep-or-dropout binary indicators $z$ (of the same shape/size as $\theta$),
specifically $z_{i}\sim\textrm{Bernoulli}(1\\!-\\!p)$, where $z_{i}\\!=\\!1$
means no-dropout of $\theta_{i}$ and $z_{i}\\!=\\!0$ implies dropout. Then the
reparametrized sample is $\theta=z\odot m+(1\\!-\\!z)\odot\overline{\theta}$.
The KL term (between the mixture of Gaussians $q$ and the Gaussian $p$) can be
approximated by the same technique as [6]. The final loss function (to be
minimized over the variational parameters $m$) is as follows:

$\displaystyle\textrm{loss}\ =\ \frac{1}{|B|}\sum_{(x,y)\in
B}l(x,y;\theta=z\odot m+(1\\!-\\!z)\odot\overline{\theta})\ +\
\frac{1}{\sigma^{2}|D|}\frac{1\\!-\\!p}{2}||m-\overline{\theta}||_{2}^{2}.$
(10)

Note that when there is no prior information on $\theta$, that is,
$\overline{\theta}\\!=\\!0$, (10) reduces to the original MC-Dropout. The loss
gradient333We may not need the loss gradient explicitly if one implements it
using the parameter-level comp-graph build-up and backprop such as the higher
library. We do not utilize this library in our current version (as of August
2023). can be easily derived using the chain rule:

$\displaystyle\frac{\partial\textrm{loss}}{\partial m}\ =\
\frac{1}{|B|}\sum_{(x,y)\in B}\frac{\partial
l(x,y;\theta)}{\partial\theta}\odot z\ +\
\frac{1\\!-\\!p}{\sigma^{2}|D|}(m-\overline{\theta}),$ (11)

where $\frac{1}{|B|}\sum_{(x,y)\in B}\frac{\partial
l(x,y;\theta)}{\partial\theta}$ can be easily computed by a backprop call
provided in most auto-gradient deep learning libraries (e.g., PyTorch or
Tensorflow).

Once $m$ is learned, at test time we can sample $\theta\sim q(\theta)$ from
the mixture density. Although this amounts to doing similar dropout sampling
used in the ENLL estimation, we often ignore dropout and use the Gaussian
sampling. Due to negligible $\epsilon$, we have a deterministic sample
$\theta=m$.

###  2.3 SG-MCMC (SGLD)

In the stochastic-gradient MCMC (SG-MCMC) approach [15, 5, 2], we can collect
posterior samples by running a certain stochastic dynamic model whose
stationary distribution coincides with the posterior distribution (3). The
stochastic-gradient Langevin dynamic method (SGLD) [15] forms a Langevin
dynamic model, which amounts to running the following recurrence to collect
posterior samples (after some burn-in steps):

$\displaystyle\theta\ \leftarrow\ \theta+\frac{\eta}{2}\nabla\bigg{(}\log
p(\theta)+\frac{|D|}{|B|}\log p(B|\theta)\bigg{)}+\epsilon\sqrt{\eta}$ (12)

where $B$ ($\subset\\!D$) is a minibatch, $\eta$ is small step size, and
$\epsilon\\!\sim\\!\mathcal{N}(0,I)$. Note that in the parentheses subject to
the derivative, the first log-prior term admits closed-form gradient while the
gradient of the second term can be computed by the conventional SGD backprop.
Thus each step in (12) is as efficient as the vanilla SGD step.

After a burn-in period, we can maintain those samples $\theta$ to approximate
the posterior $p(\theta|D)$. For instance, the running average of the $\theta$
samples, denoted by $\hat{\theta}$, is a good estimate of the mean of the
posterior $p(\theta|D)$. In the ideal case, we can save all available samples
from the posterior (i.e., the iterates from (12)), however, due to the large
number of parameters in $\theta$, this would easily incur a computational
challenge. To this end, in our current implementation444Alternatively, perhaps
more expressive solution might be to estimate a mixture-of-Gaussians density
model to fit the posterior samples. We leave this implementation as our future
work. we estimate/maintain the sample means and variances from the posterior
samples (via running estimation); and at test time the (approximate) posterior
samples are taken from the Gaussian fitted with these sample means and
variances.

###  2.4 Laplace Approximation

The Laplace approximation essentially approximates the log-posterior $\log
p(\theta|D)$ by the second-order Taylor polynomial at the maximum-a-posteriori
(MAP) estimate $\theta^{*}$. More specifically, we first obtain the MAP
estimate $\theta^{*}$ by solving the following optimization problem, typically
using the SGD:

$\displaystyle\theta^{*}=\arg\max_{\theta}\ \log
p(\theta|D)=\arg\max_{\theta}\ \log p(\theta)+\log p(D|\theta).$ (13)

Then we approximate $\log p(\theta|D)$ by the quadratic Taylor polynomial at
$\theta=\theta^{*}$, which is simplified as follows due to the vanishing
gradient at the (local) optimum (i.e., $\nabla\log p(\theta^{*}|D)=0$):

$\displaystyle\log p(\theta|D)\ \approx\
\frac{1}{2}(\theta-\theta^{*})\nabla^{2}\log
p(\theta^{*}|D)(\theta-\theta^{*})+\textrm{const.}$ (14)

Assuming that the Hessian $\nabla^{2}\log p(\theta^{*}|D)$ is negative
definite555If not, one can perform the Generalised Gauss-Newton (GGN)
approximation for the Hessian [14, 7, 9]. However, we omit this step for
simplicity, and our diagonal empirical Fisher approximation in (19) implicitly
handles this potential issue. , the equation (14) essentially leads to the
Gaussian posterior approximation,

$\displaystyle p(\theta|D)\ \approx\
\mathcal{N}\big{(}\theta;\theta^{*},-(\nabla^{2}\log
p(\theta^{*}|D))^{-1}\big{)}.$ (15)

Here arises the infamous computational challenge in Hessian evaluation and
inversion from (15). First, to circumvent the $O(d^{2})$ memory overhead for
saving the Hessian matrix and prohibitive $O(d^{3})$ matrix inversion time,
where $d=\dim(\theta)$, we consider the diagonal Hessian approximation.
Secondly, to deal with the overhead of the Hessian computation, we adopt the
famous empirical Fisher information approximation for the Hessian. For
concreteness, we here derive the details of the diagonal empirical Fisher
approximation. Letting the training data $D=\\{(x_{n},y_{n})\\}_{n=1}^{N}$,

$\displaystyle\nabla^{2}\log p(\theta^{*}|D)=\nabla^{2}\log
p(\theta^{*})+\sum_{n=1}^{N}\nabla^{2}\log p(y_{n}|x_{n},\theta^{*})$ (16)

The second term in the RHS of (16) can be approximated by the empirical Fisher
information as (17), which is essentially obtained by replacing the model
distribution $p(y|x,\theta)$ in the Fisher information by the plug-in estimate
or the empirical distribution
$\frac{1}{N}\sum_{n=1}^{N}\delta(y\\!-\\!y_{n}|x_{n})$:

$\displaystyle\nabla^{2}\log p(\theta^{*}|D)\approx\nabla^{2}\log
p(\theta^{*})-\sum_{n=1}^{N}\nabla\log p(y_{n}|x_{n},\theta^{*})\nabla\log
p(y_{n}|x_{n},\theta^{*})^{\top}.$ (17)

Now, we further approximate the dyads by the diagonal matrix (i.e., element-
wise squaring instead of outer product), leading to:

$\displaystyle\nabla^{2}\log p(\theta^{*}|D)\approx\nabla^{2}\log
p(\theta^{*})-\sum_{n=1}^{N}\textrm{Diag}\Big{(}\nabla\log
p(y_{n}|x_{n},\theta^{*})^{2}\Big{)},$ (18)

where the squaring in (18) is element-wise, and $\textrm{Diag}(v)$ is the
diagonal matrix with the vector $v$ embedded in the diagonal entries. Lastly,
assuming isotropic Gaussian prior
$p(\theta)=\mathcal{N}(\theta;\overline{\theta},\sigma^{2}I)$, we have the
final posterior approximation:

$\displaystyle p(\theta|D)\ \approx\
\prod_{i}\mathcal{N}(\theta_{i};\theta^{*}_{i},v_{i})\ \ \textrm{where}\ \
v_{i}=\frac{1}{\frac{1}{\sigma^{2}}+\sum_{n=1}^{N}\big{[}\nabla\log
p(y_{n}|x_{n},\theta^{*})\big{]}_{i}^{2}}.$ (19)

Now (19) can be computed with one forward-pass for each data instance, and all
operations are done in $O(d)$ time/memory.

Although there exist other Hessian approximation strategies, notably the block
diagonal approximation schemes such as the Kronecker factorization [13], they
can potentially introduce considerable computational overhead compared to the
diagonal one, which often hinders their applications to the large-scale
networks such as Vision Transformers. For this reason we omit the
implementation of these methods in our library.

##  3 Uncertainty Quantification

One of the key benefits of using Bayesian deep models is its capability of
capturing uncertainty in their predictive distributions. There are two popular
types of methods to quantify/measure how well the uncertainty is captured in
the underlying models: error calibration and negative log-likelihood.

###  3.1 Error Calibration

The popular error calibration metrics such as ECE and MCE [8] as well as the
visualization tool like the Reliability plot [3, 10] belong to this category.

The key idea is to measure how well the prediction accuracy and the prediction
confidence are aligned. Most approaches rely on metric evaluation based on
confidence binning. More specifically, assume that we have class predictions
by the model $p(y=j|x^{i})$ for $i=1,\dots,N$ and $j=1,\dots,K$ where $K$ is
the class cardinality. Let $y^{\textrm{true}}(x)$ be the ground-truth class
label. We consider bin size $M$ (bin index $m=0,1,\dots,M\\!-\\!1$).

  * •

Initialize: $binsize[0:M]=acc[0:M]=conf[0:M]=0$

  * •

For $i=1,\dots,N$ and $j=1,\dots,K$:

  * \- Determine $m=$ Bin ID that $p(y=j|x^{i})$ belongs to

  * \- $binsize[m]\leftarrow binsize[m]+1$

  * \- $acc[m]\leftarrow acc[m]+I(y^{\textrm{true}}(x_{i})=j)$

  * \- $conf[m]\leftarrow conf[m]+p(y=j|x^{i})$

It would be desirable to have a prediction model that leads to $acc[m]\approx
conf[m]$ for all confidence level $m=0,1,\dots,M-1$. There are several ways to
visualize or quantify this goodness of alignment.

Reliability plot is just a simple plot of $acc[0:M]$ (Y-axis) vs. $conf[0:M]$
or bin centers (X-axis). Thus in the ideal case (0 calibration error), this
plot would coincide with $Y=X$ line.

ECE and MCE can be computed by the following formulas:

ECE $\displaystyle=\ \frac{1}{M}\sum_{m=0}^{M-1}\frac{binsize[m]}{N\cdot
K}\cdot\Big{|}acc[m]-conf[m]\Big{|}$ (20) MCE $\displaystyle=\ \arg\max_{0\leq
m<M}\Big{|}acc[m]-conf[m]\Big{|}$ (21)

Temperature scaling: We typically have a logit vector $f(x)\in\mathbb{R}^{K}$
as an output of the neural network, before soft-max normalizing it to
$p(y|x)$. We consider the (temperature) scaling of this logit before soft-max,
that is,

$\displaystyle
p_{T}(y|x)=\frac{e^{f_{y}(x)/T}}{\sum_{j=1}^{K}e^{f_{j}(x)/T}}.$ (22)

Obviously $T\\!=\\!1$ is the default setting, but one can find the best $T$
that minimizes the calibration error. To this end, by regarding $T$ as an
optimization parameter, we typically form a maximum likelihood estimation
problem on the validation set. More specifically,

$\displaystyle\min_{T}\ \mathbb{E}_{(x,y)\sim D_{V}}[-\log p_{T}(y|x)],$ (23)

where $D_{V}$ is the validation data set. Once the optimal $T$ is found, we
can report the temperature-scaled calibration error metrics with $p_{T}(y|x)$.

Our BayesDLL library can produce reliability plots and ECE/MCE metrics during
model training. See Fig. 2 for the examples.

(a) Reliability plot ($T=1$) (b) Reliability plot ($T^{*}$) (c) Temp. scaling
optim. curve

Figure 2: Our BayesDLL library can produce reliability plots during model
training. (a) Reliability plot with default temperature $T\\!=\\!1$, (b)
Reliability plot after temperature scaling, and (c) Temperature scaling
optimisation learning curve. In the reliability plots we also show ECE, MCE,
and NLL metrics on the lower-right corner.

###  3.2 Negative Log-Likelihood

The negative log-likelihood (NLL) on the test data set is the standard
statistical metric that measures how close the model’s predictive distribution
is to the true labeling distribution. It can be computed as follows:

$\displaystyle\textrm{NLL}=\mathbb{E}_{(x,y)\sim D_{T}}[-\log p(y|x)],$ (24)

where $D_{T}$ is the test data set.

##  4 Implementation Notes

Our current library implements four different Bayesian deep learning methods
as well as the baseline deterministic (non-Bayesian) method. Which method is
used can be specified by the flag \--method. For instance, one can add
\--method mc_dropout flag in the command line. Within each method, we also
list the method-specific hyperparameters.

  * •

"vanilla": This is a vanilla deterministic deep learning, aka SGD (stochastic
gradient descent) learning. We also allow weight decay to L2-penalise
deviation from the pre-trained parameters or zero parameters, as well as the
bias option for the L2 penalty.

    * –

wd (eg, 1e-4): The weight decay (L2 regularisation) coefficient. The L2
penalty is measured based on either deviation from the pre-trained parameters
or from 0.

    * –

bias $\\{$"penalty","ignore"$\\}$: How to treat the bias parameters in L2
penalty. "penalty" specifies the same treatment as weight parameters, while
"ignore" simply ignores the bias deviation (analogous to uninformative bias
prior in Bayesian methods).

  * •

"vi": This is the variational inference method. The related hyperparameters
are as follows:

    * –

prior_sig (eg, 0.01): This specifies the standard deviation $\sigma$ of the
prior Gaussian distribution.

    * –

bias $\\{$"informative","uninformative"$\\}$: How to treat the bias parameters
in the prior. "informative" specifies the same treatment as weight parameters,
while "uninformative" simply adopts $p(\theta_{b})\propto 1$. This amounts to
dropping the KL terms for the bias parameters.

    * –

kld (eg, 0.1): The discount factor for the KL term. The KL term is multiplied
by this factor. This is related to the training data size inflation due to
data augmentation.

    * –

nst (eg, 5): The number of posterior samples at test time (i.e., $S$ in (9)).

  * •

"mc_dropout": This is the MC-Dropout, and the related hyperparameters are as
follows:

    * –

prior_sig (eg, 0.01): This specifies the standard deviation $\sigma$ of the
prior Gaussian distribution.

    * –

p_drop (eg, 0.1): This specifies the dropout probability.

    * –

bias $\\{$"gaussian","spikymix", "ignore"$\\}$: How to treat the bias
parameters in the prior. "gaussian" takes Gaussian $q(\theta_{b})$, thus no
dropout for bias parameters; "spikymix" specifies the same treatment as weight
parameters; while "ignore" simply ignores prior and posterior for bias
parameters.

    * –

kld (eg, 0.1): The discount factor for the KL term. The KL term is multiplied
by this factor. This is related to the training data size inflation due to
data augmentation.

    * –

nst (eg, 5): This is the number of posterior samples to be sampled at test
time (i.e., $S$ in (9)).

  * •

"sgld": This is the SGLD whose related hyperparameters are as follows:

    * –

prior_sig (eg, 0.01): This specifies the standard deviation $\sigma$ of the
prior Gaussian distribution.

    * –

Ninflate (eg, 1e3): Data inflation factor (due to data augmentation). The
training data size $|D|$ is inflated by this factor.

    * –

nd (eg, 0.1): Noise discount factor. The noise term in the SGLD iteration is
multiplied by this factor.

    * –

burnin (eg, 20): Burn-in period (in epochs).

    * –

thin (eg, 10): Thinning steps (in batch iterations).

    * –

bias $\\{$"informative","uninformative"$\\}$: How to treat the bias parameters
in the prior. "informative" specifies the same treatment as weight parameters,
while "uninformative" simply adopts $p(\theta_{b})\propto 1$. This amounts to
dropping the prior term in the SGLD iteration.

    * –

nst (eg, 5): The number of posterior samples to be sampled at test time (i.e.,
$S$ in (9)). Recall that in the current version we use a sample-estimated
Gaussian for the posterior approximation. Thus this is the number of Gaussian
samples.

  * •

"la": This is the Laplace approximation, and the related hyperparameters are
as follows:

    * –

prior_sig (eg, 0.01): This specifies the standard deviation $\sigma$ of the
prior Gaussian distribution.

    * –

Ninflate (eg, 1e3): Data inflation factor (due to data augmentation). The
training data size $|D|$ is inflated by this factor.

    * –

bias $\\{$"informative","uninformative"$\\}$: How to treat the bias parameters
in the prior. "informative" specifies the same treatment as weight parameters,
while "uninformative" simply adopts $p(\theta_{b})\propto 1$. This amounts to
dropping the prior terms for bias parameters in the MAP objective.

    * –

nst (eg, 5): The number of posterior samples at test time (i.e., $S$ in (9)).

##  5 Experiments

In this section we demonstrate the execution results of training and testing
with our Bayesian neural network library. In Sec. 5.1 we provide extensive
comparison among various running options and hyperparameters, especially the
number of posterior samples during test prediction (nst), how to treat the
bias parameters (bias, e.g., either not imposing a prior or treating the same
way as weight parameters), the choice of prior scale (prior_sig), and so on.
In Sec. 5.2 we test our library on large-scale neural networks for vision
tasks, in particular, ResNet-101 and Vision Transformer (ViT) models. For
these models, we will show that learning the models from scratch (i.e.,
uninformative zero-mean Gaussian prior $p(\theta)$) often fails. Instead, we
impose a prior that is centered at the pre-trained model parameters that are
available publicly, which leads to prediction performance comparable to
conventional warm-start/fine-tuning deterministic model learning, but with
better uncertainty calibration.

###  5.1 Testing Various Options/Hyperparameters with MLP on MNIST

Experimental setup. The original MNIST training dataset is randomly split into
$50\%$ training and $50\%$ validation sets, where the latter is used to
determinne early stopping of training iterations. The neural network we
adopted is a fully-connect network (aka, MLP), which has three hidden layers
with 1000 units, followed by the final linear prediction head. For the
nonlinearity the ReLU activation is used. For all competing methods, we set
the maximum training epochs as 100, learning rate $10^{-2}$, batch size 128,
SGD optimizer with momentum 0.5. All training starts with randomly initialized
model parameters.

Competing approaches. Vanilla is the non-Bayesian deterministic SGD learning,
where optionally we can impose the L2 regularization via the weight-decay
option (either wd=0 or wd=$10^{-4}$). VI stands for variational inference. We
take as default hyperparameters the prior scale $\sigma=1.0$ (except
$\sigma=0.01$ for Laplace) and the KL discount factor $10^{-3}$. For MC-
Dropout, we use the default dropout probability $0.1$ while the prior scale
and the KL discount factor have the same default values as VI. Recalling from
Sec. 2.2, there are three bias treatment options, which are abbreviated as: ga
(Gaussian prior, also conforming to the original version), sm (the spiky
mixture prior, thus the same treatment as weight parameters), and
uninformative prior. In SGLD, we take the burn-in steps for the first 5
epochs, followed by thinning at every 10 batch iterations. The training data
inflation factor is set to $10^{3}$. Lastly, for Laplace approximation, we use
the same inflation factor, but the prior scale is set to $\sigma=0.01$ since
having larger scale (e.g., $\sigma=1.0$) led to failure in all cases.

Table 1: MNIST test prediction errors ($\%$). For the Bayesian neural network models, we mainly vary the bias treatment option, either imposing informative bias prior (the same treatment as weight parameters) or uninformative (essentially, not imposing prior on the bias parameters). The other hyperparameters are roughly default values (See text for details), and the test-time posterior sampling is not done, but the posterior mean is used.  | Vanilla | VI | MC-Dropout | SGLD | Laplace
---|---|---|---|---|---
wd=0 | bias=1 | bias=0 | bias=1 | bias=0 | bias=ga | bisa=sm | bias=0 | bias=1 | bias=0 | bias=1 | bias=0
Error ($\%$) | 2.52 | 2.55 | 2.50 | 2.25 | 2.43 | 2.47 | 2.47 | 2.48 | 2.74 | 2.73 | 2.41 | 2.53

Results on prediction errors. As shown in Table 1, all methods perform equally
well whereas SGLD slightly falls short. Overall the different bias treatment
options have little impact on the prediction performance.

Impact of the number of posterior samples at test prediction. In the Bayesian
neural networks, one can incorporate the prediction uncertainty by
marginalizing over the posterior samples, which is in practice typically done
by Monte-Carlo averaging such as (9) for VI and similarly for other methods.
This is known to improve the uncertainty calibration, i.e., consistency
between prediction confidence and accuracy. We check this property by
comparing two different settings, the number of posterior samples nst$=0$
(using the posterior mean) and nst$=5$. As shown in Table 2, increasing the
number of posterior samples in test prediction leads to reduction in ECE and
MCE metrics, indicating that the models are better calibrated.

Table 2: Impact of the number of posterior samples at test prediction in MNIST. The number of posterior samples nst=0 amounts to using the posterior mean.  | Vanilla | VI | MC-Dropout | SGLD | Laplace
---|---|---|---|---|---
wd=0 |  wd=$10^{-4}$ | nst=0 | nst=5 | nst=0 | nst=5 | nst=0 | nst=5 | nst=0 | nst=5
Error ($\%$) | 2.52 | 2.55 | 2.25 | 2.75 | 2.47 | 2.43 | 2.74 | 2.75 | 2.41 | 1.51
ECE ($\%$) | 0.22 | 0.20 | 0.12 | 0.11 | 0.32 | 0.18 | 0.23 | 0.21 | 0.24 | 0.12
MCE ($\%$) | 22.04 | 17.33 | 14.64 | 11.71 | 21.83 | 11.19 | 13.76 | 13.56 | 17.82 | 8.04
NLL ($\times 10^{-2}$) | 9.54 | 9.38 | 8.11 | 8.57 | 11.28 | 8.75 | 10.00 | 9.27 | 9.45 | 9.36

Impact of prior scale ($\sigma$). We test how the Bayesian models behave when
we change the prior scale. From the default value $\sigma=1.0$, we reduce it
to $\sigma=0.01$. As the results in Table 3, the prediction errors barely
change, but there are slight improvement in the uncertainty calibration
scores. This may be attributed to the stronger regularisation effect, where
further deviation from the prior mean $0$ weight parameters is penalised more
severely.

Table 3: Impact of the prior scale ($\sigma$) in MNIST. In the table entries, “$a\\!\to\\!b$” indicates that score $a$ with $\sigma=1.0$ is changed to score $b$ with $\sigma=0.01$.  | VI | MC-Dropout | SGLD
---|---|---|---
nst=0 | nst=5 | nst=0 | nst=5 | nst=0 | nst=5
Error ($\%$) | $2.25\\!\to\\!2.26$ | $2.75\\!\to\\!2.63$ | $2.47\\!\to\\!2.42$ | $2.43\\!\to\\!2.52$ | $2.74\\!\to\\!2.70$ | $2.75\\!\to\\!2.70$
ECE ($\%$) | $0.12\\!\to\\!0.12$ | $0.11\\!\to\\!0.10$ | $0.32\\!\to\\!0.32$ | $0.18\\!\to\\!0.17$ | $0.23\\!\to\\!0.18$ | $0.21\\!\to\\!0.16$
MCE ($\%$) | $14.64\\!\to\\!11.50$ | $11.71\\!\to\\!10.50$ | $21.83\\!\to\\!21.72$ | $11.19\\!\to\\!22.27$ | $13.76\\!\to\\!11.95$ | $13.56\\!\to\\!12.56$
NLL ($\times 10^{-2}$) | $8.11\\!\to\\!7.75$ | $8.57\\!\to\\!8.44$ | $11.28\\!\to\\!11.24$ | $8.75\\!\to\\!8.50$ | $10.00\\!\to\\!9.49$ | $9.27\\!\to\\!9.42$

###  5.2 Large-Scale Backbones including Foundation Models

Next we test our library on the large-scale backbone networks. We consider two
popular deep networks for vision tasks: ResNet-101 and Vision Transformer
(ViT) specifically the version known as ViT-L-32, where the former consists of
about $43$ million parameters and the latter about $305$ million parameters.
For simplicity we consider the image classification vision tasks with the Pets
[12] and Flowers [11] datasets that contain images of 37 and 102 different
categories, respectively. For Pets, we randomly split the official training
data into $50\%$ training and $50\%$ validation sets. For Flowers, we merge
the official training and validation data splits, and randomly split them into
$50\%$ training and $50\%$ validation sets.

As it is widely believed that training such large-scale networks from the
scratch is very difficult and often leads to inferior solutions, we instead
adopt the pre-trained model weights in the form of prior mean parameters in
the Bayesian models. That is, instead of having $0$-mean prior (i.e.,
$\overline{\theta}=0$) as usual practice, we set the prior mean equal to the
pre-trained weights666We simply employ the network architecture definitions
and the network weights obtained from pre-training with the ImageNet subsets
[4], available at https://pytorch.org/vision/main/models.html. Note that this
feature of flexible external code incorporation, without any modification of
the original code, is one of the key benefits of the proposed library.. In our
quick experiments with the (non-Bayesian) SGD learning (denoted by Vanilla) in
Table 4, we can verify that there is huge performance difference between the
trained models with and without pre-trained weights, signifying that the use
of pre-trained weights is crucial for large-scale models.

Table 4: Comparison between trained (non-Bayesian) vanilla models with and without pre-trained weights on the Flowers dataset [11]. The weight decay hyperparameter is set to $10^{-4}$.  | ResNet-101 | ViT-L-32
---|---|---
From-scratch | Pre-trained-warm-start | From-scratch | Pre-trained-warm-start
Test error ($\%$) | 94.63 | 11.76 | 73.49 | 14.20

The overall test errors are shown in Table 5. For the detailed hyperparameters
used in this experiment, please refer to our code. We see that the variational
inference, MC-dropout, and SGLD models perform on par or better than
deterministic models. The Laplace approximation performs reliably well with
the posterior mean parameters (nst=0), but once we incorporate multiple
posterior samples (nst=5) the test accuracy dropped significantly. We still
investigate the precise reasons, but it might be due to the numerical issue in
the diagonal empirical Fisher information estimate (e.g., one may as well put
some larger regulariser in the denominator of (19) for better numerical
stability). Table 6 summarises the uncertainty quantification results.

Table 5: Test errors ($\%$) of Bayesian adaptation/fene-tuning on the Pets dataset [12].  | Vanilla | VI | MC-Dropout | SGLD | Laplace
---|---|---|---|---|---
wd=0 |  wd=$10^{-4}$ | nst=0 | nst=5 | nst=0 | nst=5 | nst=0 | nst=5 | nst=0 | nst=5
ResNet-101 | 10.03 | 10.03 | 10.03 | 9.27 | 10.03 | 9.65 | 9.21 | 9.24 | 10.19 | N/A
ViT-L-32 | 8.72 | 8.69 | 8.39 | 8.45 | 8.37 | 8.42 | 8.67 | 8.72 | 8.72 | N/A
Table 6: Uncertainty quantification of Bayesian foundation models on the Pets
dataset [12].

(a) ResNet-101
Vanilla VI MC-Dropout SGLD Laplace wd=0 wd=$10^{-4}$ nst=0 nst=5 nst=0 nst=5
nst=0 nst=5 nst=0 nst=5 ECE ($\%$) 0.16 0.16 0.18 0.11 0.19 0.09 0.13 0.14
0.16 N/A MCE ($\%$) 10.02 12.24 11.95 8.53 14.76 8.10 12.17 15.29 15.71 N/A
NLL ($\times 10^{-2}$) 33.73 33.59 33.53 31.04 33.38 31.68 31.70 31.40 33.79
N/A
(b) ViT-L-32
Vanilla VI MC-Dropout SGLD Laplace wd=0 wd=$10^{-4}$ nst=0 nst=5 nst=0 nst=5
nst=0 nst=5 nst=0 nst=5 ECE ($\%$) 0.04 0.04 0.05 0.03 0.07 0.05 0.04 0.05
0.04 N/A MCE ($\%$) 11.55 12.43 9.59 7.19 13.38 8.98 11.09 10.78 11.55 N/A NLL
($\times 10^{-2}$) 26.72 26.69 25.65 25.99 25.64 25.79 27.01 26.99 26.72 N/A

Computational overhead of Bayesian neural networks. One (often-believed)
obstacle that prevents the Bayesian neural networks from being widely applied
to large-scale foundation models in real world practice, is the computational
overhead – a sort of prejudice where one may well need to keep track of more
parameters than deterministic models with increased training time. To clarify
this, we actually compare the wall-clock training times and memory footprints
of the different Bayesian models against the deterministic vanilla SGD
training in Fig. 3. We use a RTX-2080Ti machine for ResNet-101 and Tesla A100
for ViT-L-32, with single GPUs for both cases. As shown, all Bayesian
approaches have tolerable overhead compared to base SGD models – the over head
is minor for SGLD and Laplace approximation; the worst-case overhead is at
most two times of base model’s complexity. These results imply that the
proposed Bayesian neural network library makes the Bayesianisation of large-
scale foundation models viable.

(a) Training time (b) GPU memory footprint

Figure 3: (a) (Per epoch) Training time (second) and (b) GPU memory footprint
for different Bayesian models on the Flowers dataset [11].

##  6 Conclusion

We provide full implementation, without relying on other libraries, and easy-
to-use demo codes for various Bayesian inference methods including:
variational inference, MC-dropout, stochastic-gradient Langevin dynamics, and
Laplace approximation. We also include the codes for evaluating Uncertainty
Quantification measures provided (eg, ECE, MCE, Reliability plots, Negative
log-likelihood), which can be used to report how well the uncertainty is
captured in new models. Although we have tested the library with ResNet-101
and ViT-L-32, our library can be ready to be applicable to other Foundation
Models such as LLAMA, RoBERTa, and Denoising Diffusion generative models
without code modification at all. We also demonstrate that our code incurs
minimal/acceptable use of extra computational resources (time and GPU memory).

## References

  * [1] Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra.  Weight Uncertainty in Neural Networks.  In International Conference on Machine Learning, 2015.
  * [2] T. Chen, E. B. Fox, and C. Guestrin.  Stochastic gradient Hamiltonian Monte Carlo.  International Conference on Machine Learning, 2014.
  * [3] Morris H. DeGroot and Stephen E. Fienberg.  The Comparison and Evaluation of Forecasters.  Journal of the Royal Statistical Society. Series D (The Statistician), 32(1/2):12–22, 1983.
  * [4] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  ImageNet: A large-scale hierarchical image database.  In IEEE Conference on Computer Vision and Pattern Recognition, 2009\.
  * [5] N. Ding, Y. Fang, R. Babbush, C. Chen, R. D. Skeel, and H. Neven.  Bayesian sampling using stochastic gradient thermostats.  In Advances in neural information processing systems, 2014.
  * [6] Yarin Gal and Zoubin Ghahramani.  Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning.  In International Conference on Machine Learning, 2016.
  * [7] A. Graves.  Practical variational inference for neural networks.  In Advances in Neural Information Processing Systems, 2011.
  * [8] Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger.  On Calibration of Modern Neural Networks.  In International Conference on Machine Learning, 2017.
  * [9] J. Martens.  New insights and perspectives on the natural gradient method.  arXiv preprint arXiv:1412.1193, 2014.
  * [10] Alexandru Niculescu-Mizil and Rich Caruana.  Predicting good probabilities with supervised learning.  In International Conference on Machine Learning, 2005.
  * [11] M-E. Nilsback and A. Zisserman.  Automated flower classification over a large number of classes.  In Proceedings of the Indian Conference on Computer Vision, Graphics and Image Processing, 2008.
  * [12] O. M. Parkhi, A. Vedaldi, A. Zisserman, and C. V. Jawahar.  Cats and Dogs.  In IEEE Conference on Computer Vision and Pattern Recognition, 2012\.
  * [13] Hippolyt Ritter, Aleksandar Botev, and David Barber.  A Scalable Laplace Approximation for Neural Networks.  In International Conference on Learning Representations, 2018.
  * [14] N. N. Schraudolph.  Fast curvature matrix-vector products for second-order gradient descent.  Neural computation, 14(7):1723–1738, 2002.
  * [15] Max Welling and Yee Whye Teh.  Bayesian Learning via Stochastic Gradient Langevin Dynamics.  In International Conference on Machine Learning, 2011.