# An Introduction to Bayesian Inference

### Agenda

* What is Bayesian inference ?
* Background
* Comparison with Frequentist approach
* Challenges in practical application
* Tools available
* Example-1
* Example-2
* Summary

## What is Bayesian inference ?

### Inference involves :

* Some **unknown quantities** about which we are interested in learning or testing. We call these parameters.
* Some **data** which have been observed, and hopefully contain information about these parameters.
* One or more **models** that relate the data to the parameters, and is the instrument that is used to learn.

### Bayesian Inference :

> Practical methods for making inferences from data using probability models for quantities we observe and about which we wish to learn.
*-- Gelman et al. 2013*

A Bayesian model is described by parameters, uncertainty in those parameters is described using probability distributions.

All conclusions from Bayesian statistical procedures are stated in terms of **probability statements**



## Background 

As the name suggests, Bayesian inference and Bayesian method was first thought by Thomas Bayes. He was analyzing a problem: **Can we use evidence about the natural world around us to make rational conclusion about the existence of God.**

And he came up with a principle (in ~1763):  
<div style="font-size: 120%;">
We modify our initial guess with objective new information to get a new and improved belief, which in turn carries a commitment to update that belief each time a new piece of information arrives.
</div>

Bayes' system was: **Initial Belief + New Data -> Improved Belief**

Laplace rederived it (in 1774) and called it **"Probability of Causes"**. We used this technique to calculate the mass of Saturn, amongst other things.

He came up with this formula:

\begin{equation} \Pr(M~|~D,I) = \dfrac{\Pr(D~|~M,I)\times\Pr(M ~|~I)}{\Pr(D~|~I)} \label{eqn:lapbayes} \end{equation} 

where $M$ is the mass of the Saturn ,  
       $D$ is data from various measurements of planetary orbits,    
 and $I$ is existing background information, e.g., Newton’s laws of celestial mechanics

His estimate is just **0.6%** off of the currently accepted value for  𝑀


This is basis of the formula that we studied in high school


\\[P(A|B) = \frac{P(B|A) * P(A)}{P(B)}\\]


* P(B|A) is a conditional probability: the probability of event B occurring given that A has occured.
* P(A|B) is also a conditional probability: the probability of event A occuring given that B has occured
* P(A), P(B) are the probabilities of A and B occuring


### Bayesian vs Frequentist Statistics: *What's the difference?*

### The Frequentist World View



- The **data** that have been observed are considered **random**, because they are realizations of random processes, and hence will vary each time one goes to observe the system.
- Model **parameters** are considered **fixed**. A parameter's true value is uknown and fixed, and so we *condition* on them.

In mathematical notation, this implies a (very) general model of the following form:

<div style="font-size:35px">
\\[f(y | \theta)\\]
</div>

Here, the model \\(f\\) accepts data values \\(y\\) as an argument, conditional on particular values of \\(\theta\\).

### The Bayesian World View



- Data are considered **fixed**. They used to be random, but once they were written into your lab notebook/spreadsheet/IPython notebook they do not change.
- Model parameters themselves may not be random, but Bayesians use probability distribtutions to describe their uncertainty in parameter values, and are therefore treated as **random**.

This implies the following form:

<div style="font-size:35px">
\\[p(\theta | y)\\]
</div>

This formulation used to be referred to as ***inverse probability***, because it infers from observations to parameters, or from effects to causes.

Bayesians do not seek new estimators for every estimation problem they encounter. There is only one estimator for Bayesian inference: **Bayes' Formula**.

## Bayes' Formula

Bayesian inference is based on Bayes Formula


The equation expresses how our belief about the value of \\(\theta\\), as expressed by the **prior distribution** \\(P(\theta)\\) is reallocated following the observation of the data \\(y\\).

The innocuous denominator \\(P(y)\\) usually cannot be computed directly, and is actually the expression in the numerator, integrated over all \\(\theta\\):

<div style="font-size: 150%;">  
\\[Pr(\theta|y) = \frac{Pr(y|\theta)Pr(\theta)}{\int Pr(y|\theta)Pr(\theta) d\theta}\\]
</div>

The intractability of this integral is one of the factors that has contributed to the under-utilization of Bayesian methods by statisticians.


### Priors

Once considered a controversial aspect of Bayesian analysis, the prior distribution characterizes what is known about an unknown quantity before observing the data from the present study. Thus, it represents the information state of that parameter. It can be used to reflect the information obtained in previous studies, to constrain the parameter to plausible values, or to represent the population of possible parameter values, of which the current study's parameter value can be considered a sample.

### Likelihood functions

The likelihood represents the information in the observed data, and is used to update prior distributions to posterior distributions. This updating of belief is justified becuase of the **likelihood principle**, which states:

> Following observation of \\(y\\), the likelihood \\(L(\theta|y)\\) contains all experimental information from \\(y\\) about the unknown \\(\theta\\).

Bayesian analysis satisfies the likelihood principle because the posterior distribution's dependence on the data is **only through the likelihood**. In comparison, most frequentist inference procedures violate the likelihood principle, because inference will depend on the design of the trial or experiment.

## Challenges

Even though Bayesian approach seems a very natual way of dealing with problems, it has historically had two main challenges

* Priors and the subjectivity associated with the Priors

* Calculating the normalizing constant

### Problem with the priors

Historically priors have been a problem. The main criticism for Bayesian inference and Priors has been

* They are subjective and we come up with them before seeing the data. Negates: 'Lets the data speak for itself'

* Probability as 'The degree of Uncertainty' was not the common view. Instead the frequentist view of probabilty as the 'Ratio of Occurences' held sway.


### Problem of computation


The innocuous denominator \\(P(y)\\) usually cannot be computed directly, and is actually the expression in the numerator, integrated over all \\(\theta\\).

<div style="font-size: 150%;">  
\\[{\int Pr(y|\theta)Pr(\theta) d\theta}\\]
</div>

If we have n parameters we need to integrated over n variables and it can be solved analytically and this prevented Bayesian Inference to be applied to any complex real work problem.

The solution for this was Sampling and fast computers




### Monte Carlo sampling techniques

What the method produces for us is an approximation of the posterior distribution, p(θ|D), in the form of a large number of θ values sampled from that distribution. This heap of representative θ values can be used to estimate the central tendency of the posterior, its highest density interval (HDI), etc. The posterior distribution is estimated by randomly generating a lot of values from it

The goal in Bayesian inference is to get an accurate representation of the posterior distribution. One way to do that is to sample a large number of representative points from the posterior. The question then becomes this: How can we sample a large number of representative values from a distribution?


#### Metropolis

The simplest sampling solution is the **Metropolis-Hastings** algorithm


One of the simplest and most flexible algorithms for generating reversible Markov chains is the Metropolis-Hastings algorithm. Since we cannot sample directly from the (unknown) posterior distribution, this algorithm employs an **auxilliary distribution** that is easy to sample from. These samples generate candidate state transitions, which are accepted or rejected probabilistically.




Consider a simple Metropolis-Hastings algorithm for a single parameter, $\theta$. We will use a well-known statistical distribution like Normal distribution to produce candidate variables $q_t(\theta^{\prime} | \theta)$. Each generated value, $\theta^{\prime}$, is a *possible* next value for
$\theta$ at step $t+1$. 

Whether or not $\theta^{\prime}$ is accepted depends on the relative probability of the new value versus the current value, weighted by the probabilities of the two values under the proposal distribution: 

$$a(\theta^{\prime},\theta) = \frac{q_t(\theta^{\prime} | \theta) \pi(\theta^{\prime})}{q_t(\theta | \theta^{\prime}) \pi(\theta)}$$

This is the **acceptance ratio**, and is used as a threshold value for a uniform random draw that determines acceptance:

$$
\theta^{(t+1)} = 
\begin{cases}
\theta^{\prime}  & \text{with prob. } p = \min(a(\theta^{\prime},\theta^{(t)}),1)\cr
\theta^{(t)}  & \text{with prob } 1 - p
\end{cases}
$$

This transition kernel implies that movement is not guaranteed at every step. It only occurs if the suggested transition is likely based on the acceptance ratio.

A single iteration of the Metropolis-Hastings algorithm proceeds as follows:

1.  Sample $\theta^{\prime}$ from $q(\theta^{\prime} | \theta^{(t)})$.

2.  Generate a Uniform[0,1] random variate $u$.

3.  If $a(\theta^{\prime},\theta) > u$ then
    $\theta^{(t+1)} = \theta^{\prime}$, otherwise
    $\theta^{(t+1)} = \theta^{(t)}$.

#### NUTS (No UTurns Sampler)

For a future session.

## Libraries 

There are a bunch of libraries available which can insolving problem using Bayesian inference.

* WinBugs  
* **Stan** and pyStan   (NOTE: TimeSeries libary Prophet from Facebook is based on Stan)
* Edward   
* **Pymc3**    
* Pymc4 (To be released)   

## Summary

Bayesian inference provides a powerful methods to solve problem with data, inference and prediction.

In our context, Bayesian tools can be effective for use cases where

* There is a large domain expert knowledge which can be used as prior
* There is a inflow for new labelled data which can be used to update the model to create new posterior
* There is a need to interpret and reason the model predictions
* Sufficient HW resources are avaiable. Bayesian tools are more comptutionally than classical ML algorithms but less so than Deep Neural Net

References: 

* [Doing Bayesian Data Analysis](http://www.r-5.org/files/books/computers/algo-list/statistics/data-mining/John_K_Kruschke-Doing_Bayesian_Data_Analysis-EN.pdf)

* [Introduction to Hamiltonian Monte Carlo](https://arxiv.org/pdf/1701.02434.pdf)

* [History of Bayes Theorem](https://www.lesswrong.com/posts/RTt59BtFLqQbsSiqd/a-history-of-bayes-theorem)

2f30a01d38f007db85f4bb0988d47cfd912f9dd9
ipynb
Jupyter Notebook
introduction-to-bayesian-inference.ipynb
praveen049/introduction-to-bayesian-methods
b2249ae83c9f8d9866da2dff25f16f9e608ba44d
MIT
introduction-to-bayesian-inference.ipynb
praveen049/introduction-to-bayesian-methods
b2249ae83c9f8d9866da2dff25f16f9e608ba44d
MIT
introduction-to-bayesian-inference.ipynb
praveen049/introduction-to-bayesian-methods
b2249ae83c9f8d9866da2dff25f16f9e608ba44d
MIT
Qwen/Qwen-72B
1. YES
2. YES