# Bayesian Statistics 

## Chapter Zero Review

Omission

## Chapter One Introduction 

### Conditional Probability 

As we have mentioned in `Using Python in Advanced Statistics`, **probability** is a number within in 0 to 1(included 0 and 1),and the meaning of probability is that the credibility(可信度) of a event or a prediction. According to this idea, **COnditional Probability** is the probability under some kinds of conditions. For example, if there is a rain tomorrow in Zhuhai. I check the data, found that in tomorrow's date, there are about 30% that will rain. **This is the probability**. If I want to know the possibility of rain tomorrow if it is already raining today. **This is the conditional probability**, which is a probability of an event happen based on a happened fact. 

The usual notation for conditional probability is $p(A\mid B)$, which means under B event's condition, the probability of A.

*PS: if the probability of an event is 0, we says it is **false**; in the opposite, if the probability is 1, it is **true**.*

### Conjoint Probability 

After known what is **conditional probability**, we can easily know what is **conjoint probability**, which is another way to say the probability that two things are true. This time, the notation is: $p(A B)$ or $P(A\ and \ B)$.


```python
import matplotlib.pyplot as plt
import numpy as np 
import sympy
from matplotlib_venn import venn2, venn3_circles

plt.figure(figsize=(4,4))
v = venn2(subsets=(2,2,1), set_labels = ('A', 'B'))



plt.title("Sample Venn diagram - plot one")
plt.annotate('P(AB)', xy=v.get_label_by_id('11').get_position() - np.array([0, 0.05]), xytext=(-70,-70),
             ha='center', textcoords='offset points', bbox=dict(boxstyle='round,pad=0.5', fc='gray', alpha=0.1),
             arrowprops=dict(arrowstyle='->', connectionstyle='arc3,rad=0.5',color='gray'))
plt.show()
```

The conjointed part can represent the conjoint probability that $A$ and $B$ are both true. For instance, if $A$ and $B$ are independent, we know that the $P(A\ and \ B) = P(A)P(B) \Leftrightarrow P(A \mid B) = p(A)$, for whether A is true or not will not affect B; but if $A$ and $B$ are not independent, just like the weather, if we know that it raind today, it is more likely that it will rain tomorrow, so $p(A|B) > p(A)$. 

In general, the probability of a conjunction is : $\mathbf  p(A\ and \ B) = p(A)p(B \mid A)$

### Bayes's theorem 

Known what is conditional probability and cojoint probability, we can move on to the Headlight: **Bayes's theorem**. First, we should make sure we are in the same channel: 
   1. conjoint probability: $p(A\ and \ B) = p(B\ and \ A)$ 
   2. probability of a conjunction: $p(A\ and \ B) = p(A)p(B \mid A)$
   3. interchange $A$ and $B$: $p(B\ and \ A) = p(B)p(A \mid B)$
   
In the light of point 2 and point 3, we can combine the two equations to one euqation: $\Rightarrow p(A) \ p(B \mid A) = p(B) \ p(A \mid B)$, divided by $p(B)$ in the meantime and we can get the **Bayes's theorem !** : $\Rightarrow \frac{p(A) \ p(B \mid A)}{p(B)} =  p(A \mid B)$

We can have a example, Allen B.Downey called it **The Cookie Problem**: Suppose there are two bowls of cookies. Bowl-1 contains $30$ vanilla cookies and $10$ chocolate cookies. Bowl-2 contains $20$ of each.
Now suppose you choose one of the bowls at random and, without looking, select a cookie at random. The cookie is vanilla. What is the probability that it came from Bowl-1?
This is a conditional probability; we want $p(Bowl-1|vanilla)$, but it is not obvious how to compute it. If I asked a different question — the probability of a vanilla cookie given Bowl-1 — it would be easy: 

$$p(vanilla\mid Bowl-1) = \frac{3}{4}$$

Sadly, $p(A|B)$ is not the same as $p(B|A)$, but there is a way to get from one
to the other: Bayes’s theorem.

Using **Bayes's Theorem**, we can easily solve the cookie problem: <br> 
First, assume that $B_1$ represent the probility that cookie from Bowl-1 and $p(V)$ is the probability of taking out a Vanilla. 

$$P(B_1\mid V) = \frac{p(B_1)p(V\mid B_1)}{p(V)}$$

There is another way to think of Bayes's theorem: it gives us a way to update the probability of a hypothesis, $H$, in light of some body of data,$D$. 

This way of thinking about Bayes's theorem is called the **diachronic interpretation**. "Diachronic" means that something is happening over time; in this case the probability of the hypotheses changes, over time, as we see new data. 

Rewriting Bayes's theorem with $H$ and $D$ yields:

$$P(H|D) = \frac{p(H)p(D|H)}{p(D)}$$

In this interpretation, each term has a name:

* $p(H)$ is the probability of the hypothesis before we see the data, called the prior probability, or just **prior**;
* $p(D)$ is the probability of the data under any hypothesis, called the normalizing constant(total probability);
* $P(H|D)$ is waht we want to compute, the probability of the hypothesis after we see the data, called the **posterior**;
* $P(D|H)$ is the probability of the data under the hypothesis, called the likelihood

In the classical Statistics theory, the data is the random variable. The the Cookie Problem for example, if we want to know the probability of taking out vanilla cookies from Bowl-1, the random variable is $1,2,3,\dots$, the data or the number. However, if we want to know the probability of which Bowls does the vanilla cookie come from, as we got a vanilla cookies already, this time, the random varaible is the probability, not the data. This is the biggest difference between Bayesian and Frequentist. 

## Chapter Two Computational Statistics 

In statistics, a **distribution** is a set of values and thier correponding probabilites. 

You can get more information about Probabilities and Statistics <a href="https://terenceliu98.github.io/Jupytersite//Html/Use_PY_in_Advanced_Statstics.html">here</a>

Here is a example of rolling a dice, as we all know that the probability of each side is $\frac{1}{6}$, then, this is a Discrete Uniform Distribution. 


```python
import numpy as np
from scipy import stats
import matplotlib.pyplot as plt
import matplotlib

fig, ax = plt.subplots(1, 1)
a = [1,2,3,4,5,6]
sample = np.random.randint(len(a), size = 10000)

dict = {}
for key in sample:
    dict[key] = dict.get(key, 0) + 1
len(dict)

x = [dict[i] for i in dict] # change dict to list 
pmf = [dict[i]/len(sample) for i in dict] #calculate pmf of each element in list 

ax.plot(a,pmf,'bo')
ax.vlines(a, 0, pmf, colors='b', lw=5, alpha=0.5)

line1 = [(0, 0.1666667), (10, 0.1666667)]
(line1_xs, line1_ys) = zip(*line1)
ax.add_line(matplotlib.lines.Line2D(line1_xs, line1_ys, linewidth=1, color='red') )        
plt.show()
```

## Chapter Three Naive Bayes 

Naive Bayes methods is the basic idea in both Naive Bayes Classification and Bayes Belief Network. It is totally an extension of the Bayes Theorm that we consider every pair of features given the value of the class variable are conditional independent.  Given classs vairable $y$ and dependent feature vector $x_1$ through $x_2$:

$$P(y|x_1, \dots, x_n) = \frac{P(y)P(x_1, \dots, x_n|y)}{P(x_1, \dots, x_n)}$$

Under this assumption we can rewrite the Bayes Theorm:

$$P(y|x_1, \dots, x_n) = \frac{P(y) \prod_{i=1}^n P(x_i |y)}{P(x_1, \dots, x_n)}$$

Since $P(x_1, \dots, x_n)$ is constant given the input, we can use the following classification rule:

$$P(y|x_1, \dots, x_n) \propto P(y) \prod_{i=1}^n P(x_i | y) \Rightarrow \hat{y} = arg \max \limits_{y} P(y) \prod_{i = 1}^n P(x_i | y)$$

and we can use Maximum A posterior estimation to estimate $P(y)$ and P(x_i | y)$; the former is then the relative frequency of class $y$ in the training set. 

### How to build a basic model using scikit-learn 

#### Gaussian

It is used in classificaiton and it assumes that feature follow a normal distribution (the likelihood function is assumed to be Normal)

$$P(x_i | y) = \frac{1}{\sqrt{2\pi}\sigma_y} exp( - \frac{(x_i - \mu_y)^2}{2 \sigma_y^2})$$

THe parameter $\sigma_y$ and $\mu_y$ are estimated using MLE.


```python
import numpy as np
import pandas as pd
from matplotlib import pyplot as plt
import matplotlib.colors as colors
import seaborn as sns
import itertools
from scipy.stats import norm
import scipy.stats
from sklearn.naive_bayes import GaussianNB
%matplotlib inline
sns.set()


# iris is a famous data set  and used in every data mining's reference book, including the sklearn's official doc

iris = sns.load_dataset("iris")
iris = iris.rename(index = str, columns = {'sepal_length':'1_sepal_length','sepal_width':'2_sepal_width', 'petal_length':'3_petal_length', 'petal_width':'4_petal_width'})

#Plot the scatter of sepal length vs sepal width
sns.FacetGrid(iris, hue="species", height=7) .map(plt.scatter,"1_sepal_length", "2_sepal_width", )  .add_legend()
plt.title('Scatter plot')

```


```python
from sklearn.naive_bayes import GaussianNB

#Setup X and y data
X_data = df1.iloc[:,0:2]
y_labels = df1.iloc[:,2].replace({'setosa':0,'versicolor':1,'virginica':2}).copy()

#Fit model
model_sk = GaussianNB(priors = None)
model_sk.fit(X_data,y_labels)


# Our 2-dimensional classifier will be over variables X and Y
N = 100
X = np.linspace(4, 8, N)
Y = np.linspace(1.5, 5, N)
X, Y = np.meshgrid(X, Y)

#fig = plt.figure(figsize = (10,10))
#ax = fig.gca()
color_list = ['Blues','Greens','Reds']
my_norm = colors.Normalize(vmin=-1.,vmax=1.)

g = sns.FacetGrid(iris, hue="species", height=10, palette = 'colorblind') .map(plt.scatter, "1_sepal_length", "2_sepal_width",)  .add_legend()
my_ax = g.ax


#Computing the predicted class function for each value on the grid
zz = np.array(  [model_sk.predict( [[xx,yy]])[0] for xx, yy in zip(np.ravel(X), np.ravel(Y)) ] )


#Reshaping the predicted class into the meshgrid shape
Z = zz.reshape(X.shape)


#Plot the filled and boundary contours
my_ax.contourf( X, Y, Z, 2, alpha = .1, colors = ('blue','green','red'))
my_ax.contour( X, Y, Z, 2, alpha = 1, colors = ('blue','green','red'))

# Addd axis and title
my_ax.set_xlabel('Sepal length')
my_ax.set_ylabel('Sepal width')
my_ax.set_title('Gaussian Naive Bayes decision boundaries')

plt.show()
```


```python

```

952f187a37c4bed5eb52e9ebedf1b6b715cf3a32
ipynb
Jupyter Notebook
Course File/Use PY in Bayes.ipynb
TerenceLiu98/Undergraduate-in-Statistics
e76efc789779ace123abcc2a0e94595168b444ac
MIT
2020-01-04T09:17:27.000Z
2022-03-30T16:52:44.000Z
Course File/Use PY in Bayes.ipynb
TerenceLiu98/Undergraduate-in-Statistics
e76efc789779ace123abcc2a0e94595168b444ac
MIT
2020-11-25T09:15:17.000Z
2020-11-25T09:15:17.000Z
Course File/Use PY in Bayes.ipynb
TerenceLiu98/Undergraduate-in-Statistics
e76efc789779ace123abcc2a0e94595168b444ac
MIT
2020-02-23T16:06:37.000Z
2020-11-22T07:23:12.000Z
Qwen/Qwen-72B
1. YES
2. YES