# Maximum Likelihood and Maximum A Posterior

* We looked at the regularization term as a *penalty* term in the objective function.  There is another way to interpret the regularization term as well.  Specifically, there is a *Bayesian* interpretation. 

\begin{eqnarray}
\min E^{\ast}(\mathbf{w}) &=& \max -E^{\ast}(\mathbf{w})\\
& =& \max \exp \left\{ -E^{\ast}(\mathbf{w})\right\}\\
&=& \max \exp \left\{ -\frac{1}{2}\sum_{n=1}^N \left( y(x_n, \mathbf{w}) - t_n \right)^2 - \frac{\lambda}{2}\left\| \mathbf{w} \right\|^2_2 \right\}\\
&=& \max \exp \left\{ -\frac{1}{2}\sum_{n=1}^N \left( y(x_n, \mathbf{w}) - t_n \right)^2 \right\}\exp\left\{-\frac{1}{2}\lambda\left\| \mathbf{w} \right\|^2_2\right\}\\
&=& \max \prod_{n=1}^N \exp \left\{ -\frac{1}{2} \left( y(x_n, \mathbf{w}) - t_n \right)^2 \right\}\exp\left\{-\frac{1}{2}\lambda\left\| \mathbf{w} \right\|^2_2\right\}
\end{eqnarray}

* So, this is a maximization of the *data likelihood* with a *prior*: $p(\mathbf{X}|\mathbf{w})p(\mathbf{w})$

* *Method of Maximum Likelihood:*
    * A *data likelihood* is how likely the data is given the parameter set
    * So, if we want to maximize how likely the data is to have come from the model we fit, we should find the parameters that maximize the likelihood
    * A common trick to maximizing the likelihood is to maximize the log likelihood.  Often makes the math much easier.  *Why can we maximize the log likelihood instead of the likelihood and still get the same answer?*
    * Consider: $\max \ln \exp \left\{ -\frac{1}{2}\left(y(x_n, \mathbf{w}) - t_n\right)^2\right\}$ We go back to our original objective. 


* *Method of Maximum A Posteriori (MAP):*
    * Bayes Rule: $p(Y|X) = \frac{p(X|Y)p(Y)}{p(X)}$
    * Consider: $p(\mathbf{w}|\mathscr{D}) = \frac{p(\mathscr{D}|\mathbf{w})p(\mathbf{w})}{p(\mathscr{D})}$, i.e., posterior $\propto$ likelihood $\times$ prior

## Maximum Likelihood vs. Maximum A Posteriori (MAP)

* Lets look at this in terms of binary variables, e.g., Flipping a coin:  $X =1$ is heads, $X=0$ is tails
* Let $\mu$ be the probability of heads.  If we know $\mu$, then: $P(x = 1 |\mu) = \mu$ and $P(x = 0|\mu) = 1-\mu$
\begin{eqnarray}
P(x|\mu) = \mu^x(1-\mu)^{1-x} = \left\{\begin{array}{c c}\mu & \text{ if } x=1 \\ 1-\mu & \text{ if } x = 0 \end{array}\right.
\end{eqnarray}

* This is called the *Bernoulli* distribution.  The mean and variance of a Bernoulli distribution is: 
\begin{equation}
E[x] = \mu
\end{equation}
\begin{equation}
E\left[(x-\mu)^2\right] = \mu(1-\mu)
\end{equation}
* So, suppose we conducted many Bernoulli trials (e.g., coin flips) and we want to estimate $\mu$

### Method: Maximum Likelihood
\begin{eqnarray}
p(\mathscr{D}|\mu) &=& \prod_{n=1}^N p(x_n|\mu) \\
&=& \prod_{n=1}^N \mu^{x_n}(1-\mu)^{1-x_n}
\end{eqnarray}

* Maximize : (*What trick should we use?*)
\begin{eqnarray}
\mathscr{L} = \sum_{n=1}^N x_n \ln \mu + (1-x_n)\ln(1-\mu)
\end{eqnarray}

\begin{eqnarray}
\frac{\partial \mathscr{L}}{\partial \mu} =  0 &=& \frac{1}{\mu}\sum_{n=1}^N x_n - \frac{1}{1-\mu }\sum_{n=1}^N (1 - x_n)\\
0 &=& \frac{(1-\mu) \sum_{n=1}^N x_n - \mu \sum_{n=1}^N (1- x_n)}{\mu(1-\mu)}\\
0 &=& \sum_{n=1}^N x_n - \mu \sum_{n=1}^N x_n - \mu \sum_{n=1}^N 1 + \mu \sum_{n=1}^N x_n\\
0 &=& \sum_{n=1}^N x_n - \mu N\\
\mu &=& \frac{1}{N}\sum_{n=1}^N x_n = \frac{m}{N}
\end{eqnarray}
where $m$ is the number of successful trials. 

* So, if we flip a coin 1 time and get heads, then $\mu = 1$ and probability of getting tails is 0.  *Would you believe that? We need a prior!*

### Method: Maximum A Posteriori: 

* Look at several independent trials.  Consider N = 3 and m = 2 (N is number of trials, m is number of successes) and look at all ways to get 2 H and 1 T: 
     * H H T  $\rightarrow \mu \mu (1-\mu) = \mu^2(1-\mu)$
     * H T H  $\rightarrow \mu  (1-\mu) \mu  = \mu^2(1-\mu)$
     * T H H $\rightarrow (1-\mu) \mu   \mu  = \mu^2(1-\mu)$

* $\left(\begin{array}{c} 3 \\ 2 \end{array}\right) \mu^2(1-\mu) \rightarrow \left(\begin{array}{c} N \\ m \end{array}\right) \mu^m(1-\mu)^{N-m} = \frac{N!}{(N-m)!m!}\mu^m(1-\mu)^{N-m} $
* This is the Binomial Distribution, gives the probability of $m$ observations of $x=1$ out of N independent trails
* So, what we saw is that we need a prior.  We want to incorporate our prior belief. Let us place a prior on $\mu$
\begin{equation}
Beta(\mu|a,b) = \frac{\Gamma(a + b)}{\Gamma(a)\Gamma(b)}\mu^{a-1}(1-\mu)^{b-1}
\end{equation}
\begin{equation}
E[\mu] = \frac{a}{a + b}
\end{equation}
\begin{equation}
Var[\mu] = \frac{ab}{(a+b)^2(a+b+1)}
\end{equation}
* Note: $\Gamma(x) = \int_0^\infty u^{x-1}e^{-u} du$ and when $x$ is an integer, then it simplifies to $(x-1)!$
* Calculation of the posterior, Take $N = m + l$ observations:
\begin{eqnarray}
p(\mu | m, l, a, b) &\propto& Bin(m,l|\mu)Beta(\mu|a,b) \\
&\propto& \mu^m(1-\mu)^l\mu^{a-1}(1-\mu)^{b-1}\\
&=& \mu^{m+a-1}(1-\mu)^{l+b-1}
\end{eqnarray}
* What does this look like?  Beta: $a \leftarrow m+a$, $b \leftarrow l+b$
* So, what's the posterior? 
\begin{equation}
p(\mu | m, l, a, b) = \frac{\Gamma(m+a+l+b)}{\Gamma(m+a)\Gamma(l+b)}\mu^{m+a-1}(1-\mu)^{l+b-1}
\end{equation}
* *Conjugate Prior Relationship:* When the posterior is the same form as the prior
* Now we can maximize the (log of the) posterior: 
\begin{eqnarray}
\max_\mu ((m+a-1) \ln \mu + (l+b-1) \ln (1-\mu))
\end{eqnarray}
\begin{eqnarray}
\frac{\partial \mathscr{L}}{\partial \mu} =  0&=& \frac{m + a -1}{\mu} - \frac{l + b - 1}{1-\mu}\\
&=& (1-\mu)(m+a-1) - \mu(l+b-1)\\
&=& (m+a-1) - \mu(m+a-1) - \mu(l+b-1)\\
\mu &=& \frac{m+a-1}{m+a+l+b-2}
\end{eqnarray}
* This is the MAP solution.  *So, what happens now when you flip one heads, two heads, etc.?*
* Discuss online updating of the prior.  Eventually the data takes over the prior. 



```python
import numpy as np
import matplotlib.pyplot as plt
import math 
%matplotlib inline   

priorA = 2
priorB = 2
def plotBeta(a=priorA,b=priorB):
	'''plotBeta(a=1,b=1): Plot plot beta distribution with parameters a and b'''
	xrange = np.arange(0,1,0.001)  #get equally spaced points in the xrange
	normconst = math.gamma(a+b)/(math.gamma(a)*math.gamma(b))
	beta = normconst*xrange**(a-1)*(1-xrange)**(b-1)
	fig = plt.figure()
	p1 = plt.plot(xrange,beta, 'g')
	plt.show()


#Beta Distribution
# plotBeta(priorA,priorB);

trueMu = 0.5
numFlips = 100
flipResult = []
for flip in range(numFlips):
    flipResult.append(np.random.binomial(1,trueMu,1)[0])
    print(flipResult)
    print('Frequentist/Maximum Likelihood Probability of Heads:' + str(sum(flipResult)/len(flipResult)))
    print('Bayesian/MAP Probability of Heads:' + str((sum(flipResult)+priorA-1)/(len(flipResult)+priorA+priorB-2)))
    if (input("Hit enter to continue, or q to quit...\n") == "q"):
        print("quitting...\n")
        break
```

## The Gaussian Distribution:
* Consider a univariate Gaussian distribution:
\begin{equation}
\mathscr{N}(x|\mu, \sigma^2) = \frac{1}{\sqrt{2\pi \sigma^2}}\exp\left\{ -\frac{1}{2}\frac{(x-\mu)^2}{\sigma^2} \right\}
\end{equation}
* $\sigma^2$ is the variance OR $\frac{1}{\sigma^2}$ is the *precision*
* So, as $\lambda$ gets big, variance gets smaller/tighter.  As $\lambda$ gets small, variance gets larger/wider.
* The Gaussian distribution is also called the *Normal* distribution. 
* We will often write $N(x|\mu, \sigma^2)$ to refer to a Gaussian with mean $\mu$ and variance $\sigma^2$.
* *What is the multi-variate Gaussian distribution?* 

* What is the expected value of $x$ for the Gaussian distribution?
\begin{eqnarray}
E[x] &=& \int x p(x) dx \\
     &=& \int x \frac{1}{\sqrt{2\pi \sigma^2}}\exp\left\{ -\frac{1}{2}\frac{(x-\mu)^2}{\sigma^2} \right\} dx
\end{eqnarray}
* *Change of variables:*  Let
\begin{eqnarray}
y &=& \frac{x-\mu}{\sigma} \rightarrow x = \sigma y + \mu\\
dy &=& \frac{1}{\sigma} dx \rightarrow dx = \sigma dy
\end{eqnarray}
* Plugging this into the expectation: 
\begin{eqnarray}
E[x] &=& \int \left(\sigma y + \mu  \right)\frac{1}{\sqrt{2\pi}\sigma} \exp\left\{ - \frac{1}{2} y^2 \right\} \sigma dy \\
&=& \int \frac{\sigma y}{\sqrt{2\pi}} \exp\left\{ - \frac{1}{2} y^2 \right\} dy + \int \frac{\mu}{\sqrt{2\pi}} \exp\left\{ - \frac{1}{2} y^2 \right\} dy 
\end{eqnarray}
* The first term is an odd function: $f(-y) = -f(y)$  So, $E[x] = 0 + \mu = \mu$



## MLE of Mean of Gaussian

*  Let $\mathbf{x}_1, \mathbf{x}_2, \ldots, \mathbf{x}_N$ be samples from a multi-variance Normal distribution with known covariance matrix and an unknown mean.  Given this data, obtain the ML estimate of the mean vector. 
	\begin{equation}
	p(\mathbf{x}_k| {{\mu}}) = \frac{1}{(2\pi)^{\frac{l}{2}}\left| \Sigma \right|^{\frac{1}{2}}}\exp\left( -\frac{1}{2}(\mathbf{x}_k - {{\mu}})^T\Sigma^{-1}(\mathbf{x}_k - {\mu})\right)
	\end{equation}
* We can define our likelihood given the $N$ data points.  We are assuming these data points are drawn independently but from an identical distribution (i.i.d.):
	\begin{equation}
	\prod_{n=1}^N p(\mathbf{x}_n| {{\mu}}) = \prod_{n=1}^N \frac{1}{(2\pi)^{\frac{l}{2}}\left| \Sigma \right|^{\frac{1}{2}}}\exp\left( -\frac{1}{2}(\mathbf{x}_n - {{\mu}})^T\Sigma^{-1}(\mathbf{x}_n - {\mu})\right)
	\end{equation}
*  We can apply our "trick" to simplify
	\begin{eqnarray}
	\mathscr{L} &=& \ln \prod_{n=1}^N p(\mathbf{x}_n| {{\mu}}) = \ln \prod_{n=1}^N \frac{1}{(2\pi)^{\frac{l}{2}}\left| \Sigma \right|^{\frac{1}{2}}}\exp\left( -\frac{1}{2}(\mathbf{x}_n - {{\mu}})^T\Sigma^{-1}(\mathbf{x}_n - {\mu})\right)\\
	&=& \sum_{n=1}^N  \ln \frac{1}{(2\pi)^{\frac{l}{2}}\left| \Sigma \right|^{\frac{1}{2}}}\exp\left( -\frac{1}{2}(\mathbf{x}_n - {{\mu}})^T\Sigma^{-1}(\mathbf{x}_n - {\mu})\right)\\
	&=& \sum_{n=1}^N  \left( \ln \frac{1}{(2\pi)^{\frac{l}{2}}\left| \Sigma \right|^{\frac{1}{2}}} + \left( -\frac{1}{2}(\mathbf{x}_n - {{\mu}})^T\Sigma^{-1}(\mathbf{x}_n - {\mu})\right) \right) \\
	&=&  - N \ln (2\pi)^{\frac{l}{2}}\left| \Sigma \right|^{\frac{1}{2}} + \sum_{n=1}^N  \left( -\frac{1}{2}(\mathbf{x}_n - {{\mu}})^T\Sigma^{-1}(\mathbf{x}_n - {\mu}) \right) 
	\end{eqnarray}
* Now, lets maximize:
	\begin{eqnarray}
	\frac{\partial \mathscr{L}}{\partial \mu} &=& \frac{\partial}{\partial \mu} \left[- N \ln (2\pi)^{\frac{l}{2}}\left| \Sigma \right|^{\frac{1}{2}} + \sum_{n=1}^N  \left( -\frac{1}{2}(\mathbf{x}_n - {{\mu}})^T\Sigma^{-1}(\mathbf{x}_n - {\mu}) \right)\right] = 0 \\
	&\rightarrow& \sum_{n=1}^N  \Sigma^{-1}(\mathbf{x}_n - {\mu}) = 0\\
	&\rightarrow& \sum_{n=1}^N  \Sigma^{-1}\mathbf{x}_n  = \sum_{n=1}^N  \Sigma^{-1} {\mu}\\
	&\rightarrow& \Sigma^{-1} \sum_{n=1}^N \mathbf{x}_n  = \Sigma^{-1} {\mu} N\\
	&\rightarrow& \sum_{n=1}^N \mathbf{x}_n  = {\mu} N\\
	&\rightarrow& \frac{\sum_{n=1}^N \mathbf{x}_n}{N} = {\mu}\\
	\end{eqnarray}
* So, the ML estimate of $\mu$ is the sample mean!



## MAP of Mean of Gaussian

* To get a MAP estimate of the mean of a Gaussian, we apply a prior distribution and maximize the posterior. 
* Lets use a Gaussian prior on the mean (because it has a *conjugate prior* relationship)

\begin{eqnarray}
p(\mu|X, \mu_0, \sigma_0^2, \sigma^2) &\propto& \mathscr{N}(X|\mu, \sigma^2)\mathscr{N}(\mu|\mu_0, \sigma_0^2)\\
&=& \prod_{n=1}^N \left(\frac{1}{\sqrt{2\pi \sigma^2}} \exp\left\{-\frac{1}{2\sigma^2}\left(x_n-\mu\right)^2 \right\}\right)\frac{1}{\sqrt{2\pi \sigma_0^2}} \exp\left\{-\frac{1}{2\sigma_0^2}\left(\mu-\mu_0\right)^2 \right\}\nonumber\\
\mathscr{L} &=& -\frac{N}{2}\ln(2\pi\sigma^2) - \frac{1}{2\sigma^2}\sum_{n=1}^N(x_n-\mu)^2 - \frac{N}{2}\ln(2\pi \sigma_0^2) - \frac{N}{2\sigma_0^2}(\mu - \mu_0)^2\\
\frac{\partial \mathscr{L}}{\partial \mu} &=&  - \frac{N}{\sigma^2}\mu - \frac{N}{\sigma_0^2}\mu + \frac{N}{\sigma_0^2}\mu_0 + \frac{1}{\sigma^2}\sum_{n=1}^N x_n = 0\\
N\mu\left(\frac{\sigma_0^2 + \sigma^2}{\sigma^2\sigma_0^2} \right) &=& \frac{1}{\sigma^2}\sum_{n=1}^N x_n + \frac{1}{\sigma_0^2}\mu_0 \\
\mu_{MAP} &=& \frac{\sigma_0^2}{N\sigma_0^2 + N\sigma^2}\sum_{n=1}^N x_n + \frac{\mu_0\sigma^2}{N\sigma_0^2 + N\sigma^2}
\end{eqnarray}

* *Does this result make sense?*


```python

```

51afa2e5a623849a9110ce83444bbf8413801365
ipynb
Jupyter Notebook
05_MLandMAP/ML and MAP.ipynb
zhengyul9/lecture
905b93ba713f8467887fe8de5a44a3d8a7cae45c
CC-BY-4.0 MIT
05_MLandMAP/ML and MAP.ipynb
zhengyul9/lecture
905b93ba713f8467887fe8de5a44a3d8a7cae45c
CC-BY-4.0 MIT
05_MLandMAP/ML and MAP.ipynb
zhengyul9/lecture
905b93ba713f8467887fe8de5a44a3d8a7cae45c
CC-BY-4.0 MIT
Qwen/Qwen-72B
1. YES
2. YES