$$ \LaTeX \text{ command declarations here.}
\newcommand{\R}{\mathbb{R}}
\renewcommand{\vec}[1]{\mathbf{#1}}
\newcommand{\X}{\mathcal{X}}
\newcommand{\D}{\mathcal{D}}
\newcommand{\G}{\mathcal{G}}
\newcommand{\Parents}{\mathrm{Parents}}
\newcommand{\NonDesc}{\mathrm{NonDesc}}
\newcommand{\I}{\mathcal{I}}
$$

## Exponential Family Distributions

**DEF:** $p(x | \theta)$ has **exponential family form** if:
$$
\begin{align}
p(x | \theta)
&= \frac{1}{Z(\theta)} h(x) \exp\left[ \eta(\theta)^T \phi(x) \right] \\
&= h(x) \exp\left[ \eta(\theta)^T \phi(x) - A(\theta) \right]
\end{align}
$$


- $Z(\theta)$ is the **partition function** for normalization
- $A(\theta) = \log Z(\theta)$ is the **log partition function**
- $\phi(x) \in \R^d$ is a vector of **sufficient statistics**
- $\eta(\theta)$ maps $\theta$ to a set of **natural parameters**
- $h(x)$ is a scaling constant, usually $h(x)=1$

### Problem:  Bernoulli in exponential family form

As mentioned in lecture, the *Bernoulli* distribution can be described in expontial family form:
$$
\begin{align}
\mathrm{Ber}(x | \mu)
&= \mu^x (1-\mu)^{1-x} \\ 
&= (1-\mu) \exp\left[ x \log\frac{\mu}{1-\mu} \right]
\end{align}
$$

This gives **natural parameters** $\eta = \log \frac{\mu}{1-\mu}$.

**Problem**: What is the log partition function $A(\eta)$?

## Problem: Guassian Distribution in exp family form

Recall that the gaussian distribution can be written as
$$p(x \mid \mu, \sigma^2) = \frac{1}{\sqrt{2 \pi \sigma^2}} \exp\left(-\frac{(\mu-x)^2}{2 \sigma^2} \right)$$

**Problem**: Write this in exponential family form. Questions:
1. What is $\phi(x)$?
1. What is the parameterization $\eta(\mu,\sigma^2)$?
1. What is the log-partition $A(\eta)$?

## Magic of the log partition function

Derivatives of the **log-partition function** $A(\theta)$ yield **cumulants** of the sufficient statistics (assume $h(x)=1$ and $\eta(\theta) = \theta$):
- $\nabla_\theta A(\theta) = E_{x}[\phi(x)]$
- $\nabla^2_\theta A(\theta) = \text{Cov}[ \phi(x) ]$

**Problem**: Prove the first statement using calculus.

*Hint*: While this fact isn't always true, in this case you can use the following:
$$\nabla_\theta \left( \int f(x,\theta) dx \right) =  \int \nabla_\theta f(x,\theta) dx$$

## Problem: Conditional independence in PGMs



**Which of the following are true?**:
1. $P(C,B|A) = P(C|A) P(B |A)$
2. $P(C,B) = P(C) P(B)$

Prove your answer!

## Problem: Conditional independence in PGMs



**Which of the following are true?**:
1. $P(C,B|A) = P(C|B,A)$
2. $P(C,B) = P(C) P(B)$

Prove your answer!

## Bayesian Networks

Let's consider a bayesian network representing different factors that determine whether you make it to class on time:

<a href="https://www.youtube.com/watch?v=_Tr8KRqyGJk">

</a>




### Exercise: express joint distribution in terms of product of factors

Express the joint distribution of this network using the chain rule.

P(W,R,T,S,B,C) = ...

*your answer goes here*


```python
%%html
<style>
div.probs table {
    float:left;
    margin-left: 1em;
    margin-top: 1em;
}

div.probs br.clear {
    clear:both;
}
</style>

```


<style>
div.probs table {
    float:left;
    margin-left: 1em;
    margin-top: 1em;
}

div.probs br.clear {
    clear:both;
}
</style>


### Computing Probabilities

Assuming the (conditional) distributions are as follows:

<div class=probs>

<table>
<tr>
<th>$w^0$</th>
<th>$w^1$</th>
</tr>
<tr>
<td>0.2</td>
<td>0.8</td>
</tr>
</table>

<table>
<tr>
<th>$r^0$</th>
<th>$r^1$</th>
</tr>
<tr>
<td>0.68</td>
<td>0.32</td>
</tr>
</table>

<table>
<tr>
<th>$t^0$</th>
<th>$t^1$</th>
</tr>
<tr>
<td>0.85</td>
<td>0.15</td>
</tr>
</table>

<br class="clear">

<table>
<tr>
<th></th>
<th>$s^0$</th>
<th>$s^1$</th>
</tr>
<tr>
<td>$w^0$, $r^0$</td><td>0.4</td><td>0.6</td>
</tr>
<tr>
<td>$w^0$, $r^1$</td><td>0.9</td><td>0.1</td>
</tr>
<tr>
<td>$w^1$, $r^0$</td><td>0.05</td><td>0.95</td>
</tr>
<tr>
<td>$w^1$, $r^1$</td><td>0.3</td><td>0.7</td>
</tr>
</table>

<table>
<tr>
<th><th>$b^0$</th><th>$b^1$</th>
</tr>
<tr>
<td>$t^0$</td><td>0.15</td><td>0.85</td>
</tr>
<tr>
<td>$t^1$</td><td>0.7<td>0.3</td>
</tr>
</table>

<br class="clear">

<table>
<tr>
<th></th>
<th>$c^0$</th>
<th>$c^1$</th>
</tr>
<tr>
<td>$s^0$, $b^0$</td><td>0.4</td><td>0.6</td>
</tr>
<tr>
<td>$s^0$, $b^1$</td><td>0.9</td><td>0.1</td>
</tr>
<tr>
<td>$s^1$, $b^0$</td><td>0.05</td><td>0.95</td>
</tr>
<tr>
<td>$s^1$, $b^1$</td><td>0.1</td><td>0.9</td>
</tr>
</table>

</div>

What is the probability you wake up on time, your roomate takes a long shower, there's bad traffic, you and the bus are both late to the bus stop and you make it to class on time?

*your answer goes here*

### Exercise: Marginalizatioin

What is the probability that you get to the bus stop on time given you wake up on time?

*your answer goes here*

### Exercise: v-structure

What is the probability that you get to the bus stop on time given that you make it to class on time? $P(S|C)$

What is the probability that you get to the bus stop on time given that you make it to class on time and bus arrives on time? $P(S|C,B)$


```python

```

858f95761dae4c6164ee7fcb5c3b62efbd82ca51
ipynb
Jupyter Notebook
handsOn_lecture15_exp_families_bayesian_networks/handsOn_lecture15_exp_families_bayesian_networks.ipynb
xipengwang/umich-eecs445-f16
298407af9fd417c1b6daa6127b17cb2c34c2c772
MIT
2016-09-11T23:15:35.000Z
2022-02-22T08:03:24.000Z
handsOn_lecture15_exp_families_bayesian_networks/handsOn_lecture15_exp_families_bayesian_networks.ipynb
eecs445-f16/umich-eecs445-f16
298407af9fd417c1b6daa6127b17cb2c34c2c772
MIT
handsOn_lecture15_exp_families_bayesian_networks/handsOn_lecture15_exp_families_bayesian_networks.ipynb
eecs445-f16/umich-eecs445-f16
298407af9fd417c1b6daa6127b17cb2c34c2c772
MIT
2016-09-12T20:50:46.000Z
2022-01-03T14:41:23.000Z
Qwen/Qwen-72B
1. YES
2. YES