# Mixed Dirichlet


A mixed Dirichlet random variable $Y$ takes on values in the probability simplex $\Delta_{K-1}$, an assignment $Y=y$ has probability density given by 

\begin{align}
P_{Y}(y|\alpha, w) &= \sum_{f} \mathrm{Gibbs}(f|w) \times \mathrm{Dirichlet}(y|\alpha \odot f)
\end{align}

where $w \in \mathbb R^K$, $\alpha \in \mathbb R^K_{>0}$, $f$ is one of the non-empty faces of the simplex,  by $\alpha \odot f$ we mean the sub-vector of $\alpha$ whose coordinates are associated with the vertices in $f$. 

The distribution over proper faces has probability mass function:
\begin{align}
\mathrm{Gibbs}(f|w) = \frac{\exp(w^\top \phi(f))}{\sum_{f'} \exp(w^\top \phi(f'))}
\end{align}
where $\phi(f) \in \mathbb {-1, 1}^K$ is such that $\phi_k(f) = 1$ if the vertex $\mathbf e_k$ is in the face, and $-1$, otherwise. 



```python
import probabll.distributions as pd
```


```python
import torch
import torch.distributions as td
```


```python
import matplotlib.pyplot as plt
```


```python
def plot_marginals(samples, bins=100):
    D = samples.shape[-1]
    fig, ax = plt.subplots(D, 1, figsize=(4, 2*D), sharex=True)
    for d in range(D):
        _ = ax[d].hist(samples[...,d].flatten().numpy(), bins=bins, density=True)
    return fig, ax
```

# Uniform F and Uniform Y|f


```python
p3d = pd.MixedDirichlet(concentration=torch.ones(3), scores=torch.zeros(3))
```


```python
p3d.sample([10])
```


```python
p3d.entropy(), p3d.cross_entropy(p3d), td.kl_divergence(p3d, p3d)
```


```python
_ = plot_marginals(p3d.sample([1000]), bins=100)
```


```python
_p = p3d.expand([2, 1])
_p.sample().shape
```


```python
_p.entropy(), _p.cross_entropy(_p), td.kl_divergence(_p, _p)
```


```python
_p.faces.cross_entropy(_p.faces).shape
```

# Max-Ent F and Uniform Y|f


```python
pm3d = pd.MixedDirichlet(concentration=torch.ones(3), pmf_n=pd.MaxEntropyFaces.pmf_n(3, 1))
```


```python
_ = plot_marginals(pm3d.sample(torch.Size([1000])), bins=100)
```


```python
pm3d.entropy(), pm3d.cross_entropy(pm3d), td.kl_divergence(pm3d, pm3d)
```


```python
_pm = pm3d.expand([2, 1])
_pm.sample().shape
```


```python
_pm.entropy(), _pm.cross_entropy(_pm), td.kl_divergence(_pm, _pm)
```

# VI


```python
p = pd.MixedDirichlet(concentration=torch.ones(5), pmf_n=pd.MaxEntropyFaces.pmf_n(5, 1))
q = pd.MixedDirichlet(concentration=torch.ones(5)/10, scores=torch.zeros(5))
```


```python
p.batch_shape, p.event_shape
```


```python
p.sample(torch.Size([10]))
```


```python
f = p.faces.enumerate_support()
```


```python
p.faces.log_prob(f).exp(), f.sum(-1)
```


```python
p.cross_entropy(q)
```


```python
p.Y(f).cross_entropy(q.Y(f))
```


```python
p.Y(f).entropy()
```


```python
p.cross_entropy(q)
```


```python
td.kl_divergence(p, q)
```


```python
td.kl_divergence(p.faces, q.faces)
```


```python

```


```python

```

a0adde9ef2f2edba47a809ddd81b9b035863043c
ipynb
Jupyter Notebook
notebooks/Mixed-Dirichlet.ipynb
illc-uva/dists.pt
ba6dfabedc95d0d701255d540f0c2db2f0dae734
MIT
notebooks/Mixed-Dirichlet.ipynb
illc-uva/dists.pt
ba6dfabedc95d0d701255d540f0c2db2f0dae734
MIT
notebooks/Mixed-Dirichlet.ipynb
illc-uva/dists.pt
ba6dfabedc95d0d701255d540f0c2db2f0dae734
MIT
Qwen/Qwen-72B
1. YES
2. YES