## bayespropestimation basis

The bayespropestimation module uses simple Bayesian probability to derive the posterior distributions of two samples. The parameters of these samples can then be approximated by simulation, as can the difference in the parameters.

#### Sections

- Deriving the posterior distribution
- Comparing samples
- Choice of priors
- Estimating the Bayes Factor

#### Deriving the posterior distribution

Let $Y_k$ be the count of successes from $n_k$ trials from population $k$.  $Y_k$ can be defined as:

$$Y_k \sim \textrm{Bi}(n_k, \theta_k)$$

Where $\theta_k$ is the parameter we wish to estimate.  $\theta_k$ can be defined as:

$$\theta_k \sim \textrm{Beta}(\alpha, \beta)$$

Where $\alpha$ and $\beta$ denote fixed parameters of the beta distribution.  Using Bayes theorem $\Big( p(B|A) = \frac{p(A|B)p(B)}{p(A)}\Big)$ it can be seen that:

\begin{equation}
\begin{aligned}
p(\theta_k | Y_k) &= \frac{L(Y_k | \theta_k)p(\theta_k)}{p(Y_k)}
\\
p(\theta_k | Y_k) &\propto L(Y_k | \theta_k)p(\theta_k)
\\
&\propto \Big( {n_k \choose y_k} \theta_k^{y_k} (1 - \theta_k)^{(n_k - y_k)} \Big) \Big(\frac{\theta^{(\alpha -1)}(1 - \theta_k)^{(\beta - 1)}}{\mathbb{B}(\alpha, \beta)}\Big)
\\
&\propto \theta^{(y_k + \alpha -1)}_k (1 - \theta_k)^{(n_k - y_k + \beta - 1)}
\\
&\propto \frac{\theta^{(y_k + \alpha -1)}(1 - \theta_k)^{(n_k - y_k + \beta  - 1)}}{\mathbb{B}(y_k + \alpha, n_k - y_k + \beta)}
\end{aligned}
\end{equation}

Which is identical to the beta distribution with parameters $y_k + \alpha$ and $n_k - y_k + \beta$.  Therefore:

$$ \theta_k | Y_k \sim \textrm{Beta}(y_k + \alpha, n_k - y_k + \beta)$$


#### Comparing samples

For two samples from populations $A$ and $B$, the posterior distributions can be defined as:

$$ \theta_A | Y_A \sim \textrm{Beta}(y_A + \alpha, n_A - y_A + \beta)$$

$$ \theta_B | Y_B \sim \textrm{Beta}(y_B + \alpha, n_B - y_B + \beta)$$

Credible intervals, MAP and expectations of $\theta_A | Y_A$ and $\theta_B | Y_B$ can be derived directly from the posterior density, or they can be approximated by random draws from the posterior density.

Let $\Delta_{AB} = \theta_B$ - $\theta_A$, i.e. the difference between the parameter estimates for sample $A$ and sample $B$.  $\Delta_{AB}$ can be defined as:

$$\theta_B - \theta_A = \Delta_{AB}  \propto \textrm{Beta}(y_A + \alpha, n_A - y_A + \beta) - \textrm{Beta}(y_B + \alpha, n_B - y_B + \beta)$$

This distribution is proper if a normalising constant is added.  However $\Delta_{AB}$ can be easily approximated by taking random draws from $\theta_B | Y_B$ and $\theta_A | Y_A$ and computing the difference.

#### Choice of priors

The model assumes that $\theta_B | Y_B$ and $\theta_A | Y_A$ have the same prior distribution.  Unless specified, an uninformative Jeffrey's prior is used, i.e

$$\theta_k \sim \textrm{Beta}(0.5, 0.5)$$

If an informative prior is available the user can define their own Beta prior.

#### Estimating the Bayes Factor

The Bayes Factor for comparing directional hypotheses, e.g. $\theta_B > \theta_A$ vs $\theta_B < \theta_A$, is easy to estimate from the draws from the posterior distribution of $\Delta_{AB}$.

The bayes factor ($K$) can be defined as $\frac{P(D | H_1)}{P(D | H_2)}$ where $D$ denotes the data and $H_1$ and $H_2$ denote the hypotheses $\theta_B > \theta_A$ and $\theta_B < \theta_A$ respectively.  $K$ can therefore be defined as:

$$ K = \frac{P(y_A, y_B | \theta_B > \theta_A)}{P(y_A, y_B | \theta_B < \theta_A)} $$

Given that the marginal likelihood of $P(y_k | \theta_k, \alpha, \beta)$ is defined as $\int p(y_k | \theta_k)p(\theta_k) \, d\theta_k$. The $P(y_A, y_B | \theta_B > \theta_A)$ is:

$$ P(y_A, y_B | \theta_B > \theta_A) = \int_{\theta_B > \theta_A}  L(y_A | \theta_A) p(\theta_A) L(y_B | \theta_B) p(\theta_B) \, d\theta_A d\theta_B$$

The contents of the integral are proportional to $\Delta_{AB}$, therefore:

$$ P(y_A, y_B | \theta_B > \theta_A) \propto P(\theta_B - \theta_A > 0 | y_A, y_B)$$

Where $P(\theta_A, \theta_B | y_A, y_B) \propto \Delta_{AB}$.  Consequently: 

$$ K = \frac{P(y_A, y_B | \theta_B > \theta_A)}{P(y_A, y_B | \theta_B < \theta_A)} = \frac{P(\theta_B - \theta_A > 0 | y_A, y_B)}{ P(\theta_B - \theta_A < 0 | y_A, y_B)}$$

These two numerator and denominator can be easily approximated using can therefore be approximated using random draws from $\Delta_{AB}$.  Interpretation of Bayes factors follows the [Jeffreys scale](https://en.wikipedia.org/w/index.php?title=Bayes_factor#Interpretation).



```python

```

a449e3dbe0c64a9bf556d360ad0d4f430beef867
ipynb
Jupyter Notebook
docs/bayespropestimation_basis.ipynb
oli-chipperfield/bayespropestimation
e4284029cc6c2b699c9b441acc248a1287bac05f
MIT
2020-11-28T17:45:45.000Z
2021-06-13T20:17:07.000Z
docs/bayespropestimation_basis.ipynb
oli-chipperfield/bayespropestimation
e4284029cc6c2b699c9b441acc248a1287bac05f
MIT
docs/bayespropestimation_basis.ipynb
oli-chipperfield/bayespropestimation
e4284029cc6c2b699c9b441acc248a1287bac05f
MIT
Qwen/Qwen-72B
1. YES
2. YES