<a href="https://colab.research.google.com/github/ram-anand/ram-anand.github.io/blob/main/ANOVA%2C_RM%2C_TWO_WAY%2C_F_test_with_code.ipynb" target="_parent"></a>

## Fishers F-Test
 
#### F-distribution

>The F-distribution is formed by the ratio of two independent chi-square variables divided by their respective degrees of freedom.
- Select a random sample of sample variance $s_{1}^2$ of size $n_{1}$ from a normal population, having a standard deviation equal to $\sigma_{1}$, $\chi_{1}^2$ is the chi-square statistic for the sample drawn from population 1.
- Select an independent random sample of sample variance $s_{2}^2$ of size $n_{2}$ from a normal population, having a standard deviation equal to $\sigma_{2}$,  $\chi_{2}^2$ is the chi-square statistic for the sample drawn from population 2 . 
- If both sample comes from same population then $\mu_{1}= \mu_{2}$
- Mean of F distribution = $\mu = \frac{df(num)}{df(denom)}\, -1$

>\begin{align}F = \frac{s_{1}^2/\sigma_{1}^2}{s_{2}^2/\sigma_{2}^2} = \frac{\chi_{1}^2/df_{1}}{\chi_{2}^2/df_{2}}\end{align}

>\begin{align}\frac{df_{1}s_{1}^2}{\sigma^2} \sim \chi_{1}^2, \frac{df_{2}s_{2}^2}{\sigma^2} \sim \chi_{2}^2\end{align}

>\begin{align}df_{1} = (n_{1}-1), df_{2} = (n_{2}-1)\end{align}

</img>

**Characterisitics**
- The F-values are all non-negative
- The distribution is non-symmetric
- The mean is approximately 1
- There are two independent degrees of freedom, one for the numerator, and one for the denominator.
- Distribution is positively skewed towards right. When numerator dof and denominator dof becomes larger, the distribution becomes bell shaped and symmetric.

#### F-Test for Variances

- Null hypothesis: No difference in population variances
- Alternative hypothesis: There is a difference

The F-test is designed to test if two population variances are equal. It does this by comparing the ratio of two variances. So, if the variances are equal, the ratio of the variances will be 1.

The F-test is typically used to answer one of the following questions:
1. Do two samples come from populations with equal variances?
2. Does a new treatment or process reduce the variability of some current treatment or process?
3. Are the the means between two populations significantly different or not?

#### F-statistic 

 - Is a ratio of two variances (Variance is nothing but measure of dispersion, it tells how far the data is dispersed from the mean). F statistic accounts corresponding degrees of freedom to estimate the population variance. The larger variance should always be placed in the numerator.
 - F-statistic is almost similar to t statistic. t-test states a single variable is statistically significant or not whereas F test states a group of variables are statistically significant or not.

> \begin{align} F = \frac{\text{Expained Variability}}{\text{Unexplained Variability}} = \frac{\text{Between Group Variability}}{\text{Within Group Variability}} = \frac{MS_{between}} {MS_{within}} = \frac{MS_{regression}} {MS_{error}} = \frac{\sigma_{1}^2} {\sigma_{2}^2} = \frac{s_{1}^2} {s_{2}^2}\end{align}

### F-test for linear regression model 
  - To tests **any** of the independent variables (predictor) in a multiple linear regression are significant or not. 
  - To compare two competing regression models (model1 with p1 predictor, model2 with p2 predictor) in their ability to **explain** the variance in the dependent/target variable
  - It also indicates a linear relationship between dependent variable and at least one of the independent variable.

>\begin{align}F = \frac{(SSE_{model1}-SSE_{model2})/(p_{2}-p_{1})}{SSE_{model2}/(n-p_{2})} \end{align}



### Assumptions

1. Normality – Samples in each group should follow normal distribution, only needed for small sample sizes, say n < 20 per group.
2. Equal Variances – The variances of the populations that the samples come from should be equal. Homogeneity is only needed if sample sizes are very unequal (Levene's Test).
3. Independence – The observations in each group are independent of each other and the observations within groups were obtained by a random sample.
4. Same Sample Size: The treatment groups should have the same sample size.


### Types of F-test

1. **Pouplation variance is known:**
F test is to find out whether the two independent estimates of population variance differ significantly.

- Null Hypothesis(H0): $\sigma_{1}^2 = \sigma_{2}^2$ (the population variances are equal)
- Alternate Hypothesis(H1): $\sigma_{1}^2 \neq \sigma_{2}^2$ (the population variances are not equal)

 In this case F ratio is
\begin{align} F = \frac{\sigma_{1}^2}{\sigma_{2}^2}, σ_{1}^2 > σ_{2}^2\end{align}
where
\begin{align} \sigma_{1}^2 = \frac{\sum (X_{1}-\mu_{1})^2}{n_{1}}, \sigma_{2}^2 = \frac{\sum (X_{2}-\mu_{2})^2}{n_{2}}\end{align}
  
larger estimate of variance always be in numerator and smaller estimate of variance in denominator 
 - DF of larger variance (i.e numerator) $a=n_{1}-1$
 - DF of smaller variance (i.e denominator) $b=n_{2}-1$

2. **Pouplation variance is unknown:**
To find out whether the two samples drawn from the normal population having the same variance. 

- Null Hypothesis(H0): $s_{1}^2 = s_{2}^2$ (the unbaised sample variances are equal)
- Alternate Hypothesis(H1): $s_{1}^2 \neq s_{2}^2$ (the unbaised sample variances are not equal)

In this case F ratio is

\begin{align}F = \frac{s_{1}^2}{s_{2}^2}, s_{1}^2 > s_{2}^2\end{align} where 
\begin{align}s_{1}^2 = \frac{\sum (X_{1}-\bar{x_{1}})^2}{n_{1}-1}, s_{2}^2 = \frac{\sum (X_{2}-\bar{x_{2}})^2}{n_{2}-1}\end{align}

larger estimate of variance always be in numerator and smaller estimate of variance in denominator 
 - DF of larger variance (i.e numerator) $a=n_{1}-1$
 - DF of smaller variance (i.e denominator) $b=n_{2}-1$


 F-Tests for Equality of Two population Variances

If we take a random sample from two different normally distributed population, the samples are always independent. The sampling F-statistic approximately follows an F-distribution with degrees of freedom df1=$n_{1}−1$ and df2=$n_{2}−1$.

\begin{align} F = \frac{\sigma_{1}^2} {\sigma_{2}^2} = \frac{s_{1}^2} {s_{2}^2}\end{align}
Hypotheses:

|Type of Test|Null hypothesis|Alternate hypothesis|Rejection Region|
|---|---|---|---|
|Two tailed|$\sigma_{1}^2=\sigma_{2}^2$|$\sigma_{1}^2 \neq \sigma_{1}$|$F \le F_{1-\alpha/2}$ or $F \ge F_{\alpha/2}$|
|Right tailed|$\sigma_{1}^2=\sigma_{2}^2$|$\sigma_{1}^2 \gt \sigma_{1}^2$|$F \ge F_{\alpha}$|
|Left tailed|$\sigma_{1}^2=\sigma_{2}^2$|$\sigma_{1}^2 \lt \sigma_{1}^2$|$F \le F_{1-\alpha}$|

<center>
<figure>



<figcaption>Source: (1) saylordotorg.github.io, (2) analyticsvidhya.com</figcaption>
</figure>
</center>



### Steps to conduct F-test
1. Choose the test: Note down the independent variables and dependent variable and also assume the samples are normally distributed
2. Calculate the F statistic, choose the highest variance in the numerator and lowest variance in the denominator with a degrees of freedom (n-1)
3. Determine the statistical hypothesis
4. State the level of significance
5. Compute the critical F value from F table. (use $\alpha/2$ for two tailed test)
6. Calculate the test statistic
7. Finally, draw the statistical conclusion. reject the null hypothesis; If the test statistic falls in the critical region.
  - p value < SL: Null hypothesis can be rejected, this means we have sufficient evidence to say that the two population variances are not equal.
  - p value > SL: Null hypothesis can not be rejected, two population variances are equal.

## Terminology

**Factors:**
The two independent variables in a two-way ANOVA are called factors, which affect the dependent variable. Each factor will have two or more levels within it, and the degrees of freedom for each factor is one less than the number of levels.

**Treatment Groups**
Treatement Groups are formed by making all possible combinations of the levels of two or more factors. So, the total number of treatment groups is the product of the number of levels for each factor. For example, if the first factor has 3 levels and the second factor has 2 levels, then there will be 3x2=6 different treatment groups.

**Completely Randomized Design**
In a completely randomized design, objects or subjects are assigned to groups completely at random. One standard method for assigning subjects to treatment groups is to label each subject, then use a table of random numbers to select from the labelled subjects. This may also be accomplished using a computer.

**Randomized Block Design**
If an experimenter is aware of specific differences among groups of subjects or objects within an experimental group, he or she may prefer a randomized block design to a completely randomized design. In a block design, experimental subjects are first divided into homogeneous blocks before they are randomly assigned to a treatment group.

**Main Effect due to Factor:**
The main effect involves the independent variables one at a time. The interaction is ignored for this part. Just the rows or just the columns are used, not mixed. This is the part which is similar to the one-way analysis of variance. Each of the variances calculated to analyze the main effects are like the between variances

**Variation due to Interaction Effect of Factors:**
The interaction effect is the effect that one factor has on the other factor. The degrees of freedom here is the product of the two degrees of freedom for each factor.

**Between Treatment Variation:**
The Between variation is the sum of squares within each treatment group. The Between variance is the Between variation divided by its degrees of freedom.

**Within Treatment Variation:**
The Within variation is the sum of squares within each treatment group. The within variance is the within variation divided by its degrees of freedom.

**F-Tests:**
There is an F-test for each of the hypotheses, and the F-test is the mean square for each main effect and the interaction effect divided by the within variance. The numerator degrees of freedom come from each effect, and the denominator degrees of freedom is the degrees of freedom for the within variance in each case.

**Replication Number:**
It is number of trials or observations performed for each treatment combination.

**Example:**

||Control|Fertilizer-1|Fertilizer-2|Fertilizer-3|
| --- | --- | --- | --- | --- |
|Plant A|21.0|32.0|22.5|28.0|
|Plant A|19.5|30.5|26.0|27.5|
|Plant B|23.7|30.1|30.6|36.1|
|Plant B|23.8|28.9|31.1|36.6|

- Main Effect A = Fertilizer, Factors/Levels of A = Type of Fertilizers used = 3
- Main Effect B = Plant, Factors/Levels of B = Type of Plants/species =  2
- Number of Treatment Groups = Factors of A * Factors of B = 3*2 = 6
- Number of Replications = R = 2



```
# F-Test in Python
# To perform Fishers-test we can use scipy.stats.f.cdf(f-dist-rv, numerator_dof, denomintor_dof)
import numpy as np
import pandas as pd
import scipy.stats
#define F-test function
def f_test(x, y):
    x = np.array(x)
    y = np.array(y)
    #By default, numpy.var calculates the population variance. To calculate the sample variance, we need to specify ddof=1
    f = np.var(x, ddof=1)/np.var(y, ddof=1) #calculate F test statistic , ratio of sample variance of two populations x, y
    dfn = x.size-1 #define degrees of freedom numerator 
    dfd = y.size-1 #define degrees of freedom denominator 
    p = 1-scipy.stats.f.cdf(f, dfn, dfd) #find p-value of F test statistic 
    return f, p

#perform F-test
x = [18, 19, 22, 25, 27, 28, 41, 45, 51, 55]
y = [14, 15, 15, 17, 18, 22, 25, 25, 27, 34]
f_test(x, y)
```




    (4.387122002085505, 0.01912653593238578)



## ANOVA Assumptions

1. Normality: Samples in each group should follow normal distribution, only needed for small sample sizes, say n < 20 per group.
2. Equal Variances: The variances of the populations that the samples come from should be equal. Homogeneity is only needed if sample sizes are very unequal (Levene's Test).
3. Randomness & Independence: The observations in each group are independent of each other and the observations within groups were obtained by a random sample.
4. Same Sample Size: The treatment groups should have the same sample size.

## Experimental Design

#### Total variablity sum of squares

Total variablity = Between group variability + Between subjects variability + Within variability (Random variability) +  Factor Interaction variablity

>\begin{align} SST = SSB(Group) +  SSB(Subject) + SSE(Random) + SSI(Group-Subject) \end{align}

>\begin{align} SSE(Random) = SSW(Group) + SSW(Subject) \end{align}

**Completely Randomized Design (CRD):**
>In this Experimental design, obeservations are taken in random order and the envirnment in which the treatments are used is as uniform as possible.\
Experimental design follows either Fixed Effect model or Random effect model. In Random effect model, levels of all participating factors are choosen by random process while in Fixed effect model, some specific levels are chosen by the experimenter.

#### Single Factor experimental design
>There is only one factor with 2 or more levels. The levels (columns) are called groups, or condition. These levels forms treatment combination.
1. **One Way ANOVA model**
  - SSE(Random) = SSW(Groups)
  - SST = SSB(Group) + SSE(Random)
2. **Random Measures ANOVA**
  - **Randomized Complete Block Design (RBCD)** 
  - SST = SSB(Group) + SSB(Subject) + SSE(Random)

#### Two Factor experimental design 
>**Two Way ANOVA**: There are two factors say A & B and each factor has 2 or more levels. The levels of row arranged factor A can be called subjects or blocks or rows. The levels of column arranged factor B can be called groups or condition or columns. The pair level-i(A), level-j(B) toegether forms a treatment combination for the dependent variable. The replication count (R) is the number of samples or trails for each treatment. 
  - SST = SSB(Group) + SSB(Subject) + SS(Interaction) + SSE(Random)

source: https://www.nature.com/articles/nmeth.3180</img>
a) Two common two-factor designs with 8 measurements each. In the CR scenario, each mouse is randomly assigned a single treatment. Variability among mice can be mitigated by grouping mice by similar characteristics (e.g., litter or weight). The group becomes a block. Each block is subject to all treatments. (b) Partitioning of the total sum of squares for the CR and RCB designs in a. M represents the blocking factor. Vertical axis is relative to the SST. 


## One way ANOVA (Independent ANOVA)

A one-way ANOVA compares the means of three or more independent (unrelated) groups of a factor to determine if there is a statistically significant difference between the corresponding population means.

### Hypotheses
- Null hypothesis(H0):
$\mu_{1} = \mu_{2} = \mu_{3} = … = \mu_{k}$ (all the population means are equal)
- Alternative hypothesis(H1): At least one population mean is different from the rest

</img>

**Single Factor Experiment**
A single factor (variable) experiment where factor has different levels (aslo called treatment groups), One way ANOVA compares the mean of each tretment level

> Each group / treament mean can be described in terms of overall mean of all treatment groups or levels :
\begin{align} \mu_{j} = (\mu + \tau_{j}) \end{align}
Each observation of group / treament can be described as:
\begin{align} x_{ij} = (\mu + \tau_{j}) + e_{ij} \end{align}

where 
 - $\mu$ = overall mean of factor population from which these samples for each treament / group is taken
 - $e_{ij}$ = random error for component, we assume random errors $e_{j}$ are normally distributed with mean 0 and variance $\sigma^2$
 - $\tau_{j}$ = treatment or experimental effect (deviation from factor population mean), how these treatment levels chosen gives two different models namely a) FIXED effect model, b) RANDOM effect model 

**FIXED effect model:**
  - Some specific B levels or treatment are choosen by experimenter from all possible levels (combination)
  - Sum of treatment/experimental effect of all groups is zero:
\begin{align} \sum_{j=1}^{B}\tau_{j} = 0 \end{align}
  - Hypotheses:
   - H0: $\tau_{1} = \tau_{1} = \tau_{1}...=\tau_{B}=0$, there is no difference in the mean of treatment groups
   - H1: $\tau_{j} \neq 0$ for at least one j-th group, there is at least one group mean different from others

**RANDOM effect model:** 
  - Randomly B levels or treatment are choosen from all possible levels (combination)
  - Treatment effect $\tau_{j}$ are normally distributed with mean 0 and variance $\sigma_{\tau}^2$
  - Hypotheses:
   - H0: $\sigma_{\tau}^2 = 0$, all treatment group means are identical
   - H1: $\sigma_{\tau}^2 \neq 0$, there is variability between treatment groups



### Variability & Expectation

#### Total variability - sum of squares identity
---
As there is only one factor, there is no interaction component

\begin{align} SST = SSB(Groups) + SSE(Random) \end{align}

\begin{align} SSE(Random) = SSW(Groups) = SST - SSB(Groups) \end{align}

Partitioning SST:
>\begin{align} \sum_{i=1}^{A}\sum_{j=1}^{B} (x_{ij}-\bar{x}_{grand})^2 = A*\sum_{j=1}^{B}{(\bar{x}_{j} - \bar{x}_{grand})}^2 + \sum_{j=1}^{B}\sum_{i=1}^{A}(x_{ij}-\bar{x}_{j})^2 \end{align}

Partitioning SST degree of fredom:
>\begin{align} AB-1 = (B-1) + B(A-1) \end{align}
---

#### Variance Expectation
1. Expected Between Groups Variance
 - Sum of squares:
>\begin{align} SSB(Groups) = (B-1)\sigma^2+A*\sum_{j=1}^{B}\tau_{j}^2\end{align}
 - Mean sum of squares):
>\begin{align} MSB(Groups) = \frac{SSB(Groups)}{B-1} = \sigma^2+\frac{A}{(B-1)}\,*\sum_{j=1}^{B}\tau_{j}^2 = \sigma^2+A*\sigma_{\tau}^2\end{align}
2. Expected Random Error Variance
 - Sum of squares:
>\begin{align} SSE(Random) = B(A-1)\sigma^2\end{align}
 - Mean sum of squares:
>\begin{align} MSE(Random) = \frac{SSE(Random)}{B(A-1)} = \sigma^2\end{align}


### Sum of Squares Formula

|Source of Variation| SS Form 1 | SS Form 2 | SS Form 3 | Degree of Freedom | Notes, comments, some extra details |
|---|---|---|---|---|---|
|$SST$|\begin{align} \sum_{i=1}^{A}\sum_{j=1}^{B} (x_{ij}-\bar{x}_{grand})^2 \end{align}|\begin{align}\sum_{i=1}^{A}\sum_{j=1}^{B}x_{ij}^2 - \frac{{GT}^2}{N} \end{align}|\begin{align} (N-1)*s_{grand}^2\end{align}|$(N-1)$|\begin{align} variance_{total} = s_{grand}^2 = \,\frac{1}{N-1}\sum_{i=1}^{A}\sum_{j=1}^{B}(x_{ij}-\bar{x}_{grand})^2 \end{align}|
|$SSB(Groups)$|\begin{align} A*\sum_{j=1}^{B}{(\bar{x}_{j} - \bar{x}_{grand})}^2 \end{align}|\begin{align} \frac{1}{A}\,\sum_{j=1}^{B}(\sum_{i=1}^{A}x_{ij})^2 - \frac{GT^2}{N} \end{align}||$(B-1)$||
|$SSW(Groups) = SSE$|\begin{align} \sum_{j=1}^{B}\sum_{i=1}^{A}(x_{ij}-\bar{x}_{j})^2 \end{align}||\begin{align} \sum_{j=1}^{B}(n_{j}-1)*s_{j}^2\end{align}|$B(A-1) = (N-B)$|\begin{align} variance_{group(i)} = s_{j}^2 = \, \frac{1}{A-1}\sum_{i=1}^{A}(x_{ij}-\bar{x}_{j})^2 \end{align}|

### One Way ANOVA table

| Source	of Variation | Sum of Squares (SS) |	degree of freedom calculation (df)| Mean Squares (MS)	| F-Statistic or Ratio|
|---|---|---|---|---|
| Between	Groups or Regression or Model | SSR or SSB	| $df_{groups} = B-1$|	$MSR = \frac{SSR}{df_{reg}}$ | $F = \frac{MSR}{MSE}$ |
| Within Groups or Error	or Residual| SSE or SSW	| $df_{error} = N-B$ | $MSE	= \frac{SSE}{df_{error}}$| |
| Total	| SST	| $df_{total} = N-1$ | | | | 

### Naming Convention

---
- Group = Column = Treatment = Level
- i = row/replication/sample number 
- j = group/column number
- $A$ = number of observations in each groups or columns
- $B$ = number of groups or columns
- $n_{j}$ = $A$ = the sample size or number of observartions in j-th group/column
- $N = A*B$ = total number of observations
- $x_{ij}$ = observation at i-th subject (row) and j-th group(column)
- $\bar{x}_{j}$ = the mean of j-th group/column
- $GT = \sum_{i=1}^{A}\sum_{j=1}^{B} x_{ij}$ = the grand total of the entire dataset
- $\bar{x}_{grand}$ = the grand mean or overall mean of all observations
- $s_{grand}^2$ = the unbiased sample variance for the entire dataset
- $s_{i}^2$ = the unbiased sample variance for the i-th subject/row observations
- $s_{j}^2$ = the unbiased sample variance for the j-th group/column observations
- $df_{reg} = B-1$: regression degrees of freedom
- $df_{error} = N-B$: error degrees of freedom
- $df_{total} = N-1$: total degrees of freedom
- MSR = SSR/$df_{reg}$ = regression mean square
- MSE = SSE/$df_{error}$ = error mean square
- F test statistic (F = MSR/MSE)
---


### Steps by hand

**Example:**

| | group 1 | group 2 | group 3 |
| --- | --- | --- | --- | 
|Index| j=1 | j=2 | j=3 |
| --- | --- | --- | --- | 
|i=1| - | - | - |
|i=2| - | - | - |


1. **Calculate the group means and the overall mean:**

- Grand mean or overall mean
\begin{align}\bar{x}_{grand} = \frac{1}{N}\sum_{i=1}^{A}\sum_{j=1}^{B} x_{ij} = \frac{1}{N}*{GT}\end{align}

- Group mean for j-th group
\begin{align}\bar{x}_{j} = \frac{1}{n_{j}}\sum_{i=1}^{n_{j}} x_{ij} = \frac{1}{A}\sum_{i=1}^{A} x_{ij}\end{align}

2. **Total sum of squares (SST):**
- The total group variation (not variance) is comprised the sum of squares deviations of all the observations from the grand mean
- Degree of freedom is (total number of observations - 1)

Method-1
>\begin{align} SST = \sum_{i=1}^{A}\sum_{j=1}^{B}(x_{ij}-\bar{x}_{grand})^2 = \sum_{i=1}^{A}\sum_{j=1}^{B}x_{ij}^2 - \frac{{GT}^2}{N} \end{align}
 - Calculate the difference between each observation and the grand mean irrespective of group(column) or subject(row)
 - Square each of these differences
 - Add the values together to get SST

Method-2
>\begin{align} SST = (N-1)*s_{grand}^2\end{align}
\begin{align} variance_{total} = s_{grand}^2 = \frac{1}{N-1}\,\sum_{i=1}^{A}\sum_{j=1}^{B}(x_{ij}-\bar{x}_{grand})^2 \end{align}
 - Calculate the unbiased variance for all observations together irrespective of group(column) or subject(row)
 - Multiply it with (total number of observations - 1) to get SST

Method-3
>We will calculate the total sum of squares (SST) using the following formula:
\begin{align} SST = SSB + SSW = SSR + SSE \end{align}

3. **Between Groups/Samples Sum of Squares (SSB) or Regression sum of squares (SSR):** 
- Also called SS(Model) or SS(Treatment)
- Variation between Group/column means with grand means
- It accounts the variation due to the interaction between the samples/groups. If the sample means (group means) are close to each other (and therefore the grand mean) this will be small.
- The degree of freedom is equal to (number of groups or samples - 1)

Method-1
>\begin{align} SSB = \sum_{j=1}^{B} n_{j} * (\bar{x}_{j}-\bar{x}_{grand})^2 \end{align}
 - Calculate the difference between the mean of each group and the grand mean
 - Square each of these differences
 - Multiply each result by the number of participants (sample size) within that group 
 - Add the values for each group together to get SSR

Method-2
>\begin{align} SSB = \frac{1}{A}\,\sum_{j=1}^{B}(\sum_{i=1}^{A}x_{ij})^2 - \frac{GT^2}{N} \end{align}
 - Calculate the sum of observations for each group (column)
 - Square each sum (group wise sum) and add them
 - Divide it by samplesize or total number of rows, call it A
 - Sum all observations to get grand sum (grand total)
 - Square the grand total and divide it by total number of observations, call it B
 - Calculate A-B to get the SSB

4. **Within Group/Sample sum of squares (SSW) or Error sum of squares (SSE):** or SS(Residual) 
- It account for the variation due to differences within individual samples, denoted as Sum of Squares Within groups. Each sample is considered independently, no interaction between samples is involved. 
- The degrees of freedom is equal to the sum of the individual degrees of freedom for each sample. Since each sample has degrees of freedom equal to one less than their sample sizes, and there are k samples, the total degrees of freedom is k less than the total sample size: df = N - k.

Method-1
>\begin{align} SSW = \sum_{j=1}^{B}\sum_{i=1}^{A}(x_{ij}-\bar{x}_{j})^2 \end{align}
 - Calculate the difference between each observation of the group and the group mean
 - Square each of these differences
 - Add the values for each group together and call it group error SS
 - Add the group error SS of all the groups to get the SSW

Method-2
>\begin{align} SSW = \sum_{j=1}^{B}(n_{j}-1)*s_{j}^2\end{align}
\begin{align} variance_{group(j)} = s_{j}^2 = \frac{1}{A-1}\,\sum_{i=1}^{A}(x_{ij}-\bar{x}_{j})^2 \end{align}
 - Calculate the unbiased variance for j-th group observations
 - Multiply it with (total observations in the group - 1) to get SSW

5. **ANOVA table:**
Now that we have SSR, SSE, and SST, we can fill in the ANOVA table:

6. Effect Size
>\begin{align} \eta^2 = \frac{SSB}{SST} = \frac{SSR}{SST} \end{align}


7. **Interpret the results:** To determine if this is a statistically significant result, we must compare this F-statistic to the F critical value found in the F distribution table for $F(df_{reg}, df_{error})$

  - if p-value < SL,  then we reject H0 (Null Hypothesis of all means are equal)
  - if p-value >= SL,  then we fail to reject H0 (Null Hypothesis of all means are equal)


### Post-Hoc Test
In order to determine which groups are different from one another, a post-hoc test is needed

- Tukey's Honestly Significant Differences (HSD) Test
  For each pairwise comparison for sample i and j, 
  **H0 (null hypothesis):** $\mu_{i} = \mu_{j}$,
  **Ha (alternate hypothesis):** $\mu_{i} \neq \mu_{j}$

  Tukey's critical value $w = q_{\alpha}(p, df_{error}) * s_{error}$

  where 
  - $q_{\alpha}$ = Obtained from a Table of Tukey values for given significance level,
  - p = the number of groups or treatment levels
  - $n_{samplesize}$ = number of replications/observations
  - $s_{error} = \sqrt{MSE/A}$ = standard error of group mean where A is the number of observations in each group

- Rank the group means in descending order, and calculate pairwise differences. For all groups i and j, 
- if $mean_{i} - mean_{j} <= w$, it indicates the group means do not differ significantly
- if $mean_{i} - mean_{j} > w$, it indicates the group means differ significantly

### Example One Way ANOVA

| | group 1 | group 2 | group 3 |
| --- | --- | --- | --- | 
|Index| j=1 | j=2 | j=3 |
| --- | --- | --- | --- | 
|i=1| 85 | 91 | 79 |
|i=2| 86 | 92 | 78 |
|i=3| 88 | 93 | 88 |
|i=4| 75 | 85 | 94 |
|i=5| 78 | 87 | 92 |
|i=6| 94 | 84 | 85 |
|i=7| 98 | 82 | 83 |
|i=8| 79 | 88 | 85 |
|i=9| 71 | 95 | 82 |
|i=10| 80 | 96 | 81 |
|Group Mean| 83.4| 89.3 |84.7 |

1. Overall mean = 85.8, Group-1 Mean=83.4, Group-2 Mean=89.3, Group-3 Mean=84.7
2. SST = 192.2 + 1100.6 = 1292.8
3. SSR = 10(83.4-85.8)2 + 10(89.3-85.8)2 + 10(84.7-85.8)2 = 192.2
4. We calculate SSE as follows:
 - Group 1: (85-83.4)2 + (86-83.4)2 + (88-83.4)2 + (75-83.4)2 + (78-83.4)2 + (94-83.4)2 + (98-83.4)2 +  (79-83.4)2 + (71-83.4)2 + (80-83.4)2 = 640.4
 - Group 2: (91-89.3)2 + (92-89.3)2 + (93-89.3)2 + (85-89.3)2 + (87-89.3)2 + (84-89.3)2 + (82-89.3)2 +  (88-89.3)2 + (95-89.3)2 + (96-89.3)2 = 208.1
 - Group 3: (79-84.7)2 + (78-84.7)2 + (88-84.7)2 + (94-84.7)2 + (92-84.7)2 + (85-84.7)2 + (83-84.7)2 +  (85-84.7)2 + (82-84.7)2 + (81-84.7)2 = 252.1
 - SSE = 640.4 + 208.1 + 252.1 = 1100.6

 Here is how we calculated the various numbers in the table:

5. ANOVA Table
- df treatment(groups) = 3-1 = 2
- df error = 30-3 = 27
- df total = 30-1 = 29
- MS treatment group = SST / df treatment = 192.2 / 2 = 96.1
- MS error = SSE / df error = 1100.6 / 27 = 40.8
- F = MS treatment group / MS error = 96.1 / 40.8 = 2.358

6. Intrepret result
- The F test statistic for this one-way ANOVA is 2.358. To determine if this is a statistically significant result, we must compare this to the F critical value found in the F distribution table with the following values:
- $\alpha$ (significance level) = 0.05
- DF1 (numerator degrees of freedom) = df treatment = 2
- DF2 (denominator degrees of freedom) = df error = 27
We find that the F critical value is 3.3541.

Since the F test statistic in the ANOVA table is less than the F critical value in the F distribution table, we fail to reject the null hypothesis. This means we don’t have sufficient evidence to say that there is a statistically significant difference between the mean exam scores of the three groups.



```
# One way ANOVA by Statmodels Library
import numpy as np
import pandas as pd
import statsmodels.api as sm
from statsmodels.formula.api import ols
A = [12.6, 12, 11.8, 11.9, 13, 12.5, 14]
B = [10, 10.2, 10, 12, 14, 13]
C = [10.1, 13, 13.4, 12.9, 8.9, 10.7, 13.6, 12]
all_scores = A + B + C
company_names = (['A'] * len(A)) +  (['B'] * len(B)) +  (['C'] * len(C))
data = pd.DataFrame({'company': company_names, 'score': all_scores})
data.groupby('company').mean()
lm = ols('score ~ company', data = data).fit()
table = sm.stats.anova_lm(lm)
print(table)
```

                df     sum_sq   mean_sq         F    PR(>F)
    company    2.0   3.606905  1.803452  0.821297  0.455683
    Residual  18.0  39.525476  2.195860       NaN       NaN



```
# One way ANOVA by hand

# compute group means
group_means = data.groupby('company').mean()
#group_means = group_means.rename(columns = {'score': 'group_mean'})
group_means.columns = ['group_mean']
print("group_means",group_means)

# compute overall mean
overall_mean = data['score'].mean()
print("overall_mean=",overall_mean)

# compute Sum of Squares Total
data['overall_mean'] = overall_mean
ss_total = sum((data['score'] - data['overall_mean'])**2)
print("ss_total=",ss_total)

# add group means and overall mean to the original data frame
data = data.merge(group_means, left_on = 'company', right_index = True)

# compute Sum of Squares Error/Residual/Within
ss_residual = sum((data['score'] - data['group_mean'])**2)
print("ss_residual=",ss_residual)

# compute Sum of Squares Regression/Between/Explained/Model
ss_explained = sum((data['overall_mean'] - data['group_mean'])**2)
print("ss_explained=",ss_explained)

# compute Mean Square Error/Residual/Within
n_groups = len(set(data['company']))
n_obs = data.shape[0]
df_residual = n_obs - n_groups
ms_residual = ss_residual / df_residual
print("ms_residual=",ms_residual)

# compute Mean Square Regression/Between/Explained/Model
df_explained = n_groups - 1
ms_explained = ss_explained / df_explained
print("ms_explained=",ms_explained)

# compute F-Value
f = ms_explained / ms_residual
print("F-score=",f)

# compute p-value
import scipy.stats
p_value = 1 - scipy.stats.f.cdf(f, df_explained, df_residual)
print("p-value=",p_value)
```

    group_means          group_mean
    company            
    A         12.542857
    B         11.533333
    C         11.825000
    overall_mean= 11.980952380952381
    ss_total= 43.132380952380956
    ss_residual= 39.52547619047619
    ss_explained= 3.6069047619047776
    ms_residual= 2.1958597883597886
    ms_explained= 1.8034523809523888
    F-score= 0.8212966923081592
    p-value= 0.4556832940515221


## Repeated Measures ANOVA

A repeated-measures (or within-participants) test is what you use when you want to compare the performance of the same group of participants in different experimental conditions.  
That is, when the same participants take part in all of the conditions in your study.

The term one-way simply to refers to the number of independent variables (one independent variable) you have; 
- one dependent variable
- one independent variable (with 3 or more conditions)
- each participant(subject or row) takes part in all conditions/groups/columns

### Hypotheses

A repeated measures ANOVA is used to determine whether or not there is a statistically significant difference between the means of three or more groups in which the same subjects show up in each group.

1. Measuring the mean scores of subjects(group) during three or more time points.
2. Measuring the mean scores of subjects(group) under three different conditions.


### Variability & Expectation

#### Total variability - sum of squares identity
---
As there is only one factor, there is no interaction component

\begin{align} SST = SSB(Group) + SSB(Subject) + SSE(Random) \end{align}

Partitioning SST:
>\begin{align} \sum_{i=1}^{A}\sum_{j=1}^{B} (x_{ij}-\bar{x}_{grand})^2 \,= A*\sum_{j=1}^{B}{(\bar{x}_{j} - \bar{x}_{grand})}^2 +\, B*\sum_{i=1}^{A}{(\bar{x}_{i} - \bar{x}_{grand})}^2 \,+ \sum_{i=1}^{A}\sum_{j=1}^{B}(x_{ij}-\bar{x}_{i}-\bar{x}_{j}+\bar{x}_{grand})^2 \end{align}

Partitioning SST degree of fredom:
>\begin{align} AB-1 = (B-1) + (A-1) + (A-1)(B-1) \end{align}
---

#### Variance Expectation
1. Expected Between Groups Variance
 - Sum of squares:
>\begin{align} SSB(Groups) = (B-1)*\sigma^2+A*\sum_{j=1}^{B}\tau_{j}^2\end{align}
 - Mean sum of squares):
>\begin{align} MSB(Groups) = \frac{SSB(Groups)}{B-1} = \sigma^2+\frac{A}{(B-1)}\,*\sum_{j=1}^{B}\tau_{j}^2 = \sigma^2+A*\sigma_{\tau}^2\end{align}
2. Expected Between Subjects/Blocks Variance
 - Sum of squares:
>\begin{align} SSB(Subject) = (A-1)*\sigma^2+B*\sum_{i=1}^{A}\beta_{i}^2\end{align}
 - Mean sum of squares):
>\begin{align} MSB(Subject) = \frac{SSB(Subject)}{A-1} = \sigma^2+\frac{B}{(A-1)}\,*\sum_{i=1}^{A}\beta_{i}^2 = \sigma^2+B*\sigma_{\beta}^2\end{align}
3. Expected Random Error Variance
 - Sum of squares:
>\begin{align} SSE(Random) = (A-1)(B-1)*\sigma^2\end{align}
 - Mean sum of squares:
>\begin{align} MSE(Random) = \frac{SSE(Random)}{(A-1)(B-1)} = \sigma^2\end{align}

### Sum of Squares Formula

|Source of Variation| SS Form 1 | SS Form 2 | SS Form 3 | Degree of Freedom | Notes, comments, some extra details |
|---|---|---|---|---|---|
|$SST$|\begin{align} \sum_{i=1}^{A}\sum_{j=1}^{B} (x_{ij}-\bar{x}_{grand})^2 \end{align}|\begin{align}\sum_{i=1}^{A}\sum_{j=1}^{B}x_{ij}^2 - \frac{{GT}^2}{N} \end{align}|\begin{align} (N-1)*s_{grand}^2\end{align}|$(N-1)$|\begin{align} variance_{total} = s_{grand}^2 = \,\frac{1}{N-1}\sum_{i=1}^{A}\sum_{j=1}^{B}(x_{ij}-\bar{x}_{grand})^2 \end{align}|
|$SSB(Groups)$|\begin{align} A*\sum_{j=1}^{B}{(\bar{x}_{j} - \bar{x}_{grand})}^2 \end{align}|\begin{align} \frac{1}{A}\,\sum_{j=1}^{B}(\sum_{i=1}^{A}x_{ij})^2 - \frac{GT^2}{N} \end{align}||$(B-1)$||
|$SSB(Subjects)$|\begin{align} B*\sum_{i=1}^{A}{(\bar{x}_{i} - \bar{x}_{grand})}^2 \end{align}|\begin{align} \frac{1}{B}\,\sum_{i=1}^{A}(\sum_{j=1}^{B}x_{ij})^2 - \frac{GT^2}{N} \end{align}||$(A-1)$||
|$SSW(Groups)$|\begin{align} \sum_{j=1}^{B}\sum_{i=1}^{A}(x_{ij}-\bar{x}_{j})^2 \end{align}||\begin{align} \sum_{j=1}^{B}(n_{j}-1)*s_{j}^2\end{align}|$(N-B)$|\begin{align} variance_{group(i)} = s_{j}^2 = \, \frac{1}{A-1}\sum_{i=1}^{A}(x_{ij}-\bar{x}_{j})^2 \end{align}|
|$SSW(Subjects)$|\begin{align} \sum_{i=1}^{A}\sum_{j=1}^{B}(x_{ij}-\bar{x}_{i})^2 \end{align}||\begin{align} \sum_{i=1}^{A}(n_{i}-1)*s_{i}^2 \end{align}|$(N-A)$|\begin{align} variance_{subject(i)} = s_{i}^2 = \, \frac{1}{B-1}\sum_{j=1}^{B}(x_{ij}-\bar{x}_{i})^2 \end{align}|
|$SSE = SSW$|\begin{align} \sum_{i=1}^{A}\sum_{j=1}^{B}(x_{ij}-\bar{x}_{i}-\bar{x}_{j}+\bar{x}_{grand})^2 \end{align}|$SST-SSB(Groups)-SSB(Subjects)$||$(A-1)*(B-1)$||

### Repeated measures ANOVA table

| Source of Variation | Sum of Square (SS) | Degrees of Freedom Calculation (df) | Mean Square Calculation (MS) | F ratio |
| --- | --- | --- | --- | --- |
| Groups (Between) or Participents or Columns | SSB(Groups) | $df_{groups} = (B-1)$ | $MSB(Groups) = SSB(Groups)/df_{columns}$ | F = MSB(Groups)/MSE |
| Subject (Between) or Rows | SSB(Subjects) | $df_{subjects} = (A-1)$ | $MSB(Subjects) = SSB(Subjects)/df_{subjects}$ | F = MSB(Subjects)/MSE |
| Error (Within) or Residual | SSE | $df_{error} = (A-1)(B-1))$| $MSE = SSE/df_{error}$ | |
| Total | SST | $df_{total} = (N-1)$ |||


### Naming Convention

---
- Group = Column = Treatment = Level
- Subject = Row = Block
- i = row number/subject/Block
- j = column number/group/Level
- $A$ = total number of subjects/rows
- $B$ = total number of groups/columns
- $n_{i}$ = $B$ = total number of observations in the i-th subject/column,
- $n_{j}$ = $A$ = total number of observations in the j-th group/column,
- $N = A*B$ = the total number of observations in the entire dataset
- $x_{ij}$ = observation at i-th subject (row) and j-th group(column)
- $\bar{x}_{i} = \frac{1}{n_{i}}\sum_{j=1}^{n_{i}} x_{ij} = \frac{1}{B}\sum_{i=1}^{B} x_{ij}$ = the mean of the i-th subject/row, 
- $\bar{x}_{j} = \frac{1}{n_{j}}\sum_{i=1}^{n_{j}} x_{ij} = \frac{1}{A}\sum_{i=1}^{A} x_{ij}$ = the mean of the j-th group/column, 
- $GT = \sum_{i=1}^{A}\sum_{j=1}^{B} x_{ij}$ = the grand total of the entire dataset
- $\bar{x}_{grand} = \frac{1}{N}\sum_{i=1}^{A}\sum_{j=1}^{B} x_{ij}$ = the grand mean or mean of the entire dataset
- $s_{grand}^2$ = the unbiased sample variance for the entire dataset
- $s_{i}^2$ = the unbiased sample variance for the i-th subject/row observations
- $s_{j}^2$ = the unbiased sample variance for the j-th group/column observations
---

### Steps by hand

**Example:**

| Treatment ||| group-1 | group-2 | group-3 |
| --- | --- | --- | --- | --- | --- |
||Index| i=1 | i=2 | i=3 | i=4 |
|| --- | --- | --- | --- | --- |
|subject-1|j=1| - | - | - | - |
|subject-2|j=2| - | - | - | - |
|subject-3|j=3| - | - | - | - |
|subject-4|j=4| - | - | - | - |

In a typical one-way ANOVA, different subjects are used in each group.
But in repeated measures ANOVA, the same subjects show up in each group.

**H0 (null hypothesis):** µ1 = µ2 = µ3 (the population means are all equal)

**H1 (alternative hypothesis):** at least one population mean is different from the rest

### Steps

1. **Total sum of squares (SST):**

Method-1
>\begin{align} SST = \sum_{i=1}^{A}\sum_{j=1}^{B}(x_{ij}-\bar{x}_{grand})^2 = \sum_{i=1}^{A}\sum_{j=1}^{B}x_{ij}^2 - \frac{{GT}^2}{N} \end{align}
 - Calculate the difference between each observation and the grand mean irrespective of group(column) or subject(row)
 - Square each of these differences
 - Add the values to get SST

Method-2
>\begin{align} SST = (N-1)*s_{grand}^2\end{align}
\begin{align} variance_{total} = s_{grand}^2 = \frac{1}{N-1}\,\sum_{i=1}^{n_{subjects}}\sum_{j=1}^{B}(x_{ij}-\bar{x}_{grand})^2 \end{align}
 - Calculate the unbiased variance for all observations together irrespective of group(column) or subject(row)
 - Multiply it with (total number of observations - 1) to get SST

Method-3
> \begin{align} SST = SSB + SSW =  SSB + (SSS + SSE) \end{align} 

 - Calculate the unbiased variance for all observations together irrespective of group(column) or subject(row)
 - Multiply it with (total number of observations - 1) to get SST

2. **Between Group sum of squares or SSB(Group):** 
 - Also called SS(Group) or SS(Treatment) or SS(Model)
 - Variance of group/column mean with grand mean

Method-1
>\begin{align} SSB(Group) = \sum_{j=1}^{B}{n_{j}*(\overline{x}_{j} - \overline{x}_{grand})}^2 \end{align}
 - Calculate the difference between the mean of each group/column and the grand mean
 - Square each of these differences
 - Multiply each result by the number of observations/participants within that group
 - Add the values for each group/column together

Method-2
>\begin{align} SSB(Group) = \frac{1}{A}\,\sum_{j=1}^{B}(\sum_{i=1}^{A}x_{ij})^2 - \frac{GT^2}{N} \end{align}
 - Calculate the sum of observations for each group (column)
 - Square each sum (group wise sum) and add them
 - Divide it by total number of subjects, call it A
 - Sum all observations to get grand sum(grand total)
 - Square the grand total and divide it by total number of observations, call it B
 - Calculate A-B to get the SSB(Group)

3. **Between Subject sum of squares or SSB(Subject):** 
 - Variance of subject/row mean with grand mean

Method-1
>\begin{align} SSB(Subject) = \sum_{i=1}^{A}{n_{i}*(\overline{x}_{i} - \bar{x}_{grand})}^2 \end{align}
 - Calculate the difference between the mean of each subject/row and the grand mean
 - Square each of these differences
 - Multiply each result by the number of observations within that subject/row
 - Add the values for each group together

 Method-2
>\begin{align} SSB(Subject) = \frac{1}{B}\,\sum_{i=1}^{A}(\sum_{j=1}^{B}x_{ij})^2 - \frac{GT^2}{N} \end{align}
 - Calculate the sum of observations for each subject (row)
 - Square each sum (subject wise sum) and add them
 - Divide it by total number of groups, call it A
 - Sum all observations to get grand sum(grand total)
 - Square the grand total and divide it by total number of observations, call it B
 - Calculate A-B to get the SSB(Subject)

4. **Within Group sum of squares or SSW(Groups):** 
- Variance of group/column observations with group mean

Method-1
>\begin{align} SSW(Group) = \sum_{j=1}^{B}\sum_{i=1}^{A}(x_{ij}-\bar{x}_{j})^2 \end{align}
 - Calculate the difference between each observation of the group/column and the group/column mean
 - Square each of these differences
 - Add the values for each group together and call it group error SS
 - Add the group error SS of all the groups to get the SSW

Method-2
>\begin{align} SSW(Group) = \sum_{j=1}^{B}(n_{j}-1)*s_{j}^2\end{align}
\begin{align} variance_{group(i)} = s_{j}^2 = \frac{1}{A-1}\,\sum_{i=1}^{A}(x_{ij}-\bar{x}_{j})^2 \end{align}
 - Calculate the unbiased variance for j-th group/column observations
 - Multiply it with (total observations in the group/column - 1) to get SSW

5. **Within Subjects sum of squares or SSW(Subject):** 
- Variance of subject/row observations with subject mean

Method-1
>\begin{align} SSW(Subject) = \sum_{i=1}^{A}\sum_{j=1}^{B}(x_{ij}-\bar{x}_{i})^2 \end{align}
 - Calculate the difference between each observation of the subject/rows and the subject/row mean
 - Square each of these differences
 - Add the values for each subject/row together and call it subject error SS
 - Add the subject error SS of all the groups to get the SSW(Subject)

Method-2
>\begin{align} SSW(Subject) = \sum_{i=1}^{A}(n_{i}-1)*s_{i}^2 \end{align}
\begin{align} variance_{subject(i)} = s_{i}^2 = \frac{1}{B-1}\,\sum_{j=1}^{B}(x_{ij}-\bar{x}_{i})^2 \end{align}
 - Calculate the unbiased variance for i-th subject/row observations
 - Multiply it with (total observations in the subject/row - 1) to get SSW(Subject)

6. **Error sum of squares (SSE):** Next, we will calculate the error sum of squares (SSE), which can be found using the following formula:

\begin{align} SSE = SST – SSB(Group) – SSB(Subject) \end{align}

7. **Repeated measures ANOVA table:** Now that we have SSB, SSS, and SSE, we can fill in the repeated measures ANOVA table:

8. **Interpret the results:** 


### Example Repeated Measures ANOVA

|| Patient | Drug-1 | Drug-2 | Drug-3 |
| --- | --- | --- | --- | --- |
|Index| i=1 | i=2 | i=3 | i=4 |
| --- | --- | --- | --- | --- |
|j=1| Patient 1 | 30 | 28 | 16 |
|j=2| Patient 2 | 14 | 18 | 10 |
|j=3| Patient 3 | 24 | 20 | 18 |
|j=4| Patient 4 | 38 | 34 | 20 |
|j=5| Patient 5 | 26 | 28 | 14 |


1. SST = (64.2667)(15-1) = 899.7

2. SSB = (5)(26.4-22.533)2 +(5)(25.6-22.533)2 + (5)(15.6-22.533)2 = 362.1

3. SSS = ((742 + 422 + 622 + 922 + 682)/3) – (3382/(5)(3)) = 441.1

4. SSE = 899.7 – 362.1 – 441.1 = 96.5

5. ANOVA Table
- Here is how we calculated the various numbers in the table:
  - df between: #groups – 1 = 3 – 1 = 2
  - df subject: #participants – 1 = 5 – 1 = 4
  - df error: df between * df subject = 2*4 = 8 
  - MS between: SSB / df between = 362.1 / 2 = 181.1
  - MS subject: SSS / df subject = 441.1 / 4 = 110.3
  - MS error: SSE / df error = 96.5 / 8 = 12.1
  - F: MS between / MS error = 181.1 / 12.1 = 15.006

6. Intrepret result
- The F test statistic for this one-way repeated measures ANOVA is 15.006. To determine if this is a statistically significant result, we must compare this to the F critical value found in the F distribution table with the following values:

  - α (significance level) = 0.05
  - DF1 (numerator degrees of freedom) = df between = 2
  - DF2 (denominator degrees of freedom) = df error = 8

We find that the F critical value is 4.459.

Since the F test statistic in the ANOVA table is greater than the F critical value in the F distribution table, we reject the null hypothesis. This means we have sufficient evidence to say that there is a statistically significant difference between the mean response times of the drugs.


## Two way ANOVA

The two-away ANOVA deals with the case where there are two variables or factors and there may be interaction effects between the two factors.

You should use a two-way ANOVA when you’d like to know how two factors affect a response variable and whether or not there is an interaction effect between the two factors on the response variable.

Source: https://www.nature.com/articles/nmeth.3180</img>

(a) The main effect is the difference between τ values (light gray), which is the response for a given level of a factor averaged over the levels of other factors. (b) The interaction effect is the difference between effects of A at the different levels of B or vice versa (dark gray, Δ). (c) Interaction effects may mask main effects.

### Hypotheses

There are three sets of hypothesis with the two-way ANOVA.
1. Main Effect - First Factor
  - **Null hypotheses(H0):** The population means of the first factor are equal. This is like the one-way ANOVA for the row factor.
  - **Alternate hypotheses(H0):** At least one population mean of the first factor is different from others.

2. Main Effect - Second Factor
  - **Null hypotheses(H0):** The population means of the second factor are equal. This is like the one-way ANOVA for the column factor.
  - **Alternate hypotheses(H0):** At least one population mean of the second factor is different from others.

3. Interaction Effect
  - **Null hypotheses(H0):** There is no interaction between the two factors. This is similar to performing a test for independence with contingency tables.
  - **Alternate hypotheses(H0):** There is some interaction between the two factors.


**Two Factor Factorial Experiment**

Each observation of group / treament can be described as:
\begin{align} x_{ijk} = (\mu + \tau_{j} + \beta_{i}) + (\tau_{j}\beta_{i}) + e_{ijk} \end{align}

where 
 - $\mu$ = overall mean of effect all samples
 - $e_{ijk}$ = random error for component, we assume random errors $e_{ijk}$ are normally distributed with mean 0 and variance $\sigma^2$
 - $\tau_{j}$ = main (treatment) effect of j-th level of factor B (column)
 - $\beta_{i}$ = main (treatment) effect of i-th level of factor A (row)
 - $(\tau_{j}\beta_{i})$ = interaction effect between the i-th level of factor A & j-th level of B

**FIXED effect model:**
  -  Specifically A levels or treatment called Subjects are choosen from all possible levels (combination) of factor A
  - Specifically B levels or treatment called Groups are choosen from all possible levels (combination) of factor B
  - Sum of main effect or interaction for each factor assumed to be zero:
\begin{align} \sum_{j=1}^{B}\tau_{j} = 0 \end{align}
\begin{align} \sum_{j=1}^{A}\beta_{i} = 0 \end{align}
\begin{align} \sum_{j=1}^{B}(\tau_{j}\beta_{i}) = 0 \end{align}
\begin{align} \sum_{j=1}^{A}(\tau_{j}\beta_{i}) = 0 \end{align}

  - Hypotheses:
   - H0: $\tau_{1} = \tau_{2} = \tau_{3}...=\tau_{B}=0$, (there is no main effect of factor B) 
   - H1: $\tau_{j} \neq 0$
   - H0: $\beta_{1} = \beta{2} = \beta_{3}...=\tau_{B}=0$, (there is no main effect of factor A) 
   - H1: $\beta_{i} \neq 0$
   - H0: $(\tau_{1}\beta_{1}) = (\tau_{1}\beta_{2}) = (\tau_{1}\beta_{3})...=(\tau_{B}\beta_{A})=0$, (there is no main effect of factor B) 
   - H1: $(\tau_{j}\beta_{i}) \neq 0$






### Variability & Expectation

#### Total variability - sum of squares identity
---
As there is only one factor, there is no interaction component

\begin{align} SST = SSB(Group) + SSB(Subject) + SS(Interaction) + SSE(Random) \end{align}

Partitioning SST:
> \begin{align} \sum_{i=1}^{A}\sum_{j=1}^{B} \sum_{k=1}^{R} (x_{ijk}-\bar{x}_{grand})^2 =
RA*\sum_{j=1}^{B}{(\overline{x}_{j} - \bar{x}_{grand})}^2
+
RB*\sum_{i=1}^{A}{(\bar{x}_{i} - \bar{x}_{grand})}^2
+
R*\sum_{i=1}^{A}\sum_{j=1}^{B}(x_{ijk}-\bar{x}_{i}-\bar{x}_{j}+\bar{x}_{grand})^2
+
\sum_{i=1}^{A} \sum_{j=1}^{B} \sum_{k=1}^{R} (x_{ijk} - \bar{x}_{ij}) \end{align}

Partitioning SST degree of fredom:
>\begin{align} ABR-1 = (B-1) + (A-1) + (A-1)(B-1) + AB(R-1) \end{align}
---

#### Variance Expectation
1. Expected Between Groups Variance
 - Sum of squares:
>\begin{align} SSB(Groups) = (B-1)\sigma^2+AR*\sum_{j=1}^{B}\tau_{j}^2\end{align}
 - Mean sum of squares):
>\begin{align} MSB(Groups) = \frac{SSB(Groups)}{B-1} = \sigma^2+\frac{AR}{(B-1)}\,*\sum_{j=1}^{B}\tau_{j}^2 = \sigma^2+AR*\sigma_{\tau}^2\end{align}
2. Expected Between Subjects/Blocks Variance
 - Sum of squares:
>\begin{align} SSB(Subject) = (A-1)\sigma^2+BR*\sum_{i=1}^{A}\beta_{i}^2\end{align}
 - Mean sum of squares):
>\begin{align} MSB(Subject) = \frac{SSB(Subject)}{A-1} = \sigma^2+\frac{BR}{(A-1)}\,*\sum_{i=1}^{A}\beta_{i}^2 = \sigma^2+BR*\sigma_{\beta}^2\end{align}
3. Expected Interaction Variance
 - Sum of squares:
>\begin{align} SS(Interaction) = (A-1)(B-1)\sigma^2+R*\sum_{j=1}^{B}\sum_{i=1}^{A}{(\beta_{ij}^2\tau_{ij}^2)}\end{align}
 - Mean sum of squares:
>\begin{align} MS(Interaction) = \frac{SS(Interaction)}{(A-1)(B-1)} = \sigma^2+\frac{R*\sum_{j=1}^{B}\sum_{i=1}^{A}{(\beta_{ij}^2\tau_{ij}^2)}}{(A-1)(B-1)}\end{align}
3. Expected Random Error Variance
 - Sum of squares:
>\begin{align} SSE(Random) = (AB)(R-1)\sigma^2\end{align}
 - Mean sum of squares:
>\begin{align} MSE(Random) = \frac{SSE(Random)}{(AB)(R-1)} = \sigma^2\end{align}

### Sum of Squares Formula

---
|Source of Variation| SS Form 1 | SS Form 2 | Degree of freedom | Notes, Comments and other details |
|---|---|---|---|---|
|$SST$|\begin{align} \sum_{i=1}^{A}\sum_{j=1}^{B} \sum_{k=1}^{R} (x_{ijk}-\bar{x}_{grand})^2 \end{align}||$(N-1)$||
|$SSB(Groups)$|\begin{align} R*A*\sum_{j=1}^{B}{(\bar{x}_{j} - \bar{x}_{grand})}^2 \end{align}||$(B-1)$||
|$SSB(Subjects)$|\begin{align} R*B*\sum_{i=1}^{A}{(\bar{x}_{i} - \bar{x}_{grand})}^2 \end{align}||$(A-1)$||
|$SS(Interaction)$|\begin{align} R*\sum_{i=1}^{A}\sum_{j=1}^{B}(x_{ijk}-\bar{x}_{i}-\bar{x}_{j}+\bar{x}_{grand})^2 \end{align}||$(A-1)*(B-1)$||
|$SSE$|\begin{align} \sum_{i=1}^{A} \sum_{j=1}^{B} \sum_{k=1}^{R} (x_{ijk} - \bar{x}_{ij})^2\end{align}||$(N-A*B)$||


### Two Way ANOVA table
| Source of Variation | Sum of Square (SS) | Degrees of Freedom Calculation (df) | Mean Square Calculation (MS) | F ratio |
| --- | --- | --- | --- | --- |
| Groups-Columns (Between) - Main Effect Factor A | SSB(Groups) | $df_{groups} = (B-1)$ | $MSB(Groups) = SSB(Groups)/df_{groups}$ | F = MSB(Groups)/MSE |
| Subject-Rows (Between) - Main Effect Factor B | SSB(Subjects) | $df_{subjects} = (A-1)$ | $MSB(Subjects) = SSB(Subjects)/df_{subjects}$ | F = MSB(Subjects)/MSE |
| Interaction Effect (Subject-Group) or (Row-Column) | SS(Interaction) | $df_{interaction} = (A-1)*(B-1)$ | $MS(Interaction) = SS(Interaction)/df_{interaction}$ | F = MS(Interaction)/MSE |
| Error (Within) or Random variation | SSE | $df_{error} = (N-A*B)$| $MSE = SSE/df_{error}$ | |
| Total | SST | $df_{total} = (N-1)$ |||

### Naming Convention 

---
- Group = Column = Treatment = Level
- Subject = Row = Block
- i = row number/subject/Block
- j = column number/group/Level
- Replication = Trials/Observations  per Cell(Row- Column combination)
- i = row number/subject
- j = column number/group
- $n_{i}$ = $B$ = total number of observations in the i-th subject/column,
- $n_{j}$ = $A$ = total number of observations in the j-th group/column,
- $A$ = total number of subjects/rows
- $B$ = total number of groups/columns
- $R$ =  Number of replications/trials
- $N = A*B*R$ = the total number of observations in the entire dataset
- $x_{ijk}$ = observation at i-th subject (row) and j-th group(column)
- $GT = \sum_{k=1}^{N}{x_{k}}$ = the grand total of the entire dataset
- $\bar{x}_{i} = \frac{1}{n_{i}}\sum_{j=1}^{n_{i}} x_{ij} = \frac{1}{B}\sum_{i=1}^{B} x_{ij}$ = the mean of the i-th subject/row, 
- $\bar{x}_{j} = \frac{1}{n_{j}}\sum_{i=1}^{n_{j}} x_{ij} = \frac{1}{A}\sum_{i=1}^{A} x_{ij}$ = the mean of the j-th group/column,
- $\bar{x}_{ij} = \frac{1}{R}\sum_{k=1}^{R} x_{ij} = \frac{1}{R}\sum_{k=1}^{R} x_{ij}$ = the mean of the cell (i-th subject/row, j-th group/column) = mean of observations at the i th level of subject(row) and the j th level of group(column)
- $\bar{x}_{grand} = \frac{1}{N}\sum_{ijk} x_{ijk}$ = the grand mean or mean of the entire dataset
- $s_{unbiased}^2$= the unbiased sample variance for the entire dataset
- $s_{i}^2$ = the unbiased sample variance for the i-th subject/row observations
- $s_{j}^2$ = the unbiased sample variance for the j-th group/column observations
---

### Steps by hand

**Example:**

| Treatment |||| group-1 | group-2 | group-3 |
| --- | --- | --- | --- | --- | --- | --- |
|||Index| i=1 | i=2 | i=3 | i=4 |
|Replication|| --- | --- | --- | --- | --- |
|k=1|subject-1|j=1| - | - | - | - |
|k=1|subject-2|j=2| - | - | - | - |
|k=1|subject-3|j=3| - | - | - | - |
|k=1|subject-4|j=4| - | - | - | - |
|k=2|subject-1|j=1| - | - | - | - |
|k=2|subject-2|j=2| - | - | - | - |
|k=2|subject-3|j=3| - | - | - | - |
|k=2|subject-4|j=4| - | - | - | - |

### Steps

1. **Total sum of squares (SST):**

Method-1
>\begin{align} SST = \sum_{i=1}^{A}\sum_{j=1}^{B} \sum_{k=1}^{R} (x_{ij}-\bar{x}_{grand})^2 \end{align}
 - Calculate the difference between each observation and the grand mean irrespective of group(column) or subject(row)
 - Square each of these differences
 - Add the values together to get SST

Method-2
> \begin{align} SST = SSB(Groups) + SSB(Subjects) + SS(Interaction) + SSW(Groups) \end{align}

2. **Between Group sum of squares SSB(Groups) - Main Effect due to Factor A:** 
- Also called SS(Group) or SS(Treatment) or SS(Model)
- variance of group/column mean vs grand mean

Method-1
>\begin{align} SSB(Groups) = R*A*\sum_{j=1}^{B}{(\bar{x}_{j} - \bar{x}_{grand})}^2 \end{align}
 - Calculate the difference between the mean of each group and the grand mean
 - Square each of these differences
 - Multiply each result by the number of observations/participants within that group
 - Add the values for each group together

3. **Between Subject sum of squares SSB(Subjects) - Main Effect due to Factor B:** 
- variance of subject/row mean vs grand mean

Method-1
>\begin{align} SSB(Subjects) = R*B*\sum_{i=1}^{A}{(\bar{x}_{i} - \bar{x}_{grand})}^2 \end{align}
 - Calculate the difference between the mean of each subject/row and the grand mean
 - Square each of these differences
 - Multiply each result by the number of observations/participants within that subject/row
 - Add the values for each group together

4. **Interaction Effect SS(AB):**
- INteraction between two factors Subject-Group or in other word row-column factor

Method-1
>\begin{align} SS(Interaction) = R*\sum_{i=1}^{A}\sum_{j=1}^{B}(x_{ij}-\bar{x}_{i}-\bar{x}_{j}+\bar{x}_{grand})^2 \end{align}
 - Add grand mean to each observation of entire dataset
 - Subtract subject/row mean and group/column mean to these updated observations 
 - Square each of these differences
 - Add the values together

5. **Error sum of squares (SSE):** 
- Within samples sum of squares due to chance errors

>\begin{align} SSE = \sum_{i=1}^{A} \sum_{j=1}^{B} \sum_{k=1}^{R} (x_{ijk} - \bar{x}_{ij})^2\end{align}

6. **Two Way ANOVA table:** Now that we have SSB, SSS, and SSE, we can fill in the two way ANOVA table.

6. **Interpret the results:** 


## References

- http://faculty.business.utsa.edu/manderso/STA4723/readings/Douglas-C.-Montgomery-Design-and-Analysis-of-Experiments-Wiley-2012.pdf
- https://www.statology.org/one-way-anova-by-hand/
- https://www.statology.org/repeated-measures-anova-by-hand/
- https://online.stat.psu.edu/stat200/lesson/10/10.2
- http://statweb.stanford.edu/~susan/courses/s141/hoanova.pdf
- http://statweb.stanford.edu/~susan/courses/s141/exanova.pdf
- http://web.pdx.edu/~newsomj/uvclass/ho_ANOVA.pdf
- http://web.pdx.edu/~newsomj/uvclass/ho_within.pdf
- http://web.pdx.edu/~newsomj/uvclass/ho_mixed.pdf
- https://www.spss-tutorials.com/anova-what-is-it/
- https://www.spss-tutorials.com/repeated-measures-anova/
- https://support.minitab.com/en-us/minitab-express/1/help-and-how-to/modeling-statistics/anova/how-to/one-way-anova/methods-and-formulas/methods-and-formulas/
- https://support.minitab.com/en-us/minitab-express/1/help-and-how-to/modeling-statistics/anova/how-to/two-way-anova/methods-and-formulas/methods-and-formulas/
- https://www.discoveringstatistics.com/statistics-hell-p/porus-comparing-means/one-way-anova/
- https://www.discoveringstatistics.com/statistics-hell-p/porus-comparing-means/repeated-measures-designs/
- https://learn.lboro.ac.uk/archive/olmp/olmp_resources/pages/workbooks_1_50_jan2008/Workbook44/44_1_one_way_anova.pdf
- https://learn.lboro.ac.uk/archive/olmp/olmp_resources/pages/workbooks_1_50_jan2008/Workbook44/44_2_two_way_anova.pdf
- https://people.richland.edu/james/lecture/m170/ch13-1wy.html
- https://people.richland.edu/james/lecture/m170/ch13-2wy.html
- http://pytolearn.csd.auth.gr/d1-hyptest/12/ptl_anova1.html
- http://pytolearn.csd.auth.gr/d1-hyptest/12/ptl_anovaR.html
- http://pytolearn.csd.auth.gr/d1-hyptest/12/ptl_anova2.html
- https://sixsigmastudyguide.com/anova-analysis-of-variation/
- https://www.alexanderdemos.org/ANOVA8.html
- https://courses.lumenlearning.com/introstats1/chapter/the-f-distribution-and-the-f-ratio/
- https://www.nature.com/articles/nmeth.3180
- http://www.stat.yale.edu/Courses/1997-98/101/expdes.htm
- https://online.stat.psu.edu/stat503

ee5ac29b91044c48d21f052e2cb2a32066702996
ipynb
Jupyter Notebook
ANOVA,_RM,_TWO_WAY,_F_test_with_code.ipynb
ram-anand/ram-anand.github.io
6a0bd2ad35c35810466eb072f81a59686ec55ba5
MIT
2020-12-26T04:58:35.000Z
2020-12-26T04:58:35.000Z
ANOVA,_RM,_TWO_WAY,_F_test_with_code.ipynb
ram-anand/ram-anand.github.io
6a0bd2ad35c35810466eb072f81a59686ec55ba5
MIT
ANOVA,_RM,_TWO_WAY,_F_test_with_code.ipynb
ram-anand/ram-anand.github.io
6a0bd2ad35c35810466eb072f81a59686ec55ba5
MIT
Qwen/Qwen-72B
1. YES
2. YES