# $\chi^2_\nu$ (reduced chi-squared)
## Week 4, Intro-to-Astro 2020

This week we learned about the $\chi^2$ metric and how (under certain assumptions) minimizing $\chi^2$ with respect to a certain parameter of interest can produce the best-estimate of the true value for that parameter, based on our data.

As part of your assignments this week, we'll learn about the "reduced $\chi^2$" metric (written as $\chi^2_\nu$), which is used for another common problem in data analysis: *model selection*. 

Complete this notebook by filling in the missing code snippets and answering questions when prompted.

**Assignment outline**

0. What's $\chi^2$ again?
1. What's $\chi^2_\nu$?
2. Goodness of fit with $\chi^2_\nu$
3. A cautionary tale

### Written by Joey Murphy, June 2020

### Adapted from lecture notes by Ian Czekala

## 0. $\chi^2$ refresher

In this week's tutorial on $\chi^2$ minimization, we estimated the true length of a house key based on a collection of measurements. We made assumptions about the data:

- the measurement uncertainty was the same for each data point
- the data was "drawn" from a Gaussian distribution
- each measurement was independent

In general, each of these assumptions can eventually be violated, e.g. each measurement could have a different measurement uncertainty, the data may not actually follow a Gaussian distribution (think photon counting), and data points may have some covariance (not independent measurements). For now we'll hold on to these assumptions to keep things simple, but it's always a good idea to be cognizant of the validity of the assumptions that go into your analysis.

**Question**:
What are some examples of how the three assumptions above could be violated when analyzing real astronomical data?

...

- Measurement Uncertainity can be different for different transit events discovered by TESS due to other factors such as Solar Spots, Gasses in the protoplanetary disk of the star surrounding the plannet, stellar dimming due to factors other than the planet transit (as seen in the case of Betelguese which, according to latest suggestions, accretes or barps out huge amount of gases which covers a part of the star, dimming its observable flux.
- Data may be drawn from other observations - Due to Red/Brown noise which occurs due to Brownian motion. Also, for a spatially and temporally coherent electromagnetic wave of a single frequency, the [photon counting](https://en.wikipedia.org/wiki/Photon_counting) also represents a Poisson distribution (Binomial distribution -> with probability of success, $p \rightarrow 0$ and $Np \rightarrow \mu$ (N being the number of total observations and $\mu$ being the mean), becomes the Poisson distribution, which, with a large $\mu$, becomes Gaussian distribution. 
- _(From Key)_ Nearby data points in a spectrum may have some covariance, due to wavelength-dependent sensitivity in the CCD (which is the Charged Couple Device- the array of pMOS capacitors to store the incident light as pixels in digital imaging)

...

Summarizing the rest of what we did...

We calcualted the log-likelihood of our data

\begin{align}
\mathrm{ln} \mathcal{L} & \propto \Sigma_i^N \Big[-\frac{1}{2} \big(\frac{x_i-\mu}{\sigma}\big)^2  \Big],
\end{align}

and defined the $\chi^2$ statistic as 

\begin{align}
\chi^2 & \equiv \Sigma_i^N \big(\frac{x_i - \mu}{\sigma}\big)^2,
\end{align}

giving us

\begin{align}
\mathrm{ln} \mathcal{L} & \propto -\frac{1}{2} \Sigma_i^N \big(\frac{x_i - \mu}{\sigma}\big)^2 \\
& \propto -\frac{1}{2} \chi^2.
\end{align}

We calculated our best estimate of the value of $\mu$ by minimizing the $\chi^2$ metric, finding 

\begin{align}
\hat{\mu} = \frac{1}{N} \Sigma_i^N x_i.
\end{align}

As a bonus, if we examine the width of the likelihood function at $\mu = \hat{\mu}$, we can show that the *uncertainty* on our *estimate* of the true length of the key is 

\begin{align}
\hat{\sigma} = \frac{\sigma}{\sqrt{N}},
\end{align}

where $\sigma$ is the measurement uncertainty for a single data point and $N$ is the number of data points in our sample. Our final estimate of the key's length is then 

\begin{align}
\mu = \hat{\mu} \pm \frac{\sigma}{\sqrt{N}}. 
\end{align}

*Upshot:* Our uncertainty on the estimate of the parameter's true value goes down as the square root of the number of observations. Want a more precise measurement? Get more data!

# 1. $\chi^2_\nu$ (reduced chi-squared)

What was the "model" that we used to describe our data in the previous example? 

A very simple one! Basically, a flat line. That model had a single parameter that we "fit" to the data. In general, you'll run into real datasets that require more complicated models (i.e. they contain more parameters that need fitting).

These models could be 

* a line: $\mu = m x + b$
* a quadratic function: $\mu = a x^2 + b x + c$
* a sinusoid: $\mu = A \sin(t) + B \cos(t)$
* a damped harmonic oscillator: $\mu = A e^{-t/10}\sin(5 t)$
* etc...

**Question**: What are some other physical models that you can think of?

...

- The curve of stellar flux of a planet (the transit phenomena in the lightcurve)
- The distribution of Mass and Radius of planets

...

For all of these models, good ol' $\chi^2$ still applies:

\begin{align}
\chi^2 & = \Sigma_i^N \big(\frac{x_i - \mu(\theta)}{\sigma_i}\big)^2,
\end{align}

where $\theta$ represents the model parameters.

_here $\sigma_i$ represents that there are different standard deviations/uncertainities for different observations_

We made no mention of whether or not the model that we chose was a good representation of the underlying physical process (in our case, it wasn't like the key we measured was growing or shrinking, so a constant value was actually a pretty good choice). In general, though, how do we determine which model to use to describe our data?

The $\chi^2_\nu$ statistic helps us with just that. 

\begin{align}
\chi^2_\nu & \equiv \frac{\chi^2}{\nu},
\end{align}

where $\nu = N - m$ is referred to as the number of "degrees of freedom". $N$ is the number of data points and $m$ is the number of parameters being fit in your model. For our "model" of the length of the house key, there was a single parameter being fit: $\hat{\mu}$, our estimate of the key's true length.


**Question**: Why is $\nu = N - m$ called the number of "degrees of freedom" for our model? Give your best guess!

*Hint: If I fit a model with 5 parameters to a sample with only 5 data points, what happens?*

...

Because it defines the number of ways/dimansions in which the process that data is being retreived from can change.
If we fit a model with 5 params to a sample data with 5 data points, it can caoture all these individual degrees of freedom (N-m, N and m being 5) 

...

## 2. Goodness of fit with $\chi^2_\nu$

So how do we actually use $\chi^2_\nu$ to assess whether or not the model we chose is good at describing our data?

### Goodness of fit rules of thumb

$\chi^2_\nu \gg 1 \rightarrow$ The model is inadequate (i.e. not enough parameters), or data uncertainties are underestimated

$\chi^2_\nu \approx 1 \rightarrow$ The model does a pretty good job of fitting the data

$\chi^2_\nu \ll 1 \rightarrow$ The model is too flexible (i.e. too many parameters), or data uncertainties are overestimated

**Question**: Can you try to explain why $\chi^2_\nu \gg 1$ suggests the model is too simplistic, and why $\chi^2_\nu \ll 1$ suggests the model is too complex? Again, give it your best guess!

*Hint: Look back at our definition of $\chi^2$. It's the sum of the squares of the differences between the data and the model, in units of the measurement uncertainty for each data point. Does $\chi^2_\nu$ look like an average of sorts, now?*

...

If $\chi^2_\nu$ is close to one, that means that the chi-squared (which is directly propotional to negative of log likelihood), is balanced. This is because $\chi_2$ is defined as the sum of square of differnces of data model from the mean divided by their variances, in other words, a value closer to 0 represents a better fir of the data model to the actual value. Since we are talking about average when we are talking about $\chi^2_\nu$, a value closer to 1 means a close fit of data model to the actual value (means) about the $\sigma^2$

$\chi^2_\nu \gg 1$ indicates that the data model is too far off the actual value as compared to the variance. It could be either due to underestimation of uncertainities of the data ($\sigma_i$2 being larger than what is estimated) or because the degrees of freedom of the data model are too low as compared to the actual data's degrees of freedom (due to less number of parameters being used to define the data model)

$\chi^2_\nu \ll 1$ indicates over fitting of the data. It may be because we have fed too many parameters in our data model and the actual degree of freedom of the data is smaller or because the $\sigma_i^2$ is estimated larger than it actually is
...

Now that we know how to use $\chi^2_\nu$ as a way to assess the "goodness of fit" for our model to data, let's revisit our experiment of measuring the length of the house key. After we create some "fake" data again, we'll compute $\chi^2_\nu$ and see how well we did with model selection.


```python
import numpy as np

# These lines copied from chi_squared_tutorial.ipynb

# Generate the measurement data
mu_key    = 6.0  # cm. The *actual* length of the key (unknown to us)
sigma_key = 0.05 # cm. Uncertainty on each measurement, based on the tick marks on our ruler
n_measure = 100  # Take 100 measurements
X = np.random.normal(mu_key, sigma_key, n_measure) # Generate the "measurements"- We assume a Gaussian distribution again
```


```python
# These lines copied from chi_squared_tutorial.ipynb 

from scipy.optimize import minimize

def chi_squared(mu, sigma, data):
    '''
    Args
    ----------
    mu (float): Underlying mean of distribution (unknown to us).
    sigma (float): Measurement error.
    data (ndarray): Vector of observed data
    
    Returns
    ----------
    float: chi-squared value
    '''
    return np.sum(((data - mu)/sigma)**2)

mu_0 = 4.0 # cm. Our initial guess at the true length of the key, just from eye-balling it.
#print(chi_squared(mu_0, sigma_key, X))

min_results = minimize(chi_squared, mu_0, args=(sigma_key, X)) # Minimize chi-squared with respect to mu

print(f"Best-fitting value of mu_key from chi-squared minimization = {min_results.x[0]:.3f} cm")


##############################################
############ Fill in code here ###############
##############################################



# min_chi_squared_val = ... # Look at the documentation for scipy.optimize.minimize
#                           # (https://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.minimize.html)
#                           # to see how to retrieve the minimum chi-squared value from the min_results object.
#Way 1 to show everything
min_results = minimize(chi_squared, mu_0, args=(sigma_key, X), options={'disp':True})
#Way 2 
min_chi_squared_val = min_results.fun
print(f"Minimum chi-squared = {min_chi_squared_val:.3f} ")
```

    Best-fitting value of mu_key from chi-squared minimization = 5.996 cm
    Optimization terminated successfully.
             Current function value: 101.461402
             Iterations: 3
             Function evaluations: 8
             Gradient evaluations: 4
    Minimum chi-squared = 101.461 



```python
def reduced_chi_squared(min_chi_squared, N, m):
    '''
    Args
    ----------
    min_chi_squared (float): Minimum chi-squared value
    N (int): Number of data points
    m (int): Number of parameters fit by model.
    
    Returns
    ----------
    float: reduced chi-squared metric to assess goodness of fit.
    '''
    ##############################################
    ############ Fill in code here ###############
    ##############################################
    return min_chi_squared/(N-m)

reduced_chi_sq_val = reduced_chi_squared(min_chi_squared_val, n_measure, 1)
print(f"Reduced chi-squared = {reduced_chi_sq_val:.3f}")
```

    Reduced chi-squared = 1.025


**Question**: Did our model do a good job of describing the data? Why or why not?


*Enter your answer here*

Yes, because the reduced Chi-squared value, $\chi^2_\nu \approx 1$, which represents a value which is close to actual value



## 3. A cautionary tale

If you've made it this far, you've done *a lot* of reading about measuring the length of some silly key. Where's the astronomy?!

For the last part of this assignment, read [Gould 2013](https://arxiv.org/abs/1303.0834) (< 5 pages). Don't worry if you don't understand some bits of jargon here and there—just do your best! Regardless of the way in which this author went about making their point, they highlight a very important (and potentially costly) distinction between $\chi^2$ and $\chi^2_\nu$.

When you're done reading through the paper, come back to the assignment here.

Note that we didn't do any *minimizing* when it came to $\chi^2_\nu$, we just used it as a way to estimate whether or not our model was doing a good job at describing the data we observed. $\chi^2$ is the one that's actually related to the likelihood of the data, so it's the metric that we should be worried about when it comes to the optimization of our parameters.

**Question**: What did you think about the Gould paper? Can you write a one or two sentence summary of what it was about?

...

Gould's paper discusses the case of mass calculation of an Extra-galactic blackhole from David et al(2013), who used the minimization with dof (degrees of freedom) over $\chi^2$, or $\chi_\nu^2$, which resulted in a very large uncertainity in the output. Gould argues that the minimization should be over $\chi^2$ as the likelihood is dependent only on it and not on dofs. He further shows a 7 factor decrease of uncertainity in David et al. (2013)'s mass calculation, making it the most precise (more focus as compared to accuracy here) extra-galactic black holes

...

3bb9eada7fec74a1ad10eb7d44ef21cdca710754
ipynb
Jupyter Notebook
Week4_tess_chi_squared/chi_squared_assignment.ipynb
chetanchawla/Intro-to-Astro-2020
3ec9a7b108477d552530083df8701724e3e0e31f
MIT
Week4_tess_chi_squared/chi_squared_assignment.ipynb
chetanchawla/Intro-to-Astro-2020
3ec9a7b108477d552530083df8701724e3e0e31f
MIT
Week4_tess_chi_squared/chi_squared_assignment.ipynb
chetanchawla/Intro-to-Astro-2020
3ec9a7b108477d552530083df8701724e3e0e31f
MIT
Qwen/Qwen-72B
1. YES
2. YES