## Note for Linear Regression and Data Analysis - Zhibo (Darren) Zhang
      (Partially refer from STA302 University of Toronto, prof Mark Ebden)

Given a set of data points, we want to construct a SLR (simple linear regression) model for the data of the form $y = \beta_0 + \beta_1 x$. 

Note that we assume that there is such a model that perfectly fits the data ($y=\beta_0 + \beta_1 x + e$, where $e$ is the random error term), but we never know the actual model. So we want to use some method to estimate it. To be specific, given $x$, we want to use parameters to estimate the prediction output, denoted as $\hat y$. 

In this way, if we can estimate the "tool" for the model, which is the parameter $\beta_1$ and $\beta_2$, then we can use the estimator (linear function) to make predctions.

Denote the estimated parameters as $\hat \beta_0$ and $\hat \beta_1$ accordingly. The next step is to get an estimation equation for these two parameters. 

$\bullet$ Given a set of training examples of size N, $\{x_1, x_2, ..., x_N\}$ and its according labels $\{y_1, y_2, ..., y_N\}$, denote an input matrix $\mathbf{X}$ to be:

$$ \mathbf{X} =
\begin{bmatrix}
    1 & x_1 \\
    1 & x_2  \\
    \vdots & \vdots \\
    1 & x_N
\end{bmatrix}
$$
where $\mathbf{X}$ is of size N $\times$ 2, denote the parameter vector $\boldsymbol{\hat \beta}$ to be:

$$\boldsymbol{\hat\beta} =
\begin{bmatrix}
    \hat\beta_0 \\
    \hat\beta_1
\end{bmatrix}
$$

$\bullet$ To denote the residual terms (the difference between the actual value and the prediction output, which can be regarded as an estimation for the random error), denote $ \mathbf{\hat e}$ and label vector $\mathbf{Y}$to be:
$$ \mathbf{\hat e} = 
\begin{bmatrix}
    \hat e_1 \\
    \hat e_2\\
    \vdots \\
    \hat e_N
\end{bmatrix}, \quad \mathbf{Y} = 
\begin{bmatrix}
    y_1\\
    y_2 \\
    \vdots \\
    y_N
\end{bmatrix}
$$

In this way, the equation becomes $\mathbf{Y} = \mathbf{X} \boldsymbol{\hat\beta} + \mathbf{\hat e}$

Our goal is to minimize the sum of residuals, which is the sum of residual vector, so called RSS (Residual Sum of Squares). Thus, denote an equation for RSS:
$$
\begin{equation}
\begin{split}
RSS &= (\mathbf{Y} - \mathbf{X} \boldsymbol{\hat\beta})'(\mathbf{Y} - \mathbf{X} \boldsymbol{\hat\beta})\\
      &=  (\mathbf{Y}' - (\mathbf{X} \boldsymbol{\hat\beta})')(\mathbf{Y} - \mathbf{X} \boldsymbol{\hat\beta})\\
      &=  (\mathbf{Y}' - \boldsymbol{\hat\beta}'\mathbf{X}')(\mathbf{Y} - \mathbf{X} \boldsymbol{\hat\beta})\\
      &=  \mathbf{Y}'\mathbf{Y} - \mathbf{Y}'\mathbf{X} \boldsymbol{\hat\beta} - \boldsymbol{\hat\beta}'\mathbf{X}'\mathbf{Y} + \boldsymbol{\hat\beta}'\mathbf{X}'\mathbf{X} \boldsymbol{\hat\beta}\\
      &=  \mathbf{Y}'\mathbf{Y} - 2\boldsymbol{\hat\beta}'\mathbf{X}'\mathbf{Y} + \boldsymbol{\hat\beta}'\mathbf{X}'\mathbf{X} \boldsymbol{\hat\beta}
\end{split}
\end{equation}
$$

In order to minimize the above equation, take the derivative with respect to $\boldsymbol{\hat\beta}$ and set it to zero:

$$
\frac{\partial RSS}{\partial \boldsymbol{\hat\beta}} = - 2\mathbf{X}'\mathbf{Y} + 2\mathbf{X}'\mathbf{X} \boldsymbol{\hat\beta} = 0
$$

Therefore, we get the estimation for the parameter matrix:

$$
 \boldsymbol{\hat\beta} = (\mathbf{X}'\mathbf{X})^{-1}\mathbf{X}'\mathbf{Y}
$$

_I wil realize this process using python as the next step_


```python

```

7cacfd133b8c04622759fc35a6e7254071a9e442
ipynb
Jupyter Notebook
linear_regression/linear_regression.ipynb
DarrenZhang01/Machine-Learning
f111b430330ac38d5dd711033cf068eceaf3600c
MIT
2019-01-25T19:39:36.000Z
2019-01-25T19:39:44.000Z
linear_regression/linear_regression.ipynb
DarrenZhang01/Machine-Learning
f111b430330ac38d5dd711033cf068eceaf3600c
MIT
2018-08-14T14:18:28.000Z
2018-08-18T20:59:01.000Z
linear_regression/linear_regression.ipynb
DarrenZhang01/Machine-Learning
f111b430330ac38d5dd711033cf068eceaf3600c
MIT
Qwen/Qwen-72B
1. YES
2. YES