## Linear Regression
<br/>

linear regression is a linear approach for **modelling** the relationship between a scalar **dependent variable y** and one or more **independent variables denoted X**. Using exsting data to estimate unknown data.

Linear regression model assumes that the relationship between the dependent variable **yi** and the p-vector of regressors **xi** is linear.

$$y_{i} = \beta _{0}1 + \beta _{1}x_{i1} + ... + \beta _{p}x_{ip} + \varepsilon _{i} = x_{i}^{T}\beta + \varepsilon _{i}$$

where $^{T}$ denotes the transpose, so that $x_{i}^{T}$ is the inner product between vectors $x_{i}$ and C. The equation can stacked together and written in vector form as:
$$ y = X\beta + \varepsilon $$
where,<br/>
$ y=\begin{pmatrix} y_{1}\\ y_{2}\\ ...\\ y_{n} \end{pmatrix} $ ,
$X=\begin{pmatrix} x_{1}^{T}\\  x_{2}^{T}\\ ...\\ x_{n}^{T}\end{pmatrix}
=\begin{pmatrix} 
 1 &x _{11} & ... &x _{1p} \\ 
 1& x _{21} & ... & x _{2p}\\ 
 ...& ... & ... & ...\\ 
 1&  x _{n1}& ... & x _{np}
\end{pmatrix}$ , <br/>
$\beta  = \begin{pmatrix} \beta _{0}\\ \beta _{1}\\ \beta _{2}\\ ...\\ \beta _{p} \end{pmatrix}$ , and
$\varepsilon = \begin{pmatrix} \varepsilon _{0}\\ \varepsilon _{1}\\ ...\\ \varepsilon _{n} \end{pmatrix}$

- $y_{i}$ is called dependent variable. 
- $x_{i}$ is called independent variable. 
- $\beta$ is called intercept, a (p+1)-dimension **parameter vector**, where $\beta _{0}$ is the constant (offset) term.  Its elements are also called effects, and the estimates of it are called "estimated effects" or regression coefficients. 
- $\varepsilon _{i}$ is called the _error term_ or _disturbance term_.which is diffence between estimate value $y_{i}$ and real value $x_{i}\beta$,  $$\varepsilon _{i} = y_{i} - x_{i}\beta$$
  1. they are independently
  2. they are normal distribution



#### Probability density function:
$$p(\varepsilon _{i}) = \frac{1}{\sqrt{2\pi \sigma  ^{2}}} exp(-\frac{\varepsilon _{i}^{2}}{2\sigma  ^{2}})$$
$$\Rightarrow p(y_{i}|x_{i},\beta) = \frac{1}{\sqrt{2\pi \sigma^{2}}} exp(-\frac{(y_{i}-x_{i}\beta)^{2}}{2\sigma  ^{2}})$$

#### Likelihood estimation：using data to esitimate function paramters to let argmin($\varepsilon$)
$$Likelihood(\beta ) = \prod_{i=1}^{m}p(y_{i}|x_{i},\beta) =\prod_{i=1}^{m} \frac{1}{\sqrt{2\pi \sigma^{2}}} exp(-\frac{(y_{i}-x_{i}\beta)^{2}}{2\sigma  ^{2}})$$

#### Logarithm likelihood estimation
$$log\,  L(\beta ) = log\, \prod_{i=1}^{m} \frac{1}{\sqrt{2\pi \sigma^{2}}} exp(-\frac{(y_{i}-x_{i}\beta)^{2}}{2\sigma  ^{2}})$$
$$=\sum_{i=1}^{m} log\, \frac{1}{\sqrt{2\pi \sigma^{2}}} exp(-\frac{(y_{i}-x_{i}\beta)^{2}}{2\sigma  ^{2}})$$
$$= m\, log\, \frac{1}{\sqrt{2\pi \sigma^{2}}} - \frac{1}{\sigma^{2}}\cdot \frac{1}{2}\sum_{i=1}^{m}(y_{i}-x_{i}\beta)^{2}$$

#### Least squares
$$J(\beta ) = \frac{1}{2}\sum_{i=1}^{m}(y_{i}-x_{i}\beta)^{2}$$

#### objective function

$$
\begin{equation}
\begin{split}
 J(\beta )&= \frac{1}{2}\sum_{i=1}^{m}(x_{i}\beta-y_{i})^{2}   \\
          &= \frac{1}{2}(X\beta -y)^{T}(X\beta -y) \\
          &= \frac{1}{2}(\beta ^{T}X ^{T} - y ^{T})(X\beta -y) \\
          &= \frac{1}{2}(\beta ^{T}X ^{T}X\beta -\beta ^{T}X ^{T}y- y ^{T}X\beta+y ^{T}y) \\
\end{split}
\end{equation}
$$

#### partial derivatives $\triangledown_{\beta} J(\beta)$
$$
\begin{equation}
\begin{split}
 \triangledown_{\beta } J(\beta ) &=\triangledown_{\beta } (\frac{1}{2}(\beta ^{T}X ^{T}X\beta -\beta ^{T}X ^{T}y- y ^{T}X\beta+y ^{T}y)) \\
&=  \frac{1}{2}(2X ^{T}X\beta - X ^{T}y - (y^{T}X)^{T}) \\
&= X^{T}X\beta - X^{T}y
\end{split}
\end{equation}
$$

when $\triangledown_{\beta} J(\beta) = 0$:
$$\beta = (X^{T}X)^{-1}\,X^{T}y$$

#### Evaluation:
In statistics, the coefficient of determination, denoted R2 or r2 and pronounced "R squared", is the proportion of the variance in the dependent variable that is predictable from the independent variable(s)

$$R^{2} : 1 - \frac{\sum_{i=1}^{m}(\hat{y_{i}}-y_{i})^{2}}{\sum_{i=1}^{m}(y_{i} - \overline{y})^{2}}$$

where, <br />
$\hat{y_{i}}$ is estimated y <br />
$\overline{y}$ is mean y




```python

```

d223d76f2d89cb36bb8a4f4e63835b7b419c56a9
ipynb
Jupyter Notebook
Yudi TANG/Linear Regression/Linear Regression.ipynb
JKChang2015/Machine_Learning
f8b46bf23e4d1972de6bd652dd4286e9322ed62f
Apache-2.0
2018-02-09T14:41:38.000Z
2018-02-09T14:41:38.000Z
Yudi TANG/Linear Regression/Linear Regression.ipynb
JKChang2015/Machine_Learning
f8b46bf23e4d1972de6bd652dd4286e9322ed62f
Apache-2.0
Yudi TANG/Linear Regression/Linear Regression.ipynb
JKChang2015/Machine_Learning
f8b46bf23e4d1972de6bd652dd4286e9322ed62f
Apache-2.0
2018-04-11T13:06:00.000Z
2018-04-11T13:06:00.000Z
Qwen/Qwen-72B
1. YES
2. YES