# Linear Regression

Machine learning is about using data to infer patterns.

Linear regression is the basic unit of Deep Learning. In fact personally I prefer to think of it as stacking linear regression modules.

Deep Learning has proven itself over and over again to be far superior to traditional machine learning techiques. It's strength comes from the fact that it can be scalable to large amounts of data. 

Some of the most impressive applications are in voice recognition, self driving cars and in chatbots.

If you are finding this tutorial challenging try playing around with https://www.blog.google/topics/machine-learning/now-anyone-can-explore-machine-learning-no-coding-required/ (and make sure you watch the video as well) . For a bit more mathematical take, try reading: https://ml.berkeley.edu/blog/2016/11/06/tutorial-1/. Also watch the 


```python
from IPython.display import YouTubeVideo
YouTubeVideo("jpmGxOB2YPo")
```











```python
import numpy as np
import matplotlib.pyplot as plt

%matplotlib inline
```

Consider the line $y=2x+3$. See below for a light introduction to plotting and numpy arrays. Numpy arrays are really important to understand. If you have experience with Matlab once you take out the `np.` part it should be familiar to the relevant Matlab function.


```python
x = np.linspace(-5,5,num=100)
y = 2*x+3

plt.plot(x,y)
plt.show()
```

This is how you define a function in Python. Remember to return the value.


```python
def f(w,b):
    """
    We will limit ourselves to a linear function
    """
    return w*x+b

def loss_function(w,b):
    """
    Loss function (squared error)
    Note that w and b are the ONLY variables
    """
    e = y - f(w,b)
    return np.mean(np.square(e))
```

## Find the function

Given the observation $x,y$ the goal is to find the best parameters, $w,b$ that fits the data. Ignore the fact that you can solve this as an equation. Most of the time the data isn't as linear or noiseless as what is being presented here.

The most naiive way of finding $w$ and $b$ is to do a grid search over $w$ and $b$ for the smallest loss function value. In this case the loss function is: $$\frac{1}{N}\sum_{i=1}^N \xi_i^2$$ where, $\xi_i = y_i - f(x_i|w,b)$ and $f(x_i|w,b)=wx_i+b$.

Another way of stating the above is that we need to find the best fitting function $f(x_i|w,b)$ such that we minimising the __loss function__ by varying $w$ and $b$. An illustration is shown below:


```python
y
```




    array([-7.        , -6.7979798 , -6.5959596 , -6.39393939, -6.19191919,
           -5.98989899, -5.78787879, -5.58585859, -5.38383838, -5.18181818,
           -4.97979798, -4.77777778, -4.57575758, -4.37373737, -4.17171717,
           -3.96969697, -3.76767677, -3.56565657, -3.36363636, -3.16161616,
           -2.95959596, -2.75757576, -2.55555556, -2.35353535, -2.15151515,
           -1.94949495, -1.74747475, -1.54545455, -1.34343434, -1.14141414,
           -0.93939394, -0.73737374, -0.53535354, -0.33333333, -0.13131313,
            0.07070707,  0.27272727,  0.47474747,  0.67676768,  0.87878788,
            1.08080808,  1.28282828,  1.48484848,  1.68686869,  1.88888889,
            2.09090909,  2.29292929,  2.49494949,  2.6969697 ,  2.8989899 ,
            3.1010101 ,  3.3030303 ,  3.50505051,  3.70707071,  3.90909091,
            4.11111111,  4.31313131,  4.51515152,  4.71717172,  4.91919192,
            5.12121212,  5.32323232,  5.52525253,  5.72727273,  5.92929293,
            6.13131313,  6.33333333,  6.53535354,  6.73737374,  6.93939394,
            7.14141414,  7.34343434,  7.54545455,  7.74747475,  7.94949495,
            8.15151515,  8.35353535,  8.55555556,  8.75757576,  8.95959596,
            9.16161616,  9.36363636,  9.56565657,  9.76767677,  9.96969697,
           10.17171717, 10.37373737, 10.57575758, 10.77777778, 10.97979798,
           11.18181818, 11.38383838, 11.58585859, 11.78787879, 11.98989899,
           12.19191919, 12.39393939, 12.5959596 , 12.7979798 , 13.        ])




```python
x
```




    array([-5.        , -4.8989899 , -4.7979798 , -4.6969697 , -4.5959596 ,
           -4.49494949, -4.39393939, -4.29292929, -4.19191919, -4.09090909,
           -3.98989899, -3.88888889, -3.78787879, -3.68686869, -3.58585859,
           -3.48484848, -3.38383838, -3.28282828, -3.18181818, -3.08080808,
           -2.97979798, -2.87878788, -2.77777778, -2.67676768, -2.57575758,
           -2.47474747, -2.37373737, -2.27272727, -2.17171717, -2.07070707,
           -1.96969697, -1.86868687, -1.76767677, -1.66666667, -1.56565657,
           -1.46464646, -1.36363636, -1.26262626, -1.16161616, -1.06060606,
           -0.95959596, -0.85858586, -0.75757576, -0.65656566, -0.55555556,
           -0.45454545, -0.35353535, -0.25252525, -0.15151515, -0.05050505,
            0.05050505,  0.15151515,  0.25252525,  0.35353535,  0.45454545,
            0.55555556,  0.65656566,  0.75757576,  0.85858586,  0.95959596,
            1.06060606,  1.16161616,  1.26262626,  1.36363636,  1.46464646,
            1.56565657,  1.66666667,  1.76767677,  1.86868687,  1.96969697,
            2.07070707,  2.17171717,  2.27272727,  2.37373737,  2.47474747,
            2.57575758,  2.67676768,  2.77777778,  2.87878788,  2.97979798,
            3.08080808,  3.18181818,  3.28282828,  3.38383838,  3.48484848,
            3.58585859,  3.68686869,  3.78787879,  3.88888889,  3.98989899,
            4.09090909,  4.19191919,  4.29292929,  4.39393939,  4.49494949,
            4.5959596 ,  4.6969697 ,  4.7979798 ,  4.8989899 ,  5.        ])




```python
loss_function(2,2)
```




    1.0




```python
loss_function(2,3)
```




    0.0




```python
w_try, b_try = np.meshgrid(np.linspace(-5,5), np.linspace(-5,5))
```


```python
w_try
```




    array([[-5.        , -4.79591837, -4.59183673, ...,  4.59183673,
             4.79591837,  5.        ],
           [-5.        , -4.79591837, -4.59183673, ...,  4.59183673,
             4.79591837,  5.        ],
           [-5.        , -4.79591837, -4.59183673, ...,  4.59183673,
             4.79591837,  5.        ],
           ...,
           [-5.        , -4.79591837, -4.59183673, ...,  4.59183673,
             4.79591837,  5.        ],
           [-5.        , -4.79591837, -4.59183673, ...,  4.59183673,
             4.79591837,  5.        ],
           [-5.        , -4.79591837, -4.59183673, ...,  4.59183673,
             4.79591837,  5.        ]])




```python
b_try
```




    array([[-5.        , -5.        , -5.        , ..., -5.        ,
            -5.        , -5.        ],
           [-4.79591837, -4.79591837, -4.79591837, ..., -4.79591837,
            -4.79591837, -4.79591837],
           [-4.59183673, -4.59183673, -4.59183673, ..., -4.59183673,
            -4.59183673, -4.59183673],
           ...,
           [ 4.59183673,  4.59183673,  4.59183673, ...,  4.59183673,
             4.59183673,  4.59183673],
           [ 4.79591837,  4.79591837,  4.79591837, ...,  4.79591837,
             4.79591837,  4.79591837],
           [ 5.        ,  5.        ,  5.        , ...,  5.        ,
             5.        ,  5.        ]])




```python
w_try = np.linspace(-4,4,num=100)
b_try = np.linspace(-4,4,num=100)

w_try, b_try = np.meshgrid(w_try,b_try)
e_obs = np.zeros((100,100))

for i in range(100):
    for j in range(100):
        e_obs[i,j] = loss_function(w_try[i,j],b_try[i,j])
        
plt.pcolormesh(w_try, b_try, np.log(e_obs)) #using log here helps visualise better, you may take it out
plt.xlabel('w')
plt.ylabel('b')
plt.title('Mean Squared Error')
plt.show()        
```

You can see that the error in the plot is minimised at $w=2$ and $b=3$.


```python
e_obs.max()
```




    355.060606060606




```python
e_obs.min()
```




    0.0043879944910451526



Algorithms try to be a bit smarter by following the gradients to find the minimum. Remember we can only influence the error via the weights (and bias) `w,b`.

## Maths (optional - no really)
\begin{align}
\mathcal{L} & = \sum_{i=1}^N (y_i-f(x_i,w,b))^2 \\
\frac{\delta\mathcal{L}}{\delta w} & = \sum_{i=1}^N 2(y_i-f(x_i,w,b))\frac{\delta f(x_i,w,b)}{\delta w} \\ 
& = \sum_{i=1}^N 2\xi_i\frac{\delta f(x_i,w,b)}{\delta w}
\end{align}
where $\xi_i$ is the error term $y_i-f(x_i,w,b)$ and 
$$
\frac{\delta f(x_i,w,b)}{\delta w} = x_i
$$

Similar expression can be found for $\frac{\delta\mathcal{L}}{\delta b}$ (exercise).

Finally the weights can be updated as $w_{new} = w_{current} - \gamma \frac{\delta\mathcal{L}}{\delta w}$ where $\gamma$ is a learning rate between 0 and 1.


```python
from IPython.display import YouTubeVideo
YouTubeVideo("EvnpoUTXA0E")
```











```python
from sklearn.linear_model import LinearRegression
```

Scikit learn is another Machine Learning that is quite powerful on itself. However it may not scale to millions of data points same way that DL does.

## Noisy Data
We see how to find the weights using scikit learn below. However, the data isn't as perfect as before. We add a tiny bit of noise.


```python
x = np.random.randn(10,1)
y = 2*x+3 + 0.1*np.random.randn(10,1)

plt.scatter(x,y)
plt.show()
```


```python
x
```




    array([[-0.71357596],
           [ 0.9140161 ],
           [-1.32380078],
           [ 0.28687436],
           [-1.34186068],
           [-0.49450964],
           [ 0.04626576],
           [ 0.5210283 ],
           [-0.81445593],
           [-0.2100118 ]])




```python
model = LinearRegression() # 
model.fit(x,y)
```




    LinearRegression(copy_X=True, fit_intercept=True, n_jobs=None, normalize=False)




```python
model.coef_
```




    array([[2.01513777]])




```python
model.intercept_
```




    array([2.97219557])




```python
temp = np.array(20)
temp = temp.reshape(1,-1) 

model.predict(temp)
```




    array([[43.27495091]])



## Note:
1. `model.fit(...)` and `model.predict` will be two of the most important functions you will use all throughout. Worth remembering!


```python
2*20+3
```




    43




```python
x_test = np.linspace(-3,3)
y_pred = model.predict(x_test[:,None])

plt.scatter(x,y)
plt.plot(x_test,y_pred,'r')
plt.legend(['Predicted line','Observed data'])
plt.show()
```

## A more complicated function
Remember that when we say linear regression it mean linear in the parmaters $w, b$ not $x$! See illustration below:

\begin{align}
y = b +w^T X \\
X = [x\quad x^2\quad x^3] \\
y = b +wx +w_2 x^2 +w_3 x^3 \\
\end{align}
The shapes of the matrices are:
- X = NxD matrix
- y = Nx1 vector
- w = Dx1 vector

Note in the example below that there is no $x^2$ term. i.e. it's coefficient is zero!


```python
x = np.linspace(-5,5,num=100)[:,None]
y = -0.5 + 2.2*x +0.3*x**3+ 2*np.random.randn(100,1)

plt.plot(x,y)
plt.show()
```


```python
x.shape
```




    (100, 1)




```python
x_new = np.hstack([x,x**2,x**3,x**4])
x_new.shape
```




    (100, 4)




```python
x.shape
```




    (100, 1)




```python
model = LinearRegression()
model.fit(x_new,y)
```




    LinearRegression(copy_X=True, fit_intercept=True, n_jobs=None, normalize=False)




```python
model.coef_
```




    array([[ 2.1626707 , -0.1010794 ,  0.3011717 ,  0.00270179]])



## Note:
- The coefficient of $x^2$ and $x^4$ is really close to zero.


```python
model.intercept_
```




    array([0.03650868])




```python
y_pred = model.predict(x_new)

plt.scatter(x,y)
plt.plot(x_new[:,0],y_pred,'r')
plt.legend(['Predicted line','Observed data'])
plt.show()
```


```python
from sklearn.preprocessing import PolynomialFeatures
poly = PolynomialFeatures(degree=4,include_bias=False)
x_new2 = poly.fit_transform(x)
x_new2[:2]
```




    array([[  -5.        ,   25.        , -125.        ,  625.        ],
           [  -4.8989899 ,   24.00010203, -117.57625742,  576.00489747]])




```python
x_new.shape
```




    (100, 4)




```python
x_new[:2]
```




    array([[  -5.        ,   25.        , -125.        ,  625.        ],
           [  -4.8989899 ,   24.00010203, -117.57625742,  576.00489747]])




```python

```

ed7da3a09e030918b1cde965912ba1d869dd41b6
ipynb
Jupyter Notebook
jupyter/Keras_TensorFlow_Course/Lesson 0 - LinRegression.ipynb
multivacplatform/multivac-dl
54cb33960ba14f32ed9ac185a4c151a6b72a97ca
MIT
2018-11-24T10:47:49.000Z
2018-11-24T10:47:49.000Z
jupyter/Keras_TensorFlow_Course/Lesson 0 - LinRegression.ipynb
multivacplatform/multivac-dl
54cb33960ba14f32ed9ac185a4c151a6b72a97ca
MIT
jupyter/Keras_TensorFlow_Course/Lesson 0 - LinRegression.ipynb
multivacplatform/multivac-dl
54cb33960ba14f32ed9ac185a4c151a6b72a97ca
MIT
Qwen/Qwen-72B
1. YES
2. YES