<a href="https://colab.research.google.com/github/sudheer/blog/blob/master/_notebooks/2020_04_12_interactive_and_practical_linear_regression_using_python.ipynb" target="_parent"></a>

# "Interactive and visualize linear regression in python"
> "Interactive and visualize linear regression in python"

- toc:true
- branch: master
- badges: true
- comments: true
- categories: [ml]
- tags: [jupyter, linear regression, regression]

Linear regression assumes relation between input variable and output. It predicts the output as a linear function from input variables.

Various methods of Linear Regression 
- **Simple Linear Regression**
 - Requires to calculate mean, std-deviation, co-variance, corelation 
- **Ordinary Least Squares**
 - uses Linear Algebra to estimate best values for co-efficients
- **Gradient Descent**
 - Iteratively miniize the error of model this it gets to a minimum
   - Batch Gradient descent
   - Scochastic gradient descent
   - mini batch gradient descent
- **Regularized Linear Regression**
 - Lasso Regression 
 - Ridge Regression


# Introduction and Terminology

Lets say we have below sample data. 

| size in feet | price ($) |
| ------  | ----- |
| 2104 | 460 |
| 1416 | 232 |

$m$ = number of training example

$x$ = input variable/ feature/ independent variable/ predictor

$y$ = output variable/ target variable/ dependent variable



How do we represent h ?

**Hypothesis**:
$h_\theta(x) = \theta_0 + \theta_1 * x$

$h(x)$ is predictor of $y$

This is linear regression with one variable

Let's express this as Matrix and vectors

$h_\theta(x) = \begin{pmatrix} x \\ 1\end{pmatrix}^T * \begin{pmatrix} \theta_1 \\ \theta_0\end{pmatrix}$

Let $ X = \begin{pmatrix} x \\ 1\end{pmatrix}^T$ and $\theta = \begin{pmatrix} \theta_1 \\ \theta_0\end{pmatrix}$

we can write it as $ h_\theta(x) = X * \theta $

We need to choose $\theta_0$ and $\theta_1$ so it best fits our function

# Simple Linear Regression

- uses stats to estimate co-efficients
- need to calculate mean, std-devitation, co-variance and corelation.

Lets generate some data for simple linear regression. Lets take house price prediction example

Linear equation for our calculations is $y = \theta_1 * x + \theta_0$

**Steps**:

- **Estimate the slope $\theta_1$**

\begin{equation}
\\\theta_1 = \frac{\sum(x_i – mean(x)) (y_i – mean(y)} {\sum(x_i – mean(x))^2}
\end{equation}

where 

\begin{equation}
mean(x) = \frac{1}{n}\sum_{i=1}^n x_i
\end{equation}

- **Estimate the intercept $\theta_0$**

\begin{equation}
\theta_0 = mean(y) – \theta_1 * mean(x)
\end{equation}

- **Estimating Error**

Root Mean Square Error 

$RMSE = \sqrt{(\frac{1}{n})\sum_{i=1}^{n}(h_{i} - y_{i})^{2}}$

$h_i$ is predicted value and $y_i$ is actual value



```
# imports
import pandas as pd
import numpy as np
from matplotlib import pyplot as plt
```

### Generate Data


```
# Generate 'random' data
np.random.seed(0)
no_of_records=200
sq_ft = 50 * np.random.rand(no_of_records) + 500  # Array of 100 values with mean = 1000
res = 5 * np.random.randn(no_of_records)       # Generate 100 residual terms
price = 2 * sq_ft + 1000 + res                  # Actual values of Y

# create pandas dataframe to store size and price
df = pd.DataFrame(
    {'sq_ft': sq_ft,
     'price': price}
)

# first five rows of our dataframe
df.head()
```




<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>sq_ft</th>
      <th>price</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>527.440675</td>
      <td>2060.514530</td>
    </tr>
    <tr>
      <th>1</th>
      <td>535.759468</td>
      <td>2066.119279</td>
    </tr>
    <tr>
      <th>2</th>
      <td>530.138169</td>
      <td>2054.538994</td>
    </tr>
    <tr>
      <th>3</th>
      <td>527.244159</td>
      <td>2052.299218</td>
    </tr>
    <tr>
      <th>4</th>
      <td>521.182740</td>
      <td>2039.875318</td>
    </tr>
  </tbody>
</table>
</div>



### Visualize Data


```
X = df.iloc[:, 0]
Y = df.iloc[:, 1]
plt.title('House Price vs Sq ft') 
plt.xlabel('Sq ft')  
plt.ylabel('House Price')  
plt.scatter(X,Y)
```

### Build the model

Lets calculate

\begin{equation}
\\m = \frac{\sum(x_i – mean(x)) (y_i – mean(y)} {\sum(x_i – mean(x))^2}
\end{equation}

\begin{equation}
c = mean(y) – \theta_1 * mean(x)
\end{equation}


```
# Building the model
X_mean = np.mean(X)
Y_mean = np.mean(Y)

num = 0
den = 0
for i in range(len(X)):
    num += (X[i] - X_mean)*(Y[i] - Y_mean) # sum of all differences in numerator
    den += (X[i] - X_mean)**2 # sum of all differences in denom
m = num / den # final slope \theta_1
c = Y_mean - m*X_mean # final intercept \theta_0

print (m, c)
```

    1.956195597832525 1022.4369105614524


Now our linear equation is $House Price (Y) = (1.956195597832525) * x + (1022.4369105614524)$


Lets plot and visualize it



```
m = 1.956195597832525
b = 1022.4369105614524
Y_pred = m*X+b # predicted values
plt.scatter(X,Y)
plt.plot(X,Y_pred,color='red')
plt.show()
```

### Check accuracy

Lets check how accurate is our model by estimating error. 

$RMSE = \sqrt{(\frac{1}{n})\sum_{i=1}^{n}(h_{i} - y_{i})^{2}}$


```
Y = df.iloc[:, 1] # observed
Y_pred 
summation = 0  #summation of differences
n = len(Y) #no of training examples
for i in range (0,n):  
  difference = Y[i] - Y_pred[i]  #finding the difference between observed and predicted value
  squared_difference = difference**2  #square of the differene 
  summation = summation + squared_difference  #sum of all the differences
MSE = summation/n  #dividing summation by no of training examples
print("ROOT Mean Square Error is: {}".format(MSE))
```

    ROOT Mean Square Error is: 22.826601317067663


This mean 22% of time our estimation is not accurate.

# Gradient Descent

### Cost function & Gradient Descents

Lets see how can we choose parameters $\theta_0$ and $\theta_1$

Let y be result after substitution

**Cost Function**: 

**Loss Function**
$J(\theta_0, \theta_1) = \sum\limits_{i=1}^m (y_{predicted} - y_{actual})^2$

**MSE**
$J(\theta_0, \theta_1) = \dfrac{1}{2m}\sum\limits_{i=1}^m (h_\theta(x^i) - y^i)^2$

**Goal** : minimize $J(\theta_0, \theta_1)$

Linear regression uses `Gradient Descent` to minimize the cost Function

**Update rules**
\begin{aligned}
&\theta_{0}:=\theta_{0}-\alpha \frac{\partial}{\partial \theta_{0}} J\left(\theta_{0}, \theta_{1}\right)\\
&\theta_{1}:=\theta_{1}-\alpha \frac{\partial}{\partial \theta_{1}} J\left(\theta_{0}, \theta_{1}\right)
\end{aligned}

**Partial derivatives**

\begin{array}{l}
\frac{\partial}{\partial \theta_{0}} J\left(\theta_{0}, \theta_{1}\right)=\frac{1}{m} \sum_{i=1}^{m}\left(h_{\theta}\left(x^{(i)}\right)-y^{(i)}\right) \\
\frac{\partial}{\partial \theta_{1}} J\left(\theta_{0}, \theta_{1}\right)=\frac{1}{m} \sum_{i=1}^{m}\left(h_{\theta}\left(x^{(i)}\right)-y^{(i)}\right) \cdot x^{(i)}
\end{array}

### Gradient Descent procedure

- start with initial values for coefficients

> coefficient = 0.0

- calculate cost of coefficient 

> cost = f(coefficient)

- calculate derivative of cost function

> delta = derivative(cost)

- update coefficients 

> coefficient = coefficient - (alpha * delta)

- This process is updated until cost function is minimum



### Batch Gradient descent



**Pseudo Code**
```
while True:
	Wgradient = evaluate_gradient(loss, data, W)
	W += -alpha * Wgradient
```





```
#populate x with input variable (sq_ft) and y with actual output variable (price)
x = df['sq_ft']
y = df['price']

x = (x - x.mean()) / x.std()
x = np.c_[np.ones(x.shape[0]), x]

#Gradient Descent Algo

learning_rate = 0.01 #learning rate (or) step_size
iterations = 1000 #no of iterations
m = y.size #no of training examples
np.random.seed(123) #Set the seed
theta = np.random.rand(2) #Pick some random values to start with

def  cal_cost(theta,x,y):

    m = len(y)
    
    predictions = x.dot(theta)
    cost = (1/2*m) * np.sum(np.square(predictions-y))
    return cost

def cal_co_efficients(theta, x, y):
    predictions = x.dot(theta)
    error = predictions-y
    theta = theta - (learning_rate * (1/m) * np.dot(x.T, error))
    return theta

#gradient descent function
def gradient_descent(x, y, theta, iterations, learning_rate):
    prev_costs = []
    prev_thetas = [theta]
    for i in range(iterations):
       # calculating predicted output y_predicted = m * x + c
        cost = cal_cost(theta, x, y)
        prev_costs.append(cost)
        theta = cal_co_efficients(theta, x, y)
        prev_thetas.append(theta)
        
    return prev_thetas, prev_costs

#Pass the relevant variables to the function and get the new values back...
prev_thetas, prev_costs = gradient_descent(x, y, theta, iterations, learning_rate)
theta = prev_thetas[-1]

#Print the results...
print("Gradient Descent: {:.2f}, {:.2f}".format(theta[0], theta[1]))
```

    Gradient Descent: 2049.39, 27.84



```
#Plot the cost function...
plt.title('Cost Function J')
plt.xlabel('No. of iterations')
plt.ylabel('Cost')
plt.plot(past_costs)
plt.show()
```

#### Stochastic Gradient descent pseudo code

[]

**Pseudo Code**
```
while True:
	batch = next_training_batch(data, 256)
	Wgradient = evaluate_gradient(loss, batch, W)
	W += -alpha * Wgradient
```



```
#populate x with input variable (sq_ft) and y with actual output variable (price)
x = df['sq_ft']
y = df['price']

x = (x - x.mean()) / x.std()
x = np.c_[np.ones(x.shape[0]), x]

#Gradient Descent Algo

learning_rate = 0.01 #learning rate (or) step_size
iterations = 50 #no of iterations
m = y.size #no of training examples
np.random.seed(123) #Set the seed
theta = np.random.rand(2) #Pick some random values to start with

def  cal_cost(theta,x,y):

    m = len(y)
    
    predictions = x.dot(theta)
    cost = (1/2*m) * np.sum(np.square(predictions-y))
    return cost

def cal_co_efficients(theta, x, y):
    predictions = x.dot(theta)
    error = predictions-y
    theta = theta - (learning_rate * (1/m) * np.dot(x.T, error))
    return theta

#gradient descent function
def stocashtic_gradient_descent(x, y, theta, iterations, learning_rate):
    prev_costs = []
    prev_thetas = [theta]
    cost =0.0
    for i in range(iterations):
        rand_ind = np.random.randint(0,iterations)
       # calculating predicted output y_predicted = m * x + c
        cost = cal_cost(theta, x, y)
        prev_costs.append(cost)
        theta = cal_co_efficients(theta, x, y)
        prev_thetas.append(theta)
        
    return prev_thetas, prev_costs

#Pass the relevant variables to the function and get the new values back...
prev_thetas, prev_costs = stocashtic_gradient_descent(x, y, theta, iterations, learning_rate)
theta = prev_thetas[-1]

#Print the results...
print("Stocashtic Gradient Descent: {:.2f}, {:.2f}".format(theta[0], theta[1]))
```

    Stocashtic Gradient Descent: 809.95, 11.13



```
#Plot the cost function...
plt.title('Cost Function J')
plt.xlabel('No. of iterations')
plt.ylabel('Cost')
plt.plot(prev_costs)
plt.show()
```

### Batch Gradient descent 


```
#populate x with input variable (sq_ft) and y with actual output variable (price)
x = df['sq_ft']
y = df['price']

x = (x - x.mean()) / x.std()

#Gradient Descent Algo

learning_rate = 0.01 #learning rate (or) step_size
iterations = 10 #no of iterations
batch_size = 20
m = y.size #no of training examples
np.random.seed(123) #Set the seed
theta = np.random.randn(2) #Pick some random values to start with

def  cal_cost(theta,x,y):

    m = len(y)
    
    predictions = x.dot(theta)
    cost = (1/2*m) * np.sum(np.square(predictions-y))
    return cost

def cal_co_efficients(theta, x, y):
    predictions = x.dot(theta)
    error = predictions-y
    theta = theta - (learning_rate * (1/m) * np.dot(x.T, error))
    return theta

#gradient descent function
def minibatch_gradient_descent(x, y, theta, iterations, learning_rate, batch_size):
    m = len(y)
    n_batches = int(m/batch_size)
    prev_costs = []
    prev_thetas = [theta]
    for i in range(iterations):
        cost =0.0
        indices = np.random.permutation(m)
        x = x[indices]
        y = y[indices]
        for i in range(0,m,batch_size):
            x_i = x[i:i+batch_size]
            y_i = y[i:i+batch_size]  
            x_i = np.c_[np.ones(len(x_i)),x_i]
            cost += cal_cost(theta,x_i,y_i)
            theta = cal_co_efficients(theta, x_i, y_i)
        prev_costs.append(cost)
        prev_thetas.append(theta)        
    return prev_thetas, prev_costs

#Pass the relevant variables to the function and get the new values back...
prev_thetas, prev_costs = minibatch_gradient_descent(x, y, theta, iterations, learning_rate, batch_size)
theta = prev_thetas[-1]

#Print the results...
print("Minibatch Gradient Descent: {:.2f}, {:.2f}".format(theta[0], theta[1]))
```

    Minibatch Gradient Descent: 194.15, 3.54



```
#Plot the cost function...
plt.title('Cost Function J')
plt.xlabel('No. of iterations')
plt.ylabel('Cost')
plt.plot(prev_costs)
plt.show()
```

592dd49c7aaf047c7b6ba7626e17cddb3ad0d3bd
ipynb
Jupyter Notebook
_notebooks/2020_04_12_interactive_and_practical_linear_regression_using_python.ipynb
sudheer/blog
a03beb9496dbd3a59cd5a132456ef2bf6bc9124b
Apache-2.0
_notebooks/2020_04_12_interactive_and_practical_linear_regression_using_python.ipynb
sudheer/blog
a03beb9496dbd3a59cd5a132456ef2bf6bc9124b
Apache-2.0
2021-03-30T09:27:21.000Z
2022-02-26T07:18:20.000Z
_notebooks/2020_04_12_interactive_and_practical_linear_regression_using_python.ipynb
sudheer/blog
a03beb9496dbd3a59cd5a132456ef2bf6bc9124b
Apache-2.0
Qwen/Qwen-72B
1. YES
2. YES