# Assignment #1 - Linear Regression

<font color="blue"> Abdullah Al Raqibul Islam </font>

# INDEX

I. Introduction<br/>
II. Data<br/>
&emsp;&emsp;II.I. Reading the data<br/>
&emsp;&emsp;II.II. Preprocessing of the data<br/>
&emsp;&emsp;II.III. Visualization of the data<br/>
&emsp;&emsp;&emsp;&emsp;II.III.I. Correlation Heatmap<br/>
&emsp;&emsp;&emsp;&emsp;II.III.II. Pie-chart<br/>
&emsp;&emsp;&emsp;&emsp;II.III.III. Count Plots<br/>
&emsp;&emsp;&emsp;&emsp;II.III.IV. Line Plots<br/>
&emsp;&emsp;&emsp;&emsp;II.III.V. Linear Regression Plots<br/>
&emsp;&emsp;&emsp;&emsp;II.III.VI. Box Plots<br/>
&emsp;&emsp;&emsp;&emsp;II.III.VII. Preliminary Observation About Data<br/>
III. Method<br/>
&emsp;&emsp;III.I. Super Classs Definition<br/>
&emsp;&emsp;III.II. Least Squares<br/>
&emsp;&emsp;III.III. Least Mean Squares<br/>
IV. Usage Examples<br/>
&emsp;&emsp;IV.I. Usage Example: Least Squares<br/>
&emsp;&emsp;IV.II. Usage Example: Least Mean Squares<br/>
V. Preliminary Test<br/>
VI. Experiments<br/>
&emsp;&emsp;VI.I. Plotting Functions<br/>
&emsp;&emsp;VI.II. Data Partitioning<br/>
&emsp;&emsp;**VI.III. Experiments on Least Squares**<br/>
&emsp;&emsp;&emsp;&emsp;VI.III.I. Feature Scaling: Unscaled Data<br/>
&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;VI.III.I.I. Feature Selection: All Features<br/>
&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;VI.III.I.II. Feature Selection: Significant Features<br/>
&emsp;&emsp;&emsp;&emsp;VI.III.II. Feature Scaling: Normalizaed Data<br/>
&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;VI.III.II.I. Feature Selection: All Features<br/>
&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;VI.III.II.II. Feature Selection: Significant Features<br/>
&emsp;&emsp;&emsp;&emsp;VI.III.III. Feature Scaling: Standarized Data<br/>
&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;VI.III.III.I. Feature Selection: All Features<br/>
&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;VI.III.III.II. Feature Selection: Significant Features<br/>
&emsp;&emsp;**VI.IV. Experiments on Least Mean Squares (LMS)**<br/>
&emsp;&emsp;&emsp;&emsp;VI.IV.I. Feature Scaling: Unscaled Data<br/>
&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;VI.IV.I.I. Feature Selection: All Features<br/>
&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;VI.IV.I.II. Feature Selection: Significant Features<br/>
&emsp;&emsp;&emsp;&emsp;VI.IV.II. Feature Scaling: Normalized Data<br/>
&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;VI.IV.II.I. Feature Selection: All Features<br/>
&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;VI.IV.II.II. Feature Selection: Significant Features<br/>
&emsp;&emsp;&emsp;&emsp;VI.IV.III. Feature Scaling: Standarized Data<br/>
&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;VI.IV.III.I. Feature Selection: All Features<br/>
&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;VI.IV.III.II. Feature Selection: Significant Features<br/>
VII. Conclusions<br/>
VIII. Extra Credit<br/>
IX. References<br/>

# I. Introduction

The goal of this assignment is to perform linear regression on a dataset to predict the price of house sale [[1]](https://www.kaggle.com/harlfoxem/housesalesprediction) based on the different features (i.e. size, condition, facility).

The features have been explained in the data section. I have implemented and used two linear regression models namely `Least Squares` and `Least Mean Squares`. The detail of the algorithm discussed in `section III`. Section `IV` and `V` contains the integraty testing of the implemented algorithms. In `section VI`, I discussed the performance evaluations in details of the implementation domain. This section also contains the extra credit works listed in `section VIII`.

# II. Data

In this expriment I used `House Sales in King County, USA` [[1]](https://www.kaggle.com/harlfoxem/housesalesprediction) dataset from `kaggle` [[2]](https://www.kaggle.com). This dataset contains house sale prices for King County, which includes Seattle. It includes homes sold between May 2014 and May 2015. The dataset have 21613 rows and 21 attributes. Here is the attribute list along with the data-type.

**Attribute Information:**
 0.   id (int64)
 1.   date (object)
 2.   price (float64)
 3.   bedrooms (int64)
 4.   bathrooms (float64)
 5.   sqft_living (int64)
 6.   sqft_lot (int64)
 7.   floors (float64)
 8.   waterfront (int64)
 9.   view (int64)
 10.  condition (int64)
 11.  grade (int64)
 12.  sqft_above (int64)
 13.  sqft_basement (int64)
 14.  yr_built (int64)
 15.  yr_renovated (int64)
 16.  zipcode (int64)
 17.  lat (float64)
 18.  long (float64)
 19.  sqft_living15 (int64)
 20.  sqft_lot15 (int64)

## II.I. Reading the data


```python
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
%matplotlib inline

# reading the data from csv datafile; using read_csv method from Pandas library
df_r = pd.read_csv("data/regression/kc_house_data.csv")

# displaying all the columns data of top 5 rows in the jupyter notebook
pd.set_option('max_columns', 28)
df_r.head(10)
```


```python
df_r.describe()
```

## II.II. Preprocessing of the data


```python
# get the metadata; getting familiarized with columns and data-types
df_r.info()
```

From the "Non-Null Count" column above, we can observe all the rows contain same number of `non-null` counts. From this count we can sense that there is no null values in this dataset.

To confirm this, let's check it using library function.


```python
# checking columns with null values
df_r.isna().any()
```

There is no null values in any particular columns. So we don't need to perform any data pre-processing here.


```python
# observe pairwise correlation of columns using library function
df_r.corr(method='pearson')
```

Pandas dataframe.corr() is used to find the pairwise correlation of all columns in the dataframe. Non-numeric data type columns from the dataframe is ignored (i.e. "date" column in this case).

This correlation matrix gives the measure of the strength of the association between the two columns. I used Pearson's Correlation Coefficient method to generate this matrix.

## II.III. Visualization of the data

### II.III.I. Correlation Heatmap


```python
f = plt.subplots(figsize=(10, 10))
sns.heatmap(df_r.corr(method='pearson'), annot=True, linewidths=.1,fmt= '.1f')
plt.title("Correlation Heatmap for HouseSales Data", color = 'green', fontsize = 20)
plt.show()
```

#### Observation:

1. The most correlated attributes with `price` are
    * bathrooms
    * sqft_living
    * view
    * grade
    * sqft_above
    * sqft_living15
2. Among them all, `sqft_living` and `grade` have the highest correlation coefficient (`0.7`).
3. This actually makes sense, as this data is suitable for `regrassion analysis` and house price mostly depends on the `size` and `quality`.

### II.III.II. Pie-chart


```python
# Plot number of records in the dataset for different condition

conditionList = []
conditionCount = []
for condition_name, subset in df_r.groupby('condition'):
    conditionList.append(condition_name)
    conditionCount.append(len(subset))
print(conditionList)
print(conditionCount)

plt.figure(figsize = (7, 7))
plt.pie(conditionCount, labels = conditionList)
plt.title("Pie Chart: Condition", color = 'green', fontsize = 20)
plt.show()
```

### II.III.III. Count Plots


```python
fig, ax = plt.subplots(figsize=(25,6))
sns.countplot(df_r.bathrooms)
plt.title("Count Plot: # of bathrooms", color = 'green', fontsize = 30)
```


```python
# Plot number of records in the dataset for different condition, view, waterfront, bedrooms, grade, zipcode

f, axes = plt.subplots(1, 3,figsize=(25,6))
sns.countplot(x= "condition", data=df_r, orient='v', ax=axes[0])
sns.countplot(x= "view", data=df_r, orient='v', ax=axes[1])
sns.countplot(x= "waterfront", data=df_r, orient='v', ax=axes[2])

f, axes = plt.subplots(1, 3,figsize=(25,6))
sns.countplot(x= "bedrooms", data=df_r, orient='v', ax=axes[0])
sns.countplot(x= "grade", data=df_r, orient='v', ax=axes[1])
sns.countplot(x= "zipcode", data=df_r, orient='v', ax=axes[2])
```

#### Observation:

1. The `pie-chart` and `count-plot` is used to compare parts to the whole. It helps picturing which types of data exist more in the dataset and can influence making any decision.
2. From the pie-chart we can observe that people buy more `condition 3` type house.
3. From the count-plot we can make simillar type of observations like we made in the pie-chart. That is majority people likes house with `2.5 bathrooms`.
4. Similarly, the group count-plot shows the prefered values for `condition`, `view`, `waterfront`, `bedrooms`, `grade`, and `zipcode`.

### II.III.IV. Line Plots


```python
# Plot the prices based on size

sns.set_theme(style="darkgrid")
sns.lineplot(x="sqft_living", y="price", data=df_r)
plt.title("Line Plot: Price Vs. Size", color = 'green', fontsize = 20)
```

### II.III.V. Linear Regression Plots


```python
# Plot linear regression on price based on size

g = sns.lmplot(data=df_r, x="sqft_living", y="price")
g.set_axis_labels("Living Space (sqft)", "Price (usd)")
plt.title("Linear Regression Plot: Price Vs. Size", color = 'green', fontsize = 20)
```


```python
# Plot linear regression on price based on size15

g = sns.lmplot(data=df_r, x="sqft_living15", y="price")
g.set_axis_labels("Living Space 15 (sqft)", "Price (usd)")
plt.title("Linear Regression Plot: Price Vs. Size_15", color = 'green', fontsize = 20)
```


```python
# Plot linear regression on price based on size_above

g = sns.lmplot(data=df_r, x="sqft_above", y="price")
g.set_axis_labels("Living Space Above (sqft)", "Price (usd)")
plt.title("Linear Regression Plot: Price Vs. Size_above", color = 'green', fontsize = 20)
```


```python
# Plot multiple linear regression on price based on size varying number of bedrooms

g = sns.lmplot(data=df_r, x="sqft_living", y="price", hue="bedrooms")
g.set_axis_labels("Living Space (sqft)", "Price (usd)")
plt.title("Multiple Linear Regression Plot: Price Vs. Size (w.r.t # of bedrooms)", color = 'green', fontsize = 20)
```


```python
# Plot multiple linear regression on price based on size varying waterfront

g = sns.lmplot(data=df_r, x="sqft_living", y="price", hue="waterfront")
g.set_axis_labels("Living Space (sqft)", "Price (usd)")
plt.title("Multiple Linear Regression Plot: Price Vs. Size (w.r.t waterfront)", color = 'green', fontsize = 20)
```


```python
# Plot multiple linear regression on price based on size varying view

g = sns.lmplot(data=df_r, x="sqft_living", y="price", hue="view")
g.set_axis_labels("Living Space (sqft)", "Price (usd)")
plt.title("Multiple Linear Regression Plot: Price Vs. Size (w.r.t waterfront)", color = 'green', fontsize = 20)
```


```python
# Plot multiple linear regression on price based on size varying condition

g = sns.lmplot(data=df_r, x="sqft_living", y="price", hue="condition")
g.set_axis_labels("Living Space (sqft)", "Price (usd)")
plt.title("Multiple Linear Regression Plot: Price Vs. Size (w.r.t condition)", color = 'green', fontsize = 20)
```

#### Observation:

1. The linear regression plots clearly show correlation between price and size.
2. The multiple linear regression plots indicate -
    * Price is higher with waterfront
    * House with view 4 have highest price within the same living space size
    * House with condition 5 price more

### II.III.VI. Box Plots


```python
# Plot price on different condition

sns.boxplot(x='condition', y='price', data=df_r)
plt.title("Box Plot: Price Vs. Condition", color = 'green', fontsize = 20)
```


```python
# Plot price on varying bedrooms

sns.boxplot(x='bedrooms', y='price', data=df_r)
plt.title("Box Plot: Price Vs. Bedrooms", color = 'green', fontsize = 20)
```

#### Observation:

1. The boxplot indicate -
    * Price range varies most for houses with 8 bedrooms

### II.III.VII. Preliminary Observation About Data

We made individual observations on the different plotting above. To summarize

* People mostly like houses with the following property -
    * Three bedroom
    * Condition 3
    * Seventh grade
    * 2.5 Bathrooms
    * Without waterfront
    * View 0
* This is actually true as we have observed from the linear regression graph that `with waterfront` price is prety high.

# III. Method

As I mentioned before, in this experiment I used two linear regression algorithms on the data described in `section II`. Given data, the goal of the linear regression algorithm is to find a best fit on all the data. The equation of a line can be written as

$$
f(x; a, b) = a x + b.
$$

Where we can replace the `a` and `b` by the weight symbol $w$,
$$
f(x; w) =  w_1 x + w_0.
$$

Considering multiple input features as $x$, we can extend the input $x$ to an input vector with bias input $x_0 = 1$:

$$
\begin{align}
f(x; w) &=  w_D x_D + \cdots + w_1 x_1 + w_0 \\
            &= \sum_{i=0}^{D} w_i x_i \quad\text{where } x_0 = 1\\
            &= w^\top x.
\end{align}
$$

Our goal here is to find the best value for $w$. So, this becomes an optimization problem. As this is a optimization problem, we have to define a cost function with respect whom we can figure out whether our choice is best or not. One possible solution is measuring the `Euclidean distances` between the target values and the predicted outputs. So we are defining our cost function like this:

$$
E(w) = \sum_{i=1}^N \Big( f(x_i; w_i) - t_i \Big)^2
$$

## III.I. Least Squares
Least squres is the simplest linear regression model. The parameter that gives best fit here will be

$$
w^* = \arg\min_w \sum_{i=1}^{N} \Big( f(x_i; w) - t_i \Big)^2
$$

Here the error funciton is quadratic. So this problem can be analytically solved by setting derivative with respect to $w$ to zero. Let's consider, the target values are collected in matrix $t$, and the input samples are in matrix $X$.

$$
\begin{align}
t &= [t_1, t_2, \cdots, t_N]^\top \\ 
\\
w &= [w_0, w_1, \cdots, w_D]^\top \\
\\
X &= \begin{bmatrix}
    x_{10} & x_{11} & x_{12} & \dots  & x_{1D} \\
    x_{20} & x_{21} & x_{22} & \dots  & x_{2D} \\
    \vdots & \vdots & \vdots & \ddots & \vdots \\
    x_{N0} & x_{N1} & x_{N2} & \dots  & x_{ND}
\end{bmatrix}
\end{align}
$$
where the first column is one's, $x_{*0} = [1, 1, \dots, 1]^\top$.

With this matrix, $f(x; w)$ can be written in matrix form as:
$$
f(x; w) = X w.
$$
Thus, the error function will become
$$
\begin{align}
E(w) &= \sum_{i=1}^N \Big(f(x_i; w_i) - t_i \Big)^2 \\
\end{align}
$$

Now, by doing the partial derivative w.r.t. $w$, we get - 
$$
\begin{align}
\frac{\partial E(w)}{\partial w} &= 2 X^\top X w - 2 X^\top t \\
\end{align}
$$

By setting this to zero, 
$$
\begin{align}
 2 X^\top X w - 2 X^\top t  &= 0\\
\\
w &= \big(X^\top X\big)^{-1} X^\top t
\end{align}
$$

So, given the feature matrix $X$ and our weight vector $w$, least squares find the best fitted line by this formula.

## III.II. Least Mean Squares
In `Least Squares` we observed that, it uses all the training data in finding the best fit. This is a computationally costly operation when dealing with learge features or number of entried. When data is sufficiently large, we can consider `Least Mean Squares` which learn *sequentially* instead of what doing by `Least Squares`. During the *sequential* learning process, $w$ is updated for a single data point in each iteration.

`Least Mean Squares` starts with an initial guess $w$ and changes it as it reads more data until it converges.

$$
w^{(k+1)} = w^{(k)} - \alpha \nabla E_k 
$$

Here $k$ represents the steps for the repetition and $E_k$ is the error for the $k$'th sample and $\alpha$ is a learning rate. In the $k$'th iteration with the sample $x_k$, the gradient for the sum-of-squares error is

$$
\begin{align}
\nabla E_k = \frac{\partial E}{\partial w^{(k)}} &= \frac{\partial }{\partial w^{(k)}}\Big( f(x_k; w^{(k)}) - t_k \Big)^2 \\
&= 2\Big( {w^{(k)}}^\top x_k - t_k \Big) x_k.
\end{align}
$$

By replacing $\nabla E_k$, we get:

$$
w^{(k+1)} = w^{(k)} - \alpha \Big( {w^{(k)}}^\top x_k - t_k \Big) x_k.
$$

So, given the feature matrix $X$ and the randomly initialized weight vector $w$; in each iteration of data points, `Least Mean Squares` update the $w$ by this formula.

`Least Mean Squares` solves the potential computational issue of `Least Squares`, but it could be possible that this algorithm will never converge. For dealing this, we usually run this algorithm withing a fixed number of iterations.

## III.I. Super Classs Definition

These are the super class definitions for general linear model (i.e. `LinearModel`). This is the base model for the implementation of both the models (i.e. `Least Squares Model` and `Least Mean Square Model`). The abstract methods (i.e. `train` and `use`) will enable the unified interfaces for child classes to be overridden.

`LinearModel` also contains the implementations of two helper functions `_check_matrix` and `add_ones` which helps checking dimentional constraint and adding a basis to the featur matrix respectively.


```python
import numpy as np 
from abc import ABC, abstractmethod

# Super class for machine learning models 

class BaseModel(ABC):
    """ Super class for ITCS Machine Learning Class"""
    
    @abstractmethod
    def train(self, X, T):
        pass

    @abstractmethod
    def use(self, X):
        pass

    
class LinearModel(BaseModel):
    """
        Abstract class for a linear model 
        
        Attributes
        ==========
        w       ndarray
                weight vector/matrix
    """

    def __init__(self):
        """
            weight vector w is initialized as None
        """
        self.w = np.zeros([1,1])
        # self.w = None

    # check if the matrix is 2-dimensional. if not, raise an exception    
    def _check_matrix(self, mat, name):
        if len(mat.shape) != 2:
            raise ValueError(''.join(["Wrong matrix ", name]))
        
    # add a basis
    def add_ones(self, X):
        """
            add a column basis to X input matrix
        """
        self._check_matrix(X, 'X')
        return np.hstack((np.ones((X.shape[0], 1)), X))

    ####################################################
    #### abstract funcitons ############################
    @abstractmethod
    def train(self, X, T):
        """
            train linear model
            
            parameters
            -----------
            X     2d array
                  input data
            T     2d array
                  target labels
        """        
        pass
    
    @abstractmethod
    def use(self, X):
        """
            apply the learned model to input X
            
            parameters
            ----------
            X     2d array
                  input data
            
        """        
        pass 
```

Inheriting the super class, we can define two difference classes for Least Squares and Least Mean Square algorithms. 

## III.II. Least Squares

`LinearRegress` class implements the least square solutions. Here, I implemented the `train` and `use` functions. `train()` updates the weights using least squares solution and `use()` returns the predictions for the argument X by using the trained weight w.


```python
# Linear Regression Class for least squares
class LinearRegress(LinearModel): 
    """ 
        LinearRegress class 
        
        attributes
        ===========
        w    nd.array  (column vector/matrix)
             weights
    """
    def __init__(self):
        LinearModel.__init__(self)
        
    # train lease-squares model
    def train(self, X, T):
        # First creast X1 by adding 1's column to X
        N = X.shape[0]
        X1 = self.add_ones(X)
        # Next, using inverse, solve, lstsq function to get w*
        self.w = np.linalg.inv(X1.transpose().dot(X1)).dot(X1.transpose()).dot(T)
    
    # apply the learned model to data X
    def use(self, X):
        # First creast X1 by adding 1's column to X
        N = X.shape[0]
        X1 = self.add_ones(X)
        return X1.dot(self.w)
```

## III.III. Least Mean Squares

For online learning, I have implemented LMS algorithm here. `train_step()` function updats the weights for a single input vector $x$ and one target label. I update the weight vector with $0$ if it is not initialized or it mismatches the shape with the input vector $x$. As the name suggests, `train()` function simply call `train_step()` in a loop to learn incrementally for the batch data.


```python
import collections # for checking iterable instance

# LMS class 
class LMS(LinearModel):
    """
        Lease Mean Squares. online learning algorithm
    
        attributes
        ==========
        w        nd.array
                 weight matrix
        alpha    float
                 learning rate
    """
    def __init__(self, alpha):
        LinearModel.__init__(self)
        self.alpha = alpha
    
    # batch training by using train_step function
    def train(self, X, T):
        for x, t in zip(X, T):
            self.train_step(x, t)

    # train LMS model one step 
    # here the x is 1d vector
    def train_step(self, x, t):
        N = x.shape[0]
        X1 = np.hstack((np.ones(1),x))
        N = X1.shape[0]
        #print(self.w)
        if(len(self.w)!= N):
            self.w = np.zeros(N)
        y = self.w @ X1
        self.w -= self.alpha * (y - t) * X1
       
    
    # apply the current model to data X
    def use(self, X):
        N = X.shape[0]
        X1 = np.hstack((np.ones((N, 1)), X.reshape((X.shape[0], -1))))
        y = X1 @ self.w
        N = X1.shape[0]
        y.shape = (N,1)
        return y
```

# IV. Usage Examples

In this section I used some sample linear data to test the integrary of my implementations.


```python
# HERE follow are for my code tests.

import matplotlib.pyplot as plt
%matplotlib inline
```

## IV.I. Usage Example: Least Squares

First prepare the sample data and put it in $x$. Then call `LinearRegress` model to train and predict the data. Line plot indicate the integrary of the `LinearRegress` implementation.


```python
X = np.linspace(0,10, 11).reshape((-1, 1))
T = -2 * X + 3.2

ls = LinearRegress()

ls.train(X, T)

plt.plot(ls.use(X))
```

## IV.II. Usage Example: Least Mean Squares

Here I used the same sample data $x$ and call `LMS` model to train and predict the data. In the first plotting, I draw the prediction lines in each `train_step`. And the later one showing the final prediction line after calling the `train` function. The line plots indicate the integrary of the `LMS` implementation.


```python
lms = LMS(0.1)
for x, t in zip(X, T):
    lms.train_step(x, t)
    plt.plot(lms.use(X))
```


```python
lms = LMS(0.1)
lms.train(X, T)
plt.plot(lms.use(X))
```

# V. Preliminary Test

This section contains the integrary test of the implmentation of `LinearRegress` and `LMS` classes. As all the tests passed here, getting much confidence for running my implementations of linear regression models (i.e. `LinearRegress` and `LMS`) w.r.t. the `House Sales in King County, USA` data described in `section II`.


```python
##################### WHAT I WILL RELEASE ############

# Self-Test code for accuracy of your model - DO NOT MODIFY THIS
# Primilnary test data
X = np.array([[2,5],
              [6,2],
              [1,9],
              [4,5],
              [6,3],
              [7,4],
              [8,3]])
T = X[:,0, None] * 3 - 2 * X[:, 1, None] + 3
N = X.shape[0]

def rmse(T, Y):
    return np.sqrt(np.sum((T-Y)**2))

model_names = ['LS', 'LMS_All', 'LMS_1STEP']
models = [LinearRegress(), LMS(0.02), LMS(0.02)]
#train
for i, model in enumerate(models):
    print("training ", model_names[i], "...") 
    if i == len(models) -1: 
        # train only one step for LMS2
        model.train_step(X[0], T[0])
    else:
        model.train(X, T)

def check(a, b, eps=np.finfo(float).eps):
    if abs(a-b) > eps:
        print("failed.", a, b)
    else:
        print("passed.")

errors = [1.19e-13, 2.8753214702, 38.0584918251]
for i, model in enumerate(models):
    print("---- Testing ", model_names[i], "...", end=" ") 
    
    # rmse test
    err = rmse(T, model.use(X))
    if check(err, errors[i], eps=1e-10):
        print ("check your weights: ", model.w)
        print ("oracle: ", )
```

# VI. Experiments

This section contains the experiment results of `LinearRegress` and `LMS` models (described in `section III`) w.r.t. the `House Sales in King County, USA` data (described in `section II`). Basically we will predict the `price` of the house considering the other features of the data (i.e. `sqft_living`, `view`, `grade`, etc.). My experiment domain is listed in the `Table 1` bellow.

<table>
  <thead>
    <tr>
      <th>Algorithms</th>
      <th>Feature Selection</th>
      <th>Feature Scaling</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td align="middle">Least Squares<br><br>
                           Least Mean Squares</td>
      <td align="middle">All Features<br><br>
                           Significant Features</td>
      <td align="middle">Unscaled<br><br>
                           Normalized<br><br>
                           Standarized</td>
    </tr>
    <tr>
      <td colspan="3" align="middle">Table 1: Summary of the experiment domain.</td>
    </tr>
  </tbody>
</table>

**Feature Scaling**

Differentt scales of input variables may cause problem for the regression models. Having features on a similar scale can help the gradient descent converge more quickly towards the minima.

* Dominated by one large scale inputs!
* "Standardize" the range of independent input features

There are different ways of doing the scaling. In this experiment I considered the following two feature scaling technique,
* Min-Max Normalization (I used `MinMaxScaler` from the `sklearn.preprocessing` library)
* Standardization (I used `StandardScaler` from the `sklearn.preprocessing` library)

Before jumping to the experiment results, here is the detailed descriptions of the experiment subsections:

**VI.I. Plotting Functions**

Data visualization is crutial to understand the performance of ML models. In this section, I put general graph plotting functions that will help visualizing different performance matric. Particularly the `residual plotting` is a requirement (the third one) for the `Extra Credit` (please check `section VIII`).

A residual is a measure of how far away a point is vertically from the prediction line. Basically, it is the error between a predicted value and the corresponding observed actual value.

**VI.II. Data Partitioning**

First we randomly shuffle the whole dataset, which gives two benifits,

1. Improve the ML model quality
2. Improve the predictive performance

As we know, to test the performance of linear regession model we have to partition the data into `train-data` and `test-data`. Model training will be done on the `train-data` and then the performance of the model training is done on the `test-data`. As we have total `21613` data point, I considered `19000` data for training and the rest for testing.

Besides partitioning, feature selection is also very crutial for regression models. In feature selection we actually determine which features are importent predicting the output. As discussed in `section II.III.I`, the most correlated attributes with the target attribute (i.e. `price`) are
1. bathrooms
2. sqft_living
3. view
4. grade
5. sqft_above
6. sqft_living15

So considering this, I do two types of experiments,
1. Selecting all the features
2. Selecting the most significant features (I considered the features that have `correlation_val >= 0.5` as the significant features).

I believe this covers the first two requirements of `Extra Credit` listed in `section VIII`.

## VI.I. Plotting Functions

In this section, I put general graph plotting functions (snapshot of `Prediction Vs. Actual`, `feature scaling` comparator plotting, `residual plotting`) that will help visualizing different performance matric.


```python
"""
    Prediction comparator plotting

    attributes
    ==========
    predicted     array of predicted values
    actual        array of actual values
    feature_type  feature selection
"""
def predict_comp_plotting(predicted, actual, feature_type):
    fig = plt.figure(figsize=(10, 5), dpi = 150)
    plt.plot(predicted, label="Prediction")
    plt.plot(actual, label="Actual")

    plt.legend(bbox_to_anchor=(0.83, 0.98), loc='upper left', borderaxespad=0.)

    plt.xlabel('Test data index')
    plt.ylabel('Prediction Vs. Actual')
    plt.title('Snapshot of Prediction Vs. Actual plotting for ' + feature_type, color = 'green', fontsize = 20)

    plt.show()
```


```python
"""
    Feature scaling comparator plotting

    attributes
    ==========
    predicted   array of predicted values
    actual      array of actual values
    columns     xticks of the box plot (feature title)
    scale_type  feature scaling
"""

def feature_scal_comp_plotting(unscaled, scaled, columns, scale_type):
    f, axes = plt.subplots(1, 2, figsize=(15,6))

    plt_norm = sns.boxplot(data=pd.DataFrame(unscaled), orient='v', ax=axes[0])
    axes[0].set_xticklabels(columns, rotation=90)
    axes[0].set_title('Unscaled Data')

    sns.boxplot(data=pd.DataFrame(scaled), orient='v', ax=axes[1])
    axes[1].set_xticklabels(columns, rotation=90)
    axes[1].set_title(scale_type + ' Data')

    f.suptitle('Comparing Unscaled Vs. ' + scale_type + ' data', color = 'green', fontsize = 20)
```


```python
"""
    Residual plotting

    attributes
    ==========
    predicted     array of predicted values
    actual        array of actual values
    feature_type  feature selection
"""

def residual_plotting(predicted, actual, feature_type):
    diff = actual - predicted
    sample_count = np.arange(len(actual))
    fig=plt.figure(figsize=(10, 5), dpi= 144, facecolor='w', edgecolor='k')
    plt.scatter(sample_count, diff, marker='.')
    plt.plot([0, 2600],[0, 2600], 'r-')
    plt.title('Residual Plot for ' + feature_type, color = 'green', fontsize = 20)
```

## VI.II. Data Partitioning

First we randomly shuffle the whole dataset. And then partition the data into `train-data` and `test-data` samples. As we have total 21613 data points, I considered 19000 data for training and the rest for testing.


```python
# randomly shuffel all data
df_r.sample(frac=1)
```


```python
# constants

# list of all features
all_features = ['long', 'lat', 
          'yr_renovated', 'yr_built', 
          'sqft_basement', 'sqft_above', 
          'sqft_lot', 'sqft_lot15',
          'sqft_living', 'sqft_living15',
          'condition', 'view', 'grade', 
          'floors', 'waterfront',
          'zipcode', 
          'bedrooms', 'bathrooms']

# list of significant features
sig_features = ['sqft_living', 'sqft_living15', 'sqft_above', 'view', 'grade', 'bathrooms']

# list of target features
target_features = ['price']

# feature scaling type
scale_type_norm = 'Normalized'
scale_type_stand = 'Standarized'

# feature selection type
feature_type_all = "All Features"
feature_type_sig = "Significant Features"

# training data partition threshold
train_data_th = 19000
snapshot_th = 100
```

### VI.II.I. Unscaled Data


```python
# partitioning unscaled-data of all possible features

X_All = df_r[all_features].copy()

X_All_Train = np.array(X_All.iloc[:train_data_th])
X_All_Test = np.array(X_All.iloc[train_data_th:])
```


```python
# partitioning unscaled-data of most significant co-related features (i.e. correlation_val >= 0.5)

X = df_r[sig_features].copy()
X_Train = np.array(X.iloc[:train_data_th])
X_Test = np.array(X.iloc[train_data_th:])
```


```python
# partitioning unscaled-data of target features

T = df_r[target_features].copy()
T_Train = np.array(T.iloc[:train_data_th])
T_Test = np.array(T.iloc[train_data_th:])
```

### VI.II.II. Normalized Data


```python
# partitioning normalized-data of all possible features

# data normalization with sklearn
from sklearn.preprocessing import MinMaxScaler

X_All_Train_Norm = np.array(X_All.iloc[:train_data_th])
X_All_Test_Norm = np.array(X_All.iloc[train_data_th:])

# normalize all possible feature data
norm = MinMaxScaler().fit(X_All_Train)
X_All_Train_Norm = norm.transform(X_All_Train)
X_All_Test_Norm = norm.transform(X_All_Test)

feature_scal_comp_plotting(X_All_Train, X_All_Train_Norm, all_features, scale_type_norm)
```


```python
# partitioning normalized-data of most significant co-related features (i.e. correlation_val >= 0.5)

# data normalization with sklearn
from sklearn.preprocessing import MinMaxScaler

X_Train_Norm = np.array(X.iloc[:train_data_th])
X_Test_Norm = np.array(X.iloc[train_data_th:])

# normalize significant feature data
norm = MinMaxScaler().fit(X_Train)
X_Train_Norm = norm.transform(X_Train)
X_Test_Norm = norm.transform(X_Test)

feature_scal_comp_plotting(X_Train, X_Train_Norm, sig_features, scale_type_norm)
```

### VI.II.II. Standardized Data


```python
# partitioning standarized-data of all possible features

# data standardization with  sklearn
from sklearn.preprocessing import StandardScaler

X_All_Train_Stand = np.array(X_All.iloc[:train_data_th])
X_All_Test_Stand = np.array(X_All.iloc[train_data_th:])

# define standard scaler
scaler = StandardScaler()
# standarized all possible feature data
X_All_Train_Stand = scaler.fit_transform(X_All_Train)
X_All_Test_Stand = scaler.fit_transform(X_All_Test)

feature_scal_comp_plotting(X_All_Train, X_All_Train_Stand, all_features, scale_type_stand)
```


```python
# partitioning standarized-data of most significant co-related features (i.e. correlation_val >= 0.5)

# data standardization with  sklearn
from sklearn.preprocessing import StandardScaler

X_Train_Stand = np.array(X.iloc[:train_data_th])
X_Test_Stand = np.array(X.iloc[train_data_th:])

# define standard scaler
scaler = StandardScaler()
# standarized significant feature data
X_Train_Stand = scaler.fit_transform(X_Train)
X_Test_Stand = scaler.fit_transform(X_Test)

feature_scal_comp_plotting(X_Train, X_Train_Stand, sig_features, scale_type_stand)
```

## VI.III. Experiments on Least Squares

This section contains the experiment results on `Least Squares` algorithm.

### VI.III.I. Feature Scaling: Unscaled Data

Here is the `Least Squares` algorithm's performance evaliation on unscaled data.

#### VI.III.I.I. Feature Selection: All Features


```python
lr = LinearRegress()
lr.train(X_All_Train, T_Train)
T_Predicted = lr.use(X_All_Test)

print("rmse value", rmse(T_Test, T_Predicted))
predict_comp_plotting(T_Predicted[:snapshot_th], T_Test[:snapshot_th], feature_type_all)
residual_plotting(T_Predicted, T_Test, feature_type_all)
```

#### VI.III.I.II. Feature Selection: Significant Features


```python
lr = LinearRegress()
lr.train(X_Train, T_Train)
T_Predicted = lr.use(X_Test)

print("rmse value", rmse(T_Test, T_Predicted))
predict_comp_plotting(T_Predicted[:snapshot_th], T_Test[:snapshot_th], feature_type_sig)
residual_plotting(T_Predicted, T_Test, feature_type_sig)
```

**Observations:**

From the rmse value (`464727090` for all the features Vs. `12318249` for significant features) and the residual plotting we can understand the importance of removing `insignificant features`. The rmse value gets better by `38x` when we considered only the significant features.

From the residual plot, we can observe that the data points get more clustered on the prediction line. Which also support the evidence we get from the change of rmse value.

### VI.III.II. Feature Scaling: Normalizaed Data

Here is the `Least Squares` algorithm's performance evaliation on normalized data.

#### VI.III.II.I. Feature Selection: All Features


```python
lr = LinearRegress()
lr.train(X_All_Train_Norm, T_Train)
T_Predicted = lr.use(X_All_Test_Norm)

print("rmse value", rmse(T_Test, T_Predicted))
predict_comp_plotting(T_Predicted[:snapshot_th], T_Test[:snapshot_th], feature_type_all)
residual_plotting(T_Predicted, T_Test, feature_type_all)
```

#### VI.III.II.II. Feature Selection: Significant Features


```python
lr = LinearRegress()
lr.train(X_Train_Norm, T_Train)
T_Predicted = lr.use(X_Test_Norm)

print("rmse value", rmse(T_Test, T_Predicted))
predict_comp_plotting(T_Predicted[:snapshot_th], T_Test[:snapshot_th], feature_type_sig)
residual_plotting(T_Predicted, T_Test, feature_type_sig)
```

**Observations:**

From the rmse value (`40098628` for all the features Vs. `12318249` for significant features) we again gets the proof of the importance of removing `insignificant features`. The residual plot also supports this.

Although, the performance gap is reduced comparing to the experiment result of the unscaled data (`38x` Vs. `3.25`). I believe this change occurs as we `normalized` the input feature data. Interestingly, we also can observe that for all the features `normalization` improves the performance by `11.5`, but there is no such observation found in the significant feature data. I believe this is because the impacts of normalization have been dominated by the signigicant features.

### VI.III.III. Feature Scaling: Standarized Data

#### VI.III.III.I. Feature Selection: All Features


```python
lr = LinearRegress()
lr.train(X_All_Train_Stand, T_Train)
T_Predicted = lr.use(X_All_Test_Stand)

print("rmse value", rmse(T_Test, T_Predicted))
predict_comp_plotting(T_Predicted[:snapshot_th], T_Test[:snapshot_th], feature_type_all)
residual_plotting(T_Predicted, T_Test, feature_type_all)
```

#### VI.III.III.II. Feature Selection: Significant Features


```python
lr = LinearRegress()
lr.train(X_Train_Stand, T_Train)
T_Predicted = lr.use(X_Test_Stand)

print("rmse value", rmse(T_Test, T_Predicted))
predict_comp_plotting(T_Predicted[:snapshot_th], T_Test[:snapshot_th], feature_type_sig)
residual_plotting(T_Predicted, T_Test, feature_type_sig)
```

**Observations:**

From the rmse value (`11513128` for all the features Vs. `12533302` for significant features) we again gets the proof of the importance of removing `insignificant features`. The residual plot also supports this. Interestingly, the performance gap indicate that the performance of the algorithm degrades while selecting significant features on the standarized data. I believe this change occurs as we `standarided` the input feature data and the impacts of standaraization have been dominated here due to feature selection.

## VI.IV. Experiments on Least Mean Squares (LMS)

### VI.IV.I. Feature Scaling: Unscaled Data

#### VI.IV.I.I. Feature Selection: All Features


```python
lms = LMS(0.02)
lms.train(X_All_Train, T_Train)
T_Predicted = lms.use(X_All_Test)

print("rmse value", rmse(T_Test, T_Predicted))
```

#### VI.IV.I.II. Feature Selection: Significant Features


```python
lms = LMS(0.02)
lms.train(X_Train, T_Train)
T_Predicted = lms.use(X_Test)

print("rmse value", rmse(T_Test, T_Predicted))
```

We got `overflow encountered in multiply` while running the `Least Mean Squares` on the unscaled data.

### VI.IV.II. Feature Scaling: Normalized Data

#### VI.IV.II.I. Feature Selection: All Features


```python
lms = LMS(0.02)
lms.train(X_All_Train_Norm, T_Train)
T_Predicted = lms.use(X_All_Test_Norm)

print("rmse value", rmse(T_Test, T_Predicted))
predict_comp_plotting(T_Predicted[:snapshot_th], T_Test[:snapshot_th], feature_type_all)
residual_plotting(T_Predicted, T_Test, feature_type_all)
```

#### VI.IV.II.II. Feature Selection: Significant Features


```python
lms = LMS(0.02)
lms.train(X_Train_Norm, T_Train)
T_Predicted = lms.use(X_Test_Norm)

print("rmse value", rmse(T_Test, T_Predicted))
predict_comp_plotting(T_Predicted[:100], T_Test[:100], feature_type_sig)
residual_plotting(T_Predicted, T_Test, feature_type_sig)
```

### VI.IV.III. Feature Scaling: Standarized Data

#### VI.IV.III.I. Feature Selection: All Features


```python
lms = LMS(0.02)
lms.train(X_All_Train_Stand, T_Train)
T_Predicted = lms.use(X_All_Test_Stand)

print("rmse value", rmse(T_Test, T_Predicted))
predict_comp_plotting(T_Predicted[:snapshot_th], T_Test[:snapshot_th], feature_type_all)
residual_plotting(T_Predicted, T_Test, feature_type_all)
```

#### VI.IV.III.II. Feature Selection: Significant Features


```python
lms = LMS(0.02)
lms.train(X_Train_Stand, T_Train)
T_Predicted = lms.use(X_Test_Stand)

print("rmse value", rmse(T_Test, T_Predicted))
predict_comp_plotting(T_Predicted[:100], T_Test[:100], feature_type_sig)
residual_plotting(T_Predicted, T_Test, feature_type_sig)
```

**Observation**

* Normalization works better than standarization.
* Feature scaling makes important difference.

## VI.V. Experiment Results

In this section I like to give a recap about the overall experiment result. `Table-2` bellow gives the overall rmse values and `Table-3` represents the comparative performance (w.r.t. the rmse value).

<table>
    <tr>
        <th>Table 2: Summary of the RMSE value within the experiment domain.</th>
        <th>Table 3: Comparison of RMSE value within the experiment domain.</th>
    </tr>
    <tr>
        <td>
            <table>
                <tr>
                    <th>Algorithms</th>
                    <th>Feature Selection</th>
                    <th colspan="3" align="middle">Feature Scaling</th>
                </tr>
                <tr>
                    <th></th>
                    <th></th>
                    <th>Unscaled</th>
                    <th>Normalized</th>
                    <th>Standarized</th>
                </tr>
                <tr>
                    <th rowspan="2">Least Squares</th>
                    <th>All Features</th>
                    <td>464727090</td>
                    <td>40098628</td>
                    <td>11513129</td>
                </tr>
                <tr>
                    <th>Significant Features</th>
                    <td>12318249</td>
                    <td>12318249</td>
                    <td>12533303</td>
                </tr>
                <tr>
                    <th rowspan="2">Least Mean Squares</th>
                    <th>All Features</th>
                    <td>N/A</td>
                    <td>11021941</td>
                    <td>3.25e+39</td>
                </tr>
                <tr>
                    <th>Significant Features</th>
                    <td>N/A</td>
                    <td>12703794</td>
                    <td>13333369</td>
                </tr>
                <tr>
                  <td colspan="5" align="middle"></td>
                </tr>
            </table>
        </td>
        <td>
            <table>
                <tr>
                    <th>Algorithms</th>
                    <th>Feature Selection</th>
                    <th colspan="3" align="middle">Feature Scaling</th>
                </tr>
                <tr>
                    <th></th>
                    <th></th>
                    <th>Unscaled</th>
                    <th>Normalized</th>
                    <th>Standarized</th>
                </tr>
                <tr>
                    <th rowspan="2">Least Squares</th>
                    <th>All Features</th>
                    <td>37.70 X</td>
                    <td>3.25 X</td>
                    <td>0.94 X</td>
                </tr>
                <tr>
                    <th>Significant Features</th>
                    <td>X</td>
                    <td>X</td>
                    <td>1.02 X</td>
                </tr>
                <tr>
                    <th rowspan="2">Least Mean Squares</th>
                    <th>All Features</th>
                    <td>N/A</td>
                    <td>X</td>
                    <td>2.95e32 X</td>
                </tr>
                <tr>
                    <th>Significant Features</th>
                    <td>N/A</td>
                    <td>1.15 X</td>
                    <td>1.20 X</td>
                </tr>
                <tr>
                  <td colspan="5" align="middle"></td>
                </tr>
            </table>
        </td>
    </tr>
</table>

The above tables indicate that standarization within all the features gives the best result for `Least Squares`. On the other hand, normalization within all the features gives the best result for `Least Mean Squares`. It also indicates that the feature scaling improves the performance of `Least Squares` algorithm; but as I didn't able to get the result for `Least Mean Squares` on unscaled data, it was not possible to observe such behaviour. But it was surprising that instead of improving the performance, selecting significant features degrades the performance in `Least Mean Squares`. Also, the rmse value for `Least Mean Squares` on the standarized data for all the features are quite high, but the residual plotting indicates somewhat best performance. Due to the time constraint, I didn't go further to investigate these issues. Instead, I kept these as the future work for this assignment.

# VII. Conclusions

In this assignment I have performed linear regression on the `House Sales in King County, USA` using the `Linear Square` and `Linear Mean Square` methods. The `Linear Mean Square` is more sophisticated (as I get exception on the unscaled data) due to the greater detail in its computation compared to the `Linear Square` model.

# VIII. Extra Credit

If you want to work more for an extra credit, place your work here for additional analysis: weight and residual analysis. 
Try to answer to the following questions: 
1. what is the most and least significant features for your data. (covered in `section VI. Experiments`)
2. what are the consequences if you remove those features from the model? (covered in `section VI., VI.III.II., VI.III.III., VI.IV.II., VI.IV.III.`)
3. produce residual plots and observe the patterns for the goodness of fit (covered in `section VI., VI.III.II., VI.III.III., VI.IV.II., VI.IV.III.`)

# IX. References

1. harlfoxem (2016). House Sales in King County, USA, Version 1. Retrieved September 20, 2020 from https://www.kaggle.com/harlfoxem/housesalesprediction.

2. “Kaggle: Your Machine Learning and Data Science Community,” https://www.kaggle.com, accessed Oct. 1, 2020.

0e6e2ebaa8378df7dc7520f3fc5b2b9a4cd59147
ipynb
Jupyter Notebook
programming_assignments/1_Linear-Regression/Assign1.ipynb
biqar/Fall-2020-ITCS-8156-MachineLearning
ce14609327e5fa13f7af7b904a69da3aa3606f37
MIT
programming_assignments/1_Linear-Regression/Assign1.ipynb
biqar/Fall-2020-ITCS-8156-MachineLearning
ce14609327e5fa13f7af7b904a69da3aa3606f37
MIT
programming_assignments/1_Linear-Regression/Assign1.ipynb
biqar/Fall-2020-ITCS-8156-MachineLearning
ce14609327e5fa13f7af7b904a69da3aa3606f37
MIT
Qwen/Qwen-72B
1. YES
2. YES