```python
import pandas as pd
import matplotlib.pyplot as plt
import numpy as np
from sklearn.linear_model import LinearRegression
```

# Linear Regression

### Contents
* [Bivariate Linear Regresion](#Bivariate-Linear-Regresion)
* [Bivariate Linear Regresion with Scikit Learn](#Bivariate-Linear-Regresion-with-Scikit-Learn)
* [Bivariate Linear Regression with Linear Algebra](#Bivariate-Linear-Regression-with-Linear-Algebra)
* [Multiple Regression](#Multiple-Regression)
* [Basic Model Validation](#Basic-Model-Validation)
* [Interpreting Coefficients](#Interpreting-Coefficients)

### Why is it so Important?

* **Popularity** - Linear Regression is an extremely popular technique that every data scientist **needs** to understand. It's not the most advanced technique and there are supervised learning techniques that will obtain a higher accuracy, but where it lacks in accuracy it makes up for it in interpretability and simplicity.
* **Interpretability** - Few other models possess coefficients that are so directly linked to their variables with a such a clear interpretation. Tomorrow we're going to learn about ways to make them even easier to interpret.
* **Simplicity** - A linear regression model can be communicated just by writing out its equation. It's kind of incredible that such high dimensional relationships can be described from just a linear combination of variables and coefficients. 

### What is it?
Linear Regression is a statistical model that seeks to describe the relationship between some y variable and one or more x variables.

In the simplest case, linear regression seeks to fit a straight line through a cloud of points. This line is referred to as the "regression line" or "line of best fit." This line tries to summarize the relationship between our X and Y in a way that enables us to use the equation for that line to make predictions.



### Synonyms for "y variable" 

- Dependent Variable
- Predicted Variable
- Target Variable
- Response Variable
- Outcome Variable 
- Measured Variable
- Explained Variable
- Label

### Synonyms for "x variable(s)"

- Indepent Variable
- Explanatory Variable
- Regressor
- Covariate

[Wikipedia Independent and Dependent Variables](https://en.wikipedia.org/wiki/Dependent_and_independent_variables)

# Bivariate Linear Regresion

You must first identify the component you are trying to predict based on the other information you have.

Then, Linear Regression seeks to **estimate** the slope, $m$, and intercept, $b$, values that describe a line of best fit:
\begin{align}
y = mx + b
\end{align}

### Notation

- Intercept: The $b$ value in our line equation $y=mx+b$
- Slope: The $m$ value in our line equation $y=mx+b$. These two values together define our regression line.



- $\hat{y}$ : A prediction
- Line of Best Fit (Regression Line)
- Predicted (fitted) Values: Points on our regression line
- Observed Values: Points from our dataset
- Error: The distance between predicted and observed values.





We have talked about a line of regression being represented like a regular line $y=mx+b$ but as we get to more complicated versions we're going to need to extend this equation. So lets establish the proper terminology.

**X** - Independent Variable, predictor variable, explanatory variable, regressor, covariate

**Y** - Response variable, predicted variable, measured vairable, explained variable, outcome variable

$\beta_0$ - "Beta Naught" or "Beta Zero", the intercept value. This is how much of y would exist if X were zero. This is sometimes represented by the letter "a" but I hate that. So it's "Beta 0" during my lecture.

$\beta_1$ - "Beta One" The primary coefficient of interest. This values is the slope of the line that is estimated by "minimizing the sum of the squared errors/residuals" - We'll get to that. 

$\epsilon$ - "Epsilon" The "error term", random noise, things outside of our model that affect y.

# Bivariate Linear Regresion with Scikit-Learn


```python
# Linear Regression Example
df = pd.read_csv('https://raw.githubusercontent.com/ryanleeallred/'
                 'datasets/master/Ice_Cream_Sales.csv')

print(df.shape)
df.head(11)
```

    (11, 2)





<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>Farenheit</th>
      <th>Dollars</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>32</td>
      <td>200</td>
    </tr>
    <tr>
      <th>1</th>
      <td>35</td>
      <td>400</td>
    </tr>
    <tr>
      <th>2</th>
      <td>47</td>
      <td>500</td>
    </tr>
    <tr>
      <th>3</th>
      <td>55</td>
      <td>600</td>
    </tr>
    <tr>
      <th>4</th>
      <td>73</td>
      <td>1100</td>
    </tr>
    <tr>
      <th>5</th>
      <td>78</td>
      <td>1200</td>
    </tr>
    <tr>
      <th>6</th>
      <td>81</td>
      <td>1500</td>
    </tr>
    <tr>
      <th>7</th>
      <td>86</td>
      <td>1700</td>
    </tr>
    <tr>
      <th>8</th>
      <td>89</td>
      <td>1800</td>
    </tr>
    <tr>
      <th>9</th>
      <td>90</td>
      <td>1600</td>
    </tr>
    <tr>
      <th>10</th>
      <td>110</td>
      <td>2000</td>
    </tr>
  </tbody>
</table>
</div>




```python
plt.scatter(df.Farenheit, df.Dollars)
plt.ylabel("Farenheit")
plt.xlabel("Dollars")
plt.title("Ice Cream Sales and Temperature")
plt.show()
```

We can see from the scatterplot that these data points seem to follow a somewhat linear relationship. This means that we could probably summarize their relationship well by fitting a line of best fit to these points. Lets do it.


Before we can pass our x and y values into Scikit-Learn we need our data to be the right shape.

[Linear Regression on Pandas DataFrame using Sci-kit Learn - stackoverflow](https://stackoverflow.com/questions/29934083/linear-regression-on-pandas-dataframe-using-sci-kit-learn)


```python
# Capital X, lowercase y because we typically have more than one x variable
# and our X dataframe will be a matrix rather than a vector.
X = df[['Farenheit']].values
y = df[['Dollars']].values

print('Our data needs to have this shape to be accepted by sklearn:',
      f'X: {X.shape}', f'y: {y.shape}', sep='\n')

model = LinearRegression()
model.fit(X, y)

print("Slope Coefficient:", model.coef_[0])
print("Intercept Value:", model.intercept_)
```

    Our data needs to have this shape to be accepted by sklearn:
    X: (11, 1)
    y: (11, 1)
    Slope Coefficient: [24.68849397]
    Intercept Value: [-596.20648399]


Now we can use these estimated values to plot our line of best fit.

"Predicted values" in statistics are denoted with a caret or "hat" symbol over them: 

$\hat{m}, \hat{b}$


```python
m_hat = model.coef_[0]
b_hat = model.intercept_

# Plot the points
plt.scatter(df.Farenheit, df.Dollars)

# Plot the line
y_hat = [m_hat*x + b_hat for x in X]
plt.plot(X, y_hat)

# Graph axes and title
plt.ylabel("Dollars")
plt.xlabel("Farenheit")
plt.title("Ice Cream Sales and Temperature")
plt.show()
```

## Making Predictions

Now that we have established a line of best fit that summarizes the relationship that we see in the data, we can use the line of best fit in order to make predictions. How do we make predictions? We plug x values into our equation and see what the predicted y value is.

Suppose that the weather forecast said that the temperature outside tomorrow was going to be 70 degrees Farenheit. What would we predict that our ice cream sales would be?


```python
# Make a single prediction for 70 degrees farenheit
x_value = 70

# Plug our temperature into our equation (model)
prediction = m_hat*x_value + b_hat
print("Prediction", prediction)

# Plot the points
plt.scatter(df.Farenheit, df.Dollars)

# Plot the line
y_hat = [m_hat*x + b_hat for x in X]
plt.plot(X, y_hat)
plt.plot(x_value, prediction, 'ro')

# Graph axes and title
plt.ylabel("Dollars")
plt.xlabel("Farenheit")
plt.title("Ice Cream Sales and Temperature")
plt.show()
```

## Predicted Values

You'll notice that our predicted point (red) lies on our line of best fit. The line of best fit represents all of the possible predictions that we could make given some x value. Points that represent our predictions that lie on the line of best fit are called "predicted values" or "fitted values."

## Residual Error 

The residual error is the distance between points in our dataset and our regression line. How would we calculate this? Well what if we made a prediction for an x value that we already had in our dataset? Lets make a prediction for 55 degrees Farenheit.


```python
# Make a single prediction for 55 degrees farenheit
point = df.loc[df['Farenheit']==55].values[0]
x_actual = point[0]
y_actual = point[1]

prediction = m_hat * x_actual + b_hat
print("Prediction", prediction)

# Plot the points
plt.scatter(df.Farenheit, df.Dollars)

# Plot the line
y_hat = [m_hat*x + b_hat for x in X]
plt.plot(X, y_hat)
plt.plot(x_value, prediction, 'ro')

# Graph axes and title
plt.ylabel("Farenheit")
plt.xlabel("Dollars")
plt.title("Ice Cream Sales and Temperature")

# Plot the error
plt.arrow(x_actual, y_actual, 0, prediction-600, color='r')
plt.show()
```

The difference between the predicted and measured points is what is called the **residual error** - sometimes just called the **residual**. Some amount of error is unavoidable when making predictions, but we want our line of best fit to minimize this error (make it as small as possible). That's what makes it the line of best fit. 

# Bivariate Linear Regression with Linear Algebra

### Minimizing the Sum of the Squared Error

The most common method of estimating our $\beta$ parameters  is what's known as "Ordinary Least Squares" (OLS). (There are different methods of arriving at a line of best fit). OLS estimates the parameters that minimize the squared distance between each point in our dataset and our line of best fit. 

\begin{align}
SSE = \sum(y_i - \hat{y})^2
\end{align}

This can be cumputed using linear algebra in process known as the "Least Squares Solution:"



Before we can work with this equation in its linear algebra form we have to understand how to set up the matrices that are involved in this equation. 

### $y$ Vector

The $y$ vector is simple in that it's just a vector of all of our measured y values from our dataset. 

### $X$ Matrix

The $X$ matrix is a little bit more complicated since it needs to contain values that can be used to find both our slope and our intercept values. All that this means is that we have to add an extra column of 1s in addition to our regular columns of X values. Without this we wouldn't be able to computer our intercept value.

\begin{align}
y=mx+b1
\end{align}


### $\beta$ vector

The $\beta$ vector represents all the parameters that we are trying to estimate, our $y$ vector and $X$ matrix values are full of data from our dataset. The $\beta$ vector holds the variables that we are solving for: $\beta_0$ and $\beta_1$

Now that we have all of the necessary parts we can set them up in the following equation:

\begin{align}
y = X \beta + \epsilon
\end{align}

Since our $\epsilon$ value represents **random** error we can assume that it will equal zero on average.

\begin{align}
y = X \beta
\end{align}

The objective now is to isolate the $\beta$ matrix. We can do this by pre-multiplying both sides by "X transpose" $X^{T}$.

\begin{align}
X^{T}y =  X^{T}X \beta
\end{align}

Since anything times its transpose will result in a square matrix, if that matrix is then an invertible matrix, then we should be able to multiply both sides by its inverse to remove it from the right hand side. (We'll talk tomorrow about situations that could lead to $X^{T}X$ not being invertible.)

\begin{align}
(X^{T}X)^{-1}X^{T}y =  (X^{T}X)^{-1}X^{T}X \beta
\end{align}

Since any matrix multiplied by its inverse results in the identity matrix, and anything multiplied by the identity matrix is itself, we are left with only $\beta$ on the right hand side:

\begin{align}
(X^{T}X)^{-1}X^{T}y = \hat{\beta}
\end{align}

We will now call it "beta hat" $\hat{\beta}$ because it now represents our estimated values for $\beta_0$ and $\beta_1$

# Bivariate Regression with numpy


```python
df['Ones'] = np.ones(11)
df.head()
```




<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>Farenheit</th>
      <th>Dollars</th>
      <th>Ones</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>32</td>
      <td>200</td>
      <td>1.0</td>
    </tr>
    <tr>
      <th>1</th>
      <td>35</td>
      <td>400</td>
      <td>1.0</td>
    </tr>
    <tr>
      <th>2</th>
      <td>47</td>
      <td>500</td>
      <td>1.0</td>
    </tr>
    <tr>
      <th>3</th>
      <td>55</td>
      <td>600</td>
      <td>1.0</td>
    </tr>
    <tr>
      <th>4</th>
      <td>73</td>
      <td>1100</td>
      <td>1.0</td>
    </tr>
  </tbody>
</table>
</div>




```python
## Get our X matrix and y vector in the correct shape
X = df[['Ones', 'Farenheit']].values
Y = df['Dollars'].values.reshape(-1, 1)
print('X:', X, sep='\n')

X_transpose = X.T
print("\nX Transpose:", X_transpose, sep='\n')

X_transpose_X = np.matmul(X_transpose, X)
print("\nX Transpose X", X_transpose_X, sep='\n')

X_transpose_X_inverse = np.linalg.inv(X_transpose_X)
print("\nX Transpose X Inverse", X_transpose_X_inverse, sep='\n')

X_transpose_Y = np.matmul(X_transpose, Y)
print("\nX Transpose Y", X_transpose_Y, sep='\n')

beta_hat = np.matmul(X_transpose_X_inverse, X_transpose_Y)
print("\nBeta Hat", beta_hat, sep='\n')
```

    X:
    [[  1.  32.]
     [  1.  35.]
     [  1.  47.]
     [  1.  55.]
     [  1.  73.]
     [  1.  78.]
     [  1.  81.]
     [  1.  86.]
     [  1.  89.]
     [  1.  90.]
     [  1. 110.]]
    
    X Transpose:
    [[  1.   1.   1.   1.   1.   1.   1.   1.   1.   1.   1.]
     [ 32.  35.  47.  55.  73.  78.  81.  86.  89.  90. 110.]]
    
    X Transpose X
    [[1.1000e+01 7.7600e+02]
     [7.7600e+02 6.0974e+04]]
    
    X Transpose X Inverse
    [[ 8.89637865e-01 -1.13221862e-02]
     [-1.13221862e-02  1.60494908e-04]]
    
    X Transpose Y
    [[  12600.]
     [1042700.]]
    
    Beta Hat
    [[-596.20648399]
     [  24.68849397]]


# Multiple Regression
Simple or bivariate linear regression involves a single $x$ variable and a single $y$ variable. However, we can have many $x$ variables. A linear regression model that involves multiple x variables is known as **Multiple** Regression - NOT MULTIVARIATE!

Muliple Regression is simply an extension of the bivariate case. The reason why we see the bivariate case demonstrated so often is simply because it's easier to graph and all of the intuition from the bivariate case is the same as we keep on adding explanatory variables.

As we increase the number of  𝑥  values in our model we are simply fitting a n-1-dimensional plane to an n-dimensional cloud of points within an n-dimensional hypercube.





# Multiple Regression with SciKit-Learn

Predicting **systolic blood** pressure from **age** and **weight**:

$y_i = \beta_0 + \beta_1age_i + \beta_2weight_i + \epsilon $


```python
import pandas as pd

df = pd.read_excel('https://college.cengage.com/mathematics/brase/'
                   'understandable_statistics/7e/students/datasets/'
                   'mlr/excel/mlr02.xls', names=['y', 'age', 'weight'])
print(df.shape)
df.head()
```

    *** No CODEPAGE record, no encoding_override: will use 'ascii'
    (11, 3)





<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>y</th>
      <th>age</th>
      <th>weight</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>132</td>
      <td>52</td>
      <td>173</td>
    </tr>
    <tr>
      <th>1</th>
      <td>143</td>
      <td>59</td>
      <td>184</td>
    </tr>
    <tr>
      <th>2</th>
      <td>153</td>
      <td>67</td>
      <td>194</td>
    </tr>
    <tr>
      <th>3</th>
      <td>162</td>
      <td>73</td>
      <td>211</td>
    </tr>
    <tr>
      <th>4</th>
      <td>154</td>
      <td>64</td>
      <td>196</td>
    </tr>
  </tbody>
</table>
</div>




```python
from sklearn.linear_model import LinearRegression

# The Scikit-Learn code is the same every time.
y = df[['y']].values
X = df[['age','weight']].values

print(y.shape)
print(X.shape)

model = LinearRegression()
model.fit(X, y)

beta_0 = model.intercept_
beta_i = model.coef_[0]

print("Slope Coefficients: ", beta_i)
print("Intercept Value: ", beta_0)
```

    (11, 1)
    (11, 2)
    Slope Coefficients:  [0.86141469 0.3348592 ]
    Intercept Value:  [30.99410295]



```python
from mpl_toolkits.mplot3d import Axes3D
import matplotlib.pyplot as plt
import numpy as np

fig = plt.figure()
ax = fig.add_subplot(111, projection='3d')

# Plot Points on 3D plot
ax.scatter(df['age'], df['weight'], df['y'], c='red')

# Set axes to size of scatter data
x1 = np.array(ax.get_xlim())
y1 = np.array(ax.get_ylim())
xx, yy = np.meshgrid(x1,y1)
zz = beta_i[0]*xx + beta_i[1]*yy + beta_0[0]

# Plot Plane
plt3d = plt.gca(projection='3d')

# Add Opacity to Plane 
plt3d.plot_surface(xx, yy, zz,alpha=0.5)
plt.title("Regression Plane")

plt.show()
```

# Interpreting Coefficients

One of Linear Regression's strengths is that the parameters of the model (coefficients) are readily interpretable and useful. Not only do they describe the relationship between x and y but they put a number on just how much x is associated with y. We should be careful to not speak about this relationshiop in terms of causality because these coefficients are in fact correlative measures. We would need a host of additional techniques in order to estimate a causal effect using linear regression (econometrics).

$\hat{\beta} = \frac{Cov(x,y)}{Var(y)}$

Going back to the two equations for the two models that we have estimated so far, lets replace their beta values with their actual values to see if we can make sense of how to interpret these beta coefficients.

### Bivariate Model

$y_i = \beta_0 + \beta_1temperature + \epsilon$

$sales_i = -596.2 + 24.69temperature + \epsilon$

What might $\beta_0$ in this model represent? It represents the level of sales that we would have if temperature were 0. Since this is negative one way of interpreting it is that it's so cold outside that you would have to pay people to eat ice cream. A more appropriate interpretation is probably that the ice cream store owner should close his store down long before the temperature reaches 0 degrees farenheit (-17.7 celsius). The owner can compare his predicted sales with his costs of doing business to know how warm the weather has to get before he should open his store.

What might the $beta_1$ in this model represent? it represents the increase in sales for each degree of temperature increase. For every degree that the temperature goes up outside he has $25 more in sales.

### Multiple Regression Model

$y_i = \beta_0 + \beta_1age_i + \beta_2weight_i + \epsilon$

$BloodPressure_i = 30.99+ .86age_i + .33weight_i + \epsilon$

The interpretation of coefficients in this example are similar. The intercept value repesents the blood pressure a person would have if they were 0 years old and weighed 0 pounds. This not a super useful interpretation. If we look at our data it is unlikely that we have any measurements like these in the dataset. This means that our interpretation of our intercept likely comes from extrapolating the regression line (plane). Coefficients having straightforward interpretations is a strength of linear regression if we're careful about extrapolation and only interpreting our data within the context that it was gathered.

The interpretation of our other coefficients can be a useful indicator for how much a person similar to those in our dataset's blood pressure will go up on average with each additional year of age and pound of weight.

# R Squared:  $R^2$

$R^2$ is a statistical measure of how close the data are fitted to our regression line. A helpful interpretation for the $R^2$ is the percentage of the dependent variable that is explained by the model.

In other words, the $R^2$ is the percentage of y that is explained by the x variables included in the model. For this reason the $R^2$ is also known as the "coefficient of determination," because it explains how much of y is explained (or determined) by our x varaibles. We won't go into the calculation of $R^2$ today, just know that a higher $R^2$ percentage is nearly always better and indicates a model that fits the data more closely. 

# Basic Model Validation

One of the downsides of relying on $R^2$ too much is that although it tells you when you're fitting the data well, it doesn't tell you when you're *overfitting* the data. The best way to tell if you're overfitting the data is to get some data that your model hasn't seen yet, and evaluate how your predictions do. This is essentially what "model validation" is.

### Steps of Basic Model Validation

1. Split your labeled data into two halves.
2. Fit your linear regression model to one of the two halves. (We'll call this the training dataset)
3. Run the second half of your data through your model and make predictions. (On the test dataset)
4. Compare your predictions with the true values via some measure of "accuracy." (Root Mean Squared Error)

# Model Validation with SciKit-Learn


```python
# http://lib.stat.cmu.edu/datasets/boston

import pandas as pd
from sklearn.datasets import load_boston
```

```
Variables in order:
 CRIM     per capita crime rate by town
 ZN       proportion of residential land zoned for lots over 25,000 sq.ft.
 INDUS    proportion of non-retail business acres per town
 CHAS     Charles River dummy variable (= 1 if tract bounds river; 0 otherwise)
 NOX      nitric oxides concentration (parts per 10 million)
 RM       average number of rooms per dwelling
 AGE      proportion of owner-occupied units built prior to 1940
 DIS      weighted distances to five Boston employment centres
 RAD      index of accessibility to radial highways
 TAX      full-value property-tax rate per $10,000
 PTRATIO  pupil-teacher ratio by town
 B        1000(Bk - 0.63)^2 where Bk is the proportion of blacks by town
 LSTAT    % lower status of the population
 MEDV     Median value of owner-occupied homes in $1000's
```


```python
# Our target variable is MEDV
# MEDV: Median value of owner-occupied homes in $1000s

boston_dataset = load_boston()

df = pd.DataFrame(boston_dataset.data, columns=boston_dataset.feature_names)
df['MEDV'] = boston_dataset.target
print(df.shape)
df.head()
```

    (506, 14)





<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>CRIM</th>
      <th>ZN</th>
      <th>INDUS</th>
      <th>CHAS</th>
      <th>NOX</th>
      <th>RM</th>
      <th>AGE</th>
      <th>DIS</th>
      <th>RAD</th>
      <th>TAX</th>
      <th>PTRATIO</th>
      <th>B</th>
      <th>LSTAT</th>
      <th>MEDV</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>0.00632</td>
      <td>18.0</td>
      <td>2.31</td>
      <td>0.0</td>
      <td>0.538</td>
      <td>6.575</td>
      <td>65.2</td>
      <td>4.0900</td>
      <td>1.0</td>
      <td>296.0</td>
      <td>15.3</td>
      <td>396.90</td>
      <td>4.98</td>
      <td>24.0</td>
    </tr>
    <tr>
      <th>1</th>
      <td>0.02731</td>
      <td>0.0</td>
      <td>7.07</td>
      <td>0.0</td>
      <td>0.469</td>
      <td>6.421</td>
      <td>78.9</td>
      <td>4.9671</td>
      <td>2.0</td>
      <td>242.0</td>
      <td>17.8</td>
      <td>396.90</td>
      <td>9.14</td>
      <td>21.6</td>
    </tr>
    <tr>
      <th>2</th>
      <td>0.02729</td>
      <td>0.0</td>
      <td>7.07</td>
      <td>0.0</td>
      <td>0.469</td>
      <td>7.185</td>
      <td>61.1</td>
      <td>4.9671</td>
      <td>2.0</td>
      <td>242.0</td>
      <td>17.8</td>
      <td>392.83</td>
      <td>4.03</td>
      <td>34.7</td>
    </tr>
    <tr>
      <th>3</th>
      <td>0.03237</td>
      <td>0.0</td>
      <td>2.18</td>
      <td>0.0</td>
      <td>0.458</td>
      <td>6.998</td>
      <td>45.8</td>
      <td>6.0622</td>
      <td>3.0</td>
      <td>222.0</td>
      <td>18.7</td>
      <td>394.63</td>
      <td>2.94</td>
      <td>33.4</td>
    </tr>
    <tr>
      <th>4</th>
      <td>0.06905</td>
      <td>0.0</td>
      <td>2.18</td>
      <td>0.0</td>
      <td>0.458</td>
      <td>7.147</td>
      <td>54.2</td>
      <td>6.0622</td>
      <td>3.0</td>
      <td>222.0</td>
      <td>18.7</td>
      <td>396.90</td>
      <td>5.33</td>
      <td>36.2</td>
    </tr>
  </tbody>
</table>
</div>




```python
import seaborn as sns

sns.set(style="ticks", color_codes=True)

# Generate a list of column headers not including price
x_columns = df.columns.drop('MEDV')

# Only plot the scatterplot of x variables with our y variable
fig = sns.pairplot(data=df, y_vars=['MEDV'], x_vars=x_columns)
```


```python
from sklearn.model_selection import train_test_split

y = df['MEDV']
X = df.drop('MEDV', axis=1)

X_train, X_test, Y_train, Y_test = train_test_split(X, y, test_size=0.3)

print(X_train.shape)
print(X_test.shape) 
print(Y_train.shape)
print(Y_test.shape)
```

    (354, 13)
    (152, 13)
    (354,)
    (152,)



```python
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error, r2_score

model = LinearRegression()
model.fit(X_train, Y_train)

y_test_predict = model.predict(X_test)
RMSE = np.sqrt(mean_squared_error(Y_test, y_test_predict))
R2 = r2_score(Y_test, y_test_predict)

print("RMSE:", RMSE)
print("R^2:", R2)
```

    RMSE: 5.516101900280473
    R^2: 0.6526542967095016



```python
df.describe()
```




<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>CRIM</th>
      <th>ZN</th>
      <th>INDUS</th>
      <th>CHAS</th>
      <th>NOX</th>
      <th>RM</th>
      <th>AGE</th>
      <th>DIS</th>
      <th>RAD</th>
      <th>TAX</th>
      <th>PTRATIO</th>
      <th>B</th>
      <th>LSTAT</th>
      <th>MEDV</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>count</th>
      <td>506.000000</td>
      <td>506.000000</td>
      <td>506.000000</td>
      <td>506.000000</td>
      <td>506.000000</td>
      <td>506.000000</td>
      <td>506.000000</td>
      <td>506.000000</td>
      <td>506.000000</td>
      <td>506.000000</td>
      <td>506.000000</td>
      <td>506.000000</td>
      <td>506.000000</td>
      <td>506.000000</td>
    </tr>
    <tr>
      <th>mean</th>
      <td>3.613524</td>
      <td>11.363636</td>
      <td>11.136779</td>
      <td>0.069170</td>
      <td>0.554695</td>
      <td>6.284634</td>
      <td>68.574901</td>
      <td>3.795043</td>
      <td>9.549407</td>
      <td>408.237154</td>
      <td>18.455534</td>
      <td>356.674032</td>
      <td>12.653063</td>
      <td>22.532806</td>
    </tr>
    <tr>
      <th>std</th>
      <td>8.601545</td>
      <td>23.322453</td>
      <td>6.860353</td>
      <td>0.253994</td>
      <td>0.115878</td>
      <td>0.702617</td>
      <td>28.148861</td>
      <td>2.105710</td>
      <td>8.707259</td>
      <td>168.537116</td>
      <td>2.164946</td>
      <td>91.294864</td>
      <td>7.141062</td>
      <td>9.197104</td>
    </tr>
    <tr>
      <th>min</th>
      <td>0.006320</td>
      <td>0.000000</td>
      <td>0.460000</td>
      <td>0.000000</td>
      <td>0.385000</td>
      <td>3.561000</td>
      <td>2.900000</td>
      <td>1.129600</td>
      <td>1.000000</td>
      <td>187.000000</td>
      <td>12.600000</td>
      <td>0.320000</td>
      <td>1.730000</td>
      <td>5.000000</td>
    </tr>
    <tr>
      <th>25%</th>
      <td>0.082045</td>
      <td>0.000000</td>
      <td>5.190000</td>
      <td>0.000000</td>
      <td>0.449000</td>
      <td>5.885500</td>
      <td>45.025000</td>
      <td>2.100175</td>
      <td>4.000000</td>
      <td>279.000000</td>
      <td>17.400000</td>
      <td>375.377500</td>
      <td>6.950000</td>
      <td>17.025000</td>
    </tr>
    <tr>
      <th>50%</th>
      <td>0.256510</td>
      <td>0.000000</td>
      <td>9.690000</td>
      <td>0.000000</td>
      <td>0.538000</td>
      <td>6.208500</td>
      <td>77.500000</td>
      <td>3.207450</td>
      <td>5.000000</td>
      <td>330.000000</td>
      <td>19.050000</td>
      <td>391.440000</td>
      <td>11.360000</td>
      <td>21.200000</td>
    </tr>
    <tr>
      <th>75%</th>
      <td>3.677083</td>
      <td>12.500000</td>
      <td>18.100000</td>
      <td>0.000000</td>
      <td>0.624000</td>
      <td>6.623500</td>
      <td>94.075000</td>
      <td>5.188425</td>
      <td>24.000000</td>
      <td>666.000000</td>
      <td>20.200000</td>
      <td>396.225000</td>
      <td>16.955000</td>
      <td>25.000000</td>
    </tr>
    <tr>
      <th>max</th>
      <td>88.976200</td>
      <td>100.000000</td>
      <td>27.740000</td>
      <td>1.000000</td>
      <td>0.871000</td>
      <td>8.780000</td>
      <td>100.000000</td>
      <td>12.126500</td>
      <td>24.000000</td>
      <td>711.000000</td>
      <td>22.000000</td>
      <td>396.900000</td>
      <td>37.970000</td>
      <td>50.000000</td>
    </tr>
  </tbody>
</table>
</div>



d14358bfa066373426633a86a9cc4254ac66e845
ipynb
Jupyter Notebook
06-Linear-Regression/01_Linear_Regression.ipynb
shalevy1/data-science-journal
2a6beaf5bf328e257b638a695983457a9f3cd7ce
MIT
2019-03-05T04:44:48.000Z
2022-03-24T09:47:48.000Z
06-Linear-Regression/01_Linear_Regression.ipynb
pesobreiro/data-science-journal
82a72b4ed5ce380988fac17b0acd97254c2b5c86
MIT
06-Linear-Regression/01_Linear_Regression.ipynb
pesobreiro/data-science-journal
82a72b4ed5ce380988fac17b0acd97254c2b5c86
MIT
2019-03-07T05:08:03.000Z
2022-01-05T11:32:51.000Z
Qwen/Qwen-72B
1. YES
2. YES