# Linear Regression

We will use a real-life dataset containing information about size and weight for several fish species.

1. species name of fish (Bream, Roach, Whitefish, Parkki, Perch, Pike, Smelt)
2. weight of fish in grams
3. vertical length in cm (`Length1`)
4. diagonal length in cm (`Length2`)
5. cross length in cm (`Length3`)
6. height in cm
7. diagonal width in cm

Now, it turns out that the relationship between total length (L, in inches) and total weight (W, in pounds) for nearly all species of fish can be expressed by an equation of the form

\begin{align}
W = cL^b
\end{align}

`W` is weight in pounds and `L` is length in inches. `b` is a constant that is close to 3.0 for all species and `c` is a constant that varies among species (For [northern pike](https://en.wikipedia.org/wiki/Northern_pike#Length_and_weight), for example, `b = 3.096` and `c = 0.000180`).

Nevertheless, we would like to see if we can develop a way to predict the weight of a fish given its dimensions.

## Task

**Can we predict the weight of a fish given its dimensions and species?**

## Acquiring the data
First let's load the data into a [Pandas](https://mazelife.github.io/cmsc-291/lecture_10.html#pandas) dataframe:


```python
import pandas

fish_dataframe = pandas.read_csv("Fish.csv", dtype={"Species": "category"})
fish_dataframe
```




<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>Species</th>
      <th>Weight</th>
      <th>Length1</th>
      <th>Length2</th>
      <th>Length3</th>
      <th>Height</th>
      <th>Width</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>Bream</td>
      <td>242.0</td>
      <td>23.2</td>
      <td>25.4</td>
      <td>30.0</td>
      <td>11.5200</td>
      <td>4.0200</td>
    </tr>
    <tr>
      <th>1</th>
      <td>Bream</td>
      <td>290.0</td>
      <td>24.0</td>
      <td>26.3</td>
      <td>31.2</td>
      <td>12.4800</td>
      <td>4.3056</td>
    </tr>
    <tr>
      <th>2</th>
      <td>Bream</td>
      <td>340.0</td>
      <td>23.9</td>
      <td>26.5</td>
      <td>31.1</td>
      <td>12.3778</td>
      <td>4.6961</td>
    </tr>
    <tr>
      <th>3</th>
      <td>Bream</td>
      <td>363.0</td>
      <td>26.3</td>
      <td>29.0</td>
      <td>33.5</td>
      <td>12.7300</td>
      <td>4.4555</td>
    </tr>
    <tr>
      <th>4</th>
      <td>Bream</td>
      <td>430.0</td>
      <td>26.5</td>
      <td>29.0</td>
      <td>34.0</td>
      <td>12.4440</td>
      <td>5.1340</td>
    </tr>
    <tr>
      <th>...</th>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
    </tr>
    <tr>
      <th>154</th>
      <td>Smelt</td>
      <td>12.2</td>
      <td>11.5</td>
      <td>12.2</td>
      <td>13.4</td>
      <td>2.0904</td>
      <td>1.3936</td>
    </tr>
    <tr>
      <th>155</th>
      <td>Smelt</td>
      <td>13.4</td>
      <td>11.7</td>
      <td>12.4</td>
      <td>13.5</td>
      <td>2.4300</td>
      <td>1.2690</td>
    </tr>
    <tr>
      <th>156</th>
      <td>Smelt</td>
      <td>12.2</td>
      <td>12.1</td>
      <td>13.0</td>
      <td>13.8</td>
      <td>2.2770</td>
      <td>1.2558</td>
    </tr>
    <tr>
      <th>157</th>
      <td>Smelt</td>
      <td>19.7</td>
      <td>13.2</td>
      <td>14.3</td>
      <td>15.2</td>
      <td>2.8728</td>
      <td>2.0672</td>
    </tr>
    <tr>
      <th>158</th>
      <td>Smelt</td>
      <td>19.9</td>
      <td>13.8</td>
      <td>15.0</td>
      <td>16.2</td>
      <td>2.9322</td>
      <td>1.8792</td>
    </tr>
  </tbody>
</table>
<p>159 rows × 7 columns</p>
</div>



## Reationship between weight and length in seven species of fish


```python
import seaborn
# Seaborn has various themes/styles. We'll pick a good default:
seaborn.set_theme(style="darkgrid")
seaborn.set_context("talk")

seaborn.relplot(x="Length1", y="Weight", hue="Species", data=fish_dataframe, height=8, aspect=1.5)
```

## Simplifying the problem

For the moment, let's attempt to do bivariate linear regression. Let's take *one* measure of dimension (length) and attempt to predict weight for just one species: Pike. We'll start with a scatter plot with just the Pike data:


```python
pike_dataframe = fish_dataframe[fish_dataframe.Species == "Pike"]
seaborn.relplot(x="Length1", y="Weight", data=pike_dataframe, height=8, aspect=1.5)
```

## Training a linear regression model with Scikit-Learn



Now we will train a linear regression model using the [scikit-Learn](https://scikit-learn.org/) Python library. Luciky, newer versions of this library operate pretty seamlessley with Pandas.

We will start by isolating the independent and dependent variables:


```python
x = pike_dataframe[["Length1"]]  # Independent variable, AKA feature
y = pike_dataframe["Weight"]  # Dependent varable


# Represent your features as a Pandas DataFrame, and your predictor as a Pandas Series
print("NOTE: x is a {}, y is a {}.".format(type(x), type(y)))
```

    NOTE: x is a <class 'pandas.core.frame.DataFrame'>, y is a <class 'pandas.core.series.Series'>.


Next we will create an instance of the `LinearRegression` class and then call `fit()` on it, which will train the model.


```python
from sklearn.linear_model import LinearRegression

pike_model = LinearRegression()
pike_model.fit(x, y)
```




    LinearRegression()



Once we have fit the model, we can ask it to do predictions. As we talked about last week, a linear regression model gives us an equation that describes a line. One common form for linear equations is slope-intercept form:

\begin{align}
y = mx + b \\
m = slope \\
b = intercept
\end{align}

In stats and ML the equation is formulated somewhat differently:

\begin{align}
y = \beta_0 + \beta_1x \\
\beta_0 = intercept \\
\beta_1 = slope
\end{align}

Our linear model tells us what $\beta_0$ and $\beta_1$ are (these are called the *coefficients*), so that when we plug in a value for `x`, we will have our prediction for `y`.  We can extract those coefficients from the scikit-learn model we just trained.


```python
intercept = pike_model.intercept_
slope = pike_model.coef_[0]
print(f"Intercept is: {intercept}")
print(f"Slope is: {slope}")
```

    Intercept is: -1540.824272231702
    Slope is: 53.19486584675106


We can also draw the line that our model has produced on top of the scatterplot to see how well it seems to fit:


```python
import matplotlib.pyplot as plt
import numpy


# Generate an array of 2 points at the smallest and largest x values in the dataset:
x_min = min(x["Length1"])
x_max = max(x["Length1"])

x_pred = numpy.array([x_min, x_max])
# Create the corresponding y values for the two X points using the linear equation:
y_pred = intercept + (slope * x_pred)

print(f"Predicted x values: {x_pred}")
print(f"Predicted y values: {y_pred}")


seaborn.relplot(x="Length1", y="Weight", data=pike_dataframe, height=8, aspect=1.5)
plt.plot(x_pred, y_pred)
```

## Predicting

We don't have to manually predict fish weights from a list of given lengths by manually calculating them using the model's coefficients. The instance of our `LinearRegression` class has a method called `predict()` that takes a list of lengths and gives us a the corresponding list of heights.


```python
new_x_values = pandas.DataFrame([37.8, 45.3, 52.2], columns=["Length1"])
pike_model.predict(new_x_values)
```




    array([ 469.94165678,  868.90315063, 1235.94772497])



## Evaluating the model

Remember **mean-squared error**?

$$
MSE = \frac{1}{n}\sum_{i=1}^{n} ({Y_{i} - \hat{Y_{i}}})^2
$$


    

Let's calculate it! Scikit-learn will help us here, we just need to give it two arrays: one of predicted y values and one of actual y values.


```python
from sklearn.metrics import mean_squared_error

predicted_y = pike_model.predict(x)
pike_mse = mean_squared_error(y, predicted_y)

print(f"Our model has a mean squared error of: {pike_mse:,.2f}")
pandas.DataFrame({'Actual': y, 'Predicted': predicted_y})
```

    Our model has a mean squared error of: 14,545.66





<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>Actual</th>
      <th>Predicted</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>128</th>
      <td>200.0</td>
      <td>-19.593766</td>
    </tr>
    <tr>
      <th>129</th>
      <td>300.0</td>
      <td>78.075971</td>
    </tr>
    <tr>
      <th>130</th>
      <td>300.0</td>
      <td>135.528757</td>
    </tr>
    <tr>
      <th>131</th>
      <td>300.0</td>
      <td>256.179609</td>
    </tr>
    <tr>
      <th>132</th>
      <td>430.0</td>
      <td>296.396559</td>
    </tr>
    <tr>
      <th>133</th>
      <td>345.0</td>
      <td>325.122952</td>
    </tr>
    <tr>
      <th>134</th>
      <td>456.0</td>
      <td>554.934098</td>
    </tr>
    <tr>
      <th>135</th>
      <td>510.0</td>
      <td>554.934098</td>
    </tr>
    <tr>
      <th>136</th>
      <td>540.0</td>
      <td>560.679376</td>
    </tr>
    <tr>
      <th>137</th>
      <td>500.0</td>
      <td>669.839670</td>
    </tr>
    <tr>
      <th>138</th>
      <td>567.0</td>
      <td>738.783014</td>
    </tr>
    <tr>
      <th>139</th>
      <td>770.0</td>
      <td>830.707472</td>
    </tr>
    <tr>
      <th>140</th>
      <td>950.0</td>
      <td>1031.792224</td>
    </tr>
    <tr>
      <th>141</th>
      <td>1250.0</td>
      <td>1244.367533</td>
    </tr>
    <tr>
      <th>142</th>
      <td>1600.0</td>
      <td>1474.178679</td>
    </tr>
    <tr>
      <th>143</th>
      <td>1550.0</td>
      <td>1474.178679</td>
    </tr>
    <tr>
      <th>144</th>
      <td>1650.0</td>
      <td>1646.537038</td>
    </tr>
  </tbody>
</table>
</div>



## Other goodness-of-fit measures: R-squared

Also known as the coefficient of determination. It is:

$$
R^2 = \frac{\textrm{Variance explained by model}}{\textrm{Total variance}}
$$


We calculate two values to get this:

1. $SSE$ - Sum of Squared Errors (see MSE)
2. $SST$ - Sum of squares total

$SST$ is calculated by:
$$
\textrm{SST} = \sum_{i=1}^{n} ({Y_{i} - \bar{Y_{i}}})^2
$$

Finally:

$$
R^2 = \frac{SST - SSE}{SST} = \frac{\textrm{Variance explained by model}}{\textrm{Total variance}}
$$

This is a percentage, meaning it will give you a value between 0 and 1. The closer to 1, the better your model fit is.

Luckily we don't have to calculate this by hand. Scikit-learn will do this for us:


```python
from sklearn.metrics import r2_score

# This function takes the same arguments sklearn.metrics.mean_squared_error does.
r2_score(y, predicted_y)
```




    0.9367062212730073



## Overfitting

The only problem here is we don't know if we've overfit. 




A better way to measure how well our model is to split our data into training and test datasets. Scikit-learn will help us with this too:


```python
from sklearn.model_selection import train_test_split

# Split 20% of our data off into testing data:
train, test = train_test_split(pike_dataframe, test_size=0.2)

x_train = train[["Length1"]]
y_train = train["Weight"]

x_test = test[["Length1"]]
y_test = test["Weight"]

# Train our model as usual with the 80% of the data we reserved for training:
pike_model = LinearRegression()
pike_model.fit(x_train, y_train)

# Then have it predict X values
predicted_y = pike_model.predict(x_test)
print("Mean-squared error: {:,.2f}".format(mean_squared_error(y_test, predicted_y)))
print("R-squared: {:,.2f}".format(r2_score(y_test, predicted_y) * 100))
pandas.DataFrame({'Actual Weight': y_test, 'Predicted Weight': predicted_y})
```

    Mean-squared error: 10,776.73
    R-squared: 84.70





<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>Actual Weight</th>
      <th>Predicted Weight</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>131</th>
      <td>300.0</td>
      <td>296.055034</td>
    </tr>
    <tr>
      <th>140</th>
      <td>950.0</td>
      <td>1037.354786</td>
    </tr>
    <tr>
      <th>129</th>
      <td>300.0</td>
      <td>125.830646</td>
    </tr>
    <tr>
      <th>135</th>
      <td>510.0</td>
      <td>581.592716</td>
    </tr>
  </tbody>
</table>
</div>



## Polynomial Regression

Our model works...OK. But the MSE on "real-world" data is somewhat high. You could say that out model under-fits the data. To overcome under-fitting, we need to increase the complexity of the model. What if we try to fit this line with a curve?





```python
from sklearn.preprocessing import PolynomialFeatures 

poly_reg = PolynomialFeatures(degree=2)  # More on this later!
x_poly = poly_reg.fit_transform(x)


poly_pike_model = LinearRegression()
poly_pike_model.fit(x_poly, y)
```




    LinearRegression()




```python
x_min = min(x["Length1"])
x_max = max(x["Length1"])


x_grid = numpy.arange(x_min, x_max, 0.1)
x_grid = x_grid.reshape((len(x_grid), 1))

# Create the corresponding y values for the X points using a polynomial model:
y_grid = poly_pike_model.predict(poly_reg.fit_transform(x_grid))

seaborn.relplot(x="Length1", y="Weight", data=pike_dataframe, height=8, aspect=1.5)
plt.plot(x_grid, y_grid, color="red")
```

A second-degree polynomial linear equation:

$$
y = \beta_0 + \beta_1x + \beta_2x^2
$$


```python
print("beta_0 is {}".format(poly_pike_model.intercept_))
print("beta_1 and beta_2 are {}".format(poly_pike_model.coef_[1:]))


_, beta_1, beta_2 = poly_pike_model.coef_

def predict_y(x):
    return poly_pike_model.intercept_ + (beta_1 * x) + (beta_2 * x * x)


predict_y(40)

```

    beta_0 is 1048.4181803094582
    beta_1 and beta_2 are [-68.10909876   1.36275911]





    504.46880862618514




```python
predicted_y = poly_pike_model.predict(x_poly)

print("Mean-squared error: {:,.2f}".format(mean_squared_error(y, predicted_y)))
print("R-squared: {:,.2f}".format(r2_score(y, predicted_y) * 100))
```

    Mean-squared error: 3,536.77
    R-squared: 98.46


9bf17b2cc155aeb36024a8936e8e5388d27b689f
ipynb
Jupyter Notebook
cmsc_210/examples/lecture_25/notebooks/Linear-Regression.ipynb
mazelife/cmsc-210
dbaa1604ef49bcfe5a70e09c17fbd243a8b80220
MIT
cmsc_210/examples/lecture_25/notebooks/Linear-Regression.ipynb
mazelife/cmsc-210
dbaa1604ef49bcfe5a70e09c17fbd243a8b80220
MIT
2022-01-16T23:30:12.000Z
2022-01-30T23:03:21.000Z
cmsc_210/examples/lecture_25/notebooks/Linear-Regression.ipynb
mazelife/cmsc-210
dbaa1604ef49bcfe5a70e09c17fbd243a8b80220
MIT
Qwen/Qwen-72B
1. YES
2. YES