# Linear Regression

We will use a real-life dataset containing information about size and weight for several fish species.

1. species name of fish (Bream, Roach, Whitefish, Parkki, Perch, Pike, Smelt)
2. weight of fish in grams
3. vertical length in cm (`Length1`)
4. diagonal length in cm (`Length2`)
5. cross length in cm (`Length3`)
6. height in cm
7. diagonal width in cm

We would like to see if we can develop a way to predict the weight of a fish given its dimensions.


The relationship between total length (L, in inches) and total weight (W, in pounds) for nearly all species of fish can be expressed by an equation of the form

\begin{align}
W = cL^b
\end{align}

`W` is weight in pounds and `L` is length in inches. `b` is a constant that is close to 3.0 for all species and `c` is a constance that varies among species (For [northern pike](https://en.wikipedia.org/wiki/Northern_pike#Length_and_weight), for example, `b = 3.096` and `c = 0.000180`).

## Task

**Can we predict the weight of a fish given its dimensions and species?**

## Acquiring the data
First let's load the data into a [Pandas](https://mazelife.github.io/cmsc-291/lecture_10.html#pandas) dataframe:


```python
import pandas

fish_dataframe = pandas.read_csv("Fish.csv", dtype={"Species": "category"})
fish_dataframe
```




<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>Species</th>
      <th>Weight</th>
      <th>Length1</th>
      <th>Length2</th>
      <th>Length3</th>
      <th>Height</th>
      <th>Width</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>Bream</td>
      <td>242.0</td>
      <td>23.2</td>
      <td>25.4</td>
      <td>30.0</td>
      <td>11.5200</td>
      <td>4.0200</td>
    </tr>
    <tr>
      <th>1</th>
      <td>Bream</td>
      <td>290.0</td>
      <td>24.0</td>
      <td>26.3</td>
      <td>31.2</td>
      <td>12.4800</td>
      <td>4.3056</td>
    </tr>
    <tr>
      <th>2</th>
      <td>Bream</td>
      <td>340.0</td>
      <td>23.9</td>
      <td>26.5</td>
      <td>31.1</td>
      <td>12.3778</td>
      <td>4.6961</td>
    </tr>
    <tr>
      <th>3</th>
      <td>Bream</td>
      <td>363.0</td>
      <td>26.3</td>
      <td>29.0</td>
      <td>33.5</td>
      <td>12.7300</td>
      <td>4.4555</td>
    </tr>
    <tr>
      <th>4</th>
      <td>Bream</td>
      <td>430.0</td>
      <td>26.5</td>
      <td>29.0</td>
      <td>34.0</td>
      <td>12.4440</td>
      <td>5.1340</td>
    </tr>
    <tr>
      <th>...</th>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
      <td>...</td>
    </tr>
    <tr>
      <th>154</th>
      <td>Smelt</td>
      <td>12.2</td>
      <td>11.5</td>
      <td>12.2</td>
      <td>13.4</td>
      <td>2.0904</td>
      <td>1.3936</td>
    </tr>
    <tr>
      <th>155</th>
      <td>Smelt</td>
      <td>13.4</td>
      <td>11.7</td>
      <td>12.4</td>
      <td>13.5</td>
      <td>2.4300</td>
      <td>1.2690</td>
    </tr>
    <tr>
      <th>156</th>
      <td>Smelt</td>
      <td>12.2</td>
      <td>12.1</td>
      <td>13.0</td>
      <td>13.8</td>
      <td>2.2770</td>
      <td>1.2558</td>
    </tr>
    <tr>
      <th>157</th>
      <td>Smelt</td>
      <td>19.7</td>
      <td>13.2</td>
      <td>14.3</td>
      <td>15.2</td>
      <td>2.8728</td>
      <td>2.0672</td>
    </tr>
    <tr>
      <th>158</th>
      <td>Smelt</td>
      <td>19.9</td>
      <td>13.8</td>
      <td>15.0</td>
      <td>16.2</td>
      <td>2.9322</td>
      <td>1.8792</td>
    </tr>
  </tbody>
</table>
<p>159 rows × 7 columns</p>
</div>




```python
import seaborn
# Seaborn has various themes/styles. We'll pick a good default:
seaborn.set_theme(style="darkgrid")
seaborn.set_context("talk")

seaborn.relplot(x="Length1", y="Weight", hue="Species", data=fish_dataframe, height=8, aspect=1.5)
```




    <seaborn.axisgrid.FacetGrid at 0x7fdb387193d0>



## Simplifying the problem

For the moment, let's attempt to do univariate linear regression. Let's take *one* measure of dimension (length) and attempt to predict weight for just one species: Pike. We'll start with a scatter plot with just the Pike data:


```python
pike_dataframe = fish_dataframe[fish_dataframe.Species == "Pike"]
seaborn.relplot(x="Length1", y="Weight", data=pike_dataframe, height=8, aspect=1.5)
```

Now we will train a linear regression model using the [scikit-Learn](https://scikit-learn.org/) Python library. One of the first things we will need to do is extract our `X` and `Y` values as individual arrays from the source dataframe, as scikit-learn does not work directly with Pandas. For our case, we want to extract a single DataFrame column for our X values and a single column for our Y values as a pair of arrays. The way to do this is as follows:

```
data_array = dataframe["column name"].values.reshape(-1, 1)
```

From there we will create an instance of the `LinearRegression` and then call `fit()` on it, which will train the model.


```python
from sklearn.linear_model import LinearRegression

x_values = pike_dataframe.Length1.values.reshape(-1, 1)
y_values = pike_dataframe.Weight.values.reshape(-1, 1)
pike_model = LinearRegression()
pike_model.fit(x_values, y_values)

x_values
```




    array([[30. ],
           [31.7],
           [32.7],
           [34.8],
           [35.5],
           [36. ],
           [40. ],
           [40. ],
           [40.1],
           [42. ],
           [43.2],
           [44.8],
           [48.3],
           [52. ],
           [56. ],
           [56. ],
           [59. ]])



Once we have fit the model, we can ask it to do predictions. As we talked about last week, a linear regression model gives us an equation that describes a line:

\begin{align}
y = mx + b \\
m = slope \\
b = intercept
\end{align}

The model tells us what `m` and `b` are (these are called the *coefficients*), so that when we plug in a value for `x`, we will have our prediction for `y`. We can extract those coefficients from the scikit-learn model we just trained.


```python
intercept = pike_model.intercept_[0]
slope = pike_model.coef_[0][0]
print(f"Intercept is: {intercept}")
print(f"Slope is: {slope}")
```

    Intercept is: -1540.824272231702
    Slope is: 53.19486584675106


We can also draw the line that our model has produced on top of the scatterplot to see how well it seems to fit:


```python
import matplotlib.pyplot as plt
import numpy

# Generate an array of 2 points at 27 and 60 on the x-axis:
x_pred = numpy.array([27, 60])
# Create the corresponding y values for the two X points:
y_pred = slope * x_pred + intercept

seaborn.relplot(x="Length1", y="Weight", data=pike_dataframe, height=8, aspect=1.5)
plt.plot(x_pred, y_pred)
```

## Evaluating the model

Remember **mean-squared error**?

    

Let's calculate it! Scikit-learn will help us here, we just need to give it two arrays: one of predicted y values and one of actual y values.


```python
from sklearn.metrics import mean_squared_error

predicted_y = pike_model.predict(x_values)
pike_mse = mean_squared_error(y_values, predicted_y)

print(f"MSE is {pike_mse:,.2f}")
pandas.DataFrame({'Actual': y_values.flatten(), 'Predicted': predicted_y.flatten()})

```

    MSE is 12,692.68





<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>Actual</th>
      <th>Predicted</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>200.0</td>
      <td>55.021703</td>
    </tr>
    <tr>
      <th>1</th>
      <td>300.0</td>
      <td>145.452975</td>
    </tr>
    <tr>
      <th>2</th>
      <td>300.0</td>
      <td>198.647841</td>
    </tr>
    <tr>
      <th>3</th>
      <td>300.0</td>
      <td>310.357059</td>
    </tr>
    <tr>
      <th>4</th>
      <td>430.0</td>
      <td>347.593465</td>
    </tr>
    <tr>
      <th>5</th>
      <td>345.0</td>
      <td>374.190898</td>
    </tr>
    <tr>
      <th>6</th>
      <td>456.0</td>
      <td>586.970362</td>
    </tr>
    <tr>
      <th>7</th>
      <td>510.0</td>
      <td>586.970362</td>
    </tr>
    <tr>
      <th>8</th>
      <td>540.0</td>
      <td>592.289848</td>
    </tr>
    <tr>
      <th>9</th>
      <td>500.0</td>
      <td>693.360093</td>
    </tr>
    <tr>
      <th>10</th>
      <td>567.0</td>
      <td>757.193932</td>
    </tr>
    <tr>
      <th>11</th>
      <td>770.0</td>
      <td>842.305718</td>
    </tr>
    <tr>
      <th>12</th>
      <td>950.0</td>
      <td>1028.487748</td>
    </tr>
    <tr>
      <th>13</th>
      <td>1250.0</td>
      <td>1225.308752</td>
    </tr>
    <tr>
      <th>14</th>
      <td>1600.0</td>
      <td>1438.088215</td>
    </tr>
    <tr>
      <th>15</th>
      <td>1550.0</td>
      <td>1438.088215</td>
    </tr>
    <tr>
      <th>16</th>
      <td>1650.0</td>
      <td>1597.672813</td>
    </tr>
  </tbody>
</table>
</div>



## Overfitting

The only problem here is we don't know if we've overfit. 




A better way to measure how well our model is to split our data into training and test datasets. Scikit-learn will help us with this too:


```python
from sklearn.model_selection import train_test_split


# Split 20% of our data off into testing data:
x_training_data, x_test_data, y_training_data, y_test_data = train_test_split(x_values, y_values, test_size=0.2, random_state=0)

# Train our model as usual with the 80% of the data we reserved for training:
pike_model = LinearRegression()
pike_model.fit(x_training_data, y_training_data)

# Then have it predict X values
predicted_y = pike_model.predict(x_test_data)
print("Mean-squared error: {:,.2f}".format(mean_squared_error(y_test_data, predicted_y)))
pandas.DataFrame({'Actual': y_test_data.flatten(), 'Predicted': predicted_y.flatten()})
```

    Mean-squared error: 22,682.16





<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>Actual</th>
      <th>Predicted</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>300.0</td>
      <td>153.707099</td>
    </tr>
    <tr>
      <th>1</th>
      <td>456.0</td>
      <td>601.323002</td>
    </tr>
    <tr>
      <th>2</th>
      <td>540.0</td>
      <td>606.715965</td>
    </tr>
    <tr>
      <th>3</th>
      <td>500.0</td>
      <td>709.182256</td>
    </tr>
  </tbody>
</table>
</div>



## Polynomial Regression

Our model works...OK. But the MSE on "real-world" data is somewhat high. You could say that out model under-fits the data. To overcome under-fitting, we need to increase the complexity of the model. What if we try to fit this line with a curve?



```python
from sklearn.preprocessing import PolynomialFeatures 

poly_reg = PolynomialFeatures(degree=2)
x_poly = poly_reg.fit_transform(x_values)
#poly_reg.fit(x_poly,y_values)
poly_pike_model = LinearRegression()
poly_pike_model.fit(x_poly,y_values)

```




    LinearRegression()




```python
# Generate an array of 2 points at 27 and 60 on the x-axis:
x_pred = numpy.array([27, 60])
# Create the corresponding y values for the two X points:
y_pred = slope * x_pred + intercept

seaborn.relplot(x="Length1", y="Weight", data=pike_dataframe, height=8, aspect=1.5)
plt.plot(x_pred, y_pred, color="red")


print("Mean-squared error: {}".format(mean_squared_error(y_test_data, predicted_y)))
pandas.DataFrame({'Actual': y_test_data.flatten(), 'Predicted': predicted_y.flatten()})
```


```python
# Generate an array of points:
x_grid = numpy.arange(min(x_values), max(x_values), 0.1)
x_grid = x_grid.reshape((len(x_grid), 1))

# Create the corresponding y values for the X points using a polynomial model:
y_grid = poly_pike_model.predict(poly_reg.fit_transform(x_grid))

seaborn.relplot(x="Length1", y="Weight", data=pike_dataframe, height=8, aspect=1.5)
plt.plot(x_grid, y_grid, color="red")


predicted_y = poly_pike_model.predict(poly_reg.fit_transform(x_values))
print("Mean-squared error: {}".format(mean_squared_error(y_values, predicted_y)))
```


```python
x_values
```




    array([[30. ],
           [31.7],
           [32.7],
           [34.8],
           [35.5],
           [36. ],
           [40. ],
           [40. ],
           [40.1],
           [42. ],
           [43.2],
           [44.8],
           [48.3],
           [52. ],
           [56. ],
           [56. ],
           [59. ]])




```python
x_poly
```




    array([[1.00000e+00, 3.00000e+01, 9.00000e+02],
           [1.00000e+00, 3.17000e+01, 1.00489e+03],
           [1.00000e+00, 3.27000e+01, 1.06929e+03],
           [1.00000e+00, 3.48000e+01, 1.21104e+03],
           [1.00000e+00, 3.55000e+01, 1.26025e+03],
           [1.00000e+00, 3.60000e+01, 1.29600e+03],
           [1.00000e+00, 4.00000e+01, 1.60000e+03],
           [1.00000e+00, 4.00000e+01, 1.60000e+03],
           [1.00000e+00, 4.01000e+01, 1.60801e+03],
           [1.00000e+00, 4.20000e+01, 1.76400e+03],
           [1.00000e+00, 4.32000e+01, 1.86624e+03],
           [1.00000e+00, 4.48000e+01, 2.00704e+03],
           [1.00000e+00, 4.83000e+01, 2.33289e+03],
           [1.00000e+00, 5.20000e+01, 2.70400e+03],
           [1.00000e+00, 5.60000e+01, 3.13600e+03],
           [1.00000e+00, 5.60000e+01, 3.13600e+03],
           [1.00000e+00, 5.90000e+01, 3.48100e+03]])




```python
simple_xs = numpy.array([[1], [2], [3]])
simple_xs
```




    array([[1],
           [2],
           [3]])




```python
PolynomialFeatures(degree=2).fit_transform(simple_xs)

```




    array([[1., 1., 1.],
           [1., 2., 4.],
           [1., 3., 9.]])




```python

```

bd86f120f2ad1f46137b3aa87133431bde0dffc6
ipynb
Jupyter Notebook
cmsc_210/examples/lecture_27/notebooks/Linear-Regression.ipynb
mazelife/cmsc-210
dbaa1604ef49bcfe5a70e09c17fbd243a8b80220
MIT
cmsc_210/examples/lecture_27/notebooks/Linear-Regression.ipynb
mazelife/cmsc-210
dbaa1604ef49bcfe5a70e09c17fbd243a8b80220
MIT
2022-01-16T23:30:12.000Z
2022-01-30T23:03:21.000Z
cmsc_210/examples/lecture_27/notebooks/Linear-Regression.ipynb
mazelife/cmsc-210
dbaa1604ef49bcfe5a70e09c17fbd243a8b80220
MIT
Qwen/Qwen-72B
1. YES
2. YES