# Vertical Line Test


```
# imports
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import math
```

## 1.1 Create two graphs, one that passes the vertical line test and one that does not.


```
x = np.arange(-9, 10)
y = x**3 + 2*x
plt.plot(x, y)
plt.axvline(x=0, color='purple');
```

## 1.2 Why are graphs that don't pass the vertical line test not considered "functions?"

The vertical line test can be used to determine whether a graph represents a function. If we can draw any vertical line that intersects a graph more than once, then the graph does not define a function because a function has only one output value for each input value.

# Functions as Relations

## 2.1 Which of the following relations are functions? Why?

\begin{align}
\text{Relation 1: } \{(1, 2), (3, 2), (1, 3)\}
\\
\text{Relation 2: } \{(1, 3), (2, 3), (6, 7)\}
\\
\text{Relation 3: } \{(9, 4), (2, 1), (9, 6)\}
\\
\text{Relation 4: } \{(6, 2), (8, 3), (6, 4)\}
\\
\text{Relation 5: } \{(2, 6), (2, 7), (2, 4)\}
\end{align}

**Answer: only relation 2 is the function, because a function cannot have the same input and the other functions intersects the vertical line more than once**


```
# Relation 1
x = np.array([1, 3, 1])
y = np.array([2, 2, 3])
plt.scatter(x, y, s=100)
plt.plot(x, y, color='orange')
plt.axvline(x=2, color='green');
```


```
# Relation 2
x = np.array([1, 2, 6])
y = np.array([3, 3, 7])
plt.scatter(x, y, s=100)
plt.plot(x, y, color='orange')
plt.axvline(x=2, color='green');
```


```
# Relation 3
x = np.array([9, 2, 9])
y = np.array([4, 1, 6])
plt.scatter(x, y, s=100)
plt.plot(x, y, color='orange')
plt.axvline(x=3, color='green');
```


```
# Relation 4
x = np.array([6, 8, 6])
y = np.array([2, 3, 4])
plt.scatter(x, y, s=100)
plt.plot(x, y, color='orange')
plt.axvline(x=7, color='green');
```


```
# Relation 5
x = np.array([2, 2, 2])
y = np.array([6, 7, 4])
plt.scatter(x, y, s=100)
plt.plot(x, y, color='orange')
plt.axvline(x=2, color='green');
```

# Functions as a mapping between dimensions


## 3.1 for the following functions what is the dimensionality of the domain (input) and codomain (range/output)?

\begin{align}
m(𝑥_1,𝑥_2,𝑥_3)=(x_1+x_2, x_1+x_3, x_2+x_3)
\\
n(𝑥_1,𝑥_2,𝑥_3,𝑥_4)=(x_2^2 + x_3, x_2x_4)
\end{align}


```

```

## 3.2 Do you think it's possible to create a function that maps from a lower dimensional space to a higher dimensional space? If so, provide an example.

# Vector Transformations

## 4.1 Plug the corresponding unit vectors into each function. Use the output vectors to create a transformation matrix.

\begin{align}
p(\begin{bmatrix}x_1 \\ x_2 \end{bmatrix}) = \begin{bmatrix} x_1 + 3x_2 \\2 x_2 - x_1 \\  \end{bmatrix}
\\
\\
q(\begin{bmatrix}x_1 \\ x_2 \\ x_3\end{bmatrix}) = \begin{bmatrix} 4x_1 + x_2 + 2x_3 \\2 x_2 - x_1 + 3x_3 \\ 5x_1 - 2x_3 + x_2  \end{bmatrix}
\end{align}

## 4.2 Verify that your transformation matrices are correct by choosing an input matrix and calculating the result both via the traditional functions above and also via vector-matrix multiplication.


```

```

# Eigenvalues and Eigenvectors

## 5.1 In your own words, give an explanation for the intuition behind eigenvalues and eigenvectors.

# The Curse of Dimensionality

## 6.1 What are some of the challenges of working with high dimensional spaces?

## 6.2 What is the rule of thumb for how many observations you should have compared to parameters in your model?

# Principal Component Analysis

## 7.1 Code for loading and cleaning the 2013 national dataset from the [Housing Affordability Data System (HADS)](https://www.huduser.gov/portal/datasets/hads/hads.html) --housing data, can be found below. 

## Perform PCA on the processed dataset `national_processed` (Make sure you standardize your data!) and then make a scatterplot of PC1 against PC2. Some of our discussion and work around PCA with this dataset will continue during tomorrow's lecture and assignment.

Not only does this dataset have decent amount columns to begin with (99), but in preparing the data for PCA we have also [one-hot-encoded](https://hackernoon.com/what-is-one-hot-encoding-why-and-when-do-you-have-to-use-it-e3c6186d008f#targetText=One%20hot%20encoding%20is%20a,the%20entry%20in%20the%20dataset.) all of the categorical variables. This has the effect of creating a new column for each individual category of each categorical variable. After processing this dataset has 64738 columns. --Das a lot of columns.

Don't worry too much about the mechanics of one-hot encoding right now, you will learn and experiment with a whole bunch of categorical encoding approaches in unit 2. 

The code below will read in the dataset and perform the one-hot encoding of the categorical variables. Start adding your PCA code at the bottom of the provided code.


```
from urllib.request import urlopen
from zipfile import ZipFile
from io import BytesIO
import os.path
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt

# Read Natinal Data 
national_url = 'https://www.huduser.gov/portal/datasets/hads/hads2013n_ASCII.zip'
national_file = 'thads2013n.txt'

if os.path.exists(national_file):
    national = pd.read_csv(national_file)
else: 
    z_national = urlopen(national_url)
    zip_national = ZipFile(BytesIO(z_national.read())).extract(national_file)
    national = pd.read_csv(zip_national)

print(national.shape)
national.head()
```

    (64535, 99)





<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>CONTROL</th>
      <th>AGE1</th>
      <th>METRO3</th>
      <th>REGION</th>
      <th>LMED</th>
      <th>FMR</th>
      <th>L30</th>
      <th>L50</th>
      <th>L80</th>
      <th>IPOV</th>
      <th>BEDRMS</th>
      <th>BUILT</th>
      <th>STATUS</th>
      <th>TYPE</th>
      <th>VALUE</th>
      <th>VACANCY</th>
      <th>TENURE</th>
      <th>NUNITS</th>
      <th>ROOMS</th>
      <th>WEIGHT</th>
      <th>PER</th>
      <th>ZINC2</th>
      <th>ZADEQ</th>
      <th>ZSMHC</th>
      <th>STRUCTURETYPE</th>
      <th>OWNRENT</th>
      <th>UTILITY</th>
      <th>OTHERCOST</th>
      <th>COST06</th>
      <th>COST12</th>
      <th>COST08</th>
      <th>COSTMED</th>
      <th>TOTSAL</th>
      <th>ASSISTED</th>
      <th>GLMED</th>
      <th>GL30</th>
      <th>GL50</th>
      <th>GL80</th>
      <th>APLMED</th>
      <th>ABL30</th>
      <th>...</th>
      <th>COST08RELPOVCAT</th>
      <th>COST08RELFMRPCT</th>
      <th>COST08RELFMRCAT</th>
      <th>COST12RELAMIPCT</th>
      <th>COST12RELAMICAT</th>
      <th>COST12RELPOVPCT</th>
      <th>COST12RELPOVCAT</th>
      <th>COST12RELFMRPCT</th>
      <th>COST12RELFMRCAT</th>
      <th>COSTMedRELAMIPCT</th>
      <th>COSTMedRELAMICAT</th>
      <th>COSTMedRELPOVPCT</th>
      <th>COSTMedRELPOVCAT</th>
      <th>COSTMedRELFMRPCT</th>
      <th>COSTMedRELFMRCAT</th>
      <th>FMTZADEQ</th>
      <th>FMTMETRO3</th>
      <th>FMTBUILT</th>
      <th>FMTSTRUCTURETYPE</th>
      <th>FMTBEDRMS</th>
      <th>FMTOWNRENT</th>
      <th>FMTCOST06RELPOVCAT</th>
      <th>FMTCOST08RELPOVCAT</th>
      <th>FMTCOST12RELPOVCAT</th>
      <th>FMTCOSTMEDRELPOVCAT</th>
      <th>FMTINCRELPOVCAT</th>
      <th>FMTCOST06RELFMRCAT</th>
      <th>FMTCOST08RELFMRCAT</th>
      <th>FMTCOST12RELFMRCAT</th>
      <th>FMTCOSTMEDRELFMRCAT</th>
      <th>FMTINCRELFMRCAT</th>
      <th>FMTCOST06RELAMICAT</th>
      <th>FMTCOST08RELAMICAT</th>
      <th>FMTCOST12RELAMICAT</th>
      <th>FMTCOSTMEDRELAMICAT</th>
      <th>FMTINCRELAMICAT</th>
      <th>FMTASSISTED</th>
      <th>FMTBURDEN</th>
      <th>FMTREGION</th>
      <th>FMTSTATUS</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>'100003130103'</td>
      <td>82</td>
      <td>'3'</td>
      <td>'1'</td>
      <td>73738</td>
      <td>956</td>
      <td>15738</td>
      <td>26213</td>
      <td>40322</td>
      <td>11067</td>
      <td>2</td>
      <td>2006</td>
      <td>'1'</td>
      <td>1</td>
      <td>40000</td>
      <td>-6</td>
      <td>'1'</td>
      <td>1</td>
      <td>6</td>
      <td>3117.394239</td>
      <td>1</td>
      <td>18021</td>
      <td>'1'</td>
      <td>533</td>
      <td>1</td>
      <td>'1'</td>
      <td>169.000000</td>
      <td>213.750000</td>
      <td>648.588189</td>
      <td>803.050535</td>
      <td>696.905247</td>
      <td>615.156712</td>
      <td>0</td>
      <td>-9</td>
      <td>73738</td>
      <td>15738</td>
      <td>26213</td>
      <td>40322</td>
      <td>51616.6</td>
      <td>20234.571429</td>
      <td>...</td>
      <td>4</td>
      <td>72.898038</td>
      <td>2</td>
      <td>48.402635</td>
      <td>2</td>
      <td>290.250487</td>
      <td>4</td>
      <td>84.001102</td>
      <td>2</td>
      <td>37.077624</td>
      <td>2</td>
      <td>222.339102</td>
      <td>4</td>
      <td>64.346936</td>
      <td>2</td>
      <td>'1 Adequate'</td>
      <td>'-5'</td>
      <td>'2000-2009'</td>
      <td>'1 Single Family'</td>
      <td>'2 2BR'</td>
      <td>'1 Owner'</td>
      <td>'4 200%+ Poverty'</td>
      <td>'4 200%+ Poverty'</td>
      <td>'4 200%+ Poverty'</td>
      <td>'4 200%+ Poverty'</td>
      <td>'3 150-200% Poverty'</td>
      <td>'2 50.1 - 100% FMR'</td>
      <td>'2 50.1 - 100% FMR'</td>
      <td>'2 50.1 - 100% FMR'</td>
      <td>'2 50.1 - 100% FMR'</td>
      <td>'1 LTE 50% FMR'</td>
      <td>'2 30 - 50% AMI'</td>
      <td>'2 30 - 50% AMI'</td>
      <td>'2 30 - 50% AMI'</td>
      <td>'2 30 - 50% AMI'</td>
      <td>'2 30 - 50% AMI'</td>
      <td>'.'</td>
      <td>'2 30% to 50%'</td>
      <td>'-5'</td>
      <td>'-5'</td>
    </tr>
    <tr>
      <th>1</th>
      <td>'100006110249'</td>
      <td>50</td>
      <td>'5'</td>
      <td>'3'</td>
      <td>55846</td>
      <td>1100</td>
      <td>17165</td>
      <td>28604</td>
      <td>45744</td>
      <td>24218</td>
      <td>4</td>
      <td>1980</td>
      <td>'1'</td>
      <td>1</td>
      <td>130000</td>
      <td>-6</td>
      <td>'1'</td>
      <td>1</td>
      <td>6</td>
      <td>2150.725544</td>
      <td>4</td>
      <td>122961</td>
      <td>'1'</td>
      <td>487</td>
      <td>1</td>
      <td>'1'</td>
      <td>245.333333</td>
      <td>58.333333</td>
      <td>1167.640781</td>
      <td>1669.643405</td>
      <td>1324.671218</td>
      <td>1058.988479</td>
      <td>123000</td>
      <td>-9</td>
      <td>55846</td>
      <td>17165</td>
      <td>28604</td>
      <td>45744</td>
      <td>55846.0</td>
      <td>19911.400000</td>
      <td>...</td>
      <td>4</td>
      <td>120.424656</td>
      <td>3</td>
      <td>103.094063</td>
      <td>6</td>
      <td>275.768999</td>
      <td>4</td>
      <td>151.785764</td>
      <td>3</td>
      <td>65.388468</td>
      <td>4</td>
      <td>174.909320</td>
      <td>3</td>
      <td>96.271680</td>
      <td>2</td>
      <td>'1 Adequate'</td>
      <td>'-5'</td>
      <td>'1980-1989'</td>
      <td>'1 Single Family'</td>
      <td>'4 4BR+'</td>
      <td>'1 Owner'</td>
      <td>'3 150-200% Poverty'</td>
      <td>'4 200%+ Poverty'</td>
      <td>'4 200%+ Poverty'</td>
      <td>'3 150-200% Poverty'</td>
      <td>'4 200%+ Poverty'</td>
      <td>'3 GT FMR'</td>
      <td>'3 GT FMR'</td>
      <td>'3 GT FMR'</td>
      <td>'2 50.1 - 100% FMR'</td>
      <td>'3 GT FMR'</td>
      <td>'4 60 - 80% AMI'</td>
      <td>'4 60 - 80% AMI'</td>
      <td>'6 100 - 120% AMI'</td>
      <td>'4 60 - 80% AMI'</td>
      <td>'7 120% AMI +'</td>
      <td>'.'</td>
      <td>'1 Less than 30%'</td>
      <td>'-5'</td>
      <td>'-5'</td>
    </tr>
    <tr>
      <th>2</th>
      <td>'100006370140'</td>
      <td>53</td>
      <td>'5'</td>
      <td>'3'</td>
      <td>55846</td>
      <td>1100</td>
      <td>13750</td>
      <td>22897</td>
      <td>36614</td>
      <td>15470</td>
      <td>4</td>
      <td>1985</td>
      <td>'1'</td>
      <td>1</td>
      <td>150000</td>
      <td>-6</td>
      <td>'1'</td>
      <td>1</td>
      <td>7</td>
      <td>2213.789404</td>
      <td>2</td>
      <td>27974</td>
      <td>'1'</td>
      <td>1405</td>
      <td>1</td>
      <td>'1'</td>
      <td>159.000000</td>
      <td>37.500000</td>
      <td>1193.393209</td>
      <td>1772.627006</td>
      <td>1374.582175</td>
      <td>1068.025168</td>
      <td>28000</td>
      <td>-9</td>
      <td>55846</td>
      <td>13750</td>
      <td>22897</td>
      <td>36614</td>
      <td>44676.8</td>
      <td>19937.500000</td>
      <td>...</td>
      <td>4</td>
      <td>124.962016</td>
      <td>3</td>
      <td>109.452905</td>
      <td>6</td>
      <td>458.339239</td>
      <td>4</td>
      <td>161.147910</td>
      <td>3</td>
      <td>65.946449</td>
      <td>4</td>
      <td>276.153890</td>
      <td>4</td>
      <td>97.093197</td>
      <td>2</td>
      <td>'1 Adequate'</td>
      <td>'-5'</td>
      <td>'1980-1989'</td>
      <td>'1 Single Family'</td>
      <td>'4 4BR+'</td>
      <td>'1 Owner'</td>
      <td>'4 200%+ Poverty'</td>
      <td>'4 200%+ Poverty'</td>
      <td>'4 200%+ Poverty'</td>
      <td>'4 200%+ Poverty'</td>
      <td>'3 150-200% Poverty'</td>
      <td>'3 GT FMR'</td>
      <td>'3 GT FMR'</td>
      <td>'3 GT FMR'</td>
      <td>'2 50.1 - 100% FMR'</td>
      <td>'2 50.1 - 100% FMR'</td>
      <td>'4 60 - 80% AMI'</td>
      <td>'5 80 - 100% AMI'</td>
      <td>'6 100 - 120% AMI'</td>
      <td>'4 60 - 80% AMI'</td>
      <td>'4 60 - 80% AMI'</td>
      <td>'.'</td>
      <td>'3 50% or More'</td>
      <td>'-5'</td>
      <td>'-5'</td>
    </tr>
    <tr>
      <th>3</th>
      <td>'100006520140'</td>
      <td>67</td>
      <td>'5'</td>
      <td>'3'</td>
      <td>55846</td>
      <td>949</td>
      <td>13750</td>
      <td>22897</td>
      <td>36614</td>
      <td>13964</td>
      <td>3</td>
      <td>1985</td>
      <td>'1'</td>
      <td>1</td>
      <td>200000</td>
      <td>-6</td>
      <td>'1'</td>
      <td>1</td>
      <td>6</td>
      <td>2364.585097</td>
      <td>2</td>
      <td>32220</td>
      <td>'1'</td>
      <td>279</td>
      <td>1</td>
      <td>'1'</td>
      <td>179.000000</td>
      <td>70.666667</td>
      <td>1578.857612</td>
      <td>2351.169341</td>
      <td>1820.442900</td>
      <td>1411.700224</td>
      <td>0</td>
      <td>-9</td>
      <td>55846</td>
      <td>13750</td>
      <td>22897</td>
      <td>36614</td>
      <td>44676.8</td>
      <td>17875.000000</td>
      <td>...</td>
      <td>4</td>
      <td>191.827492</td>
      <td>3</td>
      <td>161.926709</td>
      <td>7</td>
      <td>673.494512</td>
      <td>4</td>
      <td>247.752301</td>
      <td>3</td>
      <td>97.224801</td>
      <td>5</td>
      <td>404.382763</td>
      <td>4</td>
      <td>148.756610</td>
      <td>3</td>
      <td>'1 Adequate'</td>
      <td>'-5'</td>
      <td>'1980-1989'</td>
      <td>'1 Single Family'</td>
      <td>'3 3BR'</td>
      <td>'1 Owner'</td>
      <td>'4 200%+ Poverty'</td>
      <td>'4 200%+ Poverty'</td>
      <td>'4 200%+ Poverty'</td>
      <td>'4 200%+ Poverty'</td>
      <td>'4 200%+ Poverty'</td>
      <td>'3 GT FMR'</td>
      <td>'3 GT FMR'</td>
      <td>'3 GT FMR'</td>
      <td>'3 GT FMR'</td>
      <td>'2 50.1 - 100% FMR'</td>
      <td>'6 100 - 120% AMI'</td>
      <td>'7 120% AMI +'</td>
      <td>'7 120% AMI +'</td>
      <td>'5 80 - 100% AMI'</td>
      <td>'4 60 - 80% AMI'</td>
      <td>'.'</td>
      <td>'1 Less than 30%'</td>
      <td>'-5'</td>
      <td>'-5'</td>
    </tr>
    <tr>
      <th>4</th>
      <td>'100007130148'</td>
      <td>26</td>
      <td>'1'</td>
      <td>'3'</td>
      <td>60991</td>
      <td>737</td>
      <td>14801</td>
      <td>24628</td>
      <td>39421</td>
      <td>15492</td>
      <td>2</td>
      <td>1980</td>
      <td>'1'</td>
      <td>1</td>
      <td>-6</td>
      <td>-6</td>
      <td>'2'</td>
      <td>100</td>
      <td>4</td>
      <td>2314.524902</td>
      <td>2</td>
      <td>96874</td>
      <td>'1'</td>
      <td>759</td>
      <td>5</td>
      <td>'2'</td>
      <td>146.000000</td>
      <td>12.500000</td>
      <td>759.000000</td>
      <td>759.000000</td>
      <td>759.000000</td>
      <td>759.000000</td>
      <td>96900</td>
      <td>0</td>
      <td>60991</td>
      <td>14801</td>
      <td>24628</td>
      <td>39421</td>
      <td>48792.8</td>
      <td>16651.125000</td>
      <td>...</td>
      <td>3</td>
      <td>102.985075</td>
      <td>3</td>
      <td>55.308707</td>
      <td>3</td>
      <td>195.972115</td>
      <td>3</td>
      <td>102.985075</td>
      <td>3</td>
      <td>55.308707</td>
      <td>3</td>
      <td>195.972115</td>
      <td>3</td>
      <td>102.985075</td>
      <td>3</td>
      <td>'1 Adequate'</td>
      <td>'Central City'</td>
      <td>'1980-1989'</td>
      <td>'5 50+ units'</td>
      <td>'2 2BR'</td>
      <td>'2 Renter'</td>
      <td>'3 150-200% Poverty'</td>
      <td>'3 150-200% Poverty'</td>
      <td>'3 150-200% Poverty'</td>
      <td>'3 150-200% Poverty'</td>
      <td>'4 200%+ Poverty'</td>
      <td>'3 GT FMR'</td>
      <td>'3 GT FMR'</td>
      <td>'3 GT FMR'</td>
      <td>'3 GT FMR'</td>
      <td>'3 GT FMR'</td>
      <td>'3 50 - 60% AMI'</td>
      <td>'3 50 - 60% AMI'</td>
      <td>'3 50 - 60% AMI'</td>
      <td>'3 50 - 60% AMI'</td>
      <td>'7 120% AMI +'</td>
      <td>'0 Not Assisted'</td>
      <td>'1 Less than 30%'</td>
      <td>'-5'</td>
      <td>'-5'</td>
    </tr>
  </tbody>
</table>
<p>5 rows × 99 columns</p>
</div>




```
# Look at datatypes
# a lot of object datatypes even though they seem to be strings of numbers.
national.dtypes
```




    CONTROL            object
    AGE1                int64
    METRO3             object
    REGION             object
    LMED                int64
                        ...  
    FMTINCRELAMICAT    object
    FMTASSISTED        object
    FMTBURDEN          object
    FMTREGION          object
    FMTSTATUS          object
    Length: 99, dtype: object




```
# check for null values
national.isnull().sum().any()
```




    False




```
# check for number of categorical vs numeric columns
cat_cols = national.columns[national.dtypes=='object']
num_cols = national.columns[national.dtypes!='object']

print(f'{len(cat_cols)} categorical columns')
print(f'{len(num_cols)} numerical columns')
```

    32 categorical columns
    67 numerical columns



```
# We're making a copy of our data in case we mess something up.
national_processed = national.copy()

# Categorically Encode our Variables:
# They need to all be numeric before we do PCA.
# https://pbpython.com/categorical-encoding.html

# Cast categorical columns to "category" data type
national_processed[cat_cols] = national_processed[cat_cols].astype('category')

national_processed.dtypes
```




    CONTROL            category
    AGE1                  int64
    METRO3             category
    REGION             category
    LMED                  int64
                         ...   
    FMTINCRELAMICAT    category
    FMTASSISTED        category
    FMTBURDEN          category
    FMTREGION          category
    FMTSTATUS          category
    Length: 99, dtype: object




```
# Replace all category cell values with their numeric category codes
for col in cat_cols:
  national_processed[col] = national_processed[col].cat.codes

print(national_processed.shape)
national_processed.head()
```

    (64535, 99)





<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>CONTROL</th>
      <th>AGE1</th>
      <th>METRO3</th>
      <th>REGION</th>
      <th>LMED</th>
      <th>FMR</th>
      <th>L30</th>
      <th>L50</th>
      <th>L80</th>
      <th>IPOV</th>
      <th>BEDRMS</th>
      <th>BUILT</th>
      <th>STATUS</th>
      <th>TYPE</th>
      <th>VALUE</th>
      <th>VACANCY</th>
      <th>TENURE</th>
      <th>NUNITS</th>
      <th>ROOMS</th>
      <th>WEIGHT</th>
      <th>PER</th>
      <th>ZINC2</th>
      <th>ZADEQ</th>
      <th>ZSMHC</th>
      <th>STRUCTURETYPE</th>
      <th>OWNRENT</th>
      <th>UTILITY</th>
      <th>OTHERCOST</th>
      <th>COST06</th>
      <th>COST12</th>
      <th>COST08</th>
      <th>COSTMED</th>
      <th>TOTSAL</th>
      <th>ASSISTED</th>
      <th>GLMED</th>
      <th>GL30</th>
      <th>GL50</th>
      <th>GL80</th>
      <th>APLMED</th>
      <th>ABL30</th>
      <th>...</th>
      <th>COST08RELPOVCAT</th>
      <th>COST08RELFMRPCT</th>
      <th>COST08RELFMRCAT</th>
      <th>COST12RELAMIPCT</th>
      <th>COST12RELAMICAT</th>
      <th>COST12RELPOVPCT</th>
      <th>COST12RELPOVCAT</th>
      <th>COST12RELFMRPCT</th>
      <th>COST12RELFMRCAT</th>
      <th>COSTMedRELAMIPCT</th>
      <th>COSTMedRELAMICAT</th>
      <th>COSTMedRELPOVPCT</th>
      <th>COSTMedRELPOVCAT</th>
      <th>COSTMedRELFMRPCT</th>
      <th>COSTMedRELFMRCAT</th>
      <th>FMTZADEQ</th>
      <th>FMTMETRO3</th>
      <th>FMTBUILT</th>
      <th>FMTSTRUCTURETYPE</th>
      <th>FMTBEDRMS</th>
      <th>FMTOWNRENT</th>
      <th>FMTCOST06RELPOVCAT</th>
      <th>FMTCOST08RELPOVCAT</th>
      <th>FMTCOST12RELPOVCAT</th>
      <th>FMTCOSTMEDRELPOVCAT</th>
      <th>FMTINCRELPOVCAT</th>
      <th>FMTCOST06RELFMRCAT</th>
      <th>FMTCOST08RELFMRCAT</th>
      <th>FMTCOST12RELFMRCAT</th>
      <th>FMTCOSTMEDRELFMRCAT</th>
      <th>FMTINCRELFMRCAT</th>
      <th>FMTCOST06RELAMICAT</th>
      <th>FMTCOST08RELAMICAT</th>
      <th>FMTCOST12RELAMICAT</th>
      <th>FMTCOSTMEDRELAMICAT</th>
      <th>FMTINCRELAMICAT</th>
      <th>FMTASSISTED</th>
      <th>FMTBURDEN</th>
      <th>FMTREGION</th>
      <th>FMTSTATUS</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>0</td>
      <td>82</td>
      <td>2</td>
      <td>0</td>
      <td>73738</td>
      <td>956</td>
      <td>15738</td>
      <td>26213</td>
      <td>40322</td>
      <td>11067</td>
      <td>2</td>
      <td>2006</td>
      <td>0</td>
      <td>1</td>
      <td>40000</td>
      <td>-6</td>
      <td>1</td>
      <td>1</td>
      <td>6</td>
      <td>3117.394239</td>
      <td>1</td>
      <td>18021</td>
      <td>1</td>
      <td>533</td>
      <td>1</td>
      <td>0</td>
      <td>169.000000</td>
      <td>213.750000</td>
      <td>648.588189</td>
      <td>803.050535</td>
      <td>696.905247</td>
      <td>615.156712</td>
      <td>0</td>
      <td>-9</td>
      <td>73738</td>
      <td>15738</td>
      <td>26213</td>
      <td>40322</td>
      <td>51616.6</td>
      <td>20234.571429</td>
      <td>...</td>
      <td>4</td>
      <td>72.898038</td>
      <td>2</td>
      <td>48.402635</td>
      <td>2</td>
      <td>290.250487</td>
      <td>4</td>
      <td>84.001102</td>
      <td>2</td>
      <td>37.077624</td>
      <td>2</td>
      <td>222.339102</td>
      <td>4</td>
      <td>64.346936</td>
      <td>2</td>
      <td>1</td>
      <td>0</td>
      <td>5</td>
      <td>1</td>
      <td>2</td>
      <td>0</td>
      <td>4</td>
      <td>4</td>
      <td>4</td>
      <td>4</td>
      <td>3</td>
      <td>1</td>
      <td>1</td>
      <td>1</td>
      <td>1</td>
      <td>1</td>
      <td>1</td>
      <td>1</td>
      <td>1</td>
      <td>1</td>
      <td>2</td>
      <td>0</td>
      <td>2</td>
      <td>0</td>
      <td>0</td>
    </tr>
    <tr>
      <th>1</th>
      <td>1</td>
      <td>50</td>
      <td>4</td>
      <td>2</td>
      <td>55846</td>
      <td>1100</td>
      <td>17165</td>
      <td>28604</td>
      <td>45744</td>
      <td>24218</td>
      <td>4</td>
      <td>1980</td>
      <td>0</td>
      <td>1</td>
      <td>130000</td>
      <td>-6</td>
      <td>1</td>
      <td>1</td>
      <td>6</td>
      <td>2150.725544</td>
      <td>4</td>
      <td>122961</td>
      <td>1</td>
      <td>487</td>
      <td>1</td>
      <td>0</td>
      <td>245.333333</td>
      <td>58.333333</td>
      <td>1167.640781</td>
      <td>1669.643405</td>
      <td>1324.671218</td>
      <td>1058.988479</td>
      <td>123000</td>
      <td>-9</td>
      <td>55846</td>
      <td>17165</td>
      <td>28604</td>
      <td>45744</td>
      <td>55846.0</td>
      <td>19911.400000</td>
      <td>...</td>
      <td>4</td>
      <td>120.424656</td>
      <td>3</td>
      <td>103.094063</td>
      <td>6</td>
      <td>275.768999</td>
      <td>4</td>
      <td>151.785764</td>
      <td>3</td>
      <td>65.388468</td>
      <td>4</td>
      <td>174.909320</td>
      <td>3</td>
      <td>96.271680</td>
      <td>2</td>
      <td>1</td>
      <td>0</td>
      <td>3</td>
      <td>1</td>
      <td>4</td>
      <td>0</td>
      <td>3</td>
      <td>4</td>
      <td>4</td>
      <td>3</td>
      <td>4</td>
      <td>2</td>
      <td>2</td>
      <td>2</td>
      <td>1</td>
      <td>3</td>
      <td>3</td>
      <td>3</td>
      <td>5</td>
      <td>3</td>
      <td>7</td>
      <td>0</td>
      <td>1</td>
      <td>0</td>
      <td>0</td>
    </tr>
    <tr>
      <th>2</th>
      <td>2</td>
      <td>53</td>
      <td>4</td>
      <td>2</td>
      <td>55846</td>
      <td>1100</td>
      <td>13750</td>
      <td>22897</td>
      <td>36614</td>
      <td>15470</td>
      <td>4</td>
      <td>1985</td>
      <td>0</td>
      <td>1</td>
      <td>150000</td>
      <td>-6</td>
      <td>1</td>
      <td>1</td>
      <td>7</td>
      <td>2213.789404</td>
      <td>2</td>
      <td>27974</td>
      <td>1</td>
      <td>1405</td>
      <td>1</td>
      <td>0</td>
      <td>159.000000</td>
      <td>37.500000</td>
      <td>1193.393209</td>
      <td>1772.627006</td>
      <td>1374.582175</td>
      <td>1068.025168</td>
      <td>28000</td>
      <td>-9</td>
      <td>55846</td>
      <td>13750</td>
      <td>22897</td>
      <td>36614</td>
      <td>44676.8</td>
      <td>19937.500000</td>
      <td>...</td>
      <td>4</td>
      <td>124.962016</td>
      <td>3</td>
      <td>109.452905</td>
      <td>6</td>
      <td>458.339239</td>
      <td>4</td>
      <td>161.147910</td>
      <td>3</td>
      <td>65.946449</td>
      <td>4</td>
      <td>276.153890</td>
      <td>4</td>
      <td>97.093197</td>
      <td>2</td>
      <td>1</td>
      <td>0</td>
      <td>3</td>
      <td>1</td>
      <td>4</td>
      <td>0</td>
      <td>4</td>
      <td>4</td>
      <td>4</td>
      <td>4</td>
      <td>3</td>
      <td>2</td>
      <td>2</td>
      <td>2</td>
      <td>1</td>
      <td>2</td>
      <td>3</td>
      <td>4</td>
      <td>5</td>
      <td>3</td>
      <td>4</td>
      <td>0</td>
      <td>3</td>
      <td>0</td>
      <td>0</td>
    </tr>
    <tr>
      <th>3</th>
      <td>3</td>
      <td>67</td>
      <td>4</td>
      <td>2</td>
      <td>55846</td>
      <td>949</td>
      <td>13750</td>
      <td>22897</td>
      <td>36614</td>
      <td>13964</td>
      <td>3</td>
      <td>1985</td>
      <td>0</td>
      <td>1</td>
      <td>200000</td>
      <td>-6</td>
      <td>1</td>
      <td>1</td>
      <td>6</td>
      <td>2364.585097</td>
      <td>2</td>
      <td>32220</td>
      <td>1</td>
      <td>279</td>
      <td>1</td>
      <td>0</td>
      <td>179.000000</td>
      <td>70.666667</td>
      <td>1578.857612</td>
      <td>2351.169341</td>
      <td>1820.442900</td>
      <td>1411.700224</td>
      <td>0</td>
      <td>-9</td>
      <td>55846</td>
      <td>13750</td>
      <td>22897</td>
      <td>36614</td>
      <td>44676.8</td>
      <td>17875.000000</td>
      <td>...</td>
      <td>4</td>
      <td>191.827492</td>
      <td>3</td>
      <td>161.926709</td>
      <td>7</td>
      <td>673.494512</td>
      <td>4</td>
      <td>247.752301</td>
      <td>3</td>
      <td>97.224801</td>
      <td>5</td>
      <td>404.382763</td>
      <td>4</td>
      <td>148.756610</td>
      <td>3</td>
      <td>1</td>
      <td>0</td>
      <td>3</td>
      <td>1</td>
      <td>3</td>
      <td>0</td>
      <td>4</td>
      <td>4</td>
      <td>4</td>
      <td>4</td>
      <td>4</td>
      <td>2</td>
      <td>2</td>
      <td>2</td>
      <td>2</td>
      <td>2</td>
      <td>5</td>
      <td>6</td>
      <td>6</td>
      <td>4</td>
      <td>4</td>
      <td>0</td>
      <td>1</td>
      <td>0</td>
      <td>0</td>
    </tr>
    <tr>
      <th>4</th>
      <td>4</td>
      <td>26</td>
      <td>0</td>
      <td>2</td>
      <td>60991</td>
      <td>737</td>
      <td>14801</td>
      <td>24628</td>
      <td>39421</td>
      <td>15492</td>
      <td>2</td>
      <td>1980</td>
      <td>0</td>
      <td>1</td>
      <td>-6</td>
      <td>-6</td>
      <td>2</td>
      <td>100</td>
      <td>4</td>
      <td>2314.524902</td>
      <td>2</td>
      <td>96874</td>
      <td>1</td>
      <td>759</td>
      <td>5</td>
      <td>1</td>
      <td>146.000000</td>
      <td>12.500000</td>
      <td>759.000000</td>
      <td>759.000000</td>
      <td>759.000000</td>
      <td>759.000000</td>
      <td>96900</td>
      <td>0</td>
      <td>60991</td>
      <td>14801</td>
      <td>24628</td>
      <td>39421</td>
      <td>48792.8</td>
      <td>16651.125000</td>
      <td>...</td>
      <td>3</td>
      <td>102.985075</td>
      <td>3</td>
      <td>55.308707</td>
      <td>3</td>
      <td>195.972115</td>
      <td>3</td>
      <td>102.985075</td>
      <td>3</td>
      <td>55.308707</td>
      <td>3</td>
      <td>195.972115</td>
      <td>3</td>
      <td>102.985075</td>
      <td>3</td>
      <td>1</td>
      <td>1</td>
      <td>3</td>
      <td>5</td>
      <td>2</td>
      <td>1</td>
      <td>3</td>
      <td>3</td>
      <td>3</td>
      <td>3</td>
      <td>4</td>
      <td>2</td>
      <td>2</td>
      <td>2</td>
      <td>2</td>
      <td>3</td>
      <td>2</td>
      <td>2</td>
      <td>2</td>
      <td>2</td>
      <td>7</td>
      <td>1</td>
      <td>1</td>
      <td>0</td>
      <td>0</td>
    </tr>
  </tbody>
</table>
<p>5 rows × 99 columns</p>
</div>




```
# Now we only ahve numeric columns (ints and floats)
national_processed.dtypes
```




    CONTROL            int32
    AGE1               int64
    METRO3              int8
    REGION              int8
    LMED               int64
                       ...  
    FMTINCRELAMICAT     int8
    FMTASSISTED         int8
    FMTBURDEN           int8
    FMTREGION           int8
    FMTSTATUS           int8
    Length: 99, dtype: object




```
### Your Code Here
from sklearn.preprocessing import StandardScaler
myscaler = StandardScaler()
```


```
# Standardize the dataset (default is normalization)
national_processed = myscaler.fit_transform(national_processed)
pd.DataFrame(national_processed).shape
```




    (64535, 99)




```
from sklearn.decomposition import PCA
mypca = PCA(n_components = 10)
pca_features = mypca.fit_transform(national_processed)
pd.DataFrame(pca_features).shape
```




    (64535, 10)




```
#  how much variation did each principal component explain?
mypca.explained_variance_ratio_
```




    array([0.34050542, 0.14442164, 0.10287616, 0.07245401, 0.05166581,
           0.04006691, 0.02785838, 0.02220382, 0.02016894, 0.01770104])




```
print(sum(mypca.explained_variance_ratio_)) # variance explained
1 - sum(mypca.explained_variance_ratio_)
```

    0.8399221336739114





    0.16007786632608856




```
plt.scatter(pca_features[:,0], pca_features[:,1], s=10, alpha=.2)
plt.show()
```


```

```


```

```


```
# Standardize the dataset (default is normalization)
national_processed = myscaler.fit_transform(national_processed)
pd.DataFrame(national_processed).shape
```




    (64535, 99)



# Stretch Goals

## 1) Perform further data exploration on the HADS national dataset (the version before we one-hot encoded it) Make scatterplots and see if you can see any resemblance between the original scatterplots and the plot of the principal components that you made in 7.1. 

(You may or may not not see very much resemblance depending on the variables you choose, and that's ok!)

## 2) Study "Scree Plots" and then try and make one for your PCA dataset. How many principal conponents do you need to retain in order for your PCs to contain 90% of the explained variance? 

We will present this topic formally at the beginning of tomorrow's lecture, so if you figure this stretch goal out, you're ahead of the game. 

## 3) Explore further the intuition behind eigenvalues and eigenvectors by creating your very own eigenfaces:

Prioritize self-study over this stretch goal if you are not semi-comfortable with the topics of PCA, Eigenvalues, and Eigenvectors.



You don't necessarily have to use this resource, but this will get you started: 
[Eigenface Tutorial](https://sandipanweb.wordpress.com/2018/01/06/eigenfaces-and-a-simple-face-detector-with-pca-svd-in-python/)

eafcfba67a0a43a1db2b0bb0bce6322befd6f19f
ipynb
Jupyter Notebook
01_Unit_1/Sprint3-Linear-Algebra/Module3/LS_DS_133_High_Dimensional_Data_Assignment.ipynb
mark-morelos/my_DataScience_notes
2e05a3d77949cb8ccd9aac7c04cf7758d7b73d1c
MIT
01_Unit_1/Sprint3-Linear-Algebra/Module3/LS_DS_133_High_Dimensional_Data_Assignment.ipynb
mark-morelos/my_DataScience_notes
2e05a3d77949cb8ccd9aac7c04cf7758d7b73d1c
MIT
01_Unit_1/Sprint3-Linear-Algebra/Module3/LS_DS_133_High_Dimensional_Data_Assignment.ipynb
mark-morelos/my_DataScience_notes
2e05a3d77949cb8ccd9aac7c04cf7758d7b73d1c
MIT
Qwen/Qwen-72B
1. YES
2. YES