<a href="https://colab.research.google.com/github/cocoisland/AB-Demo/blob/master/High_Dimensional_Data.ipynb" target="_parent"></a>

## Functions from a Linear Algebra Perspective

In mathematics a function is a relation between a set of inputs (domain) and a set of outputs (range) with the property that each input is related to (mapped to) exactly one output. This means that for functions that are represented in $\mathbb{R}^2$ a function has to pass what is known as the "vertical line test." The vertical line test means that if you were to pass a vertical line over the function it would only be touching the function in one spot at any given time

Which of the graphs below are functions?

<center></center>

I want to re-emphasize that a function is a mapping from a set of inputs to a set of outputs where no two inputs is mapped to the same output.

## Relations as Functions

Imagine that we were trying to plot the line that created by the function $f(x) = 2x + 3$

How would we do it? Well, we would probably pick some simple $x$ values that exist near the origin and plug them in to see what their corresponding $y$ values were. It's clear that this is the equation for a line and we know that we're plotting these in $/mathbb{R}^2$ we would probably make some kind of a table or list of points that are on the line and then "connect the dots" to plot it:

Help me create some points that we would use to graph this line

\begin{align}
\text{Points: } (-2, y), (-1, y), (0, y), (1, y), (2, y)
\end{align}

These points do not completely represent that function, but we know that lines have a constant slope, so we don't need to find every ordered pair in the line in order to be able to plot it.

But say that we had some function where the only 5 inputs were the 5 that I gave you above and the only 5 outputs were the 5 that I gave you above. Tthis would be an odd function, and I don't know how we would define an equation for it. But what if we just let the ordered pairs themselves represent the function -the mapping from domain values to range values. We could create a set of ordered pairs that shows a mapping from domain values (x) to range values (y) where no domain value maps to the same range value twice. This set of ordered pairs is called a "relation"

\begin{align}
\text{Relation: } \{(-2, -1), (-1, 1), (0, 3), (1, 5), (2, 7)\}
\end{align}

Although this relation isn't represented by a specific equation (because it's just 5 points). However it meets our definition of a function, so this set of ordered pairs which show a valid mapping of domain values to range values is a function. If we plot these 5 points it would be a bit sparse but it would pass the vertical line test. 

Relations that satisfy these conditions are functions. (albeit very simple ones)

### Which of the following relations are functions?

\begin{align}
\text{Relation 1: } \{(-2, -1), (-2, 1), (2, 3)\}
\\
\text{Relation 2: } \{(-2, 5), (-1, 5), (0, 5)\}
\\
\text{Relation 3: } \{(0, -1), (-1, 1), (0, 3)\}
\end{align}

## Linear Transformations (vector transformations)

Lets look at the function $f(x) = 2x + 3$ again.

This is how lower level math classes deal with functions, they put one number in as input and they get one output, but there's no reason why we need to limit functions to having just one input.

What if I defined a function like this $g(x_1, x_2) = x_1+x_2$

**_Side Note:_** I hope that if you weren't already comfortable with it that this kind of function notation, that it makes a lot more sense since you've been writing python functions and you recognize the things in the parenthesis as being the parameters of a function just like how we would write it in code.

The above function $g$ requires a two-dimensional tuple of inputs (two inputs) and gives a single output. We would say that this function is a mapping from $\mathbb{R}^2$ to $\mathbb{R}$.

Here's another function: $h(x_1, x_2, x_3) = (x_1 + 2, x_2 - x_3 )$

What is the dimensionality of the domain compared to the range (codomain)? Can you see that this mapping takes sets of values points from a certain dimensionality and maps them to sets of values with a smaller dimensionality?

## Vector Transformations

For today, we're going to go easy on you and imagine a mapping of vectors from $\mathbb{R}^2$ to $\mathbb{R}^2$.  We're just going to work with transforming 2D vectors to some other vector in two dimensions. 

In linear algebra for a linear transformation to be valid it must meet the following two requirements:

\begin{align}
T(u+v) = T(u)+T(v)
\\
T(cu) = cT(u)
\end{align}

What these two properties mean is that scaling vectors is still valid and that a transformation of two vectors can be seen as the composition of smaller transformations (like scaled unit vectors added together). See 3Blue1Brown videos for more intuition on this.

## Linear Transformations As Matrix-Vector Products

What if I defined a mapping from $\mathbb{R}^2$ to $\mathbb{R}^2$ like so:

\begin{align}
f(\begin{bmatrix}x_1 \\ x_2\end{bmatrix}) = \begin{bmatrix} 2x_1 + x_2 \\ x_2 - x_1 \\  \end{bmatrix}
\end{align}

Well, remember how we can write any vector as a linear combination of scalars and basis vectors? How would I rewrite my input vector in those terms?

\begin{align}
\\
\begin{bmatrix} x_1 \\ x_2 \end{bmatrix} = x_1\begin{bmatrix} 1 \\ 0 \end{bmatrix} + x_2\begin{bmatrix} 0 \\ 1 \end{bmatrix}
\\
\end{align}

You might remember from the 3Blue1Brown video about linear transformations that they can be described as the composition of transformations of our basis vectors. rewriting our input vector in this way will help us see that.

What if we evaluated our expression by passing in the unit vectors $\hat{i}$ and $hat{j}$?

\begin{align}
f(\begin{bmatrix} 1 \\ 0 \end{bmatrix}) = \begin{bmatrix} 2 \\ -1 \end{bmatrix}
\\
f(\begin{bmatrix} 0 \\ 1 \end{bmatrix}) = \begin{bmatrix} 1 \\ 1 \end{bmatrix}
\end{align}

By passing in our unit vectors individually, We are essentially asking: Tell me what happens to all of the x values, and then tell me what happens to all of the y values individually. And the resulting vector will be a composition of how both the y values and x values change. 

The coolest part about passing in the unit vectors is that we've got two new vectors back that we can compose into a transformation matrix:

\begin{align}
T = \begin{bmatrix} 2 & 1 \\ -1 & 1 \end{bmatrix}
\end{align}

We can now describe the same function above as the product of matrix-vector multiplication between our resulting transformation vector and any vector in $\mathbb{R^2}$.

Lets test it! First lets use the original function to calculate the result of passing in 

\begin{align}
f(\begin{bmatrix} 3 \\ 4 \end{bmatrix}) = \begin{bmatrix} 10 \\ 1 \end{bmatrix}
\end{align}

Now lets do it via matrix multiplication:

\begin{align}
\begin{bmatrix} 2 & 1 \\ -1 & 1 \end{bmatrix}\begin{bmatrix} 3 \\ 4 \end{bmatrix} = \begin{bmatrix} 10 \\ 1 \end{bmatrix}
\end{align}

Wild right???

Because of the properties of linear transformations, we can describe any linear transformation as matrix vector multiplication where the columns of the transformation matrix come from the result of passing in the dimension appropriate unit vectors into the function. So cool that we could represent a much more traditional function (mapping) as matrix-vector multiplication because of the way that we can represent vectors as linear combinations of scalars and unit vectors. 


```
import matplotlib.pyplot as plt

# Vectors
input_vector = [3, 4]
output_vector = [10, 1]

# Plot the Vectors
plt.arrow(0,0, input_vector[0], input_vector[1],head_width=.05, head_length=0.05, color ='red')
plt.arrow(0,0, output_vector[0], output_vector[1],head_width=.05, head_length=0.05, color ='blue')
plt.xlim(0,11)          
plt.ylim(-2,6)
plt.title("Transformed Vector")
plt.show()
```

## Intro to Eigenvalues and Eigenvectors (Intuition)

All vector-matrix multiplication can be thought of as a multi-dimensional transformation of space. (I hope that by now you're realizing how critical linear algebra is for representing multi-dimensional data.) This is hard for me to demonstrate without some cool animations so I'll have to rely upon 3Blue1Brown videos to help get you that intuition. Those videos are really the only place where I've seen that principle demonstrated even a little bit effectively.

One thing that might not be immediately clear however, is that not all points in space get transformed in exactly the same way. Think about a transformation in $\mathbb{R^3}$ where we rotate 3D space. You could imagine a globe spinning on its axis.

<center></center>

When a globe rotates, does every point on the globe move the same distance in space? No, points closer to the pole move less than points near the equator (kinda weird if you think about it that people living near the equator are hurtling through space at a different rate than other latitudes). 

What about the vector represented by the earth's axis? Does it move? 

We could call this vector the **eigenvector** of the earth's rotation brecause the transformation does not reorient it. Any transformation that you could conceive of possesses eigenvectors - dimensions of space that are not pointed in a new direction by the transformation described by a certain transformation matrix. 

## So what's an eigenvalue?

An eigenvector is any vector that doesn't change its orientation during a given transformation, but an eigenvector might still get scaled by some scalar. It can get stretched or squished, it just can't change the direction in which it is pointing. An **eigenvalue** is the scalar that represents how its corresponding eigenvector was scaled during the transformation. Eigenvectors and eigenvalues always come in pairs.

What is the eigenvalue of the earth's axis when the earth rotates?

## Finding Eigenvalues and Eigenvectors

We don't have time to go over this today and it's one of the more complex processes in Linear Algebra. It usually involves a combination of a process called "Matrix Diagonalization" and Gaussian Elimination. If you're interested, you can look it up but that process is too complex for the time that we have today and I think you will be better served by focusing on an application of all of these concepts: **Principle Component Analysis (PCA)**

## One More Thing

What's the symbol commonly used to denote eigenvalues? - $\lambda$ 

\begin{align}
T(v) = \lambda v
\end{align}

# The Curse of Dimensionality

The Curse of Dimensionality is a term that is used to refer to some of the challenges and limitations that arise from trying to process or model datasets with a large number of features. (hundreds or thousands)

## What is a Dimension?

Imagine that we have a dataset with 50 numeric datapoints. How would we graph a single column of data? Well, we have nothing else in the dataset to compare it to, so all that we can do is really plot the values on a one-dimensional number line.


```
import pandas as pd
import random
import matplotlib.pyplot as plt
import matplotlib.ticker as ticker
import numpy as np

# Create a list of 50 random floats and add them to a dataframe
N = 50
x = np.random.rand(N)*100

# Crete Dataframe
data = {"x": x}
df = pd.DataFrame(data)
df.head()
```




<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>x</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>91.661852</td>
    </tr>
    <tr>
      <th>1</th>
      <td>57.389340</td>
    </tr>
    <tr>
      <th>2</th>
      <td>80.622474</td>
    </tr>
    <tr>
      <th>3</th>
      <td>40.581956</td>
    </tr>
    <tr>
      <th>4</th>
      <td>80.409636</td>
    </tr>
  </tbody>
</table>
</div>




```
# Plot the floats on a number line

# Setup a plot such that only the bottom spine is shown
def setup(ax):
    ax.spines['right'].set_color('none')
    ax.spines['left'].set_color('none')
    ax.yaxis.set_major_locator(ticker.NullLocator())
    ax.spines['top'].set_color('none')
    ax.xaxis.set_ticks_position('bottom')
    ax.tick_params(which='major', width=1.00)
    ax.tick_params(which='major', length=5)
    ax.tick_params(which='minor', width=0.75)
    ax.tick_params(which='minor', length=2.5)
    ax.set_xlim(0, 100)
    ax.set_ylim(0, 1)
    ax.patch.set_alpha(0.0)
    
plt.figure(figsize=(8, 6))
n = 8

# Create some y values to satisfy matplotlib
df['y'] = pd.Series(list(np.zeros(50)))

# Multiple Locator tick mark settings
ax = plt.subplot(n, 1, 2)
setup(ax)
ax.xaxis.set_major_locator(ticker.MultipleLocator(5))
ax.text(0.0, 0.5, "Number Line", fontsize=14,
        transform=ax.transAxes)

# Push the top of the top axes outside the figure because we only show the
# bottom spine.
plt.subplots_adjust(left=0.05, right=0.95, bottom=0.05, top=1.05)
plt.scatter(df.x, df.y, alpha=0.5)

plt.show()
```

## JUST LET ME PLOT POINTS ON A NUMBER LINE IS THAT TOO MUCH TO ASK?!?

<center></center>

### Anyway... back to dimensions.

## Two Dimensions
What if I wanted to plot plot these points on a two-dimensional graph like a scatter plot? Well, I would need y values to tell the points where to go, or else they would just sit on the x-axis of the 2D graph. If we don't have any y values, the additional axis of the coordinate plane serves no purpose.

The number line is a 1D graph with 1D data. Scatter plots are 2Ds graphs that facilitate 2D data.


```
plt.scatter(df.x, df.y, alpha=0.5)
plt.title("Not a good use of a 2D Graph")
plt.show()
```


```
# Generate 50 random y values and add them to the dataframe
df['y'] = pd.Series(list(np.random.rand(N)*100))
df.head()
```




<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>x</th>
      <th>y</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>87.224509</td>
      <td>0.828650</td>
    </tr>
    <tr>
      <th>1</th>
      <td>22.433375</td>
      <td>59.655565</td>
    </tr>
    <tr>
      <th>2</th>
      <td>54.646833</td>
      <td>69.856257</td>
    </tr>
    <tr>
      <th>3</th>
      <td>49.150725</td>
      <td>30.249769</td>
    </tr>
    <tr>
      <th>4</th>
      <td>72.895643</td>
      <td>7.607457</td>
    </tr>
  </tbody>
</table>
</div>




```
# Graph 2D data on a 2D plot
plt.scatter(df['x'], df['y'], alpha=0.5)
plt.title("A Better Use of a 2D Graph")
plt.show()
```

### That's how easy the number line should have been... Somebody fix it and submit a Pull Request to Matplotlib


## Lets do 3-Dimensional Data on a 3-Dimensional Graph!

I hope you've realized that the number of dimensions that we want to plot, the number of columns that is required in the dataframe. This is why you'll hear people refer to Dataframe columns: Columns, Features, and Dimensions, interchangeably.


```
from mpl_toolkits.mplot3d import Axes3D

# Add Z dimension to dataframe
df['z'] = pd.Series(list(np.random.rand(N)*100))

threedee = plt.figure().gca(projection='3d')
threedee.scatter(df['x'], df['y'], df['z'])
threedee.set_xlabel('X')
threedee.set_ylabel('Y')
threedee.set_zlabel('Z')
plt.show()
```

## 3D Graphs on 2D screens are a lie... unless they can rotate.

Unless this graph can rotate like a hologram it's a 2D graph that's trying really hard, yet failing to be a 3D graph. Look more closely at the pionts and you'll realize that you actually have no idea where they are at. You have no depth perception unless you can rotate the graph. This is something that's possible to do in iPython notebooks, but that I didn't want to mess with right now. You can explore it on your own. Some libraries will try and change the diameter of the points so that ones "closer" to the viewer have some kind of depth, but this is an imperfect solution.

## What if we tried a spatially represent 4-Dimensional Data in a 4D graph?

<center></center>

While it **is** technically possible to plot points on a 4-Dimensional graph, the result is by no means intuitive and suffers from even worse interpretation problems than the 3D graph.

<center></center>

A cube is 3-dimensional, so any representation larger than 3 dimensions is referred to as an n-dimensional hypercube in n-dimensional hyperspace. This is why in space movies/books they always zoom to "hyperspace" when they're trying to take a shortcut through the 3 spatial dimensions of our reality.

## Now Lets do a 1000 dimensional dataset!

You see the problem? 

While the inability for the human brain to spatially comprehend more than 3 dimensions is not **really** the curse of dimensionality, this limitation of our human brain is related to some of the challenges of dealing with high-dimensional data.

## A thought experiment

A great explanation from a Quora user: (Sorry, I can't find the actual Quora post that this originated from) I've modified it a little bit as well.

```
Let's say you have a straight line 100 yards long and you dropped 10 pennies somewhere on it. It wouldn't be too hard to find one. You walk along the line and it takes a few seconds, maybe as long as a couple of minutes if you're unlucky.

Now let's say you have a square 100 yards on each side and you dropped 10 pennies somewhere on it. It would be harder. The pennies are more spread out now. This is like searching across two football fields stuck together. It could take hours or days to find one. (on average, 10 times longer than the first time)

Now imagine a cube 100 yards across. That's like searching a 30-story building the size of a football stadium. Ugh. (It's now 100 times harder to find a penny in this space than in the first example)

The difficulty of searching through the space gets exponentially harder as you have more dimensions.
```


## Challenges of High Dimensional Spaces

### More difficult to explore the data and create visualizations



This pairplot shows us all of the possible scatterplots that could be created from a 4-dimensional dataset. Where the scatter plots would be plotted against themselves they have been replaced with histograms to show the distribution of a single variable.

Question: How many scatter plots would you have to look at if you had 1000 dimensions in a dataset?

- Remember that there are a lot of redundant scatterplots with pairplots

### Diminishing returns to adding features (increased redundancy)
Not every feature is as important as every other feature when it comes to capturing relationships in data. he likelihood of adding a feature that contributes meaningful signal to a particular predictive endeavor decreases with each additional dimension.

A good example of this is Deep Learning image recognition tasks:



How does increasing the number of pixels (dimensions of an image) increase our ability to detect what the image is? 

If an extra dimension is more expensive to process but adds less value to a model, then at some point it isn't cost effective to add additional dimensions.

### Increased computational workload

Just like how it takes more time to find the pennies in the 100x100x100 yard cube.

### Increased sparcity of data - Measures of distance (Euclidean) lose meaning 

As we add dimensions, our datapoints become more spread out. Just look at the difference between the 1-dimensional numberline and the 2-dimensional scatterplot that we did above. With each dimension that we add our datapoints spread out further, but at a decreasing rate. Eventually, in extremely high dimensional settings all datapoints appear to be more or less equidistant to one another. This is crippling to algorithms that rely upon measures of distance (Like Clustering for example). We'll talk more about this on Thursday! 

### Low number of observations relative to dimensions increases the risk of overfitting



The the degree to which increased dimensions can lead to overfitting varies from model to model. The above images are examples of a linear regression model that is attempting to fit a good line through the data. We'll talk about this extensively next week, but linear regression models can be curved, You've typically seen a straight line cutting through a cloud of points, but just no that fitting a curve to a cloud of points doesn't violate the linearity of the model equation. Don't worry about this too much right now.

With a linear regression you can create a bend in the line of best fit by introducing another parameter. This parameter has to come from your dataset as another feature (or dimension). If my cloud of points is made up of only 9 data points, then I can curve my line of best fit exactly through each point by including 9 parameters. When discussing this aspect of the Curse of Dimensionality we will abbreviate the number of parameters that we are including in the model (dimensions, features) as **P** and the number of observations (rows in a dataset) as **N**. 

If **P** $\geq$ **N** then I am at extreme risk of overfitting my model. Overfitting means that my model has fit to the noise or variance or the data rather than the acutal signal or phenomenon that we are trying to model. This damages the predictive accuracy of a model and makes it so that our model will not be able to generalize (perform well) to new data that it hasn't seen before.

### Rules of Thumb

While these rules of thumb can vary from model to model, some say that you should always make sure that you have 5 times the number of observations as you have parameters in your model. This is an easier task for some models than others. Some practitioners hold a more extreme view and say that the number of observations should be the square of the number of parameters. I think this is silly once you get to truly high numbers of dimensions, but some people are adamant about it (past professors of mine who really didn't deal with really high numbers of dimensions in their work and were blind to the existence of any other statistical approaches).


## Remember This: 

### You're in grave danger of overfitting if P $\geq$ N
### Make sure you have 5N = P (as a minimum)
### Having more observations is almost always better
### More data in terms of additional dimensions has diminishing value

## Solutions 

### Discard Redundant or Irrelevant Features 
### Get More Data - LOTS more data
### Don't measure similarity via Euclidean Distance
### Get More Data - more observations not more features

More data covereth a multitude of sins.


# Dimensionality Reduction Techniques:

The Curse of Dimensionality means that as "Big Data" gets bigger and bigger the challenges associated with it become more difficult. Not only is it difficult to explore and visualize high dimensional data, but also certain intuitions that exist in 2d or 3d space begin to betray us (distance between points for example). Also models that have a high feature to observation ratio are more prone to overfitting. 

What if we could take Big Data and make it smaller while still retaining most of its substance? In Machine Learning there are a whole host of Dimensionality Reduction Techniques that enable us to do exactly that. (with some tradeoffs - like always.)

## Feacture Selection: 

Say we have 100 features in our dataset. Instead with fighting with all of them, what if we just used the ones that contributed the most. Feature Selection refers to methods of filtering irrelevant or redundant features from our dataset.

## Feature Extraction:

Feature Extraction creates a new, smaller set of features that stills captures most of the useful information. Feature selection finds a suitable subset of a dataset's features, feature extraction creates new ones altogether. PCA is a Feature Extraction method.

# Principal Component Analysis (PCA) 

PCA is a technique that transforms a high dimensional dataset into a new lower dimensional dataset while preserving the maximum amount of information from the original data

# Variance ~~ Information

Imagine you were making shadow puppets by putting your hand in front of some light source that was projecting onto a wall.

If you want the shadow that was being cast onto the wall to be very recognizable as a hand which way would you orient your hand? (kill screenshare)

Why is that orientation more recognizeable as a hand than other orientations?

What is the following image an image of?

<center></center>

What about with clouds of data points?

<http://setosa.io/ev/principal-component-analysis/>


```
import pandas as pd
import matplotlib.pyplot as plt

x = [-2.2,-2,-2,-1,-1,0,0,1,1,2,2,2.2]
y = [0,.5,-.5,.8,-.8,.9,-.9,.8,-.8,.5,-.5,0]

print(len(x))
print(len(y))

data = {"x": x, "y": y}

df = pd.DataFrame(data)

axes = plt.gca()
plt.axes().set_aspect('equal')
plt.scatter(df['x'], df['y'])
```

Imagine that the data pictured above is a scatterplot of two explanatory variables. Lets pretend that we could only keep one of them. To remove the one that we don't need, we're going to collapse this data along one of its dimensions. 

We could choose to project all of these datapoints onto the x-axis or onto the y-axis, which axis should we project the data points onto? Which one preserves more information?


```
import math

x1 = [-2.2,-2,-2,-1,-1,0,0,1,1,2,2,2.2]
y1 = [0,.5,-.5,.8,-.8,.9,-.9,.8,-.8,.5,-.5,0]

data = {"x": x1, "y": y1}

df1 = pd.DataFrame(data)

df1["x_rotate"] = df1.apply(lambda x: (x.x+x.y)/math.sqrt(2), axis=1)
df1["y_rotate"] = df1.apply(lambda x: (x.y-x.x)/math.sqrt(2), axis=1)

axes = plt.gca()
plt.axes().set_aspect('equal')
plt.scatter(df1['x_rotate'], df1['y_rotate'])
```

What if our data didn't lie as cleanly on the x or y axis? How would we then know along what dimension to perform our linear projection?

Well, the dimension that cuts nicely through this cloud of points is the dimension that maximizes the variance of points along it. 


```

```

# The PCA Process:

## 1) Separate your data into X and Y, variables.

Remember that the y variable is the "label" or the "target" it's the thing that we want to predict. our X variables are all of the features of our data that are going ot help us to predict Y.

## 2) Center each column at 0 by subtracting its mean.

Take the matrix of independent variables $X$ and, for each column, subtract the mean of that column from each entry. (This ensures that each column has a mean of zero.)

If we don't do this then it will appear as if one component explains most of the variance in the data and the other components explain very little.  

## 3) Divide each column by its standard deviation.

This completes the process of "standardizing" our data - giving it a mean of 0 and a standard deviation of 1. Just like we standardized our variance and covariance, this now makes the variances of our different features comparable and removes some of the concerns that were brought up when we talked about the rotated elipse of points.
 
We'll call the new standardized matrix the $Z$ matrix.

## 4) Calculate the Variance-Covariance matrix of Z. 

Do this by calculating $Z^{T}Z$ Mind blown! - Multiplying a matrix by its transpose yields the variance-covariance matrix. 

## 5) Calculate the Eigenvectors and Eigenvalues of the Variance-Covariance Matrix.

What are the Eigenvectors and Eigenvalues of the Variance-Covariance Matrix? The eigenvectors are those that run along the dimensions that maximize the variance along that dimension. Remember our rotated elipse example? When we moved our elipse off of the x and y axis we then didn't know what line we should project the points onto so that they we could collapse the dimension of least variance. Well now we do!

## 6) Sort the Eigenvalue and Eigenvector Pairs.

We'll sort our eigenvalue eigenvector pairs that resulted from the eigendecomposition of the variance-covariance matrix. We'll sort them by their eigenvalues in descending order. Since the eigenvalues tell us how to scale an eigenvector. The largest eigenvalue corresponds to the direction of greatest variance in the data. The eigenvalue eigenvector pair with the largest eigenvalue will be Principal Component 1

## 7) Use a matrix transformation to project the datapoints onto our eigenvector subspaces. 

This final step is what makes it so that if we plot our transformed data our principal components will be alined with the axis of our graph. 


```
from numpy import array
from numpy import mean
from numpy import cov
from numpy.linalg import eig

# define a matrix (we'll pretend that we've already separated our X and Y matrices)
X = array([[1, 2], [3, 4], [5, 6]])
print("Data: ", X)
# calculate the mean of each column
means = mean(X.T, axis=1)
print("\n Means: \n", means)
# center columns by subtracting column means
centered_data = X - means
print("\n Centered Data: \n", centered_data)
# calculate covariance matrix of centered matrix
# We're not going to find the covaraicne matrix by the (Z^T)(Z) method, I trust this function more
covariance_matrix = cov(centered_data.T)
print("\n Covariance Matrix: \n", covariance_matrix)
# eigendecomposition of covariance matrix
values, vectors = eig(covariance_matrix)
print("\n Eigenvectors: \n", vectors)
print("\n Eigenvalues: \n", values)
# project data
P = vectors.T.dot(centered_data.T)
print("\n Projected Data: \n", P.T)
```

    [[1 2]
     [3 4]
     [5 6]]
    
     Means: 
     [3. 4.]
    
     Centered Data 
     [[-2. -2.]
     [ 0.  0.]
     [ 2.  2.]]
    
     Covariance Matrix: 
     [[4. 4.]
     [4. 4.]]
    
     Eigenvectors: 
     [[ 0.70710678 -0.70710678]
     [ 0.70710678  0.70710678]]
    
     Eigenvalues: 
     [8. 0.]
    
     Projected Data: 
     [[-2.82842712  0.        ]
     [ 0.          0.        ]
     [ 2.82842712  0.        ]]



```
x = [1,3,5] 
y = [2,4,6]
data = {"x": x, "y": y}

df = pd.DataFrame(data)

plt.scatter(df['x'], df['y'])
plt.title("Data Before PCA")
plt.show()
```


```
x = [-2.82842712,0,2.82842712] 
y = [0,0,0]
data = {"x": x, "y": y}

df = pd.DataFrame(data)

plt.scatter(df['x'], df['y'])
plt.title("Data After PCA")
plt.xlabel('PC1')
plt.show()
```

# Lets use a library to prove to ourselves how smart we are


```
# Principal Component Analysis
from numpy import array
from sklearn.decomposition import PCA
# define a matrix
X = array([[1, 2], [3, 4], [5, 6]])
print("Data: \n", X)
# create the PCA instance
pca = PCA(2)
# fit on data
pca.fit(X)
# access values and vectors
print("\n Eigenvectors: \n", pca.components_)
print("\n Eigenvalues: \n",pca.explained_variance_)
# transform data
B = pca.transform(X)
print("\n Projected Data: \n", B)
```

    Data: 
     [[1 2]
     [3 4]
     [5 6]]
    
     Eigenvectors: 
     [[ 0.70710678  0.70710678]
     [-0.70710678  0.70710678]]
    
     Eigenvalues: 
     [8. 0.]
    
     Projected Data: 
     [[-2.82842712e+00 -2.22044605e-16]
     [ 0.00000000e+00  0.00000000e+00]
     [ 2.82842712e+00  2.22044605e-16]]



```

```

09d162ec9f347a29922eb1f18caa9620ef6a0737
ipynb
Jupyter Notebook
High_Dimensional_Data.ipynb
cocoisland/AB-Demo
108e94fc44fbff0ef1ae01c9e81182caae553ff9
MIT
High_Dimensional_Data.ipynb
cocoisland/AB-Demo
108e94fc44fbff0ef1ae01c9e81182caae553ff9
MIT
High_Dimensional_Data.ipynb
cocoisland/AB-Demo
108e94fc44fbff0ef1ae01c9e81182caae553ff9
MIT
Qwen/Qwen-72B
1. YES
2. YES