much simpler than the SVM to understand and can also be used for both classification and regression. The kNN routine calculates “distances” to the $k$ nearest training data points for each point $\mathbf{X}_i$ that we are interested in predicting $Y_i$. This distance is often just a simple Euclidean distance between the features $\mathbf{X}$, however, one can imagine using other definitions as well. The predicted $Y_i$ is then just the average of the $k$ nearest training set $Y$ values. This average can be weighted according to distance such that points further away have less impact on the predicted value. Mathematically, one can represent this as $$Y_i = \frac{\sum_{k} w(d(\mathbf{X}_k,\mathbf{X}_i))Y_k }{k}$$ where $d(\mathbf{X}_k,\mathbf{X}_i)$ is the distance between $\mathbf{X}_i$ and one of its $k$ nearest neighbours $\mathbf{X}_k$, and $w(d)$ is the weight corresponding to that distance.

We again use the scikit-learn implementation of kNN [@Pea11]. The algorithm takes in a value for $k$. Again, the optimal value can be found by stepping through a pre-determined set $k=\{3,6,...,21,24\}$ and picking the value with the lowest MSE when the algorithm is applied to the validation set.

Feature Selection {#sec:fs}
-----------------

Feature selection is the process by which we select which features are the most relevant 