# Introduction

This worksheet covers the three supervised learning algorithms we looked at in week 14: k-nearest neighbours, linear regression, and the naive Bayes classifier. Similar to last week, you will do some work implementing your own versions of these algorithms, to ensure that you understand the details of them. You will also compare them with the implementations in scikit-learn to test your implementations.

# Preliminaries
Import key packages: NumPy, matplotlib, and any others that you prefer to work with. In general, when writing code, you will put all your import statements at the top. However, for these worksheets we will import as we go along.


```python
#TODO: import NumPy and matplotlib here
import numpy as np
import matplotlib.pyplot as plt
```

# Question 1: k-nearest neighbours classification
In this question we will use the k-nearest neighbours algorithm to make predictions on the breast cancer Wisconsin dataset. This is a classification problem where the aim is to classify instances as either being malignant or benign based on the following 10 features:
1. radius (mean of distances from center to points on the perimeter)
2. texture (standard deviation of gray-scale values)
3. perimeter
4. area
5. smoothness (local variation in radius lengths)
6. compactness (perimeter squared/ area −1)
7. concavity (severity of concave portions of the contour)
8. concave points (number of concave portions of the contour)
9. symmetry
10. fractal dimension (‘coastline approximation’ −1)

In this question you will (a) download the dataset from sklearn and store the data and targets in suitable variables, (b) separate your data into a training and test split, (c) write your own function to implement k-nearest neighbours, (d) check your implementation with that of sklearn. Wethen go on to (e) select the most appropriate value of $k$ using cross-validation.

## Part (a)
Import the package `datasets` from `sklearn` and then load the breast cancer dataset (function is `load_breast_cancer()`). Save the data into a variable `X` and the targets into a variable `Y`. 
Take a look at the data in `X`. How many datapoints are there? How many features does each datapoint have? (Hint: use `np.shape`).
Take a look at the targets. Is this suitable for a classification algorithm or a regression algorithm?


```python
# Import suitable packages, load the dataset, and save data and targets into variables X and Y
# import packages
from sklearn import datasets

# load dataset and save data and targets into X and Y
cancer = datasets.load_breast_cancer()
X = cancer.data
Y = cancer.target

#该数据集一共有569个样本,30个特征
print(X.shape)
print(cancer.feature_names)
```

    (569, 30)
    ['mean radius' 'mean texture' 'mean perimeter' 'mean area'
     'mean smoothness' 'mean compactness' 'mean concavity'
     'mean concave points' 'mean symmetry' 'mean fractal dimension'
     'radius error' 'texture error' 'perimeter error' 'area error'
     'smoothness error' 'compactness error' 'concavity error'
     'concave points error' 'symmetry error' 'fractal dimension error'
     'worst radius' 'worst texture' 'worst perimeter' 'worst area'
     'worst smoothness' 'worst compactness' 'worst concavity'
     'worst concave points' 'worst symmetry' 'worst fractal dimension']


## Part (b)

Use the function `train_test_split` from `sklearn.model_selection` to split your data into a training set and a held-out test set. Use a test set that is 0.2 of the original dataset. Set the parameter `random_state` to 10 to help with replication.


```python
# Import the package train_test_split from sklearn.model_selection.
from sklearn.model_selection import train_test_split
# Split the dataset into Xtr, Xtest, Ytr, Ytest. Xtest and Ytest will form your held-out
# test set. You will later split Xtr and Ytr into training and validation sets.
Xtr, Xtest, Ytr, Ytest = train_test_split(X, Y, test_size=0.2, random_state=10)
print(Xtest.shape)
```

    (114, 30)


## Part (c) 
Recall from the lecture that the k-nearest neighbours algorithm runs as follows:

Training step: Simply store the dataset

Prediction step: Given a datapoint $\vec{x}$:
 - **Find** the k datapoints $(\vec{x}_i, y_i)$ where the distance from $\vec{x}$ to $\vec{x}_i$ is smallest
 - **Return** the majority class from the $y_i$
   
 
What, if anything, do you need to do for the training step?

Write function(s) to implement the k-nearest neighbours prediction step. You may wish to break the procedure down into two functions `predict_datapoint` that makes a prediction for one datapoint and `predict_data` that loops over the whole dataset.

To select the majority class from the nearest neighbours, you can use the function `scipy.stats.mode()`



```python
from numpy.ma.core import argsort
from scipy.stats import mode
# Write function(s) to implement the prediction step in k-nearest neighbours. 
# You can use the suggested structure below if desired.

# predict_datapoint takes 4 arguments. pt (type: numpy array) is the datapoint we are making a prediction about, 
# Xtrain and Ytrain (numpy arrays) are training data and targets, k (int) is the number of neighbours.
# Returns an integer which is the predicted class for pt
def predict_datapoint(pt, Xtrain, Ytrain, k):
    distances = []
    classes = []
    length = len(Xtrain)
    # For each datapoint in Xtrain, calculate the distance to pt and store
    for i in range(length):
      distance = np.sum((pt-Xtrain[i])**2)
      distances.append(distance)
    # Sort the list of distances (hint: use np.argsort)
    sort = np.argsort(distances)
    # print(distances)
    # print(sort)
    # obtain the classes (in Ytrain) of the datapoints with the smallest distance to pt
    for j in range(k):
      classes.append(Ytrain[sort[j]])
    # return the mode of the classes
    return mode(classes)[0][0]



# predict_data takes 4 arguments: the test data Xtst (numpy array), the training data Xtrain (numpy array),
# the training targets Ytrain (numpy array), and the number of neighbours k (int, default = 3). 
# Returns: predictions (array of int) for each point in Xtst
def predict_data(Xtst, Xtrain, Ytrain, k=3):
    #Loop over the datapoints in Xtst and store the prediction for that datapoint
    length = len(Xtst)
    predicts = []
    for i in range(length):
      predicts.append(predict_datapoint(Xtst[i], Xtrain, Ytrain, k))
    # Return the predictions
    return predicts

# Predict values for the TRAINING data (we will not look at the test set yet)
Predictions = predict_data(Xtr, Xtr, Ytr, 3)
print(Predictions)
```

    [0, 1, 1, 0, 0, 1, 1, 0, 1, 0, 1, 1, 0, 0, 1, 1, 1, 1, 0, 1, 1, 0, 1, 0, 1, 0, 1, 1, 0, 1, 1, 0, 1, 1, 1, 1, 0, 1, 0, 1, 1, 1, 0, 1, 1, 0, 1, 1, 1, 1, 1, 0, 0, 1, 1, 1, 1, 1, 0, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 0, 1, 0, 1, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 0, 0, 1, 1, 1, 0, 0, 1, 1, 0, 0, 1, 1, 1, 1, 1, 0, 1, 0, 1, 1, 0, 0, 1, 0, 0, 1, 1, 0, 0, 1, 1, 1, 1, 1, 1, 0, 1, 1, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 0, 1, 0, 1, 1, 0, 1, 0, 0, 1, 0, 0, 0, 1, 0, 1, 0, 1, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 0, 1, 1, 1, 1, 0, 1, 1, 1, 1, 0, 1, 1, 0, 0, 1, 1, 0, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 1, 1, 1, 0, 1, 1, 0, 1, 1, 1, 1, 0, 1, 0, 1, 0, 0, 1, 0, 1, 0, 1, 0, 0, 1, 0, 1, 1, 0, 1, 0, 1, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 1, 0, 1, 1, 1, 1, 0, 1, 0, 1, 1, 1, 0, 1, 1, 0, 0, 0, 1, 0, 0, 1, 0, 1, 1, 1, 0, 1, 0, 1, 0, 0, 0, 1, 1, 1, 1, 1, 0, 1, 1, 1, 0, 1, 0, 1, 1, 0, 1, 1, 1, 0, 0, 1, 1, 0, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 0, 0, 1, 0, 0, 0, 0, 1, 0, 1, 1, 1, 0, 1, 1, 0, 1, 1, 0, 1, 1, 0, 1, 1, 1, 0, 1, 1, 0, 1, 0, 1, 1, 0, 0, 1, 0, 1, 1, 0, 1, 1, 1, 1, 0, 1, 1, 0, 1, 0, 1, 1, 0, 1, 1, 1, 0, 1, 1, 1, 0, 0, 1, 0, 1, 1, 1, 1, 1, 1, 0, 0, 1, 1, 1, 1, 1, 1, 1, 0, 0, 1, 1, 1, 1, 0, 1, 0, 1, 1, 0, 0, 1, 0, 1, 0, 1, 1, 1, 1, 1, 0, 1, 0, 0, 0, 1, 0, 1, 0, 1, 1, 1, 0]


## Part (d)
Now we can compare your implementation with the sklearn implementation (you should get the same results). Firstly import the classfifier `KNeighborsClassifier` from `sklearn.neighbors`. Instantiate the classifier with the same number of neighbours that you used previously. Fit the model and make a prediction on the test set.


```python
# Import KNeighborClassifier
from sklearn.neighbors import KNeighborsClassifier

# Instantiate the classifier with 3 neighbors
KNN = KNeighborsClassifier(n_neighbors=3)  

#Fit the classifier on the training data
KNN.fit(Xtr,Ytr)

#Make a prediction on the training data
KNN_Pre = KNN.predict(Xtr)
print(KNN_Pre)
```

    [0 1 1 0 0 1 1 0 1 0 1 1 0 0 1 1 1 1 0 1 1 0 1 0 1 0 1 1 0 1 1 0 1 1 1 1 0
     1 0 1 1 1 0 1 1 0 1 1 1 1 1 0 0 1 1 1 1 1 0 1 1 0 1 1 1 1 1 1 1 1 0 1 1 1
     0 1 0 1 0 0 1 1 1 1 1 1 1 1 1 0 1 0 0 1 1 1 0 0 1 1 0 0 1 1 1 1 1 0 1 0 1
     1 0 0 1 0 0 1 1 0 0 1 1 1 1 1 1 0 1 1 0 0 0 0 0 1 1 1 1 1 1 0 1 1 1 1 1 0
     1 0 1 1 0 1 0 0 1 0 0 0 1 0 1 0 1 0 0 1 1 1 1 1 1 1 1 1 0 1 1 1 1 1 1 1 0
     1 1 1 0 1 1 1 1 0 1 1 1 1 0 1 1 0 0 1 1 0 1 0 1 1 1 1 1 1 1 1 1 0 0 1 1 1
     1 1 1 1 1 1 1 0 0 0 1 1 1 0 1 1 0 1 1 1 1 0 1 0 1 0 0 1 0 1 0 1 0 0 1 0 1
     1 0 1 0 1 0 0 0 1 1 0 0 0 0 0 1 0 1 1 1 1 0 1 0 1 1 1 0 1 1 0 0 0 1 0 0 1
     0 1 1 1 0 1 0 1 0 0 0 1 1 1 1 1 0 1 1 1 0 1 0 1 1 0 1 1 1 0 0 1 1 0 1 1 1
     1 1 1 0 1 1 1 1 0 0 1 0 0 0 0 1 0 1 1 1 0 1 1 0 1 1 0 1 1 0 1 1 1 0 1 1 0
     1 0 1 1 0 0 1 0 1 1 0 1 1 1 1 0 1 1 0 1 0 1 1 0 1 1 1 0 1 1 1 0 0 1 0 1 1
     1 1 1 1 0 0 1 1 1 1 1 1 1 0 0 1 1 1 1 0 1 0 1 1 0 0 1 0 1 0 1 1 1 1 1 0 1
     0 0 0 1 0 1 0 1 1 1 0]


Check whether your predictions are the same as the predictions from `KNeighborsClassifier`.


```python
#首先要将自己写的方法最终输出的list转化为array
DIYKNN_Pre = np.array(Predictions)

check = (KNN_Pre==DIYKNN_Pre).all()
if check==True:
  print("My predictions are the same as the predictions from KNeighborsClassifier")
```

    My predictions are the same as the predictions from KNeighborsClassifier


Use the built in metrics in sklearn to calculate the accuracy of your classifier on the TRAINING set.


```python
from sklearn.metrics import accuracy_score

print(accuracy_score(Ytr, KNN_Pre)) 
```

    0.9560439560439561


## Part(e) Using cross-validation for model selection
k-nearest neighbours has the parameter $k$, and we need to decide which is the best value of $k$ to use. Last week we talked about using cross-validation for model selection.

We will use cross-validation on our training set to select the best value of $k$, in a range from 1 to 30.

NB: use sklearn's version of k-NN rather than yours, since unless you have optimised yours it is probably too slow.

Since we are using cross-validation for model selection we will cross-validate on the training set only.

Procedure:
        
 1. Import `KFold` from `sklearn.model_selection`  
 1. Instantiate `KFold` with 5 splits. Set the parameter `random_state` to help you reproduce your results if needed.
 1. Set a variable `max_k` to 30  
 1. Inititalise two variables to store the training accuracies and validation accuracies (these need to store max_k\*5 accuracies)  
 1. Loop over the values of k:  
    1. Instantiate a k-nn classifier (Use the sklearn classifier) with the current value of k  
    1. Loop over the cross-validation splits:  
       1. fit the model on the current split of data  
       1. make predictions  
       1. calculate training and validation accuracy and store  
 6. Calculate the mean training and validation accuracies across splits for each $k$

Plot the mean training and validation accuracies. Which value of $k$ will you use? Why?
    


```python
# Use cross-validation to select the value of k
# You can use the structure below if desired

# Import KFold from sklearn.model_selection
from sklearn.model_selection import KFold

# Instantiate KFold with 5 splits. 
# Set the parameter random_state to help you reproduce your results if needed.
##TODO##
KF = KFold(n_splits=5, shuffle=True, random_state=10)
# Set a variable max_k to 30 
max_k = 30

# Inititalise two variables to store the 
# training accuracies and validation accuracies 
# (these need to store max_k*5 accuracies) 
validation_accuracies =[]

# Loop over the values of k: 
for k in range(max_k):
    current_score = 0
    # Instantiate a k-nn classifier (Use the sklearn classifier)
    # with the current value of k 
    # Loop over the cross-validation splits: 
    for train_index,valid_index in KF.split(X):

        CV_KNN = KNeighborsClassifier(n_neighbors=(k+1))
 
        # fit the model on the current split of data 
        CV_KNN.fit(X[train_index],Y[train_index])
        current_score = current_score + CV_KNN.score(X[valid_index],Y[valid_index])
        
# Calculate the mean training and validation accuracies across splits for each 𝑘
    avg_score = current_score/5
    validation_accuracies.append(avg_score)
```


```python
# Plot the mean training and validation accuracies against each value of k. Which value of 𝑘 will you use? Why?
K = np.arange(1,31,1)
plt.plot(K, validation_accuracies, label="K selection by cross validation")
plt.xlabel("K value")
plt.ylabel("val_accuracy")
plt.show()
```

# The naive Bayes classifier

Recall from the lecture notes that the naive Bayes classifier works as follows. We are trying to approximate an unknown function $$f:\Omega \rightarrow \mathcal{O}$$
where $\Omega$ is our feature space and our output space $\mathcal{O} = \{c_1, c_2, ... c_K\}$ is a finite set of classes.

The naive Bayes classifier does this by building a model that assigns the class label $\hat{y} = c_k$ as follows:
$$
\hat{y} = \text{argmax}_k p(c_k)\prod_i p(x_i| c_k)
$$
i.e., the $k$ that maximises this quantity.

In practice, multiplying all the $p(x_i| c_k)$ together is going to give some very small values. Therefore, we can take the log to make it easier to compute:
\begin{align}
\hat{y} &= \text{argmax}_k p(c_k)\prod_i p(x_i| c_k)= \text{argmax}_k log(p(c_k)\prod_i p(x_i| c_k))\\
&=\text{argmax}_k log(p(c_k)) + \sum_i log(p(x_i|c_k))
\end{align}

If we choose that $p(x_i|c_k)$ is given by a normal distribution with mean $\mu_k$ and variance $\sigma_k^2$, then we obtain the following expression:

\begin{align}
\hat{y} &=\text{argmax}_k \log(p(c_k)) + \sum_i \log(p(x_i|c_k))\\
&= \text{argmax}_k \log(p(c_k)) + \sum_i \log\left(\frac{1}{\sigma_k\sqrt{2\pi}} exp\left(\frac{-(x-\mu_k)^2}{2\sigma_k}\right)\right)\\
&= \text{argmax}_k \log(p(c_k)) - \sum_i \log\left(\sigma_k\sqrt{2\pi}\right) - \sum_i\left(\frac{(x-\mu_k)^2}{2\sigma_k}\right) \quad \text{ log-likelihood}
\end{align}

Expressing the values in terms of these sums means that they do not get so small, and it is less likely that there will be errors at the machine precision level.


How do we implement this in practice? We assume that each probability $p(x_i| c_k)$ is given by some distribution, and then given a datapoint $\vec{x}$, we plug the value into the equation for the distribution.

In this question you will (a) implement your own version of the Gaussian naive Bayes classifier, (b) check your classifier against the implementation in sci-kit learn, (c) compare the accuracy of the naive Bayes classifier with the accuracy of the k-nearest neighbours classifier, and (d) run cross-validation to verify whether the kNN classfier or the Gaussian naive Bayes classifier performs better on this dataset.

## Part (a) Implementing Gaussian naive Bayes
For this question we will make the assumption that each feature is described by a normal (also called Gaussian) distribution. The procedure is as follows:
1. Divide the training data by class
2. Calculate mean and standard deviation per class and per feature
4. For each datapoint in the validation set, calculate the log-likelihood for each class and for each feature (Hint: use the function `scipy.stats.norm.logpdf`)
5. Combine these values together with the probability of the class according to the log-likelihood equation above
6. Choose the class with the highest value



```python
# Write your own implementation of naive Bayes applied to the breast cancer dataset.

# If you wish you can follow the structure below
from scipy.stats import norm
import math

#Separate the training set into classes, so you have one set of data for each class
#首先求得 是癌症或者不是癌症的概率，即求得P(class)
#为了求这个，先将训练集进行分类 即Ytr为0的分为一类，Ytr为1的分为一类
index_cancer = []
index_nocancer = []
for i in range(len(Ytr)):
  if Ytr[i]==1:
    index_cancer.append(i)
  else:
    index_nocancer.append(i)

Xtr_cancer = Xtr[index_cancer] #在这个训练集中一共有282个正样本
# print(Xtr_cancer.shape) 
Xtr_nocancer = Xtr[index_nocancer] #在这个训练集中一共有173个负样本
# print(Xtr_nocancer.shape)

# Calculate the means and standard deviations for each class, for each feature. 
# There are 30 features in the dataset, so you should have a 30-dimensional 
# array of means for each class and a 30-dimensional array of standard deviations
# for each class. Remember that you can take the average across rows or columns of 
# a matrix by specifying axis = 1 or axis = 0

feature_mean_cancer = np.sum(Xtr_cancer,axis=0)/282
feature_mean_nocancer = np.sum(Xtr_nocancer,axis=0)/173
feature_std_cancer = np.std(Xtr_cancer,axis=0)
feature_std_nocancer = np.std(Xtr_nocancer,axis=0)
# print(feature_mean_nocancer.shape)

# Calculate the prior probability p(c_i) for each class
p_cancer = 282/455
p_nocancer = 173/455

# Calculate the log-likelihood of each class for each datapoint in the validation set
# Hint: you can use the function scipy.stats.norm.logpdf to help with this
like_hood_cancer = np.log(p_cancer) + np.sum(norm.logpdf(Xtest,loc=feature_mean_cancer,scale=feature_std_cancer),axis=1)
like_hood_nocancer = np.log(p_nocancer) + np.sum(norm.logpdf(Xtest,loc=feature_mean_nocancer,scale=feature_std_nocancer),axis=1)

# Your predicted class is 0 if class 0 has the highest log-likelihood, and 1 if class 1 
arr = like_hood_cancer - like_hood_nocancer
pre = np.int64(arr>0)
print(pre)
```

    [0 1 1 0 1 0 1 1 0 0 1 1 0 1 0 0 1 1 1 0 0 0 1 1 0 0 1 0 1 1 1 1 0 1 1 1 1
     1 1 1 1 1 1 0 1 0 0 0 0 0 1 1 0 1 1 0 0 1 1 1 0 1 0 0 1 1 1 1 1 0 1 1 1 1
     1 0 1 1 0 0 1 1 0 1 1 1 1 0 1 1 1 1 0 1 1 1 0 1 0 1 0 0 1 1 1 1 0 0 1 1 1
     1 0 0]


## Part (b) Checking results
We now compare our results with the sklearn implementation. 


```python
##Import the classifier GaussianNB from sklearn.naive_bayes
from sklearn.naive_bayes import GaussianNB
# Instantiate the classifier (use the parameter var_smoothing=0.0),
# fit, and predict the classes
NB = GaussianNB()
NB.fit(Xtr,Ytr)
NB.predict(Xtest)
```




    array([0, 1, 1, 0, 1, 0, 1, 1, 0, 0, 1, 1, 0, 1, 0, 0, 1, 1, 1, 0, 0, 0,
           1, 1, 0, 0, 1, 0, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0,
           1, 0, 0, 0, 0, 0, 1, 1, 0, 1, 1, 0, 0, 1, 1, 1, 0, 1, 0, 0, 1, 1,
           1, 1, 1, 0, 1, 1, 1, 1, 1, 0, 1, 1, 0, 0, 1, 1, 0, 1, 1, 1, 1, 0,
           1, 1, 1, 1, 0, 1, 1, 1, 0, 1, 0, 1, 0, 0, 1, 1, 1, 1, 0, 0, 1, 1,
           1, 1, 0, 0])




```python
# Compare your predicted classes with those of the sklearn implementation.
# If they are not identical, this may be due to some differences in parameter setting. 
# They should be almost all the same.
check1 = (pre==NB.predict(Xtest)).all()
if check1==True:
  print("My predictions are the same as the predictions from KNeighborsClassifier")
```

    My predictions are the same as the predictions from KNeighborsClassifier


## Part (c) Comparing k-nearest neighbours and Gaussian naive Bayes
Now retrain the naive Bayes classifier using the original training set `Xtr`, `Ytr`.
Also, retrain the k-nearest neighbours classifier using `Xtr` and `Ytr`. Use the value of $k$ that you decided on using cross-validation.
You can use the sklearn implementations of knn and naive Bayes.

Compute the accuracy of the naive Bayes classifier over the training set and the held-out test set.

Compare with the accuracy of the k-nearest neighbours classifier on each set.

Is it clear which classifier is the best on this dataset? why or why not?


```python
# Instantiate the knn classifer with your chosen value of k
# Fit the Gaussian naive Bayes classifier and the knn classifier on Xtr, Ytr

KNN_C = KNeighborsClassifier(n_neighbors=10)  
KNN_C.fit(Xtr,Ytr)

NB = GaussianNB()
NB.fit(Xtr,Ytr)
# Make predictions for the training set
KNN_Train_Pre = KNN_C.predict(Xtr)
NB_Train_pre = NB.predict(Xtr)

# Take a look at the accuracy scores
print(accuracy_score(Ytr, KNN_Train_Pre)) 
print(accuracy_score(Ytr, NB_Train_pre)) 

# Make predictions for the test set
KNN_Test_Pre = KNN_C.predict(Xtest)
NB_Test_pre = NB.predict(Xtest)

# Take a look at the accuracy scores
print(accuracy_score(Ytest, KNN_Test_Pre)) 
print(accuracy_score(Ytest, NB_Test_pre))
```

    0.9428571428571428
    0.9318681318681319
    0.9298245614035088
    0.956140350877193


## Part (d) Using cross-validation for statistical validation
Earlier we used cross-validation to select the model parameters we would be using. We can also use it another way: to provide statistical information about which model is best. We will set up cross-validation on the whole dataset, with 10 folds.

 - Compute the accuracy for each model on the test set on each fold.
 - Calculate the mean accuracy across folds. Which model performs best?
 - Make a box-plot of the spread of scores of each model. Is there a clear difference between model performance?
 - Perform a paired t-test on the accuracy scores. What can you conclude about the performance of the two models?



```python
# Set up a k-fold cross-validation with 10 folds
##TODO##
```


```python
# For each fold, fit each model on the training data 
# and compute accuracy on the test data.
##TODO##
```


```python
# Compute the mean and standard devation of the accuracies for each model.
# Does one model perform better?
##TODO##
```


```python
# Make a boxplot of the accuracy scores. (Use plt.boxplot). 
# Is there a clear difference between the models?
##TODO##
```


```python
# Perform a paired t-test (you can use the function scipy.stats.ttest_rel). 
# What do you conclude about the performance of the two models?
##TODO##
```

## Linear Regression
In linear regression we make the assumption that the data $(x_i, y_i)$ can be modelled by a function of the form
$$ \hat{y_i} = f(\vec{x}_i)= \sum_j a_j x_{ij}  + b_i$$

Recall that we can express this in a matrix format by:
$$ \hat{\vec{y}} = f(X)= X\Theta$$

where 
$$ X=\begin{pmatrix}
x_{1,1} & x_{1,2} & \ldots & x_{1,n} &1 \\
\vdots & \vdots & \ldots & \vdots & \vdots \\
x_{N,1} & x_{N,2} & \ldots & x_{N,n} & 1
\end{pmatrix}, \quad \vec{y}=\begin{pmatrix} y_1 \\ \vdots \\y_N \end{pmatrix}, \quad \Theta=\begin{pmatrix} a_1 \\ \vdots \\a_n\\b \end{pmatrix}$$

We saw in lectures that the optimal value of $\Theta$ is given by setting
$$ \Theta = (X^T X)^{-1} X^T \vec{y}$$

The quantity $(X^T X)^{-1} X^T$ is called the psuedoinverse of X, and can be computed using the function `np.linalg.pinv`.

We will (a) perform a linear regression on the diabetes dataset. You can load this dataset using the function `load_diabetes` from `sklearn.datasets`. (b) compute the mean squared error and the R^2, and (c) compare your results with the built in function in sklearn (`sklearn.linear_model.LinearRegresion()`). You should get the same results.


```python
# import statments here
from sklearn import datasets, linear_model
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error, r2_score
```

## Part (a) Implementing linear regression


```python
# Load the diabetes dataset 
##TODO##

# Split the dataset into training and test, using test_size=0.2
##TODO##
```


```python
# Add a column of ones to Xtrain and Xtest for the intercept term
##TODO##
```


```python
# Calculate the value of the coefficients theta. You can use the function np.linalg.pinv
##TODO##
```

## Part (b) Computing performance metrics


```python
# Make a prediction on the test set by applying the coefficients theta to the test set
##TODO##
```


```python
# Calculate the mean squared error and the R^2. 
# You can use the built in functions from sklearn
##TODO##
```

## Part (c) Checking results
Compare your results with the built in function `sklearn.linear_model.LinearRegression()`


```python
# Instantiate the linear regression
##TODO##

```


```python
# Fit the model and make a prediction on the test set. Compare with your implementation
##TODO##

```

Visualise the perfomance of the regression by plotting your predicted values vs target values on a scatter plot, and drawing a line y=x. If all predictions were perfect, the predicted values would lie on the line.


```python
# Plot predicted values vs target values on a scatter plot, and drawing a line y=x
## TODO##

```

# Extra question: Polynomial regression
The term 'linear' in linear regression refers only to the coefficients $\theta$. We can in fact compute polynomial terms in the data and perform linear regression over this extended dataset to get a better fit to the data.

To compute polynomial terms in the data automatically, you can use the class `sklearn.preprocessing.PolynomialFeatures`. To find out how to use it, look at the guidance (you can type `help(PolynomialFeatures)` once you have imported it).

The following small dataset (in the cell below) gives a relationship between temperature and yield for an experiment. Use cross-validation to select the degree of the polynomial that best fits this data.

Plot the mean squared error against degree on the training set and on the validation set. Which degree of polynomial best fits this data?


```python
# Data
X = np.array([50,50,50,70,70,70,80,80,80,90,90,90,100,100,100]).reshape(-1, 1)
y = np.array([3.3,2.8,2.9,2.3,2.6,2.1,2.5,2.9,2.4,3,3.1,2.8,3.3,3.5,3]).reshape(-1, 1)
```


```python

```


```python

```


```python

```

957106432d69cc128c4747d4a7771897fce83812
ipynb
Jupyter Notebook
Week14/code/week14.ipynb
YifanWangCN/Introduction_to_artificial_intelligence
83cee764c6fc5e4c7245917b6eb651c546059368
MIT
Week14/code/week14.ipynb
YifanWangCN/Introduction_to_artificial_intelligence
83cee764c6fc5e4c7245917b6eb651c546059368
MIT
Week14/code/week14.ipynb
YifanWangCN/Introduction_to_artificial_intelligence
83cee764c6fc5e4c7245917b6eb651c546059368
MIT
Qwen/Qwen-72B
1. YES
2. YES