# COMP 135 Spring 2019: HW2 Solution Notebook

## Setup comp135_env package imports¶



```python
import os
import numpy as np
import warnings

import sklearn.preprocessing
import sklearn.pipeline
import sklearn.linear_model
import sklearn.metrics
import sklearn.model_selection

from matplotlib import pyplot as plt
import seaborn as sns
```


```python
%matplotlib inline
```

## Setup student-defined imports


```python
# None needed
```

## Load Auto dataset for Problem 1


```python
x_tr_MF = np.loadtxt('data_auto/x_train.csv', delimiter=',', skiprows=1)
x_va_NF = np.loadtxt('data_auto/x_valid.csv', delimiter=',', skiprows=1)
x_te_PF = np.loadtxt('data_auto/x_test.csv', delimiter=',', skiprows=1)
```


```python
y_tr_M = np.loadtxt('data_auto/y_train.csv', delimiter=',', skiprows=1)
y_va_N = np.loadtxt('data_auto/y_valid.csv', delimiter=',', skiprows=1)
y_te_P = np.loadtxt('data_auto/y_test.csv', delimiter=',', skiprows=1)
```

## Problem 1a: Polynomial Degree Selection on Fixed Validation Set

#### **1a(i):** Plot the number of polynomial features $G$ vs. degree $D$


```python
degree_list = [1, 2, 3, 4, 5, 6, 7, 8]
num_poly_feat_list = []
for degree in degree_list:    
    poly_transformer = sklearn.preprocessing.PolynomialFeatures(
        degree=degree, include_bias=False)
    # TODO make the plot
```

### 1a(ii): Fit a linear regression model to a polynomial feature transformation of the provided training set of $x$, $y$ values at each of these possible degrees: [1, 2, 3, 4, 5, 6].



```python
degree_list = [1, 2, 3, 4, 5, 6]
err_tr_list = []
err_va_list = []
coef_list = []
for degree in degree_list:
    poly_transformer = sklearn.preprocessing.PolynomialFeatures(
        degree=degree, include_bias=False)
    # TODO fit models and make the plot
    # Be sure to track the coefs of the learned weights
```

### **1a(iii):** Based on this plot, which single degree value do you recommend? Why?

TODO answer here

### **1a(iv):** Report the numerical values of the 5th percentile and 95th percentile of the coefficients observed in your linear regression model for degrees 3, 4, 5, and 6. What seems to be happening?


```python
# TODO answer here
```

### **1a(v):** Comment on the training error observed at degree 6. Based on your formulas from **1a(i)** and your knowledge of linear regression, what **should** the training error be at degree 6? What do you think is happening instead?

TODO answer here


```python

```

# 1b: Linear regression with Rescaled Polynomial Bases

Setup code below to make a pipeline


```python
def make_pipeline(degree=None):
    pipeline = sklearn.pipeline.Pipeline(
        steps=[
         ('rescaler', sklearn.preprocessing.MinMaxScaler()),
         ('poly_transformer', sklearn.preprocessing.PolynomialFeatures(degree=degree, include_bias=False)),
         ('poly_rescaler', sklearn.preprocessing.MinMaxScaler()),
         ('linear_regr', sklearn.linear_model.LinearRegression()),
        ])
    
    # Define utility method that performs all the feature transforms of the pipeline in order.
    def calc_features(x):
        ''' Maps input features to rescaled polynomial features
        '''
        feat = x
        for step_name, step_tfmr in pipeline.steps[:-1]:
            feat = step_tfmr.transform(feat)
        return feat
    pipeline.calc_features = calc_features
    
    # Define utility method that accesses the linear regression weights
    pipeline.get_linear_regr_weights = lambda: pipeline.named_steps['linear_regr'].coef_

    # Return the constructed pipeline
    # We can treat it as if it has a 'regression' API
    # e.g. a fit and a predict method
    return pipeline
```


```python
degree_list = [1, 2, 3, 4, 5, 6]
err_tr_list = []
err_va_list = []
pipeline_list = []

# TODO fit model at each degree and track error metrics
```

#### 1b(i): Plot MSE for *rescaled* polynomial features vs. degrees


```python
# TODO make plot here
```

### **1b(ii):** Using this new analysis, which degree do you recommend? 


ANSWER HERE

### **1b(iii):** Report the numerical values of the 5th percentile and 95th percentile of the coefficients observed in this most recent linear regression model at all degrees. What seems to be happening? What's different than in **1a**?


```python
# TODO Code here
```

**Answer**: goes here

# 1c: Cross Validation setup

Combine original train/valid splits into one mega dataset


```python
x_trva_LF = np.vstack([x_tr_MF, x_va_NF])
y_trva_L = np.hstack([y_tr_M, y_va_N])
```

### **Coding Step 1/1:** Complete the starter code function 


```python
def calc_mean_squared_error_across_k_folds(
        model, x_LF, y_L, K=5, random_state=0):
    ''' Calculate mean squared error on K cross-validation folds
    
    Args
    ----
    model: sklearn predictor object
        has `fit` and `predict` methods
    x_LF : 2D array, size n_examples x n_features
        
    y_L : 1D array, size n_examples
    K : int
    random_state : int or np.random.RandomState instance
    
    Returns
    -------
    mse_scores_K : 1D array, size K
        estimate of MSE on each of the K folds
    '''
    kfold_iterator = sklearn.model_selection.KFold(
        n_splits=K, shuffle=True, random_state=random_state)
    
    # TODO compute mse on each of the K folds
```

### **1c(i):** Using your `calc_mean_squared_error_across_k_folds` function with 10 folds, make a line plot of the *average* mean-squared-error at degrees 1, 2, 3, 4, 5, 6. 


```python
# TODO 
```

#### **1c(ii):** Based on this plot, what is your recommended degree? How do your recommendations differ from *1b*?

**Answer**: TODO

#### **1c(iii):** For each $K$, make a scatter plot of the $K$ fold-specific estimates of MSE ($K$ is x-axis, MSE on y-axis). Also draw a line connecting the average MSE across $K$.


```python
degree = 3
K_list = [2, 5, 10, 15, 20, 30, 40, 50, 60, 70, 80, 90, 100, 110, 120, 130, 140]
pipeline = make_pipeline(degree)
err_cv_list = []
avg_score_list = []

# TODO
```

#### **1c(iv):** What happens to the distribution of MSE as you use more and more folds? Is the trend observed in Plot **1c(iii)** what we expect based on your readings from the ISL textbook Chapter 5? 


TODO Answer here.


```python

```

# Problem 2: Regularized Linear Regression via L2 and L1 penalties

### Setup Pipeline for Problem 2a: Grid Search for L2


```python
alpha_grid = np.logspace(-9, 6, 31)
```


```python
def make_L2regularized_pipeline(degree=2, alpha=1.0):
    pipeline = sklearn.pipeline.Pipeline(
        steps=[
         ('rescaler', sklearn.preprocessing.MinMaxScaler()),
         ('poly_transformer', sklearn.preprocessing.PolynomialFeatures(degree=degree, include_bias=False)),
         ('poly_rescaler', sklearn.preprocessing.MinMaxScaler()),
         ('linear_regr', sklearn.linear_model.Ridge(alpha=alpha)),
        ])
    
    # Define utility method that performs all the feature transforms of the pipeline in order.
    def calc_features(x):
        tmp = x
        for step_name, step_tfmr in pipeline.steps[:-1]:
            tmp = step_tfmr.transform(tmp)
        return tmp
    pipeline.calc_features = calc_features
    
    pipeline.get_linear_regr_weights = lambda: pipeline.named_steps['linear_regr'].coef_
    pipeline.get_linear_regr_bias = lambda: pipeline.named_steps['linear_regr'].intercept_

    return pipeline
```


```python

```

## Problem 2a: Grid Search for L2

### **2a(i):** Train `Ridge` regression across grid of possible L2-penalty strengths $\alpha$. Using *degree 2* polynomial features, plot the MSE vs. regularization strength on train and validation.


```python
# TODO plotting code here
```

### **2a(ii):** Make the same plot at degree 6


```python
# TODO plotting code here
```

### 2a(iii): How does the recommended $\alpha$ change from deg. 2 to 6?

**Answer**: here

### 2a(iv): Plot nonzero entries vs $\alpha$ for degree 6?

CODE HERE

## Problem 2b: Grid search for L1


```python
def make_L1regularized_pipeline(degree=2, alpha=1.0):
    pipeline = sklearn.pipeline.Pipeline(
        steps=[
         ('rescaler', sklearn.preprocessing.MinMaxScaler()),
         ('poly_transformer', sklearn.preprocessing.PolynomialFeatures(degree=degree, include_bias=False)),
         ('poly_rescaler', sklearn.preprocessing.MinMaxScaler()),
         ('linear_regr', sklearn.linear_model.Lasso(alpha=alpha, max_iter=50000)),
        ])
    
    # Define utility method that performs all the feature transforms of the pipeline in order.
    def calc_features(x):
        tmp = x
        for step_name, step_tfmr in pipeline.steps[:-1]:
            tmp = step_tfmr.transform(tmp)
        return tmp
    pipeline.calc_features = calc_features
    
    pipeline.get_linear_regr_weights = lambda: pipeline.named_steps['linear_regr'].coef_
    pipeline.get_linear_regr_bias = lambda: pipeline.named_steps['linear_regr'].intercept_

    return pipeline
```


```python
alpha_grid = np.logspace(-9, 6, 31)
degree = 3
err_tr_list = []
err_va_list = []
pipeline_list = []
for alpha in [0.0001]: # TODO loop over all alpha_grid
    
    # Create pipeline with specified alpha
    pipeline = make_L1regularized_pipeline(alpha=alpha, degree=degree)

    # L1 solvers are a bit more finicky than L2 solvers
    # Here's some code that will show if each optimization has converged
    with warnings.catch_warnings(record=True) as warn_list:
        pipeline.fit(x_tr_MF, y_tr_M)
        solver = pipeline.named_steps['linear_regr']
        print("alpha % .3e | %s | completed %5d iters | gap % 10.3f" % (
            alpha, 
            'converged    ' if len(warn_list) == 0 else 'NOT converged',
            solver.n_iter_, solver.dual_gap_))

    # TODO evaluation code here
```

    alpha  1.000e-04 | NOT converged | completed 50000 iters | gap    548.999



```python

```

### 2b(i): Plot MSE vs. $\alpha$ for L1 ("Lasso")


```python
# TODO make plot
```

### 2b(ii): Plot number of non-zero coefficients


```python
# TODO make plot
```

### 2b(iii): Print out non-zero coefficients of learned L1-regularized models


```python
# TODO write code
```

## Problem 3

### 3a: Adding fake observations to a no-bias linear regression


**Question:** 
You are given a training dataset for regression $\{x_n, y_n \}_{n=1}^N$. Each feature vector $x_n$ has size $F=4$. Each $y_n$ is a scalar.

You decide to add in the following additional "fake" observations $\tilde{x}$:

\begin{align}
\begin{array}{c c c c}
    ~\sqrt{\lambda}  & 0  & 0 & 0
 \\ 0  & \sqrt{\lambda}  & 0  & 0
 \\ 0  &0 & \sqrt{\lambda} &0
 \\ 0  & 0  & 0  &\sqrt{\lambda}
\end{array}
\end{align}

You also add in 4 "fake" responses $\tilde{y}$, all equal to zero.

You then wish to fit a least squares linear regression model with no bias/intercept term to the combined labeled dataset that is the union of $x$ and $\tilde{x}$, $y$ and $\tilde{y}$.

### **3a(i):** Write down the optimization objective and simplify as much as possible.

### **3a(ii):** Does this "add fake data" process lead to an objectie that looks familiar?



**Solution**:

TODO write using Markdown / LaTeX syntax.

Or paste in an image of a handwritten solution.


```python

```


```python

```

e89262cfe19430a6214802b241aa3d1c0f2d68b7
ipynb
Jupyter Notebook
hw2/HW2_Starter.ipynb
tufts-ml-courses/comp135-19s-assignments
d54f4356e022150d85cfa58ebbf8ccdf66e0f1a9
MIT
2019-02-23T00:28:06.000Z
2020-01-28T20:45:57.000Z
hw2/HW2_Starter.ipynb
tufts-ml-courses/comp135-19s-assignments
d54f4356e022150d85cfa58ebbf8ccdf66e0f1a9
MIT
hw2/HW2_Starter.ipynb
tufts-ml-courses/comp135-19s-assignments
d54f4356e022150d85cfa58ebbf8ccdf66e0f1a9
MIT
2019-01-24T20:45:04.000Z
2022-03-21T20:27:11.000Z
Qwen/Qwen-72B
1. YES
2. YES

